builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 6 Packages Projects Releases Wiki Activity

Re-adjudicate: nothing to demote, and close the hole that would have mattered

Browse Source 
There is nothing to re-adjudicate. The proven set is empty and always has
been -- verified three ways: proven-status reports EMPTY, validatedSkills()
returns {} for every archetype, and zero conditioning entries have ever
reached PROVEN. The one PROVEN feature is recent_frequency_prior, which is the
incumbent counter itself, proven by the S78 ablation as ~100% of the
champion's resolution. It is the baseline every challenger is measured
against, not a conditioning interaction, and demoting it would leave the model
with nothing to grade from.

A correction to the premise: the cumulative gate did NOT catch a false
positive last session. It caught nothing, because there was nothing in the
proven set to catch. What it did was tighten alpha from 0.0026 to 0.0013
within one session, which demonstrated the mechanism working rather than a
demotion. So steps 3 and 4 -- demote, recalibrate -- are vacuous here, and
readjudicateAll says so plainly rather than glossing a no-op.

But the worry behind the order was well founded, and the audit found the real
exposure: promote() did not require the cumulative denominator. It checked n,
lift and CI, and nothing stopped a future session from testing eight
hypotheses, correcting by eight, and promoting on a p-value that would not
survive the programme's real denominator. That is precisely the hole that
makes a retroactive re-adjudication pass necessary later, so it is closed at
promotion time instead. isSufficient now refuses evidence carrying no
correction, evidence corrected against fewer tests than the cumulative count,
and any p-value that does not clear 0.05 over its own test count. The same
rule guards a PROVEN conditioning entry.

The second audit found two of four analysis scripts still correcting
per-session; pitcher-prove-k and tb-solo-and-interactions now use the
cumulative ledger, so the correction is native on every path.

reAblation.js is the standing second line: pure and injectable, so the
decision rule cannot drift from the gate's, and every verdict records both
p-values and both test counts so a demotion is re-derivable by anyone. A
feature promoted at alpha 0.05/20 can demote on the same p-value once the bar
is 0.05/60 -- correct, because the bar rose only after the programme had more
chances to get lucky. No fresh measurement is PENDING_RETEST and never a
demotion: absence of a re-test is not evidence, and demoting on it would
punish whichever stat happens to be off-season.

Net effect on the proven set is zero. No demotions, no recalibrations, and no
public ledger event -- announcing "recalibrated after re-adjudication" when
nothing changed would itself be a false signal of rigour.

4,238 tests green (337 suites); web build exit 0; counter byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-04 15:13:39 -04:00
parent ece2b9f5f9
commit ff037e40c2
9 changed files with 473 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/model/featureRegistry.js
+48 -9
View File
@@ -210,11 +210,27 @@ function isLive(sport, key) {
return statusOf(sport, key) === STATUS.PROVEN;
}
const BASE_ALPHA = 0.05;
/**
* Evidence sufficient to promote: a real sample, positive lift, and a CI that
* excludes zero on the good side. Anything less is a story about a number.
* Evidence sufficient to promote: a real sample, positive lift, a CI that
* excludes zero on the good side, AND a p-value that clears the CUMULATIVELY
* corrected alpha.
*
* THE CUMULATIVE REQUIREMENT IS STRUCTURAL, NOT ADVISORY. Before it, the
* correction existed but nothing forced a promotion to use it: a session could
* test eight hypotheses, correct by eight, and promote on a p-value that would
* not survive the programme's real denominator. That is precisely the hole that
* makes a retroactive "re-adjudicate everything" pass necessary later — so it is
* closed here rather than audited for afterwards.
*
* `opts.cumulativeTests` is the programme-lifetime count (from testLedger). When
* supplied, evidence corrected against FEWER tests than that is refused: you may
* not promote on a laxer bar than the programme has earned. It is injectable so
* the unit suite never needs a database, and omitted only when no cumulative
* count is available at all.
*/
function isSufficient(evidence) {
function isSufficient(evidence, opts = {}) {
if (!evidence || typeof evidence !== 'object') return false;
const n = Number(evidence.n);
const lift = Number(evidence.lift);
@@ -224,17 +240,40 @@ function isSufficient(evidence) {
if (!Array.isArray(ci) || ci.length !== 2) return false;
const [lo, hi] = ci.map(Number);
if (!Number.isFinite(lo) || !Number.isFinite(hi)) return false;
return lo > 0; // the whole interval above zero — improvement, not a coin flip
if (!(lo > 0)) return false; // the whole interval above zero
// The correction must be present, and must be the cumulative one.
const tests = Number(evidence.bonferroni_tests);
if (!Number.isFinite(tests) || tests < 1) return false;
const cum = Number(opts.cumulativeTests);
if (Number.isFinite(cum) && tests < cum) return false;
// If a p-value is supplied it must clear the corrected alpha. (It is optional
// only because some evidence is a bootstrap CI with no p — the CI check above
// already carries that case.)
const p = Number(evidence.p_value);
if (Number.isFinite(p) && !(p < BASE_ALPHA / tests)) return false;
return true;
}
/**
* Promote a CANDIDATE to PROVEN. Refuses without sufficient evidence, and there
* is no override parameter on purpose.
*/
function promote(sport, key, evidence, at = null) {
function promote(sport, key, evidence, at = null, opts = {}) {
const f = byKey.get(idOf(sport, key));
if (!f) return { ok: false, reason: 'unknown_feature' };
if (!isSufficient(evidence)) return { ok: false, reason: 'insufficient_evidence', required: { min_n: MIN_PROMOTION_N, lift: '>0', ci95_low: '>0' } };
if (!isSufficient(evidence, opts)) {
return {
ok: false,
reason: 'insufficient_evidence',
required: {
min_n: MIN_PROMOTION_N, lift: '>0', ci95_low: '>0',
bonferroni_tests: 'required, and >= the cumulative programme count',
p_value: 'if present, must be < 0.05 / bonferroni_tests',
},
};
}
f.history.push({ from: f.status, to: STATUS.PROVEN, evidence, at });
f.status = STATUS.PROVEN;
f.evidence = evidence;
@@ -302,12 +341,12 @@ const conditioning = [];
* interaction validates — an untagged proven entry cannot contribute to a
* coherent profile, so it is refused.
*/
function recordConditioning({ sport, archetype, stat, interaction, skill, status, lift = null, evidence = null }) {
function recordConditioning({ sport, archetype, stat, interaction, skill, status, lift = null, evidence = null }, opts = {}) {
if (!Object.keys(SKILLS).includes(String(skill || '').toUpperCase())) {
return { ok: false, reason: 'untagged_or_unknown_skill', known: Object.keys(SKILLS) };
}
if (!Object.values(STATUS).includes(status)) return { ok: false, reason: 'bad_status' };
if (status === STATUS.PROVEN && !isSufficient(evidence)) {
if (status === STATUS.PROVEN && !isSufficient(evidence, opts)) {
return { ok: false, reason: 'insufficient_evidence_for_proven' };
}
const row = {
@@ -349,7 +388,7 @@ function __reset() {
}
module.exports = {
STATUS, MIN_PROMOTION_N,
STATUS, MIN_PROMOTION_N, BASE_ALPHA,
allFeatures, liveFeatures, candidateFeatures, statusOf, isLive,
recordStatVerdict, statusForStat, liveFeaturesForStat, candidateFeaturesForStat,
SKILLS, recordConditioning, conditioningFor, validatedSkills,
src/services/model/reAblation.js
+108
View File
@@ -0,0 +1,108 @@
'use strict';
/**
* reAblation — THE SECOND LINE OF DEFENCE.
*
* The gate decides whether a feature earns its place. This decides whether it
* KEEPS it. Those are different questions and only the first has ever been asked
* here.
*
* WHY A PROVEN FEATURE CAN STOP BEING TRUE. Three ways, all of them real and
* none of them a bug:
*
* 1. IT WAS NEVER TRUE. It cleared the bar on a lucky draw. More data is the
* only thing that reveals this, and the cumulative correction makes it
* rarer without making it impossible.
* 2. THE GAME CHANGED. Baseball is not stationary — a league-wide shift in how
* pitchers are used, or a rule change, can retire a real effect.
* 3. THE BAR ROSE. The cumulative denominator only grows, so a feature proved
* at alpha 0.05/20 is being held to 0.05/60 a year later. A feature that
* cleared the old bar and not the new one is not being punished unfairly —
* it is being held to what the programme has since earned the right to ask.
*
* A ledger that tightens its own standard and demotes its own features is more
* credible than one that only ever adds. So the demotion is recorded with BOTH
* p-values and the test count each was corrected against — anyone can see
* exactly why, and re-derive it.
*
* PURE AND INJECTABLE: it takes evidence in and returns verdicts. It performs no
* measurement itself and reaches no database, so the decision rule is testable
* without a network and cannot quietly drift from the rule the gate uses.
*/
const { BASE_ALPHA } = require('./featureRegistry');
/**
* Re-adjudicate ONE proven entry against the current cumulative denominator.
*
* @param {object} entry what was promoted, and on what evidence
* @param {object} current the fresh measurement (may be absent)
* @param {number} cumulativeTests the programme-lifetime distinct test count
* @returns {object} an auditable verdict — never a bare boolean
*/
function readjudicate(entry, current, cumulativeTests) {
const tests = Number(cumulativeTests);
const correctedAlpha = Number.isFinite(tests) && tests >= 1 ? BASE_ALPHA / tests : BASE_ALPHA;
const originalTests = Number(entry && entry.evidence && entry.evidence.bonferroni_tests);
const originalAlpha = Number.isFinite(originalTests) && originalTests >= 1
? BASE_ALPHA / originalTests : null;
const originalP = Number(entry && entry.evidence && entry.evidence.p_value);
const base = {
key: entry && entry.key,
archetype: entry && entry.archetype,
stat: entry && entry.stat,
original_p_value: Number.isFinite(originalP) ? originalP : null,
original_bonferroni_tests: Number.isFinite(originalTests) ? originalTests : null,
original_corrected_alpha: originalAlpha,
cumulative_bonferroni_tests: Number.isFinite(tests) ? tests : null,
cumulative_corrected_alpha: correctedAlpha,
};
// NO FRESH MEASUREMENT — deliberately NOT a demotion. Absence of a re-test is
// not evidence a feature stopped working, and demoting on it would punish
// whichever stat happens to be off-season.
if (!current || !Number.isFinite(Number(current.p_value))) {
return { ...base, verdict: 'PENDING_RETEST', reason: 'no fresh measurement available' };
}
const p = Number(current.p_value);
const n = Number(current.n);
const survives = p < correctedAlpha;
return {
...base,
current_p_value: p,
current_n: Number.isFinite(n) ? n : null,
verdict: survives ? 'SURVIVES' : 'DEMOTE',
reason: survives
? `p ${p} < cumulative alpha ${correctedAlpha}`
: `p ${p} no longer clears the cumulative alpha ${correctedAlpha}`
+ (originalAlpha !== null ? ` (it cleared ${originalAlpha} when promoted)` : ''),
};
}
/**
* Re-adjudicate a whole registry. Returns the verdicts plus a summary that
* states plainly what happened, including the case that matters most right now:
* an empty proven set has nothing to re-adjudicate, and saying so is the honest
* result rather than a no-op to be glossed.
*/
function readjudicateAll(provenEntries, measurements, cumulativeTests) {
const entries = provenEntries || [];
const verdicts = entries.map((e) => readjudicate(e, (measurements || {})[e.key], cumulativeTests));
return {
cumulative_bonferroni_tests: cumulativeTests,
cumulative_corrected_alpha: Number.isFinite(Number(cumulativeTests)) && cumulativeTests >= 1
? BASE_ALPHA / cumulativeTests : BASE_ALPHA,
proven_entries_examined: entries.length,
survived: verdicts.filter((v) => v.verdict === 'SURVIVES').length,
demoted: verdicts.filter((v) => v.verdict === 'DEMOTE').length,
pending_retest: verdicts.filter((v) => v.verdict === 'PENDING_RETEST').length,
verdicts,
summary: entries.length === 0
? 'NOTHING TO RE-ADJUDICATE — the proven set is empty.'
: `${verdicts.filter((v) => v.verdict === 'DEMOTE').length} demoted of ${entries.length}.`,
};
}
module.exports = { readjudicate, readjudicateAll };