Files
vyndr/specs/re-adjudication.md
T
builtbykev ff037e40c2 Re-adjudicate: nothing to demote, and close the hole that would have mattered
There is nothing to re-adjudicate. The proven set is empty and always has
been -- verified three ways: proven-status reports EMPTY, validatedSkills()
returns {} for every archetype, and zero conditioning entries have ever
reached PROVEN. The one PROVEN feature is recent_frequency_prior, which is the
incumbent counter itself, proven by the S78 ablation as ~100% of the
champion's resolution. It is the baseline every challenger is measured
against, not a conditioning interaction, and demoting it would leave the model
with nothing to grade from.

A correction to the premise: the cumulative gate did NOT catch a false
positive last session. It caught nothing, because there was nothing in the
proven set to catch. What it did was tighten alpha from 0.0026 to 0.0013
within one session, which demonstrated the mechanism working rather than a
demotion. So steps 3 and 4 -- demote, recalibrate -- are vacuous here, and
readjudicateAll says so plainly rather than glossing a no-op.

But the worry behind the order was well founded, and the audit found the real
exposure: promote() did not require the cumulative denominator. It checked n,
lift and CI, and nothing stopped a future session from testing eight
hypotheses, correcting by eight, and promoting on a p-value that would not
survive the programme's real denominator. That is precisely the hole that
makes a retroactive re-adjudication pass necessary later, so it is closed at
promotion time instead. isSufficient now refuses evidence carrying no
correction, evidence corrected against fewer tests than the cumulative count,
and any p-value that does not clear 0.05 over its own test count. The same
rule guards a PROVEN conditioning entry.

The second audit found two of four analysis scripts still correcting
per-session; pitcher-prove-k and tb-solo-and-interactions now use the
cumulative ledger, so the correction is native on every path.

reAblation.js is the standing second line: pure and injectable, so the
decision rule cannot drift from the gate's, and every verdict records both
p-values and both test counts so a demotion is re-derivable by anyone. A
feature promoted at alpha 0.05/20 can demote on the same p-value once the bar
is 0.05/60 -- correct, because the bar rose only after the programme had more
chances to get lucky. No fresh measurement is PENDING_RETEST and never a
demotion: absence of a re-test is not evidence, and demoting on it would
punish whichever stat happens to be off-season.

Net effect on the proven set is zero. No demotions, no recalibrations, and no
public ledger event -- announcing "recalibrated after re-adjudication" when
nothing changed would itself be a false signal of rigour.

4,238 tests green (337 suites); web build exit 0; counter byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-04 15:13:39 -04:00

5.6 KiB
Raw Blame History

RE-ADJUDICATION UNDER CUMULATIVE CORRECTION

2026-08-04. Counter byte-identical. Nothing recalibrated, because nothing needed to be.

There is nothing to re-adjudicate: the proven set is empty, and has always been empty. Verified three ways below.

And the cumulative correction did not catch a false positive last session. It caught nothing — there was nothing in the proven set to catch. What it did was tighten α from 0.0026 to 0.0013 within one session, which demonstrated the mechanism working, not a demotion.

The real exposure was elsewhere, and it was real: a promotion could bypass the cumulative correction entirely. That hole is now closed structurally, so the retroactive pass this order describes can never become necessary.


1. The registry state, verified (STEP 1 + 2)

scripts/proven-status.js  → PROVEN_SET: EMPTY
                             hits LOSES · total_bases INCONCLUSIVE · strikeouts INCONCLUSIVE

featureRegistry           → PROVEN features: ["recent_frequency_prior"]
                             validatedSkills(BOMBER|GHOST|TORCH|FLAME|SCALPEL|SINKER) = {}
                             conditioning entries recorded: 0

The single PROVEN feature is recent_frequency_priorthe incumbent counter itself, proven by the S78 ablation showing it is ~100% of the champion's resolution. It is the baseline every challenger is measured against, not a conditioning interaction, and demoting it would leave the model with nothing to grade from. No conditioning entry has ever reached PROVEN, so there is no archetype-slot to re-run and no grade that depends on a demoted feature.

STEP 3 (demote) and STEP 4 (recalibrate) are therefore vacuous — correctly. readjudicateAll([], …) returns NOTHING TO RE-ADJUDICATE — the proven set is empty, which is the honest output rather than a no-op glossed over.

2. The hole that WAS open, and is now closed

The order's underlying worry — make sure the standard applied from the start, so no retroactive correction is ever needed — was well founded. Two audits:

(a) promote() did not require the cumulative denominator. It checked n, lift and CI, but nothing stopped a future session from testing eight hypotheses, correcting by eight, and promoting on a p-value that would not survive the programme's real denominator. That is exactly the hole that makes a "re-adjudicate everything" pass necessary later.

Now structural — isSufficient refuses evidence that:

  • carries no bonferroni_tests at all;
  • was corrected against fewer tests than the cumulative count;
  • has a p_value that does not clear 0.05 / bonferroni_tests.
no correction recorded                    → refused
per-session (8) when cumulative is 38     → refused
cumulative (38) but p = 0.01 > 0.0013     → refused
cumulative (38) and p = 0.0005            → accepted

The same rule guards recordConditioning(status: PROVEN). Locked by test.

(b) Two of four analysis scripts still corrected per-session. pitcher-prove-k.js and tb-solo-and-interactions.js used SOLO.length + INTERACTIONS.length; both now use the cumulative ledger, as cluster-prove.js already did. Cumulative correction is native across every analysis path.

3. Standing re-ablation (STEP 5, second line) — built

src/services/model/reAblation.js. Pure and injectable: it takes evidence and returns auditable verdicts, performs no measurement, and reaches no database, so the decision rule cannot quietly drift from the gate's.

Every verdict records both p-values and both test counts, so a demotion can be re-derived by anyone:

original_p_value · original_bonferroni_tests · original_corrected_alpha
current_p_value  · cumulative_bonferroni_tests · cumulative_corrected_alpha
verdict + reason

A feature promoted at α=0.05/20 and re-tested against α=0.05/60 can demote on the same p-value. That is not unfair — the bar rose because the programme has since had more chances to get lucky, and holding old results to the old bar is how a lucky draw becomes permanent.

No fresh measurement is PENDING_RETEST, never a demotion. Absence of a re-test is not evidence a feature stopped working, and demoting on it would punish whichever stat happens to be off-season.

Three ways a proven feature can stop being true are documented in the module: it was never true; the game changed; or the bar rose.

4. Net effect on the proven set

Zero. Nothing survived because nothing was there; nothing demoted; nothing recalibrated; no ledger event was emitted, because emitting "recalibrated after re-adjudication" when nothing changed would itself be a false signal of rigour.

Frozen clusters were not exempted — they were examined and contain no proven conditioning. The counter is byte-identical.

5. Going forward

  • Cumulative correction is native on every analysis path and required for every promotion.
  • Standing re-ablation is available and should run whenever a proven entry exists and fresh sample arrives.
  • The bar only rises. Current cumulative count 38, α = 0.0013. Every new hypothesis tightens it for everything after — so re-testing standing candidates is now mathematically the disciplined choice over inventing new ones.
  • The first promotion this programme ever makes will already have been held to the cumulative bar, which is the outcome this order was actually asking for.

Not recommended: treating the empty proven set as a problem to solve by lowering the bar, or emitting a public "re-adjudication" event for a pass that changed nothing.