vyndr/specs/challenger-scoreboard.md

CHALLENGER SCOREBOARD — 2026-08-03

Nothing was promoted. Nothing earned it yet. Not because the bar was held too high, but because no challenger's confidence interval excludes zero on the good side. The champion is byte-identical; every challenger stays wired.

The headline of this session is not the scoreboard. It is that the scoreboard was unmeasurable until a two-day-old settlement outage was found and fixed — see §1. Settled sample went 493 → 1,741 the moment it was repaired.

---

1. Why "n-blocked" was the wrong diagnosis

The order said: don't repeat "n-blocked" without counting. Counting is what found the real problem.

Three of the four axes read exactly zero settled rows — not low, zero: environment 1,496 rows / 0 settled, opportunity 922 / 0, matchup 603 / 0. Rows whose games had been played days earlier and never settled, with settle_attempts = 0 — never even attempted.

Root cause: settleLedger fetched open ids, then refetched full rows via .in('id', ids). PostgREST puts filters in the URL, so 500 UUIDs became an 18,499-character request that the fetch layer rejects with TypeError: fetch failed. The result was destructured as const { data: rows } = ... with no error binding, so rows came back null, the loop never ran, and the function returned {settled:0, voided:0, unrecoverable:0, pending:0} — byte-identical to a healthy "nothing to settle."

It hid for two days because it is volume-triggered: daily volume ran 20–260 rows and settled perfectly for weeks. 2026-08-01 was the first day past the 500-row fetch limit and settlement died that night. Worse, the zero-settle ops alarm reads these same return values, so pending: 0 told the watchdog the backlog was empty — the alarm built to catch exactly this could not see it.

Fixed, deployed, and drained: 1,444 rows from 2026-08-01 settled (1,376 hit/miss + 68 void, 0 remaining). captureClosing carried the same shape one level down and is now chunked at 100 ids.

2. THE SCOREBOARD

Bar: the challenger's own rows only, direction-aligned, paired bootstrap (4,000 resamples, deterministic seed) on the difference in resolution, because both models score the same rows and independent standard errors would overstate certainty. PROMOTE requires the CI to exclude zero on the good side. Same bar that refuted hits-v1 — no lighter test for a would-be winner.

challenger	settled n	resolution (chal / champ)	Δ vs champion	CI95	verdict
arch-v1 (market-relative nudge)	1,741	0.4599 / 0.4599	0.0000	[−0.0050, +0.0054]	STAY WIRED (inconclusive)
arch-v1 · rows it MOVED only	1,325	0.4886 / 0.4886	−0.0001	[−0.0064, +0.0061]	STAY WIRED (inconclusive)
contact-v1 (season contact quality)	1,055	0.4070 / 0.4061	+0.0008	[−0.0052, +0.0069]	STAY WIRED (inconclusive)
contact-v1 · rows it MOVED only	511	0.3966 / 0.3948	+0.0018	[−0.0107, +0.0144]	STAY WIRED (inconclusive)
proj-v1.1 ladder (all stats)	1,664	0.4339 / 0.4640	−0.0301	[−0.0543, −0.0060]	STAY WIRED (measured WORSE)
arch-v1 · environment axis rows	871	0.5651 / 0.5679	−0.0028	[−0.0089, +0.0036]	STAY WIRED (inconclusive)
arch-v1 · opportunity axis rows	539	0.5311 / 0.5310	+0.0001	[−0.0091, +0.0090]	STAY WIRED (inconclusive)
arch-v1 · matchup axis rows	0	—	—	—	STILL PENDING
tb-v1 (total_bases only)	0	—	—	—	STILL PENDING
hits-v1 (hits only)	0	—	—	—	STILL PENDING (refuted by replay, specs/hits-v1-binomial.md)

These are true prospective holdouts, not backtests. arch-v1 and contact-v1 wrote their probability at grade time, into their own columns, before the game was played. Nothing was recomputed. That is the strongest evidence available and it is why no replay was needed here.

3. What the numbers actually say

arch-v1 moves a lot and changes nothing. It moved 1,325 of 1,741 rows (76%), mean absolute move 2.5 points, max 10.9 — and resolution is identical to the champion to four decimal places, on the moved rows too. This is not "too small to detect." It is movement that carries no information about the outcome. A nudge this active with an effect this precisely zero is a finding, not a pending verdict.

The projection ladder is reliably worse than the champion. −0.0301 with a CI excluding zero, across 1,664 rows and all stats. Combined with hits-v1's refutation (specs/hits-v1-binomial.md), the projection family now has two independent measurements pointing the same way: it is not the champion's equal on any stat measured so far. That is an argument for diagnosing its inputs, not for shipping another variant of it.

Three are genuinely pending, for a legitimate reason now. matchup, tb-v1 and hits-v1 all have rows written only on 2026-08-02/03, which settle after ET midnight. matchup has ~496 rows queued, tb-v1 65, hits-v1 pending its first snapshot write. They will read within a day or two — and now that settlement works, they actually will.

4. Provenance

All 1,741 arch-v1 rows carry a single model_version (engine1@2026-07-20). The older pre-retention-unknown rows (320 settled) carry no challenger values at all, so they cannot influence any verdict. No verdict here depends on mixed-provenance rows — the split was checked, not assumed.

Contamination excluded throughout: quarantine_reason LIKE 'nontakeable_book%'.

5. Promotion mechanics — specified, deliberately unused

No flip was performed because nothing qualified. When one does, the shape is: challenger-first (write the promoted value into the served path while the champion column keeps recording), version-tagged, atomic, with the previous version one env flag away. Recorded here so a future promotion is a decision, not an improvisation.

6. Reproduce

SUPABASE_URL=... node scripts/challenger-scoreboard.js — prints the full board, the moved-rows-only slice, the per-axis slice and the provenance split.