Files
vyndr/specs/window-bug-class-audit.md
T
builtbykev 494c83cf76 Hunt the window-bug class: three more paths, and the forward re-audit rule
in code

PHASE 0 — getStatRows is the single base-rate path, so every branch is
audited, plus the feature builders since l20_avg is the season reference
projectionFor reads:

  getStatRows MLB -> estimator base    fullLog            CORRECT (929fd81)
  mlbGameLogFeatures l5/l10/l20        last10 = 10        DEFECTIVE
  espnStatsAdapter.parseGameLog        slice(0,20)        DEFECTIVE
  getStatRows NBA/WNBA ESPN branch     inherits 20-cap    DEFECTIVE via source
  getStatRows NBA/WNBA python branch   getGameLogs(...,20) dormant (offline)
  pitcherEngine / skillProjection      statcast profiles  N/A
  pitcher props via getStatRows MLB    fullLog            CORRECT
  settleSource                         full log (S64)     CORRECT

THE PITCHER ANSWER IS GOOD NEWS: pitcher props run through the same
getStatRows MLB branch, so 929fd81 repaired them too. There is no separate
defective pitcher base-rate path.

THE ONE HIDING IN PLAIN SIGHT: mlbGameLogFeatures carries the comment
"l20 = all available (the season per-game reference projectionFor needs)"
while building from last10 -- so l20_avg was a TEN-GAME AVERAGE WEARING A
SEASON LABEL, feeding both the consistency pull inside the estimator and
projectionFor, which decides refusals. It survived the previous repair
because that fix touched only getStatRows.

PHASE 1 — mlbGameLogFeatures now reads fullLog; espnStatsAdapter drops its
slice(0,20) cap. ZERO new API calls on both: each widens data already
fetched and then discarded, the same shape as the original repair. The
python branch is left alone -- the service is offline in prod and fixing it
would be speculative.

Their before/after resolution is NOT measured, deliberately: the only way
to measure today is to reconstruct the repaired forecast over old rows,
which is the reconstruction-vs-served trap this order refuses. Code fix
now, measurement at accrual.

PHASE 2 — MODEL_VERSION bumped to engine1@2026-08-07-fullwindow, so every
forward snapshot is self-identifying (retentionService already stamps it;
no new plumbing). model/reAuditEligibility.js encodes the rule: isEligible
accepts only the repaired marker, assess counts eligible DATES not rows,
and ACCRUAL is frozen at calibration 10 / hits-lift 10 / verdict-reaudit
14 / rbi-gate 14. A test locks the invisible case -- a MIXED table of 330
rows with 30 repaired returns eligible_dates 3, not 330 rows of false
confidence. Once both generations share a table a naive count would fit a
map on a blend of two forecasters.

PHASE 3 — the board, each consequence labelled: calibration WITHDRAWN
(refits at 10 dates, never on reconstructions); factor verdicts SUSPECT
(all measured against a champion worse than a frequency table, direction
UNKNOWN, not pre-priced, 14 dates); hits factor lift UN-REMEASURABLE (10
dates, factors still wired and transmitting); rbi lineup-slot RE-QUEUED
(14 dates). Pre-registered order: calibration, hits lift, verdict
re-audit, rbi gate.

Then STOP and accrue. Nothing further can be honestly measured until the
board fills with rows the repaired champion produced.

Serving-path changes by design for the MLB feature path and NBA/WNBA logs;
eleven frozen model modules verified unchanged. p_win never mutated. No
Bonferroni slot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-07 03:40:20 -04:00

6.0 KiB

The window-bug class, hunted — three more paths, and the forward re-audit rule

PHASE 0 — the audit

The defect class: a fixed short window used AS the season/base rate. getStatRows is the single path feeding meta.gameLogs, which is where estimateProbability derives its base rate, so every branch of it is a base-rate path. The feature builders are the second surface, because l20_avg is the season reference projectionFor reads.

path window classification
getStatRows MLB → estimator base rate fullLog CORRECT (fixed 929fd81)
mlbGameLogFeaturesl5/l10/l20_avg, l10_stddev last10 = 10 DEFECTIVE
espnStatsAdapter.parseGameLog → NBA/WNBA logs rows.slice(0, 20) = 20 DEFECTIVE
getStatRows NBA/WNBA ESPN branch inherits the 20-cap DEFECTIVE (via source)
getStatRows NBA/WNBA Python branch getGameLogs(..., 20) DEFECTIVE-but-dormant (service offline in prod)
pitcher engine (pitcherEngine, skillProjection) reads statcast profiles, no game log N/A
pitcher props (strikeouts) via getStatRows MLB fullLog CORRECT — fixed by the same change
settleSource already reads the full log (S64) CORRECT
playerIntelService, streaksService display/streak surfaces, not forecasts N/A

The pitcher answer matters and is good news: pitcher props run through the same getStatRows MLB branch, so 929fd81 repaired them too — there is no separate defective pitcher base-rate path.

The one that was hiding in plain sight

mlbGameLogFeatures carries this comment:

l20 = all available (the season per-game reference projectionFor needs)

Built from last10, l20_avg was a ten-game average wearing a season label — and it feeds both the consistency (cv) pull inside the estimator and projectionFor, which decides refusals. Same class as the base-rate bug, same file, and it survived the previous repair because that fix touched only getStatRows.


PHASE 1 — fixes

path fix API cost
mlbGameLogFeatures read fullLog, fall back to last10 ZERO — same response
espnStatsAdapter.parseGameLog drop the slice(0, 20) cap ZERO — same payload, already parsed
NBA/WNBA Python branch left as-is service offline in prod; fixing it would be speculative

No new API calls anywhere. Both fixes widen data that was already fetched and then discarded — the same shape as the original repair.

Measurement status, stated honestly

These are serving changes for the MLB feature path and the NBA/WNBA log path. Their before/after resolution is NOT measured here, and deliberately: the only way to measure it today would be to reconstruct the repaired forecast over old rows, which is the reconstruction-vs-served trap this order explicitly refuses. They ship as code fixes with the measurement deferred to accrual, which is the honest sequencing.


PHASE 2 — the forward re-audit rule, in code

MODEL_VERSION is bumped to engine1@2026-08-07-fullwindow, so every snapshot from this commit forward is self-identifying. retentionService already stamps it onto model_snapshots, so no new plumbing was needed.

model/reAuditEligibility.js encodes the rule:

  • isEligible(row) — true only for rows carrying the repaired marker.
  • assess(rows) — counts eligible DATES, not rows, because dates have been the binding scarcity in every interval this session.
  • ACCRUAL (frozen) — pre-stated minimum dates per measurement:
measurement minimum eligible dates
calibration re-fit 10
hits factor lift 10
prior verdict re-audit 14
rbi lineup-slot gate 14

A test locks the case that would otherwise be invisible: a MIXED table of 330 rows where only 30 carry the new marker returns eligible_dates: 3, not 330 rows' worth of false confidence. Once both generations sit in the same table, a naive count would happily fit a map on a blend of two different forecasters.


PHASE 3 — the honest board

What happened: the champion computed its season rate over ten games. Found by resolution decomposition, not by a test failing. Fixed in two lines. It no longer loses to a frequency table — it beats it CI-confirmed only on total_bases, ties on rbi and runs, and leads on the hits point estimate.

Consequences, each labelled:

  • CALIBRATION — WITHDRAWN. CALIBRATION_DEPLOYED is empty. Maps were fitted on the retired forecast. Re-fits on repaired-champion settled rows. Waiting on accrual: 10 dates. Not to be refit on reconstructions.
  • FACTOR VERDICTS — SUSPECT. Every prior null and every THEATER was measured against a champion worse than a frequency table; signal added to noise reads as noise. Re-audit on accrued rows. Direction UNKNOWN — some may pass, some may still fail. Not pre-priced. Waiting: 14 dates.
  • HITS FACTOR LIFT (1.39%) — UN-REMEASURABLE. Needs rows produced by the repaired champion. Waiting: 10 dates. The factors remain wired and transmitting (43f65d3); only the lift number is unquantified.
  • RBI LINEUP-SLOT — RE-QUEUED. Lands after the champion is sound and rows accrue. Waiting: 14 dates.

Pre-registered re-audit order (each runs only when its bar is met):

  1. Re-fit calibration on repaired-champion rows (10 dates)
  2. Re-measure hits factor lift (10 dates)
  3. Re-audit prior factor verdicts (14 dates)
  4. Run rbi lineup-slot through the two-part gate (14 dates)

Then STOP and accrue. Nothing further can be honestly measured until the board fills with rows the repaired champion produced.


Invariants

No measurement on reconstructions — hard refusal, and it is why Phase 1 ships code without numbers. Serving-path changes by design for the MLB feature path and NBA/WNBA logs; frozen model modules verified unchanged. p_win never mutated. No Bonferroni slot — base-rate repair and a bug hunt, not causal factors.