Files
vyndr/specs/window-bug-class-audit.md
T
builtbykev 494c83cf76 Hunt the window-bug class: three more paths, and the forward re-audit rule
in code

PHASE 0 — getStatRows is the single base-rate path, so every branch is
audited, plus the feature builders since l20_avg is the season reference
projectionFor reads:

  getStatRows MLB -> estimator base    fullLog            CORRECT (929fd81)
  mlbGameLogFeatures l5/l10/l20        last10 = 10        DEFECTIVE
  espnStatsAdapter.parseGameLog        slice(0,20)        DEFECTIVE
  getStatRows NBA/WNBA ESPN branch     inherits 20-cap    DEFECTIVE via source
  getStatRows NBA/WNBA python branch   getGameLogs(...,20) dormant (offline)
  pitcherEngine / skillProjection      statcast profiles  N/A
  pitcher props via getStatRows MLB    fullLog            CORRECT
  settleSource                         full log (S64)     CORRECT

THE PITCHER ANSWER IS GOOD NEWS: pitcher props run through the same
getStatRows MLB branch, so 929fd81 repaired them too. There is no separate
defective pitcher base-rate path.

THE ONE HIDING IN PLAIN SIGHT: mlbGameLogFeatures carries the comment
"l20 = all available (the season per-game reference projectionFor needs)"
while building from last10 -- so l20_avg was a TEN-GAME AVERAGE WEARING A
SEASON LABEL, feeding both the consistency pull inside the estimator and
projectionFor, which decides refusals. It survived the previous repair
because that fix touched only getStatRows.

PHASE 1 — mlbGameLogFeatures now reads fullLog; espnStatsAdapter drops its
slice(0,20) cap. ZERO new API calls on both: each widens data already
fetched and then discarded, the same shape as the original repair. The
python branch is left alone -- the service is offline in prod and fixing it
would be speculative.

Their before/after resolution is NOT measured, deliberately: the only way
to measure today is to reconstruct the repaired forecast over old rows,
which is the reconstruction-vs-served trap this order refuses. Code fix
now, measurement at accrual.

PHASE 2 — MODEL_VERSION bumped to engine1@2026-08-07-fullwindow, so every
forward snapshot is self-identifying (retentionService already stamps it;
no new plumbing). model/reAuditEligibility.js encodes the rule: isEligible
accepts only the repaired marker, assess counts eligible DATES not rows,
and ACCRUAL is frozen at calibration 10 / hits-lift 10 / verdict-reaudit
14 / rbi-gate 14. A test locks the invisible case -- a MIXED table of 330
rows with 30 repaired returns eligible_dates 3, not 330 rows of false
confidence. Once both generations share a table a naive count would fit a
map on a blend of two forecasters.

PHASE 3 — the board, each consequence labelled: calibration WITHDRAWN
(refits at 10 dates, never on reconstructions); factor verdicts SUSPECT
(all measured against a champion worse than a frequency table, direction
UNKNOWN, not pre-priced, 14 dates); hits factor lift UN-REMEASURABLE (10
dates, factors still wired and transmitting); rbi lineup-slot RE-QUEUED
(14 dates). Pre-registered order: calibration, hits lift, verdict
re-audit, rbi gate.

Then STOP and accrue. Nothing further can be honestly measured until the
board fills with rows the repaired champion produced.

Serving-path changes by design for the MLB feature path and NBA/WNBA logs;
eleven frozen model modules verified unchanged. p_win never mutated. No
Bonferroni slot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-07 03:40:20 -04:00

129 lines
6.0 KiB
Markdown

# The window-bug class, hunted — three more paths, and the forward re-audit rule
## PHASE 0 — the audit
The defect class: **a fixed short window used AS the season/base rate.**
`getStatRows` is the single path feeding `meta.gameLogs`, which is where
`estimateProbability` derives its base rate, so every branch of it is a base-rate
path. The feature builders are the second surface, because `l20_avg` is the
season reference `projectionFor` reads.
| path | window | classification |
|---|---|---|
| `getStatRows` MLB → estimator base rate | `fullLog` | **CORRECT** (fixed 929fd81) |
| `mlbGameLogFeatures``l5/l10/l20_avg`, `l10_stddev` | `last10` = **10** | **DEFECTIVE** |
| `espnStatsAdapter.parseGameLog` → NBA/WNBA logs | `rows.slice(0, 20)` = **20** | **DEFECTIVE** |
| `getStatRows` NBA/WNBA ESPN branch | inherits the 20-cap | **DEFECTIVE (via source)** |
| `getStatRows` NBA/WNBA Python branch | `getGameLogs(..., 20)` | DEFECTIVE-but-dormant (service offline in prod) |
| **pitcher engine** (`pitcherEngine`, `skillProjection`) | reads statcast **profiles**, no game log | **N/A** |
| pitcher props (strikeouts) via `getStatRows` MLB | `fullLog` | **CORRECT** — fixed by the same change |
| `settleSource` | already reads the full log (S64) | CORRECT |
| `playerIntelService`, `streaksService` | display/streak surfaces, not forecasts | N/A |
**The pitcher answer matters and is good news:** pitcher props run through the
same `getStatRows` MLB branch, so `929fd81` repaired them too — there is no
separate defective pitcher base-rate path.
### The one that was hiding in plain sight
`mlbGameLogFeatures` carries this comment:
> `l20 = all available (the season per-game reference projectionFor needs)`
Built from `last10`, **`l20_avg` was a ten-game average wearing a season label** —
and it feeds both the consistency (cv) pull inside the estimator and
`projectionFor`, which decides refusals. Same class as the base-rate bug, same
file, and it survived the previous repair because that fix touched only
`getStatRows`.
---
## PHASE 1 — fixes
| path | fix | API cost |
|---|---|---|
| `mlbGameLogFeatures` | read `fullLog`, fall back to `last10` | **ZERO** — same response |
| `espnStatsAdapter.parseGameLog` | drop the `slice(0, 20)` cap | **ZERO** — same payload, already parsed |
| NBA/WNBA Python branch | left as-is | service offline in prod; fixing it would be speculative |
**No new API calls anywhere.** Both fixes widen data that was already fetched and
then discarded — the same shape as the original repair.
### Measurement status, stated honestly
These are serving changes for the MLB feature path and the NBA/WNBA log path.
**Their before/after resolution is NOT measured here**, and deliberately: the
only way to measure it today would be to reconstruct the repaired forecast over
old rows, which is the reconstruction-vs-served trap this order explicitly
refuses. They ship as code fixes with the measurement deferred to accrual, which
is the honest sequencing.
---
## PHASE 2 — the forward re-audit rule, in code
`MODEL_VERSION` is bumped to **`engine1@2026-08-07-fullwindow`**, so every
snapshot from this commit forward is self-identifying. `retentionService` already
stamps it onto `model_snapshots`, so no new plumbing was needed.
`model/reAuditEligibility.js` encodes the rule:
- **`isEligible(row)`** — true only for rows carrying the repaired marker.
- **`assess(rows)`** — counts eligible **DATES**, not rows, because dates have
been the binding scarcity in every interval this session.
- **`ACCRUAL`** (frozen) — pre-stated minimum dates per measurement:
| measurement | minimum eligible dates |
|---|---|
| calibration re-fit | 10 |
| hits factor lift | 10 |
| prior verdict re-audit | 14 |
| rbi lineup-slot gate | 14 |
A test locks the case that would otherwise be invisible: **a MIXED table** of 330
rows where only 30 carry the new marker returns `eligible_dates: 3`, not 330
rows' worth of false confidence. Once both generations sit in the same table, a
naive count would happily fit a map on a blend of two different forecasters.
---
## PHASE 3 — the honest board
**What happened:** the champion computed its season rate over ten games. Found by
resolution decomposition, not by a test failing. Fixed in two lines. It no longer
*loses* to a frequency table — it **beats** it CI-confirmed only on total_bases,
**ties** on rbi and runs, and leads on the hits point estimate.
**Consequences, each labelled:**
- **CALIBRATION — WITHDRAWN.** `CALIBRATION_DEPLOYED` is empty. Maps were fitted
on the retired forecast. Re-fits on repaired-champion settled rows. *Waiting on
accrual: 10 dates.* Not to be refit on reconstructions.
- **FACTOR VERDICTS — SUSPECT.** Every prior null and every THEATER was measured
against a champion worse than a frequency table; signal added to noise reads as
noise. Re-audit on accrued rows. **Direction UNKNOWN** — some may pass, some
may still fail. Not pre-priced. *Waiting: 14 dates.*
- **HITS FACTOR LIFT (1.39%) — UN-REMEASURABLE.** Needs rows produced *by* the
repaired champion. *Waiting: 10 dates.* The factors remain wired and
transmitting (43f65d3); only the lift number is unquantified.
- **RBI LINEUP-SLOT — RE-QUEUED.** Lands after the champion is sound and rows
accrue. *Waiting: 14 dates.*
**Pre-registered re-audit order** (each runs only when its bar is met):
1. Re-fit calibration on repaired-champion rows (10 dates)
2. Re-measure hits factor lift (10 dates)
3. Re-audit prior factor verdicts (14 dates)
4. Run rbi lineup-slot through the two-part gate (14 dates)
**Then STOP and accrue.** Nothing further can be honestly measured until the
board fills with rows the repaired champion produced.
---
## Invariants
No measurement on reconstructions — hard refusal, and it is why Phase 1 ships
code without numbers. Serving-path changes by design for the MLB feature path and
NBA/WNBA logs; frozen model modules verified unchanged. `p_win` never mutated. No
Bonferroni slot — base-rate repair and a bug hunt, not causal factors.