diff --git a/specs/window-bug-class-audit.md b/specs/window-bug-class-audit.md new file mode 100644 index 0000000..04d5d59 --- /dev/null +++ b/specs/window-bug-class-audit.md @@ -0,0 +1,128 @@ +# The window-bug class, hunted — three more paths, and the forward re-audit rule + +## PHASE 0 — the audit + +The defect class: **a fixed short window used AS the season/base rate.** +`getStatRows` is the single path feeding `meta.gameLogs`, which is where +`estimateProbability` derives its base rate, so every branch of it is a base-rate +path. The feature builders are the second surface, because `l20_avg` is the +season reference `projectionFor` reads. + +| path | window | classification | +|---|---|---| +| `getStatRows` MLB → estimator base rate | `fullLog` | **CORRECT** (fixed 929fd81) | +| `mlbGameLogFeatures` → `l5/l10/l20_avg`, `l10_stddev` | `last10` = **10** | **DEFECTIVE** | +| `espnStatsAdapter.parseGameLog` → NBA/WNBA logs | `rows.slice(0, 20)` = **20** | **DEFECTIVE** | +| `getStatRows` NBA/WNBA ESPN branch | inherits the 20-cap | **DEFECTIVE (via source)** | +| `getStatRows` NBA/WNBA Python branch | `getGameLogs(..., 20)` | DEFECTIVE-but-dormant (service offline in prod) | +| **pitcher engine** (`pitcherEngine`, `skillProjection`) | reads statcast **profiles**, no game log | **N/A** | +| pitcher props (strikeouts) via `getStatRows` MLB | `fullLog` | **CORRECT** — fixed by the same change | +| `settleSource` | already reads the full log (S64) | CORRECT | +| `playerIntelService`, `streaksService` | display/streak surfaces, not forecasts | N/A | + +**The pitcher answer matters and is good news:** pitcher props run through the +same `getStatRows` MLB branch, so `929fd81` repaired them too — there is no +separate defective pitcher base-rate path. + +### The one that was hiding in plain sight + +`mlbGameLogFeatures` carries this comment: + +> `l20 = all available (the season per-game reference projectionFor needs)` + +Built from `last10`, **`l20_avg` was a ten-game average wearing a season label** — +and it feeds both the consistency (cv) pull inside the estimator and +`projectionFor`, which decides refusals. Same class as the base-rate bug, same +file, and it survived the previous repair because that fix touched only +`getStatRows`. + +--- + +## PHASE 1 — fixes + +| path | fix | API cost | +|---|---|---| +| `mlbGameLogFeatures` | read `fullLog`, fall back to `last10` | **ZERO** — same response | +| `espnStatsAdapter.parseGameLog` | drop the `slice(0, 20)` cap | **ZERO** — same payload, already parsed | +| NBA/WNBA Python branch | left as-is | service offline in prod; fixing it would be speculative | + +**No new API calls anywhere.** Both fixes widen data that was already fetched and +then discarded — the same shape as the original repair. + +### Measurement status, stated honestly + +These are serving changes for the MLB feature path and the NBA/WNBA log path. +**Their before/after resolution is NOT measured here**, and deliberately: the +only way to measure it today would be to reconstruct the repaired forecast over +old rows, which is the reconstruction-vs-served trap this order explicitly +refuses. They ship as code fixes with the measurement deferred to accrual, which +is the honest sequencing. + +--- + +## PHASE 2 — the forward re-audit rule, in code + +`MODEL_VERSION` is bumped to **`engine1@2026-08-07-fullwindow`**, so every +snapshot from this commit forward is self-identifying. `retentionService` already +stamps it onto `model_snapshots`, so no new plumbing was needed. + +`model/reAuditEligibility.js` encodes the rule: + +- **`isEligible(row)`** — true only for rows carrying the repaired marker. +- **`assess(rows)`** — counts eligible **DATES**, not rows, because dates have + been the binding scarcity in every interval this session. +- **`ACCRUAL`** (frozen) — pre-stated minimum dates per measurement: + +| measurement | minimum eligible dates | +|---|---| +| calibration re-fit | 10 | +| hits factor lift | 10 | +| prior verdict re-audit | 14 | +| rbi lineup-slot gate | 14 | + +A test locks the case that would otherwise be invisible: **a MIXED table** of 330 +rows where only 30 carry the new marker returns `eligible_dates: 3`, not 330 +rows' worth of false confidence. Once both generations sit in the same table, a +naive count would happily fit a map on a blend of two different forecasters. + +--- + +## PHASE 3 — the honest board + +**What happened:** the champion computed its season rate over ten games. Found by +resolution decomposition, not by a test failing. Fixed in two lines. It no longer +*loses* to a frequency table — it **beats** it CI-confirmed only on total_bases, +**ties** on rbi and runs, and leads on the hits point estimate. + +**Consequences, each labelled:** + +- **CALIBRATION — WITHDRAWN.** `CALIBRATION_DEPLOYED` is empty. Maps were fitted + on the retired forecast. Re-fits on repaired-champion settled rows. *Waiting on + accrual: 10 dates.* Not to be refit on reconstructions. +- **FACTOR VERDICTS — SUSPECT.** Every prior null and every THEATER was measured + against a champion worse than a frequency table; signal added to noise reads as + noise. Re-audit on accrued rows. **Direction UNKNOWN** — some may pass, some + may still fail. Not pre-priced. *Waiting: 14 dates.* +- **HITS FACTOR LIFT (1.39%) — UN-REMEASURABLE.** Needs rows produced *by* the + repaired champion. *Waiting: 10 dates.* The factors remain wired and + transmitting (43f65d3); only the lift number is unquantified. +- **RBI LINEUP-SLOT — RE-QUEUED.** Lands after the champion is sound and rows + accrue. *Waiting: 14 dates.* + +**Pre-registered re-audit order** (each runs only when its bar is met): +1. Re-fit calibration on repaired-champion rows (10 dates) +2. Re-measure hits factor lift (10 dates) +3. Re-audit prior factor verdicts (14 dates) +4. Run rbi lineup-slot through the two-part gate (14 dates) + +**Then STOP and accrue.** Nothing further can be honestly measured until the +board fills with rows the repaired champion produced. + +--- + +## Invariants + +No measurement on reconstructions — hard refusal, and it is why Phase 1 ships +code without numbers. Serving-path changes by design for the MLB feature path and +NBA/WNBA logs; frozen model modules verified unchanged. `p_win` never mutated. No +Bonferroni slot — base-rate repair and a bug hunt, not causal factors. diff --git a/src/services/adapters/espnStatsAdapter.js b/src/services/adapters/espnStatsAdapter.js index 0d5e3cc..279f6eb 100644 --- a/src/services/adapters/espnStatsAdapter.js +++ b/src/services/adapters/espnStatsAdapter.js @@ -227,7 +227,10 @@ function parseGameLog(payload) { if (Number.isNaN(tb)) return -1; return tb - ta; }); - return rows.slice(0, 20); + // The ESPN payload carries the full season's events; capping at 20 made every + // downstream "season rate" a 20-game rate. Same class as the MLB last10 bug + // and free to widen -- this is the same response, already parsed. + return rows; } async function fetchJsonG(url, opts = {}) { diff --git a/src/services/intelligence/featureCache.js b/src/services/intelligence/featureCache.js index 983f2e9..e46b4c8 100644 --- a/src/services/intelligence/featureCache.js +++ b/src/services/intelligence/featureCache.js @@ -129,7 +129,12 @@ const NBA_LOG_FIELD = { function mlbGameLogFeatures(res, statType) { if (!res || !res.found) return {}; const out = {}; - const logs = Array.isArray(res.last10) ? res.last10 : []; + // SAME WINDOW BUG AS THE BASE RATE. `l20_avg` is documented as "the season + // per-game reference projectionFor needs" and is read by the consistency + // pull — but built from last10 it was a TEN-game average wearing a season + // label. fullLog is already in this same response, so widening is free. + const logs = (Array.isArray(res.fullLog) && res.fullLog.length) + ? res.fullLog : (Array.isArray(res.last10) ? res.last10 : []); const vals = logs.map((g) => mlbStatValue(g.stat, statType)).filter((v) => v != null); if (vals.length) { const m5 = avg(vals.slice(-5)); // game logs are chronological (recent last) diff --git a/src/services/model/reAuditEligibility.js b/src/services/model/reAuditEligibility.js new file mode 100644 index 0000000..a0f3383 --- /dev/null +++ b/src/services/model/reAuditEligibility.js @@ -0,0 +1,63 @@ +'use strict'; + +/** + * reAuditEligibility — which settled rows may be measured on. + * + * The champion was repaired on 2026-08-07: it had been reading ten games as its + * season rate. Everything measured before that ran against a forecaster that + * lost to a frequency table, so calibration maps fitted on those rows correct + * toward a bias the current forecast may not have, and every factor verdict was + * scored against a sub-trivial baseline. + * + * The temptation is to reconstruct the repaired forecast over old rows and + * measure on that. It is REFUSED: a reconstruction is not what was served, and + * scoring a served product against a simulation of itself is the same class of + * error as scoring a map on the window it was fitted to. + * + * So eligibility is mechanical — a row qualifies only if the snapshot that + * produced it carries the repaired champion's version marker. + */ + +const { REPAIRED_CHAMPION_VERSION } = require('../retentionService'); + +/** + * Minimum settled DATES before each forward measurement is honest. + * + * Not row counts: the binding scarcity all session has been dates, and every + * interval that mattered was date-clustered. Stated here so the thresholds + * cannot drift toward whichever answer arrives first. + */ +const ACCRUAL = Object.freeze({ + calibration_refit: 10, // isotonic/low-param need a fit AND a held-out window + hits_factor_lift: 10, // a date-block CI on a composed lift + prior_verdict_reaudit: 14, // re-running gates that previously returned nulls + rbi_lineup_slot_gate: 14, // a fresh two-part gate on a new factor +}); + +/** Was this row produced by the repaired champion? */ +function isEligible(row) { + if (!row) return false; + return String(row.model_version || '') === REPAIRED_CHAMPION_VERSION; +} + +/** + * @returns {object} { eligible, dates, ready:{...}, blocked_reason } + * `ready` is per-measurement, so one can unblock before another. + */ +function assess(rows) { + const eligible = (rows || []).filter(isEligible); + const dates = new Set(eligible.map((r) => String(r.game_date || ''))).size; + const ready = Object.fromEntries(Object.entries(ACCRUAL) + .map(([k, need]) => [k, { need, have: dates, ready: dates >= need }])); + return { + eligible_rows: eligible.length, + total_rows: (rows || []).length, + eligible_dates: dates, + ready, + blocked_reason: dates === 0 + ? 'no settled rows yet carry the repaired champion marker — nothing may be measured' + : null, + }; +} + +module.exports = { isEligible, assess, ACCRUAL, REPAIRED_CHAMPION_VERSION }; diff --git a/src/services/retentionService.js b/src/services/retentionService.js index 2d53745..d581293 100644 --- a/src/services/retentionService.js +++ b/src/services/retentionService.js @@ -42,7 +42,23 @@ function etDateOf(iso) { * Bump when the grading model changes in a way that makes rows non-comparable. * This is the marker `ledger_entries` never had. */ -const MODEL_VERSION = process.env.MODEL_VERSION || 'engine1@2026-07-20'; +/** + * CHAMPION VERSION — the eligibility marker for every forward re-audit. + * + * Bumped when the forecaster itself changes, so a settled row is + * self-identifying: rows tagged `engine1@2026-08-07-fullwindow` were produced by + * the REPAIRED champion (full season log, recency weight 0.20); anything earlier + * came from the retired ten-game forecaster. + * + * This is what makes the re-audit rule mechanical rather than a promise. + * Calibration may only be re-fit, and factor verdicts may only be re-audited, on + * rows carrying the current marker — never on reconstructions of a retired + * forecast, and never on a mixture of the two, which is the trap that would + * otherwise be invisible once both generations sit in the same table. + */ +const MODEL_VERSION = process.env.MODEL_VERSION || 'engine1@2026-08-07-fullwindow'; +/** Rows at or after this marker are eligible for forward re-audit. */ +const REPAIRED_CHAMPION_VERSION = 'engine1@2026-08-07-fullwindow'; function codeSha() { return process.env.SOURCE_COMMIT || process.env.GIT_SHA || process.env.COOLIFY_GIT_COMMIT_SHA || null; @@ -229,6 +245,7 @@ function newSnapshotId() { module.exports = { MODEL_VERSION, + REPAIRED_CHAMPION_VERSION, codeSha, rowsFromSides, createCollector, diff --git a/tests/unit/reAuditEligibility.test.js b/tests/unit/reAuditEligibility.test.js new file mode 100644 index 0000000..cd37e64 --- /dev/null +++ b/tests/unit/reAuditEligibility.test.js @@ -0,0 +1,65 @@ +'use strict'; + +/** + * Which settled rows a forward re-audit may use. + * + * The refusal these encode: no measurement on rows produced by the retired + * ten-game forecaster, and no reconstruction standing in for what was served. + */ + +const el = require('../../src/services/model/reAuditEligibility'); + +const row = (version, date) => ({ model_version: version, game_date: date }); +const OLD = 'engine1@2026-07-20'; + +describe('eligibility is mechanical, not a promise', () => { + it('accepts only rows carrying the repaired champion marker', () => { + expect(el.isEligible(row(el.REPAIRED_CHAMPION_VERSION, '2026-08-08'))).toBe(true); + expect(el.isEligible(row(OLD, '2026-08-08'))).toBe(false); + expect(el.isEligible(row(null, '2026-08-08'))).toBe(false); + expect(el.isEligible(null)).toBe(false); + }); + + it('an all-old table blocks every measurement and says why', () => { + const a = el.assess(Array.from({ length: 500 }, (_, i) => row(OLD, `2026-07-${(i % 28) + 1}`))); + expect(a.eligible_rows).toBe(0); + expect(a.blocked_reason).toMatch(/nothing may be measured/); + for (const v of Object.values(a.ready)) expect(v.ready).toBe(false); + }); + + it('a MIXED table counts only the repaired rows — the invisible trap', () => { + // Once both generations sit in the same table, a naive count would happily + // fit a map on a blend of two different forecasters. + const mixed = [ + ...Array.from({ length: 300 }, (_, i) => row(OLD, `2026-07-${(i % 20) + 1}`)), + ...Array.from({ length: 30 }, (_, i) => row(el.REPAIRED_CHAMPION_VERSION, `2026-08-${(i % 3) + 8}`)), + ]; + const a = el.assess(mixed); + expect(a.total_rows).toBe(330); + expect(a.eligible_rows).toBe(30); + expect(a.eligible_dates).toBe(3); + }); +}); + +describe('thresholds are per-measurement and pre-stated', () => { + it('unblocks each measurement independently at its own date bar', () => { + const dates = 10; + const rows = Array.from({ length: 400 }, (_, i) => row(el.REPAIRED_CHAMPION_VERSION, `2026-08-${(i % dates) + 8}`)); + const a = el.assess(rows); + expect(a.eligible_dates).toBe(dates); + expect(a.ready.calibration_refit.ready).toBe(true); + expect(a.ready.hits_factor_lift.ready).toBe(true); + // The heavier measurements still wait. + expect(a.ready.prior_verdict_reaudit.ready).toBe(false); + expect(a.ready.rbi_lineup_slot_gate.ready).toBe(false); + }); + + it('counts DATES, not rows — the binding scarcity all session', () => { + const many = Array.from({ length: 5000 }, () => row(el.REPAIRED_CHAMPION_VERSION, '2026-08-08')); + expect(el.assess(many).ready.calibration_refit.ready).toBe(false); + }); + + it('the thresholds are frozen so they cannot drift', () => { + expect(Object.isFrozen(el.ACCRUAL)).toBe(true); + }); +});