Files
vyndr/tests/unit/reAuditEligibility.test.js
T
builtbykev 494c83cf76 Hunt the window-bug class: three more paths, and the forward re-audit rule
in code

PHASE 0 — getStatRows is the single base-rate path, so every branch is
audited, plus the feature builders since l20_avg is the season reference
projectionFor reads:

  getStatRows MLB -> estimator base    fullLog            CORRECT (929fd81)
  mlbGameLogFeatures l5/l10/l20        last10 = 10        DEFECTIVE
  espnStatsAdapter.parseGameLog        slice(0,20)        DEFECTIVE
  getStatRows NBA/WNBA ESPN branch     inherits 20-cap    DEFECTIVE via source
  getStatRows NBA/WNBA python branch   getGameLogs(...,20) dormant (offline)
  pitcherEngine / skillProjection      statcast profiles  N/A
  pitcher props via getStatRows MLB    fullLog            CORRECT
  settleSource                         full log (S64)     CORRECT

THE PITCHER ANSWER IS GOOD NEWS: pitcher props run through the same
getStatRows MLB branch, so 929fd81 repaired them too. There is no separate
defective pitcher base-rate path.

THE ONE HIDING IN PLAIN SIGHT: mlbGameLogFeatures carries the comment
"l20 = all available (the season per-game reference projectionFor needs)"
while building from last10 -- so l20_avg was a TEN-GAME AVERAGE WEARING A
SEASON LABEL, feeding both the consistency pull inside the estimator and
projectionFor, which decides refusals. It survived the previous repair
because that fix touched only getStatRows.

PHASE 1 — mlbGameLogFeatures now reads fullLog; espnStatsAdapter drops its
slice(0,20) cap. ZERO new API calls on both: each widens data already
fetched and then discarded, the same shape as the original repair. The
python branch is left alone -- the service is offline in prod and fixing it
would be speculative.

Their before/after resolution is NOT measured, deliberately: the only way
to measure today is to reconstruct the repaired forecast over old rows,
which is the reconstruction-vs-served trap this order refuses. Code fix
now, measurement at accrual.

PHASE 2 — MODEL_VERSION bumped to engine1@2026-08-07-fullwindow, so every
forward snapshot is self-identifying (retentionService already stamps it;
no new plumbing). model/reAuditEligibility.js encodes the rule: isEligible
accepts only the repaired marker, assess counts eligible DATES not rows,
and ACCRUAL is frozen at calibration 10 / hits-lift 10 / verdict-reaudit
14 / rbi-gate 14. A test locks the invisible case -- a MIXED table of 330
rows with 30 repaired returns eligible_dates 3, not 330 rows of false
confidence. Once both generations share a table a naive count would fit a
map on a blend of two forecasters.

PHASE 3 — the board, each consequence labelled: calibration WITHDRAWN
(refits at 10 dates, never on reconstructions); factor verdicts SUSPECT
(all measured against a champion worse than a frequency table, direction
UNKNOWN, not pre-priced, 14 dates); hits factor lift UN-REMEASURABLE (10
dates, factors still wired and transmitting); rbi lineup-slot RE-QUEUED
(14 dates). Pre-registered order: calibration, hits lift, verdict
re-audit, rbi gate.

Then STOP and accrue. Nothing further can be honestly measured until the
board fills with rows the repaired champion produced.

Serving-path changes by design for the MLB feature path and NBA/WNBA logs;
eleven frozen model modules verified unchanged. p_win never mutated. No
Bonferroni slot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-07 03:40:20 -04:00

66 lines
2.7 KiB
JavaScript

'use strict';
/**
* Which settled rows a forward re-audit may use.
*
* The refusal these encode: no measurement on rows produced by the retired
* ten-game forecaster, and no reconstruction standing in for what was served.
*/
const el = require('../../src/services/model/reAuditEligibility');
const row = (version, date) => ({ model_version: version, game_date: date });
const OLD = 'engine1@2026-07-20';
describe('eligibility is mechanical, not a promise', () => {
it('accepts only rows carrying the repaired champion marker', () => {
expect(el.isEligible(row(el.REPAIRED_CHAMPION_VERSION, '2026-08-08'))).toBe(true);
expect(el.isEligible(row(OLD, '2026-08-08'))).toBe(false);
expect(el.isEligible(row(null, '2026-08-08'))).toBe(false);
expect(el.isEligible(null)).toBe(false);
});
it('an all-old table blocks every measurement and says why', () => {
const a = el.assess(Array.from({ length: 500 }, (_, i) => row(OLD, `2026-07-${(i % 28) + 1}`)));
expect(a.eligible_rows).toBe(0);
expect(a.blocked_reason).toMatch(/nothing may be measured/);
for (const v of Object.values(a.ready)) expect(v.ready).toBe(false);
});
it('a MIXED table counts only the repaired rows — the invisible trap', () => {
// Once both generations sit in the same table, a naive count would happily
// fit a map on a blend of two different forecasters.
const mixed = [
...Array.from({ length: 300 }, (_, i) => row(OLD, `2026-07-${(i % 20) + 1}`)),
...Array.from({ length: 30 }, (_, i) => row(el.REPAIRED_CHAMPION_VERSION, `2026-08-${(i % 3) + 8}`)),
];
const a = el.assess(mixed);
expect(a.total_rows).toBe(330);
expect(a.eligible_rows).toBe(30);
expect(a.eligible_dates).toBe(3);
});
});
describe('thresholds are per-measurement and pre-stated', () => {
it('unblocks each measurement independently at its own date bar', () => {
const dates = 10;
const rows = Array.from({ length: 400 }, (_, i) => row(el.REPAIRED_CHAMPION_VERSION, `2026-08-${(i % dates) + 8}`));
const a = el.assess(rows);
expect(a.eligible_dates).toBe(dates);
expect(a.ready.calibration_refit.ready).toBe(true);
expect(a.ready.hits_factor_lift.ready).toBe(true);
// The heavier measurements still wait.
expect(a.ready.prior_verdict_reaudit.ready).toBe(false);
expect(a.ready.rbi_lineup_slot_gate.ready).toBe(false);
});
it('counts DATES, not rows — the binding scarcity all session', () => {
const many = Array.from({ length: 5000 }, () => row(el.REPAIRED_CHAMPION_VERSION, '2026-08-08'));
expect(el.assess(many).ready.calibration_refit.ready).toBe(false);
});
it('the thresholds are frozen so they cannot drift', () => {
expect(Object.isFrozen(el.ACCRUAL)).toBe(true);
});
});