Backtest harness — the validator, built refusal-first
Phase 0 gate PASSED: the join is clean. No FK exists; the natural key (sport, player_key, stat, line, side, game_date) yields 283 clean 1:1 joins with ZERO ambiguity. game_id is NOT usable — 400/550 snapshot rows carry UNK@UNK because home/away names weren't threaded into the grader until Order 1.6. Non-joining rows are EXPECTED, not errors: retention stores both sides plus refusals; the ledger keeps only the graded side. Outcomes are NOT denormalized — ledger_entries stays the source of truth. BUILT TEST-FIRST, and the first property proven is the REFUSAL, not the math. Below threshold the harness emits INSUFFICIENT with n and the shortfall and NO rate anywhere in the payload, so a downstream renderer cannot surface one by accident. A test asserts the payload contains no hit_rate number at all. - Wilson intervals (correct at the n we actually have, unlike the normal approximation which emits negative lower bounds). - Strata NEVER mix sport or model_version. - Denominator excludes quarantined, void, unrecoverable, pending, push — asserted by test. - Monotonicity refuses to RANK buckets whose intervals overlap; it reports "not distinguishable on this sample". - Probability calibration (Brier + reliability) also respects the threshold: a thin sample returns status INSUFFICIENT and a NULL score. - Replay seam reads the STORED feature vector only. A row whose input was never retained is UN-BACKTESTABLE, never scored with substituted current data. Identity replay reproduces the live prediction exactly. The tests caught a real bug in my own code: `Number(null) === 0` let a null p_win through as a confident 0% forecast — this codebase's signature fabrication bug, inside the harness whose entire purpose is refusing invented numbers. Fixed with a strict null guard. FIRST LIVE RUN — the correct, passing output: VERDICT: INSUFFICIENT_HISTORY (can_validate=false) 283 joined -> 35 scored (120 quarantined, 124 pending, 4 terminal) C n=18 (short by 2), B n=17 (short by 3) strata: mlb 7, wnba 28 — never mixed migration 028 adds harness_results (append-only trend log; INSUFFICIENT rows are expected and correct) and opsWatch.harnessStaleAlarm pages if the harness stops running — a validator that isn't running looks exactly like one that keeps passing. Suite 283/3403 green, build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
This commit is contained in:
@@ -309,3 +309,26 @@ describe('settlementRateAlarm (Session 64)', () => {
|
||||
expect(opsWatch.settlementRateAlarm([{ sport: 'soccer', skipped: 'not configured' }]).alarm).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('harnessStaleAlarm (Session 64)', () => {
|
||||
const opsWatch = require('../../src/services/opsWatch');
|
||||
const NOW = new Date('2026-07-21T12:00:00Z');
|
||||
|
||||
test('never-run is an alarm — a validator that never ran looks like one that passes', () => {
|
||||
const r = opsWatch.harnessStaleAlarm(null, NOW);
|
||||
expect(r.alarm).toBe(true);
|
||||
expect(r.reason).toMatch(/NEVER/);
|
||||
});
|
||||
|
||||
test('a stale run pages', () => {
|
||||
expect(opsWatch.harnessStaleAlarm('2026-07-19T00:00:00Z', NOW).alarm).toBe(true);
|
||||
});
|
||||
|
||||
test('a recent run is quiet', () => {
|
||||
expect(opsWatch.harnessStaleAlarm('2026-07-21T06:00:00Z', NOW).alarm).toBe(false);
|
||||
});
|
||||
|
||||
test('an unparseable timestamp is an alarm, not a silent pass', () => {
|
||||
expect(opsWatch.harnessStaleAlarm('not-a-date', NOW).alarm).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user