1 Commits

Author SHA1 Message Date
builtbykev e809a0eb3c Backtest harness — the validator, built refusal-first
Phase 0 gate PASSED: the join is clean. No FK exists; the natural key
(sport, player_key, stat, line, side, game_date) yields 283 clean 1:1
joins with ZERO ambiguity. game_id is NOT usable — 400/550 snapshot rows
carry UNK@UNK because home/away names weren't threaded into the grader
until Order 1.6. Non-joining rows are EXPECTED, not errors: retention
stores both sides plus refusals; the ledger keeps only the graded side.
Outcomes are NOT denormalized — ledger_entries stays the source of truth.

BUILT TEST-FIRST, and the first property proven is the REFUSAL, not the
math. Below threshold the harness emits INSUFFICIENT with n and the
shortfall and NO rate anywhere in the payload, so a downstream renderer
cannot surface one by accident. A test asserts the payload contains no
hit_rate number at all.

- Wilson intervals (correct at the n we actually have, unlike the normal
  approximation which emits negative lower bounds).
- Strata NEVER mix sport or model_version.
- Denominator excludes quarantined, void, unrecoverable, pending, push —
  asserted by test.
- Monotonicity refuses to RANK buckets whose intervals overlap; it reports
  "not distinguishable on this sample".
- Probability calibration (Brier + reliability) also respects the
  threshold: a thin sample returns status INSUFFICIENT and a NULL score.
- Replay seam reads the STORED feature vector only. A row whose input was
  never retained is UN-BACKTESTABLE, never scored with substituted current
  data. Identity replay reproduces the live prediction exactly.

The tests caught a real bug in my own code: `Number(null) === 0` let a
null p_win through as a confident 0% forecast — this codebase's signature
fabrication bug, inside the harness whose entire purpose is refusing
invented numbers. Fixed with a strict null guard.

FIRST LIVE RUN — the correct, passing output:
  VERDICT: INSUFFICIENT_HISTORY (can_validate=false)
  283 joined -> 35 scored (120 quarantined, 124 pending, 4 terminal)
  C n=18 (short by 2), B n=17 (short by 3)
  strata: mlb 7, wnba 28 — never mixed

migration 028 adds harness_results (append-only trend log; INSUFFICIENT
rows are expected and correct) and opsWatch.harnessStaleAlarm pages if the
harness stops running — a validator that isn't running looks exactly like
one that keeps passing.

Suite 283/3403 green, build exit 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
2026-07-20 11:31:48 -04:00