Backtest harness — the validator, built refusal-first

Phase 0 gate PASSED: the join is clean. No FK exists; the natural key
(sport, player_key, stat, line, side, game_date) yields 283 clean 1:1
joins with ZERO ambiguity. game_id is NOT usable — 400/550 snapshot rows
carry UNK@UNK because home/away names weren't threaded into the grader
until Order 1.6. Non-joining rows are EXPECTED, not errors: retention
stores both sides plus refusals; the ledger keeps only the graded side.
Outcomes are NOT denormalized — ledger_entries stays the source of truth.

BUILT TEST-FIRST, and the first property proven is the REFUSAL, not the
math. Below threshold the harness emits INSUFFICIENT with n and the
shortfall and NO rate anywhere in the payload, so a downstream renderer
cannot surface one by accident. A test asserts the payload contains no
hit_rate number at all.

- Wilson intervals (correct at the n we actually have, unlike the normal
  approximation which emits negative lower bounds).
- Strata NEVER mix sport or model_version.
- Denominator excludes quarantined, void, unrecoverable, pending, push —
  asserted by test.
- Monotonicity refuses to RANK buckets whose intervals overlap; it reports
  "not distinguishable on this sample".
- Probability calibration (Brier + reliability) also respects the
  threshold: a thin sample returns status INSUFFICIENT and a NULL score.
- Replay seam reads the STORED feature vector only. A row whose input was
  never retained is UN-BACKTESTABLE, never scored with substituted current
  data. Identity replay reproduces the live prediction exactly.

The tests caught a real bug in my own code: `Number(null) === 0` let a
null p_win through as a confident 0% forecast — this codebase's signature
fabrication bug, inside the harness whose entire purpose is refusing
invented numbers. Fixed with a strict null guard.

FIRST LIVE RUN — the correct, passing output:
  VERDICT: INSUFFICIENT_HISTORY (can_validate=false)
  283 joined -> 35 scored (120 quarantined, 124 pending, 4 terminal)
  C n=18 (short by 2), B n=17 (short by 3)
  strata: mlb 7, wnba 28 — never mixed

migration 028 adds harness_results (append-only trend log; INSUFFICIENT
rows are expected and correct) and opsWatch.harnessStaleAlarm pages if the
harness stops running — a validator that isn't running looks exactly like
one that keeps passing.

Suite 283/3403 green, build exit 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA

scripts/run-backtest.js

@@ -0,0 +1,65 @@
+#!/usr/bin/env node
+/**
+ * RUN THE BACKTEST HARNESS (Session 64).
+ *
+ * Joins model_snapshots (the model's INPUTS + prediction) to ledger_entries
+ * (the single source of truth for OUTCOMES) on the natural key, and runs the
+ * harness. Outcomes are NEVER denormalized onto snapshots.
+ *
+ * Join key: (sport, player_key, stat, line, side, game_date).
+ * Verified empirically: 283 clean 1:1 joins, ZERO ambiguity. `game_id` is NOT
+ * usable — 400/550 snapshot rows carry `UNK@UNK` because home/away team names
+ * weren't threaded into the grader until Session 64 Order 1.6.
+ *
+ * Rows that don't join are EXPECTED, not errors: retention stores BOTH sides
+ * of every prop plus refusals, while the ledger keeps only the graded side of
+ * non-refused props.
+ *
+ *   node scripts/run-backtest.js <rows.json>   # rows exported via SQL
+ */
+
+const harness = require('../src/services/backtestHarness');
+
+const rows = JSON.parse(require('fs').readFileSync(process.argv[2], 'utf8'));
+const report = harness.runBacktest(rows, {});
+
+const pad = (s, n) => String(s).padEnd(n);
+console.log('══════════ VYNDR BACKTEST HARNESS ══════════');
+console.log(`generated_at : ${report.generated_at}`);
+console.log(`min_sample   : ${report.min_sample}`);
+console.log(`VERDICT      : ${report.verdict}   (can_validate=${report.can_validate})`);
+console.log('\n--- denominator ---');
+Object.entries(report.counts).forEach(([k, v]) => console.log(`  ${pad(k, 22)} ${v}`));
+
+console.log('\n--- grade buckets (4-letter) ---');
+for (const b of report.grade_buckets.sort((a, z) => z.n - a.n)) {
+  console.log(b.status === 'OK'
+    ? `  ${pad(b.bucket, 4)} n=${pad(b.n, 5)} hit=${(b.hit_rate * 100).toFixed(1)}%  95% CI [${(b.ci_low * 100).toFixed(1)}, ${(b.ci_high * 100).toFixed(1)}]`
+    : `  ${pad(b.bucket, 4)} n=${pad(b.n, 5)} INSUFFICIENT — need ${b.need} (short by ${b.short_by})`);
+}
+
+console.log('\n--- grade buckets (11-step) ---');
+for (const b of report.grade_11_buckets.sort((a, z) => z.n - a.n)) {
+  console.log(b.status === 'OK'
+    ? `  ${pad(b.bucket, 4)} n=${pad(b.n, 5)} hit=${(b.hit_rate * 100).toFixed(1)}%`
+    : `  ${pad(b.bucket, 4)} n=${pad(b.n, 5)} INSUFFICIENT`);
+}
+
+console.log(`\n--- monotonicity: ${report.monotonicity.verdict} ---`);
+report.monotonicity.comparisons.forEach((c) => console.log(
+  `  ${c.higher} vs ${c.lower}: ${c.distinguishable ? (c.holds ? 'HOLDS' : 'BROKEN') : 'not distinguishable on this sample'}`,
+));
+
+console.log('\n--- probability calibration ---');
+console.log(report.probability.n
+  ? `  n=${report.probability.n}  Brier=${report.probability.brier.toFixed(4)}`
+  : '  n=0 — no stored p_win on any joinable settled row');
+
+console.log('\n--- strata (never mixed) ---');
+report.strata.forEach((s) => console.log(`  ${pad(s.sport, 6)} ${pad(s.model_version, 24)} n=${s.n}`));
+
+require('fs').writeFileSync(
+  process.argv[3] || '/tmp/backtest-report.json',
+  JSON.stringify(report, null, 2),
+);
+console.log(`\nfull report → ${process.argv[3] || '/tmp/backtest-report.json'}`);