LODO-gated provisional calibration: total_bases deploys, hits withdrawn

PHASE 0 — I applied factorGate's >=40 date-cluster floor to a calibration layer without challenging the binding. That floor is a cluster-robust interval bar for a CAUSAL claim. Calibration makes no causal claim, has a bounded failure mode (it can only over- or under-shrink) and consumes no Bonferroni slot. Its real risk is that the correction is DATE-DRIVEN, and leave-one-date-out tests that directly -- a STRICTER bar, since a cluster count cannot detect a single day carrying the effect. The >=40 floor is retained, correctly scoped as the PROMOTION bar. PHASE 1 — both guards codified, 11 tests, green before Phase 2. Demonstrated on live data: raw population violated=true, mean_p 0.4962, both_sides_share 0.9763; after dedup violated=false, mean_p 0.6694. The null guard's test demonstrates the trap explicitly, since (null-1)**2 is 1 and (null-0)**2 is 0 so a Brier over nulls equals the win rate. PHASE 2 — LODO: hits n=1140 dates=17 2 reversals (07-22 n=20, 07-26 n=25) FAIL total_bases n=1050 dates=7 0 reversals, 0 sign flips PASS rbi n= 630 dates=5 1 reversal (08-01 n=99) FAIL runs n= 597 dates=5 2 reversals (08-01 n=86, 08-05 n=244) FAIL Threshold sensitivity reported because the verdict moves: total_bases passes at every held-size threshold, runs fails at every one, and hits fails ONLY when 20/25-row dates are admitted. I fixed MIN_HELD_ROWS=20 before seeing which stats passed and did not move it afterwards to preserve a deploy. Honest caveat: a per-date Brier delta on 20 rows has a standard error several times the effect, so the instrument is underpowered per-drop -- an argument for pre-registering a higher threshold, which is a Roundtable call, not one to make while holding the results. PHASE 3 — total_bases DEPLOY-PROVISIONAL, band [0.6-0.8]. hits, rbi and runs REFUSE. HITS WAS BEING SERVED CALIBRATED AND IS NOT ANY MORE. snapshotService hardcoded it since S91; it fails LODO, so it is out. A stat that cannot survive dropping one day was never calibrated, it was fitted to that day. The consequence is real -- hits props become unstackable for chain.chainAcross -- and it errs toward withdrawing a claim rather than preserving one on a fragile verdict. Deployment is now driven by a frozen, tested CALIBRATION_DEPLOYED set, not a hardcoded stat name. PHASE 4 — calibrationRegistry, 14 tests. Deploy needs BOTH gates, neither waivable. reverify auto-demotes on the first breach (CI stops excluding zero, or the favourite bias flips sign) and logs the breaking date. Promotion needs the original >=40 bar. A provisional deploy that cannot be taken away is just a deploy. PHASE 5 — TB bands rebuilt on calibrated values, 625 eval rows. The two-bar rule still bites: calibrated YES, proven NO, so they stay a base-rate read, now honestly numbered. Every archetype still collapses to one band -- calibrated p_win separates within archetype no better than raw. PHASE 6 logged only: the dead gradient is buried (hits~TB > runs > RBI, and RBI has the SMALLEST bias, so the skill-driven-gradient mechanism did not survive); the refused set is a map of missing inputs; a low-parameter calibrator is queued unbuilt. p_win never mutated; calibration rides as p_win_calibrated with calibration_status provisional. No Bonferroni slot consumed. Counter and frozen clusters byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-06 18:31:19 -04:00
parent f976df47b8
commit 6ae11f1193
10 changed files with 1064 additions and 23 deletions
@@ -0,0 +1,121 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * PHASE 5 — rebuild grade bands on p_win_calibrated, for DEPLOYED stats only.
+ *
+ * total_bases is the only stat that cleared LODO, so it is the only one whose
+ * bands are rebuilt on calibrated values. The rest keep base-rate bands built on
+ * raw p_win, and the reason is named rather than left to inference.
+ *
+ * The two-bar rule still applies and still bites: TB is now CALIBRATED but no
+ * factor is PROVEN for it (barrel, exit velo and hard-contact-allowed were all
+ * THEATER), so the bands remain a base-rate read — now an honestly-numbered one.
+ */
+
+require('dotenv').config();
+const fs = require('fs');
+const path = require('path');
+const { createClient } = require('@supabase/supabase-js');
+const cal = require('../src/services/model/calibration');
+const gb = require('../src/services/model/gradeBands');
+const guards = require('../src/services/model/calibrationGuards');
+const tl = require('../src/services/model/testLedger');
+const { knownNumber } = require('../src/utils/known');
+
+const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
+const STAT = process.env.BAND_STAT || 'total_bases';
+const PAGE = 1000;
+
+async function page(sb, t, s, f) {
+  const o = [];
+  for (let i = 0; ; i += PAGE) {
+    const { data, error } = await f(sb.from(t).select(s)).order('id', { ascending: true }).range(i, i + PAGE - 1);
+    if (error) throw error;
+    if (!data.length) break;
+    o.push(...data);
+    if (data.length < PAGE) break;
+  }
+  return o;
+}
+const isPreGame = (c, g) => {
+  const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000);
+  const d = et.toISOString().slice(0, 10);
+  return d < g || (d === g && et.getUTCHours() < 19);
+};
+const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
+
+(async () => {
+  const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } });
+  const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
+
+  const snaps = await page(sb, 'model_snapshots',
+    'id, game_date, captured_at, stat, player_key, line, side, p_win, refused, archetype',
+    (q) => q.eq('sport', 'mlb').eq('stat', STAT));
+
+  const picked = new Map();
+  for (const r of snaps) {
+    if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue;
+    const k = [r.game_date, r.stat, r.player_key, r.line].join('|');
+    const prev = picked.get(k);
+    if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r);
+  }
+  guards.assertPickedSideDedup([...picked.values()].map((r) => ({
+    propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win),
+  })));
+
+  const rows = [];
+  for (const r of picked.values()) {
+    const b = lines[`${r.game_date}|${r.player_key}`];
+    const L = knownNumber(r.line);
+    if (!b || L === null || !r.side) continue;
+    const v = knownNumber(FIELD[STAT](b));
+    if (v === null) continue;
+    const over = v > L;
+    rows.push({
+      date: r.game_date, p: knownNumber(r.p_win),
+      won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0,
+      archetype: String(r.archetype || 'UNLABELLED').toUpperCase(),
+    });
+  }
+  rows.sort((a, b) => String(a.date).localeCompare(String(b.date)));
+
+  // Point-in-time map, then apply forward.
+  const dates = [...new Set(rows.map((r) => r.date))].sort();
+  const perDate = new Map();
+  for (const r of rows) perDate.set(r.date, (perDate.get(r.date) || 0) + 1);
+  let acc = 0; let cut = dates[dates.length - 1];
+  for (const d of dates) { acc += perDate.get(d); if (acc >= rows.length * 0.45) { cut = d; break; } }
+  const map = cal.fitIsotonic(rows.filter((r) => r.date < cut).map((r) => ({ p: r.p, won: r.won })));
+  const applied = guards.applyOrRefuse(map, rows.filter((r) => r.date >= cut), cal.applyIsotonic);
+  if (!applied.ok) { console.log(JSON.stringify({ stat: STAT, refused: applied.reason })); process.exit(0); }
+
+  const mc = await tl.recordAndCount(tl.supabaseStore(sb), []).catch(() => ({ cumulative_tests: 1 }));
+
+  const byArch = new Map();
+  for (const r of applied.rows) {
+    if (!byArch.has(r.archetype)) byArch.set(r.archetype, []);
+    byArch.get(r.archetype).push({ p: r.pc, won: r.won });
+  }
+
+  const out = [];
+  for (const [arch, rs] of [...byArch.entries()].sort((a, b) => b[1].length - a[1].length)) {
+    out.push(gb.buildBands(rs, {
+      archetype: arch,
+      cumulativeTests: mc.cumulative_tests,
+      // TB is CALIBRATED (provisional) but no factor is PROVEN for it.
+      proven: false,
+      calibrated: true,
+    }));
+  }
+
+  console.log(JSON.stringify({
+    stat: STAT,
+    basis: 'p_win_calibrated (PROVISIONAL)',
+    eval_rows: applied.rows.length,
+    cumulative_tests: mc.cumulative_tests,
+    two_bar_note: 'calibrated YES, proven NO -> bands stay a base-rate read, now honestly numbered',
+    bands: out,
+  }, null, 2));
+  process.exit(0);
+})().catch((e) => { console.error(e); process.exit(1); });