LODO-gated provisional calibration: total_bases deploys, hits withdrawn
PHASE 0 — I applied factorGate's >=40 date-cluster floor to a calibration layer without challenging the binding. That floor is a cluster-robust interval bar for a CAUSAL claim. Calibration makes no causal claim, has a bounded failure mode (it can only over- or under-shrink) and consumes no Bonferroni slot. Its real risk is that the correction is DATE-DRIVEN, and leave-one-date-out tests that directly -- a STRICTER bar, since a cluster count cannot detect a single day carrying the effect. The >=40 floor is retained, correctly scoped as the PROMOTION bar. PHASE 1 — both guards codified, 11 tests, green before Phase 2. Demonstrated on live data: raw population violated=true, mean_p 0.4962, both_sides_share 0.9763; after dedup violated=false, mean_p 0.6694. The null guard's test demonstrates the trap explicitly, since (null-1)**2 is 1 and (null-0)**2 is 0 so a Brier over nulls equals the win rate. PHASE 2 — LODO: hits n=1140 dates=17 2 reversals (07-22 n=20, 07-26 n=25) FAIL total_bases n=1050 dates=7 0 reversals, 0 sign flips PASS rbi n= 630 dates=5 1 reversal (08-01 n=99) FAIL runs n= 597 dates=5 2 reversals (08-01 n=86, 08-05 n=244) FAIL Threshold sensitivity reported because the verdict moves: total_bases passes at every held-size threshold, runs fails at every one, and hits fails ONLY when 20/25-row dates are admitted. I fixed MIN_HELD_ROWS=20 before seeing which stats passed and did not move it afterwards to preserve a deploy. Honest caveat: a per-date Brier delta on 20 rows has a standard error several times the effect, so the instrument is underpowered per-drop -- an argument for pre-registering a higher threshold, which is a Roundtable call, not one to make while holding the results. PHASE 3 — total_bases DEPLOY-PROVISIONAL, band [0.6-0.8]. hits, rbi and runs REFUSE. HITS WAS BEING SERVED CALIBRATED AND IS NOT ANY MORE. snapshotService hardcoded it since S91; it fails LODO, so it is out. A stat that cannot survive dropping one day was never calibrated, it was fitted to that day. The consequence is real -- hits props become unstackable for chain.chainAcross -- and it errs toward withdrawing a claim rather than preserving one on a fragile verdict. Deployment is now driven by a frozen, tested CALIBRATION_DEPLOYED set, not a hardcoded stat name. PHASE 4 — calibrationRegistry, 14 tests. Deploy needs BOTH gates, neither waivable. reverify auto-demotes on the first breach (CI stops excluding zero, or the favourite bias flips sign) and logs the breaking date. Promotion needs the original >=40 bar. A provisional deploy that cannot be taken away is just a deploy. PHASE 5 — TB bands rebuilt on calibrated values, 625 eval rows. The two-bar rule still bites: calibrated YES, proven NO, so they stay a base-rate read, now honestly numbered. Every archetype still collapses to one band -- calibrated p_win separates within archetype no better than raw. PHASE 6 logged only: the dead gradient is buried (hits~TB > runs > RBI, and RBI has the SMALLEST bias, so the skill-driven-gradient mechanism did not survive); the refused set is a map of missing inputs; a low-parameter calibrator is queued unbuilt. p_win never mutated; calibration rides as p_win_calibrated with calibration_status provisional. No Bonferroni slot consumed. Counter and frozen clusters byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,193 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* lodo-calibration — Phases 2 and 3.
|
||||
*
|
||||
* The ≥40 date-cluster floor was factorGate's interval bar for a CAUSAL claim,
|
||||
* mis-applied to a monotone shrink-to-observed layer. Calibration makes no causal
|
||||
* claim, consumes no Bonferroni slot, and has a bounded failure mode (it can only
|
||||
* over- or under-shrink). Its real risk is that the correction is DATE-DRIVEN —
|
||||
* that one unusual day's offensive environment is doing all the work.
|
||||
*
|
||||
* Leave-one-date-out tests exactly that, and it is a harder bar than a cluster
|
||||
* count: a single date whose removal reverses the improvement, or flips the
|
||||
* favourite-longshot sign, fails the stat outright.
|
||||
*
|
||||
* ── WHAT LODO IS AND IS NOT ──────────────────────────────────────────────
|
||||
* Refitting on all-but-one date uses dates that follow the held-out one, so this
|
||||
* is a STABILITY test, not a point-in-time backtest. The point-in-time result is
|
||||
* separate and already established (fit-past / apply-forward, CI excluding zero
|
||||
* on hits / TB / RBI). Both are required; neither substitutes for the other.
|
||||
*
|
||||
* SUPABASE_URL=... node scripts/lodo-calibration.js
|
||||
*/
|
||||
|
||||
require('dotenv').config();
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cal = require('../src/services/model/calibration');
|
||||
const guards = require('../src/services/model/calibrationGuards');
|
||||
const { knownNumber } = require('../src/utils/known');
|
||||
|
||||
const SB_URL = process.env.SUPABASE_URL;
|
||||
const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
|
||||
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
|
||||
const STATS = ['hits', 'total_bases', 'rbi', 'runs'];
|
||||
const PAGE = 1000;
|
||||
/** The favourite bucket where the over-prediction concentrates. */
|
||||
const FAVOURITE_FLOOR = 0.9;
|
||||
/** Minimum rows on a held-out date for that drop to be informative. */
|
||||
const MIN_HELD_ROWS = 20;
|
||||
|
||||
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
|
||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||
|
||||
async function page(sb, table, select, apply) {
|
||||
const out = [];
|
||||
for (let from = 0; ; from += PAGE) {
|
||||
const { data, error } = await apply(sb.from(table).select(select))
|
||||
.order('id', { ascending: true }).range(from, from + PAGE - 1);
|
||||
if (error) throw error;
|
||||
if (!data || data.length === 0) break;
|
||||
out.push(...data);
|
||||
if (data.length < PAGE) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const isPreGame = (capturedAt, gameDate) => {
|
||||
const et = new Date(new Date(capturedAt).getTime() - 4 * 3600 * 1000);
|
||||
const d = et.toISOString().slice(0, 10);
|
||||
return d < gameDate || (d === gameDate && et.getUTCHours() < 19);
|
||||
};
|
||||
|
||||
async function main() {
|
||||
const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
|
||||
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
|
||||
|
||||
const snaps = await page(sb, 'model_snapshots',
|
||||
'id, game_date, captured_at, stat, player_key, line, side, p_win, refused',
|
||||
(q) => q.eq('sport', 'mlb').in('stat', STATS));
|
||||
|
||||
// Build the RAW population first so the guard has something to catch.
|
||||
const raw = [];
|
||||
const picked = new Map();
|
||||
for (const r of snaps) {
|
||||
if (!isPreGame(r.captured_at, r.game_date)) continue;
|
||||
if (r.refused || knownNumber(r.p_win) === null) continue;
|
||||
const propKey = [r.game_date, r.stat, r.player_key, r.line].join('|');
|
||||
raw.push({ propKey, side: r.side, p: knownNumber(r.p_win) });
|
||||
const prev = picked.get(propKey);
|
||||
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(propKey, r);
|
||||
}
|
||||
|
||||
// GUARD 1 — prove the raw population would have lied, then prove dedup fixes it.
|
||||
const rawCheck = guards.checkPickedSideDedup(raw);
|
||||
const pickedRows = [...picked.values()].map((r) => ({
|
||||
propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'),
|
||||
side: r.side, p: knownNumber(r.p_win),
|
||||
}));
|
||||
guards.assertPickedSideDedup(pickedRows); // throws if dedup failed
|
||||
|
||||
const out = {
|
||||
guard_1_raw_population: { violated: rawCheck.violated, mean_p: rawCheck.mean_p, both_sides_share: rawCheck.both_sides_share },
|
||||
guard_1_after_dedup: guards.checkPickedSideDedup(pickedRows),
|
||||
per_stat: {},
|
||||
};
|
||||
|
||||
for (const stat of STATS) {
|
||||
const rows = [];
|
||||
for (const r of picked.values()) {
|
||||
if (r.stat !== stat) continue;
|
||||
const b = lines[`${r.game_date}|${r.player_key}`];
|
||||
const L = knownNumber(r.line);
|
||||
if (!b || L === null || !r.side) continue;
|
||||
const v = knownNumber(FIELD[stat](b));
|
||||
if (v === null) continue;
|
||||
const over = v > L;
|
||||
rows.push({
|
||||
date: r.game_date,
|
||||
p: knownNumber(r.p_win),
|
||||
won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0,
|
||||
});
|
||||
}
|
||||
const dates = [...new Set(rows.map((r) => r.date))].sort();
|
||||
|
||||
const full = cal.fitIsotonic(rows.map((r) => ({ p: r.p, won: r.won })));
|
||||
if (!full) {
|
||||
out.per_stat[stat] = {
|
||||
n: rows.length, dates: dates.length,
|
||||
lodo: 'NOT RUN', decision: 'REFUSE',
|
||||
reason: `no isotonic map is fittable at n=${rows.length} (needs ${cal.MIN_TOTAL || 200})`,
|
||||
};
|
||||
continue;
|
||||
}
|
||||
|
||||
// ── LEAVE ONE DATE OUT ──
|
||||
const table = [];
|
||||
for (const d of dates) {
|
||||
const fit = rows.filter((r) => r.date !== d);
|
||||
const held = rows.filter((r) => r.date === d);
|
||||
if (held.length < MIN_HELD_ROWS) {
|
||||
table.push({ dropped: d, held_n: held.length, verdict: 'UNINFORMATIVE', reason: 'too few rows on this date' });
|
||||
continue;
|
||||
}
|
||||
const map = cal.fitIsotonic(fit.map((r) => ({ p: r.p, won: r.won })));
|
||||
const applied = guards.applyOrRefuse(map, held, cal.applyIsotonic);
|
||||
if (!applied.ok) {
|
||||
table.push({ dropped: d, held_n: held.length, verdict: 'UNINFORMATIVE', reason: applied.reason });
|
||||
continue;
|
||||
}
|
||||
const ys = applied.rows.map((r) => r.won);
|
||||
const bRaw = guards.safeBrier(applied.rows.map((r) => r.p), ys);
|
||||
const bCal = guards.safeBrier(applied.rows.map((r) => r.pc), ys);
|
||||
if (bRaw === null || bCal === null) {
|
||||
table.push({ dropped: d, held_n: held.length, verdict: 'UNINFORMATIVE', reason: 'a null reached the metric' });
|
||||
continue;
|
||||
}
|
||||
const fav = applied.rows.filter((r) => r.p >= FAVOURITE_FLOOR);
|
||||
const favBias = fav.length >= 5 ? mean(fav.map((r) => r.p)) - mean(fav.map((r) => r.won)) : null;
|
||||
table.push({
|
||||
dropped: d,
|
||||
held_n: held.length,
|
||||
brier_delta: round4(bCal - bRaw),
|
||||
improves: bCal < bRaw,
|
||||
favourite_n: fav.length,
|
||||
favourite_bias: favBias === null ? null : round4(favBias),
|
||||
favourite_sign_holds: favBias === null ? null : favBias > 0,
|
||||
verdict: bCal < bRaw ? 'holds' : 'REVERSES',
|
||||
});
|
||||
}
|
||||
|
||||
const informative = table.filter((t) => t.verdict !== 'UNINFORMATIVE');
|
||||
const anyReversal = informative.some((t) => t.verdict === 'REVERSES');
|
||||
const signTested = informative.filter((t) => t.favourite_sign_holds !== null);
|
||||
const anySignFlip = signTested.some((t) => t.favourite_sign_holds === false);
|
||||
const passes = informative.length > 0 && !anyReversal && !anySignFlip;
|
||||
|
||||
out.per_stat[stat] = {
|
||||
n: rows.length,
|
||||
dates: dates.length,
|
||||
lodo_table: table,
|
||||
informative_drops: informative.length,
|
||||
brier_reversals: informative.filter((t) => t.verdict === 'REVERSES').length,
|
||||
favourite_sign_flips: signTested.filter((t) => t.favourite_sign_holds === false).length,
|
||||
favourite_sign_untested: informative.length - signTested.length,
|
||||
lodo: passes ? 'PASS' : 'FAIL',
|
||||
reason: passes
|
||||
? 'improvement never reverses and the favourite over-prediction never flips sign across any single-date drop'
|
||||
: (anyReversal
|
||||
? `improvement reverses when ${informative.filter((t) => t.verdict === 'REVERSES').map((t) => t.dropped).join(', ')} is dropped — the effect is date-driven`
|
||||
: `the favourite over-prediction flips sign when ${signTested.filter((t) => t.favourite_sign_holds === false).map((t) => t.dropped).join(', ')} is dropped`),
|
||||
};
|
||||
}
|
||||
|
||||
console.log(JSON.stringify(out, null, 2));
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
|
||||
main().catch((e) => { console.error(e); process.exit(1); });
|
||||
Reference in New Issue
Block a user