Instrument the calibration duel forward; diagnose the resolution ceiling
— the proven factors were never wired in
PHASE 0 — two truths recorded. The swap is a BET, not an OOS win:
isotonic beat low-param on identical held-out rows (hits +0.0028, rbi
+0.0042, TB tied) and we serve low-param anyway on an untestable prior
about shared daily structure. At 19 dates nothing here can test it. And
the MIN_SLOPE catch is preserved as standing rationale: a near-zero or
negative slope collapses toward base-rate-for-everything, which LOWERS
Brier while destroying all resolution -- a metric win that guts the
product.
PHASE 1 — the duel is now falsifiable. Both corrections computed on every
hits/TB prop; p_win_lowparam served, p_win_isotonic_shadow logged in its
own try so it can never break serving. calibrationDuel.adjudicate encodes
the rule IN CODE before any forward date exists: >=10 forward dates and
isotonic winning with a date-block CI excluding zero => REFUTED, revert;
otherwise UPHELD; under 10 dates PENDING regardless of the numbers. A
date counts as forward only if NEITHER map was fitted on it -- otherwise
we would be scoring which map memorised better. Nothing swaps now.
PHASE 2 — the ceiling, quantified via Murphy decomposition:
stat reliability RESOLUTION uncertainty variance explained
hits 0.01353 0.00252 0.24532 1.03%
TB 0.01419 0.00442 0.24329 1.82%
rbi 0.00654 0.03268 0.22531 14.51%
runs 0.00788 0.00130 0.23182 0.56%
Calibration did exactly what theory says and nothing more: hits
reliability 0.01353 -> 0.00233 (-0.0112, 83% of the error removed) while
resolution moved -0.0002. Unexpected: rbi has 13x the resolution of hits
and is the one stat we do NOT serve corrected -- it needs calibration
least and discriminates most.
PHASE 2 DIAGNOSIS — NOT-TRANSMITTED, and not weak, ABSENT. Traced in code:
sprayDefense.js and platoonSeverity.js are required by NOTHING in src/,
only by analysis scripts and their own tests. The served p_win
(intelligence/probabilityEstimator.js:54) reads exactly four inputs --
game-log frequency, opp_rank_stat +/-0.03, home_away +/-0.015, and a cv
pull -- with zero occurrences of spray, platoon, hard-hit or
contact-profile. And snapshotService grades at line 454 while computing
challenger/context at 640+, so everything proven is computed DOWNSTREAM of
the grade it would inform. The three proven hits factors have never once
moved a served number.
That reframes the recent nulls: "calibrated p_win does not separate within
archetype" was never a statement about factors. The factors were not in
the forecast.
PHASE 3 — bands rebuilt on SERVED values (hits/TB low-param, rbi/runs
raw): 28 archetype slots across four stats, ZERO show lift. No longer an
open shrug -- it is the arithmetic of resolution 0.0013-0.0327 against
uncertainty ~0.23. A forecast explaining 1% of variance cannot produce
separating bands, and no correction to its numbers will change that.
HEADLINE: calibration is complete, delivered honest numbers on two stats
and zero grade separation, because the counter has no resolution -- and
the proven factors are not wired into the forecast at all. The second is
the reason for the first, and it is plumbing rather than a modelling wall.
Per-archetype grades need proven factors that actually reach p_win. Last
calibration order.
Serving unchanged from 74cf1ce. p_win never mutated. No Bonferroni slot.
Counter and frozen clusters verified file-by-file (15 modules).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,125 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* calibrationDuel — the forward adjudication of a bet we made against the
|
||||
* measurement.
|
||||
*
|
||||
* On identical held-out rows the ISOTONIC map beat the low-parameter one (hits
|
||||
* +0.0028, rbi +0.0042, total_bases tied). We serve the low-parameter map
|
||||
* anyway, on the argument that isotonic's in-window edge is daily structure
|
||||
* shared between the fit and evaluation windows. At 19 dates that argument
|
||||
* cannot be tested — LODO has 1.4-9.3% power against it.
|
||||
*
|
||||
* So it is a BET. This module is what makes it falsifiable: both maps are
|
||||
* computed on every prop, the shadow is logged, and once enough genuinely
|
||||
* out-of-window dates settle, the season adjudicates.
|
||||
*
|
||||
* ── THE RULE IS PRE-REGISTERED, IN CODE ──────────────────────────────────
|
||||
* Written before any forward date exists, so the bar cannot drift toward
|
||||
* whichever answer arrives:
|
||||
*
|
||||
* REFUTED >=10 forward dates AND isotonic beats low-param with a date-block
|
||||
* bootstrap CI excluding zero -> revert hits/TB to isotonic
|
||||
* UPHELD >=10 forward dates and it does not -> the bet was right
|
||||
* PENDING fewer than 10 forward dates -> no verdict, keep serving
|
||||
*
|
||||
* A date is FORWARD only if NEITHER map was fitted on it. Scoring on a date
|
||||
* inside either fit window would be asking which map memorised better.
|
||||
*/
|
||||
|
||||
const { knownNumber } = require('../../utils/known');
|
||||
|
||||
/** Forward dates required before the duel may return a verdict. */
|
||||
const MIN_FORWARD_DATES = 10;
|
||||
const ITERS = 4000;
|
||||
|
||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||
|
||||
function makeRnd(seed) {
|
||||
let s = seed >>> 0;
|
||||
return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
|
||||
}
|
||||
|
||||
const brier = (rows, key) => {
|
||||
const usable = rows.filter((r) => knownNumber(r[key]) !== null && knownNumber(r.won) !== null);
|
||||
if (!usable.length) return null;
|
||||
return mean(usable.map((r) => (knownNumber(r[key]) - knownNumber(r.won)) ** 2));
|
||||
};
|
||||
|
||||
/**
|
||||
* @param {Array} rows [{ date, won, served, shadow, fitted_through }]
|
||||
* @param {object} opts { minForwardDates, seed }
|
||||
*/
|
||||
function adjudicate(rows, opts = {}) {
|
||||
const minDates = opts.minForwardDates ?? MIN_FORWARD_DATES;
|
||||
|
||||
// FORWARD ONLY: a row counts when its date postdates the window BOTH maps
|
||||
// were fitted on. Rows without that provenance are dropped, never assumed.
|
||||
const forward = (rows || []).filter((r) => {
|
||||
if (!r || !r.date) return false;
|
||||
if (knownNumber(r.served) === null || knownNumber(r.shadow) === null) return false;
|
||||
if (knownNumber(r.won) === null) return false;
|
||||
if (!r.fitted_through) return false;
|
||||
return String(r.date) > String(r.fitted_through);
|
||||
});
|
||||
|
||||
const dates = [...new Set(forward.map((r) => String(r.date)))].sort();
|
||||
if (dates.length < minDates) {
|
||||
return {
|
||||
verdict: 'PENDING',
|
||||
forward_dates: dates.length,
|
||||
forward_rows: forward.length,
|
||||
dates_needed: minDates - dates.length,
|
||||
reason: `${dates.length} forward dates < ${minDates} — the season has not spoken yet`,
|
||||
action: 'keep serving the low-parameter map',
|
||||
};
|
||||
}
|
||||
|
||||
const bServed = brier(forward, 'served');
|
||||
const bShadow = brier(forward, 'shadow');
|
||||
if (bServed === null || bShadow === null) {
|
||||
return { verdict: 'PENDING', forward_dates: dates.length, reason: 'no scorable forward rows' };
|
||||
}
|
||||
|
||||
// Paired date-block bootstrap on (isotonic - lowparam). Negative means the
|
||||
// shadow is better, which is the direction that refutes us.
|
||||
const byDate = new Map();
|
||||
for (const r of forward) {
|
||||
if (!byDate.has(String(r.date))) byDate.set(String(r.date), []);
|
||||
byDate.get(String(r.date)).push(r);
|
||||
}
|
||||
const keys = [...byDate.keys()];
|
||||
const rnd = makeRnd(opts.seed ?? 20260807);
|
||||
const diffs = [];
|
||||
for (let it = 0; it < ITERS; it += 1) {
|
||||
const s = [];
|
||||
for (let i = 0; i < keys.length; i += 1) s.push(...byDate.get(keys[Math.floor(rnd() * keys.length)]));
|
||||
const a = brier(s, 'shadow');
|
||||
const b = brier(s, 'served');
|
||||
if (a === null || b === null) continue;
|
||||
diffs.push(a - b);
|
||||
}
|
||||
diffs.sort((a, b) => a - b);
|
||||
const ci = diffs.length
|
||||
? [round5(diffs[Math.floor(diffs.length * 0.025)]), round5(diffs[Math.floor(diffs.length * 0.975)])]
|
||||
: null;
|
||||
|
||||
const shadowWins = ci !== null && ci[1] < 0;
|
||||
return {
|
||||
verdict: shadowWins ? 'REFUTED' : 'UPHELD',
|
||||
forward_dates: dates.length,
|
||||
forward_rows: forward.length,
|
||||
brier_served_lowparam: round5(bServed),
|
||||
brier_shadow_isotonic: round5(bShadow),
|
||||
delta_isotonic_minus_lowparam: round5(bShadow - bServed),
|
||||
ci,
|
||||
reason: shadowWins
|
||||
? 'isotonic beats the served low-parameter map out-of-window with a date-block interval excluding zero — the capacity argument is refuted'
|
||||
: 'the served low-parameter map is not beaten out-of-window — the bet stands',
|
||||
action: shadowWins ? 'REVERT hits and total_bases to isotonic and log the reversal' : 'keep serving the low-parameter map',
|
||||
};
|
||||
}
|
||||
|
||||
const round5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
|
||||
|
||||
module.exports = { adjudicate, MIN_FORWARD_DATES };
|
||||
@@ -734,46 +734,65 @@ async function runSnapshot(sport, opts = {}) {
|
||||
console.warn(`[challenger] ${sp} skipped:`, e.message);
|
||||
}
|
||||
|
||||
// ── FORWARD CALIBRATION (LODO-gated, per stat) ────────────────────────
|
||||
// ── FORWARD CALIBRATION + THE SHADOW DUEL ─────────────────────────────
|
||||
// Fitted on games that are OVER, applied to tonight's props. `p_win` is NOT
|
||||
// touched — the counter stays byte-identical and the calibrated value rides
|
||||
// touched — the counter stays byte-identical and the corrected value rides
|
||||
// beside it, because a calibration map is a correction TO a forecast, not a
|
||||
// different forecast.
|
||||
//
|
||||
// WHICH STATS SERVE IS MEASURED, NOT ASSUMED. The deploy bar is leave-one-
|
||||
// date-out stability: refit dropping each settled date in turn, and the
|
||||
// improvement must never reverse. That is the right instrument for a monotone
|
||||
// shrink-to-observed layer — the factor gate's >=40 date-cluster interval
|
||||
// floor was built for a CAUSAL claim and does not bind here.
|
||||
// WHAT IS SERVED, AND WHY IT IS A BET RATHER THAN A RESULT. On identical
|
||||
// held-out rows the ISOTONIC map scored BETTER than the low-parameter one
|
||||
// (hits +0.0028, rbi +0.0042, total_bases tied). We serve the low-parameter
|
||||
// map anyway, on the argument that isotonic's in-window edge is daily
|
||||
// structure shared between the fit and evaluation windows and will not
|
||||
// transmit forward. At 19 dates no instrument here can test that argument —
|
||||
// LODO has 1.4-9.3% power — so it is a BET, not evidence.
|
||||
//
|
||||
// Measured 2026-08-07: total_bases passes at every held-size threshold. hits
|
||||
// FAILS (reverses on 2026-07-22 and 2026-07-26), so it is no longer served
|
||||
// calibrated even though it was — a stat that cannot survive dropping one day
|
||||
// was never calibrated, it was fitted to that day. rbi and runs also fail.
|
||||
// So both are computed on every prop and the shadow is logged. Real
|
||||
// out-of-window dates adjudicate it:
|
||||
//
|
||||
// `calibrated` is true only inside a band certified out-of-sample, and it is
|
||||
// what `chain.chainAcross` requires before it will compound anything. Removing
|
||||
// hits here makes hits props unstackable again, which is the honest
|
||||
// consequence of the measurement rather than a regression to work around.
|
||||
// PRE-REGISTERED: once >=10 forward dates have settled that NEITHER map was
|
||||
// fitted on, if isotonic beats low-param with a date-block bootstrap CI
|
||||
// excluding zero, the capacity argument is REFUTED and hits/TB revert to
|
||||
// isotonic. If low-param wins or ties, the bet was right. The season
|
||||
// decides, not the argument.
|
||||
//
|
||||
// Serving is unchanged until that bar is met. `calibrated` is true only inside
|
||||
// a band certified out-of-sample, and it is what `chain.chainAcross` requires
|
||||
// before it will compound anything.
|
||||
if (sp === 'mlb') {
|
||||
for (const stat of CALIBRATION_DEPLOYED) {
|
||||
try {
|
||||
const calSvc = deps.calibrationService || require('./model/lowParamService');
|
||||
const shadowSvc = deps.shadowCalibrationService || require('./model/calibrationService');
|
||||
const sbc = require('../utils/supabase').getSupabaseServiceClient();
|
||||
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat }) : null;
|
||||
// The shadow must never break serving: its own try, and a null shadow
|
||||
// simply means the duel has no entry for tonight.
|
||||
let shadow = null;
|
||||
try { shadow = sbc ? await shadowSvc.fromLedger(sbc, { sport: 'mlb', stat }) : null; } catch { shadow = null; }
|
||||
|
||||
if (calibrator) {
|
||||
let marked = 0;
|
||||
let marked = 0; let shadowed = 0;
|
||||
for (const g of enriched) {
|
||||
if (String(g.stat_type || g.stat || '').toLowerCase() !== stat) continue;
|
||||
const out = calibrator.calibrate(g.p_win);
|
||||
g.p_win_calibrated = out.p_calibrated;
|
||||
g.p_win_lowparam = out.p_calibrated; // named, so the duel is legible
|
||||
g.calibrated = out.calibrated;
|
||||
g.calibration_reason = out.reason;
|
||||
g.calibration_status = 'provisional';
|
||||
g.calibration_basis = CALIBRATION_BASIS[stat] || null;
|
||||
g.calibration_basis = CALIBRATION_BASIS[stat] || null;
|
||||
if (out.calibrated) marked += 1;
|
||||
if (shadow) {
|
||||
const sh = shadow.calibrate(g.p_win);
|
||||
// SHADOW ONLY. Never read by serving, never by chainAcross.
|
||||
g.p_win_isotonic_shadow = sh.p_calibrated;
|
||||
g.calibration_duel_fitted_through = shadow.fitted_through || null;
|
||||
if (sh.p_calibrated != null) shadowed += 1;
|
||||
}
|
||||
}
|
||||
console.log(`[calibration] ${sp} ${stat} (low-param, PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, a=${calibrator.model.a} shrink=${calibrator.shrinkage}`);
|
||||
console.log(`[calibration] ${sp} ${stat} (low-param, PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, a=${calibrator.model.a} shrink=${calibrator.shrinkage}; shadow logged on ${shadowed}`);
|
||||
} else {
|
||||
console.log(`[calibration] ${sp} ${stat} — no calibrator (thin history); nothing is stackable`);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user