LODO-gated provisional calibration: total_bases deploys, hits withdrawn
PHASE 0 — I applied factorGate's >=40 date-cluster floor to a calibration layer without challenging the binding. That floor is a cluster-robust interval bar for a CAUSAL claim. Calibration makes no causal claim, has a bounded failure mode (it can only over- or under-shrink) and consumes no Bonferroni slot. Its real risk is that the correction is DATE-DRIVEN, and leave-one-date-out tests that directly -- a STRICTER bar, since a cluster count cannot detect a single day carrying the effect. The >=40 floor is retained, correctly scoped as the PROMOTION bar. PHASE 1 — both guards codified, 11 tests, green before Phase 2. Demonstrated on live data: raw population violated=true, mean_p 0.4962, both_sides_share 0.9763; after dedup violated=false, mean_p 0.6694. The null guard's test demonstrates the trap explicitly, since (null-1)**2 is 1 and (null-0)**2 is 0 so a Brier over nulls equals the win rate. PHASE 2 — LODO: hits n=1140 dates=17 2 reversals (07-22 n=20, 07-26 n=25) FAIL total_bases n=1050 dates=7 0 reversals, 0 sign flips PASS rbi n= 630 dates=5 1 reversal (08-01 n=99) FAIL runs n= 597 dates=5 2 reversals (08-01 n=86, 08-05 n=244) FAIL Threshold sensitivity reported because the verdict moves: total_bases passes at every held-size threshold, runs fails at every one, and hits fails ONLY when 20/25-row dates are admitted. I fixed MIN_HELD_ROWS=20 before seeing which stats passed and did not move it afterwards to preserve a deploy. Honest caveat: a per-date Brier delta on 20 rows has a standard error several times the effect, so the instrument is underpowered per-drop -- an argument for pre-registering a higher threshold, which is a Roundtable call, not one to make while holding the results. PHASE 3 — total_bases DEPLOY-PROVISIONAL, band [0.6-0.8]. hits, rbi and runs REFUSE. HITS WAS BEING SERVED CALIBRATED AND IS NOT ANY MORE. snapshotService hardcoded it since S91; it fails LODO, so it is out. A stat that cannot survive dropping one day was never calibrated, it was fitted to that day. The consequence is real -- hits props become unstackable for chain.chainAcross -- and it errs toward withdrawing a claim rather than preserving one on a fragile verdict. Deployment is now driven by a frozen, tested CALIBRATION_DEPLOYED set, not a hardcoded stat name. PHASE 4 — calibrationRegistry, 14 tests. Deploy needs BOTH gates, neither waivable. reverify auto-demotes on the first breach (CI stops excluding zero, or the favourite bias flips sign) and logs the breaking date. Promotion needs the original >=40 bar. A provisional deploy that cannot be taken away is just a deploy. PHASE 5 — TB bands rebuilt on calibrated values, 625 eval rows. The two-bar rule still bites: calibrated YES, proven NO, so they stay a base-rate read, now honestly numbered. Every archetype still collapses to one band -- calibrated p_win separates within archetype no better than raw. PHASE 6 logged only: the dead gradient is buried (hits~TB > runs > RBI, and RBI has the SMALLEST bias, so the skill-driven-gradient mechanism did not survive); the refused set is a map of missing inputs; a low-parameter calibrator is queued unbuilt. p_win never mutated; calibration rides as p_win_calibrated with calibration_status provisional. No Bonferroni slot consumed. Counter and frozen clusters byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,146 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* calibrationGuards — the two ways a calibration measurement lies to you.
|
||||
*
|
||||
* Both of these produced a confident, plausible, completely wrong number in the
|
||||
* settlement session, and neither was visible in the output. They are codified
|
||||
* here so the failure cannot recur silently.
|
||||
*
|
||||
* ── GUARD 1: THE BOTH-SIDES TELL ─────────────────────────────────────────
|
||||
* A prop population usually carries BOTH the over and the under. Their p_wins
|
||||
* sum to ~1 and their outcomes are complementary, so ANY population-level
|
||||
* calibration statistic over the raw set is pinned to 0.5 by construction — not
|
||||
* by the model being calibrated.
|
||||
*
|
||||
* Measured: the raw population read +0.0002 bias on hits ("perfectly
|
||||
* calibrated"); deduped to the model-picked side it read +0.0868. Same rows,
|
||||
* opposite conclusion. The tell was mean p_win sitting at 0.4998 on all four
|
||||
* stats at once, which is not something a real forecaster does.
|
||||
*
|
||||
* So picked-side dedup is MANDATORY preprocessing, and this asserts it.
|
||||
*
|
||||
* ── GUARD 2: A NULL THAT SCORES ITSELF ───────────────────────────────────
|
||||
* `fitIsotonic` returns null below its minimum and `applyIsotonic` then returns
|
||||
* null per row. In JavaScript `(null - 1) ** 2 === 1` and `(null - 0) ** 2 === 0`,
|
||||
* so a Brier score computed over nulls silently equals the WIN RATE — a number
|
||||
* in the right range, monotone in the data, and completely meaningless. It
|
||||
* reported hits at 0.5567 against a 0.5684 win rate.
|
||||
*
|
||||
* This is the `Number(null) === 0` breach the TRUTH LAW names, wearing a metric.
|
||||
* A null prediction must refuse, never score.
|
||||
*/
|
||||
|
||||
const { knownNumber } = require('../../utils/known');
|
||||
|
||||
/** How close to 0.5 counts as the both-sides signature. */
|
||||
const BALANCED_TOLERANCE = 0.02;
|
||||
|
||||
/**
|
||||
* Does this population still contain both sides of the same prop?
|
||||
*
|
||||
* @param {Array} rows [{ p, side, propKey }]
|
||||
* @returns {object} { violated, reason, ... } — never throws, so a caller can
|
||||
* decide between refusing and hard-failing.
|
||||
*/
|
||||
function checkPickedSideDedup(rows) {
|
||||
const usable = (rows || []).filter((r) => knownNumber(r && r.p) !== null);
|
||||
if (usable.length < 2) return { violated: false, reason: 'too few rows to judge', n: usable.length };
|
||||
|
||||
const sidesByProp = new Map();
|
||||
for (const r of usable) {
|
||||
const k = r.propKey == null ? null : String(r.propKey);
|
||||
if (k === null) continue;
|
||||
if (!sidesByProp.has(k)) sidesByProp.set(k, new Set());
|
||||
if (r.side) sidesByProp.get(k).add(String(r.side).toLowerCase());
|
||||
}
|
||||
let bothSides = 0;
|
||||
for (const s of sidesByProp.values()) if (s.size > 1) bothSides += 1;
|
||||
const propCount = sidesByProp.size;
|
||||
const bothShare = propCount ? bothSides / propCount : 0;
|
||||
|
||||
const meanP = usable.reduce((s, r) => s + knownNumber(r.p), 0) / usable.length;
|
||||
const balanced = Math.abs(meanP - 0.5) <= BALANCED_TOLERANCE;
|
||||
|
||||
// The violation is the CONJUNCTION: both sides present AND the mean pinned at
|
||||
// 0.5. Either alone is unremarkable — a genuinely balanced book of one-sided
|
||||
// picks is fine, and both sides present with a skewed mean means someone
|
||||
// already deduped.
|
||||
const violated = bothSides > 0 && balanced;
|
||||
return {
|
||||
violated,
|
||||
n: usable.length,
|
||||
props: propCount,
|
||||
both_sides_props: bothSides,
|
||||
both_sides_share: round4(bothShare),
|
||||
mean_p: round4(meanP),
|
||||
reason: violated
|
||||
? `both sides present on ${bothSides}/${propCount} props while mean p_win is ${round4(meanP)} — `
|
||||
+ 'the population is balanced by construction and any calibration statistic over it is meaningless. '
|
||||
+ 'Dedup to the model-picked side first.'
|
||||
: null,
|
||||
};
|
||||
}
|
||||
|
||||
/** Same check, but refuses to continue. Use at the top of a measurement. */
|
||||
function assertPickedSideDedup(rows) {
|
||||
const r = checkPickedSideDedup(rows);
|
||||
if (r.violated) throw new Error(`CALIBRATION GUARD: ${r.reason}`);
|
||||
return r;
|
||||
}
|
||||
|
||||
/**
|
||||
* Brier score that refuses rather than scoring a null.
|
||||
*
|
||||
* @param {Array} preds
|
||||
* @param {Array} outcomes
|
||||
* @param {object} opts { onNull: 'throw' | 'refuse' } default 'refuse'
|
||||
* @returns {number|null} null when any prediction is unreadable
|
||||
*/
|
||||
function safeBrier(preds, outcomes, opts = {}) {
|
||||
const ps = preds || [];
|
||||
const ys = outcomes || [];
|
||||
if (ps.length === 0 || ps.length !== ys.length) return null;
|
||||
let sum = 0;
|
||||
for (let i = 0; i < ps.length; i += 1) {
|
||||
const p = knownNumber(ps[i]);
|
||||
const y = knownNumber(ys[i]);
|
||||
if (p === null || y === null) {
|
||||
// NEVER score it. (null - 1) ** 2 === 1 would pass silently.
|
||||
if (opts.onNull === 'throw') {
|
||||
throw new Error('CALIBRATION GUARD: a null prediction reached a Brier term');
|
||||
}
|
||||
return null;
|
||||
}
|
||||
sum += (p - y) ** 2;
|
||||
}
|
||||
return sum / ps.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a population through a calibration map, refusing unreadable rows rather
|
||||
* than letting them through as nulls.
|
||||
*/
|
||||
function applyOrRefuse(map, rows, applyFn) {
|
||||
if (!map) return { ok: false, reason: 'no calibration map could be fitted', rows: [] };
|
||||
const out = [];
|
||||
let dropped = 0;
|
||||
for (const r of rows || []) {
|
||||
const pc = applyFn(map, r.p);
|
||||
if (knownNumber(pc) === null) { dropped += 1; continue; }
|
||||
out.push({ ...r, pc });
|
||||
}
|
||||
return {
|
||||
ok: out.length > 0,
|
||||
rows: out,
|
||||
dropped,
|
||||
reason: out.length === 0 ? 'every row was unmappable' : null,
|
||||
};
|
||||
}
|
||||
|
||||
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
|
||||
module.exports = {
|
||||
checkPickedSideDedup, assertPickedSideDedup, safeBrier, applyOrRefuse,
|
||||
BALANCED_TOLERANCE,
|
||||
};
|
||||
@@ -0,0 +1,110 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* calibrationRegistry — which stats are allowed to serve a calibrated number.
|
||||
*
|
||||
* ── WHY THIS IS NOT THE FACTOR REGISTRY ──────────────────────────────────
|
||||
* A factor makes a CAUSAL claim, so it needs a Bonferroni slot, a cluster-robust
|
||||
* interval, and a bar that rises with every hypothesis the programme tests.
|
||||
* Calibration makes no causal claim: it is a monotone shrink toward what was
|
||||
* actually observed, its failure mode is bounded (it can only over- or
|
||||
* under-shrink), and it consumes no test slot.
|
||||
*
|
||||
* Applying the factor gate's >=40 date-cluster interval floor to it was the
|
||||
* wrong instrument. The real risk for a calibration layer is that the correction
|
||||
* is DATE-DRIVEN, and leave-one-date-out tests that directly — and harder.
|
||||
*
|
||||
* ── TWO TIERS, AND AUTO-DEMOTION IS WHAT MAKES PROVISIONAL HONEST ────────
|
||||
* PROVISIONAL LODO passes AND the point-in-time held-out CI excludes zero.
|
||||
* Serves, labelled, inside its certified band only.
|
||||
* PROMOTED the original >=40 date-cluster bar, now correctly scoped as the
|
||||
* PROMOTION bar rather than the deploy bar.
|
||||
*
|
||||
* A provisional deploy that cannot be taken away is just a deploy. `reverify`
|
||||
* runs on every newly settled date and demotes on the first breach.
|
||||
*/
|
||||
|
||||
const { knownNumber } = require('../../utils/known');
|
||||
|
||||
const STATUS = Object.freeze({ NONE: 'none', PROVISIONAL: 'provisional', PROMOTED: 'promoted' });
|
||||
/** The ORIGINAL floor, correctly scoped: promotion, not deploy. */
|
||||
const PROMOTION_DATE_CLUSTERS = 40;
|
||||
|
||||
function createRegistry(initial = {}) {
|
||||
const state = new Map(Object.entries(initial));
|
||||
const log = [];
|
||||
|
||||
/** Deploy requires BOTH gates. Neither can be waived. */
|
||||
function deploy(stat, evidence = {}) {
|
||||
const lodo = evidence.lodo_pass === true;
|
||||
const ci = Array.isArray(evidence.ci) && evidence.ci.length === 2 && evidence.ci[1] < 0;
|
||||
if (!lodo || !ci) {
|
||||
return {
|
||||
ok: false,
|
||||
status: STATUS.NONE,
|
||||
reason: !lodo
|
||||
? 'LODO did not pass — the correction may be date-driven'
|
||||
: 'the point-in-time held-out interval does not exclude zero',
|
||||
};
|
||||
}
|
||||
if (!evidence.map) return { ok: false, status: STATUS.NONE, reason: 'no calibration map supplied' };
|
||||
|
||||
state.set(stat, {
|
||||
status: STATUS.PROVISIONAL,
|
||||
map: evidence.map,
|
||||
certified_bands: evidence.certified_bands || [],
|
||||
date_clusters: knownNumber(evidence.date_clusters) ?? 0,
|
||||
ci: evidence.ci,
|
||||
deployed_at: evidence.at || null,
|
||||
});
|
||||
log.push({ stat, event: 'deployed_provisional', at: evidence.at || null });
|
||||
return { ok: true, status: STATUS.PROVISIONAL };
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-verify on a newly settled date. Demotes on the FIRST breach — either the
|
||||
* interval ceasing to exclude zero, or the favourite over-prediction flipping
|
||||
* sign (which would mean the correction is now pushing the wrong way).
|
||||
*/
|
||||
function reverify(stat, obs = {}) {
|
||||
const cur = state.get(stat);
|
||||
if (!cur || cur.status === STATUS.NONE) return { status: STATUS.NONE, changed: false };
|
||||
|
||||
const ciHolds = Array.isArray(obs.ci) && obs.ci.length === 2 && obs.ci[1] < 0;
|
||||
const signHolds = obs.favourite_bias == null ? true : knownNumber(obs.favourite_bias) > 0;
|
||||
|
||||
if (!ciHolds || !signHolds) {
|
||||
state.set(stat, { status: STATUS.NONE, demoted_at: obs.date || null, demoted_reason: !ciHolds ? 'ci_no_longer_excludes_zero' : 'favourite_bias_flipped' });
|
||||
log.push({ stat, event: 'auto_demoted', at: obs.date || null, reason: !ciHolds ? 'ci_no_longer_excludes_zero' : 'favourite_bias_flipped' });
|
||||
return { status: STATUS.NONE, changed: true, reason: !ciHolds ? 'ci_no_longer_excludes_zero' : 'favourite_bias_flipped', breaking_date: obs.date || null };
|
||||
}
|
||||
|
||||
// PROMOTION uses the original floor, now correctly scoped.
|
||||
const dc = knownNumber(obs.date_clusters) ?? cur.date_clusters;
|
||||
if (cur.status === STATUS.PROVISIONAL && dc >= PROMOTION_DATE_CLUSTERS) {
|
||||
state.set(stat, { ...cur, status: STATUS.PROMOTED, date_clusters: dc, promoted_at: obs.date || null });
|
||||
log.push({ stat, event: 'promoted', at: obs.date || null, date_clusters: dc });
|
||||
return { status: STATUS.PROMOTED, changed: true };
|
||||
}
|
||||
if (dc !== cur.date_clusters) state.set(stat, { ...cur, date_clusters: dc });
|
||||
return { status: cur.status, changed: false };
|
||||
}
|
||||
|
||||
/** Is this stat allowed to serve a calibrated number for THIS p_win? */
|
||||
function serves(stat, p) {
|
||||
const cur = state.get(stat);
|
||||
if (!cur || cur.status === STATUS.NONE) return { serve: false, reason: 'not deployed' };
|
||||
const x = knownNumber(p);
|
||||
if (x === null) return { serve: false, reason: 'no p_win' };
|
||||
const inBand = (cur.certified_bands || []).some((b) => x >= b[0] && x < b[1]);
|
||||
if (!inBand) return { serve: false, reason: 'outside the certified band', status: cur.status };
|
||||
return { serve: true, status: cur.status, provisional: cur.status === STATUS.PROVISIONAL };
|
||||
}
|
||||
|
||||
const get = (stat) => state.get(stat) || { status: STATUS.NONE };
|
||||
const all = () => Object.fromEntries([...state.entries()].map(([k, v]) => [k, { status: v.status, certified_bands: v.certified_bands, date_clusters: v.date_clusters }]));
|
||||
|
||||
return { deploy, reverify, serves, get, all, log: () => log.slice() };
|
||||
}
|
||||
|
||||
module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS };
|
||||
@@ -275,6 +275,16 @@ async function loadPitcherArsenals(sport) {
|
||||
} catch { return out; }
|
||||
}
|
||||
|
||||
/**
|
||||
* Stats whose calibration passed leave-one-date-out and may serve a calibrated
|
||||
* number. PROVISIONAL: auto-demoted the first time the held-out interval stops
|
||||
* excluding zero or the favourite over-prediction flips sign.
|
||||
*
|
||||
* hits / rbi / runs are deliberately ABSENT — each fails LODO. See
|
||||
* specs/lodo-provisional-calibration.md.
|
||||
*/
|
||||
const CALIBRATION_DEPLOYED = Object.freeze(['total_bases']);
|
||||
|
||||
async function runSnapshot(sport, opts = {}) {
|
||||
const sp = String(sport || '').toLowerCase();
|
||||
const deps = {
|
||||
@@ -700,37 +710,51 @@ async function runSnapshot(sport, opts = {}) {
|
||||
console.warn(`[challenger] ${sp} skipped:`, e.message);
|
||||
}
|
||||
|
||||
// ── FORWARD CALIBRATION (hits) ────────────────────────────────────────
|
||||
// ── FORWARD CALIBRATION (LODO-gated, per stat) ────────────────────────
|
||||
// Fitted on games that are OVER, applied to tonight's props. `p_win` is NOT
|
||||
// touched — the counter stays byte-identical and the calibrated value rides
|
||||
// beside it, because a calibration map is a correction TO a forecast, not a
|
||||
// different forecast.
|
||||
//
|
||||
// WHICH STATS SERVE IS MEASURED, NOT ASSUMED. The deploy bar is leave-one-
|
||||
// date-out stability: refit dropping each settled date in turn, and the
|
||||
// improvement must never reverse. That is the right instrument for a monotone
|
||||
// shrink-to-observed layer — the factor gate's >=40 date-cluster interval
|
||||
// floor was built for a CAUSAL claim and does not bind here.
|
||||
//
|
||||
// Measured 2026-08-07: total_bases passes at every held-size threshold. hits
|
||||
// FAILS (reverses on 2026-07-22 and 2026-07-26), so it is no longer served
|
||||
// calibrated even though it was — a stat that cannot survive dropping one day
|
||||
// was never calibrated, it was fitted to that day. rbi and runs also fail.
|
||||
//
|
||||
// `calibrated` is true only inside a band certified out-of-sample, and it is
|
||||
// what `chain.chainAcross` requires before it will compound anything. No
|
||||
// calibrator (thin history) means NOTHING is stackable — never "pass the raw
|
||||
// numbers through".
|
||||
// what `chain.chainAcross` requires before it will compound anything. Removing
|
||||
// hits here makes hits props unstackable again, which is the honest
|
||||
// consequence of the measurement rather than a regression to work around.
|
||||
if (sp === 'mlb') {
|
||||
try {
|
||||
const calSvc = deps.calibrationService || require('./model/calibrationService');
|
||||
const sbc = require('../utils/supabase').getSupabaseServiceClient();
|
||||
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat: 'hits' }) : null;
|
||||
if (calibrator) {
|
||||
let marked = 0;
|
||||
for (const g of enriched) {
|
||||
if (String(g.stat_type || g.stat || '').toLowerCase() !== 'hits') continue;
|
||||
const out = calibrator.calibrate(g.p_win);
|
||||
g.p_win_calibrated = out.p_calibrated;
|
||||
g.calibrated = out.calibrated;
|
||||
g.calibration_reason = out.reason;
|
||||
if (out.calibrated) marked += 1;
|
||||
for (const stat of CALIBRATION_DEPLOYED) {
|
||||
try {
|
||||
const calSvc = deps.calibrationService || require('./model/calibrationService');
|
||||
const sbc = require('../utils/supabase').getSupabaseServiceClient();
|
||||
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat }) : null;
|
||||
if (calibrator) {
|
||||
let marked = 0;
|
||||
for (const g of enriched) {
|
||||
if (String(g.stat_type || g.stat || '').toLowerCase() !== stat) continue;
|
||||
const out = calibrator.calibrate(g.p_win);
|
||||
g.p_win_calibrated = out.p_calibrated;
|
||||
g.calibrated = out.calibrated;
|
||||
g.calibration_reason = out.reason;
|
||||
g.calibration_status = 'provisional';
|
||||
if (out.calibrated) marked += 1;
|
||||
}
|
||||
console.log(`[calibration] ${sp} ${stat} (PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}`);
|
||||
} else {
|
||||
console.log(`[calibration] ${sp} ${stat} — no calibrator (thin history); nothing is stackable`);
|
||||
}
|
||||
console.log(`[calibration] ${sp} hits — ${marked} stackable of ${enriched.filter((g) => String(g.stat_type || g.stat || '').toLowerCase() === 'hits').length}; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, bands ${JSON.stringify(calibrator.bands.map((b) => [b.lo, b.hi]))}`);
|
||||
} else {
|
||||
console.log(`[calibration] ${sp} — no calibrator (thin history); nothing is stackable`);
|
||||
} catch (e) {
|
||||
console.warn(`[calibration] ${stat} skipped:`, e.message);
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn('[calibration] skipped:', e.message);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -843,5 +867,6 @@ module.exports = {
|
||||
generateTickerEvents,
|
||||
pushTickerItems,
|
||||
ACTIVE_SPORTS,
|
||||
CALIBRATION_DEPLOYED,
|
||||
__internals: { propKey, gradedAtFor, indexOdds, lastName, isTopGrade, DELTA_NOISE, DELTA_MOVE, TICKER_CAP },
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user