builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Calibrate hits point-in-time: partial pass, and an honest ceiling of 0.667

Browse Source 
Fitted the isotonic map on game_date < 2026-08-02 (n=589) and evaluated it on
everything from that date forward (n=383). The map never saw the evaluation
rows, which is the only thing that makes the result mean anything -- fitting
and evaluating on the same rows always looks perfectly calibrated, because the
map is reciting the answers it was built from.

It works, on most of the distribution. Held-out after correction: 0.477 comes
back 0.506, 0.587 comes back 0.580, 0.667 comes back 0.603 -- against raw
errors of +0.191, +0.279 and +0.246 in the same bins. Ordering survived, and
that was verified pairwise rather than assumed, because a broken map would
silently destroy the one thing this model does well.

Two findings matter more than the pass.

First, the honest ceiling is 0.667. Once the numbers are truthful this model
has no 80%-plus hit reads at all -- the top of its range was miscalibration,
not confidence. A four-leg ticket at the ceiling is 0.198, where the raw
numbers implied 0.686. The high-floor parlay is a two-thirds-per-leg
proposition, and that is the number to say out loud.

Second, calibration is certified BY BAND rather than by a blanket flag.
Held-out error was -0.029 and +0.007 through the middle but -0.167 at the
bottom and +0.063 at the top: the model is trustworthy over most of its mass
and untrustworthy at both edges. A single true/false would either throw away
the 72% that works or ship the edges that do not. Only a probability inside a
certified band is marked stackable, and that flag is what chainAcross requires
before it will compound anything. The certified band is 0.40 to 0.60, n=276.

A methodological catch on the way: my first pass condition demanded honest bins
at 0.70 and above -- but honest calibration REMOVES those bins, since the
ceiling drops to 0.667. The gate would have failed the repair for succeeding.
It now tests the highest remaining band instead of a fixed threshold.

Wired forward with the same discipline: calibrationService fits strictly before
today, splits by time rather than at random, and returns null on thin history
so that "no calibrator" means nothing is stackable rather than "trust the raw
numbers". p_win is never mutated -- the calibrated value rides beside it as
p_win_calibrated, because a calibration map is a correction to a forecast, not
a different forecast, and the counter stays byte-identical.

4,275 tests green (339 suites); web build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-04 17:51:15 -04:00
parent 9c5b968351
commit 4d1803f6d7
7 changed files with 477 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/model/calibration.js
+57
View File
@@ -150,7 +150,64 @@ function applyIsotonic(map, p) {
return map[map.length - 1].value;
}
/**
* CERTIFY BANDS — where is this model honest, and where is it not?
*
* A blanket "calibrated: true/false" is the wrong shape for the decision it
* feeds. Measured on held-out hits after an isotonic fit:
*
* 0.477 -> 0.506 (err -0.029, n=83) honest
* 0.587 -> 0.580 (err +0.007, n=193) honest
* 0.667 -> 0.603 (err +0.063, n=63) NOT honest
* 0.333 -> 0.500 (err -0.167, n=30) NOT honest
*
* The model is trustworthy over most of its mass and untrustworthy at both
* edges. Rejecting it wholesale throws away the 72% that works; accepting it
* wholesale ships the edges that do not. So certification is BY BAND, and a leg
* outside a certified band is simply not stackable.
*
* Bands are built only from bins with real sample (`minBin`), so a band is never
* certified on a handful of rows.
*/
function certifyBands(heldOutRows, opts = {}) {
const tol = opts.tolerance ?? MAX_BIN_ERROR;
const minBin = opts.minBin ?? 40;
const table = reliability(heldOutRows, opts.bins ?? 10);
const bands = [];
for (const b of table) {
if (b.n < minBin) continue;
if (Math.abs(b.error) > tol) continue;
const width = 1 / (opts.bins ?? 10);
bands.push({
lo: Math.max(0, b.bin * width),
hi: Math.min(1, (b.bin + 1) * width),
observed_error: Math.round(b.error * 1000) / 1000,
n: b.n,
});
}
// Merge adjacent certified bands so the result reads as a range, not a fence.
bands.sort((a, b) => a.lo - b.lo);
const merged = [];
for (const b of bands) {
const last = merged[merged.length - 1];
if (last && Math.abs(b.lo - last.hi) < 1e-9) {
last.hi = b.hi; last.n += b.n;
last.observed_error = Math.abs(b.observed_error) > Math.abs(last.observed_error)
? b.observed_error : last.observed_error;
} else merged.push({ ...b });
}
return merged;
}
/** Is a single probability inside a certified band? */
function inCertifiedBand(bands, p) {
const x = knownNumber(p);
if (x === null || !Array.isArray(bands)) return false;
return bands.some((b) => x >= b.lo && x <= b.hi);
}
module.exports = {
reliability, isCalibrated, fitIsotonic, applyIsotonic,
certifyBands, inCertifiedBand,
MIN_BIN, MIN_TOTAL, MAX_BIN_ERROR,
};
src/services/model/calibrationService.js
+128
View File
@@ -0,0 +1,128 @@
'use strict';
/**
* calibrationService — FIT ON HISTORY, APPLY TO TONIGHT.
*
* The backtest proved the repair works out-of-sample; this is the same
* discipline running forward in production. The map is fitted ONLY on outcomes
* that have already settled, and applied to props that have not been played. If
* that cut is ever relaxed, the map has seen the answer and every number it
* produces is fiction that will look excellent in review.
*
* ── WHAT THE BACKTEST ESTABLISHED (2026-08-04, hits) ─────────────────────
* Fitted on game_date < 2026-08-02 (n=589), evaluated on >= (n=383):
*
* raw 0.746 -> 0.556 0.843 -> 0.564 0.913 -> 0.667
* calibrated 0.477 -> 0.506 0.587 -> 0.580 0.667 -> 0.603
*
* Ordering survived (isotonic is monotone — verified, not assumed). Errors at
* the top fell from +0.28 to +0.06. And the honest CEILING dropped to 0.667:
* once the numbers are truthful, this model has no 80%+ hit reads at all. The
* "high-floor parlay" is a 0.667-per-leg proposition, not a 0.9 one.
*
* ── CERTIFICATION IS BY BAND, NOT A BLANKET FLAG ─────────────────────────
* Held-out errors were −0.029 and +0.007 through the middle, but −0.167 at the
* bottom and +0.063 at the top. The model is trustworthy over most of its mass
* and untrustworthy at both edges, so a single true/false would either throw
* away the 72% that works or ship the edges that do not. Only a probability
* inside a certified band is marked stackable.
*/
const cal = require('./calibration');
const { knownNumber } = require('../../utils/known');
/** Default: hold out the most recent quarter of history to certify on. */
const HOLDOUT_FRACTION = 0.35;
const MIN_FIT = 200;
/**
* Build a calibrator from settled rows.
*
* @param {Array<{p, won, date}>} settled rows STRICTLY BEFORE the props being graded
* @returns {object|null} null when there is not enough history — the caller must
* then treat every atom as uncalibrated rather than pass it through raw.
*/
function build(settled, opts = {}) {
const rows = (settled || [])
.map((r) => ({ p: knownNumber(r && r.p), won: knownNumber(r && r.won), d: String((r && r.date) || '') }))
.filter((r) => r.p !== null && r.won !== null)
.sort((a, b) => (a.d < b.d ? -1 : a.d > b.d ? 1 : 0));
if (rows.length < (opts.minFit ?? MIN_FIT)) return null;
// Split by TIME, not at random: certifying on rows the map was fitted on
// always looks perfect, and a random split leaks the future into the fit.
const cut = Math.floor(rows.length * (1 - (opts.holdoutFraction ?? HOLDOUT_FRACTION)));
const fitRows = rows.slice(0, cut);
const certRows = rows.slice(cut);
if (fitRows.length < (opts.minFit ?? MIN_FIT) || certRows.length < 60) return null;
const map = cal.fitIsotonic(fitRows, { minTotal: opts.minFit ?? MIN_FIT });
if (!map) return null;
const corrected = certRows.map((r) => ({ ...r, p: cal.applyIsotonic(map, r.p) }));
const bands = cal.certifyBands(corrected, {
tolerance: opts.tolerance ?? 0.05,
minBin: opts.minBin ?? 40,
});
return {
map,
bands,
fit_n: fitRows.length,
certify_n: certRows.length,
fitted_through: fitRows[fitRows.length - 1].d,
certified_through: certRows[certRows.length - 1].d,
/**
* Calibrate one probability.
* `calibrated` is TRUE only inside a certified band — that flag is what
* `chain.chainAcross` requires before it will compound anything.
*/
calibrate(p) {
const raw = knownNumber(p);
if (raw === null) return { p_raw: null, p_calibrated: null, calibrated: false, reason: 'absent' };
const c = cal.applyIsotonic(map, raw);
if (c === null) return { p_raw: raw, p_calibrated: null, calibrated: false, reason: 'no_map_value' };
const inBand = cal.inCertifiedBand(bands, c);
return {
p_raw: raw,
p_calibrated: Math.round(c * 1000) / 1000,
calibrated: inBand,
reason: inBand ? null : 'outside_certified_band',
};
},
};
}
/**
* Load settled history from the ledger and build a calibrator, POINT-IN-TIME.
*
* `before` defaults to today, so the fit can only ever use games that are over.
* Injectable for tests; returns null rather than a permissive fallback, because
* "no calibrator" must mean "nothing is stackable", not "pass the raw numbers
* through".
*/
async function fromLedger(sb, { sport = 'mlb', stat = 'hits', before = null, ...opts } = {}) {
if (!sb) return null;
const cutoff = before || new Intl.DateTimeFormat('en-CA', {
timeZone: 'America/New_York', year: 'numeric', month: '2-digit', day: '2-digit',
}).format(new Date());
const rows = [];
for (let from = 0; ; from += 1000) {
const { data, error } = await sb.from('ledger_entries')
.select('p_win, outcome, game_date, quarantine_reason')
.eq('sport', sport).is('user_id', null).eq('stat', stat)
.in('outcome', ['hit', 'miss']).not('p_win', 'is', null)
.lt('game_date', cutoff) // STRICTLY before — the whole point
.range(from, from + 999);
if (error || !data || data.length === 0) break;
rows.push(...data);
if (data.length < 1000) break;
}
const clean = rows
.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book'))
.map((r) => ({ p: Number(r.p_win), won: r.outcome === 'hit' ? 1 : 0, date: String(r.game_date) }));
const built = build(clean, opts);
return built ? { ...built, cutoff } : null;
}
module.exports = { build, fromLedger, HOLDOUT_FRACTION, MIN_FIT };
src/services/snapshotService.js
+34
View File
@@ -700,6 +700,40 @@ async function runSnapshot(sport, opts = {}) {
console.warn(`[challenger] ${sp} skipped:`, e.message);
}
// ── FORWARD CALIBRATION (hits) ────────────────────────────────────────
// Fitted on games that are OVER, applied to tonight's props. `p_win` is NOT
// touched — the counter stays byte-identical and the calibrated value rides
// beside it, because a calibration map is a correction TO a forecast, not a
// different forecast.
//
// `calibrated` is true only inside a band certified out-of-sample, and it is
// what `chain.chainAcross` requires before it will compound anything. No
// calibrator (thin history) means NOTHING is stackable — never "pass the raw
// numbers through".
if (sp === 'mlb') {
try {
const calSvc = deps.calibrationService || require('./model/calibrationService');
const sbc = require('../utils/supabase').getSupabaseServiceClient();
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat: 'hits' }) : null;
if (calibrator) {
let marked = 0;
for (const g of enriched) {
if (String(g.stat_type || g.stat || '').toLowerCase() !== 'hits') continue;
const out = calibrator.calibrate(g.p_win);
g.p_win_calibrated = out.p_calibrated;
g.calibrated = out.calibrated;
g.calibration_reason = out.reason;
if (out.calibrated) marked += 1;
}
console.log(`[calibration] ${sp} hits — ${marked} stackable of ${enriched.filter((g) => String(g.stat_type || g.stat || '').toLowerCase() === 'hits').length}; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, bands ${JSON.stringify(calibrator.bands.map((b) => [b.lo, b.hi]))}`);
} else {
console.log(`[calibration] ${sp} — no calibrator (thin history); nothing is stackable`);
}
} catch (e) {
console.warn('[calibration] skipped:', e.message);
}
}
// LINEUP + BASERUNNER CONTEXT — the input RBI and runs have always needed.
// Best-effort and dated: a context failure must never break a snapshot, and a
// lineup is a PRE-GAME fact that changes by the hour, so what we knew at grade