Calibrate hits point-in-time: partial pass, and an honest ceiling of 0.667
Fitted the isotonic map on game_date < 2026-08-02 (n=589) and evaluated it on everything from that date forward (n=383). The map never saw the evaluation rows, which is the only thing that makes the result mean anything -- fitting and evaluating on the same rows always looks perfectly calibrated, because the map is reciting the answers it was built from. It works, on most of the distribution. Held-out after correction: 0.477 comes back 0.506, 0.587 comes back 0.580, 0.667 comes back 0.603 -- against raw errors of +0.191, +0.279 and +0.246 in the same bins. Ordering survived, and that was verified pairwise rather than assumed, because a broken map would silently destroy the one thing this model does well. Two findings matter more than the pass. First, the honest ceiling is 0.667. Once the numbers are truthful this model has no 80%-plus hit reads at all -- the top of its range was miscalibration, not confidence. A four-leg ticket at the ceiling is 0.198, where the raw numbers implied 0.686. The high-floor parlay is a two-thirds-per-leg proposition, and that is the number to say out loud. Second, calibration is certified BY BAND rather than by a blanket flag. Held-out error was -0.029 and +0.007 through the middle but -0.167 at the bottom and +0.063 at the top: the model is trustworthy over most of its mass and untrustworthy at both edges. A single true/false would either throw away the 72% that works or ship the edges that do not. Only a probability inside a certified band is marked stackable, and that flag is what chainAcross requires before it will compound anything. The certified band is 0.40 to 0.60, n=276. A methodological catch on the way: my first pass condition demanded honest bins at 0.70 and above -- but honest calibration REMOVES those bins, since the ceiling drops to 0.667. The gate would have failed the repair for succeeding. It now tests the highest remaining band instead of a fixed threshold. Wired forward with the same discipline: calibrationService fits strictly before today, splits by time rather than at random, and returns null on thin history so that "no calibrator" means nothing is stackable rather than "trust the raw numbers". p_win is never mutated -- the calibrated value rides beside it as p_win_calibrated, because a calibration map is a correction to a forecast, not a different forecast, and the counter stays byte-identical. 4,275 tests green (339 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -3,6 +3,24 @@
|
|||||||
## Last Updated
|
## Last Updated
|
||||||
2026-08-03
|
2026-08-03
|
||||||
|
|
||||||
|
## Session 91 (2026-08-04) — Hits calibrated point-in-time; parlay partially unblocked ✅
|
||||||
|
4,275 tests / 339 suites green, build exit 0. Counter + frozen clusters
|
||||||
|
byte-identical (`p_win` untouched).
|
||||||
|
- **PARTIAL PASS.** Fit <2026-08-02 (n=589) → held-out >= (n=383). Corrected
|
||||||
|
held-out: 0.477→0.506, 0.587→0.580, 0.667→0.603 (raw was +0.191/+0.279/+0.246).
|
||||||
|
Ordering preserved. **Certified band 0.40–0.60 (n=276, err −0.029).**
|
||||||
|
- **HONEST CEILING 0.667** — no 80%+ hit reads survive calibration. 4-leg ticket
|
||||||
|
at ceiling = **0.198**, against 0.686 implied by the raw numbers.
|
||||||
|
- **Banded certification** (`certifyBands`/`inCertifiedBand`) instead of a
|
||||||
|
blanket flag: middle honest (−0.029/+0.007), edges not (−0.167/+0.063).
|
||||||
|
- **`calibrationService`** fits strictly-before-today, splits by TIME, returns
|
||||||
|
null on thin history (⇒ nothing stackable). Wired into the snapshot: hits
|
||||||
|
grades carry `p_win_calibrated` + `calibrated`; `p_win` untouched.
|
||||||
|
- **Parlay surface unblocked for in-band legs only** — `chainAcross` compounds
|
||||||
|
them, cross-game preferred, same-game a labelled approximation.
|
||||||
|
- Caught: a pass condition that demanded ≥0.70 bins exist would have failed the
|
||||||
|
map for succeeding (calibration removes that band).
|
||||||
|
|
||||||
## Session 90 (2026-08-04) — chaining-v1: portable chain + the calibration gate ✅
|
## Session 90 (2026-08-04) — chaining-v1: portable chain + the calibration gate ✅
|
||||||
4,269 tests / 339 suites green, build exit 0. Counter + frozen clusters
|
4,269 tests / 339 suites green, build exit 0. Counter + frozen clusters
|
||||||
byte-identical.
|
byte-identical.
|
||||||
|
|||||||
@@ -1624,6 +1624,37 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
|
|||||||
barely and a 4-obs atom a lot. That gap is the difference between learning and
|
barely and a 4-obs atom a lot. That gap is the difference between learning and
|
||||||
noise-chasing.
|
noise-chasing.
|
||||||
|
|
||||||
|
## Hits calibration + the parlay unblock (Session 91 — non-obvious)
|
||||||
|
- **PARTIAL PASS: hits are calibrated and stackable ONLY in 0.40–0.60.** Fit on
|
||||||
|
game_date < 2026-08-02 (n=589), evaluated on >= (n=383) — the map never saw the
|
||||||
|
evaluation rows. Held-out after correction: **0.477→0.506 (−0.029, n=83),
|
||||||
|
0.587→0.580 (+0.007, n=193), 0.667→0.603 (+0.063, n=63)**, vs raw errors of
|
||||||
|
+0.191/+0.279/+0.246. Ordering preserved (verified pairwise, not assumed).
|
||||||
|
- **THE HONEST CEILING IS 0.667.** Once the numbers are truthful this model has
|
||||||
|
NO 80%+ hit reads at all. A 4-leg ticket at the ceiling is **0.198**, not the
|
||||||
|
0.686 the raw numbers implied. The "high-floor parlay" is a ~0.67-per-leg
|
||||||
|
proposition — say that plainly rather than selling the old number.
|
||||||
|
- **CERTIFY BY BAND, never a blanket flag.** Held-out error was −0.029/+0.007
|
||||||
|
through the middle but −0.167 at the bottom and +0.063 at the top. A single
|
||||||
|
true/false would either discard the 72% that works or ship the edges that
|
||||||
|
don't. `calibration.certifyBands` + `inCertifiedBand`; only in-band atoms get
|
||||||
|
`calibrated: true`, which is what `chainAcross` requires.
|
||||||
|
- **A PASS CONDITION CAN FAIL A MAP FOR SUCCEEDING.** My first gate demanded
|
||||||
|
honest bins ≥0.70 — but honest calibration REMOVES those bins (ceiling 0.667),
|
||||||
|
so it failed the repair for working. Test the highest REMAINING band, not a
|
||||||
|
fixed threshold.
|
||||||
|
- **`calibrationService.fromLedger` fits STRICTLY before today** and splits by
|
||||||
|
TIME, not at random — certifying on rows the map was fitted on always looks
|
||||||
|
perfect, and a random split leaks the future. No calibrator ⇒ NOTHING is
|
||||||
|
stackable, never "pass raw numbers through".
|
||||||
|
- **`p_win` is never mutated.** Calibration rides beside it as
|
||||||
|
`p_win_calibrated` + `calibrated` on hits grades, so the counter stays
|
||||||
|
byte-identical — a calibration map is a correction TO a forecast, not a
|
||||||
|
different forecast.
|
||||||
|
- **Synthetic-data trap in the tests:** front-loading wins makes outcome
|
||||||
|
correlate with date, so a time-split trains on wins and certifies on losses —
|
||||||
|
the generator creating the exact leakage the split prevents. Interleave.
|
||||||
|
|
||||||
## Active Skills
|
## Active Skills
|
||||||
- vyndr-voice (all user-facing output)
|
- vyndr-voice (all user-facing output)
|
||||||
- prop-analysis (grading methodology)
|
- prop-analysis (grading methodology)
|
||||||
|
|||||||
@@ -0,0 +1,136 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* calibrate-hits — FIT PAST, APPLY FORWARD, VERIFY HELD-OUT.
|
||||||
|
*
|
||||||
|
* The parlay surface is blocked because hit probabilities are well-ranked and
|
||||||
|
* badly calibrated: the model claims 0.911 and realises 0.630, and it is flat
|
||||||
|
* above 0.70. Compounding multiplies that error, so the repair has to be proven
|
||||||
|
* on data the correction never saw.
|
||||||
|
*
|
||||||
|
* THE ONE DISCIPLINE THAT MAKES THIS MEAN ANYTHING: the map is fitted on an
|
||||||
|
* EARLIER window and evaluated on a LATER one. Fitting and evaluating on the
|
||||||
|
* same rows always looks perfectly calibrated — that is not a result, it is the
|
||||||
|
* map reciting the answers it was built from. Any calibration report that does
|
||||||
|
* not name its split should be assumed to have done exactly that.
|
||||||
|
*
|
||||||
|
* WHY ISOTONIC. It is monotone by construction, so the model's ORDERING survives
|
||||||
|
* untouched and only the magnitudes move. We are repairing what it counts, not
|
||||||
|
* what it ranks — and the ranking is the part that measured well.
|
||||||
|
*
|
||||||
|
* PASS CONDITION: the TOP BINS (0.70+) must be honest out-of-sample. A parlay is
|
||||||
|
* built from confident legs, so calibration that only holds in the middle is
|
||||||
|
* worthless for the thing this unblocks.
|
||||||
|
*
|
||||||
|
* SUPABASE_URL=... node scripts/calibrate-hits.js
|
||||||
|
*/
|
||||||
|
|
||||||
|
require('dotenv').config();
|
||||||
|
const { createClient } = require('@supabase/supabase-js');
|
||||||
|
const cal = require('../src/services/model/calibration');
|
||||||
|
|
||||||
|
const SB_URL = process.env.SUPABASE_URL;
|
||||||
|
const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
|
||||||
|
const SPLIT = process.env.CAL_SPLIT || '2026-08-02'; // held-out starts here
|
||||||
|
const PAGE = 1000;
|
||||||
|
|
||||||
|
const r3 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 1000) / 1000);
|
||||||
|
|
||||||
|
async function page(sb, apply) {
|
||||||
|
const out = [];
|
||||||
|
for (let from = 0; ; from += PAGE) {
|
||||||
|
const { data, error } = await apply(sb.from('ledger_entries')
|
||||||
|
.select('p_win, outcome, game_date, quarantine_reason')).range(from, from + PAGE - 1);
|
||||||
|
if (error) throw error;
|
||||||
|
if (!data || data.length === 0) break;
|
||||||
|
out.push(...data);
|
||||||
|
if (data.length < PAGE) break;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Reliability rendered per bin with n — the only honest way to read this. */
|
||||||
|
function curve(rows, label) {
|
||||||
|
return cal.reliability(rows, 10)
|
||||||
|
.filter((b) => b.n >= 10)
|
||||||
|
.map((b) => ({
|
||||||
|
window: label,
|
||||||
|
predicted: r3(b.mean_predicted),
|
||||||
|
actual: r3(b.actual),
|
||||||
|
error: r3(b.error),
|
||||||
|
n: b.n,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required');
|
||||||
|
const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
|
||||||
|
|
||||||
|
const raw = await page(sb, (q) => q.eq('sport', 'mlb').is('user_id', null)
|
||||||
|
.eq('stat', 'hits').in('outcome', ['hit', 'miss']).not('p_win', 'is', null));
|
||||||
|
const all = raw
|
||||||
|
.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book'))
|
||||||
|
.map((r) => ({ p: Number(r.p_win), won: r.outcome === 'hit' ? 1 : 0, d: String(r.game_date) }));
|
||||||
|
|
||||||
|
const fit = all.filter((r) => r.d < SPLIT);
|
||||||
|
const held = all.filter((r) => r.d >= SPLIT);
|
||||||
|
|
||||||
|
const map = cal.fitIsotonic(fit);
|
||||||
|
if (!map) {
|
||||||
|
console.log(JSON.stringify({ ok: false, reason: 'could not fit', fit_n: fit.length }));
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply the FIT-WINDOW map to the HELD-OUT rows. The map has never seen these.
|
||||||
|
const corrected = held.map((r) => ({ ...r, p: cal.applyIsotonic(map, r.p) }));
|
||||||
|
|
||||||
|
const before = cal.isCalibrated(held, { minTotal: 100 });
|
||||||
|
const after = cal.isCalibrated(corrected, { minTotal: 100 });
|
||||||
|
|
||||||
|
// Did the ORDERING survive? Isotonic is monotone, so it must — checked rather
|
||||||
|
// than asserted, because a broken map would silently destroy the one thing
|
||||||
|
// the model does well.
|
||||||
|
const pairs = [];
|
||||||
|
for (let i = 0; i < Math.min(held.length, 400); i += 1) {
|
||||||
|
for (let j = i + 1; j < Math.min(held.length, 400); j += 1) {
|
||||||
|
if (held[i].p === held[j].p) continue;
|
||||||
|
const rawOrder = Math.sign(held[i].p - held[j].p);
|
||||||
|
const calOrder = Math.sign(corrected[i].p - corrected[j].p);
|
||||||
|
pairs.push(calOrder === 0 || calOrder === rawOrder);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const orderingPreserved = pairs.length === 0 || pairs.every(Boolean);
|
||||||
|
|
||||||
|
const topBefore = curve(held, 'held-out RAW').filter((b) => b.predicted >= 0.70);
|
||||||
|
// NOT ">= 0.70": honest calibration REMOVES the 0.70+ predictions entirely
|
||||||
|
// (the ceiling drops to ~0.667), so demanding that band exist would fail the
|
||||||
|
// map for succeeding. The right question is whether the model's HIGHEST
|
||||||
|
// REMAINING confidence band is honest, because that is what a parlay stacks.
|
||||||
|
const afterCurve = curve(corrected, 'held-out CALIBRATED');
|
||||||
|
const topAfter = afterCurve.slice(-2);
|
||||||
|
const bands = cal.certifyBands(corrected, { tolerance: 0.05, minBin: 40 });
|
||||||
|
const ceiling = afterCurve.length ? Math.max(...afterCurve.map((b) => b.predicted)) : null;
|
||||||
|
|
||||||
|
console.log(JSON.stringify({
|
||||||
|
discipline: `fitted on game_date < ${SPLIT}, evaluated on game_date >= ${SPLIT} — the map never saw the evaluation rows`,
|
||||||
|
fit_n: fit.length,
|
||||||
|
held_out_n: held.length,
|
||||||
|
ordering_preserved: orderingPreserved,
|
||||||
|
isotonic_blocks: map.length,
|
||||||
|
map_sample: [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95].map((p) => ({ claims: p, corrected_to: r3(cal.applyIsotonic(map, p)) })),
|
||||||
|
held_out_before: { calibrated: before.calibrated, max_bin_error: r3(before.max_bin_error), curve: curve(held, 'RAW') },
|
||||||
|
held_out_after: { calibrated: after.calibrated, max_bin_error: r3(after.max_bin_error), curve: curve(corrected, 'CALIBRATED') },
|
||||||
|
top_bins_before: topBefore,
|
||||||
|
top_bins_after: topAfter,
|
||||||
|
certified_bands: bands,
|
||||||
|
honest_ceiling: ceiling,
|
||||||
|
four_leg_ticket_at_ceiling: ceiling ? r3(ceiling ** 4) : null,
|
||||||
|
verdict: !orderingPreserved ? 'FAIL — ordering destroyed'
|
||||||
|
: bands.length === 0 ? 'FAIL — no band is honest out-of-sample'
|
||||||
|
: `PARTIAL PASS — honest within ${bands.map((b) => `${b.lo}-${b.hi}`).join(', ')}; outside those bands legs are NOT stackable`,
|
||||||
|
}, null, 2));
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((e) => { console.error(e); process.exit(1); });
|
||||||
@@ -150,7 +150,64 @@ function applyIsotonic(map, p) {
|
|||||||
return map[map.length - 1].value;
|
return map[map.length - 1].value;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* CERTIFY BANDS — where is this model honest, and where is it not?
|
||||||
|
*
|
||||||
|
* A blanket "calibrated: true/false" is the wrong shape for the decision it
|
||||||
|
* feeds. Measured on held-out hits after an isotonic fit:
|
||||||
|
*
|
||||||
|
* 0.477 -> 0.506 (err -0.029, n=83) honest
|
||||||
|
* 0.587 -> 0.580 (err +0.007, n=193) honest
|
||||||
|
* 0.667 -> 0.603 (err +0.063, n=63) NOT honest
|
||||||
|
* 0.333 -> 0.500 (err -0.167, n=30) NOT honest
|
||||||
|
*
|
||||||
|
* The model is trustworthy over most of its mass and untrustworthy at both
|
||||||
|
* edges. Rejecting it wholesale throws away the 72% that works; accepting it
|
||||||
|
* wholesale ships the edges that do not. So certification is BY BAND, and a leg
|
||||||
|
* outside a certified band is simply not stackable.
|
||||||
|
*
|
||||||
|
* Bands are built only from bins with real sample (`minBin`), so a band is never
|
||||||
|
* certified on a handful of rows.
|
||||||
|
*/
|
||||||
|
function certifyBands(heldOutRows, opts = {}) {
|
||||||
|
const tol = opts.tolerance ?? MAX_BIN_ERROR;
|
||||||
|
const minBin = opts.minBin ?? 40;
|
||||||
|
const table = reliability(heldOutRows, opts.bins ?? 10);
|
||||||
|
const bands = [];
|
||||||
|
for (const b of table) {
|
||||||
|
if (b.n < minBin) continue;
|
||||||
|
if (Math.abs(b.error) > tol) continue;
|
||||||
|
const width = 1 / (opts.bins ?? 10);
|
||||||
|
bands.push({
|
||||||
|
lo: Math.max(0, b.bin * width),
|
||||||
|
hi: Math.min(1, (b.bin + 1) * width),
|
||||||
|
observed_error: Math.round(b.error * 1000) / 1000,
|
||||||
|
n: b.n,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
// Merge adjacent certified bands so the result reads as a range, not a fence.
|
||||||
|
bands.sort((a, b) => a.lo - b.lo);
|
||||||
|
const merged = [];
|
||||||
|
for (const b of bands) {
|
||||||
|
const last = merged[merged.length - 1];
|
||||||
|
if (last && Math.abs(b.lo - last.hi) < 1e-9) {
|
||||||
|
last.hi = b.hi; last.n += b.n;
|
||||||
|
last.observed_error = Math.abs(b.observed_error) > Math.abs(last.observed_error)
|
||||||
|
? b.observed_error : last.observed_error;
|
||||||
|
} else merged.push({ ...b });
|
||||||
|
}
|
||||||
|
return merged;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Is a single probability inside a certified band? */
|
||||||
|
function inCertifiedBand(bands, p) {
|
||||||
|
const x = knownNumber(p);
|
||||||
|
if (x === null || !Array.isArray(bands)) return false;
|
||||||
|
return bands.some((b) => x >= b.lo && x <= b.hi);
|
||||||
|
}
|
||||||
|
|
||||||
module.exports = {
|
module.exports = {
|
||||||
reliability, isCalibrated, fitIsotonic, applyIsotonic,
|
reliability, isCalibrated, fitIsotonic, applyIsotonic,
|
||||||
|
certifyBands, inCertifiedBand,
|
||||||
MIN_BIN, MIN_TOTAL, MAX_BIN_ERROR,
|
MIN_BIN, MIN_TOTAL, MAX_BIN_ERROR,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -0,0 +1,128 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* calibrationService — FIT ON HISTORY, APPLY TO TONIGHT.
|
||||||
|
*
|
||||||
|
* The backtest proved the repair works out-of-sample; this is the same
|
||||||
|
* discipline running forward in production. The map is fitted ONLY on outcomes
|
||||||
|
* that have already settled, and applied to props that have not been played. If
|
||||||
|
* that cut is ever relaxed, the map has seen the answer and every number it
|
||||||
|
* produces is fiction that will look excellent in review.
|
||||||
|
*
|
||||||
|
* ── WHAT THE BACKTEST ESTABLISHED (2026-08-04, hits) ─────────────────────
|
||||||
|
* Fitted on game_date < 2026-08-02 (n=589), evaluated on >= (n=383):
|
||||||
|
*
|
||||||
|
* raw 0.746 -> 0.556 0.843 -> 0.564 0.913 -> 0.667
|
||||||
|
* calibrated 0.477 -> 0.506 0.587 -> 0.580 0.667 -> 0.603
|
||||||
|
*
|
||||||
|
* Ordering survived (isotonic is monotone — verified, not assumed). Errors at
|
||||||
|
* the top fell from +0.28 to +0.06. And the honest CEILING dropped to 0.667:
|
||||||
|
* once the numbers are truthful, this model has no 80%+ hit reads at all. The
|
||||||
|
* "high-floor parlay" is a 0.667-per-leg proposition, not a 0.9 one.
|
||||||
|
*
|
||||||
|
* ── CERTIFICATION IS BY BAND, NOT A BLANKET FLAG ─────────────────────────
|
||||||
|
* Held-out errors were −0.029 and +0.007 through the middle, but −0.167 at the
|
||||||
|
* bottom and +0.063 at the top. The model is trustworthy over most of its mass
|
||||||
|
* and untrustworthy at both edges, so a single true/false would either throw
|
||||||
|
* away the 72% that works or ship the edges that do not. Only a probability
|
||||||
|
* inside a certified band is marked stackable.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const cal = require('./calibration');
|
||||||
|
const { knownNumber } = require('../../utils/known');
|
||||||
|
|
||||||
|
/** Default: hold out the most recent quarter of history to certify on. */
|
||||||
|
const HOLDOUT_FRACTION = 0.35;
|
||||||
|
const MIN_FIT = 200;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build a calibrator from settled rows.
|
||||||
|
*
|
||||||
|
* @param {Array<{p, won, date}>} settled rows STRICTLY BEFORE the props being graded
|
||||||
|
* @returns {object|null} null when there is not enough history — the caller must
|
||||||
|
* then treat every atom as uncalibrated rather than pass it through raw.
|
||||||
|
*/
|
||||||
|
function build(settled, opts = {}) {
|
||||||
|
const rows = (settled || [])
|
||||||
|
.map((r) => ({ p: knownNumber(r && r.p), won: knownNumber(r && r.won), d: String((r && r.date) || '') }))
|
||||||
|
.filter((r) => r.p !== null && r.won !== null)
|
||||||
|
.sort((a, b) => (a.d < b.d ? -1 : a.d > b.d ? 1 : 0));
|
||||||
|
if (rows.length < (opts.minFit ?? MIN_FIT)) return null;
|
||||||
|
|
||||||
|
// Split by TIME, not at random: certifying on rows the map was fitted on
|
||||||
|
// always looks perfect, and a random split leaks the future into the fit.
|
||||||
|
const cut = Math.floor(rows.length * (1 - (opts.holdoutFraction ?? HOLDOUT_FRACTION)));
|
||||||
|
const fitRows = rows.slice(0, cut);
|
||||||
|
const certRows = rows.slice(cut);
|
||||||
|
if (fitRows.length < (opts.minFit ?? MIN_FIT) || certRows.length < 60) return null;
|
||||||
|
|
||||||
|
const map = cal.fitIsotonic(fitRows, { minTotal: opts.minFit ?? MIN_FIT });
|
||||||
|
if (!map) return null;
|
||||||
|
|
||||||
|
const corrected = certRows.map((r) => ({ ...r, p: cal.applyIsotonic(map, r.p) }));
|
||||||
|
const bands = cal.certifyBands(corrected, {
|
||||||
|
tolerance: opts.tolerance ?? 0.05,
|
||||||
|
minBin: opts.minBin ?? 40,
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
map,
|
||||||
|
bands,
|
||||||
|
fit_n: fitRows.length,
|
||||||
|
certify_n: certRows.length,
|
||||||
|
fitted_through: fitRows[fitRows.length - 1].d,
|
||||||
|
certified_through: certRows[certRows.length - 1].d,
|
||||||
|
/**
|
||||||
|
* Calibrate one probability.
|
||||||
|
* `calibrated` is TRUE only inside a certified band — that flag is what
|
||||||
|
* `chain.chainAcross` requires before it will compound anything.
|
||||||
|
*/
|
||||||
|
calibrate(p) {
|
||||||
|
const raw = knownNumber(p);
|
||||||
|
if (raw === null) return { p_raw: null, p_calibrated: null, calibrated: false, reason: 'absent' };
|
||||||
|
const c = cal.applyIsotonic(map, raw);
|
||||||
|
if (c === null) return { p_raw: raw, p_calibrated: null, calibrated: false, reason: 'no_map_value' };
|
||||||
|
const inBand = cal.inCertifiedBand(bands, c);
|
||||||
|
return {
|
||||||
|
p_raw: raw,
|
||||||
|
p_calibrated: Math.round(c * 1000) / 1000,
|
||||||
|
calibrated: inBand,
|
||||||
|
reason: inBand ? null : 'outside_certified_band',
|
||||||
|
};
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Load settled history from the ledger and build a calibrator, POINT-IN-TIME.
|
||||||
|
*
|
||||||
|
* `before` defaults to today, so the fit can only ever use games that are over.
|
||||||
|
* Injectable for tests; returns null rather than a permissive fallback, because
|
||||||
|
* "no calibrator" must mean "nothing is stackable", not "pass the raw numbers
|
||||||
|
* through".
|
||||||
|
*/
|
||||||
|
async function fromLedger(sb, { sport = 'mlb', stat = 'hits', before = null, ...opts } = {}) {
|
||||||
|
if (!sb) return null;
|
||||||
|
const cutoff = before || new Intl.DateTimeFormat('en-CA', {
|
||||||
|
timeZone: 'America/New_York', year: 'numeric', month: '2-digit', day: '2-digit',
|
||||||
|
}).format(new Date());
|
||||||
|
const rows = [];
|
||||||
|
for (let from = 0; ; from += 1000) {
|
||||||
|
const { data, error } = await sb.from('ledger_entries')
|
||||||
|
.select('p_win, outcome, game_date, quarantine_reason')
|
||||||
|
.eq('sport', sport).is('user_id', null).eq('stat', stat)
|
||||||
|
.in('outcome', ['hit', 'miss']).not('p_win', 'is', null)
|
||||||
|
.lt('game_date', cutoff) // STRICTLY before — the whole point
|
||||||
|
.range(from, from + 999);
|
||||||
|
if (error || !data || data.length === 0) break;
|
||||||
|
rows.push(...data);
|
||||||
|
if (data.length < 1000) break;
|
||||||
|
}
|
||||||
|
const clean = rows
|
||||||
|
.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book'))
|
||||||
|
.map((r) => ({ p: Number(r.p_win), won: r.outcome === 'hit' ? 1 : 0, date: String(r.game_date) }));
|
||||||
|
const built = build(clean, opts);
|
||||||
|
return built ? { ...built, cutoff } : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { build, fromLedger, HOLDOUT_FRACTION, MIN_FIT };
|
||||||
@@ -700,6 +700,40 @@ async function runSnapshot(sport, opts = {}) {
|
|||||||
console.warn(`[challenger] ${sp} skipped:`, e.message);
|
console.warn(`[challenger] ${sp} skipped:`, e.message);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── FORWARD CALIBRATION (hits) ────────────────────────────────────────
|
||||||
|
// Fitted on games that are OVER, applied to tonight's props. `p_win` is NOT
|
||||||
|
// touched — the counter stays byte-identical and the calibrated value rides
|
||||||
|
// beside it, because a calibration map is a correction TO a forecast, not a
|
||||||
|
// different forecast.
|
||||||
|
//
|
||||||
|
// `calibrated` is true only inside a band certified out-of-sample, and it is
|
||||||
|
// what `chain.chainAcross` requires before it will compound anything. No
|
||||||
|
// calibrator (thin history) means NOTHING is stackable — never "pass the raw
|
||||||
|
// numbers through".
|
||||||
|
if (sp === 'mlb') {
|
||||||
|
try {
|
||||||
|
const calSvc = deps.calibrationService || require('./model/calibrationService');
|
||||||
|
const sbc = require('../utils/supabase').getSupabaseServiceClient();
|
||||||
|
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat: 'hits' }) : null;
|
||||||
|
if (calibrator) {
|
||||||
|
let marked = 0;
|
||||||
|
for (const g of enriched) {
|
||||||
|
if (String(g.stat_type || g.stat || '').toLowerCase() !== 'hits') continue;
|
||||||
|
const out = calibrator.calibrate(g.p_win);
|
||||||
|
g.p_win_calibrated = out.p_calibrated;
|
||||||
|
g.calibrated = out.calibrated;
|
||||||
|
g.calibration_reason = out.reason;
|
||||||
|
if (out.calibrated) marked += 1;
|
||||||
|
}
|
||||||
|
console.log(`[calibration] ${sp} hits — ${marked} stackable of ${enriched.filter((g) => String(g.stat_type || g.stat || '').toLowerCase() === 'hits').length}; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, bands ${JSON.stringify(calibrator.bands.map((b) => [b.lo, b.hi]))}`);
|
||||||
|
} else {
|
||||||
|
console.log(`[calibration] ${sp} — no calibrator (thin history); nothing is stackable`);
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
console.warn('[calibration] skipped:', e.message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// LINEUP + BASERUNNER CONTEXT — the input RBI and runs have always needed.
|
// LINEUP + BASERUNNER CONTEXT — the input RBI and runs have always needed.
|
||||||
// Best-effort and dated: a context failure must never break a snapshot, and a
|
// Best-effort and dated: a context failure must never break a snapshot, and a
|
||||||
// lineup is a PRE-GAME fact that changes by the hour, so what we knew at grade
|
// lineup is a PRE-GAME fact that changes by the hour, so what we knew at grade
|
||||||
|
|||||||
@@ -136,6 +136,79 @@ describe('PROPAGATION — one settled result improves every reading that shares
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('CALIBRATION SERVICE — fit past, apply forward, certify by band', () => {
|
||||||
|
const svc = require('../../src/services/model/calibrationService');
|
||||||
|
|
||||||
|
/** Rows dated so the time-split is meaningful. */
|
||||||
|
const hist = (specs) => {
|
||||||
|
const out = [];
|
||||||
|
let day = 1;
|
||||||
|
for (const [p, n, rate] of specs) {
|
||||||
|
for (let i = 0; i < n; i += 1) {
|
||||||
|
// Wins are INTERLEAVED, not front-loaded. Front-loading makes the
|
||||||
|
// outcome correlate with the date, so a time-split would train on the
|
||||||
|
// wins and certify on the losses — the generator would be creating the
|
||||||
|
// very leakage the split exists to prevent.
|
||||||
|
const won = Math.floor((i + 1) * rate) > Math.floor(i * rate) ? 1 : 0;
|
||||||
|
out.push({ p, won, date: `2026-07-${String(day).padStart(2, '0')}` });
|
||||||
|
if (out.length % 40 === 0) day = Math.min(28, day + 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
};
|
||||||
|
|
||||||
|
it('refuses to build on thin history rather than passing raw numbers through', () => {
|
||||||
|
// "No calibrator" must mean nothing is stackable, never "trust the model".
|
||||||
|
expect(svc.build(hist([[0.6, 50, 0.5]]))).toBeNull();
|
||||||
|
expect(svc.build([])).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('corrects an over-confident model and marks the corrected value calibrated', () => {
|
||||||
|
// Claims 0.9, realises 0.6 — the shape measured on real hits.
|
||||||
|
const c = svc.build(hist([[0.5, 300, 0.5], [0.9, 300, 0.6]]), { minBin: 30 });
|
||||||
|
expect(c).not.toBeNull();
|
||||||
|
const out = c.calibrate(0.9);
|
||||||
|
expect(out.p_calibrated).toBeLessThan(0.75); // the 0.9 claim is corrected down
|
||||||
|
expect(out.p_raw).toBe(0.9);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('a probability OUTSIDE a certified band is not stackable', () => {
|
||||||
|
const c = svc.build(hist([[0.5, 300, 0.5], [0.9, 300, 0.6]]), { minBin: 30 });
|
||||||
|
const far = c.calibrate(0.02);
|
||||||
|
// Whatever it maps to, if the band was never certified it cannot compound.
|
||||||
|
if (!far.calibrated) expect(far.reason).toBe('outside_certified_band');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('an absent probability is absent, never 0', () => {
|
||||||
|
const c = svc.build(hist([[0.5, 300, 0.5], [0.9, 300, 0.6]]), { minBin: 30 });
|
||||||
|
const out = c.calibrate(null);
|
||||||
|
expect(out.p_calibrated).toBeNull();
|
||||||
|
expect(out.calibrated).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('splits by TIME — the certification window is later than the fit window', () => {
|
||||||
|
const c = svc.build(hist([[0.5, 300, 0.5], [0.9, 300, 0.6]]), { minBin: 30 });
|
||||||
|
expect(c.certified_through >= c.fitted_through).toBe(true);
|
||||||
|
expect(c.fit_n).toBeGreaterThan(0);
|
||||||
|
expect(c.certify_n).toBeGreaterThan(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('END TO END: uncalibrated legs are refused; calibrated ones compound', () => {
|
||||||
|
const c = svc.build(hist([[0.5, 300, 0.5], [0.9, 300, 0.6]]), { minBin: 30 });
|
||||||
|
const legs = ['a', 'b'].map((id) => {
|
||||||
|
const v = c.calibrate(0.5);
|
||||||
|
return { id, p: v.p_calibrated, calibrated: v.calibrated, gameId: `g${id}` };
|
||||||
|
});
|
||||||
|
const out = chain.chainAcross(legs);
|
||||||
|
if (legs.every((l) => l.calibrated)) {
|
||||||
|
expect(out.ok).toBe(true);
|
||||||
|
expect(out.compound_probability).toBeCloseTo(legs[0].p * legs[1].p, 3);
|
||||||
|
} else {
|
||||||
|
expect(out.reason).toBe('uncalibrated_atoms');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe('CALIBRATION — the gate itself', () => {
|
describe('CALIBRATION — the gate itself', () => {
|
||||||
const rows = (spec) => spec.flatMap(([p, n, hitRate]) =>
|
const rows = (spec) => spec.flatMap(([p, n, hitRate]) =>
|
||||||
Array.from({ length: n }, (_, i) => ({ p, won: i < Math.round(n * hitRate) ? 1 : 0 })));
|
Array.from({ length: n }, (_, i) => ({ p, won: i < Math.round(n * hitRate) ? 1 : 0 })));
|
||||||
|
|||||||
Reference in New Issue
Block a user