Robust bias established; low-parameter correction replaces isotonic
PHASE 0 — sample-limit truth on record: on 19 dates BOTH stability
instruments are underpowered. LODO power 0.014-0.093 (best 0.337 across
every k tried); deploy CIs rest on 2-4 date clusters, where a
cluster-robust interval has ~1 df. This is the SAMPLE, not a fixable
instrument, and the gate-refinement loop stops here. Runs corrected: its
DATE-DRIVEN label was an artefact of the coin-flip ruler (2 reversals in
3 drops never cleared cutoff 2) -- it is an ordinary no-fittable-map
refusal.
PHASE 1 — the bias is ROBUST, tested model-free and map-free with a
date-block bootstrap. Pooled over-prediction rises monotonically -0.0076
/ +0.0428 / +0.0963 / +0.1589 / +0.2451 across deciles from 0.5 to 1.0,
sign stability 0.9946 over 17 date blocks, and 4 of 4 stats replicate
(bar was 3). Also visible: realized rate PLATEAUS at 0.65-0.68 from p=0.7
upward -- the 0.9+ bucket (0.6624) does no better than the 0.8-0.9 bucket
(0.6841). The model has no high-confidence reads, only high-confidence
numbers.
PHASE 3 — Platt, two parameters over the whole curve, shrunk toward
identity by fit-date count. Validated as a NEW estimator vs RAW with
date-block CIs:
hits a=0.406 shrink 0.565 0.2626 -> 0.2540 CI [-0.0112,-0.0069] DEPLOY
total_bases a=0.472 shrink 0.333 0.2490 -> 0.2429 CI [-0.0062,-0.0059] DEPLOY
rbi a=0.775 shrink 0.231 0.2011 -> 0.2007 CI [-0.0007, 0] REFUSE
runs a=-0.032 REFUSE
A GUARD THE FIRST RUN NEEDED: runs fitted a = -0.032. A non-positive
slope inverts the forecast rather than flattening it, and near zero the
curve collapses to a constant predicting the base rate for everything --
which LOWERS Brier while destroying all resolution. It would have scored
as a win while making the product worthless. MIN_SLOPE now refuses it by
name, with a test.
STATED PLAINLY: on the identical held-out rows isotonic BEAT the
low-param on hits (+0.0028) and rbi (+0.0042) and tied on TB. The swap is
a CAPACITY JUDGEMENT, not a measurement -- the window spans 2-4 date
blocks and that is exactly what a flexible map produces when it captures
structure shared by fit and eval. Labelled as a judgement.
PHASE 4 — hits and total_bases serve the correction, basis
direction_robust_magnitude_provisional (direction bootstrap-robust,
magnitude thin-sample and shrunk). rbi is WITHDRAWN to raw -- it was
deployed on isotonic at ced4042 and the low-param does not beat raw.
runs stays raw. Auto-demotion still armed.
PHASE 5 — the standing finding, stated hard: across 18 archetype slots on
three stats, calibrated p_win separates within archetype NO BETTER than
raw. Every slot is one band indistinguishable from its base rate, zero
show lift. Per-archetype separation is not coming from calibration; it
comes from proven factors or it does not exist. Five orders of
calibration have delivered what they can -- honest numbers on two stats --
and nothing on the question the grade product turns on.
p_win never mutated; no Bonferroni slot; the robust-claim test ran before
any calibrator was built and could have ended the session at Phase 2.
Counter and frozen clusters verified file-by-file, including calibration.js
and calibrationService.js, both untouched and simply off the serving path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* PHASE 1 — is the favourite-longshot bias real WITHOUT a calibration map?
|
||||
*
|
||||
* Four orders have refined a stability gate on 19 dates. LODO turned out to be
|
||||
* structurally underpowered (0.014–0.093) and the deploy intervals rest on 2–4
|
||||
* date clusters. So we stop certifying the stability of a specific MAP, and ask
|
||||
* the one question this sample might actually answer:
|
||||
*
|
||||
* does the model over-predict its own favourites, robustly?
|
||||
*
|
||||
* That claim is MODEL-FREE and MAP-FREE — it is a property of (p_win, outcome)
|
||||
* pairs, needs no isotonic fit, and can therefore be tested without any of the
|
||||
* machinery whose stability we cannot certify.
|
||||
*
|
||||
* ── DATE-BLOCK BOOTSTRAP ─────────────────────────────────────────────────
|
||||
* Resampling ROWS would treat 200 props from one night as 200 readings of that
|
||||
* night's offensive environment. Whole DATES are resampled instead, which is the
|
||||
* honest unit and a far harsher one at 5–17 dates.
|
||||
*
|
||||
* VERDICT is pre-stated: ROBUST iff the >0.9 over-prediction sign survives in
|
||||
* >=95% of pooled date-block resamples AND replicates in >=3 of 4 stats on the
|
||||
* same criterion. Anything else is NOT-ROBUST, and NOT-ROBUST means we serve raw.
|
||||
*
|
||||
* SUPABASE_URL=... node scripts/test-favourite-bias-robust.js
|
||||
*/
|
||||
|
||||
require('dotenv').config();
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const guards = require('../src/services/model/calibrationGuards');
|
||||
const { knownNumber } = require('../src/utils/known');
|
||||
|
||||
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
|
||||
const STATS = ['hits', 'total_bases', 'rbi', 'runs'];
|
||||
const PAGE = 1000;
|
||||
const FAVOURITE_FLOOR = 0.9;
|
||||
const ITERS = 5000;
|
||||
/** Pre-stated pass marks. */
|
||||
const SIGN_STABILITY_REQUIRED = 0.95;
|
||||
const STATS_MUST_REPLICATE = 3;
|
||||
|
||||
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
|
||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||
|
||||
async function page(sb, t, s, f) {
|
||||
const o = [];
|
||||
for (let i = 0; ; i += PAGE) {
|
||||
const { data, error } = await f(sb.from(t).select(s)).order('id', { ascending: true }).range(i, i + PAGE - 1);
|
||||
if (error) throw error;
|
||||
if (!data.length) break;
|
||||
o.push(...data);
|
||||
if (data.length < PAGE) break;
|
||||
}
|
||||
return o;
|
||||
}
|
||||
const isPreGame = (c, g) => {
|
||||
const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000);
|
||||
const d = et.toISOString().slice(0, 10);
|
||||
return d < g || (d === g && et.getUTCHours() < 19);
|
||||
};
|
||||
function makeRnd(seed) {
|
||||
let s = seed >>> 0;
|
||||
return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
|
||||
}
|
||||
|
||||
/** Over-prediction in the favourite bin: predicted − realized. Positive = over. */
|
||||
function favouriteBias(rows) {
|
||||
const fav = rows.filter((r) => r.p >= FAVOURITE_FLOOR);
|
||||
if (fav.length < 5) return null;
|
||||
return mean(fav.map((r) => r.p)) - mean(fav.map((r) => r.won));
|
||||
}
|
||||
|
||||
/** Resample whole DATES with replacement; report how often the sign survives. */
|
||||
function dateBlockSignStability(rows, seed) {
|
||||
const byDate = new Map();
|
||||
for (const r of rows) {
|
||||
if (!byDate.has(r.date)) byDate.set(r.date, []);
|
||||
byDate.get(r.date).push(r);
|
||||
}
|
||||
const keys = [...byDate.keys()];
|
||||
const rnd = makeRnd(seed);
|
||||
let positive = 0; let indeterminate = 0; const draws = [];
|
||||
for (let it = 0; it < ITERS; it += 1) {
|
||||
const sample = [];
|
||||
for (let i = 0; i < keys.length; i += 1) sample.push(...byDate.get(keys[Math.floor(rnd() * keys.length)]));
|
||||
const b = favouriteBias(sample);
|
||||
// A resample with too few favourites cannot speak — counted, never guessed.
|
||||
if (b === null) { indeterminate += 1; continue; }
|
||||
draws.push(b);
|
||||
if (b > 0) positive += 1;
|
||||
}
|
||||
const usable = ITERS - indeterminate;
|
||||
draws.sort((a, b) => a - b);
|
||||
return {
|
||||
date_blocks: keys.length,
|
||||
usable_resamples: usable,
|
||||
indeterminate_resamples: indeterminate,
|
||||
sign_stability: usable ? round4(positive / usable) : null,
|
||||
ci_90: draws.length ? [round4(draws[Math.floor(draws.length * 0.05)]), round4(draws[Math.floor(draws.length * 0.95)])] : null,
|
||||
};
|
||||
}
|
||||
|
||||
function deciles(rows) {
|
||||
const out = [];
|
||||
for (let lo = 0.3; lo < 1.0; lo += 0.1) {
|
||||
const hi = lo + 0.1;
|
||||
const slice = rows.filter((r) => r.p >= lo && (hi >= 1 ? r.p <= 1 : r.p < hi));
|
||||
if (slice.length < 15) continue;
|
||||
const pred = mean(slice.map((r) => r.p));
|
||||
const real = mean(slice.map((r) => r.won));
|
||||
out.push({ bin: [round2(lo), round2(hi)], n: slice.length, predicted: round4(pred), realized: round4(real), over_prediction: round4(pred - real) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const sb = createClient(process.env.SUPABASE_URL,
|
||||
process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } });
|
||||
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
|
||||
|
||||
const snaps = await page(sb, 'model_snapshots',
|
||||
'id, game_date, captured_at, stat, player_key, line, side, p_win, refused',
|
||||
(q) => q.eq('sport', 'mlb').in('stat', STATS));
|
||||
|
||||
const picked = new Map();
|
||||
for (const r of snaps) {
|
||||
if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue;
|
||||
const k = [r.game_date, r.stat, r.player_key, r.line].join('|');
|
||||
const prev = picked.get(k);
|
||||
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r);
|
||||
}
|
||||
guards.assertPickedSideDedup([...picked.values()].map((r) => ({
|
||||
propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win),
|
||||
})));
|
||||
|
||||
const byStat = {}; const pooled = [];
|
||||
for (const stat of STATS) byStat[stat] = [];
|
||||
for (const r of picked.values()) {
|
||||
const b = lines[`${r.game_date}|${r.player_key}`];
|
||||
const L = knownNumber(r.line);
|
||||
if (!b || L === null || !r.side) continue;
|
||||
const v = knownNumber(FIELD[r.stat](b));
|
||||
if (v === null) continue;
|
||||
const over = v > L;
|
||||
const row = { date: r.game_date, p: knownNumber(r.p_win), won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0 };
|
||||
byStat[r.stat].push(row); pooled.push(row);
|
||||
}
|
||||
|
||||
const pooledResult = {
|
||||
n: pooled.length,
|
||||
deciles: deciles(pooled),
|
||||
favourite_bias: round4(favouriteBias(pooled)),
|
||||
...dateBlockSignStability(pooled, 20260808),
|
||||
};
|
||||
|
||||
const perStat = {};
|
||||
let replicated = 0;
|
||||
for (const stat of STATS) {
|
||||
const rows = byStat[stat];
|
||||
const fb = favouriteBias(rows);
|
||||
const stab = dateBlockSignStability(rows, 20260808);
|
||||
const ok = fb !== null && fb > 0 && stab.sign_stability !== null && stab.sign_stability >= SIGN_STABILITY_REQUIRED;
|
||||
if (ok) replicated += 1;
|
||||
perStat[stat] = { n: rows.length, deciles: deciles(rows), favourite_bias: fb === null ? null : round4(fb), ...stab, replicates: ok };
|
||||
}
|
||||
|
||||
const pooledOk = pooledResult.favourite_bias > 0 && pooledResult.sign_stability >= SIGN_STABILITY_REQUIRED;
|
||||
const verdict = pooledOk && replicated >= STATS_MUST_REPLICATE ? 'ROBUST' : 'NOT-ROBUST';
|
||||
|
||||
console.log(JSON.stringify({
|
||||
phase: 'PHASE 1 — model-free, map-free favourite-longshot bias test',
|
||||
criteria: { sign_stability_required: SIGN_STABILITY_REQUIRED, stats_must_replicate: STATS_MUST_REPLICATE, favourite_floor: FAVOURITE_FLOOR },
|
||||
pooled: pooledResult,
|
||||
per_stat: perStat,
|
||||
stats_replicating: replicated,
|
||||
VERDICT: verdict,
|
||||
consequence: verdict === 'ROBUST'
|
||||
? 'proceed to a low-parameter correction, validated as a NEW estimator'
|
||||
: 'serve raw; the bias is not certifiable on this sample',
|
||||
}, null, 2));
|
||||
process.exit(0);
|
||||
})().catch((e) => { console.error(e); process.exit(1); });
|
||||
|
||||
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
const round2 = (v) => Math.round(v * 100) / 100;
|
||||
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* PHASE 3 — validate the low-parameter correction as a NEW estimator.
|
||||
*
|
||||
* No grandfathering: it must beat RAW out-of-sample with a DATE-BLOCK bootstrap
|
||||
* interval excluding zero. It is also scored against the retired isotonic map on
|
||||
* the identical held-out rows, so the swap is a measured comparison rather than
|
||||
* a preference.
|
||||
*/
|
||||
|
||||
require('dotenv').config();
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cal = require('../src/services/model/calibration');
|
||||
const lp = require('../src/services/model/lowParamCalibrator');
|
||||
const guards = require('../src/services/model/calibrationGuards');
|
||||
const { knownNumber } = require('../src/utils/known');
|
||||
|
||||
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
|
||||
const STATS = ['hits', 'total_bases', 'rbi', 'runs'];
|
||||
const PAGE = 1000; const ITERS = 4000;
|
||||
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
|
||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||
|
||||
async function page(sb, t, s, f) {
|
||||
const o = [];
|
||||
for (let i = 0; ; i += PAGE) {
|
||||
const { data, error } = await f(sb.from(t).select(s)).order('id', { ascending: true }).range(i, i + PAGE - 1);
|
||||
if (error) throw error; if (!data.length) break; o.push(...data); if (data.length < PAGE) break;
|
||||
}
|
||||
return o;
|
||||
}
|
||||
const isPreGame = (c, g) => {
|
||||
const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000);
|
||||
const d = et.toISOString().slice(0, 10);
|
||||
return d < g || (d === g && et.getUTCHours() < 19);
|
||||
};
|
||||
function makeRnd(seed) { let s = seed >>> 0; return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; }; }
|
||||
|
||||
/** Paired date-block bootstrap on a Brier difference. */
|
||||
function dateBlockCI(rows, keyA, keyB, seed) {
|
||||
const byDate = new Map();
|
||||
for (const r of rows) { if (!byDate.has(r.date)) byDate.set(r.date, []); byDate.get(r.date).push(r); }
|
||||
const keys = [...byDate.keys()]; const rnd = makeRnd(seed); const diffs = [];
|
||||
for (let it = 0; it < ITERS; it += 1) {
|
||||
const s = [];
|
||||
for (let i = 0; i < keys.length; i += 1) s.push(...byDate.get(keys[Math.floor(rnd() * keys.length)]));
|
||||
const a = guards.safeBrier(s.map((r) => r[keyA]), s.map((r) => r.won));
|
||||
const b = guards.safeBrier(s.map((r) => r[keyB]), s.map((r) => r.won));
|
||||
if (a === null || b === null) continue;
|
||||
diffs.push(a - b);
|
||||
}
|
||||
diffs.sort((x, y) => x - y);
|
||||
return diffs.length
|
||||
? { ci: [round4(diffs[Math.floor(diffs.length * 0.025)]), round4(diffs[Math.floor(diffs.length * 0.975)])], date_blocks: keys.length }
|
||||
: { ci: null, date_blocks: keys.length };
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } });
|
||||
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
|
||||
const snaps = await page(sb, 'model_snapshots', 'id, game_date, captured_at, stat, player_key, line, side, p_win, refused',
|
||||
(q) => q.eq('sport', 'mlb').in('stat', STATS));
|
||||
|
||||
const picked = new Map();
|
||||
for (const r of snaps) {
|
||||
if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue;
|
||||
const k = [r.game_date, r.stat, r.player_key, r.line].join('|');
|
||||
const prev = picked.get(k);
|
||||
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r);
|
||||
}
|
||||
guards.assertPickedSideDedup([...picked.values()].map((r) => ({
|
||||
propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win) })));
|
||||
|
||||
const out = {};
|
||||
for (const stat of STATS) {
|
||||
const rows = [];
|
||||
for (const r of picked.values()) {
|
||||
if (r.stat !== stat) continue;
|
||||
const b = lines[`${r.game_date}|${r.player_key}`]; const L = knownNumber(r.line);
|
||||
if (!b || L === null || !r.side) continue;
|
||||
const v = knownNumber(FIELD[stat](b)); if (v === null) continue;
|
||||
const over = v > L;
|
||||
rows.push({ date: r.game_date, p: knownNumber(r.p_win), won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0 });
|
||||
}
|
||||
rows.sort((a, b) => String(a.date).localeCompare(String(b.date)));
|
||||
const dates = [...new Set(rows.map((r) => r.date))].sort();
|
||||
const perDate = new Map(); for (const r of rows) perDate.set(r.date, (perDate.get(r.date) || 0) + 1);
|
||||
let acc = 0; let cut = dates[dates.length - 1];
|
||||
for (const d of dates) { acc += perDate.get(d); if (acc >= rows.length * 0.45) { cut = d; break; } }
|
||||
|
||||
const fit = rows.filter((r) => r.date < cut);
|
||||
const ev = rows.filter((r) => r.date >= cut);
|
||||
const platt = lp.fitPlatt(fit);
|
||||
const iso = cal.fitIsotonic(fit.map((r) => ({ p: r.p, won: r.won })));
|
||||
|
||||
if (!platt || ev.length < 50) {
|
||||
out[stat] = { n: rows.length, fit_n: fit.length, eval_n: ev.length, decision: 'REFUSE', reason: 'no low-parameter fit or too little held out' };
|
||||
continue;
|
||||
}
|
||||
const scored = ev.map((r) => ({ ...r, plat: lp.applyPlatt(platt, r.p), isoP: iso ? cal.applyIsotonic(iso, r.p) : null }))
|
||||
.filter((r) => knownNumber(r.plat) !== null);
|
||||
const ys = scored.map((r) => r.won);
|
||||
const bRaw = guards.safeBrier(scored.map((r) => r.p), ys);
|
||||
const bPlat = guards.safeBrier(scored.map((r) => r.plat), ys);
|
||||
const isoRows = scored.filter((r) => knownNumber(r.isoP) !== null);
|
||||
const bIso = isoRows.length ? guards.safeBrier(isoRows.map((r) => r.isoP), isoRows.map((r) => r.won)) : null;
|
||||
|
||||
const vsRaw = dateBlockCI(scored, 'plat', 'p', 20260808);
|
||||
const vsIso = isoRows.length ? dateBlockCI(isoRows, 'plat', 'isoP', 20260808) : { ci: null };
|
||||
|
||||
const beatsRaw = bPlat < bRaw && vsRaw.ci && vsRaw.ci[1] < 0;
|
||||
out[stat] = {
|
||||
n: rows.length, dates: dates.length, split_at: cut, fit_n: fit.length, eval_n: scored.length,
|
||||
platt: { a: platt.a, b: platt.b, flattens: platt.flattens, fit_dates: platt.fit_dates, shrinkage: platt.shrinkage },
|
||||
brier_raw: round4(bRaw), brier_lowparam: round4(bPlat), brier_isotonic: bIso === null ? null : round4(bIso),
|
||||
delta_vs_raw: round4(bPlat - bRaw), ci_vs_raw: vsRaw.ci, eval_date_blocks: vsRaw.date_blocks,
|
||||
delta_vs_isotonic: bIso === null ? null : round4(bPlat - bIso), ci_vs_isotonic: vsIso.ci,
|
||||
decision: beatsRaw ? 'DEPLOY-PROVISIONAL' : 'REFUSE',
|
||||
reason: beatsRaw ? 'beats raw out-of-sample with a date-block interval excluding zero'
|
||||
: 'does not beat raw at a date-block interval excluding zero',
|
||||
};
|
||||
}
|
||||
console.log(JSON.stringify(out, null, 2));
|
||||
process.exit(0);
|
||||
})().catch((e) => { console.error(e); process.exit(1); });
|
||||
|
||||
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
Reference in New Issue
Block a user