Instrument the calibration duel forward; diagnose the resolution ceiling
— the proven factors were never wired in
PHASE 0 — two truths recorded. The swap is a BET, not an OOS win:
isotonic beat low-param on identical held-out rows (hits +0.0028, rbi
+0.0042, TB tied) and we serve low-param anyway on an untestable prior
about shared daily structure. At 19 dates nothing here can test it. And
the MIN_SLOPE catch is preserved as standing rationale: a near-zero or
negative slope collapses toward base-rate-for-everything, which LOWERS
Brier while destroying all resolution -- a metric win that guts the
product.
PHASE 1 — the duel is now falsifiable. Both corrections computed on every
hits/TB prop; p_win_lowparam served, p_win_isotonic_shadow logged in its
own try so it can never break serving. calibrationDuel.adjudicate encodes
the rule IN CODE before any forward date exists: >=10 forward dates and
isotonic winning with a date-block CI excluding zero => REFUTED, revert;
otherwise UPHELD; under 10 dates PENDING regardless of the numbers. A
date counts as forward only if NEITHER map was fitted on it -- otherwise
we would be scoring which map memorised better. Nothing swaps now.
PHASE 2 — the ceiling, quantified via Murphy decomposition:
stat reliability RESOLUTION uncertainty variance explained
hits 0.01353 0.00252 0.24532 1.03%
TB 0.01419 0.00442 0.24329 1.82%
rbi 0.00654 0.03268 0.22531 14.51%
runs 0.00788 0.00130 0.23182 0.56%
Calibration did exactly what theory says and nothing more: hits
reliability 0.01353 -> 0.00233 (-0.0112, 83% of the error removed) while
resolution moved -0.0002. Unexpected: rbi has 13x the resolution of hits
and is the one stat we do NOT serve corrected -- it needs calibration
least and discriminates most.
PHASE 2 DIAGNOSIS — NOT-TRANSMITTED, and not weak, ABSENT. Traced in code:
sprayDefense.js and platoonSeverity.js are required by NOTHING in src/,
only by analysis scripts and their own tests. The served p_win
(intelligence/probabilityEstimator.js:54) reads exactly four inputs --
game-log frequency, opp_rank_stat +/-0.03, home_away +/-0.015, and a cv
pull -- with zero occurrences of spray, platoon, hard-hit or
contact-profile. And snapshotService grades at line 454 while computing
challenger/context at 640+, so everything proven is computed DOWNSTREAM of
the grade it would inform. The three proven hits factors have never once
moved a served number.
That reframes the recent nulls: "calibrated p_win does not separate within
archetype" was never a statement about factors. The factors were not in
the forecast.
PHASE 3 — bands rebuilt on SERVED values (hits/TB low-param, rbi/runs
raw): 28 archetype slots across four stats, ZERO show lift. No longer an
open shrug -- it is the arithmetic of resolution 0.0013-0.0327 against
uncertainty ~0.23. A forecast explaining 1% of variance cannot produce
separating bands, and no correction to its numbers will change that.
HEADLINE: calibration is complete, delivered honest numbers on two stats
and zero grade separation, because the counter has no resolution -- and
the proven factors are not wired into the forecast at all. The second is
the reason for the first, and it is plumbing rather than a modelling wall.
Per-archetype grades need proven factors that actually reach p_win. Last
calibration order.
Serving unchanged from 74cf1ce. p_win never mutated. No Bonferroni slot.
Counter and frozen clusters verified file-by-file (15 modules).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -18,6 +18,7 @@ const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cal = require('../src/services/model/calibration');
|
||||
const lp = require('../src/services/model/lowParamCalibrator');
|
||||
const gb = require('../src/services/model/gradeBands');
|
||||
const guards = require('../src/services/model/calibrationGuards');
|
||||
const tl = require('../src/services/model/testLedger');
|
||||
@@ -86,8 +87,23 @@ const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b)
|
||||
for (const r of rows) perDate.set(r.date, (perDate.get(r.date) || 0) + 1);
|
||||
let acc = 0; let cut = dates[dates.length - 1];
|
||||
for (const d of dates) { acc += perDate.get(d); if (acc >= rows.length * 0.45) { cut = d; break; } }
|
||||
const map = cal.fitIsotonic(rows.filter((r) => r.date < cut).map((r) => ({ p: r.p, won: r.won })));
|
||||
const applied = guards.applyOrRefuse(map, rows.filter((r) => r.date >= cut), cal.applyIsotonic);
|
||||
// Bands are built on the SERVED values. hits and total_bases serve the
|
||||
// low-parameter correction; rbi and runs serve raw, so their bands are raw.
|
||||
const DEPLOYED = ['hits', 'total_bases'];
|
||||
const fitRows = rows.filter((r) => r.date < cut);
|
||||
const evalRows = rows.filter((r) => r.date >= cut);
|
||||
let applied;
|
||||
let basis;
|
||||
if (DEPLOYED.includes(STAT)) {
|
||||
const model = lp.fitPlatt(fitRows);
|
||||
applied = (!model || model.refused)
|
||||
? { ok: false, reason: 'low-parameter fit refused', rows: [] }
|
||||
: { ok: true, rows: evalRows.map((r) => ({ ...r, pc: lp.applyPlatt(model, r.p) })).filter((r) => r.pc != null) };
|
||||
basis = 'p_win_lowparam (SERVED, provisional)';
|
||||
} else {
|
||||
applied = { ok: true, rows: evalRows.map((r) => ({ ...r, pc: r.p })) };
|
||||
basis = 'raw p_win (this stat serves raw)';
|
||||
}
|
||||
if (!applied.ok) { console.log(JSON.stringify({ stat: STAT, refused: applied.reason })); process.exit(0); }
|
||||
|
||||
const mc = await tl.recordAndCount(tl.supabaseStore(sb), []).catch(() => ({ cumulative_tests: 1 }));
|
||||
@@ -105,13 +121,13 @@ const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b)
|
||||
cumulativeTests: mc.cumulative_tests,
|
||||
// TB is CALIBRATED (provisional) but no factor is PROVEN for it.
|
||||
proven: false,
|
||||
calibrated: true,
|
||||
calibrated: DEPLOYED.includes(STAT),
|
||||
}));
|
||||
}
|
||||
|
||||
console.log(JSON.stringify({
|
||||
stat: STAT,
|
||||
basis: 'p_win_calibrated (PROVISIONAL)',
|
||||
basis,
|
||||
eval_rows: applied.rows.length,
|
||||
cumulative_tests: mc.cumulative_tests,
|
||||
two_bar_note: 'calibrated YES, proven NO -> bands stay a base-rate read, now honestly numbered',
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* PHASE 2 — put a number on the resolution ceiling.
|
||||
*
|
||||
* Murphy's decomposition: Brier = reliability - resolution + uncertainty.
|
||||
*
|
||||
* reliability how far each bin's realized rate sits from its forecast (lower
|
||||
* is better; this is what calibration fixes)
|
||||
* resolution how far the bins' realized rates spread from the base rate
|
||||
* (HIGHER is better; this is discrimination, and NO amount of
|
||||
* calibration can create it)
|
||||
* uncertainty the base rate's own variance -- a property of the event
|
||||
*
|
||||
* Calibration moves reliability and leaves resolution untouched by construction:
|
||||
* a monotone map relabels bins without re-sorting the rows inside them. So if
|
||||
* resolution is near zero, honest numbers are all calibration can ever deliver.
|
||||
*/
|
||||
|
||||
require('dotenv').config();
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const lp = require('../src/services/model/lowParamCalibrator');
|
||||
const guards = require('../src/services/model/calibrationGuards');
|
||||
const { knownNumber } = require('../src/utils/known');
|
||||
|
||||
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
|
||||
const STATS = ['hits', 'total_bases', 'rbi', 'runs'];
|
||||
const DEPLOYED = ['hits', 'total_bases'];
|
||||
const PAGE = 1000;
|
||||
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
|
||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||
|
||||
async function page(sb, t, s, f) {
|
||||
const o = [];
|
||||
for (let i = 0; ; i += PAGE) {
|
||||
const { data, error } = await f(sb.from(t).select(s)).order('id', { ascending: true }).range(i, i + PAGE - 1);
|
||||
if (error) throw error; if (!data.length) break; o.push(...data); if (data.length < PAGE) break;
|
||||
}
|
||||
return o;
|
||||
}
|
||||
const isPreGame = (c, g) => {
|
||||
const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000);
|
||||
const d = et.toISOString().slice(0, 10);
|
||||
return d < g || (d === g && et.getUTCHours() < 19);
|
||||
};
|
||||
|
||||
/** Murphy decomposition over K equal-width bins. */
|
||||
function decompose(rows, bins = 10) {
|
||||
const base = mean(rows.map((r) => r.won));
|
||||
const uncertainty = base * (1 - base);
|
||||
let reliability = 0; let resolution = 0;
|
||||
const table = [];
|
||||
for (let k = 0; k < bins; k += 1) {
|
||||
const lo = k / bins; const hi = (k + 1) / bins;
|
||||
const slice = rows.filter((r) => r.p >= lo && (hi >= 1 ? r.p <= 1 : r.p < hi));
|
||||
if (!slice.length) continue;
|
||||
const w = slice.length / rows.length;
|
||||
const fk = mean(slice.map((r) => r.p));
|
||||
const ok = mean(slice.map((r) => r.won));
|
||||
reliability += w * (fk - ok) ** 2;
|
||||
resolution += w * (ok - base) ** 2;
|
||||
table.push({ bin: [round2(lo), round2(hi)], n: slice.length, forecast: round4(fk), realized: round4(ok) });
|
||||
}
|
||||
return {
|
||||
base_rate: round4(base),
|
||||
reliability: round5(reliability),
|
||||
resolution: round5(resolution),
|
||||
uncertainty: round5(uncertainty),
|
||||
brier_check: round5(reliability - resolution + uncertainty),
|
||||
/** What share of the event's variance the model actually explains. */
|
||||
resolution_share_of_uncertainty: round4(resolution / uncertainty),
|
||||
bins: table,
|
||||
};
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } });
|
||||
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
|
||||
const snaps = await page(sb, 'model_snapshots', 'id, game_date, captured_at, stat, player_key, line, side, p_win, refused',
|
||||
(q) => q.eq('sport', 'mlb').in('stat', STATS));
|
||||
|
||||
const picked = new Map();
|
||||
for (const r of snaps) {
|
||||
if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue;
|
||||
const k = [r.game_date, r.stat, r.player_key, r.line].join('|');
|
||||
const prev = picked.get(k);
|
||||
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r);
|
||||
}
|
||||
guards.assertPickedSideDedup([...picked.values()].map((r) => ({
|
||||
propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win) })));
|
||||
|
||||
const out = {};
|
||||
for (const stat of STATS) {
|
||||
const rows = [];
|
||||
for (const r of picked.values()) {
|
||||
if (r.stat !== stat) continue;
|
||||
const b = lines[`${r.game_date}|${r.player_key}`]; const L = knownNumber(r.line);
|
||||
if (!b || L === null || !r.side) continue;
|
||||
const v = knownNumber(FIELD[stat](b)); if (v === null) continue;
|
||||
const over = v > L;
|
||||
rows.push({ date: r.game_date, p: knownNumber(r.p_win), won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0 });
|
||||
}
|
||||
if (rows.length < 100) continue;
|
||||
|
||||
const raw = decompose(rows);
|
||||
let served = null;
|
||||
if (DEPLOYED.includes(stat)) {
|
||||
const m = lp.fitPlatt(rows);
|
||||
if (m && !m.refused) {
|
||||
const cal = rows.map((r) => ({ ...r, p: lp.applyPlatt(m, r.p) })).filter((r) => knownNumber(r.p) !== null);
|
||||
served = decompose(cal);
|
||||
}
|
||||
}
|
||||
out[stat] = {
|
||||
n: rows.length,
|
||||
deployed: DEPLOYED.includes(stat),
|
||||
raw,
|
||||
served,
|
||||
resolution_change_from_calibration: served ? round5(served.resolution - raw.resolution) : null,
|
||||
reliability_change_from_calibration: served ? round5(served.reliability - raw.reliability) : null,
|
||||
};
|
||||
}
|
||||
console.log(JSON.stringify(out, null, 2));
|
||||
process.exit(0);
|
||||
})().catch((e) => { console.error(e); process.exit(1); });
|
||||
|
||||
const round5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
|
||||
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
const round2 = (v) => Math.round(v * 100) / 100;
|
||||
@@ -0,0 +1,156 @@
|
||||
# The resolution ceiling — calibration is complete, and it was never the lever
|
||||
|
||||
## PHASE 0 — two honest truths, on record
|
||||
|
||||
### 1. The swap is a BET, not an OOS win
|
||||
|
||||
On identical held-out rows the **isotonic map scored BETTER**: hits +0.0028
|
||||
(CI [0.0013, 0.0045]), rbi +0.0042 (CI [0.0003, 0.0092]), total_bases tied.
|
||||
|
||||
We serve the low-parameter map anyway, on the untestable prior that isotonic's
|
||||
in-window edge is daily structure shared between the fit and evaluation windows
|
||||
and will not transmit forward. At 19 dates **no instrument here can test that
|
||||
prior** — LODO has 1.4–9.3% power against it.
|
||||
|
||||
**Named as a bet, logged, not evidence.** Phase 1 makes it falsifiable.
|
||||
|
||||
### 2. The MIN_SLOPE catch, as a standing guard rationale
|
||||
|
||||
`runs` fitted `a = −0.032`. A near-zero or negative slope collapses the curve
|
||||
toward *base-rate-for-everything*, which **lowers Brier** — shrinking a
|
||||
miscalibrated forecaster toward its base rate always does — while destroying all
|
||||
resolution.
|
||||
|
||||
**A metric win that guts the product.** Any calibration layer must refuse a
|
||||
non-positive slope on principle, not on inspection. That is now `MIN_SLOPE`.
|
||||
|
||||
---
|
||||
|
||||
## PHASE 1 — the duel, instrumented forward
|
||||
|
||||
Both corrections are computed on every hits/TB prop:
|
||||
`p_win_lowparam` (served) and `p_win_isotonic_shadow` (logged, never read by
|
||||
serving or by `chainAcross`). The shadow runs in its own try — it can never
|
||||
break serving.
|
||||
|
||||
`calibrationDuel.adjudicate` encodes the rule **in code, before any forward date
|
||||
exists**, so the bar cannot drift toward whichever answer arrives:
|
||||
|
||||
| condition | verdict | action |
|
||||
|---|---|---|
|
||||
| ≥10 forward dates AND isotonic wins, date-block CI excluding zero | **REFUTED** | revert hits/TB to isotonic, log the reversal |
|
||||
| ≥10 forward dates, isotonic does not win | **UPHELD** | keep serving low-param |
|
||||
| <10 forward dates | **PENDING** | keep serving, no verdict |
|
||||
|
||||
A date counts as forward **only if neither map was fitted on it** — scoring
|
||||
inside a fit window would ask which map memorised better. Rows lacking that
|
||||
provenance are dropped, never assumed forward. Tests lock all of it, including
|
||||
that a decisive shadow win at 5 dates is still PENDING.
|
||||
|
||||
**Nothing swaps now.** The season decides.
|
||||
|
||||
---
|
||||
|
||||
## PHASE 2 — the resolution ceiling, quantified
|
||||
|
||||
Murphy decomposition: `Brier = reliability − resolution + uncertainty`.
|
||||
Reliability is what calibration fixes. **Resolution is discrimination, and a
|
||||
monotone map cannot create it** — it relabels bins without re-sorting the rows
|
||||
inside them.
|
||||
|
||||
| stat | n | base | reliability | **resolution** | uncertainty | **share of variance explained** |
|
||||
|---|---|---|---|---|---|---|
|
||||
| hits | 1,140 | 0.5684 | 0.01353 | **0.00252** | 0.24532 | **1.03%** |
|
||||
| total_bases | 1,050 | 0.5819 | 0.01419 | **0.00442** | 0.24329 | **1.82%** |
|
||||
| rbi | 630 | 0.6571 | 0.00654 | **0.03268** | 0.22531 | **14.51%** |
|
||||
| runs | 597 | 0.6348 | 0.00788 | **0.00130** | 0.23182 | **0.56%** |
|
||||
|
||||
### What calibration did, exactly as theory predicts
|
||||
|
||||
| stat | reliability | resolution |
|
||||
|---|---|---|
|
||||
| hits | 0.01353 → 0.00233 (**−0.0112**) | 0.00252 → 0.00231 (−0.0002) |
|
||||
| total_bases | 0.01419 → 0.00527 (**−0.0089**) | 0.00442 → 0.00414 (−0.0003) |
|
||||
|
||||
**Calibration removed 83% of hits' reliability error and moved resolution by
|
||||
essentially nothing.** It did the whole of its job, and its job was never the
|
||||
one the grade product needs.
|
||||
|
||||
**Unexpected:** `rbi` has **13× the resolution of hits** and is the one stat we
|
||||
do *not* serve corrected — it needs calibration least (reliability 0.0065) and
|
||||
discriminates most. Worth carrying into the factor arc.
|
||||
|
||||
---
|
||||
|
||||
## PHASE 2 — the HITS factor-transmission diagnosis
|
||||
|
||||
**Verdict: NOT-TRANSMITTED. Not weak — absent.** This is a plumbing defect, and
|
||||
it is the highest-value finding in the order.
|
||||
|
||||
Evidence, traced in code rather than recalled:
|
||||
|
||||
1. **`sprayDefense.js` and `platoonSeverity.js` are required by NOTHING in
|
||||
`src/`.** Only by analysis scripts and their own tests. The two
|
||||
causally-correct atoms that passed the two-part gate have never been on the
|
||||
serving path.
|
||||
|
||||
2. **The served `p_win` reads exactly four inputs**
|
||||
(`intelligence/probabilityEstimator.js:54`): game-log frequency over the line,
|
||||
`opp_rank_stat` (±0.03), `home_away` (±0.015), and a cv consistency pull.
|
||||
Zero occurrences of spray, platoon, hard-hit or contact-profile.
|
||||
|
||||
3. **Ordering makes it structural.** `snapshotService` grades at line 454
|
||||
(`gradeAndCacheSlate`) and only computes challenger/context at line 640+.
|
||||
Everything proven is computed **downstream of the grade it would inform**.
|
||||
|
||||
So the three proven hits factors — `defense_by_direction`,
|
||||
`pitcher_contact_profile`, `platoon_severity` — were measured on ledger rows by
|
||||
analysis scripts and **have never once moved a served number.**
|
||||
|
||||
That reframes every null in this programme's recent history. "Calibrated p_win
|
||||
does not separate within archetype" was never a statement about factors. The
|
||||
factors were not in the forecast.
|
||||
|
||||
---
|
||||
|
||||
## PHASE 3 — bands on served values
|
||||
|
||||
| stat | basis | eval rows | slots | slots with lift |
|
||||
|---|---|---|---|---|
|
||||
| hits | `p_win_lowparam` (served) | 765 | 7 | **0** |
|
||||
| total_bases | `p_win_lowparam` (served) | 625 | 7 | **0** |
|
||||
| rbi | raw `p_win` | 425 | 7 | **0** |
|
||||
| runs | raw `p_win` | 424 | 7 | **0** |
|
||||
|
||||
**28 archetype slots across four stats. Zero show lift.** Every slot is one band
|
||||
indistinguishable from its own base rate.
|
||||
|
||||
This is no longer an open shrug. It is the arithmetic consequence of resolution
|
||||
of 0.0013–0.0327 against uncertainty of ~0.23: **a forecast explaining 1% of the
|
||||
outcome's variance cannot produce bands that separate**, and no correction to its
|
||||
numbers will change that.
|
||||
|
||||
---
|
||||
|
||||
## THE HEADLINE
|
||||
|
||||
**Calibration is complete. It delivered honest numbers on two stats and ZERO
|
||||
grade separation, because the counter has no resolution — 1.03% of variance on
|
||||
hits, 0.56% on runs.**
|
||||
|
||||
**And the three proven hits factors are NOT WIRED INTO THE FORECAST AT ALL.**
|
||||
|
||||
Those two facts together are the programme's position. The second is the reason
|
||||
for the first, and it is a plumbing defect rather than a modelling wall — which
|
||||
makes it the cheapest high-value fix available.
|
||||
|
||||
**Per-archetype grades require proven factors that actually reach `p_win`. That
|
||||
is the next and central arc.** This is the last calibration order.
|
||||
|
||||
---
|
||||
|
||||
## Invariants
|
||||
|
||||
Serving unchanged from `74cf1ce` — this order logs and diagnoses. `p_win` never
|
||||
mutated; `p_win_lowparam` served, `p_win_isotonic_shadow` logged. No Bonferroni
|
||||
slot. Counter and frozen clusters verified file-by-file.
|
||||
@@ -0,0 +1,125 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* calibrationDuel — the forward adjudication of a bet we made against the
|
||||
* measurement.
|
||||
*
|
||||
* On identical held-out rows the ISOTONIC map beat the low-parameter one (hits
|
||||
* +0.0028, rbi +0.0042, total_bases tied). We serve the low-parameter map
|
||||
* anyway, on the argument that isotonic's in-window edge is daily structure
|
||||
* shared between the fit and evaluation windows. At 19 dates that argument
|
||||
* cannot be tested — LODO has 1.4-9.3% power against it.
|
||||
*
|
||||
* So it is a BET. This module is what makes it falsifiable: both maps are
|
||||
* computed on every prop, the shadow is logged, and once enough genuinely
|
||||
* out-of-window dates settle, the season adjudicates.
|
||||
*
|
||||
* ── THE RULE IS PRE-REGISTERED, IN CODE ──────────────────────────────────
|
||||
* Written before any forward date exists, so the bar cannot drift toward
|
||||
* whichever answer arrives:
|
||||
*
|
||||
* REFUTED >=10 forward dates AND isotonic beats low-param with a date-block
|
||||
* bootstrap CI excluding zero -> revert hits/TB to isotonic
|
||||
* UPHELD >=10 forward dates and it does not -> the bet was right
|
||||
* PENDING fewer than 10 forward dates -> no verdict, keep serving
|
||||
*
|
||||
* A date is FORWARD only if NEITHER map was fitted on it. Scoring on a date
|
||||
* inside either fit window would be asking which map memorised better.
|
||||
*/
|
||||
|
||||
const { knownNumber } = require('../../utils/known');
|
||||
|
||||
/** Forward dates required before the duel may return a verdict. */
|
||||
const MIN_FORWARD_DATES = 10;
|
||||
const ITERS = 4000;
|
||||
|
||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||
|
||||
function makeRnd(seed) {
|
||||
let s = seed >>> 0;
|
||||
return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
|
||||
}
|
||||
|
||||
const brier = (rows, key) => {
|
||||
const usable = rows.filter((r) => knownNumber(r[key]) !== null && knownNumber(r.won) !== null);
|
||||
if (!usable.length) return null;
|
||||
return mean(usable.map((r) => (knownNumber(r[key]) - knownNumber(r.won)) ** 2));
|
||||
};
|
||||
|
||||
/**
|
||||
* @param {Array} rows [{ date, won, served, shadow, fitted_through }]
|
||||
* @param {object} opts { minForwardDates, seed }
|
||||
*/
|
||||
function adjudicate(rows, opts = {}) {
|
||||
const minDates = opts.minForwardDates ?? MIN_FORWARD_DATES;
|
||||
|
||||
// FORWARD ONLY: a row counts when its date postdates the window BOTH maps
|
||||
// were fitted on. Rows without that provenance are dropped, never assumed.
|
||||
const forward = (rows || []).filter((r) => {
|
||||
if (!r || !r.date) return false;
|
||||
if (knownNumber(r.served) === null || knownNumber(r.shadow) === null) return false;
|
||||
if (knownNumber(r.won) === null) return false;
|
||||
if (!r.fitted_through) return false;
|
||||
return String(r.date) > String(r.fitted_through);
|
||||
});
|
||||
|
||||
const dates = [...new Set(forward.map((r) => String(r.date)))].sort();
|
||||
if (dates.length < minDates) {
|
||||
return {
|
||||
verdict: 'PENDING',
|
||||
forward_dates: dates.length,
|
||||
forward_rows: forward.length,
|
||||
dates_needed: minDates - dates.length,
|
||||
reason: `${dates.length} forward dates < ${minDates} — the season has not spoken yet`,
|
||||
action: 'keep serving the low-parameter map',
|
||||
};
|
||||
}
|
||||
|
||||
const bServed = brier(forward, 'served');
|
||||
const bShadow = brier(forward, 'shadow');
|
||||
if (bServed === null || bShadow === null) {
|
||||
return { verdict: 'PENDING', forward_dates: dates.length, reason: 'no scorable forward rows' };
|
||||
}
|
||||
|
||||
// Paired date-block bootstrap on (isotonic - lowparam). Negative means the
|
||||
// shadow is better, which is the direction that refutes us.
|
||||
const byDate = new Map();
|
||||
for (const r of forward) {
|
||||
if (!byDate.has(String(r.date))) byDate.set(String(r.date), []);
|
||||
byDate.get(String(r.date)).push(r);
|
||||
}
|
||||
const keys = [...byDate.keys()];
|
||||
const rnd = makeRnd(opts.seed ?? 20260807);
|
||||
const diffs = [];
|
||||
for (let it = 0; it < ITERS; it += 1) {
|
||||
const s = [];
|
||||
for (let i = 0; i < keys.length; i += 1) s.push(...byDate.get(keys[Math.floor(rnd() * keys.length)]));
|
||||
const a = brier(s, 'shadow');
|
||||
const b = brier(s, 'served');
|
||||
if (a === null || b === null) continue;
|
||||
diffs.push(a - b);
|
||||
}
|
||||
diffs.sort((a, b) => a - b);
|
||||
const ci = diffs.length
|
||||
? [round5(diffs[Math.floor(diffs.length * 0.025)]), round5(diffs[Math.floor(diffs.length * 0.975)])]
|
||||
: null;
|
||||
|
||||
const shadowWins = ci !== null && ci[1] < 0;
|
||||
return {
|
||||
verdict: shadowWins ? 'REFUTED' : 'UPHELD',
|
||||
forward_dates: dates.length,
|
||||
forward_rows: forward.length,
|
||||
brier_served_lowparam: round5(bServed),
|
||||
brier_shadow_isotonic: round5(bShadow),
|
||||
delta_isotonic_minus_lowparam: round5(bShadow - bServed),
|
||||
ci,
|
||||
reason: shadowWins
|
||||
? 'isotonic beats the served low-parameter map out-of-window with a date-block interval excluding zero — the capacity argument is refuted'
|
||||
: 'the served low-parameter map is not beaten out-of-window — the bet stands',
|
||||
action: shadowWins ? 'REVERT hits and total_bases to isotonic and log the reversal' : 'keep serving the low-parameter map',
|
||||
};
|
||||
}
|
||||
|
||||
const round5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
|
||||
|
||||
module.exports = { adjudicate, MIN_FORWARD_DATES };
|
||||
@@ -734,46 +734,65 @@ async function runSnapshot(sport, opts = {}) {
|
||||
console.warn(`[challenger] ${sp} skipped:`, e.message);
|
||||
}
|
||||
|
||||
// ── FORWARD CALIBRATION (LODO-gated, per stat) ────────────────────────
|
||||
// ── FORWARD CALIBRATION + THE SHADOW DUEL ─────────────────────────────
|
||||
// Fitted on games that are OVER, applied to tonight's props. `p_win` is NOT
|
||||
// touched — the counter stays byte-identical and the calibrated value rides
|
||||
// touched — the counter stays byte-identical and the corrected value rides
|
||||
// beside it, because a calibration map is a correction TO a forecast, not a
|
||||
// different forecast.
|
||||
//
|
||||
// WHICH STATS SERVE IS MEASURED, NOT ASSUMED. The deploy bar is leave-one-
|
||||
// date-out stability: refit dropping each settled date in turn, and the
|
||||
// improvement must never reverse. That is the right instrument for a monotone
|
||||
// shrink-to-observed layer — the factor gate's >=40 date-cluster interval
|
||||
// floor was built for a CAUSAL claim and does not bind here.
|
||||
// WHAT IS SERVED, AND WHY IT IS A BET RATHER THAN A RESULT. On identical
|
||||
// held-out rows the ISOTONIC map scored BETTER than the low-parameter one
|
||||
// (hits +0.0028, rbi +0.0042, total_bases tied). We serve the low-parameter
|
||||
// map anyway, on the argument that isotonic's in-window edge is daily
|
||||
// structure shared between the fit and evaluation windows and will not
|
||||
// transmit forward. At 19 dates no instrument here can test that argument —
|
||||
// LODO has 1.4-9.3% power — so it is a BET, not evidence.
|
||||
//
|
||||
// Measured 2026-08-07: total_bases passes at every held-size threshold. hits
|
||||
// FAILS (reverses on 2026-07-22 and 2026-07-26), so it is no longer served
|
||||
// calibrated even though it was — a stat that cannot survive dropping one day
|
||||
// was never calibrated, it was fitted to that day. rbi and runs also fail.
|
||||
// So both are computed on every prop and the shadow is logged. Real
|
||||
// out-of-window dates adjudicate it:
|
||||
//
|
||||
// `calibrated` is true only inside a band certified out-of-sample, and it is
|
||||
// what `chain.chainAcross` requires before it will compound anything. Removing
|
||||
// hits here makes hits props unstackable again, which is the honest
|
||||
// consequence of the measurement rather than a regression to work around.
|
||||
// PRE-REGISTERED: once >=10 forward dates have settled that NEITHER map was
|
||||
// fitted on, if isotonic beats low-param with a date-block bootstrap CI
|
||||
// excluding zero, the capacity argument is REFUTED and hits/TB revert to
|
||||
// isotonic. If low-param wins or ties, the bet was right. The season
|
||||
// decides, not the argument.
|
||||
//
|
||||
// Serving is unchanged until that bar is met. `calibrated` is true only inside
|
||||
// a band certified out-of-sample, and it is what `chain.chainAcross` requires
|
||||
// before it will compound anything.
|
||||
if (sp === 'mlb') {
|
||||
for (const stat of CALIBRATION_DEPLOYED) {
|
||||
try {
|
||||
const calSvc = deps.calibrationService || require('./model/lowParamService');
|
||||
const shadowSvc = deps.shadowCalibrationService || require('./model/calibrationService');
|
||||
const sbc = require('../utils/supabase').getSupabaseServiceClient();
|
||||
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat }) : null;
|
||||
// The shadow must never break serving: its own try, and a null shadow
|
||||
// simply means the duel has no entry for tonight.
|
||||
let shadow = null;
|
||||
try { shadow = sbc ? await shadowSvc.fromLedger(sbc, { sport: 'mlb', stat }) : null; } catch { shadow = null; }
|
||||
|
||||
if (calibrator) {
|
||||
let marked = 0;
|
||||
let marked = 0; let shadowed = 0;
|
||||
for (const g of enriched) {
|
||||
if (String(g.stat_type || g.stat || '').toLowerCase() !== stat) continue;
|
||||
const out = calibrator.calibrate(g.p_win);
|
||||
g.p_win_calibrated = out.p_calibrated;
|
||||
g.p_win_lowparam = out.p_calibrated; // named, so the duel is legible
|
||||
g.calibrated = out.calibrated;
|
||||
g.calibration_reason = out.reason;
|
||||
g.calibration_status = 'provisional';
|
||||
g.calibration_basis = CALIBRATION_BASIS[stat] || null;
|
||||
if (out.calibrated) marked += 1;
|
||||
if (shadow) {
|
||||
const sh = shadow.calibrate(g.p_win);
|
||||
// SHADOW ONLY. Never read by serving, never by chainAcross.
|
||||
g.p_win_isotonic_shadow = sh.p_calibrated;
|
||||
g.calibration_duel_fitted_through = shadow.fitted_through || null;
|
||||
if (sh.p_calibrated != null) shadowed += 1;
|
||||
}
|
||||
console.log(`[calibration] ${sp} ${stat} (low-param, PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, a=${calibrator.model.a} shrink=${calibrator.shrinkage}`);
|
||||
}
|
||||
console.log(`[calibration] ${sp} ${stat} (low-param, PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, a=${calibrator.model.a} shrink=${calibrator.shrinkage}; shadow logged on ${shadowed}`);
|
||||
} else {
|
||||
console.log(`[calibration] ${sp} ${stat} — no calibrator (thin history); nothing is stackable`);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* The forward adjudication of a bet made against the measurement.
|
||||
*
|
||||
* We serve the map that scored WORSE in-window, on an argument the sample
|
||||
* cannot test. These lock the rule that decides whether that argument survives —
|
||||
* written before any forward date exists, so the bar cannot drift toward
|
||||
* whichever answer arrives.
|
||||
*/
|
||||
|
||||
const duel = require('../../src/services/model/calibrationDuel');
|
||||
|
||||
/** Forward rows where `edge` favours the shadow when positive. */
|
||||
function rows(dates, perDate, edge, seed = 2) {
|
||||
let s = seed;
|
||||
const rnd = () => (s = (s * 1103515245 + 12345) % 2147483648) / 2147483648;
|
||||
const out = [];
|
||||
for (let d = 0; d < dates; d += 1) {
|
||||
for (let i = 0; i < perDate; i += 1) {
|
||||
const won = rnd() < 0.6 ? 1 : 0;
|
||||
const err = 0.25 + rnd() * 0.1;
|
||||
out.push({
|
||||
date: `2026-09-${String(d + 1).padStart(2, '0')}`,
|
||||
fitted_through: '2026-08-31',
|
||||
won,
|
||||
served: won ? 1 - err : err,
|
||||
shadow: won ? 1 - err + edge : err - edge,
|
||||
});
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('the rule is pre-registered and cannot be met early', () => {
|
||||
it('is PENDING below the forward-date bar, whatever the numbers say', () => {
|
||||
// Even with the shadow winning decisively, 5 dates is not a verdict.
|
||||
const v = duel.adjudicate(rows(5, 40, 0.15));
|
||||
expect(v.verdict).toBe('PENDING');
|
||||
expect(v.dates_needed).toBe(duel.MIN_FORWARD_DATES - 5);
|
||||
expect(v.action).toMatch(/keep serving/);
|
||||
});
|
||||
|
||||
it('REFUTES the bet when isotonic wins out-of-window at the bar', () => {
|
||||
const v = duel.adjudicate(rows(12, 40, 0.15));
|
||||
expect(v.verdict).toBe('REFUTED');
|
||||
expect(v.ci[1]).toBeLessThan(0);
|
||||
expect(v.action).toMatch(/REVERT/);
|
||||
});
|
||||
|
||||
it('UPHOLDS the bet when the served map is not beaten', () => {
|
||||
const v = duel.adjudicate(rows(12, 40, -0.15));
|
||||
expect(v.verdict).toBe('UPHELD');
|
||||
expect(v.action).toMatch(/keep serving/);
|
||||
});
|
||||
|
||||
it('UPHOLDS on a tie — the burden is on refutation, not on us', () => {
|
||||
const v = duel.adjudicate(rows(12, 40, 0));
|
||||
expect(v.verdict).toBe('UPHELD');
|
||||
});
|
||||
});
|
||||
|
||||
describe('only genuinely out-of-window dates count', () => {
|
||||
it('drops rows inside the fit window — that would score memorisation', () => {
|
||||
const inWindow = rows(12, 40, 0.15).map((r) => ({ ...r, fitted_through: '2026-12-31' }));
|
||||
const v = duel.adjudicate(inWindow);
|
||||
expect(v.verdict).toBe('PENDING');
|
||||
expect(v.forward_dates).toBe(0);
|
||||
});
|
||||
|
||||
it('drops rows with no fit provenance rather than assuming they are forward', () => {
|
||||
const noProv = rows(12, 40, 0.15).map(({ fitted_through, ...r }) => r);
|
||||
expect(duel.adjudicate(noProv).forward_dates).toBe(0);
|
||||
});
|
||||
|
||||
it('drops rows missing either map — a duel needs both entrants', () => {
|
||||
const half = rows(12, 40, 0.15).map((r) => ({ ...r, shadow: null }));
|
||||
expect(duel.adjudicate(half).forward_dates).toBe(0);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user