Instrument the calibration duel forward; diagnose the resolution ceiling
— the proven factors were never wired in
PHASE 0 — two truths recorded. The swap is a BET, not an OOS win:
isotonic beat low-param on identical held-out rows (hits +0.0028, rbi
+0.0042, TB tied) and we serve low-param anyway on an untestable prior
about shared daily structure. At 19 dates nothing here can test it. And
the MIN_SLOPE catch is preserved as standing rationale: a near-zero or
negative slope collapses toward base-rate-for-everything, which LOWERS
Brier while destroying all resolution -- a metric win that guts the
product.
PHASE 1 — the duel is now falsifiable. Both corrections computed on every
hits/TB prop; p_win_lowparam served, p_win_isotonic_shadow logged in its
own try so it can never break serving. calibrationDuel.adjudicate encodes
the rule IN CODE before any forward date exists: >=10 forward dates and
isotonic winning with a date-block CI excluding zero => REFUTED, revert;
otherwise UPHELD; under 10 dates PENDING regardless of the numbers. A
date counts as forward only if NEITHER map was fitted on it -- otherwise
we would be scoring which map memorised better. Nothing swaps now.
PHASE 2 — the ceiling, quantified via Murphy decomposition:
stat reliability RESOLUTION uncertainty variance explained
hits 0.01353 0.00252 0.24532 1.03%
TB 0.01419 0.00442 0.24329 1.82%
rbi 0.00654 0.03268 0.22531 14.51%
runs 0.00788 0.00130 0.23182 0.56%
Calibration did exactly what theory says and nothing more: hits
reliability 0.01353 -> 0.00233 (-0.0112, 83% of the error removed) while
resolution moved -0.0002. Unexpected: rbi has 13x the resolution of hits
and is the one stat we do NOT serve corrected -- it needs calibration
least and discriminates most.
PHASE 2 DIAGNOSIS — NOT-TRANSMITTED, and not weak, ABSENT. Traced in code:
sprayDefense.js and platoonSeverity.js are required by NOTHING in src/,
only by analysis scripts and their own tests. The served p_win
(intelligence/probabilityEstimator.js:54) reads exactly four inputs --
game-log frequency, opp_rank_stat +/-0.03, home_away +/-0.015, and a cv
pull -- with zero occurrences of spray, platoon, hard-hit or
contact-profile. And snapshotService grades at line 454 while computing
challenger/context at 640+, so everything proven is computed DOWNSTREAM of
the grade it would inform. The three proven hits factors have never once
moved a served number.
That reframes the recent nulls: "calibrated p_win does not separate within
archetype" was never a statement about factors. The factors were not in
the forecast.
PHASE 3 — bands rebuilt on SERVED values (hits/TB low-param, rbi/runs
raw): 28 archetype slots across four stats, ZERO show lift. No longer an
open shrug -- it is the arithmetic of resolution 0.0013-0.0327 against
uncertainty ~0.23. A forecast explaining 1% of variance cannot produce
separating bands, and no correction to its numbers will change that.
HEADLINE: calibration is complete, delivered honest numbers on two stats
and zero grade separation, because the counter has no resolution -- and
the proven factors are not wired into the forecast at all. The second is
the reason for the first, and it is plumbing rather than a modelling wall.
Per-archetype grades need proven factors that actually reach p_win. Last
calibration order.
Serving unchanged from 74cf1ce. p_win never mutated. No Bonferroni slot.
Counter and frozen clusters verified file-by-file (15 modules).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -18,6 +18,7 @@ const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cal = require('../src/services/model/calibration');
|
||||
const lp = require('../src/services/model/lowParamCalibrator');
|
||||
const gb = require('../src/services/model/gradeBands');
|
||||
const guards = require('../src/services/model/calibrationGuards');
|
||||
const tl = require('../src/services/model/testLedger');
|
||||
@@ -86,8 +87,23 @@ const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b)
|
||||
for (const r of rows) perDate.set(r.date, (perDate.get(r.date) || 0) + 1);
|
||||
let acc = 0; let cut = dates[dates.length - 1];
|
||||
for (const d of dates) { acc += perDate.get(d); if (acc >= rows.length * 0.45) { cut = d; break; } }
|
||||
const map = cal.fitIsotonic(rows.filter((r) => r.date < cut).map((r) => ({ p: r.p, won: r.won })));
|
||||
const applied = guards.applyOrRefuse(map, rows.filter((r) => r.date >= cut), cal.applyIsotonic);
|
||||
// Bands are built on the SERVED values. hits and total_bases serve the
|
||||
// low-parameter correction; rbi and runs serve raw, so their bands are raw.
|
||||
const DEPLOYED = ['hits', 'total_bases'];
|
||||
const fitRows = rows.filter((r) => r.date < cut);
|
||||
const evalRows = rows.filter((r) => r.date >= cut);
|
||||
let applied;
|
||||
let basis;
|
||||
if (DEPLOYED.includes(STAT)) {
|
||||
const model = lp.fitPlatt(fitRows);
|
||||
applied = (!model || model.refused)
|
||||
? { ok: false, reason: 'low-parameter fit refused', rows: [] }
|
||||
: { ok: true, rows: evalRows.map((r) => ({ ...r, pc: lp.applyPlatt(model, r.p) })).filter((r) => r.pc != null) };
|
||||
basis = 'p_win_lowparam (SERVED, provisional)';
|
||||
} else {
|
||||
applied = { ok: true, rows: evalRows.map((r) => ({ ...r, pc: r.p })) };
|
||||
basis = 'raw p_win (this stat serves raw)';
|
||||
}
|
||||
if (!applied.ok) { console.log(JSON.stringify({ stat: STAT, refused: applied.reason })); process.exit(0); }
|
||||
|
||||
const mc = await tl.recordAndCount(tl.supabaseStore(sb), []).catch(() => ({ cumulative_tests: 1 }));
|
||||
@@ -105,13 +121,13 @@ const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b)
|
||||
cumulativeTests: mc.cumulative_tests,
|
||||
// TB is CALIBRATED (provisional) but no factor is PROVEN for it.
|
||||
proven: false,
|
||||
calibrated: true,
|
||||
calibrated: DEPLOYED.includes(STAT),
|
||||
}));
|
||||
}
|
||||
|
||||
console.log(JSON.stringify({
|
||||
stat: STAT,
|
||||
basis: 'p_win_calibrated (PROVISIONAL)',
|
||||
basis,
|
||||
eval_rows: applied.rows.length,
|
||||
cumulative_tests: mc.cumulative_tests,
|
||||
two_bar_note: 'calibrated YES, proven NO -> bands stay a base-rate read, now honestly numbered',
|
||||
|
||||
Reference in New Issue
Block a user