Instrument the calibration duel forward; diagnose the resolution ceiling

— the proven factors were never wired in

PHASE 0 — two truths recorded. The swap is a BET, not an OOS win:
isotonic beat low-param on identical held-out rows (hits +0.0028, rbi
+0.0042, TB tied) and we serve low-param anyway on an untestable prior
about shared daily structure. At 19 dates nothing here can test it. And
the MIN_SLOPE catch is preserved as standing rationale: a near-zero or
negative slope collapses toward base-rate-for-everything, which LOWERS
Brier while destroying all resolution -- a metric win that guts the
product.

PHASE 1 — the duel is now falsifiable. Both corrections computed on every
hits/TB prop; p_win_lowparam served, p_win_isotonic_shadow logged in its
own try so it can never break serving. calibrationDuel.adjudicate encodes
the rule IN CODE before any forward date exists: >=10 forward dates and
isotonic winning with a date-block CI excluding zero => REFUTED, revert;
otherwise UPHELD; under 10 dates PENDING regardless of the numbers. A
date counts as forward only if NEITHER map was fitted on it -- otherwise
we would be scoring which map memorised better. Nothing swaps now.

PHASE 2 — the ceiling, quantified via Murphy decomposition:

  stat   reliability  RESOLUTION  uncertainty  variance explained
  hits      0.01353     0.00252      0.24532        1.03%
  TB        0.01419     0.00442      0.24329        1.82%
  rbi       0.00654     0.03268      0.22531       14.51%
  runs      0.00788     0.00130      0.23182        0.56%

Calibration did exactly what theory says and nothing more: hits
reliability 0.01353 -> 0.00233 (-0.0112, 83% of the error removed) while
resolution moved -0.0002. Unexpected: rbi has 13x the resolution of hits
and is the one stat we do NOT serve corrected -- it needs calibration
least and discriminates most.

PHASE 2 DIAGNOSIS — NOT-TRANSMITTED, and not weak, ABSENT. Traced in code:
sprayDefense.js and platoonSeverity.js are required by NOTHING in src/,
only by analysis scripts and their own tests. The served p_win
(intelligence/probabilityEstimator.js:54) reads exactly four inputs --
game-log frequency, opp_rank_stat +/-0.03, home_away +/-0.015, and a cv
pull -- with zero occurrences of spray, platoon, hard-hit or
contact-profile. And snapshotService grades at line 454 while computing
challenger/context at 640+, so everything proven is computed DOWNSTREAM of
the grade it would inform. The three proven hits factors have never once
moved a served number.

That reframes the recent nulls: "calibrated p_win does not separate within
archetype" was never a statement about factors. The factors were not in
the forecast.

PHASE 3 — bands rebuilt on SERVED values (hits/TB low-param, rbi/runs
raw): 28 archetype slots across four stats, ZERO show lift. No longer an
open shrug -- it is the arithmetic of resolution 0.0013-0.0327 against
uncertainty ~0.23. A forecast explaining 1% of variance cannot produce
separating bands, and no correction to its numbers will change that.

HEADLINE: calibration is complete, delivered honest numbers on two stats
and zero grade separation, because the counter has no resolution -- and
the proven factors are not wired into the forecast at all. The second is
the reason for the first, and it is plumbing rather than a modelling wall.
Per-archetype grades need proven factors that actually reach p_win. Last
calibration order.

Serving unchanged from 74cf1ce. p_win never mutated. No Bonferroni slot.
Counter and frozen clusters verified file-by-file (15 modules).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-07 02:25:45 -04:00
parent 74cf1ce974
commit e872eff4ce
6 changed files with 550 additions and 22 deletions
+20 -4
View File
@@ -18,6 +18,7 @@ const fs = require('fs');
const path = require('path');
const { createClient } = require('@supabase/supabase-js');
const cal = require('../src/services/model/calibration');
const lp = require('../src/services/model/lowParamCalibrator');
const gb = require('../src/services/model/gradeBands');
const guards = require('../src/services/model/calibrationGuards');
const tl = require('../src/services/model/testLedger');
@@ -86,8 +87,23 @@ const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b)
for (const r of rows) perDate.set(r.date, (perDate.get(r.date) || 0) + 1);
let acc = 0; let cut = dates[dates.length - 1];
for (const d of dates) { acc += perDate.get(d); if (acc >= rows.length * 0.45) { cut = d; break; } }
const map = cal.fitIsotonic(rows.filter((r) => r.date < cut).map((r) => ({ p: r.p, won: r.won })));
const applied = guards.applyOrRefuse(map, rows.filter((r) => r.date >= cut), cal.applyIsotonic);
// Bands are built on the SERVED values. hits and total_bases serve the
// low-parameter correction; rbi and runs serve raw, so their bands are raw.
const DEPLOYED = ['hits', 'total_bases'];
const fitRows = rows.filter((r) => r.date < cut);
const evalRows = rows.filter((r) => r.date >= cut);
let applied;
let basis;
if (DEPLOYED.includes(STAT)) {
const model = lp.fitPlatt(fitRows);
applied = (!model || model.refused)
? { ok: false, reason: 'low-parameter fit refused', rows: [] }
: { ok: true, rows: evalRows.map((r) => ({ ...r, pc: lp.applyPlatt(model, r.p) })).filter((r) => r.pc != null) };
basis = 'p_win_lowparam (SERVED, provisional)';
} else {
applied = { ok: true, rows: evalRows.map((r) => ({ ...r, pc: r.p })) };
basis = 'raw p_win (this stat serves raw)';
}
if (!applied.ok) { console.log(JSON.stringify({ stat: STAT, refused: applied.reason })); process.exit(0); }
const mc = await tl.recordAndCount(tl.supabaseStore(sb), []).catch(() => ({ cumulative_tests: 1 }));
@@ -105,13 +121,13 @@ const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b)
cumulativeTests: mc.cumulative_tests,
// TB is CALIBRATED (provisional) but no factor is PROVEN for it.
proven: false,
calibrated: true,
calibrated: DEPLOYED.includes(STAT),
}));
}
console.log(JSON.stringify({
stat: STAT,
basis: 'p_win_calibrated (PROVISIONAL)',
basis,
eval_rows: applied.rows.length,
cumulative_tests: mc.cumulative_tests,
two_bar_note: 'calibrated YES, proven NO -> bands stay a base-rate read, now honestly numbered',