Instrument the calibration duel forward; diagnose the resolution ceiling

— the proven factors were never wired in PHASE 0 — two truths recorded. The swap is a BET, not an OOS win: isotonic beat low-param on identical held-out rows (hits +0.0028, rbi +0.0042, TB tied) and we serve low-param anyway on an untestable prior about shared daily structure. At 19 dates nothing here can test it. And the MIN_SLOPE catch is preserved as standing rationale: a near-zero or negative slope collapses toward base-rate-for-everything, which LOWERS Brier while destroying all resolution -- a metric win that guts the product. PHASE 1 — the duel is now falsifiable. Both corrections computed on every hits/TB prop; p_win_lowparam served, p_win_isotonic_shadow logged in its own try so it can never break serving. calibrationDuel.adjudicate encodes the rule IN CODE before any forward date exists: >=10 forward dates and isotonic winning with a date-block CI excluding zero => REFUTED, revert; otherwise UPHELD; under 10 dates PENDING regardless of the numbers. A date counts as forward only if NEITHER map was fitted on it -- otherwise we would be scoring which map memorised better. Nothing swaps now. PHASE 2 — the ceiling, quantified via Murphy decomposition: stat reliability RESOLUTION uncertainty variance explained hits 0.01353 0.00252 0.24532 1.03% TB 0.01419 0.00442 0.24329 1.82% rbi 0.00654 0.03268 0.22531 14.51% runs 0.00788 0.00130 0.23182 0.56% Calibration did exactly what theory says and nothing more: hits reliability 0.01353 -> 0.00233 (-0.0112, 83% of the error removed) while resolution moved -0.0002. Unexpected: rbi has 13x the resolution of hits and is the one stat we do NOT serve corrected -- it needs calibration least and discriminates most. PHASE 2 DIAGNOSIS — NOT-TRANSMITTED, and not weak, ABSENT. Traced in code: sprayDefense.js and platoonSeverity.js are required by NOTHING in src/, only by analysis scripts and their own tests. The served p_win (intelligence/probabilityEstimator.js:54) reads exactly four inputs -- game-log frequency, opp_rank_stat +/-0.03, home_away +/-0.015, and a cv pull -- with zero occurrences of spray, platoon, hard-hit or contact-profile. And snapshotService grades at line 454 while computing challenger/context at 640+, so everything proven is computed DOWNSTREAM of the grade it would inform. The three proven hits factors have never once moved a served number. That reframes the recent nulls: "calibrated p_win does not separate within archetype" was never a statement about factors. The factors were not in the forecast. PHASE 3 — bands rebuilt on SERVED values (hits/TB low-param, rbi/runs raw): 28 archetype slots across four stats, ZERO show lift. No longer an open shrug -- it is the arithmetic of resolution 0.0013-0.0327 against uncertainty ~0.23. A forecast explaining 1% of variance cannot produce separating bands, and no correction to its numbers will change that. HEADLINE: calibration is complete, delivered honest numbers on two stats and zero grade separation, because the counter has no resolution -- and the proven factors are not wired into the forecast at all. The second is the reason for the first, and it is plumbing rather than a modelling wall. Per-archetype grades need proven factors that actually reach p_win. Last calibration order. Serving unchanged from 74cf1ce. p_win never mutated. No Bonferroni slot. Counter and frozen clusters verified file-by-file (15 modules). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-07 02:25:45 -04:00
parent 74cf1ce974
commit e872eff4ce
6 changed files with 550 additions and 22 deletions
@@ -0,0 +1,125 @@
+'use strict';
+
+/**
+ * calibrationDuel — the forward adjudication of a bet we made against the
+ * measurement.
+ *
+ * On identical held-out rows the ISOTONIC map beat the low-parameter one (hits
+ * +0.0028, rbi +0.0042, total_bases tied). We serve the low-parameter map
+ * anyway, on the argument that isotonic's in-window edge is daily structure
+ * shared between the fit and evaluation windows. At 19 dates that argument
+ * cannot be tested — LODO has 1.4-9.3% power against it.
+ *
+ * So it is a BET. This module is what makes it falsifiable: both maps are
+ * computed on every prop, the shadow is logged, and once enough genuinely
+ * out-of-window dates settle, the season adjudicates.
+ *
+ * ── THE RULE IS PRE-REGISTERED, IN CODE ──────────────────────────────────
+ * Written before any forward date exists, so the bar cannot drift toward
+ * whichever answer arrives:
+ *
+ *   REFUTED   >=10 forward dates AND isotonic beats low-param with a date-block
+ *             bootstrap CI excluding zero  ->  revert hits/TB to isotonic
+ *   UPHELD    >=10 forward dates and it does not  ->  the bet was right
+ *   PENDING   fewer than 10 forward dates          ->  no verdict, keep serving
+ *
+ * A date is FORWARD only if NEITHER map was fitted on it. Scoring on a date
+ * inside either fit window would be asking which map memorised better.
+ */
+
+const { knownNumber } = require('../../utils/known');
+
+/** Forward dates required before the duel may return a verdict. */
+const MIN_FORWARD_DATES = 10;
+const ITERS = 4000;
+
+const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
+
+function makeRnd(seed) {
+  let s = seed >>> 0;
+  return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
+}
+
+const brier = (rows, key) => {
+  const usable = rows.filter((r) => knownNumber(r[key]) !== null && knownNumber(r.won) !== null);
+  if (!usable.length) return null;
+  return mean(usable.map((r) => (knownNumber(r[key]) - knownNumber(r.won)) ** 2));
+};
+
+/**
+ * @param {Array} rows [{ date, won, served, shadow, fitted_through }]
+ * @param {object} opts { minForwardDates, seed }
+ */
+function adjudicate(rows, opts = {}) {
+  const minDates = opts.minForwardDates ?? MIN_FORWARD_DATES;
+
+  // FORWARD ONLY: a row counts when its date postdates the window BOTH maps
+  // were fitted on. Rows without that provenance are dropped, never assumed.
+  const forward = (rows || []).filter((r) => {
+    if (!r || !r.date) return false;
+    if (knownNumber(r.served) === null || knownNumber(r.shadow) === null) return false;
+    if (knownNumber(r.won) === null) return false;
+    if (!r.fitted_through) return false;
+    return String(r.date) > String(r.fitted_through);
+  });
+
+  const dates = [...new Set(forward.map((r) => String(r.date)))].sort();
+  if (dates.length < minDates) {
+    return {
+      verdict: 'PENDING',
+      forward_dates: dates.length,
+      forward_rows: forward.length,
+      dates_needed: minDates - dates.length,
+      reason: `${dates.length} forward dates < ${minDates} — the season has not spoken yet`,
+      action: 'keep serving the low-parameter map',
+    };
+  }
+
+  const bServed = brier(forward, 'served');
+  const bShadow = brier(forward, 'shadow');
+  if (bServed === null || bShadow === null) {
+    return { verdict: 'PENDING', forward_dates: dates.length, reason: 'no scorable forward rows' };
+  }
+
+  // Paired date-block bootstrap on (isotonic - lowparam). Negative means the
+  // shadow is better, which is the direction that refutes us.
+  const byDate = new Map();
+  for (const r of forward) {
+    if (!byDate.has(String(r.date))) byDate.set(String(r.date), []);
+    byDate.get(String(r.date)).push(r);
+  }
+  const keys = [...byDate.keys()];
+  const rnd = makeRnd(opts.seed ?? 20260807);
+  const diffs = [];
+  for (let it = 0; it < ITERS; it += 1) {
+    const s = [];
+    for (let i = 0; i < keys.length; i += 1) s.push(...byDate.get(keys[Math.floor(rnd() * keys.length)]));
+    const a = brier(s, 'shadow');
+    const b = brier(s, 'served');
+    if (a === null || b === null) continue;
+    diffs.push(a - b);
+  }
+  diffs.sort((a, b) => a - b);
+  const ci = diffs.length
+    ? [round5(diffs[Math.floor(diffs.length * 0.025)]), round5(diffs[Math.floor(diffs.length * 0.975)])]
+    : null;
+
+  const shadowWins = ci !== null && ci[1] < 0;
+  return {
+    verdict: shadowWins ? 'REFUTED' : 'UPHELD',
+    forward_dates: dates.length,
+    forward_rows: forward.length,
+    brier_served_lowparam: round5(bServed),
+    brier_shadow_isotonic: round5(bShadow),
+    delta_isotonic_minus_lowparam: round5(bShadow - bServed),
+    ci,
+    reason: shadowWins
+      ? 'isotonic beats the served low-parameter map out-of-window with a date-block interval excluding zero — the capacity argument is refuted'
+      : 'the served low-parameter map is not beaten out-of-window — the bet stands',
+    action: shadowWins ? 'REVERT hits and total_bases to isotonic and log the reversal' : 'keep serving the low-parameter map',
+  };
+}
+
+const round5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
+
+module.exports = { adjudicate, MIN_FORWARD_DATES };