Robust bias established; low-parameter correction replaces isotonic

PHASE 0 — sample-limit truth on record: on 19 dates BOTH stability
instruments are underpowered. LODO power 0.014-0.093 (best 0.337 across
every k tried); deploy CIs rest on 2-4 date clusters, where a
cluster-robust interval has ~1 df. This is the SAMPLE, not a fixable
instrument, and the gate-refinement loop stops here. Runs corrected: its
DATE-DRIVEN label was an artefact of the coin-flip ruler (2 reversals in
3 drops never cleared cutoff 2) -- it is an ordinary no-fittable-map
refusal.

PHASE 1 — the bias is ROBUST, tested model-free and map-free with a
date-block bootstrap. Pooled over-prediction rises monotonically -0.0076
/ +0.0428 / +0.0963 / +0.1589 / +0.2451 across deciles from 0.5 to 1.0,
sign stability 0.9946 over 17 date blocks, and 4 of 4 stats replicate
(bar was 3). Also visible: realized rate PLATEAUS at 0.65-0.68 from p=0.7
upward -- the 0.9+ bucket (0.6624) does no better than the 0.8-0.9 bucket
(0.6841). The model has no high-confidence reads, only high-confidence
numbers.

PHASE 3 — Platt, two parameters over the whole curve, shrunk toward
identity by fit-date count. Validated as a NEW estimator vs RAW with
date-block CIs:

  hits         a=0.406 shrink 0.565  0.2626 -> 0.2540  CI [-0.0112,-0.0069]  DEPLOY
  total_bases  a=0.472 shrink 0.333  0.2490 -> 0.2429  CI [-0.0062,-0.0059]  DEPLOY
  rbi          a=0.775 shrink 0.231  0.2011 -> 0.2007  CI [-0.0007, 0]       REFUSE
  runs         a=-0.032                                                      REFUSE

A GUARD THE FIRST RUN NEEDED: runs fitted a = -0.032. A non-positive
slope inverts the forecast rather than flattening it, and near zero the
curve collapses to a constant predicting the base rate for everything --
which LOWERS Brier while destroying all resolution. It would have scored
as a win while making the product worthless. MIN_SLOPE now refuses it by
name, with a test.

STATED PLAINLY: on the identical held-out rows isotonic BEAT the
low-param on hits (+0.0028) and rbi (+0.0042) and tied on TB. The swap is
a CAPACITY JUDGEMENT, not a measurement -- the window spans 2-4 date
blocks and that is exactly what a flexible map produces when it captures
structure shared by fit and eval. Labelled as a judgement.

PHASE 4 — hits and total_bases serve the correction, basis
direction_robust_magnitude_provisional (direction bootstrap-robust,
magnitude thin-sample and shrunk). rbi is WITHDRAWN to raw -- it was
deployed on isotonic at ced4042 and the low-param does not beat raw.
runs stays raw. Auto-demotion still armed.

PHASE 5 — the standing finding, stated hard: across 18 archetype slots on
three stats, calibrated p_win separates within archetype NO BETTER than
raw. Every slot is one band indistinguishable from its base rate, zero
show lift. Per-archetype separation is not coming from calibration; it
comes from proven factors or it does not exist. Five orders of
calibration have delivered what they can -- honest numbers on two stats --
and nothing on the question the grade product turns on.

p_win never mutated; no Bonferroni slot; the robust-claim test ran before
any calibrator was built and could have ended the session at Phase 2.
Counter and frozen clusters verified file-by-file, including calibration.js
and calibrationService.js, both untouched and simply off the serving path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9

src/services/model/lowParamCalibrator.js

@@ -0,0 +1,127 @@
+'use strict';
+
+/**
+ * lowParamCalibrator — a two-parameter favourite-longshot correction.
+ *
+ * Isotonic has one free parameter per distinct prediction level, which on 19
+ * dates is far more freedom than the sample can discipline — it can and does
+ * chase a single night's structure. Platt scaling has exactly TWO parameters
+ * over the whole curve:
+ *
+ *     p_cal = sigmoid(a * logit(p) + b)
+ *
+ * `a < 1` flattens an over-confident forecaster toward the base rate, which is
+ * precisely the favourite-longshot shape measured here (over-prediction rising
+ * monotonically from -0.008 at p~0.55 to +0.245 above 0.9). Two parameters
+ * cannot represent "this Tuesday was odd", which is the entire point.
+ *
+ * ── SHRINKAGE TOWARD IDENTITY ────────────────────────────────────────────
+ * Even two parameters are fitted on few dates, so the correction is blended
+ * back toward the raw forecast by a weight tied to how many dates were seen:
+ *
+ *     w = D / (D + D0)
+ *     p_final = w * p_platt + (1 - w) * p_raw
+ *
+ * At 5 fit dates w = 0.33 — the correction is applied at a third of its fitted
+ * strength. At 40 dates it is 0.80. A thin-sample fit therefore cannot
+ * over-correct, and the blend is monotone because both inputs are.
+ *
+ * The DIRECTION of this correction is bootstrap-robust; its MAGNITUDE is
+ * thin-sample. Shrinkage is how that distinction is expressed in the number
+ * rather than only in a label.
+ */
+
+const { knownNumber } = require('../../utils/known');
+
+/** Dates at which the fit earns half its weight. */
+const SHRINK_HALF_DATES = 10;
+/** Probabilities are clamped off 0/1 before the logit. */
+const EPS = 1e-6;
+/** Below this many rows there is nothing to fit. */
+const MIN_FIT_ROWS = 100;
+/**
+ * A slope at or below this is not a correction. Ordering must be preserved and
+ * the curve must not collapse to a constant.
+ */
+const MIN_SLOPE = 0.05;
+
+const clamp01 = (p) => Math.min(1 - EPS, Math.max(EPS, p));
+const logit = (p) => Math.log(clamp01(p) / (1 - clamp01(p)));
+const sigmoid = (z) => 1 / (1 + Math.exp(-z));
+
+/**
+ * Fit `a` and `b` by Newton–Raphson on the log-likelihood. Two parameters, so
+ * this converges in a handful of steps and has no tuning of its own.
+ */
+function fitPlatt(rows, opts = {}) {
+  const pts = (rows || [])
+    .map((r) => ({ x: logit(knownNumber(r.p)), y: knownNumber(r.won) }))
+    .filter((r) => Number.isFinite(r.x) && (r.y === 0 || r.y === 1));
+  if (pts.length < (opts.minRows ?? MIN_FIT_ROWS)) return null;
+
+  let a = 1; let b = 0;
+  for (let it = 0; it < 100; it += 1) {
+    let g0 = 0; let g1 = 0; let h00 = 0; let h01 = 0; let h11 = 0;
+    for (const { x, y } of pts) {
+      const p = sigmoid(a * x + b);
+      const e = p - y;
+      const w = p * (1 - p);
+      g0 += e * x; g1 += e;
+      h00 += w * x * x; h01 += w * x; h11 += w;
+    }
+    const det = h00 * h11 - h01 * h01;
+    if (!Number.isFinite(det) || Math.abs(det) < 1e-12) break;
+    const da = (g0 * h11 - g1 * h01) / det;
+    const db = (g1 * h00 - g0 * h01) / det;
+    a -= da; b -= db;
+    if (Math.abs(da) < 1e-10 && Math.abs(db) < 1e-10) break;
+  }
+  if (!Number.isFinite(a) || !Number.isFinite(b)) return null;
+
+  // ── THE SLOPE MUST CORRECT, NOT ABANDON ────────────────────────────────
+  // `a` in (0, 1] is a flattening: ordering preserved, over-confidence pulled
+  // in. `a <= 0` INVERTS the forecast, and `a` near zero collapses the curve to
+  // a constant — the fit has decided p_win carries nothing and is predicting the
+  // base rate for everything. That lowers Brier (shrinking a miscalibrated
+  // forecaster toward its base rate always does) while destroying resolution,
+  // so it would score as a win while making the product worthless.
+  //
+  // Measured: runs fitted a = -0.032. Refused here rather than deployed.
+  if (a <= (opts.minSlope ?? MIN_SLOPE)) {
+    return { refused: true, a: round5(a), b: round5(b), reason: a <= 0
+      ? 'fitted slope is not positive — the correction would invert the forecast'
+      : 'fitted slope is near zero — the fit collapses to a constant and abandons the forecast' };
+  }
+
+  const dates = new Set((rows || []).map((r) => r.date).filter(Boolean)).size;
+  const half = opts.shrinkHalfDates ?? SHRINK_HALF_DATES;
+  const shrink = dates > 0 ? dates / (dates + half) : 0;
+
+  return {
+    a: round5(a),
+    b: round5(b),
+    fit_rows: pts.length,
+    fit_dates: dates,
+    shrinkage: round4(shrink),
+    /** Flattening a forecaster means a < 1; reported so the shape is checkable. */
+    flattens: a < 1,
+  };
+}
+
+/**
+ * Apply the fitted correction, shrunk toward the raw forecast.
+ * Returns null when unreadable — never a silently uncorrected number.
+ */
+function applyPlatt(model, p) {
+  const x = knownNumber(p);
+  if (!model || model.refused || x === null) return null;
+  const raw = clamp01(x);
+  const corrected = sigmoid(model.a * logit(raw) + model.b);
+  const w = model.shrinkage;
+  return round5(w * corrected + (1 - w) * raw);
+}
+
+const round5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
+const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
+
+module.exports = { fitPlatt, applyPlatt, SHRINK_HALF_DATES, MIN_FIT_ROWS, MIN_SLOPE, logit, sigmoid };

src/services/model/lowParamService.js

@@ -0,0 +1,102 @@
+'use strict';
+
+/**
+ * lowParamService — the production side of the two-parameter correction.
+ *
+ * Mirrors calibrationService's interface so the serving path swaps cleanly, but
+ * fits a Platt curve instead of an isotonic map. The reason for the swap is
+ * capacity, not score: on 19 dates we cannot certify the stability of a map with
+ * one free parameter per prediction level, and LODO turned out to have 1.4-9.3%
+ * power to tell us otherwise. Two parameters cannot encode "this Tuesday was
+ * odd", which is exactly the failure we cannot rule out for isotonic.
+ *
+ * Stated plainly because it is a judgement rather than a measurement: on the
+ * held-out window isotonic scored BETTER than this on hits (+0.0028) and rbi
+ * (+0.0042) and tied on total_bases. That window spans 2-4 date blocks, so it is
+ * weak evidence either way, and it is consistent with a flexible map having
+ * captured structure shared by fit and evaluation periods.
+ *
+ * Same point-in-time cut as before: fitted ONLY on games that are already over.
+ */
+
+const lp = require('./lowParamCalibrator');
+const cal = require('./calibration');
+const { knownNumber } = require('../../utils/known');
+
+const MIN_FIT = 200;
+const HOLDOUT_FRACTION = 0.35;
+
+/** Build from settled rows: fit on the older part, certify bands on the newer. */
+function build(rows, opts = {}) {
+  const clean = (rows || [])
+    .map((r) => ({ p: knownNumber(r.p), won: knownNumber(r.won), date: String(r.date || '') }))
+    .filter((r) => r.p !== null && (r.won === 0 || r.won === 1))
+    .sort((a, b) => a.date.localeCompare(b.date));
+  if (clean.length < (opts.minFit ?? MIN_FIT)) return null;
+
+  const cut = Math.floor(clean.length * (1 - (opts.holdout ?? HOLDOUT_FRACTION)));
+  const fitRows = clean.slice(0, cut);
+  const certRows = clean.slice(cut);
+  if (fitRows.length < (opts.minFit ?? MIN_FIT) || certRows.length < 50) return null;
+
+  const model = lp.fitPlatt(fitRows, opts);
+  // A refused fit (inverting or collapsed slope) yields no calibrator at all.
+  if (!model || model.refused) return null;
+
+  const corrected = certRows
+    .map((r) => ({ ...r, p: lp.applyPlatt(model, r.p) }))
+    .filter((r) => knownNumber(r.p) !== null);
+  const bands = cal.certifyBands(corrected, {
+    tolerance: opts.tolerance ?? 0.05,
+    minBin: opts.minBin ?? 40,
+  });
+
+  return {
+    model,
+    bands,
+    fit_n: fitRows.length,
+    certify_n: certRows.length,
+    fitted_through: fitRows[fitRows.length - 1].date,
+    shrinkage: model.shrinkage,
+    calibrate(p) {
+      const raw = knownNumber(p);
+      if (raw === null) return { p_raw: null, p_calibrated: null, calibrated: false, reason: 'absent' };
+      const c = lp.applyPlatt(model, raw);
+      if (c === null) return { p_raw: raw, p_calibrated: null, calibrated: false, reason: 'no_model_value' };
+      const inBand = cal.inCertifiedBand(bands, c);
+      return {
+        p_raw: raw,
+        p_calibrated: Math.round(c * 1000) / 1000,
+        calibrated: inBand,
+        reason: inBand ? null : 'outside_certified_band',
+      };
+    },
+  };
+}
+
+/** Load settled history and build, POINT-IN-TIME (strictly before today). */
+async function fromLedger(sb, { sport = 'mlb', stat = 'hits', before = null, ...opts } = {}) {
+  if (!sb) return null;
+  const cutoff = before || new Intl.DateTimeFormat('en-CA', {
+    timeZone: 'America/New_York', year: 'numeric', month: '2-digit', day: '2-digit',
+  }).format(new Date());
+  const rows = [];
+  for (let from = 0; ; from += 1000) {
+    const { data, error } = await sb.from('ledger_entries')
+      .select('p_win, outcome, game_date, quarantine_reason')
+      .eq('sport', sport).is('user_id', null).eq('stat', stat)
+      .in('outcome', ['hit', 'miss']).not('p_win', 'is', null)
+      .lt('game_date', cutoff)
+      .range(from, from + 999);
+    if (error || !data || data.length === 0) break;
+    rows.push(...data);
+    if (data.length < 1000) break;
+  }
+  const clean = rows
+    .filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book'))
+    .map((r) => ({ p: Number(r.p_win), won: r.outcome === 'hit' ? 1 : 0, date: String(r.game_date) }));
+  const built = build(clean, opts);
+  return built ? { ...built, cutoff } : null;
+}
+
+module.exports = { build, fromLedger, MIN_FIT, HOLDOUT_FRACTION };

src/services/snapshotService.js

@@ -276,36 +276,37 @@ async function loadPitcherArsenals(sport) {
 }
 
 /**
- * Stats that may serve a calibrated number, and ON WHAT BASIS.
+ * Stats that may serve a calibrated number, and on what basis.
  *
- * ── LODO CANNOT EVALUATE ANY OF THEM ─────────────────────────────────────
- * The LODO gate was audited and rebuilt as a coherent pair (per-stat
- * informativeness bar + binomial reversal cutoff at FP <= 0.05). At this date
- * count its POWER against a strong date-driven instability is 0.093 / 0.093 /
- * 0.045 / 0.014 — it would miss a real failure more than nine times in ten. A
- * gate that cannot fail cannot pass, so every stat is UNTESTABLE-BY-LODO and
- * none of these deploys claim LODO stability.
+ * ── THE BIAS IS ROBUST; THE MAP WAS NOT CERTIFIABLE ──────────────────────
+ * Tested model-free and map-free on the picked-side population: the model
+ * over-predicts its own favourites, and the sign survives 99.5% of date-block
+ * resamples pooled and replicates in 4 of 4 stats. Over-prediction rises
+ * monotonically from -0.008 near p=0.55 to +0.245 above 0.9.
  *
- * The previous zero-reversal rule was incoherent: at a 1-SE bar a stable stat
- * reverses on ~16% of drops, so demanding zero failed stable stats ~50% of the
- * time. Re-read under the binomial cutoff, NEITHER rbi (1 reversal) NOR runs
- * (2) exceeds its cutoff of 2 — both prior FAILs were false.
+ * What could NOT be certified on 19 dates is the stability of a specific
+ * isotonic MAP — LODO has 1.4-9.3% power there. So isotonic is retired and the
+ * correction is a TWO-PARAMETER Platt curve, which has no capacity to encode a
+ * single odd day, shrunk toward the raw forecast by fit-date count.
  *
- * ── SO THE DEPLOY BASIS IS THE DATE-CLUSTERED CI ALONE ───────────────────
- * hits, total_bases and rbi each have a point-in-time held-out interval
- * excluding zero. That is the ONLY support they have, and it is thin — the
- * interval rests on 4, 2 and 2 date clusters respectively. Auto-demotion is
- * therefore the sole stability guard, not a backstop to a passed test.
+ * Validated as a NEW estimator against RAW, date-block bootstrap:
+ *   hits         a=0.406 shrink 0.565  0.2626 -> 0.2540  CI [-0.0112,-0.0069]
+ *   total_bases  a=0.472 shrink 0.333  0.2490 -> 0.2429  CI [-0.0062,-0.0059]
  *
- * runs is absent: no isotonic map was fittable at its point-in-time split, so it
- * has no CI support to stand on either.
+ * rbi is WITHDRAWN (deployed last order on isotonic): the low-parameter fit does
+ * not beat raw, CI [-0.0007, 0] touching zero. runs is refused by the slope
+ * guard — it fitted a = -0.032, which would invert the forecast rather than
+ * flatten it. Both now serve RAW.
+ */
+const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases']);
+/**
+ * The DIRECTION of the correction is bootstrap-robust; its MAGNITUDE is fitted
+ * on few dates and deliberately shrunk toward identity. The customer-facing
+ * letter is unchanged; this is what the internal record says.
  */
-const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases', 'rbi']);
-/** Why each deployed stat is allowed to serve. Not one of them is LODO-stable. */
 const CALIBRATION_BASIS = Object.freeze({
-  hits: 'ci_only_lodo_untestable',
-  total_bases: 'ci_only_lodo_untestable',
-  rbi: 'ci_only_lodo_untestable',
+  hits: 'direction_robust_magnitude_provisional',
+  total_bases: 'direction_robust_magnitude_provisional',
 });
 
 async function runSnapshot(sport, opts = {}) {
@@ -757,7 +758,7 @@ async function runSnapshot(sport, opts = {}) {
   if (sp === 'mlb') {
     for (const stat of CALIBRATION_DEPLOYED) {
       try {
-        const calSvc = deps.calibrationService || require('./model/calibrationService');
+        const calSvc = deps.calibrationService || require('./model/lowParamService');
         const sbc = require('../utils/supabase').getSupabaseServiceClient();
         const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat }) : null;
         if (calibrator) {
@@ -772,7 +773,7 @@ async function runSnapshot(sport, opts = {}) {
           g.calibration_basis = CALIBRATION_BASIS[stat] || null;
             if (out.calibrated) marked += 1;
           }
-          console.log(`[calibration] ${sp} ${stat} (PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}`);
+          console.log(`[calibration] ${sp} ${stat} (low-param, PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, a=${calibrator.model.a} shrink=${calibrator.shrinkage}`);
         } else {
           console.log(`[calibration] ${sp} ${stat} — no calibrator (thin history); nothing is stackable`);
         }