From ced40421ed328f48f34c9cfc3365a9aea2bd0825 Mon Sep 17 00:00:00 2001 From: Kev Date: Thu, 6 Aug 2026 23:01:57 -0400 Subject: [PATCH] Audit the LODO instrument: it cannot evaluate any stat, and both prior FAILs were false MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PHASE 0 — the gate at 1f40014 was mine and was an incoherent pair. A 1-SE informativeness bar with a ZERO-reversal rule: at exactly 1 SE a stable stat's drop reverses with prob Phi(-1)=0.1587, so on four informative drops P(>=1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50. It failed stable stats half the time by construction. And the pooled n*=70 mis-credited EVERY stat -- too low for hits (own 77) and runs (81), too high for total_bases (60) and rbi (54). PHASE 1, blind. Per-stat (g, sigma_row): hits -0.01288/0.11251, TB -0.01380/0.10680, rbi -0.00884/0.06459, runs -0.00902/0.08080. All four clear z=1.96 at full n, so none is NO-EFFECT. Committed k=1 with per-stat n* and a binomial cutoff holding FP at 0.004-0.031. THE FINDING THAT DOMINATES: the test has no power. Against a strong instability (date-to-date SD equal to the effect) it detects a failure 1.4%-9.3% of the time, and across every k from 1.0 to 2.0 the best any stat reaches is 0.337. A gate that cannot fail cannot pass, so LODO_POWER_FLOOR=0.50 makes UNTESTABLE structural -- "could not test" can never read as "passed". PHASE 2/3 cold, at each stat's OWN n*: hits 5 informative, 0 reversals, cutoff 2, power 0.093 UNTESTABLE TB 5 informative, 0 reversals, cutoff 2, power 0.093 UNTESTABLE rbi 4 informative, 1 reversal, cutoff 2, power 0.045 UNTESTABLE runs 3 informative, 2 reversals, cutoff 2, power 0.014 UNTESTABLE Setting the power floor aside entirely, NOT ONE STAT EXCEEDS ITS CUTOFF. PHASE 4 — rbi's FAIL was false, as the order suspected. So was RUNS' -- which the order did not anticipate, having classified it DATE-DRIVEN on a 244-row reversal; two reversals in three drops does not clear a cutoff of 2. TB's PASS was vacuous: the test could not have failed it. hits' own n* is LARGER than the pooled one (77 vs 70), and it remains untestable. PHASE 5 — deploy basis is now the date-clustered CI alone: hits CI [-0.0139,-0.0097], 4 date clusters relabelled ci_only TB CI [-0.0061,-0.0045], 2 date clusters RELABELLED, kept rbi CI [-0.0092,-0.0010], 2 date clusters NEWLY DEPLOYED runs no fittable map at its split REFUSE, no CI either Every deployed stat carries calibration_basis ci_only_lodo_untestable and auto-demotion is the SOLE stability guard, not a backstop to a passed test. Stated plainly: those intervals rest on 2-4 date clusters, which is thin, and it is now the only support. rbi gains chainAcross stackability; its bands rebuilt on p_win_calibrated (425 rows) are every-archetype base_rate. runs is queued for the low-param calibrator for the ordinary reason -- no fittable map -- not on the date-driven finding, which was an artefact. PHASE 6 — the deploy set was set by a coin-flip-power ruler; it is now set by a per-stat power-coherent pre-committed test whose first act was to report that it cannot evaluate anything. The audit was permitted to wound the live deploy and did: total_bases lost its LODO claim. Standing question unchanged -- 18 archetype slots across three deployed stats, every one a single band indistinguishable from base rate. Blind ordering held. p_win never mutated. No Bonferroni slot. Counter and frozen clusters verified file-by-file. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9 --- scripts/derive-lodo-test.js | 198 ++++++++++++++++++++++ scripts/lodo-calibration.js | 44 +++-- specs/lodo-instrument-audit.md | 170 +++++++++++++++++++ src/services/model/calibrationRegistry.js | 72 +++++--- src/services/snapshotService.js | 46 +++-- tests/unit/calibrationDeployGate.test.js | 52 +++--- tests/unit/calibrationRegistry.test.js | 70 +++++--- 7 files changed, 545 insertions(+), 107 deletions(-) create mode 100644 scripts/derive-lodo-test.js create mode 100644 specs/lodo-instrument-audit.md diff --git a/scripts/derive-lodo-test.js b/scripts/derive-lodo-test.js new file mode 100644 index 0000000..4fbd025 --- /dev/null +++ b/scripts/derive-lodo-test.js @@ -0,0 +1,198 @@ +#!/usr/bin/env node +'use strict'; + +/** + * PHASE 1 — derive a COHERENT LODO test, blind to reversals. + * + * ── THE DEFECT BEING FIXED ─────────────────────────────────────────────── + * The gate at 1f40014 paired a 1-SE per-drop informativeness bar with a + * zero-reversal decision rule. Those two are incoherent. At exactly 1 SE, a + * genuinely STABLE stat's drop reverses with probability Phi(-1) = 0.159, so on + * four informative drops the chance of at least one reversal is + * 1 - 0.841^4 = 0.50. The rule failed stable stats half the time by construction. + * + * And n* was pooled across four stats whose signed effects differ several-fold, + * so "informative" meant different things for different stats while being + * treated as one number. + * + * ── THE FIX ────────────────────────────────────────────────────────────── + * The two halves have to be chosen together: + * + * informative bar n*_k = k^2 * (sigma_row / |g|)^2 PER STAT + * decision rule FAIL iff reversals > c, where under stability + * R ~ Binomial(D, Phi(-k)) and c is the smallest cutoff + * with P(R > c) <= 0.05 + * + * `g` is the mean SIGNED per-row improvement — the quantity whose sign a + * reversal flips. Not a mean-absolute, and not pooled: a reversal is a claim + * about THIS stat's effect changing sign. + * + * THIS SCRIPT PRINTS NO REVERSAL AND NO VERDICT. It is blind by construction and + * must run, and its constants be committed, before any stat is re-read. + * + * SUPABASE_URL=... node scripts/derive-lodo-test.js + */ + +require('dotenv').config(); +const fs = require('fs'); +const path = require('path'); +const { createClient } = require('@supabase/supabase-js'); +const cal = require('../src/services/model/calibration'); +const guards = require('../src/services/model/calibrationGuards'); +const { knownNumber } = require('../src/utils/known'); + +const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json'); +const STATS = ['hits', 'total_bases', 'rbi', 'runs']; +const PAGE = 1000; +/** |g| must clear this many SE at the stat's full n or there is no effect to test. */ +const EFFECT_Z = 1.96; +/** Target false-positive rate for the whole per-stat test. */ +const TARGET_FP = 0.05; + +const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs }; +const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null); + +/** Standard normal CDF (Abramowitz–Stegun 7.1.26 via erf). */ +function normCdf(z) { + const t = 1 / (1 + 0.2316419 * Math.abs(z)); + const d = 0.3989422804014327 * Math.exp(-z * z / 2); + const p = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + t * (-1.821255978 + t * 1.330274429)))); + return z >= 0 ? 1 - p : p; +} + +const binomPmf = (n, k, p) => { + let logC = 0; + for (let i = 0; i < k; i += 1) logC += Math.log(n - i) - Math.log(i + 1); + return Math.exp(logC + k * Math.log(p) + (n - k) * Math.log(1 - p)); +}; +/** P(R > c) for R ~ Binomial(n, p). */ +const binomTail = (n, c, p) => { + let s = 0; + for (let k = c + 1; k <= n; k += 1) s += binomPmf(n, k, p); + return s; +}; +/** Smallest cutoff c with P(R > c) <= target. */ +function cutoffFor(D, p, target) { + for (let c = 0; c <= D; c += 1) if (binomTail(D, c, p) <= target) return { cutoff: c, fp: binomTail(D, c, p) }; + return { cutoff: D, fp: 0 }; +} + +async function page(sb, t, s, f) { + const o = []; + for (let i = 0; ; i += PAGE) { + const { data, error } = await f(sb.from(t).select(s)).order('id', { ascending: true }).range(i, i + PAGE - 1); + if (error) throw error; + if (!data.length) break; + o.push(...data); + if (data.length < PAGE) break; + } + return o; +} +const isPreGame = (c, g) => { + const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000); + const d = et.toISOString().slice(0, 10); + return d < g || (d === g && et.getUTCHours() < 19); +}; + +(async () => { + const sb = createClient(process.env.SUPABASE_URL, + process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } }); + const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines; + + const snaps = await page(sb, 'model_snapshots', + 'id, game_date, captured_at, stat, player_key, line, side, p_win, refused', + (q) => q.eq('sport', 'mlb').in('stat', STATS)); + + const picked = new Map(); + for (const r of snaps) { + if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue; + const k = [r.game_date, r.stat, r.player_key, r.line].join('|'); + const prev = picked.get(k); + if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r); + } + guards.assertPickedSideDedup([...picked.values()].map((r) => ({ + propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win), + }))); + + const perStat = {}; + const dateSizes = {}; + + for (const stat of STATS) { + const rows = []; + for (const r of picked.values()) { + if (r.stat !== stat) continue; + const b = lines[`${r.game_date}|${r.player_key}`]; + const L = knownNumber(r.line); + if (!b || L === null || !r.side) continue; + const v = knownNumber(FIELD[stat](b)); + if (v === null) continue; + const over = v > L; + rows.push({ date: r.game_date, p: knownNumber(r.p_win), won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0 }); + } + const map = cal.fitIsotonic(rows.map((r) => ({ p: r.p, won: r.won }))); + if (!map) { perStat[stat] = { n: rows.length, fittable: false }; continue; } + + const d = []; + for (const r of rows) { + const pc = cal.applyIsotonic(map, r.p); + if (knownNumber(pc) === null) continue; + d.push((pc - r.won) ** 2 - (r.p - r.won) ** 2); + } + const g = mean(d); + const sigma = Math.sqrt(d.reduce((s, x) => s + (x - g) ** 2, 0) / (d.length - 1)); + const seFull = sigma / Math.sqrt(d.length); + + // Date sizes are sample STRUCTURE, not outcomes — safe to read here. + const sizes = new Map(); + for (const r of rows) sizes.set(r.date, (sizes.get(r.date) || 0) + 1); + dateSizes[stat] = [...sizes.values()].sort((a, b) => b - a); + + perStat[stat] = { + n: d.length, + g_signed: round5(g), + sigma_row: round5(sigma), + se_full: round5(seFull), + effect_z_at_full_n: round3(Math.abs(g) / seFull), + improves: g < 0, + no_effect: Math.abs(g) / seFull < EFFECT_Z, + fittable: true, + }; + } + + // ── Choose k jointly. Blind: uses only (g, sigma) and date SIZES. ── + const kTable = []; + for (const k of [1.0, 1.25, 1.5, 1.75, 2.0]) { + const pNoise = normCdf(-k); + const row = { k, per_drop_noise_prob: round4(pNoise), stats: {} }; + for (const stat of STATS) { + const ps = perStat[stat]; + if (!ps || !ps.fittable) continue; + const nStar = Math.ceil(k * k * (ps.sigma_row / Math.abs(ps.g_signed)) ** 2); + const D = (dateSizes[stat] || []).filter((n) => n >= nStar).length; + const { cutoff, fp } = D > 0 ? cutoffFor(D, pNoise, TARGET_FP) : { cutoff: null, fp: null }; + // FN at a stated alternative: date-to-date SD of the effect equals |g|. + const pAlt = D > 0 ? normCdf(-Math.abs(ps.g_signed) / Math.sqrt(ps.g_signed ** 2 + (ps.sigma_row ** 2) / nStar)) : null; + const fn = D > 0 && cutoff !== null ? 1 - binomTail(D, cutoff, pAlt) : null; + row.stats[stat] = { + n_star: nStar, informative_drops: D, cutoff, fp: fp === null ? null : round4(fp), + fn_at_tau_equals_g: fn === null ? null : round4(fn), + }; + } + kTable.push(row); + } + + console.log(JSON.stringify({ + phase: 'PHASE 1 — coherent LODO test derivation, BLIND', + defect_being_fixed: 'a 1-SE informative bar with a zero-reversal rule: P(>=1 reversal | stable, 4 drops) = 0.50', + per_stat_effect: perStat, + date_sizes: dateSizes, + k_selection_table: kTable, + target_fp: TARGET_FP, + blind: 'no reversal, no verdict, no reversing date referenced anywhere in this output', + }, null, 2)); + process.exit(0); +})().catch((e) => { console.error(e); process.exit(1); }); + +const round5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000); +const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000); +const round3 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 1000) / 1000); diff --git a/scripts/lodo-calibration.js b/scripts/lodo-calibration.js index 7c9e408..2b87c78 100644 --- a/scripts/lodo-calibration.js +++ b/scripts/lodo-calibration.js @@ -45,7 +45,7 @@ const FAVOURITE_FLOOR = 0.9; * from here -- it is imported so the value that decides the verdicts cannot be * edited alongside them. */ -const { LODO_MIN_HELD_ROWS: MIN_HELD_ROWS } = require('../src/services/model/calibrationRegistry'); +const { LODO_TEST, LODO_POWER_FLOOR } = require('../src/services/model/calibrationRegistry'); const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs }; const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null); @@ -131,7 +131,9 @@ async function main() { continue; } - // ── LEAVE ONE DATE OUT ── + // ── LEAVE ONE DATE OUT, at THIS stat's own informative bar ── + const spec = LODO_TEST[stat]; + const MIN_HELD_ROWS = spec ? spec.n_star : Infinity; const table = []; for (const d of dates) { const fit = rows.filter((r) => r.date !== d); @@ -168,10 +170,19 @@ async function main() { } const informative = table.filter((t) => t.verdict !== 'UNINFORMATIVE'); - const anyReversal = informative.some((t) => t.verdict === 'REVERSES'); - const signTested = informative.filter((t) => t.favourite_sign_holds !== null); - const anySignFlip = signTested.some((t) => t.favourite_sign_holds === false); - const passes = informative.length > 0 && !anyReversal && !anySignFlip; + const reversals = informative.filter((t) => t.verdict === 'REVERSES'); + + // THE DECISION RULE IS BINOMIAL, not zero-tolerance. Under stability each + // informative drop reverses with prob Phi(-k), so demanding zero reversals + // failed stable stats roughly half the time. + const cutoff = spec ? spec.cutoff : 0; + const exceedsCutoff = reversals.length > cutoff; + + // AND THE TEST MUST BE ABLE TO FAIL. Below the power floor it cannot, so it + // cannot pass either -- "could not test" must never read as "passed". + const underpowered = !spec || spec.power < LODO_POWER_FLOOR; + const verdict = underpowered ? 'UNTESTABLE_BY_LODO' : (exceedsCutoff ? 'FAIL' : 'PASS'); + const passes = verdict === 'PASS'; out.per_stat[stat] = { n: rows.length, @@ -179,14 +190,19 @@ async function main() { lodo_table: table, informative_drops: informative.length, brier_reversals: informative.filter((t) => t.verdict === 'REVERSES').length, - favourite_sign_flips: signTested.filter((t) => t.favourite_sign_holds === false).length, - favourite_sign_untested: informative.length - signTested.length, - lodo: passes ? 'PASS' : 'FAIL', - reason: passes - ? 'improvement never reverses and the favourite over-prediction never flips sign across any single-date drop' - : (anyReversal - ? `improvement reverses when ${informative.filter((t) => t.verdict === 'REVERSES').map((t) => t.dropped).join(', ')} is dropped — the effect is date-driven` - : `the favourite over-prediction flips sign when ${signTested.filter((t) => t.favourite_sign_holds === false).map((t) => t.dropped).join(', ')} is dropped`), + favourite_sign_flips: informative.filter((t) => t.favourite_sign_holds === false).length, + favourite_sign_untested: informative.filter((t) => t.favourite_sign_holds === null).length, + n_star: spec ? spec.n_star : null, + cutoff, + reversal_count: reversals.length, + reversing_dates: reversals.map((t) => ({ date: t.dropped, held_n: t.held_n, delta: t.brier_delta })), + test_power: spec ? spec.power : null, + lodo: verdict, + reason: underpowered + ? `power ${spec ? spec.power : 0} < floor ${LODO_POWER_FLOOR} — this test would miss a real date-driven failure more than nine times in ten, so it can neither pass nor fail the stat` + : (exceedsCutoff + ? `${reversals.length} reversals among ${informative.length} informative drops exceeds the cutoff of ${cutoff}` + : `${reversals.length} reversals among ${informative.length} informative drops is within the cutoff of ${cutoff}`), }; } diff --git a/specs/lodo-instrument-audit.md b/specs/lodo-instrument-audit.md new file mode 100644 index 0000000..32ac6dd --- /dev/null +++ b/specs/lodo-instrument-audit.md @@ -0,0 +1,170 @@ +# Auditing the LODO instrument — it cannot evaluate any stat + +**All four stats are UNTESTABLE-BY-LODO. Not one exceeds its reversal cutoff, so +rbi's and runs' prior FAILs were both false. And no stat may claim LODO +stability, because at this date count the test cannot fail.** + +--- + +## PHASE 0 — the defect, on record + +The gate at `1f40014` was mine, and it was an incoherent pair: + +**1. A 1-SE informativeness bar coupled to a ZERO-reversal rule.** At exactly 1 +SE, a genuinely STABLE stat's drop reverses with probability Φ(−1) = 0.1587. On +four informative drops: + +``` +P(>= 1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50 +``` + +**The rule failed stable stats half the time by construction.** A test cannot +have a 1-SE noise floor and a zero-tolerance decision rule; the two have to be +chosen together. + +**2. A pooled n\*.** The four stats' signed effects differ several-fold, so one +threshold meant four different things. Measured, the pooled 70 was: + +| stat | own n\* | pooled 70 was | +|---|---|---| +| hits | 77 | **too low** | +| total_bases | 60 | too high | +| rbi | 54 | too high | +| runs | 81 | **too low** | + +It mis-credited **every** stat, in both directions. + +Neither defect touched the counter or `p_win`. Both touched only which +calibrations were judged stable. + +--- + +## PHASE 1 — the coherent test, derived blind + +### (a)(b)(c) per-stat effect + +| stat | n | g (signed) | σ_row | SE_full | effect z | NO-EFFECT? | +|---|---|---|---|---|---|---| +| hits | 1,140 | −0.01288 | 0.11251 | 0.00333 | 3.87 | no | +| total_bases | 1,050 | −0.01380 | 0.10680 | 0.00330 | 4.19 | no | +| rbi | 630 | −0.00884 | 0.06459 | 0.00257 | 3.44 | no | +| runs | 597 | −0.00902 | 0.08080 | 0.00331 | 2.73 | no | + +All four have a real effect at full n. None is NO-EFFECT — there is something for +stability to be tested *of* in every case. + +### The committed test pair + +`k = 1`, chosen because informative drops (D) are the binding scarcity here and +k=1 maximises them while the binomial cutoff holds the false-positive rate. + +| stat | n\*=k²(σ/\|g\|)² | informative drops D | cutoff | FP | **power at τ=\|g\|** | +|---|---|---|---|---|---| +| hits | 77 | 5 | 2 | 0.031 | **0.093** | +| total_bases | 60 | 5 | 2 | 0.031 | **0.093** | +| rbi | 54 | 4 | 2 | 0.014 | **0.045** | +| runs | 81 | 3 | 2 | 0.004 | **0.014** | + +Per-drop noise probability under stability Φ(−1) = 0.1587. FAIL iff reversals > cutoff. + +### The finding that dominates everything else: the test has no power + +Against a **strong** instability — date-to-date SD of the effect equal to the +effect itself — this test detects a failure between **1.4% and 9.3%** of the +time. Across every k examined (1.0 → 2.0), the best any stat reaches is 0.337, +and reaching even that costs all but two informative drops. + +**A gate that cannot fail cannot pass.** `LODO_POWER_FLOOR = 0.50` makes that +structural: below it a stat is UNTESTABLE-BY-LODO regardless of its reversal +count, so "could not test" can never be read as "passed". + +Committed as `LODO_K`, `LODO_TEST`, `LODO_POWER_FLOOR`; a test recomputes each +n\* from (σ, g) and each cutoff from the binomial tail, and asserts the old +zero-reversal rule's ~0.50 false-fail rate. The stale pooled constant is nulled +so nothing can read it. + +--- + +## PHASE 2/3 — cold re-read at each stat's own n\* + +| stat | own n\* | informative | reversals | cutoff | power | verdict | +|---|---|---|---|---|---|---| +| hits | 77 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** | +| total_bases | 60 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** | +| rbi | 54 | 4 | 1 | 2 | 0.045 | **UNTESTABLE** | +| runs | 81 | 3 | 2 | 2 | 0.014 | **UNTESTABLE** | + +**Setting the power floor aside entirely, not one stat exceeds its cutoff.** + +--- + +## PHASE 4 — reconcile against 1f40014 + +| stat | 1f40014 | now | why it changed | +|---|---|---|---| +| hits | PASS | UNTESTABLE | the PASS was from a test that cannot fail; 0 reversals is uninformative at 9% power | +| total_bases | PASS | UNTESTABLE | same — and it "passed" at a pooled n\*=70 above its own 60, so its drop count was under-credited too | +| **rbi** | **FAIL** | UNTESTABLE (1 reversal, cutoff 2) | **FALSE FAIL.** One reversal on four drops is a ~16%-per-drop coin flip, not evidence | +| **runs** | **FAIL** | UNTESTABLE (2 reversals, cutoff 2) | **ALSO A FALSE FAIL** under the coherent rule — this was not anticipated | + +Answering the order's three questions directly: + +- **Is rbi's FAIL a false fail?** Yes. And so is runs' — which the order did not + anticipate, having classified runs as DATE-DRIVEN on the strength of a 244-row + reversal. Under a rule with a stated error rate, two reversals in three drops + does not clear the cutoff. +- **Was TB's PASS real?** No. It was vacuous: the test could not have failed it. +- **Does hits still pass at its own smaller n\*?** Its own n\* is *larger* (77 vs + the pooled 70), it still shows zero reversals, and it is still untestable. + +--- + +## PHASE 5 — deploy, withdraw, route + +| stat | basis | decision | +|---|---|---| +| hits | date-clustered CI [−0.0139, −0.0097], **4 date clusters** | DEPLOY-PROVISIONAL, relabelled `ci_only_lodo_untestable` | +| total_bases | CI [−0.0061, −0.0045], **2 date clusters** | **RELABELLED** — kept, no longer claims LODO stability | +| **rbi** | CI [−0.0092, −0.0010], **2 date clusters** | **NEWLY DEPLOYED** — its FAIL was false | +| runs | **no fittable map** at its point-in-time split | REFUSE — no CI to stand on either | + +Every deployed stat now carries `calibration_basis: 'ci_only_lodo_untestable'`. +**Auto-demotion is the sole stability guard**, not a backstop to a passed test. + +The honest weakness, stated rather than buried: those intervals rest on **2 to 4 +date clusters**. That is thin support, and it is now the *only* support. + +**rbi chainAcross stackability is newly granted**; hits' remains from `1f40014`. +rbi bands rebuilt on `p_win_calibrated` (425 eval rows): every archetype +indistinguishable from its base rate, two-bar rule keeping them `base_rate`. + +runs is **not** routed to the low-parameter calibrator on a date-driven finding — +that finding was an artefact. It is queued for the ordinary reason: no isotonic +map is fittable at its sample. + +--- + +## PHASE 6 — logged + +**The deploy set at `1f40014` was set by a coin-flip-power ruler.** It is now set +by a per-stat, power-coherent, pre-committed test with a stated false-positive +rate — and that test's first act was to report that it cannot evaluate anything, +which is a more useful answer than either verdict it replaced. + +**The audit was permitted to wound the live deploy, and did**: total_bases lost +its LODO claim and now stands on a two-date-cluster interval. That it could is +the integrity property. + +**Standing question unchanged.** Calibrated `p_win` still separates within +archetype no better than raw — 18 archetype slots across three deployed stats, +every one a single band indistinguishable from its base rate. Per-archetype +separation comes from proven factors or it does not exist. + +--- + +## Invariants + +Blind ordering held: (g, σ_row) and the committed (n\*, cutoff, power floor) were +derived and locked with no reversal or verdict in view, before any stat was +re-read. `p_win` never mutated. No Bonferroni slot consumed. Counter and frozen +clusters verified file-by-file. diff --git a/src/services/model/calibrationRegistry.js b/src/services/model/calibrationRegistry.js index 3c70a97..002c80f 100644 --- a/src/services/model/calibrationRegistry.js +++ b/src/services/model/calibrationRegistry.js @@ -29,41 +29,57 @@ const { knownNumber } = require('../../utils/known'); const STATUS = Object.freeze({ NONE: 'none', PROVISIONAL: 'provisional', PROMOTED: 'promoted' }); /** - * LODO_MIN_HELD_ROWS — POWER-DERIVED, PRE-COMMITTED, NOT OPERATOR-CHOSEN. + * THE LODO TEST — a COHERENT pair, replacing the incoherent one at 1f40014. * - * A leave-one-date-out reversal is only informative if that date's held-out - * Brier delta is distinguishable from zero at its row count. Below that, a - * "reversal" is a coin flip wearing a decimal point — which is what made the - * previous verdict depend on a number someone picked. + * ── WHAT WAS WRONG ─────────────────────────────────────────────────────── + * The previous gate paired a 1-SE per-drop informativeness bar with a + * ZERO-reversal decision rule. At exactly 1 SE a genuinely STABLE stat's drop + * reverses with probability Phi(-1) = 0.159, so on four informative drops the + * chance of at least one reversal is 1 - 0.841^4 = 0.50. The rule failed stable + * stats half the time BY CONSTRUCTION. And n* was pooled across four stats whose + * signed effects differ several-fold, so one number meant four different things: + * measured, the pooled 70 was too LOW for hits (77) and runs (81) and too HIGH + * for total_bases (60) and rbi (54). * - * The per-row Brier difference is d_i = (pc_i - y_i)^2 - (p_i - y_i)^2, so a - * date's delta is mean(d) and SE(n) = SD(d)/sqrt(n). The smallest n at which a - * typical effect clears one standard error is n* = (SD(d)/|effect|)^2. + * ── THE COHERENT PAIR ──────────────────────────────────────────────────── + * The bar and the rule are chosen TOGETHER, per stat, for a stated error rate: * - * Measured 2026-08-07, pooled across all four stats so that no single stat's - * verdict could shape the threshold that decides it: + * informative bar n*_k = k^2 * (sigma_row / |g|)^2 per stat + * decision rule FAIL iff reversals > cutoff, where under stability + * R ~ Binomial(D, Phi(-k)) and cutoff is the smallest c + * with P(R > c) <= 0.05 * - * pooled rows 3,417 - * SD(d) 0.09816 - * |effect| 0.01175 - * n* = (0.09816/0.01175)^2 = 69.8 -> 70 + * `g` is the mean SIGNED per-row improvement — the quantity whose sign a + * reversal flips. k = 1 is chosen because it maximises informative drops (D), + * which is the binding scarcity here, while the binomial cutoff holds the + * false-positive rate at 0.004-0.031 across the four stats. * - * SE-vs-n: n=20 gives effect/SE 0.54, n=50 gives 0.85, n=75 gives 1.04. So - * anything under ~70 held rows cannot tell a real reversal from noise. + * ── AND THE TEST STILL HAS ALMOST NO POWER ─────────────────────────────── + * At the stated alternative (date-to-date SD of the effect equal to |g| — a + * strong instability), power is 0.093 / 0.093 / 0.045 / 0.014. The test would + * MISS a real date-driven failure more than nine times in ten. Across every k + * examined, the best any stat reaches is 0.337. * - * DERIVED BLIND — the derivation script prints no stat verdict, no date and no - * reversal. It ran, and this constant was committed, before any stat was - * re-read. That ordering is the integrity property; a test locks the value so it - * cannot be silently tuned afterwards. + * So a PASS here means "no instability was detected", NOT "it is stable", and a + * gate that cannot fail is not a gate. LODO_POWER_FLOOR makes that structural: a + * stat whose test power falls below it is UNTESTABLE-BY-LODO and may not claim + * LODO stability at all, whatever its reversal count. + * + * Derived BLIND — the derivation script prints no reversal, no verdict and no + * reversing date. It ran, and these were committed, before any stat was re-read. */ -const LODO_MIN_HELD_ROWS = 70; -const LODO_THRESHOLD_BASIS = Object.freeze({ - pooled_rows: 3417, - per_row_brier_diff_sd: 0.09816, - pooled_effect_abs_mean: 0.01175, - rule: 'n* = (SD(d) / |effect|)^2', - derived_blind: true, +const LODO_K = 1.0; +/** Below this power the test cannot fail, so it cannot pass either. */ +const LODO_POWER_FLOOR = 0.50; +/** Per-stat, from (g, sigma_row) measured blind. */ +const LODO_TEST = Object.freeze({ + hits: { g: -0.01288, sigma_row: 0.11251, n_star: 77, informative_drops: 5, cutoff: 2, fp: 0.0310, power: 0.093 }, + total_bases: { g: -0.01380, sigma_row: 0.10680, n_star: 60, informative_drops: 5, cutoff: 2, fp: 0.0310, power: 0.093 }, + rbi: { g: -0.00884, sigma_row: 0.06459, n_star: 54, informative_drops: 4, cutoff: 2, fp: 0.0141, power: 0.045 }, + runs: { g: -0.00902, sigma_row: 0.08080, n_star: 81, informative_drops: 3, cutoff: 2, fp: 0.0040, power: 0.014 }, }); +/** Legacy name kept so nothing silently reads a stale pooled value. */ +const LODO_MIN_HELD_ROWS = null; /** The ORIGINAL floor, correctly scoped: promotion, not deploy. */ const PROMOTION_DATE_CLUSTERS = 40; @@ -144,4 +160,4 @@ function createRegistry(initial = {}) { return { deploy, reverify, serves, get, all, log: () => log.slice() }; } -module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS, LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS }; +module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS, LODO_K, LODO_TEST, LODO_POWER_FLOOR, LODO_MIN_HELD_ROWS }; diff --git a/src/services/snapshotService.js b/src/services/snapshotService.js index cf795b8..da6ce49 100644 --- a/src/services/snapshotService.js +++ b/src/services/snapshotService.js @@ -276,27 +276,37 @@ async function loadPitcherArsenals(sport) { } /** - * Stats whose calibration passed leave-one-date-out and may serve a calibrated - * number. PROVISIONAL: auto-demoted the first time the held-out interval stops - * excluding zero or the favourite over-prediction flips sign. + * Stats that may serve a calibrated number, and ON WHAT BASIS. * - * Gated at LODO_MIN_HELD_ROWS = 70, which is POWER-DERIVED and pre-committed: - * below ~70 held rows a date's Brier delta cannot be told from a coin flip, so a - * "reversal" there carries no information. + * ── LODO CANNOT EVALUATE ANY OF THEM ───────────────────────────────────── + * The LODO gate was audited and rebuilt as a coherent pair (per-stat + * informativeness bar + binomial reversal cutoff at FP <= 0.05). At this date + * count its POWER against a strong date-driven instability is 0.093 / 0.093 / + * 0.045 / 0.014 — it would miss a real failure more than nine times in ten. A + * gate that cannot fail cannot pass, so every stat is UNTESTABLE-BY-LODO and + * none of these deploys claim LODO stability. * - * hits was WITHDRAWN at 6ae11f1 and is RESTORED here. That is not a reversal of - * the earlier call — it was correct on the instrument available then, which - * admitted 20- and 25-row dates as evidence. With the threshold derived from - * power rather than chosen, hits reverses on nothing. The restoration came - * through the gate, not around it. + * The previous zero-reversal rule was incoherent: at a 1-SE bar a stable stat + * reverses on ~16% of drops, so demanding zero failed stable stats ~50% of the + * time. Re-read under the binomial cutoff, NEITHER rbi (1 reversal) NOR runs + * (2) exceeds its cutoff of 2 — both prior FAILs were false. * - * rbi and runs remain ABSENT, and their failures are NOT underpowered: each - * reverses on a date comfortably above the threshold (rbi 2026-08-01 n=99; runs - * 2026-08-01 n=86 and 2026-08-05 n=244). Those are DATE-DRIVEN failures — no - * threshold and no further accrual rescues them, and isotonic is fitting - * day-structure. Routed to the low-parameter calibrator queue. + * ── SO THE DEPLOY BASIS IS THE DATE-CLUSTERED CI ALONE ─────────────────── + * hits, total_bases and rbi each have a point-in-time held-out interval + * excluding zero. That is the ONLY support they have, and it is thin — the + * interval rests on 4, 2 and 2 date clusters respectively. Auto-demotion is + * therefore the sole stability guard, not a backstop to a passed test. + * + * runs is absent: no isotonic map was fittable at its point-in-time split, so it + * has no CI support to stand on either. */ -const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases']); +const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases', 'rbi']); +/** Why each deployed stat is allowed to serve. Not one of them is LODO-stable. */ +const CALIBRATION_BASIS = Object.freeze({ + hits: 'ci_only_lodo_untestable', + total_bases: 'ci_only_lodo_untestable', + rbi: 'ci_only_lodo_untestable', +}); async function runSnapshot(sport, opts = {}) { const sp = String(sport || '').toLowerCase(); @@ -759,6 +769,7 @@ async function runSnapshot(sport, opts = {}) { g.calibrated = out.calibrated; g.calibration_reason = out.reason; g.calibration_status = 'provisional'; + g.calibration_basis = CALIBRATION_BASIS[stat] || null; if (out.calibrated) marked += 1; } console.log(`[calibration] ${sp} ${stat} (PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}`); @@ -881,5 +892,6 @@ module.exports = { pushTickerItems, ACTIVE_SPORTS, CALIBRATION_DEPLOYED, + CALIBRATION_BASIS, __internals: { propKey, gradedAtFor, indexOdds, lastName, isTopGrade, DELTA_NOISE, DELTA_MOVE, TICKER_CAP }, }; diff --git a/tests/unit/calibrationDeployGate.test.js b/tests/unit/calibrationDeployGate.test.js index f580019..df9e015 100644 --- a/tests/unit/calibrationDeployGate.test.js +++ b/tests/unit/calibrationDeployGate.test.js @@ -1,43 +1,45 @@ 'use strict'; /** - * Which stats serve a calibrated number in the live pipeline. + * Which stats serve a calibrated number, and on what basis. * - * The rule this locks: a stat that cannot survive dropping a single settled date - * was never calibrated — it was fitted to that date. hits WAS served calibrated - * and is not any more, which is the honest consequence of measuring it. + * The thing these lock is that "could not test" never reads as "passed". LODO + * has 1.4%-9.3% power at this date count, so no deployed stat claims stability + * from it — each rides a thin date-clustered interval and auto-demotion alone. */ const snapshotService = require('../../src/services/snapshotService'); +const { LODO_TEST, LODO_POWER_FLOOR } = require('../../src/services/model/calibrationRegistry'); -describe('the deployed set is LODO-gated', () => { - it('serves the stats that passed LODO at the powered threshold', () => { - expect(snapshotService.CALIBRATION_DEPLOYED).toContain('total_bases'); - // hits was withdrawn at 6ae11f1 under a hand-chosen threshold that admitted - // 20-row dates as evidence, and is restored here because the powered - // instrument finds no reversal. Through the gate, not around it. - expect(snapshotService.CALIBRATION_DEPLOYED).toContain('hits'); +describe('the deploy set rests on the CI, not on a passed LODO', () => { + it('serves the three stats with a point-in-time interval excluding zero', () => { + expect(snapshotService.CALIBRATION_DEPLOYED).toEqual(['hits', 'total_bases', 'rbi']); }); - it('does NOT serve rbi or runs — both fail on dates ABOVE the threshold', () => { - // These are DATE-DRIVEN failures, not underpowered ones: rbi reverses on a - // 99-row date and runs on 86- and 244-row dates. No threshold rescues them. - for (const stat of ['rbi', 'runs']) { - expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain(stat); + it('does NOT serve runs — no fittable map, so no CI to stand on', () => { + expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain('runs'); + }); + + it('labels every deployed stat as LODO-untestable rather than LODO-stable', () => { + for (const stat of snapshotService.CALIBRATION_DEPLOYED) { + expect(snapshotService.CALIBRATION_BASIS[stat]).toBe('ci_only_lodo_untestable'); } }); - it('every deployed stat cleared the power-derived threshold, not a chosen one', () => { - const { LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS } = require('../../src/services/model/calibrationRegistry'); - expect(LODO_MIN_HELD_ROWS).toBe(70); - expect(LODO_THRESHOLD_BASIS.derived_blind).toBe(true); - // The reversing dates that keep rbi/runs out are all at or above it, so - // their exclusion cannot be an artefact of the threshold. - for (const n of [99, 86, 244]) expect(n).toBeGreaterThanOrEqual(LODO_MIN_HELD_ROWS); + it('no stat may claim LODO stability, because the test cannot fail', () => { + for (const t of Object.values(LODO_TEST)) expect(t.power).toBeLessThan(LODO_POWER_FLOOR); }); - it('is frozen, so a stat cannot be added at runtime without a code change', () => { + it('rbi and runs were FALSE FAILS under the old zero-reversal rule', () => { + // Re-read under the binomial cutoff: 1 and 2 reversals, cutoff 2 for both. + expect(LODO_TEST.rbi.cutoff).toBe(2); + expect(LODO_TEST.runs.cutoff).toBe(2); + // rbi returns to the deploy set on CI support; runs still has none. + expect(snapshotService.CALIBRATION_DEPLOYED).toContain('rbi'); + }); + + it('is frozen, so a stat cannot be added at runtime', () => { expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true); - expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('hits'); }).toThrow(); + expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('runs'); }).toThrow(); }); }); diff --git a/tests/unit/calibrationRegistry.test.js b/tests/unit/calibrationRegistry.test.js index da8e25f..149768c 100644 --- a/tests/unit/calibrationRegistry.test.js +++ b/tests/unit/calibrationRegistry.test.js @@ -119,34 +119,58 @@ describe('serving is band-limited', () => { }); }); -describe('the LODO held-row threshold is power-derived, not operator-chosen', () => { - const { LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS } = require('../../src/services/model/calibrationRegistry'); +describe('the LODO test is a COHERENT pair, not a bar plus an unrelated rule', () => { + const { LODO_K, LODO_TEST, LODO_POWER_FLOOR } = require('../../src/services/model/calibrationRegistry'); - it('is the value its own stated derivation produces', () => { - // n* = (SD(d) / |effect|)^2 -- recomputed here so the constant cannot drift - // away from the basis that justifies it. - const { per_row_brier_diff_sd: sd, pooled_effect_abs_mean: eff } = LODO_THRESHOLD_BASIS; - expect(Math.ceil((sd / eff) ** 2)).toBe(LODO_MIN_HELD_ROWS); + const normCdf = (z) => { + const t = 1 / (1 + 0.2316419 * Math.abs(z)); + const d = 0.3989422804014327 * Math.exp(-z * z / 2); + const p = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + t * (-1.821255978 + t * 1.330274429)))); + return z >= 0 ? 1 - p : p; + }; + const binomPmf = (n, k, p) => { + let logC = 0; + for (let i = 0; i < k; i += 1) logC += Math.log(n - i) - Math.log(i + 1); + return Math.exp(logC + k * Math.log(p) + (n - k) * Math.log(1 - p)); + }; + const tail = (n, c, p) => { let s = 0; for (let k = c + 1; k <= n; k += 1) s += binomPmf(n, k, p); return s; }; + + it('each n* is what its own (sigma_row, g) produce — no pooled value', () => { + for (const [stat, t] of Object.entries(LODO_TEST)) { + expect(Math.ceil(LODO_K ** 2 * (t.sigma_row / Math.abs(t.g)) ** 2)).toBe(t.n_star); + } + // And the four differ, which is exactly why one pooled number mis-credited them. + const stars = Object.values(LODO_TEST).map((t) => t.n_star); + expect(new Set(stars).size).toBeGreaterThan(1); }); - it('carries its derivation basis, and was derived blind', () => { - expect(LODO_THRESHOLD_BASIS.rule).toBe('n* = (SD(d) / |effect|)^2'); - expect(LODO_THRESHOLD_BASIS.derived_blind).toBe(true); - expect(LODO_THRESHOLD_BASIS.pooled_rows).toBeGreaterThan(1000); - }); - - it('rejects the thresholds that were previously chosen by hand', () => { - // 20 was the operator-chosen value whose verdict moved with it; anything - // below n* cannot distinguish a reversal from a coin flip. - for (const weak of [10, 20, 25, 30, 50]) { - const se = LODO_THRESHOLD_BASIS.per_row_brier_diff_sd / Math.sqrt(weak); - expect(LODO_THRESHOLD_BASIS.pooled_effect_abs_mean).toBeLessThan(se); - expect(weak).toBeLessThan(LODO_MIN_HELD_ROWS); + it('each cutoff is the smallest one holding the false-positive rate at 0.05', () => { + const p = normCdf(-LODO_K); + for (const [stat, t] of Object.entries(LODO_TEST)) { + expect(tail(t.informative_drops, t.cutoff, p)).toBeLessThanOrEqual(0.05); + if (t.cutoff > 0) expect(tail(t.informative_drops, t.cutoff - 1, p)).toBeGreaterThan(0.05); + expect(t.fp).toBeCloseTo(tail(t.informative_drops, t.cutoff, p), 3); } }); - it('is informative at the committed value', () => { - const se = LODO_THRESHOLD_BASIS.per_row_brier_diff_sd / Math.sqrt(LODO_MIN_HELD_ROWS); - expect(LODO_THRESHOLD_BASIS.pooled_effect_abs_mean).toBeGreaterThanOrEqual(se * 0.99); + it('the OLD rule is demonstrably incoherent — it failed stable stats ~half the time', () => { + // Zero-reversal rule at a 1-SE bar, on four informative drops. + const pNoise = normCdf(-1); + const falseFail = 1 - (1 - pNoise) ** 4; + expect(falseFail).toBeGreaterThan(0.45); + expect(falseFail).toBeLessThan(0.55); + }); + + it('every stat falls below the power floor, so none may claim LODO stability', () => { + // A gate that cannot fail is not a gate. This is the honest state at this + // date count, and the floor makes it structural rather than a footnote. + for (const [stat, t] of Object.entries(LODO_TEST)) { + expect(t.power).toBeLessThan(LODO_POWER_FLOOR); + } + }); + + it('the stale pooled threshold is nulled so nothing can read it', () => { + const { LODO_MIN_HELD_ROWS } = require('../../src/services/model/calibrationRegistry'); + expect(LODO_MIN_HELD_ROWS).toBeNull(); }); });