Audit the LODO instrument: it cannot evaluate any stat, and both prior
FAILs were false
PHASE 0 — the gate at 1f40014 was mine and was an incoherent pair. A 1-SE
informativeness bar with a ZERO-reversal rule: at exactly 1 SE a stable
stat's drop reverses with prob Phi(-1)=0.1587, so on four informative
drops P(>=1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50. It failed
stable stats half the time by construction. And the pooled n*=70
mis-credited EVERY stat -- too low for hits (own 77) and runs (81), too
high for total_bases (60) and rbi (54).
PHASE 1, blind. Per-stat (g, sigma_row): hits -0.01288/0.11251, TB
-0.01380/0.10680, rbi -0.00884/0.06459, runs -0.00902/0.08080. All four
clear z=1.96 at full n, so none is NO-EFFECT. Committed k=1 with per-stat
n* and a binomial cutoff holding FP at 0.004-0.031.
THE FINDING THAT DOMINATES: the test has no power. Against a strong
instability (date-to-date SD equal to the effect) it detects a failure
1.4%-9.3% of the time, and across every k from 1.0 to 2.0 the best any
stat reaches is 0.337. A gate that cannot fail cannot pass, so
LODO_POWER_FLOOR=0.50 makes UNTESTABLE structural -- "could not test" can
never read as "passed".
PHASE 2/3 cold, at each stat's OWN n*:
hits 5 informative, 0 reversals, cutoff 2, power 0.093 UNTESTABLE
TB 5 informative, 0 reversals, cutoff 2, power 0.093 UNTESTABLE
rbi 4 informative, 1 reversal, cutoff 2, power 0.045 UNTESTABLE
runs 3 informative, 2 reversals, cutoff 2, power 0.014 UNTESTABLE
Setting the power floor aside entirely, NOT ONE STAT EXCEEDS ITS CUTOFF.
PHASE 4 — rbi's FAIL was false, as the order suspected. So was RUNS' --
which the order did not anticipate, having classified it DATE-DRIVEN on a
244-row reversal; two reversals in three drops does not clear a cutoff of
2. TB's PASS was vacuous: the test could not have failed it. hits' own n*
is LARGER than the pooled one (77 vs 70), and it remains untestable.
PHASE 5 — deploy basis is now the date-clustered CI alone:
hits CI [-0.0139,-0.0097], 4 date clusters relabelled ci_only
TB CI [-0.0061,-0.0045], 2 date clusters RELABELLED, kept
rbi CI [-0.0092,-0.0010], 2 date clusters NEWLY DEPLOYED
runs no fittable map at its split REFUSE, no CI either
Every deployed stat carries calibration_basis ci_only_lodo_untestable and
auto-demotion is the SOLE stability guard, not a backstop to a passed
test. Stated plainly: those intervals rest on 2-4 date clusters, which is
thin, and it is now the only support. rbi gains chainAcross stackability;
its bands rebuilt on p_win_calibrated (425 rows) are every-archetype
base_rate. runs is queued for the low-param calibrator for the ordinary
reason -- no fittable map -- not on the date-driven finding, which was an
artefact.
PHASE 6 — the deploy set was set by a coin-flip-power ruler; it is now set
by a per-stat power-coherent pre-committed test whose first act was to
report that it cannot evaluate anything. The audit was permitted to wound
the live deploy and did: total_bases lost its LODO claim. Standing
question unchanged -- 18 archetype slots across three deployed stats, every
one a single band indistinguishable from base rate.
Blind ordering held. p_win never mutated. No Bonferroni slot. Counter and
frozen clusters verified file-by-file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,198 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* PHASE 1 — derive a COHERENT LODO test, blind to reversals.
|
||||||
|
*
|
||||||
|
* ── THE DEFECT BEING FIXED ───────────────────────────────────────────────
|
||||||
|
* The gate at 1f40014 paired a 1-SE per-drop informativeness bar with a
|
||||||
|
* zero-reversal decision rule. Those two are incoherent. At exactly 1 SE, a
|
||||||
|
* genuinely STABLE stat's drop reverses with probability Phi(-1) = 0.159, so on
|
||||||
|
* four informative drops the chance of at least one reversal is
|
||||||
|
* 1 - 0.841^4 = 0.50. The rule failed stable stats half the time by construction.
|
||||||
|
*
|
||||||
|
* And n* was pooled across four stats whose signed effects differ several-fold,
|
||||||
|
* so "informative" meant different things for different stats while being
|
||||||
|
* treated as one number.
|
||||||
|
*
|
||||||
|
* ── THE FIX ──────────────────────────────────────────────────────────────
|
||||||
|
* The two halves have to be chosen together:
|
||||||
|
*
|
||||||
|
* informative bar n*_k = k^2 * (sigma_row / |g|)^2 PER STAT
|
||||||
|
* decision rule FAIL iff reversals > c, where under stability
|
||||||
|
* R ~ Binomial(D, Phi(-k)) and c is the smallest cutoff
|
||||||
|
* with P(R > c) <= 0.05
|
||||||
|
*
|
||||||
|
* `g` is the mean SIGNED per-row improvement — the quantity whose sign a
|
||||||
|
* reversal flips. Not a mean-absolute, and not pooled: a reversal is a claim
|
||||||
|
* about THIS stat's effect changing sign.
|
||||||
|
*
|
||||||
|
* THIS SCRIPT PRINTS NO REVERSAL AND NO VERDICT. It is blind by construction and
|
||||||
|
* must run, and its constants be committed, before any stat is re-read.
|
||||||
|
*
|
||||||
|
* SUPABASE_URL=... node scripts/derive-lodo-test.js
|
||||||
|
*/
|
||||||
|
|
||||||
|
require('dotenv').config();
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const { createClient } = require('@supabase/supabase-js');
|
||||||
|
const cal = require('../src/services/model/calibration');
|
||||||
|
const guards = require('../src/services/model/calibrationGuards');
|
||||||
|
const { knownNumber } = require('../src/utils/known');
|
||||||
|
|
||||||
|
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
|
||||||
|
const STATS = ['hits', 'total_bases', 'rbi', 'runs'];
|
||||||
|
const PAGE = 1000;
|
||||||
|
/** |g| must clear this many SE at the stat's full n or there is no effect to test. */
|
||||||
|
const EFFECT_Z = 1.96;
|
||||||
|
/** Target false-positive rate for the whole per-stat test. */
|
||||||
|
const TARGET_FP = 0.05;
|
||||||
|
|
||||||
|
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
|
||||||
|
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||||
|
|
||||||
|
/** Standard normal CDF (Abramowitz–Stegun 7.1.26 via erf). */
|
||||||
|
function normCdf(z) {
|
||||||
|
const t = 1 / (1 + 0.2316419 * Math.abs(z));
|
||||||
|
const d = 0.3989422804014327 * Math.exp(-z * z / 2);
|
||||||
|
const p = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + t * (-1.821255978 + t * 1.330274429))));
|
||||||
|
return z >= 0 ? 1 - p : p;
|
||||||
|
}
|
||||||
|
|
||||||
|
const binomPmf = (n, k, p) => {
|
||||||
|
let logC = 0;
|
||||||
|
for (let i = 0; i < k; i += 1) logC += Math.log(n - i) - Math.log(i + 1);
|
||||||
|
return Math.exp(logC + k * Math.log(p) + (n - k) * Math.log(1 - p));
|
||||||
|
};
|
||||||
|
/** P(R > c) for R ~ Binomial(n, p). */
|
||||||
|
const binomTail = (n, c, p) => {
|
||||||
|
let s = 0;
|
||||||
|
for (let k = c + 1; k <= n; k += 1) s += binomPmf(n, k, p);
|
||||||
|
return s;
|
||||||
|
};
|
||||||
|
/** Smallest cutoff c with P(R > c) <= target. */
|
||||||
|
function cutoffFor(D, p, target) {
|
||||||
|
for (let c = 0; c <= D; c += 1) if (binomTail(D, c, p) <= target) return { cutoff: c, fp: binomTail(D, c, p) };
|
||||||
|
return { cutoff: D, fp: 0 };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function page(sb, t, s, f) {
|
||||||
|
const o = [];
|
||||||
|
for (let i = 0; ; i += PAGE) {
|
||||||
|
const { data, error } = await f(sb.from(t).select(s)).order('id', { ascending: true }).range(i, i + PAGE - 1);
|
||||||
|
if (error) throw error;
|
||||||
|
if (!data.length) break;
|
||||||
|
o.push(...data);
|
||||||
|
if (data.length < PAGE) break;
|
||||||
|
}
|
||||||
|
return o;
|
||||||
|
}
|
||||||
|
const isPreGame = (c, g) => {
|
||||||
|
const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000);
|
||||||
|
const d = et.toISOString().slice(0, 10);
|
||||||
|
return d < g || (d === g && et.getUTCHours() < 19);
|
||||||
|
};
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
const sb = createClient(process.env.SUPABASE_URL,
|
||||||
|
process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } });
|
||||||
|
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
|
||||||
|
|
||||||
|
const snaps = await page(sb, 'model_snapshots',
|
||||||
|
'id, game_date, captured_at, stat, player_key, line, side, p_win, refused',
|
||||||
|
(q) => q.eq('sport', 'mlb').in('stat', STATS));
|
||||||
|
|
||||||
|
const picked = new Map();
|
||||||
|
for (const r of snaps) {
|
||||||
|
if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue;
|
||||||
|
const k = [r.game_date, r.stat, r.player_key, r.line].join('|');
|
||||||
|
const prev = picked.get(k);
|
||||||
|
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r);
|
||||||
|
}
|
||||||
|
guards.assertPickedSideDedup([...picked.values()].map((r) => ({
|
||||||
|
propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win),
|
||||||
|
})));
|
||||||
|
|
||||||
|
const perStat = {};
|
||||||
|
const dateSizes = {};
|
||||||
|
|
||||||
|
for (const stat of STATS) {
|
||||||
|
const rows = [];
|
||||||
|
for (const r of picked.values()) {
|
||||||
|
if (r.stat !== stat) continue;
|
||||||
|
const b = lines[`${r.game_date}|${r.player_key}`];
|
||||||
|
const L = knownNumber(r.line);
|
||||||
|
if (!b || L === null || !r.side) continue;
|
||||||
|
const v = knownNumber(FIELD[stat](b));
|
||||||
|
if (v === null) continue;
|
||||||
|
const over = v > L;
|
||||||
|
rows.push({ date: r.game_date, p: knownNumber(r.p_win), won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0 });
|
||||||
|
}
|
||||||
|
const map = cal.fitIsotonic(rows.map((r) => ({ p: r.p, won: r.won })));
|
||||||
|
if (!map) { perStat[stat] = { n: rows.length, fittable: false }; continue; }
|
||||||
|
|
||||||
|
const d = [];
|
||||||
|
for (const r of rows) {
|
||||||
|
const pc = cal.applyIsotonic(map, r.p);
|
||||||
|
if (knownNumber(pc) === null) continue;
|
||||||
|
d.push((pc - r.won) ** 2 - (r.p - r.won) ** 2);
|
||||||
|
}
|
||||||
|
const g = mean(d);
|
||||||
|
const sigma = Math.sqrt(d.reduce((s, x) => s + (x - g) ** 2, 0) / (d.length - 1));
|
||||||
|
const seFull = sigma / Math.sqrt(d.length);
|
||||||
|
|
||||||
|
// Date sizes are sample STRUCTURE, not outcomes — safe to read here.
|
||||||
|
const sizes = new Map();
|
||||||
|
for (const r of rows) sizes.set(r.date, (sizes.get(r.date) || 0) + 1);
|
||||||
|
dateSizes[stat] = [...sizes.values()].sort((a, b) => b - a);
|
||||||
|
|
||||||
|
perStat[stat] = {
|
||||||
|
n: d.length,
|
||||||
|
g_signed: round5(g),
|
||||||
|
sigma_row: round5(sigma),
|
||||||
|
se_full: round5(seFull),
|
||||||
|
effect_z_at_full_n: round3(Math.abs(g) / seFull),
|
||||||
|
improves: g < 0,
|
||||||
|
no_effect: Math.abs(g) / seFull < EFFECT_Z,
|
||||||
|
fittable: true,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Choose k jointly. Blind: uses only (g, sigma) and date SIZES. ──
|
||||||
|
const kTable = [];
|
||||||
|
for (const k of [1.0, 1.25, 1.5, 1.75, 2.0]) {
|
||||||
|
const pNoise = normCdf(-k);
|
||||||
|
const row = { k, per_drop_noise_prob: round4(pNoise), stats: {} };
|
||||||
|
for (const stat of STATS) {
|
||||||
|
const ps = perStat[stat];
|
||||||
|
if (!ps || !ps.fittable) continue;
|
||||||
|
const nStar = Math.ceil(k * k * (ps.sigma_row / Math.abs(ps.g_signed)) ** 2);
|
||||||
|
const D = (dateSizes[stat] || []).filter((n) => n >= nStar).length;
|
||||||
|
const { cutoff, fp } = D > 0 ? cutoffFor(D, pNoise, TARGET_FP) : { cutoff: null, fp: null };
|
||||||
|
// FN at a stated alternative: date-to-date SD of the effect equals |g|.
|
||||||
|
const pAlt = D > 0 ? normCdf(-Math.abs(ps.g_signed) / Math.sqrt(ps.g_signed ** 2 + (ps.sigma_row ** 2) / nStar)) : null;
|
||||||
|
const fn = D > 0 && cutoff !== null ? 1 - binomTail(D, cutoff, pAlt) : null;
|
||||||
|
row.stats[stat] = {
|
||||||
|
n_star: nStar, informative_drops: D, cutoff, fp: fp === null ? null : round4(fp),
|
||||||
|
fn_at_tau_equals_g: fn === null ? null : round4(fn),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
kTable.push(row);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(JSON.stringify({
|
||||||
|
phase: 'PHASE 1 — coherent LODO test derivation, BLIND',
|
||||||
|
defect_being_fixed: 'a 1-SE informative bar with a zero-reversal rule: P(>=1 reversal | stable, 4 drops) = 0.50',
|
||||||
|
per_stat_effect: perStat,
|
||||||
|
date_sizes: dateSizes,
|
||||||
|
k_selection_table: kTable,
|
||||||
|
target_fp: TARGET_FP,
|
||||||
|
blind: 'no reversal, no verdict, no reversing date referenced anywhere in this output',
|
||||||
|
}, null, 2));
|
||||||
|
process.exit(0);
|
||||||
|
})().catch((e) => { console.error(e); process.exit(1); });
|
||||||
|
|
||||||
|
const round5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
|
||||||
|
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||||
|
const round3 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 1000) / 1000);
|
||||||
+30
-14
@@ -45,7 +45,7 @@ const FAVOURITE_FLOOR = 0.9;
|
|||||||
* from here -- it is imported so the value that decides the verdicts cannot be
|
* from here -- it is imported so the value that decides the verdicts cannot be
|
||||||
* edited alongside them.
|
* edited alongside them.
|
||||||
*/
|
*/
|
||||||
const { LODO_MIN_HELD_ROWS: MIN_HELD_ROWS } = require('../src/services/model/calibrationRegistry');
|
const { LODO_TEST, LODO_POWER_FLOOR } = require('../src/services/model/calibrationRegistry');
|
||||||
|
|
||||||
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
|
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
|
||||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||||
@@ -131,7 +131,9 @@ async function main() {
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── LEAVE ONE DATE OUT ──
|
// ── LEAVE ONE DATE OUT, at THIS stat's own informative bar ──
|
||||||
|
const spec = LODO_TEST[stat];
|
||||||
|
const MIN_HELD_ROWS = spec ? spec.n_star : Infinity;
|
||||||
const table = [];
|
const table = [];
|
||||||
for (const d of dates) {
|
for (const d of dates) {
|
||||||
const fit = rows.filter((r) => r.date !== d);
|
const fit = rows.filter((r) => r.date !== d);
|
||||||
@@ -168,10 +170,19 @@ async function main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const informative = table.filter((t) => t.verdict !== 'UNINFORMATIVE');
|
const informative = table.filter((t) => t.verdict !== 'UNINFORMATIVE');
|
||||||
const anyReversal = informative.some((t) => t.verdict === 'REVERSES');
|
const reversals = informative.filter((t) => t.verdict === 'REVERSES');
|
||||||
const signTested = informative.filter((t) => t.favourite_sign_holds !== null);
|
|
||||||
const anySignFlip = signTested.some((t) => t.favourite_sign_holds === false);
|
// THE DECISION RULE IS BINOMIAL, not zero-tolerance. Under stability each
|
||||||
const passes = informative.length > 0 && !anyReversal && !anySignFlip;
|
// informative drop reverses with prob Phi(-k), so demanding zero reversals
|
||||||
|
// failed stable stats roughly half the time.
|
||||||
|
const cutoff = spec ? spec.cutoff : 0;
|
||||||
|
const exceedsCutoff = reversals.length > cutoff;
|
||||||
|
|
||||||
|
// AND THE TEST MUST BE ABLE TO FAIL. Below the power floor it cannot, so it
|
||||||
|
// cannot pass either -- "could not test" must never read as "passed".
|
||||||
|
const underpowered = !spec || spec.power < LODO_POWER_FLOOR;
|
||||||
|
const verdict = underpowered ? 'UNTESTABLE_BY_LODO' : (exceedsCutoff ? 'FAIL' : 'PASS');
|
||||||
|
const passes = verdict === 'PASS';
|
||||||
|
|
||||||
out.per_stat[stat] = {
|
out.per_stat[stat] = {
|
||||||
n: rows.length,
|
n: rows.length,
|
||||||
@@ -179,14 +190,19 @@ async function main() {
|
|||||||
lodo_table: table,
|
lodo_table: table,
|
||||||
informative_drops: informative.length,
|
informative_drops: informative.length,
|
||||||
brier_reversals: informative.filter((t) => t.verdict === 'REVERSES').length,
|
brier_reversals: informative.filter((t) => t.verdict === 'REVERSES').length,
|
||||||
favourite_sign_flips: signTested.filter((t) => t.favourite_sign_holds === false).length,
|
favourite_sign_flips: informative.filter((t) => t.favourite_sign_holds === false).length,
|
||||||
favourite_sign_untested: informative.length - signTested.length,
|
favourite_sign_untested: informative.filter((t) => t.favourite_sign_holds === null).length,
|
||||||
lodo: passes ? 'PASS' : 'FAIL',
|
n_star: spec ? spec.n_star : null,
|
||||||
reason: passes
|
cutoff,
|
||||||
? 'improvement never reverses and the favourite over-prediction never flips sign across any single-date drop'
|
reversal_count: reversals.length,
|
||||||
: (anyReversal
|
reversing_dates: reversals.map((t) => ({ date: t.dropped, held_n: t.held_n, delta: t.brier_delta })),
|
||||||
? `improvement reverses when ${informative.filter((t) => t.verdict === 'REVERSES').map((t) => t.dropped).join(', ')} is dropped — the effect is date-driven`
|
test_power: spec ? spec.power : null,
|
||||||
: `the favourite over-prediction flips sign when ${signTested.filter((t) => t.favourite_sign_holds === false).map((t) => t.dropped).join(', ')} is dropped`),
|
lodo: verdict,
|
||||||
|
reason: underpowered
|
||||||
|
? `power ${spec ? spec.power : 0} < floor ${LODO_POWER_FLOOR} — this test would miss a real date-driven failure more than nine times in ten, so it can neither pass nor fail the stat`
|
||||||
|
: (exceedsCutoff
|
||||||
|
? `${reversals.length} reversals among ${informative.length} informative drops exceeds the cutoff of ${cutoff}`
|
||||||
|
: `${reversals.length} reversals among ${informative.length} informative drops is within the cutoff of ${cutoff}`),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,170 @@
|
|||||||
|
# Auditing the LODO instrument — it cannot evaluate any stat
|
||||||
|
|
||||||
|
**All four stats are UNTESTABLE-BY-LODO. Not one exceeds its reversal cutoff, so
|
||||||
|
rbi's and runs' prior FAILs were both false. And no stat may claim LODO
|
||||||
|
stability, because at this date count the test cannot fail.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 0 — the defect, on record
|
||||||
|
|
||||||
|
The gate at `1f40014` was mine, and it was an incoherent pair:
|
||||||
|
|
||||||
|
**1. A 1-SE informativeness bar coupled to a ZERO-reversal rule.** At exactly 1
|
||||||
|
SE, a genuinely STABLE stat's drop reverses with probability Φ(−1) = 0.1587. On
|
||||||
|
four informative drops:
|
||||||
|
|
||||||
|
```
|
||||||
|
P(>= 1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50
|
||||||
|
```
|
||||||
|
|
||||||
|
**The rule failed stable stats half the time by construction.** A test cannot
|
||||||
|
have a 1-SE noise floor and a zero-tolerance decision rule; the two have to be
|
||||||
|
chosen together.
|
||||||
|
|
||||||
|
**2. A pooled n\*.** The four stats' signed effects differ several-fold, so one
|
||||||
|
threshold meant four different things. Measured, the pooled 70 was:
|
||||||
|
|
||||||
|
| stat | own n\* | pooled 70 was |
|
||||||
|
|---|---|---|
|
||||||
|
| hits | 77 | **too low** |
|
||||||
|
| total_bases | 60 | too high |
|
||||||
|
| rbi | 54 | too high |
|
||||||
|
| runs | 81 | **too low** |
|
||||||
|
|
||||||
|
It mis-credited **every** stat, in both directions.
|
||||||
|
|
||||||
|
Neither defect touched the counter or `p_win`. Both touched only which
|
||||||
|
calibrations were judged stable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 1 — the coherent test, derived blind
|
||||||
|
|
||||||
|
### (a)(b)(c) per-stat effect
|
||||||
|
|
||||||
|
| stat | n | g (signed) | σ_row | SE_full | effect z | NO-EFFECT? |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| hits | 1,140 | −0.01288 | 0.11251 | 0.00333 | 3.87 | no |
|
||||||
|
| total_bases | 1,050 | −0.01380 | 0.10680 | 0.00330 | 4.19 | no |
|
||||||
|
| rbi | 630 | −0.00884 | 0.06459 | 0.00257 | 3.44 | no |
|
||||||
|
| runs | 597 | −0.00902 | 0.08080 | 0.00331 | 2.73 | no |
|
||||||
|
|
||||||
|
All four have a real effect at full n. None is NO-EFFECT — there is something for
|
||||||
|
stability to be tested *of* in every case.
|
||||||
|
|
||||||
|
### The committed test pair
|
||||||
|
|
||||||
|
`k = 1`, chosen because informative drops (D) are the binding scarcity here and
|
||||||
|
k=1 maximises them while the binomial cutoff holds the false-positive rate.
|
||||||
|
|
||||||
|
| stat | n\*=k²(σ/\|g\|)² | informative drops D | cutoff | FP | **power at τ=\|g\|** |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| hits | 77 | 5 | 2 | 0.031 | **0.093** |
|
||||||
|
| total_bases | 60 | 5 | 2 | 0.031 | **0.093** |
|
||||||
|
| rbi | 54 | 4 | 2 | 0.014 | **0.045** |
|
||||||
|
| runs | 81 | 3 | 2 | 0.004 | **0.014** |
|
||||||
|
|
||||||
|
Per-drop noise probability under stability Φ(−1) = 0.1587. FAIL iff reversals > cutoff.
|
||||||
|
|
||||||
|
### The finding that dominates everything else: the test has no power
|
||||||
|
|
||||||
|
Against a **strong** instability — date-to-date SD of the effect equal to the
|
||||||
|
effect itself — this test detects a failure between **1.4% and 9.3%** of the
|
||||||
|
time. Across every k examined (1.0 → 2.0), the best any stat reaches is 0.337,
|
||||||
|
and reaching even that costs all but two informative drops.
|
||||||
|
|
||||||
|
**A gate that cannot fail cannot pass.** `LODO_POWER_FLOOR = 0.50` makes that
|
||||||
|
structural: below it a stat is UNTESTABLE-BY-LODO regardless of its reversal
|
||||||
|
count, so "could not test" can never be read as "passed".
|
||||||
|
|
||||||
|
Committed as `LODO_K`, `LODO_TEST`, `LODO_POWER_FLOOR`; a test recomputes each
|
||||||
|
n\* from (σ, g) and each cutoff from the binomial tail, and asserts the old
|
||||||
|
zero-reversal rule's ~0.50 false-fail rate. The stale pooled constant is nulled
|
||||||
|
so nothing can read it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 2/3 — cold re-read at each stat's own n\*
|
||||||
|
|
||||||
|
| stat | own n\* | informative | reversals | cutoff | power | verdict |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| hits | 77 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** |
|
||||||
|
| total_bases | 60 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** |
|
||||||
|
| rbi | 54 | 4 | 1 | 2 | 0.045 | **UNTESTABLE** |
|
||||||
|
| runs | 81 | 3 | 2 | 2 | 0.014 | **UNTESTABLE** |
|
||||||
|
|
||||||
|
**Setting the power floor aside entirely, not one stat exceeds its cutoff.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 4 — reconcile against 1f40014
|
||||||
|
|
||||||
|
| stat | 1f40014 | now | why it changed |
|
||||||
|
|---|---|---|---|
|
||||||
|
| hits | PASS | UNTESTABLE | the PASS was from a test that cannot fail; 0 reversals is uninformative at 9% power |
|
||||||
|
| total_bases | PASS | UNTESTABLE | same — and it "passed" at a pooled n\*=70 above its own 60, so its drop count was under-credited too |
|
||||||
|
| **rbi** | **FAIL** | UNTESTABLE (1 reversal, cutoff 2) | **FALSE FAIL.** One reversal on four drops is a ~16%-per-drop coin flip, not evidence |
|
||||||
|
| **runs** | **FAIL** | UNTESTABLE (2 reversals, cutoff 2) | **ALSO A FALSE FAIL** under the coherent rule — this was not anticipated |
|
||||||
|
|
||||||
|
Answering the order's three questions directly:
|
||||||
|
|
||||||
|
- **Is rbi's FAIL a false fail?** Yes. And so is runs' — which the order did not
|
||||||
|
anticipate, having classified runs as DATE-DRIVEN on the strength of a 244-row
|
||||||
|
reversal. Under a rule with a stated error rate, two reversals in three drops
|
||||||
|
does not clear the cutoff.
|
||||||
|
- **Was TB's PASS real?** No. It was vacuous: the test could not have failed it.
|
||||||
|
- **Does hits still pass at its own smaller n\*?** Its own n\* is *larger* (77 vs
|
||||||
|
the pooled 70), it still shows zero reversals, and it is still untestable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 5 — deploy, withdraw, route
|
||||||
|
|
||||||
|
| stat | basis | decision |
|
||||||
|
|---|---|---|
|
||||||
|
| hits | date-clustered CI [−0.0139, −0.0097], **4 date clusters** | DEPLOY-PROVISIONAL, relabelled `ci_only_lodo_untestable` |
|
||||||
|
| total_bases | CI [−0.0061, −0.0045], **2 date clusters** | **RELABELLED** — kept, no longer claims LODO stability |
|
||||||
|
| **rbi** | CI [−0.0092, −0.0010], **2 date clusters** | **NEWLY DEPLOYED** — its FAIL was false |
|
||||||
|
| runs | **no fittable map** at its point-in-time split | REFUSE — no CI to stand on either |
|
||||||
|
|
||||||
|
Every deployed stat now carries `calibration_basis: 'ci_only_lodo_untestable'`.
|
||||||
|
**Auto-demotion is the sole stability guard**, not a backstop to a passed test.
|
||||||
|
|
||||||
|
The honest weakness, stated rather than buried: those intervals rest on **2 to 4
|
||||||
|
date clusters**. That is thin support, and it is now the *only* support.
|
||||||
|
|
||||||
|
**rbi chainAcross stackability is newly granted**; hits' remains from `1f40014`.
|
||||||
|
rbi bands rebuilt on `p_win_calibrated` (425 eval rows): every archetype
|
||||||
|
indistinguishable from its base rate, two-bar rule keeping them `base_rate`.
|
||||||
|
|
||||||
|
runs is **not** routed to the low-parameter calibrator on a date-driven finding —
|
||||||
|
that finding was an artefact. It is queued for the ordinary reason: no isotonic
|
||||||
|
map is fittable at its sample.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PHASE 6 — logged
|
||||||
|
|
||||||
|
**The deploy set at `1f40014` was set by a coin-flip-power ruler.** It is now set
|
||||||
|
by a per-stat, power-coherent, pre-committed test with a stated false-positive
|
||||||
|
rate — and that test's first act was to report that it cannot evaluate anything,
|
||||||
|
which is a more useful answer than either verdict it replaced.
|
||||||
|
|
||||||
|
**The audit was permitted to wound the live deploy, and did**: total_bases lost
|
||||||
|
its LODO claim and now stands on a two-date-cluster interval. That it could is
|
||||||
|
the integrity property.
|
||||||
|
|
||||||
|
**Standing question unchanged.** Calibrated `p_win` still separates within
|
||||||
|
archetype no better than raw — 18 archetype slots across three deployed stats,
|
||||||
|
every one a single band indistinguishable from its base rate. Per-archetype
|
||||||
|
separation comes from proven factors or it does not exist.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
Blind ordering held: (g, σ_row) and the committed (n\*, cutoff, power floor) were
|
||||||
|
derived and locked with no reversal or verdict in view, before any stat was
|
||||||
|
re-read. `p_win` never mutated. No Bonferroni slot consumed. Counter and frozen
|
||||||
|
clusters verified file-by-file.
|
||||||
@@ -29,41 +29,57 @@ const { knownNumber } = require('../../utils/known');
|
|||||||
const STATUS = Object.freeze({ NONE: 'none', PROVISIONAL: 'provisional', PROMOTED: 'promoted' });
|
const STATUS = Object.freeze({ NONE: 'none', PROVISIONAL: 'provisional', PROMOTED: 'promoted' });
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* LODO_MIN_HELD_ROWS — POWER-DERIVED, PRE-COMMITTED, NOT OPERATOR-CHOSEN.
|
* THE LODO TEST — a COHERENT pair, replacing the incoherent one at 1f40014.
|
||||||
*
|
*
|
||||||
* A leave-one-date-out reversal is only informative if that date's held-out
|
* ── WHAT WAS WRONG ───────────────────────────────────────────────────────
|
||||||
* Brier delta is distinguishable from zero at its row count. Below that, a
|
* The previous gate paired a 1-SE per-drop informativeness bar with a
|
||||||
* "reversal" is a coin flip wearing a decimal point — which is what made the
|
* ZERO-reversal decision rule. At exactly 1 SE a genuinely STABLE stat's drop
|
||||||
* previous verdict depend on a number someone picked.
|
* reverses with probability Phi(-1) = 0.159, so on four informative drops the
|
||||||
|
* chance of at least one reversal is 1 - 0.841^4 = 0.50. The rule failed stable
|
||||||
|
* stats half the time BY CONSTRUCTION. And n* was pooled across four stats whose
|
||||||
|
* signed effects differ several-fold, so one number meant four different things:
|
||||||
|
* measured, the pooled 70 was too LOW for hits (77) and runs (81) and too HIGH
|
||||||
|
* for total_bases (60) and rbi (54).
|
||||||
*
|
*
|
||||||
* The per-row Brier difference is d_i = (pc_i - y_i)^2 - (p_i - y_i)^2, so a
|
* ── THE COHERENT PAIR ────────────────────────────────────────────────────
|
||||||
* date's delta is mean(d) and SE(n) = SD(d)/sqrt(n). The smallest n at which a
|
* The bar and the rule are chosen TOGETHER, per stat, for a stated error rate:
|
||||||
* typical effect clears one standard error is n* = (SD(d)/|effect|)^2.
|
|
||||||
*
|
*
|
||||||
* Measured 2026-08-07, pooled across all four stats so that no single stat's
|
* informative bar n*_k = k^2 * (sigma_row / |g|)^2 per stat
|
||||||
* verdict could shape the threshold that decides it:
|
* decision rule FAIL iff reversals > cutoff, where under stability
|
||||||
|
* R ~ Binomial(D, Phi(-k)) and cutoff is the smallest c
|
||||||
|
* with P(R > c) <= 0.05
|
||||||
*
|
*
|
||||||
* pooled rows 3,417
|
* `g` is the mean SIGNED per-row improvement — the quantity whose sign a
|
||||||
* SD(d) 0.09816
|
* reversal flips. k = 1 is chosen because it maximises informative drops (D),
|
||||||
* |effect| 0.01175
|
* which is the binding scarcity here, while the binomial cutoff holds the
|
||||||
* n* = (0.09816/0.01175)^2 = 69.8 -> 70
|
* false-positive rate at 0.004-0.031 across the four stats.
|
||||||
*
|
*
|
||||||
* SE-vs-n: n=20 gives effect/SE 0.54, n=50 gives 0.85, n=75 gives 1.04. So
|
* ── AND THE TEST STILL HAS ALMOST NO POWER ───────────────────────────────
|
||||||
* anything under ~70 held rows cannot tell a real reversal from noise.
|
* At the stated alternative (date-to-date SD of the effect equal to |g| — a
|
||||||
|
* strong instability), power is 0.093 / 0.093 / 0.045 / 0.014. The test would
|
||||||
|
* MISS a real date-driven failure more than nine times in ten. Across every k
|
||||||
|
* examined, the best any stat reaches is 0.337.
|
||||||
*
|
*
|
||||||
* DERIVED BLIND — the derivation script prints no stat verdict, no date and no
|
* So a PASS here means "no instability was detected", NOT "it is stable", and a
|
||||||
* reversal. It ran, and this constant was committed, before any stat was
|
* gate that cannot fail is not a gate. LODO_POWER_FLOOR makes that structural: a
|
||||||
* re-read. That ordering is the integrity property; a test locks the value so it
|
* stat whose test power falls below it is UNTESTABLE-BY-LODO and may not claim
|
||||||
* cannot be silently tuned afterwards.
|
* LODO stability at all, whatever its reversal count.
|
||||||
|
*
|
||||||
|
* Derived BLIND — the derivation script prints no reversal, no verdict and no
|
||||||
|
* reversing date. It ran, and these were committed, before any stat was re-read.
|
||||||
*/
|
*/
|
||||||
const LODO_MIN_HELD_ROWS = 70;
|
const LODO_K = 1.0;
|
||||||
const LODO_THRESHOLD_BASIS = Object.freeze({
|
/** Below this power the test cannot fail, so it cannot pass either. */
|
||||||
pooled_rows: 3417,
|
const LODO_POWER_FLOOR = 0.50;
|
||||||
per_row_brier_diff_sd: 0.09816,
|
/** Per-stat, from (g, sigma_row) measured blind. */
|
||||||
pooled_effect_abs_mean: 0.01175,
|
const LODO_TEST = Object.freeze({
|
||||||
rule: 'n* = (SD(d) / |effect|)^2',
|
hits: { g: -0.01288, sigma_row: 0.11251, n_star: 77, informative_drops: 5, cutoff: 2, fp: 0.0310, power: 0.093 },
|
||||||
derived_blind: true,
|
total_bases: { g: -0.01380, sigma_row: 0.10680, n_star: 60, informative_drops: 5, cutoff: 2, fp: 0.0310, power: 0.093 },
|
||||||
|
rbi: { g: -0.00884, sigma_row: 0.06459, n_star: 54, informative_drops: 4, cutoff: 2, fp: 0.0141, power: 0.045 },
|
||||||
|
runs: { g: -0.00902, sigma_row: 0.08080, n_star: 81, informative_drops: 3, cutoff: 2, fp: 0.0040, power: 0.014 },
|
||||||
});
|
});
|
||||||
|
/** Legacy name kept so nothing silently reads a stale pooled value. */
|
||||||
|
const LODO_MIN_HELD_ROWS = null;
|
||||||
/** The ORIGINAL floor, correctly scoped: promotion, not deploy. */
|
/** The ORIGINAL floor, correctly scoped: promotion, not deploy. */
|
||||||
const PROMOTION_DATE_CLUSTERS = 40;
|
const PROMOTION_DATE_CLUSTERS = 40;
|
||||||
|
|
||||||
@@ -144,4 +160,4 @@ function createRegistry(initial = {}) {
|
|||||||
return { deploy, reverify, serves, get, all, log: () => log.slice() };
|
return { deploy, reverify, serves, get, all, log: () => log.slice() };
|
||||||
}
|
}
|
||||||
|
|
||||||
module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS, LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS };
|
module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS, LODO_K, LODO_TEST, LODO_POWER_FLOOR, LODO_MIN_HELD_ROWS };
|
||||||
|
|||||||
@@ -276,27 +276,37 @@ async function loadPitcherArsenals(sport) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Stats whose calibration passed leave-one-date-out and may serve a calibrated
|
* Stats that may serve a calibrated number, and ON WHAT BASIS.
|
||||||
* number. PROVISIONAL: auto-demoted the first time the held-out interval stops
|
|
||||||
* excluding zero or the favourite over-prediction flips sign.
|
|
||||||
*
|
*
|
||||||
* Gated at LODO_MIN_HELD_ROWS = 70, which is POWER-DERIVED and pre-committed:
|
* ── LODO CANNOT EVALUATE ANY OF THEM ─────────────────────────────────────
|
||||||
* below ~70 held rows a date's Brier delta cannot be told from a coin flip, so a
|
* The LODO gate was audited and rebuilt as a coherent pair (per-stat
|
||||||
* "reversal" there carries no information.
|
* informativeness bar + binomial reversal cutoff at FP <= 0.05). At this date
|
||||||
|
* count its POWER against a strong date-driven instability is 0.093 / 0.093 /
|
||||||
|
* 0.045 / 0.014 — it would miss a real failure more than nine times in ten. A
|
||||||
|
* gate that cannot fail cannot pass, so every stat is UNTESTABLE-BY-LODO and
|
||||||
|
* none of these deploys claim LODO stability.
|
||||||
*
|
*
|
||||||
* hits was WITHDRAWN at 6ae11f1 and is RESTORED here. That is not a reversal of
|
* The previous zero-reversal rule was incoherent: at a 1-SE bar a stable stat
|
||||||
* the earlier call — it was correct on the instrument available then, which
|
* reverses on ~16% of drops, so demanding zero failed stable stats ~50% of the
|
||||||
* admitted 20- and 25-row dates as evidence. With the threshold derived from
|
* time. Re-read under the binomial cutoff, NEITHER rbi (1 reversal) NOR runs
|
||||||
* power rather than chosen, hits reverses on nothing. The restoration came
|
* (2) exceeds its cutoff of 2 — both prior FAILs were false.
|
||||||
* through the gate, not around it.
|
|
||||||
*
|
*
|
||||||
* rbi and runs remain ABSENT, and their failures are NOT underpowered: each
|
* ── SO THE DEPLOY BASIS IS THE DATE-CLUSTERED CI ALONE ───────────────────
|
||||||
* reverses on a date comfortably above the threshold (rbi 2026-08-01 n=99; runs
|
* hits, total_bases and rbi each have a point-in-time held-out interval
|
||||||
* 2026-08-01 n=86 and 2026-08-05 n=244). Those are DATE-DRIVEN failures — no
|
* excluding zero. That is the ONLY support they have, and it is thin — the
|
||||||
* threshold and no further accrual rescues them, and isotonic is fitting
|
* interval rests on 4, 2 and 2 date clusters respectively. Auto-demotion is
|
||||||
* day-structure. Routed to the low-parameter calibrator queue.
|
* therefore the sole stability guard, not a backstop to a passed test.
|
||||||
|
*
|
||||||
|
* runs is absent: no isotonic map was fittable at its point-in-time split, so it
|
||||||
|
* has no CI support to stand on either.
|
||||||
*/
|
*/
|
||||||
const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases']);
|
const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases', 'rbi']);
|
||||||
|
/** Why each deployed stat is allowed to serve. Not one of them is LODO-stable. */
|
||||||
|
const CALIBRATION_BASIS = Object.freeze({
|
||||||
|
hits: 'ci_only_lodo_untestable',
|
||||||
|
total_bases: 'ci_only_lodo_untestable',
|
||||||
|
rbi: 'ci_only_lodo_untestable',
|
||||||
|
});
|
||||||
|
|
||||||
async function runSnapshot(sport, opts = {}) {
|
async function runSnapshot(sport, opts = {}) {
|
||||||
const sp = String(sport || '').toLowerCase();
|
const sp = String(sport || '').toLowerCase();
|
||||||
@@ -759,6 +769,7 @@ async function runSnapshot(sport, opts = {}) {
|
|||||||
g.calibrated = out.calibrated;
|
g.calibrated = out.calibrated;
|
||||||
g.calibration_reason = out.reason;
|
g.calibration_reason = out.reason;
|
||||||
g.calibration_status = 'provisional';
|
g.calibration_status = 'provisional';
|
||||||
|
g.calibration_basis = CALIBRATION_BASIS[stat] || null;
|
||||||
if (out.calibrated) marked += 1;
|
if (out.calibrated) marked += 1;
|
||||||
}
|
}
|
||||||
console.log(`[calibration] ${sp} ${stat} (PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}`);
|
console.log(`[calibration] ${sp} ${stat} (PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}`);
|
||||||
@@ -881,5 +892,6 @@ module.exports = {
|
|||||||
pushTickerItems,
|
pushTickerItems,
|
||||||
ACTIVE_SPORTS,
|
ACTIVE_SPORTS,
|
||||||
CALIBRATION_DEPLOYED,
|
CALIBRATION_DEPLOYED,
|
||||||
|
CALIBRATION_BASIS,
|
||||||
__internals: { propKey, gradedAtFor, indexOdds, lastName, isTopGrade, DELTA_NOISE, DELTA_MOVE, TICKER_CAP },
|
__internals: { propKey, gradedAtFor, indexOdds, lastName, isTopGrade, DELTA_NOISE, DELTA_MOVE, TICKER_CAP },
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -1,43 +1,45 @@
|
|||||||
'use strict';
|
'use strict';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Which stats serve a calibrated number in the live pipeline.
|
* Which stats serve a calibrated number, and on what basis.
|
||||||
*
|
*
|
||||||
* The rule this locks: a stat that cannot survive dropping a single settled date
|
* The thing these lock is that "could not test" never reads as "passed". LODO
|
||||||
* was never calibrated — it was fitted to that date. hits WAS served calibrated
|
* has 1.4%-9.3% power at this date count, so no deployed stat claims stability
|
||||||
* and is not any more, which is the honest consequence of measuring it.
|
* from it — each rides a thin date-clustered interval and auto-demotion alone.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
const snapshotService = require('../../src/services/snapshotService');
|
const snapshotService = require('../../src/services/snapshotService');
|
||||||
|
const { LODO_TEST, LODO_POWER_FLOOR } = require('../../src/services/model/calibrationRegistry');
|
||||||
|
|
||||||
describe('the deployed set is LODO-gated', () => {
|
describe('the deploy set rests on the CI, not on a passed LODO', () => {
|
||||||
it('serves the stats that passed LODO at the powered threshold', () => {
|
it('serves the three stats with a point-in-time interval excluding zero', () => {
|
||||||
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('total_bases');
|
expect(snapshotService.CALIBRATION_DEPLOYED).toEqual(['hits', 'total_bases', 'rbi']);
|
||||||
// hits was withdrawn at 6ae11f1 under a hand-chosen threshold that admitted
|
|
||||||
// 20-row dates as evidence, and is restored here because the powered
|
|
||||||
// instrument finds no reversal. Through the gate, not around it.
|
|
||||||
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('hits');
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('does NOT serve rbi or runs — both fail on dates ABOVE the threshold', () => {
|
it('does NOT serve runs — no fittable map, so no CI to stand on', () => {
|
||||||
// These are DATE-DRIVEN failures, not underpowered ones: rbi reverses on a
|
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain('runs');
|
||||||
// 99-row date and runs on 86- and 244-row dates. No threshold rescues them.
|
});
|
||||||
for (const stat of ['rbi', 'runs']) {
|
|
||||||
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain(stat);
|
it('labels every deployed stat as LODO-untestable rather than LODO-stable', () => {
|
||||||
|
for (const stat of snapshotService.CALIBRATION_DEPLOYED) {
|
||||||
|
expect(snapshotService.CALIBRATION_BASIS[stat]).toBe('ci_only_lodo_untestable');
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
it('every deployed stat cleared the power-derived threshold, not a chosen one', () => {
|
it('no stat may claim LODO stability, because the test cannot fail', () => {
|
||||||
const { LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS } = require('../../src/services/model/calibrationRegistry');
|
for (const t of Object.values(LODO_TEST)) expect(t.power).toBeLessThan(LODO_POWER_FLOOR);
|
||||||
expect(LODO_MIN_HELD_ROWS).toBe(70);
|
|
||||||
expect(LODO_THRESHOLD_BASIS.derived_blind).toBe(true);
|
|
||||||
// The reversing dates that keep rbi/runs out are all at or above it, so
|
|
||||||
// their exclusion cannot be an artefact of the threshold.
|
|
||||||
for (const n of [99, 86, 244]) expect(n).toBeGreaterThanOrEqual(LODO_MIN_HELD_ROWS);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('is frozen, so a stat cannot be added at runtime without a code change', () => {
|
it('rbi and runs were FALSE FAILS under the old zero-reversal rule', () => {
|
||||||
|
// Re-read under the binomial cutoff: 1 and 2 reversals, cutoff 2 for both.
|
||||||
|
expect(LODO_TEST.rbi.cutoff).toBe(2);
|
||||||
|
expect(LODO_TEST.runs.cutoff).toBe(2);
|
||||||
|
// rbi returns to the deploy set on CI support; runs still has none.
|
||||||
|
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('rbi');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is frozen, so a stat cannot be added at runtime', () => {
|
||||||
expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true);
|
expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true);
|
||||||
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('hits'); }).toThrow();
|
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('runs'); }).toThrow();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -119,34 +119,58 @@ describe('serving is band-limited', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('the LODO held-row threshold is power-derived, not operator-chosen', () => {
|
describe('the LODO test is a COHERENT pair, not a bar plus an unrelated rule', () => {
|
||||||
const { LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS } = require('../../src/services/model/calibrationRegistry');
|
const { LODO_K, LODO_TEST, LODO_POWER_FLOOR } = require('../../src/services/model/calibrationRegistry');
|
||||||
|
|
||||||
it('is the value its own stated derivation produces', () => {
|
const normCdf = (z) => {
|
||||||
// n* = (SD(d) / |effect|)^2 -- recomputed here so the constant cannot drift
|
const t = 1 / (1 + 0.2316419 * Math.abs(z));
|
||||||
// away from the basis that justifies it.
|
const d = 0.3989422804014327 * Math.exp(-z * z / 2);
|
||||||
const { per_row_brier_diff_sd: sd, pooled_effect_abs_mean: eff } = LODO_THRESHOLD_BASIS;
|
const p = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + t * (-1.821255978 + t * 1.330274429))));
|
||||||
expect(Math.ceil((sd / eff) ** 2)).toBe(LODO_MIN_HELD_ROWS);
|
return z >= 0 ? 1 - p : p;
|
||||||
|
};
|
||||||
|
const binomPmf = (n, k, p) => {
|
||||||
|
let logC = 0;
|
||||||
|
for (let i = 0; i < k; i += 1) logC += Math.log(n - i) - Math.log(i + 1);
|
||||||
|
return Math.exp(logC + k * Math.log(p) + (n - k) * Math.log(1 - p));
|
||||||
|
};
|
||||||
|
const tail = (n, c, p) => { let s = 0; for (let k = c + 1; k <= n; k += 1) s += binomPmf(n, k, p); return s; };
|
||||||
|
|
||||||
|
it('each n* is what its own (sigma_row, g) produce — no pooled value', () => {
|
||||||
|
for (const [stat, t] of Object.entries(LODO_TEST)) {
|
||||||
|
expect(Math.ceil(LODO_K ** 2 * (t.sigma_row / Math.abs(t.g)) ** 2)).toBe(t.n_star);
|
||||||
|
}
|
||||||
|
// And the four differ, which is exactly why one pooled number mis-credited them.
|
||||||
|
const stars = Object.values(LODO_TEST).map((t) => t.n_star);
|
||||||
|
expect(new Set(stars).size).toBeGreaterThan(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('carries its derivation basis, and was derived blind', () => {
|
it('each cutoff is the smallest one holding the false-positive rate at 0.05', () => {
|
||||||
expect(LODO_THRESHOLD_BASIS.rule).toBe('n* = (SD(d) / |effect|)^2');
|
const p = normCdf(-LODO_K);
|
||||||
expect(LODO_THRESHOLD_BASIS.derived_blind).toBe(true);
|
for (const [stat, t] of Object.entries(LODO_TEST)) {
|
||||||
expect(LODO_THRESHOLD_BASIS.pooled_rows).toBeGreaterThan(1000);
|
expect(tail(t.informative_drops, t.cutoff, p)).toBeLessThanOrEqual(0.05);
|
||||||
});
|
if (t.cutoff > 0) expect(tail(t.informative_drops, t.cutoff - 1, p)).toBeGreaterThan(0.05);
|
||||||
|
expect(t.fp).toBeCloseTo(tail(t.informative_drops, t.cutoff, p), 3);
|
||||||
it('rejects the thresholds that were previously chosen by hand', () => {
|
|
||||||
// 20 was the operator-chosen value whose verdict moved with it; anything
|
|
||||||
// below n* cannot distinguish a reversal from a coin flip.
|
|
||||||
for (const weak of [10, 20, 25, 30, 50]) {
|
|
||||||
const se = LODO_THRESHOLD_BASIS.per_row_brier_diff_sd / Math.sqrt(weak);
|
|
||||||
expect(LODO_THRESHOLD_BASIS.pooled_effect_abs_mean).toBeLessThan(se);
|
|
||||||
expect(weak).toBeLessThan(LODO_MIN_HELD_ROWS);
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
it('is informative at the committed value', () => {
|
it('the OLD rule is demonstrably incoherent — it failed stable stats ~half the time', () => {
|
||||||
const se = LODO_THRESHOLD_BASIS.per_row_brier_diff_sd / Math.sqrt(LODO_MIN_HELD_ROWS);
|
// Zero-reversal rule at a 1-SE bar, on four informative drops.
|
||||||
expect(LODO_THRESHOLD_BASIS.pooled_effect_abs_mean).toBeGreaterThanOrEqual(se * 0.99);
|
const pNoise = normCdf(-1);
|
||||||
|
const falseFail = 1 - (1 - pNoise) ** 4;
|
||||||
|
expect(falseFail).toBeGreaterThan(0.45);
|
||||||
|
expect(falseFail).toBeLessThan(0.55);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('every stat falls below the power floor, so none may claim LODO stability', () => {
|
||||||
|
// A gate that cannot fail is not a gate. This is the honest state at this
|
||||||
|
// date count, and the floor makes it structural rather than a footnote.
|
||||||
|
for (const [stat, t] of Object.entries(LODO_TEST)) {
|
||||||
|
expect(t.power).toBeLessThan(LODO_POWER_FLOOR);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('the stale pooled threshold is nulled so nothing can read it', () => {
|
||||||
|
const { LODO_MIN_HELD_ROWS } = require('../../src/services/model/calibrationRegistry');
|
||||||
|
expect(LODO_MIN_HELD_ROWS).toBeNull();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user