LODO-gated provisional calibration: total_bases deploys, hits withdrawn

PHASE 0 — I applied factorGate's >=40 date-cluster floor to a calibration
layer without challenging the binding. That floor is a cluster-robust
interval bar for a CAUSAL claim. Calibration makes no causal claim, has a
bounded failure mode (it can only over- or under-shrink) and consumes no
Bonferroni slot. Its real risk is that the correction is DATE-DRIVEN, and
leave-one-date-out tests that directly -- a STRICTER bar, since a cluster
count cannot detect a single day carrying the effect. The >=40 floor is
retained, correctly scoped as the PROMOTION bar.

PHASE 1 — both guards codified, 11 tests, green before Phase 2.
Demonstrated on live data: raw population violated=true, mean_p 0.4962,
both_sides_share 0.9763; after dedup violated=false, mean_p 0.6694. The
null guard's test demonstrates the trap explicitly, since (null-1)**2 is
1 and (null-0)**2 is 0 so a Brier over nulls equals the win rate.

PHASE 2 — LODO:

  hits         n=1140 dates=17  2 reversals (07-22 n=20, 07-26 n=25)  FAIL
  total_bases  n=1050 dates=7   0 reversals, 0 sign flips             PASS
  rbi          n= 630 dates=5   1 reversal  (08-01 n=99)              FAIL
  runs         n= 597 dates=5   2 reversals (08-01 n=86, 08-05 n=244) FAIL

Threshold sensitivity reported because the verdict moves: total_bases
passes at every held-size threshold, runs fails at every one, and hits
fails ONLY when 20/25-row dates are admitted. I fixed MIN_HELD_ROWS=20
before seeing which stats passed and did not move it afterwards to
preserve a deploy. Honest caveat: a per-date Brier delta on 20 rows has a
standard error several times the effect, so the instrument is
underpowered per-drop -- an argument for pre-registering a higher
threshold, which is a Roundtable call, not one to make while holding the
results.

PHASE 3 — total_bases DEPLOY-PROVISIONAL, band [0.6-0.8]. hits, rbi and
runs REFUSE.

HITS WAS BEING SERVED CALIBRATED AND IS NOT ANY MORE. snapshotService
hardcoded it since S91; it fails LODO, so it is out. A stat that cannot
survive dropping one day was never calibrated, it was fitted to that day.
The consequence is real -- hits props become unstackable for
chain.chainAcross -- and it errs toward withdrawing a claim rather than
preserving one on a fragile verdict. Deployment is now driven by a frozen,
tested CALIBRATION_DEPLOYED set, not a hardcoded stat name.

PHASE 4 — calibrationRegistry, 14 tests. Deploy needs BOTH gates, neither
waivable. reverify auto-demotes on the first breach (CI stops excluding
zero, or the favourite bias flips sign) and logs the breaking date.
Promotion needs the original >=40 bar. A provisional deploy that cannot be
taken away is just a deploy.

PHASE 5 — TB bands rebuilt on calibrated values, 625 eval rows. The
two-bar rule still bites: calibrated YES, proven NO, so they stay a
base-rate read, now honestly numbered. Every archetype still collapses to
one band -- calibrated p_win separates within archetype no better than raw.

PHASE 6 logged only: the dead gradient is buried (hits~TB > runs > RBI,
and RBI has the SMALLEST bias, so the skill-driven-gradient mechanism did
not survive); the refused set is a map of missing inputs; a low-parameter
calibrator is queued unbuilt.

p_win never mutated; calibration rides as p_win_calibrated with
calibration_status provisional. No Bonferroni slot consumed. Counter and
frozen clusters byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-06 18:31:19 -04:00
parent f976df47b8
commit 6ae11f1193
10 changed files with 1064 additions and 23 deletions
+193
View File
@@ -0,0 +1,193 @@
#!/usr/bin/env node
'use strict';
/**
* lodo-calibration — Phases 2 and 3.
*
* The ≥40 date-cluster floor was factorGate's interval bar for a CAUSAL claim,
* mis-applied to a monotone shrink-to-observed layer. Calibration makes no causal
* claim, consumes no Bonferroni slot, and has a bounded failure mode (it can only
* over- or under-shrink). Its real risk is that the correction is DATE-DRIVEN —
* that one unusual day's offensive environment is doing all the work.
*
* Leave-one-date-out tests exactly that, and it is a harder bar than a cluster
* count: a single date whose removal reverses the improvement, or flips the
* favourite-longshot sign, fails the stat outright.
*
* ── WHAT LODO IS AND IS NOT ──────────────────────────────────────────────
* Refitting on all-but-one date uses dates that follow the held-out one, so this
* is a STABILITY test, not a point-in-time backtest. The point-in-time result is
* separate and already established (fit-past / apply-forward, CI excluding zero
* on hits / TB / RBI). Both are required; neither substitutes for the other.
*
* SUPABASE_URL=... node scripts/lodo-calibration.js
*/
require('dotenv').config();
const fs = require('fs');
const path = require('path');
const { createClient } = require('@supabase/supabase-js');
const cal = require('../src/services/model/calibration');
const guards = require('../src/services/model/calibrationGuards');
const { knownNumber } = require('../src/utils/known');
const SB_URL = process.env.SUPABASE_URL;
const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
const STATS = ['hits', 'total_bases', 'rbi', 'runs'];
const PAGE = 1000;
/** The favourite bucket where the over-prediction concentrates. */
const FAVOURITE_FLOOR = 0.9;
/** Minimum rows on a held-out date for that drop to be informative. */
const MIN_HELD_ROWS = 20;
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
async function page(sb, table, select, apply) {
const out = [];
for (let from = 0; ; from += PAGE) {
const { data, error } = await apply(sb.from(table).select(select))
.order('id', { ascending: true }).range(from, from + PAGE - 1);
if (error) throw error;
if (!data || data.length === 0) break;
out.push(...data);
if (data.length < PAGE) break;
}
return out;
}
const isPreGame = (capturedAt, gameDate) => {
const et = new Date(new Date(capturedAt).getTime() - 4 * 3600 * 1000);
const d = et.toISOString().slice(0, 10);
return d < gameDate || (d === gameDate && et.getUTCHours() < 19);
};
async function main() {
const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
const snaps = await page(sb, 'model_snapshots',
'id, game_date, captured_at, stat, player_key, line, side, p_win, refused',
(q) => q.eq('sport', 'mlb').in('stat', STATS));
// Build the RAW population first so the guard has something to catch.
const raw = [];
const picked = new Map();
for (const r of snaps) {
if (!isPreGame(r.captured_at, r.game_date)) continue;
if (r.refused || knownNumber(r.p_win) === null) continue;
const propKey = [r.game_date, r.stat, r.player_key, r.line].join('|');
raw.push({ propKey, side: r.side, p: knownNumber(r.p_win) });
const prev = picked.get(propKey);
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(propKey, r);
}
// GUARD 1 — prove the raw population would have lied, then prove dedup fixes it.
const rawCheck = guards.checkPickedSideDedup(raw);
const pickedRows = [...picked.values()].map((r) => ({
propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'),
side: r.side, p: knownNumber(r.p_win),
}));
guards.assertPickedSideDedup(pickedRows); // throws if dedup failed
const out = {
guard_1_raw_population: { violated: rawCheck.violated, mean_p: rawCheck.mean_p, both_sides_share: rawCheck.both_sides_share },
guard_1_after_dedup: guards.checkPickedSideDedup(pickedRows),
per_stat: {},
};
for (const stat of STATS) {
const rows = [];
for (const r of picked.values()) {
if (r.stat !== stat) continue;
const b = lines[`${r.game_date}|${r.player_key}`];
const L = knownNumber(r.line);
if (!b || L === null || !r.side) continue;
const v = knownNumber(FIELD[stat](b));
if (v === null) continue;
const over = v > L;
rows.push({
date: r.game_date,
p: knownNumber(r.p_win),
won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0,
});
}
const dates = [...new Set(rows.map((r) => r.date))].sort();
const full = cal.fitIsotonic(rows.map((r) => ({ p: r.p, won: r.won })));
if (!full) {
out.per_stat[stat] = {
n: rows.length, dates: dates.length,
lodo: 'NOT RUN', decision: 'REFUSE',
reason: `no isotonic map is fittable at n=${rows.length} (needs ${cal.MIN_TOTAL || 200})`,
};
continue;
}
// ── LEAVE ONE DATE OUT ──
const table = [];
for (const d of dates) {
const fit = rows.filter((r) => r.date !== d);
const held = rows.filter((r) => r.date === d);
if (held.length < MIN_HELD_ROWS) {
table.push({ dropped: d, held_n: held.length, verdict: 'UNINFORMATIVE', reason: 'too few rows on this date' });
continue;
}
const map = cal.fitIsotonic(fit.map((r) => ({ p: r.p, won: r.won })));
const applied = guards.applyOrRefuse(map, held, cal.applyIsotonic);
if (!applied.ok) {
table.push({ dropped: d, held_n: held.length, verdict: 'UNINFORMATIVE', reason: applied.reason });
continue;
}
const ys = applied.rows.map((r) => r.won);
const bRaw = guards.safeBrier(applied.rows.map((r) => r.p), ys);
const bCal = guards.safeBrier(applied.rows.map((r) => r.pc), ys);
if (bRaw === null || bCal === null) {
table.push({ dropped: d, held_n: held.length, verdict: 'UNINFORMATIVE', reason: 'a null reached the metric' });
continue;
}
const fav = applied.rows.filter((r) => r.p >= FAVOURITE_FLOOR);
const favBias = fav.length >= 5 ? mean(fav.map((r) => r.p)) - mean(fav.map((r) => r.won)) : null;
table.push({
dropped: d,
held_n: held.length,
brier_delta: round4(bCal - bRaw),
improves: bCal < bRaw,
favourite_n: fav.length,
favourite_bias: favBias === null ? null : round4(favBias),
favourite_sign_holds: favBias === null ? null : favBias > 0,
verdict: bCal < bRaw ? 'holds' : 'REVERSES',
});
}
const informative = table.filter((t) => t.verdict !== 'UNINFORMATIVE');
const anyReversal = informative.some((t) => t.verdict === 'REVERSES');
const signTested = informative.filter((t) => t.favourite_sign_holds !== null);
const anySignFlip = signTested.some((t) => t.favourite_sign_holds === false);
const passes = informative.length > 0 && !anyReversal && !anySignFlip;
out.per_stat[stat] = {
n: rows.length,
dates: dates.length,
lodo_table: table,
informative_drops: informative.length,
brier_reversals: informative.filter((t) => t.verdict === 'REVERSES').length,
favourite_sign_flips: signTested.filter((t) => t.favourite_sign_holds === false).length,
favourite_sign_untested: informative.length - signTested.length,
lodo: passes ? 'PASS' : 'FAIL',
reason: passes
? 'improvement never reverses and the favourite over-prediction never flips sign across any single-date drop'
: (anyReversal
? `improvement reverses when ${informative.filter((t) => t.verdict === 'REVERSES').map((t) => t.dropped).join(', ')} is dropped — the effect is date-driven`
: `the favourite over-prediction flips sign when ${signTested.filter((t) => t.favourite_sign_holds === false).map((t) => t.dropped).join(', ')} is dropped`),
};
}
console.log(JSON.stringify(out, null, 2));
process.exit(0);
}
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
main().catch((e) => { console.error(e); process.exit(1); });
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env node
'use strict';
/**
* PHASE 5 — rebuild grade bands on p_win_calibrated, for DEPLOYED stats only.
*
* total_bases is the only stat that cleared LODO, so it is the only one whose
* bands are rebuilt on calibrated values. The rest keep base-rate bands built on
* raw p_win, and the reason is named rather than left to inference.
*
* The two-bar rule still applies and still bites: TB is now CALIBRATED but no
* factor is PROVEN for it (barrel, exit velo and hard-contact-allowed were all
* THEATER), so the bands remain a base-rate read — now an honestly-numbered one.
*/
require('dotenv').config();
const fs = require('fs');
const path = require('path');
const { createClient } = require('@supabase/supabase-js');
const cal = require('../src/services/model/calibration');
const gb = require('../src/services/model/gradeBands');
const guards = require('../src/services/model/calibrationGuards');
const tl = require('../src/services/model/testLedger');
const { knownNumber } = require('../src/utils/known');
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
const STAT = process.env.BAND_STAT || 'total_bases';
const PAGE = 1000;
async function page(sb, t, s, f) {
const o = [];
for (let i = 0; ; i += PAGE) {
const { data, error } = await f(sb.from(t).select(s)).order('id', { ascending: true }).range(i, i + PAGE - 1);
if (error) throw error;
if (!data.length) break;
o.push(...data);
if (data.length < PAGE) break;
}
return o;
}
const isPreGame = (c, g) => {
const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000);
const d = et.toISOString().slice(0, 10);
return d < g || (d === g && et.getUTCHours() < 19);
};
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
(async () => {
const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } });
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
const snaps = await page(sb, 'model_snapshots',
'id, game_date, captured_at, stat, player_key, line, side, p_win, refused, archetype',
(q) => q.eq('sport', 'mlb').eq('stat', STAT));
const picked = new Map();
for (const r of snaps) {
if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue;
const k = [r.game_date, r.stat, r.player_key, r.line].join('|');
const prev = picked.get(k);
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r);
}
guards.assertPickedSideDedup([...picked.values()].map((r) => ({
propKey: [r.game_date, r.stat, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win),
})));
const rows = [];
for (const r of picked.values()) {
const b = lines[`${r.game_date}|${r.player_key}`];
const L = knownNumber(r.line);
if (!b || L === null || !r.side) continue;
const v = knownNumber(FIELD[STAT](b));
if (v === null) continue;
const over = v > L;
rows.push({
date: r.game_date, p: knownNumber(r.p_win),
won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0,
archetype: String(r.archetype || 'UNLABELLED').toUpperCase(),
});
}
rows.sort((a, b) => String(a.date).localeCompare(String(b.date)));
// Point-in-time map, then apply forward.
const dates = [...new Set(rows.map((r) => r.date))].sort();
const perDate = new Map();
for (const r of rows) perDate.set(r.date, (perDate.get(r.date) || 0) + 1);
let acc = 0; let cut = dates[dates.length - 1];
for (const d of dates) { acc += perDate.get(d); if (acc >= rows.length * 0.45) { cut = d; break; } }
const map = cal.fitIsotonic(rows.filter((r) => r.date < cut).map((r) => ({ p: r.p, won: r.won })));
const applied = guards.applyOrRefuse(map, rows.filter((r) => r.date >= cut), cal.applyIsotonic);
if (!applied.ok) { console.log(JSON.stringify({ stat: STAT, refused: applied.reason })); process.exit(0); }
const mc = await tl.recordAndCount(tl.supabaseStore(sb), []).catch(() => ({ cumulative_tests: 1 }));
const byArch = new Map();
for (const r of applied.rows) {
if (!byArch.has(r.archetype)) byArch.set(r.archetype, []);
byArch.get(r.archetype).push({ p: r.pc, won: r.won });
}
const out = [];
for (const [arch, rs] of [...byArch.entries()].sort((a, b) => b[1].length - a[1].length)) {
out.push(gb.buildBands(rs, {
archetype: arch,
cumulativeTests: mc.cumulative_tests,
// TB is CALIBRATED (provisional) but no factor is PROVEN for it.
proven: false,
calibrated: true,
}));
}
console.log(JSON.stringify({
stat: STAT,
basis: 'p_win_calibrated (PROVISIONAL)',
eval_rows: applied.rows.length,
cumulative_tests: mc.cumulative_tests,
two_bar_note: 'calibrated YES, proven NO -> bands stay a base-rate read, now honestly numbered',
bands: out,
}, null, 2));
process.exit(0);
})().catch((e) => { console.error(e); process.exit(1); });
+184
View File
@@ -0,0 +1,184 @@
# LODO-gated provisional calibration — total_bases deploys, three stats refuse
## PHASE 0 — Record correction
**The ≥40 date-cluster deploy floor applied to CALIBRATION was the wrong
instrument, and I applied it without challenging the binding.**
It is `factorGate`'s cluster-robust interval floor, built for a factor making a
CAUSAL claim, where the risk is a false positive dressed as mechanism. A
calibration layer is different in kind:
- it makes **no causal claim** — it is a monotone shrink toward observed
- its failure mode is **bounded**: it can only over- or under-shrink
- it consumes **no Bonferroni slot**
Its real risk is that the correction is **date-driven**, and leave-one-date-out
tests that directly. The replacement bar is **stricter on stability**, not looser
on standard: LODO fails a stat if removing any single day reverses the
improvement, which a cluster count cannot detect at all.
The prior order's date premise was also wrong (05-01→08-04, "~90 dates"); the
snapshots span 07-19→08-06 = 19 dates. That was corrected in the settlement
session. The mis-bound instrument is mine.
**The ≥40 floor is retained, correctly scoped as the PROMOTION bar** — the point
at which a stat leaves provisional status.
---
## PHASE 1 — Both guards codified (11 tests)
`src/services/model/calibrationGuards.js`
**GUARD 1 — the both-sides tell.** Picked-side dedup is now mandatory
preprocessing, asserted. The guard fires on the CONJUNCTION of both sides being
present AND mean p_win pinned near 0.5 — either alone is unremarkable, and
flagging a genuinely balanced one-sided book would be a false alarm.
Demonstrated on live data in this run:
```
raw population violated=true mean_p 0.4962 both_sides_share 0.9763
after dedup violated=false mean_p 0.6694
```
**GUARD 2 — a null that scores itself.** `safeBrier` refuses when any prediction
is null; `applyOrRefuse` drops unmappable rows rather than passing nulls
downstream. A test demonstrates the trap explicitly — `(null1)² === 1` and
`(null0)² === 0`, so a Brier over nulls silently equals the win rate.
---
## PHASE 2 — LODO table
Refit dropping each date; measure held-out Brier delta and the sign of the >0.9
favourite bias. Drops with fewer than 20 held rows are marked UNINFORMATIVE
rather than counted either way.
**A note on what LODO is:** refitting on all-but-one date uses dates that follow
the held-out one, so this is a STABILITY test, not a point-in-time backtest. The
point-in-time result is separate and already established. Both are required.
| stat | n | dates | informative drops | reversals | sign flips | LODO |
|---|---|---|---|---|---|---|
| hits | 1,140 | 17 | 7 | **2** (07-22 n=20, 07-26 n=25) | 0 | **FAIL** |
| **total_bases** | 1,050 | 7 | 5 | 0 | 0 | **PASS** |
| rbi | 630 | 5 | 5 | **1** (08-01 n=99) | 0 | **FAIL** |
| runs | 597 | 5 | 5 | **2** (08-01 n=86, 08-05 n=244) | 0 | **FAIL** |
### Threshold sensitivity — reported because the verdict moves
| min held rows | hits | total_bases | rbi | runs |
|---|---|---|---|---|
| **20** (applied) | FAIL | **PASS** | FAIL | FAIL |
| 30 | PASS | **PASS** | FAIL | FAIL |
| 50 / 75 | PASS | **PASS** | FAIL | FAIL |
| 100 | PASS | **PASS** | PASS | FAIL |
- **total_bases passes at every threshold** — the only unambiguous result.
- **runs fails at every threshold**, reversing on a 244-row date.
- **hits' failure is threshold-fragile**: it fails only when 20- and 25-row dates
are admitted, and those are the two smallest informative drops in the set.
I chose `MIN_HELD_ROWS = 20` before seeing which stats passed, and did not move
it afterwards to preserve a deploy. The honest caveat: a per-date Brier delta on
20 rows has a standard error several times the effect being tested, so the LODO
instrument is underpowered per-drop at this sample size. That argues for
pre-registering a higher threshold — a Roundtable decision, not one to make while
holding the results.
---
## PHASE 3 — Deploy decisions
| stat | LODO | point-in-time CI | decision |
|---|---|---|---|
| **total_bases** | PASS | [0.0061, 0.0045] | **DEPLOY-PROVISIONAL** |
| hits | FAIL | [0.0139, 0.0097] | REFUSE — improvement reverses on 07-22 / 07-26 |
| rbi | FAIL | [0.0092, 0.0010] | REFUSE — improvement reverses on 08-01 |
| runs | FAIL | no fittable map at the point-in-time split | REFUSE — honest null |
Certified band for total_bases: **[0.60.8]**. Outside it → refuse, fall to base
rate.
### hits was being served calibrated, and is not any more
`snapshotService` hardcoded hits calibration since S91. hits fails LODO, so it
has been removed from the deployed set. **A stat that cannot survive dropping one
day was never calibrated — it was fitted to that day.** The consequence is real:
hits props become unstackable again for `chain.chainAcross`. That is the honest
result of measuring it, not a regression to route around, and it errs toward
withdrawing a claim rather than preserving one on a fragile verdict.
Deployment is now driven by `CALIBRATION_DEPLOYED` (frozen, tested), not a
hardcoded stat name.
---
## PHASE 4 — Auto-demotion (14 tests)
`src/services/model/calibrationRegistry.js`
- **Deploy needs BOTH gates** — LODO pass AND a point-in-time CI excluding zero.
Neither is waivable.
- **`reverify` demotes on the first breach**: the CI ceasing to exclude zero, or
the favourite over-prediction flipping sign (which would mean the correction is
now pushing the wrong way). The breaking date is logged.
- **Promotion to non-provisional** requires the original ≥40 date-cluster bar,
with the interval still holding.
A provisional deploy that cannot be taken away is just a deploy; `reverify` is
what makes the label mean something.
---
## PHASE 5 — Bands rebuilt on p_win_calibrated (total_bases only)
625 eval rows on calibrated values. **The two-bar rule still bites**: TB is now
CALIBRATED but no factor is PROVEN for it (barrel, exit velo and
hard-contact-allowed were all THEATER), so bands remain a base-rate read — now an
honestly-numbered one.
| archetype | n | base rate | bands | separation |
|---|---|---|---|---|
| UNLABELLED | 275 | 0.6255 | 1 | indistinguishable from base rate |
| BOMBER | 200 | 0.6100 | 1 | indistinguishable |
| GHOST | 87 | 0.5747 | 1 | indistinguishable |
| DRIVER | 24 | 0.7917 | 1 (PROVISIONAL) | indistinguishable |
| BRUSH | 19 | 0.4737 | 1 (PROVISIONAL) | indistinguishable |
| MIRROR | 6 | — | REFUSED | insufficient outcomes |
Calibration compressed the served range to 0.42861.0. Every archetype still
collapses to a single band — calibrated p_win does not separate within archetype
any better than raw p_win did. Refused stats keep base-rate bands on raw p_win.
---
## PHASE 6 — Logged, not acted on
**The dead gradient is buried.** Over-prediction ordering on the fuller settled
set is **hits ≈ TB > runs > RBI**, not TB > RBI > runs. The skill-driven-gradient
mechanism did not survive — **RBI has the SMALLEST bias** (+0.0164). Descriptive
only; no mechanism claimed.
**Refusal coverage.** Refused props are predictable-but-input-less rather than
genuinely uncertain (3.20 vs 3.39 AB rules out playing time). The refused set is
a MAP OF MISSING INPUTS and feeds the input-coverage roadmap. Not this order.
**Queued candidate:** a low-parameter calibrator (Platt / beta) fits a
favourite-longshot shape on far fewer points than isotonic needs, which is
exactly the constraint that refused runs. It is a NEW estimator requiring its own
out-of-sample validation. Not built here.
**Programme-level finding:** calibration beats every factor tried on TB / RBI /
runs, and the defect is systematic over-prediction **concentrated in favourites**
(+0.21 to +0.28 above p_win 0.9 on all four stats) rather than a uniform shift.
---
## Invariants
`p_win` never mutated — calibration rides as `p_win_calibrated` with
`calibration_status: 'provisional'`. No Bonferroni slot consumed; testLedger
factor count untouched. Counter and frozen clusters byte-identical.
+146
View File
@@ -0,0 +1,146 @@
'use strict';
/**
* calibrationGuards — the two ways a calibration measurement lies to you.
*
* Both of these produced a confident, plausible, completely wrong number in the
* settlement session, and neither was visible in the output. They are codified
* here so the failure cannot recur silently.
*
* ── GUARD 1: THE BOTH-SIDES TELL ─────────────────────────────────────────
* A prop population usually carries BOTH the over and the under. Their p_wins
* sum to ~1 and their outcomes are complementary, so ANY population-level
* calibration statistic over the raw set is pinned to 0.5 by construction — not
* by the model being calibrated.
*
* Measured: the raw population read +0.0002 bias on hits ("perfectly
* calibrated"); deduped to the model-picked side it read +0.0868. Same rows,
* opposite conclusion. The tell was mean p_win sitting at 0.4998 on all four
* stats at once, which is not something a real forecaster does.
*
* So picked-side dedup is MANDATORY preprocessing, and this asserts it.
*
* ── GUARD 2: A NULL THAT SCORES ITSELF ───────────────────────────────────
* `fitIsotonic` returns null below its minimum and `applyIsotonic` then returns
* null per row. In JavaScript `(null - 1) ** 2 === 1` and `(null - 0) ** 2 === 0`,
* so a Brier score computed over nulls silently equals the WIN RATE — a number
* in the right range, monotone in the data, and completely meaningless. It
* reported hits at 0.5567 against a 0.5684 win rate.
*
* This is the `Number(null) === 0` breach the TRUTH LAW names, wearing a metric.
* A null prediction must refuse, never score.
*/
const { knownNumber } = require('../../utils/known');
/** How close to 0.5 counts as the both-sides signature. */
const BALANCED_TOLERANCE = 0.02;
/**
* Does this population still contain both sides of the same prop?
*
* @param {Array} rows [{ p, side, propKey }]
* @returns {object} { violated, reason, ... } — never throws, so a caller can
* decide between refusing and hard-failing.
*/
function checkPickedSideDedup(rows) {
const usable = (rows || []).filter((r) => knownNumber(r && r.p) !== null);
if (usable.length < 2) return { violated: false, reason: 'too few rows to judge', n: usable.length };
const sidesByProp = new Map();
for (const r of usable) {
const k = r.propKey == null ? null : String(r.propKey);
if (k === null) continue;
if (!sidesByProp.has(k)) sidesByProp.set(k, new Set());
if (r.side) sidesByProp.get(k).add(String(r.side).toLowerCase());
}
let bothSides = 0;
for (const s of sidesByProp.values()) if (s.size > 1) bothSides += 1;
const propCount = sidesByProp.size;
const bothShare = propCount ? bothSides / propCount : 0;
const meanP = usable.reduce((s, r) => s + knownNumber(r.p), 0) / usable.length;
const balanced = Math.abs(meanP - 0.5) <= BALANCED_TOLERANCE;
// The violation is the CONJUNCTION: both sides present AND the mean pinned at
// 0.5. Either alone is unremarkable — a genuinely balanced book of one-sided
// picks is fine, and both sides present with a skewed mean means someone
// already deduped.
const violated = bothSides > 0 && balanced;
return {
violated,
n: usable.length,
props: propCount,
both_sides_props: bothSides,
both_sides_share: round4(bothShare),
mean_p: round4(meanP),
reason: violated
? `both sides present on ${bothSides}/${propCount} props while mean p_win is ${round4(meanP)}`
+ 'the population is balanced by construction and any calibration statistic over it is meaningless. '
+ 'Dedup to the model-picked side first.'
: null,
};
}
/** Same check, but refuses to continue. Use at the top of a measurement. */
function assertPickedSideDedup(rows) {
const r = checkPickedSideDedup(rows);
if (r.violated) throw new Error(`CALIBRATION GUARD: ${r.reason}`);
return r;
}
/**
* Brier score that refuses rather than scoring a null.
*
* @param {Array} preds
* @param {Array} outcomes
* @param {object} opts { onNull: 'throw' | 'refuse' } default 'refuse'
* @returns {number|null} null when any prediction is unreadable
*/
function safeBrier(preds, outcomes, opts = {}) {
const ps = preds || [];
const ys = outcomes || [];
if (ps.length === 0 || ps.length !== ys.length) return null;
let sum = 0;
for (let i = 0; i < ps.length; i += 1) {
const p = knownNumber(ps[i]);
const y = knownNumber(ys[i]);
if (p === null || y === null) {
// NEVER score it. (null - 1) ** 2 === 1 would pass silently.
if (opts.onNull === 'throw') {
throw new Error('CALIBRATION GUARD: a null prediction reached a Brier term');
}
return null;
}
sum += (p - y) ** 2;
}
return sum / ps.length;
}
/**
* Map a population through a calibration map, refusing unreadable rows rather
* than letting them through as nulls.
*/
function applyOrRefuse(map, rows, applyFn) {
if (!map) return { ok: false, reason: 'no calibration map could be fitted', rows: [] };
const out = [];
let dropped = 0;
for (const r of rows || []) {
const pc = applyFn(map, r.p);
if (knownNumber(pc) === null) { dropped += 1; continue; }
out.push({ ...r, pc });
}
return {
ok: out.length > 0,
rows: out,
dropped,
reason: out.length === 0 ? 'every row was unmappable' : null,
};
}
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
module.exports = {
checkPickedSideDedup, assertPickedSideDedup, safeBrier, applyOrRefuse,
BALANCED_TOLERANCE,
};
+110
View File
@@ -0,0 +1,110 @@
'use strict';
/**
* calibrationRegistry — which stats are allowed to serve a calibrated number.
*
* ── WHY THIS IS NOT THE FACTOR REGISTRY ──────────────────────────────────
* A factor makes a CAUSAL claim, so it needs a Bonferroni slot, a cluster-robust
* interval, and a bar that rises with every hypothesis the programme tests.
* Calibration makes no causal claim: it is a monotone shrink toward what was
* actually observed, its failure mode is bounded (it can only over- or
* under-shrink), and it consumes no test slot.
*
* Applying the factor gate's >=40 date-cluster interval floor to it was the
* wrong instrument. The real risk for a calibration layer is that the correction
* is DATE-DRIVEN, and leave-one-date-out tests that directly — and harder.
*
* ── TWO TIERS, AND AUTO-DEMOTION IS WHAT MAKES PROVISIONAL HONEST ────────
* PROVISIONAL LODO passes AND the point-in-time held-out CI excludes zero.
* Serves, labelled, inside its certified band only.
* PROMOTED the original >=40 date-cluster bar, now correctly scoped as the
* PROMOTION bar rather than the deploy bar.
*
* A provisional deploy that cannot be taken away is just a deploy. `reverify`
* runs on every newly settled date and demotes on the first breach.
*/
const { knownNumber } = require('../../utils/known');
const STATUS = Object.freeze({ NONE: 'none', PROVISIONAL: 'provisional', PROMOTED: 'promoted' });
/** The ORIGINAL floor, correctly scoped: promotion, not deploy. */
const PROMOTION_DATE_CLUSTERS = 40;
function createRegistry(initial = {}) {
const state = new Map(Object.entries(initial));
const log = [];
/** Deploy requires BOTH gates. Neither can be waived. */
function deploy(stat, evidence = {}) {
const lodo = evidence.lodo_pass === true;
const ci = Array.isArray(evidence.ci) && evidence.ci.length === 2 && evidence.ci[1] < 0;
if (!lodo || !ci) {
return {
ok: false,
status: STATUS.NONE,
reason: !lodo
? 'LODO did not pass — the correction may be date-driven'
: 'the point-in-time held-out interval does not exclude zero',
};
}
if (!evidence.map) return { ok: false, status: STATUS.NONE, reason: 'no calibration map supplied' };
state.set(stat, {
status: STATUS.PROVISIONAL,
map: evidence.map,
certified_bands: evidence.certified_bands || [],
date_clusters: knownNumber(evidence.date_clusters) ?? 0,
ci: evidence.ci,
deployed_at: evidence.at || null,
});
log.push({ stat, event: 'deployed_provisional', at: evidence.at || null });
return { ok: true, status: STATUS.PROVISIONAL };
}
/**
* Re-verify on a newly settled date. Demotes on the FIRST breach — either the
* interval ceasing to exclude zero, or the favourite over-prediction flipping
* sign (which would mean the correction is now pushing the wrong way).
*/
function reverify(stat, obs = {}) {
const cur = state.get(stat);
if (!cur || cur.status === STATUS.NONE) return { status: STATUS.NONE, changed: false };
const ciHolds = Array.isArray(obs.ci) && obs.ci.length === 2 && obs.ci[1] < 0;
const signHolds = obs.favourite_bias == null ? true : knownNumber(obs.favourite_bias) > 0;
if (!ciHolds || !signHolds) {
state.set(stat, { status: STATUS.NONE, demoted_at: obs.date || null, demoted_reason: !ciHolds ? 'ci_no_longer_excludes_zero' : 'favourite_bias_flipped' });
log.push({ stat, event: 'auto_demoted', at: obs.date || null, reason: !ciHolds ? 'ci_no_longer_excludes_zero' : 'favourite_bias_flipped' });
return { status: STATUS.NONE, changed: true, reason: !ciHolds ? 'ci_no_longer_excludes_zero' : 'favourite_bias_flipped', breaking_date: obs.date || null };
}
// PROMOTION uses the original floor, now correctly scoped.
const dc = knownNumber(obs.date_clusters) ?? cur.date_clusters;
if (cur.status === STATUS.PROVISIONAL && dc >= PROMOTION_DATE_CLUSTERS) {
state.set(stat, { ...cur, status: STATUS.PROMOTED, date_clusters: dc, promoted_at: obs.date || null });
log.push({ stat, event: 'promoted', at: obs.date || null, date_clusters: dc });
return { status: STATUS.PROMOTED, changed: true };
}
if (dc !== cur.date_clusters) state.set(stat, { ...cur, date_clusters: dc });
return { status: cur.status, changed: false };
}
/** Is this stat allowed to serve a calibrated number for THIS p_win? */
function serves(stat, p) {
const cur = state.get(stat);
if (!cur || cur.status === STATUS.NONE) return { serve: false, reason: 'not deployed' };
const x = knownNumber(p);
if (x === null) return { serve: false, reason: 'no p_win' };
const inBand = (cur.certified_bands || []).some((b) => x >= b[0] && x < b[1]);
if (!inBand) return { serve: false, reason: 'outside the certified band', status: cur.status };
return { serve: true, status: cur.status, provisional: cur.status === STATUS.PROVISIONAL };
}
const get = (stat) => state.get(stat) || { status: STATUS.NONE };
const all = () => Object.fromEntries([...state.entries()].map(([k, v]) => [k, { status: v.status, certified_bands: v.certified_bands, date_clusters: v.date_clusters }]));
return { deploy, reverify, serves, get, all, log: () => log.slice() };
}
module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS };
+47 -22
View File
@@ -275,6 +275,16 @@ async function loadPitcherArsenals(sport) {
} catch { return out; }
}
/**
* Stats whose calibration passed leave-one-date-out and may serve a calibrated
* number. PROVISIONAL: auto-demoted the first time the held-out interval stops
* excluding zero or the favourite over-prediction flips sign.
*
* hits / rbi / runs are deliberately ABSENT — each fails LODO. See
* specs/lodo-provisional-calibration.md.
*/
const CALIBRATION_DEPLOYED = Object.freeze(['total_bases']);
async function runSnapshot(sport, opts = {}) {
const sp = String(sport || '').toLowerCase();
const deps = {
@@ -700,37 +710,51 @@ async function runSnapshot(sport, opts = {}) {
console.warn(`[challenger] ${sp} skipped:`, e.message);
}
// ── FORWARD CALIBRATION (hits) ────────────────────────────────────────
// ── FORWARD CALIBRATION (LODO-gated, per stat) ────────────────────────
// Fitted on games that are OVER, applied to tonight's props. `p_win` is NOT
// touched — the counter stays byte-identical and the calibrated value rides
// beside it, because a calibration map is a correction TO a forecast, not a
// different forecast.
//
// WHICH STATS SERVE IS MEASURED, NOT ASSUMED. The deploy bar is leave-one-
// date-out stability: refit dropping each settled date in turn, and the
// improvement must never reverse. That is the right instrument for a monotone
// shrink-to-observed layer — the factor gate's >=40 date-cluster interval
// floor was built for a CAUSAL claim and does not bind here.
//
// Measured 2026-08-07: total_bases passes at every held-size threshold. hits
// FAILS (reverses on 2026-07-22 and 2026-07-26), so it is no longer served
// calibrated even though it was — a stat that cannot survive dropping one day
// was never calibrated, it was fitted to that day. rbi and runs also fail.
//
// `calibrated` is true only inside a band certified out-of-sample, and it is
// what `chain.chainAcross` requires before it will compound anything. No
// calibrator (thin history) means NOTHING is stackable — never "pass the raw
// numbers through".
// what `chain.chainAcross` requires before it will compound anything. Removing
// hits here makes hits props unstackable again, which is the honest
// consequence of the measurement rather than a regression to work around.
if (sp === 'mlb') {
try {
const calSvc = deps.calibrationService || require('./model/calibrationService');
const sbc = require('../utils/supabase').getSupabaseServiceClient();
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat: 'hits' }) : null;
if (calibrator) {
let marked = 0;
for (const g of enriched) {
if (String(g.stat_type || g.stat || '').toLowerCase() !== 'hits') continue;
const out = calibrator.calibrate(g.p_win);
g.p_win_calibrated = out.p_calibrated;
g.calibrated = out.calibrated;
g.calibration_reason = out.reason;
if (out.calibrated) marked += 1;
for (const stat of CALIBRATION_DEPLOYED) {
try {
const calSvc = deps.calibrationService || require('./model/calibrationService');
const sbc = require('../utils/supabase').getSupabaseServiceClient();
const calibrator = sbc ? await calSvc.fromLedger(sbc, { sport: 'mlb', stat }) : null;
if (calibrator) {
let marked = 0;
for (const g of enriched) {
if (String(g.stat_type || g.stat || '').toLowerCase() !== stat) continue;
const out = calibrator.calibrate(g.p_win);
g.p_win_calibrated = out.p_calibrated;
g.calibrated = out.calibrated;
g.calibration_reason = out.reason;
g.calibration_status = 'provisional';
if (out.calibrated) marked += 1;
}
console.log(`[calibration] ${sp} ${stat} (PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}`);
} else {
console.log(`[calibration] ${sp} ${stat} — no calibrator (thin history); nothing is stackable`);
}
console.log(`[calibration] ${sp} hits — ${marked} stackable of ${enriched.filter((g) => String(g.stat_type || g.stat || '').toLowerCase() === 'hits').length}; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}, bands ${JSON.stringify(calibrator.bands.map((b) => [b.lo, b.hi]))}`);
} else {
console.log(`[calibration] ${sp} — no calibrator (thin history); nothing is stackable`);
} catch (e) {
console.warn(`[calibration] ${stat} skipped:`, e.message);
}
} catch (e) {
console.warn('[calibration] skipped:', e.message);
}
}
@@ -843,5 +867,6 @@ module.exports = {
generateTickerEvents,
pushTickerItems,
ACTIVE_SPORTS,
CALIBRATION_DEPLOYED,
__internals: { propKey, gradedAtFor, indexOdds, lastName, isTopGrade, DELTA_NOISE, DELTA_MOVE, TICKER_CAP },
};
+30
View File
@@ -0,0 +1,30 @@
'use strict';
/**
* Which stats serve a calibrated number in the live pipeline.
*
* The rule this locks: a stat that cannot survive dropping a single settled date
* was never calibrated — it was fitted to that date. hits WAS served calibrated
* and is not any more, which is the honest consequence of measuring it.
*/
const snapshotService = require('../../src/services/snapshotService');
describe('the deployed set is LODO-gated', () => {
it('serves total_bases — it passed at every held-size threshold', () => {
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('total_bases');
});
it('does NOT serve hits, rbi or runs — each fails LODO', () => {
// hits reverses when 2026-07-22 or 2026-07-26 is dropped; rbi on 2026-08-01;
// runs on 2026-08-01 and 2026-08-05.
for (const stat of ['hits', 'rbi', 'runs']) {
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain(stat);
}
});
it('is frozen, so a stat cannot be added at runtime without a code change', () => {
expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true);
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('hits'); }).toThrow();
});
});
+112
View File
@@ -0,0 +1,112 @@
'use strict';
/**
* The two ways a calibration measurement lies.
*
* Both produced a confident, plausible, completely wrong number in the
* settlement session, and neither was visible in the output. These lock them out.
*/
const g = require('../../src/services/model/calibrationGuards');
const cal = require('../../src/services/model/calibration');
/** A population carrying BOTH sides of each prop, as the snapshot table does. */
function bothSides(n) {
const rows = [];
for (let i = 0; i < n; i += 1) {
const p = 0.55 + (i % 7) * 0.05;
rows.push({ propKey: `prop${i}`, side: 'over', p });
rows.push({ propKey: `prop${i}`, side: 'under', p: 1 - p });
}
return rows;
}
describe('GUARD 1 — the both-sides tell', () => {
it('catches the 0.4998 signature: both sides present AND mean pinned at 0.5', () => {
const rows = bothSides(200);
const r = g.checkPickedSideDedup(rows);
expect(r.violated).toBe(true);
expect(r.both_sides_share).toBe(1);
expect(Math.abs(r.mean_p - 0.5)).toBeLessThanOrEqual(g.BALANCED_TOLERANCE);
expect(r.reason).toMatch(/balanced by construction/);
});
it('assert form REFUSES rather than returning a number', () => {
expect(() => g.assertPickedSideDedup(bothSides(100))).toThrow(/CALIBRATION GUARD/);
});
it('passes once deduped to the model-picked side', () => {
// The picked side is the one the model favoured, so the mean sits well
// above 0.5 — which is what a real forecaster's book looks like.
const picked = bothSides(200).filter((r) => r.p > 0.5);
const r = g.checkPickedSideDedup(picked);
expect(r.violated).toBe(false);
expect(r.mean_p).toBeGreaterThan(0.5 + g.BALANCED_TOLERANCE);
});
it('does NOT fire on a genuinely balanced one-sided book', () => {
// Either condition alone is unremarkable. A book of one-sided picks that
// happens to average 0.5 is honest, and flagging it would be a false alarm.
const rows = Array.from({ length: 300 }, (_, i) => ({
propKey: `p${i}`, side: 'over', p: i % 2 ? 0.45 : 0.55,
}));
const r = g.checkPickedSideDedup(rows);
expect(r.both_sides_props).toBe(0);
expect(r.violated).toBe(false);
});
it('does NOT fire when both sides are present but the mean is skewed', () => {
const rows = bothSides(50).concat(
Array.from({ length: 400 }, (_, i) => ({ propKey: `x${i}`, side: 'over', p: 0.8 })));
const r = g.checkPickedSideDedup(rows);
expect(r.both_sides_props).toBeGreaterThan(0);
expect(r.violated).toBe(false); // already deduped elsewhere
});
});
describe('GUARD 2 — a null must never score itself', () => {
it('(null-1)**2 can no longer pass as a metric', () => {
// This is the exact breach: JS scores null as 1 against a win and 0 against
// a loss, so the "Brier" silently equals the win rate.
const outcomes = [1, 1, 0, 1, 0];
const naive = outcomes.reduce((s, y, i) => s + ((null - y) ** 2), 0) / outcomes.length;
const winRate = outcomes.reduce((a, b) => a + b, 0) / outcomes.length;
expect(naive).toBeCloseTo(winRate, 10); // the trap, demonstrated
expect(g.safeBrier([null, null, null, null, null], outcomes)).toBeNull();
});
it('refuses when ANY single prediction is null', () => {
expect(g.safeBrier([0.6, 0.4, null], [1, 0, 1])).toBeNull();
});
it('can be made to hard-fail instead of refusing', () => {
expect(() => g.safeBrier([0.6, null], [1, 0], { onNull: 'throw' }))
.toThrow(/null prediction reached a Brier term/);
});
it('scores normally when every prediction is real', () => {
expect(g.safeBrier([1, 0], [1, 0])).toBe(0);
expect(g.safeBrier([0.5, 0.5], [1, 0])).toBeCloseTo(0.25, 10);
});
it('an unfittable map refuses instead of producing null predictions', () => {
// fitIsotonic returns null below its minimum; this is what must happen next.
const map = cal.fitIsotonic([{ p: 0.6, won: 1 }, { p: 0.4, won: 0 }]);
expect(map).toBeNull();
const out = g.applyOrRefuse(map, [{ p: 0.6 }], cal.applyIsotonic);
expect(out.ok).toBe(false);
expect(out.reason).toMatch(/no calibration map/);
expect(out.rows).toEqual([]);
});
it('drops unmappable rows rather than passing nulls downstream', () => {
const fit = [];
for (let i = 0; i < 400; i += 1) fit.push({ p: 0.3 + (i % 60) / 100, won: i % 3 === 0 ? 1 : 0 });
const map = cal.fitIsotonic(fit);
expect(map).not.toBeNull();
const out = g.applyOrRefuse(map, [{ p: 0.5 }, { p: null }], cal.applyIsotonic);
expect(out.rows.length).toBe(1);
expect(out.dropped).toBe(1);
});
});
+120
View File
@@ -0,0 +1,120 @@
'use strict';
/**
* Which stats may serve a calibrated number.
*
* The thing these protect is the meaning of PROVISIONAL: a provisional deploy
* that cannot be taken away is just a deploy.
*/
const { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS } = require('../../src/services/model/calibrationRegistry');
const MAP = [{ lo: 0.5, hi: 0.7, value: 0.55, n: 300 }];
const GOOD = { lodo_pass: true, ci: [-0.0061, -0.0045], map: MAP, certified_bands: [[0.6, 0.8]], date_clusters: 7, at: '2026-08-06' };
describe('deploy needs BOTH gates', () => {
it('deploys when LODO passes and the interval excludes zero', () => {
const r = createRegistry();
expect(r.deploy('total_bases', GOOD).status).toBe(STATUS.PROVISIONAL);
});
it('refuses on a LODO failure however good the interval', () => {
const r = createRegistry();
const out = r.deploy('runs', { ...GOOD, lodo_pass: false });
expect(out.ok).toBe(false);
expect(out.reason).toMatch(/date-driven/);
});
it('refuses when the interval spans zero however clean the LODO', () => {
const r = createRegistry();
const out = r.deploy('hits', { ...GOOD, ci: [-0.01, 0.002] });
expect(out.ok).toBe(false);
expect(out.reason).toMatch(/does not exclude zero/);
});
it('refuses without a map — there is nothing to serve', () => {
const r = createRegistry();
expect(r.deploy('hits', { ...GOOD, map: null }).ok).toBe(false);
});
});
describe('auto-demotion is what makes provisional honest', () => {
it('demotes on the first date where the interval stops excluding zero', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.004, 0.001], date: '2026-08-07' });
expect(out.status).toBe(STATUS.NONE);
expect(out.reason).toBe('ci_no_longer_excludes_zero');
expect(out.breaking_date).toBe('2026-08-07');
expect(r.serves('total_bases', 0.65).serve).toBe(false);
});
it('demotes when the favourite over-prediction flips sign', () => {
// A flip means the correction is now pushing the wrong way.
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.006, -0.004], favourite_bias: -0.03, date: '2026-08-08' });
expect(out.status).toBe(STATUS.NONE);
expect(out.reason).toBe('favourite_bias_flipped');
});
it('logs the demotion with its breaking date', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
r.reverify('total_bases', { ci: [0.001, 0.004], date: '2026-08-09' });
const ev = r.log().find((e) => e.event === 'auto_demoted');
expect(ev).toMatchObject({ stat: 'total_bases', at: '2026-08-09' });
});
it('stays deployed while both conditions hold', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.007, -0.003], favourite_bias: 0.17, date: '2026-08-07' });
expect(out.status).toBe(STATUS.PROVISIONAL);
expect(out.changed).toBe(false);
});
});
describe('the >=40 date-cluster bar is the PROMOTION bar, not the deploy bar', () => {
it('does not block deployment', () => {
const r = createRegistry();
expect(r.deploy('total_bases', { ...GOOD, date_clusters: 7 }).ok).toBe(true);
});
it('promotes out of provisional once it is met', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.006, -0.004], date_clusters: PROMOTION_DATE_CLUSTERS, date: '2026-09-15' });
expect(out.status).toBe(STATUS.PROMOTED);
});
it('does not promote while the interval has stopped holding', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.001, 0.003], date_clusters: 60, date: '2026-09-15' });
expect(out.status).toBe(STATUS.NONE);
});
});
describe('serving is band-limited', () => {
it('serves inside the certified band and refuses outside it', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
expect(r.serves('total_bases', 0.65).serve).toBe(true);
expect(r.serves('total_bases', 0.65).provisional).toBe(true);
expect(r.serves('total_bases', 0.95).serve).toBe(false);
expect(r.serves('total_bases', 0.95).reason).toMatch(/outside the certified band/);
});
it('an undeployed stat never serves', () => {
const r = createRegistry();
expect(r.serves('hits', 0.6).serve).toBe(false);
expect(r.serves('hits', 0.6).reason).toBe('not deployed');
});
it('a missing p_win serves nothing', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
expect(r.serves('total_bases', null).serve).toBe(false);
});
});
+1 -1
View File
File diff suppressed because one or more lines are too long