Files
vyndr/tests/unit/calibrationRegistry.test.js
T
builtbykev 6ae11f1193 LODO-gated provisional calibration: total_bases deploys, hits withdrawn
PHASE 0 — I applied factorGate's >=40 date-cluster floor to a calibration
layer without challenging the binding. That floor is a cluster-robust
interval bar for a CAUSAL claim. Calibration makes no causal claim, has a
bounded failure mode (it can only over- or under-shrink) and consumes no
Bonferroni slot. Its real risk is that the correction is DATE-DRIVEN, and
leave-one-date-out tests that directly -- a STRICTER bar, since a cluster
count cannot detect a single day carrying the effect. The >=40 floor is
retained, correctly scoped as the PROMOTION bar.

PHASE 1 — both guards codified, 11 tests, green before Phase 2.
Demonstrated on live data: raw population violated=true, mean_p 0.4962,
both_sides_share 0.9763; after dedup violated=false, mean_p 0.6694. The
null guard's test demonstrates the trap explicitly, since (null-1)**2 is
1 and (null-0)**2 is 0 so a Brier over nulls equals the win rate.

PHASE 2 — LODO:

  hits         n=1140 dates=17  2 reversals (07-22 n=20, 07-26 n=25)  FAIL
  total_bases  n=1050 dates=7   0 reversals, 0 sign flips             PASS
  rbi          n= 630 dates=5   1 reversal  (08-01 n=99)              FAIL
  runs         n= 597 dates=5   2 reversals (08-01 n=86, 08-05 n=244) FAIL

Threshold sensitivity reported because the verdict moves: total_bases
passes at every held-size threshold, runs fails at every one, and hits
fails ONLY when 20/25-row dates are admitted. I fixed MIN_HELD_ROWS=20
before seeing which stats passed and did not move it afterwards to
preserve a deploy. Honest caveat: a per-date Brier delta on 20 rows has a
standard error several times the effect, so the instrument is
underpowered per-drop -- an argument for pre-registering a higher
threshold, which is a Roundtable call, not one to make while holding the
results.

PHASE 3 — total_bases DEPLOY-PROVISIONAL, band [0.6-0.8]. hits, rbi and
runs REFUSE.

HITS WAS BEING SERVED CALIBRATED AND IS NOT ANY MORE. snapshotService
hardcoded it since S91; it fails LODO, so it is out. A stat that cannot
survive dropping one day was never calibrated, it was fitted to that day.
The consequence is real -- hits props become unstackable for
chain.chainAcross -- and it errs toward withdrawing a claim rather than
preserving one on a fragile verdict. Deployment is now driven by a frozen,
tested CALIBRATION_DEPLOYED set, not a hardcoded stat name.

PHASE 4 — calibrationRegistry, 14 tests. Deploy needs BOTH gates, neither
waivable. reverify auto-demotes on the first breach (CI stops excluding
zero, or the favourite bias flips sign) and logs the breaking date.
Promotion needs the original >=40 bar. A provisional deploy that cannot be
taken away is just a deploy.

PHASE 5 — TB bands rebuilt on calibrated values, 625 eval rows. The
two-bar rule still bites: calibrated YES, proven NO, so they stay a
base-rate read, now honestly numbered. Every archetype still collapses to
one band -- calibrated p_win separates within archetype no better than raw.

PHASE 6 logged only: the dead gradient is buried (hits~TB > runs > RBI,
and RBI has the SMALLEST bias, so the skill-driven-gradient mechanism did
not survive); the refused set is a map of missing inputs; a low-parameter
calibrator is queued unbuilt.

p_win never mutated; calibration rides as p_win_calibrated with
calibration_status provisional. No Bonferroni slot consumed. Counter and
frozen clusters byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-06 18:31:19 -04:00

121 lines
4.7 KiB
JavaScript

'use strict';
/**
* Which stats may serve a calibrated number.
*
* The thing these protect is the meaning of PROVISIONAL: a provisional deploy
* that cannot be taken away is just a deploy.
*/
const { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS } = require('../../src/services/model/calibrationRegistry');
const MAP = [{ lo: 0.5, hi: 0.7, value: 0.55, n: 300 }];
const GOOD = { lodo_pass: true, ci: [-0.0061, -0.0045], map: MAP, certified_bands: [[0.6, 0.8]], date_clusters: 7, at: '2026-08-06' };
describe('deploy needs BOTH gates', () => {
it('deploys when LODO passes and the interval excludes zero', () => {
const r = createRegistry();
expect(r.deploy('total_bases', GOOD).status).toBe(STATUS.PROVISIONAL);
});
it('refuses on a LODO failure however good the interval', () => {
const r = createRegistry();
const out = r.deploy('runs', { ...GOOD, lodo_pass: false });
expect(out.ok).toBe(false);
expect(out.reason).toMatch(/date-driven/);
});
it('refuses when the interval spans zero however clean the LODO', () => {
const r = createRegistry();
const out = r.deploy('hits', { ...GOOD, ci: [-0.01, 0.002] });
expect(out.ok).toBe(false);
expect(out.reason).toMatch(/does not exclude zero/);
});
it('refuses without a map — there is nothing to serve', () => {
const r = createRegistry();
expect(r.deploy('hits', { ...GOOD, map: null }).ok).toBe(false);
});
});
describe('auto-demotion is what makes provisional honest', () => {
it('demotes on the first date where the interval stops excluding zero', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.004, 0.001], date: '2026-08-07' });
expect(out.status).toBe(STATUS.NONE);
expect(out.reason).toBe('ci_no_longer_excludes_zero');
expect(out.breaking_date).toBe('2026-08-07');
expect(r.serves('total_bases', 0.65).serve).toBe(false);
});
it('demotes when the favourite over-prediction flips sign', () => {
// A flip means the correction is now pushing the wrong way.
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.006, -0.004], favourite_bias: -0.03, date: '2026-08-08' });
expect(out.status).toBe(STATUS.NONE);
expect(out.reason).toBe('favourite_bias_flipped');
});
it('logs the demotion with its breaking date', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
r.reverify('total_bases', { ci: [0.001, 0.004], date: '2026-08-09' });
const ev = r.log().find((e) => e.event === 'auto_demoted');
expect(ev).toMatchObject({ stat: 'total_bases', at: '2026-08-09' });
});
it('stays deployed while both conditions hold', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.007, -0.003], favourite_bias: 0.17, date: '2026-08-07' });
expect(out.status).toBe(STATUS.PROVISIONAL);
expect(out.changed).toBe(false);
});
});
describe('the >=40 date-cluster bar is the PROMOTION bar, not the deploy bar', () => {
it('does not block deployment', () => {
const r = createRegistry();
expect(r.deploy('total_bases', { ...GOOD, date_clusters: 7 }).ok).toBe(true);
});
it('promotes out of provisional once it is met', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.006, -0.004], date_clusters: PROMOTION_DATE_CLUSTERS, date: '2026-09-15' });
expect(out.status).toBe(STATUS.PROMOTED);
});
it('does not promote while the interval has stopped holding', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
const out = r.reverify('total_bases', { ci: [-0.001, 0.003], date_clusters: 60, date: '2026-09-15' });
expect(out.status).toBe(STATUS.NONE);
});
});
describe('serving is band-limited', () => {
it('serves inside the certified band and refuses outside it', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
expect(r.serves('total_bases', 0.65).serve).toBe(true);
expect(r.serves('total_bases', 0.65).provisional).toBe(true);
expect(r.serves('total_bases', 0.95).serve).toBe(false);
expect(r.serves('total_bases', 0.95).reason).toMatch(/outside the certified band/);
});
it('an undeployed stat never serves', () => {
const r = createRegistry();
expect(r.serves('hits', 0.6).serve).toBe(false);
expect(r.serves('hits', 0.6).reason).toBe('not deployed');
});
it('a missing p_win serves nothing', () => {
const r = createRegistry();
r.deploy('total_bases', GOOD);
expect(r.serves('total_bases', null).serve).toBe(false);
});
});