builtbykev
6ae11f1193
PHASE 0 — I applied factorGate's >=40 date-cluster floor to a calibration layer without challenging the binding. That floor is a cluster-robust interval bar for a CAUSAL claim. Calibration makes no causal claim, has a bounded failure mode (it can only over- or under-shrink) and consumes no Bonferroni slot. Its real risk is that the correction is DATE-DRIVEN, and leave-one-date-out tests that directly -- a STRICTER bar, since a cluster count cannot detect a single day carrying the effect. The >=40 floor is retained, correctly scoped as the PROMOTION bar. PHASE 1 — both guards codified, 11 tests, green before Phase 2. Demonstrated on live data: raw population violated=true, mean_p 0.4962, both_sides_share 0.9763; after dedup violated=false, mean_p 0.6694. The null guard's test demonstrates the trap explicitly, since (null-1)**2 is 1 and (null-0)**2 is 0 so a Brier over nulls equals the win rate. PHASE 2 — LODO: hits n=1140 dates=17 2 reversals (07-22 n=20, 07-26 n=25) FAIL total_bases n=1050 dates=7 0 reversals, 0 sign flips PASS rbi n= 630 dates=5 1 reversal (08-01 n=99) FAIL runs n= 597 dates=5 2 reversals (08-01 n=86, 08-05 n=244) FAIL Threshold sensitivity reported because the verdict moves: total_bases passes at every held-size threshold, runs fails at every one, and hits fails ONLY when 20/25-row dates are admitted. I fixed MIN_HELD_ROWS=20 before seeing which stats passed and did not move it afterwards to preserve a deploy. Honest caveat: a per-date Brier delta on 20 rows has a standard error several times the effect, so the instrument is underpowered per-drop -- an argument for pre-registering a higher threshold, which is a Roundtable call, not one to make while holding the results. PHASE 3 — total_bases DEPLOY-PROVISIONAL, band [0.6-0.8]. hits, rbi and runs REFUSE. HITS WAS BEING SERVED CALIBRATED AND IS NOT ANY MORE. snapshotService hardcoded it since S91; it fails LODO, so it is out. A stat that cannot survive dropping one day was never calibrated, it was fitted to that day. The consequence is real -- hits props become unstackable for chain.chainAcross -- and it errs toward withdrawing a claim rather than preserving one on a fragile verdict. Deployment is now driven by a frozen, tested CALIBRATION_DEPLOYED set, not a hardcoded stat name. PHASE 4 — calibrationRegistry, 14 tests. Deploy needs BOTH gates, neither waivable. reverify auto-demotes on the first breach (CI stops excluding zero, or the favourite bias flips sign) and logs the breaking date. Promotion needs the original >=40 bar. A provisional deploy that cannot be taken away is just a deploy. PHASE 5 — TB bands rebuilt on calibrated values, 625 eval rows. The two-bar rule still bites: calibrated YES, proven NO, so they stay a base-rate read, now honestly numbered. Every archetype still collapses to one band -- calibrated p_win separates within archetype no better than raw. PHASE 6 logged only: the dead gradient is buried (hits~TB > runs > RBI, and RBI has the SMALLEST bias, so the skill-driven-gradient mechanism did not survive); the refused set is a map of missing inputs; a low-parameter calibrator is queued unbuilt. p_win never mutated; calibration rides as p_win_calibrated with calibration_status provisional. No Bonferroni slot consumed. Counter and frozen clusters byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
31 lines
1.2 KiB
JavaScript
31 lines
1.2 KiB
JavaScript
'use strict';
|
|
|
|
/**
|
|
* Which stats serve a calibrated number in the live pipeline.
|
|
*
|
|
* The rule this locks: a stat that cannot survive dropping a single settled date
|
|
* was never calibrated — it was fitted to that date. hits WAS served calibrated
|
|
* and is not any more, which is the honest consequence of measuring it.
|
|
*/
|
|
|
|
const snapshotService = require('../../src/services/snapshotService');
|
|
|
|
describe('the deployed set is LODO-gated', () => {
|
|
it('serves total_bases — it passed at every held-size threshold', () => {
|
|
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('total_bases');
|
|
});
|
|
|
|
it('does NOT serve hits, rbi or runs — each fails LODO', () => {
|
|
// hits reverses when 2026-07-22 or 2026-07-26 is dropped; rbi on 2026-08-01;
|
|
// runs on 2026-08-01 and 2026-08-05.
|
|
for (const stat of ['hits', 'rbi', 'runs']) {
|
|
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain(stat);
|
|
}
|
|
});
|
|
|
|
it('is frozen, so a stat cannot be added at runtime without a code change', () => {
|
|
expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true);
|
|
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('hits'); }).toThrow();
|
|
});
|
|
});
|