builtbykev
74cf1ce974
PHASE 0 — sample-limit truth on record: on 19 dates BOTH stability
instruments are underpowered. LODO power 0.014-0.093 (best 0.337 across
every k tried); deploy CIs rest on 2-4 date clusters, where a
cluster-robust interval has ~1 df. This is the SAMPLE, not a fixable
instrument, and the gate-refinement loop stops here. Runs corrected: its
DATE-DRIVEN label was an artefact of the coin-flip ruler (2 reversals in
3 drops never cleared cutoff 2) -- it is an ordinary no-fittable-map
refusal.
PHASE 1 — the bias is ROBUST, tested model-free and map-free with a
date-block bootstrap. Pooled over-prediction rises monotonically -0.0076
/ +0.0428 / +0.0963 / +0.1589 / +0.2451 across deciles from 0.5 to 1.0,
sign stability 0.9946 over 17 date blocks, and 4 of 4 stats replicate
(bar was 3). Also visible: realized rate PLATEAUS at 0.65-0.68 from p=0.7
upward -- the 0.9+ bucket (0.6624) does no better than the 0.8-0.9 bucket
(0.6841). The model has no high-confidence reads, only high-confidence
numbers.
PHASE 3 — Platt, two parameters over the whole curve, shrunk toward
identity by fit-date count. Validated as a NEW estimator vs RAW with
date-block CIs:
hits a=0.406 shrink 0.565 0.2626 -> 0.2540 CI [-0.0112,-0.0069] DEPLOY
total_bases a=0.472 shrink 0.333 0.2490 -> 0.2429 CI [-0.0062,-0.0059] DEPLOY
rbi a=0.775 shrink 0.231 0.2011 -> 0.2007 CI [-0.0007, 0] REFUSE
runs a=-0.032 REFUSE
A GUARD THE FIRST RUN NEEDED: runs fitted a = -0.032. A non-positive
slope inverts the forecast rather than flattening it, and near zero the
curve collapses to a constant predicting the base rate for everything --
which LOWERS Brier while destroying all resolution. It would have scored
as a win while making the product worthless. MIN_SLOPE now refuses it by
name, with a test.
STATED PLAINLY: on the identical held-out rows isotonic BEAT the
low-param on hits (+0.0028) and rbi (+0.0042) and tied on TB. The swap is
a CAPACITY JUDGEMENT, not a measurement -- the window spans 2-4 date
blocks and that is exactly what a flexible map produces when it captures
structure shared by fit and eval. Labelled as a judgement.
PHASE 4 — hits and total_bases serve the correction, basis
direction_robust_magnitude_provisional (direction bootstrap-robust,
magnitude thin-sample and shrunk). rbi is WITHDRAWN to raw -- it was
deployed on isotonic at ced4042 and the low-param does not beat raw.
runs stays raw. Auto-demotion still armed.
PHASE 5 — the standing finding, stated hard: across 18 archetype slots on
three stats, calibrated p_win separates within archetype NO BETTER than
raw. Every slot is one band indistinguishable from its base rate, zero
show lift. Per-archetype separation is not coming from calibration; it
comes from proven factors or it does not exist. Five orders of
calibration have delivered what they can -- honest numbers on two stats --
and nothing on the question the grade product turns on.
p_win never mutated; no Bonferroni slot; the robust-claim test ran before
any calibrator was built and could have ended the session at Phase 2.
Counter and frozen clusters verified file-by-file, including calibration.js
and calibrationService.js, both untouched and simply off the serving path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
46 lines
1.8 KiB
JavaScript
46 lines
1.8 KiB
JavaScript
'use strict';
|
|
|
|
/**
|
|
* Which stats serve a calibrated number, and on what basis.
|
|
*
|
|
* The claim that survived on this sample is the bias DIRECTION, tested model-free
|
|
* and map-free. What did not survive is any certification of a specific map's
|
|
* stability. These lock that distinction into the serving path.
|
|
*/
|
|
|
|
const snapshotService = require('../../src/services/snapshotService');
|
|
const lp = require('../../src/services/model/lowParamCalibrator');
|
|
|
|
describe('the deploy set rides a low-parameter correction, not isotonic', () => {
|
|
it('serves the two stats whose correction beat RAW out-of-sample', () => {
|
|
expect(snapshotService.CALIBRATION_DEPLOYED).toEqual(['hits', 'total_bases']);
|
|
});
|
|
|
|
it('WITHDRAWS rbi — the low-parameter fit does not beat raw', () => {
|
|
// rbi was deployed at ced4042 on isotonic. Its CI vs raw is [-0.0007, 0],
|
|
// which touches zero, so it serves raw again.
|
|
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain('rbi');
|
|
});
|
|
|
|
it('does NOT serve runs — its fitted slope would invert the forecast', () => {
|
|
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain('runs');
|
|
});
|
|
|
|
it('labels the basis honestly: direction robust, magnitude provisional', () => {
|
|
for (const stat of snapshotService.CALIBRATION_DEPLOYED) {
|
|
expect(snapshotService.CALIBRATION_BASIS[stat]).toBe('direction_robust_magnitude_provisional');
|
|
}
|
|
});
|
|
|
|
it('the served correction cannot encode a single odd day', () => {
|
|
// Two parameters over the whole curve is the entire reason for the swap.
|
|
expect(typeof lp.fitPlatt).toBe('function');
|
|
expect(lp.MIN_SLOPE).toBeGreaterThan(0);
|
|
});
|
|
|
|
it('is frozen, so a stat cannot be added at runtime', () => {
|
|
expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true);
|
|
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('rbi'); }).toThrow();
|
|
});
|
|
});
|