Robust bias established; low-parameter correction replaces isotonic
PHASE 0 — sample-limit truth on record: on 19 dates BOTH stability
instruments are underpowered. LODO power 0.014-0.093 (best 0.337 across
every k tried); deploy CIs rest on 2-4 date clusters, where a
cluster-robust interval has ~1 df. This is the SAMPLE, not a fixable
instrument, and the gate-refinement loop stops here. Runs corrected: its
DATE-DRIVEN label was an artefact of the coin-flip ruler (2 reversals in
3 drops never cleared cutoff 2) -- it is an ordinary no-fittable-map
refusal.
PHASE 1 — the bias is ROBUST, tested model-free and map-free with a
date-block bootstrap. Pooled over-prediction rises monotonically -0.0076
/ +0.0428 / +0.0963 / +0.1589 / +0.2451 across deciles from 0.5 to 1.0,
sign stability 0.9946 over 17 date blocks, and 4 of 4 stats replicate
(bar was 3). Also visible: realized rate PLATEAUS at 0.65-0.68 from p=0.7
upward -- the 0.9+ bucket (0.6624) does no better than the 0.8-0.9 bucket
(0.6841). The model has no high-confidence reads, only high-confidence
numbers.
PHASE 3 — Platt, two parameters over the whole curve, shrunk toward
identity by fit-date count. Validated as a NEW estimator vs RAW with
date-block CIs:
hits a=0.406 shrink 0.565 0.2626 -> 0.2540 CI [-0.0112,-0.0069] DEPLOY
total_bases a=0.472 shrink 0.333 0.2490 -> 0.2429 CI [-0.0062,-0.0059] DEPLOY
rbi a=0.775 shrink 0.231 0.2011 -> 0.2007 CI [-0.0007, 0] REFUSE
runs a=-0.032 REFUSE
A GUARD THE FIRST RUN NEEDED: runs fitted a = -0.032. A non-positive
slope inverts the forecast rather than flattening it, and near zero the
curve collapses to a constant predicting the base rate for everything --
which LOWERS Brier while destroying all resolution. It would have scored
as a win while making the product worthless. MIN_SLOPE now refuses it by
name, with a test.
STATED PLAINLY: on the identical held-out rows isotonic BEAT the
low-param on hits (+0.0028) and rbi (+0.0042) and tied on TB. The swap is
a CAPACITY JUDGEMENT, not a measurement -- the window spans 2-4 date
blocks and that is exactly what a flexible map produces when it captures
structure shared by fit and eval. Labelled as a judgement.
PHASE 4 — hits and total_bases serve the correction, basis
direction_robust_magnitude_provisional (direction bootstrap-robust,
magnitude thin-sample and shrunk). rbi is WITHDRAWN to raw -- it was
deployed on isotonic at ced4042 and the low-param does not beat raw.
runs stays raw. Auto-demotion still armed.
PHASE 5 — the standing finding, stated hard: across 18 archetype slots on
three stats, calibrated p_win separates within archetype NO BETTER than
raw. Every slot is one band indistinguishable from its base rate, zero
show lift. Per-archetype separation is not coming from calibration; it
comes from proven factors or it does not exist. Five orders of
calibration have delivered what they can -- honest numbers on two stats --
and nothing on the question the grade product turns on.
p_win never mutated; no Bonferroni slot; the robust-claim test ran before
any calibrator was built and could have ended the session at Phase 2.
Counter and frozen clusters verified file-by-file, including calibration.js
and calibrationService.js, both untouched and simply off the serving path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -3,43 +3,43 @@
|
||||
/**
|
||||
* Which stats serve a calibrated number, and on what basis.
|
||||
*
|
||||
* The thing these lock is that "could not test" never reads as "passed". LODO
|
||||
* has 1.4%-9.3% power at this date count, so no deployed stat claims stability
|
||||
* from it — each rides a thin date-clustered interval and auto-demotion alone.
|
||||
* The claim that survived on this sample is the bias DIRECTION, tested model-free
|
||||
* and map-free. What did not survive is any certification of a specific map's
|
||||
* stability. These lock that distinction into the serving path.
|
||||
*/
|
||||
|
||||
const snapshotService = require('../../src/services/snapshotService');
|
||||
const { LODO_TEST, LODO_POWER_FLOOR } = require('../../src/services/model/calibrationRegistry');
|
||||
const lp = require('../../src/services/model/lowParamCalibrator');
|
||||
|
||||
describe('the deploy set rests on the CI, not on a passed LODO', () => {
|
||||
it('serves the three stats with a point-in-time interval excluding zero', () => {
|
||||
expect(snapshotService.CALIBRATION_DEPLOYED).toEqual(['hits', 'total_bases', 'rbi']);
|
||||
describe('the deploy set rides a low-parameter correction, not isotonic', () => {
|
||||
it('serves the two stats whose correction beat RAW out-of-sample', () => {
|
||||
expect(snapshotService.CALIBRATION_DEPLOYED).toEqual(['hits', 'total_bases']);
|
||||
});
|
||||
|
||||
it('does NOT serve runs — no fittable map, so no CI to stand on', () => {
|
||||
it('WITHDRAWS rbi — the low-parameter fit does not beat raw', () => {
|
||||
// rbi was deployed at ced4042 on isotonic. Its CI vs raw is [-0.0007, 0],
|
||||
// which touches zero, so it serves raw again.
|
||||
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain('rbi');
|
||||
});
|
||||
|
||||
it('does NOT serve runs — its fitted slope would invert the forecast', () => {
|
||||
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain('runs');
|
||||
});
|
||||
|
||||
it('labels every deployed stat as LODO-untestable rather than LODO-stable', () => {
|
||||
it('labels the basis honestly: direction robust, magnitude provisional', () => {
|
||||
for (const stat of snapshotService.CALIBRATION_DEPLOYED) {
|
||||
expect(snapshotService.CALIBRATION_BASIS[stat]).toBe('ci_only_lodo_untestable');
|
||||
expect(snapshotService.CALIBRATION_BASIS[stat]).toBe('direction_robust_magnitude_provisional');
|
||||
}
|
||||
});
|
||||
|
||||
it('no stat may claim LODO stability, because the test cannot fail', () => {
|
||||
for (const t of Object.values(LODO_TEST)) expect(t.power).toBeLessThan(LODO_POWER_FLOOR);
|
||||
});
|
||||
|
||||
it('rbi and runs were FALSE FAILS under the old zero-reversal rule', () => {
|
||||
// Re-read under the binomial cutoff: 1 and 2 reversals, cutoff 2 for both.
|
||||
expect(LODO_TEST.rbi.cutoff).toBe(2);
|
||||
expect(LODO_TEST.runs.cutoff).toBe(2);
|
||||
// rbi returns to the deploy set on CI support; runs still has none.
|
||||
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('rbi');
|
||||
it('the served correction cannot encode a single odd day', () => {
|
||||
// Two parameters over the whole curve is the entire reason for the swap.
|
||||
expect(typeof lp.fitPlatt).toBe('function');
|
||||
expect(lp.MIN_SLOPE).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('is frozen, so a stat cannot be added at runtime', () => {
|
||||
expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true);
|
||||
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('runs'); }).toThrow();
|
||||
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('rbi'); }).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* The two-parameter correction that replaces isotonic on a thin sample.
|
||||
*
|
||||
* What these protect: that it CANNOT chase day-structure (two parameters over
|
||||
* the whole curve), and that a thin fit is applied at reduced strength rather
|
||||
* than at face value.
|
||||
*/
|
||||
|
||||
const lp = require('../../src/services/model/lowParamCalibrator');
|
||||
|
||||
/** An over-confident forecaster: predicts p, actually hits closer to the mean. */
|
||||
function overConfident(n, dates = 10, seed = 3) {
|
||||
let s = seed;
|
||||
const rnd = () => (s = (s * 1103515245 + 12345) % 2147483648) / 2147483648;
|
||||
const rows = [];
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
const p = 0.5 + rnd() * 0.45;
|
||||
const truth = 0.5 + (p - 0.5) * 0.4; // real skill is 40% of claimed
|
||||
rows.push({ date: `d${i % dates}`, p, won: rnd() < truth ? 1 : 0 });
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
describe('it fits the favourite-longshot shape', () => {
|
||||
it('FLATTENS an over-confident forecaster (a < 1)', () => {
|
||||
const m = lp.fitPlatt(overConfident(2000));
|
||||
expect(m).not.toBeNull();
|
||||
expect(m.flattens).toBe(true);
|
||||
expect(m.a).toBeLessThan(1);
|
||||
});
|
||||
|
||||
it('pulls high predictions down and leaves the middle nearly alone', () => {
|
||||
const m = lp.fitPlatt(overConfident(2000));
|
||||
const hi = lp.applyPlatt(m, 0.92);
|
||||
const mid = lp.applyPlatt(m, 0.55);
|
||||
expect(hi).toBeLessThan(0.92);
|
||||
expect(Math.abs(mid - 0.55)).toBeLessThan(Math.abs(hi - 0.92));
|
||||
});
|
||||
|
||||
it('stays monotone — ordering is never disturbed', () => {
|
||||
const m = lp.fitPlatt(overConfident(2000));
|
||||
let prev = -1;
|
||||
for (let p = 0.05; p <= 0.95; p += 0.05) {
|
||||
const v = lp.applyPlatt(m, p);
|
||||
expect(v).toBeGreaterThan(prev);
|
||||
prev = v;
|
||||
}
|
||||
});
|
||||
|
||||
it('leaves an already-honest forecaster essentially alone', () => {
|
||||
let s = 11;
|
||||
const rnd = () => (s = (s * 1103515245 + 12345) % 2147483648) / 2147483648;
|
||||
const rows = [];
|
||||
for (let i = 0; i < 2000; i += 1) {
|
||||
const p = 0.3 + rnd() * 0.6;
|
||||
rows.push({ date: `d${i % 12}`, p, won: rnd() < p ? 1 : 0 });
|
||||
}
|
||||
const m = lp.fitPlatt(rows);
|
||||
expect(Math.abs(lp.applyPlatt(m, 0.8) - 0.8)).toBeLessThan(0.06);
|
||||
});
|
||||
});
|
||||
|
||||
describe('shrinkage — a thin fit is applied at reduced strength', () => {
|
||||
it('scales with the number of fit DATES, not rows', () => {
|
||||
const few = lp.fitPlatt(overConfident(2000, 5));
|
||||
const many = lp.fitPlatt(overConfident(2000, 40));
|
||||
expect(few.fit_rows).toBe(many.fit_rows); // same rows
|
||||
expect(few.shrinkage).toBeLessThan(many.shrinkage); // different dates
|
||||
expect(few.shrinkage).toBeCloseTo(5 / 15, 3);
|
||||
expect(many.shrinkage).toBeCloseTo(40 / 50, 3);
|
||||
});
|
||||
|
||||
it('a 5-date fit corrects less than a 40-date fit on the same input', () => {
|
||||
const few = lp.fitPlatt(overConfident(2000, 5));
|
||||
const many = lp.fitPlatt(overConfident(2000, 40));
|
||||
// Both flatten; the thin one is held closer to the raw number.
|
||||
expect(Math.abs(lp.applyPlatt(few, 0.92) - 0.92))
|
||||
.toBeLessThan(Math.abs(lp.applyPlatt(many, 0.92) - 0.92));
|
||||
});
|
||||
});
|
||||
|
||||
describe('honesty', () => {
|
||||
it('refuses to fit below the row floor', () => {
|
||||
expect(lp.fitPlatt(overConfident(40))).toBeNull();
|
||||
expect(lp.fitPlatt([])).toBeNull();
|
||||
expect(lp.fitPlatt(null)).toBeNull();
|
||||
});
|
||||
|
||||
it('an unreadable input returns null, never an uncorrected number', () => {
|
||||
const m = lp.fitPlatt(overConfident(2000));
|
||||
expect(lp.applyPlatt(m, null)).toBeNull();
|
||||
expect(lp.applyPlatt(null, 0.7)).toBeNull();
|
||||
});
|
||||
|
||||
it('has exactly two parameters — it CANNOT encode a single odd day', () => {
|
||||
// This is the whole reason it replaces isotonic here.
|
||||
const m = lp.fitPlatt(overConfident(2000));
|
||||
const shape = Object.keys(m).filter((k) => k === 'a' || k === 'b');
|
||||
expect(shape.sort()).toEqual(['a', 'b']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the slope must CORRECT, not abandon the forecast', () => {
|
||||
/** A forecaster whose p_win carries no information at all. */
|
||||
function uninformative(n, seed = 7) {
|
||||
let s = seed;
|
||||
const rnd = () => (s = (s * 1103515245 + 12345) % 2147483648) / 2147483648;
|
||||
const rows = [];
|
||||
for (let i = 0; i < n; i += 1) rows.push({ date: `d${i % 8}`, p: 0.4 + rnd() * 0.5, won: rnd() < 0.55 ? 1 : 0 });
|
||||
return rows;
|
||||
}
|
||||
|
||||
it('REFUSES a fit whose slope collapses to a constant', () => {
|
||||
// Shrinking a miscalibrated forecaster toward its base rate always lowers
|
||||
// Brier, so this would score as a win while destroying all resolution.
|
||||
const m = lp.fitPlatt(uninformative(3000));
|
||||
expect(m.refused).toBe(true);
|
||||
expect(m.reason).toMatch(/collapses to a constant|invert/);
|
||||
});
|
||||
|
||||
it('a refused fit produces no calibrated number at all', () => {
|
||||
const m = lp.fitPlatt(uninformative(3000));
|
||||
expect(lp.applyPlatt(m, 0.8)).toBeNull();
|
||||
});
|
||||
|
||||
it('an INVERTING slope is refused by name', () => {
|
||||
// Real case: runs fitted a = -0.032, which would reverse every ordering.
|
||||
let s = 5;
|
||||
const rnd = () => (s = (s * 1103515245 + 12345) % 2147483648) / 2147483648;
|
||||
const rows = [];
|
||||
for (let i = 0; i < 3000; i += 1) {
|
||||
const p = 0.4 + rnd() * 0.5;
|
||||
rows.push({ date: `d${i % 8}`, p, won: rnd() < (0.9 - p) ? 1 : 0 }); // backwards
|
||||
}
|
||||
const m = lp.fitPlatt(rows);
|
||||
expect(m.refused).toBe(true);
|
||||
expect(m.a).toBeLessThanOrEqual(lp.MIN_SLOPE);
|
||||
});
|
||||
|
||||
it('still accepts a genuine flattening', () => {
|
||||
const m = lp.fitPlatt(overConfident(2000));
|
||||
expect(m.refused).toBeUndefined();
|
||||
expect(m.a).toBeGreaterThan(lp.MIN_SLOPE);
|
||||
expect(m.a).toBeLessThan(1);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user