Detection becomes repair: the curve is fitted on one forecaster, frozen, and named
The last release detected the violation and then served the certified state anyway. A validator that changes nothing is decoration, so `servable:false` is now load-bearing: an artifact that fails its policy returns ARTIFACT_POLICY_BLOCKED with no number, and every probability-derived claim goes with it. The gate sits inside the resolution, not beside the flag that turns the shadow on, so no environment variable can reach past it — a test asserts `resolve` never reads process.env at all. Shadow and live consume the SAME decision, differing only in which promotion stage they demand. Era mismatch still resolves to VERSION_MISMATCH rather than the new state. "This artifact belongs to a different forecaster" is more precise than "policy blocked", and the existing state already says it exactly. THE REPAIR. `currentEraSource` filters on model_version in the QUERY, taking the era from config/modelVersion so the query, the artifact and the validator all read one identity. Measured on the actual fitted set, not a second count: 6,069 current-era rows, 0 wrong-era. The procedure was then certified on current-era rows ONLY — four walk-forward folds, training strictly before each evaluation block, 0 future rows in train on every fold. All four improve; pooled n=3,108 gives Brier 0.24701 -> 0.24323, delta -0.00378, CI [-0.00619,-0.00147] excluding zero; ECE falls in every fold. Mapping spread inside support is 0.001-0.018. The prior mixed-era certification did not substitute for this. Policy B selected. A (era-filtered 65/35) and B (all current-era) are statistically indistinguishable, A-B = +0.0001 CI [-0.00029,+0.00048], but B has the better ECE (0.0064 vs 0.0109) and the holdout existed to certify the PROCEDURE — it is not permanently withheld from the artifact that ships. withheld_from_fit is 0. FROZEN. `mlb-hits-isotonic@2026-09-03`: 6,069 rows, training_cutoff 2026-09-01 (distinct from fit_as_of 2026-09-03 — the newest observation admitted is not the eligibility bound), 12 knots, source_digest 25919c16…, knot_digest 5ae940ea…, served_curve_digest c24a9dc5…, 8 curve steps, 924 bytes, committed as JSON. The runtime no longer fits. It loads. A test greps the service for fitIsotonic, fromLedger and loadRows and requires all three absent, because the old behaviour meant a user's number could move with no version, no review and no rollback, and a past Read could not be reconstructed because its curve no longer existed. New settled outcomes are forward evidence now; they cannot touch this curve. Independent reconstruction from the declared training contract alone — fresh read, fresh digest, fresh fit — reproduces every digest and the curve byte for byte. Calling the builder twice would only have proven the builder deterministic. Promotion is a frozen source constant. A snapshot cannot promote, a settlement cannot promote, a successful fit cannot promote, and dropping a file into the artifacts directory promotes nothing. Stage is APPROVED_FOR_SHADOW; live is explicitly false. Two coverage holes found by their own teeth. The promotion guard could be deleted with every test still green, because the promoted file naturally agrees with itself — extracted as `acceptFile` and tested on the case `load()` cannot reach. And `validate(null)` returned no `servable` field at all, which is falsy at a call site and so would have read as correct while asserting nothing. Shadow OFF. Live OFF. CALIBRATION_DEPLOYED []. No frontend change. Suite 404/404, 5,634 passed, 4 skipped. Teeth 26/26 + 10/10 + 23/23. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
@@ -0,0 +1,174 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* A VALIDATOR IS NOT A REPAIR. `servable:false` must mean something mechanically.
|
||||
*
|
||||
* The previous release detected the violation correctly and then served the
|
||||
* certified state anyway. These tests exist so that cannot recur.
|
||||
*/
|
||||
const pc = require('../../src/services/model/probabilityContract');
|
||||
const svc = require('../../src/services/model/probabilityContractService');
|
||||
const registry = require('../../src/services/model/artifactRegistry');
|
||||
const fp = require('../../src/services/model/fitPolicy');
|
||||
|
||||
const ERA = 'engine1@2026-08-07-fullwindow';
|
||||
const good = registry.load('mlb', 'hits');
|
||||
const read = (p) => ({ sport: 'mlb', stat: 'hits', model_version: ERA, p_win: p });
|
||||
const est = () => 0.6;
|
||||
|
||||
describe('an unservable artifact can never emit a certified state', () => {
|
||||
// Era and estimator mismatch resolve to VERSION_MISMATCH rather than the new
|
||||
// state — that is DELIBERATE. "This artifact belongs to a different
|
||||
// forecaster" is a more precise thing to say than "policy blocked", and the
|
||||
// existing state already says it exactly. The invariant asserted for all six
|
||||
// is the one that matters: never certified, never a number.
|
||||
const violations = {
|
||||
ERA_NOT_RESTRICTED: [{ era_audit: { era_counts: { [ERA]: 10, 'engine1@2026-07-20': 5 }, wrong_era_rows: 5 } }, 'ARTIFACT_POLICY_BLOCKED'],
|
||||
MODEL_VERSION_MISMATCH: [{ model_version: 'engine1@2026-07-20' }, 'VERSION_MISMATCH'],
|
||||
ESTIMATOR_MISMATCH: [{ estimator_type: 'low_param' }, 'VERSION_MISMATCH'],
|
||||
MISSING_IDENTITY: [{ knot_digest: null }, 'ARTIFACT_POLICY_BLOCKED'],
|
||||
THIN_FIT: [{ fit_n: 3 }, 'ARTIFACT_POLICY_BLOCKED'],
|
||||
SUPPORT_WIDENING: [{ certified_bands: [[0.50, 0.99]] }, 'ARTIFACT_POLICY_BLOCKED'],
|
||||
};
|
||||
|
||||
for (const [name, [over, expected]] of Object.entries(violations)) {
|
||||
it(`${name} -> ${expected}, never CERTIFIED_CALIBRATED`, () => {
|
||||
const base = { ...good, ...over };
|
||||
const check = fp.validate(base, { era_counts: base.era_audit ? base.era_audit.era_counts : null });
|
||||
expect(check.servable).toBe(false);
|
||||
const artifact = { ...base, servable: check.servable, fit_policy_violations: check.violations };
|
||||
const r = pc.resolve(read(0.65), { estimate: est, artifact });
|
||||
expect(r.probability_state).toBe(pc.STATE[expected]);
|
||||
expect(r.probability_state).not.toBe(pc.STATE.CERTIFIED_CALIBRATED);
|
||||
expect(r.served_probability).toBeNull();
|
||||
expect(r.raw_model_probability).toBe(0.65); // raw is still evidence
|
||||
expect(pc.derivedClaims(r, -115).available).toBe(false);
|
||||
});
|
||||
}
|
||||
|
||||
it('and every probability-derived claim is withheld with it', () => {
|
||||
const artifact = { ...good, servable: false, fit_policy_violations: ['ERA_NOT_RESTRICTED'] };
|
||||
const d = pc.derivedClaims(pc.resolve(read(0.65), { estimate: est, artifact }), -115);
|
||||
expect(d.available).toBe(false);
|
||||
expect(d.ev_pct).toBeNull();
|
||||
expect(d.kelly).toBeNull();
|
||||
expect(d.value).toBeNull();
|
||||
});
|
||||
|
||||
it('the blocked state names the violation rather than shrugging', () => {
|
||||
const artifact = { ...good, servable: false, fit_policy_violations: ['ERA_NOT_RESTRICTED'] };
|
||||
expect(pc.resolve(read(0.65), { estimate: est, artifact }).reason).toContain('ERA_NOT_RESTRICTED');
|
||||
});
|
||||
});
|
||||
|
||||
describe('no environment variable can override the gate', () => {
|
||||
const artifact = { ...good, servable: false, fit_policy_violations: ['ERA_NOT_RESTRICTED'] };
|
||||
|
||||
it('shadow ON does not make an unservable artifact certify', () => {
|
||||
expect(pc.shadowState({ PROBABILITY_CONTRACT_SHADOW: '1' }).shadow).toBe('ON');
|
||||
// the gate lives in the resolution, not beside the flag
|
||||
const r = pc.resolve(read(0.65), { estimate: est, artifact });
|
||||
expect(r.probability_state).toBe(pc.STATE.ARTIFACT_POLICY_BLOCKED);
|
||||
});
|
||||
|
||||
it('resolve takes no flag, so there is nothing for a flag to reach', () => {
|
||||
const src = require('fs').readFileSync(
|
||||
require('path').join(__dirname, '../../src/services/model/probabilityContract.js'), 'utf8');
|
||||
const body = src.slice(src.indexOf('function resolve'), src.indexOf('const isCertified'));
|
||||
expect(body).not.toContain('process.env');
|
||||
expect(body).not.toContain('PROBABILITY_CONTRACT_SHADOW');
|
||||
});
|
||||
});
|
||||
|
||||
describe('one validity decision for shadow AND live', () => {
|
||||
it('a shadow-approved artifact is refused for live use', () => {
|
||||
expect(good.approved_for_shadow).toBe(true);
|
||||
expect(good.approved_for_live).toBe(false);
|
||||
const live = pc.resolve(read(0.65), { estimate: est, artifact: good, usage: 'live' });
|
||||
expect(live.probability_state).toBe(pc.STATE.ARTIFACT_POLICY_BLOCKED);
|
||||
expect(live.reason).toContain('not promoted for live');
|
||||
const shadow = pc.resolve(read(0.65), { estimate: est, artifact: good, usage: 'shadow' });
|
||||
expect(shadow.probability_state).toBe(pc.STATE.CERTIFIED_CALIBRATED);
|
||||
});
|
||||
|
||||
it('live consumes the SAME servable decision, not a parallel one', () => {
|
||||
const bad = { ...good, servable: false, approved_for_live: true, fit_policy_violations: ['X'] };
|
||||
expect(pc.resolve(read(0.65), { estimate: est, artifact: bad, usage: 'live' }).probability_state)
|
||||
.toBe(pc.STATE.ARTIFACT_POLICY_BLOCKED);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the active curve does not move', () => {
|
||||
it('the runtime holds no fitter — nothing to refit on a snapshot', () => {
|
||||
const src = require('fs').readFileSync(
|
||||
require('path').join(__dirname, '../../src/services/model/probabilityContractService.js'), 'utf8');
|
||||
const code = src.replace(/\/\*[\s\S]*?\*\//g, '').replace(/^\s*\/\/.*$/gm, '');
|
||||
expect(code).not.toContain('fitIsotonic');
|
||||
expect(code).not.toContain('fromLedger');
|
||||
expect(code).not.toContain('loadRows');
|
||||
});
|
||||
|
||||
it('repeated builds return the same artifact object and the same numbers', async () => {
|
||||
const a = await svc.build(null, { sport: 'mlb', stat: 'hits' });
|
||||
const b = await svc.build(null, { sport: 'mlb', stat: 'hits' });
|
||||
expect(a.artifact).toBe(b.artifact); // cached, identical identity
|
||||
expect(a.artifact.source_digest).toBe(b.artifact.source_digest);
|
||||
expect(a.resolve(read(0.72)).served_probability).toBe(b.resolve(read(0.72)).served_probability);
|
||||
});
|
||||
|
||||
it('a new settled outcome cannot alter the curve — it is a committed file', () => {
|
||||
const before = registry.applyCurve(good, 0.65);
|
||||
registry.__resetCache();
|
||||
const after = registry.applyCurve(registry.load('mlb', 'hits'), 0.65);
|
||||
expect(after).toBe(before);
|
||||
});
|
||||
});
|
||||
|
||||
describe('promotion is a deliberate act', () => {
|
||||
it('the promotion table is a frozen source constant, not runtime state', () => {
|
||||
expect(Object.isFrozen(registry.PROMOTED)).toBe(true);
|
||||
expect(Object.isFrozen(registry.PROMOTED['mlb:hits'])).toBe(true);
|
||||
// strict mode makes the write throw rather than fail silently — either way
|
||||
// the table is unchanged, which is the property under test
|
||||
expect(() => { registry.PROMOTED['mlb:rbi'] = { artifact_id: 'x', stage: 'APPROVED_FOR_LIVE' }; }).toThrow();
|
||||
expect(registry.PROMOTED['mlb:rbi']).toBeUndefined();
|
||||
});
|
||||
|
||||
it('an artifact file that exists but is not named is NOT loaded', () => {
|
||||
// load() resolves the file FROM the promotion table, so an unnamed artifact
|
||||
// sitting in the directory is inert.
|
||||
expect(registry.load('mlb', 'rbi')).toBeNull();
|
||||
});
|
||||
|
||||
it('the loaded artifact id must equal the promoted id', () => {
|
||||
expect(registry.load('mlb', 'hits').artifact_id).toBe(registry.PROMOTED['mlb:hits'].artifact_id);
|
||||
});
|
||||
|
||||
it('a file declaring a DIFFERENT id is refused — dropping one in promotes nothing', () => {
|
||||
const promoted = registry.PROMOTED['mlb:hits'];
|
||||
expect(registry.acceptFile({ artifact_id: promoted.artifact_id }, promoted)).toBe(true);
|
||||
// the case load() can never reach on the happy path, and the reason the
|
||||
// guard existed unprotected until a teeth injection came back green
|
||||
expect(registry.acceptFile({ artifact_id: 'mlb-hits-isotonic@2099-01-01' }, promoted)).toBe(false);
|
||||
expect(registry.acceptFile({ artifact_id: null }, promoted)).toBe(false);
|
||||
expect(registry.acceptFile({}, promoted)).toBe(false);
|
||||
expect(registry.acceptFile(null, promoted)).toBe(false);
|
||||
expect(registry.acceptFile({ artifact_id: 'x' }, null)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('model-era transition law', () => {
|
||||
it('a NEW model era does not inherit this artifact', () => {
|
||||
const r = pc.resolve({ sport: 'mlb', stat: 'hits', model_version: 'engine1@2027-01-01', p_win: 0.65 },
|
||||
{ estimate: est, artifact: good });
|
||||
expect(r.probability_state).toBe(pc.STATE.VERSION_MISMATCH);
|
||||
expect(r.served_probability).toBeNull();
|
||||
});
|
||||
|
||||
it('and an artifact relabelled to a new era fails its own policy', () => {
|
||||
const relabelled = { ...good, model_version: 'engine1@2027-01-01' };
|
||||
const check = fp.validate(relabelled, { era_counts: good.era_audit.era_counts });
|
||||
expect(check.valid).toBe(false);
|
||||
expect(check.servable).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,84 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* THE INVARIANT: artifact.model_version = X => distinct(training.model_version) = {X}.
|
||||
*/
|
||||
const src = require('../../src/services/model/currentEraSource');
|
||||
|
||||
const ERA = 'engine1@2026-08-07-fullwindow';
|
||||
const OLD = 'engine1@2026-07-20';
|
||||
|
||||
/** A fake that HONOURS its filters — a pass-through would prove nothing. */
|
||||
function client(rows) {
|
||||
const f = {};
|
||||
const q = {
|
||||
select: () => q,
|
||||
eq: (c, v) => { f[c] = v; return q; },
|
||||
is: () => q,
|
||||
in: (c, v) => { f[`in_${c}`] = v; return q; },
|
||||
not: () => q,
|
||||
lt: (c, v) => { f[`lt_${c}`] = v; return q; },
|
||||
order: () => q,
|
||||
range: async () => {
|
||||
let out = rows;
|
||||
for (const [k, v] of Object.entries(f)) {
|
||||
if (k.startsWith('in_') || k.startsWith('lt_')) continue;
|
||||
out = out.filter((r) => r[k] === v);
|
||||
}
|
||||
if (f.lt_game_date) out = out.filter((r) => r.game_date < f.lt_game_date);
|
||||
return { data: out, error: null, count: out.length };
|
||||
},
|
||||
};
|
||||
return { client: { from: () => q }, filters: f };
|
||||
}
|
||||
|
||||
const ROWS = [
|
||||
{ id: '1', p_win: 0.6, outcome: 'hit', game_date: '2026-08-12', model_version: ERA, sport: 'mlb', stat: 'hits' },
|
||||
{ id: '2', p_win: 0.7, outcome: 'miss', game_date: '2026-08-13', model_version: ERA, sport: 'mlb', stat: 'hits' },
|
||||
{ id: '3', p_win: 0.55, outcome: 'hit', game_date: '2026-08-01', model_version: OLD, sport: 'mlb', stat: 'hits' },
|
||||
];
|
||||
|
||||
describe('the era restriction is in the QUERY', () => {
|
||||
it('loads only the requested era', async () => {
|
||||
const { client: c, filters } = client(ROWS);
|
||||
const out = await src.loadRows(c, { sport: 'mlb', stat: 'hits', modelVersion: ERA, before: '2026-09-03' });
|
||||
expect(filters.model_version).toBe(ERA);
|
||||
expect(out.map((r) => r.id).sort()).toEqual(['1', '2']);
|
||||
expect(out.every((r) => r.model_version === ERA)).toBe(true);
|
||||
});
|
||||
|
||||
it('the era audit is computed from the ACTUAL rows, not a second query', () => {
|
||||
const mixed = [{ model_version: ERA }, { model_version: ERA }, { model_version: OLD }];
|
||||
const a = src.eraAudit(mixed, ERA);
|
||||
expect(a.current_era_rows).toBe(2);
|
||||
expect(a.wrong_era_rows).toBe(1);
|
||||
expect(src.eraAudit([{ model_version: ERA }], ERA).wrong_era_rows).toBe(0);
|
||||
});
|
||||
|
||||
it('refuses to run without an explicit era and horizon', async () => {
|
||||
const { client: c } = client(ROWS);
|
||||
await expect(src.loadRows(c, { sport: 'mlb', stat: 'hits', before: '2026-09-03' })).rejects.toThrow();
|
||||
await expect(src.loadRows(c, { sport: 'mlb', stat: 'hits', modelVersion: ERA })).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('the source digest', () => {
|
||||
const R = [
|
||||
{ id: 'b', p: 0.6, won: 1, date: 'd2', model_version: ERA },
|
||||
{ id: 'a', p: 0.5, won: 0, date: 'd1', model_version: ERA },
|
||||
];
|
||||
const base = src.sourceDigest(R);
|
||||
|
||||
it('ignores database return order', () => {
|
||||
expect(src.sourceDigest([...R].reverse())).toBe(base);
|
||||
});
|
||||
|
||||
it('moves when a row is added, removed, or materially changed', () => {
|
||||
expect(src.sourceDigest([R[0]])).not.toBe(base);
|
||||
expect(src.sourceDigest([...R, { id: 'c', p: 0.7, won: 1, date: 'd3', model_version: ERA }])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], won: 0 }, R[1]])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], p: 0.61 }, R[1]])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], model_version: OLD }, R[1]])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], date: 'd9' }, R[1]])).not.toBe(base);
|
||||
});
|
||||
});
|
||||
@@ -84,28 +84,26 @@ describe('the minimum automatic gates', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('the built artifact carries its policy state', () => {
|
||||
const map = cal.fitIsotonic(Array.from({ length: 900 }, (_, i) => {
|
||||
const p = Math.round((0.35 + (i % 60) / 100) * 1000) / 1000;
|
||||
return { p, won: ((i * 2654435761) % 1000) / 1000 < (0.5 + 0.45 * (p - 0.5)) ? 1 : 0, date: `d${i % 14}` };
|
||||
}), { minTotal: 200 });
|
||||
const build = () => svc.build({}, { calibrationService: { fromLedger: async () => ({
|
||||
map, fit_n: 900, fitted_through: '2026-08-21', cutoff: '2026-09-02' }) } });
|
||||
describe('the promoted artifact carries its policy state', () => {
|
||||
const registry = require('../../src/services/model/artifactRegistry');
|
||||
|
||||
it('records the policy version, the violation and servable:false', async () => {
|
||||
const b = await build();
|
||||
expect(b.artifact.fit_policy_version).toBe('mlb-hits-isotonic-refit@v1');
|
||||
expect(b.artifact.fit_policy_valid).toBe(false);
|
||||
expect(b.artifact.fit_policy_violations).toContain(fp.VIOLATION.ERA_NOT_RESTRICTED);
|
||||
expect(b.artifact.servable).toBe(false);
|
||||
it('the shipped artifact PASSES its own policy — detection became repair', () => {
|
||||
const a = registry.load('mlb', 'hits');
|
||||
expect(a.fit_policy_version || a.procedure_version).toBe('mlb-hits-isotonic-refit@v1');
|
||||
expect(a.fit_policy_valid).toBe(true);
|
||||
expect(a.fit_policy_violations).toEqual([]);
|
||||
expect(a.servable).toBe(true);
|
||||
});
|
||||
|
||||
it('the policy violation does NOT distort what the shadow measures', async () => {
|
||||
// The shadow's job is to measure the certified contract on real rows. If a
|
||||
// policy violation flipped every row to UNCERTIFIED the shadow would
|
||||
// measure the violation instead, and the receipt would be worthless.
|
||||
const b = await build();
|
||||
expect(b.resolve({ model_version: ERA, p_win: 0.65 }).probability_state).toBe('CERTIFIED_CALIBRATED');
|
||||
expect(b.resolve({ model_version: ERA, p_win: 0.91 }).probability_state).toBe('UNCERTIFIED');
|
||||
it('the policy is RE-VALIDATED at load, so a hand-edited file cannot smuggle one in', () => {
|
||||
const a = registry.load('mlb', 'hits');
|
||||
// the loader recomputes from era_audit rather than trusting the file's flag
|
||||
const recomputed = fp.validate({ ...a, estimator_type: 'isotonic' },
|
||||
{ era_counts: a.era_audit.era_counts });
|
||||
expect(recomputed.valid).toBe(true);
|
||||
const lying = fp.validate({ ...a, estimator_type: 'isotonic' },
|
||||
{ era_counts: { [ERA]: 100, 'engine1@2026-07-20': 1 } });
|
||||
expect(lying.valid).toBe(false);
|
||||
expect(lying.servable).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -191,93 +191,85 @@ describe('the registry already asked the right question', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('probabilityContractService', () => {
|
||||
it('returns null — not raw — when there is no settled history', async () => {
|
||||
const built = await svc.build({}, { calibrationService: { fromLedger: async () => null } });
|
||||
expect(built).toBeNull();
|
||||
describe('probabilityContractService — loads, never fits', () => {
|
||||
const registry = require('../../src/services/model/artifactRegistry');
|
||||
|
||||
it('returns the PROMOTED frozen artifact, not a fresh fit', async () => {
|
||||
const b = await svc.build(null, { sport: 'mlb', stat: 'hits' });
|
||||
expect(b).not.toBeNull();
|
||||
expect(b.artifact.artifact_id).toBe(registry.PROMOTED['mlb:hits'].artifact_id);
|
||||
expect(b.artifact.servable).toBe(true);
|
||||
});
|
||||
|
||||
it('takes the MAP and never the blocked calibrate() gate', async () => {
|
||||
const cal = require('../../src/services/model/calibration');
|
||||
const map = cal.fitIsotonic(Array.from({ length: 600 }, (_, i) => {
|
||||
const p = 0.40 + (i % 55) / 100;
|
||||
return { p, won: i % 3 === 0 ? 0 : 1, date: `d${i % 12}` };
|
||||
}), { minTotal: 200 });
|
||||
const calibrateSpy = jest.fn(() => ({ p_calibrated: 0.99, calibrated: true }));
|
||||
const built = await svc.build({}, { calibrationService: {
|
||||
fromLedger: async () => ({ map, fit_n: 600, fitted_through: 'd11', calibrate: calibrateSpy }) } });
|
||||
expect(built).not.toBeNull();
|
||||
const r = built.resolve({ model_version: ERA, p_win: 0.65 });
|
||||
expect(r.probability_state).toBe(pc.STATE.CERTIFIED_CALIBRATED);
|
||||
expect(calibrateSpy).not.toHaveBeenCalled();
|
||||
it('returns null — never a fallback curve — when nothing is promoted', async () => {
|
||||
for (const stat of ['rbi', 'total_bases', 'runs']) {
|
||||
expect(await svc.build(null, { sport: 'mlb', stat })).toBeNull();
|
||||
}
|
||||
expect(await svc.build(null, { sport: 'wnba', stat: 'points' })).toBeNull();
|
||||
});
|
||||
|
||||
it('support comes from the artifact, so a nightly refit cannot widen it', async () => {
|
||||
const built = await svc.build({}, { calibrationService: {
|
||||
fromLedger: async () => ({ map: { x: [0, 1], y: [0.5, 0.9] }, fit_n: 900, fitted_through: 'd9',
|
||||
bands: [[0.0, 1.0]] }) } }); // fit claims the whole range
|
||||
expect(built.resolve({ model_version: ERA, p_win: 0.95 }).probability_state).toBe(pc.STATE.UNCERTIFIED);
|
||||
it('needs no database client at all — there is nothing left to read', async () => {
|
||||
const b = await svc.build(undefined, { sport: 'mlb', stat: 'hits' });
|
||||
expect(b.artifact.artifact_id).toBeTruthy();
|
||||
});
|
||||
|
||||
it('two resolutions of the same input are identical, forever', async () => {
|
||||
const b1 = await svc.build(null, { sport: 'mlb', stat: 'hits' });
|
||||
const b2 = await svc.build(null, { sport: 'mlb', stat: 'hits' });
|
||||
for (const p of [0.50, 0.55, 0.601, 0.72, 0.799]) {
|
||||
const a = b1.resolve({ model_version: ERA, p_win: p });
|
||||
const c = b2.resolve({ model_version: ERA, p_win: p });
|
||||
expect(a.served_probability).toBe(c.served_probability);
|
||||
expect(a.artifact.knot_digest).toBe(c.artifact.knot_digest);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('artifact identity — the mapping that actually ran', () => {
|
||||
const cal = require('../../src/services/model/calibration');
|
||||
const mk = (seed) => cal.fitIsotonic(Array.from({ length: 900 }, (_, i) => {
|
||||
const p = Math.round((0.35 + (i % 60) / 100) * 1000) / 1000;
|
||||
return { p, won: ((i * seed) % 1000) / 1000 < (0.5 + 0.45 * (p - 0.5)) ? 1 : 0, date: `d${i % 14}` };
|
||||
}), { minTotal: 200 });
|
||||
const fitted = (map, over = {}) => ({ calibrationService: { fromLedger: async () => ({
|
||||
map, fit_n: 900, fitted_through: 'd13', cutoff: '2026-09-03', ...over }) } });
|
||||
describe('artifact identity — the frozen curve that actually ran', () => {
|
||||
const registry = require('../../src/services/model/artifactRegistry');
|
||||
const a = registry.load('mlb', 'hits');
|
||||
|
||||
it('the same evidence reconstructs the same artifact, digest for digest', async () => {
|
||||
const map = mk(2654435761);
|
||||
const a = await svc.build({}, fitted(map));
|
||||
const b = await svc.build({}, fitted(map));
|
||||
expect(a.artifact.knot_digest).toBe(b.artifact.knot_digest);
|
||||
expect(a.artifact.served_curve_digest).toBe(b.artifact.served_curve_digest);
|
||||
expect(a.artifact.served_curve).toEqual(b.artifact.served_curve);
|
||||
});
|
||||
|
||||
it('DIFFERENT evidence produces a different identity — the digest is not decorative', async () => {
|
||||
const a = await svc.build({}, fitted(mk(2654435761)));
|
||||
const b = await svc.build({}, fitted(mk(40503)));
|
||||
expect(a.artifact.knot_digest).not.toBe(b.artifact.knot_digest);
|
||||
});
|
||||
|
||||
it('carries the point-in-time bound and the training cutoff, distinctly', async () => {
|
||||
const a = await svc.build({}, fitted(mk(2654435761)));
|
||||
expect(a.artifact.fit_as_of).toBe('2026-09-03'); // the lt(game_date) bound
|
||||
expect(a.artifact.training_cutoff).toBe('d13'); // last date inside the fit
|
||||
expect(a.artifact.fit_n).toBe(900);
|
||||
expect(a.artifact.knot_count).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('the served curve IS the served function over certified support, not a sample', async () => {
|
||||
const a = await svc.build({}, fitted(mk(2654435761)));
|
||||
const lookup = (raw) => {
|
||||
let v = null;
|
||||
for (const [from, val] of a.artifact.served_curve) if (raw >= from) v = val;
|
||||
return v;
|
||||
};
|
||||
for (let x = 0.50; x < 0.80 - 1e-9; x += 0.001) {
|
||||
const raw = Math.round(x * 1000) / 1000;
|
||||
const r = a.resolve({ model_version: ERA, p_win: raw });
|
||||
expect(r.served_probability).toBe(Math.round(lookup(raw) * 1000) / 1000);
|
||||
it('carries every field needed to find and check it again', () => {
|
||||
for (const k of ['artifact_id', 'procedure_version', 'sport', 'stat', 'model_version',
|
||||
'fit_as_of', 'training_cutoff', 'fit_n', 'source_digest', 'algorithm', 'algorithm_version',
|
||||
'knot_count', 'knot_digest', 'served_curve', 'served_curve_digest', 'certified_bands']) {
|
||||
expect(a[k]).toBeDefined();
|
||||
expect(a[k]).not.toBeNull();
|
||||
}
|
||||
});
|
||||
|
||||
it('the curve covers ONLY certified support — never the unsupported tail', async () => {
|
||||
const a = await svc.build({}, fitted(mk(2654435761)));
|
||||
for (const [from] of a.artifact.served_curve) {
|
||||
it('fit_as_of and training_cutoff are DIFFERENT questions', () => {
|
||||
// the eligibility bound, and the newest observation actually admitted
|
||||
expect(a.fit_as_of).not.toBe(a.training_cutoff);
|
||||
expect(a.training_cutoff < a.fit_as_of).toBe(true);
|
||||
});
|
||||
|
||||
it('was fitted on the current era ONLY', () => {
|
||||
expect(a.model_version).toBe(ERA);
|
||||
expect(a.era_audit.wrong_era_rows).toBe(0);
|
||||
expect(Object.keys(a.era_audit.era_counts)).toEqual([ERA]);
|
||||
});
|
||||
|
||||
it('withheld nothing from the promoted fit', () => {
|
||||
expect(a.withheld_from_fit).toBe(0);
|
||||
expect(a.fit_n).toBe(a.era_audit.current_era_rows);
|
||||
});
|
||||
|
||||
it('the served curve IS the served function over support, not a sample', () => {
|
||||
for (let x = 0.50; x < 0.80 - 1e-9; x += 0.001) {
|
||||
const raw = Math.round(x * 1000) / 1000;
|
||||
let want = null;
|
||||
for (const [from, val] of a.served_curve) { if (raw >= from) want = val; else break; }
|
||||
expect(registry.applyCurve(a, raw)).toBe(want);
|
||||
}
|
||||
});
|
||||
|
||||
it('the curve covers ONLY certified support', () => {
|
||||
for (const [from] of a.served_curve) {
|
||||
expect(from).toBeGreaterThanOrEqual(0.50);
|
||||
expect(from).toBeLessThan(0.80);
|
||||
}
|
||||
});
|
||||
|
||||
it('a resolution with no artifact records null rather than inventing one', () => {
|
||||
const r = pc.resolve(read(0.65), { estimate: iso });
|
||||
expect(r.artifact).toBeNull();
|
||||
expect(r.served_probability).not.toBeNull();
|
||||
for (const p of [0.499, 0.80, 0.9, 0.99]) expect(registry.applyCurve(a, p)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -313,17 +305,11 @@ describe('point in time — a Read can only see settlements before its own day',
|
||||
expect(bound.some((a) => a[0] === 'gte')).toBe(false);
|
||||
});
|
||||
|
||||
it('the artifact records the bound it was fitted under, so a Read names its own evidence horizon', async () => {
|
||||
const cal = require('../../src/services/model/calibration');
|
||||
const map = cal.fitIsotonic(Array.from({ length: 900 }, (_, i) => {
|
||||
const p = Math.round((0.35 + (i % 60) / 100) * 1000) / 1000;
|
||||
return { p, won: ((i * 2654435761) % 1000) / 1000 < (0.5 + 0.45 * (p - 0.5)) ? 1 : 0, date: `d${i % 14}` };
|
||||
}), { minTotal: 200 });
|
||||
const built = await svc.build({}, { calibrationService: { fromLedger: async () => ({
|
||||
map, fit_n: 900, fitted_through: '2026-08-21', cutoff: '2026-09-02' }) } });
|
||||
expect(built.artifact.fit_as_of).toBe('2026-09-02');
|
||||
expect(built.artifact.training_cutoff).toBe('2026-08-21');
|
||||
// and the two are DIFFERENT questions — the bound, and the last date inside it
|
||||
expect(built.artifact.fit_as_of).not.toBe(built.artifact.training_cutoff);
|
||||
it('the artifact names its own evidence horizon', async () => {
|
||||
const built = await svc.build(null, { sport: 'mlb', stat: 'hits' });
|
||||
expect(built.artifact.fit_as_of).toBeTruthy();
|
||||
expect(built.artifact.training_cutoff).toBeTruthy();
|
||||
// the bound, and the last date inside it — never the same claim
|
||||
expect(built.artifact.training_cutoff < built.artifact.fit_as_of).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user