Detection becomes repair: the curve is fitted on one forecaster, frozen, and named
The last release detected the violation and then served the certified state anyway. A validator that changes nothing is decoration, so `servable:false` is now load-bearing: an artifact that fails its policy returns ARTIFACT_POLICY_BLOCKED with no number, and every probability-derived claim goes with it. The gate sits inside the resolution, not beside the flag that turns the shadow on, so no environment variable can reach past it — a test asserts `resolve` never reads process.env at all. Shadow and live consume the SAME decision, differing only in which promotion stage they demand. Era mismatch still resolves to VERSION_MISMATCH rather than the new state. "This artifact belongs to a different forecaster" is more precise than "policy blocked", and the existing state already says it exactly. THE REPAIR. `currentEraSource` filters on model_version in the QUERY, taking the era from config/modelVersion so the query, the artifact and the validator all read one identity. Measured on the actual fitted set, not a second count: 6,069 current-era rows, 0 wrong-era. The procedure was then certified on current-era rows ONLY — four walk-forward folds, training strictly before each evaluation block, 0 future rows in train on every fold. All four improve; pooled n=3,108 gives Brier 0.24701 -> 0.24323, delta -0.00378, CI [-0.00619,-0.00147] excluding zero; ECE falls in every fold. Mapping spread inside support is 0.001-0.018. The prior mixed-era certification did not substitute for this. Policy B selected. A (era-filtered 65/35) and B (all current-era) are statistically indistinguishable, A-B = +0.0001 CI [-0.00029,+0.00048], but B has the better ECE (0.0064 vs 0.0109) and the holdout existed to certify the PROCEDURE — it is not permanently withheld from the artifact that ships. withheld_from_fit is 0. FROZEN. `mlb-hits-isotonic@2026-09-03`: 6,069 rows, training_cutoff 2026-09-01 (distinct from fit_as_of 2026-09-03 — the newest observation admitted is not the eligibility bound), 12 knots, source_digest 25919c16…, knot_digest 5ae940ea…, served_curve_digest c24a9dc5…, 8 curve steps, 924 bytes, committed as JSON. The runtime no longer fits. It loads. A test greps the service for fitIsotonic, fromLedger and loadRows and requires all three absent, because the old behaviour meant a user's number could move with no version, no review and no rollback, and a past Read could not be reconstructed because its curve no longer existed. New settled outcomes are forward evidence now; they cannot touch this curve. Independent reconstruction from the declared training contract alone — fresh read, fresh digest, fresh fit — reproduces every digest and the curve byte for byte. Calling the builder twice would only have proven the builder deterministic. Promotion is a frozen source constant. A snapshot cannot promote, a settlement cannot promote, a successful fit cannot promote, and dropping a file into the artifacts directory promotes nothing. Stage is APPROVED_FOR_SHADOW; live is explicitly false. Two coverage holes found by their own teeth. The promotion guard could be deleted with every test still green, because the promoted file naturally agrees with itself — extracted as `acceptFile` and tested on the case `load()` cannot reach. And `validate(null)` returned no `servable` field at all, which is falsy at a call site and so would have read as correct while asserting nothing. Shadow OFF. Live OFF. CALIBRATION_DEPLOYED []. No frontend change. Suite 404/404, 5,634 passed, 4 skipped. Teeth 26/26 + 10/10 + 23/23. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* THE INVARIANT: artifact.model_version = X => distinct(training.model_version) = {X}.
|
||||
*/
|
||||
const src = require('../../src/services/model/currentEraSource');
|
||||
|
||||
const ERA = 'engine1@2026-08-07-fullwindow';
|
||||
const OLD = 'engine1@2026-07-20';
|
||||
|
||||
/** A fake that HONOURS its filters — a pass-through would prove nothing. */
|
||||
function client(rows) {
|
||||
const f = {};
|
||||
const q = {
|
||||
select: () => q,
|
||||
eq: (c, v) => { f[c] = v; return q; },
|
||||
is: () => q,
|
||||
in: (c, v) => { f[`in_${c}`] = v; return q; },
|
||||
not: () => q,
|
||||
lt: (c, v) => { f[`lt_${c}`] = v; return q; },
|
||||
order: () => q,
|
||||
range: async () => {
|
||||
let out = rows;
|
||||
for (const [k, v] of Object.entries(f)) {
|
||||
if (k.startsWith('in_') || k.startsWith('lt_')) continue;
|
||||
out = out.filter((r) => r[k] === v);
|
||||
}
|
||||
if (f.lt_game_date) out = out.filter((r) => r.game_date < f.lt_game_date);
|
||||
return { data: out, error: null, count: out.length };
|
||||
},
|
||||
};
|
||||
return { client: { from: () => q }, filters: f };
|
||||
}
|
||||
|
||||
const ROWS = [
|
||||
{ id: '1', p_win: 0.6, outcome: 'hit', game_date: '2026-08-12', model_version: ERA, sport: 'mlb', stat: 'hits' },
|
||||
{ id: '2', p_win: 0.7, outcome: 'miss', game_date: '2026-08-13', model_version: ERA, sport: 'mlb', stat: 'hits' },
|
||||
{ id: '3', p_win: 0.55, outcome: 'hit', game_date: '2026-08-01', model_version: OLD, sport: 'mlb', stat: 'hits' },
|
||||
];
|
||||
|
||||
describe('the era restriction is in the QUERY', () => {
|
||||
it('loads only the requested era', async () => {
|
||||
const { client: c, filters } = client(ROWS);
|
||||
const out = await src.loadRows(c, { sport: 'mlb', stat: 'hits', modelVersion: ERA, before: '2026-09-03' });
|
||||
expect(filters.model_version).toBe(ERA);
|
||||
expect(out.map((r) => r.id).sort()).toEqual(['1', '2']);
|
||||
expect(out.every((r) => r.model_version === ERA)).toBe(true);
|
||||
});
|
||||
|
||||
it('the era audit is computed from the ACTUAL rows, not a second query', () => {
|
||||
const mixed = [{ model_version: ERA }, { model_version: ERA }, { model_version: OLD }];
|
||||
const a = src.eraAudit(mixed, ERA);
|
||||
expect(a.current_era_rows).toBe(2);
|
||||
expect(a.wrong_era_rows).toBe(1);
|
||||
expect(src.eraAudit([{ model_version: ERA }], ERA).wrong_era_rows).toBe(0);
|
||||
});
|
||||
|
||||
it('refuses to run without an explicit era and horizon', async () => {
|
||||
const { client: c } = client(ROWS);
|
||||
await expect(src.loadRows(c, { sport: 'mlb', stat: 'hits', before: '2026-09-03' })).rejects.toThrow();
|
||||
await expect(src.loadRows(c, { sport: 'mlb', stat: 'hits', modelVersion: ERA })).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('the source digest', () => {
|
||||
const R = [
|
||||
{ id: 'b', p: 0.6, won: 1, date: 'd2', model_version: ERA },
|
||||
{ id: 'a', p: 0.5, won: 0, date: 'd1', model_version: ERA },
|
||||
];
|
||||
const base = src.sourceDigest(R);
|
||||
|
||||
it('ignores database return order', () => {
|
||||
expect(src.sourceDigest([...R].reverse())).toBe(base);
|
||||
});
|
||||
|
||||
it('moves when a row is added, removed, or materially changed', () => {
|
||||
expect(src.sourceDigest([R[0]])).not.toBe(base);
|
||||
expect(src.sourceDigest([...R, { id: 'c', p: 0.7, won: 1, date: 'd3', model_version: ERA }])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], won: 0 }, R[1]])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], p: 0.61 }, R[1]])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], model_version: OLD }, R[1]])).not.toBe(base);
|
||||
expect(src.sourceDigest([{ ...R[0], date: 'd9' }, R[1]])).not.toBe(base);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user