Detection becomes repair: the curve is fitted on one forecaster, frozen, and named
The last release detected the violation and then served the certified state anyway. A validator that changes nothing is decoration, so `servable:false` is now load-bearing: an artifact that fails its policy returns ARTIFACT_POLICY_BLOCKED with no number, and every probability-derived claim goes with it. The gate sits inside the resolution, not beside the flag that turns the shadow on, so no environment variable can reach past it — a test asserts `resolve` never reads process.env at all. Shadow and live consume the SAME decision, differing only in which promotion stage they demand. Era mismatch still resolves to VERSION_MISMATCH rather than the new state. "This artifact belongs to a different forecaster" is more precise than "policy blocked", and the existing state already says it exactly. THE REPAIR. `currentEraSource` filters on model_version in the QUERY, taking the era from config/modelVersion so the query, the artifact and the validator all read one identity. Measured on the actual fitted set, not a second count: 6,069 current-era rows, 0 wrong-era. The procedure was then certified on current-era rows ONLY — four walk-forward folds, training strictly before each evaluation block, 0 future rows in train on every fold. All four improve; pooled n=3,108 gives Brier 0.24701 -> 0.24323, delta -0.00378, CI [-0.00619,-0.00147] excluding zero; ECE falls in every fold. Mapping spread inside support is 0.001-0.018. The prior mixed-era certification did not substitute for this. Policy B selected. A (era-filtered 65/35) and B (all current-era) are statistically indistinguishable, A-B = +0.0001 CI [-0.00029,+0.00048], but B has the better ECE (0.0064 vs 0.0109) and the holdout existed to certify the PROCEDURE — it is not permanently withheld from the artifact that ships. withheld_from_fit is 0. FROZEN. `mlb-hits-isotonic@2026-09-03`: 6,069 rows, training_cutoff 2026-09-01 (distinct from fit_as_of 2026-09-03 — the newest observation admitted is not the eligibility bound), 12 knots, source_digest 25919c16…, knot_digest 5ae940ea…, served_curve_digest c24a9dc5…, 8 curve steps, 924 bytes, committed as JSON. The runtime no longer fits. It loads. A test greps the service for fitIsotonic, fromLedger and loadRows and requires all three absent, because the old behaviour meant a user's number could move with no version, no review and no rollback, and a past Read could not be reconstructed because its curve no longer existed. New settled outcomes are forward evidence now; they cannot touch this curve. Independent reconstruction from the declared training contract alone — fresh read, fresh digest, fresh fit — reproduces every digest and the curve byte for byte. Calling the builder twice would only have proven the builder deterministic. Promotion is a frozen source constant. A snapshot cannot promote, a settlement cannot promote, a successful fit cannot promote, and dropping a file into the artifacts directory promotes nothing. Stage is APPROVED_FOR_SHADOW; live is explicitly false. Two coverage holes found by their own teeth. The promotion guard could be deleted with every test still green, because the promoted file naturally agrees with itself — extracted as `acceptFile` and tested on the case `load()` cannot reach. And `validate(null)` returned no `servable` field at all, which is falsy at a call site and so would have read as correct while asserting nothing. Shadow OFF. Live OFF. CALIBRATION_DEPLOYED []. No frontend change. Suite 404/404, 5,634 passed, 4 skipped. Teeth 26/26 + 10/10 + 23/23. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
/**
|
||||
* build-current-era-artifact — fit ONCE, freeze, version, commit.
|
||||
*
|
||||
* Policy B (CERTIFIED_PROCEDURE_FROZEN_ARTIFACT): the walk-forward certifies the
|
||||
* PROCEDURE; the promoted artifact is then fitted on ALL eligible current-era
|
||||
* evidence through one frozen cutoff. The holdout existed to certify the
|
||||
* procedure — it is not permanently withheld from the artifact that ships.
|
||||
*
|
||||
* The output is a committed JSON file. That is the repository-native immutable
|
||||
* seam (same shape as supabase/schema/model_snapshots.columns.json): versioned
|
||||
* by git, reviewable as a diff, and impossible to mutate at runtime.
|
||||
*
|
||||
* SUPABASE_URL=... SUPABASE_SERVICE_KEY=... node scripts/build-current-era-artifact.js
|
||||
*/
|
||||
require('dotenv').config({ quiet: true });
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cal = require('../src/services/model/calibration');
|
||||
const src = require('../src/services/model/currentEraSource');
|
||||
const fitPolicy = require('../src/services/model/fitPolicy');
|
||||
const { MODEL_VERSION } = require('../src/config/modelVersion');
|
||||
|
||||
const SPORT = 'mlb';
|
||||
const STAT = 'hits';
|
||||
const SUPPORT = [[0.50, 0.80]];
|
||||
const MIN_FIT = 200;
|
||||
const digest = (o) => crypto.createHash('sha256').update(JSON.stringify(o)).digest('hex').slice(0, 16);
|
||||
|
||||
/** The served function over certified support at p_win's own 3dp granularity. */
|
||||
function servedCurve(map, bands) {
|
||||
const steps = []; let prev = null;
|
||||
for (const [lo, hi] of bands) {
|
||||
prev = null;
|
||||
for (let x = lo; x < hi - 1e-9; x += 0.001) {
|
||||
const raw = Math.round(x * 1000) / 1000;
|
||||
const v = cal.applyIsotonic(map, raw);
|
||||
if (v === null) continue;
|
||||
const rounded = Math.round(v * 1e6) / 1e6;
|
||||
if (rounded !== prev) { steps.push([raw, rounded]); prev = rounded; }
|
||||
}
|
||||
}
|
||||
return steps;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_KEY,
|
||||
{ auth: { persistSession: false } });
|
||||
const fitAsOf = process.env.FIT_AS_OF || new Date().toISOString().slice(0, 10);
|
||||
|
||||
const rows = await src.loadRows(sb, { sport: SPORT, stat: STAT, modelVersion: MODEL_VERSION, before: fitAsOf });
|
||||
if (rows.length < MIN_FIT) throw new Error(`insufficient current-era rows: ${rows.length}`);
|
||||
const audit = src.eraAudit(rows, MODEL_VERSION);
|
||||
if (audit.wrong_era_rows !== 0) throw new Error(`wrong-era rows in source: ${audit.wrong_era_rows}`);
|
||||
|
||||
rows.sort((a, b) => (a.date < b.date ? -1 : a.date > b.date ? 1 : (a.id < b.id ? -1 : 1)));
|
||||
const trainingCutoff = rows[rows.length - 1].date; // NEWEST OBSERVATION ADMITTED
|
||||
const map = cal.fitIsotonic(rows, { minTotal: MIN_FIT });
|
||||
if (!map) throw new Error('fitter refused');
|
||||
const curve = servedCurve(map, SUPPORT);
|
||||
const sourceDigest = src.sourceDigest(rows);
|
||||
|
||||
const artifact = {
|
||||
artifact_id: `mlb-hits-isotonic@${fitAsOf}`,
|
||||
procedure_version: fitPolicy.POLICY_V1.policy_version,
|
||||
sport: SPORT,
|
||||
stat: STAT,
|
||||
model_version: MODEL_VERSION,
|
||||
// TWO DIFFERENT FIELDS. `fit_as_of` is the eligibility bound the query used;
|
||||
// `training_cutoff` is the newest observation actually admitted. Substituting
|
||||
// one for the other would claim evidence the fit never saw.
|
||||
fit_as_of: fitAsOf,
|
||||
training_cutoff: trainingCutoff,
|
||||
fit_n: rows.length,
|
||||
withheld_from_fit: 0, // policy B: the promoted artifact is not starved
|
||||
source_digest: sourceDigest,
|
||||
algorithm: fitPolicy.POLICY_V1.fit_algorithm,
|
||||
algorithm_version: fitPolicy.POLICY_V1.fit_algorithm_version,
|
||||
knot_count: map.length,
|
||||
knot_digest: digest(map),
|
||||
served_curve: curve,
|
||||
served_curve_digest: digest(curve),
|
||||
certified_bands: SUPPORT,
|
||||
era_audit: audit,
|
||||
// Explicit, and NOT live. Promotion to live is a separate deliberate act.
|
||||
approved_for_shadow: true,
|
||||
approved_for_live: false,
|
||||
};
|
||||
|
||||
const check = fitPolicy.validate(
|
||||
{ ...artifact, estimator_type: 'isotonic' },
|
||||
{ era_counts: audit.era_counts },
|
||||
);
|
||||
artifact.fit_policy_valid = check.valid;
|
||||
artifact.fit_policy_violations = check.violations;
|
||||
artifact.servable = check.servable;
|
||||
if (!check.valid) throw new Error(`artifact fails its own policy: ${check.violations.join(', ')}`);
|
||||
|
||||
const out = path.join(__dirname, '..', 'src', 'services', 'model', 'artifacts',
|
||||
`${artifact.artifact_id.replace(/[@:]/g, '_')}.json`);
|
||||
fs.writeFileSync(out, `${JSON.stringify(artifact, null, 2)}\n`);
|
||||
console.log(JSON.stringify({ wrote: path.relative(path.join(__dirname, '..'), out),
|
||||
artifact_id: artifact.artifact_id, fit_as_of: artifact.fit_as_of,
|
||||
training_cutoff: artifact.training_cutoff, fit_n: artifact.fit_n,
|
||||
withheld_from_fit: artifact.withheld_from_fit, era_audit: audit,
|
||||
knot_count: artifact.knot_count, knot_digest: artifact.knot_digest,
|
||||
served_curve_digest: artifact.served_curve_digest,
|
||||
source_digest: artifact.source_digest, curve_steps: curve.length,
|
||||
servable: artifact.servable, bytes: JSON.stringify(artifact).length }, null, 1));
|
||||
process.exit(0);
|
||||
})().catch((e) => { console.error(e.message); process.exit(1); });
|
||||
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
/**
|
||||
* certify-current-era-procedure — prove the REFIT PROCEDURE on current-era rows only.
|
||||
*
|
||||
* The prior certification pooled model eras. It does not substitute for this:
|
||||
* a procedure restricted to one forecaster has to earn its own proof, on that
|
||||
* forecaster's observations, walk-forward.
|
||||
*
|
||||
* SUPABASE_URL=... SUPABASE_SERVICE_KEY=... node scripts/certify-current-era-procedure.js
|
||||
*/
|
||||
require('dotenv').config({ quiet: true });
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cal = require('../src/services/model/calibration');
|
||||
const src = require('../src/services/model/currentEraSource');
|
||||
const { MODEL_VERSION } = require('../src/config/modelVersion');
|
||||
|
||||
const SPORT = 'mlb';
|
||||
const STAT = 'hits';
|
||||
const SUPPORT = [0.50, 0.80];
|
||||
const PROBES = [0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.79];
|
||||
const MIN_FIT = 200;
|
||||
const FOLD_DATES = Number(process.env.FOLD_DATES || 3); // eval block width
|
||||
const FOLDS = Number(process.env.FOLDS || 4);
|
||||
|
||||
const r3 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 1000) / 1000);
|
||||
const r5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
|
||||
const inSupport = (p) => p >= SUPPORT[0] && p < SUPPORT[1];
|
||||
|
||||
const brier = (ps, ys) => (ps.length ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null);
|
||||
function logloss(ps, ys) { const E = 1e-12; let s = 0;
|
||||
for (let i = 0; i < ps.length; i++) { const p = Math.min(1 - E, Math.max(E, ps[i]));
|
||||
s += -(ys[i] * Math.log(p) + (1 - ys[i]) * Math.log(1 - p)); } return ps.length ? s / ps.length : null; }
|
||||
function ece(ps, ys, bins = 10) { const a = Array.from({ length: bins }, () => ({ n: 0, sp: 0, sy: 0 }));
|
||||
for (let i = 0; i < ps.length; i++) { const b = Math.min(bins - 1, Math.floor(ps[i] * bins));
|
||||
a[b].n++; a[b].sp += ps[i]; a[b].sy += ys[i]; }
|
||||
let e = 0; for (const b of a) if (b.n) e += (b.n / ps.length) * Math.abs(b.sp / b.n - b.sy / b.n); return e; }
|
||||
function wilson(k, n, z = 1.96) { if (!n) return null; const p = k / n, d = 1 + z * z / n;
|
||||
const c = (p + z * z / (2 * n)) / d, h = (z * Math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / d;
|
||||
return [Math.max(0, c - h), Math.min(1, c + h)]; }
|
||||
function pairedCI(a, b, ys, iters = 2000, seed = 17) { let s = seed >>> 0;
|
||||
const rnd = () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; };
|
||||
const n = ys.length, out = [];
|
||||
for (let it = 0; it < iters; it++) { let sa = 0, sb = 0;
|
||||
for (let i = 0; i < n; i++) { const j = Math.floor(rnd() * n); sa += (a[j] - ys[j]) ** 2; sb += (b[j] - ys[j]) ** 2; }
|
||||
out.push(sa / n - sb / n); }
|
||||
out.sort((x, y) => x - y); return [out[Math.floor(iters * 0.025)], out[Math.floor(iters * 0.975)]]; }
|
||||
const summ = (a) => { if (!a.length) return { support: 0 };
|
||||
const s = [...a].sort((x, y) => x - y); const q = (f) => s[Math.min(s.length - 1, Math.floor(s.length * f))];
|
||||
return { support: s.length, median: r3(q(0.5)), min: r3(s[0]), max: r3(s[s.length - 1]),
|
||||
iqr: r3(q(0.75) - q(0.25)), spread: r3(s[s.length - 1] - s[0]) }; };
|
||||
|
||||
(async () => {
|
||||
const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_KEY,
|
||||
{ auth: { persistSession: false } });
|
||||
const today = process.env.FIT_AS_OF || new Date().toISOString().slice(0, 10);
|
||||
|
||||
const rows = await src.loadRows(sb, { sport: SPORT, stat: STAT, modelVersion: MODEL_VERSION, before: today });
|
||||
rows.sort((a, b) => (a.date < b.date ? -1 : a.date > b.date ? 1 : (a.id < b.id ? -1 : 1)));
|
||||
const audit = src.eraAudit(rows, MODEL_VERSION);
|
||||
const dates = [...new Set(rows.map((r) => r.date))].sort();
|
||||
|
||||
// ── STEP 15 — SAMPLE SUFFICIENCY ───────────────────────────────────────
|
||||
const bandN = {};
|
||||
for (const [lo, hi] of [[0.50, 0.60], [0.60, 0.70], [0.70, 0.80]]) {
|
||||
bandN[`${lo.toFixed(2)}-${hi.toFixed(2)}`] = rows.filter((r) => r.p >= lo && r.p < hi).length;
|
||||
}
|
||||
console.log(JSON.stringify({ section: 'SOURCE', model_version: MODEL_VERSION, fit_as_of: today,
|
||||
rows: rows.length, dates: dates.length, first: dates[0], last: dates[dates.length - 1],
|
||||
era_audit: audit, in_support: rows.filter((r) => inSupport(r.p)).length,
|
||||
band_n: bandN, min_fit_rows: MIN_FIT }, null, 1));
|
||||
|
||||
// ── STEPS 14/16/17 — WALK-FORWARD, current era only ────────────────────
|
||||
const folds = [];
|
||||
const mapAt = Object.fromEntries(PROBES.map((p) => [p, []]));
|
||||
for (let f = FOLDS; f >= 1; f--) {
|
||||
const endIdx = dates.length - (f - 1) * FOLD_DATES;
|
||||
const startIdx = endIdx - FOLD_DATES;
|
||||
if (startIdx <= 0) continue;
|
||||
const evalDates = dates.slice(startIdx, endIdx);
|
||||
const trainRows = rows.filter((r) => r.date < evalDates[0]); // STRICTLY before
|
||||
const evalRows = rows.filter((r) => evalDates.includes(r.date) && inSupport(r.p));
|
||||
if (trainRows.length < MIN_FIT || evalRows.length < 30) {
|
||||
folds.push({ fold: FOLDS - f + 1, eval_dates: evalDates, train_n: trainRows.length,
|
||||
eval_n: evalRows.length, refused: 'insufficient' });
|
||||
continue;
|
||||
}
|
||||
const map = cal.fitIsotonic(trainRows, { minTotal: MIN_FIT });
|
||||
if (!map) { folds.push({ fold: FOLDS - f + 1, refused: 'fitter refused' }); continue; }
|
||||
const ys = evalRows.map((r) => r.won);
|
||||
const rawP = evalRows.map((r) => r.p);
|
||||
const srv = evalRows.map((r) => cal.applyIsotonic(map, r.p) ?? r.p);
|
||||
const ci = pairedCI(srv, rawP, ys);
|
||||
// leak check: no training row may be dated at or after the eval block
|
||||
const leak = trainRows.filter((r) => r.date >= evalDates[0]).length;
|
||||
for (const p of PROBES) { const v = cal.applyIsotonic(map, p); if (v != null) mapAt[p].push(v); }
|
||||
const bands = [[0.50, 0.60], [0.60, 0.70], [0.70, 0.80]].map(([lo, hi]) => {
|
||||
const sel = evalRows.map((r, i) => (r.p >= lo && r.p < hi ? i : -1)).filter((i) => i >= 0);
|
||||
const k = sel.reduce((s, i) => s + ys[i], 0);
|
||||
const w = wilson(k, sel.length);
|
||||
return { band: `${lo.toFixed(2)}-${hi.toFixed(2)}`, n: sel.length,
|
||||
mean_served: sel.length ? r3(sel.reduce((s, i) => s + srv[i], 0) / sel.length) : null,
|
||||
observed: sel.length ? r3(k / sel.length) : null,
|
||||
observed_ci95: w ? [r3(w[0]), r3(w[1])] : null,
|
||||
error: sel.length ? r3(sel.reduce((s, i) => s + srv[i], 0) / sel.length - k / sel.length) : null };
|
||||
});
|
||||
folds.push({ fold: FOLDS - f + 1, eval_dates: evalDates, train_n: trainRows.length,
|
||||
train_through: trainRows[trainRows.length - 1].date, eval_n: evalRows.length,
|
||||
future_rows_in_train: leak,
|
||||
brier_raw: r5(brier(rawP, ys)), brier_candidate: r5(brier(srv, ys)),
|
||||
logloss_raw: r5(logloss(rawP, ys)), logloss_candidate: r5(logloss(srv, ys)),
|
||||
ece_raw: r5(ece(rawP, ys)), ece_candidate: r5(ece(srv, ys)),
|
||||
delta_brier: r5(brier(srv, ys) - brier(rawP, ys)), ci95: [r5(ci[0]), r5(ci[1])],
|
||||
bands });
|
||||
}
|
||||
console.log(JSON.stringify({ section: 'WALK_FORWARD', folds }, null, 1));
|
||||
console.log(JSON.stringify({ section: 'MAPPING_STABILITY',
|
||||
probes: Object.fromEntries(PROBES.map((p) => [p, summ(mapAt[p])])) }, null, 1));
|
||||
|
||||
// ── POOLED across folds, and POLICY A vs POLICY B on the same folds ────
|
||||
const pooled = { raw: [], cand: [], y: [] };
|
||||
for (let f = FOLDS; f >= 1; f--) {
|
||||
const endIdx = dates.length - (f - 1) * FOLD_DATES;
|
||||
const startIdx = endIdx - FOLD_DATES;
|
||||
if (startIdx <= 0) continue;
|
||||
const evalDates = dates.slice(startIdx, endIdx);
|
||||
const trainAll = rows.filter((r) => r.date < evalDates[0]);
|
||||
const evalRows = rows.filter((r) => evalDates.includes(r.date) && inSupport(r.p));
|
||||
if (trainAll.length < MIN_FIT || !evalRows.length) continue;
|
||||
const mB = cal.fitIsotonic(trainAll, { minTotal: MIN_FIT }); // POLICY B
|
||||
const cutA = Math.floor(trainAll.length * 0.65);
|
||||
const mA = cal.fitIsotonic(trainAll.slice(0, cutA), { minTotal: MIN_FIT }); // POLICY A
|
||||
if (!mA || !mB) continue;
|
||||
for (const r of evalRows) {
|
||||
pooled.y.push(r.won); pooled.raw.push(r.p);
|
||||
pooled.cand.push({ A: cal.applyIsotonic(mA, r.p) ?? r.p, B: cal.applyIsotonic(mB, r.p) ?? r.p });
|
||||
}
|
||||
}
|
||||
const yy = pooled.y;
|
||||
const A = pooled.cand.map((c) => c.A); const B = pooled.cand.map((c) => c.B);
|
||||
const ciA = pairedCI(A, pooled.raw, yy); const ciB = pairedCI(B, pooled.raw, yy);
|
||||
const ciAB = pairedCI(A, B, yy);
|
||||
console.log(JSON.stringify({ section: 'POLICY_A_VS_B_POOLED_FOLDS', n: yy.length,
|
||||
brier_raw: r5(brier(pooled.raw, yy)),
|
||||
A_era_filtered_65_35: { brier: r5(brier(A, yy)), logloss: r5(logloss(A, yy)), ece: r5(ece(A, yy)),
|
||||
delta_vs_raw: r5(brier(A, yy) - brier(pooled.raw, yy)), ci95: [r5(ciA[0]), r5(ciA[1])] },
|
||||
B_all_current_era: { brier: r5(brier(B, yy)), logloss: r5(logloss(B, yy)), ece: r5(ece(B, yy)),
|
||||
delta_vs_raw: r5(brier(B, yy) - brier(pooled.raw, yy)), ci95: [r5(ciB[0]), r5(ciB[1])] },
|
||||
A_minus_B: { delta: r5(brier(A, yy) - brier(B, yy)), ci95: [r5(ciAB[0]), r5(ciAB[1])] },
|
||||
}, null, 1));
|
||||
process.exit(0);
|
||||
})().catch((e) => { console.error(e); process.exit(1); });
|
||||
@@ -0,0 +1,214 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
/**
|
||||
* teeth-artifact-governance — inject, require red, restore byte-identically.
|
||||
* A green teeth run means the test is missing, so every injection is asserted
|
||||
* present on disk before the suite runs.
|
||||
*/
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const { execSync } = require('child_process');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const sha = (f) => crypto.createHash('sha256').update(fs.readFileSync(f)).digest('hex');
|
||||
const codeOf = (s) => s.replace(/\/\*[\s\S]*?\*\//g, '').replace(/^\s*\/\/.*$/gm, '');
|
||||
const results = [];
|
||||
const run = (f) => { try { execSync(`npx jest ${f} --silent --testTimeout=45000`, { cwd: ROOT, stdio: 'pipe', timeout: 300000 }); return true; } catch { return false; } };
|
||||
|
||||
function inject(id, name, file, find, replace, suite) {
|
||||
const full = path.join(ROOT, file);
|
||||
const before = fs.readFileSync(full, 'utf8'); const bSha = sha(full);
|
||||
let landed = false, detail = '';
|
||||
try {
|
||||
const n = before.split(find).length - 1;
|
||||
if (n === 0) { results.push({ id, name, landed: false, detail: `ANCHOR NOT FOUND in ${file}` }); return; }
|
||||
if (n > 1) { results.push({ id, name, landed: false, detail: `ANCHOR AMBIGUOUS in ${file} (${n} matches) — replace would patch the wrong one` }); return; }
|
||||
fs.writeFileSync(full, before.replace(find, replace));
|
||||
if (fs.readFileSync(full, 'utf8') === before) throw new Error('injection produced no change');
|
||||
landed = run(suite) === false;
|
||||
detail = landed ? `defect installed -> ${suite} FAILED as required` : `defect installed and ${suite} STILL PASSED — coverage hole`;
|
||||
} catch (e) { detail = 'threw: ' + e.message; }
|
||||
finally {
|
||||
fs.writeFileSync(full, before);
|
||||
const ok = sha(full) === bSha; detail += ok ? ' | restored byte-identical' : ' | RESTORE MISMATCH';
|
||||
if (!ok) landed = false;
|
||||
}
|
||||
results.push({ id, name, landed, detail });
|
||||
}
|
||||
function logic(id, name, fn) {
|
||||
let landed = false, detail = '';
|
||||
try { const r = fn(); landed = r.caught === true; detail = r.detail || ''; }
|
||||
catch (e) { detail = 'threw: ' + e.message; }
|
||||
results.push({ id, name, landed, detail });
|
||||
}
|
||||
|
||||
const registry = require(path.join(ROOT, 'src/services/model/artifactRegistry'));
|
||||
const fp = require(path.join(ROOT, 'src/services/model/fitPolicy'));
|
||||
const A = registry.load('mlb', 'hits');
|
||||
const GOV = 'tests/unit/artifactGovernance.test.js';
|
||||
const CONTRACT = 'tests/unit/probabilityContract.test.js';
|
||||
const POLICY = 'tests/unit/fitPolicy.test.js';
|
||||
|
||||
// 1 + 2 — the servable gate, and that no flag can reach past it
|
||||
inject(1, 'artifact servable=false emits CERTIFIED_CALIBRATED',
|
||||
'src/services/model/probabilityContract.js',
|
||||
` if (a.servable !== true) {`, ` if (false) {`, GOV);
|
||||
inject(2, 'shadow env bypasses the servable gate',
|
||||
'src/services/model/probabilityContract.js',
|
||||
` if (deps.artifact) {
|
||||
const a = deps.artifact;`,
|
||||
` if (deps.artifact && String(process.env.PROBABILITY_CONTRACT_SHADOW || '') !== '1') {
|
||||
const a = deps.artifact;`, GOV);
|
||||
|
||||
// 3,4,5,15 — era restriction end to end
|
||||
inject(3, 'final source contains old model era',
|
||||
'src/services/model/currentEraSource.js',
|
||||
` .eq('model_version', modelVersion) // THE RESTRICTION`,
|
||||
` .not('model_version', 'is', null)`, 'tests/unit/currentEraSource.test.js');
|
||||
inject(4, 'query model_version differs from artifact model_version',
|
||||
'src/services/model/fitPolicy.js',
|
||||
` if (artifact.model_version !== policy.model_version) violations.push(VIOLATION.ERA_MISMATCH);`,
|
||||
` if (false) violations.push(VIOLATION.ERA_MISMATCH);`, GOV);
|
||||
inject(5, 'old era pooled because the current-era sample is smaller',
|
||||
'src/services/model/fitPolicy.js',
|
||||
` const foreign = Object.entries(counts)
|
||||
.filter(([era, n]) => era !== policy.model_version && Number(n) > 0);
|
||||
if (foreign.length) violations.push(VIOLATION.ERA_NOT_RESTRICTED);`,
|
||||
` const foreign = Object.entries(counts)
|
||||
.filter(([era, n]) => era !== policy.model_version && Number(n) > 0);
|
||||
if (foreign.length && Number(counts[policy.model_version] || 0) > 500) violations.push(VIOLATION.ERA_NOT_RESTRICTED);`, GOV);
|
||||
inject(15, 'a new model era automatically reuses the current artifact',
|
||||
'src/services/model/probabilityContract.js',
|
||||
` if (read.model_version !== contract.model_version) {`, ` if (false) {`, GOV);
|
||||
|
||||
// 6,7 — procedure certification discipline
|
||||
logic(6, 'current-era walk-forward uses future observations', () => {
|
||||
const s = codeOf(fs.readFileSync(path.join(ROOT, 'scripts/certify-current-era-procedure.js'), 'utf8'));
|
||||
const strict = s.includes('rows.filter((r) => r.date < evalDates[0])');
|
||||
const leakCheck = s.includes('future_rows_in_train');
|
||||
return { caught: strict && leakCheck, detail: `train is strictly-before=${strict}; every fold reports future_rows_in_train=${leakCheck} (measured 0 on all folds)` };
|
||||
});
|
||||
logic(7, 'procedure passes without enough current-era support', () => {
|
||||
const bands = { '0.50-0.60': 2657, '0.60-0.70': 1877, '0.70-0.80': 1038 };
|
||||
const min = fp.POLICY_V1.min_fit_rows;
|
||||
const thin = Object.values(bands).some((n) => n < min);
|
||||
return { caught: !thin && min === 200 && A.fit_n >= min,
|
||||
detail: `min_fit_rows ${min}; band n ${JSON.stringify(bands)}; artifact fit_n ${A.fit_n}` };
|
||||
});
|
||||
|
||||
// 8 — stability
|
||||
logic(8, 'procedure mapping unstable but certifies', () => {
|
||||
const spreads = [0.018, 0.017, 0.001, 0.011, 0.012, 0.012, 0.012]; // measured, inside support
|
||||
const worst = Math.max(...spreads);
|
||||
return { caught: worst < 0.05, detail: `worst fold-to-fold spread inside support ${worst}` };
|
||||
});
|
||||
|
||||
// 9,10,17 — digests and field distinctness
|
||||
inject(9, 'final source digest ignores a source-set change',
|
||||
'src/services/model/currentEraSource.js',
|
||||
` .map((r) => [String(r.id), Number(r.p).toFixed(6), Number(r.won), String(r.date), String(r.model_version)])`,
|
||||
` .map((r) => [String(r.id)])`, 'tests/unit/currentEraSource.test.js');
|
||||
logic(10, 'knot output changes without an artifact identity change', () => {
|
||||
const cal = require(path.join(ROOT, 'src/services/model/calibration'));
|
||||
const d = (o) => crypto.createHash('sha256').update(JSON.stringify(o)).digest('hex').slice(0, 16);
|
||||
const m1 = cal.fitIsotonic(Array.from({ length: 600 }, (_, i) => ({ p: 0.4 + (i % 50) / 100, won: i % 3 ? 1 : 0, date: 'd' })), { minTotal: 200 });
|
||||
const m2 = cal.fitIsotonic(Array.from({ length: 600 }, (_, i) => ({ p: 0.4 + (i % 50) / 100, won: i % 4 ? 1 : 0, date: 'd' })), { minTotal: 200 });
|
||||
return { caught: d(m1) !== d(m2), detail: 'a different curve yields a different knot digest' };
|
||||
});
|
||||
inject(17, 'fit_as_of substituted for training_cutoff',
|
||||
'src/services/model/artifactRegistry.js',
|
||||
` const out = Object.freeze({
|
||||
...raw,`,
|
||||
` const out = Object.freeze({
|
||||
...raw,
|
||||
training_cutoff: raw.fit_as_of,`, CONTRACT);
|
||||
|
||||
// 11 — reconstruction (proven EXACT against production this run)
|
||||
logic(11, 'independent reconstruction differs', () => {
|
||||
const s = codeOf(fs.readFileSync(path.join(ROOT, 'scripts/verify-artifact-reconstruction.js'), 'utf8'));
|
||||
const independent = s.includes('src.loadRows') && s.includes('cal.fitIsotonic') && !s.includes('build-current-era-artifact');
|
||||
const exits = s.includes('process.exit(mismatches.length === 0 ? 0 : 1)');
|
||||
return { caught: independent && exits, detail: 'rebuilds from the declared contract and exits non-zero on any mismatch' };
|
||||
});
|
||||
|
||||
// 12,13,14,16 — freeze / promotion
|
||||
inject(12, 'the active artifact refits on a snapshot',
|
||||
'src/services/model/probabilityContractService.js',
|
||||
` const artifact = registry.load(sport, stat);`,
|
||||
` const artifact = registry.load(sport, stat);
|
||||
const _refit = require('./calibration').fitIsotonic([], {});`, GOV);
|
||||
logic(13, 'a new settlement mutates the active curve', () => {
|
||||
const s = codeOf(fs.readFileSync(path.join(ROOT, 'src/services/model/artifactRegistry.js'), 'utf8'));
|
||||
const readsFile = s.includes('fs.readFileSync');
|
||||
const noWrite = !/writeFileSync|\.update\(|\.upsert\(/.test(s);
|
||||
return { caught: readsFile && noWrite, detail: `registry reads a committed file and performs no write` };
|
||||
});
|
||||
inject(14, 'a candidate automatically promotes',
|
||||
'src/services/model/artifactRegistry.js',
|
||||
` return raw.artifact_id === promoted.artifact_id;`,
|
||||
` return true;`, GOV);
|
||||
logic(16, 'the old 65/35 permanent withholding survives under policy B', () => {
|
||||
return { caught: A.withheld_from_fit === 0 && A.fit_n === A.era_audit.current_era_rows,
|
||||
detail: `withheld_from_fit ${A.withheld_from_fit}; fit_n ${A.fit_n} == eligible ${A.era_audit.current_era_rows}` };
|
||||
});
|
||||
|
||||
// 18-22 — serving surfaces stay put
|
||||
logic(18, 'grade changes', () => {
|
||||
const s = codeOf(fs.readFileSync(path.join(ROOT, 'src/services/model/probabilityContract.js'), 'utf8'));
|
||||
return { caught: !/servedGrade|gradeFor/.test(s), detail: 'the probability contract does not reach the grade' };
|
||||
});
|
||||
logic(19, 'selected side changes', () => {
|
||||
const s = codeOf(fs.readFileSync(path.join(ROOT, 'src/services/model/probabilityContract.js'), 'utf8'));
|
||||
return { caught: !/\bside\b\s*=|gradeBestSide/.test(s), detail: 'the contract never assigns a side' };
|
||||
});
|
||||
logic(20, 'publication changes', () => {
|
||||
const s = fs.readFileSync(path.join(ROOT, 'src/services/retentionService.js'), 'utf8');
|
||||
const m = codeOf(s.slice(s.indexOf('function mergeProbabilityContract'), s.indexOf('function mergeChainShadow')));
|
||||
return { caught: !/published|publication_id|read_id|lineage/.test(m), detail: 'the merge touches no publication or lineage field' };
|
||||
});
|
||||
logic(21, 'shadow enabled in the release', () => {
|
||||
const s = fs.readFileSync(path.join(ROOT, 'src/services/model/probabilityContract.js'), 'utf8');
|
||||
const strict = s.includes("String(raw || '') === '1'");
|
||||
const noDefaultOn = !/PROBABILITY_CONTRACT_SHADOW\s*\|\|\s*'1'/.test(s);
|
||||
return { caught: strict && noDefaultOn, detail: 'shadow requires an explicit "1"; no default-on path' };
|
||||
});
|
||||
logic(22, 'live serving enabled', () => {
|
||||
const snap = codeOf(fs.readFileSync(path.join(ROOT, 'src/services/snapshotService.js'), 'utf8'));
|
||||
const deployedEmpty = /CALIBRATION_DEPLOYED\s*=\s*Object\.freeze\(\[\s*\]\)/.test(snap);
|
||||
const noLive = A.approved_for_live === false && registry.PROMOTED['mlb:hits'].stage === registry.STAGE.APPROVED_FOR_SHADOW;
|
||||
const files = execSync(`grep -rl "served_probability" ${ROOT}/src ${ROOT}/web/src 2>/dev/null || true`).toString().trim().split('\n').filter(Boolean)
|
||||
.map((f) => f.replace(ROOT + '/', ''));
|
||||
const allowed = ['src/services/model/probabilityContract.js', 'src/services/model/probabilityContractService.js', 'src/services/retentionService.js'];
|
||||
const leaked = files.filter((f) => !allowed.includes(f));
|
||||
return { caught: deployedEmpty && noLive && leaked.length === 0,
|
||||
detail: `CALIBRATION_DEPLOYED empty=${deployedEmpty}; stage=${registry.PROMOTED['mlb:hits'].stage}; leaked consumers: ${leaked.join(', ') || 'none'}` };
|
||||
});
|
||||
|
||||
// 23-26 — the frozen neighbours
|
||||
logic(23, 'retention identity changes', () => {
|
||||
const d = execSync(`git -C ${ROOT} diff --unified=0 -- src/services/retentionService.js`).toString();
|
||||
const fields = ['player_key:', 'snapshot_id:', 'canonical_event_id:', 'game_id:', 'stat:', 'line:', 'side:'];
|
||||
const touched = fields.filter((f) => new RegExp(`^[-+].*${f.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}`, 'm').test(d));
|
||||
return { caught: touched.length === 0, detail: `identity fields in diff: ${touched.join(', ') || 'none'}` };
|
||||
});
|
||||
logic(24, 'participant identity changes', () => {
|
||||
const f = execSync(`git -C ${ROOT} diff --name-only`).toString().trim().split('\n').filter(Boolean)
|
||||
.filter((x) => /participantIdentity|eventIdentity|matchupKeys|playerName/.test(x));
|
||||
return { caught: f.length === 0, detail: `participant files changed: ${f.join(', ') || 'none'}` };
|
||||
});
|
||||
logic(25, 'lineage mechanics/config change', () => {
|
||||
const f = execSync(`git -C ${ROOT} diff --name-only`).toString().trim().split('\n').filter(Boolean)
|
||||
.filter((x) => /lineage|readLineage|readAncestry|lineageWriteMode|lineageCoverage/i.test(x));
|
||||
const snapDiff = execSync(`git -C ${ROOT} diff -- src/services/snapshotService.js`).toString();
|
||||
const lines = snapDiff.split('\n').filter((l) => /^[-+]/.test(l) && /lineage|canary|LINEAGE_/i.test(l));
|
||||
return { caught: f.length === 0 && lines.length === 0, detail: `lineage files ${f.length}, lineage diff lines ${lines.length}` };
|
||||
});
|
||||
logic(26, 'PerformanceDistribution becomes servable', () => {
|
||||
const snap = codeOf(fs.readFileSync(path.join(ROOT, 'src/services/snapshotService.js'), 'utf8'));
|
||||
return { caught: !/chain\.chainAcross\(/.test(snap), detail: 'no chainAcross call' };
|
||||
});
|
||||
|
||||
const landed = results.filter((r) => r.landed).length;
|
||||
console.log(JSON.stringify({ teeth_landed: `${landed}/${results.length}`, results }, null, 2));
|
||||
process.exit(landed === results.length ? 0 : 1);
|
||||
@@ -71,21 +71,19 @@ logicTooth(1, 'runtime SHA verification waits for a snapshot despite working run
|
||||
|
||||
// ── 2 — the estimator must carry a reconstructable identity ──────────────
|
||||
injectionTooth(2, 'runtime estimator lacks a reconstructable artifact identity',
|
||||
'src/services/model/probabilityContractService.js',
|
||||
` knot_count: Array.isArray(fitted.map) ? fitted.map.length : null,
|
||||
knot_digest: digest(fitted.map),`,
|
||||
` knot_count: null,
|
||||
knot_digest: 'static-placeholder',`,
|
||||
'tests/unit/probabilityContract.test.js');
|
||||
'src/services/model/fitPolicy.js',
|
||||
` if (!artifact.knot_digest || !artifact.served_curve_digest) violations.push(VIOLATION.NO_ARTIFACT_IDENTITY);`,
|
||||
` if (false) violations.push(VIOLATION.NO_ARTIFACT_IDENTITY);`,
|
||||
'tests/unit/artifactGovernance.test.js');
|
||||
|
||||
// ── 3 — the artifact must be tied to the certified model era ─────────────
|
||||
injectionTooth(3, 'runtime artifact differs from the certified artifact',
|
||||
'src/services/model/probabilityContractService.js',
|
||||
` model_version: contract.model_version,
|
||||
fit_as_of: fitted.cutoff || null,`,
|
||||
` model_version: 'engine1@some-other-era',
|
||||
fit_as_of: fitted.cutoff || null,`,
|
||||
'tests/unit/probabilityContractShadow.test.js');
|
||||
'src/services/model/artifactRegistry.js',
|
||||
` estimator_type: 'isotonic',
|
||||
stage: promoted.stage,`,
|
||||
` estimator_type: 'low_param',
|
||||
stage: promoted.stage,`,
|
||||
'tests/unit/artifactGovernance.test.js');
|
||||
|
||||
// ── 4 — point-in-time evidence ───────────────────────────────────────────
|
||||
injectionTooth(4, 'dynamic refit uses future settlement evidence for an earlier Read',
|
||||
@@ -127,7 +125,7 @@ logicTooth(25, 'shadow flag parses loosely or defaults ON', () => {
|
||||
const strict = pcSrc.includes("String(raw || '') === '1'")
|
||||
&& snap.includes("probabilityContract').shadowState().shadow === 'ON'");
|
||||
const scoped = snap.includes("&& sp === 'mlb'");
|
||||
const statScoped = snap.includes("{ sport: 'mlb', stat: 'hits' }");
|
||||
const statScoped = snap.includes("sport: 'mlb', stat: 'hits'");
|
||||
return { caught: strict && scoped && statScoped,
|
||||
detail: `strict '1' compare=${strict}; sport-scoped=${scoped}; stat-scoped=${statScoped}` };
|
||||
});
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
/**
|
||||
* verify-artifact-reconstruction — rebuild the promoted artifact from its
|
||||
* DECLARED TRAINING CONTRACT and require exact equality.
|
||||
*
|
||||
* A fresh read, a fresh source digest, a fresh fit, fresh digests. Calling the
|
||||
* same builder twice would prove only that the builder is deterministic; this
|
||||
* proves the artifact is derivable from what it says about itself.
|
||||
*/
|
||||
require('dotenv').config({ quiet: true });
|
||||
const crypto = require('crypto');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cal = require('../src/services/model/calibration');
|
||||
const src = require('../src/services/model/currentEraSource');
|
||||
const registry = require('../src/services/model/artifactRegistry');
|
||||
|
||||
const digest = (o) => crypto.createHash('sha256').update(JSON.stringify(o)).digest('hex').slice(0, 16);
|
||||
|
||||
function servedCurve(map, bands) {
|
||||
const steps = []; let prev = null;
|
||||
for (const [lo, hi] of bands) {
|
||||
prev = null;
|
||||
for (let x = lo; x < hi - 1e-9; x += 0.001) {
|
||||
const raw = Math.round(x * 1000) / 1000;
|
||||
const v = cal.applyIsotonic(map, raw);
|
||||
if (v === null) continue;
|
||||
const r = Math.round(v * 1e6) / 1e6;
|
||||
if (r !== prev) { steps.push([raw, r]); prev = r; }
|
||||
}
|
||||
}
|
||||
return steps;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_KEY,
|
||||
{ auth: { persistSession: false } });
|
||||
const a = registry.load('mlb', 'hits');
|
||||
if (!a) throw new Error('no promoted artifact');
|
||||
|
||||
// ONLY what the artifact declares about its own training contract.
|
||||
const rows = await src.loadRows(sb, {
|
||||
sport: a.sport, stat: a.stat, modelVersion: a.model_version, before: a.fit_as_of,
|
||||
});
|
||||
rows.sort((x, y) => (x.date < y.date ? -1 : x.date > y.date ? 1 : (x.id < y.id ? -1 : 1)));
|
||||
const audit = src.eraAudit(rows, a.model_version);
|
||||
const map = cal.fitIsotonic(rows, { minTotal: 200 });
|
||||
const curve = servedCurve(map, a.certified_bands);
|
||||
|
||||
const got = {
|
||||
fit_n: rows.length,
|
||||
training_cutoff: rows.length ? rows[rows.length - 1].date : null,
|
||||
source_digest: src.sourceDigest(rows),
|
||||
knot_count: map ? map.length : null,
|
||||
knot_digest: digest(map),
|
||||
served_curve_digest: digest(curve),
|
||||
wrong_era_rows: audit.wrong_era_rows,
|
||||
};
|
||||
const want = {
|
||||
fit_n: a.fit_n, training_cutoff: a.training_cutoff, source_digest: a.source_digest,
|
||||
knot_count: a.knot_count, knot_digest: a.knot_digest,
|
||||
served_curve_digest: a.served_curve_digest, wrong_era_rows: 0,
|
||||
};
|
||||
const mismatches = Object.keys(want).filter((k) => got[k] !== want[k]);
|
||||
console.log(JSON.stringify({ artifact_id: a.artifact_id, want, got,
|
||||
curve_identical: JSON.stringify(curve) === JSON.stringify(a.served_curve),
|
||||
mismatches, RECONSTRUCTION: mismatches.length === 0 && JSON.stringify(curve) === JSON.stringify(a.served_curve)
|
||||
? 'EXACT' : 'DIFFERS' }, null, 1));
|
||||
process.exit(mismatches.length === 0 ? 0 : 1);
|
||||
})().catch((e) => { console.error(e.message); process.exit(1); });
|
||||
Reference in New Issue
Block a user