The live machinery ships dark, behind two gates that cannot substitute for each other
ATTRIBUTION RECEIPT (cohort 27ce152f, writer 94f7c3c, 01:02:19Z): 2,316 non-hits
rows — certified 0, numeric 0, and artifact_id / estimator / certification
version / source / knot / curve ALL ZERO. The hits slice held: 209 published,
201 certified, curve mismatches 0/201, artifact identity deviations 0, support
violations 0, raw erased 0. Shadow tranche frozen.
AUTHENTICATED NON-LEAK: obtained through the owner magic-link flow — the real
auth system, no bypass, no stored password, token never printed, session
discarded. /api/ledger, /api/ledger/accuracy, /api/preferences, /api/accuracy
and the authenticated /api/snapshot/mlb (677KB, full model fields) carry zero
shadow keys. One scanner hit adjudicated: `served_grade.calibrated` is false on
all 504 rows — servedGrade's hardcoded constant from before this work, a name
collision with my keyword list, not the shadow.
A REAL MONITOR DEFECT, asked for and found. The row floor alone let 400
observations from ONE slate reach HEALTHY or DRIFT — one correlated draw wearing
the costume of four hundred. The monitor now also requires settled DATES, and
the floor is not invented: it reads
`fitPolicy.POLICY_V1.certification.eval_block_dates`, the 3-date fold the
walk-forward was actually certified with. One date and two dates now return
INSUFFICIENT_SAMPLE regardless of row count.
That fix had a bug of its own that a test caught: `scored` never carried `date`,
so the distinct-date count read `undefined` for every row and always returned 1.
The gate looked correct while measuring nothing.
THE SEAM. EV, Kelly and VALUE are produced in exactly one place at grade time,
all from p_win, and every served row passes exactly one boundary on the way out
(`snapshotGating.stripModelPrice`, used by the snapshot route, hero route,
topGraded and props). So `servedProbability.applyToRows` sits there — one place,
not four call sites and four chances to miss one. Consumers never see the
artifact, the curve, the support region or the environment; a test forbids
`applyCurve`, `applyIsotonic`, `artifactRegistry` and `served_curve` in all three
serving consumers.
Placed BEFORE the tier strip deliberately: calibration decides what the number
IS, entitlement decides who may see it. Reversed, it would calibrate fields that
had already been removed.
TWO GATES, NEITHER SUFFICIENT. The artifact is promoted APPROVED_FOR_LIVE — stage
only, same id, same source/knot/curve digests, same cutoff, no refit. Behaviour
is still OFF because PROBABILITY_CONTRACT_LIVE is unset. Flag without approval:
OFF. Approval without flag: OFF. Both: ON. Live OFF returns rows byte-identical
and consults no artifact.
Two teeth had to be rewritten rather than satisfied: they pinned "artifact
unapproved" as the safety property, which would have blocked the deliberate
promotion. The property is that live BEHAVIOUR is off, and they now test that.
And my own splice while fixing one of them silently deleted ten newly-added
teeth — caught by counting ids, not by the runner going green.
Suite 406/406, 5,681 passed. Teeth 41/41 + 10/10 + 23/23. Live OFF.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
@@ -174,18 +174,22 @@ logic(21, 'shadow enabled in the release', () => {
|
||||
return { caught: strict && noDefaultOn, detail: 'shadow requires an explicit "1"; no default-on path' };
|
||||
});
|
||||
logic(22, 'live serving enabled', () => {
|
||||
const pcm = require(path.join(ROOT, 'src/services/model/probabilityContract'));
|
||||
const live = pcm.liveState({});
|
||||
const snap = codeOf(fs.readFileSync(path.join(ROOT, 'src/services/snapshotService.js'), 'utf8'));
|
||||
const deployedEmpty = /CALIBRATION_DEPLOYED\s*=\s*Object\.freeze\(\[\s*\]\)/.test(snap);
|
||||
const noLive = A.approved_for_live === false && registry.PROMOTED['mlb:hits'].stage === registry.STAGE.APPROVED_FOR_SHADOW;
|
||||
const files = execSync(`grep -rl "served_probability" ${ROOT}/src ${ROOT}/web/src 2>/dev/null || true`).toString().trim().split('\n').filter(Boolean)
|
||||
.map((f) => f.replace(ROOT + '/', ''));
|
||||
const allowed = ['src/services/model/probabilityContract.js', 'src/services/model/probabilityContractService.js', 'src/services/retentionService.js'];
|
||||
// the artifact IS promoted now, so the property under test is BEHAVIOUR:
|
||||
// live must resolve OFF, and the reason must be the unset runtime flag.
|
||||
const files = execSync(`grep -rl "served_probability" ${ROOT}/src ${ROOT}/web/src 2>/dev/null || true`)
|
||||
.toString().trim().split('\n').filter(Boolean).map((f) => f.replace(ROOT + '/', ''));
|
||||
const allowed = ['src/services/model/probabilityContract.js',
|
||||
'src/services/model/probabilityContractService.js', 'src/services/model/servedProbability.js',
|
||||
'src/services/retentionService.js'];
|
||||
const leaked = files.filter((f) => !allowed.includes(f));
|
||||
return { caught: deployedEmpty && noLive && leaked.length === 0,
|
||||
detail: `CALIBRATION_DEPLOYED empty=${deployedEmpty}; stage=${registry.PROMOTED['mlb:hits'].stage}; leaked consumers: ${leaked.join(', ') || 'none'}` };
|
||||
return { caught: deployedEmpty && live.live === 'OFF' && live.flag_set === false && leaked.length === 0,
|
||||
detail: `live=${live.live} (${live.blocked_reason}); CALIBRATION_DEPLOYED empty=${deployedEmpty}; leaked: ${leaked.join(', ') || 'none'}` };
|
||||
});
|
||||
|
||||
// 23-26 — the frozen neighbours
|
||||
logic(23, 'retention identity changes', () => {
|
||||
// BEHAVIOURAL, not a diff grep. The grep version fired on `stat: r.stat` —
|
||||
// a line that READS identity to pass it to a reader, not one that changes
|
||||
@@ -262,43 +266,67 @@ inject(30, 'the monitor scores evidence the fit already saw',
|
||||
'tests/unit/forwardMonitor.test.js');
|
||||
|
||||
logic(31, 'live serving turns on before all gates pass', () => {
|
||||
const snap = codeOf(fs.readFileSync(path.join(ROOT, 'src/services/snapshotService.js'), 'utf8'));
|
||||
const deployedEmpty = /CALIBRATION_DEPLOYED\s*=\s*Object\.freeze\(\[\s*\]\)/.test(snap);
|
||||
const stage = registry.PROMOTED['mlb:hits'].stage === registry.STAGE.APPROVED_FOR_SHADOW;
|
||||
const notLive = A.approved_for_live === false;
|
||||
return { caught: deployedEmpty && stage && notLive,
|
||||
detail: `CALIBRATION_DEPLOYED empty=${deployedEmpty}; stage=${registry.PROMOTED['mlb:hits'].stage}; approved_for_live=${A.approved_for_live}` };
|
||||
const pcm = require(path.join(ROOT, 'src/services/model/probabilityContract'));
|
||||
// BOTH gates are the property. The artifact is approved; the flag is not set;
|
||||
// therefore behaviour is off. Asserting "artifact unapproved" would have
|
||||
// blocked the deliberate promotion this tranche performed.
|
||||
const off = pcm.liveState({});
|
||||
const flagOnly = pcm.liveState({ PROBABILITY_CONTRACT_LIVE: '1' },
|
||||
{ load: () => ({ ...A, approved_for_live: false }) });
|
||||
const approvalOnly = pcm.liveState({});
|
||||
return { caught: off.live === 'OFF' && flagOnly.live === 'OFF' && approvalOnly.live === 'OFF'
|
||||
&& approvalOnly.artifact_approved_for_live === true,
|
||||
detail: `default OFF; flag-without-approval OFF; approval-without-flag OFF (approved=${approvalOnly.artifact_approved_for_live})` };
|
||||
});
|
||||
|
||||
// ── 32 — THE CROSS-STAT LEAK (found by the first real cohort) ────────────
|
||||
inject(32, 'the hits curve is applied to other stats in a mixed batch',
|
||||
'src/services/model/probabilityContractService.js',
|
||||
` resolve(read) {
|
||||
return pc.resolve(read,`,
|
||||
` resolve(read) {
|
||||
return pc.resolve({ ...read, sport, stat },`,
|
||||
GOV);
|
||||
inject(33, 'the merge drops the row identity the resolver needs',
|
||||
'src/services/retentionService.js',
|
||||
` res = contract.resolve({
|
||||
sport: r.sport, stat: r.stat,
|
||||
model_version: r.model_version, p_win: numOrNull(r.p_win),
|
||||
});`,
|
||||
` res = contract.resolve({ model_version: r.model_version, p_win: numOrNull(r.p_win) });`,
|
||||
GOV);
|
||||
inject(34, 'an artifact for another stat is accepted',
|
||||
// ── LIVE MACHINERY (dark) + MONITOR DATE-AWARENESS ───────────────────────
|
||||
inject(36, 'the runtime LIVE flag alone bypasses artifact approval',
|
||||
'src/services/model/probabilityContract.js',
|
||||
` || (deps.artifact.stat && deps.artifact.stat !== contract.stat))) {`,
|
||||
` || false)) {`,
|
||||
GOV);
|
||||
|
||||
inject(35, 'an UNSUPPORTED row is attributed to the hits artifact',
|
||||
` const on = flag && approved;`, ` const on = flag;`,
|
||||
'tests/unit/servedProbability.test.js');
|
||||
inject(37, 'artifact approval alone activates live behaviour',
|
||||
'src/services/model/probabilityContract.js',
|
||||
` estimator_type: null, estimator_version: null, certification_version: null,
|
||||
contract_model_version: null, artifact: null, artifact_id: null,
|
||||
procedure_version: null,`,
|
||||
``,
|
||||
GOV);
|
||||
` const on = flag && approved;`, ` const on = approved;`,
|
||||
'tests/unit/servedProbability.test.js');
|
||||
inject(38, 'live default is not OFF',
|
||||
'src/services/model/probabilityContract.js',
|
||||
` const flag = String(raw || '') === '1';`, ` const flag = String(raw ?? '1') !== '0';`,
|
||||
'tests/unit/servedProbability.test.js');
|
||||
inject(39, 'an uncertified row keeps a probability-derived claim',
|
||||
'src/services/model/servedProbability.js',
|
||||
` for (const f of DERIVED_FIELDS) if (f in out) out[f] = null;`, ``,
|
||||
'tests/unit/servedProbability.test.js');
|
||||
inject(40, 'a certified row derives EV from raw instead of served',
|
||||
'src/services/model/servedProbability.js',
|
||||
` out.ev_pct = derived.ev_pct;`, ` out.ev_pct = row.ev_pct;`,
|
||||
'tests/unit/servedProbability.test.js');
|
||||
inject(41, 'raw model probability is erased when live serves',
|
||||
'src/services/model/servedProbability.js',
|
||||
` const out = { ...row, raw_model_probability: resolution.raw_model_probability,`,
|
||||
` const out = { ...row, raw_model_probability: null,`,
|
||||
'tests/unit/servedProbability.test.js');
|
||||
inject(42, 'the serving boundary bypasses the seam',
|
||||
'src/utils/snapshotGating.js',
|
||||
` try { rows = require('../services/model/servedProbability').applyToRows(rows); }`,
|
||||
` try { rows = rows; }`,
|
||||
'tests/unit/servedProbability.test.js');
|
||||
inject(43, 'the monitor reaches a verdict from one settled date',
|
||||
'src/services/model/forwardMonitor.js',
|
||||
` if (settledDates.length < MIN_FORWARD_DATES) {`, ` if (false) {`,
|
||||
'tests/unit/forwardMonitor.test.js');
|
||||
inject(44, 'the date count is taken from rows that never carried a date',
|
||||
'src/services/model/forwardMonitor.js',
|
||||
` scored.push({ raw, served, won, date: String(r.date) });`,
|
||||
` scored.push({ raw, served, won });`,
|
||||
'tests/unit/forwardMonitor.test.js');
|
||||
logic(45, 'the stage promotion changed the artifact', () => {
|
||||
const A2 = registry.load('mlb', 'hits');
|
||||
return { caught: A2.artifact_id === 'mlb-hits-isotonic@2026-09-03'
|
||||
&& A2.knot_digest === '5ae940ea163b7da2'
|
||||
&& A2.served_curve_digest === 'c24a9dc5c2a96068'
|
||||
&& A2.training_cutoff === '2026-09-01' && A2.fit_n === 6069,
|
||||
detail: `id/knot/curve/cutoff/fit_n unchanged across promotion; stage=${A2.stage}` };
|
||||
});
|
||||
|
||||
const landed = results.filter((r) => r.landed).length;
|
||||
console.log(JSON.stringify({ teeth_landed: `${landed}/${results.length}`, results }, null, 2));
|
||||
|
||||
@@ -112,10 +112,14 @@ logicTooth(16, 'live serving activates before the shadow has passed', () => {
|
||||
const srcFiles = execSync(`grep -rl "served_probability" ${ROOT}/src ${ROOT}/web/src 2>/dev/null || true`)
|
||||
.toString().trim().split('\n').filter(Boolean).map((f) => f.replace(ROOT + '/', ''));
|
||||
const allowed = ['src/services/model/probabilityContract.js',
|
||||
'src/services/model/probabilityContractService.js', 'src/services/retentionService.js'];
|
||||
'src/services/model/probabilityContractService.js', 'src/services/model/servedProbability.js',
|
||||
'src/services/retentionService.js'];
|
||||
const leaked = srcFiles.filter((f) => !allowed.includes(f));
|
||||
return { caught: deployedEmpty && leaked.length === 0,
|
||||
detail: `CALIBRATION_DEPLOYED empty=${deployedEmpty}; served_probability referenced outside the contract layer: ${leaked.join(', ') || 'none'}` };
|
||||
// BEHAVIOUR, not stage: the artifact is deliberately promoted now, so the
|
||||
// property is that live still resolves OFF because the runtime flag is unset.
|
||||
const live = require(path.join(ROOT, 'src/services/model/probabilityContract')).liveState({});
|
||||
return { caught: deployedEmpty && leaked.length === 0 && live.live === 'OFF' && live.flag_set === false,
|
||||
detail: `live=${live.live} (${live.blocked_reason}); CALIBRATION_DEPLOYED empty=${deployedEmpty}; leaked: ${leaked.join(', ') || 'none'}` };
|
||||
});
|
||||
|
||||
// ── the shadow flag itself ───────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user