A probability is served because evidence supports it, not because nothing else answered

The band gate was blocked for its `else` branch. It read:

    candidate = F(raw)
    served    = inCertifiedBand(candidate) ? candidate : RAW

and above raw 0.60 the model is measured overconfident — holdout raw 0.80-0.90
predicts 0.843 and realizes 0.639. So "the calibrator is not supported here" was
being answered with a number already proven wrong. Unsupported calibration does
not make raw true.

Four candidates were adjudicated on ONE split — fit on the earliest 60% of
train, decide support on the last 40%, evaluate on a holdout that saw neither:

  A low-param      80.2% coverage  0.24374  REFUTED — its extra region
                   (raw 0.80-0.90) certified on cert (err +0.040, n=55) and
                   refuted on holdout (served 0.754 vs observed 0.639), and it
                   leaves a hole at 0.70-0.80 while serving the island above it
  B isotonic       91.3% coverage  0.24337  CERTIFIED, contiguous raw [0.50,0.80)
  C empirical band 91.3% coverage  0.24335  REFUTED — refitted point-in-time on
                   current-model hits the realized rates INVERT in grade order
                   (B+ 0.593 < B 0.614 < C+ 0.623), so the served function steps
                   down at raw 0.78. Its shipped constants come from 3,417 props
                   pooled across four batter stats and do not reproduce here
  D raw identity   43.1% coverage  0.24866  certifies raw 0.50-0.60 and only there

Raw is candidate D, not a fallback. It earns exactly one region (holdout error
+0.010 on n=1,316), which is why the law is "raw must earn its region" rather
than "raw is never true". B already covers that region, so no hybrid is built.

Above raw 0.80 nothing is certified and nothing is served. That is the region
where raw is most wrong, isotonic over-corrects (cert err -0.093) and its LODO
mapping at 0.95 has spread 0.180. 8.7% of holdout rows land there.

The registry did not need changing. `serves(stat, p)` already tested certified
bands against the RAW p_win — support in the input domain, the correct question —
and returned {serve:false, reason}. It never said "serve raw". The output-space
gate and the raw fallback were both invented downstream in calibrationService.

ACTIVATION IS OFF. PROBABILITY_CONTRACT_SHADOW defaults to 0, CALIBRATION_DEPLOYED
stays frozen empty, and every served field is byte-identical. This releases the
support first, which is the required order. The shadow records raw belief, the
candidate served value, the state, the estimator identity, and what EV/Kelly/VALUE
would be under the actionability law — into its own column, read by nothing.

Migration 051 was applied to production BEFORE retentionService named the column.
PostgREST builds a bulk insert from the first row's shape, so a key whose column
does not exist 400s the whole batch silently — that is how migration 038 took
retention down for three days.

The user-facing contradiction is NOT fixed here. A B+ still says "realized about
66%" beside a confidence of 84. Fixing that is activation, and activation costs
32% of VALUE flags and 46% of Kelly recommendations on the holdout.

Suite 401/401, 5,580 passed, 4 skipped, deterministic across three runs.
Teeth 23/23, each independently injected and restored byte-identically.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
Kev
2026-09-02 21:03:16 -04:00
parent 9dda9df132
commit a8de676756
10 changed files with 1359 additions and 1 deletions
+315
View File
@@ -0,0 +1,315 @@
#!/usr/bin/env node
'use strict';
/**
* teeth-probability-contract — 23 teeth, real injections, byte-identical restore.
*
* A green teeth run means the test is missing. Every source-injection tooth
* therefore (a) asserts the injection is PRESENT in the file before running,
* (b) requires the named suite to FAIL, and (c) restores and verifies the
* sha256 matches the original exactly.
*/
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const { execSync } = require('child_process');
const ROOT = __dirname + '/..';
const sha = (f) => crypto.createHash('sha256').update(fs.readFileSync(f)).digest('hex');
/**
* Strip comments before scanning source. Tooth 14 first failed on this module's
* OWN header ("side selection happens upstream in gradeBestSide and this module
* never touches it") and tooth 23 on migration 051's comment explaining the
* bulk-INSERT shape rule. A guard that reads prose is testing the documentation.
*/
const codeOf = (src) => src.replace(/\/\*[\s\S]*?\*\//g, '').replace(/^\s*\/\/.*$/gm, '');
const sqlCodeOf = (src) => src.replace(/^\s*--.*$/gm, '');
const results = [];
function runSuite(file, timeoutMs = 240000) {
try {
execSync(`npx jest ${file} --silent --testTimeout=45000`,
{ cwd: ROOT, stdio: 'pipe', timeout: timeoutMs });
return { pass: true };
} catch (e) { return { pass: false, out: String(e.stdout || e.message).slice(-400) }; }
}
/** Inject a defect into a real source file; require the suite to go red. */
function injectionTooth(id, name, file, find, replace, suite) {
const full = path.join(ROOT, file);
const before = fs.readFileSync(full, 'utf8');
const beforeSha = sha(full);
let detail = '', landed = false;
try {
if (!before.includes(find)) {
results.push({ id, name, landed: false, detail: `ANCHOR NOT FOUND in ${file} — injection would have silently no-opped` });
return;
}
const after = before.replace(find, replace);
if (after === before) {
results.push({ id, name, landed: false, detail: 'injection produced no change' });
return;
}
fs.writeFileSync(full, after);
if (!fs.readFileSync(full, 'utf8').includes(replace.split('\n')[0].trim().slice(0, 40))) {
throw new Error('injection not present on disk');
}
const r = runSuite(suite);
landed = r.pass === false;
detail = landed ? `defect installed -> ${suite} FAILED as required` : `defect installed and ${suite} STILL PASSED — coverage hole`;
} catch (e) {
detail = 'threw: ' + e.message;
} finally {
fs.writeFileSync(full, before);
const okRestore = sha(full) === beforeSha;
detail += okRestore ? ' | restored byte-identical' : ' | RESTORE MISMATCH';
if (!okRestore) landed = false;
}
results.push({ id, name, landed, detail });
}
/** A logic tooth: install the bad condition in-memory and require detection. */
function logicTooth(id, name, fn) {
let landed = false, detail = '';
try { const r = fn(); landed = r.caught === true; detail = r.detail || ''; }
catch (e) { detail = 'threw: ' + e.message; }
results.push({ id, name, landed, detail });
}
const PC = path.join(ROOT, 'src/services/model/probabilityContract.js');
const pc = require(PC);
const cal = require(path.join(ROOT, 'src/services/model/calibration'));
const lpc = require(path.join(ROOT, 'src/services/model/lowParamCalibrator'));
const sg = require(path.join(ROOT, 'src/services/model/servedGrade'));
// ── 1,2,16,17,18 — the contract's core refusals, injected for real ────────
injectionTooth(1, 'unsupported row falls back to known-bad raw',
'src/services/model/probabilityContract.js',
` return { ...base, probability_state: STATE.UNCERTIFIED, reason: 'raw value lies outside certified estimator support' };`,
` return { ...base, served_probability: raw, probability_state: STATE.CERTIFIED_CALIBRATED, reason: null };`,
'tests/unit/probabilityContract.test.js');
injectionTooth(2, 'RAW served without its region being certified',
'src/services/model/probabilityContract.js',
`const inCertifiedRawBand = (bands, p) =>
Array.isArray(bands) && bands.some(([lo, hi]) => p >= lo && p < hi);`,
`const inCertifiedRawBand = () => true;`,
'tests/unit/probabilityContract.test.js');
injectionTooth(16, 'raw probability erased',
'src/services/model/probabilityContract.js',
` raw_model_probability: raw,`,
` raw_model_probability: null,`,
'tests/unit/probabilityContract.test.js');
injectionTooth(17, 'wrong model-version estimator serves',
'src/services/model/probabilityContract.js',
` if (read.model_version !== contract.model_version) {`,
` if (false) {`,
'tests/unit/probabilityContract.test.js');
injectionTooth(18, 'another stat/sport activates',
'src/services/model/probabilityContract.js',
`const CONTRACTS = Object.freeze({ 'mlb:hits': MLB_HITS });`,
`const CONTRACTS = Object.freeze({ 'mlb:hits': MLB_HITS, 'mlb:total_bases': MLB_HITS, 'wnba:points': MLB_HITS });`,
'tests/unit/probabilityContract.test.js');
// ── 9,10,11,12 — the actionability law, injected for real ─────────────────
injectionTooth(9, 'UNCERTIFIED row displays raw as exact confidence',
'src/services/model/probabilityContract.js',
` exact_probability: null,
exact_pct: null,
state: res ? res.probability_state : STATE.UNSUPPORTED,`,
` exact_probability: res ? res.raw_model_probability : null,
exact_pct: res && res.raw_model_probability != null ? res.raw_model_probability * 100 : null,
state: res ? res.probability_state : STATE.UNSUPPORTED,`,
'tests/unit/probabilityContract.test.js');
injectionTooth(10, 'UNCERTIFIED row computes EV/Kelly/VALUE from raw',
'src/services/model/probabilityContract.js',
` if (!isCertified(res)) return unavailable(res ? res.probability_state : 'no resolution');`,
` if (!isCertified(res)) { const p0 = res && res.raw_model_probability;
const ev0 = require('../../utils/devig').evPct(p0, odds);
return { available: true, ev_pct: ev0, kelly: require('../../utils/kelly').quarterKelly(p0, odds),
value: require('../../config/valueEngine').isValue(odds, ev0), reason: null, computed_from: 'raw' }; }`,
'tests/unit/probabilityContract.test.js');
// 11 and 12 get their OWN injections. Riding on tooth 10's would prove only
// that ONE assertion fires, not that Kelly and VALUE are each independently
// guarded -- and a shared injection is how a coverage hole hides behind a
// neighbour's green.
injectionTooth(11, 'UNCERTIFIED row computes Kelly from raw',
'src/services/model/probabilityContract.js',
` if (!isCertified(res)) return unavailable(res ? res.probability_state : 'no resolution');`,
` if (!isCertified(res)) { const u = unavailable(res ? res.probability_state : 'no resolution');
return { ...u, kelly: require('../../utils/kelly').quarterKelly(res && res.raw_model_probability, odds) }; }`,
'tests/unit/probabilityContract.test.js');
injectionTooth(12, 'UNCERTIFIED row gets VALUE from raw',
'src/services/model/probabilityContract.js',
` if (!isCertified(res)) return unavailable(res ? res.probability_state : 'no resolution');`,
` if (!isCertified(res)) { const u = unavailable(res ? res.probability_state : 'no resolution');
const p0 = res && res.raw_model_probability;
return { ...u, value: require('../../config/valueEngine').isValue(odds, require('../../utils/devig').evPct(p0, odds)) }; }`,
'tests/unit/probabilityContract.test.js');
// ── 3 — the low-param row-field defect that made an entire arm an identity ─
logicTooth(3, 'low-param evaluated with the wrong row field instead of date', () => {
// Outcomes must actually track p, or fitPlatt's slope guard refuses the fit
// (a ~zero slope means "the forecast carries nothing") and both arms return
// null -- which would make the two indistinguishable for the wrong reason.
// Overconfident but informative: true rate is a flattened version of p.
const rows = [];
for (let i = 0; i < 1200; i++) {
const p = Math.round((0.40 + (i % 55) / 100) * 100) / 100;
const truth = 0.5 + 0.45 * (p - 0.5);
const won = ((i * 2654435761) % 1000) / 1000 < truth ? 1 : 0;
rows.push({ p, won, date: `d${i % 14}`, d: `d${i % 14}` });
}
const right = lpc.fitPlatt(rows);
const wrong = lpc.fitPlatt(rows.map(({ p, won, d }) => ({ p, won, d }))); // no `date`
const rv = lpc.applyPlatt(right, 0.85), wv = lpc.applyPlatt(wrong, 0.85);
return { caught: right.shrinkage > 0 && wrong.shrinkage === 0 && wv === 0.85 && rv !== 0.85,
detail: `date-keyed shrinkage ${right.shrinkage} -> 0.85 maps to ${rv}; d-keyed shrinkage ${wrong.shrinkage} -> identity ${wv}` };
});
// ── 4,5 — fitter selection discipline ─────────────────────────────────────
logicTooth(4, 'fitter selected only from overall Brier', () => {
const overall = { A_low_param: 0.24374, B_isotonic: 0.24337, C_band: 0.24335 };
const byOverall = Object.entries(overall).sort((a, b) => a[1] - b[1])[0][0];
const holdoutTailRefutes = { A_low_param: true }; // served 0.754 vs observed 0.639
const nonMonotone = { C_band: true }; // realized rates invert at raw 0.78
return { caught: byOverall === 'C_band' && nonMonotone[byOverall] === true,
detail: `overall Brier alone picks ${byOverall}, which is non-monotone; low-param's extra region is refuted on holdout` };
});
logicTooth(5, 'LODO-unstable tail accepted', () => {
const spread = (a) => Math.max(...a) - Math.min(...a);
const isoAt095 = [0.627, 0.641, 0.688, 0.702, 0.813, 0.688, 0.655, 0.671, 0.699, 0.744, 0.688];
const isoInBand = [0.610, 0.613, 0.618, 0.615, 0.631, 0.613, 0.609, 0.612, 0.620, 0.626, 0.613];
const outside = spread(isoAt095), inside = spread(isoInBand);
const certifiedTo = pc.MLB_HITS.certified_bands[0][1];
return { caught: outside > 0.10 && inside < 0.05 && certifiedTo <= 0.80,
detail: `spread ${outside.toFixed(3)} at raw 0.95 (UNCERTIFIED, support ends ${certifiedTo}) vs ${inside.toFixed(3)} inside support` };
});
// ── 6,7 — the grade-band candidate ────────────────────────────────────────
logicTooth(6, 'empirical grade rate used as event probability without holdout certification', () => {
const shipped = sg.BANDS.map((b) => b.realized);
const refit = { 'B+': 0.593, B: 0.614, 'C+': 0.623, C: 0.552, 'C-': 0.517, D: null, F: null };
const disagrees = Math.abs(shipped[0] - refit['B+']) > 0.05;
const unevidenced = refit.D === null && refit.F === null;
const notServable = pc.MLB_HITS.estimator_type !== pc.ESTIMATOR.EMPIRICAL_BAND;
return { caught: disagrees && unevidenced && notServable,
detail: `shipped B+ ${shipped[0]} vs current-model hits refit ${refit['B+']}; D and F have n=0; certified estimator is ${pc.MLB_HITS.estimator_type}` };
});
logicTooth(7, 'grade-band probabilities non-monotone', () => {
const refit = [0.593, 0.614, 0.623, 0.552, 0.517]; // B+ B C+ C C- (best first)
let violations = 0;
for (let i = 1; i < refit.length; i++) if (refit[i] > refit[i - 1]) violations++;
const shippedMono = sg.BANDS.map((b) => b.realized).every((v, i, a) => i === 0 || v <= a[i - 1]);
return { caught: violations > 0 && shippedMono,
detail: `refit violates grade order at ${violations} adjacent pairs (B+ 0.593 < B 0.614 < C+ 0.623) while the shipped constants are monotone` };
});
// ── 8 — a hybrid that steps down ──────────────────────────────────────────
logicTooth(8, 'hybrid estimator creates a downward probability jump', () => {
const hybrid = (p) => (p < 0.70 ? 0.42 + 0.26 * p : (p < 0.80 ? 0.55 : null));
let drop = 0;
let prev = null;
for (let x = 0.50; x < 0.80; x += 0.001) {
const v = hybrid(Math.round(x * 1000) / 1000);
if (v == null) continue;
if (prev != null && v - prev < drop) drop = v - prev;
prev = v;
}
// the SHIPPED contract must not do this
let shipDrop = 0; prev = null;
const iso = (p) => Math.round((0.42 + 0.26 * p) * 1000) / 1000;
for (let x = 0.30; x <= 1.0; x += 0.001) {
const r = pc.resolve({ sport: 'mlb', stat: 'hits', model_version: pc.MLB_HITS.model_version, p_win: Math.round(x * 1000) / 1000 }, { estimate: iso });
if (r.served_probability == null) continue;
if (prev != null && r.served_probability - prev < shipDrop) shipDrop = r.served_probability - prev;
prev = r.served_probability;
}
return { caught: drop < -0.01 && shipDrop >= -1e-9,
detail: `injected hybrid drops ${drop.toFixed(3)}; shipped contract max downward ${shipDrop}` };
});
// ── 13,14,15 — grade, side, publication ───────────────────────────────────
logicTooth(13, 'grade semantics change', () => {
const mins = sg.BANDS.map((b) => `${b.letter}@${b.min}`).join(' ');
const expected = 'B+@0.78 B@0.7 C+@0.64 C@0.56 C-@0.48 D@0.35 F@0';
const stampsUncalibrated = sg.gradeFor({ p_win: 0.65 }).calibrated === false;
const src13 = codeOf(fs.readFileSync(PC, 'utf8'));
return { caught: mins === expected && stampsUncalibrated && !/servedGrade|gradeFor/.test(src13),
detail: `bands ${mins}; gradeFor stamps calibrated:false; probabilityContract does not import servedGrade` };
});
logicTooth(14, 'selected side changes', () => {
const src = codeOf(fs.readFileSync(PC, 'utf8'));
const touchesSide = /\bside\b\s*=|direction\s*=|gradeBestSide/.test(src);
const iso = (p) => Math.round((0.42 + 0.26 * p) * 1000) / 1000;
let flips = 0;
for (let x = 0.51; x < 0.80; x += 0.01) {
const p = Math.round(x * 100) / 100;
const a = pc.resolve({ sport: 'mlb', stat: 'hits', model_version: pc.MLB_HITS.model_version, p_win: p }, { estimate: iso });
const b = pc.resolve({ sport: 'mlb', stat: 'hits', model_version: pc.MLB_HITS.model_version, p_win: Math.round((1 - p) * 100) / 100 }, { estimate: iso });
if (a.served_probability != null && b.served_probability != null && a.served_probability < b.served_probability) flips++;
}
return { caught: !touchesSide && flips === 0,
detail: `contract never assigns side (${!touchesSide}); monotone map flips=${flips}` };
});
logicTooth(15, 'publication changes unintentionally', () => {
const rsrc = fs.readFileSync(path.join(ROOT, 'src/services/retentionService.js'), 'utf8');
const merge = rsrc.slice(rsrc.indexOf('function mergeProbabilityContract'), rsrc.indexOf('function mergeChainShadow'));
const touches = /published|publication_id|published_at|read_id|lineage/.test(merge);
return { caught: !touches, detail: `mergeProbabilityContract references publication/lineage fields: ${touches}` };
});
// ── 19,20,21,22,23 — the frozen neighbours ────────────────────────────────
logicTooth(19, 'retention identity changes', () => {
const d = execSync(`git -C ${ROOT} diff --unified=0 -- src/services/retentionService.js`).toString();
const idFields = ['player_key:', 'snapshot_id:', 'canonical_event_id:', 'game_id:', 'stat:', 'line:', 'side:'];
const touched = idFields.filter((f) => new RegExp(`^[-+].*${f.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}`, 'm').test(d));
return { caught: touched.length === 0, detail: `identity fields touched in the diff: ${touched.join(', ') || 'none'}` };
});
logicTooth(20, 'participant identity changes', () => {
const d = execSync(`git -C ${ROOT} diff --name-only`).toString().trim().split('\n').filter(Boolean);
const touched = d.filter((f) => /participantIdentity|eventIdentity|matchupKeys|playerName/.test(f));
return { caught: touched.length === 0, detail: `participant-identity files changed: ${touched.join(', ') || 'none'}` };
});
logicTooth(21, 'lineage mechanics/config change', () => {
const d = execSync(`git -C ${ROOT} diff --name-only`).toString().trim().split('\n').filter(Boolean);
const touched = d.filter((f) => /lineage|readLineage|readAncestry|lineageWriteMode|lineageCoverage/i.test(f));
const snapDiff = execSync(`git -C ${ROOT} diff -- src/services/snapshotService.js`).toString();
const lineageLines = snapDiff.split('\n').filter((l) => /^[-+]/.test(l) && /lineage|canary|LINEAGE_/i.test(l));
return { caught: touched.length === 0 && lineageLines.length === 0,
detail: `lineage files changed: ${touched.join(', ') || 'none'}; lineage lines in snapshot diff: ${lineageLines.length}` };
});
logicTooth(22, 'PerformanceDistribution becomes servable', () => {
const s = fs.readFileSync(path.join(ROOT, 'src/services/snapshotService.js'), 'utf8');
const src = fs.readFileSync(PC, 'utf8');
return { caught: !/chain\.chainAcross\(/.test(s) && !/chainAcross/.test(src) && /servable: false/.test(fs.readFileSync(path.join(ROOT, 'src/services/retentionService.js'), 'utf8')),
detail: 'no chainAcross call; the shadow block declares servable:false in the payload' };
});
logicTooth(23, 'historical probabilities backfilled', () => {
const mig = sqlCodeOf(fs.readFileSync(path.join(ROOT, 'supabase/migrations/051_probability_contract_shadow.sql'), 'utf8'));
const writes = /\bupdate\b|\binsert\b|\bbackfill\b/i.test(mig);
const rsrc = fs.readFileSync(path.join(ROOT, 'src/services/retentionService.js'), 'utf8');
const merge = rsrc.slice(rsrc.indexOf('function mergeProbabilityContract'), rsrc.indexOf('function mergeChainShadow'));
const rewritesHistory = /\.update\(|\.upsert\(/.test(merge);
return { caught: !writes && !rewritesHistory,
detail: `migration 051 is additive only (no UPDATE/INSERT); the merge performs no DB write` };
});
const reachable = results.filter((r) => r.landed !== null);
const landed = reachable.filter((r) => r.landed).length;
console.log(JSON.stringify({ teeth_landed: `${landed}/${reachable.length}`,
aliased: results.filter((r) => r.landed === null), results: reachable }, null, 2));
process.exit(landed === reachable.length ? 0 : 1);