A probability is served because evidence supports it, not because nothing else answered

The band gate was blocked for its `else` branch. It read: candidate = F(raw) served = inCertifiedBand(candidate) ? candidate : RAW and above raw 0.60 the model is measured overconfident — holdout raw 0.80-0.90 predicts 0.843 and realizes 0.639. So "the calibrator is not supported here" was being answered with a number already proven wrong. Unsupported calibration does not make raw true. Four candidates were adjudicated on ONE split — fit on the earliest 60% of train, decide support on the last 40%, evaluate on a holdout that saw neither: A low-param 80.2% coverage 0.24374 REFUTED — its extra region (raw 0.80-0.90) certified on cert (err +0.040, n=55) and refuted on holdout (served 0.754 vs observed 0.639), and it leaves a hole at 0.70-0.80 while serving the island above it B isotonic 91.3% coverage 0.24337 CERTIFIED, contiguous raw [0.50,0.80) C empirical band 91.3% coverage 0.24335 REFUTED — refitted point-in-time on current-model hits the realized rates INVERT in grade order (B+ 0.593 < B 0.614 < C+ 0.623), so the served function steps down at raw 0.78. Its shipped constants come from 3,417 props pooled across four batter stats and do not reproduce here D raw identity 43.1% coverage 0.24866 certifies raw 0.50-0.60 and only there Raw is candidate D, not a fallback. It earns exactly one region (holdout error +0.010 on n=1,316), which is why the law is "raw must earn its region" rather than "raw is never true". B already covers that region, so no hybrid is built. Above raw 0.80 nothing is certified and nothing is served. That is the region where raw is most wrong, isotonic over-corrects (cert err -0.093) and its LODO mapping at 0.95 has spread 0.180. 8.7% of holdout rows land there. The registry did not need changing. `serves(stat, p)` already tested certified bands against the RAW p_win — support in the input domain, the correct question — and returned {serve:false, reason}. It never said "serve raw". The output-space gate and the raw fallback were both invented downstream in calibrationService. ACTIVATION IS OFF. PROBABILITY_CONTRACT_SHADOW defaults to 0, CALIBRATION_DEPLOYED stays frozen empty, and every served field is byte-identical. This releases the support first, which is the required order. The shadow records raw belief, the candidate served value, the state, the estimator identity, and what EV/Kelly/VALUE would be under the actionability law — into its own column, read by nothing. Migration 051 was applied to production BEFORE retentionService named the column. PostgREST builds a bulk insert from the first row's shape, so a key whose column does not exist 400s the whole batch silently — that is how migration 038 took retention down for three days. The user-facing contradiction is NOT fixed here. A B+ still says "realized about 66%" beside a confidence of 84. Fixing that is activation, and activation costs 32% of VALUE flags and 46% of Kelly recommendations on the holdout. Suite 401/401, 5,580 passed, 4 skipped, deterministic across three runs. Teeth 23/23, each independently injected and restored byte-identically. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
2026-09-02 21:03:16 -04:00
parent 9dda9df132
commit a8de676756
10 changed files with 1359 additions and 1 deletions
@@ -0,0 +1,126 @@
+'use strict';
+
+/**
+ * THE SHADOW MUST BE GENERATED, PERSISTED, AND HARMLESS.
+ *
+ * A5's whole finding was a factor that was built, correct, and never invoked.
+ * Evidence that is computed but never reaches a row is that same failure one
+ * layer along — so this drives the REAL retention row builder rather than
+ * asserting the merge function in isolation.
+ */
+const retention = require('../../src/services/retentionService');
+const pc = require('../../src/services/model/probabilityContract');
+
+const ERA = 'engine1@2026-08-07-fullwindow';
+const iso = (p) => Math.round((0.42 + 0.26 * p) * 1000) / 1000;
+const contract = {
+  fit_n: 1798, fitted_through: '2026-08-17',
+  resolve: (read) => pc.resolve({ ...read, sport: 'mlb', stat: 'hits' }, { estimate: iso }),
+};
+
+const row = (over) => ({
+  player_key: 'x', stat: 'hits', line: 0.5, side: 'over', model_version: ERA,
+  p_win: 0.65, confidence: 65, grade: 'C+', ev_pct: 12.3, value: true, takeable: true,
+  book_odds: -115, over_odds: -115, under_odds: -105, probability_contract: null, ...over,
+});
+
+describe('the shadow reaches the row', () => {
+  it('the REAL collector declares the column on every row, refusals included', () => {
+    const collector = retention.createCollector({
+      snapshotId: 'snap-1', sport: 'mlb', modelVersion: ERA, codeSha: 'test',
+      gameDate: '2026-09-01', gameIdFor: () => 'mlb:2026-09-01:AAA@BBB',
+    });
+    collector.onGraded(
+      { player: 'A Hitter', stat_type: 'hits', line: 0.5, sport: 'mlb',
+        over_odds: -115, under_odds: -105 },
+      [{ direction: 'over', grade: 'C+', p_win: 0.65, confidence: 65 },
+       { direction: 'under', insufficient_data: true }],
+    );
+    // the collector is the real path; if it collected nothing the assertion
+    // below would vacuously pass, so require the rows first
+    expect(collector.rows.length).toBe(2);
+    for (const r of collector.rows) expect('probability_contract' in r).toBe(true);
+    const merged = retention.mergeProbabilityContract(collector.rows, contract);
+    expect(merged.find((r) => r.side === 'over').probability_contract.probability_state)
+      .toBe(pc.STATE.CERTIFIED_CALIBRATED);
+  });
+
+  it('every row in a batch carries the key — a partial shape drops the column for all', () => {
+    const rows = [row(), row({ p_win: 0.91 }), row({ p_win: null, grade: null })];
+    const out = retention.mergeProbabilityContract(rows, contract);
+    expect(out).toHaveLength(3);
+    for (const r of out) expect('probability_contract' in r).toBe(true);
+  });
+
+  it('records the served candidate inside support', () => {
+    const [r] = retention.mergeProbabilityContract([row()], contract);
+    expect(r.probability_contract.probability_state).toBe(pc.STATE.CERTIFIED_CALIBRATED);
+    expect(r.probability_contract.served_probability).toBe(iso(0.65));
+    expect(r.probability_contract.estimator_type).toBe('isotonic');
+    expect(r.probability_contract.servable).toBe(false);
+  });
+
+  it('records the REFUSAL, and keeps the raw belief on it', () => {
+    const [r] = retention.mergeProbabilityContract([row({ p_win: 0.91 })], contract);
+    expect(r.probability_contract.probability_state).toBe(pc.STATE.UNCERTIFIED);
+    expect(r.probability_contract.served_probability).toBeNull();
+    expect(r.probability_contract.raw_model_probability).toBe(0.91);
+    expect(r.probability_contract.derived.available).toBe(false);
+  });
+
+  it('prices the SIDE, not the opposite bet', () => {
+    const { evPct } = require('../../src/utils/devig');
+    const [o] = retention.mergeProbabilityContract([row({ side: 'over', book_odds: null })], contract);
+    const [u] = retention.mergeProbabilityContract([row({ side: 'under', book_odds: null })], contract);
+    expect(o.probability_contract.derived.ev_pct).toBe(evPct(iso(0.65), -115));
+    expect(u.probability_contract.derived.ev_pct).toBe(evPct(iso(0.65), -105));
+  });
+});
+
+describe('the shadow changes NOTHING that is served', () => {
+  const SERVED = ['p_win', 'confidence', 'grade', 'ev_pct', 'value', 'takeable', 'side', 'line'];
+  it('leaves every served field byte-identical', () => {
+    const before = [row(), row({ p_win: 0.91, grade: 'B+' }), row({ side: 'under', p_win: 0.55 })];
+    const snapshot = JSON.parse(JSON.stringify(before));
+    const after = retention.mergeProbabilityContract(before, contract);
+    after.forEach((r, i) => { for (const k of SERVED) expect(r[k]).toEqual(snapshot[i][k]); });
+  });
+
+  it('does not mutate the input rows in place', () => {
+    const rows = [row()];
+    retention.mergeProbabilityContract(rows, contract);
+    expect(rows[0].probability_contract).toBeNull();
+  });
+
+  it('a null contract is a no-op, not a raw passthrough', () => {
+    const out = retention.mergeProbabilityContract([row()], null);
+    expect(out[0].probability_contract).toBeNull();
+  });
+
+  it('a throwing contract leaves the row intact rather than losing it', () => {
+    const bad = { resolve: () => { throw new Error('boom'); } };
+    const out = retention.mergeProbabilityContract([row()], bad);
+    expect(out).toHaveLength(1);
+    expect(out[0].p_win).toBe(0.65);
+    expect(out[0].probability_contract).toBeNull();
+  });
+});
+
+describe('the flag is OFF by default', () => {
+  it('snapshotService reads PROBABILITY_CONTRACT_SHADOW and defaults to off', () => {
+    const src = require('fs').readFileSync(
+      require('path').join(__dirname, '../../src/services/snapshotService.js'), 'utf8');
+    expect(src).toContain("process.env.PROBABILITY_CONTRACT_SHADOW || '') === '1'");
+    // and it must not be able to write a served field
+    const block = src.slice(src.indexOf('PROBABILITY CONTRACT SHADOW'), src.indexOf('let lineageIndex'));
+    for (const served of ['g.p_win =', 'g.confidence =', 'g.grade =', 'g.ev_pct =', 'g.value =']) {
+      expect(block).not.toContain(served);
+    }
+  });
+
+  it('nothing is added to CALIBRATION_DEPLOYED', () => {
+    const src = require('fs').readFileSync(
+      require('path').join(__dirname, '../../src/services/snapshotService.js'), 'utf8');
+    expect(src).toMatch(/CALIBRATION_DEPLOYED\s*=\s*Object\.freeze\(\[\s*\]\)/);
+  });
+});