Layer 3 Step 2: archetype-aware CHALLENGER, measured not claimed

The champion (probabilityEstimator -> p_win) keeps serving and grading users, completely unchanged. The challenger is a second probability computed from the same inputs at the same instant, landing on the same ledger row so it joins to the same outcome and the same close. Identical conditions, one difference — the only clean A/B. NOTHING IS CLAIMED. Running a challenger is honest beta; asserting it is better before the settled ledger says so is not. Promotion stays a later decision gated on Brier and calibration over sufficient segmented volume. INTERPRETABLE, NOT A RE-ESTIMATION. The challenger is the champion's probability adjusted by the Layer-2 axes, applied in log-odds space so a nudge cannot push past 0 or 1 and means the same thing at p=0.5 as at p=0.9. Every deviation is attributable to a named axis and a signed nudge, stored as challenger_adjustments, and the total is capped at 0.45 log-odds — a lean on a real signal, never a re-forecast. Only mechanically obvious stat/axis relationships are mapped; a speculative mapping would be the same guessing this layer exists to replace. IDENTICAL WHERE THERE IS NO SIGNAL, by construction. An unremarkable player, a thin sample, an unmapped stat or a missing classification all return the champion's probability byte-for-byte with an empty adjustment list and a stated reason. The experiment therefore differs only where archetype-awareness could possibly help or hurt, with no dilution from rows the treatment never touched. Induced on real players. Judge home runs over: 0.42 -> 0.447, via BOMBER +0.22 and WHIFF RISK -0.11 — two real opposing signals netting positive. The same prop under mirrors it exactly to -0.027. Judge strikeouts: delta exactly 0, because WHIFF RISK and GRINDER cancel — an honest "no lean" with both signals still recorded. Skubal strikeouts over: 0.60 -> 0.702 via WHIFF, TRAPDOOR and CANNON all aligned; his hits-allowed goes the other way, 0.50 -> 0.392, because a strikeout arm makes hits less likely. Josh Bell and a 12-PA sample are untouched. Isolation is structural: adjust() is pure, the champion field is read and never written, the served snapshot payload is still the untouched champion object, and a challenger failure is caught so it can never break the pipeline it is measured inside. Statcast aggregates load once per snapshot run rather than per prop, so grade-time I/O stays at zero. Migration 034. Tests 3634 passed / 295 suites, web build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCNgGSt5qvcLxaeQqa7Zpj
2026-07-20 23:53:32 -04:00
parent 80f7100fc3
commit f2da9dd7e8
5 changed files with 385 additions and 2 deletions
@@ -0,0 +1,149 @@
+/* ============================================================
+   Session 71 — CHAMPION / CHALLENGER.
+
+   The champion serves and grades users, unchanged. The challenger runs live and
+   is measured. These lock the three properties that make the A/B worth running:
+   isolation, interpretability, and identity-where-there-is-no-signal.
+   ============================================================ */
+
+const ch = require('../../src/services/challengerProjection');
+const axes = require('../../src/services/archetypeAxes');
+
+const JUDGE = axes.classifyPlayer({
+  role: 'batter', bats: 'R', sample_pa: 261, k_pct: 27.6, bb_pct: 16.1, chase_pct: 25.8,
+  barrel_pct: 21.7, hard_hit_pct: 57.3, avg_launch_angle: 14.6, sweet_spot_pct: 33.6,
+});
+const BELL = axes.classifyPlayer({
+  role: 'batter', bats: 'S', sample_pa: 387, k_pct: 21.7, bb_pct: 7.5, chase_pct: 30.6,
+  barrel_pct: 10.3, hard_hit_pct: 43.4, avg_launch_angle: 13.9, sweet_spot_pct: 33.8,
+});
+const SKUBAL = axes.classifyPlayer({
+  role: 'pitcher', role_detail: 'starter', throws: 'L', sample_ip: 82.2, k_pct: 30.5,
+  bb_pct: 3.4, chase_pct: 36.7, barrel_pct: 6.7, gb_pct: 49, fb_pct: 22.6, arm_angle: 46.9,
+  pitch_mix: [{ type: 'FF', velo: 96.7 }],
+});
+const THIN = axes.classifyPlayer({ role: 'batter', sample_pa: 12, barrel_pct: 30 });
+
+describe('identical where there is no signal — the clean-experiment property', () => {
+  it.each([
+    ['unremarkable player', BELL, 'no_distinctive_axis_for_stat'],
+    ['thin sample', THIN, 'archetype_absent_or_thin'],
+    ['no classification', null, 'archetype_absent_or_thin'],
+  ])('%s → challenger === champion, exactly', (_n, cls, reason) => {
+    const r = ch.adjust({ pWin: 0.42, direction: 'over', statType: 'home_runs', classification: cls });
+    expect(r.p_win_challenger).toBe(0.42);
+    expect(r.delta).toBe(0);
+    expect(r.adjustments).toEqual([]);
+    expect(r.reason).toBe(reason);
+  });
+
+  it('an unmapped stat leaves the champion untouched — no speculative mapping', () => {
+    const r = ch.adjust({ pWin: 0.5, direction: 'over', statType: 'nonsense_stat', classification: JUDGE });
+    expect(r.p_win_challenger).toBe(0.5);
+    expect(r.reason).toBe('stat_not_mapped');
+  });
+
+  it('no champion probability → nothing to adjust', () => {
+    for (const p of [null, undefined, 0, 1]) {
+      expect(ch.adjust({ pWin: p, direction: 'over', statType: 'hits', classification: JUDGE }).reason)
+        .toBe('no_champion_probability');
+    }
+  });
+});
+
+describe('interpretable — every difference is attributable', () => {
+  it('Judge HR: BOMBER lifts, WHIFF RISK offsets, and both are named', () => {
+    const r = ch.adjust({ pWin: 0.42, direction: 'over', statType: 'home_runs', classification: JUDGE });
+    const labels = r.adjustments.map((a) => a.label);
+    expect(labels).toContain('BOMBER');
+    expect(labels).toContain('WHIFF RISK');
+    expect(r.adjustments.find((a) => a.label === 'BOMBER').nudge).toBeGreaterThan(0);
+    expect(r.adjustments.find((a) => a.label === 'WHIFF RISK').nudge).toBeLessThan(0);
+    expect(r.p_win_challenger).toBeGreaterThan(0.42);
+  });
+
+  it('Skubal strikeouts: WHIFF + TRAPDOOR + CANNON all push the same way', () => {
+    const r = ch.adjust({ pWin: 0.6, direction: 'over', statType: 'strikeouts', classification: SKUBAL });
+    expect(r.p_win_challenger).toBeGreaterThan(0.6);
+    expect(r.adjustments.every((a) => a.nudge > 0)).toBe(true);
+  });
+
+  it('a strikeout arm makes hits-allowed LESS likely — sign is per stat', () => {
+    const r = ch.adjust({ pWin: 0.5, direction: 'over', statType: 'hits_allowed', classification: SKUBAL });
+    expect(r.p_win_challenger).toBeLessThan(0.5);
+  });
+
+  it('opposing signals can cancel to exactly zero — an honest "no lean"', () => {
+    const r = ch.adjust({ pWin: 0.55, direction: 'over', statType: 'strikeouts', classification: JUDGE });
+    expect(r.delta).toBe(0);
+    expect(r.adjustments.length).toBe(2); // both still recorded
+  });
+});
+
+describe('direction and bounds', () => {
+  it('UNDER mirrors OVER exactly', () => {
+    const over = ch.adjust({ pWin: 0.42, direction: 'over', statType: 'home_runs', classification: JUDGE });
+    const under = ch.adjust({ pWin: 0.58, direction: 'under', statType: 'home_runs', classification: JUDGE });
+    expect(under.delta).toBeCloseTo(-over.delta, 3);
+  });
+
+  it('log-odds space keeps every output a valid probability', () => {
+    for (const p of [0.02, 0.5, 0.98]) {
+      const r = ch.adjust({ pWin: p, direction: 'over', statType: 'strikeouts', classification: SKUBAL });
+      expect(r.p_win_challenger).toBeGreaterThan(0);
+      expect(r.p_win_challenger).toBeLessThan(1);
+    }
+  });
+
+  it('the total nudge is capped — a stack of axes cannot re-forecast', () => {
+    const r = ch.adjust({ pWin: 0.5, direction: 'over', statType: 'strikeouts', classification: SKUBAL });
+    expect(Math.abs(r.delta)).toBeLessThan(0.12); // a lean, not a re-forecast
+  });
+});
+
+describe('isolation — the champion is never touched', () => {
+  it('adjust() is pure: same input, same output, no shared state', () => {
+    const a = ch.adjust({ pWin: 0.42, direction: 'over', statType: 'home_runs', classification: JUDGE });
+    const b = ch.adjust({ pWin: 0.42, direction: 'over', statType: 'home_runs', classification: JUDGE });
+    expect(a).toEqual(b);
+  });
+
+  it('attachChallenger preserves p_win byte-for-byte on every grade', () => {
+    const grades = [
+      { player: 'Aaron Judge', stat_type: 'home_runs', direction: 'over', p_win: 0.42, grade: 'B' },
+      { player: 'Josh Bell', stat_type: 'home_runs', direction: 'over', p_win: 0.33, grade: 'C' },
+    ];
+    const out = ch.attachChallenger(grades, (n) => (n === 'Aaron Judge' ? JUDGE : BELL));
+    expect(out[0].p_win).toBe(0.42);
+    expect(out[1].p_win).toBe(0.33);
+    expect(out[0].p_win_challenger).toBeGreaterThan(0.42); // moved
+    expect(out[1].p_win_challenger).toBe(0.33);            // identical
+    expect(out[1].challenger_delta).toBe(0);
+  });
+
+  it('stamps a version so a future adjustment is distinguishable', () => {
+    const out = ch.attachChallenger([{ player: 'x', stat_type: 'hits', p_win: 0.5 }], () => null);
+    expect(out[0].challenger_version).toBe(ch.CHALLENGER_VERSION);
+  });
+});
+
+describe('retention + serving contract', () => {
+  const fs = require('fs');
+  it('the ledger retains the challenger beside the champion on one row', () => {
+    const src = fs.readFileSync(require.resolve('../../src/services/ledgerService'), 'utf8');
+    expect(src).toMatch(/p_win_challenger: numOrNull\(g\.p_win_challenger\)/);
+    expect(src).toMatch(/challenger_adjustments/);
+  });
+
+  it('the snapshot writes the CHALLENGER-carrying grades but serves `enriched`', () => {
+    const src = fs.readFileSync(require.resolve('../../src/services/snapshotService'), 'utf8');
+    expect(src).toMatch(/recordPipelineGrades\(sp, withChallenger, props/);
+    // the served snapshot payload is still the untouched champion object
+    expect(src).toMatch(/grades: enriched/);
+  });
+
+  it('a challenger failure can never break the pipeline it is measured inside', () => {
+    const src = fs.readFileSync(require.resolve('../../src/services/snapshotService'), 'utf8');
+    expect(src).toMatch(/\[challenger\] \$\{sp\} skipped/);
+  });
+});