runs + RBI: mostly-base-rate confirmed, and one level deeper than expected

Nothing proved. For RBI even the ARCHETYPE split is theatre, so the honest grade is the POOLED base rate. PREMISE NOTE: the order's closing line says the batter board is per-archetype-graded after this. Nothing has been rescaled for hits or total_bases either -- no archetype slot has ever reached sample and gradeBands remains built, gated and unwired. This is the fourth stat measured, not the completion of three. AUDIT: RBI 935 clean / 43 games; RUNS 617 clean / 33 games. Zero quarantined. No archetype slot reaches 500 -- and the signature archetypes the order names are the two SMALLEST slots on the board, RBI->DRIVER at n=24 and runs->CATALYST at n=9. RUNS is refused structurally before any factor is tested: 33 game clusters against a 40 floor. INPUTS RECONSTRUCTED rather than declared missing. lineup_context only covers 08-04 onward while settled rows start 07-31, so 187/617 runs rows joined. But the play-by-play cache runs from 05-01 and the batting order IS the order batters first appear -- slot, power-behind and reach-base all rebuilt point-in-time, coverage 187 -> 574. RBI, all THEATER: risp_opportunity +0.0047, extra_base_skill +0.0010, risp x extra_base +0.0056. RUNS, all refused on clusters and all pointing the wrong way: +0.0043 / +0.0008 / +0.0054. THE COMPOUND IS THE WORST VERSION IN BOTH STATS. The causally-correct compound was the most promising factor on the sheet and is the most harmful in each. Two multipliers that individually carry nothing do not cancel -- they compound each other's noise. Distinct from the collapsed-sequence lesson: there the product of two REAL effects was too small to use; here the product of two NULL effects is worse than either. THE ARCHETYPE DOES NOT RESCUE IT, and this is where the session nearly went wrong. The base rates look strongly differentiated (RBI DRIVER 0.609 vs BOMBER 0.413; runs GHOST 0.716 vs BOMBER 0.460). Gated directly against the pooled base rate: RBI +0.0010 CI [-0.0034,+0.0050] THEATER; runs -0.0028 CI [-0.0147,+0.0108] candidate at k=33. DRIVER's 0.609 is n=23 -- small-slot noise wearing a decimal point. Read off the table instead of gated, this would have shipped as "archetype differentiation is real and large". It is not. THE CROSS-STAT PATTERN THAT IS REAL -- the counter over-predicts every batter counting stat measured: total_bases p_win 0.5698 vs actual 0.5074 bias +0.0624 rbi p_win 0.4860 vs actual 0.4313 bias +0.0547 runs p_win 0.5949 vs actual 0.5749 bias +0.0200 Across four stats and three sessions, calibration is the systematic defect and factor scarcity is not. TB's held-out isotonic fix (-0.0039) still outperforms every factor tried on any stat, all null or theatre. NO RESCALE. Nothing proved, nothing certified calibrated, no slot at sample, and for RBI the archetype split is itself theatre -- so the honest band is the pooled base rate, which gradeBands returns by construction. Counter and frozen clusters byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-06 14:42:33 -04:00
parent 6a327d9114
commit 23d1b13176
3 changed files with 445 additions and 1 deletions
@@ -0,0 +1,301 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * prove-runs-rbi — THE CONTEXT-HEAVY STATS, WHERE INFLATION IS EASIEST.
+ *
+ * A large share of both stats is genuinely outside the hitter's control: a run
+ * needs someone behind you, an RBI needs someone in front of you. The job is to
+ * prove the HITTER-CONTROLLABLE part above the archetype's own base rate and
+ * grade the rest honestly as base-rate — which is the CORRECT answer for a
+ * context stat, not a failure to find something.
+ *
+ * ── THE NULL IS THE ARCHETYPE'S BASE RATE ────────────────────────────────
+ * Deliberately, and per the order: these base rates are spread and
+ * context-inflated, so beating "hitters like him" is the only meaningful bar. A
+ * per-player leave-one-out rate is not available here — 935 RBI rows over 344
+ * players is ~2.7 rows each, and estimating a personal rate from two rows would
+ * be inventing one. Leave-one-out is applied at the ARCHETYPE level so a row
+ * never contributes to its own baseline.
+ *
+ * ── INPUTS RECONSTRUCTED RATHER THAN DECLARED MISSING ────────────────────
+ * `lineup_context` only starts 2026-08-04 (ingest began last week) while settled
+ * rows run from 07-31, so only 187 of 617 runs rows join to a batting order.
+ * That would be input-blocked — except the play-by-play cache covers 05-01
+ * onward, and the batting order IS the order batters first appear. Reach-base
+ * skill and lineup power behind are derived from the same cache, point-in-time.
+ *
+ *   SUPABASE_URL=... node scripts/prove-runs-rbi.js
+ */
+
+require('dotenv').config();
+const fs = require('fs');
+const path = require('path');
+const { createClient } = require('@supabase/supabase-js');
+const fg = require('../src/services/model/factorGate');
+const tl = require('../src/services/model/testLedger');
+const sk = require('../src/services/model/skillProjection');
+const { knownNumber, knownRate } = require('../src/utils/known');
+const { nameKey } = require('../src/utils/playerName');
+
+const SB_URL = process.env.SUPABASE_URL;
+const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
+const CACHE = process.env.SEQ_OUT || path.join(process.cwd(), '.seq-cache', 'sequences.json');
+const PAGE = 1000;
+const ARCHS = (process.env.RR_ARCHETYPES || 'ALL,BOMBER,GHOST,BRUSH,DRIVER').split(',');
+
+const HIT = new Set(['single', 'double', 'triple', 'home_run']);
+const ONBASE = new Set(['single', 'double', 'triple', 'home_run', 'walk', 'hit_by_pitch', 'intent_walk']);
+const PA_EVENT = new Set([...HIT, 'field_out', 'strikeout', 'grounded_into_double_play', 'force_out',
+  'field_error', 'fielders_choice', 'fielders_choice_out', 'double_play', 'sac_fly', 'pop_out',
+  'line_out', 'fly_out', 'strikeout_double_play', 'walk', 'hit_by_pitch', 'intent_walk']);
+
+const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
+
+async function page(sb, table, select, apply) {
+  const out = [];
+  for (let from = 0; ; from += PAGE) {
+    const { data, error } = await apply(sb.from(table).select(select)).range(from, from + PAGE - 1);
+    if (error) throw error;
+    if (!data || data.length === 0) break;
+    out.push(...data);
+    if (data.length < PAGE) break;
+  }
+  return out;
+}
+
+/**
+ * Reconstruct, point-in-time, from play-by-play:
+ *   order[date|nameKey]      the hitter's batting slot that game
+ *   behind[date|nameKey]     mean barrel-ish power of the three slots after him
+ *   onbase[nameKey]          his reach-base rate over PRIOR games only
+ */
+function reconstruct(barrelByKey) {
+  const { games } = JSON.parse(fs.readFileSync(CACHE, 'utf8'));
+  games.sort((a, b) => String(a.date).localeCompare(String(b.date)) || a.gamePk - b.gamePk);
+
+  const order = new Map();
+  const behind = new Map();
+  const onbaseNow = new Map();      // running totals, folded in AFTER each game
+  const onbasePrior = new Map();    // snapshot used for that game's rows
+
+  for (const g of games) {
+    for (const half of ['top', 'bottom']) {
+      const pas = g.pas.filter((p) => p.half === half && PA_EVENT.has(p.event));
+      if (!pas.length) continue;
+
+      // The batting order IS the order batters first appear.
+      const seen = [];
+      const seenSet = new Set();
+      for (const p of pas) {
+        if (!seenSet.has(p.batter)) { seenSet.add(p.batter); seen.push(p); }
+        if (seen.length >= 9) break;
+      }
+      const slots = seen.map((p) => ({ id: p.batter, key: nameKey(p.batter_name || '') }));
+
+      for (let i = 0; i < slots.length; i += 1) {
+        const k = `${g.date}|${slots[i].key}`;
+        order.set(k, i + 1);
+        // Power BEHIND him — the hitters who would drive him in.
+        const nxt = [1, 2, 3].map((d) => slots[(i + d) % slots.length])
+          .map((s) => (s ? knownRate(barrelByKey.get(s.key)) : null))
+          .filter((v) => v !== null);
+        if (nxt.length) behind.set(k, mean(nxt));
+        const prior = onbaseNow.get(slots[i].key);
+        if (prior && prior.pa >= 60) onbasePrior.set(k, prior.ob / prior.pa);
+      }
+      for (const p of pas) {
+        const key = nameKey(p.batter_name || '');
+        const cur = onbaseNow.get(key) || { pa: 0, ob: 0 };
+        cur.pa += 1; cur.ob += ONBASE.has(p.event) ? 1 : 0;
+        onbaseNow.set(key, cur);
+      }
+    }
+  }
+  return { order, behind, onbase: onbasePrior };
+}
+
+/** RBI and RUNS have different causal stories, so different factors. */
+const FACTORS = {
+  rbi: [
+    {
+      key: 'risp_opportunity',
+      needs: ['risp_share'],
+      entity: (r) => r.player_key,
+      mechanism: 'HOW OFTEN HE BATS WITH RUNNERS IN SCORING POSITION. Half of an RBI is opportunity, and this is the ingested measure of it.',
+      apply: (r) => 1 + Math.max(-0.20, Math.min(0.20, (r.risp_share - 0.22) * 1.6)),
+    },
+    {
+      key: 'extra_base_skill',
+      needs: ['barrel_pct'],
+      entity: (r) => r.player_key,
+      mechanism: 'The other half — having batted with runners on, can he drive them in.',
+      apply: (r) => 1 + Math.max(-0.20, Math.min(0.20, (r.barrel_pct - 0.078) * 1.8)),
+    },
+    {
+      key: 'risp_x_extra_base',
+      needs: ['risp_share', 'barrel_pct'],
+      entity: (r) => r.player_key,
+      mechanism: 'THE CAUSALLY-CORRECT COMPOUND: opportunity AND the power to convert it. Neither half alone is an RBI.',
+      apply: (r) => (1 + Math.max(-0.20, Math.min(0.20, (r.risp_share - 0.22) * 1.6)))
+        * (1 + Math.max(-0.20, Math.min(0.20, (r.barrel_pct - 0.078) * 1.8))),
+    },
+  ],
+  runs: [
+    {
+      key: 'reach_base',
+      needs: ['onbase'],
+      entity: (r) => r.player_key,
+      mechanism: 'You cannot score without first reaching base. The most hitter-controllable component of a run.',
+      apply: (r) => 1 + Math.max(-0.25, Math.min(0.25, (r.onbase - 0.318) * 2.2)),
+    },
+    {
+      key: 'lineup_power_behind',
+      needs: ['power_behind'],
+      entity: (r) => `${r.game_id}|${r.batting_order}`,
+      mechanism: 'Who bats after him — the hitters who would drive him in. Pure context, and the part he does not control.',
+      apply: (r) => 1 + Math.max(-0.20, Math.min(0.20, (r.power_behind - 0.078) * 1.8)),
+    },
+    {
+      key: 'reach_x_power_behind',
+      needs: ['onbase', 'power_behind'],
+      entity: (r) => r.player_key,
+      mechanism: 'THE CAUSALLY-CORRECT COMPOUND: reach base AND have someone behind you who can drive you in.',
+      apply: (r) => (1 + Math.max(-0.25, Math.min(0.25, (r.onbase - 0.318) * 2.2)))
+        * (1 + Math.max(-0.20, Math.min(0.20, (r.power_behind - 0.078) * 1.8))),
+    },
+  ],
+};
+
+async function main() {
+  const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
+
+  const statcast = await page(sb, 'statcast_aggregates', '*', (q) => q.eq('sport', 'mlb').eq('role', 'batter'));
+  const batByKey = new Map();
+  const barrelByKey = new Map();
+  for (const r of statcast) {
+    if (!r.player_key) continue;
+    const prof = sk.fromStatcastRow(r);
+    batByKey.set(r.player_key, prof);
+    if (prof.barrel_pct != null) barrelByKey.set(r.player_key, prof.barrel_pct);
+  }
+
+  const oppRows = await page(sb, 'hitter_opportunity', '*', (q) => q.eq('sport', 'mlb'));
+  const oppByKey = new Map();
+  for (const r of oppRows) {
+    const prev = oppByKey.get(r.player_key);
+    if (!prev || String(r.as_of_date) > String(prev.as_of_date)) oppByKey.set(r.player_key, r);
+  }
+
+  const recon = reconstruct(barrelByKey);
+
+  const out = { generated_note: 'null is the ARCHETYPE base rate, leave-one-out' };
+
+  for (const stat of ['rbi', 'runs']) {
+    const snaps = await page(sb, 'model_snapshots', 'player_key, game_date, archetype',
+      (q) => q.eq('sport', 'mlb').eq('stat', stat).not('archetype', 'is', null));
+    const archOf = new Map();
+    for (const s of snaps) archOf.set(`${s.player_key}|${s.game_date}`, s.archetype);
+
+    const led = await page(sb, 'ledger_entries',
+      'id, game_id, player_key, player_name, line, side, outcome, game_date, p_win, quarantine_reason',
+      (q) => q.eq('sport', 'mlb').is('user_id', null).eq('stat', stat).in('outcome', ['hit', 'miss']));
+    const clean = led.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book')
+      && knownNumber(r.line) === 0.5);
+
+    // Archetype-level leave-one-out base rate — the context-inflated null.
+    const byArch = new Map();
+    for (const r of clean) {
+      const a = String(archOf.get(`${r.player_key}|${r.game_date}`) || 'UNLABELLED').toUpperCase();
+      const cur = byArch.get(a) || { n: 0, w: 0 };
+      cur.n += 1; cur.w += r.outcome === 'hit' ? 1 : 0;
+      byArch.set(a, cur);
+    }
+
+    const rows = [];
+    const loss = { no_archetype_base: 0, kept: 0 };
+    for (const r of clean) {
+      const a = String(archOf.get(`${r.player_key}|${r.game_date}`) || 'UNLABELLED').toUpperCase();
+      const ab = byArch.get(a);
+      if (!ab || ab.n < 4) { loss.no_archetype_base += 1; continue; }
+      const baseline = (ab.w - (r.outcome === 'hit' ? 1 : 0)) / (ab.n - 1);
+      const bat = batByKey.get(r.player_key);
+      const opp = oppByKey.get(r.player_key);
+      const okey = `${r.game_date}|${r.player_key}`;
+      rows.push({
+        archetype: a,
+        player_key: r.player_key,
+        game_id: r.game_id,
+        cluster: r.game_id,
+        baseline,
+        won: r.outcome === 'hit' ? 1 : 0,
+        risp_share: opp ? knownNumber(opp.risp_share) : null,
+        barrel_pct: bat ? knownRate(bat.barrel_pct) : null,
+        onbase: recon.onbase.has(okey) ? recon.onbase.get(okey) : null,
+        power_behind: recon.behind.has(okey) ? recon.behind.get(okey) : null,
+        batting_order: recon.order.get(okey) ?? null,
+      });
+      loss.kept += 1;
+    }
+
+    const mc = await tl.recordAndCount(tl.supabaseStore(sb), FACTORS[stat].flatMap((f) =>
+      ARCHS.map((a) => ({
+        sport: 'mlb', stat, archetype: a === 'ALL' ? null : a,
+        interaction: `factor:${f.key}`, target: 'outcome',
+      }))));
+
+    const audit = [];
+    const results = [];
+    for (const arch of ARCHS) {
+      const slot = arch === 'ALL' ? rows : rows.filter((r) => r.archetype === arch);
+      for (const f of FACTORS[stat]) {
+        const usable = slot.filter((r) => f.needs.every((k) => knownNumber(r[k]) !== null));
+        const ents = new Set(usable.map((r) => String(f.entity(r))));
+        const games = new Set(usable.map((r) => String(r.cluster)));
+        if (arch === 'ALL') {
+          audit.push({ factor: f.key, archetype: arch, rows: usable.length, games: games.size, entities: ents.size });
+        }
+        const useEntity = ents.size < games.size;
+        const paired = usable.map((r) => {
+          const m = f.apply(r);
+          return {
+            baseline: r.baseline,
+            conditioned: m === null ? null : Math.min(0.99, Math.max(0.01, r.baseline * m)),
+            won: r.won,
+            cluster: useEntity ? `e:${f.entity(r)}` : r.cluster,
+          };
+        });
+        const v = fg.adjudicate(paired, { factor: f.key, archetype: arch, stat, cumulativeTests: mc.cumulative_tests });
+        results.push({
+          archetype: arch, factor: f.key, n: v.movement.n,
+          clusters: v.improvement ? v.improvement.effective_n : null,
+          clustered_on: useEntity ? 'treatment_entity' : 'game',
+          distinct_games: games.size,
+          mean_abs_shift: v.movement.mean_abs_shift,
+          brier_delta: v.improvement ? v.improvement.brier_delta : null,
+          ci: v.improvement ? v.improvement.ci : null,
+          verdict: v.verdict,
+        });
+      }
+    }
+
+    out[stat] = {
+      clean_rows_line_0_5: clean.length,
+      rows_built: rows.length,
+      row_loss: loss,
+      distinct_games: new Set(rows.map((r) => r.cluster)).size,
+      archetype_base_rates: Object.fromEntries([...byArch.entries()]
+        .sort((a, b) => b[1].n - a[1].n)
+        .map(([a, v]) => [a, { n: v.n, base_rate: Math.round((v.w / v.n) * 10000) / 10000 }])),
+      cumulative_tests: mc.cumulative_tests,
+      input_audit: audit,
+      results,
+      proven: results.filter((r) => r.verdict === 'PROVES'),
+    };
+  }
+
+  console.log(JSON.stringify(out, null, 2));
+  process.exit(0);
+}
+
+main().catch((e) => { console.error(e); process.exit(1); });
@@ -0,0 +1,143 @@
+# runs + RBI — mostly-base-rate confirmed, and stronger than expected
+
+**Nothing proved. For RBI even the ARCHETYPE split is theatre, so the honest
+grade is the POOLED base rate.** The order anticipated that mostly-base-rate
+would be the correct answer for context-heavy stats. It is, and one level deeper
+than predicted.
+
+## Premise note
+
+The order's closing line says the batter board is "per-archetype-graded" after
+this. It is not: **nothing has been rescaled for hits or total_bases either** —
+no archetype slot has ever reached sample, and `gradeBands` remains built, gated
+and unwired. This is the fourth stat measured, not the completion of three.
+
+---
+
+## STEP 1 — Full-history audit
+
+| | clean | with p_win | players | dates | **games** | lines |
+|---|---|---|---|---|---|---|
+| RBI | 935 | 931 | 344 | 5 | **43** | 0.5 on 885 |
+| RUNS | 617 | 614 | 334 | 5 | **33** | 0.5 on all |
+
+Zero quarantined in either. **No archetype slot reaches 500.**
+
+| archetype | RBI n | RUNS n |
+|---|---|---|
+| UNLABELLED | 546 | 341 |
+| BOMBER | 207 | 148 |
+| GHOST | 92 | 67 |
+| BRUSH | 43 | 26 |
+| **DRIVER** | **24** | 20 |
+| **CATALYST** | 13 | **9** |
+
+**The signature archetypes the order names are the two smallest slots on the
+board** — RBI→DRIVER at n=24, runs→CATALYST at n=9. The hypothesis is reasonable
+and we are three orders of magnitude from being able to test it.
+
+**RUNS is refused structurally before any factor is tested: 33 game clusters
+against a 40 floor.** Reported as such rather than dressed up as a factor result.
+
+## Inputs reconstructed rather than declared missing
+
+`lineup_context` only covers 2026-08-04/05/06 (ingest began last week) while
+settled rows start 07-31, so just 187 of 617 runs rows join to a batting order.
+That reads as input-blocked — but the play-by-play cache runs from 05-01, and
+**the batting order IS the order batters first appear.** Batting slot, power
+behind (mean barrel of the next three slots) and reach-base rate were all
+reconstructed point-in-time from it: coverage went 187 → 574.
+
+---
+
+## STEP 2/3 — The gate (153 / 168 cumulative tests)
+
+### RBI — all THEATER
+
+| factor | n | clusters | shift | Brier Δ | CI | verdict |
+|---|---|---|---|---|---|---|
+| `risp_opportunity` | 803 | 43 | 0.0585 | **+0.0047** | [−0.0023, +0.0122] | **THEATER** |
+| `extra_base_skill` | 881 | 43 | 0.0271 | **+0.0010** | [−0.0023, +0.0042] | **THEATER** |
+| `risp × extra_base` | 803 | 43 | 0.0624 | **+0.0056** | [−0.0042, +0.0135] | **THEATER** |
+
+### RUNS — all refused on cluster count, all pointing the wrong way
+
+| factor | n | clusters | shift | Brier Δ | verdict |
+|---|---|---|---|---|---|
+| `reach_base` | 525 | 32 | 0.0383 | +0.0043 | PENDING — k<40 |
+| `lineup_power_behind` | 574 | 32 | 0.0232 | +0.0008 | PENDING — k<40 |
+| `reach × power_behind` | 525 | 32 | 0.0482 | +0.0054 | PENDING — k<40 |
+
+### The compound is the WORST version, in both stats
+
+RBI: 0.0047 / 0.0010 → **0.0056** compounded. RUNS: 0.0043 / 0.0008 →
+**0.0054** compounded.
+
+The causally-correct compound was the most promising factor on the sheet and is
+the most harmful in both. Two multipliers that individually carry nothing do not
+cancel — they compound each other's noise. Related to but distinct from the
+collapsed-sequence lesson: there the product of two REAL effects was too small to
+use; here the product of two NULL effects is actively worse than either.
+
+---
+
+## The archetype itself does not rescue it
+
+The archetype base rates look strongly differentiated, and that appearance is
+most of the trap:
+
+| | RBI | RUNS |
+|---|---|---|
+| DRIVER | **0.609** (n=23) | 0.500 (n=20) |
+| GHOST | 0.467 | **0.716** (n=67) |
+| BOMBER | 0.413 | 0.460 |
+| pooled | 0.4305 | 0.5721 |
+
+Tested directly — is the archetype's leave-one-out base rate better than the
+pooled one?
+
+| | shift | Brier Δ | CI | verdict |
+|---|---|---|---|---|
+| **RBI** | 0.0231 | +0.0010 | [−0.0034, +0.0050] | **THEATER** |
+| **RUNS** | 0.0642 | −0.0028 | [−0.0147, +0.0108] | CANDIDATE — k=33 |
+
+**For RBI, knowing the archetype makes the forecast worse.** DRIVER's 0.609 is
+n=23 — the spread is small-slot noise wearing a decimal point. Runs is at least
+directionally favourable, and unproven.
+
+Had this been read off the base-rate table instead of gated, the session would
+have shipped "archetype differentiation is real and large" as a finding. It is
+not one.
+
+---
+
+## The cross-stat pattern that IS real
+
+| stat | mean p_win | actual | counter bias |
+|---|---|---|---|
+| total_bases | 0.5698 | 0.5074 | **+0.0624** |
+| RBI | 0.4860 | 0.4313 | **+0.0547** |
+| runs | 0.5949 | 0.5749 | **+0.0200** |
+
+**The counter over-predicts every batter counting stat measured.** Across four
+stats and three sessions, calibration is the systematic defect and factor
+scarcity is not — TB's held-out isotonic fix (−0.0039 Brier) still outperforms
+every factor tried on any stat, all of which have been null or theatre.
+
+## STEP 4 — No rescale
+
+Two-bar rule: nothing proved, nothing certified calibrated, no slot at sample —
+and for RBI the archetype split is itself theatre, so the honest band is the
+POOLED base rate rather than a per-archetype one. `gradeBands` returns exactly
+that by construction.
+
+## Next, by value
+
+1. **Calibrate the counter across all four stats.** One systematic bias,
+   measured four times, larger than anything else on the board.
+2. **Stop adding factors to context stats.** Six tested across runs/RBI, six
+   null-or-worse, and the compounds worst of all.
+3. Runs needs game-date accrual to clear the cluster floor before it can be
+   gated at all.
+
+Counter and frozen clusters byte-identical.