Decompose the rbi anomaly: it is lineup ROLE, and the counter is

out-resolved by a frequency table on three of four stats

PHASE 0 — the 14.51% is REAL. Re-derived with a paged pull asserted
against an exact count (rbi 7,930 == 7,930; hits 11,690; TB 12,086; runs
6,440), since this harness produced a false null three times tonight. rbi
resolution 0.03268 reproduces, deciles are monotone through the middle,
and 20 raw rows are in the artifact for hand audit.

CAVEAT GOVERNING EVERYTHING BELOW: the naive forecasts are leave-one-out
ON THE EVALUATION WINDOW, so they see the rows they are scored on while
the model is strictly point-in-time. They are upper bounds on available
resolution, not fair competitors, and every comparison is read that way.

PHASE 1 — the split:

  stat  MODEL    (a)player-base  (b)lineup-slot  (c)within-stratum
  rbi   0.03268     0.01167         0.03608          0.01908
  hits  0.00252     0.00446         0.00100          0.00473
  TB    0.00442     0.01331         0.03448          0.00607
  runs  0.00130     0.00262         0.01156          0.01170

FINDING 1 — rbi's resolution is LINEUP ROLE almost exactly. Batting-order
slot alone resolves 0.03608 against the model's 0.03268. A single integer
accounts for the whole anomaly and slightly more. That is opportunity, not
skill -- the cleanup hitter bats with runners on. 36% is matched by player
identity alone. Within similar-base-rate strata the model still resolves
0.01908, 58% of its total and higher than any other stat's ENTIRE model
resolution, so genuine within-role discrimination exists on top.

FINDING 2 — on three of four stats the model is beaten by "he's a .270
hitter". Base-rate-only out-resolves the model 1.8x on hits, 3.0x on TB,
2.0x on runs. Even allowing for the window-peeking advantage, a 1.8-3.0x
gap is not explained by that alone: the served counter appears to DESTROY
discrimination relative to the player's own rate. rbi is the one stat
where the model beats the naive baseline.

FINDING 3 — lineup slot out-resolves the MODEL on three stats: TB 7.8x,
runs 8.9x, rbi 1.1x. Hits is the only stat where batting order carries
less, which is mechanically right -- a hit is a hit wherever you bat, but
runs, RBI and total bases all scale with opportunity.

PHASE 2 — all three worlds are partly true, in measured proportions.
World A ~90% true (slot covers rbi's entire resolution). World B ~36% true
for rbi, but the WHOLE story for hits/TB/runs where base rate alone wins.
World C true with a low ceiling: hits' total available spread resolution
is 0.00446, i.e. 1.8% of variance from a forecast that has seen the
answers.

PHASE 3 — the next arc is NOT "strengthen hits factors". Hits has the
lowest available resolution on the board and last order's wiring already
took it to 1.39% of a ~1.8% ceiling. Named first factor order for next
session: LINEUP SLOT / RISP OPPORTUNITY on rbi through the two-part gate --
input already ingested and prod-verified (S89), resolution measured not
hypothesised, causally-correct unit is plate appearances with runners on.
Measured availability is not a pass; it still faces the gate.

And higher-value than either: the counter being out-resolved by a
frequency table on three of four stats is a defect in the CHAMPION, not a
factor problem, and it costs nothing to test -- the recency blend and the
+/-0.03 / +/-0.015 nudges are three lines in probabilityEstimator.

The hits transmission win from 43f65d3 stands: the conduit is real and
permanent. This order changes only which stat has the most worth flowing
through it.

Diagnostic only -- no factor wired, no serving path changed, p_win
untouched, all frozen modules byte-identical. No Bonferroni slot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-07 03:09:44 -04:00
parent 43f65d30cb
commit 65ca6493db
2 changed files with 325 additions and 0 deletions
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env node
'use strict';
/**
* PHASES 0-1 — is rbi's 14.51% real, and where does it come from?
*
* PHASE 0 is a Beck gate. The 14.51% came from the same decomposition harness
* whose paging helper produced a false null three times tonight (composite PKs,
* order-by-id, error swallowed). So the row count is asserted against an
* independent exact count, every page is error-checked, and 20 raw rows are
* printed so the decile arithmetic can be audited by hand.
*
* PHASE 1 asks what the resolution IS. Resolution rewards a forecast for
* separating outcomes — but a forecast can separate outcomes by knowing WHO is
* batting rather than anything about tonight. Three nested forecasts:
*
* PLAYER BASE RATE leave-one-out frequency for that hitter, nothing else.
* Its resolution is pure across-player spread.
* LINEUP SLOT mean rate for that batting-order position. Real
* predictive signal, but ROLE, not skill.
* THE MODEL served p_win.
*
* The part that behaves like our doctrine's "skill" is what the model resolves
* WITHIN a stratum of similar players — measured directly by stratifying on the
* player's own base rate and pooling the within-stratum resolutions.
*/
require('dotenv').config();
const fs = require('fs');
const path = require('path');
const { createClient } = require('@supabase/supabase-js');
const guards = require('../src/services/model/calibrationGuards');
const { knownNumber } = require('../src/utils/known');
const { nameKey } = require('../src/utils/playerName');
const BOX = path.join(process.cwd(), '.seq-cache', 'batting-lines.json');
const SEQ = path.join(process.cwd(), '.seq-cache', 'sequences.json');
const STATS = ['rbi', 'hits', 'total_bases', 'runs'];
const FIELD = { hits: (b) => b.hits, total_bases: (b) => b.totalBases, rbi: (b) => b.rbi, runs: (b) => b.runs };
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
/** Error-checked pager with an explicit order column. */
async function page(sb, t, sel, orderBy, apply) {
const out = [];
for (let i = 0; ; i += 1000) {
const q = apply ? apply(sb.from(t).select(sel)) : sb.from(t).select(sel);
const { data, error } = await q.order(orderBy, { ascending: true }).range(i, i + 999);
if (error) throw new Error(`${t}: ${error.message}`);
if (!data || !data.length) break;
out.push(...data);
if (data.length < 1000) break;
}
return out;
}
const isPreGame = (c, g) => {
const et = new Date(new Date(c).getTime() - 4 * 3600 * 1000);
const d = et.toISOString().slice(0, 10);
return d < g || (d === g && et.getUTCHours() < 19);
};
/** Resolution alone: weighted spread of bin realized rates about the base rate. */
function resolutionOf(rows, bins = 10) {
const base = mean(rows.map((r) => r.won));
let res = 0;
const table = [];
for (let k = 0; k < bins; k += 1) {
const lo = k / bins; const hi = (k + 1) / bins;
const sl = rows.filter((r) => r.p >= lo && (hi >= 1 ? r.p <= 1 : r.p < hi));
if (!sl.length) continue;
const w = sl.length / rows.length;
const ok = mean(sl.map((r) => r.won));
res += w * (ok - base) ** 2;
table.push({ bin: `${lo.toFixed(1)}-${hi.toFixed(1)}`, n: sl.length, forecast: r4(mean(sl.map((r) => r.p))), realized: r4(ok) });
}
return { base_rate: r4(base), resolution: r5(res), uncertainty: r5(base * (1 - base)), share: r4(res / (base * (1 - base))), table };
}
(async () => {
const sb = createClient(process.env.SUPABASE_URL, process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY, { auth: { persistSession: false } });
const lines = JSON.parse(fs.readFileSync(BOX, 'utf8')).lines;
// Batting-order slot, reconstructed point-in-time from play-by-play.
const { games } = JSON.parse(fs.readFileSync(SEQ, 'utf8'));
const slotOf = new Map();
for (const g of games) {
for (const half of ['top', 'bottom']) {
const seen = []; const set = new Set();
for (const p of g.pas.filter((x) => x.half === half)) {
if (!set.has(p.batter)) { set.add(p.batter); seen.push(p); }
if (seen.length >= 9) break;
}
seen.forEach((p, i) => slotOf.set(`${g.date}|${nameKey(p.batter_name || '')}`, i + 1));
}
}
const out = { PHASE_0: {}, PHASE_1: {} };
for (const stat of STATS) {
// PHASE 0 — exact count first, then the paged pull must match it.
const { count: exact, error: cErr } = await sb.from('model_snapshots')
.select('*', { count: 'exact', head: true }).eq('sport', 'mlb').eq('stat', stat);
if (cErr) throw new Error(`count ${stat}: ${cErr.message}`);
const snaps = await page(sb, 'model_snapshots',
'id, game_date, captured_at, stat, player_key, player_name, line, side, p_win, refused', 'id',
(q) => q.eq('sport', 'mlb').eq('stat', stat));
if (snaps.length !== exact) throw new Error(`PHASE 0 FAIL ${stat}: paged ${snaps.length} != exact ${exact}`);
const picked = new Map();
for (const r of snaps) {
if (!isPreGame(r.captured_at, r.game_date) || r.refused || knownNumber(r.p_win) === null) continue;
const k = [r.game_date, r.player_key, r.line].join('|');
const prev = picked.get(k);
if (!prev || knownNumber(r.p_win) > knownNumber(prev.p_win)) picked.set(k, r);
}
guards.assertPickedSideDedup([...picked.values()].map((r) => ({ propKey: [r.game_date, r.player_key, r.line].join('|'), side: r.side, p: knownNumber(r.p_win) })));
const rows = [];
for (const r of picked.values()) {
const b = lines[`${r.game_date}|${r.player_key}`]; const L = knownNumber(r.line);
if (!b || L === null || !r.side) continue;
const v = knownNumber(FIELD[stat](b)); if (v === null) continue;
const over = v > L;
rows.push({
date: r.game_date, key: r.player_key, name: r.player_name, line: L, side: r.side,
actual: v, p: knownNumber(r.p_win),
won: (String(r.side).toLowerCase() === 'under' ? !over : over) ? 1 : 0,
slot: slotOf.get(`${r.game_date}|${r.player_key}`) ?? null,
});
}
const model = resolutionOf(rows);
out.PHASE_0[stat] = { exact_rows: exact, paged_rows: snaps.length, scorable: rows.length, model_resolution: model.resolution, model_share: model.share, deciles: model.table };
if (stat === 'rbi') {
out.PHASE_0.rbi_raw_sample = rows.slice(0, 20).map((r) => ({ name: r.name, date: r.date, line: r.line, side: r.side, p_win: r.p, actual_rbi: r.actual, won: r.won }));
}
// ── (a) PLAYER BASE RATE, leave-one-out ──
const byPlayer = new Map();
for (const r of rows) { const c = byPlayer.get(r.key) || { n: 0, w: 0 }; c.n += 1; c.w += r.won; byPlayer.set(r.key, c); }
const baseRows = rows.filter((r) => byPlayer.get(r.key).n >= 3)
.map((r) => { const c = byPlayer.get(r.key); return { ...r, p: (c.w - r.won) / (c.n - 1) }; });
const baseOnly = resolutionOf(baseRows);
// ── (b) LINEUP SLOT, leave-one-out ──
const bySlot = new Map();
for (const r of rows) { if (r.slot == null) continue; const c = bySlot.get(r.slot) || { n: 0, w: 0 }; c.n += 1; c.w += r.won; bySlot.set(r.slot, c); }
const slotRows = rows.filter((r) => r.slot != null && bySlot.get(r.slot).n >= 10)
.map((r) => { const c = bySlot.get(r.slot); return { ...r, p: (c.w - r.won) / (c.n - 1) }; });
const slotOnly = slotRows.length ? resolutionOf(slotRows) : null;
// ── (c) WITHIN-STRATUM: does the model still separate similar players? ──
// Stratify on the player's own base rate, then pool the model's resolution
// computed INSIDE each stratum. Across-player spread is held constant, so
// what survives is discrimination between comparable hitters.
const strata = [[0, 0.45], [0.45, 0.6], [0.6, 0.75], [0.75, 1.01]];
let within = 0; let wTot = 0; const strataDetail = [];
for (const [lo, hi] of strata) {
const sl = rows.filter((r) => { const c = byPlayer.get(r.key); if (!c || c.n < 3) return false; const b = c.w / c.n; return b >= lo && b < hi; });
if (sl.length < 40) continue;
const rr = resolutionOf(sl);
within += sl.length * rr.resolution; wTot += sl.length;
strataDetail.push({ stratum: `${lo}-${hi}`, n: sl.length, base: rr.base_rate, resolution: rr.resolution });
}
const withinRes = wTot ? within / wTot : null;
out.PHASE_1[stat] = {
n: rows.length,
model_resolution: model.resolution,
model_share_of_variance: model.share,
a_player_base_rate_only: { n: baseRows.length, resolution: baseOnly.resolution, share: baseOnly.share },
b_lineup_slot_only: slotOnly ? { n: slotRows.length, resolution: slotOnly.resolution, share: slotOnly.share } : null,
c_within_stratum_resolution: r5(withinRes),
strata: strataDetail,
base_rate_explains_pct: baseOnly.resolution ? r4(Math.min(1, baseOnly.resolution / model.resolution)) : null,
within_stratum_share_of_model: withinRes != null && model.resolution ? r4(withinRes / model.resolution) : null,
};
}
console.log(JSON.stringify(out, null, 2));
process.exit(0);
})().catch((e) => { console.error('FAILED:', e.message); process.exit(1); });
const r5 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 100000) / 100000);
const r4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
+141
View File
@@ -0,0 +1,141 @@
# The rbi anomaly, decomposed — the model is out-resolved by a batting-order integer
## PHASE 0 — the 14.51% is REAL
The figure came from the harness that produced a false null three times tonight,
so it was re-derived with a paged pull **asserted against an exact count**:
| stat | exact rows | paged rows | scorable | resolution | share |
|---|---|---|---|---|---|
| rbi | 7,930 | 7,930 ✓ | 630 | **0.03268** | **14.51%** |
| hits | 11,690 | 11,690 ✓ | 1,140 | 0.00252 | 1.03% |
| total_bases | 12,086 | 12,086 ✓ | 1,050 | 0.00442 | 1.82% |
| runs | 6,440 | 6,440 ✓ | 597 | 0.00130 | 0.56% |
**Reproduces exactly.** rbi deciles are monotone through the middle (0.552→0.577,
0.646→0.691, 0.746→0.784) with a genuinely low bin at 0.13→0.122, and 20 raw
rows are printed in the artifact for hand audit.
---
## The caveat that governs every number below
The naive forecasts are **leave-one-out on the evaluation window itself** — they
see that player's performance in the very rows being scored, while the model is
strictly point-in-time. They are therefore **upper bounds on available
resolution, not fair competitors.** Every comparison is read that way.
---
## PHASE 1 — the three-way split
| stat | MODEL | (a) player base rate | (b) lineup slot | (c) within-stratum |
|---|---|---|---|---|
| **rbi** | **0.03268** | 0.01167 | **0.03608** | 0.01908 |
| hits | 0.00252 | **0.00446** | 0.00100 | 0.00473 |
| total_bases | 0.00442 | **0.01331** | **0.03448** | 0.00607 |
| runs | 0.00130 | **0.00262** | **0.01156** | 0.01170 |
### Finding 1 — rbi's resolution is LINEUP ROLE, almost exactly
**Batting-order slot alone resolves 0.03608 against the model's 0.03268.** A
single integer — where he hits in the order — accounts for the entire anomaly and
slightly more. That is real predictive signal and it is **opportunity, not
skill**: the cleanup hitter bats with runners on, the 8-hole hitter does not.
Player base rate alone gives 0.01167, so ~36% of the model's rbi resolution is
matched by knowing only *who* is batting.
Within strata of similar-base-rate players the model still resolves **0.01908**
58% of its total, and **higher than any other stat's entire model resolution.**
So rbi does carry genuine within-role discrimination on top of the role effect.
### Finding 2 — on three of four stats the model is beaten by "he's a .270 hitter"
| stat | model | player base rate alone | |
|---|---|---|---|
| hits | 0.00252 | **0.00446** | base-rate-only resolves **1.8×** the model |
| total_bases | 0.00442 | **0.01331** | **3.0×** |
| runs | 0.00130 | **0.00262** | **2.0×** |
| rbi | **0.03268** | 0.01167 | model wins, 2.8× |
Even allowing that the naive forecast peeks at the window, a **1.83.0× gap is
not explained by that advantage alone.** The served counter — a frequency over
the line, blended with the last five games, nudged by opponent rank and home/away
— appears to **destroy discrimination relative to the player's own rate.** The
recency blend and the ±0.03/±0.015 nudges move predictions in ways that do not
track outcomes.
**rbi is the one stat where the model beats the naive baseline.**
### Finding 3 — lineup slot out-resolves the model on THREE stats
| stat | model | slot alone |
|---|---|---|
| total_bases | 0.00442 | **0.03448** (7.8×) |
| runs | 0.00130 | **0.01156** (8.9×) |
| rbi | 0.03268 | **0.03608** (1.1×) |
| hits | 0.00252 | 0.00100 (model wins) |
**Hits is the only stat where batting order carries less than the model** — which
makes sense: a hit is a hit whether you bat first or ninth, but runs, RBI and
total bases all scale with opportunity.
---
## PHASE 2 — which world
**All three are partly true, in measured proportions:**
- **WORLD A — rbi resolution is real and role-driven: ~90% TRUE.** Slot alone
(0.03608) covers the model's entire rbi resolution. It is real, it is
contextual rather than skill-based, and 58% survives within similar-player
strata as genuine discrimination.
- **WORLD B — base-rate-spread artefact: ~36% TRUE for rbi.** Player identity
alone accounts for about a third. Not the main story for rbi — but for
**hits, TB and runs it is the whole story and then some**, since base rate
alone out-resolves the model on all three.
- **WORLD C — hits is intrinsically compressed: TRUE, and the ceiling is low.**
Total available spread resolution for hits is 0.00446 — **1.8% of variance even
from a forecast that has seen the answers.** Perfect factors cannot make hits a
high-resolution grade.
---
## PHASE 3 — the roadmap-deciding log
**The next arc is not "strengthen hits factors."** Hits has the lowest available
resolution on the board (1.8% ceiling from an oracle-ish baseline) and last
order's wiring already lifted it to 1.39% of a 1.8% ceiling. There is very little
left there.
**Named first factor order for next session: LINEUP SLOT / RISP OPPORTUNITY on
rbi**, through the two-part gate.
- The input is **already ingested and prod-verified** (`lineup_context` batting
order, `hitter_opportunity` RISP share, S89).
- Its resolution is **measured, not hypothesised**: 0.03608 slot-only on rbi,
0.03448 on TB, 0.01156 on runs.
- The causally-correct unit is **plate appearances with runners on**, which is
what RISP share measures directly — the crude version is the slot integer.
- It must clear the two-part gate like anything else. Measured availability is
not a pass.
**Second, and higher-value than either:** the counter is out-resolved by a player
frequency table on three of four stats. That is not a factor problem — it is a
defect in the champion. **Diagnosing whether the recency blend and the
±0.03/±0.015 nudges are destroying discrimination is the biggest single lever
this decomposition found**, and it costs nothing to test: they are three lines in
`probabilityEstimator`.
**The hits transmission win stands.** The conduit is real and permanent —
sign-verified, 75% coverage. This order changes only *which stat* has the most
worth flowing through it.
---
## Invariants
Diagnostic only. No factor wired, no serving path changed, no calibration refit,
`p_win` untouched, all frozen modules byte-identical including the hits path from
`43f65d3`. No Bonferroni slot — this is resolution accounting, not a causal claim.