tb-v1: model total_bases as a compound outcome (challenger)

Current ladder (proj_p_over_line) and champion p_win are BYTE-IDENTICAL.
tb-v1 writes alongside them, on total_bases props only.

STEP 0 -- components confirmed on real data, not assumed. statsapi has no
singles field, but hits - doubles - triples - homeRuns reproduces stored
totalBases EXACTLY on a real 10-game log. So the decomposition is exact,
not an approximation.

THE MODEL. Each component gets its own per-game Poisson rate; TB is their
weighted sum, and the PMF is built by exact convolution rather than
simulated (TB support is small). It inherits the SAME combined multiplier
proj-v1.1 computes, so the two models differ only in STRUCTURE.

Why this is the fix: with identical mean TB of 1.0, a pure-HR hitter and a
pure-singles hitter get P(TB>=4) of 0.221 vs 0.019 -- a 12x difference an NB
on TB alone cannot express, because it treats one home run as four events.
A test asserts that separation, and asserts P(TB>=4) for a pure-HR hitter
equals P(at least one HR) exactly.

INDEPENDENCE IS AN APPROXIMATION AND IS LABELLED AS ONE: a plate appearance
that becomes a double cannot also become a single, so the components are
weakly negatively correlated and independent Poissons slightly overstate
the tail. Closer to the truth than what it replaces; not a solved problem.

HONEST-ABSENT throughout: fewer than 3 usable games, or no derivable
component, returns null and the prop keeps the current ladder value. An
inconsistent row (hits < extra-base hits) is SKIPPED rather than clamped to
zero -- clamping would invent a plausible line out of a broken one.

I HIT THE Number(null)===0 TRAP IN MY OWN CODE and a test caught it: a null
rate passed a naive finite check and was treated as a measured zero, which
is the difference between "this player never triples" and "we do not know
his triple rate". Both tbPmf and tbMean now reject null/''/boolean strictly.

Holdout committed: TB ROWS ONLY (49 of 437 settled -- averaging into other
stats would hide the effect) and DIRECTION-ALIGNED, since the unaligned
comparison is the artifact that accounted for 41% of the ladder's apparent
loss. If tb-v1 does NOT improve, the family-mismatch hypothesis is wrong
and the mean/similarity branch reopens -- recorded in the query header.

Migration applied: proj_tb_p_over + proj_tb_meta, NULL-meaningful.

Gates: 4,104 tests / 329 suites green; next build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc

scripts/tb-compound-holdout.sql

@@ -0,0 +1,42 @@
+-- tb-compound-holdout.sql — TOTAL BASES ONLY, direction-aligned.
+--
+-- TWO guards this query exists to enforce:
+--   1. TB ROWS ONLY. Averaging into other stats would hide the effect, since
+--      total_bases is 49 of 437 settled rows.
+--   2. DIRECTION-ALIGNED. p_win is P(GRADED SIDE); proj_p_over_line and
+--      proj_tb_p_over are P(OVER). 31.4% of rows are under-graded, and comparing
+--      raw P(over) against an under-side outcome measures the model BACKWARDS —
+--      that artifact alone accounted for 41% of the ladder's apparent loss.
+--
+-- Promote tb-v1 ONLY if it materially improves TB resolution toward/past the
+-- champion. If it does NOT, the family-mismatch hypothesis is WRONG and the
+-- mean-weakness / similarity branch REOPENS. Record which.
+
+with tb as (
+  select
+    game_date, id, lower(side) side, (outcome='hit')::int won,
+    p_win::numeric champ,
+    case when lower(side)='under' then 1 - proj_p_over_line::numeric
+         else proj_p_over_line::numeric end ladder_al,
+    case when lower(side)='under' then 1 - proj_tb_p_over::numeric
+         else proj_tb_p_over::numeric end tbv1_al
+  from public.ledger_entries
+  where sport='mlb' and user_id is null
+    and stat = 'total_bases'
+    and outcome in ('hit','miss')
+    and p_win is not null
+    and proj_p_over_line is not null
+    and proj_tb_p_over is not null          -- matched rows: all three present
+)
+select
+  count(*)                                            n,
+  count(*) filter (where side='under')                under_rows,
+  round(avg(won::numeric),3)                          base_rate,
+  round(corr(champ,     won::numeric)::numeric,4)     res_champion,
+  round(corr(ladder_al, won::numeric)::numeric,4)     res_ladder_v11,
+  round(corr(tbv1_al,   won::numeric)::numeric,4)     res_tb_v1,
+  round(stddev(ladder_al)::numeric,4)                 sd_ladder,
+  round(stddev(tbv1_al)::numeric,4)                   sd_tb_v1,
+  round(avg(ladder_al)::numeric,4)                    mean_ladder,
+  round(avg(tbv1_al)::numeric,4)                      mean_tb_v1
+from tb;

src/services/ledgerService.js

@@ -293,6 +293,11 @@ function rowsFromSnapshot(sport, grades, oddsProps, nowIso) {
       proj_distribution: g.proj_distribution || null,
       proj_ladder: g.proj_ladder || null,
       proj_factors: g.proj_factors || null,
+      // tb-v1 CHALLENGER — total_bases as a compound outcome. Written alongside
+      // proj_p_over_line, never in place of it. NULL on non-TB props and when
+      // the components are underivable; never a fabricated 0.
+      proj_tb_p_over: numOrNull(g.proj_tb_p_over),
+      proj_tb_meta: g.proj_tb_meta || null,
       // Session 75 — the ENVIRONMENT that drove this projection. The FORECAST,
       // not the actual: this is what we knew when we projected, and it is what
       // the instrument measures. The actual lands in game_context and is never

src/services/projection/compoundTotalBases.js

@@ -0,0 +1,200 @@
+'use strict';
+
+/**
+ * compoundTotalBases — TOTAL BASES modelled as the compound outcome it is.
+ *
+ * WHY THIS EXISTS. proj-v1.1 models every stat as a single negative binomial
+ * count. That is right for genuine low-rate event counts (walks, runs, doubles —
+ * measured resolution 0.519 / 0.345 / 0.207) and WRONG for total bases, which is
+ * not a count of events at all but a WEIGHTED SUM of them:
+ *
+ *     TB = 1·singles + 2·doubles + 3·triples + 4·home_runs
+ *
+ * An NB fitted to TB treats one home run as "four events", which mis-states the
+ * variance badly — a 4-base outcome from ONE plate appearance is not the same
+ * random object as four separate 1-base outcomes. Measured: total_bases had the
+ * worst result in the ladder, resolution 0.009 (mean −0.019) against the
+ * champion's 0.273. See `specs/proj-v11-diagnosis.md`.
+ *
+ * THE MODEL. Each component is its own per-game Poisson rate; TB is their
+ * weighted sum. The exact PMF is built by convolution rather than simulated, so
+ * the result is deterministic and cheap (TB support is small — a cap of 20 bases
+ * covers every realistic game).
+ *
+ * INDEPENDENCE IS AN APPROXIMATION, and a stated one: a plate appearance that
+ * becomes a double cannot also become a single, so the components are weakly
+ * negatively correlated. Modelling them as independent Poissons slightly
+ * OVERSTATES the tail. That is still far closer to the truth than treating one
+ * home run as four independent events, which is what it replaces — but it is a
+ * known limitation, not a solved problem.
+ *
+ * DOCTRINE: this is the per-stat rule one level deeper — model the stat by its
+ * actual generative structure, not by a family that happens to fit its name.
+ */
+
+const TB_CAP = 20;        // bases per game; beyond this is not a real outcome
+const N_CAP = 8;          // per-component events per game
+
+/** Poisson pmf, computed iteratively so no factorial overflows. */
+function poissonPmf(lambda, nMax) {
+  const l = Number(lambda);
+  if (!Number.isFinite(l) || l < 0) return null;
+  const out = new Array(nMax + 1).fill(0);
+  let term = Math.exp(-l);        // n = 0
+  out[0] = term;
+  for (let n = 1; n <= nMax; n += 1) {
+    term = (term * l) / n;
+    out[n] = term;
+  }
+  return out;
+}
+
+/**
+ * The exact PMF of TB = Σ weightᵢ · Nᵢ, with Nᵢ ~ Poisson(rateᵢ) independent.
+ *
+ * @param {{singles:number,doubles:number,triples:number,home_runs:number}} rates
+ *        per-GAME expected counts. Any missing/negative component is treated as
+ *        ABSENT (rate 0) rather than guessed — a player with no recorded triples
+ *        genuinely has ~0 triple rate, and inventing one would add tail mass.
+ * @returns {number[]|null} pmf indexed by total bases, or null if no component
+ *          has a usable rate (caller must fall back, never fabricate).
+ */
+function tbPmf(rates = {}) {
+  const components = [
+    { w: 1, rate: rates.singles },
+    { w: 2, rate: rates.doubles },
+    { w: 3, rate: rates.triples },
+    { w: 4, rate: rates.home_runs },
+  ];
+  // STRICT: `Number(null) === 0`, so a null rate would pass a naive finite check
+  // and be treated as a real, measured zero — the difference between "this
+  // player never triples" and "we do not know his triple rate". Absent stays
+  // absent; only a genuine number counts.
+  const isRate = (v) => v != null && v !== '' && typeof v !== 'boolean'
+    && Number.isFinite(Number(v)) && Number(v) >= 0;
+  const usable = components.filter((c) => isRate(c.rate));
+  if (usable.length === 0) return null;
+
+  let pmf = new Array(TB_CAP + 1).fill(0);
+  pmf[0] = 1;
+
+  for (const c of usable) {
+    const rate = Number(c.rate);
+    if (rate === 0) continue;                 // contributes nothing, skip cleanly
+    const comp = poissonPmf(rate, N_CAP);
+    if (!comp) continue;
+    const next = new Array(TB_CAP + 1).fill(0);
+    for (let t = 0; t <= TB_CAP; t += 1) {
+      const pt = pmf[t];
+      if (pt === 0) continue;
+      for (let n = 0; n <= N_CAP; n += 1) {
+        const bases = t + c.w * n;
+        if (bases > TB_CAP) break;            // truncated tail, accounted below
+        next[bases] += pt * comp[n];
+      }
+    }
+    pmf = next;
+  }
+
+  // Truncation leaves a small mass deficit (the >TB_CAP tail). Push it onto the
+  // top bucket rather than renormalising: renormalising would silently inflate
+  // every low bucket, and the deficit genuinely belongs at the top.
+  const total = pmf.reduce((a, b) => a + b, 0);
+  if (total > 0 && total < 1) pmf[TB_CAP] += 1 - total;
+  return pmf;
+}
+
+/** P(TB >= k) from a pmf. */
+function pAtLeast(pmf, k) {
+  if (!Array.isArray(pmf)) return null;
+  const kk = Math.max(0, Math.ceil(Number(k)));
+  if (!Number.isFinite(kk)) return null;
+  if (kk === 0) return 1;
+  let s = 0;
+  for (let i = kk; i < pmf.length; i += 1) s += pmf[i];
+  return Math.min(1, Math.max(0, s));
+}
+
+/** Expected total bases implied by the component rates. */
+function tbMean(rates = {}) {
+  // Same strictness as tbPmf: an absent component contributes nothing, and is
+  // not silently read as a measured zero.
+  const n = (v) => ((v != null && v !== '' && typeof v !== 'boolean'
+    && Number.isFinite(Number(v)) && Number(v) >= 0) ? Number(v) : 0);
+  return n(rates.singles) + 2 * n(rates.doubles) + 3 * n(rates.triples) + 4 * n(rates.home_runs);
+}
+
+/**
+ * Derive component rates from per-game log rows.
+ *
+ * SINGLES ARE DERIVED, not read: statsapi has no `singles` field, and
+ * hits − doubles − triples − homeRuns reproduces stored totalBases EXACTLY on
+ * real logs (verified). A negative result means the row is inconsistent, so that
+ * ROW is skipped rather than clamped to 0 — clamping would quietly invent a
+ * plausible line out of a broken one.
+ *
+ * @param {Array<{stat:object}>|Array<object>} rows game-log rows
+ * @param {number} [minGames=3]
+ */
+function ratesFromLog(rows, minGames = 3) {
+  const games = [];
+  for (const r of rows || []) {
+    const s = (r && r.stat) || r || {};
+    const h = Number(s.hits);
+    const d = Number(s.doubles);
+    const t = Number(s.triples);
+    const hr = Number(s.homeRuns);
+    if (![h, d, t, hr].every((v) => Number.isFinite(v))) continue;
+    const singles = h - d - t - hr;
+    if (singles < 0) continue;               // inconsistent row — skip, never clamp
+    games.push({ singles, doubles: d, triples: t, home_runs: hr });
+  }
+  if (games.length < minGames) return null;  // honest-absent: caller falls back
+  const mean = (k) => games.reduce((a, g) => a + g[k], 0) / games.length;
+  return {
+    singles: mean('singles'),
+    doubles: mean('doubles'),
+    triples: mean('triples'),
+    home_runs: mean('home_runs'),
+    games: games.length,
+  };
+}
+
+/**
+ * The full read: P(TB >= line) plus the mean, or null when the components are
+ * not derivable. `multiplier` scales every component rate together (the same
+ * park × weather × platoon × matchup product proj-v1.1 already computes), so the
+ * compound model inherits the adjustments rather than ignoring them.
+ */
+function projectTotalBases({ rows, line, multiplier = 1, minGames = 3 } = {}) {
+  const base = ratesFromLog(rows, minGames);
+  if (!base) return null;
+  const m = Number.isFinite(Number(multiplier)) && Number(multiplier) > 0 ? Number(multiplier) : 1;
+  const rates = {
+    singles: base.singles * m,
+    doubles: base.doubles * m,
+    triples: base.triples * m,
+    home_runs: base.home_runs * m,
+  };
+  const pmf = tbPmf(rates);
+  if (!pmf) return null;
+  const target = Math.max(1, Math.ceil(Number(line)));
+  if (!Number.isFinite(target)) return null;
+  return {
+    p_over_line: Math.round(pAtLeast(pmf, target) * 1000) / 1000,
+    mean: Math.round(tbMean(rates) * 1000) / 1000,
+    rates: {
+      singles: Math.round(rates.singles * 1000) / 1000,
+      doubles: Math.round(rates.doubles * 1000) / 1000,
+      triples: Math.round(rates.triples * 1000) / 1000,
+      home_runs: Math.round(rates.home_runs * 1000) / 1000,
+    },
+    games_used: base.games,
+    family: 'compound_weighted_poisson',
+    independence_caveat: true,
+  };
+}
+
+module.exports = {
+  tbPmf, pAtLeast, tbMean, ratesFromLog, projectTotalBases, poissonPmf, TB_CAP, N_CAP,
+};

src/services/projectionChallenger.js

@@ -18,6 +18,7 @@
  */
 
 const dist = require('./projection/distribution');
+const compoundTb = require('./projection/compoundTotalBases');
 const matchup = require('./projection/matchupRead');
 const parkBase = require('./parkBase');
 const { NAME_TO_ABBR } = require('./environmentContext');
@@ -204,8 +205,35 @@ function projectProp({
   }));
   const pOverLine = tradedRung != null ? dist.round3(dist.nbSurvival(nb.r, nb.p, tradedRung)) : null;
 
+  // ── TOTAL BASES, modelled as the COMPOUND OUTCOME it is (tb-v1) ───────────
+  // TB is a weighted sum (1B..HR = 1..4), not a count of events, so the single
+  // negative binomial above treats one home run as four events. Measured, that
+  // gave total_bases the worst result in the ladder (resolution 0.009 vs the
+  // champion's 0.273). This computes the exact PMF from per-component Poisson
+  // rates instead, and inherits the SAME combined multiplier so the two models
+  // differ only in structure.
+  //
+  // CHALLENGER ONLY: it is written alongside, never substituted for
+  // `proj_p_over_line`. The current ladder and the champion are byte-identical.
+  // Components underivable (thin/inconsistent log) → null, and the prop keeps
+  // the current ladder value. Never fabricated.
+  let tbCompound = null;
+  if (stat === 'total_bases' && tradedRung != null) {
+    try {
+      tbCompound = compoundTb.projectTotalBases({
+        rows: gameLog, line, multiplier: M,
+      });
+    } catch { tbCompound = null; }
+  }
+
   return {
     proj_version: PROJ_VERSION,
+    proj_tb_p_over: tbCompound ? tbCompound.p_over_line : null,
+    proj_tb_meta: tbCompound ? {
+      version: 'tb-v1', mean: tbCompound.mean, rates: tbCompound.rates,
+      games_used: tbCompound.games_used, family: tbCompound.family,
+      independence_caveat: tbCompound.independence_caveat,
+    } : null,
     proj_point: point,
     proj_line: line,
     proj_p_over_line: pOverLine,

tests/unit/compoundTotalBases.test.js

@@ -0,0 +1,101 @@
+'use strict';
+
+/**
+ * compoundTotalBases (tb-v1) — total bases as the compound outcome it is.
+ *
+ * The property that matters is the one an NB on TB cannot express: two hitters
+ * with the SAME mean total bases but different STRUCTURE must get different
+ * curves. Everything else here guards the honest-absent paths.
+ */
+
+const c = require('../../src/services/projection/compoundTotalBases');
+
+const R = (s, d, t, hr) => ({ singles: s, doubles: d, triples: t, home_runs: hr });
+const row = (h, d, t, hr) => ({ stat: { hits: h, doubles: d, triples: t, homeRuns: hr } });
+
+describe('the PMF is a real distribution', () => {
+  it('sums to 1 and its mean matches the analytic mean', () => {
+    const rates = R(0.5, 0.15, 0.01, 0.12);
+    const pmf = c.tbPmf(rates);
+    expect(pmf.reduce((a, b) => a + b, 0)).toBeCloseTo(1, 6);
+    const pmfMean = pmf.reduce((a, p, i) => a + p * i, 0);
+    expect(pmfMean).toBeCloseTo(c.tbMean(rates), 4);
+  });
+
+  it('P(TB>=0) is 1 and P(>=k) is monotonically non-increasing', () => {
+    const pmf = c.tbPmf(R(0.6, 0.2, 0.02, 0.1));
+    expect(c.pAtLeast(pmf, 0)).toBe(1);
+    let prev = 1;
+    for (let k = 1; k <= 8; k += 1) {
+      const p = c.pAtLeast(pmf, k);
+      expect(p).toBeLessThanOrEqual(prev + 1e-12);
+      prev = p;
+    }
+  });
+});
+
+describe('THE POINT — structure separates hitters an NB would merge', () => {
+  it('a slugger and a slap hitter with the SAME mean get different curves', () => {
+    const slugger = c.tbPmf(R(0, 0, 0, 0.25));   // mean 1.0, all from home runs
+    const slap    = c.tbPmf(R(1.0, 0, 0, 0));    // mean 1.0, all from singles
+    expect(c.tbMean(R(0, 0, 0, 0.25))).toBeCloseTo(c.tbMean(R(1.0, 0, 0, 0)), 6);
+    // The 4-base tail is where an NB on TB alone is blind.
+    expect(c.pAtLeast(slugger, 4)).toBeGreaterThan(10 * c.pAtLeast(slap, 4));
+  });
+
+  it('a home run is ONE event worth four bases, not four events', () => {
+    // P(TB>=4) for a pure HR hitter equals P(at least one HR) exactly.
+    const lambda = 0.25;
+    const pmf = c.tbPmf(R(0, 0, 0, lambda));
+    expect(c.pAtLeast(pmf, 4)).toBeCloseTo(1 - Math.exp(-lambda), 4);
+  });
+});
+
+describe('honest-absent — never fabricate a rate', () => {
+  it('derives singles exactly, reproducing stored total bases', () => {
+    const rates = c.ratesFromLog([row(2, 1, 0, 1), row(1, 0, 0, 0), row(0, 0, 0, 0)]);
+    // game 1: singles = 2-1-0-1 = 0
+    expect(rates.singles).toBeCloseTo((0 + 1 + 0) / 3, 6);
+    expect(rates.home_runs).toBeCloseTo(1 / 3, 6);
+  });
+
+  it('SKIPS an inconsistent row rather than clamping it to zero', () => {
+    // hits < extra-base hits is impossible; clamping would invent a plausible line.
+    const rates = c.ratesFromLog([row(0, 2, 0, 0), row(1, 0, 0, 0), row(1, 0, 0, 0), row(1, 0, 0, 0)]);
+    expect(rates.games).toBe(3);
+  });
+
+  it('returns null below the minimum games — caller falls back to the ladder', () => {
+    expect(c.ratesFromLog([row(1, 0, 0, 0), row(1, 0, 0, 0)])).toBeNull();
+    expect(c.projectTotalBases({ rows: [row(1, 0, 0, 0)], line: 1.5 })).toBeNull();
+  });
+
+  it('a missing component is absent (rate 0), not guessed', () => {
+    const pmf = c.tbPmf({ singles: 0.8 });          // no doubles/triples/HR keys
+    expect(pmf.reduce((a, b) => a + b, 0)).toBeCloseTo(1, 6);
+    expect(c.pAtLeast(pmf, 2)).toBeLessThan(c.pAtLeast(pmf, 1));
+  });
+
+  it('no usable component at all returns null, never a flat distribution', () => {
+    expect(c.tbPmf({})).toBeNull();
+    expect(c.tbPmf({ singles: 'x', doubles: null })).toBeNull();
+  });
+});
+
+describe('projectTotalBases — the full read', () => {
+  const rows = [row(2, 1, 0, 1), row(1, 0, 0, 0), row(0, 0, 0, 0), row(3, 1, 0, 1), row(1, 1, 0, 0)];
+
+  it('inherits the combined multiplier rather than ignoring adjustments', () => {
+    const base = c.projectTotalBases({ rows, line: 1.5, multiplier: 1 });
+    const up   = c.projectTotalBases({ rows, line: 1.5, multiplier: 1.2 });
+    expect(up.mean).toBeGreaterThan(base.mean);
+    expect(up.p_over_line).toBeGreaterThan(base.p_over_line);
+  });
+
+  it('labels its family and carries the independence caveat', () => {
+    const out = c.projectTotalBases({ rows, line: 1.5 });
+    expect(out.family).toBe('compound_weighted_poisson');
+    expect(out.independence_caveat).toBe(true);
+    expect(out.games_used).toBe(5);
+  });
+});