From aa1228ec42f5373a960378a610c4939a48664234 Mon Sep 17 00:00:00 2001 From: Kev Date: Sun, 2 Aug 2026 03:34:14 -0400 Subject: [PATCH] tb-v1 report + plan: diagnosis on trial, branch pre-registered Firing verified on a real prod snapshot: 10/10 total_bases props carry proj_tb_p_over. The snapshot HTTP call returned 524 (Cloudflare's 100s origin timeout vs a ~115s snapshot) but the work completed server-side -- confirmed from the ledger rather than assumed. Face validity is good and diagnostic: means agree almost exactly with the ladder (1.813 vs 1.833), so this is a SHAPE-ONLY intervention, which is what was intended. Component rates are plausible, and Carroll's triples rate (0.112, far above his peers) is a clean check -- he is a speed player and the model sees it. AN OBSERVATION I AM NOT RESOLVING BY EYE: tb-v1 reads systematically LOWER than the ladder (0.424 vs 0.540 at the same mean). That is the expected DIRECTION, since the NB overstates P(>=2) by treating a home run as four accumulating events -- but whether 0.424 is right or an overcorrection is not knowable from face validity. A ~1.8-TB hitter clearing 1.5 empirically sits nearer 45-50%, between the two. I am not claiming tb-v1 is better; the holdout decides. BRANCH PRE-REGISTERED, before the result, so the verdict cannot be reinterpreted afterward: improves -> family-mismatch HOLDS, similarity stays off the critical path, hits is next; does not improve -> hypothesis WRONG and the mean-weakness/similarity branch REOPENS. Also recorded: I hit Number(null)===0 in my own new module -- a null component rate treated as a measured zero, the difference between "never triples" and "we don't know his triple rate". A test caught it. Sixth appearance of this trap in this codebase, and it caught the person writing the warnings about it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc --- specs/MASTER-PLAN.md | 39 +++++------ specs/tb-v1-compound.md | 139 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 156 insertions(+), 22 deletions(-) create mode 100644 specs/tb-v1-compound.md diff --git a/specs/MASTER-PLAN.md b/specs/MASTER-PLAN.md index 6b388ef..babb643 100644 --- a/specs/MASTER-PLAN.md +++ b/specs/MASTER-PLAN.md @@ -29,33 +29,27 @@ stay provisional until re-run** · documented ≠ verified. ## ▶ NEXT EXECUTABLE ORDER -**TARGETED PER-STAT FIX for `hits` and `total_bases` in proj-v1.1.** Diagnosis: -`specs/proj-v11-diagnosis.md`. +**AWAIT the tb-v1 verdict, then `hits`.** `specs/tb-v1-compound.md`. -The ladder's loss is **concentrated, not systemic**. On matched, direction-aligned -rows (n=437) the real gap is **0.252 vs 0.352** — not the 0.108 vs 0.331 previously -reported, which was **unaligned on direction across 31.4% under-graded rows**. +**tb-v1 is LIVE as a challenger** — total_bases modelled as the compound outcome +it is (per-component Poisson rates convolved exactly) instead of one negative +binomial that treats a home run as four events. Firing on **10/10 TB props**; +means agree with the ladder, **only the shape differs** — the intended +intervention. -**`hits` (n=245) and `total_bases` (n=49) are 67% of the sample and carry -essentially no signal** (0.060 and 0.009). Everything else is fine or better: -**`walks` 0.519 vs champion 0.544**, and on **`doubles` the ladder's MEAN beats the -champion's** (0.207 vs −0.062). Priors and plumbing are clean; compression was -tested and **disproved** (spread ratio 0.80–0.94). +**The diagnosis is on trial, and the branch is pre-registered:** +- **tb-v1 improves TB resolution** → family-mismatch **HOLDS**, similarity stays + **off** the critical path, **`hits` is next** (AB-bounded, mostly 0.5 lines, so + it rides on P(0)). +- **tb-v1 does not improve** → hypothesis **WRONG**, the **mean-weakness / + MLB-similarity branch REOPENS**. -**Hypothesis to test in that order:** the negative binomial mismatches those two -stats' structure — `total_bases` is a **weighted sum**, not an event count, and -`hits` is **bounded by at-bats**, so both violate the unbounded-count assumption -that `walks`/`runs`/`doubles` satisfy. +**Verdict n-blocked:** 0 settled TB rows carry tb-v1 yet; first settle is +tomorrow. `scripts/tb-compound-holdout.sql` is TB-rows-only and direction-aligned. -> **This diagnosis REMOVES the MLB similarity build from the critical path.** That -> branch assumed a *global* mean weakness; the mean is fine or better on three of -> six stat families. Similarity may be worth building later — **on evidence, not -> on this.** +**Then:** `archetype_x_archetype` → retire/rewrite `bayesianEngine`. -**Then:** `archetype_x_archetype` (matchup's upper rung) → retire/rewrite -`bayesianEngine` (keys on 7 non-live stat names; defaults count stats to Gaussian). - -### FOUR challengers accruing in parallel — do NOT re-run early +### FIVE challengers accruing in parallel — do NOT re-run early Verified firing on a real prod snapshot (293 grades), not inferred: | axis | coverage | mean \|nudge\| | holdout query | @@ -63,6 +57,7 @@ Verified firing on a real prod snapshot (293 grades), not inferred: | **environment** | **84.6%** | 0.057 | *(shares the arch-v1 pattern)* | | **matchup** (`batter_own_split`) | **82.9%** | 0.015 | `scripts/matchup-axis-holdout.sql` | | **opportunity** | 30.0% | 0.142 | `scripts/opportunity-axis-holdout.sql` | +| **tb-v1** (compound total_bases) | **10/10 TB props** | *(shape, not a nudge)* | pre-registered: improves → `hits` next; doesn't → similarity reopens | | **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — aligned gap **0.252 vs 0.352**, concentrated in `hits`+`total_bases` | All three orthogonal (r ≈ 0 vs projection, `p_win`, line and each other). Each diff --git a/specs/tb-v1-compound.md b/specs/tb-v1-compound.md new file mode 100644 index 0000000..8c74b33 --- /dev/null +++ b/specs/tb-v1-compound.md @@ -0,0 +1,139 @@ +# tb-v1 — TOTAL BASES AS A COMPOUND OUTCOME (challenger) + +**Date:** 2026-08-02 · current ladder + champion `p_win` **byte-identical** · +ledger-only · **firing verified on a real prod snapshot**. + +**Gates:** 4,104 tests / 329 suites green · `next build` exit 0 · tb-v1 on +**10/10 total_bases props** · holdout committed, verdict n-blocked. + +--- + +## STEP 0 — COMPONENTS CONFIRMED ON REAL DATA + +statsapi has **no `singles` field**. But on a real 10-game log: + +``` +singles = hits − doubles − triples − homeRuns +singles + 2·doubles + 3·triples + 4·homeRuns == stored totalBases ✓ exact +``` + +**The decomposition is exact, not an approximation.** Verified before any code +was written — this fix depended on it. + +--- + +## THE MODEL + +Each component gets its own per-game Poisson rate; TB is their weighted sum, and +the PMF is built by **exact convolution** rather than simulated (TB support is +small). It inherits the **same combined multiplier** proj-v1.1 computes, so the +two models differ **only in structure**. + +### Why this is the fix, in one number + +With **identical mean TB of 1.0**: + +| hitter | P(TB≥2) | **P(TB≥4)** | +|---|---:|---:| +| pure home-run hitter | 0.221 | **0.221** | +| pure singles hitter | 0.264 | **0.019** | + +**A 12× difference at the 4-base rung that an NB on TB alone cannot express**, +because it treats one home run as four independent events. A test asserts this +separation, and asserts that P(TB≥4) for a pure-HR hitter equals P(at least one +HR) **exactly**. + +### The approximation, stated + +Components are modelled as **independent** Poissons. They are not: a plate +appearance that becomes a double cannot also become a single, so they are weakly +negatively correlated and **independence slightly overstates the tail**. Closer +to the truth than what it replaces; **not a solved problem**, and labelled +`independence_caveat: true` on every record. + +--- + +## FIRING ON PROD — 10/10 TB PROPS + +| | | +|---|---:| +| rows written | 241 | +| total_bases props | 10 | +| **with `proj_tb_p_over`** | **10 (100%)** | +| range / sd | 0.295–0.424 / 0.045 | + +Real records: + +| player | line | ladder mean | **ladder P** | **tb-v1 P** | tb-v1 mean | rates (1B/2B/3B/HR) | games | +|---|---:|---:|---:|---:|---:|---|---:| +| Ketel Marte | 1.5 | 1.813 | **0.540** | **0.424** | 1.833 | .614/.208/.028/.180 | 106 | +| Corbin Carroll | 1.5 | 1.814 | 0.540 | 0.424 | 1.815 | .468/.206/**.112**/.150 | 107 | +| Gabriel Moreno | 1.5 | 1.590 | 0.471 | 0.405 | 1.608 | .773/.221/.000/.098 | 84 | + +**The means agree almost exactly** (1.813 vs 1.833) — both use the same rate +machinery. **Only the shape differs**, which is precisely the intended +intervention. Carroll's triples rate (0.112, far above the others) is a good +face-validity check: he is a speed player, and the model sees it. + +### An observation the holdout must settle + +**tb-v1 is systematically LOWER than the ladder** (0.424 vs 0.540 at the same +mean). That is the expected *direction* — the NB overstates P(≥2) by treating a +home run as four accumulating events. But **whether 0.424 is right, or an +overcorrection, is not knowable from face validity.** A ~1.8-TB hitter clearing +1.5 empirically sits nearer 45–50%, which is between the two. + +**I am not claiming tb-v1 is better. The holdout decides.** + +--- + +## HOLDOUT — COMMITTED, VERDICT n-BLOCKED + +`scripts/tb-compound-holdout.sql`, with two guards baked in: + +1. **TB rows ONLY.** total_bases is 49 of 437 settled rows; averaging into other + stats would hide the effect entirely. +2. **DIRECTION-ALIGNED.** `p_win` is P(graded side); both projection values are + P(over). 31.4% of rows are under-graded, and the unaligned comparison is the + artifact that accounted for **41%** of the ladder's apparent loss. + +**Settled TB rows carrying tb-v1: 0** — it went live today, on games not yet +played. First settle pass is tomorrow. + +--- + +## THE DIAGNOSIS IS ON TRIAL — recorded in advance + +> **If tb-v1 materially improves TB resolution** → the family-mismatch hypothesis +> **HOLDS**, similarity stays **off** the critical path, and **`hits` is next** +> (its own order: AB-bounded, mostly 0.5 lines, so it rides on P(0)). +> +> **If it does NOT improve** → the hypothesis is **WRONG**, and the +> **mean-weakness / MLB-similarity branch REOPENS**. + +Written down now, before the result, so the verdict cannot be reinterpreted after +the fact. + +--- + +## A TRAP I HIT IN MY OWN CODE + +`Number(null) === 0` — again, and this time in the new module. A null component +rate passed a naive finite check and was treated as a **measured zero**. That is +the difference between *"this player never triples"* and *"we do not know his +triple rate"*, and it would have silently narrowed the distribution. + +**A test caught it.** Both `tbPmf` and `tbMean` now reject `null`/`''`/boolean +strictly. Recording it because this is the sixth time this specific trap has +appeared in this codebase, and it caught the person who has been writing the +warnings about it. + +## TAGS + +**VERIFIED:** the TB decomposition is exact on real logs · tb-v1 fires on 10/10 TB +props · means agree with the ladder (shape-only intervention) · component rates +face-valid (Carroll's triples) · 12× separation at the 4-base rung. + +**NOT CLAIMED:** that tb-v1 is better. n-blocked until TB rows settle. + +**STATED LIMITATION:** component independence overstates the tail.