diff --git a/specs/MASTER-PLAN.md b/specs/MASTER-PLAN.md index 80d0282..46a1d97 100644 --- a/specs/MASTER-PLAN.md +++ b/specs/MASTER-PLAN.md @@ -29,16 +29,26 @@ stay provisional until re-run** · documented ≠ verified. ## ▶ NEXT EXECUTABLE ORDER -**CONNECT THE STILL-DORMANT LAYERS** — similarity, Bayesian, and the distribution -ladder (`projection/distribution.js`, proj-v1.1, built and unconnected). Same -pattern that just worked three times: audit what is actually firing, repair the -joins, add as a labelled challenger, prove on an axis-filtered holdout. +**DIAGNOSE WHY proj-v1.1 LOSES.** The distribution ladder is **not dormant** — it +is a **fourth accruing challenger**, live at **94.2%** coverage with **437 settled +rows**, and it is **losing**: resolution **0.108** vs the champion's **0.331**. -*Blocked on nothing.* The alternative next order is -**`archetype_x_archetype`** — the matchup ladder's upper rung, which needs the -opposing starter classified per game. +It is the only one of the candidate layers already carrying real settled evidence, +it is the **per-stat distribution §10.3 called the biggest modelling gap**, and its +diagnosis decides whether an MLB similarity layer is worth building at all. -### THREE challengers accruing in parallel — do NOT re-run early +**Then, in order:** `archetype_x_archetype` (matchup's upper rung — ready, small +radius) → **build** an MLB similarity layer *only if* the diagnosis says a better +mean is what proj-v1.1 needs → retire/rewrite `bayesianEngine`. + +**Nothing is left to "connect"** — see `specs/dormant-layer-audit.md`: +- **similarity** — BUILT but **NBA-shaped** (pace, referees, score state). For MLB + it is **CONSTRUCT, not connect**; wiring it would be the sport-stubbed-in breach. +- **bayesian** — BUILT but keys on **7 stat names that are not live** and defaults + to `'normal'`, so it would silently model count stats as Gaussian. Also + **superseded** by `distribution.js`. **Do not connect.** + +### FOUR challengers accruing in parallel — do NOT re-run early Verified firing on a real prod snapshot (293 grades), not inferred: | axis | coverage | mean \|nudge\| | holdout query | @@ -46,6 +56,7 @@ Verified firing on a real prod snapshot (293 grades), not inferred: | **environment** | **84.6%** | 0.057 | *(shares the arch-v1 pattern)* | | **matchup** (`batter_own_split`) | **82.9%** | 0.015 | `scripts/matchup-axis-holdout.sql` | | **opportunity** | 30.0% | 0.142 | `scripts/opportunity-axis-holdout.sql` | +| **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — **currently LOSING 0.108 vs 0.331** | All three orthogonal (r ≈ 0 vs projection, `p_win`, line and each other). Each promotes ONLY on its own axis-filtered holdout, and ONLY if **reliability AND diff --git a/specs/dormant-layer-audit.md b/specs/dormant-layer-audit.md new file mode 100644 index 0000000..50abff4 --- /dev/null +++ b/specs/dormant-layer-audit.md @@ -0,0 +1,152 @@ +# DORMANT-LAYER AUDIT — similarity / Bayesian / distribution ladder + +**Date:** 2026-08-02 · **READ-ONLY** — nothing connected, built or wired. The +three accruing challengers were not touched. + +--- + +## VERDICT: NONE OF THE THREE SHOULD BE CONNECTED + +One is **already live**, one is **wrong-sport**, one is **superseded and would +silently regress the model**. "Dormant" meant three different things, exactly as +the order suspected — and in no case is the answer "connect it". + +| layer | state | blast radius | verdict | +|---|---|---|---| +| **distribution ladder** | 🟢 **LIVE — 94.2% coverage, 437 settled rows** | projection-formation | **not dormant. Already a 4th challenger — and currently losing** | +| **similarity** | BUILT · **0 callers** · **NBA-shaped** | projection-formation | **CONSTRUCT, not connect** (for MLB) | +| **bayesian** | BUILT · **0 callers** · **stale stat names** · **superseded** | projection-formation | **do not connect — retire or rewrite** | + +--- + +## 1. DISTRIBUTION LADDER — NOT DORMANT + +`projection/distribution.js` is consumed by `projectionChallenger` (**proj-v1.1**), +which `snapshotService` calls on every snapshot. Measured on a real prod snapshot: + +| field | coverage | +|---|---:| +| `proj_version` | 293/293 | +| `proj_point`, `proj_distribution`, `proj_ladder`, `proj_p_over_line`, `proj_factors` | **276/293 = 94.2%** | + +**It is PER-STAT — doctrine-correct.** `STAT_FIELD` carries a distinct league +prior per stat (hits 0.90 · total_bases 1.45 · home_runs 0.15 · doubles 0.18 · +triples 0.02 · strikeouts 1.05 · rbi 0.50 · runs 0.50 · walks 0.32), each feeding +a gamma-Poisson posterior → negative binomial. **It is not a single additive +index.** + +> **§10.3's "single additive index across hits/Ks/TB" refers to `engine1`'s GRADE, +> not to this ladder.** The ladder was built correctly. Worth correcting in the +> plan, because it made a solved problem look open. + +### It is a fourth accruing challenger — and it is losing + +``` +settled rows with proj-v1.1: 437 (2026-07-23 → 2026-08-02) +resolution, proj-v1.1: 0.108 +resolution, champion p_win: 0.331 +``` + +**The "it lost to the champion" verdict is still current — and no longer thin.** +437 settled rows is well-powered. + +*Caveat: those two correlations are over different row sets (proj is present on +437 of 848 settled). Indicative, not like-for-like — the committed per-axis +holdout is what settles it properly.* + +--- + +## 2. SIMILARITY — BUILT, BUT FOR BASKETBALL + +`similarityEngine.js` — **zero callers**, genuinely unwired. But look at what it +weights: + +``` +pace 0.15 · referee_tendency 0.06 · lineup_context 0.12 +score_state_context 0.05 · travel_fatigue 0.08 · opponent_defensive_rating 0.14 +``` + +**Pace, referees, score state, defensive rating — this is NBA vocabulary.** MLB has +no pace and no referee tendency; "lineup context" means batting order, not a +five-man unit. + +**Connecting it to MLB would be "a sport stubbed in on another sport's template" — +the exact breach CLAUDE.md forbids.** It would produce similarity scores dominated +by factors that are absent or meaningless, and the failure would be quiet: missing +factors are skipped, so the score would silently collapse onto whatever few +dimensions happened to be present. + +**Verdict: CONSTRUCT, not connect.** An MLB similarity layer needs MLB comparables +— park, opposing hand and arsenal, lineup slot, rest. That is a **build order**, +and it should be judged on its own holdout, not inherited from a basketball model. + +--- + +## 3. BAYESIAN — SUPERSEDED, AND A TRAP IF CONNECTED + +`bayesianEngine.js` — zero callers. Two independent reasons not to wire it: + +**(a) Its stat names are stale.** `DISTRIBUTION_SHAPES` keys on +`rbis`, `runs_scored`, `strikeouts_batter`, `outs_recorded`, `pitcher_strikeouts`, +`walks_allowed`, `pitches_thrown`. + +**None of those are live stat keys.** Per CLAUDE.md S41 the live names are `rbi`, +`runs`, `outs`, `strikeouts`. `getDistributionShape` falls through to +**`'normal'`** on an unknown key — so connecting it as-is would model **count +stats as Gaussian**, silently, on most MLB props. A negative-binomial engine that +defaults to normal for the stats it was built for is worse than no engine. + +**(b) It duplicates a layer that is already live and better.** `distribution.js` +does the same job — per-stat count distributions — with a gamma-Poisson posterior +and real per-stat priors, and it is already running at 94.2%. `bayesianEngine` is +an older, hand-declared-shape version of it. + +**Verdict: do not connect. Retire it or rewrite it against the live stat +vocabulary** — and only if it offers something `distribution.js` does not. + +--- + +## 4. DEPENDENCY — foundation-first still applies, just not as posed + +The order asked whether the ladder depends on the projection similarity/Bayesian +would strengthen. **Yes** — proj-v1.1 forms its own mean from game-log rates plus a +league prior, then wraps a negative binomial around it. **A better mean would make +the shape more useful.** + +But the foundation-first conclusion inverts: **the ladder is already connected and +the two would-be foundations are not usable.** So this is not "connect similarity +before the ladder" — it is **"the ladder is live and underperforming, and +strengthening its mean requires building an MLB similarity layer that does not +exist yet."** + +--- + +## THE CONNECT ORDER — as an honest sequence + +**There is nothing to connect. The next orders are, in priority:** + +1. **🔴 Diagnose why proj-v1.1 loses (0.108 vs 0.331 on 437 settled rows).** + Most foundational, already connected, well-powered, and it is the **per-stat + distribution §10.3 called the biggest modelling gap**. If the shape is right + but the mean is weak, that is a specific, findable defect — and it tells us + whether an MLB similarity layer is worth building at all. +2. **`archetype_x_archetype`** — the matchup ladder's upper rung. Ready, scoped, + small blast radius (adjustment, not projection-formation). +3. **Build an MLB similarity layer** — only if (1) says a better mean is what + proj-v1.1 needs. Construct, not connect. +4. **Retire or rewrite `bayesianEngine`** — housekeeping; it is a live trap for + whoever wires it next without reading the stat keys. + +**Recommended first: (1).** It is the only one of the four that is already +carrying real settled evidence, and it decides whether (3) is worth doing. + +## TAGS + +**VERIFIED:** distribution ladder live at 94.2% (276/293), per-stat with 9 distinct +priors, 437 settled rows, resolution 0.108 vs champion 0.331 · similarityEngine and +bayesianEngine both have zero callers · similarity weights are NBA-shaped · +bayesianEngine keys on 7 stat names that are not live and defaults to `'normal'`. + +**CORRECTED:** the distribution ladder is **not dormant** — it is a fourth accruing +challenger. And §10.3's "single additive index" is `engine1`'s grade, **not** this +ladder, which is properly per-stat.