Diagnose proj-v1.1: concentrated mean failure, NOT a similarity problem
READ-ONLY. Nothing built or fixed; the four challengers untouched. 41% OF THE REPORTED GAP WAS A MEASUREMENT ARTIFACT. p_win is P(graded side); proj_p_over_line is P(over); 31.4% of settled rows are UNDER-graded, so comparing them raw measures the ladder backwards on a third of the sample. Matched + direction-aligned (n=437): 0.252 vs champion 0.352, not 0.108 vs 0.331. The PRODUCT is not making this mistake -- I checked; projectionChallenger normalises both to the over basis deliberately. The error was in the measurement. THE LOSS IS CONCENTRATED. hits (n=245, res 0.060) and total_bases (n=49, res 0.009) are 67% of rows and carry essentially no signal. Everything else is fine or better: walks 0.519 vs champion 0.544, runs mean 0.345 vs 0.392, and on DOUBLES the ladder's mean BEATS the champion's (0.207 vs -0.062). IT IS THE MEAN, NOT THE SHAPE. On the two failing families the mean itself carries no signal (0.052, -0.019) against the champion's 0.158 and 0.085. Where the mean is good the probability is good -- shape follows mean. A HYPOTHESIS I TESTED AND DISPROVED: prediction compression. I expected P(>=1 hit) to sit in a narrow band and fail to rank. It does not -- spread ratio 0.94 overall, 0.80 for hits, 0.94 for total_bases. The ladder has comparable spread; it is spread in a direction uncorrelated with outcomes. Recorded because it was a plausible story the data refused. PRIORS AND PLUMBING CLEAN. proj_factors carries form_rate, combined_multiplier and breakdown on every row; proj_point 100% populated with sane centres (hits 0.830 vs line 0.578). Not the environment-style silent-null failure. NAMED CAUSE (structural, flagged as hypothesis not finding): the count model mismatches those two stats. total_bases is a WEIGHTED SUM (1B..HR = 1..4), so an NB treats one home run as four events and mis-states variance -- and TB has the worst result in the table. hits is BOUNDED BY AT-BATS and mostly traded at 0.5, so almost everything rides on P(0), the region where the wrong family hurts most. walks/runs/doubles ARE genuine low-rate counts and are exactly the ones that work. FIX BRANCH: targeted per-stat fix for hits and total_bases. THIS REMOVES THE MLB SIMILARITY BUILD FROM THE CRITICAL PATH -- that branch assumed a GLOBAL mean weakness, and the mean is fine or better on three of six stat families. Similarity may be worth building later, on evidence, not on this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
This commit is contained in:
+23
-16
@@ -29,24 +29,31 @@ stay provisional until re-run** · documented ≠ verified.
|
||||
|
||||
## ▶ NEXT EXECUTABLE ORDER
|
||||
|
||||
**DIAGNOSE WHY proj-v1.1 LOSES.** The distribution ladder is **not dormant** — it
|
||||
is a **fourth accruing challenger**, live at **94.2%** coverage with **437 settled
|
||||
rows**, and it is **losing**: resolution **0.108** vs the champion's **0.331**.
|
||||
**TARGETED PER-STAT FIX for `hits` and `total_bases` in proj-v1.1.** Diagnosis:
|
||||
`specs/proj-v11-diagnosis.md`.
|
||||
|
||||
It is the only one of the candidate layers already carrying real settled evidence,
|
||||
it is the **per-stat distribution §10.3 called the biggest modelling gap**, and its
|
||||
diagnosis decides whether an MLB similarity layer is worth building at all.
|
||||
The ladder's loss is **concentrated, not systemic**. On matched, direction-aligned
|
||||
rows (n=437) the real gap is **0.252 vs 0.352** — not the 0.108 vs 0.331 previously
|
||||
reported, which was **unaligned on direction across 31.4% under-graded rows**.
|
||||
|
||||
**Then, in order:** `archetype_x_archetype` (matchup's upper rung — ready, small
|
||||
radius) → **build** an MLB similarity layer *only if* the diagnosis says a better
|
||||
mean is what proj-v1.1 needs → retire/rewrite `bayesianEngine`.
|
||||
**`hits` (n=245) and `total_bases` (n=49) are 67% of the sample and carry
|
||||
essentially no signal** (0.060 and 0.009). Everything else is fine or better:
|
||||
**`walks` 0.519 vs champion 0.544**, and on **`doubles` the ladder's MEAN beats the
|
||||
champion's** (0.207 vs −0.062). Priors and plumbing are clean; compression was
|
||||
tested and **disproved** (spread ratio 0.80–0.94).
|
||||
|
||||
**Nothing is left to "connect"** — see `specs/dormant-layer-audit.md`:
|
||||
- **similarity** — BUILT but **NBA-shaped** (pace, referees, score state). For MLB
|
||||
it is **CONSTRUCT, not connect**; wiring it would be the sport-stubbed-in breach.
|
||||
- **bayesian** — BUILT but keys on **7 stat names that are not live** and defaults
|
||||
to `'normal'`, so it would silently model count stats as Gaussian. Also
|
||||
**superseded** by `distribution.js`. **Do not connect.**
|
||||
**Hypothesis to test in that order:** the negative binomial mismatches those two
|
||||
stats' structure — `total_bases` is a **weighted sum**, not an event count, and
|
||||
`hits` is **bounded by at-bats**, so both violate the unbounded-count assumption
|
||||
that `walks`/`runs`/`doubles` satisfy.
|
||||
|
||||
> **This diagnosis REMOVES the MLB similarity build from the critical path.** That
|
||||
> branch assumed a *global* mean weakness; the mean is fine or better on three of
|
||||
> six stat families. Similarity may be worth building later — **on evidence, not
|
||||
> on this.**
|
||||
|
||||
**Then:** `archetype_x_archetype` (matchup's upper rung) → retire/rewrite
|
||||
`bayesianEngine` (keys on 7 non-live stat names; defaults count stats to Gaussian).
|
||||
|
||||
### FOUR challengers accruing in parallel — do NOT re-run early
|
||||
Verified firing on a real prod snapshot (293 grades), not inferred:
|
||||
@@ -56,7 +63,7 @@ Verified firing on a real prod snapshot (293 grades), not inferred:
|
||||
| **environment** | **84.6%** | 0.057 | *(shares the arch-v1 pattern)* |
|
||||
| **matchup** (`batter_own_split`) | **82.9%** | 0.015 | `scripts/matchup-axis-holdout.sql` |
|
||||
| **opportunity** | 30.0% | 0.142 | `scripts/opportunity-axis-holdout.sql` |
|
||||
| **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — **currently LOSING 0.108 vs 0.331** |
|
||||
| **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — aligned gap **0.252 vs 0.352**, concentrated in `hits`+`total_bases` |
|
||||
|
||||
All three orthogonal (r ≈ 0 vs projection, `p_win`, line and each other). Each
|
||||
promotes ONLY on its own axis-filtered holdout, and ONLY if **reliability AND
|
||||
|
||||
Reference in New Issue
Block a user