Diagnose proj-v1.1: concentrated mean failure, NOT a similarity problem

READ-ONLY. Nothing built or fixed; the four challengers untouched.

41% OF THE REPORTED GAP WAS A MEASUREMENT ARTIFACT. p_win is P(graded
side); proj_p_over_line is P(over); 31.4% of settled rows are UNDER-graded,
so comparing them raw measures the ladder backwards on a third of the
sample. Matched + direction-aligned (n=437): 0.252 vs champion 0.352, not
0.108 vs 0.331. The PRODUCT is not making this mistake -- I checked;
projectionChallenger normalises both to the over basis deliberately. The
error was in the measurement.

THE LOSS IS CONCENTRATED. hits (n=245, res 0.060) and total_bases (n=49,
res 0.009) are 67% of rows and carry essentially no signal. Everything else
is fine or better: walks 0.519 vs champion 0.544, runs mean 0.345 vs 0.392,
and on DOUBLES the ladder's mean BEATS the champion's (0.207 vs -0.062).

IT IS THE MEAN, NOT THE SHAPE. On the two failing families the mean itself
carries no signal (0.052, -0.019) against the champion's 0.158 and 0.085.
Where the mean is good the probability is good -- shape follows mean.

A HYPOTHESIS I TESTED AND DISPROVED: prediction compression. I expected
P(>=1 hit) to sit in a narrow band and fail to rank. It does not -- spread
ratio 0.94 overall, 0.80 for hits, 0.94 for total_bases. The ladder has
comparable spread; it is spread in a direction uncorrelated with outcomes.
Recorded because it was a plausible story the data refused.

PRIORS AND PLUMBING CLEAN. proj_factors carries form_rate,
combined_multiplier and breakdown on every row; proj_point 100% populated
with sane centres (hits 0.830 vs line 0.578). Not the environment-style
silent-null failure.

NAMED CAUSE (structural, flagged as hypothesis not finding): the count
model mismatches those two stats. total_bases is a WEIGHTED SUM (1B..HR =
1..4), so an NB treats one home run as four events and mis-states variance
-- and TB has the worst result in the table. hits is BOUNDED BY AT-BATS and
mostly traded at 0.5, so almost everything rides on P(0), the region where
the wrong family hurts most. walks/runs/doubles ARE genuine low-rate counts
and are exactly the ones that work.

FIX BRANCH: targeted per-stat fix for hits and total_bases. THIS REMOVES
THE MLB SIMILARITY BUILD FROM THE CRITICAL PATH -- that branch assumed a
GLOBAL mean weakness, and the mean is fine or better on three of six stat
families. Similarity may be worth building later, on evidence, not on this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
This commit is contained in:
Kev
2026-08-02 02:50:13 -04:00
parent f5997778a2
commit 48706210fe
2 changed files with 182 additions and 16 deletions
+23 -16
View File
@@ -29,24 +29,31 @@ stay provisional until re-run** · documented ≠ verified.
## ▶ NEXT EXECUTABLE ORDER
**DIAGNOSE WHY proj-v1.1 LOSES.** The distribution ladder is **not dormant** — it
is a **fourth accruing challenger**, live at **94.2%** coverage with **437 settled
rows**, and it is **losing**: resolution **0.108** vs the champion's **0.331**.
**TARGETED PER-STAT FIX for `hits` and `total_bases` in proj-v1.1.** Diagnosis:
`specs/proj-v11-diagnosis.md`.
It is the only one of the candidate layers already carrying real settled evidence,
it is the **per-stat distribution §10.3 called the biggest modelling gap**, and its
diagnosis decides whether an MLB similarity layer is worth building at all.
The ladder's loss is **concentrated, not systemic**. On matched, direction-aligned
rows (n=437) the real gap is **0.252 vs 0.352** — not the 0.108 vs 0.331 previously
reported, which was **unaligned on direction across 31.4% under-graded rows**.
**Then, in order:** `archetype_x_archetype` (matchup's upper rung — ready, small
radius) → **build** an MLB similarity layer *only if* the diagnosis says a better
mean is what proj-v1.1 needs → retire/rewrite `bayesianEngine`.
**`hits` (n=245) and `total_bases` (n=49) are 67% of the sample and carry
essentially no signal** (0.060 and 0.009). Everything else is fine or better:
**`walks` 0.519 vs champion 0.544**, and on **`doubles` the ladder's MEAN beats the
champion's** (0.207 vs 0.062). Priors and plumbing are clean; compression was
tested and **disproved** (spread ratio 0.800.94).
**Nothing is left to "connect"** — see `specs/dormant-layer-audit.md`:
- **similarity** — BUILT but **NBA-shaped** (pace, referees, score state). For MLB
it is **CONSTRUCT, not connect**; wiring it would be the sport-stubbed-in breach.
- **bayesian** — BUILT but keys on **7 stat names that are not live** and defaults
to `'normal'`, so it would silently model count stats as Gaussian. Also
**superseded** by `distribution.js`. **Do not connect.**
**Hypothesis to test in that order:** the negative binomial mismatches those two
stats' structure — `total_bases` is a **weighted sum**, not an event count, and
`hits` is **bounded by at-bats**, so both violate the unbounded-count assumption
that `walks`/`runs`/`doubles` satisfy.
> **This diagnosis REMOVES the MLB similarity build from the critical path.** That
> branch assumed a *global* mean weakness; the mean is fine or better on three of
> six stat families. Similarity may be worth building later — **on evidence, not
> on this.**
**Then:** `archetype_x_archetype` (matchup's upper rung) → retire/rewrite
`bayesianEngine` (keys on 7 non-live stat names; defaults count stats to Gaussian).
### FOUR challengers accruing in parallel — do NOT re-run early
Verified firing on a real prod snapshot (293 grades), not inferred:
@@ -56,7 +63,7 @@ Verified firing on a real prod snapshot (293 grades), not inferred:
| **environment** | **84.6%** | 0.057 | *(shares the arch-v1 pattern)* |
| **matchup** (`batter_own_split`) | **82.9%** | 0.015 | `scripts/matchup-axis-holdout.sql` |
| **opportunity** | 30.0% | 0.142 | `scripts/opportunity-axis-holdout.sql` |
| **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — **currently LOSING 0.108 vs 0.331** |
| **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — aligned gap **0.252 vs 0.352**, concentrated in `hits`+`total_bases` |
All three orthogonal (r ≈ 0 vs projection, `p_win`, line and each other). Each
promotes ONLY on its own axis-filtered holdout, and ONLY if **reliability AND