builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Diagnose proj-v1.1: concentrated mean failure, NOT a similarity problem

Browse Source 
READ-ONLY. Nothing built or fixed; the four challengers untouched.

41% OF THE REPORTED GAP WAS A MEASUREMENT ARTIFACT. p_win is P(graded
side); proj_p_over_line is P(over); 31.4% of settled rows are UNDER-graded,
so comparing them raw measures the ladder backwards on a third of the
sample. Matched + direction-aligned (n=437): 0.252 vs champion 0.352, not
0.108 vs 0.331. The PRODUCT is not making this mistake -- I checked;
projectionChallenger normalises both to the over basis deliberately. The
error was in the measurement.

THE LOSS IS CONCENTRATED. hits (n=245, res 0.060) and total_bases (n=49,
res 0.009) are 67% of rows and carry essentially no signal. Everything else
is fine or better: walks 0.519 vs champion 0.544, runs mean 0.345 vs 0.392,
and on DOUBLES the ladder's mean BEATS the champion's (0.207 vs -0.062).

IT IS THE MEAN, NOT THE SHAPE. On the two failing families the mean itself
carries no signal (0.052, -0.019) against the champion's 0.158 and 0.085.
Where the mean is good the probability is good -- shape follows mean.

A HYPOTHESIS I TESTED AND DISPROVED: prediction compression. I expected
P(>=1 hit) to sit in a narrow band and fail to rank. It does not -- spread
ratio 0.94 overall, 0.80 for hits, 0.94 for total_bases. The ladder has
comparable spread; it is spread in a direction uncorrelated with outcomes.
Recorded because it was a plausible story the data refused.

PRIORS AND PLUMBING CLEAN. proj_factors carries form_rate,
combined_multiplier and breakdown on every row; proj_point 100% populated
with sane centres (hits 0.830 vs line 0.578). Not the environment-style
silent-null failure.

NAMED CAUSE (structural, flagged as hypothesis not finding): the count
model mismatches those two stats. total_bases is a WEIGHTED SUM (1B..HR =
1..4), so an NB treats one home run as four events and mis-states variance
-- and TB has the worst result in the table. hits is BOUNDED BY AT-BATS and
mostly traded at 0.5, so almost everything rides on P(0), the region where
the wrong family hurts most. walks/runs/doubles ARE genuine low-rate counts
and are exactly the ones that work.

FIX BRANCH: targeted per-stat fix for hits and total_bases. THIS REMOVES
THE MLB SIMILARITY BUILD FROM THE CRITICAL PATH -- that branch assumed a
GLOBAL mean weakness, and the mean is fine or better on three of six stat
families. Similarity may be worth building later, on evidence, not on this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
This commit is contained in:
Kev
2026-08-02 02:50:13 -04:00
parent f5997778a2
commit 48706210fe
2 changed files with 182 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/MASTER-PLAN.md
+23 -16
View File
@@ -29,24 +29,31 @@ stay provisional until re-run** · documented ≠ verified.
## ▶ NEXT EXECUTABLE ORDER
**DIAGNOSE WHY proj-v1.1 LOSES.** The distribution ladder is **not dormant** — it
is a **fourth accruing challenger**, live at **94.2%** coverage with **437 settled
rows**, and it is **losing**: resolution **0.108** vs the champion's **0.331**.
**TARGETED PER-STAT FIX for `hits` and `total_bases` in proj-v1.1.** Diagnosis:
`specs/proj-v11-diagnosis.md`.
It is the only one of the candidate layers already carrying real settled evidence,
it is the **per-stat distribution §10.3 called the biggest modelling gap**, and its
diagnosis decides whether an MLB similarity layer is worth building at all.
The ladder's loss is **concentrated, not systemic**. On matched, direction-aligned
rows (n=437) the real gap is **0.252 vs 0.352** — not the 0.108 vs 0.331 previously
reported, which was **unaligned on direction across 31.4% under-graded rows**.
**Then, in order:** `archetype_x_archetype` (matchup's upper rung — ready, small
radius) → **build** an MLB similarity layer *only if* the diagnosis says a better
mean is what proj-v1.1 needs → retire/rewrite `bayesianEngine`.
**`hits` (n=245) and `total_bases` (n=49) are 67% of the sample and carry
essentially no signal** (0.060 and 0.009). Everything else is fine or better:
**`walks` 0.519 vs champion 0.544**, and on **`doubles` the ladder's MEAN beats the
champion's** (0.207 vs −0.062). Priors and plumbing are clean; compression was
tested and **disproved** (spread ratio 0.80–0.94).
**Nothing is left to "connect"** — see `specs/dormant-layer-audit.md`:
- **similarity** — BUILT but **NBA-shaped** (pace, referees, score state). For MLB
it is **CONSTRUCT, not connect**; wiring it would be the sport-stubbed-in breach.
- **bayesian** — BUILT but keys on **7 stat names that are not live** and defaults
to `'normal'`, so it would silently model count stats as Gaussian. Also
**superseded** by `distribution.js`. **Do not connect.**
**Hypothesis to test in that order:** the negative binomial mismatches those two
stats' structure — `total_bases` is a **weighted sum**, not an event count, and
`hits` is **bounded by at-bats**, so both violate the unbounded-count assumption
that `walks`/`runs`/`doubles` satisfy.
> **This diagnosis REMOVES the MLB similarity build from the critical path.** That
> branch assumed a *global* mean weakness; the mean is fine or better on three of
> six stat families. Similarity may be worth building later — **on evidence, not
> on this.**
**Then:** `archetype_x_archetype` (matchup's upper rung) → retire/rewrite
`bayesianEngine` (keys on 7 non-live stat names; defaults count stats to Gaussian).
### FOUR challengers accruing in parallel — do NOT re-run early
Verified firing on a real prod snapshot (293 grades), not inferred:
@@ -56,7 +63,7 @@ Verified firing on a real prod snapshot (293 grades), not inferred:
| **environment** | **84.6%** | 0.057 | *(shares the arch-v1 pattern)* |
| **matchup** (`batter_own_split`) | **82.9%** | 0.015 | `scripts/matchup-axis-holdout.sql` |
| **opportunity** | 30.0% | 0.142 | `scripts/opportunity-axis-holdout.sql` |
| **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — **currently LOSING 0.108 vs 0.331** |
| **proj-v1.1** (distribution ladder) | **94.2%** | *(forms the projection, not a nudge)* | 437 settled — aligned gap **0.252 vs 0.352**, concentrated in `hits`+`total_bases` |
All three orthogonal (r ≈ 0 vs projection, `p_win`, line and each other). Each
promotes ONLY on its own axis-filtered holdout, and ONLY if **reliability AND
specs/proj-v11-diagnosis.md
+159
View File
@@ -0,0 +1,159 @@
# WHY proj-v1.1 LOSES — DIAGNOSIS
**Date:** 2026-08-02 · **READ-ONLY** — nothing built, fixed or wired. The four
accruing challengers were not touched.
---
## HEADLINE: THE GAP IS REAL BUT WAS OVERSTATED, AND IT IS NOT A SIMILARITY PROBLEM
> **CAUSE: a bad MEAN on TWO stat families — `hits` and `total_bases` — which are
> 67% of the sample. Everything else works.**
>
> **FIX BRANCH: targeted per-stat model fix. NOT an MLB similarity build.**
The diagnosis removes the biggest remaining build, which is what it was for.
---
## STEP 1 — MATCHED ROWS: 41% OF THE "GAP" WAS A MEASUREMENT ARTIFACT
`p_win` is P(**graded side**). `proj_p_over_line` is P(**over**). **31.4% of settled
rows are UNDER-graded**, so comparing raw P(over) against a graded-under outcome
measures the ladder backwards on a third of the sample.
Matched rows, n=437:
| | resolution |
|---|---:|
| champion `p_win` | **0.3523** |
| proj-v1.1, **unaligned** (as previously reported) | 0.1494 |
| proj-v1.1, **direction-aligned** | **0.2521** |
**Aligning direction recovers 41% of the apparent gap.** The previously reported
*0.108 vs 0.331* substantially overstated the loss.
**The product itself is NOT making this mistake** — I checked. `projectionChallenger`
deliberately normalises both quantities to the over basis (`proj_book_implied` =
`fairOver`, converting the graded-side fair when direction is under). **The error
was in the measurement, not the model.** The real gap is **0.252 vs 0.352**.
---
## STEP 2 — PER-STAT: THE LOSS IS CONCENTRATED, NOT SYSTEMIC
| stat | n | base | **res champ** | **res proj** | champ MEAN | proj MEAN |
|---|---:|---:|---:|---:|---:|---:|
| **hits** | **245** | .588 | 0.204 | **0.060** | 0.158 | **0.052** |
| **total_bases** | **49** | .612 | 0.273 | **0.009** | 0.085 | **−0.019** |
| rbi | 42 | .333 | 0.412 | 0.218 | 0.349 | 0.159 |
| runs | 38 | .579 | 0.467 | 0.270 | 0.392 | **0.345** |
| **walks** | 29 | .517 | 0.544 | **0.519** | 0.297 | **0.519** |
| **doubles** | 28 | .179 | 0.315 | 0.213 | **−0.062** | **0.207** |
| **ALL** | 437 | .533 | 0.352 | 0.252 | 0.199 | 0.166 |
**`hits` + `total_bases` = 294 of 437 rows (67%), and on both the ladder has
essentially NO signal.** They drag the aggregate on their own.
**Where the ladder works, it works well:**
- **`walks`: 0.519 vs the champion's 0.544** — competitive.
- **`doubles`: the ladder's MEAN (0.207) BEATS the champion's (−0.062).**
- `runs`: mean 0.345 vs 0.392 — close.
**The machinery is not broken. Two stat families are.**
---
## STEP 3 — MEAN vs SHAPE: IT IS THE MEAN
For the two failing families the **mean itself carries no signal** — `hits` 0.052,
`total_bases` **−0.019** — while the champion's mean on the same rows carries 0.158
and 0.085. The distribution cannot rescue a mean that does not rank.
Conversely, where the mean is good (`walks` 0.519, `runs` 0.345, `doubles` 0.207)
the ladder's probability is good. **Shape follows mean, cleanly.**
### A hypothesis I tested and DISPROVED
I expected **prediction compression** — that P(≥1 hit) would land in a narrow band
across players and so could not rank. **Wrong:**
| stat | sd champion | sd proj | ratio |
|---|---:|---:|---:|
| ALL | 0.2048 | 0.1932 | **0.94** |
| hits | 0.1703 | 0.1369 | 0.80 |
| total_bases | 0.1876 | 0.1758 | **0.94** |
**The ladder has comparable spread. It is not compressed — it is spread in a
direction uncorrelated with outcomes.** Recording this because it was a plausible
story that the data refused.
---
## STEP 4 — PRIOR / PLUMBING INTEGRITY: CLEAN
No silent default found. On a real snapshot, `proj_factors` carries `form_rate`,
`combined_multiplier`, `breakdown`, `book_implied_basis` on every row, and
`proj_point` is populated 100%. The central values are sane:
| stat | mean `proj_point` | mean line |
|---|---:|---:|
| hits | 0.830 | 0.578 |
| total_bases | 1.639 | 1.500 |
| strikeouts | 4.972 | 5.204 |
**The priors are reaching the posterior. This is not the environment-style
silent-null failure.**
---
## THE CAUSE, AND WHY THOSE TWO STATS
**Named cause: the count model does not match the generative structure of `hits`
and `total_bases`.**
Stated as a **hypothesis** — it follows from the structure, and this diagnosis did
not test it directly:
- **`total_bases` is not a count of events — it is a WEIGHTED SUM** (1B=1, 2B=2,
3B=3, HR=4). A negative binomial fitted to TB treats one home run as "four
events", which mis-states the variance badly. This is a structural mismatch, not
a tuning error — and TB has the worst result in the table (**−0.019**).
- **`hits` is bounded by at-bats** (~4/game). It is closer to
binomial(AB, avg) than to an unbounded Poisson/NB, and most lines are 0.5
(mean line 0.578), so almost everything rides on P(0) — the exact region where
the wrong family hurts most.
- **`walks`, `runs`, `doubles` ARE genuine low-rate event counts** — and they are
precisely the ones that work.
---
## FIX BRANCH — and what it rules OUT
**RECOMMENDED NEXT ORDER: a targeted per-stat model fix for `hits` and
`total_bases`.** Small, contained, and aimed at 67% of the sample.
**This diagnosis REMOVES the MLB similarity build from the critical path.** The
order's "bad MEAN → build similarity (large)" branch assumed a *global* mean
weakness. It is not global: the mean is fine or better than the champion's on
`walks`, `runs` and `doubles`. **A similarity layer would not fix a
family-mismatched count model, and building one now would be a large project aimed
at the wrong defect.**
Similarity may still be worth building later — but **on evidence, not on this.**
## TAGS
**VERIFIED:** matched-row aligned gap 0.252 vs 0.352 (n=437) · 31.4% under-graded ·
`projectionChallenger` normalises direction correctly, so the misalignment was
measurement-only · per-stat table above · mean-vs-shape isolation · priors and
plumbing clean.
**DISPROVED:** prediction compression (spread ratio 0.80–0.94).
**HYPOTHESIS, NOT VERIFIED:** that the NB family mismatch is *why* hits and TB fail.
Structurally motivated; the fix order should test it before committing to a family
change.
**CORRECTED:** "proj-v1.1 loses 0.108 vs 0.331" — that comparison was unaligned on
direction and across different row sets.