Build the gate, run it, and find we were proving things on the wrong stat
PREMISE CORRECTION FIRST. statModel.js and correlateValidator.js do not exist in this repository. The validation spec's only prior form is src/services/python/blueprints/unconventional.py -- a Flask blueprint in the Python service that is offline in production, scoring NBA factors against a warehouse that was never populated -- and tests/unit/supplementSystems.test.js requires only fs and path while defining its own validateFactor inline at line 368. Those tests assert a re-implementation of the thresholds, not an implementation, which is exactly why they passed for months while nothing was connected. The diagnosis behind the order is right -- every challenger was measured without a gate -- but the cause is that there was no gate on the Node side to import. So it is built, to the exact spec. correlateValidator: n>=500, |r|>=0.15, p<0.05, Bonferroni across the sweep. The p-value is exact rather than approximated (t-transform through a regularized incomplete beta) and is verified in the suite against known values, because scipy is not available here. Pairs with an unknown side are dropped, never zero-filled -- a zero-fill inside a correlation does not add noise, it invents a point at the origin. THE RUN, hits, n=570, Bonferroni-8: every skill feature fails, and not narrowly. The strongest marginal correlation against the counter's residual is 0.062 against a 0.15 bar. That is an effect-size failure at a sample that would have found a real effect comfortably -- a clean, well-powered negative. The head-to-head agrees: value engine 0.0499 against the counter's 0.166, delta -0.116 with CI [-0.189, -0.043]. Not promoted. THE RUN, total bases, n=295: cannot be tested, and that is the finding. hard_hit_pct shows a marginal r of 0.153 -- above the threshold -- and exit velo 0.124, refused solely because n is 205 short of 500. It is the most encouraging number this work has produced, and it is what the physics predicts: contact quality governs extra bases, not whether a grounder finds a hole. We have been testing skill inputs on the one stat where they should not matter much. Two things the run forced. Feature verdicts are now PER STAT, because marking these DEAD sport-wide on hits evidence would have killed, for total bases, the features that look most alive there -- per-sport doctrine one level deeper. And the gate now reports r and p even when underpowered, because "not enough data yet" and "nothing here" demand opposite decisions and a bare refusal was hiding the best signal on the board. Next: build the compound TB projection (skillProjection still refuses total bases by design, since a deterministic bases-per-hit made P(TB>=2) identical to P(hits>=1)), accrue to n>=500, re-run this gate. Leave hits alone. 4,200 tests green (334 suites); web build exit 0; counter byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,158 @@
|
||||
# STAGE A THROUGH THE GATE — and the stat we should have picked
|
||||
|
||||
**2026-08-03.** Challenger-only. The counter is byte-identical (zero diff on
|
||||
`probabilityEstimator.js` / `analyzeViaEngine1.js`).
|
||||
|
||||
> **RESULT: no skill feature passes the gate for HITS, and the value engine
|
||||
> loses to the counter on hits. But the same features show the strongest
|
||||
> correlations yet measured for TOTAL BASES — and total bases is 205 settled rows
|
||||
> short of being testable at all.** We have been proving things on the wrong stat.
|
||||
|
||||
---
|
||||
|
||||
## 0. Premise correction, with evidence
|
||||
|
||||
The order describes `statModel.js` and `correlateValidator.js` as *built, tested,
|
||||
disconnected*. **Neither file exists in this repository.**
|
||||
|
||||
```
|
||||
find . -name "statModel*" → nothing (outside .claude/worktrees)
|
||||
find . -name "correlateValidator*" → nothing
|
||||
```
|
||||
|
||||
The validation spec's only prior existence is
|
||||
`src/services/python/blueprints/unconventional.py` — a Flask blueprint in the
|
||||
**Python service that is offline in production**, scoring NBA "unconventional
|
||||
factors" (altitude, contract year, referee crew) against a data warehouse that
|
||||
was never populated. And `tests/unit/supplementSystems.test.js` has exactly
|
||||
**two `require()` calls — `fs` and `path`** — and defines its own `validateFactor`
|
||||
inline at line 368. **Those tests assert a re-implementation of the thresholds,
|
||||
not an implementation of them**, which is precisely why they passed for months
|
||||
while nothing existed to connect.
|
||||
|
||||
So the diagnosis behind the order is right — **every challenger was measured
|
||||
without a gate** — but the cause is simpler than "someone forgot to import it":
|
||||
**there was no gate on the Node side to import.** It is built now, to the exact
|
||||
spec, and this document is its first real run.
|
||||
|
||||
## 1. The gate (`src/services/model/correlateValidator.js`)
|
||||
|
||||
`n ≥ 500 · |Pearson r| ≥ 0.15 · p < 0.05 · Bonferroni across the sweep`,
|
||||
ported from `VALIDATION_REQUIREMENTS` in the Python blueprint.
|
||||
|
||||
The p-value is exact, not approximated — scipy is unavailable in Node, so it is
|
||||
computed from the t transform via a regularized incomplete beta (Lentz continued
|
||||
fraction) and **verified against known values** in the unit suite (r=0.5,n=10 →
|
||||
p≈0.1411; r=0.8,n=10 → p≈0.00546; r=0.15,n=500 → p≈0.00076). Pairs with an
|
||||
unknown side are **dropped, never zero-filled** — a zero-fill inside a
|
||||
correlation doesn't add noise, it invents a point at the origin.
|
||||
|
||||
**Bonferroni is not decoration here.** The S78 residual scan ran 70 tests at
|
||||
α=.05 and produced six "significant" hits — almost exactly the 3–4 chance
|
||||
predicts. The correction is what makes PROVEN mean anything.
|
||||
|
||||
## 2. THE GATE RUN — hits (n=570, clears the bar)
|
||||
|
||||
Bonferroni denominator = 8 (the features in this sweep), corrected α = 0.00625.
|
||||
Two questions per feature: does it relate to the outcome at all, and — the one
|
||||
that matters — is any of it **new**, i.e. correlated with what the counter
|
||||
*misses* (`won − p_win`)?
|
||||
|
||||
| feature | r vs outcome | **r vs counter residual** | p | verdict |
|
||||
|---|---|---|---|---|
|
||||
| batter_bb_pct | −0.047 | **−0.062** | 0.138 | weak + not significant |
|
||||
| batter_exit_velo | −0.031 | −0.056 | 0.180 | weak + not significant |
|
||||
| batter_hard_hit_pct | −0.007 | −0.039 | 0.355 | weak + not significant |
|
||||
| batter_barrel_pct | −0.020 | −0.037 | 0.378 | weak + not significant |
|
||||
| batter_k_pct | −0.014 | −0.015 | 0.722 | weak + not significant |
|
||||
| batter_launch_angle | −0.010 | −0.005 | 0.910 | weak + not significant |
|
||||
| pitcher_k_pct | +0.018 | +0.012 | 0.777 | weak + not significant |
|
||||
| pitcher_hard_hit_allowed | +0.016 | +0.011 | 0.806 | weak + not significant |
|
||||
|
||||
**Nothing passes. Nothing is close.** The strongest marginal |r| is 0.062 against
|
||||
a 0.15 bar — not "failed on significance", failed on effect size, at a sample
|
||||
that would have detected a real effect easily.
|
||||
|
||||
**This is a clean, well-powered negative result for hits**, and it explains every
|
||||
earlier failure at a stroke: the skill features carry essentially no information
|
||||
about whether a hitter gets one hit. The counter isn't beating them because it's
|
||||
clever; it's beating them because at the 0.5 hits line there is very little for
|
||||
them to know.
|
||||
|
||||
## 3. THE GATE RUN — total bases (n=295, CANNOT be tested)
|
||||
|
||||
| feature | r vs outcome | **r vs counter residual** | status |
|
||||
|---|---|---|---|
|
||||
| **batter_hard_hit_pct** | **+0.196** | **+0.153** | insufficient data (needs 205 more rows) |
|
||||
| **batter_exit_velo** | **+0.167** | +0.124 | insufficient data |
|
||||
| batter_barrel_pct | +0.121 | +0.088 | insufficient data |
|
||||
| pitcher_k_pct | −0.049 | −0.077 | insufficient data |
|
||||
|
||||
**`hard_hit_pct` marginal r = 0.153 — above the 0.15 threshold — and it is
|
||||
refused solely because n = 295 < 500.** That is the gate working correctly, and
|
||||
it is also the most encouraging number this line of work has produced.
|
||||
|
||||
The gate now reports `r` and `p` even when underpowered, with
|
||||
`underpowered: true` and `rows_needed`. **"Not enough data yet" and "nothing
|
||||
here" require opposite decisions** — one waits, the other stops — and collapsing
|
||||
them into a bare refusal was hiding the single most useful signal on the board.
|
||||
|
||||
### And it is exactly what the physics predicts
|
||||
|
||||
Exit velocity and hard-hit rate govern whether contact becomes **extra bases**.
|
||||
They barely govern whether a ground ball finds a hole for a single. The archetype
|
||||
doctrine says the same thing: contact quality is the BOMBER lane, and total bases
|
||||
is the power stat. **We tested skill inputs on the one stat where skill inputs
|
||||
shouldn't matter much.**
|
||||
|
||||
## 4. Head-to-head — hits (the stat we could test)
|
||||
|
||||
Out-of-sample: profiles frozen 2026-07-21, only later games scored. 570 rows,
|
||||
91.9% opposing-pitcher coverage.
|
||||
|
||||
```
|
||||
resolution value engine 0.0499 counter 0.1660
|
||||
brier value engine 0.2839 counter 0.2506
|
||||
delta −0.1161 CI95 [−0.1892, −0.0426] ← excludes zero
|
||||
```
|
||||
|
||||
**LOSES. Not promoted.** Consistent with §2: a model built from features that
|
||||
carry no hits signal cannot beat a counter on hits.
|
||||
|
||||
## 5. What changed in the architecture because of this run
|
||||
|
||||
**Verdicts are now PER STAT** (`recordStatVerdict` / `statusForStat` /
|
||||
`candidateFeaturesForStat`). Marking these features DEAD sport-wide on hits
|
||||
evidence would have killed, for total bases, the exact features that look most
|
||||
alive there. That is the per-sport doctrine one level deeper: **a feature earns
|
||||
or loses its place per stat, because the physics differ per stat.** Locked by
|
||||
test.
|
||||
|
||||
## 6. Verdict and the next order
|
||||
|
||||
**Stage A on hits: the windshield loses, and the gate says why — there is no
|
||||
skill signal there to find.** Do not iterate the hits model; that is polishing a
|
||||
read of a stat whose outcome our best inputs are blind to.
|
||||
|
||||
**Stage A should move to TOTAL BASES**, where:
|
||||
- the leading marginal correlation (0.153) is already at the threshold,
|
||||
- the gate needs **205 more settled TB rows** — days, now that settlement works,
|
||||
- and `skillProjection` currently **refuses** total bases by design, because a
|
||||
deterministic bases-per-hit multiplier made P(TB≥2) identical to P(hits≥1).
|
||||
**That refusal is now the thing to build:** a real per-hit extra-base
|
||||
distribution off the launch/barrel profile — which is tb-v1's compound shape,
|
||||
fed by skill inputs instead of raw counts.
|
||||
|
||||
Ranked next steps:
|
||||
|
||||
1. **Build the compound TB value projection** (per-hit bases distribution from
|
||||
launch/barrel), so TB can be projected at all.
|
||||
2. **Accrue to n ≥ 500 on TB** and re-run this exact gate.
|
||||
3. **Then** head-to-head on TB. If `hard_hit_pct` holds its 0.153 through the
|
||||
Bonferroni-corrected gate at full n, that is the first feature in this
|
||||
codebase to genuinely earn its place.
|
||||
4. Leave hits alone.
|
||||
|
||||
**Not recommended:** promoting anything now, lowering the n≥500 bar to make TB
|
||||
testable today, or re-running the hits model with more features. The bar being
|
||||
inconvenient is what makes it worth having.
|
||||
Reference in New Issue
Block a user