builtbykev
4aab18096f
Nothing passed. Nothing promoted. Counter byte-identical. THE BLOCKER, which is the real finding. statcast_aggregates is upserted in place and holds exactly one as-of date. Yesterday's skill backtest was honest only by accident: the nightly refresh was unreachable code, so the profiles sat frozen at 2026-07-21 -- before the settled window. Repairing that cron was right for production and it refreshed them to today, destroying every prior version. Scoring a 2026-07-25 game now uses a season aggregate that contains that game. Point-in-time validation is structurally impossible from that table, so every number in this run is contaminated and directional, and none of it is a gate verdict. Fixed forward: statcast_history retains a dated snapshot on every refresh, so point-in-time becomes "as_of_date < game_date, most recent". Retention is best-effort and cannot fail the refresh; both properties are unit-tested. It has one day of data, which is not yet a window. SOLO BASELINE, n=383, Bonferroni across 12 tests (alpha 0.00417): nothing passes. hard_hit_pct is closest at marginal r 0.135 with p 0.0080, failing both the 0.15 effect bar and the corrected alpha. And it drifted DOWN from 0.153 at n=295 -- an estimate regressing as noise averages out, not an effect firming up. I called that number encouraging yesterday; on 88 more rows it is fading, and it should not keep being quoted at its best value. INTERACTIONS, each scored by partial correlation against the counter residual controlling for both of its own components: none pass. Only barrel x power archetype has an incremental exceeding its parts (-0.101 against 0.019) at n=260 -- the shape Discipline 2 predicts, but a lead, not a finding. A methodological catch worth keeping. The archetype conditioner was first built as barrel_pct over league barrel -- a monotone transform of one of its own components -- so the "interaction" was barrel squared, measuring nonlinearity in barrel rate rather than any archetype effect, and it produced this run's only positive result. A Gauss-Jordan pivot test does not catch that, because the two columns differ by a scale factor. Fixed with a scale-free collinearity check plus real archetype labels joined from model_snapshots. Without it this document would have reported a fabricated interaction as the session's finding. COMBINED vs COUNTER on total bases: 0.2718 against 0.2647, delta +0.0071, CI [-0.065, +0.079] -- inconclusive, and the first time a challenger has not lost. The same engine on hits was -0.116 with a CI excluding zero. That contrast is the whole argument for total bases, and it is what the physics said: contact quality governs extra bases, not whether a grounder finds a hole. Also built: the compound TB projection. skillProjection no longer refuses total bases -- a deterministic bases-per-hit multiplier had made P(TB>=2) exactly P(hits>=1), a relabelled hits curve. It is now a convolution over per-PA base outcomes with hit-type shares shifted by skill. Non-degeneracy is locked by test. 4,204 tests green (334 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
163 lines
8.6 KiB
Markdown
163 lines
8.6 KiB
Markdown
# TOTAL BASES — solo baseline, interactions, and the validity blocker
|
||
|
||
**2026-08-03.** Challenger-only. Counter byte-identical (zero diff on
|
||
`probabilityEstimator.js` / `analyzeViaEngine1.js`).
|
||
|
||
> **HEADLINE 1 — nothing can be gate-validated today, and the reason is my own
|
||
> fix.** `statcast_aggregates` is upserted in place and keeps **one as-of date**.
|
||
> The first skill backtest was honest only by accident: the nightly refresh was
|
||
> unreachable code, so profiles sat frozen at 2026-07-21 — *before* the settled
|
||
> window. Repairing that cron (correct for production) refreshed them to today
|
||
> and destroyed every prior version. **Point-in-time validation is now
|
||
> structurally impossible from that table**, so every number below is
|
||
> **contaminated / directional** and none is a gate verdict.
|
||
>
|
||
> **HEADLINE 2 — on total bases the skill model is at PARITY with the counter**
|
||
> (0.2718 vs 0.2647, CI includes zero). On hits it lost by 0.116 with a CI
|
||
> excluding zero. Same engine, same day. **The stat choice was the whole story.**
|
||
|
||
---
|
||
|
||
## 0. Premise correction
|
||
|
||
The order states "barrel rate PASSED, exit velo/launch/matchup FAILED solo."
|
||
**Nothing passed.** Yesterday's run refused *every* TB feature on
|
||
`insufficient_data` (n=295 < 500); barrel rate's marginal r was 0.088 and the
|
||
strongest was hard-hit at 0.153. Today at n=383, **hard-hit has fallen to
|
||
0.135** — see §3, because that drift is itself the most informative number here.
|
||
|
||
## 1. The validity blocker, and the fix
|
||
|
||
```
|
||
select updated_at::date, count(*) from statcast_aggregates group by 1;
|
||
→ 2026-08-03 | 1387 ← ONE row. No history. 2026-07-21 is gone.
|
||
```
|
||
|
||
A season aggregate dated 2026-08-03 used to score a 2026-07-25 game **contains
|
||
that game**. That is leakage, and it is not repairable after the fact.
|
||
|
||
**Fixed forward:** `statcast_history` (migration applied) retains a dated
|
||
snapshot on every refresh, keyed `(as_of_date, sport, season, source_id, role)`.
|
||
Point-in-time becomes `where as_of_date < game_date order by as_of_date desc
|
||
limit 1` — the profile as it stood *before* the game, by construction. Retention
|
||
is best-effort and **cannot fail the refresh**; both properties are unit-tested.
|
||
It has one day of data, which is not yet a window.
|
||
|
||
## 2. Feature → conditioning variable → mechanism (STEP 2)
|
||
|
||
| interaction | conditioning on | mechanism |
|
||
|---|---|---|
|
||
| `launch × exit_velo` | each other | Extra bases need **both**: hit hard AND in the air. A 105-mph ground ball is an out; a 25° popup is an out. Neither predicts alone — which is exactly why each can fail solo and the product might not. |
|
||
| `exit_velo × pitcher_hard_hit_allowed` | pitcher suppression | A hitter only realises contact quality against a pitcher who permits it. Elite suppression attenuates a power bat; a contact-permitting arm amplifies it. Conditional by construction. |
|
||
| `barrel × power archetype` | **real archetype label** | Barrels convert to extra bases for hitters whose lane is power. This is Discipline 2 as a testable claim. |
|
||
| `batter_K × pitcher_K` | each other | Strikeout risk compounds (log5 is this shape). A PA lost to a strikeout is a base opportunity that never happens — suppression through **opportunity**, not contact. |
|
||
| `pull_rate × park` | park | **UNTESTABLE** — park is passed as 1.0 everywhere and pull rate is not ingested. |
|
||
|
||
## 3. STEP 1 — solo baseline (the control)
|
||
|
||
n=383, Bonferroni across 12 tests → corrected α = **0.00417**.
|
||
The decider is correlation with the **counter's residual** (`won − p_win`) — a
|
||
feature correlating with the raw outcome may simply know what the counter
|
||
already knows.
|
||
|
||
| feature | r vs outcome | **r vs residual** | p | would it pass at n≥500? |
|
||
|---|---|---|---|---|
|
||
| **batter_hard_hit_pct** | 0.172 | **0.135** | 0.0080 | **No** — |r| < 0.15 *and* p > 0.00417 |
|
||
| batter_exit_velo | 0.120 | 0.070 | 0.173 | No |
|
||
| batter_barrel_pct | 0.109 | 0.078 | 0.127 | No |
|
||
| pitcher_k_pct | −0.094 | −0.087 | 0.093 | No |
|
||
| batter_launch_angle | 0.032 | 0.003 | 0.948 | No |
|
||
| batter_bb_pct | −0.019 | −0.026 | 0.607 | No |
|
||
| pitcher_hard_hit_allowed | 0.024 | 0.035 | 0.501 | No |
|
||
| batter_k_pct | −0.009 | −0.003 | 0.952 | No |
|
||
|
||
**No feature passes solo, and hard-hit is drifting the wrong way.** Its marginal
|
||
r went **0.153 (n=295) → 0.135 (n=383)** as sample grew. That is the signature of
|
||
an estimate regressing toward its true value as noise averages out, not of a real
|
||
effect firming up. Yesterday I called it "the most encouraging number this work
|
||
has produced"; on 88 more rows it is fading. **Worth stating plainly, because the
|
||
temptation is to keep quoting the older, better number.**
|
||
|
||
## 4. STEP 3 — interactions, each against its own parts
|
||
|
||
Incremental signal = partial correlation of the interaction with the residual,
|
||
**controlling for both components**. If an interaction is barrel rate in a hat,
|
||
regressing out barrel rate collapses it.
|
||
|
||
| interaction | n | raw r | best component r | **incremental r** | verdict |
|
||
|---|---|---|---|---|---|
|
||
| `barrel × power archetype` | 260 | −0.046 | 0.019 | **−0.101** | UNDERPOWERED — n below gate |
|
||
| `launch × exit_velo` | 383 | 0.007 | 0.070 | −0.053 | UNDERPOWERED |
|
||
| `batter_K × pitcher_K` | 378 | −0.050 | 0.087 | −0.024 | UNDERPOWERED |
|
||
| `exit_velo × pitcher_suppression` | 378 | 0.048 | 0.073 | 0.013 | UNDERPOWERED |
|
||
|
||
Only the archetype interaction has an incremental exceeding its parts (0.101 vs
|
||
0.019) — the shape Discipline 2 predicts — but at n=260, with a negative sign,
|
||
and nowhere near the gate. **It is a lead, not a finding.**
|
||
|
||
### A methodological catch worth recording
|
||
|
||
The archetype conditioning variable was first built as
|
||
`barrel_pct / LEAGUE.barrel_pct` — a monotone transform of one of its own
|
||
components. That interaction was therefore **barrel², measuring nonlinearity in
|
||
barrel rate, not an archetype effect at all**, and it produced the run's only
|
||
"interaction adds" result (−0.132). Two fixes: a scale-free collinearity check on
|
||
the control columns (the Gauss-Jordan pivot test missed it, because the columns
|
||
differ by a *scale factor*), and real archetype labels joined from
|
||
`model_snapshots` (260 labelled rows: 140 BOMBER, 120 other) so the conditioning
|
||
variable is categorical and independent of barrel by construction.
|
||
|
||
**Without that catch this document would have reported a fabricated interaction
|
||
as the session's finding.**
|
||
|
||
## 5. STEP 4 — combined skill model vs the counter
|
||
|
||
The compound TB projection was built for this (see §6). Paired bootstrap is
|
||
valid at this n even though the *gate* is not.
|
||
|
||
```
|
||
n 383 · pitcher coverage 100% · base rate 0.483
|
||
resolution skill TB 0.2718 counter 0.2647
|
||
brier skill TB 0.2439 counter 0.2426
|
||
delta +0.0071 CI95 [−0.0648, +0.0789] → INCONCLUSIVE
|
||
```
|
||
|
||
**Not a win — but the first time a challenger has not LOST.** For contrast, the
|
||
same engine on hits: −0.1161, CI [−0.1892, −0.0426], excluding zero. The
|
||
difference between those two lines is the entire argument for moving to total
|
||
bases, and it is what the physics said: contact quality governs extra bases, not
|
||
whether a grounder finds a hole.
|
||
|
||
## 6. Built this session: the compound TB projection
|
||
|
||
`skillProjection` no longer refuses total bases. A deterministic bases-per-hit
|
||
multiplier made `P(TB≥2)` **exactly** `P(hits≥1)` — a relabelled hits curve — so
|
||
it was refused. It is now a real convolution: each PA yields 0/1/2/3/4 bases with
|
||
hit-type shares shifted by skill (barrel → home-run share, exit velo → doubles
|
||
and triples), convolved over a PA distribution. A BOMBER gets a 37.5% home-run
|
||
share against a slap hitter's 4.5%. Non-degeneracy is locked by test.
|
||
|
||
## 7. Verdict
|
||
|
||
- **Solo-proven set for total_bases: EMPTY.** Nothing passes; hard-hit is the
|
||
closest and is drifting down.
|
||
- **Interaction-proven set: EMPTY.** One lead (archetype × barrel), underpowered.
|
||
- **Combined vs counter: INCONCLUSIVE at parity** — genuinely better news than
|
||
hits, and still not promotable.
|
||
- **Nothing promoted. Counter untouched.**
|
||
|
||
## 8. Next, in order
|
||
|
||
1. **Let `statcast_history` accrue.** Nothing here can be validated honestly
|
||
until there is a point-in-time window — a week gives a real one. This is now
|
||
the binding constraint on the entire programme, and it is a waiting problem,
|
||
not a building problem.
|
||
2. **Re-run this exact script** once history exists and TB clears n≥500 (383
|
||
today, ~117 short). Both arrive on roughly the same timescale.
|
||
3. **Archetype × barrel is the one interaction worth carrying forward**, with
|
||
real labels and adequate n.
|
||
4. **Do not re-run hits.** §5 settles it.
|
||
|
||
**Not recommended:** quoting the 0.153, lowering n≥500, or promoting on a
|
||
parity result. Parity with contamination is not a win.
|