Prove both on total bases -- and find that my own fix destroyed the backtest

Nothing passed. Nothing promoted. Counter byte-identical.

THE BLOCKER, which is the real finding. statcast_aggregates is upserted in
place and holds exactly one as-of date. Yesterday's skill backtest was honest
only by accident: the nightly refresh was unreachable code, so the profiles
sat frozen at 2026-07-21 -- before the settled window. Repairing that cron was
right for production and it refreshed them to today, destroying every prior
version. Scoring a 2026-07-25 game now uses a season aggregate that contains
that game. Point-in-time validation is structurally impossible from that
table, so every number in this run is contaminated and directional, and none
of it is a gate verdict.

Fixed forward: statcast_history retains a dated snapshot on every refresh, so
point-in-time becomes "as_of_date < game_date, most recent". Retention is
best-effort and cannot fail the refresh; both properties are unit-tested. It
has one day of data, which is not yet a window.

SOLO BASELINE, n=383, Bonferroni across 12 tests (alpha 0.00417): nothing
passes. hard_hit_pct is closest at marginal r 0.135 with p 0.0080, failing
both the 0.15 effect bar and the corrected alpha. And it drifted DOWN from
0.153 at n=295 -- an estimate regressing as noise averages out, not an effect
firming up. I called that number encouraging yesterday; on 88 more rows it is
fading, and it should not keep being quoted at its best value.

INTERACTIONS, each scored by partial correlation against the counter residual
controlling for both of its own components: none pass. Only barrel x power
archetype has an incremental exceeding its parts (-0.101 against 0.019) at
n=260 -- the shape Discipline 2 predicts, but a lead, not a finding.

A methodological catch worth keeping. The archetype conditioner was first
built as barrel_pct over league barrel -- a monotone transform of one of its
own components -- so the "interaction" was barrel squared, measuring
nonlinearity in barrel rate rather than any archetype effect, and it produced
this run's only positive result. A Gauss-Jordan pivot test does not catch that,
because the two columns differ by a scale factor. Fixed with a scale-free
collinearity check plus real archetype labels joined from model_snapshots.
Without it this document would have reported a fabricated interaction as the
session's finding.

COMBINED vs COUNTER on total bases: 0.2718 against 0.2647, delta +0.0071, CI
[-0.065, +0.079] -- inconclusive, and the first time a challenger has not
lost. The same engine on hits was -0.116 with a CI excluding zero. That
contrast is the whole argument for total bases, and it is what the physics
said: contact quality governs extra bases, not whether a grounder finds a hole.

Also built: the compound TB projection. skillProjection no longer refuses
total bases -- a deterministic bases-per-hit multiplier had made P(TB>=2)
exactly P(hits>=1), a relabelled hits curve. It is now a convolution over
per-PA base outcomes with hit-type shares shifted by skill. Non-degeneracy is
locked by test.

4,204 tests green (334 suites); web build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-03 16:27:11 -04:00
parent c7cc8f5e52
commit 4aab18096f
8 changed files with 880 additions and 10 deletions
+38
View File
@@ -1337,6 +1337,44 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
the per-hit extra-base distribution off launch/barrel (tb-v1's shape, fed by
skill inputs), accrue to n>=500, re-run this gate. Do NOT lower the bar.
## Point-in-time skill validation + TB solo/interactions (Session 82 — non-obvious)
- **`statcast_aggregates` KEEPS NO HISTORY** — upserted in place on
(sport,season,source_id,role), ONE as-of date, prior versions destroyed. The
first skill backtest was honest only BY ACCIDENT: the nightly refresh was
unreachable code so profiles sat frozen at 2026-07-21, BEFORE the settled
window. Fixing that cron refreshed them to today and made point-in-time
validation impossible from that table. **`statcast_history` (new) retains a
dated snapshot per refresh** — query `where as_of_date < game_date order by
as_of_date desc limit 1`. Retention is best-effort and must NEVER fail the
refresh (unit-tested). Until it accrues a window, ALL skill-feature results are
CONTAMINATED/DIRECTIONAL, never gate verdicts.
- **TB solo pass: NOTHING passes.** n=383, Bonferroni-12 (α=0.00417).
`hard_hit_pct` is closest at marginal r=0.135, p=0.0080 — fails BOTH the 0.15
effect bar and corrected α. **It DRIFTED DOWN from 0.153 (n=295) → 0.135
(n=383)**: an estimate regressing as noise averages out, not an effect firming.
Don't keep quoting the older better number.
- **Interactions: none pass.** Only `barrel × power_archetype` has incremental
(partial, controlling for both components) exceeding its parts — 0.101 vs
0.019 at n=260. A lead, not a finding.
- **INTERACTION-PROXY TRAP:** the archetype conditioner was first
`barrel_pct/LEAGUE.barrel_pct` — a monotone transform of its own component — so
the "interaction" was barrel² measuring NONLINEARITY, and it produced the run's
only positive result (0.132). A Gauss-Jordan pivot test does NOT catch this
(the columns differ by a scale factor); use a **scale-free pairwise correlation
check** on control columns. Real archetype labels come from
`model_snapshots.archetype` (260 labelled TB rows: 140 BOMBER / 120 other).
- **Interactions must be scored by PARTIAL correlation** vs the counter residual,
controlling for both components — raw correlation can't distinguish
PASSES-AND-ADDS from PASSES-BUT-REDUNDANT.
- **TB is at PARITY with the counter** (0.2718 vs 0.2647, CI [0.065,+0.079],
inconclusive) where HITS lost by 0.116 with CI excluding zero. Same engine,
same day — the stat choice was the whole story. Parity under contamination is
NOT a win; nothing promoted.
- **`skillProjection` now models total_bases** as a compound convolution (per-PA
0/1/2/3/4 bases; barrel→HR share, exit velo→2B/3B share). The old
deterministic bases-per-hit made P(TB>=2) EXACTLY P(hits>=1); non-degeneracy is
locked by test.
## Active Skills
- vyndr-voice (all user-facing output)
- prop-analysis (grading methodology)