hits-v1: built on the right structure, measured honestly, REFUTED
Hits was diagnosed as a family mismatch: 84% of hits rows trade at 0.5, so the stat rides on P(0), and a negative binomial has unbounded support and no notion of opportunity at all. hits-v1 models it as the bounded conversion it is -- N ~ the player's empirical at-bat distribution, hits|N ~ Binomial(N,q), with the multiplier scaling q (conversion) and never N (opportunity). STEP 0 confirmed the inputs before the model existed: 30/30 real ledger players, 100% combined-input coverage. Every read goes through knownRate -- a row with no atBats is dropped, never counted as a 0-at-bat game. It FIRES: 158/159 hits props (99.4%) on the live production snapshot, through the real attachProjection path. Scoping by book IDENTITY rather than price shape kept 94 out-of-promotion-band props on the board, 93 of them modelled -- 59% that a price rule would have deleted. And it LOST. Point-in-time replay (game log truncated strictly before each row's game_date, real grade-time multiplier), hits-only, direction-aligned, n=242: resolution champion 0.195 / ladder 0.048 / hits-v1 0.026. Paired bootstrap on the same rows: hits-v1 - ladder = -0.022, CI95 excluding zero. Not promoted. The value is in what it eliminates. The family was wrong AND the mean was not the constraint -- hits-v1 moved the line-0.5 mean 0.554 -> 0.581 toward a 0.598 base rate while resolution fell. What is left is per-prop discrimination: the ladder's inputs, not its distribution. The pre-registered fallback is recorded as WRONG rather than deleted. It said hits might be genuinely low-resolution for anyone; the champion scores 0.276 on the identical 189 rows, so there is real signal and the ceiling claim was the comfortable reading, not the honest one. Its own control refuted it, and that control was already in hand when the branch was written. hits-v1 stays wired as a challenger writing its own ledger columns so the forward accrual can confirm the backtest. Champion, ladder, ranking, calibration, reference ruler and the four accruing verdicts are byte-identical -- the diff has zero deleted lines. Tests 4,156 green (332 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -1096,6 +1096,42 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
|
||||
Redis runs degraded locally, so the script must `process.exit(0)` — otherwise a
|
||||
reconnect timer holds the process open and piped output is lost to SIGTERM.
|
||||
|
||||
## hits-v1 — a REFUTED challenger, and why it stays (Session 76 — non-obvious)
|
||||
- **`specs/hits-v1-binomial.md` is the record.** hits-v1 models hits as a
|
||||
binomial over the player's EMPIRICAL at-bat distribution (P(0) stated directly,
|
||||
since 84% of hits rows trade at 0.5). It FIRES at 99.4% on the live board and
|
||||
it DOES NOT WORK: point-in-time replay, hits-only, direction-aligned, n=242 —
|
||||
resolution champion 0.195 / ladder 0.048 / hits-v1 0.026. Paired bootstrap
|
||||
(same rows) puts hits-v1 − ladder at −0.022, CI95 excluding 0. NOT PROMOTED.
|
||||
- **Two explanations are now ELIMINATED for hits, which is the useful part.**
|
||||
The family was wrong (swapping it made things slightly worse) AND the mean was
|
||||
not the constraint (hits-v1 moved the line-0.5 mean 0.554→0.581 against a 0.598
|
||||
base rate — closer — while resolution FELL). What remains is per-prop
|
||||
DISCRIMINATION: the ladder's inputs don't separate hitters. Don't build another
|
||||
projection variant for hits; diagnose what the champion's `p_win` reads first.
|
||||
- **"Honest ceiling" needs its control checked before you claim it.** The spec's
|
||||
own pre-registered fallback ("hits may be genuinely low-resolution for anyone")
|
||||
was REFUTED by the champion scoring 0.276 on the identical 189 rows. A ceiling
|
||||
claim is only honest if no instrument on the same rows beats it — check that
|
||||
BEFORE writing the branch, not after.
|
||||
- **Backtest ≠ verdict.** The replay truncates each player's game log strictly
|
||||
BEFORE the row's `game_date` and reuses the row's stored grade-time
|
||||
`combined_multiplier` (both live on real ledger rows) — without that truncation
|
||||
it would be scoring predictions with the answer in hand. The verdict of record
|
||||
is still the forward accrual, so hits-v1 stays wired, writing
|
||||
`proj_hits_p_over`/`proj_hits_meta` only. Never served.
|
||||
- **Paired bootstrap, not two independent SEs.** Challengers score the SAME rows;
|
||||
comparing independent standard errors overstates uncertainty and would have
|
||||
read a reliable −0.022 regression as noise. `scripts/hits-v1-holdout.js` has
|
||||
the seeded implementation — reuse it for the next challenger.
|
||||
- **The takeable axis paid off measurably:** 94 of 159 live hits props are
|
||||
OUTSIDE the promotion band and 93 were modelled anyway. Scope by
|
||||
`isTakeableMarket` (book identity); record `isWithinPromotionBand` and never
|
||||
let it gate a model — a price-shape rule would have deleted 59% of the board.
|
||||
- **Local `.env` has a transposed Supabase ref** (`zmdnczhtdxcddszxttub`; real is
|
||||
`zmdnczhtdxcddsxzttub`), so local scripts hitting Supabase need an explicit
|
||||
`SUPABASE_URL=` override. Prod + the MCP connection are fine.
|
||||
|
||||
## Active Skills
|
||||
- vyndr-voice (all user-facing output)
|
||||
- prop-analysis (grading methodology)
|
||||
|
||||
Reference in New Issue
Block a user