builtbykev
43f65d30cb
inconclusive THE BUG THIS NEARLY SHIPPED AS A FINDING. The first audit reported 0 factors fired on all 1,140 rows. Not a result -- my paging helper ordered by `id`, and batter_spray, team_defense, platoon_splits and statcast_aggregates have composite primary keys with NO id column. The query errored, the loop broke on error, and four fully-populated tables read as empty. hitsFactorContext.js -- the PRODUCTION loader -- had the identical defect, so live wiring would have loaded nothing and served unadjusted while logging success. Third occurrence of this class in one session. Both loaders now order by a real column and THROW rather than degrade. The Phase 2 gate is what caught it: no resolution number was quoted until transmission was proved. PHASE 1 — pipeline is now base -> FACTORS -> CALIBRATE -> GRADE. Context built in snapshotService BEFORE gradeAndCacheSlate (was line 640+, grade at 454), threaded per prop, applied to p_over before p_win is set with p_win_prefactor and a full trace retained. Hits only. Coverage 859/1140 rows (75%): 474 with all three factors, 256 two, 129 one, 281 none. PHASE 2 — TRANSMISSION PROVEN, 12/12 sign-correct, 4/4 per factor, each applied IN ISOLATION. My first table compared each factor's expected sign against the COMPOSITE change and showed 3 false failures -- with three factors firing the net can oppose any single member; that was a flaw in the test, not the wiring. Two under-side rows confirm the flip is handled: a factor raising p(over) correctly lowers p_win. Switch hitters (Bailey, Bell, Rocchio) took no spray adjustment while their other factors fired normally -- the refusal is selective, not a blanket skip. PHASE 3/4 — both maps refit on the factor-adjusted forecast; the shadow-duel baseline is VOID and restarts, since it accumulated against a different forecast. Point-in-time, 765 held-out rows: reliability 0.00795 -> 0.00828 RESOLUTION 0.00229 -> 0.00345 (variance explained 0.93% -> 1.39%) Brier 0.25398 -> 0.25305 delta -0.00093 CI [-0.00225,+0.00002] Resolution rose 51% relative. The CI TOUCHES ZERO on 4 eval dates, so the composition does NOT earn a proven keep -- three isolated passes did not grant a composed pass. INCONCLUSIVE, reported as such. The gain is far below the sum of the isolated effects, which is expected: all three run through the same pitcher-batter confrontation and share signal. PHASE 5 — 1.39% of variance is still far below what band separation needs. The pivot was correct and incomplete: the plumbing defect was real and is fixed, three proven factors reach the served number for the first time, and transmission alone did not buy grade separation. Next arc is factor STRENGTH and BREADTH, not more plumbing. PHASE 6 — rbi anomaly logged, not chased: 14.51% variance explained vs hits 1.03%, on the stat we do not serve corrected and which has no proven factors. Either the biggest lever on the board or a mirage; it deserves its own order. The byte-identical invariant INVERTED for hits by design. All 13 frozen non-hits modules verified unchanged, probabilityEstimator included -- the factors ride outside it. No new Bonferroni slot; the composed OOS claim is reported with its CI and not claimed as a pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
143 lines
6.6 KiB
Markdown
143 lines
6.6 KiB
Markdown
# Wiring the three proven hits factors — transmission proven, gain inconclusive
|
||
|
||
## The bug this order nearly shipped as a finding
|
||
|
||
The first audit run reported **0 factors fired on all 1,140 rows**. Not a
|
||
modelling result — my paging helper ordered by `id`, and `batter_spray`,
|
||
`team_defense`, `platoon_splits` and `statcast_aggregates` have **composite
|
||
primary keys with no `id` column**. The query errored, the loop broke on error,
|
||
and four fully-populated tables read as empty.
|
||
|
||
`hitsFactorContext.js` — the *production* loader — had the identical defect, so
|
||
the live wiring would have loaded nothing and served unadjusted forecasts while
|
||
logging success.
|
||
|
||
**Third occurrence of this class in one session** (doubled `/leaderboard`, the
|
||
silent settlement outage, this). Both loaders now order by a real column and
|
||
**throw** rather than degrade, because a wiring fault must not be able to wear
|
||
the costume of an honest absence.
|
||
|
||
The Phase 2 transmission gate is what caught it: no resolution number was quoted
|
||
until transmission was proved mechanically.
|
||
|
||
---
|
||
|
||
## PHASE 1 — the pipeline, in order
|
||
|
||
```
|
||
base rate -> FACTORS (pre-grade) -> CALIBRATE -> GRADE
|
||
```
|
||
|
||
- `model/hitsFactors.js` — the three proven factors composed, each applying only
|
||
where it proved, bounded at ±0.25 combined.
|
||
- `model/hitsFactorContext.js` — loads the inputs **once per slate**, indexed.
|
||
- `snapshotService` builds the context **before** `gradeAndCacheSlate`; it was
|
||
previously computed at line 640+, downstream of the grade at 454.
|
||
- `gradeSlateService` threads it per prop; `analyzeViaEngine1` applies it to
|
||
`p_over` **before** `p_win` is set, recording `p_win_prefactor` and a full
|
||
`factor_adjustment` trace.
|
||
|
||
Hits only. TB/rbi/runs have no proven factors and are untouched.
|
||
|
||
**Coverage: 859 of 1,140 rows (75%) have at least one factor fire** — 474 with
|
||
all three, 256 with two, 129 with one, 281 with none.
|
||
|
||
---
|
||
|
||
## PHASE 2 — transmission, proven mechanically
|
||
|
||
Each factor applied **in isolation** to the same base. (My first table compared
|
||
each factor's expected sign against the *composite* change and showed 3 false
|
||
failures — with three factors firing, the net can oppose any single member. That
|
||
was a flaw in the test, not the wiring.)
|
||
|
||
| factor | player | side | expected | mult | p before | p after (solo) | sign |
|
||
|---|---|---|---|---|---|---|---|
|
||
| pitcher_contact | Travis Bazzana | over | raise | 1.0396 | 0.604 | 0.6279 | ✓ |
|
||
| platoon_severity | Travis Bazzana | over | raise | 1.0860 | 0.604 | 0.6559 | ✓ |
|
||
| pitcher_contact | Patrick Bailey | over | raise | 1.0396 | 0.604 | 0.6279 | ✓ |
|
||
| pitcher_contact | Kyle Manzardo | **under** | raise p(over) | 1.0396 | 0.684 | 0.6715 | ✓ |
|
||
| platoon_severity | Kyle Manzardo | **under** | raise p(over) | 1.0350 | 0.684 | 0.6729 | ✓ |
|
||
| defense_by_direction | Royce Lewis | over | lower | 0.9690 | 0.577 | 0.5591 | ✓ |
|
||
| platoon_severity | Royce Lewis | over | lower | 0.9650 | 0.577 | 0.5568 | ✓ |
|
||
| pitcher_contact | Petey Halpin | over | raise | 1.0396 | 0.662 | 0.6882 | ✓ |
|
||
| platoon_severity | Chase DeLauter | over | lower | 0.9620 | 0.838 | 0.8062 | ✓ |
|
||
| defense_by_direction | Gabriel Arias | over | raise | 1.0400 | 0.536 | 0.5574 | ✓ |
|
||
| defense_by_direction | Austin Hedges | over | raise | 1.0390 | 0.685 | 0.7117 | ✓ |
|
||
| defense_by_direction | Ryan Kreidler | over | lower | 0.9560 | 0.523 | 0.5000 | ✓ |
|
||
|
||
**12/12 sign-correct — 4/4 for each of the three factors.** The two `under` rows
|
||
confirm the flip is handled: a factor raising p(over) correctly *lowers* p_win.
|
||
|
||
**Unreadables static:** Patrick Bailey, Josh Bell and Brayan Rocchio are switch
|
||
hitters — spray applied to none of them, while their other factors fired
|
||
normally. The refusal is selective, not a blanket skip.
|
||
|
||
**TRANSMISSION PASSES.** Resolution may now be quoted.
|
||
|
||
---
|
||
|
||
## PHASE 3/4 — refit and OOS measurement
|
||
|
||
Both calibration maps refit on the factor-adjusted forecast (the un-factored
|
||
distribution no longer exists). **The shadow-duel baseline is VOID and restarts**
|
||
— it accumulated against a different forecast.
|
||
|
||
Point-in-time, fit on dates < 2026-08-02, evaluated on 765 held-out rows:
|
||
|
||
| | reliability | **resolution** | uncertainty | **variance explained** |
|
||
|---|---|---|---|---|
|
||
| before (four-input) | 0.00795 | 0.00229 | 0.2476 | **0.93%** |
|
||
| **after (factor-adjusted)** | 0.00828 | **0.00345** | 0.2476 | **1.39%** |
|
||
|
||
**Resolution rose 51% relative (+0.00116).** Held-out Brier 0.25398 → 0.25305,
|
||
**delta −0.00093, date-block CI [−0.00225, +0.00002]**.
|
||
|
||
**The CI touches zero on 4 eval dates. The composition does NOT earn a proven
|
||
keep.** The point estimate favours the factors and the resolution gain is real in
|
||
sample, but the honest verdict is **INCONCLUSIVE** — three isolated passes did
|
||
not grant a composed pass, exactly as the order anticipated.
|
||
|
||
**Double-counting note:** the gain is far below the sum of the isolated factor
|
||
effects. Expected — defence, pitcher contact and platoon all run through the same
|
||
pitcher-batter confrontation and share signal.
|
||
|
||
---
|
||
|
||
## PHASE 5 — bands and the honest headline
|
||
|
||
Resolution moved from 0.93% to 1.39% of variance. **Both are far below what band
|
||
separation requires** — a forecast explaining 1.4% of an outcome's variance
|
||
cannot produce archetype bands that clear their own base rate.
|
||
|
||
**The headline, landed as it fell:** the pivot was correct and incomplete. The
|
||
plumbing defect was real and is fixed — three proven factors now reach the served
|
||
number for the first time, verified sign-by-sign. But **transmission alone did
|
||
not buy grade separation.** The factors are real and too weak *in combination* at
|
||
current strength.
|
||
|
||
So the next arc is **factor STRENGTH and BREADTH, not more plumbing.** The wiring
|
||
is now a working conduit with three things flowing through it; it needs more, and
|
||
stronger.
|
||
|
||
---
|
||
|
||
## PHASE 6 — the rbi anomaly, logged only
|
||
|
||
`rbi` shows **14.51% variance explained vs hits 1.03%** — 13×, on the stat we do
|
||
*not* serve corrected and which has **no proven factors**. Open question for the
|
||
next order: real counter structure (its four inputs happen to discriminate on a
|
||
stat where opportunity is lumpier), or an artefact of line placement and
|
||
base-rate spread? **Not investigated here.** It is either the biggest lever on
|
||
the board or a mirage, and it deserves its own order.
|
||
|
||
---
|
||
|
||
## Invariants
|
||
|
||
The byte-identical invariant **inverted** for hits by design — the served hits
|
||
number should move, and does. TB/rbi/runs paths and all frozen non-hits modules
|
||
verified unchanged. `p_win_prefactor` preserves the un-factored forecast in the
|
||
trace. No new Bonferroni slot (the factors were already proven); the composed
|
||
OOS claim is reported with its CI and is **not** claimed as a pass.
|