builtbykev
7b85934dc3
The platoon test's n=452 described how much of the JOIN survived, not how much data exists. There are 1,266 clean settled hits rows and zero quarantined ones. platoon_splits had been ingested from tonight's lineups only (315 players), so any hitter who settled a prop without appearing in an ingest-day lineup was silently absent from every test. Backfilled all 380 hitters (81 fetched, 0 unresolved). Re-ran on 1,059 rows, up from 452. THE DEMOTION IS THE HEADLINE. pitcher_contact_profile, the strongest proven factor in the programme (-0.0064, CI [-0.0113,-0.0014]), roughly halved to -0.0034 on more than double the sample and its corrected interval now spans zero. The Bonferroni denominator also rose to 55, which widens every interval -- but a denominator cannot move a point estimate, and that halved on its own. platoon and platoon_severity now clear the bar and are NOT promoted. Upper bound -0.0001, on season-to-date splits that contain the games they predict: measured contamination is 4.5% median, 12.4% at p90, 137% worst. I had assumed ~1%. They stay CANDIDATE pending point-in-time splits. GAME-LEVEL IS A DIFFERENT PROBLEM. game_context held zero weather rows ever -- not because the fetcher was wrong (it correctly targets Open-Meteo's archive) but because ledger_entries keys a game as mlb:2026-08-03:Away@Home and game_context keys it as mlb:823437. Every lookup missed and NULL columns read as honest absence. Third occurrence of that class. Fixed the join: 96/101 settled games now carry actual archived weather, park dimensions backfilled 15 -> 30 venues. But 928 total_bases rows sit on 47 games at 17.6 rows per game. Park and weather assign one value per game, so resampling rows would have manufactured a pass. factorGate now resamples clusters when rows carry one and judges sample against effective_n; unclustered rows keep the original path byte-for-byte. Verdict: 47 clusters < 500, and the point estimate is +0.0011 -- worse, not merely unproven. Weather needs ~57 more days. Park dimensions need never: there are 30 ballparks in MLB, so a venue-constant factor can never reach 500 independent units. That bar was built for player-level factors and does not transfer. Wind is refused. We have speed and bearing for all 96 games; we lack park orientation, and 220 degrees is blowing out at one park and in at another. Using speed alone would assert an effect while discarding the sign that decides what it is. Counter and frozen clusters untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
158 lines
7.1 KiB
Markdown
158 lines
7.1 KiB
Markdown
# Were we out of data, or not using what we had?
|
||
|
||
**Both — and which one it is depends entirely on what unit a factor varies over.**
|
||
That distinction turned out to matter more than the sample counts themselves.
|
||
|
||
---
|
||
|
||
## 1. Player-level factors: we were under-querying
|
||
|
||
The platoon test reported n=452 and "48 short of the gate." That number described
|
||
how much of the JOIN survived, not how much data exists.
|
||
|
||
| | used | actually available |
|
||
|---|---|---|
|
||
| clean settled `hits` rows | 452 | **1,266** |
|
||
| clean settled `total_bases` rows | 383 | **928** |
|
||
| quarantined `hits` rows | — | **0** |
|
||
| hitters with platoon splits | 298 | 380 needed |
|
||
|
||
`platoon_splits` had been ingested from *tonight's lineups only* — 315 players —
|
||
so any hitter who settled a prop but was not in a lineup on an ingest day was
|
||
silently absent from every test. Backfilling all 380 (`scripts/backfill-context.js`)
|
||
took one pass and no waiting: **81 hitters fetched, 0 unresolved, coverage now
|
||
380/380.**
|
||
|
||
Re-run on the full clean history (`scripts/prove-hit-factors.js`, rows 452 → **1,059**):
|
||
|
||
| factor | n | mean shift | Brier Δ | CI (corrected, 55 tests) | verdict |
|
||
|---|---|---|---|---|---|
|
||
| `defense_by_direction` | 782 | 0.0127 | −0.0031 | [−0.0050, −0.0013] | **PROVES** |
|
||
| `platoon` | 1,056 | 0.0268 | −0.0033 | [−0.0062, −0.0001] | PROVES\* |
|
||
| `platoon_severity` | 700 | 0.0218 | −0.0038 | [−0.0076, −0.0001] | PROVES\* |
|
||
| `defense` | 912 | 0.0289 | −0.0038 | [−0.0079, +0.0002] | NOT_PROVEN |
|
||
| `pitcher_contact_profile` | 1,059 | 0.0259 | −0.0034 | [−0.0078, +0.0006] | **NOT_PROVEN — demoted** |
|
||
| `park_hits` | 619 | 0.0190 | −0.0037 | [−0.0076, +0.0005] | NOT_PROVEN |
|
||
|
||
### 1a. The demotion is the real headline
|
||
|
||
`pitcher_contact_profile` was the strongest proven factor in the programme
|
||
(Brier −0.0064, CI [−0.0113, −0.0014]). On more than double the sample its point
|
||
estimate **roughly halved to −0.0034** and the corrected interval now spans zero.
|
||
|
||
Two things moved at once and honesty requires naming both: the cumulative
|
||
Bonferroni denominator also rose to 55, which widens every interval. But the
|
||
denominator cannot touch a *point estimate*, and that halved on its own. This is
|
||
the standing second line doing exactly what it exists for — more data demoting a
|
||
favourite rather than confirming it.
|
||
|
||
### 1b. \*The two platoon passes are NOT promoted
|
||
|
||
Both clear the bar with an upper bound of **−0.0001**. That is as marginal as a
|
||
pass can be, and they ride a reconstructed input:
|
||
|
||
`platoon_splits` are **season-to-date**, so applying today's split to a game from
|
||
2026-07-15 means the split contains that game. Measured, not assumed:
|
||
|
||
- median contamination **4.5%** of the split's plate appearances
|
||
- p90 **12.4%**
|
||
- worst **137%** (call-ups whose scored games outnumber their split sample)
|
||
|
||
I had originally estimated ~1%. It is four and a half times that, and it runs in
|
||
the flattering direction on a result whose margin is one ten-thousandth. These
|
||
stay **CANDIDATE — pending point-in-time splits**. Promoting a 4.5%-contaminated
|
||
input on a −0.0001 bound would be exactly the kind of pass this programme keeps
|
||
having to retract.
|
||
|
||
---
|
||
|
||
## 2. Game-level factors: genuinely short, and no backfill fixes it
|
||
|
||
`game_context` held **zero** weather readings, ever. The fetcher was correct and
|
||
already pointed at Open-Meteo's **archive** endpoint. The failure was that the
|
||
two tables had never been introduced:
|
||
|
||
```
|
||
ledger_entries.game_id = mlb:2026-08-03:WashingtonNationals@PhiladelphiaPhillies
|
||
game_context.game_id = mlb:823437
|
||
```
|
||
|
||
Every lookup missed, and NULL weather columns read exactly like "the weather was
|
||
unavailable." **Same class as the doubled `/leaderboard` path: graceful
|
||
degradation wearing the mask of honest absence.** That is now three occurrences;
|
||
it is the failure mode this codebase produces most reliably.
|
||
|
||
Fixed in `scripts/reconstruct-game-environment.js` — resolves each ledger slug to
|
||
its real statsapi game and venue, writes `game_context` keyed by the *ledger's*
|
||
key, then pulls actual archived weather:
|
||
|
||
- 101 settled games → **96 matched**, 30 venues
|
||
- **96/96 venue-days returned real archived weather** (`open_meteo_archive`)
|
||
- park dimensions backfilled 15 → **30 venues**, zero dimension changes observed
|
||
|
||
### 2a. Park dimensions: what I verified and what I did not
|
||
|
||
statsapi serves only **current** venue geometry — it has no historical record. My
|
||
capture window is 2026-08-04 to 08-05, so "no mid-season change" is verified
|
||
across *two days*, which is nearly no verification at all. Applying current
|
||
dimensions to July games is the order's stated allowance and it is almost
|
||
certainly fine, but I did not verify it and will not claim to have.
|
||
|
||
### 2b. Why 928 rows are 47 readings
|
||
|
||
Park and weather assign **one value per game**. The 928 clean settled
|
||
`total_bases` rows sit on **47 distinct games — median 17.6 rows per game.**
|
||
Eighteen hitters in one ballpark on one night are one reading of that ballpark,
|
||
not eighteen.
|
||
|
||
Resampling rows would treat them as independent and return an interval far
|
||
tighter than the evidence supports. `factorGate.improvement` now resamples
|
||
**clusters** when rows carry one, and `adjudicate` judges sample against
|
||
`effective_n`. Rows without a cluster keep the original path byte-for-byte.
|
||
|
||
`scripts/prove-park-weather.js`:
|
||
|
||
```
|
||
rows_built 828 | distinct_games 47 | rows_per_game 17.6
|
||
brier_delta +0.0011 (WORSE, not merely unproven)
|
||
effective_n 47
|
||
VERDICT: CANDIDATE_PENDING_SAMPLE — 47 independent clusters < 500
|
||
```
|
||
|
||
---
|
||
|
||
## 3. The verdict: tested-now vs real-wait
|
||
|
||
| factor | unit it varies over | units held | ceiling | real wait |
|
||
|---|---|---|---|---|
|
||
| platoon, defence-by-direction | **hitter-game** | 1,059 | none | **none — answered now** |
|
||
| weather | **game** | 47 | none | **~57 days** at 7 games/settled-day |
|
||
| park dimensions | **venue** | 30 | **30, permanently** | **never** |
|
||
|
||
The last row is arithmetic, not pessimism. **There are 30 ballparks in MLB.** A
|
||
factor constant per venue can never accumulate 500 independent units no matter
|
||
how long the ledger runs. A park-geometry effect is only ever validatable as a
|
||
fixed effect with many games per park under a hierarchical model — never under a
|
||
bar expressed in independent units. The n≥500 bar was designed for player-level
|
||
factors and quietly does not transfer.
|
||
|
||
**So: we were under-querying at the player level, and genuinely short at the game
|
||
level — and for park geometry specifically, "wait for more data" was never going
|
||
to be the answer.**
|
||
|
||
---
|
||
|
||
## 4. Wind is refused
|
||
|
||
`parkWeather` reads temperature, elevation and geometry. It does **not** read
|
||
wind, and says so on every read (`wind_readable: false`).
|
||
|
||
We have the wind — Open-Meteo returns speed and bearing for all 96 games. What we
|
||
lack is **park orientation**: which compass direction each stadium's centre field
|
||
faces. A 15 mph wind from 220° is blowing out to right at one park and straight in
|
||
at another, and those are opposite predictions.
|
||
|
||
The tempting move is to use wind *speed* alone as a magnitude of disruption. That
|
||
asserts an effect while discarding the sign that determines what the effect is.
|
||
Wind stays unreadable until orientation is a real column.
|