Files

T

builtbykev 7b85934dc3 Under-querying vs out of data: the answer depends on the unit

The platoon test's n=452 described how much of the JOIN survived, not how
much data exists. There are 1,266 clean settled hits rows and zero
quarantined ones. platoon_splits had been ingested from tonight's lineups
only (315 players), so any hitter who settled a prop without appearing in
an ingest-day lineup was silently absent from every test.

Backfilled all 380 hitters (81 fetched, 0 unresolved). Re-ran on 1,059
rows, up from 452.

THE DEMOTION IS THE HEADLINE. pitcher_contact_profile, the strongest
proven factor in the programme (-0.0064, CI [-0.0113,-0.0014]), roughly
halved to -0.0034 on more than double the sample and its corrected
interval now spans zero. The Bonferroni denominator also rose to 55,
which widens every interval -- but a denominator cannot move a point
estimate, and that halved on its own.

platoon and platoon_severity now clear the bar and are NOT promoted.
Upper bound -0.0001, on season-to-date splits that contain the games they
predict: measured contamination is 4.5% median, 12.4% at p90, 137% worst.
I had assumed ~1%. They stay CANDIDATE pending point-in-time splits.

GAME-LEVEL IS A DIFFERENT PROBLEM. game_context held zero weather rows
ever -- not because the fetcher was wrong (it correctly targets
Open-Meteo's archive) but because ledger_entries keys a game as
mlb:2026-08-03:Away@Home and game_context keys it as mlb:823437. Every
lookup missed and NULL columns read as honest absence. Third occurrence
of that class.

Fixed the join: 96/101 settled games now carry actual archived weather,
park dimensions backfilled 15 -> 30 venues.

But 928 total_bases rows sit on 47 games at 17.6 rows per game. Park and
weather assign one value per game, so resampling rows would have
manufactured a pass. factorGate now resamples clusters when rows carry
one and judges sample against effective_n; unclustered rows keep the
original path byte-for-byte. Verdict: 47 clusters < 500, and the point
estimate is +0.0011 -- worse, not merely unproven.

Weather needs ~57 more days. Park dimensions need never: there are 30
ballparks in MLB, so a venue-constant factor can never reach 500
independent units. That bar was built for player-level factors and does
not transfer.

Wind is refused. We have speed and bearing for all 96 games; we lack park
orientation, and 220 degrees is blowing out at one park and in at
another. Using speed alone would assert an effect while discarding the
sign that decides what it is.

Counter and frozen clusters untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9

2026-08-05 19:30:17 -04:00

7.1 KiB

Raw Blame History

Were we out of data, or not using what we had?

Both — and which one it is depends entirely on what unit a factor varies over. That distinction turned out to matter more than the sample counts themselves.

1. Player-level factors: we were under-querying

The platoon test reported n=452 and "48 short of the gate." That number described how much of the JOIN survived, not how much data exists.

	used	actually available
clean settled `hits` rows	452	1,266
clean settled `total_bases` rows	383	928
quarantined `hits` rows	—	0
hitters with platoon splits	298	380 needed

platoon_splits had been ingested from tonight's lineups only — 315 players — so any hitter who settled a prop but was not in a lineup on an ingest day was silently absent from every test. Backfilling all 380 (scripts/backfill-context.js) took one pass and no waiting: 81 hitters fetched, 0 unresolved, coverage now 380/380.

Re-run on the full clean history (scripts/prove-hit-factors.js, rows 452 → 1,059):

factor	n	mean shift	Brier Δ	CI (corrected, 55 tests)	verdict
`defense_by_direction`	782	0.0127	−0.0031	[−0.0050, −0.0013]	PROVES
`platoon`	1,056	0.0268	−0.0033	[−0.0062, −0.0001]	PROVES*
`platoon_severity`	700	0.0218	−0.0038	[−0.0076, −0.0001]	PROVES*
`defense`	912	0.0289	−0.0038	[−0.0079, +0.0002]	NOT_PROVEN
`pitcher_contact_profile`	1,059	0.0259	−0.0034	[−0.0078, +0.0006]	NOT_PROVEN — demoted
`park_hits`	619	0.0190	−0.0037	[−0.0076, +0.0005]	NOT_PROVEN

1a. The demotion is the real headline

pitcher_contact_profile was the strongest proven factor in the programme (Brier −0.0064, CI [−0.0113, −0.0014]). On more than double the sample its point estimate roughly halved to −0.0034 and the corrected interval now spans zero.

Two things moved at once and honesty requires naming both: the cumulative Bonferroni denominator also rose to 55, which widens every interval. But the denominator cannot touch a point estimate, and that halved on its own. This is the standing second line doing exactly what it exists for — more data demoting a favourite rather than confirming it.

1b. *The two platoon passes are NOT promoted

Both clear the bar with an upper bound of −0.0001. That is as marginal as a pass can be, and they ride a reconstructed input:

platoon_splits are season-to-date, so applying today's split to a game from 2026-07-15 means the split contains that game. Measured, not assumed:

median contamination 4.5% of the split's plate appearances
p90 12.4%
worst 137% (call-ups whose scored games outnumber their split sample)

I had originally estimated ~1%. It is four and a half times that, and it runs in the flattering direction on a result whose margin is one ten-thousandth. These stay CANDIDATE — pending point-in-time splits. Promoting a 4.5%-contaminated input on a −0.0001 bound would be exactly the kind of pass this programme keeps having to retract.

2. Game-level factors: genuinely short, and no backfill fixes it

game_context held zero weather readings, ever. The fetcher was correct and already pointed at Open-Meteo's archive endpoint. The failure was that the two tables had never been introduced:

ledger_entries.game_id  =  mlb:2026-08-03:WashingtonNationals@PhiladelphiaPhillies
game_context.game_id    =  mlb:823437

Every lookup missed, and NULL weather columns read exactly like "the weather was unavailable." Same class as the doubled /leaderboard path: graceful degradation wearing the mask of honest absence. That is now three occurrences; it is the failure mode this codebase produces most reliably.

Fixed in scripts/reconstruct-game-environment.js — resolves each ledger slug to its real statsapi game and venue, writes game_context keyed by the ledger's key, then pulls actual archived weather:

101 settled games → 96 matched, 30 venues
96/96 venue-days returned real archived weather (open_meteo_archive)
park dimensions backfilled 15 → 30 venues, zero dimension changes observed

2a. Park dimensions: what I verified and what I did not

statsapi serves only current venue geometry — it has no historical record. My capture window is 2026-08-04 to 08-05, so "no mid-season change" is verified across two days, which is nearly no verification at all. Applying current dimensions to July games is the order's stated allowance and it is almost certainly fine, but I did not verify it and will not claim to have.

2b. Why 928 rows are 47 readings

Park and weather assign one value per game. The 928 clean settled total_bases rows sit on 47 distinct games — median 17.6 rows per game. Eighteen hitters in one ballpark on one night are one reading of that ballpark, not eighteen.

Resampling rows would treat them as independent and return an interval far tighter than the evidence supports. factorGate.improvement now resamples clusters when rows carry one, and adjudicate judges sample against effective_n. Rows without a cluster keep the original path byte-for-byte.

scripts/prove-park-weather.js:

rows_built 828 | distinct_games 47 | rows_per_game 17.6
brier_delta +0.0011   (WORSE, not merely unproven)
effective_n 47
VERDICT: CANDIDATE_PENDING_SAMPLE — 47 independent clusters < 500

3. The verdict: tested-now vs real-wait

factor	unit it varies over	units held	ceiling	real wait
platoon, defence-by-direction	hitter-game	1,059	none	none — answered now
weather	game	47	none	~57 days at 7 games/settled-day
park dimensions	venue	30	30, permanently	never

The last row is arithmetic, not pessimism. There are 30 ballparks in MLB. A factor constant per venue can never accumulate 500 independent units no matter how long the ledger runs. A park-geometry effect is only ever validatable as a fixed effect with many games per park under a hierarchical model — never under a bar expressed in independent units. The n≥500 bar was designed for player-level factors and quietly does not transfer.

So: we were under-querying at the player level, and genuinely short at the game level — and for park geometry specifically, "wait for more data" was never going to be the answer.

4. Wind is refused

parkWeather reads temperature, elevation and geometry. It does not read wind, and says so on every read (wind_readable: false).

We have the wind — Open-Meteo returns speed and bearing for all 96 games. What we lack is park orientation: which compass direction each stadium's centre field faces. A 15 mph wind from 220° is blowing out to right at one park and straight in at another, and those are opposite predictions.

The tempting move is to use wind speed alone as a magnitude of disruption. That asserts an effect while discarding the sign that determines what the effect is. Wind stays unreadable until orientation is a real column.

7.1 KiB Raw Blame History Unescape Escape