The data foundation for the archetype and projection layers, built as the
pattern every sport inherits. Layers 2 and 3 are not touched.
PHASE 0 GATE — both match rates measured live, both 100%. Batters 40/40;
PITCHERS 66/66 across five real rosters (CLE, DET, MIN, NYY, LAD) joined by
MLBAM id against the 713-pitcher Savant feed. Zero honest-absent on identity,
because the join is an integer both systems use natively — and the snapshot
pipeline already stores it per graded row.
SOURCE — five Baseball Savant CSV leaderboards, free and public, pulled with
axios and the CSV parser savantAdapter already runs in prod. pybaseball is
deliberately NOT used: it is an MIT wrapper over these same URLs, and adding it
would reintroduce a Python runtime in a stack where the existing Python service
is already offline. min=1 on every feed, not Savant's default min=q, so the
long tail arrives and OUR minimum-sample gate decides what is thin — explicit
and testable rather than silently dropped upstream.
Measured: 1,354 rows per season (604 batters, 750 pitchers), all five feeds in
about five seconds. Pitcher mechanism includes arm angle, GB/FB/LD, chase and
whiff; batters get exit velo, launch angle, barrel and hard-hit, chase and
z-swing. Handedness rides in free on the movement feed (677 pitchers); batter
handedness stays absent pending a roster join rather than being guessed.
BACKFILL AND REFRESH ARE THE SAME CALL — a full re-pull upserted on
(sport, season, source_id). Idempotent and self-healing: a missed night
self-corrects on the next run, with no incremental who-played bookkeeping to
drift out of sync. At 1,354 rows the simple thing is also the robust one.
HONESTY RULES, each with a test: a metric the feed did not carry is null and
never 0; a thin sample is STORED and flagged rather than dropped or inflated,
because thin and missing are different claims; an unjoined player is stored
with a null player_key and joins later; and if every feed comes back empty the
job REFUSES to write, so a bad night can never blank a good table.
Freshness is treated as a truth property. updated_at on every row, and the
scheduler pages on a failed run AND on silent staleness — a job that stops
being scheduled never produces a failure, so staleness has to alarm on its own.
Never-built is deliberately not stale: different condition, different fix, and
paging on a fresh install teaches the operator to ignore the alarm.
Nightly at STATCAST_HOUR_UTC (default 11 UTC, after every game is final), kill
switch STATCAST=0, and induce-able at POST /api/internal/statcast/refresh with
a freshness probe at /statcast/status — we verify a refresh by running it, not
by waiting for the slot.
Migration 030 applied. Promoted columns for the classification-critical metrics
plus a metrics JSONB carrying every raw field, so Layer 2 can reach something we
did not promote without a re-ingest. Raw per-pitch stays out of Postgres on
purpose: one season is ~0.85 GB against a 500 MB plan ceiling, and it is
re-pullable from the free source if Layer 3 ever needs it.
Pattern documented in docs/MECHANISM-DATA.md for NBA tracking and NFL Next Gen.
Tests 3581 passed / 292 suites, web build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCNgGSt5qvcLxaeQqa7Zpj
