Files

T

builtbykev 528cb1a6d0 Layer 1: Statcast mechanism-data ingestion (backfill + nightly refresh)

The data foundation for the archetype and projection layers, built as the
pattern every sport inherits. Layers 2 and 3 are not touched.

PHASE 0 GATE — both match rates measured live, both 100%. Batters 40/40;
PITCHERS 66/66 across five real rosters (CLE, DET, MIN, NYY, LAD) joined by
MLBAM id against the 713-pitcher Savant feed. Zero honest-absent on identity,
because the join is an integer both systems use natively — and the snapshot
pipeline already stores it per graded row.

SOURCE — five Baseball Savant CSV leaderboards, free and public, pulled with
axios and the CSV parser savantAdapter already runs in prod. pybaseball is
deliberately NOT used: it is an MIT wrapper over these same URLs, and adding it
would reintroduce a Python runtime in a stack where the existing Python service
is already offline. min=1 on every feed, not Savant's default min=q, so the
long tail arrives and OUR minimum-sample gate decides what is thin — explicit
and testable rather than silently dropped upstream.

Measured: 1,354 rows per season (604 batters, 750 pitchers), all five feeds in
about five seconds. Pitcher mechanism includes arm angle, GB/FB/LD, chase and
whiff; batters get exit velo, launch angle, barrel and hard-hit, chase and
z-swing. Handedness rides in free on the movement feed (677 pitchers); batter
handedness stays absent pending a roster join rather than being guessed.

BACKFILL AND REFRESH ARE THE SAME CALL — a full re-pull upserted on
(sport, season, source_id). Idempotent and self-healing: a missed night
self-corrects on the next run, with no incremental who-played bookkeeping to
drift out of sync. At 1,354 rows the simple thing is also the robust one.

HONESTY RULES, each with a test: a metric the feed did not carry is null and
never 0; a thin sample is STORED and flagged rather than dropped or inflated,
because thin and missing are different claims; an unjoined player is stored
with a null player_key and joins later; and if every feed comes back empty the
job REFUSES to write, so a bad night can never blank a good table.

Freshness is treated as a truth property. updated_at on every row, and the
scheduler pages on a failed run AND on silent staleness — a job that stops
being scheduled never produces a failure, so staleness has to alarm on its own.
Never-built is deliberately not stale: different condition, different fix, and
paging on a fresh install teaches the operator to ignore the alarm.

Nightly at STATCAST_HOUR_UTC (default 11 UTC, after every game is final), kill
switch STATCAST=0, and induce-able at POST /api/internal/statcast/refresh with
a freshness probe at /statcast/status — we verify a refresh by running it, not
by waiting for the slot.

Migration 030 applied. Promoted columns for the classification-critical metrics
plus a metrics JSONB carrying every raw field, so Layer 2 can reach something we
did not promote without a re-ingest. Raw per-pitch stays out of Postgres on
purpose: one season is ~0.85 GB against a 500 MB plan ceiling, and it is
re-pullable from the free source if Layer 3 ever needs it.

Pattern documented in docs/MECHANISM-DATA.md for NBA tracking and NFL Next Gen.

Tests 3581 passed / 292 suites, web build exit 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCNgGSt5qvcLxaeQqa7Zpj

2026-07-20 21:37:30 -04:00

4.8 KiB

Raw Blame History

MECHANISM DATA — the Layer-1 pattern every sport inherits

Layer 1 of the archetype+projection build. It lands mechanism data — how a player actually does what he does — and keeps it current. It does not classify (Layer 2) or project (Layer 3).

MLB is the first instance. NBA tracking and NFL Next Gen slot into the same four boxes with a different adapter and a different source id.

The pattern

  SOURCE ADAPTER      free/public first · absent-not-zero · injectable fetch
        │             no new runtime deps · one file per sport
        ▼
  IDENTITY BRIDGE     source-native id ↔ our player_key
        │             the match rate is MEASURED and REPORTED, never assumed —
        │             the unmatched rate IS the honest-absent rate
        ▼
  AGGREGATE           pure, re-runnable, explicit MIN-SAMPLE gates
  DERIVATION          thin ≠ missing: both stored, distinguishable
        ▼
  HOT STORE           small, indexed, upserted on the natural key
  (Supabase)          + updated_at, because freshness is a truth property

The five rules that make it a pattern

Backfill and refresh are the same call. A full re-pull upserted on the natural key is idempotent and self-healing: a missed night self-corrects on the next run. No incremental "who played today" bookkeeping to drift out of sync. Only viable because the aggregate grain is small — which is the point of aggregating.
Aggregate grain, not raw. One season of raw MLB per-pitch is ~0.85 GB in Postgres against a 500 MB plan ceiling; the aggregate set is ~1,350 rows. Raw stays retrievable from the free source if Layer 3 ever needs it.
Absent is absent. A metric the feed did not carry is null, never 0. A player below the minimum sample is stored and flagged, not dropped and not inflated. A player we cannot join is stored with a null player_key and joins later — storing him is not a claim about him.
Refuse to write nothing. If every feed returns empty that is a source failure, not "there is no mechanism data". The job refuses the write so a bad night can never blank a good table.
Freshness is monitored. updated_at on every row; the scheduler pages on a failed run AND on silent staleness — a job that stops being scheduled never produces a failure, so staleness must alarm on its own. Never-built is not stale: different condition, different fix, and paging on a fresh install teaches the operator to ignore the alarm.

MLB instance

Adapter src/services/adapters/statcastAdapter.js — five Baseball Savant CSV leaderboards (free, public, no key). Direct axios + the existing CSV parser; pybaseball is deliberately not used — it is a Python wrapper over these same URLs, and the stack's Python service is already offline in prod.
Service src/services/statcastAggregateService.js — buildRows (pure), refreshSeason (the job), getFreshness / isStale (the alarm predicates).
Store statcast_aggregates (migration 030), PK (sport, season, source_id). Promoted columns for the classification-critical metrics + a metrics JSONB carrying every raw field, so Layer 2/3 can reach something we did not promote without a re-ingest.
Schedule nightly at STATCAST_HOUR_UTC (default 11 UTC ≈ 7 AM ET, after every game is final). Kill switch STATCAST=0.
Induce POST /api/internal/statcast/refresh · probe GET /api/internal/statcast/status (internal key). We verify a refresh by running it, never by waiting for the slot.

Measured (2026-07-20)


Batter match rate	100% (40/40 real players)
Pitcher match rate	100% (66/66 real roster pitchers)
Rows per season	1,354 (604 batters + 750 pitchers)
Join rate	1,354 / 1,354
Handedness present	677 pitchers (from the movement feed)
Sufficient / thin	998 / 356 at PA≥50, IP≥10
Pull time	~5 s for all five feeds

Adding a sport

Write src/services/adapters/{sport}Adapter.js returning the same shape: indexes keyed by source id, metrics strictly parsed.
Confirm and report the identity match rate before building on it.
Extend buildRows with the sport's role vocabulary and its minimum-sample gates.
Add the scheduler hour and the induce endpoint.

Nothing else changes: the store, the alarm and the upsert semantics are shared.

Commodity, not moat

Raw Statcast is public — every competitor can pull the same numbers in about a second. Ingesting it is table stakes. The edge is Layer 2 (which mechanism signals define an archetype, and where the boundaries sit), Layer 3 (projections built on them), and the settled ledger that proves whether any of it predicts anything. Having the data is not having an edge. Having it plus an attributed record is.

4.8 KiB Raw Blame History