builtbykev
528cb1a6d0
The data foundation for the archetype and projection layers, built as the pattern every sport inherits. Layers 2 and 3 are not touched. PHASE 0 GATE — both match rates measured live, both 100%. Batters 40/40; PITCHERS 66/66 across five real rosters (CLE, DET, MIN, NYY, LAD) joined by MLBAM id against the 713-pitcher Savant feed. Zero honest-absent on identity, because the join is an integer both systems use natively — and the snapshot pipeline already stores it per graded row. SOURCE — five Baseball Savant CSV leaderboards, free and public, pulled with axios and the CSV parser savantAdapter already runs in prod. pybaseball is deliberately NOT used: it is an MIT wrapper over these same URLs, and adding it would reintroduce a Python runtime in a stack where the existing Python service is already offline. min=1 on every feed, not Savant's default min=q, so the long tail arrives and OUR minimum-sample gate decides what is thin — explicit and testable rather than silently dropped upstream. Measured: 1,354 rows per season (604 batters, 750 pitchers), all five feeds in about five seconds. Pitcher mechanism includes arm angle, GB/FB/LD, chase and whiff; batters get exit velo, launch angle, barrel and hard-hit, chase and z-swing. Handedness rides in free on the movement feed (677 pitchers); batter handedness stays absent pending a roster join rather than being guessed. BACKFILL AND REFRESH ARE THE SAME CALL — a full re-pull upserted on (sport, season, source_id). Idempotent and self-healing: a missed night self-corrects on the next run, with no incremental who-played bookkeeping to drift out of sync. At 1,354 rows the simple thing is also the robust one. HONESTY RULES, each with a test: a metric the feed did not carry is null and never 0; a thin sample is STORED and flagged rather than dropped or inflated, because thin and missing are different claims; an unjoined player is stored with a null player_key and joins later; and if every feed comes back empty the job REFUSES to write, so a bad night can never blank a good table. Freshness is treated as a truth property. updated_at on every row, and the scheduler pages on a failed run AND on silent staleness — a job that stops being scheduled never produces a failure, so staleness has to alarm on its own. Never-built is deliberately not stale: different condition, different fix, and paging on a fresh install teaches the operator to ignore the alarm. Nightly at STATCAST_HOUR_UTC (default 11 UTC, after every game is final), kill switch STATCAST=0, and induce-able at POST /api/internal/statcast/refresh with a freshness probe at /statcast/status — we verify a refresh by running it, not by waiting for the slot. Migration 030 applied. Promoted columns for the classification-critical metrics plus a metrics JSONB carrying every raw field, so Layer 2 can reach something we did not promote without a re-ingest. Raw per-pitch stays out of Postgres on purpose: one season is ~0.85 GB against a 500 MB plan ceiling, and it is re-pullable from the free source if Layer 3 ever needs it. Pattern documented in docs/MECHANISM-DATA.md for NBA tracking and NFL Next Gen. Tests 3581 passed / 292 suites, web build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCNgGSt5qvcLxaeQqa7Zpj
99 lines
4.8 KiB
Markdown
99 lines
4.8 KiB
Markdown
# MECHANISM DATA — the Layer-1 pattern every sport inherits
|
|
|
|
Layer 1 of the archetype+projection build. It lands **mechanism data** — how a
|
|
player actually does what he does — and keeps it current. It does **not**
|
|
classify (Layer 2) or project (Layer 3).
|
|
|
|
MLB is the first instance. NBA tracking and NFL Next Gen slot into the same
|
|
four boxes with a different adapter and a different source id.
|
|
|
|
## The pattern
|
|
|
|
```
|
|
SOURCE ADAPTER free/public first · absent-not-zero · injectable fetch
|
|
│ no new runtime deps · one file per sport
|
|
▼
|
|
IDENTITY BRIDGE source-native id ↔ our player_key
|
|
│ the match rate is MEASURED and REPORTED, never assumed —
|
|
│ the unmatched rate IS the honest-absent rate
|
|
▼
|
|
AGGREGATE pure, re-runnable, explicit MIN-SAMPLE gates
|
|
DERIVATION thin ≠ missing: both stored, distinguishable
|
|
▼
|
|
HOT STORE small, indexed, upserted on the natural key
|
|
(Supabase) + updated_at, because freshness is a truth property
|
|
```
|
|
|
|
### The five rules that make it a pattern
|
|
|
|
1. **Backfill and refresh are the same call.** A full re-pull upserted on the
|
|
natural key is idempotent and self-healing: a missed night self-corrects on
|
|
the next run. No incremental "who played today" bookkeeping to drift out of
|
|
sync. Only viable because the aggregate grain is small — which is the point
|
|
of aggregating.
|
|
2. **Aggregate grain, not raw.** One season of raw MLB per-pitch is ~0.85 GB in
|
|
Postgres against a 500 MB plan ceiling; the aggregate set is ~1,350 rows.
|
|
Raw stays retrievable from the free source if Layer 3 ever needs it.
|
|
3. **Absent is absent.** A metric the feed did not carry is `null`, never `0`.
|
|
A player below the minimum sample is stored and flagged, not dropped and not
|
|
inflated. A player we cannot join is stored with a null `player_key` and
|
|
joins later — storing him is not a claim about him.
|
|
4. **Refuse to write nothing.** If every feed returns empty that is a source
|
|
failure, not "there is no mechanism data". The job refuses the write so a
|
|
bad night can never blank a good table.
|
|
5. **Freshness is monitored.** `updated_at` on every row; the scheduler pages on
|
|
a failed run AND on silent staleness — a job that stops being scheduled never
|
|
produces a failure, so staleness must alarm on its own. **Never-built is not
|
|
stale**: different condition, different fix, and paging on a fresh install
|
|
teaches the operator to ignore the alarm.
|
|
|
|
## MLB instance
|
|
|
|
- **Adapter** `src/services/adapters/statcastAdapter.js` — five Baseball Savant
|
|
CSV leaderboards (free, public, no key). Direct `axios` + the existing CSV
|
|
parser; **pybaseball is deliberately not used** — it is a Python wrapper over
|
|
these same URLs, and the stack's Python service is already offline in prod.
|
|
- **Service** `src/services/statcastAggregateService.js` — `buildRows` (pure),
|
|
`refreshSeason` (the job), `getFreshness` / `isStale` (the alarm predicates).
|
|
- **Store** `statcast_aggregates` (migration 030), PK `(sport, season, source_id)`.
|
|
Promoted columns for the classification-critical metrics + a `metrics` JSONB
|
|
carrying every raw field, so Layer 2/3 can reach something we did not promote
|
|
**without a re-ingest**.
|
|
- **Schedule** nightly at `STATCAST_HOUR_UTC` (default 11 UTC ≈ 7 AM ET, after
|
|
every game is final). Kill switch `STATCAST=0`.
|
|
- **Induce** `POST /api/internal/statcast/refresh` · **probe**
|
|
`GET /api/internal/statcast/status` (internal key). We verify a refresh by
|
|
running it, never by waiting for the slot.
|
|
|
|
### Measured (2026-07-20)
|
|
|
|
| | |
|
|
|---|---|
|
|
| Batter match rate | **100%** (40/40 real players) |
|
|
| Pitcher match rate | **100%** (66/66 real roster pitchers) |
|
|
| Rows per season | **1,354** (604 batters + 750 pitchers) |
|
|
| Join rate | **1,354 / 1,354** |
|
|
| Handedness present | 677 pitchers (from the movement feed) |
|
|
| Sufficient / thin | 998 / 356 at PA≥50, IP≥10 |
|
|
| Pull time | ~5 s for all five feeds |
|
|
|
|
## Adding a sport
|
|
|
|
1. Write `src/services/adapters/{sport}Adapter.js` returning the same shape:
|
|
indexes keyed by source id, metrics strictly parsed.
|
|
2. Confirm and **report** the identity match rate before building on it.
|
|
3. Extend `buildRows` with the sport's role vocabulary and its minimum-sample
|
|
gates.
|
|
4. Add the scheduler hour and the induce endpoint.
|
|
|
|
Nothing else changes: the store, the alarm and the upsert semantics are shared.
|
|
|
|
## Commodity, not moat
|
|
|
|
Raw Statcast is public — every competitor can pull the same numbers in about a
|
|
second. Ingesting it is table stakes. The edge is Layer 2 (which mechanism
|
|
signals define an archetype, and where the boundaries sit), Layer 3
|
|
(projections built on them), and the settled ledger that proves whether any of
|
|
it predicts anything. **Having the data is not having an edge. Having it plus
|
|
an attributed record is.**
|