diff --git a/specs/STATE.md b/specs/STATE.md index 94abdaf..18e7fa0 100644 --- a/specs/STATE.md +++ b/specs/STATE.md @@ -1,5 +1,5 @@ # VYNDR — STATE OF THE WORLD -### As of `4e2f488` (main, DEPLOYED + fingerprinted live), 2026-07-20. This file opens every future session. **Start with the CURRENT STATUS + OPEN ITEMS block below.** +### As of `a499598` (main, DEPLOYED + fingerprinted live), 2026-07-21. This file opens every future session. **Start with the CURRENT STATUS + OPEN ITEMS block below.** --- @@ -131,6 +131,93 @@ exist locally; harmless, the data restores completely. `specs/snapshot-retention.md` (Phase 2 report + schema) · `specs/audit-data/grade-collapse.md` · `specs/audit-data/gate-simulation.md`. +--- + +# 🟢 SESSION 68 — LAYER 1: STATCAST MECHANISM DATA (2026-07-21) +Ingest + stay-current. **Layers 2 (archetypes) and 3 (projection) NOT built.** +Blueprint: **`docs/MECHANISM-DATA.md`** — the pattern every sport inherits. + +## Phase 0 gate — both match rates 100% +- **Batters 40/40 · PITCHERS 66/66** (real rosters: CLE, DET, MIN, NYY, LAD), + joined by **MLBAM id** against the 713-pitcher Savant feed. Zero honest-absent + on identity — the join is an integer both systems use natively, and the + snapshot pipeline **already stores it** (`playerId`) per graded row. +- **pybaseball deliberately NOT used** — it is an MIT wrapper over the same CSV + URLs; adding it reintroduces a Python runtime in a stack where the existing + Python service is already offline. Direct axios + `savantAdapter`'s parser. + +## What landed +`statcastAdapter` (6 free Savant feeds) → `statcastAggregateService` → +`statcast_aggregates` (migrations 030 + 031). **Live in prod:** + +| | | +|---|---| +| Rows | **1,354** (604 batters · 750 pitchers) | +| Distinct players | 1,312 → **42 two-way players carry BOTH profiles** | +| Join rate | **1,354 / 1,354 · 0 unjoined** | +| Handedness | 677 pitchers (movement feed) | +| Sufficient / thin | 998 / 356 (PA≥50, IP≥10) | +| Arsenals | avg **4.4 pitches**, max 8, 679 with ≥3 | +| Table size | **5.3 MB** (vs 500 MB plan ceiling) | +| Pull time | ~5 s, all six feeds | + +**Backfill and refresh are the SAME call** — full re-pull upserted on +`(sport, season, source_id, role)`. Idempotent + self-healing: **5 induced runs +→ still exactly 1,354 rows.** No incremental who-played bookkeeping to drift. + +## 🔴 TWO REAL BUGS CAUGHT BY INDUCING, NOT BY REVIEW +1. **Two-way players broke the write.** PK `(sport, season, source_id)` + collapsed a player's batter and pitcher profiles into one key → one batch hit + the same row twice → `ON CONFLICT DO UPDATE command cannot affect row a + second time`, first chunk written, rest refused. **ROLE is in the key now** + (migration 031). Ohtani stores 419 PA / 17.2% barrel as a batter AND 85.2 IP + / 33.7° arm angle / 52.1% GB as a pitcher — two real profiles, not one + invented player. +2. **Pitch mix was primary-pitch-only.** `pitch-movement` with an empty + `pitch_type` returns **ONE row per pitcher** (677 rows / ~700 pitchers), so a + five-pitch arsenal stored as one pitch — which would have classified every + pitcher as a one-pitch arm. Mix now comes from **`pitch-arsenal-stats` + (3,205 rows = pitcher × pitch)**; movement still supplies velo/break/ + handedness folded onto the primary pitch. Skubal: 5 pitches, L, FF 96.7. + +## Honesty rules (each has a test) +Absent metric → `null`, never 0 · thin sample **stored and flagged** (thin ≠ +missing) · unjoined player stored with null `player_key`, joins later · **all +feeds empty → REFUSES to write** so a bad night can never blank a good table · +`updated_at` on every row. + +## Staleness — verified across simulated conditions +`0.1h → false · 26h → false · 50h → TRUE · 168h → TRUE · NEVER-BUILT → false` +(threshold 48h). **Never-built is deliberately NOT stale** — different +condition, different fix; paging on a fresh install teaches the operator to +ignore the alarm. The scheduler pages on a **failed run** AND on **silent +staleness** separately: a job that stops being scheduled never produces a +failure. + +## Ops +Nightly at `STATCAST_HOUR_UTC` (default **11 UTC**, after every game is final); +kill switch `STATCAST=0`; gates `STATCAST_MIN_PA` / `MIN_IP` / `MAX_AGE_HOURS`. +**Induce:** `POST /api/internal/statcast/refresh` · **probe:** +`GET /api/internal/statcast/status`. +**⚠️ `/api/internal/*` is NOT reachable on `vyndr.app`** (no Next proxy, by +design) — use **`api.vyndr.app`**. Also: local `@supabase/supabase-js` egress +from the WSL box FAILS (`fetch failed`); the MCP tool works. Induce DB-writing +jobs on the server, not locally. + +## Raw per-pitch — measured, and deliberately NOT stored +2026-07-19: **4,791 pitches · 119 cols · 3.26 MB · 680 B/row · 1.2 s** → +**~776k pitches/season = 503 MB CSV, 171 MB gzipped, ~0.85 GB in Postgres** +against a **500 MB free-plan ceiling** (DB is 26 MB today). Aggregate grain is +5.3 MB. Raw stays re-pullable from the free source if Layer 3 needs it; the +cold tier (Storage Box) is the two-tier home when it does. + +## Commodity, not moat +Raw Statcast is public — anyone can pull it in a second. The edge is Layer 2 +(which signals define an archetype, and where the boundaries sit), Layer 3 +(projections), and the settled ledger that proves them. **Having the data is not +having an edge.** + + --- # 🟢 SESSION 67 — PRICE LAYER GATED + WIRED (2026-07-20)