Layer 1: Statcast mechanism-data ingestion (backfill + nightly refresh)
The data foundation for the archetype and projection layers, built as the pattern every sport inherits. Layers 2 and 3 are not touched. PHASE 0 GATE — both match rates measured live, both 100%. Batters 40/40; PITCHERS 66/66 across five real rosters (CLE, DET, MIN, NYY, LAD) joined by MLBAM id against the 713-pitcher Savant feed. Zero honest-absent on identity, because the join is an integer both systems use natively — and the snapshot pipeline already stores it per graded row. SOURCE — five Baseball Savant CSV leaderboards, free and public, pulled with axios and the CSV parser savantAdapter already runs in prod. pybaseball is deliberately NOT used: it is an MIT wrapper over these same URLs, and adding it would reintroduce a Python runtime in a stack where the existing Python service is already offline. min=1 on every feed, not Savant's default min=q, so the long tail arrives and OUR minimum-sample gate decides what is thin — explicit and testable rather than silently dropped upstream. Measured: 1,354 rows per season (604 batters, 750 pitchers), all five feeds in about five seconds. Pitcher mechanism includes arm angle, GB/FB/LD, chase and whiff; batters get exit velo, launch angle, barrel and hard-hit, chase and z-swing. Handedness rides in free on the movement feed (677 pitchers); batter handedness stays absent pending a roster join rather than being guessed. BACKFILL AND REFRESH ARE THE SAME CALL — a full re-pull upserted on (sport, season, source_id). Idempotent and self-healing: a missed night self-corrects on the next run, with no incremental who-played bookkeeping to drift out of sync. At 1,354 rows the simple thing is also the robust one. HONESTY RULES, each with a test: a metric the feed did not carry is null and never 0; a thin sample is STORED and flagged rather than dropped or inflated, because thin and missing are different claims; an unjoined player is stored with a null player_key and joins later; and if every feed comes back empty the job REFUSES to write, so a bad night can never blank a good table. Freshness is treated as a truth property. updated_at on every row, and the scheduler pages on a failed run AND on silent staleness — a job that stops being scheduled never produces a failure, so staleness has to alarm on its own. Never-built is deliberately not stale: different condition, different fix, and paging on a fresh install teaches the operator to ignore the alarm. Nightly at STATCAST_HOUR_UTC (default 11 UTC, after every game is final), kill switch STATCAST=0, and induce-able at POST /api/internal/statcast/refresh with a freshness probe at /statcast/status — we verify a refresh by running it, not by waiting for the slot. Migration 030 applied. Promoted columns for the classification-critical metrics plus a metrics JSONB carrying every raw field, so Layer 2 can reach something we did not promote without a re-ingest. Raw per-pitch stays out of Postgres on purpose: one season is ~0.85 GB against a 500 MB plan ceiling, and it is re-pullable from the free source if Layer 3 ever needs it. Pattern documented in docs/MECHANISM-DATA.md for NBA tracking and NFL Next Gen. Tests 3581 passed / 292 suites, web build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCNgGSt5qvcLxaeQqa7Zpj
This commit is contained in:
@@ -0,0 +1,98 @@
|
|||||||
|
# MECHANISM DATA — the Layer-1 pattern every sport inherits
|
||||||
|
|
||||||
|
Layer 1 of the archetype+projection build. It lands **mechanism data** — how a
|
||||||
|
player actually does what he does — and keeps it current. It does **not**
|
||||||
|
classify (Layer 2) or project (Layer 3).
|
||||||
|
|
||||||
|
MLB is the first instance. NBA tracking and NFL Next Gen slot into the same
|
||||||
|
four boxes with a different adapter and a different source id.
|
||||||
|
|
||||||
|
## The pattern
|
||||||
|
|
||||||
|
```
|
||||||
|
SOURCE ADAPTER free/public first · absent-not-zero · injectable fetch
|
||||||
|
│ no new runtime deps · one file per sport
|
||||||
|
▼
|
||||||
|
IDENTITY BRIDGE source-native id ↔ our player_key
|
||||||
|
│ the match rate is MEASURED and REPORTED, never assumed —
|
||||||
|
│ the unmatched rate IS the honest-absent rate
|
||||||
|
▼
|
||||||
|
AGGREGATE pure, re-runnable, explicit MIN-SAMPLE gates
|
||||||
|
DERIVATION thin ≠ missing: both stored, distinguishable
|
||||||
|
▼
|
||||||
|
HOT STORE small, indexed, upserted on the natural key
|
||||||
|
(Supabase) + updated_at, because freshness is a truth property
|
||||||
|
```
|
||||||
|
|
||||||
|
### The five rules that make it a pattern
|
||||||
|
|
||||||
|
1. **Backfill and refresh are the same call.** A full re-pull upserted on the
|
||||||
|
natural key is idempotent and self-healing: a missed night self-corrects on
|
||||||
|
the next run. No incremental "who played today" bookkeeping to drift out of
|
||||||
|
sync. Only viable because the aggregate grain is small — which is the point
|
||||||
|
of aggregating.
|
||||||
|
2. **Aggregate grain, not raw.** One season of raw MLB per-pitch is ~0.85 GB in
|
||||||
|
Postgres against a 500 MB plan ceiling; the aggregate set is ~1,350 rows.
|
||||||
|
Raw stays retrievable from the free source if Layer 3 ever needs it.
|
||||||
|
3. **Absent is absent.** A metric the feed did not carry is `null`, never `0`.
|
||||||
|
A player below the minimum sample is stored and flagged, not dropped and not
|
||||||
|
inflated. A player we cannot join is stored with a null `player_key` and
|
||||||
|
joins later — storing him is not a claim about him.
|
||||||
|
4. **Refuse to write nothing.** If every feed returns empty that is a source
|
||||||
|
failure, not "there is no mechanism data". The job refuses the write so a
|
||||||
|
bad night can never blank a good table.
|
||||||
|
5. **Freshness is monitored.** `updated_at` on every row; the scheduler pages on
|
||||||
|
a failed run AND on silent staleness — a job that stops being scheduled never
|
||||||
|
produces a failure, so staleness must alarm on its own. **Never-built is not
|
||||||
|
stale**: different condition, different fix, and paging on a fresh install
|
||||||
|
teaches the operator to ignore the alarm.
|
||||||
|
|
||||||
|
## MLB instance
|
||||||
|
|
||||||
|
- **Adapter** `src/services/adapters/statcastAdapter.js` — five Baseball Savant
|
||||||
|
CSV leaderboards (free, public, no key). Direct `axios` + the existing CSV
|
||||||
|
parser; **pybaseball is deliberately not used** — it is a Python wrapper over
|
||||||
|
these same URLs, and the stack's Python service is already offline in prod.
|
||||||
|
- **Service** `src/services/statcastAggregateService.js` — `buildRows` (pure),
|
||||||
|
`refreshSeason` (the job), `getFreshness` / `isStale` (the alarm predicates).
|
||||||
|
- **Store** `statcast_aggregates` (migration 030), PK `(sport, season, source_id)`.
|
||||||
|
Promoted columns for the classification-critical metrics + a `metrics` JSONB
|
||||||
|
carrying every raw field, so Layer 2/3 can reach something we did not promote
|
||||||
|
**without a re-ingest**.
|
||||||
|
- **Schedule** nightly at `STATCAST_HOUR_UTC` (default 11 UTC ≈ 7 AM ET, after
|
||||||
|
every game is final). Kill switch `STATCAST=0`.
|
||||||
|
- **Induce** `POST /api/internal/statcast/refresh` · **probe**
|
||||||
|
`GET /api/internal/statcast/status` (internal key). We verify a refresh by
|
||||||
|
running it, never by waiting for the slot.
|
||||||
|
|
||||||
|
### Measured (2026-07-20)
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| Batter match rate | **100%** (40/40 real players) |
|
||||||
|
| Pitcher match rate | **100%** (66/66 real roster pitchers) |
|
||||||
|
| Rows per season | **1,354** (604 batters + 750 pitchers) |
|
||||||
|
| Join rate | **1,354 / 1,354** |
|
||||||
|
| Handedness present | 677 pitchers (from the movement feed) |
|
||||||
|
| Sufficient / thin | 998 / 356 at PA≥50, IP≥10 |
|
||||||
|
| Pull time | ~5 s for all five feeds |
|
||||||
|
|
||||||
|
## Adding a sport
|
||||||
|
|
||||||
|
1. Write `src/services/adapters/{sport}Adapter.js` returning the same shape:
|
||||||
|
indexes keyed by source id, metrics strictly parsed.
|
||||||
|
2. Confirm and **report** the identity match rate before building on it.
|
||||||
|
3. Extend `buildRows` with the sport's role vocabulary and its minimum-sample
|
||||||
|
gates.
|
||||||
|
4. Add the scheduler hour and the induce endpoint.
|
||||||
|
|
||||||
|
Nothing else changes: the store, the alarm and the upsert semantics are shared.
|
||||||
|
|
||||||
|
## Commodity, not moat
|
||||||
|
|
||||||
|
Raw Statcast is public — every competitor can pull the same numbers in about a
|
||||||
|
second. Ingesting it is table stakes. The edge is Layer 2 (which mechanism
|
||||||
|
signals define an archetype, and where the boundaries sit), Layer 3
|
||||||
|
(projections built on them), and the settled ledger that proves whether any of
|
||||||
|
it predicts anything. **Having the data is not having an edge. Having it plus
|
||||||
|
an attributed record is.**
|
||||||
@@ -431,6 +431,46 @@ router.post('/newsletter/send', async (req, res) => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* POST /api/internal/statcast/refresh — INDUCE the Layer-1 mechanism refresh.
|
||||||
|
*
|
||||||
|
* Same call the nightly scheduler makes. Exists so the job is verifiable ON
|
||||||
|
* DEMAND: we prove a refresh works by running it and reading the result, never
|
||||||
|
* by waiting for the cron slot. Idempotent — running it twice is a no-op beyond
|
||||||
|
* refreshing values and updated_at.
|
||||||
|
*/
|
||||||
|
router.post('/statcast/refresh', async (req, res) => {
|
||||||
|
try {
|
||||||
|
const agg = require('../services/statcastAggregateService');
|
||||||
|
const season = req.query.season ? Number(req.query.season) : undefined;
|
||||||
|
const out = await agg.refreshSeason({ season });
|
||||||
|
const fresh = await agg.getFreshness({});
|
||||||
|
return res.status(out.ok ? 200 : 500).json({ ...out, freshness: fresh, stale: agg.isStale(fresh) });
|
||||||
|
} catch (err) {
|
||||||
|
console.error('[internal/statcast]', err.message);
|
||||||
|
return res.status(500).json({ ok: false, error: err.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
/** GET /api/internal/statcast/status — freshness probe for the mechanism tier. */
|
||||||
|
router.get('/statcast/status', async (req, res) => {
|
||||||
|
try {
|
||||||
|
const agg = require('../services/statcastAggregateService');
|
||||||
|
const fresh = await agg.getFreshness({});
|
||||||
|
return res.json({
|
||||||
|
...fresh,
|
||||||
|
stale: agg.isStale(fresh),
|
||||||
|
max_age_hours: agg.MAX_AGE_HOURS,
|
||||||
|
min_pa: agg.MIN_PA,
|
||||||
|
min_ip: agg.MIN_IP,
|
||||||
|
cron_hour_utc: Number(process.env.STATCAST_HOUR_UTC || 11),
|
||||||
|
enabled: process.env.STATCAST !== '0',
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
return res.status(500).json({ error: err.message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
router.post('/outcomes/:sport', async (req, res) => {
|
router.post('/outcomes/:sport', async (req, res) => {
|
||||||
const outcomes = require('../services/outcomeService');
|
const outcomes = require('../services/outcomeService');
|
||||||
try {
|
try {
|
||||||
|
|||||||
@@ -0,0 +1,217 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* STATCAST ADAPTER (Layer 1) — the SOURCE ADAPTER of the mechanism-data pattern.
|
||||||
|
*
|
||||||
|
* Pulls per-player-per-season Statcast aggregates from Baseball Savant. Free,
|
||||||
|
* public, no key, no quota — the same host `savantAdapter` already uses in prod.
|
||||||
|
*
|
||||||
|
* WHY NOT pybaseball: it is an MIT-licensed Python wrapper around these exact
|
||||||
|
* CSV URLs. Adding it would reintroduce a Python runtime dependency in a stack
|
||||||
|
* where the existing Python service is already offline in production (nba_api
|
||||||
|
* degrades to found:false for exactly that reason). We call the endpoints
|
||||||
|
* directly with axios and reuse savantAdapter's quote-aware CSV parser — no new
|
||||||
|
* dependency, one less thing to keep alive.
|
||||||
|
*
|
||||||
|
* FIVE FEEDS (all measured live 2026-07-20, all `csv=true`, all min=1 so the
|
||||||
|
* long tail is included rather than silently dropped by Savant's `min=q`):
|
||||||
|
* 1. batter discipline 604 rows — k/bb/whiff/swing/CHASE/z-swing/sweetspot/
|
||||||
|
* barrel/hard-hit/best-speed
|
||||||
|
* 2. batter batted-ball 600 rows — exit velo (avg/max/ev50), launch angle,
|
||||||
|
* GB/FB-LD, barrels, ev95+
|
||||||
|
* 3. pitcher discipline 713 rows — k/bb/whiff/swing/chase, GB/FB/LD%,
|
||||||
|
* barrel/hard-hit allowed, ARM ANGLE
|
||||||
|
* 4. pitcher batted-ball 749 rows — contact quality allowed
|
||||||
|
* 5. pitch movement 677 rows — per pitch type: velo, break_x/z, induced
|
||||||
|
* break, usage — and PITCH_HAND (handedness)
|
||||||
|
*
|
||||||
|
* HONESTY DOCTRINE (inherited from savantAdapter / espnStatsAdapter):
|
||||||
|
* - A missing metric is NULL, never 0. `Number(null) === 0` is the fabrication
|
||||||
|
* trap this whole layer exists to avoid.
|
||||||
|
* - Any unrecognised shape → that field absent; never a guessed default.
|
||||||
|
* - Injectable (`opts.fetchImpl`) so tests never touch the network.
|
||||||
|
*
|
||||||
|
* COMMODITY, NOT MOAT: this data is public and every competitor can pull it in
|
||||||
|
* about a second. The edge is Layer 2 (archetype definitions) and Layer 3
|
||||||
|
* (projections) built on top — plus the settled ledger that proves them.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const axios = require('axios');
|
||||||
|
const { __internals: savant } = require('./savantAdapter');
|
||||||
|
|
||||||
|
const { parseCsv, numOrNull } = savant;
|
||||||
|
|
||||||
|
const HTTP_TIMEOUT_MS = 60_000;
|
||||||
|
const DEFAULT_SEASON = Number(process.env.STATCAST_SEASON) || 2026;
|
||||||
|
const BASE = 'https://baseballsavant.mlb.com/leaderboard';
|
||||||
|
|
||||||
|
/** `min=1` deliberately: Savant's default `min=q` drops every non-qualified
|
||||||
|
* player, which would silently make a thin-sample player look absent rather
|
||||||
|
* than thin. We take the long tail and apply our OWN minimum-sample gate at
|
||||||
|
* the aggregate layer, where it is explicit and testable. */
|
||||||
|
const FEEDS = Object.freeze({
|
||||||
|
batter_discipline: (y) => `${BASE}/custom?year=${y}&type=batter&filter=&min=1&selections=pa,k_percent,bb_percent,whiff_percent,swing_percent,oz_swing_percent,z_swing_percent,sweet_spot_percent,barrel_batted_rate,hard_hit_percent,avg_best_speed,avg_hyper_speed&chart=false&x=pa&y=pa&r=no&chartType=beeswarm&sort=pa&sortDir=desc&csv=true`,
|
||||||
|
batter_batted_ball: (y) => `${BASE}/statcast?type=batter&year=${y}&position=&team=&min=1&csv=true`,
|
||||||
|
pitcher_discipline: (y) => `${BASE}/custom?year=${y}&type=pitcher&filter=&min=1&selections=p_formatted_ip,k_percent,bb_percent,whiff_percent,swing_percent,oz_swing_percent,groundballs_percent,flyballs_percent,linedrives_percent,barrel_batted_rate,hard_hit_percent,arm_angle&chart=false&x=p_formatted_ip&y=p_formatted_ip&r=no&chartType=beeswarm&sort=p_formatted_ip&sortDir=desc&csv=true`,
|
||||||
|
pitcher_batted_ball: (y) => `${BASE}/statcast?type=pitcher&year=${y}&position=&team=&min=1&csv=true`,
|
||||||
|
pitch_movement: (y) => `${BASE}/pitch-movement?year=${y}&team=&min=1&pitch_type=&hand=&csv=true`,
|
||||||
|
});
|
||||||
|
|
||||||
|
async function fetchText(url, opts = {}) {
|
||||||
|
if (opts.fetchImpl) return opts.fetchImpl(url);
|
||||||
|
const res = await axios.get(url, {
|
||||||
|
timeout: HTTP_TIMEOUT_MS,
|
||||||
|
responseType: 'text',
|
||||||
|
headers: { Accept: 'text/csv,*/*' },
|
||||||
|
// Savant occasionally returns a large body; don't let axios truncate.
|
||||||
|
maxContentLength: 64 * 1024 * 1024,
|
||||||
|
transformResponse: [(d) => d],
|
||||||
|
});
|
||||||
|
return typeof res.data === 'string' ? res.data : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Savant's first column is the literal header `last_name, first_name` — a
|
||||||
|
* comma INSIDE a quoted header. The value is likewise `"Last, First"`. */
|
||||||
|
function flipName(v) {
|
||||||
|
const s = String(v || '').trim();
|
||||||
|
if (!s) return null;
|
||||||
|
const i = s.indexOf(',');
|
||||||
|
if (i < 0) return s;
|
||||||
|
const last = s.slice(0, i).trim();
|
||||||
|
const first = s.slice(i + 1).trim();
|
||||||
|
return first ? `${first} ${last}` : last;
|
||||||
|
}
|
||||||
|
|
||||||
|
function idOf(row) {
|
||||||
|
const raw = row.player_id ?? row.pitcher_id ?? row.batter_id;
|
||||||
|
const n = Number(raw);
|
||||||
|
return Number.isFinite(n) && n > 0 ? n : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Row → { id, name, metrics } with EVERY metric strictly parsed (null, not 0). */
|
||||||
|
function normalizeRow(row, fields) {
|
||||||
|
const id = idOf(row);
|
||||||
|
if (id == null) return null;
|
||||||
|
const metrics = {};
|
||||||
|
for (const [out, src] of Object.entries(fields)) {
|
||||||
|
const v = numOrNull(row[src]);
|
||||||
|
if (v != null) metrics[out] = v;
|
||||||
|
}
|
||||||
|
return { id, name: flipName(row['last_name, first_name'] || row.last_name), metrics };
|
||||||
|
}
|
||||||
|
|
||||||
|
const BATTER_DISCIPLINE = {
|
||||||
|
pa: 'pa', k_pct: 'k_percent', bb_pct: 'bb_percent', whiff_pct: 'whiff_percent',
|
||||||
|
swing_pct: 'swing_percent', chase_pct: 'oz_swing_percent', z_swing_pct: 'z_swing_percent',
|
||||||
|
sweet_spot_pct: 'sweet_spot_percent', barrel_pct: 'barrel_batted_rate',
|
||||||
|
hard_hit_pct: 'hard_hit_percent', avg_best_speed: 'avg_best_speed',
|
||||||
|
};
|
||||||
|
const BATTED_BALL = {
|
||||||
|
bip: 'attempts', avg_launch_angle: 'avg_hit_angle', sweet_spot_pct_bb: 'anglesweetspotpercent',
|
||||||
|
max_exit_velo: 'max_hit_speed', avg_exit_velo: 'avg_hit_speed', ev50: 'ev50',
|
||||||
|
fb_ld_pct: 'fbld', gb_pct_bb: 'gb', max_distance: 'max_distance',
|
||||||
|
avg_distance: 'avg_distance', avg_hr_distance: 'avg_hr_distance',
|
||||||
|
ev95_pct: 'ev95percent', barrels: 'barrels', barrel_pct_bb: 'brl_percent', barrel_per_pa: 'brl_pa',
|
||||||
|
};
|
||||||
|
const PITCHER_DISCIPLINE = {
|
||||||
|
ip: 'p_formatted_ip', k_pct: 'k_percent', bb_pct: 'bb_percent', whiff_pct: 'whiff_percent',
|
||||||
|
swing_pct: 'swing_percent', chase_pct: 'oz_swing_percent', gb_pct: 'groundballs_percent',
|
||||||
|
fb_pct: 'flyballs_percent', ld_pct: 'linedrives_percent',
|
||||||
|
barrel_pct_allowed: 'barrel_batted_rate', hard_hit_pct_allowed: 'hard_hit_percent',
|
||||||
|
arm_angle: 'arm_angle',
|
||||||
|
};
|
||||||
|
|
||||||
|
function indexBy(rows, fields) {
|
||||||
|
const out = new Map();
|
||||||
|
for (const r of rows) {
|
||||||
|
const n = normalizeRow(r, fields);
|
||||||
|
if (!n) continue;
|
||||||
|
const prev = out.get(n.id);
|
||||||
|
out.set(n.id, prev ? { ...prev, ...n, metrics: { ...prev.metrics, ...n.metrics } } : n);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pitch movement is LONG (one row per pitcher × pitch type). Collapses to a
|
||||||
|
* pitch-mix array per pitcher, sorted by usage. Also the only feed carrying
|
||||||
|
* HANDEDNESS (`pitch_hand`), so it doubles as the pitcher-handedness source.
|
||||||
|
*/
|
||||||
|
function indexPitchMix(rows) {
|
||||||
|
const out = new Map();
|
||||||
|
for (const r of rows) {
|
||||||
|
const id = idOf(r);
|
||||||
|
if (id == null) continue;
|
||||||
|
const entry = out.get(id) || { id, name: flipName(r['last_name, first_name']), throws: null, pitches: [] };
|
||||||
|
if (!entry.throws && r.pitch_hand) entry.throws = String(r.pitch_hand).trim().toUpperCase() || null;
|
||||||
|
// `pitch_per` arrives as a FRACTION (0.371), unlike every other _pct field
|
||||||
|
// on these feeds. Normalise to a percentage so downstream never has to know
|
||||||
|
// which endpoint a number came from.
|
||||||
|
const rawUsage = numOrNull(r.pitch_per);
|
||||||
|
const usage = rawUsage == null ? null : (rawUsage <= 1 ? Math.round(rawUsage * 1000) / 10 : rawUsage);
|
||||||
|
const type = (r.pitch_type || '').trim();
|
||||||
|
if (type) {
|
||||||
|
entry.pitches.push({
|
||||||
|
type,
|
||||||
|
name: (r.pitch_type_name || '').trim() || null,
|
||||||
|
usage_pct: usage,
|
||||||
|
velo: numOrNull(r.avg_speed),
|
||||||
|
break_z_induced: numOrNull(r.pitcher_break_z_induced),
|
||||||
|
break_x: numOrNull(r.pitcher_break_x),
|
||||||
|
thrown: numOrNull(r.pitches_thrown),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
out.set(id, entry);
|
||||||
|
}
|
||||||
|
for (const e of out.values()) {
|
||||||
|
e.pitches.sort((a, b) => (b.usage_pct ?? 0) - (a.usage_pct ?? 0));
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* fetchSeason(season, opts) — pull all five feeds and return normalized indexes.
|
||||||
|
* Any single feed failing degrades to an EMPTY index for that feed (its metrics
|
||||||
|
* become absent) rather than failing the whole pull: partial mechanism data is
|
||||||
|
* honest, a failed night is not a reason to have none.
|
||||||
|
*/
|
||||||
|
async function fetchSeason(season = DEFAULT_SEASON, opts = {}) {
|
||||||
|
const get = async (name) => {
|
||||||
|
try {
|
||||||
|
const text = await fetchText(FEEDS[name](season), opts);
|
||||||
|
const rows = text ? parseCsv(text) : [];
|
||||||
|
return Array.isArray(rows) ? rows : [];
|
||||||
|
} catch (err) {
|
||||||
|
console.warn(`[statcast] feed ${name} failed:`, err.message);
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const [bd, bbb, pd, pbb, mv] = await Promise.all([
|
||||||
|
get('batter_discipline'), get('batter_batted_ball'),
|
||||||
|
get('pitcher_discipline'), get('pitcher_batted_ball'), get('pitch_movement'),
|
||||||
|
]);
|
||||||
|
|
||||||
|
return {
|
||||||
|
season,
|
||||||
|
batterDiscipline: indexBy(bd, BATTER_DISCIPLINE),
|
||||||
|
batterBattedBall: indexBy(bbb, BATTED_BALL),
|
||||||
|
pitcherDiscipline: indexBy(pd, PITCHER_DISCIPLINE),
|
||||||
|
pitcherBattedBall: indexBy(pbb, BATTED_BALL),
|
||||||
|
pitchMix: indexPitchMix(mv),
|
||||||
|
counts: {
|
||||||
|
batter_discipline: bd.length, batter_batted_ball: bbb.length,
|
||||||
|
pitcher_discipline: pd.length, pitcher_batted_ball: pbb.length, pitch_movement: mv.length,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
fetchSeason,
|
||||||
|
DEFAULT_SEASON,
|
||||||
|
FEEDS,
|
||||||
|
__internals: {
|
||||||
|
flipName, idOf, normalizeRow, indexBy, indexPitchMix,
|
||||||
|
BATTER_DISCIPLINE, BATTED_BALL, PITCHER_DISCIPLINE, fetchText,
|
||||||
|
},
|
||||||
|
};
|
||||||
@@ -0,0 +1,273 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* STATCAST AGGREGATE SERVICE (Layer 1) — derivation + join + store.
|
||||||
|
*
|
||||||
|
* The AGGREGATE DERIVATION and IDENTITY BRIDGE boxes of the mechanism-data
|
||||||
|
* pattern (docs/MECHANISM-DATA.md). Pure where it can be, injectable
|
||||||
|
* everywhere, so tests never touch the network or the database.
|
||||||
|
*
|
||||||
|
* BACKFILL and REFRESH are the SAME operation: a full re-pull of the ~1,300-row
|
||||||
|
* aggregate set, upserted on (sport, season, source_id). That makes the nightly
|
||||||
|
* job idempotent and self-healing — a missed night self-corrects on the next
|
||||||
|
* run, with no incremental "who played today" bookkeeping to drift out of sync.
|
||||||
|
* At this volume the simple thing is also the robust thing.
|
||||||
|
*
|
||||||
|
* HONESTY RULES:
|
||||||
|
* - A metric the feed did not carry is ABSENT (null), never 0.
|
||||||
|
* - A player below the minimum sample keeps his row but is marked
|
||||||
|
* `sufficient: false` — thin is not the same as missing, and Layer 2 must
|
||||||
|
* be able to tell them apart.
|
||||||
|
* - A player we cannot join to our own roster still stores (player_key null)
|
||||||
|
* and joins later. Storing him is not a claim about him.
|
||||||
|
* - `updated_at` is written every run. Freshness is a truth property; the
|
||||||
|
* staleness alarm reads it.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const { nameKey } = require('../utils/playerName');
|
||||||
|
const statcast = require('./adapters/statcastAdapter');
|
||||||
|
|
||||||
|
/** Minimum samples for a mechanism claim. Below these the row is stored but
|
||||||
|
* flagged insufficient — the explicit gate that prevents a 12-PA sample from
|
||||||
|
* being classified as confidently as a 400-PA one (the FLEX-at-weight-1.0
|
||||||
|
* failure, structurally prevented). */
|
||||||
|
const MIN_PA = Number(process.env.STATCAST_MIN_PA) || 50;
|
||||||
|
const MIN_IP = Number(process.env.STATCAST_MIN_IP) || 10;
|
||||||
|
|
||||||
|
const num = (v) => {
|
||||||
|
if (v == null || v === '') return null;
|
||||||
|
const n = typeof v === 'number' ? v : Number(v);
|
||||||
|
return Number.isFinite(n) ? n : null;
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* buildRows(season, feeds, opts) — PURE. Merges the five feed indexes into one
|
||||||
|
* row per player per role. No I/O.
|
||||||
|
*/
|
||||||
|
function buildRows(season, feeds, opts = {}) {
|
||||||
|
const sport = opts.sport || 'mlb';
|
||||||
|
const now = opts.now || new Date().toISOString();
|
||||||
|
const rows = [];
|
||||||
|
|
||||||
|
// ---- BATTERS -----------------------------------------------------------
|
||||||
|
const batterIds = new Set([...feeds.batterDiscipline.keys(), ...feeds.batterBattedBall.keys()]);
|
||||||
|
for (const id of batterIds) {
|
||||||
|
const d = feeds.batterDiscipline.get(id);
|
||||||
|
const b = feeds.batterBattedBall.get(id);
|
||||||
|
const m = { ...(b ? b.metrics : {}), ...(d ? d.metrics : {}) };
|
||||||
|
const name = (d && d.name) || (b && b.name) || null;
|
||||||
|
const pa = num(m.pa);
|
||||||
|
rows.push({
|
||||||
|
sport,
|
||||||
|
season,
|
||||||
|
source_id: id,
|
||||||
|
player_key: name ? nameKey(name) : null,
|
||||||
|
player_name: name,
|
||||||
|
role: 'batter',
|
||||||
|
throws: null,
|
||||||
|
bats: null, // roster join fills this; absent otherwise (never guessed)
|
||||||
|
sample_pa: pa,
|
||||||
|
sample_ip: null,
|
||||||
|
sample_bip: num(m.bip),
|
||||||
|
k_pct: num(m.k_pct),
|
||||||
|
bb_pct: num(m.bb_pct),
|
||||||
|
whiff_pct: num(m.whiff_pct),
|
||||||
|
swing_pct: num(m.swing_pct),
|
||||||
|
chase_pct: num(m.chase_pct),
|
||||||
|
barrel_pct: num(m.barrel_pct) ?? num(m.barrel_pct_bb),
|
||||||
|
hard_hit_pct: num(m.hard_hit_pct),
|
||||||
|
avg_exit_velo: num(m.avg_exit_velo),
|
||||||
|
max_exit_velo: num(m.max_exit_velo),
|
||||||
|
avg_launch_angle: num(m.avg_launch_angle),
|
||||||
|
sweet_spot_pct: num(m.sweet_spot_pct) ?? num(m.sweet_spot_pct_bb),
|
||||||
|
ev95_pct: num(m.ev95_pct),
|
||||||
|
arm_angle: null,
|
||||||
|
gb_pct: null,
|
||||||
|
fb_pct: null,
|
||||||
|
ld_pct: null,
|
||||||
|
pitch_mix: null,
|
||||||
|
metrics: m,
|
||||||
|
updated_at: now,
|
||||||
|
_sufficient: pa != null && pa >= MIN_PA,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- PITCHERS ----------------------------------------------------------
|
||||||
|
const pitcherIds = new Set([
|
||||||
|
...feeds.pitcherDiscipline.keys(),
|
||||||
|
...feeds.pitcherBattedBall.keys(),
|
||||||
|
...feeds.pitchMix.keys(),
|
||||||
|
]);
|
||||||
|
for (const id of pitcherIds) {
|
||||||
|
const d = feeds.pitcherDiscipline.get(id);
|
||||||
|
const b = feeds.pitcherBattedBall.get(id);
|
||||||
|
const mix = feeds.pitchMix.get(id);
|
||||||
|
const m = { ...(b ? b.metrics : {}), ...(d ? d.metrics : {}) };
|
||||||
|
const name = (d && d.name) || (mix && mix.name) || (b && b.name) || null;
|
||||||
|
const ip = num(m.ip);
|
||||||
|
rows.push({
|
||||||
|
sport,
|
||||||
|
season,
|
||||||
|
source_id: id,
|
||||||
|
player_key: name ? nameKey(name) : null,
|
||||||
|
player_name: name,
|
||||||
|
role: 'pitcher',
|
||||||
|
// Handedness comes free on the movement feed — the ONLY feed that carries
|
||||||
|
// it. Absent when a pitcher has thrown too few tracked pitches to appear.
|
||||||
|
throws: (mix && mix.throws) || null,
|
||||||
|
bats: null,
|
||||||
|
sample_pa: null,
|
||||||
|
sample_ip: ip,
|
||||||
|
sample_bip: num(m.bip),
|
||||||
|
k_pct: num(m.k_pct),
|
||||||
|
bb_pct: num(m.bb_pct),
|
||||||
|
whiff_pct: num(m.whiff_pct),
|
||||||
|
swing_pct: num(m.swing_pct),
|
||||||
|
chase_pct: num(m.chase_pct),
|
||||||
|
barrel_pct: num(m.barrel_pct_allowed),
|
||||||
|
hard_hit_pct: num(m.hard_hit_pct_allowed),
|
||||||
|
avg_exit_velo: num(m.avg_exit_velo),
|
||||||
|
max_exit_velo: num(m.max_exit_velo),
|
||||||
|
avg_launch_angle: num(m.avg_launch_angle),
|
||||||
|
sweet_spot_pct: num(m.sweet_spot_pct_bb),
|
||||||
|
ev95_pct: num(m.ev95_pct),
|
||||||
|
arm_angle: num(m.arm_angle),
|
||||||
|
gb_pct: num(m.gb_pct),
|
||||||
|
fb_pct: num(m.fb_pct),
|
||||||
|
ld_pct: num(m.ld_pct),
|
||||||
|
pitch_mix: mix && mix.pitches && mix.pitches.length ? mix.pitches : null,
|
||||||
|
metrics: m,
|
||||||
|
updated_at: now,
|
||||||
|
_sufficient: ip != null && ip >= MIN_IP,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return rows;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Strip the internal flag before the row reaches Postgres. */
|
||||||
|
function toDbRow(r) {
|
||||||
|
const { _sufficient, ...rest } = r;
|
||||||
|
return rest;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* refreshSeason(opts) — THE JOB. Backfill and nightly refresh are the same call.
|
||||||
|
*
|
||||||
|
* opts: { season, sport, supabase, fetchSeason, now, chunkSize }
|
||||||
|
* Returns a summary — never throws on a partial failure; a broken feed degrades
|
||||||
|
* to fewer rows, which is honest, rather than to a wrong number.
|
||||||
|
*/
|
||||||
|
async function refreshSeason(opts = {}) {
|
||||||
|
const season = opts.season || statcast.DEFAULT_SEASON;
|
||||||
|
const sport = opts.sport || 'mlb';
|
||||||
|
const started = opts.now || new Date().toISOString();
|
||||||
|
|
||||||
|
const fetchSeason = opts.fetchSeason || statcast.fetchSeason;
|
||||||
|
const feeds = await fetchSeason(season, opts);
|
||||||
|
const rows = buildRows(season, feeds, { sport, now: started });
|
||||||
|
|
||||||
|
const summary = {
|
||||||
|
ok: true,
|
||||||
|
sport,
|
||||||
|
season,
|
||||||
|
started_at: started,
|
||||||
|
feed_counts: feeds.counts || null,
|
||||||
|
rows: rows.length,
|
||||||
|
batters: rows.filter((r) => r.role === 'batter').length,
|
||||||
|
pitchers: rows.filter((r) => r.role === 'pitcher').length,
|
||||||
|
sufficient: rows.filter((r) => r._sufficient).length,
|
||||||
|
thin: rows.filter((r) => !r._sufficient).length,
|
||||||
|
joined: rows.filter((r) => r.player_key).length,
|
||||||
|
unjoined: rows.filter((r) => !r.player_key).length,
|
||||||
|
with_handedness: rows.filter((r) => r.throws).length,
|
||||||
|
written: 0,
|
||||||
|
};
|
||||||
|
|
||||||
|
if (!rows.length) {
|
||||||
|
// Every feed empty = the source failed, NOT "there is no mechanism data".
|
||||||
|
// Refuse to write, so a bad night can never blank a good table.
|
||||||
|
summary.ok = false;
|
||||||
|
summary.reason = 'no rows from source — refusing to write';
|
||||||
|
return summary;
|
||||||
|
}
|
||||||
|
|
||||||
|
const sb = opts.supabase || getServiceClient();
|
||||||
|
if (!sb) {
|
||||||
|
summary.ok = false;
|
||||||
|
summary.reason = 'no supabase client (env not configured)';
|
||||||
|
return summary;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Chunked upsert — idempotent on the natural key, so re-running is a no-op
|
||||||
|
// beyond refreshing values and `updated_at`.
|
||||||
|
const chunk = opts.chunkSize || 500;
|
||||||
|
for (let i = 0; i < rows.length; i += chunk) {
|
||||||
|
const batch = rows.slice(i, i + chunk).map(toDbRow);
|
||||||
|
const { error } = await sb
|
||||||
|
.from('statcast_aggregates')
|
||||||
|
.upsert(batch, { onConflict: 'sport,season,source_id' });
|
||||||
|
if (error) {
|
||||||
|
summary.ok = false;
|
||||||
|
summary.reason = `upsert failed: ${error.message}`;
|
||||||
|
return summary;
|
||||||
|
}
|
||||||
|
summary.written += batch.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
return summary;
|
||||||
|
}
|
||||||
|
|
||||||
|
function getServiceClient() {
|
||||||
|
try {
|
||||||
|
return require('../utils/supabase').getSupabaseServiceClient();
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* getFreshness(opts) — how current is the mechanism tier?
|
||||||
|
* Reads the newest `updated_at`. Used by the staleness alarm and the status
|
||||||
|
* probe. Returns { updated_at, age_hours, rows } or nulls — never a guess.
|
||||||
|
*/
|
||||||
|
async function getFreshness(opts = {}) {
|
||||||
|
const sb = opts.supabase || getServiceClient();
|
||||||
|
if (!sb) return { updated_at: null, age_hours: null, rows: null };
|
||||||
|
const sport = opts.sport || 'mlb';
|
||||||
|
const { data, error } = await sb
|
||||||
|
.from('statcast_aggregates')
|
||||||
|
.select('updated_at')
|
||||||
|
.eq('sport', sport)
|
||||||
|
.order('updated_at', { ascending: false })
|
||||||
|
.limit(1);
|
||||||
|
if (error || !data || !data.length) return { updated_at: null, age_hours: null, rows: null };
|
||||||
|
const updated = data[0].updated_at;
|
||||||
|
const nowMs = opts.nowMs || Date.now();
|
||||||
|
const ageH = (nowMs - new Date(updated).getTime()) / 3_600_000;
|
||||||
|
return { updated_at: updated, age_hours: Math.round(ageH * 10) / 10 };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* isStale(freshness, maxAgeHours) — PURE. The alarm predicate.
|
||||||
|
*
|
||||||
|
* NEVER-REFRESHED (updated_at null) is NOT stale — it is "not built yet", a
|
||||||
|
* different condition with a different fix. Conflating them would page on a
|
||||||
|
* fresh install and teach the operator to ignore the alarm.
|
||||||
|
*/
|
||||||
|
const MAX_AGE_HOURS = Number(process.env.STATCAST_MAX_AGE_HOURS) || 48;
|
||||||
|
function isStale(freshness, maxAgeHours = MAX_AGE_HOURS) {
|
||||||
|
if (!freshness || freshness.updated_at == null) return false;
|
||||||
|
return Number.isFinite(freshness.age_hours) && freshness.age_hours > maxAgeHours;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
refreshSeason,
|
||||||
|
buildRows,
|
||||||
|
getFreshness,
|
||||||
|
isStale,
|
||||||
|
MIN_PA,
|
||||||
|
MIN_IP,
|
||||||
|
MAX_AGE_HOURS,
|
||||||
|
__internals: { toDbRow, num, getServiceClient },
|
||||||
|
};
|
||||||
@@ -286,6 +286,34 @@ function startSnapshotScheduler(opts = {}) {
|
|||||||
}
|
}
|
||||||
} catch (e) { console.warn('[harness] nightly run failed:', e.message); }
|
} catch (e) { console.warn('[harness] nightly run failed:', e.message); }
|
||||||
|
|
||||||
|
// Session 68 — LAYER 1 MECHANISM DATA. Nightly full re-pull of the
|
||||||
|
// ~1,350-row Statcast aggregate set at STATCAST_HOUR_UTC. Backfill and
|
||||||
|
// refresh are the SAME call, upserted on the natural key, so the job is
|
||||||
|
// idempotent and self-healing: a missed night self-corrects on the next
|
||||||
|
// run with no incremental bookkeeping to drift. Kill switch: STATCAST=0.
|
||||||
|
try {
|
||||||
|
const statcastHour = Number(process.env.STATCAST_HOUR_UTC || 11); // ~7 AM ET, after every game is final
|
||||||
|
if (h === statcastHour && process.env.STATCAST !== '0') {
|
||||||
|
const agg = opts.statcastService || require('./services/statcastAggregateService');
|
||||||
|
const res = await agg.refreshSeason({});
|
||||||
|
console.log(`[statcast] nightly refresh — ok=${res.ok} rows=${res.rows ?? 'n/a'} written=${res.written ?? 0} joined=${res.joined ?? 'n/a'} thin=${res.thin ?? 'n/a'}${res.reason ? ` reason: ${res.reason}` : ''}`);
|
||||||
|
if (!res.ok) {
|
||||||
|
await notify(`Statcast refresh failed — ${res.reason || 'unknown'}. Mechanism data is not updating.`, {
|
||||||
|
title: 'VYNDR statcast', priority: 'high', tags: ['rotating_light'],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
// Serving a stale aggregate as current is a quiet fabrication of
|
||||||
|
// currency, so staleness pages on its own — separate from a failed run,
|
||||||
|
// because a silently-not-scheduled job never produces a failure.
|
||||||
|
const fresh = await agg.getFreshness({});
|
||||||
|
if (agg.isStale(fresh)) {
|
||||||
|
await notify(`Statcast aggregates stale — last updated ${fresh.updated_at} (${fresh.age_hours}h). Mechanism data is being served as current.`, {
|
||||||
|
title: 'VYNDR statcast', priority: 'high', tags: ['rotating_light'],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (e) { console.warn('[statcast] nightly refresh failed:', e.message); }
|
||||||
|
|
||||||
// Session 8 — quota check after each snapshot run: odds-api >= 80% alerts
|
// Session 8 — quota check after each snapshot run: odds-api >= 80% alerts
|
||||||
// once per day (Redis-deduped). Never throws (guarded inside checkQuotaDaily).
|
// once per day (Redis-deduped). Never throws (guarded inside checkQuotaDaily).
|
||||||
await opsWatch.checkQuotaDaily({ getStatus: getQuotaStatus, cacheGet, cacheSet, notify, now: () => now() });
|
await opsWatch.checkQuotaDaily({ getStatus: getQuotaStatus, cacheGet, cacheSet, notify, now: () => now() });
|
||||||
@@ -324,7 +352,7 @@ function startSnapshotScheduler(opts = {}) {
|
|||||||
|
|
||||||
const interval = setInterval(() => { void tick(); void refreshTick(); }, 60_000);
|
const interval = setInterval(() => { void tick(); void refreshTick(); }, 60_000);
|
||||||
if (interval.unref) interval.unref();
|
if (interval.unref) interval.unref();
|
||||||
console.log(`[snapshotScheduler] armed — SNAPSHOT_CRON=${process.env.SNAPSHOT_CRON}, hours=${HOURS_UTC.join(',')} UTC, intraday=${process.env.INTRADAY_REFRESH === '0' ? 'off' : `${REFRESH_MINUTES}m (slate hours)`}`);
|
console.log(`[snapshotScheduler] armed — SNAPSHOT_CRON=${process.env.SNAPSHOT_CRON}, hours=${HOURS_UTC.join(',')} UTC, intraday=${process.env.INTRADAY_REFRESH === '0' ? 'off' : `${REFRESH_MINUTES}m (slate hours)`}, statcast=${process.env.STATCAST === '0' ? 'off' : `${process.env.STATCAST_HOUR_UTC || 11}h UTC`}`);
|
||||||
// Job 1 — state each sport's cadence at boot so "when does WNBA grade?" is
|
// Job 1 — state each sport's cadence at boot so "when does WNBA grade?" is
|
||||||
// answerable from logs, and warn if a sport is scheduled at an hour the
|
// answerable from logs, and warn if a sport is scheduled at an hour the
|
||||||
// scheduler never fires (it would silently never run).
|
// scheduler never fires (it would silently never run).
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
-- Layer 1 — MECHANISM DATA (Session 68).
|
||||||
|
-- Per-player-per-season Statcast aggregates. The HOT tier of the two-tier
|
||||||
|
-- mechanism pattern: small, indexed, joined at classification time. Raw
|
||||||
|
-- per-pitch stays out of Postgres on purpose (one season ~0.85 GB vs this
|
||||||
|
-- table's ~2,300 rows) — see docs/MECHANISM-DATA.md.
|
||||||
|
--
|
||||||
|
-- The table is SPORT-SCOPED from day one: NBA tracking and NFL Next Gen land
|
||||||
|
-- in the same shape with a different `sport` and their own source ids.
|
||||||
|
|
||||||
|
create table if not exists statcast_aggregates (
|
||||||
|
sport text not null default 'mlb',
|
||||||
|
season int not null,
|
||||||
|
-- Source-native player id (MLB: MLBAM). The join key — an integer both
|
||||||
|
-- systems use natively, which is why the measured match rate is 100%.
|
||||||
|
source_id bigint not null,
|
||||||
|
-- Our canonical key. Nullable ON PURPOSE: a player Statcast has but we have
|
||||||
|
-- never graded still gets stored, and joins later.
|
||||||
|
player_key text,
|
||||||
|
player_name text,
|
||||||
|
role text not null check (role in ('batter','pitcher')),
|
||||||
|
-- Handedness: pitchers from the movement feed, batters from the roster join.
|
||||||
|
-- NULL = honest-absent, never guessed.
|
||||||
|
throws text,
|
||||||
|
bats text,
|
||||||
|
|
||||||
|
-- Sample size — the gate for every downstream claim.
|
||||||
|
sample_pa numeric,
|
||||||
|
sample_ip numeric,
|
||||||
|
sample_bip numeric,
|
||||||
|
|
||||||
|
-- Plate discipline / contact quality (shared vocabulary, both roles).
|
||||||
|
k_pct numeric,
|
||||||
|
bb_pct numeric,
|
||||||
|
whiff_pct numeric,
|
||||||
|
swing_pct numeric,
|
||||||
|
chase_pct numeric,
|
||||||
|
barrel_pct numeric,
|
||||||
|
hard_hit_pct numeric,
|
||||||
|
|
||||||
|
-- Batter mechanism.
|
||||||
|
avg_exit_velo numeric,
|
||||||
|
max_exit_velo numeric,
|
||||||
|
avg_launch_angle numeric,
|
||||||
|
sweet_spot_pct numeric,
|
||||||
|
ev95_pct numeric,
|
||||||
|
|
||||||
|
-- Pitcher mechanism.
|
||||||
|
arm_angle numeric,
|
||||||
|
gb_pct numeric,
|
||||||
|
fb_pct numeric,
|
||||||
|
ld_pct numeric,
|
||||||
|
pitch_mix jsonb,
|
||||||
|
|
||||||
|
-- Everything else the feeds carry, verbatim. Layer 2/3 can reach a field we
|
||||||
|
-- did not promote to a column WITHOUT a re-ingest.
|
||||||
|
metrics jsonb not null default '{}'::jsonb,
|
||||||
|
|
||||||
|
-- Freshness is a TRUTH property: the staleness alarm reads this, and no
|
||||||
|
-- surface may present a stale aggregate as current.
|
||||||
|
updated_at timestamptz not null default now(),
|
||||||
|
source text not null default 'baseball_savant',
|
||||||
|
|
||||||
|
primary key (sport, season, source_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
create index if not exists statcast_agg_player_key_idx on statcast_aggregates (sport, season, player_key);
|
||||||
|
create index if not exists statcast_agg_role_idx on statcast_aggregates (sport, season, role);
|
||||||
|
create index if not exists statcast_agg_updated_idx on statcast_aggregates (updated_at desc);
|
||||||
|
|
||||||
|
alter table statcast_aggregates enable row level security;
|
||||||
|
|
||||||
|
-- Commodity public data: readable by anyone, written ONLY by the service role
|
||||||
|
-- (the pipeline). Same posture as the public ledger rows.
|
||||||
|
drop policy if exists statcast_agg_read on statcast_aggregates;
|
||||||
|
create policy statcast_agg_read on statcast_aggregates for select using (true);
|
||||||
@@ -0,0 +1,202 @@
|
|||||||
|
/* ============================================================
|
||||||
|
Session 68 — LAYER 1: mechanism-data ingestion.
|
||||||
|
|
||||||
|
Pure derivation + the honesty rules. No network, no database: the adapter's
|
||||||
|
fetch and the Supabase client are both injected.
|
||||||
|
============================================================ */
|
||||||
|
|
||||||
|
const svc = require('../../src/services/statcastAggregateService');
|
||||||
|
const adapter = require('../../src/services/adapters/statcastAdapter');
|
||||||
|
|
||||||
|
const { flipName, indexBy, indexPitchMix, BATTER_DISCIPLINE } = adapter.__internals;
|
||||||
|
|
||||||
|
/** Minimal feed indexes, shaped exactly like adapter.fetchSeason() output. */
|
||||||
|
function feeds({ batter = [], batterBB = [], pitcher = [], pitcherBB = [], mix = [] } = {}) {
|
||||||
|
return {
|
||||||
|
batterDiscipline: indexBy(batter, BATTER_DISCIPLINE),
|
||||||
|
batterBattedBall: indexBy(batterBB, adapter.__internals.BATTED_BALL),
|
||||||
|
pitcherDiscipline: indexBy(pitcher, adapter.__internals.PITCHER_DISCIPLINE),
|
||||||
|
pitcherBattedBall: indexBy(pitcherBB, adapter.__internals.BATTED_BALL),
|
||||||
|
pitchMix: indexPitchMix(mix),
|
||||||
|
counts: {},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const BELL = {
|
||||||
|
'last_name, first_name': 'Bell, Josh', player_id: '605137', pa: '387',
|
||||||
|
k_percent: '21.7', bb_percent: '7.5', whiff_percent: '24.8', swing_percent: '51',
|
||||||
|
oz_swing_percent: '30.6', barrel_batted_rate: '10.3', hard_hit_percent: '43.4',
|
||||||
|
};
|
||||||
|
const SKUBAL = {
|
||||||
|
'last_name, first_name': 'Skubal, Tarik', player_id: '669373', p_formatted_ip: '82.2',
|
||||||
|
k_percent: '30.5', whiff_percent: '32.1', groundballs_percent: '49', arm_angle: '46.9',
|
||||||
|
};
|
||||||
|
const SKUBAL_MIX = {
|
||||||
|
'last_name, first_name': 'Skubal, Tarik', pitcher_id: '669373', pitch_hand: 'L',
|
||||||
|
pitch_type: 'FF', pitch_type_name: '4-Seam Fastball', pitch_per: '0.371',
|
||||||
|
avg_speed: '96.7', pitcher_break_z_induced: '17.1', pitcher_break_x: '3.6', pitches_thrown: '463',
|
||||||
|
};
|
||||||
|
|
||||||
|
describe('adapter normalisation', () => {
|
||||||
|
it('flips Savant "Last, First" into a real name', () => {
|
||||||
|
expect(flipName('Bell, Josh')).toBe('Josh Bell');
|
||||||
|
expect(flipName('Guerrero Jr., Vladimir')).toBe('Vladimir Guerrero Jr.');
|
||||||
|
expect(flipName('Ohtani')).toBe('Ohtani');
|
||||||
|
expect(flipName('')).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('normalises pitch usage from a FRACTION to a percentage', () => {
|
||||||
|
const [entry] = [...indexPitchMix([SKUBAL_MIX]).values()];
|
||||||
|
expect(entry.pitches[0].usage_pct).toBe(37.1); // not 0.371
|
||||||
|
expect(entry.throws).toBe('L');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('a missing metric is ABSENT, never 0', () => {
|
||||||
|
const [entry] = [...indexBy([{ 'last_name, first_name': 'X, Y', player_id: '1' }], BATTER_DISCIPLINE).values()];
|
||||||
|
expect(entry.metrics.k_pct).toBeUndefined();
|
||||||
|
expect(entry.metrics.whiff_pct).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('buildRows — derivation', () => {
|
||||||
|
const rows = () => svc.buildRows(2026, feeds({
|
||||||
|
batter: [BELL], pitcher: [SKUBAL], mix: [SKUBAL_MIX],
|
||||||
|
}), { now: '2026-07-21T00:00:00.000Z' });
|
||||||
|
|
||||||
|
it('produces one row per player per role, keyed by the source id', () => {
|
||||||
|
const r = rows();
|
||||||
|
expect(r).toHaveLength(2);
|
||||||
|
expect(r.find((x) => x.role === 'batter').source_id).toBe(605137);
|
||||||
|
expect(r.find((x) => x.role === 'pitcher').source_id).toBe(669373);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('joins to our canonical player_key', () => {
|
||||||
|
const b = rows().find((x) => x.role === 'batter');
|
||||||
|
expect(b.player_name).toBe('Josh Bell');
|
||||||
|
expect(b.player_key).toBe(require('../../src/utils/playerName').nameKey('Josh Bell'));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('carries pitcher handedness from the movement feed', () => {
|
||||||
|
expect(rows().find((x) => x.role === 'pitcher').throws).toBe('L');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('leaves batter handedness ABSENT (roster join, never guessed)', () => {
|
||||||
|
expect(rows().find((x) => x.role === 'batter').bats).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('flags thin samples rather than dropping or inflating them', () => {
|
||||||
|
const thin = svc.buildRows(2026, feeds({
|
||||||
|
batter: [{ ...BELL, pa: '12' }], pitcher: [{ ...SKUBAL, p_formatted_ip: '2.1' }],
|
||||||
|
}), {});
|
||||||
|
expect(thin.every((r) => r._sufficient === false)).toBe(true);
|
||||||
|
// Thin is STORED, not dropped — "thin" and "missing" are different claims.
|
||||||
|
expect(thin).toHaveLength(2);
|
||||||
|
expect(thin[0].sample_pa).toBe(12);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('a player with no sample at all is insufficient, not zero', () => {
|
||||||
|
const [r] = svc.buildRows(2026, feeds({ batter: [{ 'last_name, first_name': 'A, B', player_id: '9' }] }), {});
|
||||||
|
expect(r.sample_pa).toBeNull(); // NOT 0
|
||||||
|
expect(r._sufficient).toBe(false);
|
||||||
|
expect(r.k_pct).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps every raw metric in `metrics` so Layer 2/3 never needs a re-ingest', () => {
|
||||||
|
const b = rows().find((x) => x.role === 'batter');
|
||||||
|
expect(b.metrics.chase_pct).toBe(30.6);
|
||||||
|
expect(b.chase_pct).toBe(30.6);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('refreshSeason — the job', () => {
|
||||||
|
const okClient = () => {
|
||||||
|
const calls = [];
|
||||||
|
return {
|
||||||
|
calls,
|
||||||
|
from: () => ({ upsert: async (batch) => { calls.push(batch); return { error: null }; } }),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
it('backfill and refresh are the SAME idempotent call', async () => {
|
||||||
|
const fetchSeason = async () => feeds({ batter: [BELL], pitcher: [SKUBAL], mix: [SKUBAL_MIX] });
|
||||||
|
const a = okClient(); const b = okClient();
|
||||||
|
const r1 = await svc.refreshSeason({ supabase: a, fetchSeason, now: '2026-07-21T00:00:00.000Z' });
|
||||||
|
const r2 = await svc.refreshSeason({ supabase: b, fetchSeason, now: '2026-07-21T00:00:00.000Z' });
|
||||||
|
expect(r1.ok).toBe(true);
|
||||||
|
expect(r1.written).toBe(2);
|
||||||
|
expect(JSON.stringify(a.calls)).toBe(JSON.stringify(b.calls)); // identical → idempotent
|
||||||
|
});
|
||||||
|
|
||||||
|
it('upserts on the natural key so a re-run never duplicates', async () => {
|
||||||
|
let opts = null;
|
||||||
|
const sb = { from: () => ({ upsert: async (_b, o) => { opts = o; return { error: null }; } }) };
|
||||||
|
await svc.refreshSeason({ supabase: sb, fetchSeason: async () => feeds({ batter: [BELL] }) });
|
||||||
|
expect(opts.onConflict).toBe('sport,season,source_id');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('REFUSES to write when every feed is empty — a bad night cannot blank a good table', async () => {
|
||||||
|
const sb = okClient();
|
||||||
|
const out = await svc.refreshSeason({ supabase: sb, fetchSeason: async () => feeds({}) });
|
||||||
|
expect(out.ok).toBe(false);
|
||||||
|
expect(out.reason).toMatch(/refusing to write/);
|
||||||
|
expect(sb.calls).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('strips the internal flag before it reaches Postgres', async () => {
|
||||||
|
const sb = okClient();
|
||||||
|
await svc.refreshSeason({ supabase: sb, fetchSeason: async () => feeds({ batter: [BELL] }) });
|
||||||
|
expect(sb.calls[0][0]._sufficient).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports the join rate — the honest-absent rate for the mechanism tier', async () => {
|
||||||
|
const out = await svc.refreshSeason({
|
||||||
|
supabase: okClient(),
|
||||||
|
fetchSeason: async () => feeds({ batter: [BELL, { player_id: '77', pa: '100' }] }),
|
||||||
|
});
|
||||||
|
expect(out.joined).toBe(1);
|
||||||
|
expect(out.unjoined).toBe(1); // no name → no key → stored, joins later
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('staleness — never serve stale as fresh', () => {
|
||||||
|
it('fires past the max age', () => {
|
||||||
|
expect(svc.isStale({ updated_at: '2026-07-01T00:00:00Z', age_hours: 72 })).toBe(true);
|
||||||
|
expect(svc.isStale({ updated_at: '2026-07-20T00:00:00Z', age_hours: 6 })).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('NEVER-BUILT is not STALE — different condition, different fix', () => {
|
||||||
|
// Paging on a fresh install teaches the operator to ignore the alarm.
|
||||||
|
expect(svc.isStale({ updated_at: null, age_hours: null })).toBe(false);
|
||||||
|
expect(svc.isStale(null)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('respects an explicit threshold', () => {
|
||||||
|
expect(svc.isStale({ updated_at: 'x', age_hours: 30 }, 24)).toBe(true);
|
||||||
|
expect(svc.isStale({ updated_at: 'x', age_hours: 30 }, 48)).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('scheduler + route wiring', () => {
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const read = (r) => fs.readFileSync(path.join(__dirname, '..', '..', r), 'utf8');
|
||||||
|
|
||||||
|
it('is server-scheduled with a kill switch', () => {
|
||||||
|
const s = read('src/snapshotScheduler.js');
|
||||||
|
expect(s).toContain('STATCAST_HOUR_UTC');
|
||||||
|
expect(s).toMatch(/process\.env\.STATCAST !== '0'/);
|
||||||
|
expect(s).toContain('refreshSeason');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('pages on BOTH a failed run and silent staleness', () => {
|
||||||
|
const s = read('src/snapshotScheduler.js');
|
||||||
|
expect(s).toMatch(/Statcast refresh failed/);
|
||||||
|
expect(s).toMatch(/Statcast aggregates stale/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is induce-able on demand (never wait on the cron)', () => {
|
||||||
|
const s = read('src/routes/internal.js');
|
||||||
|
expect(s).toContain("router.post('/statcast/refresh'");
|
||||||
|
expect(s).toContain("router.get('/statcast/status'");
|
||||||
|
});
|
||||||
|
});
|
||||||
+1
-1
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user