Mirror statcast_history on its source so retention cannot drift

The first production run of the point-in-time retention failed with
"Could not find the 'swing_pct' column of 'statcast_history'" -- the
hand-enumerated column list had already drifted from the table it was copying.
The refresh itself still succeeded and wrote all 1,387 aggregate rows, which
verified the best-effort guard in prod: a retention failure does not fail the
refresh.

The table is now created with LIKE statcast_aggregates, and the writer passes
the row through whole instead of hand-stripping columns, so there is no drift
surface left. Recreating was safe -- nothing had been retained.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-03 16:49:15 -04:00
parent 4aab18096f
commit c2d6e8ee7d
+7 -8
View File
@@ -272,14 +272,13 @@ async function refreshSeason(opts = {}) {
const asOf = (opts.asOfDate || started).slice(0, 10);
let retained = 0;
for (let i = 0; i < rows.length; i += chunk) {
const batch = rows.slice(i, i + chunk).map((r) => {
const row = toDbRow(r);
delete row.updated_at; delete row.source;
delete row.position; delete row.role_detail;
delete row.games_started; delete row.games_pitched;
delete row.saves; delete row.holds;
return { ...row, as_of_date: asOf };
});
// The history table MIRRORS statcast_aggregates (created via LIKE), so the
// row goes in whole. The first prod attempt hand-stripped columns against a
// hand-enumerated schema and failed on the first one that had drifted
// ('swing_pct'); passing the same shape through removes the drift surface
// entirely. `updated_at` rides along as the refresh timestamp; `as_of_date`
// is the day the snapshot describes and is what point-in-time queries use.
const batch = rows.slice(i, i + chunk).map((r) => ({ ...toDbRow(r), as_of_date: asOf }));
const { error } = await sb.from('statcast_history')
.upsert(batch, { onConflict: 'as_of_date,sport,season,source_id,role' });
if (error) { summary.history_error = error.message; break; }