Mirror statcast_history on its source so retention cannot drift
The first production run of the point-in-time retention failed with "Could not find the 'swing_pct' column of 'statcast_history'" -- the hand-enumerated column list had already drifted from the table it was copying. The refresh itself still succeeded and wrote all 1,387 aggregate rows, which verified the best-effort guard in prod: a retention failure does not fail the refresh. The table is now created with LIKE statcast_aggregates, and the writer passes the row through whole instead of hand-stripping columns, so there is no drift surface left. Recreating was safe -- nothing had been retained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -272,14 +272,13 @@ async function refreshSeason(opts = {}) {
|
||||
const asOf = (opts.asOfDate || started).slice(0, 10);
|
||||
let retained = 0;
|
||||
for (let i = 0; i < rows.length; i += chunk) {
|
||||
const batch = rows.slice(i, i + chunk).map((r) => {
|
||||
const row = toDbRow(r);
|
||||
delete row.updated_at; delete row.source;
|
||||
delete row.position; delete row.role_detail;
|
||||
delete row.games_started; delete row.games_pitched;
|
||||
delete row.saves; delete row.holds;
|
||||
return { ...row, as_of_date: asOf };
|
||||
});
|
||||
// The history table MIRRORS statcast_aggregates (created via LIKE), so the
|
||||
// row goes in whole. The first prod attempt hand-stripped columns against a
|
||||
// hand-enumerated schema and failed on the first one that had drifted
|
||||
// ('swing_pct'); passing the same shape through removes the drift surface
|
||||
// entirely. `updated_at` rides along as the refresh timestamp; `as_of_date`
|
||||
// is the day the snapshot describes and is what point-in-time queries use.
|
||||
const batch = rows.slice(i, i + chunk).map((r) => ({ ...toDbRow(r), as_of_date: asOf }));
|
||||
const { error } = await sb.from('statcast_history')
|
||||
.upsert(batch, { onConflict: 'as_of_date,sport,season,source_id,role' });
|
||||
if (error) { summary.history_error = error.message; break; }
|
||||
|
||||
Reference in New Issue
Block a user