Mirror statcast_history on its source so retention cannot drift
The first production run of the point-in-time retention failed with "Could not find the 'swing_pct' column of 'statcast_history'" -- the hand-enumerated column list had already drifted from the table it was copying. The refresh itself still succeeded and wrote all 1,387 aggregate rows, which verified the best-effort guard in prod: a retention failure does not fail the refresh. The table is now created with LIKE statcast_aggregates, and the writer passes the row through whole instead of hand-stripping columns, so there is no drift surface left. Recreating was safe -- nothing had been retained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -272,14 +272,13 @@ async function refreshSeason(opts = {}) {
|
|||||||
const asOf = (opts.asOfDate || started).slice(0, 10);
|
const asOf = (opts.asOfDate || started).slice(0, 10);
|
||||||
let retained = 0;
|
let retained = 0;
|
||||||
for (let i = 0; i < rows.length; i += chunk) {
|
for (let i = 0; i < rows.length; i += chunk) {
|
||||||
const batch = rows.slice(i, i + chunk).map((r) => {
|
// The history table MIRRORS statcast_aggregates (created via LIKE), so the
|
||||||
const row = toDbRow(r);
|
// row goes in whole. The first prod attempt hand-stripped columns against a
|
||||||
delete row.updated_at; delete row.source;
|
// hand-enumerated schema and failed on the first one that had drifted
|
||||||
delete row.position; delete row.role_detail;
|
// ('swing_pct'); passing the same shape through removes the drift surface
|
||||||
delete row.games_started; delete row.games_pitched;
|
// entirely. `updated_at` rides along as the refresh timestamp; `as_of_date`
|
||||||
delete row.saves; delete row.holds;
|
// is the day the snapshot describes and is what point-in-time queries use.
|
||||||
return { ...row, as_of_date: asOf };
|
const batch = rows.slice(i, i + chunk).map((r) => ({ ...toDbRow(r), as_of_date: asOf }));
|
||||||
});
|
|
||||||
const { error } = await sb.from('statcast_history')
|
const { error } = await sb.from('statcast_history')
|
||||||
.upsert(batch, { onConflict: 'as_of_date,sport,season,source_id,role' });
|
.upsert(batch, { onConflict: 'as_of_date,sport,season,source_id,role' });
|
||||||
if (error) { summary.history_error = error.message; break; }
|
if (error) { summary.history_error = error.message; break; }
|
||||||
|
|||||||
Reference in New Issue
Block a user