Settlement has been dead since 2026-08-01: a 500-id filter overflowed the URL

The self-learning loop stopped two days ago and reported success the whole
time. 1,444 ledger rows from 2026-08-01 sit unsettled with settle_attempts=0
-- never even attempted -- and every accruing challenger has been starved of
settled sample as a result.

ROOT CAUSE. settleLedger fetched open ids, then REFETCHED the full rows with
.in('id', ids). PostgREST puts filters in the URL, so 500 UUIDs became an
18,499-character request that the fetch layer rejects with "TypeError: fetch
failed". The result was destructured as `const { data: rows } = ...` with NO
error binding, so rows came back null, the loop body never executed, and the
function returned {settled:0, voided:0, unrecoverable:0, pending:0} --
byte-identical to a clean "nothing to settle". Reproduced against prod before
changing anything.

WHY IT HID FOR TWO DAYS. It is volume-triggered. Daily volume ran 20-260 rows
and settled perfectly for weeks; 2026-08-01 was the first day past the 500-row
fetch limit. And the zero-settle ops alarm reads these very return values, so
pending:0 told the watchdog the backlog was empty -- the alarm built to catch
exactly this could not see it.

THE FIX. The refetch existed only to add game_date/settle_attempts/
dclv_computed_at. Selecting them in the first query removes the id list
entirely, so there is no URL to overflow at any volume. A failed fetch now
surfaces its error instead of being reported as an empty backlog.

captureClosing carried the same shape one level down -- .in('id', g.ids) on an
UPDATE, which fails identically once a single line|odds group gets large on a
big slate. Its id filters are now chunked at 100 (~3.7 KB).

Tests: the regression is locked by asserting settlement issues NO id-list
filter at 500 rows, and that a failed fetch is never reported as an empty
backlog -- the two properties that would have caught this. Two existing
suites asserted the old two-query shape and were updated to the real one.

4,159 tests green (332 suites); web build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-02 21:49:30 -04:00
parent 2394fb04a1
commit b06a84af80
4 changed files with 176 additions and 19 deletions
+54 -12
View File
@@ -37,6 +37,16 @@ const { LEDGER_TAKEABLE_FLOOR, isLedgerTakeable: takeableFor } = require('../con
const CONFLICT = 'user_id,player_key,stat,line,side,game_id';
const UPSERT_CHUNK = 200;
const SETTLE_FETCH_LIMIT = 500;
/**
* Max ids in a single `.in('id', …)` filter.
*
* PostgREST puts filters in the URL, so an id list is bounded by URL length, not
* by row count: 500 UUIDs is ~18.5 KB and the fetch layer rejects it outright
* with `TypeError: fetch failed`. 100 keeps it near 3.7 KB, comfortably inside
* every proxy default. This is not a tuning knob — it is the guard for the
* defect that silently killed settlement for two days (see settleLedger).
*/
const ID_FILTER_CHUNK = 100;
// Session 64 — bounded retry: a row that cannot be resolved after this many
// date-targeted attempts becomes 'unrecoverable' rather than pending forever.
const SETTLE_ATTEMPT_CAP = Number(process.env.SETTLE_ATTEMPT_CAP || 4);
@@ -520,10 +530,23 @@ async function captureClosing(sport, oddsProps, opts = {}) {
groups.get(gk).ids.push(row.id);
}
for (const g of groups.values()) {
const { error: upErr } = await sb.from('ledger_entries')
.update({ closing_line: g.line, closing_odds: g.odds })
.in('id', g.ids);
if (!upErr) updated += g.ids.length;
// CHUNKED, for the same reason settleLedger no longer refetches by id: an
// `.in('id', …)` filter travels in the URL, and a few hundred UUIDs exceed
// what the fetch layer will send (500 ids ≈ 18.5 KB). That is the defect
// that silently killed settlement on 2026-08-01; this is the same shape,
// one prop-price group deep, and it fails the same way once a single
// line|odds pair collects enough rows on a big slate.
for (let i = 0; i < g.ids.length; i += ID_FILTER_CHUNK) {
const idChunk = g.ids.slice(i, i + ID_FILTER_CHUNK);
const { error: upErr } = await sb.from('ledger_entries')
.update({ closing_line: g.line, closing_odds: g.odds })
.in('id', idChunk);
if (upErr) {
console.warn(`[ledger] closing capture chunk failed (${idChunk.length} rows): ${upErr.message}`);
continue;
}
updated += idChunk.length;
}
}
return { updated };
}
@@ -614,20 +637,39 @@ async function settleLedger(sport, opts = {}) {
const getSchedule = opts.getSchedule || getScheduleFn;
const cutoff = opts.beforeDate || todayET(); // settle strictly-before today
const { data: open, error } = await sb.from('ledger_entries')
.select('id, player_key, player_name, stat, line, side, closing_line')
// ONE query, selecting everything the settle loop needs.
//
// THIS USED TO BE TWO QUERIES, and the second one silently killed the entire
// self-learning loop (found 2026-08-03). It refetched by primary key —
// `.in('id', open.map(r => r.id))` — which PostgREST sends as a GET query
// string: 500 UUIDs is an 18,499-character URL, and the fetch layer rejects it
// with `TypeError: fetch failed`. The result was destructured as
// `const { data: rows } = ...` with NO error binding, so `rows` came back null,
// the loop body never executed, and settleLedger returned
// `{settled:0, voided:0, unrecoverable:0, pending:0}` — BYTE-IDENTICAL to a
// clean "nothing to settle".
//
// It stayed invisible because it is volume-triggered: daily volume ran 20260
// rows and settled perfectly for weeks. 2026-08-01 was the first day over the
// 500-row fetch limit, and settlement died that night — 1,444 rows with
// `settle_attempts = 0`, never even attempted. Worse, the zero-settle ops alarm
// reads THESE return values, so `pending: 0` told the watchdog there was
// nothing pending and nobody was paged.
//
// The refetch existed only to add game_date/settle_attempts/dclv_computed_at.
// Selecting them up front removes the URL entirely — there is no id list to
// send at any volume.
const { data: rows, error } = await sb.from('ledger_entries')
.select('id, player_key, player_name, stat, line, side, closing_line, game_date, settle_attempts, dclv_computed_at')
.eq('sport', sp)
.is('outcome', null)
.lt('game_date', cutoff)
.order('game_date', { ascending: true })
.limit(SETTLE_FETCH_LIMIT);
// An errored fetch is NOT an empty backlog. Returning zeros here is what made
// a dead loop look like a healthy one for two days — surface it instead.
if (error) return { settled: 0, pending: 0, error: error.message };
if (!open || open.length === 0) return { settled: 0, pending: 0 };
// We need each row's game_date for the log match — refetch with it included.
const { data: rows } = await sb.from('ledger_entries')
.select('id, player_name, stat, line, side, closing_line, game_date, settle_attempts, dclv_computed_at, player_key')
.in('id', open.map((r) => r.id));
if (!rows || rows.length === 0) return { settled: 0, pending: 0 };
// One game-log fetch per unique player.
const players = [...new Set((rows || []).map((r) => r.player_name))];