Settlement has been dead since 2026-08-01: a 500-id filter overflowed the URL
The self-learning loop stopped two days ago and reported success the whole
time. 1,444 ledger rows from 2026-08-01 sit unsettled with settle_attempts=0
-- never even attempted -- and every accruing challenger has been starved of
settled sample as a result.
ROOT CAUSE. settleLedger fetched open ids, then REFETCHED the full rows with
.in('id', ids). PostgREST puts filters in the URL, so 500 UUIDs became an
18,499-character request that the fetch layer rejects with "TypeError: fetch
failed". The result was destructured as `const { data: rows } = ...` with NO
error binding, so rows came back null, the loop body never executed, and the
function returned {settled:0, voided:0, unrecoverable:0, pending:0} --
byte-identical to a clean "nothing to settle". Reproduced against prod before
changing anything.
WHY IT HID FOR TWO DAYS. It is volume-triggered. Daily volume ran 20-260 rows
and settled perfectly for weeks; 2026-08-01 was the first day past the 500-row
fetch limit. And the zero-settle ops alarm reads these very return values, so
pending:0 told the watchdog the backlog was empty -- the alarm built to catch
exactly this could not see it.
THE FIX. The refetch existed only to add game_date/settle_attempts/
dclv_computed_at. Selecting them in the first query removes the id list
entirely, so there is no URL to overflow at any volume. A failed fetch now
surfaces its error instead of being reported as an empty backlog.
captureClosing carried the same shape one level down -- .in('id', g.ids) on an
UPDATE, which fails identically once a single line|odds group gets large on a
big slate. Its id filters are now chunked at 100 (~3.7 KB).
Tests: the regression is locked by asserting settlement issues NO id-list
filter at 500 rows, and that a failed fetch is never reported as an empty
backlog -- the two properties that would have caught this. Two existing
suites asserted the old two-query shape and were updated to the real one.
4,159 tests green (332 suites); web build exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -37,6 +37,16 @@ const { LEDGER_TAKEABLE_FLOOR, isLedgerTakeable: takeableFor } = require('../con
|
||||
const CONFLICT = 'user_id,player_key,stat,line,side,game_id';
|
||||
const UPSERT_CHUNK = 200;
|
||||
const SETTLE_FETCH_LIMIT = 500;
|
||||
/**
|
||||
* Max ids in a single `.in('id', …)` filter.
|
||||
*
|
||||
* PostgREST puts filters in the URL, so an id list is bounded by URL length, not
|
||||
* by row count: 500 UUIDs is ~18.5 KB and the fetch layer rejects it outright
|
||||
* with `TypeError: fetch failed`. 100 keeps it near 3.7 KB, comfortably inside
|
||||
* every proxy default. This is not a tuning knob — it is the guard for the
|
||||
* defect that silently killed settlement for two days (see settleLedger).
|
||||
*/
|
||||
const ID_FILTER_CHUNK = 100;
|
||||
// Session 64 — bounded retry: a row that cannot be resolved after this many
|
||||
// date-targeted attempts becomes 'unrecoverable' rather than pending forever.
|
||||
const SETTLE_ATTEMPT_CAP = Number(process.env.SETTLE_ATTEMPT_CAP || 4);
|
||||
@@ -520,10 +530,23 @@ async function captureClosing(sport, oddsProps, opts = {}) {
|
||||
groups.get(gk).ids.push(row.id);
|
||||
}
|
||||
for (const g of groups.values()) {
|
||||
const { error: upErr } = await sb.from('ledger_entries')
|
||||
.update({ closing_line: g.line, closing_odds: g.odds })
|
||||
.in('id', g.ids);
|
||||
if (!upErr) updated += g.ids.length;
|
||||
// CHUNKED, for the same reason settleLedger no longer refetches by id: an
|
||||
// `.in('id', …)` filter travels in the URL, and a few hundred UUIDs exceed
|
||||
// what the fetch layer will send (500 ids ≈ 18.5 KB). That is the defect
|
||||
// that silently killed settlement on 2026-08-01; this is the same shape,
|
||||
// one prop-price group deep, and it fails the same way once a single
|
||||
// line|odds pair collects enough rows on a big slate.
|
||||
for (let i = 0; i < g.ids.length; i += ID_FILTER_CHUNK) {
|
||||
const idChunk = g.ids.slice(i, i + ID_FILTER_CHUNK);
|
||||
const { error: upErr } = await sb.from('ledger_entries')
|
||||
.update({ closing_line: g.line, closing_odds: g.odds })
|
||||
.in('id', idChunk);
|
||||
if (upErr) {
|
||||
console.warn(`[ledger] closing capture chunk failed (${idChunk.length} rows): ${upErr.message}`);
|
||||
continue;
|
||||
}
|
||||
updated += idChunk.length;
|
||||
}
|
||||
}
|
||||
return { updated };
|
||||
}
|
||||
@@ -614,20 +637,39 @@ async function settleLedger(sport, opts = {}) {
|
||||
const getSchedule = opts.getSchedule || getScheduleFn;
|
||||
const cutoff = opts.beforeDate || todayET(); // settle strictly-before today
|
||||
|
||||
const { data: open, error } = await sb.from('ledger_entries')
|
||||
.select('id, player_key, player_name, stat, line, side, closing_line')
|
||||
// ONE query, selecting everything the settle loop needs.
|
||||
//
|
||||
// THIS USED TO BE TWO QUERIES, and the second one silently killed the entire
|
||||
// self-learning loop (found 2026-08-03). It refetched by primary key —
|
||||
// `.in('id', open.map(r => r.id))` — which PostgREST sends as a GET query
|
||||
// string: 500 UUIDs is an 18,499-character URL, and the fetch layer rejects it
|
||||
// with `TypeError: fetch failed`. The result was destructured as
|
||||
// `const { data: rows } = ...` with NO error binding, so `rows` came back null,
|
||||
// the loop body never executed, and settleLedger returned
|
||||
// `{settled:0, voided:0, unrecoverable:0, pending:0}` — BYTE-IDENTICAL to a
|
||||
// clean "nothing to settle".
|
||||
//
|
||||
// It stayed invisible because it is volume-triggered: daily volume ran 20–260
|
||||
// rows and settled perfectly for weeks. 2026-08-01 was the first day over the
|
||||
// 500-row fetch limit, and settlement died that night — 1,444 rows with
|
||||
// `settle_attempts = 0`, never even attempted. Worse, the zero-settle ops alarm
|
||||
// reads THESE return values, so `pending: 0` told the watchdog there was
|
||||
// nothing pending and nobody was paged.
|
||||
//
|
||||
// The refetch existed only to add game_date/settle_attempts/dclv_computed_at.
|
||||
// Selecting them up front removes the URL entirely — there is no id list to
|
||||
// send at any volume.
|
||||
const { data: rows, error } = await sb.from('ledger_entries')
|
||||
.select('id, player_key, player_name, stat, line, side, closing_line, game_date, settle_attempts, dclv_computed_at')
|
||||
.eq('sport', sp)
|
||||
.is('outcome', null)
|
||||
.lt('game_date', cutoff)
|
||||
.order('game_date', { ascending: true })
|
||||
.limit(SETTLE_FETCH_LIMIT);
|
||||
// An errored fetch is NOT an empty backlog. Returning zeros here is what made
|
||||
// a dead loop look like a healthy one for two days — surface it instead.
|
||||
if (error) return { settled: 0, pending: 0, error: error.message };
|
||||
if (!open || open.length === 0) return { settled: 0, pending: 0 };
|
||||
|
||||
// We need each row's game_date for the log match — refetch with it included.
|
||||
const { data: rows } = await sb.from('ledger_entries')
|
||||
.select('id, player_name, stat, line, side, closing_line, game_date, settle_attempts, dclv_computed_at, player_key')
|
||||
.in('id', open.map((r) => r.id));
|
||||
if (!rows || rows.length === 0) return { settled: 0, pending: 0 };
|
||||
|
||||
// One game-log fetch per unique player.
|
||||
const players = [...new Set((rows || []).map((r) => r.player_name))];
|
||||
|
||||
Reference in New Issue
Block a user