Settlement has been dead since 2026-08-01: a 500-id filter overflowed the URL
The self-learning loop stopped two days ago and reported success the whole
time. 1,444 ledger rows from 2026-08-01 sit unsettled with settle_attempts=0
-- never even attempted -- and every accruing challenger has been starved of
settled sample as a result.
ROOT CAUSE. settleLedger fetched open ids, then REFETCHED the full rows with
.in('id', ids). PostgREST puts filters in the URL, so 500 UUIDs became an
18,499-character request that the fetch layer rejects with "TypeError: fetch
failed". The result was destructured as `const { data: rows } = ...` with NO
error binding, so rows came back null, the loop body never executed, and the
function returned {settled:0, voided:0, unrecoverable:0, pending:0} --
byte-identical to a clean "nothing to settle". Reproduced against prod before
changing anything.
WHY IT HID FOR TWO DAYS. It is volume-triggered. Daily volume ran 20-260 rows
and settled perfectly for weeks; 2026-08-01 was the first day past the 500-row
fetch limit. And the zero-settle ops alarm reads these very return values, so
pending:0 told the watchdog the backlog was empty -- the alarm built to catch
exactly this could not see it.
THE FIX. The refetch existed only to add game_date/settle_attempts/
dclv_computed_at. Selecting them in the first query removes the id list
entirely, so there is no URL to overflow at any volume. A failed fetch now
surfaces its error instead of being reported as an empty backlog.
captureClosing carried the same shape one level down -- .in('id', g.ids) on an
UPDATE, which fails identically once a single line|odds group gets large on a
big slate. Its id filters are now chunked at 100 (~3.7 KB).
Tests: the regression is locked by asserting settlement issues NO id-list
filter at 500 rows, and that a failed fetch is never reported as an empty
backlog -- the two properties that would have caught this. Two existing
suites asserted the old two-query shape and were updated to the real one.
4,159 tests green (332 suites); web build exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -78,6 +78,13 @@ describe('IMMUTABILITY — a shown badge never silently flips', () => {
|
||||
expect(svc).toMatch(/row\.dclv_computed_at\s*\n?\s*\?\s*null/);
|
||||
});
|
||||
test('the settle read fetches dclv_computed_at so the guard can see it', () => {
|
||||
expect(svc).toMatch(/dclv_computed_at, player_key/);
|
||||
// Asserted on the SETTLE SELECT itself rather than on an adjacent column
|
||||
// pair: the settle read is now a single query (the id-refetch that used to
|
||||
// follow it overflowed the URL and silently returned null), so column order
|
||||
// changed while the requirement did not. What matters is that the column the
|
||||
// immutability guard reads is actually fetched.
|
||||
const settleSelect = svc.match(/\.select\('id, player_key[^']*'\)/g) || [];
|
||||
expect(settleSelect.length).toBeGreaterThan(0);
|
||||
expect(settleSelect.some((sel) => sel.includes('dclv_computed_at'))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -157,9 +157,9 @@ describe('computeClv — signed by side (Phase 1 amendment)', () => {
|
||||
describe('settleLedger — outcome + CLV vs the real result', () => {
|
||||
test('settles hit/miss/push from the game log and stamps CLV', async () => {
|
||||
const sb = fakeSb();
|
||||
// 1st select: open ids; 2nd: full rows.
|
||||
// ONE select — the id-refetch that used to follow it is gone (it built an
|
||||
// 18.5 KB URL and silently returned null; see settleLedger's header).
|
||||
sb._state.selectResults = [
|
||||
[{ id: 'r1' }, { id: 'r2' }],
|
||||
[
|
||||
{ id: 'r1', player_name: 'Aaron Judge', stat: 'home_runs', line: 0.5, side: 'over', closing_line: 0.5, game_date: '2026-07-09' },
|
||||
{ id: 'r2', player_name: 'Aaron Judge', stat: 'hits', line: 1.5, side: 'over', closing_line: 2.5, game_date: '2026-07-09' },
|
||||
@@ -182,7 +182,6 @@ describe('settleLedger — outcome + CLV vs the real result', () => {
|
||||
test('no row + game FINAL → VOID (genuine DNP; dates verified correct)', async () => {
|
||||
const sb = fakeSb();
|
||||
sb._state.selectResults = [
|
||||
[{ id: 'r1' }],
|
||||
[{ id: 'r1', player_name: 'Aaron Judge', stat: 'home_runs', line: 0.5, side: 'over', closing_line: null, game_date: '2026-07-09' }],
|
||||
];
|
||||
const getPlayerStats = async () => ({ found: true, last10: [{ date: '2026-07-08', stat: { homeRuns: 2 } }] });
|
||||
@@ -198,7 +197,6 @@ describe('settleLedger — outcome + CLV vs the real result', () => {
|
||||
test('no row + game NOT FINAL → stays pending, never voided', async () => {
|
||||
const sb = fakeSb();
|
||||
sb._state.selectResults = [
|
||||
[{ id: 'r1' }],
|
||||
[{ id: 'r1', player_name: 'Aaron Judge', stat: 'home_runs', line: 0.5, side: 'over', closing_line: null, game_date: '2026-07-09' }],
|
||||
];
|
||||
const getPlayerStats = async () => ({ found: true, last10: [{ date: '2026-07-08', stat: { homeRuns: 2 } }] });
|
||||
@@ -216,7 +214,6 @@ describe('settleLedger — outcome + CLV vs the real result', () => {
|
||||
test('undetermined state becomes UNRECOVERABLE at the retry cap', async () => {
|
||||
const sb = fakeSb();
|
||||
sb._state.selectResults = [
|
||||
[{ id: 'r1' }],
|
||||
[{ id: 'r1', player_name: 'Aaron Judge', stat: 'home_runs', line: 0.5, side: 'over', closing_line: null, game_date: '2026-07-09', settle_attempts: 3 }],
|
||||
];
|
||||
const getPlayerStats = async () => ({ found: true, last10: [] });
|
||||
@@ -435,3 +432,113 @@ describe('Session 64 Order 2 — quarantine vs analysis_flags (two scopes)', ()
|
||||
expect(agg).toMatch(/quarantine_reason/);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* THE SILENT-SETTLEMENT REGRESSION (2026-08-03).
|
||||
*
|
||||
* settleLedger used to fetch open ids, then REFETCH the full rows by
|
||||
* `.in('id', ids)`. PostgREST puts filters in the URL, so 500 UUIDs became an
|
||||
* 18,499-character request that the fetch layer rejected outright. The result
|
||||
* was destructured as `const { data: rows } = ...` with no error binding, so
|
||||
* `rows` was null, the loop never ran, and the function returned
|
||||
* `{settled:0, voided:0, unrecoverable:0, pending:0}` — byte-identical to a
|
||||
* healthy "nothing to settle".
|
||||
*
|
||||
* It was invisible for two days because it is VOLUME-triggered: daily volume ran
|
||||
* 20–260 rows and settled perfectly for weeks; 2026-08-01 was the first day past
|
||||
* the 500-row fetch limit and 1,444 rows were left with `settle_attempts = 0`,
|
||||
* never attempted. The zero-settle ops alarm reads these same return values, so
|
||||
* `pending: 0` told the watchdog the backlog was empty and nobody was paged.
|
||||
*
|
||||
* These tests assert the two properties that would have caught it: settlement
|
||||
* issues NO id-list filter at any volume, and a failed fetch is never reported
|
||||
* as an empty backlog.
|
||||
*/
|
||||
describe('settleLedger — the silent-settlement regression stays fixed', () => {
|
||||
function bigSb(n) {
|
||||
const calls = { selects: [], updates: [], inFilters: [] };
|
||||
const rows = Array.from({ length: n }, (_, i) => ({
|
||||
id: `00000000-0000-4000-8000-${String(i).padStart(12, '0')}`,
|
||||
player_name: 'Aaron Judge', stat: 'home_runs', line: 0.5, side: 'over',
|
||||
closing_line: 0.5, game_date: '2026-07-09',
|
||||
}));
|
||||
let served = false;
|
||||
function builder() {
|
||||
const b = {
|
||||
_update: null,
|
||||
update(v) { b._update = v; return b; },
|
||||
select() { return b; }, eq() { return b; }, is() { return b; },
|
||||
not() { return b; }, lt() { return b; }, gte() { return b; }, gt() { return b; },
|
||||
in(col, ids) {
|
||||
calls.inFilters.push({ col, count: ids.length });
|
||||
if (b._update) { calls.updates.push({ values: b._update, ids }); return Promise.resolve({ error: null }); }
|
||||
return terminal();
|
||||
},
|
||||
order() { return b; },
|
||||
limit() { return terminal(); },
|
||||
then(res, rej) { return terminal().then(res, rej); },
|
||||
};
|
||||
function terminal() {
|
||||
if (b._update) { calls.updates.push({ values: b._update }); return Promise.resolve({ error: null }); }
|
||||
if (served) return Promise.resolve({ data: [], error: null });
|
||||
served = true;
|
||||
return Promise.resolve({ data: rows, error: null });
|
||||
}
|
||||
return b;
|
||||
}
|
||||
return { from: () => builder(), _calls: calls };
|
||||
}
|
||||
|
||||
test('settles a 500-row backlog WITHOUT ever sending an id-list filter', async () => {
|
||||
const sb = bigSb(500);
|
||||
const getPlayerStats = async () => ({
|
||||
found: true, last10: [{ date: '2026-07-09', stat: { homeRuns: 1 } }],
|
||||
});
|
||||
const res = await ledger.settleLedger('mlb', {
|
||||
sb, getPlayerStats, now: () => NOW, beforeDate: '2026-07-10',
|
||||
});
|
||||
expect(res.settled).toBe(500);
|
||||
// The whole defect in one assertion: no `.in('id', …)` at read time means no
|
||||
// URL to overflow, at any volume.
|
||||
expect(sb._calls.inFilters.filter((f) => f.col === 'id')).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('a FAILED fetch is surfaced, never reported as an empty backlog', async () => {
|
||||
// The exact production failure: the query rejects, so `data` is null.
|
||||
const sb = { from: () => ({
|
||||
select() { return this; }, eq() { return this; }, is() { return this; },
|
||||
lt() { return this; }, order() { return this; },
|
||||
limit() { return Promise.resolve({ data: null, error: { message: 'TypeError: fetch failed' } }); },
|
||||
}) };
|
||||
const res = await ledger.settleLedger('mlb', {
|
||||
sb, getPlayerStats: async () => ({ found: false }), now: () => NOW, beforeDate: '2026-07-10',
|
||||
});
|
||||
// A dead loop must not be indistinguishable from a healthy one.
|
||||
expect(res.error).toBe('TypeError: fetch failed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('captureClosing — id filters are chunked (same defect family)', () => {
|
||||
test('a 250-row same-price group updates in bounded chunks, all rows counted', async () => {
|
||||
const calls = { updates: [] };
|
||||
const rows = Array.from({ length: 250 }, (_, i) => ({
|
||||
id: `id-${i}`, player_key: 'aaron judge', stat: 'home_runs', side: 'over',
|
||||
}));
|
||||
let served = false;
|
||||
const sb = { from: () => {
|
||||
const b = {
|
||||
_update: null,
|
||||
update(v) { b._update = v; return b; },
|
||||
select() { return b; }, eq() { return b; }, is() { return b; },
|
||||
in(col, ids) { calls.updates.push({ values: b._update, ids }); return Promise.resolve({ error: null }); },
|
||||
limit() { if (served) return Promise.resolve({ data: [], error: null }); served = true; return Promise.resolve({ data: rows, error: null }); },
|
||||
};
|
||||
return b;
|
||||
} };
|
||||
const res = await ledger.captureClosing('mlb', [PROP], { sb, gameDate: '2026-07-10' });
|
||||
expect(res.updated).toBe(250);
|
||||
// Every chunk stays small enough that the id list cannot overflow the URL.
|
||||
expect(calls.updates.length).toBeGreaterThan(1);
|
||||
for (const u of calls.updates) expect(u.ids.length).toBeLessThanOrEqual(100);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -175,8 +175,9 @@ describe('ledgerService.settleLedger — WNBA settles vs the ESPN game log', ()
|
||||
|
||||
test('settles a WNBA row (points) via the ESPN gamelog ET-date match', async () => {
|
||||
const sb = fakeSb();
|
||||
// ONE select — settleLedger no longer refetches rows by id (that filter
|
||||
// built an 18.5 KB URL and silently returned null; see ledgerService).
|
||||
sb._state.selectResults = [
|
||||
[{ id: 'r1' }],
|
||||
[{ id: 'r1', player_name: "A'ja Wilson", stat: 'points', line: 19.5, side: 'over', closing_line: 19.5, game_date: '2026-07-12' }],
|
||||
];
|
||||
// ESPN ISO date normalizes to ET 2026-07-12 → matches game_date.
|
||||
|
||||
Reference in New Issue
Block a user