Quota guard: close the silent odds-api drain + reserve floor for MLB
Diagnosis (why 500/500 went unpaged): the only regular odds-api burner was
futuresService, which called axios DIRECTLY — bypassing the gateway, so it
never hit recordCall (the ONE place the WARN/BLOCK pager fires) and never
respected the 95% block. It only syncFromHeaders, which updated the counter's
number SILENTLY. oddsService (which does go through the gateway) only touches
odds-api when PropLine fails, so recordCall for odds-api effectively never ran.
Result: the counter could reach 100% with neither pager firing.
Fixes (a silent drain is now impossible, not just guarded):
- futuresService routes through gateway.fetch('odds-api', …) → counted, blocked
at 95%, and reserve-gated. Closes the raw-axios bypass.
- Reserve floor in the gateway: a DISCRETIONARY call (futures/soccer) passes
reserve=ODDS_API_RESERVE (default 50) and is refused while remaining <= reserve.
The ESSENTIAL MLB prop-backup passes no reserve and may spend to the 95% block.
→ a futures/soccer drain can NEVER starve MLB's backup path.
- quotaTracker.syncFromHeaders (the AUTHORITATIVE number) now fires the same
once-per-period WARN/BLOCK alert on a crossing — extracted fireThresholdAlert
shared with recordCall. The header-only drain now pages.
- POST /api/internal/quota/test-alert (internal-key) test-fires the pager
end-to-end so ntfy delivery is verifiable on demand.
Also (reality-corrected cadence): WNBA restored to the full grid. 2026-07-15
had two AFTERNOON WNBA games finished before the 22 UTC slot — 14 UTC (10am ET)
is the only slot early enough for a 1pm ET game's props, and on PropLine the
extra slots cost a rounding error. Soccer stays the only trimmed sport (the
real odds-api discipline). Assumption corrected by observed data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -43,7 +43,19 @@ class QuotaExhaustedError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
async function tryOne(providerId, callbackFn, syncHeadersFrom) {
|
||||
async function tryOne(providerId, callbackFn, syncHeadersFrom, reserve = 0) {
|
||||
// Reserve floor (Job 1 / quota guard) — a DISCRETIONARY call (futures,
|
||||
// soccer outrights) is refused while fewer than `reserve` credits remain, so
|
||||
// it can never drain the last credits that the ESSENTIAL path (MLB prop
|
||||
// backup when PropLine fails) depends on. Essential calls pass reserve=0 and
|
||||
// use the quota down to the normal 95% block. Checked BEFORE the optimistic
|
||||
// increment so we don't consume-then-refund. Degraded Redis fails open.
|
||||
if (reserve > 0) {
|
||||
const pre = await quotaTracker.getQuotaStatus(providerId);
|
||||
if (pre && !pre.degraded && Number.isFinite(pre.remaining) && pre.remaining <= reserve) {
|
||||
return { ok: false, reason: `reserve_floor(${pre.remaining}<=${reserve})`, status: pre };
|
||||
}
|
||||
}
|
||||
// Optimistic increment — if the call throws we roll back below.
|
||||
// recordCall also evaluates the post-increment threshold; if the
|
||||
// very next call would put us at 95%+, we still execute THIS one
|
||||
@@ -89,10 +101,11 @@ async function fetch(primaryId, callbackFn, opts = {}) {
|
||||
sport,
|
||||
fallbackProviders,
|
||||
syncHeadersFrom,
|
||||
reserve = 0,
|
||||
} = opts;
|
||||
|
||||
const attempts = [];
|
||||
const result = await tryOne(primaryId, callbackFn, syncHeadersFrom);
|
||||
const result = await tryOne(primaryId, callbackFn, syncHeadersFrom, reserve);
|
||||
if (result.ok) return result.result;
|
||||
|
||||
// Generic adapter error on the primary — propagate, don't shift.
|
||||
@@ -112,7 +125,7 @@ async function fetch(primaryId, callbackFn, opts = {}) {
|
||||
: [];
|
||||
|
||||
for (const fallbackId of chain) {
|
||||
const fb = await tryOne(fallbackId, callbackFn, syncHeadersFrom);
|
||||
const fb = await tryOne(fallbackId, callbackFn, syncHeadersFrom, reserve);
|
||||
if (fb.ok) {
|
||||
console.log(`[gateway] primary=${primaryId} blocked; succeeded via fallback=${fallbackId}`);
|
||||
return fb.result;
|
||||
|
||||
Reference in New Issue
Block a user