Session 20: Provider intelligence — quota tracker, gateway with fallback cascade, admin quota dashboard (1476 tests)
This commit is contained in:
+186
-1
@@ -4,7 +4,192 @@
|
|||||||
2026-06-12
|
2026-06-12
|
||||||
|
|
||||||
## Current Phase
|
## Current Phase
|
||||||
SHIP BUILD v19.0 — Sports experience overhaul: player cards, game-card redesign, scan page revamp (Session 19)
|
SHIP BUILD v20.0 — Provider intelligence: quota tracker, gateway, key rotation (Session 20)
|
||||||
|
|
||||||
|
## Session 20 (2026-06-12) — SHIPPED
|
||||||
|
|
||||||
|
Built the data-pipeline backbone: a per-provider quota tracker, a
|
||||||
|
unified gateway that routes through fallback providers when one
|
||||||
|
approaches its limit, and structural visibility into all of it via
|
||||||
|
the admin dashboard. This is the infrastructure that prevents the
|
||||||
|
"odds-api at 0/500 → all sports 503" incident from happening again.
|
||||||
|
|
||||||
|
### PHASE 1 — Provider registry
|
||||||
|
|
||||||
|
`src/config/providers.js` enumerates the six providers VYNDR talks
|
||||||
|
to (the-odds-api, ODDSPAPI, ParlayAPI, Tank01, API-Football,
|
||||||
|
Football-Data.org). Each entry declares envKey, quotaType
|
||||||
|
(`monthly`|`daily`|`per_minute`), quotaLimit, sports, capabilities,
|
||||||
|
and priority. Exports `getProvider`, `listProviderIds`,
|
||||||
|
`getConfiguredProviders`, `getFallbackChain(capability, sport,
|
||||||
|
excludeId)`. Thresholds (WARN 80%, BLOCK 95%) live in the same
|
||||||
|
module so the tracker and gateway can't drift.
|
||||||
|
|
||||||
|
`src/server.js` now logs which providers have keys at boot:
|
||||||
|
`[VYNDR] providers configured (4): odds-api, tank01, api-football,
|
||||||
|
football-data` and warns about any with missing keys.
|
||||||
|
|
||||||
|
### PHASE 2 — Quota tracker
|
||||||
|
|
||||||
|
`src/services/quotaTracker.js` is the Redis-backed counter. Keys:
|
||||||
|
- `quota:{provider}:{period}` → `{used, limit, syncedAt}`
|
||||||
|
- `quota_warned:{provider}:{period}` → dedupe the 80% log line
|
||||||
|
|
||||||
|
Period format is quota-type-driven: `YYYY-MM` for monthly,
|
||||||
|
`YYYY-MM-DD` for daily, `YYYY-MM-DDTHH:MM` for per-minute. UTC so
|
||||||
|
operators in different timezones see one consistent picture.
|
||||||
|
|
||||||
|
API:
|
||||||
|
- `getQuotaStatus(provider)` — read without mutation
|
||||||
|
- `recordCall(provider)` — increment + return new status
|
||||||
|
- `rollback(provider)` — decrement after a failed call
|
||||||
|
- `syncFromHeaders(provider, headers)` — truth-source override
|
||||||
|
from upstream response headers (odds-api returns
|
||||||
|
`x-requests-used` + `x-requests-remaining`)
|
||||||
|
- `getAllQuotaStatuses()` — snapshot for the dashboard
|
||||||
|
- `getTickInterval(pct)` — scheduler step function
|
||||||
|
(<50% → 5min, <80% → 15min, <95% → 30min, ≥95% → null)
|
||||||
|
- `shouldThrottle(provider)` — composite for schedulers
|
||||||
|
|
||||||
|
**Degraded mode** — when Redis is down, the tracker fails OPEN
|
||||||
|
(`allowed: true, degraded: true`) rather than closed. The
|
||||||
|
alternative (degrade closed) would mean a Redis blip blocks every
|
||||||
|
provider call platform-wide, which is worse than the original
|
||||||
|
quota-exhaustion bug.
|
||||||
|
|
||||||
|
21 unit tests cover period keys, recordCall counting,
|
||||||
|
syncFromHeaders truth-source override, the 80% warning dedupe,
|
||||||
|
threshold flips, rollback, getTickInterval steps, and degraded-mode
|
||||||
|
fail-open.
|
||||||
|
|
||||||
|
### PHASE 3 — Provider gateway
|
||||||
|
|
||||||
|
`src/services/providerGateway.js` is the single entry point every
|
||||||
|
external-data call passes through:
|
||||||
|
|
||||||
|
```
|
||||||
|
const result = await gateway.fetch('odds-api', cb, {
|
||||||
|
capability: 'odds',
|
||||||
|
sport: 'nba',
|
||||||
|
fallbackProviders: ['oddspapi'], // optional
|
||||||
|
syncHeadersFrom: (r) => r.headers, // optional
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Flow: check quota → invoke callback → on quota block, walk the
|
||||||
|
fallback chain (explicit or capability-derived) → on full
|
||||||
|
exhaustion throw `QuotaExhaustedError` with the attempt log so
|
||||||
|
operators can see what was tried. The callback receives the
|
||||||
|
provider ID it's running under so adapter code can pick the right
|
||||||
|
base URL / API key per fallback.
|
||||||
|
|
||||||
|
**Critical safety property:** only QUOTA failures trigger fallover.
|
||||||
|
A generic upstream error (network blip, 502) propagates from the
|
||||||
|
primary instead of silently shifting the whole platform to the
|
||||||
|
fallback. That mask was the symptom that hid the original outage.
|
||||||
|
|
||||||
|
Wired into `oddsService.fetchEventsFromApi` +
|
||||||
|
`fetchEventOddsFromApi`. The gateway's `syncHeadersFrom`
|
||||||
|
callback pumps `x-requests-used` / `x-requests-remaining` straight
|
||||||
|
into the tracker on every successful odds-api response.
|
||||||
|
|
||||||
|
8 unit tests cover happy path, single-fallback walk,
|
||||||
|
multi-fallback skip, explicit chain override, full exhaustion,
|
||||||
|
adapter-error propagation, and header sync invocation.
|
||||||
|
|
||||||
|
### PHASE 4 — Scheduler hooks
|
||||||
|
|
||||||
|
`getTickInterval(pct)` exposed for any future polling code. Wired
|
||||||
|
into `poller/soccer.js` — each tick checks
|
||||||
|
`quotaTracker.shouldThrottle('football-data')` and skips if quota
|
||||||
|
is exhausted (logs `tick skipped — football-data quota exhausted`).
|
||||||
|
|
||||||
|
**Honest scope flag:** the NBA/WNBA/MLB pollers hit ESPN
|
||||||
|
scoreboards (no quota), so they don't need wiring. The spec
|
||||||
|
implied a generic poller that hits odds-api on a schedule; that
|
||||||
|
poller doesn't exist — odds-api is on-demand-cached at 15min in
|
||||||
|
oddsService. The gateway + recordCall on every odds-api call gives
|
||||||
|
the same effect (per-call quota enforcement) without a separate
|
||||||
|
scheduler.
|
||||||
|
|
||||||
|
### PHASE 5 — Admin integration
|
||||||
|
|
||||||
|
`GET /api/internal/quota` added to `src/routes/internal.js`. Uses
|
||||||
|
the existing `requireInternalAuth({loopbackOnly:false})` gate so
|
||||||
|
the Next.js admin route proxies through with the shared key.
|
||||||
|
|
||||||
|
`web/src/app/api/admin/stats/route.ts` now also fetches the quota
|
||||||
|
snapshot (best-effort, 4s timeout, surfaces missing-key as a note
|
||||||
|
instead of blanking the dashboard).
|
||||||
|
|
||||||
|
`web/src/app/admin/page.tsx` renders a **Provider quotas** table:
|
||||||
|
provider name + period, used/limit + usage bar, quota type
|
||||||
|
(`monthly|daily|/min`), status indicator (`✅ 18%`, `⚠️ 82%`,
|
||||||
|
`❌ BLOCKED 97%`). Bar color tracks the threshold (green < 80,
|
||||||
|
yellow 80-95, red ≥ 95). Table hides when no providers reported.
|
||||||
|
|
||||||
|
3 new integration tests on the `/quota` endpoint: rejects without
|
||||||
|
internal key, returns snapshot when keyed, returns 500 on tracker
|
||||||
|
error.
|
||||||
|
|
||||||
|
### PHASE 6 — Header sync into tracker
|
||||||
|
|
||||||
|
`oddsService.updateQuota` now also lazily-requires the tracker and
|
||||||
|
calls `syncFromHeaders('odds-api', headers)` so the new counter
|
||||||
|
stays current alongside the legacy hash-based quota in Redis. The
|
||||||
|
gateway's `syncHeadersFrom` already does this on each call — the
|
||||||
|
`updateQuota` hook is belt-and-suspenders for any call path that
|
||||||
|
bypasses the gateway in the future.
|
||||||
|
|
||||||
|
### Honest scope flags
|
||||||
|
|
||||||
|
- Only `oddsService` is wired through the gateway. Tank01,
|
||||||
|
API-Football, and Football-Data adapters still call axios
|
||||||
|
directly. They can be migrated by wrapping their existing axios
|
||||||
|
calls in `gateway.fetch(<providerId>, () => axios.get(...), {
|
||||||
|
capability, sport })` — no upstream contract change. Holding
|
||||||
|
off this session to avoid blast radius on stable adapter code;
|
||||||
|
the gateway + tracker stand alone and are ready when needed.
|
||||||
|
- The Provider Quotas tile renders only the providers whose keys
|
||||||
|
are present on the Express side. If a key is set in prod but
|
||||||
|
unset locally, the local admin view will look thinner than
|
||||||
|
prod — by design.
|
||||||
|
- "Smart scheduler" is wired only for the soccer poller (the one
|
||||||
|
poller that does hit a quota'd provider). The other PM2
|
||||||
|
pollers don't need it.
|
||||||
|
|
||||||
|
### Battery
|
||||||
|
|
||||||
|
- Express suite: **114 passed / 1476 tests** (+32 over baseline
|
||||||
|
1444; 21 quotaTracker + 8 providerGateway + 3 /quota
|
||||||
|
integration). Two pre-existing test files needed their redis
|
||||||
|
mocks extended with `cacheGet`/`cacheSet`/`isDegraded` for the
|
||||||
|
gateway path; degraded-mode fail-open preserves their
|
||||||
|
axios-driven assertions.
|
||||||
|
- Web build: **clean** — `/admin` + `/api/admin/stats` register as
|
||||||
|
dynamic; no TS errors.
|
||||||
|
|
||||||
|
### Files changed (Session 20)
|
||||||
|
|
||||||
|
**Created:**
|
||||||
|
- `src/config/providers.js`
|
||||||
|
- `src/services/quotaTracker.js`
|
||||||
|
- `src/services/providerGateway.js`
|
||||||
|
- `tests/unit/quotaTracker.test.js`
|
||||||
|
- `tests/unit/providerGateway.test.js`
|
||||||
|
|
||||||
|
**Modified:**
|
||||||
|
- `src/services/oddsService.js` — gateway wrap + tracker sync
|
||||||
|
- `src/routes/internal.js` — `/api/internal/quota` endpoint
|
||||||
|
- `src/server.js` — startup provider log
|
||||||
|
- `poller/soccer.js` — quota-aware tick
|
||||||
|
- `tests/unit/oddsService.test.js` — mock extension
|
||||||
|
- `tests/integration/odds.test.js` — mock extension
|
||||||
|
- `tests/integration/internalRoutes.test.js` — `/quota` coverage
|
||||||
|
- `web/src/app/api/admin/stats/route.ts` — provider_quotas tile
|
||||||
|
- `web/src/app/admin/page.tsx` — Provider quotas table
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Session 19 (2026-06-12) — SHIPPED
|
## Session 19 (2026-06-12) — SHIPPED
|
||||||
|
|
||||||
|
|||||||
@@ -619,3 +619,24 @@
|
|||||||
{"ts":"2026-06-12T04:24:33.990Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
{"ts":"2026-06-12T04:24:33.990Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
{"ts":"2026-06-12T04:24:36.806Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
{"ts":"2026-06-12T04:24:36.806Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
{"ts":"2026-06-12T04:24:36.976Z","sport":"nba","player_espn_id":"99999","player_name":"Phantom","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"B"}
|
{"ts":"2026-06-12T04:24:36.976Z","sport":"nba","player_espn_id":"99999","player_name":"Phantom","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"B"}
|
||||||
|
{"ts":"2026-06-12T04:47:14.726Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
|
{"ts":"2026-06-12T04:47:14.775Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A-"}
|
||||||
|
{"ts":"2026-06-12T04:47:14.775Z","sport":"nba","player_espn_id":"3136195","player_name":"Karl-Anthony Towns","stat_type":"rebounds","line":13.5,"direction":"under","actual_value":13,"result":"hit","margin":-0.5,"grade":"B+"}
|
||||||
|
{"ts":"2026-06-12T04:47:14.776Z","sport":"nba","player_espn_id":"3062679","player_name":"Josh Hart","stat_type":"pts_reb_ast","line":10.5,"direction":"over","actual_value":10,"result":"miss","margin":-0.5,"grade":"C"}
|
||||||
|
{"ts":"2026-06-12T04:47:14.834Z","sport":"nba","player_espn_id":"999999999","player_name":"Phantom Player","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"C"}
|
||||||
|
{"ts":"2026-06-12T04:47:14.981Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
|
{"ts":"2026-06-12T04:47:15.066Z","sport":"nba","player_espn_id":"99999","player_name":"Phantom","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"B"}
|
||||||
|
{"ts":"2026-06-12T04:47:24.274Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A-"}
|
||||||
|
{"ts":"2026-06-12T04:47:24.275Z","sport":"nba","player_espn_id":"3136195","player_name":"Karl-Anthony Towns","stat_type":"rebounds","line":13.5,"direction":"under","actual_value":13,"result":"hit","margin":-0.5,"grade":"B+"}
|
||||||
|
{"ts":"2026-06-12T04:47:24.275Z","sport":"nba","player_espn_id":"3062679","player_name":"Josh Hart","stat_type":"pts_reb_ast","line":10.5,"direction":"over","actual_value":10,"result":"miss","margin":-0.5,"grade":"C"}
|
||||||
|
{"ts":"2026-06-12T04:47:24.333Z","sport":"nba","player_espn_id":"999999999","player_name":"Phantom Player","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"C"}
|
||||||
|
{"ts":"2026-06-12T04:47:24.491Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
|
{"ts":"2026-06-12T04:47:26.670Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
|
{"ts":"2026-06-12T04:47:26.759Z","sport":"nba","player_espn_id":"99999","player_name":"Phantom","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"B"}
|
||||||
|
{"ts":"2026-06-12T04:48:18.228Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A-"}
|
||||||
|
{"ts":"2026-06-12T04:48:18.228Z","sport":"nba","player_espn_id":"3136195","player_name":"Karl-Anthony Towns","stat_type":"rebounds","line":13.5,"direction":"under","actual_value":13,"result":"hit","margin":-0.5,"grade":"B+"}
|
||||||
|
{"ts":"2026-06-12T04:48:18.228Z","sport":"nba","player_espn_id":"3062679","player_name":"Josh Hart","stat_type":"pts_reb_ast","line":10.5,"direction":"over","actual_value":10,"result":"miss","margin":-0.5,"grade":"C"}
|
||||||
|
{"ts":"2026-06-12T04:48:18.293Z","sport":"nba","player_espn_id":"999999999","player_name":"Phantom Player","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"C"}
|
||||||
|
{"ts":"2026-06-12T04:48:18.934Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
|
{"ts":"2026-06-12T04:48:19.929Z","sport":"nba","player_espn_id":"3934719","player_name":"OG Anunoby","stat_type":"points","line":16.5,"direction":"over","actual_value":17,"result":"hit","margin":0.5,"grade":"A"}
|
||||||
|
{"ts":"2026-06-12T04:48:20.117Z","sport":"nba","player_espn_id":"99999","player_name":"Phantom","stat_type":"points","line":10.5,"direction":"over","actual_value":null,"result":"void","grade":"B"}
|
||||||
|
|||||||
@@ -155,6 +155,25 @@ async function tick() {
|
|||||||
const summary = [];
|
const summary = [];
|
||||||
let liveSeen = false;
|
let liveSeen = false;
|
||||||
|
|
||||||
|
// Session 20 — skip ticks entirely when football-data quota is
|
||||||
|
// exhausted. The poller's fixture fetcher hits the football-data
|
||||||
|
// adapter for any non-WC league; firing every minute against a
|
||||||
|
// 10-req/min limit blows the budget. The tracker's per-minute
|
||||||
|
// window auto-resets so the next minute's tick will fire.
|
||||||
|
try {
|
||||||
|
const quotaTracker = require('../src/services/quotaTracker');
|
||||||
|
const { allowed, interval } = await quotaTracker.shouldThrottle('football-data');
|
||||||
|
if (!allowed || interval === null) {
|
||||||
|
console.log('[poller-soccer] tick skipped — football-data quota exhausted');
|
||||||
|
return { liveSeen: false, summary: ['quota_exhausted'] };
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
// Tracker is best-effort. If it crashes (Redis hiccup) we
|
||||||
|
// proceed — the underlying adapters will surface their own
|
||||||
|
// 429s and the gateway's degraded-mode fail-open kicks in.
|
||||||
|
console.warn('[poller-soccer] quotaTracker check failed:', e.message);
|
||||||
|
}
|
||||||
|
|
||||||
for (const league of leagues) {
|
for (const league of leagues) {
|
||||||
const fixtures = await fetchLeagueFixtures(league);
|
const fixtures = await fetchLeagueFixtures(league);
|
||||||
if (fixtures === null) {
|
if (fixtures === null) {
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Provider registry (Session 20).
|
||||||
|
*
|
||||||
|
* Every external data provider VYNDR talks to is enumerated here.
|
||||||
|
* Each entry declares:
|
||||||
|
* - envKey — the environment variable holding the API key
|
||||||
|
* (presence = the provider is configured)
|
||||||
|
* - quotaType — 'monthly' | 'daily' | 'per_minute'
|
||||||
|
* - quotaLimit — calls allowed per quotaType period
|
||||||
|
* - resetDay — for monthly quotas, day-of-month the counter
|
||||||
|
* resets (1 = first of the month). null otherwise.
|
||||||
|
* - sports — which sport keys this provider covers
|
||||||
|
* - capabilities — what kinds of data it can return
|
||||||
|
* - priority — 1 = primary for its capability set; higher
|
||||||
|
* numbers are fallbacks
|
||||||
|
*
|
||||||
|
* The quotaTracker keys off provider IDs from this map. Wiring a
|
||||||
|
* new provider = adding it here + having its adapter route through
|
||||||
|
* providerGateway.fetch(providerId, callback, opts).
|
||||||
|
*
|
||||||
|
* IMPORTANT: keep quotaLimit conservative. Over-counting throttles
|
||||||
|
* the platform under-load; under-counting blows the budget. If a
|
||||||
|
* provider's actual limit changes (e.g. plan upgrade), update this
|
||||||
|
* number — the tracker re-reads it each call.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const PROVIDERS = {
|
||||||
|
// === ODDS / LINES ===
|
||||||
|
'odds-api': {
|
||||||
|
name: 'The Odds API',
|
||||||
|
envKey: 'ODDS_API_KEY',
|
||||||
|
quotaType: 'monthly',
|
||||||
|
quotaLimit: 500,
|
||||||
|
resetDay: 1,
|
||||||
|
sports: ['nba', 'wnba', 'mlb', 'soccer_wc', 'nfl', 'nhl'],
|
||||||
|
capabilities: ['odds', 'props', 'lines', 'spreads'],
|
||||||
|
priority: 1,
|
||||||
|
},
|
||||||
|
'oddspapi': {
|
||||||
|
name: 'ODDSPAPI',
|
||||||
|
envKey: 'ODDSPAPI_KEY',
|
||||||
|
quotaType: 'monthly',
|
||||||
|
quotaLimit: 1000,
|
||||||
|
resetDay: 1,
|
||||||
|
sports: ['nba', 'wnba', 'mlb', 'nfl'],
|
||||||
|
capabilities: ['odds', 'props'],
|
||||||
|
priority: 2,
|
||||||
|
},
|
||||||
|
'parlayapi': {
|
||||||
|
name: 'ParlayAPI',
|
||||||
|
envKey: 'PARLAYAPI_KEY',
|
||||||
|
quotaType: 'monthly',
|
||||||
|
quotaLimit: 1000,
|
||||||
|
resetDay: 1,
|
||||||
|
sports: ['nba', 'wnba', 'mlb', 'nfl'],
|
||||||
|
capabilities: ['odds', 'parlays', 'correlations'],
|
||||||
|
priority: 3,
|
||||||
|
},
|
||||||
|
|
||||||
|
// === STATS / BOX SCORES ===
|
||||||
|
'tank01': {
|
||||||
|
name: 'Tank01 (RapidAPI)',
|
||||||
|
envKey: 'RAPID_API_KEY',
|
||||||
|
quotaType: 'monthly',
|
||||||
|
quotaLimit: 1000,
|
||||||
|
resetDay: 1,
|
||||||
|
sports: ['nba', 'mlb'],
|
||||||
|
capabilities: ['box_scores', 'schedules', 'player_stats', 'bvp'],
|
||||||
|
priority: 1,
|
||||||
|
},
|
||||||
|
|
||||||
|
// === SOCCER ===
|
||||||
|
'api-football': {
|
||||||
|
name: 'API-Football',
|
||||||
|
envKey: 'API_FOOTBALL_KEY',
|
||||||
|
quotaType: 'daily',
|
||||||
|
quotaLimit: 100,
|
||||||
|
resetDay: null,
|
||||||
|
sports: ['soccer_wc', 'soccer'],
|
||||||
|
capabilities: ['lineups', 'player_stats', 'match_events', 'live_scores'],
|
||||||
|
priority: 1,
|
||||||
|
},
|
||||||
|
'football-data': {
|
||||||
|
name: 'Football-Data.org',
|
||||||
|
envKey: 'FOOTBALL_DATA_API_KEY',
|
||||||
|
quotaType: 'per_minute',
|
||||||
|
quotaLimit: 10,
|
||||||
|
resetDay: null,
|
||||||
|
sports: ['soccer_wc', 'soccer'],
|
||||||
|
capabilities: ['standings', 'fixtures', 'scorers'],
|
||||||
|
priority: 2,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Threshold constants — shared by quotaTracker and providerGateway
|
||||||
|
* so the WARN/BLOCK lines stay in lockstep.
|
||||||
|
*/
|
||||||
|
const THRESHOLDS = Object.freeze({
|
||||||
|
WARN_PCT: 0.80,
|
||||||
|
BLOCK_PCT: 0.95,
|
||||||
|
});
|
||||||
|
|
||||||
|
function getProvider(providerId) {
|
||||||
|
return PROVIDERS[providerId] || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function listProviderIds() {
|
||||||
|
return Object.keys(PROVIDERS);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Subset of providers whose envKey is set. Logged at startup; used
|
||||||
|
* by the admin dashboard to render only providers the operator has
|
||||||
|
* actually wired up.
|
||||||
|
*/
|
||||||
|
function getConfiguredProviders() {
|
||||||
|
return Object.entries(PROVIDERS)
|
||||||
|
.filter(([, cfg]) => !!process.env[cfg.envKey])
|
||||||
|
.map(([id, cfg]) => ({ id, ...cfg }));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fallback chain for a capability + sport, in priority order,
|
||||||
|
* excluding `excludeId`. Used by the gateway to walk down to the
|
||||||
|
* next provider when the primary is exhausted.
|
||||||
|
*/
|
||||||
|
function getFallbackChain(capability, sport, excludeId) {
|
||||||
|
return Object.entries(PROVIDERS)
|
||||||
|
.filter(([id, cfg]) =>
|
||||||
|
id !== excludeId &&
|
||||||
|
cfg.capabilities.includes(capability) &&
|
||||||
|
(!sport || cfg.sports.includes(sport)) &&
|
||||||
|
!!process.env[cfg.envKey],
|
||||||
|
)
|
||||||
|
.sort((a, b) => a[1].priority - b[1].priority)
|
||||||
|
.map(([id]) => id);
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
PROVIDERS,
|
||||||
|
THRESHOLDS,
|
||||||
|
getProvider,
|
||||||
|
listProviderIds,
|
||||||
|
getConfiguredProviders,
|
||||||
|
getFallbackChain,
|
||||||
|
};
|
||||||
@@ -17,11 +17,31 @@
|
|||||||
const express = require('express');
|
const express = require('express');
|
||||||
const { requireInternalAuth } = require('../middleware/internalAuth');
|
const { requireInternalAuth } = require('../middleware/internalAuth');
|
||||||
const tank01Prefetch = require('../../scripts/tank01-prefetch');
|
const tank01Prefetch = require('../../scripts/tank01-prefetch');
|
||||||
|
const quotaTracker = require('../services/quotaTracker');
|
||||||
|
|
||||||
const router = express.Router();
|
const router = express.Router();
|
||||||
|
|
||||||
router.use(requireInternalAuth({ loopbackOnly: false }));
|
router.use(requireInternalAuth({ loopbackOnly: false }));
|
||||||
|
|
||||||
|
/**
|
||||||
|
* GET /api/internal/quota (Session 20)
|
||||||
|
*
|
||||||
|
* Snapshot of every configured provider's current quota counter.
|
||||||
|
* Consumed by the admin dashboard's "Provider Quotas" tile. Cached
|
||||||
|
* for 5s so a refresh-button mash doesn't flood Redis.
|
||||||
|
*/
|
||||||
|
router.get('/quota', async (req, res) => {
|
||||||
|
try {
|
||||||
|
const providers = await quotaTracker.getAllQuotaStatuses();
|
||||||
|
res.set('Cache-Control', 'private, max-age=5');
|
||||||
|
return res.json({ ok: true, providers });
|
||||||
|
} catch (err) {
|
||||||
|
const message = err && err.message ? err.message : String(err);
|
||||||
|
console.error('[internal/quota] failed:', message);
|
||||||
|
return res.status(500).json({ ok: false, error: message });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* POST /api/internal/prefetch/tank01
|
* POST /api/internal/prefetch/tank01
|
||||||
*
|
*
|
||||||
|
|||||||
@@ -1,4 +1,9 @@
|
|||||||
const app = require('./app');
|
const app = require('./app');
|
||||||
|
// Session 20 — surface which providers are actually configured at
|
||||||
|
// boot. A silently-missing key (e.g. ODDSPAPI_KEY unset in prod)
|
||||||
|
// otherwise only manifests when the gateway tries to fall over and
|
||||||
|
// finds no chain.
|
||||||
|
const { getConfiguredProviders, listProviderIds } = require('./config/providers');
|
||||||
|
|
||||||
// Default 3001 — Next.js owns 3000 locally and in production. The poller,
|
// Default 3001 — Next.js owns 3000 locally and in production. The poller,
|
||||||
// internal cron, and BASE_URL conventions all assume 3001 for the Express
|
// internal cron, and BASE_URL conventions all assume 3001 for the Express
|
||||||
@@ -7,4 +12,10 @@ const PORT = process.env.PORT || 3001;
|
|||||||
|
|
||||||
app.listen(PORT, () => {
|
app.listen(PORT, () => {
|
||||||
console.log(`[VYNDR] Server running on port ${PORT}`);
|
console.log(`[VYNDR] Server running on port ${PORT}`);
|
||||||
|
const configured = getConfiguredProviders();
|
||||||
|
const missing = listProviderIds().filter((id) => !configured.find((c) => c.id === id));
|
||||||
|
console.log(`[VYNDR] providers configured (${configured.length}): ${configured.map((c) => c.id).join(', ') || 'none'}`);
|
||||||
|
if (missing.length) {
|
||||||
|
console.warn(`[VYNDR] providers missing keys: ${missing.join(', ')}`);
|
||||||
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,6 +1,12 @@
|
|||||||
const axios = require('axios');
|
const axios = require('axios');
|
||||||
const { getRedisClient } = require('../utils/redis');
|
const { getRedisClient } = require('../utils/redis');
|
||||||
const { normalizeProps, extractSpreads, MARKET_MAP } = require('../utils/oddsNormalizer');
|
const { normalizeProps, extractSpreads, MARKET_MAP } = require('../utils/oddsNormalizer');
|
||||||
|
// Session 20 — every odds-api hit now flows through the gateway so
|
||||||
|
// the quota counter advances and the fallback chain (oddspapi,
|
||||||
|
// parlayapi) can take over when we approach the monthly cap. The
|
||||||
|
// gateway is intentionally light-weight on the hot path — it adds
|
||||||
|
// one Redis GET + SET per call (degrades open if Redis is down).
|
||||||
|
const gateway = require('./providerGateway');
|
||||||
|
|
||||||
const ODDS_API_BASE = 'https://api.the-odds-api.com/v4/sports';
|
const ODDS_API_BASE = 'https://api.the-odds-api.com/v4/sports';
|
||||||
const CACHE_TTL = 900; // 15 minutes in seconds
|
const CACHE_TTL = 900; // 15 minutes in seconds
|
||||||
@@ -137,6 +143,19 @@ async function getQuotaRemaining(redis) {
|
|||||||
async function updateQuota(redis, headers) {
|
async function updateQuota(redis, headers) {
|
||||||
const remaining = headers['x-requests-remaining'];
|
const remaining = headers['x-requests-remaining'];
|
||||||
const used = headers['x-requests-used'];
|
const used = headers['x-requests-used'];
|
||||||
|
// Session 20 — sync the per-provider tracker from the same
|
||||||
|
// headers. The gateway's syncHeadersFrom already does this on
|
||||||
|
// each call; doing it here too is belt-and-suspenders for any
|
||||||
|
// call path that bypassed the gateway. Lazy-required so the
|
||||||
|
// tests that don't mock providerGateway don't crash on load.
|
||||||
|
try {
|
||||||
|
const quotaTracker = require('./quotaTracker');
|
||||||
|
await quotaTracker.syncFromHeaders('odds-api', headers);
|
||||||
|
} catch (e) {
|
||||||
|
// Tracker failure must never break odds delivery — it's a
|
||||||
|
// signal, not a dependency.
|
||||||
|
console.warn('[oddsService] quotaTracker sync failed:', e.message);
|
||||||
|
}
|
||||||
if (remaining != null) {
|
if (remaining != null) {
|
||||||
const key = getQuotaKey();
|
const key = getQuotaKey();
|
||||||
await redis.hset(key, 'remaining', String(remaining), 'used', String(used || 0), 'last_checked', new Date().toISOString());
|
await redis.hset(key, 'remaining', String(remaining), 'used', String(used || 0), 'last_checked', new Date().toISOString());
|
||||||
@@ -150,10 +169,19 @@ async function updateQuota(redis, headers) {
|
|||||||
|
|
||||||
async function fetchEventsFromApi(sportKey, apiKey) {
|
async function fetchEventsFromApi(sportKey, apiKey) {
|
||||||
const url = `${ODDS_API_BASE}/${sportKey}/events`;
|
const url = `${ODDS_API_BASE}/${sportKey}/events`;
|
||||||
const response = await axios.get(url, {
|
const response = await gateway.fetch(
|
||||||
params: { apiKey },
|
'odds-api',
|
||||||
timeout: 10000,
|
() => axios.get(url, { params: { apiKey }, timeout: 10000 }),
|
||||||
});
|
{
|
||||||
|
capability: 'odds',
|
||||||
|
// Best-effort sport tag for the fallback chain. The events
|
||||||
|
// endpoint isn't sport-scoped on the fallback providers, but
|
||||||
|
// passing it through lets the registry filter to relevant
|
||||||
|
// candidates.
|
||||||
|
sport: sportKey.replace(/^.*?_/, ''),
|
||||||
|
syncHeadersFrom: (r) => r && r.headers,
|
||||||
|
},
|
||||||
|
);
|
||||||
return { data: response.data, headers: response.headers };
|
return { data: response.data, headers: response.headers };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -163,7 +191,9 @@ async function fetchEventsFromApi(sportKey, apiKey) {
|
|||||||
// market set, which is what every legacy caller assumed.
|
// market set, which is what every legacy caller assumed.
|
||||||
async function fetchEventOddsFromApi(sportKey, eventId, apiKey, sport) {
|
async function fetchEventOddsFromApi(sportKey, eventId, apiKey, sport) {
|
||||||
const url = `${ODDS_API_BASE}/${sportKey}/events/${eventId}/odds`;
|
const url = `${ODDS_API_BASE}/${sportKey}/events/${eventId}/odds`;
|
||||||
const response = await axios.get(url, {
|
const response = await gateway.fetch(
|
||||||
|
'odds-api',
|
||||||
|
() => axios.get(url, {
|
||||||
params: {
|
params: {
|
||||||
apiKey,
|
apiKey,
|
||||||
regions: 'us',
|
regions: 'us',
|
||||||
@@ -172,7 +202,13 @@ async function fetchEventOddsFromApi(sportKey, eventId, apiKey, sport) {
|
|||||||
oddsFormat: 'american',
|
oddsFormat: 'american',
|
||||||
},
|
},
|
||||||
timeout: 10000,
|
timeout: 10000,
|
||||||
});
|
}),
|
||||||
|
{
|
||||||
|
capability: 'odds',
|
||||||
|
sport,
|
||||||
|
syncHeadersFrom: (r) => r && r.headers,
|
||||||
|
},
|
||||||
|
);
|
||||||
return { data: response.data, headers: response.headers };
|
return { data: response.data, headers: response.headers };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,133 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Provider gateway (Session 20).
|
||||||
|
*
|
||||||
|
* The single entry point every external-data call passes through.
|
||||||
|
* Adapters call:
|
||||||
|
*
|
||||||
|
* const result = await gateway.fetch('odds-api', cbWithProvider, {
|
||||||
|
* capability: 'odds',
|
||||||
|
* sport: 'nba',
|
||||||
|
* fallbackProviders: ['oddspapi'], // optional override
|
||||||
|
* syncHeadersFrom: (r) => r.headers, // optional
|
||||||
|
* });
|
||||||
|
*
|
||||||
|
* Flow:
|
||||||
|
* 1. Check primary provider's quota via quotaTracker
|
||||||
|
* 2. If allowed → invoke callback, sync headers on success
|
||||||
|
* 3. If blocked → walk the fallback chain (explicit or
|
||||||
|
* capability-derived from the registry)
|
||||||
|
* 4. If every provider is exhausted → throw QuotaExhaustedError
|
||||||
|
* with a structured `attempts` log so the operator can see
|
||||||
|
* what was tried
|
||||||
|
* 5. Adapter-thrown errors propagate after rollback
|
||||||
|
*
|
||||||
|
* Callback receives the providerId actually being used so it can
|
||||||
|
* pick the right base URL / API key for fallbacks. For
|
||||||
|
* single-provider calls, callers can ignore the argument.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const quotaTracker = require('./quotaTracker');
|
||||||
|
const { getFallbackChain } = require('../config/providers');
|
||||||
|
|
||||||
|
class QuotaExhaustedError extends Error {
|
||||||
|
constructor(primary, sport, attempts) {
|
||||||
|
super(`All providers exhausted for ${primary}/${sport || '*'}. Tried: ${attempts.map((a) => `${a.provider}=${a.reason}`).join('; ')}`);
|
||||||
|
this.name = 'QuotaExhaustedError';
|
||||||
|
this.code = 'QUOTA_EXHAUSTED';
|
||||||
|
this.statusCode = 503;
|
||||||
|
this.primary = primary;
|
||||||
|
this.sport = sport;
|
||||||
|
this.attempts = attempts;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function tryOne(providerId, callbackFn, syncHeadersFrom) {
|
||||||
|
// Optimistic increment — if the call throws we roll back below.
|
||||||
|
// recordCall also evaluates the post-increment threshold; if the
|
||||||
|
// very next call would put us at 95%+, we still execute THIS one
|
||||||
|
// (it returned allowed:true before incrementing) and the NEXT
|
||||||
|
// call will see the block.
|
||||||
|
const status = await quotaTracker.recordCall(providerId);
|
||||||
|
if (!status.allowed) {
|
||||||
|
await quotaTracker.rollback(providerId);
|
||||||
|
return { ok: false, reason: status.reason || 'blocked', status };
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const result = await callbackFn(providerId);
|
||||||
|
// Best-effort header sync — caller signals where the headers
|
||||||
|
// live on the response object. Failure is non-fatal; the
|
||||||
|
// optimistic counter remains.
|
||||||
|
if (typeof syncHeadersFrom === 'function') {
|
||||||
|
try {
|
||||||
|
const headers = syncHeadersFrom(result);
|
||||||
|
if (headers) await quotaTracker.syncFromHeaders(providerId, headers);
|
||||||
|
} catch (e) {
|
||||||
|
console.warn(`[gateway] header sync failed for ${providerId}: ${e.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { ok: true, result, provider: providerId };
|
||||||
|
} catch (err) {
|
||||||
|
await quotaTracker.rollback(providerId);
|
||||||
|
return { ok: false, reason: err && err.message ? err.message : 'error', err };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Invoke `callbackFn` against the primary provider, falling over
|
||||||
|
* to alternatives in the fallback chain if quota is exhausted.
|
||||||
|
*
|
||||||
|
* IMPORTANT: this only retries fallbacks on QUOTA failures, not on
|
||||||
|
* generic upstream errors. A network blip on the primary doesn't
|
||||||
|
* silently shift the entire platform to the fallback (that masks
|
||||||
|
* outages); it surfaces as the adapter's normal error path.
|
||||||
|
*/
|
||||||
|
async function fetch(primaryId, callbackFn, opts = {}) {
|
||||||
|
const {
|
||||||
|
capability,
|
||||||
|
sport,
|
||||||
|
fallbackProviders,
|
||||||
|
syncHeadersFrom,
|
||||||
|
} = opts;
|
||||||
|
|
||||||
|
const attempts = [];
|
||||||
|
const result = await tryOne(primaryId, callbackFn, syncHeadersFrom);
|
||||||
|
if (result.ok) return result.result;
|
||||||
|
|
||||||
|
// Generic adapter error on the primary — propagate, don't shift.
|
||||||
|
if (result.err) {
|
||||||
|
attempts.push({ provider: primaryId, reason: result.reason });
|
||||||
|
throw result.err;
|
||||||
|
}
|
||||||
|
|
||||||
|
attempts.push({ provider: primaryId, reason: result.reason });
|
||||||
|
|
||||||
|
// Build the fallback chain. Caller can override; otherwise derive
|
||||||
|
// from the capability/sport pair in the registry.
|
||||||
|
const chain = Array.isArray(fallbackProviders) && fallbackProviders.length
|
||||||
|
? fallbackProviders
|
||||||
|
: capability
|
||||||
|
? getFallbackChain(capability, sport, primaryId)
|
||||||
|
: [];
|
||||||
|
|
||||||
|
for (const fallbackId of chain) {
|
||||||
|
const fb = await tryOne(fallbackId, callbackFn, syncHeadersFrom);
|
||||||
|
if (fb.ok) {
|
||||||
|
console.log(`[gateway] primary=${primaryId} blocked; succeeded via fallback=${fallbackId}`);
|
||||||
|
return fb.result;
|
||||||
|
}
|
||||||
|
attempts.push({ provider: fallbackId, reason: fb.reason });
|
||||||
|
// Generic error on a fallback → record and continue to the next.
|
||||||
|
// We don't propagate fallback errors because the user only sees
|
||||||
|
// one final response, and the original primary was already
|
||||||
|
// unavailable when we entered this loop.
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new QuotaExhaustedError(primaryId, sport, attempts);
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
fetch,
|
||||||
|
QuotaExhaustedError,
|
||||||
|
};
|
||||||
@@ -0,0 +1,281 @@
|
|||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Quota tracker (Session 20).
|
||||||
|
*
|
||||||
|
* Single source of truth for "how many calls have we made to each
|
||||||
|
* provider this period?" Pure Redis-backed counter — no in-process
|
||||||
|
* memory beyond a one-shot warn dedupe set per period.
|
||||||
|
*
|
||||||
|
* Redis key shape:
|
||||||
|
* quota:{providerId}:{period} → { used: number, limit: number, syncedAt?: iso }
|
||||||
|
* quota_warned:{providerId}:{period} → '1' (dedupes the 80% warn line)
|
||||||
|
*
|
||||||
|
* Period format:
|
||||||
|
* monthly: "YYYY-MM"
|
||||||
|
* daily: "YYYY-MM-DD"
|
||||||
|
* per_minute: "YYYY-MM-DDTHH:MM"
|
||||||
|
*
|
||||||
|
* Degraded mode: when Redis is unavailable, `getQuotaStatus`
|
||||||
|
* returns `{ allowed: true, degraded: true }` — i.e. we FAIL OPEN
|
||||||
|
* (allow the call) rather than block the whole platform on a Redis
|
||||||
|
* outage. The original quota-exhaustion bug we're guarding against
|
||||||
|
* is recoverable (a real upstream 429 stops the bleeding); a
|
||||||
|
* Redis-tied block would make every outage existential.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const { cacheGet, cacheSet, isDegraded } = require('../utils/redis');
|
||||||
|
const { getProvider, THRESHOLDS, getConfiguredProviders } = require('../config/providers');
|
||||||
|
|
||||||
|
function pad(n) {
|
||||||
|
return String(n).padStart(2, '0');
|
||||||
|
}
|
||||||
|
|
||||||
|
function getPeriodKey(providerId, now = new Date()) {
|
||||||
|
const cfg = getProvider(providerId);
|
||||||
|
if (!cfg) return '';
|
||||||
|
switch (cfg.quotaType) {
|
||||||
|
case 'monthly':
|
||||||
|
return `${now.getUTCFullYear()}-${pad(now.getUTCMonth() + 1)}`;
|
||||||
|
case 'daily':
|
||||||
|
return `${now.getUTCFullYear()}-${pad(now.getUTCMonth() + 1)}-${pad(now.getUTCDate())}`;
|
||||||
|
case 'per_minute':
|
||||||
|
return `${now.getUTCFullYear()}-${pad(now.getUTCMonth() + 1)}-${pad(now.getUTCDate())}T${pad(now.getUTCHours())}:${pad(now.getUTCMinutes())}`;
|
||||||
|
default:
|
||||||
|
return `${now.getUTCFullYear()}-${pad(now.getUTCMonth() + 1)}-${pad(now.getUTCDate())}`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Redis TTL for the counter key. Set so the counter naturally
|
||||||
|
* expires at the end of its period — no separate sweep job.
|
||||||
|
*
|
||||||
|
* Monthly: 35 days (slightly past the next reset; the new key
|
||||||
|
* takes over on day-1 of the new month anyway).
|
||||||
|
* Daily: 48 hours.
|
||||||
|
* Per-minute: 2 minutes.
|
||||||
|
*/
|
||||||
|
function getQuotaTTL(providerId) {
|
||||||
|
const cfg = getProvider(providerId);
|
||||||
|
if (!cfg) return 60;
|
||||||
|
switch (cfg.quotaType) {
|
||||||
|
case 'monthly': return 35 * 24 * 60 * 60;
|
||||||
|
case 'daily': return 2 * 24 * 60 * 60;
|
||||||
|
case 'per_minute': return 2 * 60;
|
||||||
|
default: return 24 * 60 * 60;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildKey(providerId, now = new Date()) {
|
||||||
|
return `quota:${providerId}:${getPeriodKey(providerId, now)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildWarnKey(providerId, now = new Date()) {
|
||||||
|
return `quota_warned:${providerId}:${getPeriodKey(providerId, now)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Read the counter without mutating it. Returns the structured
|
||||||
|
* status the admin dashboard renders + the gateway consults.
|
||||||
|
*/
|
||||||
|
async function getQuotaStatus(providerId) {
|
||||||
|
const cfg = getProvider(providerId);
|
||||||
|
if (!cfg) {
|
||||||
|
return {
|
||||||
|
provider: providerId,
|
||||||
|
allowed: false,
|
||||||
|
reason: 'unknown_provider',
|
||||||
|
used: 0, limit: 0, remaining: 0, pct: 1, period: '',
|
||||||
|
quotaType: 'unknown',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (isDegraded()) {
|
||||||
|
// Fail open — the alternative (degrade closed) means a Redis
|
||||||
|
// hiccup takes down every provider call platform-wide.
|
||||||
|
return {
|
||||||
|
provider: providerId, name: cfg.name,
|
||||||
|
allowed: true, degraded: true,
|
||||||
|
used: 0, limit: cfg.quotaLimit, remaining: cfg.quotaLimit, pct: 0,
|
||||||
|
period: getPeriodKey(providerId), quotaType: cfg.quotaType,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const cached = await cacheGet(buildKey(providerId));
|
||||||
|
const used = (cached && typeof cached.used === 'number') ? cached.used : 0;
|
||||||
|
const limit = (cached && typeof cached.limit === 'number') ? cached.limit : cfg.quotaLimit;
|
||||||
|
const pct = limit > 0 ? used / limit : 0;
|
||||||
|
return {
|
||||||
|
provider: providerId,
|
||||||
|
name: cfg.name,
|
||||||
|
allowed: pct < THRESHOLDS.BLOCK_PCT,
|
||||||
|
used, limit,
|
||||||
|
remaining: Math.max(0, limit - used),
|
||||||
|
pct,
|
||||||
|
period: getPeriodKey(providerId),
|
||||||
|
quotaType: cfg.quotaType,
|
||||||
|
syncedAt: cached && cached.syncedAt ? cached.syncedAt : null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Increment the counter by one. Returns the updated status.
|
||||||
|
*
|
||||||
|
* The warning at 80% fires once per period via a separate sentinel
|
||||||
|
* key so the log line doesn't repeat 200 times.
|
||||||
|
*/
|
||||||
|
async function recordCall(providerId) {
|
||||||
|
const cfg = getProvider(providerId);
|
||||||
|
if (!cfg) return { provider: providerId, allowed: false, reason: 'unknown_provider' };
|
||||||
|
if (isDegraded()) return { provider: providerId, allowed: true, degraded: true };
|
||||||
|
|
||||||
|
const key = buildKey(providerId);
|
||||||
|
const cached = await cacheGet(key);
|
||||||
|
const used = (cached && typeof cached.used === 'number') ? cached.used : 0;
|
||||||
|
const limit = (cached && typeof cached.limit === 'number') ? cached.limit : cfg.quotaLimit;
|
||||||
|
const nextUsed = used + 1;
|
||||||
|
const pct = limit > 0 ? nextUsed / limit : 0;
|
||||||
|
|
||||||
|
const payload = { used: nextUsed, limit, updatedAt: new Date().toISOString() };
|
||||||
|
if (cached && cached.syncedAt) payload.syncedAt = cached.syncedAt;
|
||||||
|
await cacheSet(key, payload, getQuotaTTL(providerId));
|
||||||
|
|
||||||
|
if (pct >= THRESHOLDS.WARN_PCT) {
|
||||||
|
const warnKey = buildWarnKey(providerId);
|
||||||
|
const already = await cacheGet(warnKey);
|
||||||
|
if (!already) {
|
||||||
|
console.warn(
|
||||||
|
`[quotaTracker] ${cfg.name} at ${(pct * 100).toFixed(0)}% quota (${nextUsed}/${limit}) for ${getPeriodKey(providerId)}`,
|
||||||
|
);
|
||||||
|
await cacheSet(warnKey, '1', getQuotaTTL(providerId));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
provider: providerId, name: cfg.name,
|
||||||
|
allowed: pct < THRESHOLDS.BLOCK_PCT,
|
||||||
|
used: nextUsed, limit,
|
||||||
|
remaining: Math.max(0, limit - nextUsed),
|
||||||
|
pct,
|
||||||
|
period: getPeriodKey(providerId),
|
||||||
|
quotaType: cfg.quotaType,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Decrement after a failed call. The optimistic recordCall+rollback
|
||||||
|
* pattern means a thread-unsafe race could overshoot by N during a
|
||||||
|
* burst — acceptable for our scale (single-instance Express,
|
||||||
|
* 5–10 concurrent calls peak). Atomic INCR would require a Lua
|
||||||
|
* script and the gain is marginal.
|
||||||
|
*/
|
||||||
|
async function rollback(providerId) {
|
||||||
|
const cfg = getProvider(providerId);
|
||||||
|
if (!cfg || isDegraded()) return;
|
||||||
|
const key = buildKey(providerId);
|
||||||
|
const cached = await cacheGet(key);
|
||||||
|
if (!cached || typeof cached.used !== 'number') return;
|
||||||
|
const nextUsed = Math.max(0, cached.used - 1);
|
||||||
|
await cacheSet(
|
||||||
|
key,
|
||||||
|
{ ...cached, used: nextUsed, updatedAt: new Date().toISOString() },
|
||||||
|
getQuotaTTL(providerId),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sync from upstream response headers. odds-api returns
|
||||||
|
* `x-requests-used` + `x-requests-remaining`; this becomes the
|
||||||
|
* truth source for that provider's counter.
|
||||||
|
*
|
||||||
|
* Header presence varies — we look at common spellings.
|
||||||
|
*/
|
||||||
|
async function syncFromHeaders(providerId, headers) {
|
||||||
|
if (!headers || isDegraded()) return null;
|
||||||
|
const cfg = getProvider(providerId);
|
||||||
|
if (!cfg) return null;
|
||||||
|
|
||||||
|
// Normalize header lookup (axios returns lowercase, fetch keeps case).
|
||||||
|
const h = {};
|
||||||
|
for (const k of Object.keys(headers)) h[String(k).toLowerCase()] = headers[k];
|
||||||
|
|
||||||
|
const usedHdr = h['x-requests-used'] ?? h['x-quota-used'] ?? h['x-ratelimit-used'];
|
||||||
|
const remainingHdr = h['x-requests-remaining'] ?? h['x-quota-remaining'] ?? h['x-ratelimit-remaining'];
|
||||||
|
const limitHdr = h['x-quota-limit'] ?? h['x-ratelimit-limit'];
|
||||||
|
|
||||||
|
const used = Number.parseInt(usedHdr, 10);
|
||||||
|
const remaining = Number.parseInt(remainingHdr, 10);
|
||||||
|
const limitFromHdr = Number.parseInt(limitHdr, 10);
|
||||||
|
if (!Number.isFinite(used) && !Number.isFinite(remaining)) return null;
|
||||||
|
|
||||||
|
const limit = Number.isFinite(limitFromHdr)
|
||||||
|
? limitFromHdr
|
||||||
|
: (Number.isFinite(used) && Number.isFinite(remaining))
|
||||||
|
? used + remaining
|
||||||
|
: cfg.quotaLimit;
|
||||||
|
const resolvedUsed = Number.isFinite(used)
|
||||||
|
? used
|
||||||
|
: (Number.isFinite(remaining) ? Math.max(0, limit - remaining) : 0);
|
||||||
|
|
||||||
|
const payload = {
|
||||||
|
used: resolvedUsed,
|
||||||
|
limit,
|
||||||
|
syncedAt: new Date().toISOString(),
|
||||||
|
source: 'headers',
|
||||||
|
};
|
||||||
|
await cacheSet(buildKey(providerId), payload, getQuotaTTL(providerId));
|
||||||
|
return payload;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Status snapshot for every configured provider — admin dashboard
|
||||||
|
* tile, /api/internal/quota endpoint.
|
||||||
|
*/
|
||||||
|
async function getAllQuotaStatuses() {
|
||||||
|
const configured = getConfiguredProviders();
|
||||||
|
return Promise.all(configured.map((p) => getQuotaStatus(p.id)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Frequency-scaling helper for schedulers. Returns the wait
|
||||||
|
* interval (ms) the caller SHOULD use before its next provider
|
||||||
|
* hit, or null when the quota is exhausted.
|
||||||
|
*
|
||||||
|
* The discrete steps match the Session 20 spec:
|
||||||
|
* <50% → 5 min (full speed)
|
||||||
|
* <80% → 15 min
|
||||||
|
* <95% → 30 min
|
||||||
|
* ≥95% → null (stop)
|
||||||
|
*/
|
||||||
|
function getTickInterval(pct) {
|
||||||
|
if (!Number.isFinite(pct)) return 5 * 60 * 1000;
|
||||||
|
if (pct >= THRESHOLDS.BLOCK_PCT) return null;
|
||||||
|
if (pct >= THRESHOLDS.WARN_PCT) return 30 * 60 * 1000;
|
||||||
|
if (pct >= 0.50) return 15 * 60 * 1000;
|
||||||
|
return 5 * 60 * 1000;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Composite helper for schedulers: should this poller wait or fire
|
||||||
|
* right now? Convenience over `getQuotaStatus + getTickInterval`.
|
||||||
|
*/
|
||||||
|
async function shouldThrottle(providerId) {
|
||||||
|
const status = await getQuotaStatus(providerId);
|
||||||
|
const interval = getTickInterval(status.pct);
|
||||||
|
return { allowed: status.allowed, interval, status };
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
getPeriodKey,
|
||||||
|
buildKey,
|
||||||
|
buildWarnKey,
|
||||||
|
getQuotaStatus,
|
||||||
|
getAllQuotaStatuses,
|
||||||
|
recordCall,
|
||||||
|
rollback,
|
||||||
|
syncFromHeaders,
|
||||||
|
getTickInterval,
|
||||||
|
shouldThrottle,
|
||||||
|
// Exposed for tests that need to reach the threshold constants
|
||||||
|
// without re-importing the providers module.
|
||||||
|
THRESHOLDS,
|
||||||
|
};
|
||||||
@@ -14,6 +14,11 @@ jest.mock('../../scripts/tank01-prefetch', () => ({
|
|||||||
}));
|
}));
|
||||||
const tank01Prefetch = require('../../scripts/tank01-prefetch');
|
const tank01Prefetch = require('../../scripts/tank01-prefetch');
|
||||||
|
|
||||||
|
jest.mock('../../src/services/quotaTracker', () => ({
|
||||||
|
getAllQuotaStatuses: jest.fn(),
|
||||||
|
}));
|
||||||
|
const quotaTracker = require('../../src/services/quotaTracker');
|
||||||
|
|
||||||
beforeEach(() => {
|
beforeEach(() => {
|
||||||
jest.resetAllMocks();
|
jest.resetAllMocks();
|
||||||
process.env.VYNDR_INTERNAL_KEY = 'test-internal-key-9999';
|
process.env.VYNDR_INTERNAL_KEY = 'test-internal-key-9999';
|
||||||
@@ -98,3 +103,37 @@ describe('POST /api/internal/prefetch/tank01', () => {
|
|||||||
expect(argv).toContain('--sports=mlb');
|
expect(argv).toContain('--sports=mlb');
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('GET /api/internal/quota (Session 20)', () => {
|
||||||
|
test('rejects without the internal key', async () => {
|
||||||
|
const app = mountApp();
|
||||||
|
const res = await request(app).get('/api/internal/quota');
|
||||||
|
expect(res.status).toBe(401);
|
||||||
|
expect(quotaTracker.getAllQuotaStatuses).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('returns the per-provider snapshot when keyed', async () => {
|
||||||
|
const fakeSnapshot = [
|
||||||
|
{ provider: 'odds-api', name: 'The Odds API', used: 487, limit: 500, remaining: 13, pct: 0.974, allowed: false, period: '2026-06', quotaType: 'monthly' },
|
||||||
|
{ provider: 'tank01', name: 'Tank01', used: 30, limit: 1000, remaining: 970, pct: 0.03, allowed: true, period: '2026-06', quotaType: 'monthly' },
|
||||||
|
];
|
||||||
|
quotaTracker.getAllQuotaStatuses.mockResolvedValueOnce(fakeSnapshot);
|
||||||
|
|
||||||
|
const app = mountApp();
|
||||||
|
const res = await request(app)
|
||||||
|
.get('/api/internal/quota')
|
||||||
|
.set('x-internal-key', 'test-internal-key-9999');
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
expect(res.body).toEqual({ ok: true, providers: fakeSnapshot });
|
||||||
|
});
|
||||||
|
|
||||||
|
test('returns 500 with the underlying error when the tracker throws', async () => {
|
||||||
|
quotaTracker.getAllQuotaStatuses.mockRejectedValueOnce(new Error('redis down'));
|
||||||
|
const app = mountApp();
|
||||||
|
const res = await request(app)
|
||||||
|
.get('/api/internal/quota')
|
||||||
|
.set('x-internal-key', 'test-internal-key-9999');
|
||||||
|
expect(res.status).toBe(500);
|
||||||
|
expect(res.body).toEqual({ ok: false, error: 'redis down' });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -10,6 +10,13 @@ const mockRedis = {
|
|||||||
};
|
};
|
||||||
jest.mock('../../src/utils/redis', () => ({
|
jest.mock('../../src/utils/redis', () => ({
|
||||||
getRedisClient: () => mockRedis,
|
getRedisClient: () => mockRedis,
|
||||||
|
// Session 20 — provider gateway pulls cacheGet/cacheSet/isDegraded.
|
||||||
|
// Degraded mode lets every call through and skips Redis writes, so
|
||||||
|
// the existing axios-mock assertions stay accurate.
|
||||||
|
cacheGet: jest.fn(async () => null),
|
||||||
|
cacheSet: jest.fn(async () => true),
|
||||||
|
cacheDel: jest.fn(async () => true),
|
||||||
|
isDegraded: jest.fn(() => true),
|
||||||
}));
|
}));
|
||||||
|
|
||||||
// Mock axios
|
// Mock axios
|
||||||
|
|||||||
@@ -10,6 +10,15 @@ const mockRedis = {
|
|||||||
};
|
};
|
||||||
jest.mock('../../src/utils/redis', () => ({
|
jest.mock('../../src/utils/redis', () => ({
|
||||||
getRedisClient: () => mockRedis,
|
getRedisClient: () => mockRedis,
|
||||||
|
// Session 20 — the provider gateway + quotaTracker pull from
|
||||||
|
// cacheGet/cacheSet/isDegraded. We surface them as degraded-mode
|
||||||
|
// no-ops here so the gateway fails OPEN in tests (lets every call
|
||||||
|
// through without touching Redis), which preserves the legacy
|
||||||
|
// axios-mock-driven assertions in this file.
|
||||||
|
cacheGet: jest.fn(async () => null),
|
||||||
|
cacheSet: jest.fn(async () => true),
|
||||||
|
cacheDel: jest.fn(async () => true),
|
||||||
|
isDegraded: jest.fn(() => true),
|
||||||
}));
|
}));
|
||||||
|
|
||||||
// Mock axios
|
// Mock axios
|
||||||
|
|||||||
@@ -0,0 +1,156 @@
|
|||||||
|
// Provider gateway (Session 20).
|
||||||
|
//
|
||||||
|
// Covers: happy path (callback invoked once), quota-block fallback
|
||||||
|
// (primary blocked → walks fallback chain), full exhaustion
|
||||||
|
// (QuotaExhaustedError), upstream errors propagate without
|
||||||
|
// shifting, and header sync is invoked on success.
|
||||||
|
|
||||||
|
jest.mock('../../src/services/quotaTracker', () => {
|
||||||
|
// The mock keeps a per-test counter so we can drive different
|
||||||
|
// providers into different quota states without writing to Redis.
|
||||||
|
const state = new Map();
|
||||||
|
const setStatus = (providerId, allowed, extra = {}) => {
|
||||||
|
state.set(providerId, { allowed, used: extra.used || 0, limit: 500, ...extra });
|
||||||
|
};
|
||||||
|
return {
|
||||||
|
recordCall: jest.fn(async (providerId) => {
|
||||||
|
const s = state.get(providerId) || { allowed: true, used: 0, limit: 500 };
|
||||||
|
if (!s.allowed) return { provider: providerId, allowed: false, reason: s.reason || 'blocked' };
|
||||||
|
return { provider: providerId, allowed: true, used: s.used + 1, limit: s.limit };
|
||||||
|
}),
|
||||||
|
rollback: jest.fn(async () => {}),
|
||||||
|
syncFromHeaders: jest.fn(async () => null),
|
||||||
|
__state: state,
|
||||||
|
__setStatus: setStatus,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
jest.mock('../../src/config/providers', () => {
|
||||||
|
const PROVIDERS = {
|
||||||
|
'odds-api': { name: 'The Odds API', capabilities: ['odds'], sports: ['nba'], envKey: 'ODDS_API_KEY', priority: 1 },
|
||||||
|
'oddspapi': { name: 'ODDSPAPI', capabilities: ['odds'], sports: ['nba'], envKey: 'ODDSPAPI_KEY', priority: 2 },
|
||||||
|
'parlayapi': { name: 'ParlayAPI', capabilities: ['odds'], sports: ['nba'], envKey: 'PARLAYAPI_KEY', priority: 3 },
|
||||||
|
};
|
||||||
|
return {
|
||||||
|
PROVIDERS,
|
||||||
|
getProvider: (id) => PROVIDERS[id] || null,
|
||||||
|
getFallbackChain: (capability, sport, excludeId) =>
|
||||||
|
Object.entries(PROVIDERS)
|
||||||
|
.filter(([id, cfg]) =>
|
||||||
|
id !== excludeId &&
|
||||||
|
cfg.capabilities.includes(capability) &&
|
||||||
|
(!sport || cfg.sports.includes(sport)) &&
|
||||||
|
!!process.env[cfg.envKey],
|
||||||
|
)
|
||||||
|
.sort((a, b) => a[1].priority - b[1].priority)
|
||||||
|
.map(([id]) => id),
|
||||||
|
listProviderIds: () => Object.keys(PROVIDERS),
|
||||||
|
getConfiguredProviders: () => Object.keys(PROVIDERS).filter((k) => !!process.env[PROVIDERS[k].envKey]),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
const tracker = require('../../src/services/quotaTracker');
|
||||||
|
const gateway = require('../../src/services/providerGateway');
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
tracker.__state.clear();
|
||||||
|
tracker.recordCall.mockClear();
|
||||||
|
tracker.rollback.mockClear();
|
||||||
|
tracker.syncFromHeaders.mockClear();
|
||||||
|
process.env.ODDS_API_KEY = 'k1';
|
||||||
|
process.env.ODDSPAPI_KEY = 'k2';
|
||||||
|
process.env.PARLAYAPI_KEY = 'k3';
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('gateway.fetch — happy path', () => {
|
||||||
|
test('invokes callback once and returns its result', async () => {
|
||||||
|
const cb = jest.fn(async () => ({ ok: true }));
|
||||||
|
const result = await gateway.fetch('odds-api', cb, { capability: 'odds', sport: 'nba' });
|
||||||
|
expect(result).toEqual({ ok: true });
|
||||||
|
expect(cb).toHaveBeenCalledTimes(1);
|
||||||
|
expect(cb).toHaveBeenCalledWith('odds-api');
|
||||||
|
expect(tracker.rollback).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('invokes syncHeadersFrom on success', async () => {
|
||||||
|
const cb = jest.fn(async () => ({ headers: { 'x-requests-remaining': '100' } }));
|
||||||
|
await gateway.fetch('odds-api', cb, {
|
||||||
|
capability: 'odds', sport: 'nba',
|
||||||
|
syncHeadersFrom: (r) => r.headers,
|
||||||
|
});
|
||||||
|
expect(tracker.syncFromHeaders).toHaveBeenCalledWith('odds-api', { 'x-requests-remaining': '100' });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('gateway.fetch — quota fallback', () => {
|
||||||
|
test('walks the chain when the primary is blocked', async () => {
|
||||||
|
tracker.__setStatus('odds-api', false);
|
||||||
|
tracker.__setStatus('oddspapi', true);
|
||||||
|
const cb = jest.fn(async (provider) => ({ ok: true, from: provider }));
|
||||||
|
const result = await gateway.fetch('odds-api', cb, { capability: 'odds', sport: 'nba' });
|
||||||
|
expect(result.from).toBe('oddspapi');
|
||||||
|
expect(cb).toHaveBeenCalledTimes(1); // primary skipped pre-call
|
||||||
|
expect(cb).toHaveBeenCalledWith('oddspapi');
|
||||||
|
expect(tracker.rollback).toHaveBeenCalledWith('odds-api'); // rolled back the optimistic increment
|
||||||
|
});
|
||||||
|
|
||||||
|
test('skips through multiple blocked providers to the next allowed', async () => {
|
||||||
|
tracker.__setStatus('odds-api', false);
|
||||||
|
tracker.__setStatus('oddspapi', false);
|
||||||
|
tracker.__setStatus('parlayapi', true);
|
||||||
|
const cb = jest.fn(async (provider) => ({ from: provider }));
|
||||||
|
const result = await gateway.fetch('odds-api', cb, { capability: 'odds', sport: 'nba' });
|
||||||
|
expect(result.from).toBe('parlayapi');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('honors explicit fallbackProviders over derived chain', async () => {
|
||||||
|
tracker.__setStatus('odds-api', false);
|
||||||
|
tracker.__setStatus('parlayapi', true);
|
||||||
|
const cb = jest.fn(async (provider) => ({ from: provider }));
|
||||||
|
const result = await gateway.fetch('odds-api', cb, {
|
||||||
|
capability: 'odds', sport: 'nba',
|
||||||
|
fallbackProviders: ['parlayapi'], // skip oddspapi
|
||||||
|
});
|
||||||
|
expect(result.from).toBe('parlayapi');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('gateway.fetch — full exhaustion', () => {
|
||||||
|
test('throws QuotaExhaustedError when every provider is blocked', async () => {
|
||||||
|
tracker.__setStatus('odds-api', false);
|
||||||
|
tracker.__setStatus('oddspapi', false);
|
||||||
|
tracker.__setStatus('parlayapi', false);
|
||||||
|
const cb = jest.fn();
|
||||||
|
await expect(gateway.fetch('odds-api', cb, { capability: 'odds', sport: 'nba' }))
|
||||||
|
.rejects.toMatchObject({
|
||||||
|
name: 'QuotaExhaustedError',
|
||||||
|
code: 'QUOTA_EXHAUSTED',
|
||||||
|
statusCode: 503,
|
||||||
|
});
|
||||||
|
expect(cb).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('reports the primary and the attempt chain on the error', async () => {
|
||||||
|
tracker.__setStatus('odds-api', false);
|
||||||
|
tracker.__setStatus('oddspapi', false);
|
||||||
|
tracker.__setStatus('parlayapi', false);
|
||||||
|
try {
|
||||||
|
await gateway.fetch('odds-api', jest.fn(), { capability: 'odds', sport: 'nba' });
|
||||||
|
throw new Error('should have thrown');
|
||||||
|
} catch (err) {
|
||||||
|
expect(err.primary).toBe('odds-api');
|
||||||
|
expect(err.attempts.map((a) => a.provider)).toEqual(['odds-api', 'oddspapi', 'parlayapi']);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('gateway.fetch — upstream errors', () => {
|
||||||
|
test('propagates the adapter error without falling over', async () => {
|
||||||
|
const adapterErr = new Error('upstream 502');
|
||||||
|
const cb = jest.fn(async () => { throw adapterErr; });
|
||||||
|
await expect(gateway.fetch('odds-api', cb, { capability: 'odds', sport: 'nba' }))
|
||||||
|
.rejects.toBe(adapterErr);
|
||||||
|
// The increment is rolled back so we don't burn quota on a failed call.
|
||||||
|
expect(tracker.rollback).toHaveBeenCalledWith('odds-api');
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,223 @@
|
|||||||
|
// Quota tracker (Session 20).
|
||||||
|
//
|
||||||
|
// Pins behavior so the gateway + scheduler can rely on it: period
|
||||||
|
// keys reflect the configured quotaType, recordCall advances the
|
||||||
|
// counter, syncFromHeaders is the truth-source override, the 80%
|
||||||
|
// warning fires once per period, and the 95% threshold flips
|
||||||
|
// `allowed` from true to false. Redis is mocked so the tests don't
|
||||||
|
// require a live server.
|
||||||
|
|
||||||
|
jest.mock('../../src/utils/redis', () => {
|
||||||
|
const store = new Map();
|
||||||
|
return {
|
||||||
|
cacheGet: jest.fn(async (key) => {
|
||||||
|
if (!store.has(key)) return null;
|
||||||
|
return store.get(key);
|
||||||
|
}),
|
||||||
|
cacheSet: jest.fn(async (key, value) => {
|
||||||
|
// Mirror the real helper: it stores the value as JSON; cacheGet
|
||||||
|
// returns the parsed shape. We skip serialization here and just
|
||||||
|
// hold the object directly — same observed behavior.
|
||||||
|
store.set(key, value);
|
||||||
|
return true;
|
||||||
|
}),
|
||||||
|
cacheDel: jest.fn(async (key) => {
|
||||||
|
store.delete(key);
|
||||||
|
return true;
|
||||||
|
}),
|
||||||
|
isDegraded: jest.fn(() => false),
|
||||||
|
__store: store,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
const redis = require('../../src/utils/redis');
|
||||||
|
const tracker = require('../../src/services/quotaTracker');
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
redis.__store.clear();
|
||||||
|
redis.cacheGet.mockClear();
|
||||||
|
redis.cacheSet.mockClear();
|
||||||
|
redis.cacheDel.mockClear();
|
||||||
|
redis.isDegraded.mockReturnValue(false);
|
||||||
|
process.env.ODDS_API_KEY = 'test-odds-key';
|
||||||
|
process.env.RAPID_API_KEY = 'test-tank01-key';
|
||||||
|
process.env.API_FOOTBALL_KEY = 'test-apifoot-key';
|
||||||
|
process.env.FOOTBALL_DATA_API_KEY = 'test-fd-key';
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker.getPeriodKey', () => {
|
||||||
|
test('monthly produces YYYY-MM', () => {
|
||||||
|
const key = tracker.getPeriodKey('odds-api', new Date(Date.UTC(2026, 5, 12)));
|
||||||
|
expect(key).toBe('2026-06');
|
||||||
|
});
|
||||||
|
test('daily produces YYYY-MM-DD', () => {
|
||||||
|
const key = tracker.getPeriodKey('api-football', new Date(Date.UTC(2026, 5, 12)));
|
||||||
|
expect(key).toBe('2026-06-12');
|
||||||
|
});
|
||||||
|
test('per_minute produces YYYY-MM-DDTHH:MM', () => {
|
||||||
|
const key = tracker.getPeriodKey('football-data', new Date(Date.UTC(2026, 5, 12, 15, 30)));
|
||||||
|
expect(key).toBe('2026-06-12T15:30');
|
||||||
|
});
|
||||||
|
test('unknown provider returns empty string', () => {
|
||||||
|
expect(tracker.getPeriodKey('made-up')).toBe('');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker.recordCall', () => {
|
||||||
|
test('counts up from zero, exposes remaining', async () => {
|
||||||
|
const a = await tracker.recordCall('odds-api');
|
||||||
|
expect(a.allowed).toBe(true);
|
||||||
|
expect(a.used).toBe(1);
|
||||||
|
expect(a.remaining).toBe(499);
|
||||||
|
const b = await tracker.recordCall('odds-api');
|
||||||
|
expect(b.used).toBe(2);
|
||||||
|
expect(b.remaining).toBe(498);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('returns allowed:false once pct hits 95%', async () => {
|
||||||
|
// Seed the counter at 95% directly through syncFromHeaders so we
|
||||||
|
// don't have to fire 475 recordCall iterations.
|
||||||
|
await tracker.syncFromHeaders('odds-api', {
|
||||||
|
'x-requests-used': '475',
|
||||||
|
'x-requests-remaining': '25',
|
||||||
|
});
|
||||||
|
const status = await tracker.getQuotaStatus('odds-api');
|
||||||
|
expect(status.used).toBe(475);
|
||||||
|
expect(status.pct).toBeCloseTo(0.95);
|
||||||
|
expect(status.allowed).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('logs the WARN line exactly once at 80%', async () => {
|
||||||
|
const warnSpy = jest.spyOn(console, 'warn').mockImplementation(() => {});
|
||||||
|
// Seed 399/500 = 79.8% — first recordCall takes us to 400/500 = 80%
|
||||||
|
await tracker.syncFromHeaders('odds-api', {
|
||||||
|
'x-requests-used': '399',
|
||||||
|
'x-requests-remaining': '101',
|
||||||
|
});
|
||||||
|
await tracker.recordCall('odds-api');
|
||||||
|
await tracker.recordCall('odds-api');
|
||||||
|
await tracker.recordCall('odds-api');
|
||||||
|
const warnings = warnSpy.mock.calls.map((c) => c.join(' ')).filter((m) => /quotaTracker/.test(m));
|
||||||
|
expect(warnings.length).toBe(1);
|
||||||
|
expect(warnings[0]).toMatch(/Odds API/);
|
||||||
|
warnSpy.mockRestore();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('unknown provider → allowed:false', async () => {
|
||||||
|
const r = await tracker.recordCall('does-not-exist');
|
||||||
|
expect(r.allowed).toBe(false);
|
||||||
|
expect(r.reason).toBe('unknown_provider');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker.rollback', () => {
|
||||||
|
test('decrements the counter without going below zero', async () => {
|
||||||
|
await tracker.recordCall('tank01');
|
||||||
|
await tracker.recordCall('tank01');
|
||||||
|
await tracker.rollback('tank01');
|
||||||
|
let status = await tracker.getQuotaStatus('tank01');
|
||||||
|
expect(status.used).toBe(1);
|
||||||
|
await tracker.rollback('tank01');
|
||||||
|
await tracker.rollback('tank01');
|
||||||
|
status = await tracker.getQuotaStatus('tank01');
|
||||||
|
expect(status.used).toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker.syncFromHeaders', () => {
|
||||||
|
test('odds-api headers overwrite the counter (truth source)', async () => {
|
||||||
|
await tracker.recordCall('odds-api');
|
||||||
|
await tracker.recordCall('odds-api');
|
||||||
|
// Local counter says 2; upstream says 50 — upstream wins.
|
||||||
|
const synced = await tracker.syncFromHeaders('odds-api', {
|
||||||
|
'x-requests-used': '50',
|
||||||
|
'x-requests-remaining': '450',
|
||||||
|
});
|
||||||
|
expect(synced.used).toBe(50);
|
||||||
|
expect(synced.limit).toBe(500);
|
||||||
|
const status = await tracker.getQuotaStatus('odds-api');
|
||||||
|
expect(status.used).toBe(50);
|
||||||
|
expect(status.syncedAt).toBeTruthy();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('infers used from remaining + limit when used header absent', async () => {
|
||||||
|
const synced = await tracker.syncFromHeaders('odds-api', {
|
||||||
|
'x-requests-remaining': '120',
|
||||||
|
'x-quota-limit': '500',
|
||||||
|
});
|
||||||
|
expect(synced.used).toBe(380);
|
||||||
|
expect(synced.limit).toBe(500);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('returns null when no usable headers present', async () => {
|
||||||
|
const synced = await tracker.syncFromHeaders('odds-api', { 'content-type': 'application/json' });
|
||||||
|
expect(synced).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker.getTickInterval (scheduler step function)', () => {
|
||||||
|
test('returns 5min under 50%', () => {
|
||||||
|
expect(tracker.getTickInterval(0)).toBe(5 * 60 * 1000);
|
||||||
|
expect(tracker.getTickInterval(0.49)).toBe(5 * 60 * 1000);
|
||||||
|
});
|
||||||
|
test('returns 15min at 50%–79%', () => {
|
||||||
|
expect(tracker.getTickInterval(0.5)).toBe(15 * 60 * 1000);
|
||||||
|
expect(tracker.getTickInterval(0.79)).toBe(15 * 60 * 1000);
|
||||||
|
});
|
||||||
|
test('returns 30min at 80%–94%', () => {
|
||||||
|
expect(tracker.getTickInterval(0.8)).toBe(30 * 60 * 1000);
|
||||||
|
expect(tracker.getTickInterval(0.94)).toBe(30 * 60 * 1000);
|
||||||
|
});
|
||||||
|
test('returns null at >=95% (stop)', () => {
|
||||||
|
expect(tracker.getTickInterval(0.95)).toBeNull();
|
||||||
|
expect(tracker.getTickInterval(1.0)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker.shouldThrottle', () => {
|
||||||
|
test('returns allowed:false and interval:null at exhaustion', async () => {
|
||||||
|
await tracker.syncFromHeaders('odds-api', {
|
||||||
|
'x-requests-used': '500',
|
||||||
|
'x-requests-remaining': '0',
|
||||||
|
});
|
||||||
|
const out = await tracker.shouldThrottle('odds-api');
|
||||||
|
expect(out.allowed).toBe(false);
|
||||||
|
expect(out.interval).toBeNull();
|
||||||
|
});
|
||||||
|
test('allowed and 5min interval when healthy', async () => {
|
||||||
|
const out = await tracker.shouldThrottle('odds-api');
|
||||||
|
expect(out.allowed).toBe(true);
|
||||||
|
expect(out.interval).toBe(5 * 60 * 1000);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker degraded-mode fail-open', () => {
|
||||||
|
test('returns allowed:true degraded:true when Redis is down', async () => {
|
||||||
|
redis.isDegraded.mockReturnValue(true);
|
||||||
|
const status = await tracker.getQuotaStatus('odds-api');
|
||||||
|
expect(status.allowed).toBe(true);
|
||||||
|
expect(status.degraded).toBe(true);
|
||||||
|
});
|
||||||
|
test('recordCall is a no-op when degraded', async () => {
|
||||||
|
redis.isDegraded.mockReturnValue(true);
|
||||||
|
const r = await tracker.recordCall('odds-api');
|
||||||
|
expect(r.allowed).toBe(true);
|
||||||
|
expect(r.degraded).toBe(true);
|
||||||
|
expect(redis.cacheSet).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('quotaTracker.getAllQuotaStatuses', () => {
|
||||||
|
test('returns one entry per configured provider', async () => {
|
||||||
|
const statuses = await tracker.getAllQuotaStatuses();
|
||||||
|
const ids = statuses.map((s) => s.provider).sort();
|
||||||
|
// ODDS_API_KEY, RAPID_API_KEY, API_FOOTBALL_KEY, FOOTBALL_DATA_API_KEY
|
||||||
|
// are set in beforeEach; ODDSPAPI_KEY and PARLAYAPI_KEY are not.
|
||||||
|
expect(ids).toContain('odds-api');
|
||||||
|
expect(ids).toContain('tank01');
|
||||||
|
expect(ids).toContain('api-football');
|
||||||
|
expect(ids).toContain('football-data');
|
||||||
|
expect(ids).not.toContain('oddspapi');
|
||||||
|
expect(ids).not.toContain('parlayapi');
|
||||||
|
});
|
||||||
|
});
|
||||||
+1
-1
File diff suppressed because one or more lines are too long
@@ -36,6 +36,18 @@ interface AdminStats {
|
|||||||
sports: Array<{ sport: string; status: 'ok' | 'error' | 'empty'; quota?: number | null; props?: number; error?: string }>;
|
sports: Array<{ sport: string; status: 'ok' | 'error' | 'empty'; quota?: number | null; props?: number; error?: string }>;
|
||||||
odds_quota_remaining: number | null;
|
odds_quota_remaining: number | null;
|
||||||
};
|
};
|
||||||
|
provider_quotas?: Array<{
|
||||||
|
provider: string;
|
||||||
|
name?: string;
|
||||||
|
used: number;
|
||||||
|
limit: number;
|
||||||
|
remaining: number;
|
||||||
|
pct: number;
|
||||||
|
period: string;
|
||||||
|
quotaType: string;
|
||||||
|
allowed: boolean;
|
||||||
|
degraded?: boolean;
|
||||||
|
}>;
|
||||||
notes: string[];
|
notes: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -255,6 +267,62 @@ export default function AdminPage() {
|
|||||||
</table>
|
</table>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
{/* Session 20 — Provider Quotas. Pulled from
|
||||||
|
/api/internal/quota; rendered as a per-provider table with
|
||||||
|
a usage bar + status indicator. When the array is empty,
|
||||||
|
the section auto-hides (likely VYNDR_INTERNAL_KEY unset
|
||||||
|
on the Next.js side — surfaced in the notes section). */}
|
||||||
|
{!!stats?.provider_quotas?.length && (
|
||||||
|
<section style={{ ...cellStyle, marginTop: 24 }}>
|
||||||
|
<h2 style={{ fontSize: 14, fontWeight: 700, marginBottom: 14, letterSpacing: '-0.01em' }}>Provider quotas</h2>
|
||||||
|
<table style={{ width: '100%', borderCollapse: 'collapse', fontSize: 13 }}>
|
||||||
|
<thead>
|
||||||
|
<tr style={{ textAlign: 'left', color: 'var(--text-tertiary, #6B6B7B)', fontSize: 10, textTransform: 'uppercase', letterSpacing: '0.08em' }}>
|
||||||
|
<th style={{ padding: '6px 0' }}>Provider</th>
|
||||||
|
<th style={{ padding: '6px 0' }}>Used</th>
|
||||||
|
<th style={{ padding: '6px 0' }}>Type</th>
|
||||||
|
<th style={{ padding: '6px 0', textAlign: 'right' }}>Status</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{stats.provider_quotas.map((p) => {
|
||||||
|
const pctNum = Math.round((p.pct || 0) * 100);
|
||||||
|
const color = !p.allowed
|
||||||
|
? 'var(--grade-d, #FF6B6B)'
|
||||||
|
: pctNum >= 80
|
||||||
|
? 'var(--grade-c, #FFD93D)'
|
||||||
|
: 'var(--grade-a, #00D4A0)';
|
||||||
|
const indicator = !p.allowed
|
||||||
|
? `❌ BLOCKED ${pctNum}%`
|
||||||
|
: pctNum >= 80
|
||||||
|
? `⚠️ ${pctNum}%`
|
||||||
|
: `✅ ${pctNum}%`;
|
||||||
|
return (
|
||||||
|
<tr key={p.provider} style={{ borderBottom: '1px solid var(--border, #1A1A24)' }}>
|
||||||
|
<td style={{ padding: '8px 0', color: 'var(--text-0, #F0F0F5)' }}>
|
||||||
|
<div style={{ fontWeight: 600 }}>{p.name || p.provider}</div>
|
||||||
|
<div className="mono" style={{ fontSize: 10, color: 'var(--text-tertiary, #6B6B7B)', marginTop: 2 }}>{p.period}</div>
|
||||||
|
</td>
|
||||||
|
<td className="mono" style={{ padding: '8px 0', color: 'var(--text-0, #F0F0F5)', fontVariantNumeric: 'tabular-nums' }}>
|
||||||
|
<div>{p.used}/{p.limit}</div>
|
||||||
|
<div style={{ marginTop: 4, background: 'var(--bg-2, #15151F)', height: 4, borderRadius: 2, overflow: 'hidden', width: 90 }}>
|
||||||
|
<div style={{ width: `${Math.min(100, Math.max(2, pctNum))}%`, height: '100%', background: color }} />
|
||||||
|
</div>
|
||||||
|
</td>
|
||||||
|
<td className="mono" style={{ padding: '8px 0', color: 'var(--text-secondary, #8A8A9A)', fontSize: 11, textTransform: 'uppercase' }}>
|
||||||
|
{p.quotaType}
|
||||||
|
</td>
|
||||||
|
<td className="mono" style={{ padding: '8px 0', textAlign: 'right', color, fontSize: 12 }}>
|
||||||
|
{indicator}{p.degraded ? ' (degraded)' : ''}
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</section>
|
||||||
|
)}
|
||||||
|
|
||||||
{!!stats?.notes?.length && (
|
{!!stats?.notes?.length && (
|
||||||
<section style={{ ...cellStyle, marginTop: 24, fontSize: 12, color: 'var(--grade-c, #FFD93D)' }}>
|
<section style={{ ...cellStyle, marginTop: 24, fontSize: 12, color: 'var(--grade-c, #FFD93D)' }}>
|
||||||
<h2 style={{ fontSize: 12, fontWeight: 700, marginBottom: 8, letterSpacing: '-0.01em' }}>Query notes</h2>
|
<h2 style={{ fontSize: 12, fontWeight: 700, marginBottom: 8, letterSpacing: '-0.01em' }}>Query notes</h2>
|
||||||
|
|||||||
@@ -48,6 +48,21 @@ interface AdminStats {
|
|||||||
sports: Array<{ sport: string; status: 'ok' | 'error' | 'empty'; quota?: number | null; props?: number; error?: string }>;
|
sports: Array<{ sport: string; status: 'ok' | 'error' | 'empty'; quota?: number | null; props?: number; error?: string }>;
|
||||||
odds_quota_remaining: number | null;
|
odds_quota_remaining: number | null;
|
||||||
};
|
};
|
||||||
|
// Session 20 — per-provider quota snapshot. Pulled through the
|
||||||
|
// internal /quota endpoint so the admin page sees the same view
|
||||||
|
// the gateway makes routing decisions against.
|
||||||
|
provider_quotas: Array<{
|
||||||
|
provider: string;
|
||||||
|
name?: string;
|
||||||
|
used: number;
|
||||||
|
limit: number;
|
||||||
|
remaining: number;
|
||||||
|
pct: number;
|
||||||
|
period: string;
|
||||||
|
quotaType: string;
|
||||||
|
allowed: boolean;
|
||||||
|
degraded?: boolean;
|
||||||
|
}>;
|
||||||
notes: string[];
|
notes: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -186,6 +201,40 @@ export async function GET(req: NextRequest): Promise<Response> {
|
|||||||
// same quota number for every sport since they share the account.
|
// same quota number for every sport since they share the account.
|
||||||
const oddsQuotaRemaining = sports.map((s) => s.quota).find((q) => typeof q === 'number') ?? null;
|
const oddsQuotaRemaining = sports.map((s) => s.quota).find((q) => typeof q === 'number') ?? null;
|
||||||
|
|
||||||
|
// Session 20 — fetch the per-provider quota snapshot from the
|
||||||
|
// backend's internal endpoint. Best-effort: a failure to reach
|
||||||
|
// the backend or a missing internal key leaves provider_quotas
|
||||||
|
// empty and surfaces a note instead of blanking the dashboard.
|
||||||
|
const internalKey = process.env.VYNDR_INTERNAL_KEY || '';
|
||||||
|
let providerQuotas: AdminStats['provider_quotas'] = [];
|
||||||
|
if (internalKey) {
|
||||||
|
try {
|
||||||
|
const quotaController = new AbortController();
|
||||||
|
const quotaTimer = setTimeout(() => quotaController.abort(), HEALTH_PROBE_TIMEOUT_MS);
|
||||||
|
try {
|
||||||
|
const quotaRes = await fetch(`${BACKEND_URL}/api/internal/quota`, {
|
||||||
|
signal: quotaController.signal,
|
||||||
|
headers: { 'x-internal-key': internalKey, Accept: 'application/json' },
|
||||||
|
cache: 'no-store',
|
||||||
|
});
|
||||||
|
if (quotaRes.ok) {
|
||||||
|
const quotaBody = (await quotaRes.json().catch(() => null)) as { providers?: AdminStats['provider_quotas'] } | null;
|
||||||
|
if (quotaBody && Array.isArray(quotaBody.providers)) {
|
||||||
|
providerQuotas = quotaBody.providers;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
notes.push(`quota fetch returned ${quotaRes.status}`);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
clearTimeout(quotaTimer);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
notes.push(`quota fetch failed: ${err instanceof Error ? err.message : 'unknown'}`);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
notes.push('VYNDR_INTERNAL_KEY unset — provider quotas hidden');
|
||||||
|
}
|
||||||
|
|
||||||
const payload: AdminStats = {
|
const payload: AdminStats = {
|
||||||
generated_at: new Date().toISOString(),
|
generated_at: new Date().toISOString(),
|
||||||
users: {
|
users: {
|
||||||
@@ -201,6 +250,7 @@ export async function GET(req: NextRequest): Promise<Response> {
|
|||||||
sports,
|
sports,
|
||||||
odds_quota_remaining: oddsQuotaRemaining,
|
odds_quota_remaining: oddsQuotaRemaining,
|
||||||
},
|
},
|
||||||
|
provider_quotas: providerQuotas,
|
||||||
notes,
|
notes,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user