From 3ba3dd28f32963e3e2bfcad6251fbb88b6beeec3 Mon Sep 17 00:00:00 2001 From: Kev Date: Sun, 2 Aug 2026 22:07:43 -0400 Subject: [PATCH] Scoreboard every challenger; diagnose the 429 as odds-api, not PropLine PROMOTE-THE-EARNED. Nothing was promoted, because nothing earned it -- not because the bar was held high. Measured on the same bar that refuted hits-v1: own rows only, direction-aligned, paired bootstrap, promote only on a CI excluding zero. arch-v1 n=1741 delta 0.0000 CI[-0.0050,+0.0054] inconclusive contact-v1 n=1055 delta +0.0008 CI[-0.0052,+0.0069] inconclusive proj-v1.1 n=1664 delta -0.0301 CI[-0.0543,-0.0060] reliably WORSE matchup/tb-v1/hits-v1 n=0 genuinely pending (rows dated 08-02+) arch-v1 is the interesting one: it MOVED 76% of rows by 2.5 points on average and resolution is identical to the champion to four decimals, on the moved rows too. That is active movement carrying no information -- a finding, not a pending verdict. These are true prospective holdouts: arch-v1 and contact-v1 wrote p_win at grade time into their own columns before the game. Nothing recomputed. THE 429, read-only. The premise was that we re-pull the full picture every slot and blow the quota. Measured: PropLine is at 5 calls of 3,000/day -- 0.17%. One snapshot is ONE PropLine call per sport, all markets comma-joined. There is no request-pattern problem, so a change-based pull cannot fix it and no tier upgrade is needed. The 429 is odds-api: 478/500 MONTHLY, blocked at 95%. oddsService falls through silently when PropLine returns empty, and the backup's quota gate throws the error -- so an empty slate is indistinguishable from an outage and the message names the wrong provider. Flagged for its own order. Could NOT verify PropLine movement endpoints: docs are auth-gated and the keys are production-only. Not asserted either way. The movement-as-data argument stands on its own merits and should be justified that way, not as a quota fix it isn't. Book-breadth invariant written down: we never discard books. All are kept and shown (DISPLAY_BOOKS = MODEL + REFERENCE + DFS); DFS pick'em is excluded from PRICING only, because a fixed-payout shaded number is not a market price. Verified this is already what bookRoles.js does. Champion byte-identical; every challenger stays wired. 4,159 tests green (332 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9 --- BUILD-STATE.md | 32 +++++- CLAUDE.md | 51 +++++++++ scripts/challenger-scoreboard.js | 191 +++++++++++++++++++++++++++++++ specs/challenger-scoreboard.md | 109 ++++++++++++++++++ specs/odds-429-diagnosis.md | 145 +++++++++++++++++++++++ 5 files changed, 527 insertions(+), 1 deletion(-) create mode 100644 scripts/challenger-scoreboard.js create mode 100644 specs/challenger-scoreboard.md create mode 100644 specs/odds-429-diagnosis.md diff --git a/BUILD-STATE.md b/BUILD-STATE.md index 5e22828..ce919a2 100755 --- a/BUILD-STATE.md +++ b/BUILD-STATE.md @@ -1,7 +1,37 @@ # VYNDR — Build State ## Last Updated -2026-08-02 +2026-08-03 + +## Session 77 (2026-08-03) — Settlement was dead for two days; scoreboard now readable ✅ +Specs: `specs/challenger-scoreboard.md`, `specs/odds-429-diagnosis.md`. +4,159 tests / 332 suites green, web build exit 0. +- **THE FIND.** Three challenger axes read exactly ZERO settled rows. Not low — + zero, on games played days earlier, with `settle_attempts = 0`. `settleLedger` + refetched rows via `.in('id', ids)`; 500 UUIDs = an 18,499-char URL the fetch + layer rejects, and the result was destructured with no error binding, so it + returned all-zeros indistinguishable from a clean "nothing to settle". + Volume-triggered: 2026-08-01 was the first day past the 500-row limit. + The zero-settle ops alarm reads those same return values and was blind to it. +- **FIXED + DRAINED.** One query, all columns, no id list; failed fetches surface. + `captureClosing` chunked at 100 (same defect family). 1,444 rows from 08-01 + settled (1,376 hit/miss + 68 void, 0 remaining). Settled n **493 → 1,741**. +- **SCOREBOARD — nothing promoted, nothing earned it.** arch-v1 n=1,741 Δ0.0000 + CI[−0.0050,+0.0054] (moves 76% of rows by 2.5pp mean = active movement carrying + no information); contact-v1 n=1,055 +0.0008 inconclusive; proj-v1.1 ladder + n=1,664 **−0.0301 CI[−0.0543,−0.0060] = reliably WORSE**. matchup/tb-v1/hits-v1 + STILL PENDING (rows dated 08-02+, settle after ET midnight). Champion + byte-identical; all challengers stay wired. +- **429 DIAGNOSED (read-only) — premise refuted with numbers.** PropLine 5/3,000 + daily (0.17%); the 429 is **odds-api at 478/500 monthly, blocked at 95%**, + surfacing whenever PropLine returns empty. One snapshot = ONE PropLine call per + sport. Change-based pull is NOT the fix and no tier upgrade is needed. Could + NOT verify PropLine movement endpoints (auth-gated docs, prod-only keys) — not + asserted. Book-breadth invariant recorded: we never discard books; DFS is + excluded from PRICING only. +- **Next:** the silent PropLine fall-through (empty slate must not report the + backup's 429); diagnose the projection family's INPUTS (two independent + measurements now say it trails the champion). ## Session 76 (2026-08-02) — hits-v1: a challenger built, measured, and REFUTED ✅ Spec: `specs/hits-v1-binomial.md`. 4,156 tests / 332 suites green, web build exit 0. diff --git a/CLAUDE.md b/CLAUDE.md index 393c3f9..2e364ab 100755 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1132,6 +1132,57 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section). `zmdnczhtdxcddsxzttub`), so local scripts hitting Supabase need an explicit `SUPABASE_URL=` override. Prod + the MCP connection are fine. +## Settlement outage + challenger scoreboard (Session 77 — non-obvious) +- **`.in('id', [...])` IS A URL, NOT A QUERY.** PostgREST puts filters in the + URL: 500 UUIDs = an 18,499-char request that the fetch layer rejects with + `TypeError: fetch failed`. This silently killed `settleLedger` for two days + (2026-08-01/02) — it refetched rows by id, destructured `const { data: rows }` + with NO error binding, so rows was null, the loop never ran, and it returned + `{settled:0,voided:0,unrecoverable:0,pending:0}`, byte-identical to a healthy + "nothing to settle". 1,444 rows sat with `settle_attempts=0`. NEVER send an + unbounded id list; `ID_FILTER_CHUNK` (100) is the guard, and settleLedger now + selects every column it needs in ONE query. +- **It was VOLUME-TRIGGERED, which is why it hid.** Daily volume ran 20–260 rows + for weeks; 2026-08-01 was the first day past `SETTLE_FETCH_LIMIT` (500). If a + pipeline "works for weeks then stops", suspect a threshold that volume just + crossed, not a code change. +- **The ops alarm was blind to it BY CONSTRUCTION.** The zero-settle watchdog + reads settleLedger's own return values, so `pending: 0` told it the backlog was + empty. An alarm that trusts the return value of the thing it watches cannot see + that thing fail silently — the signal must come from OUTSIDE (a direct + `game_date < today AND outcome IS NULL` count). +- **Settled n went 493 → 1,741 the moment it was fixed.** Anything reading + "n-blocked" across MULTIPLE independent challengers at once is a pipeline + symptom, not a sampling fact. Count settled rows before believing it. +- **`specs/challenger-scoreboard.md` is the board.** Nothing promoted: arch-v1 + Δ0.0000 CI[−0.005,+0.005] on 1,741 (and it MOVED 76% of rows by 2.5pp mean — + active movement carrying zero information), contact-v1 +0.0008 inconclusive, + proj-v1.1 ladder −0.0301 CI excluding zero = reliably WORSE. matchup/tb-v1/ + hits-v1 genuinely pending (rows dated 08-02+, settle after ET midnight). +- **arch-v1/contact-v1 need no replay** — they wrote p_win at grade time into + their own columns, so scoring them is a TRUE prospective holdout. Only a + challenger that did not exist at grade time (hits-v1) needs a point-in-time + replay. Don't conflate the two kinds of evidence. +- **Score a nudge on the rows it MOVED**, not on all rows — otherwise the + unmoved rows are the champion measured against itself and dilute any real + effect toward zero. `scripts/challenger-scoreboard.js` does both slices. + +## The 429 is odds-api, NOT PropLine (Session 77 — non-obvious) +- **`specs/odds-429-diagnosis.md`.** MEASURED: PropLine 5/3,000 daily (0.17%); + odds-api 478/500 MONTHLY, `allowed:false` (tracker blocks at 95%). One snapshot + = ONE PropLine call per sport (all markets comma-joined) — there is no + per-prop/per-book fan-out and no request-pattern problem to optimize. +- **The 429 text is the BACKUP's.** `oddsService.getOdds` falls through silently + when PropLine returns null/empty, then odds-api's quota gate throws + `429 "Odds data temporarily unavailable"`. So an EMPTY PropLine slate is + indistinguishable from an outage, and the error names the wrong provider. Open + order — don't read a 429 as "PropLine exhausted" without checking + `GET /api/internal/quota`. +- **BOOK BREADTH INVARIANT: we never discard books.** All books are KEPT and + SHOWN (`DISPLAY_BOOKS = MODEL ∪ REFERENCE ∪ DFS`). The ONLY selectivity is that + DFS pick'em is excluded from PRICING/consensus (`EXCLUDED_FROM_PRICING`) — a + fixed-payout shaded number is not a market price. Never "clean up" breadth. + ## Active Skills - vyndr-voice (all user-facing output) - prop-analysis (grading methodology) diff --git a/scripts/challenger-scoreboard.js b/scripts/challenger-scoreboard.js new file mode 100644 index 0000000..6c676e8 --- /dev/null +++ b/scripts/challenger-scoreboard.js @@ -0,0 +1,191 @@ +#!/usr/bin/env node +'use strict'; + +/** + * challenger-scoreboard — every accruing challenger, measured on the same bar. + * + * WHY THIS IS A REAL HOLDOUT AND NOT A BACKTEST. Unlike hits-v1 (which did not + * exist when these rows were graded and therefore needed a point-in-time + * replay), arch-v1 and contact-v1 wrote their probability AT GRADE TIME, into + * their own columns, before the game was played. Nothing here is recomputed. + * These numbers are genuinely out-of-sample — the strongest evidence available. + * + * THE BAR IS THE SAME ONE THAT REFUTED hits-v1: + * - the challenger's OWN rows only (a challenger that abstains is not scored + * on the rows it declined — averaging those in measures the champion twice) + * - direction handled: p_win and p_win_challenger/p_win_contact are all + * P(GRADED SIDE), so they are already aligned. The projection ladder is + * P(OVER) and IS realigned here. + * - paired bootstrap on matched rows, because both models score the SAME rows + * and treating their errors as independent overstates the uncertainty + * - PROMOTE only when the CI on (challenger − champion) excludes zero + * + * CONTAMINATION EXCLUSION: rows whose price/book were stamped from a + * non-takeable book are tagged `quarantine_reason LIKE 'nontakeable_book%'` and + * are excluded — their locked price describes a market you could not have bet. + * + * PROVENANCE: results are reported for all rows AND split by `model_version`, + * so if a verdict depends on the older `pre-retention-unknown` era that fact is + * visible rather than buried. + * + * SUPABASE_URL=... node scripts/challenger-scoreboard.js + */ + +require('dotenv').config(); +const { createClient } = require('@supabase/supabase-js'); + +const SB_URL = process.env.SUPABASE_URL; +const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY; +const PAGE = 1000; + +function corr(xs, ys) { + const n = xs.length; + if (n < 3) return null; + const mx = xs.reduce((a, b) => a + b, 0) / n; + const my = ys.reduce((a, b) => a + b, 0) / n; + let sxy = 0; let sxx = 0; let syy = 0; + for (let i = 0; i < n; i += 1) { + const dx = xs[i] - mx; const dy = ys[i] - my; + sxy += dx * dy; sxx += dx * dx; syy += dy * dy; + } + if (sxx <= 0 || syy <= 0) return null; + return sxy / Math.sqrt(sxx * syy); +} +const r4 = (v) => (v == null ? null : Math.round(v * 10000) / 10000); +const meanOf = (a) => (a.length ? a.reduce((x, y) => x + y, 0) / a.length : null); +const brier = (ps, ys) => (ps.length ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null); + +/** Paired bootstrap on the DIFFERENCE of resolutions. Deterministic seed. */ +function bootstrapDiff(rows, keyA, keyB, iters = 4000, seed = 20260803) { + if (rows.length < 30) return null; + let s = seed >>> 0; + const rnd = () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; }; + const n = rows.length; + const diffs = []; + for (let it = 0; it < iters; it += 1) { + const ys = []; const a = []; const b = []; + for (let i = 0; i < n; i += 1) { + const r = rows[Math.floor(rnd() * n)]; + ys.push(r.won); a.push(r[keyA]); b.push(r[keyB]); + } + const ca = corr(a, ys); const cb = corr(b, ys); + if (ca == null || cb == null) continue; + diffs.push(ca - cb); + } + if (diffs.length < 100) return null; + diffs.sort((x, y) => x - y); + const q = (p) => r4(diffs[Math.floor(p * (diffs.length - 1))]); + const point = r4(corr(rows.map((r) => r[keyA]), rows.map((r) => r.won)) + - corr(rows.map((r) => r[keyB]), rows.map((r) => r.won))); + const ci = [q(0.025), q(0.975)]; + return { point, ci95: ci, p_improves: r4(diffs.filter((d) => d > 0).length / diffs.length), + ci_excludes_zero: ci[0] > 0 || ci[1] < 0 }; +} + +function score(rows, challKey, label) { + const ys = rows.map((r) => r.won); + const ch = rows.map((r) => r[challKey]); + const cp = rows.map((r) => r.champ); + const bs = bootstrapDiff(rows, challKey, 'champ'); + let verdict = 'STILL PENDING'; + if (rows.length >= 30 && bs) { + if (bs.ci_excludes_zero && bs.point > 0) verdict = 'PROMOTE'; + else if (bs.ci_excludes_zero && bs.point < 0) verdict = 'STAY WIRED (measured worse)'; + else verdict = 'STAY WIRED (inconclusive)'; + } + return { + challenger: label, + settled_n: rows.length, + base_rate: r4(meanOf(ys)), + resolution_challenger: r4(corr(ch, ys)), + resolution_champion: r4(corr(cp, ys)), + brier_challenger: r4(brier(ch, ys)), + brier_champion: r4(brier(cp, ys)), + delta_vs_champion: bs, + verdict, + }; +} + +async function fetchAll(sb) { + const out = []; + for (let from = 0; ; from += PAGE) { + const { data, error } = await sb.from('ledger_entries') + .select('id, stat, side, outcome, model_version, quarantine_reason, p_win, p_win_challenger, p_win_contact, challenger_delta, contact_delta, proj_p_over_line, proj_tb_p_over, proj_hits_p_over, challenger_adjustments') + .eq('sport', 'mlb').is('user_id', null) + .in('outcome', ['hit', 'miss']) + .not('p_win', 'is', null) + .range(from, from + PAGE - 1); + if (error) throw error; + if (!data || data.length === 0) break; + out.push(...data); + if (data.length < PAGE) break; + } + return out; +} + +async function main() { + if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required'); + const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } }); + + const raw = (await fetchAll(sb)) + .filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book')); + + const base = raw.map((r) => ({ + won: r.outcome === 'hit' ? 1 : 0, + champ: Number(r.p_win), + arch: r.p_win_challenger == null ? null : Number(r.p_win_challenger), + contact: r.p_win_contact == null ? null : Number(r.p_win_contact), + // The ladder is P(OVER); realign it to the graded side before comparing. + ladder: r.proj_p_over_line == null ? null + : (String(r.side).toLowerCase() === 'under' ? 1 - Number(r.proj_p_over_line) : Number(r.proj_p_over_line)), + tb: r.proj_tb_p_over == null ? null + : (String(r.side).toLowerCase() === 'under' ? 1 - Number(r.proj_tb_p_over) : Number(r.proj_tb_p_over)), + hits: r.proj_hits_p_over == null ? null + : (String(r.side).toLowerCase() === 'under' ? 1 - Number(r.proj_hits_p_over) : Number(r.proj_hits_p_over)), + stat: r.stat, + // Did the challenger actually MOVE this row? A nudge that leaves p_win + // untouched is the champion wearing a different name, and scoring it on + // those rows measures the champion against itself — which is exactly how a + // real effect gets averaged down to zero. + archMoved: r.challenger_delta != null && Number(r.challenger_delta) !== 0, + contactMoved: r.contact_delta != null && Number(r.contact_delta) !== 0, + era: r.model_version || 'unknown', + axes: new Set(((r.challenger_adjustments) || []).map((a) => a && a.axis).filter(Boolean)), + })); + + const withKey = (k, extra = () => true) => base.filter((r) => r[k] != null && Number.isFinite(r[k]) && extra(r)); + + const board = [ + score(withKey('arch'), 'arch', 'arch-v1 (market-relative nudge)'), + score(withKey('contact'), 'contact', 'contact-v1 (season contact quality)'), + score(withKey('ladder'), 'ladder', 'proj-v1.1 ladder (all stats)'), + score(withKey('tb', (r) => r.stat === 'total_bases'), 'tb', 'tb-v1 (total_bases only)'), + score(withKey('hits', (r) => r.stat === 'hits'), 'hits', 'hits-v1 (hits only)'), + ]; + + // THE SHARPEST TEST OF A NUDGE — only the rows it actually moved. + const movedBoard = [ + score(withKey('arch', (r) => r.archMoved), 'arch', 'arch-v1 · rows it MOVED only'), + score(withKey('contact', (r) => r.contactMoved), 'contact', 'contact-v1 · rows it MOVED only'), + ]; + + // Per-AXIS: arch-v1 restricted to the rows where that axis actually fired. + const axisBoard = ['environment', 'opportunity', 'matchup'].map((ax) => + score(withKey('arch', (r) => r.axes.has(ax)), 'arch', `arch-v1 · ${ax} axis rows only`)); + + // PROVENANCE split — does any verdict depend on the older era? + const eras = [...new Set(base.map((r) => r.era))]; + const provenance = eras.map((era) => ({ + era, + ...score(withKey('arch', (r) => r.era === era), 'arch', `arch-v1 · ${era}`), + })); + + console.log(JSON.stringify({ + measurement: 'PROSPECTIVE HOLDOUT — challenger values were written at grade time, before the game. No recomputation, no lookahead.', + total_settled_rows: base.length, + board, movedBoard, axisBoard, provenance, + }, null, 2)); + process.exit(0); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/specs/challenger-scoreboard.md b/specs/challenger-scoreboard.md new file mode 100644 index 0000000..a3d5e98 --- /dev/null +++ b/specs/challenger-scoreboard.md @@ -0,0 +1,109 @@ +# CHALLENGER SCOREBOARD — 2026-08-03 + +> **Nothing was promoted. Nothing earned it yet.** Not because the bar was held +> too high, but because no challenger's confidence interval excludes zero on the +> good side. The champion is byte-identical; every challenger stays wired. + +The headline of this session is not the scoreboard. It is that **the scoreboard +was unmeasurable until a two-day-old settlement outage was found and fixed** — +see §1. Settled sample went **493 → 1,741** the moment it was repaired. + +--- + +## 1. Why "n-blocked" was the wrong diagnosis + +The order said: don't repeat "n-blocked" without counting. Counting is what found +the real problem. + +Three of the four axes read **exactly zero** settled rows — not low, *zero*: +environment 1,496 rows / 0 settled, opportunity 922 / 0, matchup 603 / 0. Rows +whose games had been **played days earlier** and never settled, with +`settle_attempts = 0` — never even attempted. + +**Root cause:** `settleLedger` fetched open ids, then refetched full rows via +`.in('id', ids)`. PostgREST puts filters in the URL, so 500 UUIDs became an +**18,499-character request** that the fetch layer rejects with `TypeError: fetch +failed`. The result was destructured as `const { data: rows } = ...` with **no +error binding**, so `rows` came back null, the loop never ran, and the function +returned `{settled:0, voided:0, unrecoverable:0, pending:0}` — byte-identical to +a healthy "nothing to settle." + +It hid for two days because it is **volume-triggered**: daily volume ran 20–260 +rows and settled perfectly for weeks. **2026-08-01 was the first day past the +500-row fetch limit** and settlement died that night. Worse, the zero-settle ops +alarm reads these same return values, so `pending: 0` told the watchdog the +backlog was empty — *the alarm built to catch exactly this could not see it.* + +Fixed, deployed, and drained: **1,444 rows from 2026-08-01 settled (1,376 +hit/miss + 68 void, 0 remaining).** `captureClosing` carried the same shape one +level down and is now chunked at 100 ids. + +## 2. THE SCOREBOARD + +Bar: the challenger's **own rows only**, direction-aligned, **paired bootstrap** +(4,000 resamples, deterministic seed) on the difference in resolution, because +both models score the same rows and independent standard errors would overstate +certainty. **PROMOTE requires the CI to exclude zero on the good side.** Same bar +that refuted hits-v1 — no lighter test for a would-be winner. + +| challenger | settled n | resolution (chal / champ) | Δ vs champion | CI95 | verdict | +|---|---|---|---|---|---| +| arch-v1 (market-relative nudge) | **1,741** | 0.4599 / 0.4599 | 0.0000 | [−0.0050, +0.0054] | **STAY WIRED** (inconclusive) | +| arch-v1 · rows it MOVED only | 1,325 | 0.4886 / 0.4886 | −0.0001 | [−0.0064, +0.0061] | **STAY WIRED** (inconclusive) | +| contact-v1 (season contact quality) | **1,055** | 0.4070 / 0.4061 | +0.0008 | [−0.0052, +0.0069] | **STAY WIRED** (inconclusive) | +| contact-v1 · rows it MOVED only | 511 | 0.3966 / 0.3948 | +0.0018 | [−0.0107, +0.0144] | **STAY WIRED** (inconclusive) | +| proj-v1.1 ladder (all stats) | **1,664** | 0.4339 / 0.4640 | **−0.0301** | **[−0.0543, −0.0060]** | **STAY WIRED** (measured WORSE) | +| arch-v1 · environment axis rows | 871 | 0.5651 / 0.5679 | −0.0028 | [−0.0089, +0.0036] | **STAY WIRED** (inconclusive) | +| arch-v1 · opportunity axis rows | 539 | 0.5311 / 0.5310 | +0.0001 | [−0.0091, +0.0090] | **STAY WIRED** (inconclusive) | +| arch-v1 · matchup axis rows | **0** | — | — | — | **STILL PENDING** | +| tb-v1 (total_bases only) | **0** | — | — | — | **STILL PENDING** | +| hits-v1 (hits only) | **0** | — | — | — | **STILL PENDING** (refuted by replay, `specs/hits-v1-binomial.md`) | + +**These are true prospective holdouts, not backtests.** arch-v1 and contact-v1 +wrote their probability at grade time, into their own columns, before the game +was played. Nothing was recomputed. That is the strongest evidence available and +it is why no replay was needed here. + +## 3. What the numbers actually say + +**arch-v1 moves a lot and changes nothing.** It moved **1,325 of 1,741 rows +(76%)**, mean absolute move **2.5 points**, max 10.9 — and resolution is +identical to the champion to four decimal places, on the moved rows too. This is +not "too small to detect." It is movement that carries **no information about the +outcome**. A nudge this active with an effect this precisely zero is a finding, +not a pending verdict. + +**The projection ladder is reliably worse than the champion.** −0.0301 with a CI +excluding zero, across 1,664 rows and all stats. Combined with hits-v1's refutation +(`specs/hits-v1-binomial.md`), the projection family now has two independent +measurements pointing the same way: it is not the champion's equal on any stat +measured so far. That is an argument for diagnosing its *inputs*, not for shipping +another variant of it. + +**Three are genuinely pending, for a legitimate reason now.** matchup, tb-v1 and +hits-v1 all have rows written only on 2026-08-02/03, which settle after ET +midnight. matchup has ~496 rows queued, tb-v1 65, hits-v1 pending its first +snapshot write. They will read within a day or two — and now that settlement +works, they actually will. + +## 4. Provenance + +All 1,741 arch-v1 rows carry a single `model_version` (`engine1@2026-07-20`). The +older `pre-retention-unknown` rows (320 settled) carry no challenger values at +all, so they cannot influence any verdict. **No verdict here depends on +mixed-provenance rows** — the split was checked, not assumed. + +Contamination excluded throughout: `quarantine_reason LIKE 'nontakeable_book%'`. + +## 5. Promotion mechanics — specified, deliberately unused + +No flip was performed because nothing qualified. When one does, the shape is: +challenger-first (write the promoted value into the served path while the +champion column keeps recording), version-tagged, atomic, with the previous +version one env flag away. Recorded here so a future promotion is a decision, +not an improvisation. + +## 6. Reproduce + +`SUPABASE_URL=... node scripts/challenger-scoreboard.js` — prints the full board, +the moved-rows-only slice, the per-axis slice and the provenance split. diff --git a/specs/odds-429-diagnosis.md b/specs/odds-429-diagnosis.md new file mode 100644 index 0000000..0f36327 --- /dev/null +++ b/specs/odds-429-diagnosis.md @@ -0,0 +1,145 @@ +# THE 429 — DIAGNOSED, AND IT IS NOT WHAT WE THOUGHT + +**Read-only investigation, 2026-08-03.** Nothing was implemented, no tier bought. + +> **VERDICT: the change-based pull is NOT the fix, and no tier upgrade is +> needed. There is no request-pattern problem to solve.** PropLine is running at +> **5 calls out of 3,000/day — 0.17% of the free allowance.** The 429 comes from a +> different provider entirely. + +--- + +## 1. BOOK BREADTH — the invariant, written down (and already true in code) + +**WE DO NOT DISCARD BOOKS.** All books are KEPT and SHOWN. More breadth is +strictly good for display, line-shopping and consensus. The **only** selectivity +is that **DFS pick'em (prizepicks / underdog / sleeper / dabble) does not vote on +the fair-price CONSENSUS**, because a fixed-payout shaded number is not a market +price. That is one category exclusion from *pricing*, not a reduction in breadth. + +Verified in `src/config/bookRoles.js` — this is already exactly what the code +does, and the invariant is recorded so no future "cleanup" undoes it: + +``` +DISPLAY_BOOKS = MODEL_BOOKS ∪ REFERENCE_BOOKS ∪ DFS_PLATFORMS ← everything shows +EXCLUDED_FROM_PRICING = DFS_PLATFORMS ∪ OFFSHORE_OR_INTL ← pricing only +``` + +The historical real discarding — `ALLOWED_BOOKS` cutting the slate to 5 keys and +making 64.8% of MLB props invisible — was fixed in Order Zero and must not +return. No language in this repo should describe book handling as "discarding". + +## 2. Post-reset status (STEP 1) — the pipeline RECOVERED + +``` +GET /api/internal/snapshot/status → overdue: false + mlb 2026-08-03T01:00:24Z gradeCount 235 + wnba 2026-08-03T01:02:58Z gradeCount 139 +``` + +The 14/19/22 UTC slots on 2026-08-02 all failed; the 01:00 UTC slot after the +reset succeeded. So the failure was time-bounded and self-healing — which already +argued against a structural request-pattern defect. + +## 3. What one snapshot actually requests (STEP 2) + +Measured in `proplineAdapter.fetchRaw`: **one snapshot = ONE PropLine HTTP call +per sport.** All markets ride the same request, comma-joined (12 markets for MLB). +There is no per-prop, per-book or per-event fan-out. + +``` +snapshot 1 call × 2 active sports × 5 slots/day ≈ 10 calls/day +intraday refresh every 20 min, slate hours only (12h) ≈ 72 calls/day +public /api/odds cache-aside, 1h TTL bounded, small + ── total order-of-10s/day +``` + +**MEASURED, not estimated** — `GET /api/internal/quota`, 2026-08-03: + +| provider | used | limit | period | allowed | +|---|---|---|---|---| +| **propline** | **5** | **3,000** | daily | ✅ true | +| **odds-api** | **478** | **500** | **monthly** | ❌ **false (95.6%)** | +| oddspapi | 0 | 1,000 | monthly | true | +| tank01 | 86 | 1,000 | monthly | true | + +**There is no waste to reclaim.** We are using 0.17% of the free PropLine +allowance. Re-pulling "the full picture" every slot costs one request; a +change-based pull would replace one request with one request. + +## 4. Where the 429 actually comes from + +`oddsService.getOdds` tries PropLine first, and **falls through silently** when it +returns null or an empty array. The 429 is thrown *after* that, by the **odds-api +backup's** quota gate: + +```js +const quotaStatus = await quotaTracker.getQuotaStatus('odds-api'); +if (!quotaStatus.allowed) { + const error = new Error('Odds data temporarily unavailable. Try again later.'); + error.statusCode = 429; // ← THIS is the 429 we saw +``` + +So the real sequence on 2026-08-02 was: + +1. PropLine returned **no props** for that moment (it had 2,995 calls remaining — + this was an empty/failed slate, **not** exhaustion), +2. control fell through to odds-api, +3. odds-api sits at **478/500 for the month** and the tracker blocks at ≥95%, +4. → user-facing `429 Odds data temporarily unavailable`. + +**The 429 is the dead backup announcing itself.** It is not a PropLine quota +signal, and tuning PropLine request volume cannot affect it. + +### The real defect this exposes + +**An empty PropLine slate is indistinguishable from a provider outage.** A +legitimately empty board (off-hours, no props posted yet) produces the identical +429 as a genuine failure, because the fall-through is silent and the error text +belongs to whichever provider happened to be asked last. That is a +truth-in-reporting bug in the odds path — the operator cannot tell "no games" from +"we are broken." Flagged for its own order; **not** fixed here (read-only). + +## 5. Change-based capture (STEP 3) — NOT VERIFIED, and moot for quota + +**I could not verify whether PropLine supports `/movement`, odds-history, or a +`since=` parameter.** PropLine's docs are auth-gated (HTTP 401) and the API keys +are production-only — not present in the local environment. **I am not going to +assert an endpoint exists or costs less when I could not test it**; "documented ≠ +verified" cuts both ways. + +What can be said without testing anything: + +- **For quota, it is moot.** At 5/3,000 there is nothing to optimise. A + change-based pull would save at most a handful of calls per day out of an + allowance we are not close to touching. +- **The movement-data argument stands on its own, and is the better reason.** A + change-based capture records line movement as `from → to at time T`, which is + genuine model input we currently lack: steam detection, true CLV attribution, + and the market-movement axis. Today `intradayRefreshService.trackHistory` + reconstructs a coarse version of this by diffing full pulls. If PropLine + exposes movement natively it is worth building **as a data feature** — and it + should be justified that way, not as a quota fix it isn't. + +## 6. VERDICT (STEP 4), with the numbers + +| question | answer | +|---|---| +| Did the pipeline recover post-reset? | **Yes** — `overdue: false`, mlb 235 grades at 01:00Z | +| Requests per snapshot | **1 per sport**, all markets in one call | +| Requests/day vs allowance | **~10s/day vs 3,000/day free — 0.17% used** | +| Is the 429 a request-pattern problem? | **No.** Measured, decisively | +| Is the 429 a PropLine quota problem? | **No.** 2,995 calls remaining, `allowed: true` | +| What is the 429? | **odds-api backup at 478/500 monthly, blocked at 95%** | +| Does change-based pull fix it? | **No** — it cannot; wrong provider, and no volume problem | +| Is a tier upgrade needed? | **No.** Not for PropLine. Nothing is constrained | + +**What actually deserves an order**, in priority sequence: + +1. **The silent fall-through** — an empty PropLine slate must report "no props + available", never the backup's quota 429 (§4). +2. **odds-api at 478/500 monthly** — it is a dead backup for the rest of August. + Either accept it (PropLine is primary and healthy) and stop letting its quota + gate speak for the whole odds path, or budget it deliberately. +3. **Movement capture as a DATA feature** — worth scoping on its own merits, with + PropLine's endpoint support verified first from an environment that has keys.