Scoreboard every challenger; diagnose the 429 as odds-api, not PropLine
PROMOTE-THE-EARNED. Nothing was promoted, because nothing earned it -- not because the bar was held high. Measured on the same bar that refuted hits-v1: own rows only, direction-aligned, paired bootstrap, promote only on a CI excluding zero. arch-v1 n=1741 delta 0.0000 CI[-0.0050,+0.0054] inconclusive contact-v1 n=1055 delta +0.0008 CI[-0.0052,+0.0069] inconclusive proj-v1.1 n=1664 delta -0.0301 CI[-0.0543,-0.0060] reliably WORSE matchup/tb-v1/hits-v1 n=0 genuinely pending (rows dated 08-02+) arch-v1 is the interesting one: it MOVED 76% of rows by 2.5 points on average and resolution is identical to the champion to four decimals, on the moved rows too. That is active movement carrying no information -- a finding, not a pending verdict. These are true prospective holdouts: arch-v1 and contact-v1 wrote p_win at grade time into their own columns before the game. Nothing recomputed. THE 429, read-only. The premise was that we re-pull the full picture every slot and blow the quota. Measured: PropLine is at 5 calls of 3,000/day -- 0.17%. One snapshot is ONE PropLine call per sport, all markets comma-joined. There is no request-pattern problem, so a change-based pull cannot fix it and no tier upgrade is needed. The 429 is odds-api: 478/500 MONTHLY, blocked at 95%. oddsService falls through silently when PropLine returns empty, and the backup's quota gate throws the error -- so an empty slate is indistinguishable from an outage and the message names the wrong provider. Flagged for its own order. Could NOT verify PropLine movement endpoints: docs are auth-gated and the keys are production-only. Not asserted either way. The movement-as-data argument stands on its own merits and should be justified that way, not as a quota fix it isn't. Book-breadth invariant written down: we never discard books. All are kept and shown (DISPLAY_BOOKS = MODEL + REFERENCE + DFS); DFS pick'em is excluded from PRICING only, because a fixed-payout shaded number is not a market price. Verified this is already what bookRoles.js does. Champion byte-identical; every challenger stays wired. 4,159 tests green (332 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
+31
-1
@@ -1,7 +1,37 @@
|
|||||||
# VYNDR — Build State
|
# VYNDR — Build State
|
||||||
|
|
||||||
## Last Updated
|
## Last Updated
|
||||||
2026-08-02
|
2026-08-03
|
||||||
|
|
||||||
|
## Session 77 (2026-08-03) — Settlement was dead for two days; scoreboard now readable ✅
|
||||||
|
Specs: `specs/challenger-scoreboard.md`, `specs/odds-429-diagnosis.md`.
|
||||||
|
4,159 tests / 332 suites green, web build exit 0.
|
||||||
|
- **THE FIND.** Three challenger axes read exactly ZERO settled rows. Not low —
|
||||||
|
zero, on games played days earlier, with `settle_attempts = 0`. `settleLedger`
|
||||||
|
refetched rows via `.in('id', ids)`; 500 UUIDs = an 18,499-char URL the fetch
|
||||||
|
layer rejects, and the result was destructured with no error binding, so it
|
||||||
|
returned all-zeros indistinguishable from a clean "nothing to settle".
|
||||||
|
Volume-triggered: 2026-08-01 was the first day past the 500-row limit.
|
||||||
|
The zero-settle ops alarm reads those same return values and was blind to it.
|
||||||
|
- **FIXED + DRAINED.** One query, all columns, no id list; failed fetches surface.
|
||||||
|
`captureClosing` chunked at 100 (same defect family). 1,444 rows from 08-01
|
||||||
|
settled (1,376 hit/miss + 68 void, 0 remaining). Settled n **493 → 1,741**.
|
||||||
|
- **SCOREBOARD — nothing promoted, nothing earned it.** arch-v1 n=1,741 Δ0.0000
|
||||||
|
CI[−0.0050,+0.0054] (moves 76% of rows by 2.5pp mean = active movement carrying
|
||||||
|
no information); contact-v1 n=1,055 +0.0008 inconclusive; proj-v1.1 ladder
|
||||||
|
n=1,664 **−0.0301 CI[−0.0543,−0.0060] = reliably WORSE**. matchup/tb-v1/hits-v1
|
||||||
|
STILL PENDING (rows dated 08-02+, settle after ET midnight). Champion
|
||||||
|
byte-identical; all challengers stay wired.
|
||||||
|
- **429 DIAGNOSED (read-only) — premise refuted with numbers.** PropLine 5/3,000
|
||||||
|
daily (0.17%); the 429 is **odds-api at 478/500 monthly, blocked at 95%**,
|
||||||
|
surfacing whenever PropLine returns empty. One snapshot = ONE PropLine call per
|
||||||
|
sport. Change-based pull is NOT the fix and no tier upgrade is needed. Could
|
||||||
|
NOT verify PropLine movement endpoints (auth-gated docs, prod-only keys) — not
|
||||||
|
asserted. Book-breadth invariant recorded: we never discard books; DFS is
|
||||||
|
excluded from PRICING only.
|
||||||
|
- **Next:** the silent PropLine fall-through (empty slate must not report the
|
||||||
|
backup's 429); diagnose the projection family's INPUTS (two independent
|
||||||
|
measurements now say it trails the champion).
|
||||||
|
|
||||||
## Session 76 (2026-08-02) — hits-v1: a challenger built, measured, and REFUTED ✅
|
## Session 76 (2026-08-02) — hits-v1: a challenger built, measured, and REFUTED ✅
|
||||||
Spec: `specs/hits-v1-binomial.md`. 4,156 tests / 332 suites green, web build exit 0.
|
Spec: `specs/hits-v1-binomial.md`. 4,156 tests / 332 suites green, web build exit 0.
|
||||||
|
|||||||
@@ -1132,6 +1132,57 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
|
|||||||
`zmdnczhtdxcddsxzttub`), so local scripts hitting Supabase need an explicit
|
`zmdnczhtdxcddsxzttub`), so local scripts hitting Supabase need an explicit
|
||||||
`SUPABASE_URL=` override. Prod + the MCP connection are fine.
|
`SUPABASE_URL=` override. Prod + the MCP connection are fine.
|
||||||
|
|
||||||
|
## Settlement outage + challenger scoreboard (Session 77 — non-obvious)
|
||||||
|
- **`.in('id', [...])` IS A URL, NOT A QUERY.** PostgREST puts filters in the
|
||||||
|
URL: 500 UUIDs = an 18,499-char request that the fetch layer rejects with
|
||||||
|
`TypeError: fetch failed`. This silently killed `settleLedger` for two days
|
||||||
|
(2026-08-01/02) — it refetched rows by id, destructured `const { data: rows }`
|
||||||
|
with NO error binding, so rows was null, the loop never ran, and it returned
|
||||||
|
`{settled:0,voided:0,unrecoverable:0,pending:0}`, byte-identical to a healthy
|
||||||
|
"nothing to settle". 1,444 rows sat with `settle_attempts=0`. NEVER send an
|
||||||
|
unbounded id list; `ID_FILTER_CHUNK` (100) is the guard, and settleLedger now
|
||||||
|
selects every column it needs in ONE query.
|
||||||
|
- **It was VOLUME-TRIGGERED, which is why it hid.** Daily volume ran 20–260 rows
|
||||||
|
for weeks; 2026-08-01 was the first day past `SETTLE_FETCH_LIMIT` (500). If a
|
||||||
|
pipeline "works for weeks then stops", suspect a threshold that volume just
|
||||||
|
crossed, not a code change.
|
||||||
|
- **The ops alarm was blind to it BY CONSTRUCTION.** The zero-settle watchdog
|
||||||
|
reads settleLedger's own return values, so `pending: 0` told it the backlog was
|
||||||
|
empty. An alarm that trusts the return value of the thing it watches cannot see
|
||||||
|
that thing fail silently — the signal must come from OUTSIDE (a direct
|
||||||
|
`game_date < today AND outcome IS NULL` count).
|
||||||
|
- **Settled n went 493 → 1,741 the moment it was fixed.** Anything reading
|
||||||
|
"n-blocked" across MULTIPLE independent challengers at once is a pipeline
|
||||||
|
symptom, not a sampling fact. Count settled rows before believing it.
|
||||||
|
- **`specs/challenger-scoreboard.md` is the board.** Nothing promoted: arch-v1
|
||||||
|
Δ0.0000 CI[−0.005,+0.005] on 1,741 (and it MOVED 76% of rows by 2.5pp mean —
|
||||||
|
active movement carrying zero information), contact-v1 +0.0008 inconclusive,
|
||||||
|
proj-v1.1 ladder −0.0301 CI excluding zero = reliably WORSE. matchup/tb-v1/
|
||||||
|
hits-v1 genuinely pending (rows dated 08-02+, settle after ET midnight).
|
||||||
|
- **arch-v1/contact-v1 need no replay** — they wrote p_win at grade time into
|
||||||
|
their own columns, so scoring them is a TRUE prospective holdout. Only a
|
||||||
|
challenger that did not exist at grade time (hits-v1) needs a point-in-time
|
||||||
|
replay. Don't conflate the two kinds of evidence.
|
||||||
|
- **Score a nudge on the rows it MOVED**, not on all rows — otherwise the
|
||||||
|
unmoved rows are the champion measured against itself and dilute any real
|
||||||
|
effect toward zero. `scripts/challenger-scoreboard.js` does both slices.
|
||||||
|
|
||||||
|
## The 429 is odds-api, NOT PropLine (Session 77 — non-obvious)
|
||||||
|
- **`specs/odds-429-diagnosis.md`.** MEASURED: PropLine 5/3,000 daily (0.17%);
|
||||||
|
odds-api 478/500 MONTHLY, `allowed:false` (tracker blocks at 95%). One snapshot
|
||||||
|
= ONE PropLine call per sport (all markets comma-joined) — there is no
|
||||||
|
per-prop/per-book fan-out and no request-pattern problem to optimize.
|
||||||
|
- **The 429 text is the BACKUP's.** `oddsService.getOdds` falls through silently
|
||||||
|
when PropLine returns null/empty, then odds-api's quota gate throws
|
||||||
|
`429 "Odds data temporarily unavailable"`. So an EMPTY PropLine slate is
|
||||||
|
indistinguishable from an outage, and the error names the wrong provider. Open
|
||||||
|
order — don't read a 429 as "PropLine exhausted" without checking
|
||||||
|
`GET /api/internal/quota`.
|
||||||
|
- **BOOK BREADTH INVARIANT: we never discard books.** All books are KEPT and
|
||||||
|
SHOWN (`DISPLAY_BOOKS = MODEL ∪ REFERENCE ∪ DFS`). The ONLY selectivity is that
|
||||||
|
DFS pick'em is excluded from PRICING/consensus (`EXCLUDED_FROM_PRICING`) — a
|
||||||
|
fixed-payout shaded number is not a market price. Never "clean up" breadth.
|
||||||
|
|
||||||
## Active Skills
|
## Active Skills
|
||||||
- vyndr-voice (all user-facing output)
|
- vyndr-voice (all user-facing output)
|
||||||
- prop-analysis (grading methodology)
|
- prop-analysis (grading methodology)
|
||||||
|
|||||||
@@ -0,0 +1,191 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
'use strict';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* challenger-scoreboard — every accruing challenger, measured on the same bar.
|
||||||
|
*
|
||||||
|
* WHY THIS IS A REAL HOLDOUT AND NOT A BACKTEST. Unlike hits-v1 (which did not
|
||||||
|
* exist when these rows were graded and therefore needed a point-in-time
|
||||||
|
* replay), arch-v1 and contact-v1 wrote their probability AT GRADE TIME, into
|
||||||
|
* their own columns, before the game was played. Nothing here is recomputed.
|
||||||
|
* These numbers are genuinely out-of-sample — the strongest evidence available.
|
||||||
|
*
|
||||||
|
* THE BAR IS THE SAME ONE THAT REFUTED hits-v1:
|
||||||
|
* - the challenger's OWN rows only (a challenger that abstains is not scored
|
||||||
|
* on the rows it declined — averaging those in measures the champion twice)
|
||||||
|
* - direction handled: p_win and p_win_challenger/p_win_contact are all
|
||||||
|
* P(GRADED SIDE), so they are already aligned. The projection ladder is
|
||||||
|
* P(OVER) and IS realigned here.
|
||||||
|
* - paired bootstrap on matched rows, because both models score the SAME rows
|
||||||
|
* and treating their errors as independent overstates the uncertainty
|
||||||
|
* - PROMOTE only when the CI on (challenger − champion) excludes zero
|
||||||
|
*
|
||||||
|
* CONTAMINATION EXCLUSION: rows whose price/book were stamped from a
|
||||||
|
* non-takeable book are tagged `quarantine_reason LIKE 'nontakeable_book%'` and
|
||||||
|
* are excluded — their locked price describes a market you could not have bet.
|
||||||
|
*
|
||||||
|
* PROVENANCE: results are reported for all rows AND split by `model_version`,
|
||||||
|
* so if a verdict depends on the older `pre-retention-unknown` era that fact is
|
||||||
|
* visible rather than buried.
|
||||||
|
*
|
||||||
|
* SUPABASE_URL=... node scripts/challenger-scoreboard.js
|
||||||
|
*/
|
||||||
|
|
||||||
|
require('dotenv').config();
|
||||||
|
const { createClient } = require('@supabase/supabase-js');
|
||||||
|
|
||||||
|
const SB_URL = process.env.SUPABASE_URL;
|
||||||
|
const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
|
||||||
|
const PAGE = 1000;
|
||||||
|
|
||||||
|
function corr(xs, ys) {
|
||||||
|
const n = xs.length;
|
||||||
|
if (n < 3) return null;
|
||||||
|
const mx = xs.reduce((a, b) => a + b, 0) / n;
|
||||||
|
const my = ys.reduce((a, b) => a + b, 0) / n;
|
||||||
|
let sxy = 0; let sxx = 0; let syy = 0;
|
||||||
|
for (let i = 0; i < n; i += 1) {
|
||||||
|
const dx = xs[i] - mx; const dy = ys[i] - my;
|
||||||
|
sxy += dx * dy; sxx += dx * dx; syy += dy * dy;
|
||||||
|
}
|
||||||
|
if (sxx <= 0 || syy <= 0) return null;
|
||||||
|
return sxy / Math.sqrt(sxx * syy);
|
||||||
|
}
|
||||||
|
const r4 = (v) => (v == null ? null : Math.round(v * 10000) / 10000);
|
||||||
|
const meanOf = (a) => (a.length ? a.reduce((x, y) => x + y, 0) / a.length : null);
|
||||||
|
const brier = (ps, ys) => (ps.length ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null);
|
||||||
|
|
||||||
|
/** Paired bootstrap on the DIFFERENCE of resolutions. Deterministic seed. */
|
||||||
|
function bootstrapDiff(rows, keyA, keyB, iters = 4000, seed = 20260803) {
|
||||||
|
if (rows.length < 30) return null;
|
||||||
|
let s = seed >>> 0;
|
||||||
|
const rnd = () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
|
||||||
|
const n = rows.length;
|
||||||
|
const diffs = [];
|
||||||
|
for (let it = 0; it < iters; it += 1) {
|
||||||
|
const ys = []; const a = []; const b = [];
|
||||||
|
for (let i = 0; i < n; i += 1) {
|
||||||
|
const r = rows[Math.floor(rnd() * n)];
|
||||||
|
ys.push(r.won); a.push(r[keyA]); b.push(r[keyB]);
|
||||||
|
}
|
||||||
|
const ca = corr(a, ys); const cb = corr(b, ys);
|
||||||
|
if (ca == null || cb == null) continue;
|
||||||
|
diffs.push(ca - cb);
|
||||||
|
}
|
||||||
|
if (diffs.length < 100) return null;
|
||||||
|
diffs.sort((x, y) => x - y);
|
||||||
|
const q = (p) => r4(diffs[Math.floor(p * (diffs.length - 1))]);
|
||||||
|
const point = r4(corr(rows.map((r) => r[keyA]), rows.map((r) => r.won))
|
||||||
|
- corr(rows.map((r) => r[keyB]), rows.map((r) => r.won)));
|
||||||
|
const ci = [q(0.025), q(0.975)];
|
||||||
|
return { point, ci95: ci, p_improves: r4(diffs.filter((d) => d > 0).length / diffs.length),
|
||||||
|
ci_excludes_zero: ci[0] > 0 || ci[1] < 0 };
|
||||||
|
}
|
||||||
|
|
||||||
|
function score(rows, challKey, label) {
|
||||||
|
const ys = rows.map((r) => r.won);
|
||||||
|
const ch = rows.map((r) => r[challKey]);
|
||||||
|
const cp = rows.map((r) => r.champ);
|
||||||
|
const bs = bootstrapDiff(rows, challKey, 'champ');
|
||||||
|
let verdict = 'STILL PENDING';
|
||||||
|
if (rows.length >= 30 && bs) {
|
||||||
|
if (bs.ci_excludes_zero && bs.point > 0) verdict = 'PROMOTE';
|
||||||
|
else if (bs.ci_excludes_zero && bs.point < 0) verdict = 'STAY WIRED (measured worse)';
|
||||||
|
else verdict = 'STAY WIRED (inconclusive)';
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
challenger: label,
|
||||||
|
settled_n: rows.length,
|
||||||
|
base_rate: r4(meanOf(ys)),
|
||||||
|
resolution_challenger: r4(corr(ch, ys)),
|
||||||
|
resolution_champion: r4(corr(cp, ys)),
|
||||||
|
brier_challenger: r4(brier(ch, ys)),
|
||||||
|
brier_champion: r4(brier(cp, ys)),
|
||||||
|
delta_vs_champion: bs,
|
||||||
|
verdict,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchAll(sb) {
|
||||||
|
const out = [];
|
||||||
|
for (let from = 0; ; from += PAGE) {
|
||||||
|
const { data, error } = await sb.from('ledger_entries')
|
||||||
|
.select('id, stat, side, outcome, model_version, quarantine_reason, p_win, p_win_challenger, p_win_contact, challenger_delta, contact_delta, proj_p_over_line, proj_tb_p_over, proj_hits_p_over, challenger_adjustments')
|
||||||
|
.eq('sport', 'mlb').is('user_id', null)
|
||||||
|
.in('outcome', ['hit', 'miss'])
|
||||||
|
.not('p_win', 'is', null)
|
||||||
|
.range(from, from + PAGE - 1);
|
||||||
|
if (error) throw error;
|
||||||
|
if (!data || data.length === 0) break;
|
||||||
|
out.push(...data);
|
||||||
|
if (data.length < PAGE) break;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required');
|
||||||
|
const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
|
||||||
|
|
||||||
|
const raw = (await fetchAll(sb))
|
||||||
|
.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book'));
|
||||||
|
|
||||||
|
const base = raw.map((r) => ({
|
||||||
|
won: r.outcome === 'hit' ? 1 : 0,
|
||||||
|
champ: Number(r.p_win),
|
||||||
|
arch: r.p_win_challenger == null ? null : Number(r.p_win_challenger),
|
||||||
|
contact: r.p_win_contact == null ? null : Number(r.p_win_contact),
|
||||||
|
// The ladder is P(OVER); realign it to the graded side before comparing.
|
||||||
|
ladder: r.proj_p_over_line == null ? null
|
||||||
|
: (String(r.side).toLowerCase() === 'under' ? 1 - Number(r.proj_p_over_line) : Number(r.proj_p_over_line)),
|
||||||
|
tb: r.proj_tb_p_over == null ? null
|
||||||
|
: (String(r.side).toLowerCase() === 'under' ? 1 - Number(r.proj_tb_p_over) : Number(r.proj_tb_p_over)),
|
||||||
|
hits: r.proj_hits_p_over == null ? null
|
||||||
|
: (String(r.side).toLowerCase() === 'under' ? 1 - Number(r.proj_hits_p_over) : Number(r.proj_hits_p_over)),
|
||||||
|
stat: r.stat,
|
||||||
|
// Did the challenger actually MOVE this row? A nudge that leaves p_win
|
||||||
|
// untouched is the champion wearing a different name, and scoring it on
|
||||||
|
// those rows measures the champion against itself — which is exactly how a
|
||||||
|
// real effect gets averaged down to zero.
|
||||||
|
archMoved: r.challenger_delta != null && Number(r.challenger_delta) !== 0,
|
||||||
|
contactMoved: r.contact_delta != null && Number(r.contact_delta) !== 0,
|
||||||
|
era: r.model_version || 'unknown',
|
||||||
|
axes: new Set(((r.challenger_adjustments) || []).map((a) => a && a.axis).filter(Boolean)),
|
||||||
|
}));
|
||||||
|
|
||||||
|
const withKey = (k, extra = () => true) => base.filter((r) => r[k] != null && Number.isFinite(r[k]) && extra(r));
|
||||||
|
|
||||||
|
const board = [
|
||||||
|
score(withKey('arch'), 'arch', 'arch-v1 (market-relative nudge)'),
|
||||||
|
score(withKey('contact'), 'contact', 'contact-v1 (season contact quality)'),
|
||||||
|
score(withKey('ladder'), 'ladder', 'proj-v1.1 ladder (all stats)'),
|
||||||
|
score(withKey('tb', (r) => r.stat === 'total_bases'), 'tb', 'tb-v1 (total_bases only)'),
|
||||||
|
score(withKey('hits', (r) => r.stat === 'hits'), 'hits', 'hits-v1 (hits only)'),
|
||||||
|
];
|
||||||
|
|
||||||
|
// THE SHARPEST TEST OF A NUDGE — only the rows it actually moved.
|
||||||
|
const movedBoard = [
|
||||||
|
score(withKey('arch', (r) => r.archMoved), 'arch', 'arch-v1 · rows it MOVED only'),
|
||||||
|
score(withKey('contact', (r) => r.contactMoved), 'contact', 'contact-v1 · rows it MOVED only'),
|
||||||
|
];
|
||||||
|
|
||||||
|
// Per-AXIS: arch-v1 restricted to the rows where that axis actually fired.
|
||||||
|
const axisBoard = ['environment', 'opportunity', 'matchup'].map((ax) =>
|
||||||
|
score(withKey('arch', (r) => r.axes.has(ax)), 'arch', `arch-v1 · ${ax} axis rows only`));
|
||||||
|
|
||||||
|
// PROVENANCE split — does any verdict depend on the older era?
|
||||||
|
const eras = [...new Set(base.map((r) => r.era))];
|
||||||
|
const provenance = eras.map((era) => ({
|
||||||
|
era,
|
||||||
|
...score(withKey('arch', (r) => r.era === era), 'arch', `arch-v1 · ${era}`),
|
||||||
|
}));
|
||||||
|
|
||||||
|
console.log(JSON.stringify({
|
||||||
|
measurement: 'PROSPECTIVE HOLDOUT — challenger values were written at grade time, before the game. No recomputation, no lookahead.',
|
||||||
|
total_settled_rows: base.length,
|
||||||
|
board, movedBoard, axisBoard, provenance,
|
||||||
|
}, null, 2));
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((e) => { console.error(e); process.exit(1); });
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# CHALLENGER SCOREBOARD — 2026-08-03
|
||||||
|
|
||||||
|
> **Nothing was promoted. Nothing earned it yet.** Not because the bar was held
|
||||||
|
> too high, but because no challenger's confidence interval excludes zero on the
|
||||||
|
> good side. The champion is byte-identical; every challenger stays wired.
|
||||||
|
|
||||||
|
The headline of this session is not the scoreboard. It is that **the scoreboard
|
||||||
|
was unmeasurable until a two-day-old settlement outage was found and fixed** —
|
||||||
|
see §1. Settled sample went **493 → 1,741** the moment it was repaired.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why "n-blocked" was the wrong diagnosis
|
||||||
|
|
||||||
|
The order said: don't repeat "n-blocked" without counting. Counting is what found
|
||||||
|
the real problem.
|
||||||
|
|
||||||
|
Three of the four axes read **exactly zero** settled rows — not low, *zero*:
|
||||||
|
environment 1,496 rows / 0 settled, opportunity 922 / 0, matchup 603 / 0. Rows
|
||||||
|
whose games had been **played days earlier** and never settled, with
|
||||||
|
`settle_attempts = 0` — never even attempted.
|
||||||
|
|
||||||
|
**Root cause:** `settleLedger` fetched open ids, then refetched full rows via
|
||||||
|
`.in('id', ids)`. PostgREST puts filters in the URL, so 500 UUIDs became an
|
||||||
|
**18,499-character request** that the fetch layer rejects with `TypeError: fetch
|
||||||
|
failed`. The result was destructured as `const { data: rows } = ...` with **no
|
||||||
|
error binding**, so `rows` came back null, the loop never ran, and the function
|
||||||
|
returned `{settled:0, voided:0, unrecoverable:0, pending:0}` — byte-identical to
|
||||||
|
a healthy "nothing to settle."
|
||||||
|
|
||||||
|
It hid for two days because it is **volume-triggered**: daily volume ran 20–260
|
||||||
|
rows and settled perfectly for weeks. **2026-08-01 was the first day past the
|
||||||
|
500-row fetch limit** and settlement died that night. Worse, the zero-settle ops
|
||||||
|
alarm reads these same return values, so `pending: 0` told the watchdog the
|
||||||
|
backlog was empty — *the alarm built to catch exactly this could not see it.*
|
||||||
|
|
||||||
|
Fixed, deployed, and drained: **1,444 rows from 2026-08-01 settled (1,376
|
||||||
|
hit/miss + 68 void, 0 remaining).** `captureClosing` carried the same shape one
|
||||||
|
level down and is now chunked at 100 ids.
|
||||||
|
|
||||||
|
## 2. THE SCOREBOARD
|
||||||
|
|
||||||
|
Bar: the challenger's **own rows only**, direction-aligned, **paired bootstrap**
|
||||||
|
(4,000 resamples, deterministic seed) on the difference in resolution, because
|
||||||
|
both models score the same rows and independent standard errors would overstate
|
||||||
|
certainty. **PROMOTE requires the CI to exclude zero on the good side.** Same bar
|
||||||
|
that refuted hits-v1 — no lighter test for a would-be winner.
|
||||||
|
|
||||||
|
| challenger | settled n | resolution (chal / champ) | Δ vs champion | CI95 | verdict |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| arch-v1 (market-relative nudge) | **1,741** | 0.4599 / 0.4599 | 0.0000 | [−0.0050, +0.0054] | **STAY WIRED** (inconclusive) |
|
||||||
|
| arch-v1 · rows it MOVED only | 1,325 | 0.4886 / 0.4886 | −0.0001 | [−0.0064, +0.0061] | **STAY WIRED** (inconclusive) |
|
||||||
|
| contact-v1 (season contact quality) | **1,055** | 0.4070 / 0.4061 | +0.0008 | [−0.0052, +0.0069] | **STAY WIRED** (inconclusive) |
|
||||||
|
| contact-v1 · rows it MOVED only | 511 | 0.3966 / 0.3948 | +0.0018 | [−0.0107, +0.0144] | **STAY WIRED** (inconclusive) |
|
||||||
|
| proj-v1.1 ladder (all stats) | **1,664** | 0.4339 / 0.4640 | **−0.0301** | **[−0.0543, −0.0060]** | **STAY WIRED** (measured WORSE) |
|
||||||
|
| arch-v1 · environment axis rows | 871 | 0.5651 / 0.5679 | −0.0028 | [−0.0089, +0.0036] | **STAY WIRED** (inconclusive) |
|
||||||
|
| arch-v1 · opportunity axis rows | 539 | 0.5311 / 0.5310 | +0.0001 | [−0.0091, +0.0090] | **STAY WIRED** (inconclusive) |
|
||||||
|
| arch-v1 · matchup axis rows | **0** | — | — | — | **STILL PENDING** |
|
||||||
|
| tb-v1 (total_bases only) | **0** | — | — | — | **STILL PENDING** |
|
||||||
|
| hits-v1 (hits only) | **0** | — | — | — | **STILL PENDING** (refuted by replay, `specs/hits-v1-binomial.md`) |
|
||||||
|
|
||||||
|
**These are true prospective holdouts, not backtests.** arch-v1 and contact-v1
|
||||||
|
wrote their probability at grade time, into their own columns, before the game
|
||||||
|
was played. Nothing was recomputed. That is the strongest evidence available and
|
||||||
|
it is why no replay was needed here.
|
||||||
|
|
||||||
|
## 3. What the numbers actually say
|
||||||
|
|
||||||
|
**arch-v1 moves a lot and changes nothing.** It moved **1,325 of 1,741 rows
|
||||||
|
(76%)**, mean absolute move **2.5 points**, max 10.9 — and resolution is
|
||||||
|
identical to the champion to four decimal places, on the moved rows too. This is
|
||||||
|
not "too small to detect." It is movement that carries **no information about the
|
||||||
|
outcome**. A nudge this active with an effect this precisely zero is a finding,
|
||||||
|
not a pending verdict.
|
||||||
|
|
||||||
|
**The projection ladder is reliably worse than the champion.** −0.0301 with a CI
|
||||||
|
excluding zero, across 1,664 rows and all stats. Combined with hits-v1's refutation
|
||||||
|
(`specs/hits-v1-binomial.md`), the projection family now has two independent
|
||||||
|
measurements pointing the same way: it is not the champion's equal on any stat
|
||||||
|
measured so far. That is an argument for diagnosing its *inputs*, not for shipping
|
||||||
|
another variant of it.
|
||||||
|
|
||||||
|
**Three are genuinely pending, for a legitimate reason now.** matchup, tb-v1 and
|
||||||
|
hits-v1 all have rows written only on 2026-08-02/03, which settle after ET
|
||||||
|
midnight. matchup has ~496 rows queued, tb-v1 65, hits-v1 pending its first
|
||||||
|
snapshot write. They will read within a day or two — and now that settlement
|
||||||
|
works, they actually will.
|
||||||
|
|
||||||
|
## 4. Provenance
|
||||||
|
|
||||||
|
All 1,741 arch-v1 rows carry a single `model_version` (`engine1@2026-07-20`). The
|
||||||
|
older `pre-retention-unknown` rows (320 settled) carry no challenger values at
|
||||||
|
all, so they cannot influence any verdict. **No verdict here depends on
|
||||||
|
mixed-provenance rows** — the split was checked, not assumed.
|
||||||
|
|
||||||
|
Contamination excluded throughout: `quarantine_reason LIKE 'nontakeable_book%'`.
|
||||||
|
|
||||||
|
## 5. Promotion mechanics — specified, deliberately unused
|
||||||
|
|
||||||
|
No flip was performed because nothing qualified. When one does, the shape is:
|
||||||
|
challenger-first (write the promoted value into the served path while the
|
||||||
|
champion column keeps recording), version-tagged, atomic, with the previous
|
||||||
|
version one env flag away. Recorded here so a future promotion is a decision,
|
||||||
|
not an improvisation.
|
||||||
|
|
||||||
|
## 6. Reproduce
|
||||||
|
|
||||||
|
`SUPABASE_URL=... node scripts/challenger-scoreboard.js` — prints the full board,
|
||||||
|
the moved-rows-only slice, the per-axis slice and the provenance split.
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
# THE 429 — DIAGNOSED, AND IT IS NOT WHAT WE THOUGHT
|
||||||
|
|
||||||
|
**Read-only investigation, 2026-08-03.** Nothing was implemented, no tier bought.
|
||||||
|
|
||||||
|
> **VERDICT: the change-based pull is NOT the fix, and no tier upgrade is
|
||||||
|
> needed. There is no request-pattern problem to solve.** PropLine is running at
|
||||||
|
> **5 calls out of 3,000/day — 0.17% of the free allowance.** The 429 comes from a
|
||||||
|
> different provider entirely.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. BOOK BREADTH — the invariant, written down (and already true in code)
|
||||||
|
|
||||||
|
**WE DO NOT DISCARD BOOKS.** All books are KEPT and SHOWN. More breadth is
|
||||||
|
strictly good for display, line-shopping and consensus. The **only** selectivity
|
||||||
|
is that **DFS pick'em (prizepicks / underdog / sleeper / dabble) does not vote on
|
||||||
|
the fair-price CONSENSUS**, because a fixed-payout shaded number is not a market
|
||||||
|
price. That is one category exclusion from *pricing*, not a reduction in breadth.
|
||||||
|
|
||||||
|
Verified in `src/config/bookRoles.js` — this is already exactly what the code
|
||||||
|
does, and the invariant is recorded so no future "cleanup" undoes it:
|
||||||
|
|
||||||
|
```
|
||||||
|
DISPLAY_BOOKS = MODEL_BOOKS ∪ REFERENCE_BOOKS ∪ DFS_PLATFORMS ← everything shows
|
||||||
|
EXCLUDED_FROM_PRICING = DFS_PLATFORMS ∪ OFFSHORE_OR_INTL ← pricing only
|
||||||
|
```
|
||||||
|
|
||||||
|
The historical real discarding — `ALLOWED_BOOKS` cutting the slate to 5 keys and
|
||||||
|
making 64.8% of MLB props invisible — was fixed in Order Zero and must not
|
||||||
|
return. No language in this repo should describe book handling as "discarding".
|
||||||
|
|
||||||
|
## 2. Post-reset status (STEP 1) — the pipeline RECOVERED
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/internal/snapshot/status → overdue: false
|
||||||
|
mlb 2026-08-03T01:00:24Z gradeCount 235
|
||||||
|
wnba 2026-08-03T01:02:58Z gradeCount 139
|
||||||
|
```
|
||||||
|
|
||||||
|
The 14/19/22 UTC slots on 2026-08-02 all failed; the 01:00 UTC slot after the
|
||||||
|
reset succeeded. So the failure was time-bounded and self-healing — which already
|
||||||
|
argued against a structural request-pattern defect.
|
||||||
|
|
||||||
|
## 3. What one snapshot actually requests (STEP 2)
|
||||||
|
|
||||||
|
Measured in `proplineAdapter.fetchRaw`: **one snapshot = ONE PropLine HTTP call
|
||||||
|
per sport.** All markets ride the same request, comma-joined (12 markets for MLB).
|
||||||
|
There is no per-prop, per-book or per-event fan-out.
|
||||||
|
|
||||||
|
```
|
||||||
|
snapshot 1 call × 2 active sports × 5 slots/day ≈ 10 calls/day
|
||||||
|
intraday refresh every 20 min, slate hours only (12h) ≈ 72 calls/day
|
||||||
|
public /api/odds cache-aside, 1h TTL bounded, small
|
||||||
|
── total order-of-10s/day
|
||||||
|
```
|
||||||
|
|
||||||
|
**MEASURED, not estimated** — `GET /api/internal/quota`, 2026-08-03:
|
||||||
|
|
||||||
|
| provider | used | limit | period | allowed |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **propline** | **5** | **3,000** | daily | ✅ true |
|
||||||
|
| **odds-api** | **478** | **500** | **monthly** | ❌ **false (95.6%)** |
|
||||||
|
| oddspapi | 0 | 1,000 | monthly | true |
|
||||||
|
| tank01 | 86 | 1,000 | monthly | true |
|
||||||
|
|
||||||
|
**There is no waste to reclaim.** We are using 0.17% of the free PropLine
|
||||||
|
allowance. Re-pulling "the full picture" every slot costs one request; a
|
||||||
|
change-based pull would replace one request with one request.
|
||||||
|
|
||||||
|
## 4. Where the 429 actually comes from
|
||||||
|
|
||||||
|
`oddsService.getOdds` tries PropLine first, and **falls through silently** when it
|
||||||
|
returns null or an empty array. The 429 is thrown *after* that, by the **odds-api
|
||||||
|
backup's** quota gate:
|
||||||
|
|
||||||
|
```js
|
||||||
|
const quotaStatus = await quotaTracker.getQuotaStatus('odds-api');
|
||||||
|
if (!quotaStatus.allowed) {
|
||||||
|
const error = new Error('Odds data temporarily unavailable. Try again later.');
|
||||||
|
error.statusCode = 429; // ← THIS is the 429 we saw
|
||||||
|
```
|
||||||
|
|
||||||
|
So the real sequence on 2026-08-02 was:
|
||||||
|
|
||||||
|
1. PropLine returned **no props** for that moment (it had 2,995 calls remaining —
|
||||||
|
this was an empty/failed slate, **not** exhaustion),
|
||||||
|
2. control fell through to odds-api,
|
||||||
|
3. odds-api sits at **478/500 for the month** and the tracker blocks at ≥95%,
|
||||||
|
4. → user-facing `429 Odds data temporarily unavailable`.
|
||||||
|
|
||||||
|
**The 429 is the dead backup announcing itself.** It is not a PropLine quota
|
||||||
|
signal, and tuning PropLine request volume cannot affect it.
|
||||||
|
|
||||||
|
### The real defect this exposes
|
||||||
|
|
||||||
|
**An empty PropLine slate is indistinguishable from a provider outage.** A
|
||||||
|
legitimately empty board (off-hours, no props posted yet) produces the identical
|
||||||
|
429 as a genuine failure, because the fall-through is silent and the error text
|
||||||
|
belongs to whichever provider happened to be asked last. That is a
|
||||||
|
truth-in-reporting bug in the odds path — the operator cannot tell "no games" from
|
||||||
|
"we are broken." Flagged for its own order; **not** fixed here (read-only).
|
||||||
|
|
||||||
|
## 5. Change-based capture (STEP 3) — NOT VERIFIED, and moot for quota
|
||||||
|
|
||||||
|
**I could not verify whether PropLine supports `/movement`, odds-history, or a
|
||||||
|
`since=` parameter.** PropLine's docs are auth-gated (HTTP 401) and the API keys
|
||||||
|
are production-only — not present in the local environment. **I am not going to
|
||||||
|
assert an endpoint exists or costs less when I could not test it**; "documented ≠
|
||||||
|
verified" cuts both ways.
|
||||||
|
|
||||||
|
What can be said without testing anything:
|
||||||
|
|
||||||
|
- **For quota, it is moot.** At 5/3,000 there is nothing to optimise. A
|
||||||
|
change-based pull would save at most a handful of calls per day out of an
|
||||||
|
allowance we are not close to touching.
|
||||||
|
- **The movement-data argument stands on its own, and is the better reason.** A
|
||||||
|
change-based capture records line movement as `from → to at time T`, which is
|
||||||
|
genuine model input we currently lack: steam detection, true CLV attribution,
|
||||||
|
and the market-movement axis. Today `intradayRefreshService.trackHistory`
|
||||||
|
reconstructs a coarse version of this by diffing full pulls. If PropLine
|
||||||
|
exposes movement natively it is worth building **as a data feature** — and it
|
||||||
|
should be justified that way, not as a quota fix it isn't.
|
||||||
|
|
||||||
|
## 6. VERDICT (STEP 4), with the numbers
|
||||||
|
|
||||||
|
| question | answer |
|
||||||
|
|---|---|
|
||||||
|
| Did the pipeline recover post-reset? | **Yes** — `overdue: false`, mlb 235 grades at 01:00Z |
|
||||||
|
| Requests per snapshot | **1 per sport**, all markets in one call |
|
||||||
|
| Requests/day vs allowance | **~10s/day vs 3,000/day free — 0.17% used** |
|
||||||
|
| Is the 429 a request-pattern problem? | **No.** Measured, decisively |
|
||||||
|
| Is the 429 a PropLine quota problem? | **No.** 2,995 calls remaining, `allowed: true` |
|
||||||
|
| What is the 429? | **odds-api backup at 478/500 monthly, blocked at 95%** |
|
||||||
|
| Does change-based pull fix it? | **No** — it cannot; wrong provider, and no volume problem |
|
||||||
|
| Is a tier upgrade needed? | **No.** Not for PropLine. Nothing is constrained |
|
||||||
|
|
||||||
|
**What actually deserves an order**, in priority sequence:
|
||||||
|
|
||||||
|
1. **The silent fall-through** — an empty PropLine slate must report "no props
|
||||||
|
available", never the backup's quota 429 (§4).
|
||||||
|
2. **odds-api at 478/500 monthly** — it is a dead backup for the rest of August.
|
||||||
|
Either accept it (PropLine is primary and healthy) and stop letting its quota
|
||||||
|
gate speak for the whole odds path, or budget it deliberately.
|
||||||
|
3. **Movement capture as a DATA feature** — worth scoping on its own merits, with
|
||||||
|
PropLine's endpoint support verified first from an environment that has keys.
|
||||||
Reference in New Issue
Block a user