Scoreboard every challenger; diagnose the 429 as odds-api, not PropLine
PROMOTE-THE-EARNED. Nothing was promoted, because nothing earned it -- not because the bar was held high. Measured on the same bar that refuted hits-v1: own rows only, direction-aligned, paired bootstrap, promote only on a CI excluding zero. arch-v1 n=1741 delta 0.0000 CI[-0.0050,+0.0054] inconclusive contact-v1 n=1055 delta +0.0008 CI[-0.0052,+0.0069] inconclusive proj-v1.1 n=1664 delta -0.0301 CI[-0.0543,-0.0060] reliably WORSE matchup/tb-v1/hits-v1 n=0 genuinely pending (rows dated 08-02+) arch-v1 is the interesting one: it MOVED 76% of rows by 2.5 points on average and resolution is identical to the champion to four decimals, on the moved rows too. That is active movement carrying no information -- a finding, not a pending verdict. These are true prospective holdouts: arch-v1 and contact-v1 wrote p_win at grade time into their own columns before the game. Nothing recomputed. THE 429, read-only. The premise was that we re-pull the full picture every slot and blow the quota. Measured: PropLine is at 5 calls of 3,000/day -- 0.17%. One snapshot is ONE PropLine call per sport, all markets comma-joined. There is no request-pattern problem, so a change-based pull cannot fix it and no tier upgrade is needed. The 429 is odds-api: 478/500 MONTHLY, blocked at 95%. oddsService falls through silently when PropLine returns empty, and the backup's quota gate throws the error -- so an empty slate is indistinguishable from an outage and the message names the wrong provider. Flagged for its own order. Could NOT verify PropLine movement endpoints: docs are auth-gated and the keys are production-only. Not asserted either way. The movement-as-data argument stands on its own merits and should be justified that way, not as a quota fix it isn't. Book-breadth invariant written down: we never discard books. All are kept and shown (DISPLAY_BOOKS = MODEL + REFERENCE + DFS); DFS pick'em is excluded from PRICING only, because a fixed-payout shaded number is not a market price. Verified this is already what bookRoles.js does. Champion byte-identical; every challenger stays wired. 4,159 tests green (332 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,109 @@
|
||||
# CHALLENGER SCOREBOARD — 2026-08-03
|
||||
|
||||
> **Nothing was promoted. Nothing earned it yet.** Not because the bar was held
|
||||
> too high, but because no challenger's confidence interval excludes zero on the
|
||||
> good side. The champion is byte-identical; every challenger stays wired.
|
||||
|
||||
The headline of this session is not the scoreboard. It is that **the scoreboard
|
||||
was unmeasurable until a two-day-old settlement outage was found and fixed** —
|
||||
see §1. Settled sample went **493 → 1,741** the moment it was repaired.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why "n-blocked" was the wrong diagnosis
|
||||
|
||||
The order said: don't repeat "n-blocked" without counting. Counting is what found
|
||||
the real problem.
|
||||
|
||||
Three of the four axes read **exactly zero** settled rows — not low, *zero*:
|
||||
environment 1,496 rows / 0 settled, opportunity 922 / 0, matchup 603 / 0. Rows
|
||||
whose games had been **played days earlier** and never settled, with
|
||||
`settle_attempts = 0` — never even attempted.
|
||||
|
||||
**Root cause:** `settleLedger` fetched open ids, then refetched full rows via
|
||||
`.in('id', ids)`. PostgREST puts filters in the URL, so 500 UUIDs became an
|
||||
**18,499-character request** that the fetch layer rejects with `TypeError: fetch
|
||||
failed`. The result was destructured as `const { data: rows } = ...` with **no
|
||||
error binding**, so `rows` came back null, the loop never ran, and the function
|
||||
returned `{settled:0, voided:0, unrecoverable:0, pending:0}` — byte-identical to
|
||||
a healthy "nothing to settle."
|
||||
|
||||
It hid for two days because it is **volume-triggered**: daily volume ran 20–260
|
||||
rows and settled perfectly for weeks. **2026-08-01 was the first day past the
|
||||
500-row fetch limit** and settlement died that night. Worse, the zero-settle ops
|
||||
alarm reads these same return values, so `pending: 0` told the watchdog the
|
||||
backlog was empty — *the alarm built to catch exactly this could not see it.*
|
||||
|
||||
Fixed, deployed, and drained: **1,444 rows from 2026-08-01 settled (1,376
|
||||
hit/miss + 68 void, 0 remaining).** `captureClosing` carried the same shape one
|
||||
level down and is now chunked at 100 ids.
|
||||
|
||||
## 2. THE SCOREBOARD
|
||||
|
||||
Bar: the challenger's **own rows only**, direction-aligned, **paired bootstrap**
|
||||
(4,000 resamples, deterministic seed) on the difference in resolution, because
|
||||
both models score the same rows and independent standard errors would overstate
|
||||
certainty. **PROMOTE requires the CI to exclude zero on the good side.** Same bar
|
||||
that refuted hits-v1 — no lighter test for a would-be winner.
|
||||
|
||||
| challenger | settled n | resolution (chal / champ) | Δ vs champion | CI95 | verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| arch-v1 (market-relative nudge) | **1,741** | 0.4599 / 0.4599 | 0.0000 | [−0.0050, +0.0054] | **STAY WIRED** (inconclusive) |
|
||||
| arch-v1 · rows it MOVED only | 1,325 | 0.4886 / 0.4886 | −0.0001 | [−0.0064, +0.0061] | **STAY WIRED** (inconclusive) |
|
||||
| contact-v1 (season contact quality) | **1,055** | 0.4070 / 0.4061 | +0.0008 | [−0.0052, +0.0069] | **STAY WIRED** (inconclusive) |
|
||||
| contact-v1 · rows it MOVED only | 511 | 0.3966 / 0.3948 | +0.0018 | [−0.0107, +0.0144] | **STAY WIRED** (inconclusive) |
|
||||
| proj-v1.1 ladder (all stats) | **1,664** | 0.4339 / 0.4640 | **−0.0301** | **[−0.0543, −0.0060]** | **STAY WIRED** (measured WORSE) |
|
||||
| arch-v1 · environment axis rows | 871 | 0.5651 / 0.5679 | −0.0028 | [−0.0089, +0.0036] | **STAY WIRED** (inconclusive) |
|
||||
| arch-v1 · opportunity axis rows | 539 | 0.5311 / 0.5310 | +0.0001 | [−0.0091, +0.0090] | **STAY WIRED** (inconclusive) |
|
||||
| arch-v1 · matchup axis rows | **0** | — | — | — | **STILL PENDING** |
|
||||
| tb-v1 (total_bases only) | **0** | — | — | — | **STILL PENDING** |
|
||||
| hits-v1 (hits only) | **0** | — | — | — | **STILL PENDING** (refuted by replay, `specs/hits-v1-binomial.md`) |
|
||||
|
||||
**These are true prospective holdouts, not backtests.** arch-v1 and contact-v1
|
||||
wrote their probability at grade time, into their own columns, before the game
|
||||
was played. Nothing was recomputed. That is the strongest evidence available and
|
||||
it is why no replay was needed here.
|
||||
|
||||
## 3. What the numbers actually say
|
||||
|
||||
**arch-v1 moves a lot and changes nothing.** It moved **1,325 of 1,741 rows
|
||||
(76%)**, mean absolute move **2.5 points**, max 10.9 — and resolution is
|
||||
identical to the champion to four decimal places, on the moved rows too. This is
|
||||
not "too small to detect." It is movement that carries **no information about the
|
||||
outcome**. A nudge this active with an effect this precisely zero is a finding,
|
||||
not a pending verdict.
|
||||
|
||||
**The projection ladder is reliably worse than the champion.** −0.0301 with a CI
|
||||
excluding zero, across 1,664 rows and all stats. Combined with hits-v1's refutation
|
||||
(`specs/hits-v1-binomial.md`), the projection family now has two independent
|
||||
measurements pointing the same way: it is not the champion's equal on any stat
|
||||
measured so far. That is an argument for diagnosing its *inputs*, not for shipping
|
||||
another variant of it.
|
||||
|
||||
**Three are genuinely pending, for a legitimate reason now.** matchup, tb-v1 and
|
||||
hits-v1 all have rows written only on 2026-08-02/03, which settle after ET
|
||||
midnight. matchup has ~496 rows queued, tb-v1 65, hits-v1 pending its first
|
||||
snapshot write. They will read within a day or two — and now that settlement
|
||||
works, they actually will.
|
||||
|
||||
## 4. Provenance
|
||||
|
||||
All 1,741 arch-v1 rows carry a single `model_version` (`engine1@2026-07-20`). The
|
||||
older `pre-retention-unknown` rows (320 settled) carry no challenger values at
|
||||
all, so they cannot influence any verdict. **No verdict here depends on
|
||||
mixed-provenance rows** — the split was checked, not assumed.
|
||||
|
||||
Contamination excluded throughout: `quarantine_reason LIKE 'nontakeable_book%'`.
|
||||
|
||||
## 5. Promotion mechanics — specified, deliberately unused
|
||||
|
||||
No flip was performed because nothing qualified. When one does, the shape is:
|
||||
challenger-first (write the promoted value into the served path while the
|
||||
champion column keeps recording), version-tagged, atomic, with the previous
|
||||
version one env flag away. Recorded here so a future promotion is a decision,
|
||||
not an improvisation.
|
||||
|
||||
## 6. Reproduce
|
||||
|
||||
`SUPABASE_URL=... node scripts/challenger-scoreboard.js` — prints the full board,
|
||||
the moved-rows-only slice, the per-axis slice and the provenance split.
|
||||
@@ -0,0 +1,145 @@
|
||||
# THE 429 — DIAGNOSED, AND IT IS NOT WHAT WE THOUGHT
|
||||
|
||||
**Read-only investigation, 2026-08-03.** Nothing was implemented, no tier bought.
|
||||
|
||||
> **VERDICT: the change-based pull is NOT the fix, and no tier upgrade is
|
||||
> needed. There is no request-pattern problem to solve.** PropLine is running at
|
||||
> **5 calls out of 3,000/day — 0.17% of the free allowance.** The 429 comes from a
|
||||
> different provider entirely.
|
||||
|
||||
---
|
||||
|
||||
## 1. BOOK BREADTH — the invariant, written down (and already true in code)
|
||||
|
||||
**WE DO NOT DISCARD BOOKS.** All books are KEPT and SHOWN. More breadth is
|
||||
strictly good for display, line-shopping and consensus. The **only** selectivity
|
||||
is that **DFS pick'em (prizepicks / underdog / sleeper / dabble) does not vote on
|
||||
the fair-price CONSENSUS**, because a fixed-payout shaded number is not a market
|
||||
price. That is one category exclusion from *pricing*, not a reduction in breadth.
|
||||
|
||||
Verified in `src/config/bookRoles.js` — this is already exactly what the code
|
||||
does, and the invariant is recorded so no future "cleanup" undoes it:
|
||||
|
||||
```
|
||||
DISPLAY_BOOKS = MODEL_BOOKS ∪ REFERENCE_BOOKS ∪ DFS_PLATFORMS ← everything shows
|
||||
EXCLUDED_FROM_PRICING = DFS_PLATFORMS ∪ OFFSHORE_OR_INTL ← pricing only
|
||||
```
|
||||
|
||||
The historical real discarding — `ALLOWED_BOOKS` cutting the slate to 5 keys and
|
||||
making 64.8% of MLB props invisible — was fixed in Order Zero and must not
|
||||
return. No language in this repo should describe book handling as "discarding".
|
||||
|
||||
## 2. Post-reset status (STEP 1) — the pipeline RECOVERED
|
||||
|
||||
```
|
||||
GET /api/internal/snapshot/status → overdue: false
|
||||
mlb 2026-08-03T01:00:24Z gradeCount 235
|
||||
wnba 2026-08-03T01:02:58Z gradeCount 139
|
||||
```
|
||||
|
||||
The 14/19/22 UTC slots on 2026-08-02 all failed; the 01:00 UTC slot after the
|
||||
reset succeeded. So the failure was time-bounded and self-healing — which already
|
||||
argued against a structural request-pattern defect.
|
||||
|
||||
## 3. What one snapshot actually requests (STEP 2)
|
||||
|
||||
Measured in `proplineAdapter.fetchRaw`: **one snapshot = ONE PropLine HTTP call
|
||||
per sport.** All markets ride the same request, comma-joined (12 markets for MLB).
|
||||
There is no per-prop, per-book or per-event fan-out.
|
||||
|
||||
```
|
||||
snapshot 1 call × 2 active sports × 5 slots/day ≈ 10 calls/day
|
||||
intraday refresh every 20 min, slate hours only (12h) ≈ 72 calls/day
|
||||
public /api/odds cache-aside, 1h TTL bounded, small
|
||||
── total order-of-10s/day
|
||||
```
|
||||
|
||||
**MEASURED, not estimated** — `GET /api/internal/quota`, 2026-08-03:
|
||||
|
||||
| provider | used | limit | period | allowed |
|
||||
|---|---|---|---|---|
|
||||
| **propline** | **5** | **3,000** | daily | ✅ true |
|
||||
| **odds-api** | **478** | **500** | **monthly** | ❌ **false (95.6%)** |
|
||||
| oddspapi | 0 | 1,000 | monthly | true |
|
||||
| tank01 | 86 | 1,000 | monthly | true |
|
||||
|
||||
**There is no waste to reclaim.** We are using 0.17% of the free PropLine
|
||||
allowance. Re-pulling "the full picture" every slot costs one request; a
|
||||
change-based pull would replace one request with one request.
|
||||
|
||||
## 4. Where the 429 actually comes from
|
||||
|
||||
`oddsService.getOdds` tries PropLine first, and **falls through silently** when it
|
||||
returns null or an empty array. The 429 is thrown *after* that, by the **odds-api
|
||||
backup's** quota gate:
|
||||
|
||||
```js
|
||||
const quotaStatus = await quotaTracker.getQuotaStatus('odds-api');
|
||||
if (!quotaStatus.allowed) {
|
||||
const error = new Error('Odds data temporarily unavailable. Try again later.');
|
||||
error.statusCode = 429; // ← THIS is the 429 we saw
|
||||
```
|
||||
|
||||
So the real sequence on 2026-08-02 was:
|
||||
|
||||
1. PropLine returned **no props** for that moment (it had 2,995 calls remaining —
|
||||
this was an empty/failed slate, **not** exhaustion),
|
||||
2. control fell through to odds-api,
|
||||
3. odds-api sits at **478/500 for the month** and the tracker blocks at ≥95%,
|
||||
4. → user-facing `429 Odds data temporarily unavailable`.
|
||||
|
||||
**The 429 is the dead backup announcing itself.** It is not a PropLine quota
|
||||
signal, and tuning PropLine request volume cannot affect it.
|
||||
|
||||
### The real defect this exposes
|
||||
|
||||
**An empty PropLine slate is indistinguishable from a provider outage.** A
|
||||
legitimately empty board (off-hours, no props posted yet) produces the identical
|
||||
429 as a genuine failure, because the fall-through is silent and the error text
|
||||
belongs to whichever provider happened to be asked last. That is a
|
||||
truth-in-reporting bug in the odds path — the operator cannot tell "no games" from
|
||||
"we are broken." Flagged for its own order; **not** fixed here (read-only).
|
||||
|
||||
## 5. Change-based capture (STEP 3) — NOT VERIFIED, and moot for quota
|
||||
|
||||
**I could not verify whether PropLine supports `/movement`, odds-history, or a
|
||||
`since=` parameter.** PropLine's docs are auth-gated (HTTP 401) and the API keys
|
||||
are production-only — not present in the local environment. **I am not going to
|
||||
assert an endpoint exists or costs less when I could not test it**; "documented ≠
|
||||
verified" cuts both ways.
|
||||
|
||||
What can be said without testing anything:
|
||||
|
||||
- **For quota, it is moot.** At 5/3,000 there is nothing to optimise. A
|
||||
change-based pull would save at most a handful of calls per day out of an
|
||||
allowance we are not close to touching.
|
||||
- **The movement-data argument stands on its own, and is the better reason.** A
|
||||
change-based capture records line movement as `from → to at time T`, which is
|
||||
genuine model input we currently lack: steam detection, true CLV attribution,
|
||||
and the market-movement axis. Today `intradayRefreshService.trackHistory`
|
||||
reconstructs a coarse version of this by diffing full pulls. If PropLine
|
||||
exposes movement natively it is worth building **as a data feature** — and it
|
||||
should be justified that way, not as a quota fix it isn't.
|
||||
|
||||
## 6. VERDICT (STEP 4), with the numbers
|
||||
|
||||
| question | answer |
|
||||
|---|---|
|
||||
| Did the pipeline recover post-reset? | **Yes** — `overdue: false`, mlb 235 grades at 01:00Z |
|
||||
| Requests per snapshot | **1 per sport**, all markets in one call |
|
||||
| Requests/day vs allowance | **~10s/day vs 3,000/day free — 0.17% used** |
|
||||
| Is the 429 a request-pattern problem? | **No.** Measured, decisively |
|
||||
| Is the 429 a PropLine quota problem? | **No.** 2,995 calls remaining, `allowed: true` |
|
||||
| What is the 429? | **odds-api backup at 478/500 monthly, blocked at 95%** |
|
||||
| Does change-based pull fix it? | **No** — it cannot; wrong provider, and no volume problem |
|
||||
| Is a tier upgrade needed? | **No.** Not for PropLine. Nothing is constrained |
|
||||
|
||||
**What actually deserves an order**, in priority sequence:
|
||||
|
||||
1. **The silent fall-through** — an empty PropLine slate must report "no props
|
||||
available", never the backup's quota 429 (§4).
|
||||
2. **odds-api at 478/500 monthly** — it is a dead backup for the rest of August.
|
||||
Either accept it (PropLine is primary and healthy) and stop letting its quota
|
||||
gate speak for the whole odds path, or budget it deliberately.
|
||||
3. **Movement capture as a DATA feature** — worth scoping on its own merits, with
|
||||
PropLine's endpoint support verified first from an environment that has keys.
|
||||
Reference in New Issue
Block a user