Raise the grade cap 25 -> 500 on measured cost; refusals are correct
PART 1 (read-only, measured on a live prod slate, n=80) OVERTURNS THE
PREMISE. The refusal rate is not a data problem -- it is 98% correct
behaviour. The cap is the entire problem, and it is worse than "25 of 546".
Composition: GRADED 44 (55.0%) | POLICY-SUPPRESSION 35 (43.8%) |
FETCHABLE-GAP 1 (1.3%) | FALSE-THRESHOLD 0 | ARCHETYPE-GAP 0 |
GENUINE-ABSENCE 0.
THE FIFTH BUCKET the order did not anticipate: all 35 "refusals" are
rare_event_over_below_line -- the 2026-07-19 betting-logic audit
deliberately refusing 0.5-line rare events, setting the SAME
insufficient_data flag as a real data gap, which is why they read as one.
They are entirely doubles (18) and stolen_bases (17), while hits (19/19),
rbi (19/19) and total_bases (5/5) grade at ~100%. Had we "fixed" this we
would have re-introduced exactly the bets a previous audit removed, and the
count would have looked like progress.
THE CAP: 585 unique gradeable props, cap 25 -> 560 discarded (95.7%).
Traced to Session 32 (f0c8b4f), commented "bound the herd" -- a guard
written before anyone measured what a grade costs. So I measured it:
721ms mean / 666ms median / 1024ms p90 per grade => ~72s for 500 props at
concurrency 5. Both callers tolerate that: the cron runs 5x/day and
recordDownstream is fire-and-forget.
PART 2 -- item 3 ONLY, because that is what the diagnosis supports.
DEFAULT_LIMIT 25 -> 500, env-tunable via GRADE_SLATE_LIMIT. Concurrency
stays 5 deliberately: the cap raise already multiplies load ~20x, and
concurrency decides how hard we hit statsapi at once. One variable at a
time.
Items 4/5/6 have nothing to act on and I am not manufacturing work for
them: 0 false thresholds to loosen (loosening would be manufacturing
grades); /context wiring is worth doing for grade QUALITY but would not
have graded one extra prop here, so it is not claimed as a coverage win;
archetypes are display-side and do not gate grading at all.
THE REFUSAL RATE DOES NOT DROP, AND THAT IS CORRECT. No threshold lowered,
no grade forced. The board grows because the cap stops discarding 95.7% of
the slate.
Flagged in advance rather than discovered later: snapshot payload and
ledger volume both scale with the same multiple. If the response gets
unwieldy the fix is a response-side cap on what the BOARD returns, never a
re-cap on what gets graded -- grading everything and serving a slice is
honest; grading a slice and calling it the slate is what this fixes.
Gates: 4,052 tests / 324 suites green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
This commit is contained in:
@@ -0,0 +1,153 @@
|
||||
# THE 25-CAP + THE "72% REFUSAL" — DIAGNOSIS AND FIX
|
||||
|
||||
**Date:** 2026-08-01 · Part 1 **read-only, measured on prod** · Part 2 fix gated
|
||||
on it. Queries/tooling committed (`src/services/refusalDiagnostics.js`,
|
||||
`GET /api/internal/diagnose-refusals`).
|
||||
|
||||
---
|
||||
|
||||
## VERDICT UP FRONT — the premise is half right, and the wrong half is the one we were chasing
|
||||
|
||||
> **The refusal rate is not a data problem. It is 98% correct behaviour.**
|
||||
> **The cap is the entire problem, and it is worse than "25 of 546".**
|
||||
|
||||
Measured on a live MLB slate (n=80 graded through the real path):
|
||||
|
||||
| bucket | count | share | verdict |
|
||||
|---|---:|---:|---|
|
||||
| **GRADED** | **44** | **55.0%** | — |
|
||||
| **(e) POLICY-SUPPRESSION** | **35** | **43.8%** | ✅ **correct by design — keep** |
|
||||
| (b) FETCHABLE-GAP | **1** | 1.3% | fixable, but n=1 |
|
||||
| (a) FALSE-THRESHOLD | **0** | 0% | nothing to fix |
|
||||
| (c) ARCHETYPE-GAP | **0** | 0% | archetypes do not gate grading at all |
|
||||
| (d) GENUINE-ABSENCE | **0** | 0% | — |
|
||||
|
||||
**There is no meaningful data gap.** Part 2 items 4, 5 and 6 (fix thresholds,
|
||||
wire `/context`, build archetypes) have **nothing to act on** — the diagnosis
|
||||
says so, and I am not going to manufacture work for them.
|
||||
|
||||
---
|
||||
|
||||
## PART 1a — THE FIFTH BUCKET (the order did not anticipate this, and it changes the reading)
|
||||
|
||||
**All 35 "refusals" are `rare_event_over_below_line`.** They are not missing
|
||||
data. They are the **2026-07-19 betting-logic audit deliberately refusing
|
||||
rare-event 0.5 markets** — and they set the **same `insufficient_data: true`
|
||||
flag** as a genuine data gap, which is why they read as one.
|
||||
|
||||
From the code that produces them:
|
||||
|
||||
> *"a rare-event 0.5 OVER is a read ONLY when the model genuinely projects the
|
||||
> event ABOVE the line. Below that, the over carries the same |edge| as the
|
||||
> (already-suppressed) under and would just take its place on the board — so
|
||||
> refuse it."*
|
||||
|
||||
Entirely concentrated in two stats:
|
||||
|
||||
| stat | graded | suppressed |
|
||||
|---|---:|---:|
|
||||
| `doubles` | 1 | **18** |
|
||||
| `stolen_bases` | 0 | **17** |
|
||||
| `hits` | **19** | 0 |
|
||||
| `rbi` | **19** | 0 |
|
||||
| `total_bases` | **5** | 0 |
|
||||
|
||||
**The stats that can be graded grade at ~100%.** The refusals are entirely
|
||||
0.5-line rare events, refused on purpose.
|
||||
|
||||
**Had we "fixed" this**, we would have re-introduced exactly the bets a previous
|
||||
audit removed — and the count would have looked like progress. This is the
|
||||
single most important thing in the diagnosis.
|
||||
|
||||
The one real gap (bucket b) is `Miguel Rojas / stolen_bases`, who has **10 log
|
||||
rows with the stat** but produced no projection. Real, worth a look, **n=1** — not
|
||||
a program of work.
|
||||
|
||||
---
|
||||
|
||||
## PART 1b — THE CAP, AND WHAT IT ACTUALLY COSTS
|
||||
|
||||
| | |
|
||||
|---|---:|
|
||||
| rows in feed | 11,737 |
|
||||
| **unique gradeable props** | **585** |
|
||||
| **cap** | **25** |
|
||||
| **discarded by the cap** | **560 — 95.7%** |
|
||||
|
||||
**Why 25?** Traced to Session 32 (`f0c8b4f`), commented *"we still bound the
|
||||
herd."* It is a **herd guard written before anyone measured what a grade costs** —
|
||||
not a performance limit, not a cost limit, not a display choice. Nothing was ever
|
||||
measured against it.
|
||||
|
||||
**So I measured it.** Per grade, through the real path:
|
||||
|
||||
| mean | median | p90 | max |
|
||||
|---:|---:|---:|---:|
|
||||
| **721 ms** | 666 ms | 1,024 ms | 1,730 ms |
|
||||
|
||||
→ **~72s for 500 props at concurrency 5.** 80 props took **11.9s** wall.
|
||||
|
||||
**Both callers tolerate that comfortably:** the snapshot cron runs 5×/day, and
|
||||
`oddsService.recordDownstream` is fire-and-forget and never holds an HTTP
|
||||
response.
|
||||
|
||||
---
|
||||
|
||||
## PART 2 — THE FIX (item 3 only, because that is what the diagnosis supports)
|
||||
|
||||
**`DEFAULT_LIMIT` 25 → 500**, env-tunable via `GRADE_SLATE_LIMIT`.
|
||||
|
||||
**`DEFAULT_CONCURRENCY` left at 5, deliberately.** Raising the cap already
|
||||
multiplies total load ~20×, and concurrency is the knob that decides how hard we
|
||||
hit statsapi *at once*. One variable at a time.
|
||||
|
||||
**Items 4, 5, 6, 7 — no action, and the reason is the measurement:**
|
||||
|
||||
- **(4) false-thresholds — 0 found.** Nothing to loosen. Loosening anything here
|
||||
would be manufacturing grades, which the order's absolute rule forbids and
|
||||
which this whole run has been removing.
|
||||
- **(5) fetchable gaps — 1 of 80.** `/context` wiring is worth doing for *grade
|
||||
quality* (umpire, roof, handedness) but it is **not** a coverage fix; it would
|
||||
not have graded a single additional prop in this sample. Keeping it out of this
|
||||
order rather than claiming it as a win.
|
||||
- **(6) archetype-gap — 0.** Archetypes are display-side and **do not gate
|
||||
grading at all**. Building MLB batter archetypes remains worthwhile for the
|
||||
model, but it is **not** what is suppressing the board, and presenting it as
|
||||
such would have been a misdiagnosis.
|
||||
- **(7) genuine-absence — still refusing.** Unchanged.
|
||||
|
||||
### The refusal rate does not drop, and that is the correct outcome
|
||||
|
||||
**No threshold was lowered. No grade was forced.** The board grows because the
|
||||
cap stops discarding 95.7% of the slate — not because the bar moved.
|
||||
|
||||
---
|
||||
|
||||
## EXPECTED EFFECT
|
||||
|
||||
At the measured 55% grade rate on 585 unique props, the MLB board should go from
|
||||
**~7 graded props to roughly 300**, at a cost of about **70 seconds per snapshot
|
||||
run, five times a day.**
|
||||
|
||||
**Two things to watch after deploy, stated in advance rather than discovered
|
||||
later:**
|
||||
|
||||
1. **Payload size.** `snapshot:{sport}:latest` and `/api/snapshot/mlb` currently
|
||||
carry 7 grades; at ~300 they will be substantially larger. If the response
|
||||
becomes unwieldy, the fix is a response-side cap on what the *board* returns —
|
||||
**not** a re-cap on what gets graded. Grading everything and serving a slice
|
||||
is honest; grading a slice and calling it the slate is what we just fixed.
|
||||
2. **Ledger volume.** `recordPipelineGrades` writes one row per graded prop, so
|
||||
ledger growth scales with the same multiple. That is desirable — it is the
|
||||
accrual the model verdict needs — but it changes the shape of the table.
|
||||
|
||||
## TAGS
|
||||
|
||||
**VERIFIED:** 585 unique gradeable props vs a cap of 25 · per-grade cost
|
||||
721ms mean · 43.8% of refusals are deliberate policy suppression · 0 false
|
||||
thresholds · 0 archetype-gated refusals · the gradeable stats grade at ~100%.
|
||||
|
||||
**CORRECTED:** "72% refuse for insufficient_data" — the flag is shared between a
|
||||
genuine data gap and a deliberate suppression, and it is almost entirely the
|
||||
latter. The product was never failing to grade for want of data; it was
|
||||
discarding 95.7% of the slate before grading began.
|
||||
Reference in New Issue
Block a user