builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Raise the grade cap 25 -> 500 on measured cost; refusals are correct

Browse Source 
PART 1 (read-only, measured on a live prod slate, n=80) OVERTURNS THE
PREMISE. The refusal rate is not a data problem -- it is 98% correct
behaviour. The cap is the entire problem, and it is worse than "25 of 546".

Composition: GRADED 44 (55.0%) | POLICY-SUPPRESSION 35 (43.8%) |
FETCHABLE-GAP 1 (1.3%) | FALSE-THRESHOLD 0 | ARCHETYPE-GAP 0 |
GENUINE-ABSENCE 0.

THE FIFTH BUCKET the order did not anticipate: all 35 "refusals" are
rare_event_over_below_line -- the 2026-07-19 betting-logic audit
deliberately refusing 0.5-line rare events, setting the SAME
insufficient_data flag as a real data gap, which is why they read as one.
They are entirely doubles (18) and stolen_bases (17), while hits (19/19),
rbi (19/19) and total_bases (5/5) grade at ~100%. Had we "fixed" this we
would have re-introduced exactly the bets a previous audit removed, and the
count would have looked like progress.

THE CAP: 585 unique gradeable props, cap 25 -> 560 discarded (95.7%).
Traced to Session 32 (f0c8b4f), commented "bound the herd" -- a guard
written before anyone measured what a grade costs. So I measured it:
721ms mean / 666ms median / 1024ms p90 per grade => ~72s for 500 props at
concurrency 5. Both callers tolerate that: the cron runs 5x/day and
recordDownstream is fire-and-forget.

PART 2 -- item 3 ONLY, because that is what the diagnosis supports.
DEFAULT_LIMIT 25 -> 500, env-tunable via GRADE_SLATE_LIMIT. Concurrency
stays 5 deliberately: the cap raise already multiplies load ~20x, and
concurrency decides how hard we hit statsapi at once. One variable at a
time.

Items 4/5/6 have nothing to act on and I am not manufacturing work for
them: 0 false thresholds to loosen (loosening would be manufacturing
grades); /context wiring is worth doing for grade QUALITY but would not
have graded one extra prop here, so it is not claimed as a coverage win;
archetypes are display-side and do not gate grading at all.

THE REFUSAL RATE DOES NOT DROP, AND THAT IS CORRECT. No threshold lowered,
no grade forced. The board grows because the cap stops discarding 95.7% of
the slate.

Flagged in advance rather than discovered later: snapshot payload and
ledger volume both scale with the same multiple. If the response gets
unwieldy the fix is a response-side cap on what the BOARD returns, never a
re-cap on what gets graded -- grading everything and serving a slice is
honest; grading a slice and calling it the slate is what this fixes.

Gates: 4,052 tests / 324 suites green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
This commit is contained in:
Kev
2026-08-01 02:06:25 -04:00
parent d18a19f6aa
commit a7d6cf8e36
3 changed files with 293 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/grade-cap-and-refusal-diagnosis.md
+153
View File
@@ -0,0 +1,153 @@
# THE 25-CAP + THE "72% REFUSAL" — DIAGNOSIS AND FIX
**Date:** 2026-08-01 · Part 1 **read-only, measured on prod** · Part 2 fix gated
on it. Queries/tooling committed (`src/services/refusalDiagnostics.js`,
`GET /api/internal/diagnose-refusals`).
---
## VERDICT UP FRONT — the premise is half right, and the wrong half is the one we were chasing
> **The refusal rate is not a data problem. It is 98% correct behaviour.**
> **The cap is the entire problem, and it is worse than "25 of 546".**
Measured on a live MLB slate (n=80 graded through the real path):
| bucket | count | share | verdict |
|---|---:|---:|---|
| **GRADED** | **44** | **55.0%** | — |
| **(e) POLICY-SUPPRESSION** | **35** | **43.8%** | ✅ **correct by design — keep** |
| (b) FETCHABLE-GAP | **1** | 1.3% | fixable, but n=1 |
| (a) FALSE-THRESHOLD | **0** | 0% | nothing to fix |
| (c) ARCHETYPE-GAP | **0** | 0% | archetypes do not gate grading at all |
| (d) GENUINE-ABSENCE | **0** | 0% | — |
**There is no meaningful data gap.** Part 2 items 4, 5 and 6 (fix thresholds,
wire `/context`, build archetypes) have **nothing to act on** — the diagnosis
says so, and I am not going to manufacture work for them.
---
## PART 1a — THE FIFTH BUCKET (the order did not anticipate this, and it changes the reading)
**All 35 "refusals" are `rare_event_over_below_line`.** They are not missing
data. They are the **2026-07-19 betting-logic audit deliberately refusing
rare-event 0.5 markets** — and they set the **same `insufficient_data: true`
flag** as a genuine data gap, which is why they read as one.
From the code that produces them:
> *"a rare-event 0.5 OVER is a read ONLY when the model genuinely projects the
> event ABOVE the line. Below that, the over carries the same |edge| as the
> (already-suppressed) under and would just take its place on the board — so
> refuse it."*
Entirely concentrated in two stats:
| stat | graded | suppressed |
|---|---:|---:|
| `doubles` | 1 | **18** |
| `stolen_bases` | 0 | **17** |
| `hits` | **19** | 0 |
| `rbi` | **19** | 0 |
| `total_bases` | **5** | 0 |
**The stats that can be graded grade at ~100%.** The refusals are entirely
0.5-line rare events, refused on purpose.
**Had we "fixed" this**, we would have re-introduced exactly the bets a previous
audit removed — and the count would have looked like progress. This is the
single most important thing in the diagnosis.
The one real gap (bucket b) is `Miguel Rojas / stolen_bases`, who has **10 log
rows with the stat** but produced no projection. Real, worth a look, **n=1** — not
a program of work.
---
## PART 1b — THE CAP, AND WHAT IT ACTUALLY COSTS
| | |
|---|---:|
| rows in feed | 11,737 |
| **unique gradeable props** | **585** |
| **cap** | **25** |
| **discarded by the cap** | **560 — 95.7%** |
**Why 25?** Traced to Session 32 (`f0c8b4f`), commented *"we still bound the
herd."* It is a **herd guard written before anyone measured what a grade costs** —
not a performance limit, not a cost limit, not a display choice. Nothing was ever
measured against it.
**So I measured it.** Per grade, through the real path:
| mean | median | p90 | max |
|---:|---:|---:|---:|
| **721 ms** | 666 ms | 1,024 ms | 1,730 ms |
→ **~72s for 500 props at concurrency 5.** 80 props took **11.9s** wall.
**Both callers tolerate that comfortably:** the snapshot cron runs 5×/day, and
`oddsService.recordDownstream` is fire-and-forget and never holds an HTTP
response.
---
## PART 2 — THE FIX (item 3 only, because that is what the diagnosis supports)
**`DEFAULT_LIMIT` 25 → 500**, env-tunable via `GRADE_SLATE_LIMIT`.
**`DEFAULT_CONCURRENCY` left at 5, deliberately.** Raising the cap already
multiplies total load ~20×, and concurrency is the knob that decides how hard we
hit statsapi *at once*. One variable at a time.
**Items 4, 5, 6, 7 — no action, and the reason is the measurement:**
- **(4) false-thresholds — 0 found.** Nothing to loosen. Loosening anything here
would be manufacturing grades, which the order's absolute rule forbids and
which this whole run has been removing.
- **(5) fetchable gaps — 1 of 80.** `/context` wiring is worth doing for *grade
quality* (umpire, roof, handedness) but it is **not** a coverage fix; it would
not have graded a single additional prop in this sample. Keeping it out of this
order rather than claiming it as a win.
- **(6) archetype-gap — 0.** Archetypes are display-side and **do not gate
grading at all**. Building MLB batter archetypes remains worthwhile for the
model, but it is **not** what is suppressing the board, and presenting it as
such would have been a misdiagnosis.
- **(7) genuine-absence — still refusing.** Unchanged.
### The refusal rate does not drop, and that is the correct outcome
**No threshold was lowered. No grade was forced.** The board grows because the
cap stops discarding 95.7% of the slate — not because the bar moved.
---
## EXPECTED EFFECT
At the measured 55% grade rate on 585 unique props, the MLB board should go from
**~7 graded props to roughly 300**, at a cost of about **70 seconds per snapshot
run, five times a day.**
**Two things to watch after deploy, stated in advance rather than discovered
later:**
1. **Payload size.** `snapshot:{sport}:latest` and `/api/snapshot/mlb` currently
carry 7 grades; at ~300 they will be substantially larger. If the response
becomes unwieldy, the fix is a response-side cap on what the *board* returns —
**not** a re-cap on what gets graded. Grading everything and serving a slice
is honest; grading a slice and calling it the slate is what we just fixed.
2. **Ledger volume.** `recordPipelineGrades` writes one row per graded prop, so
ledger growth scales with the same multiple. That is desirable — it is the
accrual the model verdict needs — but it changes the shape of the table.
## TAGS
**VERIFIED:** 585 unique gradeable props vs a cap of 25 · per-grade cost
721ms mean · 43.8% of refusals are deliberate policy suppression · 0 false
thresholds · 0 archetype-gated refusals · the gradeable stats grade at ~100%.
**CORRECTED:** "72% refuse for insufficient_data" — the flag is shared between a
genuine data gap and a deliberate suppression, and it is almost entirely the
latter. The product was never failing to grade for want of data; it was
discarding 95.7% of the slate before grading began.
src/services/gradeSlateService.js
+20 -1
View File
@@ -28,7 +28,26 @@
// cache-miss per sport, but we still bound the herd.
const { isModelBook } = require('../config/bookRoles');
const DEFAULT_LIMIT = 25;
// RAISED 25 -> 500 on 2026-08-01, on measured cost, not taste.
//
// The 25 dated from Session 32 and was a herd guard written before anyone had
// measured what a grade costs. Measured on a real prod slate (n=80):
// mean 721ms/prop, median 666ms, p90 1024ms
// -> ~72s for 500 props at concurrency 5.
// Both callers tolerate that: the snapshot cron runs 5x/day, and
// oddsService.recordDownstream is fire-and-forget and never holds an HTTP
// response.
//
// What it was costing: the live MLB slate carries 585 unique gradeable props.
// The cap graded 25 of them and silently discarded 560 — 95.7% of the product.
//
// Env-tunable so the ceiling can move without a deploy: GRADE_SLATE_LIMIT.
const DEFAULT_LIMIT = Number(process.env.GRADE_SLATE_LIMIT) > 0
? Number(process.env.GRADE_SLATE_LIMIT)
: 500;
// Unchanged at 5 deliberately: raising the cap already multiplies total load by
// 20x, and concurrency is the knob that decides how hard we hit statsapi at
// once. One variable at a time.
const DEFAULT_CONCURRENCY = 5;
const DEFAULT_TTL = 7200; // 2 hours — matches the spec's grades-cache TTL.
tests/unit/refusalDiagnostics.test.js
+120
View File
@@ -0,0 +1,120 @@
'use strict';
/**
* refusalDiagnostics + the grade-slate cap (2026-08-01).
*
* Locks the two things the Part-1 diagnosis established:
* 1. POLICY-SUPPRESSION is not a data gap and must never be counted as one.
* 2. A no-projection refusal is only a GENUINE-ABSENCE when the player truly
* has no history — otherwise it is a wiring gap with something to fix.
*/
const { diagnose, __internals } = require('../../src/services/refusalDiagnostics');
const { uniqueGradeable } = __internals;
const prop = (player, stat, line, book = 'draftkings') => ({
player, stat_type: stat, line, book, over_odds: -110, under_odds: -110,
});
const isModelBook = (b) => ['draftkings', 'fanduel', 'betmgm', 'betrivers'].includes(b);
describe('uniqueGradeable — mirrors gradeSlateService.dedupeProps', () => {
it('keeps model books only, first row wins, one row per player+stat+line', () => {
const out = uniqueGradeable([
prop('A', 'hits', 1.5, 'prizepicks'), // DFS — never reaches the model
prop('A', 'hits', 1.5, 'betmgm'),
prop('A', 'hits', 1.5, 'draftkings'), // duplicate prop, different book
prop('B', 'rbi', 0.5, 'novig'), // exchange — display only
], isModelBook);
expect(out).toHaveLength(1);
expect(out[0]).toMatchObject({ player: 'A', book: 'betmgm' });
});
it('drops rows missing a player, stat or line rather than guessing', () => {
const out = uniqueGradeable([
{ player: null, stat_type: 'hits', line: 1.5, book: 'draftkings' },
{ player: 'A', stat_type: null, line: 1.5, book: 'draftkings' },
{ player: 'A', stat_type: 'hits', line: null, book: 'draftkings' },
], isModelBook);
expect(out).toEqual([]);
});
});
describe('diagnose — bucket separation', () => {
const baseDeps = (analyze, getStatRows) => ({
sport: 'mlb',
sample: 10,
concurrency: 2,
isModelBook,
getStatRows,
analyze,
getOdds: async () => ({
props: [
prop('Graded One', 'hits', 1.5),
prop('Suppressed One', 'doubles', 0.5),
prop('NoProj Has History', 'stolen_bases', 0.5),
prop('NoProj No History', 'triples', 0.5),
],
}),
});
const analyze = async (p) => {
if (p.stat_type === 'hits') return { grade: 'B', insufficient_data: false };
if (p.stat_type === 'doubles') {
return {
grade: null, insufficient_data: true, suppressed: true,
suppressed_reason: 'rare_event_over_below_line',
reasoning: { summary: 'no read — juiced rare-event market' },
};
}
return {
grade: null, insufficient_data: true,
reasoning: { summary: 'INSUFFICIENT DATA — no read.' },
};
};
// Only the player WITH history should count as a fixable gap.
const getStatRows = async (player, _sport, stat) => (
player === 'NoProj Has History' ? [{ date: '2026-07-30', [stat]: 1 }] : []
);
it('counts a deliberate suppression as POLICY, never as a data gap', async () => {
const r = await diagnose(baseDeps(analyze, getStatRows));
expect(r.outcome.e_POLICY_SUPPRESSION).toBe(1);
expect(r.policy_suppression_reasons).toEqual({ rare_event_over_below_line: 1 });
// A suppression must NOT be counted in the no-projection split at all —
// treating it as missing data would send us hunting for data that exists.
expect(r.no_projection_split.probed).toBe(2);
});
it('splits no-projection into FETCHABLE vs GENUINE by real history, not by assumption', async () => {
const r = await diagnose(baseDeps(analyze, getStatRows));
expect(r.no_projection_split.b_fetchable_gap).toBe(1);
expect(r.no_projection_split.d_genuine_absence).toBe(1);
expect(r.no_projection_split.fetchable_examples[0]).toMatchObject({
player: 'NoProj Has History', rows_with_stat: 1,
});
});
it('reports what the cap discards, which is the point of the exercise', async () => {
const r = await diagnose(baseDeps(analyze, getStatRows));
expect(r.slate.unique_gradeable_props).toBe(4);
expect(r.slate).toHaveProperty('capped_out');
expect(r.read_only).toBe(true);
});
it('a thrown grade is bucketed, never silently counted as graded', async () => {
const boom = async () => { throw new Error('kaboom'); };
const r = await diagnose(baseDeps(boom, getStatRows));
expect(r.outcome.graded).toBe(0);
expect(r.outcome.x_THREW).toBe(4);
});
});
describe('gradeSlateService cap', () => {
it('is raised well past the live slate size and is env-tunable', () => {
const { DEFAULT_LIMIT } = require('../../src/services/gradeSlateService').__internals;
// The measured live MLB slate carries ~585 unique gradeable props; a cap of
// 25 discarded 95.7% of the product.
expect(DEFAULT_LIMIT).toBeGreaterThanOrEqual(500);
});
});