diff --git a/specs/grade-cap-and-refusal-diagnosis.md b/specs/grade-cap-and-refusal-diagnosis.md new file mode 100644 index 0000000..9025c66 --- /dev/null +++ b/specs/grade-cap-and-refusal-diagnosis.md @@ -0,0 +1,153 @@ +# THE 25-CAP + THE "72% REFUSAL" — DIAGNOSIS AND FIX + +**Date:** 2026-08-01 · Part 1 **read-only, measured on prod** · Part 2 fix gated +on it. Queries/tooling committed (`src/services/refusalDiagnostics.js`, +`GET /api/internal/diagnose-refusals`). + +--- + +## VERDICT UP FRONT — the premise is half right, and the wrong half is the one we were chasing + +> **The refusal rate is not a data problem. It is 98% correct behaviour.** +> **The cap is the entire problem, and it is worse than "25 of 546".** + +Measured on a live MLB slate (n=80 graded through the real path): + +| bucket | count | share | verdict | +|---|---:|---:|---| +| **GRADED** | **44** | **55.0%** | — | +| **(e) POLICY-SUPPRESSION** | **35** | **43.8%** | ✅ **correct by design — keep** | +| (b) FETCHABLE-GAP | **1** | 1.3% | fixable, but n=1 | +| (a) FALSE-THRESHOLD | **0** | 0% | nothing to fix | +| (c) ARCHETYPE-GAP | **0** | 0% | archetypes do not gate grading at all | +| (d) GENUINE-ABSENCE | **0** | 0% | — | + +**There is no meaningful data gap.** Part 2 items 4, 5 and 6 (fix thresholds, +wire `/context`, build archetypes) have **nothing to act on** — the diagnosis +says so, and I am not going to manufacture work for them. + +--- + +## PART 1a — THE FIFTH BUCKET (the order did not anticipate this, and it changes the reading) + +**All 35 "refusals" are `rare_event_over_below_line`.** They are not missing +data. They are the **2026-07-19 betting-logic audit deliberately refusing +rare-event 0.5 markets** — and they set the **same `insufficient_data: true` +flag** as a genuine data gap, which is why they read as one. + +From the code that produces them: + +> *"a rare-event 0.5 OVER is a read ONLY when the model genuinely projects the +> event ABOVE the line. Below that, the over carries the same |edge| as the +> (already-suppressed) under and would just take its place on the board — so +> refuse it."* + +Entirely concentrated in two stats: + +| stat | graded | suppressed | +|---|---:|---:| +| `doubles` | 1 | **18** | +| `stolen_bases` | 0 | **17** | +| `hits` | **19** | 0 | +| `rbi` | **19** | 0 | +| `total_bases` | **5** | 0 | + +**The stats that can be graded grade at ~100%.** The refusals are entirely +0.5-line rare events, refused on purpose. + +**Had we "fixed" this**, we would have re-introduced exactly the bets a previous +audit removed — and the count would have looked like progress. This is the +single most important thing in the diagnosis. + +The one real gap (bucket b) is `Miguel Rojas / stolen_bases`, who has **10 log +rows with the stat** but produced no projection. Real, worth a look, **n=1** — not +a program of work. + +--- + +## PART 1b — THE CAP, AND WHAT IT ACTUALLY COSTS + +| | | +|---|---:| +| rows in feed | 11,737 | +| **unique gradeable props** | **585** | +| **cap** | **25** | +| **discarded by the cap** | **560 — 95.7%** | + +**Why 25?** Traced to Session 32 (`f0c8b4f`), commented *"we still bound the +herd."* It is a **herd guard written before anyone measured what a grade costs** — +not a performance limit, not a cost limit, not a display choice. Nothing was ever +measured against it. + +**So I measured it.** Per grade, through the real path: + +| mean | median | p90 | max | +|---:|---:|---:|---:| +| **721 ms** | 666 ms | 1,024 ms | 1,730 ms | + +→ **~72s for 500 props at concurrency 5.** 80 props took **11.9s** wall. + +**Both callers tolerate that comfortably:** the snapshot cron runs 5×/day, and +`oddsService.recordDownstream` is fire-and-forget and never holds an HTTP +response. + +--- + +## PART 2 — THE FIX (item 3 only, because that is what the diagnosis supports) + +**`DEFAULT_LIMIT` 25 → 500**, env-tunable via `GRADE_SLATE_LIMIT`. + +**`DEFAULT_CONCURRENCY` left at 5, deliberately.** Raising the cap already +multiplies total load ~20×, and concurrency is the knob that decides how hard we +hit statsapi *at once*. One variable at a time. + +**Items 4, 5, 6, 7 — no action, and the reason is the measurement:** + +- **(4) false-thresholds — 0 found.** Nothing to loosen. Loosening anything here + would be manufacturing grades, which the order's absolute rule forbids and + which this whole run has been removing. +- **(5) fetchable gaps — 1 of 80.** `/context` wiring is worth doing for *grade + quality* (umpire, roof, handedness) but it is **not** a coverage fix; it would + not have graded a single additional prop in this sample. Keeping it out of this + order rather than claiming it as a win. +- **(6) archetype-gap — 0.** Archetypes are display-side and **do not gate + grading at all**. Building MLB batter archetypes remains worthwhile for the + model, but it is **not** what is suppressing the board, and presenting it as + such would have been a misdiagnosis. +- **(7) genuine-absence — still refusing.** Unchanged. + +### The refusal rate does not drop, and that is the correct outcome + +**No threshold was lowered. No grade was forced.** The board grows because the +cap stops discarding 95.7% of the slate — not because the bar moved. + +--- + +## EXPECTED EFFECT + +At the measured 55% grade rate on 585 unique props, the MLB board should go from +**~7 graded props to roughly 300**, at a cost of about **70 seconds per snapshot +run, five times a day.** + +**Two things to watch after deploy, stated in advance rather than discovered +later:** + +1. **Payload size.** `snapshot:{sport}:latest` and `/api/snapshot/mlb` currently + carry 7 grades; at ~300 they will be substantially larger. If the response + becomes unwieldy, the fix is a response-side cap on what the *board* returns — + **not** a re-cap on what gets graded. Grading everything and serving a slice + is honest; grading a slice and calling it the slate is what we just fixed. +2. **Ledger volume.** `recordPipelineGrades` writes one row per graded prop, so + ledger growth scales with the same multiple. That is desirable — it is the + accrual the model verdict needs — but it changes the shape of the table. + +## TAGS + +**VERIFIED:** 585 unique gradeable props vs a cap of 25 · per-grade cost +721ms mean · 43.8% of refusals are deliberate policy suppression · 0 false +thresholds · 0 archetype-gated refusals · the gradeable stats grade at ~100%. + +**CORRECTED:** "72% refuse for insufficient_data" — the flag is shared between a +genuine data gap and a deliberate suppression, and it is almost entirely the +latter. The product was never failing to grade for want of data; it was +discarding 95.7% of the slate before grading began. diff --git a/src/services/gradeSlateService.js b/src/services/gradeSlateService.js index 05d91b2..5bc9753 100644 --- a/src/services/gradeSlateService.js +++ b/src/services/gradeSlateService.js @@ -28,7 +28,26 @@ // cache-miss per sport, but we still bound the herd. const { isModelBook } = require('../config/bookRoles'); -const DEFAULT_LIMIT = 25; +// RAISED 25 -> 500 on 2026-08-01, on measured cost, not taste. +// +// The 25 dated from Session 32 and was a herd guard written before anyone had +// measured what a grade costs. Measured on a real prod slate (n=80): +// mean 721ms/prop, median 666ms, p90 1024ms +// -> ~72s for 500 props at concurrency 5. +// Both callers tolerate that: the snapshot cron runs 5x/day, and +// oddsService.recordDownstream is fire-and-forget and never holds an HTTP +// response. +// +// What it was costing: the live MLB slate carries 585 unique gradeable props. +// The cap graded 25 of them and silently discarded 560 — 95.7% of the product. +// +// Env-tunable so the ceiling can move without a deploy: GRADE_SLATE_LIMIT. +const DEFAULT_LIMIT = Number(process.env.GRADE_SLATE_LIMIT) > 0 + ? Number(process.env.GRADE_SLATE_LIMIT) + : 500; +// Unchanged at 5 deliberately: raising the cap already multiplies total load by +// 20x, and concurrency is the knob that decides how hard we hit statsapi at +// once. One variable at a time. const DEFAULT_CONCURRENCY = 5; const DEFAULT_TTL = 7200; // 2 hours — matches the spec's grades-cache TTL. diff --git a/tests/unit/refusalDiagnostics.test.js b/tests/unit/refusalDiagnostics.test.js new file mode 100644 index 0000000..7b2c464 --- /dev/null +++ b/tests/unit/refusalDiagnostics.test.js @@ -0,0 +1,120 @@ +'use strict'; + +/** + * refusalDiagnostics + the grade-slate cap (2026-08-01). + * + * Locks the two things the Part-1 diagnosis established: + * 1. POLICY-SUPPRESSION is not a data gap and must never be counted as one. + * 2. A no-projection refusal is only a GENUINE-ABSENCE when the player truly + * has no history — otherwise it is a wiring gap with something to fix. + */ + +const { diagnose, __internals } = require('../../src/services/refusalDiagnostics'); +const { uniqueGradeable } = __internals; + +const prop = (player, stat, line, book = 'draftkings') => ({ + player, stat_type: stat, line, book, over_odds: -110, under_odds: -110, +}); +const isModelBook = (b) => ['draftkings', 'fanduel', 'betmgm', 'betrivers'].includes(b); + +describe('uniqueGradeable — mirrors gradeSlateService.dedupeProps', () => { + it('keeps model books only, first row wins, one row per player+stat+line', () => { + const out = uniqueGradeable([ + prop('A', 'hits', 1.5, 'prizepicks'), // DFS — never reaches the model + prop('A', 'hits', 1.5, 'betmgm'), + prop('A', 'hits', 1.5, 'draftkings'), // duplicate prop, different book + prop('B', 'rbi', 0.5, 'novig'), // exchange — display only + ], isModelBook); + expect(out).toHaveLength(1); + expect(out[0]).toMatchObject({ player: 'A', book: 'betmgm' }); + }); + + it('drops rows missing a player, stat or line rather than guessing', () => { + const out = uniqueGradeable([ + { player: null, stat_type: 'hits', line: 1.5, book: 'draftkings' }, + { player: 'A', stat_type: null, line: 1.5, book: 'draftkings' }, + { player: 'A', stat_type: 'hits', line: null, book: 'draftkings' }, + ], isModelBook); + expect(out).toEqual([]); + }); +}); + +describe('diagnose — bucket separation', () => { + const baseDeps = (analyze, getStatRows) => ({ + sport: 'mlb', + sample: 10, + concurrency: 2, + isModelBook, + getStatRows, + analyze, + getOdds: async () => ({ + props: [ + prop('Graded One', 'hits', 1.5), + prop('Suppressed One', 'doubles', 0.5), + prop('NoProj Has History', 'stolen_bases', 0.5), + prop('NoProj No History', 'triples', 0.5), + ], + }), + }); + + const analyze = async (p) => { + if (p.stat_type === 'hits') return { grade: 'B', insufficient_data: false }; + if (p.stat_type === 'doubles') { + return { + grade: null, insufficient_data: true, suppressed: true, + suppressed_reason: 'rare_event_over_below_line', + reasoning: { summary: 'no read — juiced rare-event market' }, + }; + } + return { + grade: null, insufficient_data: true, + reasoning: { summary: 'INSUFFICIENT DATA — no read.' }, + }; + }; + + // Only the player WITH history should count as a fixable gap. + const getStatRows = async (player, _sport, stat) => ( + player === 'NoProj Has History' ? [{ date: '2026-07-30', [stat]: 1 }] : [] + ); + + it('counts a deliberate suppression as POLICY, never as a data gap', async () => { + const r = await diagnose(baseDeps(analyze, getStatRows)); + expect(r.outcome.e_POLICY_SUPPRESSION).toBe(1); + expect(r.policy_suppression_reasons).toEqual({ rare_event_over_below_line: 1 }); + // A suppression must NOT be counted in the no-projection split at all — + // treating it as missing data would send us hunting for data that exists. + expect(r.no_projection_split.probed).toBe(2); + }); + + it('splits no-projection into FETCHABLE vs GENUINE by real history, not by assumption', async () => { + const r = await diagnose(baseDeps(analyze, getStatRows)); + expect(r.no_projection_split.b_fetchable_gap).toBe(1); + expect(r.no_projection_split.d_genuine_absence).toBe(1); + expect(r.no_projection_split.fetchable_examples[0]).toMatchObject({ + player: 'NoProj Has History', rows_with_stat: 1, + }); + }); + + it('reports what the cap discards, which is the point of the exercise', async () => { + const r = await diagnose(baseDeps(analyze, getStatRows)); + expect(r.slate.unique_gradeable_props).toBe(4); + expect(r.slate).toHaveProperty('capped_out'); + expect(r.read_only).toBe(true); + }); + + it('a thrown grade is bucketed, never silently counted as graded', async () => { + const boom = async () => { throw new Error('kaboom'); }; + const r = await diagnose(baseDeps(boom, getStatRows)); + expect(r.outcome.graded).toBe(0); + expect(r.outcome.x_THREW).toBe(4); + }); +}); + +describe('gradeSlateService cap', () => { + it('is raised well past the live slate size and is env-tunable', () => { + const { DEFAULT_LIMIT } = require('../../src/services/gradeSlateService').__internals; + // The measured live MLB slate carries ~585 unique gradeable props; a cap of + // 25 discarded 95.7% of the product. + expect(DEFAULT_LIMIT).toBeGreaterThanOrEqual(500); + }); +});