diff --git a/specs/audit-data/grade-collapse.md b/specs/audit-data/grade-collapse.md new file mode 100644 index 0000000..183da03 --- /dev/null +++ b/specs/audit-data/grade-collapse.md @@ -0,0 +1,189 @@ +# GRADE COLLAPSE + DEAD PROBABILITY LAYER — diagnosis + +**REPORT ONLY. No grade logic, thresholds, or engine code changed** (Kev's instruction). +Two findings. The second one is bigger than the question I was asked. + +Data: live Supabase `ledger_entries` (604 rows, all users) + live prod API +`api.vyndr.app`, 2026-07-19 ~22:05 UTC. + +--- + +## FINDING 1 — THE COLLAPSE IS REAL, LIVE, AND STRUCTURAL + +Not a thin-slate artifact. Across **604 ledger rows and both sports**: + +- **2 distinct grades ever emitted: B and C.** Zero A+, A, A−, B+, B−, C−, D, F. +- **9 distinct confidence values ever emitted:** 63, 57, 55, 52, 47, 45, 35, 25, 20. +- **Confidence ceiling = 63.** It has never exceeded 63 in the recorded era. + +Still true **today** (Jul 18 + 19, post every fix, both sports): 4 confidence values +(63/57/52/47), 2 grades. + +| Sport | Grade | n | conf min | conf max | +|---|---|---|---|---| +| mlb | B | 276 | 45 | 63 | +| mlb | C | 107 | 20 | 52 | +| wnba | B | 130 | 45 | 63 | +| wnba | C | 91 | 35 | 52 | + +### Confidence does NOT determine the letter + +| conf | grade | n | +|---|---|---| +| 63 | B | 54 | +| 57 | B | 175 | +| 55 | B | 29 | +| 52 | **C** | 100 | +| 47 | **C** | 14 | +| **45** | **B** | **148** | +| 35 | C | 80 | + +**conf 45 → B, but conf 47 and 52 → C.** The mapping is non-monotonic, so the surfaced +`confidence` is not the quantity the letter was derived from. This confirms the +`mlb-grade-degradation.md` "grade↔confidence mismatch" as a *display* artifact: two +different quantities are being shown as if one explains the other. + +(At conf 45→B the avg edge is 103; at conf 52→C it is 49 — so the letter tracks the +engine composite/edge, not the displayed confidence.) + +### Collateral: the edge scale is still broken and still live + +- **311 of 604 rows (51.5 %) have |edge| > 40** — the frontend's `EDGE_BOARD_SANE_MAX`, + i.e. over half the board's edge is nulled at render. +- **39 rows have |edge| > 100** — impossible as a percentage. Worst: **620**. +- Live today: edges of 140, 180, 220 on Jul 18–19 rows. + +`U-deg`'s `projection == 0` leak IS closed (0 since 07-18). The **edge_pct scale is +not** — it remains open and is now quantified. + +--- + +## FINDING 2 — 🔴 THE ENTIRE PROBABILITY LAYER IS DEAD IN PRODUCTION + +Found while fingerprinting Arc 1 (U-fp). This is the headline. + +### Live fingerprint, `GET /api/snapshot/mlb`, 8 graded props + +| Field | Present | +|---|---| +| `projection`, `confidence`, `book_odds`, `fair_odds`, `takeable`, `devig_method`, `alt_lines` | **8 / 8** | +| **`p_win`** | **0 / 8** | +| **`kelly`** | **0 / 8** | +| **`ev_pct`** | **0 / 8** | +| **`model_odds`** | **0 / 8** | +| **`value`** | **0 / 8** | + +**Control:** `alt_lines` is present 8/8 and is Desk-gated in `tierGating.js:55`, which +proves the payload is **not** being tier-stripped. These fields are genuinely never +computed — not hidden. + +### Root cause — a one-line sport gate, and an S46 fix that was only half-applied + +`analyzeViaEngine1.js:509` feeds the estimator from `meta.gameLogs`: + +```js +const est = estimateProbability({ gameLogs: meta.gameLogs, line: prop.line, ... }); +``` + +`meta.gameLogs` comes from `computeFeatures.js:173-181` → `gameLogService.getGameLogs`. +And `gameLogService.js:21-26`: + +```js +function pythonPath(sport) { + switch (sport) { + case 'nba': return '/stats/last-n'; + case 'wnba': return '/wnba/stats/last-n'; + default: return null; // ← MLB exits here + } +} +``` +with `getGameLogs` line 31: `if (!path) return null;` + +So: +- **MLB** — returns `null` by construction. Never had game logs on this path. +- **NBA/WNBA** — hits the Python stats service, which is **offline in prod** (documented + in CLAUDE.md; degrades to null). + +⇒ `meta.gameLogs` is `[]` for **every sport in production** ⇒ +`estimateProbability` returns `{p_over: null, reason:'insufficient_data'}` +(`probabilityEstimator.js:55-57`) ⇒ `pWin` is null ⇒ **every field guarded by +`if (pWin != null)` is skipped**: `p_win`, `kelly`, `model_odds`, `ev_pct`, `value`. + +**This is the S46 bug, second location, never fixed.** CLAUDE.md records that +`gameLogService.getGameLogs` being NBA/WNBA-only starved MLB, and that the fix was an +MLB branch in **`featureCache.gameLogFeatures`**. That fixed the *feature* path — which +is why `projection`, `confidence`, and grades still work. The **estimator path was never +given the same branch**, so it has been silently dead the whole time. + +### What this actually breaks + +1. **EV — the Model Train's entire ranking signal — does not exist in production.** + Arc 1 shipped `ev_pct` and it has never once been computed on a live prop. +2. **Hero v2 is non-functional.** `pickHeroProp` requires a finite `ev_pct` + (`heroPropService.js:84`), so the EV loop matches **nothing** and always falls through + to the "most recent graded read" fallback. Live proof: `/api/hero-prop` returns + `"is_recent": true` — the fallback path, every time. The hero has not been an EV pick + since the day it shipped. +3. **Quarter-Kelly is dead** — same `pWin` dependency (`analyzeViaEngine1.js:516-520`). + This is a **promise-audit issue**: Kelly sizing is sold on the pricing page and + `PROMISE-AUDIT.md` lists it as BUILT. It is built and never runs. +4. **The "value triplet" is a duet live** — `book_odds` + `fair_odds` render; + `model_odds` is always absent. +5. **`value` is never true**, so the VALUE marker can never light up. + +### Why this reframes the whole train + +- **C-led would persist a column of nulls.** Do not build EV persistence until EV exists. +- **G-a's `EV_FLEX_THRESHOLD` would gate on a permanently-null value.** With + `EV_FLEX_ENFORCE=0` (Kev's ruling) this is harmless today — but had we enforced it, + the flex band would have been cut to **zero**, because `ev_pct >= 4` can never be true. + The ruling to ship it disabled accidentally prevented an outage. +- **S-b (rank board on EV)** would rank on nulls. + +--- + +## RELATIONSHIP BETWEEN THE TWO FINDINGS + +They are **adjacent, not identical**, and both trace to the same missing input: + +- The dead estimator explains **why no probability-derived output exists** (EV, Kelly, + model_odds, p_win). +- It does **not by itself** explain the B/C letter collapse, because the letter comes + from engine1's rule-based composite over the *feature vector*, which is alive. +- But they share a root: **the model is running on a partial input set.** One of its two + probability inputs (the empirical quantile distribution over real game logs) is absent + for 100 % of props, so whatever spread the composite was designed to produce is being + generated from the surviving features only. + +**The 9-discrete-confidence-values pattern is consistent with a small set of additive +rule hits** — a scorer landing on a lattice rather than a continuum. Confirming exactly +where the letter range compresses requires the engine1/threshold trace, which is the one +open thread in this report. + +--- + +## RECOMMENDATION (no code changed pending Kev's call) + +**Re-sequence: fix the dead estimator FIRST — before G-a, before C-led.** + +Rationale: it is the cheapest fix on the board (an MLB branch in the estimator's log +source, mirroring the one already written for `featureCache`), and it simultaneously +restores EV, Kelly, `model_odds`, the VALUE flag, and hero v2. Every other Arc 2-5 item +is downstream of it. Building the gate, the persistence layer, or the board ranking on a +null signal is building on nothing. + +Suggested order: +1. **Revive the probability layer** (MLB branch + a real NBA/WNBA fallback, since Python + is offline). Fingerprint that `p_win`/`ev_pct` appear live. +2. **Then C-led** — persist EV that now has values. +3. **Then G-a** — with the flex band still disabled per the standing ruling. +4. **Separately: the grade-range investigation** (engine1 composite + thresholds), which + may be partly a consequence of step 1 and should be re-measured after it. + +Open question for Kev: NBA/WNBA have no free game-log source on this path with Python +down. `espnStatsAdapter.getPlayerGameLog` (Wave 0) already solves exactly this for +`featureCache` — reusing it here is the obvious candidate, and costs no quota. + +--- + +*Diagnosis 2026-07-19. Data + live API. No engine code changed.* diff --git a/specs/model-train.md b/specs/model-train.md index caf3b10..2b57818 100644 --- a/specs/model-train.md +++ b/specs/model-train.md @@ -176,6 +176,25 @@ Full arc definitions live in the Session-63 order. Status only here; update as e | **U-deg** MLB degradation | ✅ **STATUS REPORTED** | `projection==0` leak **already closed** (0 occurrences since 07-18). `edge_pct` scale still broken. | | **U-fp** Arc 1 fingerprint | open | Do it on the first deploy this train ships. | +### 🔶 OPEN DECISION — FLEX BAND ENFORCEMENT (Kev, 2026-07-19) + +**Ruling:** build `EDGE_FLEX_WALL` (−250) + `EV_FLEX_THRESHOLD` (default **4 %**, = 2× +`VALUE_EV_THRESHOLD`) but ship with **`EV_FLEX_ENFORCE=0`**. Flex-band props +(−161…−250) **grade exactly as they do today — the band is NOT cut.** The ledger shows +it is our most profitable segment (**+2.2 % ROI, n=70**) and we will not restrict a +proven-profitable band on an unvalidated threshold. Record `ev_pct` on every flex prop +now so real in-production data accumulates. + +**TRIGGER TO REVISIT:** once ~2 weeks of production `ev_pct` data exists on the +−161…−250 band, report the **`ev_pct` distribution vs settled outcomes for that band**, +then Kev decides the threshold — or whether to enforce at all. +**Do NOT flip `EV_FLEX_ENFORCE=1` without that report and explicit sign-off.** It comes +back as a data-backed decision, never a silent flip. + +**Unaffected — these ship and enforce normally:** `HARD_JUICE_WALL` (−250, never grades), +`LADDER_ODDS_MAX` (+400), `MIN_RUNG_PROBABILITY` (0.25), the no-odds refusal, and the +folded `projection > 0` check. + ### Backtest harness **None exists** (grep-verified). `migrations/006` defines `grade_outcomes` + `player_calibrated_weights` and **no code reads or writes them**. The G-b/C-cal replay was