builtbykev
1a94ef5fcf
Folds re-sequenced steps 1+2 into one change (Kev's call): same bug
family — features wired to sources that return null.
THE PROBABILITY LAYER WAS DEAD IN PRODUCTION. p_win/ev_pct/kelly/
model_odds/value were absent on 0/8 live grades because
gameLogService.getGameLogs returns null for MLB by construction and
depends on the offline Python service for NBA/WNBA, so meta.gameLogs was
[] for every sport. This was the S46 bug in a second location — that fix
gave featureCache an MLB branch (why grades still worked) but never the
estimator. featureCache.getStatRows now supplies normalized rows
([{date,[statType]:v}], most-recent-first) for every sport, feeding the
estimator AND consistency AND game_count_in_7d from one fetch.
VERIFIED on real props: p_win 25/25 WNBA, 8/8 MLB (was 0).
GRADE RANGE, ON MERIT — never by rescaling (permanent founder ruling:
minting A's without new information is a relabelled B sold as an A and
corrupts an append-only ledger).
- refreshTeamStats wired into runSnapshot — it had ZERO production
callers, so opp_rank_stat was permanently null and a +/-1.0 factor
could never fire. Test-env no-op (opsNotify precedent).
- L20 made SYMMETRIC: both branches were delta +1.0, so the season
baseline could only ever ADD. No negative path was a structural reason
D was unreachable. New l20_contradicts_* carries -1.0.
- game_count_in_7d derived from real logged dates (heavy_workload_7d).
- NOT wired, deliberately, with reasons inline: teamId (no team_id
column; getFeatures reads it top-level; factor also needs a starter-id
list) and season_type (ESPN 2 = REGULAR season; threading it raw would
fire veteran_in_playoffs in July). Dead code dressed as a fix is the
thing we are removing, not adding.
CALIBRATION GUARD (found by verifying, not assuming): consistency CV is
NBA-tuned; for a Poisson-ish stat cv ~ 1/sqrt(mean), so any stat with
mean < 4 auto-classifies boom_bust. First verification run showed 8/8 MLB
props boom_bust — a blanket -1.0 that dropped the board to all-C. Floored
at CONSISTENCY_MIN_MEAN=4 -> 'unknown' below. Absent beats wrong. MLB
low-count stats therefore still get no consistency factor: honest, not
fixed. Scale-free index-of-dispersion classifier is the open follow-up.
CONFIDENCE IS NOT A PROBABILITY: payloads carry confidence_basis:
'grade_band'. Corrected mlb-grade-degradation.md — its "25/25
grade<->confidence agreement" is a TAUTOLOGY (confidence is derived FROM
the letter, so it would report 25/25 even if every grade were wrong), not
a validation. Removed dead mlbGrader.js (referenced only by its own test)
and the stale computeFeatures comment claiming a penalty that never ran.
VERIFICATION (scripts/verify-grade-range.js, real props/logs/engine):
WNBA 25 props B 68%->32%, C 32%->64%, D 0->1 (4%); 11-step spread went
from 2 steps to 5 (C/C+/B-/D). The D is earned: Angel Reese assists o2.5,
p_win 0.365. Nothing flooded — grades got HARDER. A did not emit locally
because opp_rank_stat needs the Redis cache only prod populates (local
ceiling +3.0 vs the +4.5 A needs); reachability is proven arithmetically
and locked in tests. Prod A-emission is the outstanding fingerprint.
MARKETING HOLD: "A-RATED" (AccuracyBadge, TopSignals) is unsupported
until that fingerprint. Confirmed honest fallbacks render today —
/api/ledger/accuracy has B and C buckets only, so the badge shows
"MODEL · 63% HIT" and TopSignals self-hides. Nothing fabricated ships.
Suite 276/3286 green, web build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
107 lines
5.9 KiB
Markdown
107 lines
5.9 KiB
Markdown
# MLB Grade Pipeline Degradation — FIXED (2026-07-17)
|
||
|
||
Source: phone-audit P1-7 (broken edge board) + P2-9 (B grades at 45% confidence).
|
||
Diagnosed against LIVE `GET /api/snapshot/mlb` on 2026-07-17. Backend grading
|
||
bug, fixed at the source in the generic grade path (`engine1` +
|
||
`analyzeViaEngine1`), which grades EVERY sport.
|
||
|
||
## Before (25 live MLB grades, degraded)
|
||
- **projection == 0 for 9/25** — graded on a zero projection.
|
||
- **edge_pct quantized to {20, 60, 100, 140}** — the 100s were the proj=0
|
||
degeneracy `(line - 0)/line = 100%`.
|
||
- **grade ↔ confidence mismatch** — 10/25 disagreed even at the 4-letter level
|
||
(25/25 vs the stricter 11-step bands).
|
||
|
||
## Root causes + fixes (commit `888d103`)
|
||
1. **projection=0 bypassed the refusal gate.** `projectionFor` returned
|
||
`l5_avg` even when 0 (finite → the `== null` gate passed it). FIX: a
|
||
non-positive reference is not a projection — `projectionFor` skips it and
|
||
falls through to the next POSITIVE reference (l5 → l20 → per_90 → xg); when
|
||
none is positive it returns null and the read REFUSES (`insufficient_data`).
|
||
The gate also gained an explicit `> 0` guard — the invariant is structural.
|
||
2. **edge_pct.** Formula was already `(model - line)/line` (the intended
|
||
semantics); the {100} cluster was purely the proj=0 degeneracy. With fix 1
|
||
those refuse. Main-line edge now reuses the VALIDATED projection so edge and
|
||
the persisted `projection` can't diverge.
|
||
3. **confidence/letter split.** `engine1.GRADE_TO_CONFIDENCE` was hand-rolled
|
||
and drifted a full sub-tier low (B → 0.55, which `grade_thresholds.json`
|
||
calls B-). FIX: confidence is now DERIVED from each grade's band MIDPOINT in
|
||
`grade_thresholds.json` — one source of truth. Applying the threshold table
|
||
to any grade's displayed confidence resolves back to the same letter (proven
|
||
for all 11 grades in `tests/unit/mlbGradeDegradation.test.js`).
|
||
|
||
> ### ⚠️ CORRECTION (Session 63, 2026-07-19) — THE "25/25 AGREEMENT" WAS A TAUTOLOGY
|
||
>
|
||
> **Do not cite the 25/25 grade↔confidence agreement below as validation of
|
||
> grade quality. It validates nothing.**
|
||
>
|
||
> The fix above made `confidence` a *deterministic function of the letter*:
|
||
> engine1 picks a letter via an additive factor index, then looks up that
|
||
> letter's band midpoint to produce the number (`engine1.js:29-36`). Feeding
|
||
> that number back through the same table can only ever return the letter it
|
||
> came from. **The round-trip would report 25/25 even if every grade were
|
||
> wrong.**
|
||
>
|
||
> It is a real fix for a real bug (the two encodings had drifted a sub-tier
|
||
> apart) — it is simply a *consistency* check, not an *accuracy* check.
|
||
> `confidence` carries ZERO information beyond the letter. The genuinely
|
||
> independent probability is `p_win` (the quantile estimate over real game
|
||
> logs), which Session 63 discovered had never been computed in production at
|
||
> all. Payloads now carry `confidence_basis: 'grade_band'` so no consumer can
|
||
> mistake the derived number for a model probability.
|
||
>
|
||
> Full diagnosis: `specs/audit-data/grade-collapse.md`.
|
||
|
||
## Blast radius (commit `9fc4edf`) — work-order #6
|
||
The degraded grades (projection=0 → `model_value = 0`) are already settled in
|
||
the append-only `ledger_entries` and are NOT deleted. Functional marking:
|
||
`getModelAggregate` now filters `.gt('model_value', 0)` on the settled AND
|
||
pending queries — the rows stay in the ledger but leave the public model record
|
||
(their hit/miss is noise, not skill). Post-fix no new such row can be written.
|
||
|
||
**Exact count NOT queryable from the dev box** (`*.supabase.co` is unreachable
|
||
here — curl 000; only `vyndr.app`/`api.vyndr.app` resolve; no `VYNDR_INTERNAL_KEY`
|
||
locally). Proxy signal: 9/25 (36%) of the current live slate. For the precise
|
||
figure, run in Supabase SQL:
|
||
```sql
|
||
SELECT count(*) FILTER (WHERE outcome IS NOT NULL) AS settled_degraded,
|
||
count(*) AS total_degraded
|
||
FROM ledger_entries
|
||
WHERE user_id IS NULL AND model_value = 0;
|
||
```
|
||
|
||
## Other sports — work-order #5
|
||
NBA/WNBA/soccer grade through the SAME `analyzeViaEngine1` → `engine1` path
|
||
(`gradeSlateService` does not branch by sport; `mlbGrader.js` is dead code). So
|
||
they SHARE the disease and are fixed by the same commit. They rarely grade in
|
||
prod today (stats service offline off-season → refuse anyway). No separate fix.
|
||
|
||
## Live validation — work-order #4
|
||
The fix deploys immediately, but the SNAPSHOT only re-grades on the full cron
|
||
(UTC hours 14,19,22,1,3). Run after the next 14:00 UTC snapshot post-deploy:
|
||
```
|
||
node scripts/validate-grade-fix.js
|
||
```
|
||
PASS criteria: projection==0 count → 0; edge_pct no longer contains 100 and is
|
||
not the four-value cluster; grade↔confidence agreement 25/25. The before-state
|
||
(this file's "Before") is the diff baseline; the script's output is the
|
||
fingerprint.
|
||
|
||
## AFTER — VALIDATED LIVE (2026-07-17 14:00:53 UTC regrade, fingerprint)
|
||
Ran `scripts/validate-grade-fix.js` against the first post-deploy snapshot.
|
||
|
||
| Signal | Before (03:00 UTC) | After (14:00 UTC) |
|
||
|---|---|---|
|
||
| projection == 0 | **9 / 25** | **0 / 25** ✅ |
|
||
| grade ↔ confidence agree | 15 / 25 (4-letter); 0/25 vs 11-step | **25 / 25** ✅ |
|
||
| edge_pct distinct values | 4 — {20, 60, 100, 140} | 7 — {20, 60, 70, 76, 78, 82, 100} ✅ |
|
||
| projection distribution | contained 0 | all positive (min 0.09, no zeros) ✅ |
|
||
|
||
The nine projection-0 grades vanished (those props now refuse). Grade and
|
||
confidence agree on every row. Edges are continuous, not the degenerate cluster.
|
||
The single remaining 100 is NOT the old bug: Wilyer Abreu · hits · line 0.5 ·
|
||
**projection 1.0** · over → `(1.0 − 0.5)/0.5 = 100%`, a real model call of double
|
||
a small line. The `(model − line)/line` metric inherently produces large % on
|
||
0.5-step lines — the frontend `|edge| > 40` guard is the intended safety net for
|
||
exactly that, and it stays. **All three bugs resolved; validated in production.**
|