Revive the dead probability layer + restore grade range ON MERIT
Folds re-sequenced steps 1+2 into one change (Kev's call): same bug
family — features wired to sources that return null.
THE PROBABILITY LAYER WAS DEAD IN PRODUCTION. p_win/ev_pct/kelly/
model_odds/value were absent on 0/8 live grades because
gameLogService.getGameLogs returns null for MLB by construction and
depends on the offline Python service for NBA/WNBA, so meta.gameLogs was
[] for every sport. This was the S46 bug in a second location — that fix
gave featureCache an MLB branch (why grades still worked) but never the
estimator. featureCache.getStatRows now supplies normalized rows
([{date,[statType]:v}], most-recent-first) for every sport, feeding the
estimator AND consistency AND game_count_in_7d from one fetch.
VERIFIED on real props: p_win 25/25 WNBA, 8/8 MLB (was 0).
GRADE RANGE, ON MERIT — never by rescaling (permanent founder ruling:
minting A's without new information is a relabelled B sold as an A and
corrupts an append-only ledger).
- refreshTeamStats wired into runSnapshot — it had ZERO production
callers, so opp_rank_stat was permanently null and a +/-1.0 factor
could never fire. Test-env no-op (opsNotify precedent).
- L20 made SYMMETRIC: both branches were delta +1.0, so the season
baseline could only ever ADD. No negative path was a structural reason
D was unreachable. New l20_contradicts_* carries -1.0.
- game_count_in_7d derived from real logged dates (heavy_workload_7d).
- NOT wired, deliberately, with reasons inline: teamId (no team_id
column; getFeatures reads it top-level; factor also needs a starter-id
list) and season_type (ESPN 2 = REGULAR season; threading it raw would
fire veteran_in_playoffs in July). Dead code dressed as a fix is the
thing we are removing, not adding.
CALIBRATION GUARD (found by verifying, not assuming): consistency CV is
NBA-tuned; for a Poisson-ish stat cv ~ 1/sqrt(mean), so any stat with
mean < 4 auto-classifies boom_bust. First verification run showed 8/8 MLB
props boom_bust — a blanket -1.0 that dropped the board to all-C. Floored
at CONSISTENCY_MIN_MEAN=4 -> 'unknown' below. Absent beats wrong. MLB
low-count stats therefore still get no consistency factor: honest, not
fixed. Scale-free index-of-dispersion classifier is the open follow-up.
CONFIDENCE IS NOT A PROBABILITY: payloads carry confidence_basis:
'grade_band'. Corrected mlb-grade-degradation.md — its "25/25
grade<->confidence agreement" is a TAUTOLOGY (confidence is derived FROM
the letter, so it would report 25/25 even if every grade were wrong), not
a validation. Removed dead mlbGrader.js (referenced only by its own test)
and the stale computeFeatures comment claiming a penalty that never ran.
VERIFICATION (scripts/verify-grade-range.js, real props/logs/engine):
WNBA 25 props B 68%->32%, C 32%->64%, D 0->1 (4%); 11-step spread went
from 2 steps to 5 (C/C+/B-/D). The D is earned: Angel Reese assists o2.5,
p_win 0.365. Nothing flooded — grades got HARDER. A did not emit locally
because opp_rank_stat needs the Redis cache only prod populates (local
ceiling +3.0 vs the +4.5 A needs); reachability is proven arithmetically
and locked in tests. Prod A-emission is the outstanding fingerprint.
MARKETING HOLD: "A-RATED" (AccuracyBadge, TopSignals) is unsupported
until that fingerprint. Confirmed honest fallbacks render today —
/api/ledger/accuracy has B and C buckets only, so the badge shows
"MODEL · 63% HIT" and TopSignals self-hides. Nothing fabricated ships.
Suite 276/3286 green, web build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
This commit is contained in:
@@ -277,4 +277,104 @@ down. `espnStatsAdapter.getPlayerGameLog` (Wave 0) already solves exactly this f
|
||||
|
||||
---
|
||||
|
||||
*Diagnosis 2026-07-19. Data + live API. No engine code changed.*
|
||||
# RESOLUTION — Session 63 (shipped)
|
||||
|
||||
Kev's ruling: **(a) fix on merit, never (b) rescale.** Rescaling would mint A's
|
||||
without adding information — a relabelled B marketed as an A, corrupting an
|
||||
append-only ledger permanently. That option is permanently rejected.
|
||||
|
||||
## What shipped
|
||||
|
||||
| Fix | File | Effect |
|
||||
|---|---|---|
|
||||
| Normalized per-game rows for ALL sports | `featureCache.getStatRows` | Revives `p_win` → `ev_pct`, `kelly`, `model_odds`, `value`, hero v2. Also feeds consistency. |
|
||||
| Rows wired into the grade path | `computeFeatures.safeGetConsistency` | One fetch per prop, shared by 3 starving consumers |
|
||||
| `refreshTeamStats` called in production | `snapshotService.runSnapshot` | `opp_rank_stat` populated → the ±1.0 opponent factor can fire (it had ZERO callers) |
|
||||
| `game_count_in_7d` derived from real logs | `computeFeatures` gameContext | `heavy_workload_7d` (−0.5) can fire |
|
||||
| **L20 symmetry** | `engine1.computeFactors` | NEW `l20_contradicts_*` −1.0. There was no negative L20 path at all — a structural reason D was unreachable |
|
||||
| Consistency CV floor | `consistencyScore` | See calibration finding below |
|
||||
| `confidence_basis: 'grade_band'` | `gradeAdapter.toLegacyShape` | Confidence labelled as derived, not a probability |
|
||||
| Dead `mlbGrader.js` **removed** | — | Referenced only by its own test. Described-but-dead penalty eliminated |
|
||||
|
||||
**Deliberately NOT wired** (would have been dead code dressed as a fix, documented
|
||||
inline): `teamId` (no `team_id` column exists; `getFeatures` reads it top-level not
|
||||
off gameContext; and the factor needs a starter-id list that doesn't exist) and
|
||||
`season_type` (engine1 gates playoff factors on `season_type >= 2`, but ESPN's 2
|
||||
means REGULAR season — threading it raw would fire "veteran_in_playoffs" in July).
|
||||
|
||||
## 🔶 CALIBRATION FINDING — consistency was NBA-tuned and would have flooded `boom_bust`
|
||||
|
||||
Reviving consistency exposed a latent bug. The CV thresholds (`cv >= 0.5` →
|
||||
`boom_bust`) were calibrated for NBA points (mean ~20). For a Poisson-ish counting
|
||||
stat, **cv ≈ 1/√mean**, so any stat with mean < 4 forces `cv > 0.5` — it classifies
|
||||
`boom_bust` regardless of actual behaviour. Verified on real logs:
|
||||
|
||||
- Alonso hits `[0,0,0,1,2,1,0,1,1,0]` → mean 0.60, **cv 1.17** → boom_bust
|
||||
- Henderson hits `[1,0,0,3,1,1,0,0,1,0]` → mean 0.70, **cv 1.36** → boom_bust
|
||||
|
||||
First verification run confirmed it: **8/8 MLB props classified boom_bust**, a
|
||||
blanket −1.0 that dropped the whole board to C. That is a systematic downgrade
|
||||
masquerading as a signal — the mirror image of the "flooding A's" failure Kev
|
||||
warned about.
|
||||
|
||||
**Guard shipped:** `MIN_MEAN_FOR_CV = 4` (env `CONSISTENCY_MIN_MEAN`). Below it,
|
||||
consistency returns `unknown` (no factor) with `reason: 'low_mean_cv_unreliable'`.
|
||||
Absent beats wrong. **Consequence: MLB low-count stats still get no consistency
|
||||
factor** — honest, not fixed. The correct long-term fix is an index-of-dispersion
|
||||
(variance/mean vs the Poisson baseline) classifier, which is scale-free. Tracked
|
||||
as an open item; it is a modelling change needing its own validation.
|
||||
|
||||
## VERIFICATION ON MERIT — real props, real logs, real engine
|
||||
|
||||
`scripts/verify-grade-range.js` replays live-board props through the repaired
|
||||
engine using free feeds (statsapi/ESPN). **Caveat stated up front: `opp_rank_stat`
|
||||
needs the Redis team-stats cache that only production populates, so these local
|
||||
runs OMIT a ±1.0 factor and therefore UNDERSTATE the restored range.**
|
||||
|
||||
**WNBA — 25 real props**
|
||||
|
||||
| | BEFORE (live board) | AFTER (repaired) |
|
||||
|---|---|---|
|
||||
| A | 0 | 0 |
|
||||
| B | 17 (68 %) | 8 (32 %) |
|
||||
| C | 8 (32 %) | 16 (64 %) |
|
||||
| **D** | **0** | **1 (4 %)** |
|
||||
|
||||
11-step spread: `C 6 · C+ 10 · B− 8 · D 1` — five distinct steps where there were
|
||||
two. Revived signals: **`p_win` 25/25 (was 0)**, rows 25/25, consistency known
|
||||
15/25 (the floor correctly abstains on low-mean assists/rebounds).
|
||||
|
||||
The D is earned, not manufactured: *Angel Reese assists over 2.5, p_win 0.365* —
|
||||
the model gives it 36.5 % and says so.
|
||||
|
||||
**MLB — 8 real props:** B 5 / C 3, `p_win` 8/8 (was 0). No A or D on a thin
|
||||
8-prop late-night board of near-identical 0.5-hits props.
|
||||
|
||||
**Reading it honestly:**
|
||||
- **D emits on merit. ✅**
|
||||
- **A did not emit locally** — expected: A needs Σδ ≥ +4.5 and the local ceiling is
|
||||
+3.0 without `opp_rank_stat`. Structural reachability is proven arithmetically
|
||||
and locked in `tests/unit/gradeRangeRestore.test.js`; **empirical A emission
|
||||
requires production and is the outstanding fingerprint.**
|
||||
- **Nothing flooded.** Grades got *harder*, not easier — B fell 68 % → 32 %. The
|
||||
B→C movers are driven by the new L20 negative branch: props whose season
|
||||
baseline contradicts the graded side no longer get a free pass. That is the
|
||||
intended correction.
|
||||
|
||||
## 🔴 MARKETING HOLD — A-rated copy is UNSUPPORTED until A verifiably emits
|
||||
|
||||
Confirmed the honest fallbacks are what render today:
|
||||
- `/api/ledger/accuracy` returns buckets **B and C only** — no A bucket. So
|
||||
`AccuracyBadge`'s `aRated` sample is 0, below `minSample`, and it falls through
|
||||
to **"MODEL · 63% HIT"**. No fabricated A-RATED is displayed.
|
||||
- `TopSignals` self-hides when there are no A-rated grades.
|
||||
|
||||
**Nothing fabricated is shipping — but the copy describes a grade the engine has
|
||||
never emitted.** Do not promote "A-RATED" in marketing, and do not build new
|
||||
surfaces on an A bucket, until a production fingerprint shows real A grades. Lift
|
||||
this hold only against live data.
|
||||
|
||||
---
|
||||
|
||||
*Diagnosed + resolved 2026-07-19 (Session 63). Verified on real props; production
|
||||
A-emission fingerprint outstanding.*
|
||||
|
||||
@@ -30,6 +30,28 @@ bug, fixed at the source in the generic grade path (`engine1` +
|
||||
to any grade's displayed confidence resolves back to the same letter (proven
|
||||
for all 11 grades in `tests/unit/mlbGradeDegradation.test.js`).
|
||||
|
||||
> ### ⚠️ CORRECTION (Session 63, 2026-07-19) — THE "25/25 AGREEMENT" WAS A TAUTOLOGY
|
||||
>
|
||||
> **Do not cite the 25/25 grade↔confidence agreement below as validation of
|
||||
> grade quality. It validates nothing.**
|
||||
>
|
||||
> The fix above made `confidence` a *deterministic function of the letter*:
|
||||
> engine1 picks a letter via an additive factor index, then looks up that
|
||||
> letter's band midpoint to produce the number (`engine1.js:29-36`). Feeding
|
||||
> that number back through the same table can only ever return the letter it
|
||||
> came from. **The round-trip would report 25/25 even if every grade were
|
||||
> wrong.**
|
||||
>
|
||||
> It is a real fix for a real bug (the two encodings had drifted a sub-tier
|
||||
> apart) — it is simply a *consistency* check, not an *accuracy* check.
|
||||
> `confidence` carries ZERO information beyond the letter. The genuinely
|
||||
> independent probability is `p_win` (the quantile estimate over real game
|
||||
> logs), which Session 63 discovered had never been computed in production at
|
||||
> all. Payloads now carry `confidence_basis: 'grade_band'` so no consumer can
|
||||
> mistake the derived number for a model probability.
|
||||
>
|
||||
> Full diagnosis: `specs/audit-data/grade-collapse.md`.
|
||||
|
||||
## Blast radius (commit `9fc4edf`) — work-order #6
|
||||
The degraded grades (projection=0 → `model_value = 0`) are already settled in
|
||||
the append-only `ledger_entries` and are NOT deleted. Functional marking:
|
||||
|
||||
Reference in New Issue
Block a user