diff --git a/scripts/grade-correlation-proof.sql b/scripts/grade-correlation-proof.sql new file mode 100644 index 0000000..10e528a --- /dev/null +++ b/scripts/grade-correlation-proof.sql @@ -0,0 +1,31 @@ +-- GRADE CORRELATION PROOF (2026-07-31). Re-runnable evidence for +-- specs/grade-fix-part1-investigation.md. LOOKAHEAD GUARD: fair_prob_lock is the +-- LOCK-TIME market probability; closing_prob (the close) is deliberately unused. + +-- 1) Candidate edge formulations vs outcome, per sport. +with r as ( + select sport, (outcome='hit')::int as won, p_win::numeric as p, fair_prob_lock::numeric as f, + case grade when 'A+' then 10 when 'A' then 9 when 'A-' then 8 when 'B+' then 7 when 'B' then 6 + when 'B-' then 5 when 'C+' then 4 when 'C' then 3 when 'C-' then 2 when 'D' then 1 else 0 end as champ_idx + from ledger_entries + where user_id is null and outcome in ('hit','miss') + and p_win is not null and fair_prob_lock is not null and grade is not null +) +select sport, count(*) n, + corr(champ_idx, won) champ_letter_r, + corr(p, won) p_win_alone_r, + corr(p - f, won) additive_edge_r, + corr(p / nullif(f,0), won) ratio_edge_r, + corr(ln(p/(1-p)) - ln(f/(1-f)), won) logodds_edge_r +from r group by sport; + +-- 2) Time-forward split (overfitting guard), MLB. +with r as ( + select game_date, (outcome='hit')::int as won, p_win::numeric as p, fair_prob_lock::numeric as f + from ledger_entries + where user_id is null and outcome in ('hit','miss') and sport='mlb' + and p_win is not null and fair_prob_lock is not null +), s as (select *, ntile(2) over (order by game_date) half from r) +select half, count(*) n, min(game_date) from_date, max(game_date) to_date, + corr(p, won) p_win_alone_r, corr(p - f, won) additive_edge_r +from s group by half order by half; diff --git a/specs/grade-fix-part1-investigation.md b/specs/grade-fix-part1-investigation.md new file mode 100644 index 0000000..9393106 --- /dev/null +++ b/specs/grade-fix-part1-investigation.md @@ -0,0 +1,105 @@ +# THE GRADE FIX β€” PART 1 INVESTIGATION (report; STOPPED at the gate) +2026-07-31. Nothing rebuilt, no grade changed, no surface cut over. + +# πŸ”΄ HEADLINE: THE PROPOSED REBUILD IS REFUTED BY THE DATA + +**Grading on p_win-vs-fair_prob does NOT work. Every edge formulation correlates +NEGATIVELY with outcomes β€” on both sports, overall, and in both time splits.** +The order said to verify rather than assume, and to STOP on a wrong premise. This +is that case. + +## The evidence (n=432 decided rows carrying BOTH p_win and fair_prob_lock) + +| sport | n | champ letter r | **p_win ALONE r** | additive (pβˆ’f) r | ratio r | log-odds r | +|---|---|---|---|---|---|---| +| **ALL** | 432 | βˆ’0.0016 | **+0.1221** | **βˆ’0.0615** | **βˆ’0.1161** | **βˆ’0.0438** | +| **MLB** | 240 | +0.0984 | **+0.2278** | **βˆ’0.0336** | **βˆ’0.1350** | **βˆ’0.0124** | +| **WNBA** | 192 | βˆ’0.1143 | **βˆ’0.0842** | **βˆ’0.1326** | **βˆ’0.1281** | **βˆ’0.1243** | + +**All three candidate edge definitions (2a) are negative everywhere.** Subtracting +the market's lock-time fair probability does not sharpen the signal β€” it destroys +it and inverts it. + +**The plain reading:** the props where our model most disagrees with the market are +*less* likely to hit. That is the market being better than the model. "Edge vs the +market" is therefore anti-predictive here, while the raw probability estimate +retains some skill on its own. + +## What DOES carry signal: `p_win` alone, MLB only β€” and it is modest + +Time-forward split (overfitting guard β€” fit-free, just correlation on each half): + +| MLB split | dates | n | **p_win alone r** | additive edge r | hit% | +|---|---|---|---|---|---| +| TRAIN (older) | 07-21 β†’ 07-26 | 120 | **+0.2770** | βˆ’0.0027 | 50.8 | +| **HOLDOUT (newer)** | 07-26 β†’ 07-30 | 120 | **+0.1647** | **βˆ’0.0688** | 57.5 | + +`p_win` alone **survives forward validation directionally** (positive in both halves). +**But the holdout is NOT statistically significant: r=0.165 at n=120 β†’ tβ‰ˆ1.81, pβ‰ˆ0.07.** +Suggestive, not proven. The additive edge is negative in both halves. + +**WNBA must ABSTAIN (2c/2d).** Every measure is negative, including p_win itself +(βˆ’0.084). There is no honest grade to build there β€” forcing one threshold across both +sports would make a coin-flip sport look sharp, which 2c explicitly forbids. + +## Sample-size reality (2c per-sport honesty) +1,103 decided rows exist, but only **442 carry p_win and 432 carry `fair_prob_lock`** β€” +the instrument fields only began populating recently. After a per-sport split that is +**MLB 240 / WNBA 192**, and a train/holdout split leaves **~120 per half**. Threshold +fitting on 120 rows, validated on 120, is thin: it can show direction but cannot +certify a letter ladder. + +## LOOKAHEAD GUARD β€” satisfied, and the field is named +**`ledger_entries.fair_prob_lock` is the lock-time truth** and is populated on 432 +decided rows, range 0.145–0.713 (a probability, correct scale). `p_win` ranges +0.100–0.950 (the estimator's clamp). `closing_prob` (415 rows) is the CLOSE and was +**not** used in any correlation above β€” using it would have manufactured a +correlation out of lookahead. + +## Why the served letter is rβ‰ˆ0 (confirmed) +`engine1.gradeProp` sums flat Β±1.0/Β±0.5 factor deltas onto a neutral 'C'; the file +contains **zero references to any probability**. `edge_pct` β€” the price-free +`(projβˆ’line)/line` β€” is not an input to the letter either (engine1 has no `edge` +reference); it drives the *board sort* and *was* the display figure until it was +retired 07-31. So the letter is a factor index that never sees price or probability, +which is why it lands at rβ‰ˆ0.005 and inverts (B 52.4% < C 56.9%). + +## Provenance of the r=0.236 figure +It is mine, measured this session on 354 settled rows (MLB 224) β€” the probability +letter banded through `grade_thresholds.json`. Today's recomputation on the +p_win/fair_prob_lock population gives **MLB p_win r = 0.2278 (n=240)**, consistent. + +--- + +# WHAT I RECOMMEND INSTEAD (not built β€” needs your call) + +**Do not ship the specified rebuild.** The defensible options: + +1. **Grade MLB on `p_win` alone; WNBA abstains.** Honest, matches the evidence. + But the holdout is pβ‰ˆ0.07 β€” so it ships as *"modest, directional, accruing"*, not + as a proven edge, and the A-RATED hold stays. +2. **Wait for sample.** The instrument fields are ~10 days old. At ~90 decided + MLB rows/week, nβ‰ˆ500 MLB (a 250/250 split) is ~6 weeks out and would move pβ‰ˆ0.07 + toward significance if the effect is real. +3. **Investigate WHY the market-relative edge inverts.** That negative correlation is + itself information: it says the model's disagreement with the market is noise. That + is the more valuable question than any threshold ladder. + +**What I will not do:** tune a formulation until a positive number appears. Three +pre-registered candidates were tested and all three failed; picking a fourth because +the first three lost is exactly the overfitting the order guards against. + +## Part 4 loose ends (unchanged, tracked) +`dclv` 0 rows/'unknown' β€” its own follow-up. NBA/soccer never settle β€” a per-sport +feed decision, and the dead `/api/grading/resolve` must NOT be wired (it would +double-count the working settlement). **Working settlement untouched by this order.** + +## REPRODUCIBLE +Both queries are committed at `scripts/grade-correlation-proof.sql` β€” re-runnable, +so no number here has to be taken on trust. + +## TAGS +VERIFIED: all correlations above; `fair_prob_lock` is lock-time and populated; the +served letter has no probability input. CANNOT DETERMINE: whether p_win-alone clears +significance β€” n is too small (holdout pβ‰ˆ0.07). **BLOCKED: Parts 2/3/4 β€” the +specified rebuild is refuted, so Part 2's ship gate cannot be met as written.**