diff --git a/scripts/pwin-timeforward.sql b/scripts/pwin-timeforward.sql new file mode 100644 index 0000000..1a4ebde --- /dev/null +++ b/scripts/pwin-timeforward.sql @@ -0,0 +1,33 @@ +-- pwin-timeforward.sql — calibration refresh on the CURRENT MLB sample (2026-08-01) +-- MEASURE-ONLY. Time-forward: earlier games fit/observe, later games prove. +-- +-- Deliberately RULER-INDEPENDENT, and that is the finding: reliability +-- (predicted vs actual hit rate) and resolution (does higher p_win hit more) +-- are both p_win-vs-outcome measures. No fair_prob appears anywhere below, +-- because none can. This is why the consensus ruler cannot change the +-- calibration verdict. +-- +-- reliability = n-weighted mean |predicted - actual| across deciles. NOTE: +-- mean|p - outcome| on 0/1 rows is NOT calibration -- it is noise-dominated +-- individual error. Bucket first. +-- +-- MLB only. WNBA abstains on its own data and is not re-litigated here. + +with base as ( + select sport, game_date, p_win::numeric p, (outcome='hit')::int won, + ntile(2) over (order by game_date, id) half + from public.ledger_entries + where sport='mlb' and user_id is null and outcome in ('hit','miss') and p_win is not null), +s as (select *, case when half=1 then 'train' else 'holdout' end split from base), +b as (select split, width_bucket(p, 0.0, 1.0, 10) bkt, + count(*) n, avg(p) pred, avg(won::numeric) actual + from s group by 1,2) +select split, + sum(n) total_n, + count(*) buckets, + round(sum(n*abs(pred-actual))/sum(n),4) reliability_mean_abs_dev, + round((select corr(p, won::numeric) from s s2 where s2.split=b.split)::numeric,4) resolution_corr, + round((select avg(won::numeric) from s s3 where s3.split=b.split)::numeric,4) base_rate, + (select min(game_date) from s s4 where s4.split=b.split) first_game, + (select max(game_date) from s s5 where s5.split=b.split) last_game +from b group by split order by split desc; diff --git a/scripts/ruler-comparison.sql b/scripts/ruler-comparison.sql new file mode 100644 index 0000000..4835683 --- /dev/null +++ b/scripts/ruler-comparison.sql @@ -0,0 +1,75 @@ +-- ruler-comparison.sql — Order: MLB CALIBRATION RE-RUN vs CONSENSUS RULER (2026-08-01) +-- MEASURE-ONLY. Run against prod (Supabase MCP execute_sql). +-- +-- WHAT THIS DOES AND DOES NOT ANSWER +-- +-- It does NOT re-fit the p_win calibration. It cannot: `estimateProbability` +-- takes {gameLogs, line, statType, features} and never sees a market price, and +-- the calibration query fits p_win against OUTCOMES. Reliability and resolution +-- are both p_win-vs-outcome measures, so the ruler cannot enter either. See +-- pwin-timeforward.sql for the (ruler-independent) calibration refresh. +-- +-- What IS ruler-dependent is EDGE (p_win - fair_prob). This measures whether +-- swapping the incumbent single-book ruler for a median consensus rescues it. +-- +-- TIMING IS HELD CONSTANT: both rulers are read at CLOSE. The lock-time +-- reconstruction is impossible at usable n (only 43 settled rows join +-- lock_lines with >=2 two-sided books), and mixing a lock-time incumbent with +-- a close-time consensus would confound WHEN with WHAT. +-- +-- LIMITATION, load-bearing: closing_captures contains ONLY MODEL books +-- (draftkings/betmgm/betrivers/fanduel/pinnacle). Exchange quotes were never +-- stored, because normalizeProps discarded them until 2026-08-01. So this can +-- only test a US-books-median ruler, NOT the exchange-inclusive consensus. The +-- exchange ruler is untestable on existing data at any n. + +with imp as ( + select id, player_key, stat, game_date, line, side, p_win, book, (outcome='hit')::int won + from public.ledger_entries + where sport='mlb' and user_id is null and outcome in ('hit','miss') and p_win is not null), + +-- latest CLOSE capture per (prop, book); two-sided only -- a one-sided quote +-- cannot be de-vigged, so it cannot price a ruler. +cap as ( + select distinct on (player_key,stat,game_date,line,book) + player_key,stat,game_date,line,book,over_odds,under_odds + from public.closing_captures + where sport='mlb' and over_odds is not null and under_odds is not null + order by player_key,stat,game_date,line,book,captured_at desc), + +d as (select *, + case when over_odds>0 then 100.0/(over_odds+100) else (-over_odds)/((-over_odds)+100.0) end po, + case when under_odds>0 then 100.0/(under_odds+100) else (-under_odds)/((-under_odds)+100.0) end pu + from cap), + +-- multiplicative two-way de-vig, per book (matches src/utils/devig.js) +f as (select player_key,stat,game_date,line,book, po/(po+pu) fo, pu/(po+pu) fu + from d where po+pu > 0), + +j as (select i.*, f.book ref_book, + case when lower(i.side)='under' then f.fu else f.fo end fair_side + from imp i join f + on f.player_key=i.player_key and f.stat=i.stat + and f.game_date=i.game_date and f.line=i.line), + +a as (select id, p_win, won, game_date, + count(*) n_books, + percentile_cont(0.5) within group (order by fair_side) cons, -- v2: MEDIAN + max(case when ref_book=book then fair_side end) own -- v1: the locked book + from j group by 1,2,3,4), + +e as (select *, p_win-own edge_v1, p_win-cons edge_v2 + from a where n_books>=2 and own is not null) + +select count(*) n, + round(avg(won)::numeric,4) base_rate, + round(avg(abs(cons-own))::numeric,4) mean_abs_ruler_gap, + round(avg(cons-own)::numeric,4) mean_signed_ruler_gap, + round(corr(edge_v1, won::numeric)::numeric,4) corr_edge_v1_won, + round(corr(edge_v2, won::numeric)::numeric,4) corr_edge_v2_won, + round(corr(p_win, won::numeric)::numeric,4) corr_pwin_won, + round(avg(edge_v1) filter (where won=1)::numeric,4) edge_v1_winners, + round(avg(edge_v1) filter (where won=0)::numeric,4) edge_v1_losers, + round(avg(edge_v2) filter (where won=1)::numeric,4) edge_v2_winners, + round(avg(edge_v2) filter (where won=0)::numeric,4) edge_v2_losers +from e; diff --git a/specs/MASTER-PLAN.md b/specs/MASTER-PLAN.md index c9d16e7..0901406 100644 --- a/specs/MASTER-PLAN.md +++ b/specs/MASTER-PLAN.md @@ -29,16 +29,25 @@ stay provisional until re-run** · documented ≠ verified. ## ▶ NEXT EXECUTABLE ORDER -**The MLB calibration RE-RUN against the consensus ruler.** Everything -model-shaped is downstream of it: +**DONE 2026-08-01** — the MLB re-run landed, and it dissolved rather than answered +its own question: **calibration is ruler-independent**, so MLB isotonic was never +gated on the ruler. See `specs/mlb-recalibration-vs-consensus-ruler.md`. -- MLB isotonic `p_win` cannot promote until re-run (it was calibrated against - `v1_first_book`). -- The newly-visible props cannot feed the model until it promotes. -- Every edge/CLV number resets to `ruler_version = v2_consensus` at that boundary - and **must not be pooled** with what came before. +**The next order follows from what that measurement found, not from the plan:** -*Blocked on nothing. This is the next build.* +> **`p_win` predicts outcomes (r = +0.26). `p_win − fair_prob` does not +> (r = −0.01 to −0.02, either ruler).** Subtracting the market destroys the +> signal. So the build is: **make the served product rank on `p_win`, not on +> edge** — and retire market-relative edge from every ranking, gate and display +> where it still sits. + +Two things gate on accrual instead of on code, and cannot be rushed: +- The **exchange-inclusive ruler** is untestable until v2-era captures accrue. +- The **edge verdict** may change under that ruler — or may not; today it is + unproven either way, and 'unproven' is the honest label. + +**Open, cheap, and unrelated to the model:** the 🔴 pinnacle feed regression, and +the display layer still shows nothing of the widened multi-book data. --- @@ -51,8 +60,11 @@ model-shaped is downstream of it: | **books/prop, WNBA** | **4.21** feed → 1.20 | **WNBA is BETTER covered than MLB** | | **consensus ruler** | **MARKET, not SHARP** | `pinnacle`/`matchbook`/`polymarket` = **0%** on both sports. No sharp anchor exists in our feed. Permanent limitation, not a milestone | | **ruler delta** (consensus − incumbent) | MLB mean +1.50 pts, median 0, **17% of props move ≥5 pts** | rulers genuinely differ; "better" is unproven | -| **MLB isotonic `p_win`** | **PROVISIONAL** | calibrated on the bent ruler; does not promote until re-run | +| **MLB isotonic `p_win`** | **DECIDED** — reliability **0.0846**, resolution **0.190**, holdout **n=125** | **PROVISIONAL label RETRACTED 2026-08-01.** Calibration is **ruler-independent** (`estimateProbability` never sees a price; the fit is p_win-vs-outcome). Replicated on a fresh later window, both metrics improved | +| **edge vs the ruler** | corr(edge, outcome) **−0.010** (v1) → **−0.022** (v2), n=200 · corr(**p_win**, outcome) **+0.26** | **Subtracting the market DESTROYS the signal.** The consensus ruler does not rescue edge: *differs ≠ better* | +| **exchange-inclusive ruler** | **UNTESTABLE on existing data** | exchange quotes were never stored (discarded until 2026-08-01). Becomes testable only as v2-era captures accrue | | **WNBA** | **still abstains** | a MODEL problem, not a coverage problem — coverage was never its constraint | +| 🔴 **pinnacle feed** | **0 captures since 2026-07-31** (103,940 in the prior 10 days) | a live regression; **we had a sharp anchor and lost it.** Not caused by our changes | | **soccer** | **settles** — ~15 competitions, 30d | "grades into a void" is a **$19/mo Pro-tier** problem, not a data problem | | **CLV + results feeds** | `/odds/closing` + `/movement` **redacted**; `/results` **403 `required_tier: hobby`**; `/exports/resolved-props` **403 `required_tier: pro`** | **verified on our keys** — plain tier exclusion, not a key or plan fault. **$9/mo** buys CLV + steam + results; **$19/mo** adds the 90-day settlement export | | **books SERVED** | 5 → **13**; props rendered **546 → 2,780** (5.1×); mean **4.22** books/prop | **AGGREGATOR widening is LIVE.** Of 2,234 newly-visible props, **31.2% carry a real non-DFS price**; **68.8% are DFS-only** — shown, tagged, never a market | diff --git a/specs/mlb-recalibration-vs-consensus-ruler.md b/specs/mlb-recalibration-vs-consensus-ruler.md new file mode 100644 index 0000000..f4d3a96 --- /dev/null +++ b/specs/mlb-recalibration-vs-consensus-ruler.md @@ -0,0 +1,181 @@ +# MLB CALIBRATION RE-RUN vs THE CONSENSUS RULER + +**Date:** 2026-08-01 · **MEASURE-ONLY** · no tier spend · live path byte-identical · +queries committed (`scripts/ruler-comparison.sql`, `scripts/pwin-timeforward.sql`). + +--- + +## VERDICT UP FRONT + +**Mandate 1's premise does not hold, and I have to say so before reporting numbers +against it.** + +> **The p_win calibration is RULER-INDEPENDENT. Re-deriving `fair_prob` against the +> consensus ruler cannot change a single number in the isotonic result.** + +Two independent confirmations: + +1. **`estimateProbability({ gameLogs, line, statType, features })`** + (`src/services/intelligence/probabilityEstimator.js:54`) — **no market price, no + book odds, no `fair_prob` anywhere in its inputs.** `p_win` is computed from game + logs against the line. +2. **The calibration query fits `p_win` against OUTCOMES.** Reliability (predicted vs + actual hit rate) and resolution (does higher `p_win` hit more often) are both + `p_win`-vs-outcome measures. `fair_prob` cannot enter either. + +**So there is nothing to re-fit.** The ruler changes **edge** (`p_win − fair_prob`), +**CLV**, and **takeable** — not calibration. + +### I have to retract my own label + +**I declared the MLB isotonic result PROVISIONAL "because it was measured against +the bent ruler." That was wrong** — I over-applied the ruler caveat to a +measurement the ruler never touched. **The isotonic result was never contaminated.** + +It moves from **PROVISIONAL → DECIDED**, not by re-running, but because the gate I +attached to it does not apply. It stands on its own terms. + +--- + +## WHAT I RAN INSTEAD (the genuinely ruler-dependent question) + +**Does swapping the incumbent single-book ruler for a median consensus rescue +EDGE?** That question *is* ruler-dependent, and it is the one that came back +negative in three formulations. + +**Timing held constant** — both rulers read at CLOSE. A lock-time reconstruction is +impossible at usable n (**only 43** settled rows join `lock_lines` with ≥2 +two-sided books), and pairing a lock-time incumbent with a close-time consensus +would confound *when* with *what*. + +### Result — MLB, n=200 settled rows, 2026-07-21 → 07-30 + +| measure | value | +|---|---:| +| base rate | 0.525 | +| **mean \|ruler gap\|** | **0.0085** (0.85 prob points) | +| mean signed ruler gap | −0.0004 | +| **corr(edge **v1 single-book**, outcome)** | **−0.0101** | +| **corr(edge **v2 consensus**, outcome)** | **−0.0220** | +| **corr(`p_win` alone, outcome)** | **+0.2598** | +| edge v1: winners / losers | 0.0281 / 0.0315 | +| edge v2: winners / losers | 0.0265 / 0.0339 | + +**The consensus ruler does not rescue edge.** Both rulers give a correlation +indistinguishable from zero, and both are very slightly *negative* — higher edge +associates marginally with *losing*, under either ruler. + +### The number that actually matters + +**`p_win` alone correlates +0.26 with outcome. Edge — `p_win` minus the market — +correlates −0.01 to −0.02.** + +**Subtracting the market price destroys the signal.** The model has genuine +forecasting signal; the market-relative transformation removes it. That is a +direct empirical vindication of the identity now at the top of CLAUDE.md: *market +edge is a byproduct, never the success criterion.* Here it is not merely a poor +criterion — **it is a strictly worse instrument than the raw forecast.** + +--- + +## CALIBRATION REFRESH (ruler-independent, but n has grown) + +Worth running for a different reason: the prior report used **n=119** holdout. MLB +now has **250** settled rows with `p_win`. Time-forward, earlier fits, later proves. + +| split | n | buckets | reliability (mean \|dev\|) | resolution (corr) | base rate | window | +|---|---:|---:|---:|---:|---:|---| +| train | 125 | 9 | 0.0951 | 0.2557 | 0.528 | 07-21 → 07-26 | +| **holdout** | **125** | **9** | **0.0846** | **0.1902** | 0.560 | 07-26 → 07-30 | + +**Both hold on a fresh, later window — and both improved** versus the prior report +(reliability 0.0939 → **0.0846**; resolution 0.123 → **0.190**). Resolution degrades +train → holdout (0.256 → 0.190), which is expected and honest; it stays clearly +positive rather than crushing to base rate. + +**This is independent replication on data the earlier fit never saw.** It is the +strongest evidence to date that MLB `p_win` is honest *and* still ranks. + +--- + +## THE LIMITATION THAT PREVENTS A FULL VERDICT + +**The ruler I could test is not the ruler that was built.** + +`closing_captures` contains **only MODEL books** — draftkings, betmgm, betrivers, +fanduel, pinnacle. **Exchange quotes were never stored**, because `normalizeProps` +discarded them until 2026-08-01. Mean books per prop in the historical join: +**1.97**. + +So this measured a **US-books-median vs one-US-book** ruler change — which is small +by construction (0.85 prob points), and nothing like the exchange-inclusive +consensus, where the live delta showed **p90 +10 points and 17% of props moving +≥5 points**. + +> **The exchange-inclusive consensus ruler is UNTESTED and UNTESTABLE on existing +> data at any n.** It becomes testable only as v2-era captures accumulate — which +> starts now, because the display widening went live yesterday. + +**I am not forcing a verdict on it.** Per Mandate 4's third outcome: **inconclusive +on holdout n.** + +--- + +## VERDICT, PER MANDATE 4 + +| question | verdict | +|---|---| +| Does the consensus ruler improve the **calibration**? | **Question does not apply** — calibration is ruler-independent (proven in code and in the query). | +| Is the MLB isotonic result still provisional? | **No — DECIDED.** My PROVISIONAL label was over-applied and is retracted. | +| Does the consensus ruler rescue **EDGE** (US-books version, n=200)? | **DIFFERS BUT NOT BETTER.** −0.010 → −0.022. Rulers differ ≠ new ruler better. | +| Does the **exchange-inclusive** ruler rescue edge? | **INCONCLUSIVE — n-blocked.** Untestable on existing data; the quotes were never stored. | +| MLB `p_win` honest **and** ranking, held out? | **YES, replicated on fresh data.** reliability 0.0846, resolution 0.190, n=125. | +| WNBA | **Abstains, unchanged.** Not re-litigated. | + +**Nothing is promoted. Nothing is flipped.** `CURRENT_RULER_VERSION` remains +`v1_first_book`; the model still consumes `MODEL_BOOKS` only. + +**No promotion is scoped**, because the trigger condition — "new ruler is better" — +was not met on the testable portion and could not be evaluated on the rest. + +--- + +## 🔴 SEPARATE FINDING — A LIVE FEED REGRESSION + +While establishing feasibility I found this, and it is unrelated to the order: + +| date | pinnacle MLB captures | other books | +|---|---:|---:| +| 07-26 | 15,008 | 115,638 | +| 07-29 | 10,406 | 82,080 | +| 07-30 | 4,022 | 63,602 | +| **07-31** | **0** | 47,606 | +| **08-01** | **0** | 45,092 | + +**Pinnacle prop coverage went to zero on 2026-07-31 and has not returned**, while +every other book continued normally. 103,940 captures over the preceding 10 days, +then a clean cliff. + +**This also corrects an Order Zero claim of mine.** I reported "pinnacle 0% — no +sharp anchor exists in our feed." That was accurate *for the day I measured* but +wrong as a general statement: **Pinnacle was in the MLB props feed until 07-30**, +with 17,090 two-sided captures. (`line_type='sharp'` is a label applied in +`closingCapture.js` via `SHARP_BOOKS`, not a separate provider — I checked.) + +**We had a sharp anchor and lost it two days ago.** Whether that is a PropLine +change, a regional restriction, or a bug is unknown and worth one question to the +provider. It is not caused by anything in this session's work — the widening only +adds books. + +## TAGS + +**VERIFIED:** calibration is ruler-independent (code + query) · consensus ruler +does not rescue edge at n=200 · MLB p_win replicates honest+ranking on a fresh +holdout (n=125) · pinnacle coverage stopped 07-31. + +**CANNOT DETERMINE:** whether the exchange-inclusive consensus ruler improves edge +— the quotes were never stored, so it is untestable at any n until v2-era captures +accumulate. + +**RETRACTED:** my own "MLB isotonic is PROVISIONAL pending the ruler fix" — the +ruler never touched it.