diff --git a/scripts/opportunity-axis-holdout.sql b/scripts/opportunity-axis-holdout.sql new file mode 100644 index 0000000..947b4ad --- /dev/null +++ b/scripts/opportunity-axis-holdout.sql @@ -0,0 +1,49 @@ +-- opportunity-axis-holdout.sql — the Step 4 proof, RUN WHEN n IS ADEQUATE. +-- +-- Cannot run yet, by construction: the axis went live 2026-08-01 and its first +-- rows carry game_date 2026-08-01 (games not yet played). Settled rows carrying +-- the axis: 0. The earliest possible run is the next morning settle pass, and a +-- defensible n is several days out at ~140 opportunity-axis rows per snapshot. +-- +-- BOTH reliability AND resolution must improve for the axis to promote. One or +-- neither => SHELVE and record why. +-- +-- reliability = n-weighted mean |predicted - actual| across deciles. Bucket +-- FIRST: mean|p - outcome| on 0/1 rows is noise-dominated individual error, not +-- calibration. + +with rows_ as ( + select game_date, id, + p_win::numeric champ, + p_win_challenger::numeric chal, + (outcome='hit')::int won + from public.ledger_entries + where sport='mlb' and user_id is null + and outcome in ('hit','miss') + and p_win is not null and p_win_challenger is not null + -- ONLY rows the opportunity axis actually touched. Including untouched rows + -- would dilute the comparison with rows where challenger === champion by + -- construction, and make a null result look like a small positive one. + and challenger_adjustments::text like '%opportunity%' +), +split as ( + select *, case when ntile(2) over (order by game_date, id) = 1 then 'train' else 'holdout' end split + from rows_ +), +b_champ as ( + select split, width_bucket(champ, 0.0, 1.0, 10) bkt, count(*) n, avg(champ) pred, avg(won::numeric) actual + from split group by 1,2), +b_chal as ( + select split, width_bucket(chal, 0.0, 1.0, 10) bkt, count(*) n, avg(chal) pred, avg(won::numeric) actual + from split group by 1,2) +select + s.split, + count(*) n, + round((select sum(n*abs(pred-actual))/nullif(sum(n),0) from b_champ c where c.split=s.split),4) reliability_champion, + round((select sum(n*abs(pred-actual))/nullif(sum(n),0) from b_chal c where c.split=s.split),4) reliability_challenger, + round(corr(s.champ, s.won::numeric)::numeric,4) resolution_champion, + round(corr(s.chal, s.won::numeric)::numeric,4) resolution_challenger, + round(avg(s.won::numeric),4) base_rate +from split s +group by s.split +order by s.split desc; diff --git a/specs/opportunity-drift-axis.md b/specs/opportunity-drift-axis.md new file mode 100644 index 0000000..e0d3ae8 --- /dev/null +++ b/specs/opportunity-drift-axis.md @@ -0,0 +1,155 @@ +# OPPORTUNITY_DRIFT AXIS — BUILT, LIVE AS A CHALLENGER, PROOF PENDING + +**Date:** 2026-08-01 · champion `p_win` and the live grade path **byte-identical** · +ledger-only challenger · queries committed (`scripts/opportunity-axis-holdout.sql`). + +**Gates:** 4,073 tests / 326 suites green · `next build` exit 0 · input mapped · +coverage measured · **collinearity guard PASSED** · axis writing on prod. + +--- + +## STEP 1 — INPUT MAPPED, AND MEASURED + +`MLB_LOG_FIELD` now maps `at_bats → 'atBats'`. + +**Deliberately NOT added to the other two MLB maps** (`outcomeService`'s +`MLB_LOG_FIELD`, `liveTrackingService`'s `LIVE_BOX_FIELD`). Those exist to +**settle** and **track** graded props; nothing grades at-bats, so adding it there +would imply a settlement path for a market we do not carry. A test asserts the +settle map still lacks it. + +### Coverage on real props (n=100, MLB) + +| feature | coverage | +|---|---:| +| `opportunity_drift` | **94%** | +| `recent_ab_per_game` | 94% | +| `ab_per_game` | 94% | + +By stat — and the split is exactly what it should be: + +| stat | n | drift | +|---|---:|---:| +| `total_bases` | 42 | **100%** | +| `hits` | 36 | **100%** | +| `home_runs` | 8 | **100%** | +| `strikeouts` | 5 | 40% | +| `outs` / `earned_runs` / `hits_allowed` | 3 each | 66.7% | + +**Batters 100%, pitchers partial** — correct: pitchers accumulate few at-bats, so +the drift is genuinely undefined for many of them and **abstains** rather than +inventing a ratio. + +--- + +## STEP 2 — 🟢 THE COLLINEARITY GUARD PASSES DECISIVELY + +This was the gate that could have killed the axis before it was worth building. + +| correlation | n | r | +|---|---:|---:| +| `drift` vs `l20_avg` | 94 | **−0.020** | +| `drift` vs `l5_avg` | 94 | **+0.027** | +| `drift` vs `ab_per_game` | 94 | **−0.029** | + +**All three are essentially zero.** The drift ratio is **orthogonal to every +existing projection input** — it carries information the projection does not +already contain. + +**This also validates the ratio-over-level decision empirically.** `ab_per_game` +(the level) is the same quantity divided by the same denominator as `l20_avg`; +had we used it, we would have added a redundant input. **Dividing by the player's +own baseline removed the collinearity** — r = −0.029 against the very quantity it +is built from. + +--- + +## STEP 3 — THE AXIS IS LIVE AS A CHALLENGER + +`opportunityNudge` composes in the same log-odds space as park and platoon +(`log` of a ratio), with **two guards the measured axes do not get**: + +- **A ±10% deadband.** A rest day, a pinch-hit appearance or a blowout can move a + 5-game window without any change in role. +- **A tighter cap — 0.15 vs the environment's 0.30.** This is a *proxy*, not a + measurement, and a noisy proxy must not outvote measured signals. + +Every adjustment carries **`is_proxy: true`** and +**`proxy_for: 'confirmed_batting_order'`**, so nothing downstream can mistake it +for a lineup feed. + +The axis can **stand alone** — without that, the challenger's early return would +have gated opportunity off on exactly the thin-classification rows it is most +likely to help. + +**Zero extra I/O:** `analyzeViaEngine1` attaches drift from the feature vector it +has already built; `attachChallenger` reads it off the grade. Nothing re-fetches +in a loop running over hundreds of props. + +### Verified writing on prod (induced snapshot, 416 grades) + +| | | +|---|---:| +| rows with a challenger value | 275 / 276 | +| **rows carrying the opportunity axis** | **142 (51.4%)** | +| rows where the challenger moved | 190 | +| mean \|delta\| when it moved | **0.034** | +| delta range | −0.089 … +0.108 | + +**Champion `p_win` unchanged. Live grade path unchanged.** The axis writes only +to `p_win_challenger` / `challenger_adjustments`. + +--- + +## STEP 4 — ⏳ HOLDOUT PROOF IS n-BLOCKED BY CONSTRUCTION + +``` +settled MLB rows (all): 605 +settled rows with a challenger value: 250 +settled rows carrying the OPPORTUNITY axis: 0 +first game_date carrying the axis: 2026-08-01 (tonight — not yet played) +``` + +**The axis went live today. Its first rows are on games that have not happened.** +No holdout can be run, and **I am not going to manufacture one** — running the +test on rows the axis never touched would dilute the comparison with rows where +challenger ≡ champion by construction, and **make a null result look like a small +positive one.** + +**Earliest possible run:** tomorrow's settle pass. **Defensible n:** several days, +at ~140 axis-carrying rows per snapshot. + +The query is written and committed (`scripts/opportunity-axis-holdout.sql`). It +filters to `challenger_adjustments LIKE '%opportunity%'` for exactly the reason +above, buckets before measuring reliability, and splits time-forward. + +**BOTH reliability and resolution must improve. One or neither → SHELVE.** + +--- + +## STEP 5 — NOT REACHED + +Nothing promoted. Nothing flipped. The axis sits in the ledger accruing evidence. + +## THE PROXY CAVEAT, CARRIED FORWARD + +**This is a proxy for tonight's batting order, not a measurement of it.** The real +input is the confirmed lineup slot, which no wired source exposes +(`depthChartService` returns `battingOrder: null` for MLB; PropLine `/context` +carries only a `lineup_confirmed` boolean). **If a lineup feed becomes available, +this axis should be replaced by the real slot, not supplemented with it.** + +## A MEASUREMENT TRAP WORTH RECORDING + +The first prod coverage run reported **`opportunity_drift` at 0%** while +`ab_per_game` read 94% — which looks exactly like "the feature does not compute". + +It was the **120-second feature-vector cache** (`VECTOR_TTL_SECONDS`): the probe +ran inside the window and read payloads written by the previous image, which had +no such field. Re-running past the TTL gave 94%. + +**A new feature field is invisible for one cache generation after deploy.** Worth +knowing before concluding a feature is broken — I very nearly reported this one +as absent, having already confirmed against the live statsapi payload that +`atBats` was present and that the code produced `drift = 1.05` locally on that +exact data. **The contradiction between those two facts is what saved it.**