builtbykev
48a2f764ac
STEP 1 -- input mapped and measured. opportunity_drift 94% coverage on 100 real props: 100% for batters (total_bases, hits, home_runs), 40-67% for pitchers, which is correct -- pitchers accumulate few at-bats so the ratio is genuinely undefined and ABSTAINS rather than being invented. STEP 2 -- THE COLLINEARITY GUARD PASSES DECISIVELY. Pearson r on n=94: drift vs l20_avg -0.020, vs l5_avg +0.027, vs ab_per_game -0.029. All essentially zero, so the axis is orthogonal to every existing projection input and carries information the projection does not already contain. That also validates the ratio-over-level decision EMPIRICALLY: ab_per_game is the same quantity over the same denominator as l20_avg, so the level would have been redundant. Dividing by the player's own baseline removed the collinearity -- r = -0.029 against the very quantity it is built from. STEP 3 -- live as a challenger, verified on prod over an induced 416-grade snapshot: 142 of 276 rows (51.4%) carry the opportunity axis, the challenger moved on 190 rows, mean |delta| 0.034, range -0.089..+0.108. Champion p_win and the live grade path are unchanged. STEP 4 -- HOLDOUT IS n-BLOCKED BY CONSTRUCTION and I am not manufacturing one. Settled rows carrying the axis: 0. Its first rows carry game_date 2026-08-01 -- games that have not been played. Running the test on rows the axis never touched would dilute the comparison with rows where challenger === champion by construction, making a null result look like a small positive one. Query committed for when n arrives; it filters to axis-carrying rows for exactly that reason, buckets before measuring reliability, and splits time-forward. BOTH metrics must improve or the axis is shelved. A MEASUREMENT TRAP RECORDED: the first prod run showed drift at 0% while ab_per_game read 94% -- indistinguishable from "the feature does not compute". It was the 120-second feature-vector cache serving payloads written by the previous image. A new feature field is invisible for one cache generation after deploy. I nearly reported it absent, having already confirmed atBats is present in the live statsapi payload and that the code produced drift = 1.05 locally on that exact data; the contradiction between those two facts is what saved it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
156 lines
6.0 KiB
Markdown
156 lines
6.0 KiB
Markdown
# OPPORTUNITY_DRIFT AXIS — BUILT, LIVE AS A CHALLENGER, PROOF PENDING
|
||
|
||
**Date:** 2026-08-01 · champion `p_win` and the live grade path **byte-identical** ·
|
||
ledger-only challenger · queries committed (`scripts/opportunity-axis-holdout.sql`).
|
||
|
||
**Gates:** 4,073 tests / 326 suites green · `next build` exit 0 · input mapped ·
|
||
coverage measured · **collinearity guard PASSED** · axis writing on prod.
|
||
|
||
---
|
||
|
||
## STEP 1 — INPUT MAPPED, AND MEASURED
|
||
|
||
`MLB_LOG_FIELD` now maps `at_bats → 'atBats'`.
|
||
|
||
**Deliberately NOT added to the other two MLB maps** (`outcomeService`'s
|
||
`MLB_LOG_FIELD`, `liveTrackingService`'s `LIVE_BOX_FIELD`). Those exist to
|
||
**settle** and **track** graded props; nothing grades at-bats, so adding it there
|
||
would imply a settlement path for a market we do not carry. A test asserts the
|
||
settle map still lacks it.
|
||
|
||
### Coverage on real props (n=100, MLB)
|
||
|
||
| feature | coverage |
|
||
|---|---:|
|
||
| `opportunity_drift` | **94%** |
|
||
| `recent_ab_per_game` | 94% |
|
||
| `ab_per_game` | 94% |
|
||
|
||
By stat — and the split is exactly what it should be:
|
||
|
||
| stat | n | drift |
|
||
|---|---:|---:|
|
||
| `total_bases` | 42 | **100%** |
|
||
| `hits` | 36 | **100%** |
|
||
| `home_runs` | 8 | **100%** |
|
||
| `strikeouts` | 5 | 40% |
|
||
| `outs` / `earned_runs` / `hits_allowed` | 3 each | 66.7% |
|
||
|
||
**Batters 100%, pitchers partial** — correct: pitchers accumulate few at-bats, so
|
||
the drift is genuinely undefined for many of them and **abstains** rather than
|
||
inventing a ratio.
|
||
|
||
---
|
||
|
||
## STEP 2 — 🟢 THE COLLINEARITY GUARD PASSES DECISIVELY
|
||
|
||
This was the gate that could have killed the axis before it was worth building.
|
||
|
||
| correlation | n | r |
|
||
|---|---:|---:|
|
||
| `drift` vs `l20_avg` | 94 | **−0.020** |
|
||
| `drift` vs `l5_avg` | 94 | **+0.027** |
|
||
| `drift` vs `ab_per_game` | 94 | **−0.029** |
|
||
|
||
**All three are essentially zero.** The drift ratio is **orthogonal to every
|
||
existing projection input** — it carries information the projection does not
|
||
already contain.
|
||
|
||
**This also validates the ratio-over-level decision empirically.** `ab_per_game`
|
||
(the level) is the same quantity divided by the same denominator as `l20_avg`;
|
||
had we used it, we would have added a redundant input. **Dividing by the player's
|
||
own baseline removed the collinearity** — r = −0.029 against the very quantity it
|
||
is built from.
|
||
|
||
---
|
||
|
||
## STEP 3 — THE AXIS IS LIVE AS A CHALLENGER
|
||
|
||
`opportunityNudge` composes in the same log-odds space as park and platoon
|
||
(`log` of a ratio), with **two guards the measured axes do not get**:
|
||
|
||
- **A ±10% deadband.** A rest day, a pinch-hit appearance or a blowout can move a
|
||
5-game window without any change in role.
|
||
- **A tighter cap — 0.15 vs the environment's 0.30.** This is a *proxy*, not a
|
||
measurement, and a noisy proxy must not outvote measured signals.
|
||
|
||
Every adjustment carries **`is_proxy: true`** and
|
||
**`proxy_for: 'confirmed_batting_order'`**, so nothing downstream can mistake it
|
||
for a lineup feed.
|
||
|
||
The axis can **stand alone** — without that, the challenger's early return would
|
||
have gated opportunity off on exactly the thin-classification rows it is most
|
||
likely to help.
|
||
|
||
**Zero extra I/O:** `analyzeViaEngine1` attaches drift from the feature vector it
|
||
has already built; `attachChallenger` reads it off the grade. Nothing re-fetches
|
||
in a loop running over hundreds of props.
|
||
|
||
### Verified writing on prod (induced snapshot, 416 grades)
|
||
|
||
| | |
|
||
|---|---:|
|
||
| rows with a challenger value | 275 / 276 |
|
||
| **rows carrying the opportunity axis** | **142 (51.4%)** |
|
||
| rows where the challenger moved | 190 |
|
||
| mean \|delta\| when it moved | **0.034** |
|
||
| delta range | −0.089 … +0.108 |
|
||
|
||
**Champion `p_win` unchanged. Live grade path unchanged.** The axis writes only
|
||
to `p_win_challenger` / `challenger_adjustments`.
|
||
|
||
---
|
||
|
||
## STEP 4 — ⏳ HOLDOUT PROOF IS n-BLOCKED BY CONSTRUCTION
|
||
|
||
```
|
||
settled MLB rows (all): 605
|
||
settled rows with a challenger value: 250
|
||
settled rows carrying the OPPORTUNITY axis: 0
|
||
first game_date carrying the axis: 2026-08-01 (tonight — not yet played)
|
||
```
|
||
|
||
**The axis went live today. Its first rows are on games that have not happened.**
|
||
No holdout can be run, and **I am not going to manufacture one** — running the
|
||
test on rows the axis never touched would dilute the comparison with rows where
|
||
challenger ≡ champion by construction, and **make a null result look like a small
|
||
positive one.**
|
||
|
||
**Earliest possible run:** tomorrow's settle pass. **Defensible n:** several days,
|
||
at ~140 axis-carrying rows per snapshot.
|
||
|
||
The query is written and committed (`scripts/opportunity-axis-holdout.sql`). It
|
||
filters to `challenger_adjustments LIKE '%opportunity%'` for exactly the reason
|
||
above, buckets before measuring reliability, and splits time-forward.
|
||
|
||
**BOTH reliability and resolution must improve. One or neither → SHELVE.**
|
||
|
||
---
|
||
|
||
## STEP 5 — NOT REACHED
|
||
|
||
Nothing promoted. Nothing flipped. The axis sits in the ledger accruing evidence.
|
||
|
||
## THE PROXY CAVEAT, CARRIED FORWARD
|
||
|
||
**This is a proxy for tonight's batting order, not a measurement of it.** The real
|
||
input is the confirmed lineup slot, which no wired source exposes
|
||
(`depthChartService` returns `battingOrder: null` for MLB; PropLine `/context`
|
||
carries only a `lineup_confirmed` boolean). **If a lineup feed becomes available,
|
||
this axis should be replaced by the real slot, not supplemented with it.**
|
||
|
||
## A MEASUREMENT TRAP WORTH RECORDING
|
||
|
||
The first prod coverage run reported **`opportunity_drift` at 0%** while
|
||
`ab_per_game` read 94% — which looks exactly like "the feature does not compute".
|
||
|
||
It was the **120-second feature-vector cache** (`VECTOR_TTL_SECONDS`): the probe
|
||
ran inside the window and read payloads written by the previous image, which had
|
||
no such field. Re-running past the TTL gave 94%.
|
||
|
||
**A new feature field is invisible for one cache generation after deploy.** Worth
|
||
knowing before concluding a feature is broken — I very nearly reported this one
|
||
as absent, having already confirmed against the live statsapi payload that
|
||
`atBats` was present and that the code produced `drift = 1.05` locally on that
|
||
exact data. **The contradiction between those two facts is what saved it.**
|