builtbykev
48a2f764ac
STEP 1 -- input mapped and measured. opportunity_drift 94% coverage on 100 real props: 100% for batters (total_bases, hits, home_runs), 40-67% for pitchers, which is correct -- pitchers accumulate few at-bats so the ratio is genuinely undefined and ABSTAINS rather than being invented. STEP 2 -- THE COLLINEARITY GUARD PASSES DECISIVELY. Pearson r on n=94: drift vs l20_avg -0.020, vs l5_avg +0.027, vs ab_per_game -0.029. All essentially zero, so the axis is orthogonal to every existing projection input and carries information the projection does not already contain. That also validates the ratio-over-level decision EMPIRICALLY: ab_per_game is the same quantity over the same denominator as l20_avg, so the level would have been redundant. Dividing by the player's own baseline removed the collinearity -- r = -0.029 against the very quantity it is built from. STEP 3 -- live as a challenger, verified on prod over an induced 416-grade snapshot: 142 of 276 rows (51.4%) carry the opportunity axis, the challenger moved on 190 rows, mean |delta| 0.034, range -0.089..+0.108. Champion p_win and the live grade path are unchanged. STEP 4 -- HOLDOUT IS n-BLOCKED BY CONSTRUCTION and I am not manufacturing one. Settled rows carrying the axis: 0. Its first rows carry game_date 2026-08-01 -- games that have not been played. Running the test on rows the axis never touched would dilute the comparison with rows where challenger === champion by construction, making a null result look like a small positive one. Query committed for when n arrives; it filters to axis-carrying rows for exactly that reason, buckets before measuring reliability, and splits time-forward. BOTH metrics must improve or the axis is shelved. A MEASUREMENT TRAP RECORDED: the first prod run showed drift at 0% while ab_per_game read 94% -- indistinguishable from "the feature does not compute". It was the 120-second feature-vector cache serving payloads written by the previous image. A new feature field is invisible for one cache generation after deploy. I nearly reported it absent, having already confirmed atBats is present in the live statsapi payload and that the code produced drift = 1.05 locally on that exact data; the contradiction between those two facts is what saved it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
50 lines
2.3 KiB
SQL
50 lines
2.3 KiB
SQL
-- opportunity-axis-holdout.sql — the Step 4 proof, RUN WHEN n IS ADEQUATE.
|
|
--
|
|
-- Cannot run yet, by construction: the axis went live 2026-08-01 and its first
|
|
-- rows carry game_date 2026-08-01 (games not yet played). Settled rows carrying
|
|
-- the axis: 0. The earliest possible run is the next morning settle pass, and a
|
|
-- defensible n is several days out at ~140 opportunity-axis rows per snapshot.
|
|
--
|
|
-- BOTH reliability AND resolution must improve for the axis to promote. One or
|
|
-- neither => SHELVE and record why.
|
|
--
|
|
-- reliability = n-weighted mean |predicted - actual| across deciles. Bucket
|
|
-- FIRST: mean|p - outcome| on 0/1 rows is noise-dominated individual error, not
|
|
-- calibration.
|
|
|
|
with rows_ as (
|
|
select game_date, id,
|
|
p_win::numeric champ,
|
|
p_win_challenger::numeric chal,
|
|
(outcome='hit')::int won
|
|
from public.ledger_entries
|
|
where sport='mlb' and user_id is null
|
|
and outcome in ('hit','miss')
|
|
and p_win is not null and p_win_challenger is not null
|
|
-- ONLY rows the opportunity axis actually touched. Including untouched rows
|
|
-- would dilute the comparison with rows where challenger === champion by
|
|
-- construction, and make a null result look like a small positive one.
|
|
and challenger_adjustments::text like '%opportunity%'
|
|
),
|
|
split as (
|
|
select *, case when ntile(2) over (order by game_date, id) = 1 then 'train' else 'holdout' end split
|
|
from rows_
|
|
),
|
|
b_champ as (
|
|
select split, width_bucket(champ, 0.0, 1.0, 10) bkt, count(*) n, avg(champ) pred, avg(won::numeric) actual
|
|
from split group by 1,2),
|
|
b_chal as (
|
|
select split, width_bucket(chal, 0.0, 1.0, 10) bkt, count(*) n, avg(chal) pred, avg(won::numeric) actual
|
|
from split group by 1,2)
|
|
select
|
|
s.split,
|
|
count(*) n,
|
|
round((select sum(n*abs(pred-actual))/nullif(sum(n),0) from b_champ c where c.split=s.split),4) reliability_champion,
|
|
round((select sum(n*abs(pred-actual))/nullif(sum(n),0) from b_chal c where c.split=s.split),4) reliability_challenger,
|
|
round(corr(s.champ, s.won::numeric)::numeric,4) resolution_champion,
|
|
round(corr(s.chal, s.won::numeric)::numeric,4) resolution_challenger,
|
|
round(avg(s.won::numeric),4) base_rate
|
|
from split s
|
|
group by s.split
|
|
order by s.split desc;
|