builtbykev
e29ab6fd6a
PART 1 verified by inducing the REAL rowsFromSnapshot over REAL lock_lines
rows from prod. Three cases, 0 non-takeable anchors:
Narvaez (dabble/kalshi/prizepicks/smarkets, NO takeable book)
-> book=null, price=null, takeable=null [honest absent]
Schwarber(bovada/dabble/novig/PINNACLE before draftkings)
-> draftkings +102 [pinnacle SKIPPED, proving TAKEABLE not MODEL]
Ohtani (dabble/onexbet before draftkings) -> draftkings -266
Narvaez is the case that matters: pre-fix he was stamped dabble +104
takeable=true; he is now honestly absent.
A HARNESS BUG RECORDED: my first verification pulled live /api/odds/mlb,
which returned {"error":"Odds data temporarily unavailable"}. The script
read that as 0 props and printed "all from takeable books? true" -- a
VACUOUSLY TRUE pass. I caught it only because I also printed the book list
and it was empty. Same family as the silent-false traps: a probe that finds
nothing looks identical to a probe that finds nothing wrong.
PART 2: 1,006 rows tagged via the purpose-built quarantine_reason at ROW
level with three sub-cases (recoverable_same_line 936, no_takeable_quote
49, takeable_line_differs 21). getModelAggregate ALREADY excluded
quarantined rows, so the public record and the n>=20 gate were clean
automatically; all five committed holdout scripts now carry the exclusion
explicitly.
PART 3 -- the re-stamp call is now fact-based. The takeable LOCK-TIME price
is recoverable for 936/1,006 (93.0%) from lock_lines, the correct
instrument. Only 431 appear in closing_captures, which is the wrong timing
for a lock price anyway.
LINE CONTAMINATION ANSWERED (previously unverified): the stored line
MATCHES a takeable book's line on 936 (93.0%), DIFFERS on 21 (2.1%), and is
unverifiable on 49 (4.9%) where no takeable book quoted the prop at all.
That makes it cleanly row-level: re-stamp the 936 as an honest JOIN and
recover 886 pending rows for the holdouts, or leave all 1,006 excluded.
Either way the 21 + 49 stay out -- re-stamping those would invent a lock
price, or a line, we never captured. Nothing re-stamped; Kev's call.
Gates: 4,111 tests / 330 suites green; next build exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
85 lines
4.4 KiB
SQL
85 lines
4.4 KiB
SQL
-- ruler-comparison.sql — Order: MLB CALIBRATION RE-RUN vs CONSENSUS RULER (2026-08-01)
|
|
-- MEASURE-ONLY. Run against prod (Supabase MCP execute_sql).
|
|
--
|
|
-- WHAT THIS DOES AND DOES NOT ANSWER
|
|
--
|
|
-- It does NOT re-fit the p_win calibration. It cannot: `estimateProbability`
|
|
-- takes {gameLogs, line, statType, features} and never sees a market price, and
|
|
-- the calibration query fits p_win against OUTCOMES. Reliability and resolution
|
|
-- are both p_win-vs-outcome measures, so the ruler cannot enter either. See
|
|
-- pwin-timeforward.sql for the (ruler-independent) calibration refresh.
|
|
--
|
|
-- What IS ruler-dependent is EDGE (p_win - fair_prob). This measures whether
|
|
-- swapping the incumbent single-book ruler for a median consensus rescues it.
|
|
--
|
|
-- TIMING IS HELD CONSTANT: both rulers are read at CLOSE. The lock-time
|
|
-- reconstruction is impossible at usable n (only 43 settled rows join
|
|
-- lock_lines with >=2 two-sided books), and mixing a lock-time incumbent with
|
|
-- a close-time consensus would confound WHEN with WHAT.
|
|
--
|
|
-- LIMITATION, load-bearing: closing_captures contains ONLY MODEL books
|
|
-- (draftkings/betmgm/betrivers/fanduel/pinnacle). Exchange quotes were never
|
|
-- stored, because normalizeProps discarded them until 2026-08-01. So this can
|
|
-- only test a US-books-median ruler, NOT the exchange-inclusive consensus. The
|
|
-- exchange ruler is untestable on existing data at any n.
|
|
|
|
--
|
|
-- CONTAMINATION EXCLUSION (2026-08-02, MANDATORY). Rows whose price/book/takeable
|
|
-- were stamped from a NON-TAKEABLE book (DFS / offshore / exchange) between
|
|
-- 2026-08-01 and the write-path fix are tagged `quarantine_reason LIKE
|
|
-- 'nontakeable_book%'`. They are EXCLUDED here and must never be pooled with
|
|
-- clean rows: their locked price -- and therefore the `takeable` flag computed
|
|
-- from it -- describes a market you could not have bet.
|
|
|
|
with imp as (
|
|
select id, player_key, stat, game_date, line, side, p_win, book, (outcome='hit')::int won
|
|
from public.ledger_entries
|
|
where sport='mlb' and user_id is null
|
|
and (quarantine_reason is null or quarantine_reason not like 'nontakeable_book%') and outcome in ('hit','miss') and p_win is not null),
|
|
|
|
-- latest CLOSE capture per (prop, book); two-sided only -- a one-sided quote
|
|
-- cannot be de-vigged, so it cannot price a ruler.
|
|
cap as (
|
|
select distinct on (player_key,stat,game_date,line,book)
|
|
player_key,stat,game_date,line,book,over_odds,under_odds
|
|
from public.closing_captures
|
|
where sport='mlb' and over_odds is not null and under_odds is not null
|
|
order by player_key,stat,game_date,line,book,captured_at desc),
|
|
|
|
d as (select *,
|
|
case when over_odds>0 then 100.0/(over_odds+100) else (-over_odds)/((-over_odds)+100.0) end po,
|
|
case when under_odds>0 then 100.0/(under_odds+100) else (-under_odds)/((-under_odds)+100.0) end pu
|
|
from cap),
|
|
|
|
-- multiplicative two-way de-vig, per book (matches src/utils/devig.js)
|
|
f as (select player_key,stat,game_date,line,book, po/(po+pu) fo, pu/(po+pu) fu
|
|
from d where po+pu > 0),
|
|
|
|
j as (select i.*, f.book ref_book,
|
|
case when lower(i.side)='under' then f.fu else f.fo end fair_side
|
|
from imp i join f
|
|
on f.player_key=i.player_key and f.stat=i.stat
|
|
and f.game_date=i.game_date and f.line=i.line),
|
|
|
|
a as (select id, p_win, won, game_date,
|
|
count(*) n_books,
|
|
percentile_cont(0.5) within group (order by fair_side) cons, -- v2: MEDIAN
|
|
max(case when ref_book=book then fair_side end) own -- v1: the locked book
|
|
from j group by 1,2,3,4),
|
|
|
|
e as (select *, p_win-own edge_v1, p_win-cons edge_v2
|
|
from a where n_books>=2 and own is not null)
|
|
|
|
select count(*) n,
|
|
round(avg(won)::numeric,4) base_rate,
|
|
round(avg(abs(cons-own))::numeric,4) mean_abs_ruler_gap,
|
|
round(avg(cons-own)::numeric,4) mean_signed_ruler_gap,
|
|
round(corr(edge_v1, won::numeric)::numeric,4) corr_edge_v1_won,
|
|
round(corr(edge_v2, won::numeric)::numeric,4) corr_edge_v2_won,
|
|
round(corr(p_win, won::numeric)::numeric,4) corr_pwin_won,
|
|
round(avg(edge_v1) filter (where won=1)::numeric,4) edge_v1_winners,
|
|
round(avg(edge_v1) filter (where won=0)::numeric,4) edge_v1_losers,
|
|
round(avg(edge_v2) filter (where won=1)::numeric,4) edge_v2_winners,
|
|
round(avg(edge_v2) filter (where won=0)::numeric,4) edge_v2_losers
|
|
from e;
|