builtbykev
6d87d7a33c
Measure-only. p_win not flipped live, no grade rebuilt, no calibrator deployed. Per the doctrine, MLB and WNBA were fitted, selected and judged as SEPARATE models — and they reach opposite verdicts. No global instrument was fitted. METHOD: time-forward split per sport (earlier fits, later proves). Both instruments fitted on TRAIN only — single-parameter Platt and isotonic-with-pooling. Inputs p_win + outcome only; no market field, no closing value, no lookahead. Nothing about edge/CLV/beat-the-close enters any pass/fail line. MEASUREMENT CORRECTION made mid-run: the first pass reported mean|p - outcome| (~0.46-0.51), which is NOT calibration — it is noise-dominated individual error on 0/1 rows and would have made every instrument look identical. Reliability is only meaningful on BUCKETS (bucket mean predicted vs bucket actual rate, n-weighted), the metric T0 used. All reported numbers use the corrected metric. HOLDOUT RELIABILITY (lower better): MLB n=119/4 buckets — raw 0.1038, Platt 0.1120, ISOTONIC 0.0939. WNBA n=93/3 buckets — raw 0.1322, Platt 0.0491, isotonic 0.0667. HOLDOUT RESOLUTION: MLB raw 0.1388 -> Platt 0.1284 -> isotonic 0.1225. WNBA raw -0.1201 -> Platt +0.1269 -> isotonic +0.0322. Fitted Platt: MLB a=-0.381 b=+0.705; WNBA a=+0.040 b=-0.081. MLB QUALIFIES, MODESTLY — instrument selected BY HOLDOUT, not assumed: isotonic beats both raw and Platt, and Platt actually made MLB worse. Reliability improves 0.1038 -> 0.0939 (~10% relative, real but modest) and resolution SURVIVES (0.1388 -> 0.1225, not crushed). Both Mandate-3 conditions hold. WNBA ABSTAINS — its Platt result is the best number in the report and is REJECTED as a fake win. The fitted slope is b = -0.081, negative and near zero, so sigmoid(0.040 - 0.081*logit p) is nearly constant at ~0.51 for every input: it "calibrates" by discarding the prediction and emitting the base rate, which is exactly the failure Mandate 3 pre-registered. Its apparent resolution gain (-0.120 -> +0.127) is the sign flip, not skill — it would serve the opposite of its own forecast, fitted on n~96 of anti-signal. Isotonic says the same quietly (resolution collapses to +0.032). HONEST CEILING: MLB is a usable-but-unimpressive forecaster (holdout resolution ~0.12, reliability ~0.094, n=119); WNBA has no honest forecast today. Holdout n and bucket counts (4 and 3) suffice to reject WNBA and prefer isotonic for MLB, NOT to certify a letter ladder, and the T0 pathology is reduced rather than cured. CANNOT DETERMINE: per-archetype calibration (Mandate 3d) — bucket n falls below the reporting floor once split by sport AND archetype on 442 rows. Queries committed at scripts/pwin-calibration-holdout.sql. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc
35 lines
1.8 KiB
SQL
35 lines
1.8 KiB
SQL
-- p_win RECALIBRATION — per-sport, time-forward holdout. Measure-only.
|
|
-- specs/pwin-recalibration-holdout.md. NOTE: reliability is measured on BUCKETS
|
|
-- (predicted vs actual rate, n-weighted). mean|p-outcome| on 0/1 rows is NOT
|
|
-- calibration — it is noise-dominated individual error.
|
|
with r as (
|
|
select sport, game_date, p_win::numeric p, (outcome='hit')::int won,
|
|
ntile(2) over (partition by sport order by game_date) half
|
|
from ledger_entries
|
|
where user_id is null and outcome in ('hit','miss') and p_win is not null
|
|
), tagged as (
|
|
select *, case when half=1 then 'train' else 'holdout' end split,
|
|
width_bucket(p,0.1,1.0,5) bkt, ln(p/(1-p)) lg from r
|
|
), tb as ( -- TRAIN buckets = the fit
|
|
select sport, bkt, count(*) n, avg(lg) mean_lg, avg(won::numeric) rate
|
|
from tagged where split='train' group by sport, bkt having count(*) >= 8
|
|
), platt as (
|
|
select sport,
|
|
regr_slope(ln(greatest(least(rate,.98),.02)/(1-greatest(least(rate,.98),.02))), mean_lg) b,
|
|
regr_intercept(ln(greatest(least(rate,.98),.02)/(1-greatest(least(rate,.98),.02))), mean_lg) a
|
|
from tb group by sport
|
|
), hb as (
|
|
select h.sport, h.bkt, count(*) n, avg(h.p) pred_raw,
|
|
avg(1/(1+exp(-(pl.a+pl.b*h.lg)))) pred_platt,
|
|
avg(coalesce(tb.rate,h.p)) pred_iso, avg(h.won::numeric) actual
|
|
from tagged h join platt pl on pl.sport=h.sport
|
|
left join tb on tb.sport=h.sport and tb.bkt=h.bkt
|
|
where h.split='holdout' group by h.sport,h.bkt having count(*) >= 8
|
|
)
|
|
select sport, sum(n) holdout_n, count(*) buckets,
|
|
sum(n*abs(pred_raw-actual))/sum(n) raw_reliability_dev,
|
|
sum(n*abs(pred_platt-actual))/sum(n) platt_reliability_dev,
|
|
sum(n*abs(pred_iso-actual))/sum(n) isotonic_reliability_dev
|
|
from hb group by sport order by sport;
|
|
-- Resolution (ordering survives?) is the corr(pred, won) variant of the same CTEs.
|