Files

T

builtbykev ea1157d709 report: grade fix Part 1 — the p_win-vs-fair_prob rebuild is REFUTED by the data

STOPPED at the Part 1 gate. Nothing rebuilt, no grade changed, no cutover.

THE FINDING: grading on p_win vs fair_prob does not work. All three candidate
edge formulations correlate NEGATIVELY with outcomes, on both sports, overall,
and in both time splits (n=432 decided rows carrying p_win AND fair_prob_lock):

ALL n=432 champ -0.0016 p_win ALONE +0.1221 additive -0.0615 ratio -0.1161 logodds -0.0438
MLB n=240 champ +0.0984 p_win ALONE +0.2278 additive -0.0336 ratio -0.1350 logodds -0.0124
WNBA n=192 champ -0.1143 p_win ALONE -0.0842 additive -0.1326 ratio -0.1281 logodds -0.1243

Subtracting the market's lock-time fair probability destroys and inverts the
signal. The plain reading: props where the model most disagrees with the market
are LESS likely to hit — the market is better than the model, so "edge vs market"
is anti-predictive here, while the raw probability retains some skill alone.

WHAT DOES CARRY SIGNAL: p_win alone, MLB only, and it is modest. Time-forward
split — TRAIN (07-21..07-26, n=120) r=0.2770; HOLDOUT (07-26..07-30, n=120)
r=0.1647, with the additive edge negative in BOTH halves. So p_win survives
forward validation directionally but the holdout is NOT significant (t~1.81,
p~0.07). Suggestive, not proven.

WNBA MUST ABSTAIN: every measure negative including p_win itself (-0.084). Forcing
one threshold across both sports would make a coin-flip sport look sharp, which the
order forbids.

LOOKAHEAD GUARD SATISFIED: fair_prob_lock is the lock-time field, populated on 432
decided rows, range 0.145-0.713. closing_prob (415 rows) is the CLOSE and was NOT
used in any correlation — using it would have manufactured a correlation.

SAMPLE REALITY: 1103 decided rows but only 432 carry both instrument fields, so a
per-sport train/holdout split leaves ~120 per half — enough to show direction, not
to certify a letter ladder.

I did not tune toward a win: three pre-registered candidates were tested and all
three failed; picking a fourth because the first three lost is the overfitting the
order guards against. Recommended instead: grade MLB on p_win alone with WNBA
abstaining and label it modest/accruing (A-RATED hold stays); or wait ~6 weeks for
n~500 MLB; or investigate WHY the market-relative edge inverts, which is the more
valuable question.

Both queries committed at scripts/grade-correlation-proof.sql so no number here
has to be taken on trust. Working settlement untouched; dead resolve endpoint not
wired.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc

2026-07-31 19:57:21 -04:00

5.6 KiB

Raw Blame History

THE GRADE FIX — PART 1 INVESTIGATION (report; STOPPED at the gate)

2026-07-31. Nothing rebuilt, no grade changed, no surface cut over.

🔴 HEADLINE: THE PROPOSED REBUILD IS REFUTED BY THE DATA

Grading on p_win-vs-fair_prob does NOT work. Every edge formulation correlates NEGATIVELY with outcomes — on both sports, overall, and in both time splits. The order said to verify rather than assume, and to STOP on a wrong premise. This is that case.

The evidence (n=432 decided rows carrying BOTH p_win and fair_prob_lock)

sport	n	champ letter r	p_win ALONE r	additive (p−f) r	ratio r	log-odds r
ALL	432	−0.0016	+0.1221	−0.0615	−0.1161	−0.0438
MLB	240	+0.0984	+0.2278	−0.0336	−0.1350	−0.0124
WNBA	192	−0.1143	−0.0842	−0.1326	−0.1281	−0.1243

All three candidate edge definitions (2a) are negative everywhere. Subtracting the market's lock-time fair probability does not sharpen the signal — it destroys it and inverts it.

The plain reading: the props where our model most disagrees with the market are less likely to hit. That is the market being better than the model. "Edge vs the market" is therefore anti-predictive here, while the raw probability estimate retains some skill on its own.

What DOES carry signal: `p_win` alone, MLB only — and it is modest

Time-forward split (overfitting guard — fit-free, just correlation on each half):

MLB split	dates	n	p_win alone r	additive edge r	hit%
TRAIN (older)	07-21 → 07-26	120	+0.2770	−0.0027	50.8
HOLDOUT (newer)	07-26 → 07-30	120	+0.1647	−0.0688	57.5

p_win alone survives forward validation directionally (positive in both halves). But the holdout is NOT statistically significant: r=0.165 at n=120 → t≈1.81, p≈0.07. Suggestive, not proven. The additive edge is negative in both halves.

WNBA must ABSTAIN (2c/2d). Every measure is negative, including p_win itself (−0.084). There is no honest grade to build there — forcing one threshold across both sports would make a coin-flip sport look sharp, which 2c explicitly forbids.

Sample-size reality (2c per-sport honesty)

1,103 decided rows exist, but only 442 carry p_win and 432 carry fair_prob_lock — the instrument fields only began populating recently. After a per-sport split that is MLB 240 / WNBA 192, and a train/holdout split leaves ~120 per half. Threshold fitting on 120 rows, validated on 120, is thin: it can show direction but cannot certify a letter ladder.

LOOKAHEAD GUARD — satisfied, and the field is named

ledger_entries.fair_prob_lock is the lock-time truth and is populated on 432 decided rows, range 0.145–0.713 (a probability, correct scale). p_win ranges 0.100–0.950 (the estimator's clamp). closing_prob (415 rows) is the CLOSE and was not used in any correlation above — using it would have manufactured a correlation out of lookahead.

Why the served letter is r≈0 (confirmed)

engine1.gradeProp sums flat ±1.0/±0.5 factor deltas onto a neutral 'C'; the file contains zero references to any probability. edge_pct — the price-free (proj−line)/line — is not an input to the letter either (engine1 has no edge reference); it drives the board sort and was the display figure until it was retired 07-31. So the letter is a factor index that never sees price or probability, which is why it lands at r≈0.005 and inverts (B 52.4% < C 56.9%).

Provenance of the r=0.236 figure

It is mine, measured this session on 354 settled rows (MLB 224) — the probability letter banded through grade_thresholds.json. Today's recomputation on the p_win/fair_prob_lock population gives MLB p_win r = 0.2278 (n=240), consistent.

Do not ship the specified rebuild. The defensible options:

Grade MLB on p_win alone; WNBA abstains. Honest, matches the evidence. But the holdout is p≈0.07 — so it ships as "modest, directional, accruing", not as a proven edge, and the A-RATED hold stays.
Wait for sample. The instrument fields are ~10 days old. At ~90 decided MLB rows/week, n≈500 MLB (a 250/250 split) is ~6 weeks out and would move p≈0.07 toward significance if the effect is real.
Investigate WHY the market-relative edge inverts. That negative correlation is itself information: it says the model's disagreement with the market is noise. That is the more valuable question than any threshold ladder.

What I will not do: tune a formulation until a positive number appears. Three pre-registered candidates were tested and all three failed; picking a fourth because the first three lost is exactly the overfitting the order guards against.

Part 4 loose ends (unchanged, tracked)

dclv 0 rows/'unknown' — its own follow-up. NBA/soccer never settle — a per-sport feed decision, and the dead /api/grading/resolve must NOT be wired (it would double-count the working settlement). Working settlement untouched by this order.

REPRODUCIBLE

Both queries are committed at scripts/grade-correlation-proof.sql — re-runnable, so no number here has to be taken on trust.

5.6 KiB Raw Blame History Unescape Escape