Files

T

builtbykev 7140e62b65 MLB re-run vs consensus ruler: premise dissolved, isotonic DECIDED

MEASURE-ONLY. No promotion, no flip, no tier spend. Live path
byte-identical: CURRENT_RULER_VERSION still v1_first_book, model still
consumes MODEL_BOOKS only.

MANDATE 1'S PREMISE DOES NOT HOLD. The p_win calibration is
RULER-INDEPENDENT, confirmed two ways: estimateProbability takes
{gameLogs, line, statType, features} and never sees a market price, and
the calibration fits p_win against OUTCOMES. Reliability and resolution
are both p_win-vs-outcome measures, so fair_prob cannot enter either.
There is nothing to re-fit -- the ruler changes edge, CLV and takeable,
not calibration.

I RETRACT MY OWN LABEL. I declared the MLB isotonic result PROVISIONAL
"because it was measured against the bent ruler". That over-applied the
ruler caveat to a measurement the ruler never touched. The result was
never contaminated; it moves PROVISIONAL -> DECIDED, not by re-running but
because the gate I attached does not apply.

RAN THE GENUINELY RULER-DEPENDENT QUESTION INSTEAD -- does a median
consensus rescue EDGE? Timing held constant (both rulers at close; a
lock-time reconstruction joins only 43 rows, and mixing lock-incumbent
with close-consensus would confound WHEN with WHAT).

n=200 MLB settled rows: mean |ruler gap| 0.0085. corr(edge_v1, outcome)
-0.0101; corr(edge_v2, outcome) -0.0220; corr(p_win, outcome) +0.2598.

THE HEADLINE: p_win predicts outcomes at +0.26 while p_win minus the
market predicts nothing under EITHER ruler. Subtracting the market price
destroys the signal -- a direct empirical vindication of the identity now
at the top of CLAUDE.md. Market edge is not merely a poor criterion here;
it is a strictly worse instrument than the raw forecast.

CALIBRATION REFRESH (ruler-independent, but n grew 119 -> 250):
time-forward holdout n=125, reliability 0.0846 (was 0.0939), resolution
0.190 (was 0.123). Both hold and both improved on a fresh later window
the earlier fit never saw. Independent replication.

THE LIMITATION THAT BLOCKS A FULL VERDICT: closing_captures holds only
MODEL books -- exchange quotes were never stored, because normalizeProps
discarded them until yesterday. Mean 1.97 books in the historical join. So
this tested a US-books-median ruler, not the exchange-inclusive consensus
whose live delta showed p90 +10 points. That ruler is UNTESTABLE on
existing data at any n. Per Mandate 4's third outcome: inconclusive, not
forced.

SEPARATE FINDING -- LIVE FEED REGRESSION: pinnacle MLB captures went 4,022
-> 0 on 2026-07-31 and have not returned, while every other book continued
(103,940 captures in the prior 10 days). This also corrects an Order Zero
claim of mine: "no sharp anchor exists in our feed" was accurate for the
day measured but wrong generally -- pinnacle was there until 07-30 with
17,090 two-sided captures. line_type='sharp' is a label in closingCapture
via SHARP_BOOKS, not a separate provider. We had a sharp anchor and lost
it two days ago; not caused by anything in this session.

Both queries committed: scripts/ruler-comparison.sql,
scripts/pwin-timeforward.sql.

Gates: 4,028 tests / 322 suites green; next build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJs13VsyiSKYQP6rj3NNmc

2026-08-01 01:12:47 -04:00

8.0 KiB

Raw Blame History

MLB CALIBRATION RE-RUN vs THE CONSENSUS RULER

Date: 2026-08-01 · MEASURE-ONLY · no tier spend · live path byte-identical · queries committed (scripts/ruler-comparison.sql, scripts/pwin-timeforward.sql).

VERDICT UP FRONT

Mandate 1's premise does not hold, and I have to say so before reporting numbers against it.

The p_win calibration is RULER-INDEPENDENT. Re-deriving fair_prob against the consensus ruler cannot change a single number in the isotonic result.

Two independent confirmations:

estimateProbability({ gameLogs, line, statType, features }) (src/services/intelligence/probabilityEstimator.js:54) — no market price, no book odds, no fair_prob anywhere in its inputs. p_win is computed from game logs against the line.
The calibration query fits p_win against OUTCOMES. Reliability (predicted vs actual hit rate) and resolution (does higher p_win hit more often) are both p_win-vs-outcome measures. fair_prob cannot enter either.

So there is nothing to re-fit. The ruler changes edge (p_win − fair_prob), CLV, and takeable — not calibration.

I have to retract my own label

I declared the MLB isotonic result PROVISIONAL "because it was measured against the bent ruler." That was wrong — I over-applied the ruler caveat to a measurement the ruler never touched. The isotonic result was never contaminated.

It moves from PROVISIONAL → DECIDED, not by re-running, but because the gate I attached to it does not apply. It stands on its own terms.

WHAT I RAN INSTEAD (the genuinely ruler-dependent question)

Does swapping the incumbent single-book ruler for a median consensus rescue EDGE? That question is ruler-dependent, and it is the one that came back negative in three formulations.

Timing held constant — both rulers read at CLOSE. A lock-time reconstruction is impossible at usable n (only 43 settled rows join lock_lines with ≥2 two-sided books), and pairing a lock-time incumbent with a close-time consensus would confound when with what.

Result — MLB, n=200 settled rows, 2026-07-21 → 07-30

measure	value
base rate	0.525
mean \|ruler gap\|	0.0085 (0.85 prob points)
mean signed ruler gap	−0.0004
corr(edge v1 single-book, outcome)	−0.0101
corr(edge v2 consensus, outcome)	−0.0220
corr(`p_win` alone, outcome)	+0.2598
edge v1: winners / losers	0.0281 / 0.0315
edge v2: winners / losers	0.0265 / 0.0339

The consensus ruler does not rescue edge. Both rulers give a correlation indistinguishable from zero, and both are very slightly negative — higher edge associates marginally with losing, under either ruler.

The number that actually matters

p_win alone correlates +0.26 with outcome. Edge — p_win minus the market — correlates −0.01 to −0.02.

Subtracting the market price destroys the signal. The model has genuine forecasting signal; the market-relative transformation removes it. That is a direct empirical vindication of the identity now at the top of CLAUDE.md: market edge is a byproduct, never the success criterion. Here it is not merely a poor criterion — it is a strictly worse instrument than the raw forecast.

CALIBRATION REFRESH (ruler-independent, but n has grown)

Worth running for a different reason: the prior report used n=119 holdout. MLB now has 250 settled rows with p_win. Time-forward, earlier fits, later proves.

split	n	buckets	reliability (mean \|dev\|)	resolution (corr)	base rate	window
train	125	9	0.0951	0.2557	0.528	07-21 → 07-26
holdout	125	9	0.0846	0.1902	0.560	07-26 → 07-30

Both hold on a fresh, later window — and both improved versus the prior report (reliability 0.0939 → 0.0846; resolution 0.123 → 0.190). Resolution degrades train → holdout (0.256 → 0.190), which is expected and honest; it stays clearly positive rather than crushing to base rate.

This is independent replication on data the earlier fit never saw. It is the strongest evidence to date that MLB p_win is honest and still ranks.

THE LIMITATION THAT PREVENTS A FULL VERDICT

The ruler I could test is not the ruler that was built.

closing_captures contains only MODEL books — draftkings, betmgm, betrivers, fanduel, pinnacle. Exchange quotes were never stored, because normalizeProps discarded them until 2026-08-01. Mean books per prop in the historical join: 1.97.

So this measured a US-books-median vs one-US-book ruler change — which is small by construction (0.85 prob points), and nothing like the exchange-inclusive consensus, where the live delta showed p90 +10 points and 17% of props moving ≥5 points.

The exchange-inclusive consensus ruler is UNTESTED and UNTESTABLE on existing data at any n. It becomes testable only as v2-era captures accumulate — which starts now, because the display widening went live yesterday.

I am not forcing a verdict on it. Per Mandate 4's third outcome: inconclusive on holdout n.

VERDICT, PER MANDATE 4

question	verdict
Does the consensus ruler improve the calibration?	Question does not apply — calibration is ruler-independent (proven in code and in the query).
Is the MLB isotonic result still provisional?	No — DECIDED. My PROVISIONAL label was over-applied and is retracted.
Does the consensus ruler rescue EDGE (US-books version, n=200)?	DIFFERS BUT NOT BETTER. −0.010 → −0.022. Rulers differ ≠ new ruler better.
Does the exchange-inclusive ruler rescue edge?	INCONCLUSIVE — n-blocked. Untestable on existing data; the quotes were never stored.
MLB `p_win` honest and ranking, held out?	YES, replicated on fresh data. reliability 0.0846, resolution 0.190, n=125.
WNBA	Abstains, unchanged. Not re-litigated.

Nothing is promoted. Nothing is flipped. CURRENT_RULER_VERSION remains v1_first_book; the model still consumes MODEL_BOOKS only.

No promotion is scoped, because the trigger condition — "new ruler is better" — was not met on the testable portion and could not be evaluated on the rest.

🔴 SEPARATE FINDING — A LIVE FEED REGRESSION

While establishing feasibility I found this, and it is unrelated to the order:

date	pinnacle MLB captures	other books
07-26	15,008	115,638
07-29	10,406	82,080
07-30	4,022	63,602
07-31	0	47,606
08-01	0	45,092

Pinnacle prop coverage went to zero on 2026-07-31 and has not returned, while every other book continued normally. 103,940 captures over the preceding 10 days, then a clean cliff.

This also corrects an Order Zero claim of mine. I reported "pinnacle 0% — no sharp anchor exists in our feed." That was accurate for the day I measured but wrong as a general statement: Pinnacle was in the MLB props feed until 07-30, with 17,090 two-sided captures. (line_type='sharp' is a label applied in closingCapture.js via SHARP_BOOKS, not a separate provider — I checked.)

We had a sharp anchor and lost it two days ago. Whether that is a PropLine change, a regional restriction, or a bug is unknown and worth one question to the provider. It is not caused by anything in this session's work — the widening only adds books.

8.0 KiB Raw Blame History Unescape Escape