Files
vyndr/specs/resolution-ceiling-and-the-duel.md
builtbykev e872eff4ce Instrument the calibration duel forward; diagnose the resolution ceiling
— the proven factors were never wired in

PHASE 0 — two truths recorded. The swap is a BET, not an OOS win:
isotonic beat low-param on identical held-out rows (hits +0.0028, rbi
+0.0042, TB tied) and we serve low-param anyway on an untestable prior
about shared daily structure. At 19 dates nothing here can test it. And
the MIN_SLOPE catch is preserved as standing rationale: a near-zero or
negative slope collapses toward base-rate-for-everything, which LOWERS
Brier while destroying all resolution -- a metric win that guts the
product.

PHASE 1 — the duel is now falsifiable. Both corrections computed on every
hits/TB prop; p_win_lowparam served, p_win_isotonic_shadow logged in its
own try so it can never break serving. calibrationDuel.adjudicate encodes
the rule IN CODE before any forward date exists: >=10 forward dates and
isotonic winning with a date-block CI excluding zero => REFUTED, revert;
otherwise UPHELD; under 10 dates PENDING regardless of the numbers. A
date counts as forward only if NEITHER map was fitted on it -- otherwise
we would be scoring which map memorised better. Nothing swaps now.

PHASE 2 — the ceiling, quantified via Murphy decomposition:

  stat   reliability  RESOLUTION  uncertainty  variance explained
  hits      0.01353     0.00252      0.24532        1.03%
  TB        0.01419     0.00442      0.24329        1.82%
  rbi       0.00654     0.03268      0.22531       14.51%
  runs      0.00788     0.00130      0.23182        0.56%

Calibration did exactly what theory says and nothing more: hits
reliability 0.01353 -> 0.00233 (-0.0112, 83% of the error removed) while
resolution moved -0.0002. Unexpected: rbi has 13x the resolution of hits
and is the one stat we do NOT serve corrected -- it needs calibration
least and discriminates most.

PHASE 2 DIAGNOSIS — NOT-TRANSMITTED, and not weak, ABSENT. Traced in code:
sprayDefense.js and platoonSeverity.js are required by NOTHING in src/,
only by analysis scripts and their own tests. The served p_win
(intelligence/probabilityEstimator.js:54) reads exactly four inputs --
game-log frequency, opp_rank_stat +/-0.03, home_away +/-0.015, and a cv
pull -- with zero occurrences of spray, platoon, hard-hit or
contact-profile. And snapshotService grades at line 454 while computing
challenger/context at 640+, so everything proven is computed DOWNSTREAM of
the grade it would inform. The three proven hits factors have never once
moved a served number.

That reframes the recent nulls: "calibrated p_win does not separate within
archetype" was never a statement about factors. The factors were not in
the forecast.

PHASE 3 — bands rebuilt on SERVED values (hits/TB low-param, rbi/runs
raw): 28 archetype slots across four stats, ZERO show lift. No longer an
open shrug -- it is the arithmetic of resolution 0.0013-0.0327 against
uncertainty ~0.23. A forecast explaining 1% of variance cannot produce
separating bands, and no correction to its numbers will change that.

HEADLINE: calibration is complete, delivered honest numbers on two stats
and zero grade separation, because the counter has no resolution -- and
the proven factors are not wired into the forecast at all. The second is
the reason for the first, and it is plumbing rather than a modelling wall.
Per-archetype grades need proven factors that actually reach p_win. Last
calibration order.

Serving unchanged from 74cf1ce. p_win never mutated. No Bonferroni slot.
Counter and frozen clusters verified file-by-file (15 modules).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-07 02:25:45 -04:00

6.5 KiB
Raw Permalink Blame History

The resolution ceiling — calibration is complete, and it was never the lever

PHASE 0 — two honest truths, on record

1. The swap is a BET, not an OOS win

On identical held-out rows the isotonic map scored BETTER: hits +0.0028 (CI [0.0013, 0.0045]), rbi +0.0042 (CI [0.0003, 0.0092]), total_bases tied.

We serve the low-parameter map anyway, on the untestable prior that isotonic's in-window edge is daily structure shared between the fit and evaluation windows and will not transmit forward. At 19 dates no instrument here can test that prior — LODO has 1.49.3% power against it.

Named as a bet, logged, not evidence. Phase 1 makes it falsifiable.

2. The MIN_SLOPE catch, as a standing guard rationale

runs fitted a = 0.032. A near-zero or negative slope collapses the curve toward base-rate-for-everything, which lowers Brier — shrinking a miscalibrated forecaster toward its base rate always does — while destroying all resolution.

A metric win that guts the product. Any calibration layer must refuse a non-positive slope on principle, not on inspection. That is now MIN_SLOPE.


PHASE 1 — the duel, instrumented forward

Both corrections are computed on every hits/TB prop: p_win_lowparam (served) and p_win_isotonic_shadow (logged, never read by serving or by chainAcross). The shadow runs in its own try — it can never break serving.

calibrationDuel.adjudicate encodes the rule in code, before any forward date exists, so the bar cannot drift toward whichever answer arrives:

condition verdict action
≥10 forward dates AND isotonic wins, date-block CI excluding zero REFUTED revert hits/TB to isotonic, log the reversal
≥10 forward dates, isotonic does not win UPHELD keep serving low-param
<10 forward dates PENDING keep serving, no verdict

A date counts as forward only if neither map was fitted on it — scoring inside a fit window would ask which map memorised better. Rows lacking that provenance are dropped, never assumed forward. Tests lock all of it, including that a decisive shadow win at 5 dates is still PENDING.

Nothing swaps now. The season decides.


PHASE 2 — the resolution ceiling, quantified

Murphy decomposition: Brier = reliability resolution + uncertainty. Reliability is what calibration fixes. Resolution is discrimination, and a monotone map cannot create it — it relabels bins without re-sorting the rows inside them.

stat n base reliability resolution uncertainty share of variance explained
hits 1,140 0.5684 0.01353 0.00252 0.24532 1.03%
total_bases 1,050 0.5819 0.01419 0.00442 0.24329 1.82%
rbi 630 0.6571 0.00654 0.03268 0.22531 14.51%
runs 597 0.6348 0.00788 0.00130 0.23182 0.56%

What calibration did, exactly as theory predicts

stat reliability resolution
hits 0.01353 → 0.00233 (0.0112) 0.00252 → 0.00231 (0.0002)
total_bases 0.01419 → 0.00527 (0.0089) 0.00442 → 0.00414 (0.0003)

Calibration removed 83% of hits' reliability error and moved resolution by essentially nothing. It did the whole of its job, and its job was never the one the grade product needs.

Unexpected: rbi has 13× the resolution of hits and is the one stat we do not serve corrected — it needs calibration least (reliability 0.0065) and discriminates most. Worth carrying into the factor arc.


PHASE 2 — the HITS factor-transmission diagnosis

Verdict: NOT-TRANSMITTED. Not weak — absent. This is a plumbing defect, and it is the highest-value finding in the order.

Evidence, traced in code rather than recalled:

  1. sprayDefense.js and platoonSeverity.js are required by NOTHING in src/. Only by analysis scripts and their own tests. The two causally-correct atoms that passed the two-part gate have never been on the serving path.

  2. The served p_win reads exactly four inputs (intelligence/probabilityEstimator.js:54): game-log frequency over the line, opp_rank_stat (±0.03), home_away (±0.015), and a cv consistency pull. Zero occurrences of spray, platoon, hard-hit or contact-profile.

  3. Ordering makes it structural. snapshotService grades at line 454 (gradeAndCacheSlate) and only computes challenger/context at line 640+. Everything proven is computed downstream of the grade it would inform.

So the three proven hits factors — defense_by_direction, pitcher_contact_profile, platoon_severity — were measured on ledger rows by analysis scripts and have never once moved a served number.

That reframes every null in this programme's recent history. "Calibrated p_win does not separate within archetype" was never a statement about factors. The factors were not in the forecast.


PHASE 3 — bands on served values

stat basis eval rows slots slots with lift
hits p_win_lowparam (served) 765 7 0
total_bases p_win_lowparam (served) 625 7 0
rbi raw p_win 425 7 0
runs raw p_win 424 7 0

28 archetype slots across four stats. Zero show lift. Every slot is one band indistinguishable from its own base rate.

This is no longer an open shrug. It is the arithmetic consequence of resolution of 0.00130.0327 against uncertainty of ~0.23: a forecast explaining 1% of the outcome's variance cannot produce bands that separate, and no correction to its numbers will change that.


THE HEADLINE

Calibration is complete. It delivered honest numbers on two stats and ZERO grade separation, because the counter has no resolution — 1.03% of variance on hits, 0.56% on runs.

And the three proven hits factors are NOT WIRED INTO THE FORECAST AT ALL.

Those two facts together are the programme's position. The second is the reason for the first, and it is a plumbing defect rather than a modelling wall — which makes it the cheapest high-value fix available.

Per-archetype grades require proven factors that actually reach p_win. That is the next and central arc. This is the last calibration order.


Invariants

Serving unchanged from 74cf1ce — this order logs and diagnoses. p_win never mutated; p_win_lowparam served, p_win_isotonic_shadow logged. No Bonferroni slot. Counter and frozen clusters verified file-by-file.