# LODO-gated provisional calibration — total_bases deploys, three stats refuse ## PHASE 0 — Record correction **The ≥40 date-cluster deploy floor applied to CALIBRATION was the wrong instrument, and I applied it without challenging the binding.** It is `factorGate`'s cluster-robust interval floor, built for a factor making a CAUSAL claim, where the risk is a false positive dressed as mechanism. A calibration layer is different in kind: - it makes **no causal claim** — it is a monotone shrink toward observed - its failure mode is **bounded**: it can only over- or under-shrink - it consumes **no Bonferroni slot** Its real risk is that the correction is **date-driven**, and leave-one-date-out tests that directly. The replacement bar is **stricter on stability**, not looser on standard: LODO fails a stat if removing any single day reverses the improvement, which a cluster count cannot detect at all. The prior order's date premise was also wrong (05-01→08-04, "~90 dates"); the snapshots span 07-19→08-06 = 19 dates. That was corrected in the settlement session. The mis-bound instrument is mine. **The ≥40 floor is retained, correctly scoped as the PROMOTION bar** — the point at which a stat leaves provisional status. --- ## PHASE 1 — Both guards codified (11 tests) `src/services/model/calibrationGuards.js` **GUARD 1 — the both-sides tell.** Picked-side dedup is now mandatory preprocessing, asserted. The guard fires on the CONJUNCTION of both sides being present AND mean p_win pinned near 0.5 — either alone is unremarkable, and flagging a genuinely balanced one-sided book would be a false alarm. Demonstrated on live data in this run: ``` raw population violated=true mean_p 0.4962 both_sides_share 0.9763 after dedup violated=false mean_p 0.6694 ``` **GUARD 2 — a null that scores itself.** `safeBrier` refuses when any prediction is null; `applyOrRefuse` drops unmappable rows rather than passing nulls downstream. A test demonstrates the trap explicitly — `(null−1)² === 1` and `(null−0)² === 0`, so a Brier over nulls silently equals the win rate. --- ## PHASE 2 — LODO table Refit dropping each date; measure held-out Brier delta and the sign of the >0.9 favourite bias. Drops with fewer than 20 held rows are marked UNINFORMATIVE rather than counted either way. **A note on what LODO is:** refitting on all-but-one date uses dates that follow the held-out one, so this is a STABILITY test, not a point-in-time backtest. The point-in-time result is separate and already established. Both are required. | stat | n | dates | informative drops | reversals | sign flips | LODO | |---|---|---|---|---|---|---| | hits | 1,140 | 17 | 7 | **2** (07-22 n=20, 07-26 n=25) | 0 | **FAIL** | | **total_bases** | 1,050 | 7 | 5 | 0 | 0 | **PASS** | | rbi | 630 | 5 | 5 | **1** (08-01 n=99) | 0 | **FAIL** | | runs | 597 | 5 | 5 | **2** (08-01 n=86, 08-05 n=244) | 0 | **FAIL** | ### Threshold sensitivity — reported because the verdict moves | min held rows | hits | total_bases | rbi | runs | |---|---|---|---|---| | **20** (applied) | FAIL | **PASS** | FAIL | FAIL | | 30 | PASS | **PASS** | FAIL | FAIL | | 50 / 75 | PASS | **PASS** | FAIL | FAIL | | 100 | PASS | **PASS** | PASS | FAIL | - **total_bases passes at every threshold** — the only unambiguous result. - **runs fails at every threshold**, reversing on a 244-row date. - **hits' failure is threshold-fragile**: it fails only when 20- and 25-row dates are admitted, and those are the two smallest informative drops in the set. I chose `MIN_HELD_ROWS = 20` before seeing which stats passed, and did not move it afterwards to preserve a deploy. The honest caveat: a per-date Brier delta on 20 rows has a standard error several times the effect being tested, so the LODO instrument is underpowered per-drop at this sample size. That argues for pre-registering a higher threshold — a Roundtable decision, not one to make while holding the results. --- ## PHASE 3 — Deploy decisions | stat | LODO | point-in-time CI | decision | |---|---|---|---| | **total_bases** | PASS | [−0.0061, −0.0045] | **DEPLOY-PROVISIONAL** | | hits | FAIL | [−0.0139, −0.0097] | REFUSE — improvement reverses on 07-22 / 07-26 | | rbi | FAIL | [−0.0092, −0.0010] | REFUSE — improvement reverses on 08-01 | | runs | FAIL | no fittable map at the point-in-time split | REFUSE — honest null | Certified band for total_bases: **[0.6–0.8]**. Outside it → refuse, fall to base rate. ### hits was being served calibrated, and is not any more `snapshotService` hardcoded hits calibration since S91. hits fails LODO, so it has been removed from the deployed set. **A stat that cannot survive dropping one day was never calibrated — it was fitted to that day.** The consequence is real: hits props become unstackable again for `chain.chainAcross`. That is the honest result of measuring it, not a regression to route around, and it errs toward withdrawing a claim rather than preserving one on a fragile verdict. Deployment is now driven by `CALIBRATION_DEPLOYED` (frozen, tested), not a hardcoded stat name. --- ## PHASE 4 — Auto-demotion (14 tests) `src/services/model/calibrationRegistry.js` - **Deploy needs BOTH gates** — LODO pass AND a point-in-time CI excluding zero. Neither is waivable. - **`reverify` demotes on the first breach**: the CI ceasing to exclude zero, or the favourite over-prediction flipping sign (which would mean the correction is now pushing the wrong way). The breaking date is logged. - **Promotion to non-provisional** requires the original ≥40 date-cluster bar, with the interval still holding. A provisional deploy that cannot be taken away is just a deploy; `reverify` is what makes the label mean something. --- ## PHASE 5 — Bands rebuilt on p_win_calibrated (total_bases only) 625 eval rows on calibrated values. **The two-bar rule still bites**: TB is now CALIBRATED but no factor is PROVEN for it (barrel, exit velo and hard-contact-allowed were all THEATER), so bands remain a base-rate read — now an honestly-numbered one. | archetype | n | base rate | bands | separation | |---|---|---|---|---| | UNLABELLED | 275 | 0.6255 | 1 | indistinguishable from base rate | | BOMBER | 200 | 0.6100 | 1 | indistinguishable | | GHOST | 87 | 0.5747 | 1 | indistinguishable | | DRIVER | 24 | 0.7917 | 1 (PROVISIONAL) | indistinguishable | | BRUSH | 19 | 0.4737 | 1 (PROVISIONAL) | indistinguishable | | MIRROR | 6 | — | REFUSED | insufficient outcomes | Calibration compressed the served range to 0.4286–1.0. Every archetype still collapses to a single band — calibrated p_win does not separate within archetype any better than raw p_win did. Refused stats keep base-rate bands on raw p_win. --- ## PHASE 6 — Logged, not acted on **The dead gradient is buried.** Over-prediction ordering on the fuller settled set is **hits ≈ TB > runs > RBI**, not TB > RBI > runs. The skill-driven-gradient mechanism did not survive — **RBI has the SMALLEST bias** (+0.0164). Descriptive only; no mechanism claimed. **Refusal coverage.** Refused props are predictable-but-input-less rather than genuinely uncertain (3.20 vs 3.39 AB rules out playing time). The refused set is a MAP OF MISSING INPUTS and feeds the input-coverage roadmap. Not this order. **Queued candidate:** a low-parameter calibrator (Platt / beta) fits a favourite-longshot shape on far fewer points than isotonic needs, which is exactly the constraint that refused runs. It is a NEW estimator requiring its own out-of-sample validation. Not built here. **Programme-level finding:** calibration beats every factor tried on TB / RBI / runs, and the defect is systematic over-prediction **concentrated in favourites** (+0.21 to +0.28 above p_win 0.9 on all four stats) rather than a uniform shift. --- ## Invariants `p_win` never mutated — calibration rides as `p_win_calibrated` with `calibration_status: 'provisional'`. No Bonferroni slot consumed; testLedger factor count untouched. Counter and frozen clusters byte-identical.