vyndr/specs/lodo-provisional-calibration.md

# LODO-gated provisional calibration — total_bases deploys, three stats refuse

## PHASE 0 — Record correction

**The ≥40 date-cluster deploy floor applied to CALIBRATION was the wrong
instrument, and I applied it without challenging the binding.**

It is `factorGate`'s cluster-robust interval floor, built for a factor making a
CAUSAL claim, where the risk is a false positive dressed as mechanism. A
calibration layer is different in kind:

- it makes **no causal claim** — it is a monotone shrink toward observed
- its failure mode is **bounded**: it can only over- or under-shrink
- it consumes **no Bonferroni slot**

Its real risk is that the correction is **date-driven**, and leave-one-date-out
tests that directly. The replacement bar is **stricter on stability**, not looser
on standard: LODO fails a stat if removing any single day reverses the
improvement, which a cluster count cannot detect at all.

The prior order's date premise was also wrong (05-01→08-04, "~90 dates"); the
snapshots span 07-19→08-06 = 19 dates. That was corrected in the settlement
session. The mis-bound instrument is mine.

**The ≥40 floor is retained, correctly scoped as the PROMOTION bar** — the point
at which a stat leaves provisional status.

---

## PHASE 1 — Both guards codified (11 tests)

`src/services/model/calibrationGuards.js`

**GUARD 1 — the both-sides tell.** Picked-side dedup is now mandatory
preprocessing, asserted. The guard fires on the CONJUNCTION of both sides being
present AND mean p_win pinned near 0.5 — either alone is unremarkable, and
flagging a genuinely balanced one-sided book would be a false alarm.

Demonstrated on live data in this run:

```
raw population    violated=true   mean_p 0.4962   both_sides_share 0.9763
after dedup       violated=false  mean_p 0.6694
```

**GUARD 2 — a null that scores itself.** `safeBrier` refuses when any prediction
is null; `applyOrRefuse` drops unmappable rows rather than passing nulls
downstream. A test demonstrates the trap explicitly — `(null−1)² === 1` and
`(null−0)² === 0`, so a Brier over nulls silently equals the win rate.

---

## PHASE 2 — LODO table

Refit dropping each date; measure held-out Brier delta and the sign of the >0.9
favourite bias. Drops with fewer than 20 held rows are marked UNINFORMATIVE
rather than counted either way.

**A note on what LODO is:** refitting on all-but-one date uses dates that follow
the held-out one, so this is a STABILITY test, not a point-in-time backtest. The
point-in-time result is separate and already established. Both are required.

| stat | n | dates | informative drops | reversals | sign flips | LODO |
|---|---|---|---|---|---|---|
| hits | 1,140 | 17 | 7 | **2** (07-22 n=20, 07-26 n=25) | 0 | **FAIL** |
| **total_bases** | 1,050 | 7 | 5 | 0 | 0 | **PASS** |
| rbi | 630 | 5 | 5 | **1** (08-01 n=99) | 0 | **FAIL** |
| runs | 597 | 5 | 5 | **2** (08-01 n=86, 08-05 n=244) | 0 | **FAIL** |

### Threshold sensitivity — reported because the verdict moves

| min held rows | hits | total_bases | rbi | runs |
|---|---|---|---|---|
| **20** (applied) | FAIL | **PASS** | FAIL | FAIL |
| 30 | PASS | **PASS** | FAIL | FAIL |
| 50 / 75 | PASS | **PASS** | FAIL | FAIL |
| 100 | PASS | **PASS** | PASS | FAIL |

- **total_bases passes at every threshold** — the only unambiguous result.
- **runs fails at every threshold**, reversing on a 244-row date.
- **hits' failure is threshold-fragile**: it fails only when 20- and 25-row dates
  are admitted, and those are the two smallest informative drops in the set.

I chose `MIN_HELD_ROWS = 20` before seeing which stats passed, and did not move
it afterwards to preserve a deploy. The honest caveat: a per-date Brier delta on
20 rows has a standard error several times the effect being tested, so the LODO
instrument is underpowered per-drop at this sample size. That argues for
pre-registering a higher threshold — a Roundtable decision, not one to make while
holding the results.

---

## PHASE 3 — Deploy decisions

| stat | LODO | point-in-time CI | decision |
|---|---|---|---|
| **total_bases** | PASS | [−0.0061, −0.0045] | **DEPLOY-PROVISIONAL** |
| hits | FAIL | [−0.0139, −0.0097] | REFUSE — improvement reverses on 07-22 / 07-26 |
| rbi | FAIL | [−0.0092, −0.0010] | REFUSE — improvement reverses on 08-01 |
| runs | FAIL | no fittable map at the point-in-time split | REFUSE — honest null |

Certified band for total_bases: **[0.6–0.8]**. Outside it → refuse, fall to base
rate.

### hits was being served calibrated, and is not any more

`snapshotService` hardcoded hits calibration since S91. hits fails LODO, so it
has been removed from the deployed set. **A stat that cannot survive dropping one
day was never calibrated — it was fitted to that day.** The consequence is real:
hits props become unstackable again for `chain.chainAcross`. That is the honest
result of measuring it, not a regression to route around, and it errs toward
withdrawing a claim rather than preserving one on a fragile verdict.

Deployment is now driven by `CALIBRATION_DEPLOYED` (frozen, tested), not a
hardcoded stat name.

---

## PHASE 4 — Auto-demotion (14 tests)

`src/services/model/calibrationRegistry.js`

- **Deploy needs BOTH gates** — LODO pass AND a point-in-time CI excluding zero.
  Neither is waivable.
- **`reverify` demotes on the first breach**: the CI ceasing to exclude zero, or
  the favourite over-prediction flipping sign (which would mean the correction is
  now pushing the wrong way). The breaking date is logged.
- **Promotion to non-provisional** requires the original ≥40 date-cluster bar,
  with the interval still holding.

A provisional deploy that cannot be taken away is just a deploy; `reverify` is
what makes the label mean something.

---

## PHASE 5 — Bands rebuilt on p_win_calibrated (total_bases only)

625 eval rows on calibrated values. **The two-bar rule still bites**: TB is now
CALIBRATED but no factor is PROVEN for it (barrel, exit velo and
hard-contact-allowed were all THEATER), so bands remain a base-rate read — now an
honestly-numbered one.

| archetype | n | base rate | bands | separation |
|---|---|---|---|---|
| UNLABELLED | 275 | 0.6255 | 1 | indistinguishable from base rate |
| BOMBER | 200 | 0.6100 | 1 | indistinguishable |
| GHOST | 87 | 0.5747 | 1 | indistinguishable |
| DRIVER | 24 | 0.7917 | 1 (PROVISIONAL) | indistinguishable |
| BRUSH | 19 | 0.4737 | 1 (PROVISIONAL) | indistinguishable |
| MIRROR | 6 | — | REFUSED | insufficient outcomes |

Calibration compressed the served range to 0.4286–1.0. Every archetype still
collapses to a single band — calibrated p_win does not separate within archetype
any better than raw p_win did. Refused stats keep base-rate bands on raw p_win.

---

## PHASE 6 — Logged, not acted on

**The dead gradient is buried.** Over-prediction ordering on the fuller settled
set is **hits ≈ TB > runs > RBI**, not TB > RBI > runs. The skill-driven-gradient
mechanism did not survive — **RBI has the SMALLEST bias** (+0.0164). Descriptive
only; no mechanism claimed.

**Refusal coverage.** Refused props are predictable-but-input-less rather than
genuinely uncertain (3.20 vs 3.39 AB rules out playing time). The refused set is
a MAP OF MISSING INPUTS and feeds the input-coverage roadmap. Not this order.

**Queued candidate:** a low-parameter calibrator (Platt / beta) fits a
favourite-longshot shape on far fewer points than isotonic needs, which is
exactly the constraint that refused runs. It is a NEW estimator requiring its own
out-of-sample validation. Not built here.

**Programme-level finding:** calibration beats every factor tried on TB / RBI /
runs, and the defect is systematic over-prediction **concentrated in favourites**
(+0.21 to +0.28 above p_win 0.9 on all four stats) rather than a uniform shift.

---

## Invariants

`p_win` never mutated — calibration rides as `p_win_calibrated` with
`calibration_status: 'provisional'`. No Bonferroni slot consumed; testLedger
factor count untouched. Counter and frozen clusters byte-identical.