# Auditing the LODO instrument — it cannot evaluate any stat **All four stats are UNTESTABLE-BY-LODO. Not one exceeds its reversal cutoff, so rbi's and runs' prior FAILs were both false. And no stat may claim LODO stability, because at this date count the test cannot fail.** --- ## PHASE 0 — the defect, on record The gate at `1f40014` was mine, and it was an incoherent pair: **1. A 1-SE informativeness bar coupled to a ZERO-reversal rule.** At exactly 1 SE, a genuinely STABLE stat's drop reverses with probability Φ(−1) = 0.1587. On four informative drops: ``` P(>= 1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50 ``` **The rule failed stable stats half the time by construction.** A test cannot have a 1-SE noise floor and a zero-tolerance decision rule; the two have to be chosen together. **2. A pooled n\*.** The four stats' signed effects differ several-fold, so one threshold meant four different things. Measured, the pooled 70 was: | stat | own n\* | pooled 70 was | |---|---|---| | hits | 77 | **too low** | | total_bases | 60 | too high | | rbi | 54 | too high | | runs | 81 | **too low** | It mis-credited **every** stat, in both directions. Neither defect touched the counter or `p_win`. Both touched only which calibrations were judged stable. --- ## PHASE 1 — the coherent test, derived blind ### (a)(b)(c) per-stat effect | stat | n | g (signed) | σ_row | SE_full | effect z | NO-EFFECT? | |---|---|---|---|---|---|---| | hits | 1,140 | −0.01288 | 0.11251 | 0.00333 | 3.87 | no | | total_bases | 1,050 | −0.01380 | 0.10680 | 0.00330 | 4.19 | no | | rbi | 630 | −0.00884 | 0.06459 | 0.00257 | 3.44 | no | | runs | 597 | −0.00902 | 0.08080 | 0.00331 | 2.73 | no | All four have a real effect at full n. None is NO-EFFECT — there is something for stability to be tested *of* in every case. ### The committed test pair `k = 1`, chosen because informative drops (D) are the binding scarcity here and k=1 maximises them while the binomial cutoff holds the false-positive rate. | stat | n\*=k²(σ/\|g\|)² | informative drops D | cutoff | FP | **power at τ=\|g\|** | |---|---|---|---|---|---| | hits | 77 | 5 | 2 | 0.031 | **0.093** | | total_bases | 60 | 5 | 2 | 0.031 | **0.093** | | rbi | 54 | 4 | 2 | 0.014 | **0.045** | | runs | 81 | 3 | 2 | 0.004 | **0.014** | Per-drop noise probability under stability Φ(−1) = 0.1587. FAIL iff reversals > cutoff. ### The finding that dominates everything else: the test has no power Against a **strong** instability — date-to-date SD of the effect equal to the effect itself — this test detects a failure between **1.4% and 9.3%** of the time. Across every k examined (1.0 → 2.0), the best any stat reaches is 0.337, and reaching even that costs all but two informative drops. **A gate that cannot fail cannot pass.** `LODO_POWER_FLOOR = 0.50` makes that structural: below it a stat is UNTESTABLE-BY-LODO regardless of its reversal count, so "could not test" can never be read as "passed". Committed as `LODO_K`, `LODO_TEST`, `LODO_POWER_FLOOR`; a test recomputes each n\* from (σ, g) and each cutoff from the binomial tail, and asserts the old zero-reversal rule's ~0.50 false-fail rate. The stale pooled constant is nulled so nothing can read it. --- ## PHASE 2/3 — cold re-read at each stat's own n\* | stat | own n\* | informative | reversals | cutoff | power | verdict | |---|---|---|---|---|---|---| | hits | 77 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** | | total_bases | 60 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** | | rbi | 54 | 4 | 1 | 2 | 0.045 | **UNTESTABLE** | | runs | 81 | 3 | 2 | 2 | 0.014 | **UNTESTABLE** | **Setting the power floor aside entirely, not one stat exceeds its cutoff.** --- ## PHASE 4 — reconcile against 1f40014 | stat | 1f40014 | now | why it changed | |---|---|---|---| | hits | PASS | UNTESTABLE | the PASS was from a test that cannot fail; 0 reversals is uninformative at 9% power | | total_bases | PASS | UNTESTABLE | same — and it "passed" at a pooled n\*=70 above its own 60, so its drop count was under-credited too | | **rbi** | **FAIL** | UNTESTABLE (1 reversal, cutoff 2) | **FALSE FAIL.** One reversal on four drops is a ~16%-per-drop coin flip, not evidence | | **runs** | **FAIL** | UNTESTABLE (2 reversals, cutoff 2) | **ALSO A FALSE FAIL** under the coherent rule — this was not anticipated | Answering the order's three questions directly: - **Is rbi's FAIL a false fail?** Yes. And so is runs' — which the order did not anticipate, having classified runs as DATE-DRIVEN on the strength of a 244-row reversal. Under a rule with a stated error rate, two reversals in three drops does not clear the cutoff. - **Was TB's PASS real?** No. It was vacuous: the test could not have failed it. - **Does hits still pass at its own smaller n\*?** Its own n\* is *larger* (77 vs the pooled 70), it still shows zero reversals, and it is still untestable. --- ## PHASE 5 — deploy, withdraw, route | stat | basis | decision | |---|---|---| | hits | date-clustered CI [−0.0139, −0.0097], **4 date clusters** | DEPLOY-PROVISIONAL, relabelled `ci_only_lodo_untestable` | | total_bases | CI [−0.0061, −0.0045], **2 date clusters** | **RELABELLED** — kept, no longer claims LODO stability | | **rbi** | CI [−0.0092, −0.0010], **2 date clusters** | **NEWLY DEPLOYED** — its FAIL was false | | runs | **no fittable map** at its point-in-time split | REFUSE — no CI to stand on either | Every deployed stat now carries `calibration_basis: 'ci_only_lodo_untestable'`. **Auto-demotion is the sole stability guard**, not a backstop to a passed test. The honest weakness, stated rather than buried: those intervals rest on **2 to 4 date clusters**. That is thin support, and it is now the *only* support. **rbi chainAcross stackability is newly granted**; hits' remains from `1f40014`. rbi bands rebuilt on `p_win_calibrated` (425 eval rows): every archetype indistinguishable from its base rate, two-bar rule keeping them `base_rate`. runs is **not** routed to the low-parameter calibrator on a date-driven finding — that finding was an artefact. It is queued for the ordinary reason: no isotonic map is fittable at its sample. --- ## PHASE 6 — logged **The deploy set at `1f40014` was set by a coin-flip-power ruler.** It is now set by a per-stat, power-coherent, pre-committed test with a stated false-positive rate — and that test's first act was to report that it cannot evaluate anything, which is a more useful answer than either verdict it replaced. **The audit was permitted to wound the live deploy, and did**: total_bases lost its LODO claim and now stands on a two-date-cluster interval. That it could is the integrity property. **Standing question unchanged.** Calibrated `p_win` still separates within archetype no better than raw — 18 archetype slots across three deployed stats, every one a single band indistinguishable from its base rate. Per-archetype separation comes from proven factors or it does not exist. --- ## Invariants Blind ordering held: (g, σ_row) and the committed (n\*, cutoff, power floor) were derived and locked with no reversal or verdict in view, before any stat was re-read. `p_win` never mutated. No Bonferroni slot consumed. Counter and frozen clusters verified file-by-file.