vyndr/specs/lodo-instrument-audit.md

Auditing the LODO instrument — it cannot evaluate any stat

All four stats are UNTESTABLE-BY-LODO. Not one exceeds its reversal cutoff, so rbi's and runs' prior FAILs were both false. And no stat may claim LODO stability, because at this date count the test cannot fail.

---

PHASE 0 — the defect, on record

The gate at 1f40014 was mine, and it was an incoherent pair:

1. A 1-SE informativeness bar coupled to a ZERO-reversal rule. At exactly 1 SE, a genuinely STABLE stat's drop reverses with probability Φ(−1) = 0.1587. On four informative drops:

P(>= 1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50

The rule failed stable stats half the time by construction. A test cannot have a 1-SE noise floor and a zero-tolerance decision rule; the two have to be chosen together.

2. A pooled n*. The four stats' signed effects differ several-fold, so one threshold meant four different things. Measured, the pooled 70 was:

stat	own n*	pooled 70 was
hits	77	too low
total_bases	60	too high
rbi	54	too high
runs	81	too low

It mis-credited every stat, in both directions.

Neither defect touched the counter or p_win. Both touched only which calibrations were judged stable.

---

PHASE 1 — the coherent test, derived blind

(a)(b)(c) per-stat effect

stat	n	g (signed)	σ_row	SE_full	effect z	NO-EFFECT?
hits	1,140	−0.01288	0.11251	0.00333	3.87	no
total_bases	1,050	−0.01380	0.10680	0.00330	4.19	no
rbi	630	−0.00884	0.06459	0.00257	3.44	no
runs	597	−0.00902	0.08080	0.00331	2.73	no

All four have a real effect at full n. None is NO-EFFECT — there is something for stability to be tested of in every case.

The committed test pair

k = 1, chosen because informative drops (D) are the binding scarcity here and k=1 maximises them while the binomial cutoff holds the false-positive rate.

stat	n*=k²(σ/|g|)²	informative drops D	cutoff	FP	power at τ=|g|
hits	77	5	2	0.031	0.093
total_bases	60	5	2	0.031	0.093
rbi	54	4	2	0.014	0.045
runs	81	3	2	0.004	0.014

Per-drop noise probability under stability Φ(−1) = 0.1587. FAIL iff reversals > cutoff.

The finding that dominates everything else: the test has no power

Against a strong instability — date-to-date SD of the effect equal to the effect itself — this test detects a failure between 1.4% and 9.3% of the time. Across every k examined (1.0 → 2.0), the best any stat reaches is 0.337, and reaching even that costs all but two informative drops.

A gate that cannot fail cannot pass. LODO_POWER_FLOOR = 0.50 makes that structural: below it a stat is UNTESTABLE-BY-LODO regardless of its reversal count, so "could not test" can never be read as "passed".

Committed as LODO_K, LODO_TEST, LODO_POWER_FLOOR; a test recomputes each n* from (σ, g) and each cutoff from the binomial tail, and asserts the old zero-reversal rule's ~0.50 false-fail rate. The stale pooled constant is nulled so nothing can read it.

---

PHASE 2/3 — cold re-read at each stat's own n*

stat	own n*	informative	reversals	cutoff	power	verdict
hits	77	5	0	2	0.093	UNTESTABLE
total_bases	60	5	0	2	0.093	UNTESTABLE
rbi	54	4	1	2	0.045	UNTESTABLE
runs	81	3	2	2	0.014	UNTESTABLE

Setting the power floor aside entirely, not one stat exceeds its cutoff.

---

PHASE 4 — reconcile against 1f40014

stat	1f40014	now	why it changed
hits	PASS	UNTESTABLE	the PASS was from a test that cannot fail; 0 reversals is uninformative at 9% power
total_bases	PASS	UNTESTABLE	same — and it "passed" at a pooled n*=70 above its own 60, so its drop count was under-credited too
rbi	FAIL	UNTESTABLE (1 reversal, cutoff 2)	FALSE FAIL. One reversal on four drops is a ~16%-per-drop coin flip, not evidence
runs	FAIL	UNTESTABLE (2 reversals, cutoff 2)	ALSO A FALSE FAIL under the coherent rule — this was not anticipated

Answering the order's three questions directly:

• Is rbi's FAIL a false fail? Yes. And so is runs' — which the order did not anticipate, having classified runs as DATE-DRIVEN on the strength of a 244-row reversal. Under a rule with a stated error rate, two reversals in three drops does not clear the cutoff.
• Was TB's PASS real? No. It was vacuous: the test could not have failed it.
• Does hits still pass at its own smaller n*? Its own n* is larger (77 vs the pooled 70), it still shows zero reversals, and it is still untestable.

---

PHASE 5 — deploy, withdraw, route

stat	basis	decision
hits	date-clustered CI [−0.0139, −0.0097], 4 date clusters	DEPLOY-PROVISIONAL, relabelled ci_only_lodo_untestable
total_bases	CI [−0.0061, −0.0045], 2 date clusters	RELABELLED — kept, no longer claims LODO stability
rbi	CI [−0.0092, −0.0010], 2 date clusters	NEWLY DEPLOYED — its FAIL was false
runs	no fittable map at its point-in-time split	REFUSE — no CI to stand on either

Every deployed stat now carries calibration_basis: 'ci_only_lodo_untestable'. Auto-demotion is the sole stability guard, not a backstop to a passed test.

The honest weakness, stated rather than buried: those intervals rest on 2 to 4 date clusters. That is thin support, and it is now the only support.

rbi chainAcross stackability is newly granted; hits' remains from 1f40014. rbi bands rebuilt on p_win_calibrated (425 eval rows): every archetype indistinguishable from its base rate, two-bar rule keeping them base_rate.

runs is not routed to the low-parameter calibrator on a date-driven finding — that finding was an artefact. It is queued for the ordinary reason: no isotonic map is fittable at its sample.

---

PHASE 6 — logged

The deploy set at 1f40014 was set by a coin-flip-power ruler. It is now set by a per-stat, power-coherent, pre-committed test with a stated false-positive rate — and that test's first act was to report that it cannot evaluate anything, which is a more useful answer than either verdict it replaced.

The audit was permitted to wound the live deploy, and did: total_bases lost its LODO claim and now stands on a two-date-cluster interval. That it could is the integrity property.

Standing question unchanged. Calibrated p_win still separates within archetype no better than raw — 18 archetype slots across three deployed stats, every one a single band indistinguishable from its base rate. Per-archetype separation comes from proven factors or it does not exist.

---

Invariants

Blind ordering held: (g, σ_row) and the committed (n*, cutoff, power floor) were derived and locked with no reversal or verdict in view, before any stat was re-read. p_win never mutated. No Bonferroni slot consumed. Counter and frozen clusters verified file-by-file.