builtbykev
ced40421ed
FAILs were false
PHASE 0 — the gate at 1f40014 was mine and was an incoherent pair. A 1-SE
informativeness bar with a ZERO-reversal rule: at exactly 1 SE a stable
stat's drop reverses with prob Phi(-1)=0.1587, so on four informative
drops P(>=1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50. It failed
stable stats half the time by construction. And the pooled n*=70
mis-credited EVERY stat -- too low for hits (own 77) and runs (81), too
high for total_bases (60) and rbi (54).
PHASE 1, blind. Per-stat (g, sigma_row): hits -0.01288/0.11251, TB
-0.01380/0.10680, rbi -0.00884/0.06459, runs -0.00902/0.08080. All four
clear z=1.96 at full n, so none is NO-EFFECT. Committed k=1 with per-stat
n* and a binomial cutoff holding FP at 0.004-0.031.
THE FINDING THAT DOMINATES: the test has no power. Against a strong
instability (date-to-date SD equal to the effect) it detects a failure
1.4%-9.3% of the time, and across every k from 1.0 to 2.0 the best any
stat reaches is 0.337. A gate that cannot fail cannot pass, so
LODO_POWER_FLOOR=0.50 makes UNTESTABLE structural -- "could not test" can
never read as "passed".
PHASE 2/3 cold, at each stat's OWN n*:
hits 5 informative, 0 reversals, cutoff 2, power 0.093 UNTESTABLE
TB 5 informative, 0 reversals, cutoff 2, power 0.093 UNTESTABLE
rbi 4 informative, 1 reversal, cutoff 2, power 0.045 UNTESTABLE
runs 3 informative, 2 reversals, cutoff 2, power 0.014 UNTESTABLE
Setting the power floor aside entirely, NOT ONE STAT EXCEEDS ITS CUTOFF.
PHASE 4 — rbi's FAIL was false, as the order suspected. So was RUNS' --
which the order did not anticipate, having classified it DATE-DRIVEN on a
244-row reversal; two reversals in three drops does not clear a cutoff of
2. TB's PASS was vacuous: the test could not have failed it. hits' own n*
is LARGER than the pooled one (77 vs 70), and it remains untestable.
PHASE 5 — deploy basis is now the date-clustered CI alone:
hits CI [-0.0139,-0.0097], 4 date clusters relabelled ci_only
TB CI [-0.0061,-0.0045], 2 date clusters RELABELLED, kept
rbi CI [-0.0092,-0.0010], 2 date clusters NEWLY DEPLOYED
runs no fittable map at its split REFUSE, no CI either
Every deployed stat carries calibration_basis ci_only_lodo_untestable and
auto-demotion is the SOLE stability guard, not a backstop to a passed
test. Stated plainly: those intervals rest on 2-4 date clusters, which is
thin, and it is now the only support. rbi gains chainAcross stackability;
its bands rebuilt on p_win_calibrated (425 rows) are every-archetype
base_rate. runs is queued for the low-param calibrator for the ordinary
reason -- no fittable map -- not on the date-driven finding, which was an
artefact.
PHASE 6 — the deploy set was set by a coin-flip-power ruler; it is now set
by a per-stat power-coherent pre-committed test whose first act was to
report that it cannot evaluate anything. The audit was permitted to wound
the live deploy and did: total_bases lost its LODO claim. Standing
question unchanged -- 18 archetype slots across three deployed stats, every
one a single band indistinguishable from base rate.
Blind ordering held. p_win never mutated. No Bonferroni slot. Counter and
frozen clusters verified file-by-file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
171 lines
7.1 KiB
Markdown
171 lines
7.1 KiB
Markdown
# Auditing the LODO instrument — it cannot evaluate any stat
|
||
|
||
**All four stats are UNTESTABLE-BY-LODO. Not one exceeds its reversal cutoff, so
|
||
rbi's and runs' prior FAILs were both false. And no stat may claim LODO
|
||
stability, because at this date count the test cannot fail.**
|
||
|
||
---
|
||
|
||
## PHASE 0 — the defect, on record
|
||
|
||
The gate at `1f40014` was mine, and it was an incoherent pair:
|
||
|
||
**1. A 1-SE informativeness bar coupled to a ZERO-reversal rule.** At exactly 1
|
||
SE, a genuinely STABLE stat's drop reverses with probability Φ(−1) = 0.1587. On
|
||
four informative drops:
|
||
|
||
```
|
||
P(>= 1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50
|
||
```
|
||
|
||
**The rule failed stable stats half the time by construction.** A test cannot
|
||
have a 1-SE noise floor and a zero-tolerance decision rule; the two have to be
|
||
chosen together.
|
||
|
||
**2. A pooled n\*.** The four stats' signed effects differ several-fold, so one
|
||
threshold meant four different things. Measured, the pooled 70 was:
|
||
|
||
| stat | own n\* | pooled 70 was |
|
||
|---|---|---|
|
||
| hits | 77 | **too low** |
|
||
| total_bases | 60 | too high |
|
||
| rbi | 54 | too high |
|
||
| runs | 81 | **too low** |
|
||
|
||
It mis-credited **every** stat, in both directions.
|
||
|
||
Neither defect touched the counter or `p_win`. Both touched only which
|
||
calibrations were judged stable.
|
||
|
||
---
|
||
|
||
## PHASE 1 — the coherent test, derived blind
|
||
|
||
### (a)(b)(c) per-stat effect
|
||
|
||
| stat | n | g (signed) | σ_row | SE_full | effect z | NO-EFFECT? |
|
||
|---|---|---|---|---|---|---|
|
||
| hits | 1,140 | −0.01288 | 0.11251 | 0.00333 | 3.87 | no |
|
||
| total_bases | 1,050 | −0.01380 | 0.10680 | 0.00330 | 4.19 | no |
|
||
| rbi | 630 | −0.00884 | 0.06459 | 0.00257 | 3.44 | no |
|
||
| runs | 597 | −0.00902 | 0.08080 | 0.00331 | 2.73 | no |
|
||
|
||
All four have a real effect at full n. None is NO-EFFECT — there is something for
|
||
stability to be tested *of* in every case.
|
||
|
||
### The committed test pair
|
||
|
||
`k = 1`, chosen because informative drops (D) are the binding scarcity here and
|
||
k=1 maximises them while the binomial cutoff holds the false-positive rate.
|
||
|
||
| stat | n\*=k²(σ/\|g\|)² | informative drops D | cutoff | FP | **power at τ=\|g\|** |
|
||
|---|---|---|---|---|---|
|
||
| hits | 77 | 5 | 2 | 0.031 | **0.093** |
|
||
| total_bases | 60 | 5 | 2 | 0.031 | **0.093** |
|
||
| rbi | 54 | 4 | 2 | 0.014 | **0.045** |
|
||
| runs | 81 | 3 | 2 | 0.004 | **0.014** |
|
||
|
||
Per-drop noise probability under stability Φ(−1) = 0.1587. FAIL iff reversals > cutoff.
|
||
|
||
### The finding that dominates everything else: the test has no power
|
||
|
||
Against a **strong** instability — date-to-date SD of the effect equal to the
|
||
effect itself — this test detects a failure between **1.4% and 9.3%** of the
|
||
time. Across every k examined (1.0 → 2.0), the best any stat reaches is 0.337,
|
||
and reaching even that costs all but two informative drops.
|
||
|
||
**A gate that cannot fail cannot pass.** `LODO_POWER_FLOOR = 0.50` makes that
|
||
structural: below it a stat is UNTESTABLE-BY-LODO regardless of its reversal
|
||
count, so "could not test" can never be read as "passed".
|
||
|
||
Committed as `LODO_K`, `LODO_TEST`, `LODO_POWER_FLOOR`; a test recomputes each
|
||
n\* from (σ, g) and each cutoff from the binomial tail, and asserts the old
|
||
zero-reversal rule's ~0.50 false-fail rate. The stale pooled constant is nulled
|
||
so nothing can read it.
|
||
|
||
---
|
||
|
||
## PHASE 2/3 — cold re-read at each stat's own n\*
|
||
|
||
| stat | own n\* | informative | reversals | cutoff | power | verdict |
|
||
|---|---|---|---|---|---|---|
|
||
| hits | 77 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** |
|
||
| total_bases | 60 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** |
|
||
| rbi | 54 | 4 | 1 | 2 | 0.045 | **UNTESTABLE** |
|
||
| runs | 81 | 3 | 2 | 2 | 0.014 | **UNTESTABLE** |
|
||
|
||
**Setting the power floor aside entirely, not one stat exceeds its cutoff.**
|
||
|
||
---
|
||
|
||
## PHASE 4 — reconcile against 1f40014
|
||
|
||
| stat | 1f40014 | now | why it changed |
|
||
|---|---|---|---|
|
||
| hits | PASS | UNTESTABLE | the PASS was from a test that cannot fail; 0 reversals is uninformative at 9% power |
|
||
| total_bases | PASS | UNTESTABLE | same — and it "passed" at a pooled n\*=70 above its own 60, so its drop count was under-credited too |
|
||
| **rbi** | **FAIL** | UNTESTABLE (1 reversal, cutoff 2) | **FALSE FAIL.** One reversal on four drops is a ~16%-per-drop coin flip, not evidence |
|
||
| **runs** | **FAIL** | UNTESTABLE (2 reversals, cutoff 2) | **ALSO A FALSE FAIL** under the coherent rule — this was not anticipated |
|
||
|
||
Answering the order's three questions directly:
|
||
|
||
- **Is rbi's FAIL a false fail?** Yes. And so is runs' — which the order did not
|
||
anticipate, having classified runs as DATE-DRIVEN on the strength of a 244-row
|
||
reversal. Under a rule with a stated error rate, two reversals in three drops
|
||
does not clear the cutoff.
|
||
- **Was TB's PASS real?** No. It was vacuous: the test could not have failed it.
|
||
- **Does hits still pass at its own smaller n\*?** Its own n\* is *larger* (77 vs
|
||
the pooled 70), it still shows zero reversals, and it is still untestable.
|
||
|
||
---
|
||
|
||
## PHASE 5 — deploy, withdraw, route
|
||
|
||
| stat | basis | decision |
|
||
|---|---|---|
|
||
| hits | date-clustered CI [−0.0139, −0.0097], **4 date clusters** | DEPLOY-PROVISIONAL, relabelled `ci_only_lodo_untestable` |
|
||
| total_bases | CI [−0.0061, −0.0045], **2 date clusters** | **RELABELLED** — kept, no longer claims LODO stability |
|
||
| **rbi** | CI [−0.0092, −0.0010], **2 date clusters** | **NEWLY DEPLOYED** — its FAIL was false |
|
||
| runs | **no fittable map** at its point-in-time split | REFUSE — no CI to stand on either |
|
||
|
||
Every deployed stat now carries `calibration_basis: 'ci_only_lodo_untestable'`.
|
||
**Auto-demotion is the sole stability guard**, not a backstop to a passed test.
|
||
|
||
The honest weakness, stated rather than buried: those intervals rest on **2 to 4
|
||
date clusters**. That is thin support, and it is now the *only* support.
|
||
|
||
**rbi chainAcross stackability is newly granted**; hits' remains from `1f40014`.
|
||
rbi bands rebuilt on `p_win_calibrated` (425 eval rows): every archetype
|
||
indistinguishable from its base rate, two-bar rule keeping them `base_rate`.
|
||
|
||
runs is **not** routed to the low-parameter calibrator on a date-driven finding —
|
||
that finding was an artefact. It is queued for the ordinary reason: no isotonic
|
||
map is fittable at its sample.
|
||
|
||
---
|
||
|
||
## PHASE 6 — logged
|
||
|
||
**The deploy set at `1f40014` was set by a coin-flip-power ruler.** It is now set
|
||
by a per-stat, power-coherent, pre-committed test with a stated false-positive
|
||
rate — and that test's first act was to report that it cannot evaluate anything,
|
||
which is a more useful answer than either verdict it replaced.
|
||
|
||
**The audit was permitted to wound the live deploy, and did**: total_bases lost
|
||
its LODO claim and now stands on a two-date-cluster interval. That it could is
|
||
the integrity property.
|
||
|
||
**Standing question unchanged.** Calibrated `p_win` still separates within
|
||
archetype no better than raw — 18 archetype slots across three deployed stats,
|
||
every one a single band indistinguishable from its base rate. Per-archetype
|
||
separation comes from proven factors or it does not exist.
|
||
|
||
---
|
||
|
||
## Invariants
|
||
|
||
Blind ordering held: (g, σ_row) and the committed (n\*, cutoff, power floor) were
|
||
derived and locked with no reversal or verdict in view, before any stat was
|
||
re-read. `p_win` never mutated. No Bonferroni slot consumed. Counter and frozen
|
||
clusters verified file-by-file.
|