builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Audit the LODO instrument: it cannot evaluate any stat, and both prior

Browse Source 
FAILs were false

PHASE 0 — the gate at 1f40014 was mine and was an incoherent pair. A 1-SE
informativeness bar with a ZERO-reversal rule: at exactly 1 SE a stable
stat's drop reverses with prob Phi(-1)=0.1587, so on four informative
drops P(>=1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50. It failed
stable stats half the time by construction. And the pooled n*=70
mis-credited EVERY stat -- too low for hits (own 77) and runs (81), too
high for total_bases (60) and rbi (54).

PHASE 1, blind. Per-stat (g, sigma_row): hits -0.01288/0.11251, TB
-0.01380/0.10680, rbi -0.00884/0.06459, runs -0.00902/0.08080. All four
clear z=1.96 at full n, so none is NO-EFFECT. Committed k=1 with per-stat
n* and a binomial cutoff holding FP at 0.004-0.031.

THE FINDING THAT DOMINATES: the test has no power. Against a strong
instability (date-to-date SD equal to the effect) it detects a failure
1.4%-9.3% of the time, and across every k from 1.0 to 2.0 the best any
stat reaches is 0.337. A gate that cannot fail cannot pass, so
LODO_POWER_FLOOR=0.50 makes UNTESTABLE structural -- "could not test" can
never read as "passed".

PHASE 2/3 cold, at each stat's OWN n*:

  hits  5 informative, 0 reversals, cutoff 2, power 0.093  UNTESTABLE
  TB    5 informative, 0 reversals, cutoff 2, power 0.093  UNTESTABLE
  rbi   4 informative, 1 reversal,  cutoff 2, power 0.045  UNTESTABLE
  runs  3 informative, 2 reversals, cutoff 2, power 0.014  UNTESTABLE

Setting the power floor aside entirely, NOT ONE STAT EXCEEDS ITS CUTOFF.

PHASE 4 — rbi's FAIL was false, as the order suspected. So was RUNS' --
which the order did not anticipate, having classified it DATE-DRIVEN on a
244-row reversal; two reversals in three drops does not clear a cutoff of
2. TB's PASS was vacuous: the test could not have failed it. hits' own n*
is LARGER than the pooled one (77 vs 70), and it remains untestable.

PHASE 5 — deploy basis is now the date-clustered CI alone:

  hits  CI [-0.0139,-0.0097], 4 date clusters   relabelled ci_only
  TB    CI [-0.0061,-0.0045], 2 date clusters   RELABELLED, kept
  rbi   CI [-0.0092,-0.0010], 2 date clusters   NEWLY DEPLOYED
  runs  no fittable map at its split            REFUSE, no CI either

Every deployed stat carries calibration_basis ci_only_lodo_untestable and
auto-demotion is the SOLE stability guard, not a backstop to a passed
test. Stated plainly: those intervals rest on 2-4 date clusters, which is
thin, and it is now the only support. rbi gains chainAcross stackability;
its bands rebuilt on p_win_calibrated (425 rows) are every-archetype
base_rate. runs is queued for the low-param calibrator for the ordinary
reason -- no fittable map -- not on the date-driven finding, which was an
artefact.

PHASE 6 — the deploy set was set by a coin-flip-power ruler; it is now set
by a per-stat power-coherent pre-committed test whose first act was to
report that it cannot evaluate anything. The audit was permitted to wound
the live deploy and did: total_bases lost its LODO claim. Standing
question unchanged -- 18 archetype slots across three deployed stats, every
one a single band indistinguishable from base rate.

Blind ordering held. p_win never mutated. No Bonferroni slot. Counter and
frozen clusters verified file-by-file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-06 23:01:57 -04:00
parent 1f40014256
commit ced40421ed
7 changed files with 545 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/lodo-instrument-audit.md
+170
View File
@@ -0,0 +1,170 @@
# Auditing the LODO instrument — it cannot evaluate any stat
**All four stats are UNTESTABLE-BY-LODO. Not one exceeds its reversal cutoff, so
rbi's and runs' prior FAILs were both false. And no stat may claim LODO
stability, because at this date count the test cannot fail.**
---
## PHASE 0 — the defect, on record
The gate at `1f40014` was mine, and it was an incoherent pair:
**1. A 1-SE informativeness bar coupled to a ZERO-reversal rule.** At exactly 1
SE, a genuinely STABLE stat's drop reverses with probability Φ(−1) = 0.1587. On
four informative drops:
```
P(>= 1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50
```
**The rule failed stable stats half the time by construction.** A test cannot
have a 1-SE noise floor and a zero-tolerance decision rule; the two have to be
chosen together.
**2. A pooled n\*.** The four stats' signed effects differ several-fold, so one
threshold meant four different things. Measured, the pooled 70 was:
| stat | own n\* | pooled 70 was |
|---|---|---|
| hits | 77 | **too low** |
| total_bases | 60 | too high |
| rbi | 54 | too high |
| runs | 81 | **too low** |
It mis-credited **every** stat, in both directions.
Neither defect touched the counter or `p_win`. Both touched only which
calibrations were judged stable.
---
## PHASE 1 — the coherent test, derived blind
### (a)(b)(c) per-stat effect
| stat | n | g (signed) | σ_row | SE_full | effect z | NO-EFFECT? |
|---|---|---|---|---|---|---|
| hits | 1,140 | −0.01288 | 0.11251 | 0.00333 | 3.87 | no |
| total_bases | 1,050 | −0.01380 | 0.10680 | 0.00330 | 4.19 | no |
| rbi | 630 | −0.00884 | 0.06459 | 0.00257 | 3.44 | no |
| runs | 597 | −0.00902 | 0.08080 | 0.00331 | 2.73 | no |
All four have a real effect at full n. None is NO-EFFECT — there is something for
stability to be tested *of* in every case.
### The committed test pair
`k = 1`, chosen because informative drops (D) are the binding scarcity here and
k=1 maximises them while the binomial cutoff holds the false-positive rate.
| stat | n\*=k²(σ/\|g\|)² | informative drops D | cutoff | FP | **power at τ=\|g\|** |
|---|---|---|---|---|---|
| hits | 77 | 5 | 2 | 0.031 | **0.093** |
| total_bases | 60 | 5 | 2 | 0.031 | **0.093** |
| rbi | 54 | 4 | 2 | 0.014 | **0.045** |
| runs | 81 | 3 | 2 | 0.004 | **0.014** |
Per-drop noise probability under stability Φ(−1) = 0.1587. FAIL iff reversals > cutoff.
### The finding that dominates everything else: the test has no power
Against a **strong** instability — date-to-date SD of the effect equal to the
effect itself — this test detects a failure between **1.4% and 9.3%** of the
time. Across every k examined (1.0 → 2.0), the best any stat reaches is 0.337,
and reaching even that costs all but two informative drops.
**A gate that cannot fail cannot pass.** `LODO_POWER_FLOOR = 0.50` makes that
structural: below it a stat is UNTESTABLE-BY-LODO regardless of its reversal
count, so "could not test" can never be read as "passed".
Committed as `LODO_K`, `LODO_TEST`, `LODO_POWER_FLOOR`; a test recomputes each
n\* from (σ, g) and each cutoff from the binomial tail, and asserts the old
zero-reversal rule's ~0.50 false-fail rate. The stale pooled constant is nulled
so nothing can read it.
---
## PHASE 2/3 — cold re-read at each stat's own n\*
| stat | own n\* | informative | reversals | cutoff | power | verdict |
|---|---|---|---|---|---|---|
| hits | 77 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** |
| total_bases | 60 | 5 | 0 | 2 | 0.093 | **UNTESTABLE** |
| rbi | 54 | 4 | 1 | 2 | 0.045 | **UNTESTABLE** |
| runs | 81 | 3 | 2 | 2 | 0.014 | **UNTESTABLE** |
**Setting the power floor aside entirely, not one stat exceeds its cutoff.**
---
## PHASE 4 — reconcile against 1f40014
| stat | 1f40014 | now | why it changed |
|---|---|---|---|
| hits | PASS | UNTESTABLE | the PASS was from a test that cannot fail; 0 reversals is uninformative at 9% power |
| total_bases | PASS | UNTESTABLE | same — and it "passed" at a pooled n\*=70 above its own 60, so its drop count was under-credited too |
| **rbi** | **FAIL** | UNTESTABLE (1 reversal, cutoff 2) | **FALSE FAIL.** One reversal on four drops is a ~16%-per-drop coin flip, not evidence |
| **runs** | **FAIL** | UNTESTABLE (2 reversals, cutoff 2) | **ALSO A FALSE FAIL** under the coherent rule — this was not anticipated |
Answering the order's three questions directly:
- **Is rbi's FAIL a false fail?** Yes. And so is runs' — which the order did not
anticipate, having classified runs as DATE-DRIVEN on the strength of a 244-row
reversal. Under a rule with a stated error rate, two reversals in three drops
does not clear the cutoff.
- **Was TB's PASS real?** No. It was vacuous: the test could not have failed it.
- **Does hits still pass at its own smaller n\*?** Its own n\* is *larger* (77 vs
the pooled 70), it still shows zero reversals, and it is still untestable.
---
## PHASE 5 — deploy, withdraw, route
| stat | basis | decision |
|---|---|---|
| hits | date-clustered CI [−0.0139, −0.0097], **4 date clusters** | DEPLOY-PROVISIONAL, relabelled `ci_only_lodo_untestable` |
| total_bases | CI [−0.0061, −0.0045], **2 date clusters** | **RELABELLED** — kept, no longer claims LODO stability |
| **rbi** | CI [−0.0092, −0.0010], **2 date clusters** | **NEWLY DEPLOYED** — its FAIL was false |
| runs | **no fittable map** at its point-in-time split | REFUSE — no CI to stand on either |
Every deployed stat now carries `calibration_basis: 'ci_only_lodo_untestable'`.
**Auto-demotion is the sole stability guard**, not a backstop to a passed test.
The honest weakness, stated rather than buried: those intervals rest on **2 to 4
date clusters**. That is thin support, and it is now the *only* support.
**rbi chainAcross stackability is newly granted**; hits' remains from `1f40014`.
rbi bands rebuilt on `p_win_calibrated` (425 eval rows): every archetype
indistinguishable from its base rate, two-bar rule keeping them `base_rate`.
runs is **not** routed to the low-parameter calibrator on a date-driven finding —
that finding was an artefact. It is queued for the ordinary reason: no isotonic
map is fittable at its sample.
---
## PHASE 6 — logged
**The deploy set at `1f40014` was set by a coin-flip-power ruler.** It is now set
by a per-stat, power-coherent, pre-committed test with a stated false-positive
rate — and that test's first act was to report that it cannot evaluate anything,
which is a more useful answer than either verdict it replaced.
**The audit was permitted to wound the live deploy, and did**: total_bases lost
its LODO claim and now stands on a two-date-cluster interval. That it could is
the integrity property.
**Standing question unchanged.** Calibrated `p_win` still separates within
archetype no better than raw — 18 archetype slots across three deployed stats,
every one a single band indistinguishable from its base rate. Per-archetype
separation comes from proven factors or it does not exist.
---
## Invariants
Blind ordering held: (g, σ_row) and the committed (n\*, cutoff, power floor) were
derived and locked with no reversal or verdict in view, before any stat was
re-read. `p_win` never mutated. No Bonferroni slot consumed. Counter and frozen
clusters verified file-by-file.