builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Audit the LODO instrument: it cannot evaluate any stat, and both prior

Browse Source 
FAILs were false

PHASE 0 — the gate at 1f40014 was mine and was an incoherent pair. A 1-SE
informativeness bar with a ZERO-reversal rule: at exactly 1 SE a stable
stat's drop reverses with prob Phi(-1)=0.1587, so on four informative
drops P(>=1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50. It failed
stable stats half the time by construction. And the pooled n*=70
mis-credited EVERY stat -- too low for hits (own 77) and runs (81), too
high for total_bases (60) and rbi (54).

PHASE 1, blind. Per-stat (g, sigma_row): hits -0.01288/0.11251, TB
-0.01380/0.10680, rbi -0.00884/0.06459, runs -0.00902/0.08080. All four
clear z=1.96 at full n, so none is NO-EFFECT. Committed k=1 with per-stat
n* and a binomial cutoff holding FP at 0.004-0.031.

THE FINDING THAT DOMINATES: the test has no power. Against a strong
instability (date-to-date SD equal to the effect) it detects a failure
1.4%-9.3% of the time, and across every k from 1.0 to 2.0 the best any
stat reaches is 0.337. A gate that cannot fail cannot pass, so
LODO_POWER_FLOOR=0.50 makes UNTESTABLE structural -- "could not test" can
never read as "passed".

PHASE 2/3 cold, at each stat's OWN n*:

  hits  5 informative, 0 reversals, cutoff 2, power 0.093  UNTESTABLE
  TB    5 informative, 0 reversals, cutoff 2, power 0.093  UNTESTABLE
  rbi   4 informative, 1 reversal,  cutoff 2, power 0.045  UNTESTABLE
  runs  3 informative, 2 reversals, cutoff 2, power 0.014  UNTESTABLE

Setting the power floor aside entirely, NOT ONE STAT EXCEEDS ITS CUTOFF.

PHASE 4 — rbi's FAIL was false, as the order suspected. So was RUNS' --
which the order did not anticipate, having classified it DATE-DRIVEN on a
244-row reversal; two reversals in three drops does not clear a cutoff of
2. TB's PASS was vacuous: the test could not have failed it. hits' own n*
is LARGER than the pooled one (77 vs 70), and it remains untestable.

PHASE 5 — deploy basis is now the date-clustered CI alone:

  hits  CI [-0.0139,-0.0097], 4 date clusters   relabelled ci_only
  TB    CI [-0.0061,-0.0045], 2 date clusters   RELABELLED, kept
  rbi   CI [-0.0092,-0.0010], 2 date clusters   NEWLY DEPLOYED
  runs  no fittable map at its split            REFUSE, no CI either

Every deployed stat carries calibration_basis ci_only_lodo_untestable and
auto-demotion is the SOLE stability guard, not a backstop to a passed
test. Stated plainly: those intervals rest on 2-4 date clusters, which is
thin, and it is now the only support. rbi gains chainAcross stackability;
its bands rebuilt on p_win_calibrated (425 rows) are every-archetype
base_rate. runs is queued for the low-param calibrator for the ordinary
reason -- no fittable map -- not on the date-driven finding, which was an
artefact.

PHASE 6 — the deploy set was set by a coin-flip-power ruler; it is now set
by a per-stat power-coherent pre-committed test whose first act was to
report that it cannot evaluate anything. The audit was permitted to wound
the live deploy and did: total_bases lost its LODO claim. Standing
question unchanged -- 18 archetype slots across three deployed stats, every
one a single band indistinguishable from base rate.

Blind ordering held. p_win never mutated. No Bonferroni slot. Counter and
frozen clusters verified file-by-file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-06 23:01:57 -04:00
parent 1f40014256
commit ced40421ed
7 changed files with 545 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/unit/calibrationDeployGate.test.js
+27 -25
View File
@@ -1,43 +1,45 @@
'use strict';
/**
* Which stats serve a calibrated number in the live pipeline.
* Which stats serve a calibrated number, and on what basis.
*
* The rule this locks: a stat that cannot survive dropping a single settled date
* was never calibrated — it was fitted to that date. hits WAS served calibrated
* and is not any more, which is the honest consequence of measuring it.
* The thing these lock is that "could not test" never reads as "passed". LODO
* has 1.4%-9.3% power at this date count, so no deployed stat claims stability
* from it — each rides a thin date-clustered interval and auto-demotion alone.
*/
const snapshotService = require('../../src/services/snapshotService');
const { LODO_TEST, LODO_POWER_FLOOR } = require('../../src/services/model/calibrationRegistry');
describe('the deployed set is LODO-gated', () => {
it('serves the stats that passed LODO at the powered threshold', () => {
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('total_bases');
// hits was withdrawn at 6ae11f1 under a hand-chosen threshold that admitted
// 20-row dates as evidence, and is restored here because the powered
// instrument finds no reversal. Through the gate, not around it.
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('hits');
describe('the deploy set rests on the CI, not on a passed LODO', () => {
it('serves the three stats with a point-in-time interval excluding zero', () => {
expect(snapshotService.CALIBRATION_DEPLOYED).toEqual(['hits', 'total_bases', 'rbi']);
});
it('does NOT serve rbi or runs — both fail on dates ABOVE the threshold', () => {
// These are DATE-DRIVEN failures, not underpowered ones: rbi reverses on a
// 99-row date and runs on 86- and 244-row dates. No threshold rescues them.
for (const stat of ['rbi', 'runs']) {
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain(stat);
it('does NOT serve runs — no fittable map, so no CI to stand on', () => {
expect(snapshotService.CALIBRATION_DEPLOYED).not.toContain('runs');
});
it('labels every deployed stat as LODO-untestable rather than LODO-stable', () => {
for (const stat of snapshotService.CALIBRATION_DEPLOYED) {
expect(snapshotService.CALIBRATION_BASIS[stat]).toBe('ci_only_lodo_untestable');
}
});
it('every deployed stat cleared the power-derived threshold, not a chosen one', () => {
const { LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS } = require('../../src/services/model/calibrationRegistry');
expect(LODO_MIN_HELD_ROWS).toBe(70);
expect(LODO_THRESHOLD_BASIS.derived_blind).toBe(true);
// The reversing dates that keep rbi/runs out are all at or above it, so
// their exclusion cannot be an artefact of the threshold.
for (const n of [99, 86, 244]) expect(n).toBeGreaterThanOrEqual(LODO_MIN_HELD_ROWS);
it('no stat may claim LODO stability, because the test cannot fail', () => {
for (const t of Object.values(LODO_TEST)) expect(t.power).toBeLessThan(LODO_POWER_FLOOR);
});
it('is frozen, so a stat cannot be added at runtime without a code change', () => {
it('rbi and runs were FALSE FAILS under the old zero-reversal rule', () => {
// Re-read under the binomial cutoff: 1 and 2 reversals, cutoff 2 for both.
expect(LODO_TEST.rbi.cutoff).toBe(2);
expect(LODO_TEST.runs.cutoff).toBe(2);
// rbi returns to the deploy set on CI support; runs still has none.
expect(snapshotService.CALIBRATION_DEPLOYED).toContain('rbi');
});
it('is frozen, so a stat cannot be added at runtime', () => {
expect(Object.isFrozen(snapshotService.CALIBRATION_DEPLOYED)).toBe(true);
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('hits'); }).toThrow();
expect(() => { snapshotService.CALIBRATION_DEPLOYED.push('runs'); }).toThrow();
});
});
tests/unit/calibrationRegistry.test.js
+47 -23
View File
@@ -119,34 +119,58 @@ describe('serving is band-limited', () => {
});
});
describe('the LODO held-row threshold is power-derived, not operator-chosen', () => {
const { LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS } = require('../../src/services/model/calibrationRegistry');
describe('the LODO test is a COHERENT pair, not a bar plus an unrelated rule', () => {
const { LODO_K, LODO_TEST, LODO_POWER_FLOOR } = require('../../src/services/model/calibrationRegistry');
it('is the value its own stated derivation produces', () => {
// n* = (SD(d) / |effect|)^2 -- recomputed here so the constant cannot drift
// away from the basis that justifies it.
const { per_row_brier_diff_sd: sd, pooled_effect_abs_mean: eff } = LODO_THRESHOLD_BASIS;
expect(Math.ceil((sd / eff) ** 2)).toBe(LODO_MIN_HELD_ROWS);
const normCdf = (z) => {
const t = 1 / (1 + 0.2316419 * Math.abs(z));
const d = 0.3989422804014327 * Math.exp(-z * z / 2);
const p = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + t * (-1.821255978 + t * 1.330274429))));
return z >= 0 ? 1 - p : p;
};
const binomPmf = (n, k, p) => {
let logC = 0;
for (let i = 0; i < k; i += 1) logC += Math.log(n - i) - Math.log(i + 1);
return Math.exp(logC + k * Math.log(p) + (n - k) * Math.log(1 - p));
};
const tail = (n, c, p) => { let s = 0; for (let k = c + 1; k <= n; k += 1) s += binomPmf(n, k, p); return s; };
it('each n* is what its own (sigma_row, g) produce — no pooled value', () => {
for (const [stat, t] of Object.entries(LODO_TEST)) {
expect(Math.ceil(LODO_K ** 2 * (t.sigma_row / Math.abs(t.g)) ** 2)).toBe(t.n_star);
}
// And the four differ, which is exactly why one pooled number mis-credited them.
const stars = Object.values(LODO_TEST).map((t) => t.n_star);
expect(new Set(stars).size).toBeGreaterThan(1);
});
it('carries its derivation basis, and was derived blind', () => {
expect(LODO_THRESHOLD_BASIS.rule).toBe('n* = (SD(d) / |effect|)^2');
expect(LODO_THRESHOLD_BASIS.derived_blind).toBe(true);
expect(LODO_THRESHOLD_BASIS.pooled_rows).toBeGreaterThan(1000);
});
it('rejects the thresholds that were previously chosen by hand', () => {
// 20 was the operator-chosen value whose verdict moved with it; anything
// below n* cannot distinguish a reversal from a coin flip.
for (const weak of [10, 20, 25, 30, 50]) {
const se = LODO_THRESHOLD_BASIS.per_row_brier_diff_sd / Math.sqrt(weak);
expect(LODO_THRESHOLD_BASIS.pooled_effect_abs_mean).toBeLessThan(se);
expect(weak).toBeLessThan(LODO_MIN_HELD_ROWS);
it('each cutoff is the smallest one holding the false-positive rate at 0.05', () => {
const p = normCdf(-LODO_K);
for (const [stat, t] of Object.entries(LODO_TEST)) {
expect(tail(t.informative_drops, t.cutoff, p)).toBeLessThanOrEqual(0.05);
if (t.cutoff > 0) expect(tail(t.informative_drops, t.cutoff - 1, p)).toBeGreaterThan(0.05);
expect(t.fp).toBeCloseTo(tail(t.informative_drops, t.cutoff, p), 3);
}
});
it('is informative at the committed value', () => {
const se = LODO_THRESHOLD_BASIS.per_row_brier_diff_sd / Math.sqrt(LODO_MIN_HELD_ROWS);
expect(LODO_THRESHOLD_BASIS.pooled_effect_abs_mean).toBeGreaterThanOrEqual(se * 0.99);
it('the OLD rule is demonstrably incoherent — it failed stable stats ~half the time', () => {
// Zero-reversal rule at a 1-SE bar, on four informative drops.
const pNoise = normCdf(-1);
const falseFail = 1 - (1 - pNoise) ** 4;
expect(falseFail).toBeGreaterThan(0.45);
expect(falseFail).toBeLessThan(0.55);
});
it('every stat falls below the power floor, so none may claim LODO stability', () => {
// A gate that cannot fail is not a gate. This is the honest state at this
// date count, and the floor makes it structural rather than a footnote.
for (const [stat, t] of Object.entries(LODO_TEST)) {
expect(t.power).toBeLessThan(LODO_POWER_FLOOR);
}
});
it('the stale pooled threshold is nulled so nothing can read it', () => {
const { LODO_MIN_HELD_ROWS } = require('../../src/services/model/calibrationRegistry');
expect(LODO_MIN_HELD_ROWS).toBeNull();
});
});