builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Audit the LODO instrument: it cannot evaluate any stat, and both prior

Browse Source 
FAILs were false

PHASE 0 — the gate at 1f40014 was mine and was an incoherent pair. A 1-SE
informativeness bar with a ZERO-reversal rule: at exactly 1 SE a stable
stat's drop reverses with prob Phi(-1)=0.1587, so on four informative
drops P(>=1 reversal | perfectly stable) = 1 - 0.8413^4 = 0.50. It failed
stable stats half the time by construction. And the pooled n*=70
mis-credited EVERY stat -- too low for hits (own 77) and runs (81), too
high for total_bases (60) and rbi (54).

PHASE 1, blind. Per-stat (g, sigma_row): hits -0.01288/0.11251, TB
-0.01380/0.10680, rbi -0.00884/0.06459, runs -0.00902/0.08080. All four
clear z=1.96 at full n, so none is NO-EFFECT. Committed k=1 with per-stat
n* and a binomial cutoff holding FP at 0.004-0.031.

THE FINDING THAT DOMINATES: the test has no power. Against a strong
instability (date-to-date SD equal to the effect) it detects a failure
1.4%-9.3% of the time, and across every k from 1.0 to 2.0 the best any
stat reaches is 0.337. A gate that cannot fail cannot pass, so
LODO_POWER_FLOOR=0.50 makes UNTESTABLE structural -- "could not test" can
never read as "passed".

PHASE 2/3 cold, at each stat's OWN n*:

  hits  5 informative, 0 reversals, cutoff 2, power 0.093  UNTESTABLE
  TB    5 informative, 0 reversals, cutoff 2, power 0.093  UNTESTABLE
  rbi   4 informative, 1 reversal,  cutoff 2, power 0.045  UNTESTABLE
  runs  3 informative, 2 reversals, cutoff 2, power 0.014  UNTESTABLE

Setting the power floor aside entirely, NOT ONE STAT EXCEEDS ITS CUTOFF.

PHASE 4 — rbi's FAIL was false, as the order suspected. So was RUNS' --
which the order did not anticipate, having classified it DATE-DRIVEN on a
244-row reversal; two reversals in three drops does not clear a cutoff of
2. TB's PASS was vacuous: the test could not have failed it. hits' own n*
is LARGER than the pooled one (77 vs 70), and it remains untestable.

PHASE 5 — deploy basis is now the date-clustered CI alone:

  hits  CI [-0.0139,-0.0097], 4 date clusters   relabelled ci_only
  TB    CI [-0.0061,-0.0045], 2 date clusters   RELABELLED, kept
  rbi   CI [-0.0092,-0.0010], 2 date clusters   NEWLY DEPLOYED
  runs  no fittable map at its split            REFUSE, no CI either

Every deployed stat carries calibration_basis ci_only_lodo_untestable and
auto-demotion is the SOLE stability guard, not a backstop to a passed
test. Stated plainly: those intervals rest on 2-4 date clusters, which is
thin, and it is now the only support. rbi gains chainAcross stackability;
its bands rebuilt on p_win_calibrated (425 rows) are every-archetype
base_rate. runs is queued for the low-param calibrator for the ordinary
reason -- no fittable map -- not on the date-driven finding, which was an
artefact.

PHASE 6 — the deploy set was set by a coin-flip-power ruler; it is now set
by a per-stat power-coherent pre-committed test whose first act was to
report that it cannot evaluate anything. The audit was permitted to wound
the live deploy and did: total_bases lost its LODO claim. Standing
question unchanged -- 18 archetype slots across three deployed stats, every
one a single band indistinguishable from base rate.

Blind ordering held. p_win never mutated. No Bonferroni slot. Counter and
frozen clusters verified file-by-file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-06 23:01:57 -04:00
parent 1f40014256
commit ced40421ed
7 changed files with 545 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/model/calibrationRegistry.js
+44 -28
View File
@@ -29,41 +29,57 @@ const { knownNumber } = require('../../utils/known');
const STATUS = Object.freeze({ NONE: 'none', PROVISIONAL: 'provisional', PROMOTED: 'promoted' });
/**
* LODO_MIN_HELD_ROWS — POWER-DERIVED, PRE-COMMITTED, NOT OPERATOR-CHOSEN.
* THE LODO TEST — a COHERENT pair, replacing the incoherent one at 1f40014.
*
* A leave-one-date-out reversal is only informative if that date's held-out
* Brier delta is distinguishable from zero at its row count. Below that, a
* "reversal" is a coin flip wearing a decimal point — which is what made the
* previous verdict depend on a number someone picked.
* ── WHAT WAS WRONG ───────────────────────────────────────────────────────
* The previous gate paired a 1-SE per-drop informativeness bar with a
* ZERO-reversal decision rule. At exactly 1 SE a genuinely STABLE stat's drop
* reverses with probability Phi(-1) = 0.159, so on four informative drops the
* chance of at least one reversal is 1 - 0.841^4 = 0.50. The rule failed stable
* stats half the time BY CONSTRUCTION. And n* was pooled across four stats whose
* signed effects differ several-fold, so one number meant four different things:
* measured, the pooled 70 was too LOW for hits (77) and runs (81) and too HIGH
* for total_bases (60) and rbi (54).
*
* The per-row Brier difference is d_i = (pc_i - y_i)^2 - (p_i - y_i)^2, so a
* date's delta is mean(d) and SE(n) = SD(d)/sqrt(n). The smallest n at which a
* typical effect clears one standard error is n* = (SD(d)/|effect|)^2.
* ── THE COHERENT PAIR ────────────────────────────────────────────────────
* The bar and the rule are chosen TOGETHER, per stat, for a stated error rate:
*
* Measured 2026-08-07, pooled across all four stats so that no single stat's
* verdict could shape the threshold that decides it:
* informative bar n*_k = k^2 * (sigma_row / |g|)^2 per stat
* decision rule FAIL iff reversals > cutoff, where under stability
* R ~ Binomial(D, Phi(-k)) and cutoff is the smallest c
* with P(R > c) <= 0.05
*
* pooled rows 3,417
* SD(d) 0.09816
* |effect| 0.01175
* n* = (0.09816/0.01175)^2 = 69.8 -> 70
* `g` is the mean SIGNED per-row improvement — the quantity whose sign a
* reversal flips. k = 1 is chosen because it maximises informative drops (D),
* which is the binding scarcity here, while the binomial cutoff holds the
* false-positive rate at 0.004-0.031 across the four stats.
*
* SE-vs-n: n=20 gives effect/SE 0.54, n=50 gives 0.85, n=75 gives 1.04. So
* anything under ~70 held rows cannot tell a real reversal from noise.
* ── AND THE TEST STILL HAS ALMOST NO POWER ───────────────────────────────
* At the stated alternative (date-to-date SD of the effect equal to |g| — a
* strong instability), power is 0.093 / 0.093 / 0.045 / 0.014. The test would
* MISS a real date-driven failure more than nine times in ten. Across every k
* examined, the best any stat reaches is 0.337.
*
* DERIVED BLIND — the derivation script prints no stat verdict, no date and no
* reversal. It ran, and this constant was committed, before any stat was
* re-read. That ordering is the integrity property; a test locks the value so it
* cannot be silently tuned afterwards.
* So a PASS here means "no instability was detected", NOT "it is stable", and a
* gate that cannot fail is not a gate. LODO_POWER_FLOOR makes that structural: a
* stat whose test power falls below it is UNTESTABLE-BY-LODO and may not claim
* LODO stability at all, whatever its reversal count.
*
* Derived BLIND — the derivation script prints no reversal, no verdict and no
* reversing date. It ran, and these were committed, before any stat was re-read.
*/
const LODO_MIN_HELD_ROWS = 70;
const LODO_THRESHOLD_BASIS = Object.freeze({
pooled_rows: 3417,
per_row_brier_diff_sd: 0.09816,
pooled_effect_abs_mean: 0.01175,
rule: 'n* = (SD(d) / |effect|)^2',
derived_blind: true,
const LODO_K = 1.0;
/** Below this power the test cannot fail, so it cannot pass either. */
const LODO_POWER_FLOOR = 0.50;
/** Per-stat, from (g, sigma_row) measured blind. */
const LODO_TEST = Object.freeze({
hits: { g: -0.01288, sigma_row: 0.11251, n_star: 77, informative_drops: 5, cutoff: 2, fp: 0.0310, power: 0.093 },
total_bases: { g: -0.01380, sigma_row: 0.10680, n_star: 60, informative_drops: 5, cutoff: 2, fp: 0.0310, power: 0.093 },
rbi: { g: -0.00884, sigma_row: 0.06459, n_star: 54, informative_drops: 4, cutoff: 2, fp: 0.0141, power: 0.045 },
runs: { g: -0.00902, sigma_row: 0.08080, n_star: 81, informative_drops: 3, cutoff: 2, fp: 0.0040, power: 0.014 },
});
/** Legacy name kept so nothing silently reads a stale pooled value. */
const LODO_MIN_HELD_ROWS = null;
/** The ORIGINAL floor, correctly scoped: promotion, not deploy. */
const PROMOTION_DATE_CLUSTERS = 40;
@@ -144,4 +160,4 @@ function createRegistry(initial = {}) {
return { deploy, reverify, serves, get, all, log: () => log.slice() };
}
module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS, LODO_MIN_HELD_ROWS, LODO_THRESHOLD_BASIS };
module.exports = { createRegistry, STATUS, PROMOTION_DATE_CLUSTERS, LODO_K, LODO_TEST, LODO_POWER_FLOOR, LODO_MIN_HELD_ROWS };
src/services/snapshotService.js
+29 -17
View File
@@ -276,27 +276,37 @@ async function loadPitcherArsenals(sport) {
}
/**
* Stats whose calibration passed leave-one-date-out and may serve a calibrated
* number. PROVISIONAL: auto-demoted the first time the held-out interval stops
* excluding zero or the favourite over-prediction flips sign.
* Stats that may serve a calibrated number, and ON WHAT BASIS.
*
* Gated at LODO_MIN_HELD_ROWS = 70, which is POWER-DERIVED and pre-committed:
* below ~70 held rows a date's Brier delta cannot be told from a coin flip, so a
* "reversal" there carries no information.
* ── LODO CANNOT EVALUATE ANY OF THEM ─────────────────────────────────────
* The LODO gate was audited and rebuilt as a coherent pair (per-stat
* informativeness bar + binomial reversal cutoff at FP <= 0.05). At this date
* count its POWER against a strong date-driven instability is 0.093 / 0.093 /
* 0.045 / 0.014 — it would miss a real failure more than nine times in ten. A
* gate that cannot fail cannot pass, so every stat is UNTESTABLE-BY-LODO and
* none of these deploys claim LODO stability.
*
* hits was WITHDRAWN at 6ae11f1 and is RESTORED here. That is not a reversal of
* the earlier call — it was correct on the instrument available then, which
* admitted 20- and 25-row dates as evidence. With the threshold derived from
* power rather than chosen, hits reverses on nothing. The restoration came
* through the gate, not around it.
* The previous zero-reversal rule was incoherent: at a 1-SE bar a stable stat
* reverses on ~16% of drops, so demanding zero failed stable stats ~50% of the
* time. Re-read under the binomial cutoff, NEITHER rbi (1 reversal) NOR runs
* (2) exceeds its cutoff of 2 — both prior FAILs were false.
*
* rbi and runs remain ABSENT, and their failures are NOT underpowered: each
* reverses on a date comfortably above the threshold (rbi 2026-08-01 n=99; runs
* 2026-08-01 n=86 and 2026-08-05 n=244). Those are DATE-DRIVEN failures — no
* threshold and no further accrual rescues them, and isotonic is fitting
* day-structure. Routed to the low-parameter calibrator queue.
* ── SO THE DEPLOY BASIS IS THE DATE-CLUSTERED CI ALONE ───────────────────
* hits, total_bases and rbi each have a point-in-time held-out interval
* excluding zero. That is the ONLY support they have, and it is thin — the
* interval rests on 4, 2 and 2 date clusters respectively. Auto-demotion is
* therefore the sole stability guard, not a backstop to a passed test.
*
* runs is absent: no isotonic map was fittable at its point-in-time split, so it
* has no CI support to stand on either.
*/
const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases']);
const CALIBRATION_DEPLOYED = Object.freeze(['hits', 'total_bases', 'rbi']);
/** Why each deployed stat is allowed to serve. Not one of them is LODO-stable. */
const CALIBRATION_BASIS = Object.freeze({
hits: 'ci_only_lodo_untestable',
total_bases: 'ci_only_lodo_untestable',
rbi: 'ci_only_lodo_untestable',
});
async function runSnapshot(sport, opts = {}) {
const sp = String(sport || '').toLowerCase();
@@ -759,6 +769,7 @@ async function runSnapshot(sport, opts = {}) {
g.calibrated = out.calibrated;
g.calibration_reason = out.reason;
g.calibration_status = 'provisional';
g.calibration_basis = CALIBRATION_BASIS[stat] || null;
if (out.calibrated) marked += 1;
}
console.log(`[calibration] ${sp} ${stat} (PROVISIONAL) — ${marked} stackable; fit n=${calibrator.fit_n} through ${calibrator.fitted_through}`);
@@ -881,5 +892,6 @@ module.exports = {
pushTickerItems,
ACTIVE_SPORTS,
CALIBRATION_DEPLOYED,
CALIBRATION_BASIS,
__internals: { propKey, gradedAtFor, indexOdds, lastName, isTopGrade, DELTA_NOISE, DELTA_MOVE, TICKER_CAP },
};