Build the two-part factor gate: one factor proves, and zero are theatre
The question was whether the hit grade reads tonight's game or just says he is due. Answering it needed a gate that correlation cannot provide, because correlation cannot separate the two ways a factor looks alive: it reads the game, or it moves the number and reads nothing. The second is what a product ships by accident -- arch-v1 moved 76% of rows by 2.5 points, changed resolution by 0.0000, and was live for months, and no user could have told. So a factor must now clear both conditions: move the prediction off the player's own leave-one-out base rate, AND improve out-of-sample Brier. Brier rather than correlation, because correlation asks whether the ordering improved and this asks whether the NUMBER got closer to what happened -- and for a graded probability the number is the product. The correction applies to the interval itself, which turned out to matter more than expected. A plain 95% CI is the right bar for one test; at fifty cumulative tests roughly two or three intervals exclude zero by chance alone. Widening to 1 - 0.05/tests, currently 99.9%, flipped both defence and platoon out of "proves". A 95% interval would have shipped two unproven factors into the grade, with reasoning text explaining them to users. That forced a distinction I had initially collapsed. Defence and platoon have FAVOURABLE point estimates whose corrected intervals merely span zero, and calling that THEATER would repeat the error this codebase keeps correcting: insufficient evidence is not evidence of absence. THEATER is now reserved for its one real meaning -- moves the number, reads nothing -- and NOT_PROVEN_AT_CORRECTED_BAR names a real candidate held to a bar that rises with every hypothesis the programme tests. Result on 741 settled hits rows: pitcher_contact_profile PROVES, improving Brier by 0.0066 with a 99.9% interval of [-0.0114, -0.0016]. Defence (-0.0043) and platoon (-0.0039) are not proven at the corrected bar. Park is sample-blocked at n=405. Zero factors are theatre, which is the genuinely good news: nothing decorative is being wired. Per-archetype every slot is sample-blocked (BOMBER 252-294, GHOST 67-125). Two spec gaps worth recording. The approach identities the order names -- SPRAY, DAMAGE-DEALER, COUNT-WORKER -- do not exist in the registry; the MLB batter archetypes are BOMBER, GHOST, TORCH, BRUSH, DRIVER, FLEX, ALPHA, HYBRID and CATALYST. And parkFactors maps hits to run_base, so there is no hits-specific park factor at all: a park that turns outs into hits without producing runs is invisible to the input we have. The grade rescale is NOT run. It was explicitly gated on the factor proving, and one pooled factor worth 0.0066 of Brier is not a factor-informed distribution -- rescaling on it would dress a base-rate model as a matchup model, which is the exact thing this gate was built to prevent. 4,286 tests green (340 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,179 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* factorGate — DOES THIS FACTOR READ TONIGHT'S GAME, OR JUST MOVE THE NUMBER?
|
||||
*
|
||||
* Every gate in this codebase so far asks one question: is there a correlation.
|
||||
* That is necessary and it is not sufficient, because it cannot tell apart the
|
||||
* two ways a factor can look alive:
|
||||
*
|
||||
* PROVES the factor moves the prediction off the player's base rate AND the
|
||||
* moved prediction is MORE ACCURATE out-of-sample. It is reading the
|
||||
* game.
|
||||
* THEATER the factor moves the prediction — sometimes a lot — and accuracy
|
||||
* does not improve, or gets worse. The number looks responsive. It is
|
||||
* responding to nothing.
|
||||
*
|
||||
* THEATER IS THE DANGEROUS ONE, and it is what a product ships by accident. A
|
||||
* grade that swings on park and platoon LOOKS like it read tonight's matchup;
|
||||
* a user cannot tell the difference from the outside, and neither can a
|
||||
* correlation test. arch-v1 was exactly this: it moved 76% of rows by 2.5 points
|
||||
* and changed resolution by 0.0000. It was live for months.
|
||||
*
|
||||
* So a factor must clear BOTH:
|
||||
*
|
||||
* (a) movement mean |Δp| against the base-rate baseline is real
|
||||
* (b) improvement paired bootstrap on Brier score, CI excluding zero
|
||||
*
|
||||
* (a) alone is rejected BY NAME as THEATER rather than filed as "inconclusive",
|
||||
* because the distinction is the whole point: an inconclusive factor might work
|
||||
* with more data, and a theatrical one is actively misleading the user now.
|
||||
*
|
||||
* ── WHY BRIER AND NOT CORRELATION ────────────────────────────────────────
|
||||
* Correlation asks whether the ORDERING improved. This asks whether the NUMBER
|
||||
* got closer to what happened, which is what a probability claims. A factor can
|
||||
* improve ordering while degrading the number, and for a graded probability the
|
||||
* number is the product.
|
||||
*/
|
||||
|
||||
const { knownNumber } = require('../../utils/known');
|
||||
|
||||
/** Minimum mean |Δp| for a factor to count as having moved anything at all. */
|
||||
const MIN_MOVEMENT = 0.01;
|
||||
const MIN_N = 500;
|
||||
|
||||
const brier = (ps, ys) => (ps.length
|
||||
? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null);
|
||||
|
||||
function makeRnd(seed) {
|
||||
let s = seed >>> 0;
|
||||
return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
|
||||
}
|
||||
|
||||
/**
|
||||
* How far does the factor move the prediction off the baseline?
|
||||
*
|
||||
* Reported as the MEAN ABSOLUTE shift and its spread. A factor that shifts every
|
||||
* prediction by the same amount is not reading the game either — it is a
|
||||
* constant — so the spread matters as much as the mean.
|
||||
*/
|
||||
function movement(rows) {
|
||||
const deltas = [];
|
||||
for (const r of rows || []) {
|
||||
const b = knownNumber(r && r.baseline);
|
||||
const c = knownNumber(r && r.conditioned);
|
||||
if (b === null || c === null) continue; // absent, never assumed equal
|
||||
deltas.push(c - b);
|
||||
}
|
||||
if (deltas.length === 0) return { n: 0, mean_abs_shift: null, sd_shift: null, max_abs_shift: null };
|
||||
const meanAbs = deltas.reduce((s, d) => s + Math.abs(d), 0) / deltas.length;
|
||||
const mean = deltas.reduce((s, d) => s + d, 0) / deltas.length;
|
||||
const sd = deltas.length > 1
|
||||
? Math.sqrt(deltas.reduce((s, d) => s + (d - mean) ** 2, 0) / (deltas.length - 1)) : 0;
|
||||
return {
|
||||
n: deltas.length,
|
||||
mean_abs_shift: round4(meanAbs),
|
||||
mean_signed_shift: round4(mean),
|
||||
sd_shift: round4(sd),
|
||||
max_abs_shift: round4(Math.max(...deltas.map(Math.abs))),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Did the moved prediction get CLOSER to what happened?
|
||||
*
|
||||
* Paired bootstrap on the Brier difference — the same rows score both models, so
|
||||
* treating their errors as independent would overstate certainty. NEGATIVE delta
|
||||
* means the conditioned model has lower Brier, i.e. it improved.
|
||||
*/
|
||||
function improvement(rows, iters = 4000, seed = 20260805, cumulativeTests = 1) {
|
||||
const usable = (rows || []).filter((r) =>
|
||||
knownNumber(r.baseline) !== null && knownNumber(r.conditioned) !== null && knownNumber(r.won) !== null);
|
||||
if (usable.length < 30) return null;
|
||||
const rnd = makeRnd(seed);
|
||||
const diffs = [];
|
||||
for (let it = 0; it < iters; it += 1) {
|
||||
const b = []; const c = []; const y = [];
|
||||
for (let i = 0; i < usable.length; i += 1) {
|
||||
const r = usable[Math.floor(rnd() * usable.length)];
|
||||
b.push(r.baseline); c.push(r.conditioned); y.push(r.won > 0 ? 1 : 0);
|
||||
}
|
||||
diffs.push(brier(c, y) - brier(b, y));
|
||||
}
|
||||
diffs.sort((x, y) => x - y);
|
||||
// CUMULATIVE CORRECTION APPLIED TO THE INTERVAL ITSELF. A plain 95% CI is the
|
||||
// right bar for ONE test and far too lenient for a programme that has run
|
||||
// dozens: at 50 cumulative tests, roughly two or three 95% intervals exclude
|
||||
// zero by chance alone. So the interval widens to 1 − 0.05/tests, which is the
|
||||
// same discipline the p-value gate applies, expressed as an interval.
|
||||
const tests = Math.max(1, Math.round(knownNumber(cumulativeTests) ?? 1));
|
||||
const alpha = 0.05 / tests;
|
||||
const q = (p) => round4(diffs[Math.floor(Math.min(diffs.length - 1, Math.max(0, p * (diffs.length - 1))))]);
|
||||
const ci = [q(alpha / 2), q(1 - alpha / 2)];
|
||||
const ys = usable.map((r) => (r.won > 0 ? 1 : 0));
|
||||
return {
|
||||
n: usable.length,
|
||||
brier_baseline: round4(brier(usable.map((r) => r.baseline), ys)),
|
||||
brier_conditioned: round4(brier(usable.map((r) => r.conditioned), ys)),
|
||||
brier_delta: round4(brier(usable.map((r) => r.conditioned), ys) - brier(usable.map((r) => r.baseline), ys)),
|
||||
ci: ci,
|
||||
ci_level: round4(1 - alpha),
|
||||
bonferroni_tests: tests,
|
||||
improves: ci[1] < 0, // whole interval below zero = genuinely better
|
||||
degrades: ci[0] > 0,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* THE VERDICT. Both conditions, named outcomes.
|
||||
*
|
||||
* `cumulativeTests` is the programme-lifetime Bonferroni denominator; it tightens
|
||||
* the improvement requirement the same way it does everywhere else.
|
||||
*/
|
||||
function adjudicate(rows, opts = {}) {
|
||||
const minN = opts.minN ?? MIN_N;
|
||||
const minMove = opts.minMovement ?? MIN_MOVEMENT;
|
||||
const mv = movement(rows);
|
||||
const imp = improvement(rows, opts.iters, opts.seed, opts.cumulativeTests);
|
||||
|
||||
const base = { factor: opts.factor || null, archetype: opts.archetype || null, stat: opts.stat || 'hits', movement: mv, improvement: imp };
|
||||
|
||||
if (mv.n < minN) {
|
||||
return { ...base, verdict: 'CANDIDATE_PENDING_SAMPLE', reason: `n ${mv.n} < ${minN}`, rows_needed: minN - mv.n };
|
||||
}
|
||||
if (mv.mean_abs_shift === null || mv.mean_abs_shift < minMove) {
|
||||
// It never moved the number, so it cannot be reading anything.
|
||||
return { ...base, verdict: 'INERT', reason: `mean |shift| ${mv.mean_abs_shift} < ${minMove}` };
|
||||
}
|
||||
if (!imp) return { ...base, verdict: 'CANDIDATE_PENDING_SAMPLE', reason: 'too few paired rows to bootstrap' };
|
||||
|
||||
// NOT PROVEN is not the same as THEATER, and collapsing them would repeat the
|
||||
// error this codebase keeps having to correct: insufficient evidence is not
|
||||
// evidence of absence. A factor whose POINT ESTIMATE improves accuracy but
|
||||
// whose corrected interval still spans zero has not earned its place — and it
|
||||
// is not decorative either. It is a real candidate held to a bar that rises
|
||||
// with every hypothesis the programme tests. Saying so keeps THEATER meaning
|
||||
// the one thing it must mean: moves the number, reads nothing.
|
||||
if (!imp.improves && imp.brier_delta < 0) {
|
||||
return {
|
||||
...base,
|
||||
verdict: 'NOT_PROVEN_AT_CORRECTED_BAR',
|
||||
reason: `moves ${mv.mean_abs_shift} and the point estimate improves Brier by ${-imp.brier_delta}, but the interval corrected for ${imp.bonferroni_tests} tests still spans zero (${JSON.stringify(imp.ci)} at level ${imp.ci_level})`,
|
||||
note: 'a real candidate, not theatre — it improves on the point estimate and needs more sample, or a tighter bar than the programme can currently afford it',
|
||||
};
|
||||
}
|
||||
if (imp.improves) {
|
||||
return { ...base, verdict: 'PROVES', reason: `moves ${mv.mean_abs_shift} and improves Brier by ${-imp.brier_delta} (CI ${JSON.stringify(imp.ci)} at level ${imp.ci_level}, corrected for ${imp.bonferroni_tests} tests)` };
|
||||
}
|
||||
// MOVED BUT DID NOT IMPROVE. Named, not softened.
|
||||
return {
|
||||
...base,
|
||||
verdict: 'THEATER',
|
||||
reason: `moves the prediction by ${mv.mean_abs_shift} on average (max ${mv.max_abs_shift}) while accuracy does NOT improve (Brier delta ${imp.brier_delta}, CI ${JSON.stringify(imp.ci)} at level ${imp.ci_level})`,
|
||||
consequence: "wiring this would make the grade LOOK like it read tonight's game while reading nothing — the failure mode a user cannot detect from outside",
|
||||
};
|
||||
}
|
||||
|
||||
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
|
||||
module.exports = { movement, improvement, adjudicate, MIN_MOVEMENT, MIN_N };
|
||||
Reference in New Issue
Block a user