Ingest defence, and make Bonferroni cumulative across the programme
Two things shipped that stand regardless of sample. DEFENCE. Statcast Outs Above Average is free on the host we already pull six feeds from, so there was nothing to decide. 514 fielders, aggregated to team level -- the unit a batter's prop actually needs, the defence behind the pitcher he faces -- and persisted as 31 team rows. Verified in production. Cubs +56 best, Mariners -29 worst. Unknown is not zero, and it bites unusually hard here: an OAA of 0 is a REAL reading meaning exactly average, so coercing absence to 0 would assert that every unmeasured fielder is league-average, which is the commonest defensive profile there is. team_defense also carries as_of_date in its primary key from the first row -- statcast_aggregates was built upsert-in-place and that silently made every backtest leak the games it predicted, so point-in-time is available here before it is needed rather than after a wrong answer. A bug worth recording as a class: BASE already ends in /leaderboard, so the new feed built a doubled path and 404'd. Because a failing feed degrades to an empty index by design -- correct, so one broken source cannot fail the whole pull -- it surfaced as "fielding_oaa: 0 rows", which reads exactly like "Statcast has no fielding data". Graceful degradation makes a wiring bug look like an honest absence. CUMULATIVE CORRECTION. Bonferroni had been applied per session throughout: a run testing eight features corrected by eight. Across a programme's lifetime that is wrong in the dangerous direction, because every order gets a fresh generous alpha and the false-positive rate compounds quietly. Correcting by 8 when sixty have been tried is how a noise result eventually gets recorded as PROVEN with a p-value to point at. The denominator is now distinct hypotheses ever tested, persisted, and it moved 19 -> 38 within this session alone, alpha 0.0026 -> 0.0013. Re-tests deliberately do not inflate it: re-asking the same question on more data is not a new shot on goal, and counting it would punish the discipline of waiting for sample. THE MEASUREMENT. The differential the theory predicted is present: defence correlates with the counter's residual at +0.130 for GHOST, the contact and speed archetype, and -0.018 for BOMBER, the power archetype. A GHOST's hits depend on whether anyone can range to the ball; a BOMBER's barrels clear the defence entirely. So a flat BOMBER result is the theory working rather than the test failing. It is not a result. GHOST is n=104 against a 500 bar, with p=0.188 against a corrected alpha of 0.0013 -- three orders of magnitude short. Both are recorded as CANDIDATE with their measured lift, tagged contact-skill, so the re-run at full sample compares against a recorded baseline. Nothing proved, so nothing was recalibrated and nothing shipped. 4,228 tests green (336 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,83 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* The cumulative Bonferroni denominator, asserted as behaviour.
|
||||
*
|
||||
* The failure this prevents is silent and slow: correcting by 8 in a session
|
||||
* that tries 8 hypotheses, forever, while the programme as a whole has tried
|
||||
* sixty. Nothing looks wrong in any single run.
|
||||
*/
|
||||
|
||||
const tl = require('../../src/services/model/testLedger');
|
||||
|
||||
const H = (o) => ({ sport: 'mlb', target: 'counter_residual', ...o });
|
||||
|
||||
describe('the denominator counts DISTINCT hypotheses across the lifetime', () => {
|
||||
it('accumulates across separate runs — it does not reset per session', async () => {
|
||||
const store = tl.memoryStore();
|
||||
const run1 = await tl.recordAndCount(store, [
|
||||
H({ stat: 'hits', archetype: 'BOMBER', interaction: 'barrel_x_park' }),
|
||||
H({ stat: 'hits', archetype: 'BOMBER', interaction: 'launch_x_gb' }),
|
||||
]);
|
||||
expect(run1.cumulative_tests).toBe(2);
|
||||
|
||||
const run2 = await tl.recordAndCount(store, [
|
||||
H({ stat: 'hits', archetype: 'GHOST', interaction: 'defense_x_contact' }),
|
||||
]);
|
||||
// A per-session correction would have said 1 here. The whole point is 3.
|
||||
expect(run2.cumulative_tests).toBe(3);
|
||||
expect(run2.new_this_run).toBe(1);
|
||||
});
|
||||
|
||||
it('RE-TESTING the same hypothesis does not inflate the denominator', async () => {
|
||||
const store = tl.memoryStore();
|
||||
const h = H({ stat: 'hits', archetype: 'BOMBER', interaction: 'barrel_x_park' });
|
||||
await tl.recordAndCount(store, [h]);
|
||||
const again = await tl.recordAndCount(store, [h]);
|
||||
// Waiting for more sample and re-asking is the SAME question, and must not
|
||||
// be punished — that discipline is what the programme depends on.
|
||||
expect(again.cumulative_tests).toBe(1);
|
||||
expect(again.repeat_this_run).toBe(1);
|
||||
expect(again.new_this_run).toBe(0);
|
||||
});
|
||||
|
||||
it('the current batch IS included — a run is corrected for its own tests', async () => {
|
||||
const store = tl.memoryStore();
|
||||
const out = await tl.recordAndCount(store, [
|
||||
H({ stat: 'rbi', interaction: 'a' }), H({ stat: 'rbi', interaction: 'b' }),
|
||||
]);
|
||||
expect(out.cumulative_tests).toBe(2);
|
||||
});
|
||||
|
||||
it('the same interaction on a DIFFERENT stat/archetype/target is a NEW shot on goal', async () => {
|
||||
const store = tl.memoryStore();
|
||||
await tl.recordAndCount(store, [H({ stat: 'hits', archetype: 'BOMBER', interaction: 'x' })]);
|
||||
const out = await tl.recordAndCount(store, [
|
||||
H({ stat: 'total_bases', archetype: 'BOMBER', interaction: 'x' }), // new stat
|
||||
H({ stat: 'hits', archetype: 'GHOST', interaction: 'x' }), // new archetype
|
||||
H({ stat: 'hits', archetype: 'BOMBER', interaction: 'x', target: 'outcome' }), // new target
|
||||
]);
|
||||
expect(out.cumulative_tests).toBe(4);
|
||||
expect(out.new_this_run).toBe(3);
|
||||
});
|
||||
|
||||
it('the corrected alpha only ever gets HARDER as the programme runs', async () => {
|
||||
const cv = require('../../src/services/model/correlateValidator');
|
||||
const store = tl.memoryStore();
|
||||
const a = await tl.recordAndCount(store, [H({ stat: 's', interaction: 'i1' })]);
|
||||
const b = await tl.recordAndCount(store, [H({ stat: 's', interaction: 'i2' })]);
|
||||
const alphaA = cv.VALIDATION_REQUIREMENTS.max_p_value / a.cumulative_tests;
|
||||
const alphaB = cv.VALIDATION_REQUIREMENTS.max_p_value / b.cumulative_tests;
|
||||
expect(alphaB).toBeLessThan(alphaA);
|
||||
});
|
||||
|
||||
it('the key is stable and case/shape insensitive', () => {
|
||||
expect(tl.testKey({ sport: 'MLB', stat: 'Hits', archetype: 'bomber', interaction: 'x', target: 'outcome' }))
|
||||
.toBe(tl.testKey({ sport: 'mlb', stat: 'hits', archetype: 'BOMBER', interaction: 'x', target: 'outcome' }));
|
||||
});
|
||||
|
||||
it('a missing archetype is a real pooled test, keyed distinctly from a per-archetype one', () => {
|
||||
expect(tl.testKey(H({ stat: 'hits', interaction: 'x' })))
|
||||
.not.toBe(tl.testKey(H({ stat: 'hits', archetype: 'BOMBER', interaction: 'x' })));
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user