Ingest defence, and make Bonferroni cumulative across the programme

Two things shipped that stand regardless of sample.

DEFENCE. Statcast Outs Above Average is free on the host we already pull six
feeds from, so there was nothing to decide. 514 fielders, aggregated to team
level -- the unit a batter's prop actually needs, the defence behind the
pitcher he faces -- and persisted as 31 team rows. Verified in production.
Cubs +56 best, Mariners -29 worst.

Unknown is not zero, and it bites unusually hard here: an OAA of 0 is a REAL
reading meaning exactly average, so coercing absence to 0 would assert that
every unmeasured fielder is league-average, which is the commonest defensive
profile there is. team_defense also carries as_of_date in its primary key from
the first row -- statcast_aggregates was built upsert-in-place and that
silently made every backtest leak the games it predicted, so point-in-time is
available here before it is needed rather than after a wrong answer.

A bug worth recording as a class: BASE already ends in /leaderboard, so the
new feed built a doubled path and 404'd. Because a failing feed degrades to an
empty index by design -- correct, so one broken source cannot fail the whole
pull -- it surfaced as "fielding_oaa: 0 rows", which reads exactly like
"Statcast has no fielding data". Graceful degradation makes a wiring bug look
like an honest absence.

CUMULATIVE CORRECTION. Bonferroni had been applied per session throughout: a
run testing eight features corrected by eight. Across a programme's lifetime
that is wrong in the dangerous direction, because every order gets a fresh
generous alpha and the false-positive rate compounds quietly. Correcting by 8
when sixty have been tried is how a noise result eventually gets recorded as
PROVEN with a p-value to point at. The denominator is now distinct hypotheses
ever tested, persisted, and it moved 19 -> 38 within this session alone, alpha
0.0026 -> 0.0013. Re-tests deliberately do not inflate it: re-asking the same
question on more data is not a new shot on goal, and counting it would punish
the discipline of waiting for sample.

THE MEASUREMENT. The differential the theory predicted is present: defence
correlates with the counter's residual at +0.130 for GHOST, the contact and
speed archetype, and -0.018 for BOMBER, the power archetype. A GHOST's hits
depend on whether anyone can range to the ball; a BOMBER's barrels clear the
defence entirely. So a flat BOMBER result is the theory working rather than
the test failing.

It is not a result. GHOST is n=104 against a 500 bar, with p=0.188 against a
corrected alpha of 0.0013 -- three orders of magnitude short. Both are
recorded as CANDIDATE with their measured lift, tagged contact-skill, so the
re-run at full sample compares against a recorded baseline.

Nothing proved, so nothing was recalibrated and nothing shipped.

4,228 tests green (336 suites); web build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-03 22:20:30 -04:00
parent 25e36c0257
commit ece2b9f5f9
6 changed files with 435 additions and 6 deletions
+21
View File
@@ -3,6 +3,27 @@
## Last Updated ## Last Updated
2026-08-03 2026-08-03
## Session 87 (2026-08-03) — Defence ingested; cumulative correction locked ✅
Spec: `specs/defense-ingest-and-cumulative-correction.md`. 4,228 tests / 336
suites green, build exit 0. Counter + clusters byte-identical.
- **DEFENCE INGESTED (free):** Statcast OAA feed → 514 fielders → `team_defense`
(31 teams, dated from row one). Prod-verified: fielding rows 514,
team_defense_written 31. Cubs +56 best, Mariners 29 worst.
- **CUMULATIVE BONFERRONI LOCKED** (`testLedger.js` + `mc_test_ledger`): the
denominator is now distinct hypotheses across the programme lifetime, not the
session. Demonstrated 19 → 38, α 0.0026 → 0.0013. Re-tests don't inflate it.
- **THE PREDICTED DIFFERENTIAL APPEARS:** defence solo r = **+0.130 GHOST**
(contact/speed) vs **0.018 BOMBER** (power). Exactly "defence matters, and for
whom". Both UNDERPOWERED (n=104/245, p=0.188 vs α=0.0013) — signal shape only.
- **Bug class recorded:** the feed 404'd on a doubled `/leaderboard` path and,
because feeds degrade to an empty index by design, reported "0 rows" — which
reads like an honest absence. Any feed reporting 0 is suspect.
- **Nothing proved → nothing recalibrated, nothing shipped.** validatedSkills()
is {} everywhere; proven set still EMPTY.
- **Next:** sample only. GHOST×hits needs ~396 more rows, BOMBER×hits ~213.
Prefer re-testing standing candidates — every new hypothesis now tightens α
for everything after it.
## Session 86 (2026-08-03) — Conditioning registry + a probe so "proven" stops drifting ✅ ## Session 86 (2026-08-03) — Conditioning registry + a probe so "proven" stops drifting ✅
Spec: `specs/conditioning-registry.md`. 4,221 tests / 335 suites green, build exit Spec: `specs/conditioning-registry.md`. 4,221 tests / 335 suites green, build exit
0. Counter + batter model + pitcher engine byte-identical. 0. Counter + batter model + pitcher engine byte-identical.
+33
View File
@@ -1496,6 +1496,39 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
- Within BOMBER, the counter still leads on hits (0.218 vs 0.160) — consistent - Within BOMBER, the counter still leads on hits (0.218 vs 0.160) — consistent
with the closed pooled hits negative. with the closed pooled hits negative.
## Defence ingest + cumulative Bonferroni (Session 87 — non-obvious)
- **BONFERRONI IS NOW CUMULATIVE ACROSS THE PROGRAMME LIFETIME**
(`src/services/model/testLedger.js` + `mc_test_ledger`). Correcting per-session
(8 tests → /8, forever) let the false-positive rate compound silently; the
denominator is now DISTINCT hypotheses ever tested. Demonstrated: 19 → 38 in
one session, α 0.0026 → **0.0013**. RE-TESTS DO NOT INFLATE IT — re-asking the
same question on more data is not a new shot on goal, and counting it would
punish waiting for sample. The alpha only ever shrinks, so prefer re-testing
standing candidates over inventing new hypotheses — that is now mathematically
the disciplined choice.
- **DEFENCE IS INGESTED** — free Statcast OAA (`FEEDS.fielding_oaa`), 514
fielders → `team_defense` (31 teams, dated). Team-level is the right unit (the
defence behind the pitcher faced). `oaa_sum` + `oaa_mean` (mean because a team
with more measured fielders would else look better for being measured more);
<3 fielders → absent. **OAA 0 is a REAL "exactly average" reading** — coercing
absence to 0 asserts every unmeasured fielder is league-average, the commonest
profile there is. `team_defense` carries `as_of_date` in the PK FROM ROW ONE
(the statcast_aggregates lesson, applied before it was needed).
- **`BASE` in statcastAdapter ALREADY ENDS IN `/leaderboard`** — the new feed
doubled it and 404'd. And because a failing feed degrades to an EMPTY index by
design, it surfaced as "fielding_oaa: 0 rows", which reads exactly like
"Statcast has no fielding data". **Graceful degradation makes a wiring bug look
like an honest absence — treat any feed reporting 0 as suspect until the URL is
fetched by hand.**
- **THE DEFENCE DIFFERENTIAL APPEARS AS THEORY PREDICTS:** solo r vs counter
residual is **+0.130 for GHOST** (contact/speed, n=104) and **0.018 for
BOMBER** (power, n=245). A GHOST's hits depend on fielder range; a BOMBER's
barrels clear the defence. **A flat BOMBER result is the theory working, not
the test failing.** Both UNDERPOWERED (p=0.188 vs corrected α 0.0013) — a
signal shape, not a result.
- `team_defense` keys on Savant's DISPLAY NAME (a nickname, "Cubs") while game
logs give full names ("Chicago Cubs") — match on both.
## Active Skills ## Active Skills
- vyndr-voice (all user-facing output) - vyndr-voice (all user-facing output)
- prop-analysis (grading methodology) - prop-analysis (grading methodology)
+50 -6
View File
@@ -199,7 +199,7 @@ async function opponentByPlayerDate(players) {
const SOLO = ['batter_barrel_pct', 'batter_hard_hit_pct', 'batter_exit_velo', const SOLO = ['batter_barrel_pct', 'batter_hard_hit_pct', 'batter_exit_velo',
'batter_launch_angle', 'batter_k_pct', 'batter_bb_pct', 'batter_launch_angle', 'batter_k_pct', 'batter_bb_pct',
'pitcher_k_pct', 'pitcher_hard_hit_allowed', 'pitcher_k_pct', 'pitcher_hard_hit_allowed',
'pitcher_gb_pct', 'pitcher_fb_pct', 'pitcher_breaking_share']; 'pitcher_gb_pct', 'pitcher_fb_pct', 'pitcher_breaking_share', 'team_defense'];
/** /**
* PER-STAT INTERACTION SETS — the total_bases conditioning map RE-WEIGHTED, not * PER-STAT INTERACTION SETS — the total_bases conditioning map RE-WEIGHTED, not
@@ -209,8 +209,8 @@ const SOLO = ['batter_barrel_pct', 'batter_hard_hit_pct', 'batter_exit_velo',
* and RBI through another (does anyone happen to be on base when it does). * and RBI through another (does anyone happen to be on base when it does).
*/ */
const STAT_INTERACTIONS = { const STAT_INTERACTIONS = {
total_bases: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'barrel_x_power_archetype', 'batterK_x_pitcherK', 'launch_x_pitcher_gb', 'barrel_x_breaking_share'], total_bases: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'barrel_x_power_archetype', 'batterK_x_pitcherK', 'launch_x_pitcher_gb', 'barrel_x_breaking_share', 'defense_x_contact', 'defense_x_speed_profile'],
hits: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'batterK_x_pitcherK', 'launch_x_pitcher_gb', 'barrel_x_breaking_share'], hits: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'batterK_x_pitcherK', 'launch_x_pitcher_gb', 'barrel_x_breaking_share', 'defense_x_contact', 'defense_x_speed_profile'],
// HOME RUNS are the purest barrel stat: the ball must be hit hard AND at the // HOME RUNS are the purest barrel stat: the ball must be hit hard AND at the
// right angle, and the pitcher must be the kind who allows that combination. // right angle, and the pitcher must be the kind who allows that combination.
home_runs: ['launch_x_exit_velo', 'barrel_x_power_archetype', 'exitvelo_x_pitcher_suppression'], home_runs: ['launch_x_exit_velo', 'barrel_x_power_archetype', 'exitvelo_x_pitcher_suppression'],
@@ -243,6 +243,18 @@ const INTERACTIONS = [
mechanism: 'ARCHETYPE-CONDITIONAL. Barrels convert to extra bases for hitters whose lane is power; for a speed/contact profile the same barrel rate is a rarer event on a swing built for something else. This is Discipline 2 stated as a testable interaction. NOTE: it is currently UNTESTABLE — statcast rows carry no archetype label, and the barrel-relative proxy is an exact linear function of barrel_pct, so controlling for both components is rank-deficient. It needs a real archetype classification joined in.', mechanism: 'ARCHETYPE-CONDITIONAL. Barrels convert to extra bases for hitters whose lane is power; for a speed/contact profile the same barrel rate is a rarer event on a swing built for something else. This is Discipline 2 stated as a testable interaction. NOTE: it is currently UNTESTABLE — statcast rows carry no archetype label, and the barrel-relative proxy is an exact linear function of barrel_pct, so controlling for both components is rank-deficient. It needs a real archetype classification joined in.',
build: (r) => r.batter_barrel_pct * r.archetype_power, build: (r) => r.batter_barrel_pct * r.archetype_power,
}, },
{
key: 'defense_x_contact',
components: ['team_defense', 'batter_hard_hit_pct'],
mechanism: 'DEFENCE. A ball in play becomes a hit or an out partly by who is standing behind the pitcher. This should matter MOST for hitters whose value is contact that stays in the park, and LEAST for power hitters whose barrels clear the defence entirely — so a DEAD result for BOMBER is not a failure, it is the differential the theory predicts.',
build: (r) => r.team_defense * r.batter_hard_hit_pct,
},
{
key: 'defense_x_speed_profile',
components: ['team_defense', 'batter_launch_angle'],
mechanism: 'DEFENCE x BATTED-BALL PROFILE. A low-launch (ground-ball) hitter puts the ball where fielders range; a high-launch hitter does not. Launch angle stands in for the profile, so defence should condition the ground-ball hitter far more.',
build: (r) => r.team_defense * r.batter_launch_angle,
},
{ {
key: 'launch_x_pitcher_gb', key: 'launch_x_pitcher_gb',
components: ['batter_launch_angle', 'pitcher_gb_pct'], components: ['batter_launch_angle', 'pitcher_gb_pct'],
@@ -356,6 +368,14 @@ async function main() {
console.error(`[debug] sample statcast keys=${JSON.stringify([...batters.keys()].slice(0, 5))}`); console.error(`[debug] sample statcast keys=${JSON.stringify([...batters.keys()].slice(0, 5))}`);
console.error(`[debug] matches in sample=${sampleKeys.filter((k) => batters.has(k)).length}/5`); console.error(`[debug] matches in sample=${sampleKeys.filter((k) => batters.has(k)).length}/5`);
} }
// TEAM DEFENCE — the newly-ingested Statcast OAA, per team, dated.
const defRows = await page(sb, 'team_defense', '*', (q) => q.eq('sport', 'mlb'));
const defByTeam = new Map();
for (const d of defRows) {
const prev = defByTeam.get(d.team);
if (!prev || String(d.as_of_date) > String(prev.as_of_date)) defByTeam.set(d.team, d);
}
const allowed = reg.candidateFeaturesForStat('mlb', STAT); const allowed = reg.candidateFeaturesForStat('mlb', STAT);
const rowsAll = []; const rowsAll = [];
@@ -405,7 +425,17 @@ async function main() {
pitcher_breaking_share: pit && pit.pitch_mix ? breakingShare(pit.pitch_mix) : null, pitcher_breaking_share: pit && pit.pitch_mix ? breakingShare(pit.pitch_mix) : null,
// DEFENSE: NOT DERIVABLE from what we ingest — see the report. Recorded as // DEFENSE: NOT DERIVABLE from what we ingest — see the report. Recorded as
// null rather than proxied by something that is really pitching quality. // null rather than proxied by something that is really pitching quality.
team_defense: null, // DEFENCE behind the pitcher he faces. knownRate: an unmeasured team is
// ABSENT, never league-average — OAA 0 is a real "exactly average" reading
// and the two must stay distinguishable.
team_defense: (() => {
if (!faced) return null;
// team_defense keys on Savant's display name (a nickname, "Cubs"),
// while the game log gives the full name ("Chicago Cubs"). Try both.
const nick = String(faced).split(' ').pop();
const d = defByTeam.get(faced) || defByTeam.get(nick);
return d ? knownNumber(d.oaa_sum) : null;
})(),
}); });
} }
@@ -413,8 +443,21 @@ async function main() {
const archRows = ARCH ? rowsAll.filter((r) => String(r.archetype || '').toUpperCase() === ARCH) : rowsAll; const archRows = ARCH ? rowsAll.filter((r) => String(r.archetype || '').toUpperCase() === ARCH) : rowsAll;
rows.length = 0; rows.push(...archRows); rows.length = 0; rows.push(...archRows);
// Bonferroni denominator = every test in this family (solo + interaction). // ── CUMULATIVE BONFERRONI ─────────────────────────────────────────────
const TESTS = SOLO.length + INTERACTIONS.length; // The denominator is every DISTINCT hypothesis this programme has tested,
// not just this run's. Correcting by 8 in a session that tries 8, forever,
// while the programme as a whole has tried sixty, is how a noise result
// eventually gets recorded as PROVEN with a p-value to point at.
const tl = require('../src/services/model/testLedger');
const store = tl.supabaseStore(sb);
const chosenKeys = new Set(STAT_INTERACTIONS[STAT] || []);
const entries = [
...SOLO.map((f) => ({ sport: 'mlb', stat: STAT, archetype: ARCH, interaction: `solo:${f}`, target: 'counter_residual' })),
...INTERACTIONS.filter((x) => chosenKeys.has(x.key))
.map((x) => ({ sport: 'mlb', stat: STAT, archetype: ARCH, interaction: x.key, target: 'counter_residual' })),
];
const mc = await tl.recordAndCount(store, entries);
const TESTS = mc.cumulative_tests;
// ── STEP 1 — SOLO PASS (the control) ──────────────────────────────────── // ── STEP 1 — SOLO PASS (the control) ────────────────────────────────────
const solo = {}; const solo = {};
@@ -482,6 +525,7 @@ async function main() {
rows_scored: rows.length, rows_scored: rows.length,
gate_spec: cv.VALIDATION_REQUIREMENTS, gate_spec: cv.VALIDATION_REQUIREMENTS,
bonferroni_tests: TESTS, bonferroni_tests: TESTS,
multiple_comparisons: { ...mc, note: 'denominator is DISTINCT hypotheses across the programme lifetime, not this session' },
n_gap_note: `the gate needs ${cv.VALIDATION_REQUIREMENTS.min_historical_instances} rows; this run has ${rows.length}`, n_gap_note: `the gate needs ${cv.VALIDATION_REQUIREMENTS.min_historical_instances} rows; this run has ${rows.length}`,
archetype_coverage: { archetype_coverage: {
labelled: rows.filter((r) => r.archetype).length, labelled: rows.filter((r) => r.archetype).length,
@@ -0,0 +1,138 @@
# DEFENCE INGESTED + CUMULATIVE CORRECTION LOCKED
**2026-08-03.** Challenger-only. Counter, batter model and pitcher engine
byte-identical (verified by diff).
> **Two durable things shipped, and they stand regardless of sample:** the free
> Statcast fielding feed is ingested and persisted (514 fielders, 31 teams,
> verified in production), and Bonferroni is now corrected against the
> **programme's lifetime test count**, not the session's.
>
> **The differential the theory predicted actually appears:** defence correlates
> with the counter's residual for GHOST (contact/speed, **+0.130**) and is flat
> for BOMBER (power, **0.018**). That is "defence matters, and for whom" showing
> up in the data — at n=104 and n=245, so it is a signal shape, not a result.
>
> **Nothing proved. Nothing recalibrated. Nothing shipped.**
---
## 1. Defence ingested (STEP 1)
Statcast Outs Above Average, free, same host as the six feeds already pulled.
```
feed rows 514 fielders
team defence 31 teams
best Cubs oaa_sum +56
worst Mariners oaa_sum 29
prod verified fielding rows 514 · team_defense_written 31
```
Stored per fielder (`oaa`, `runs_prevented`, `success_diff`, position, team) and
aggregated to **team level**, which is the unit a batter's prop needs: the
defence behind the pitcher he faces. Summed OAA is the team's outs converted
above average; the mean rides along because a team with more measured fielders
would otherwise look better merely for being measured more. Under three measured
fielders → **absent**, not thin.
**Unknown is not zero, and it bites unusually hard here.** An OAA of 0 is a REAL
reading meaning *exactly average*. Coercing absence to 0 would assert that every
unmeasured fielder is league-average — the most common defensive profile there
is — which is a fabricated fact wearing the costume of a neutral default. Every
read goes through `knownRate`/`knownNumber`.
**`team_defense` carries `as_of_date` in its primary key from the first row.**
`statcast_aggregates` was built upsert-in-place with a single as-of date, which
silently made every backtest leak the games it was predicting and cost a full
session to discover. Point-in-time is available here *before* it is needed.
### A bug worth recording as a class
The first prod run reported **`fielding_oaa: 0 rows`**. `BASE` already ends in
`/leaderboard`, so the new feed built `.../leaderboard/leaderboard/...` and 404'd.
Because a failing feed **degrades to an empty index by design** — correct, so one
broken source cannot fail the whole mechanism pull — it surfaced as *zero
fielders*, which reads exactly like "Statcast has no fielding data."
**Graceful degradation makes a wiring bug look like an honest absence.** Any feed
reporting 0 should be treated as suspect until the URL is fetched by hand.
## 2. Defence conditioning per archetype (STEP 2)
Cumulative Bonferroni (see §3): denominator **38**, corrected α = **0.0013**.
| archetype | n | **defence solo r** | p | defence × contact (incr) | defence × launch profile (incr) |
|---|---|---|---|---|---|
| **GHOST** (contact/speed) | 104 | **+0.130** | 0.188 | 0.148 | +0.029 |
| **BOMBER** (power) | 245 | **0.018** | 0.782 | +0.030 | +0.025 |
**The differential is the point, and it is present.** Defence carries a signal
against the counter's residual for the contact/speed archetype and essentially
nothing for the power archetype — which is the causal story: a GHOST's hits
depend on whether anyone can range to the ball, while a BOMBER's barrels clear
the defence entirely. **A flat BOMBER result is the theory working, not the test
failing.**
**But neither is a result.** GHOST is n=104 against a 500 bar with p=0.188 against
a corrected α of 0.0013 — three orders of magnitude short. The direction matches
the theory, which is worth carrying forward; it is not worth acting on.
Both recorded in the registry as CANDIDATE with measured lift, tagged
`CONTACT`-skill, so re-running at n≥500 compares against a recorded baseline.
## 3. Cumulative multiple-comparisons correction (STEP 4) — locked
`src/services/model/testLedger.js` + `mc_test_ledger`.
Bonferroni had been applied **per session** throughout: a run testing 8 features
corrected by 8. Across a programme's lifetime that is wrong in the dangerous
direction — every order gets a fresh, generous alpha, so the false-positive rate
compounds quietly. **Correcting by 8 when sixty have been tried is exactly how a
noise result eventually gets recorded as PROVEN, with a p-value to point at.**
The denominator is now the count of **distinct hypotheses ever tested**,
persisted. Demonstrated live this session:
```
GHOST run → cumulative 19 (19 new)
BOMBER run → cumulative 38 (19 new) corrected α: 0.0026 → 0.0013
```
**Re-tests do not inflate it.** Re-running the same hypothesis on more data is
the same question asked again, not a new shot on goal — counting it again would
punish the discipline of waiting for sample, which is the behaviour this
programme depends on. `times_tested` increments; the denominator does not.
**The alpha only ever shrinks**, which is the correct ordering: an interaction
proved late has cleared a genuinely higher bar than one proved on day one,
because by then we have had far more chances to get lucky. Seven tests lock this.
## 4. Ship + registry (STEP 5)
**Nothing proved → nothing recalibrated, nothing shipped.** The counter continues
to grade everything. `validatedSkills()` returns `{}` for every archetype.
## 5. Programme state
| | status |
|---|---|
| proven set | **EMPTY** (`node scripts/proven-status.js`) |
| gate-ready archetype × stat | **none** — BOMBER×hits 287 is closest, short by 213 |
| defence | **ingested**, testable, underpowered |
| cumulative correction | **locked**, α now 0.0013 and falling |
| point-in-time window | `statcast_history` 1 day; `team_defense` dated from row one |
## 6. Next
1. **Sample is still the only constraint.** The cap fix (907 grades/snapshot vs
334) is compounding it; GHOST × hits needs ~396 more rows, BOMBER × hits ~213.
2. **Re-run `scripts/cluster-prove.js` per archetype at n≥500.** The GHOST
defence differential is the single most theory-consistent signal the programme
has produced — it deserves a fair test, and it will get one.
3. **Note the moving bar:** every new hypothesis tightens α for everything that
follows. Prefer re-testing the standing candidates over inventing new ones —
that is now mathematically, not just methodologically, the disciplined choice.
**Not recommended:** reading GHOST's +0.130 as evidence, recalibrating anything,
or adding new hypotheses while the standing ones are unresolved.
+110
View File
@@ -0,0 +1,110 @@
'use strict';
/**
* testLedger — THE CUMULATIVE MULTIPLE-COMPARISONS DENOMINATOR.
*
* Bonferroni has been applied PER SESSION throughout this programme: a run
* testing eight features corrected by eight. Across a programme's lifetime that
* is wrong, and wrong in the dangerous direction. Every order that tests a fresh
* batch gets a fresh, generous alpha, so the false-positive rate compounds
* quietly with each session — and by now this programme has tried dozens of
* hypotheses across batters, pitchers, archetypes and interactions. Correcting
* by 8 when 60 have been tried is precisely how a noise result eventually gets
* recorded as PROVEN, with a p-value to point at.
*
* So the denominator is the count of DISTINCT hypotheses ever tested, persisted.
*
* ── WHY RE-TESTS DO NOT COUNT ────────────────────────────────────────────
* Re-running the same hypothesis on more data is the SAME question asked again,
* not a new shot on goal. Counting it again would punish the discipline of
* waiting for sample — exactly the behaviour this programme depends on. So
* `times_tested` increments while the denominator does not.
*
* ── WHY THIS MAKES THE BAR HARDER, ON PURPOSE ────────────────────────────
* The corrected alpha only ever shrinks. That is the point: it means an
* interaction proved late in the programme has cleared a genuinely higher bar
* than one proved on day one, which is the correct ordering — by then we have
* had many more chances to get lucky.
*
* Storage is injectable so the unit suite never touches the network.
*/
/** Stable identity for a hypothesis. Same question → same key → not re-counted. */
function testKey({ sport, stat, archetype, interaction, target }) {
return [
String(sport || '').toLowerCase(),
String(stat || '').toLowerCase(),
String(archetype || 'ALL').toUpperCase(),
String(interaction || ''),
String(target || 'counter_residual'),
].join('|');
}
/** In-memory store — the default for tests and for any caller without Supabase. */
function memoryStore() {
const rows = new Map();
return {
async record(entry) {
const key = testKey(entry);
const cur = rows.get(key);
if (cur) { cur.times_tested += 1; return { key, isNew: false }; }
rows.set(key, { ...entry, test_key: key, times_tested: 1 });
return { key, isNew: true };
},
async distinctCount() { return rows.size; },
async all() { return [...rows.values()]; },
};
}
/** Supabase-backed store. */
function supabaseStore(sb) {
return {
async record(entry) {
const key = testKey(entry);
const { data } = await sb.from('mc_test_ledger').select('test_key, times_tested').eq('test_key', key).limit(1);
if (data && data.length) {
await sb.from('mc_test_ledger')
.update({ times_tested: data[0].times_tested + 1, last_tested_at: new Date().toISOString() })
.eq('test_key', key);
return { key, isNew: false };
}
await sb.from('mc_test_ledger').insert({
test_key: key,
sport: String(entry.sport || '').toLowerCase(),
stat: entry.stat ? String(entry.stat).toLowerCase() : null,
archetype: entry.archetype ? String(entry.archetype).toUpperCase() : null,
interaction: String(entry.interaction || ''),
target: String(entry.target || 'counter_residual'),
});
return { key, isNew: true };
},
async distinctCount() {
const { count } = await sb.from('mc_test_ledger').select('test_key', { count: 'exact', head: true });
return count || 0;
},
async all() {
const { data } = await sb.from('mc_test_ledger').select('*');
return data || [];
},
};
}
/**
* Record a batch of hypotheses and return the CUMULATIVE distinct count to use
* as the Bonferroni denominator for this run.
*
* The count is taken AFTER recording, so the current batch is included — a run
* must be corrected for the tests it is itself performing, not only for history.
*/
async function recordAndCount(store, entries) {
const results = [];
for (const e of entries) results.push(await store.record(e));
const total = await store.distinctCount();
return {
cumulative_tests: total,
new_this_run: results.filter((r) => r.isNew).length,
repeat_this_run: results.filter((r) => !r.isNew).length,
};
}
module.exports = { testKey, memoryStore, supabaseStore, recordAndCount };
+83
View File
@@ -0,0 +1,83 @@
'use strict';
/**
* The cumulative Bonferroni denominator, asserted as behaviour.
*
* The failure this prevents is silent and slow: correcting by 8 in a session
* that tries 8 hypotheses, forever, while the programme as a whole has tried
* sixty. Nothing looks wrong in any single run.
*/
const tl = require('../../src/services/model/testLedger');
const H = (o) => ({ sport: 'mlb', target: 'counter_residual', ...o });
describe('the denominator counts DISTINCT hypotheses across the lifetime', () => {
it('accumulates across separate runs — it does not reset per session', async () => {
const store = tl.memoryStore();
const run1 = await tl.recordAndCount(store, [
H({ stat: 'hits', archetype: 'BOMBER', interaction: 'barrel_x_park' }),
H({ stat: 'hits', archetype: 'BOMBER', interaction: 'launch_x_gb' }),
]);
expect(run1.cumulative_tests).toBe(2);
const run2 = await tl.recordAndCount(store, [
H({ stat: 'hits', archetype: 'GHOST', interaction: 'defense_x_contact' }),
]);
// A per-session correction would have said 1 here. The whole point is 3.
expect(run2.cumulative_tests).toBe(3);
expect(run2.new_this_run).toBe(1);
});
it('RE-TESTING the same hypothesis does not inflate the denominator', async () => {
const store = tl.memoryStore();
const h = H({ stat: 'hits', archetype: 'BOMBER', interaction: 'barrel_x_park' });
await tl.recordAndCount(store, [h]);
const again = await tl.recordAndCount(store, [h]);
// Waiting for more sample and re-asking is the SAME question, and must not
// be punished — that discipline is what the programme depends on.
expect(again.cumulative_tests).toBe(1);
expect(again.repeat_this_run).toBe(1);
expect(again.new_this_run).toBe(0);
});
it('the current batch IS included — a run is corrected for its own tests', async () => {
const store = tl.memoryStore();
const out = await tl.recordAndCount(store, [
H({ stat: 'rbi', interaction: 'a' }), H({ stat: 'rbi', interaction: 'b' }),
]);
expect(out.cumulative_tests).toBe(2);
});
it('the same interaction on a DIFFERENT stat/archetype/target is a NEW shot on goal', async () => {
const store = tl.memoryStore();
await tl.recordAndCount(store, [H({ stat: 'hits', archetype: 'BOMBER', interaction: 'x' })]);
const out = await tl.recordAndCount(store, [
H({ stat: 'total_bases', archetype: 'BOMBER', interaction: 'x' }), // new stat
H({ stat: 'hits', archetype: 'GHOST', interaction: 'x' }), // new archetype
H({ stat: 'hits', archetype: 'BOMBER', interaction: 'x', target: 'outcome' }), // new target
]);
expect(out.cumulative_tests).toBe(4);
expect(out.new_this_run).toBe(3);
});
it('the corrected alpha only ever gets HARDER as the programme runs', async () => {
const cv = require('../../src/services/model/correlateValidator');
const store = tl.memoryStore();
const a = await tl.recordAndCount(store, [H({ stat: 's', interaction: 'i1' })]);
const b = await tl.recordAndCount(store, [H({ stat: 's', interaction: 'i2' })]);
const alphaA = cv.VALIDATION_REQUIREMENTS.max_p_value / a.cumulative_tests;
const alphaB = cv.VALIDATION_REQUIREMENTS.max_p_value / b.cumulative_tests;
expect(alphaB).toBeLessThan(alphaA);
});
it('the key is stable and case/shape insensitive', () => {
expect(tl.testKey({ sport: 'MLB', stat: 'Hits', archetype: 'bomber', interaction: 'x', target: 'outcome' }))
.toBe(tl.testKey({ sport: 'mlb', stat: 'hits', archetype: 'BOMBER', interaction: 'x', target: 'outcome' }));
});
it('a missing archetype is a real pooled test, keyed distinctly from a per-archetype one', () => {
expect(tl.testKey(H({ stat: 'hits', interaction: 'x' })))
.not.toBe(tl.testKey(H({ stat: 'hits', archetype: 'BOMBER', interaction: 'x' })));
});
});