Ingest defence, and make Bonferroni cumulative across the programme

Two things shipped that stand regardless of sample. DEFENCE. Statcast Outs Above Average is free on the host we already pull six feeds from, so there was nothing to decide. 514 fielders, aggregated to team level -- the unit a batter's prop actually needs, the defence behind the pitcher he faces -- and persisted as 31 team rows. Verified in production. Cubs +56 best, Mariners -29 worst. Unknown is not zero, and it bites unusually hard here: an OAA of 0 is a REAL reading meaning exactly average, so coercing absence to 0 would assert that every unmeasured fielder is league-average, which is the commonest defensive profile there is. team_defense also carries as_of_date in its primary key from the first row -- statcast_aggregates was built upsert-in-place and that silently made every backtest leak the games it predicted, so point-in-time is available here before it is needed rather than after a wrong answer. A bug worth recording as a class: BASE already ends in /leaderboard, so the new feed built a doubled path and 404'd. Because a failing feed degrades to an empty index by design -- correct, so one broken source cannot fail the whole pull -- it surfaced as "fielding_oaa: 0 rows", which reads exactly like "Statcast has no fielding data". Graceful degradation makes a wiring bug look like an honest absence. CUMULATIVE CORRECTION. Bonferroni had been applied per session throughout: a run testing eight features corrected by eight. Across a programme's lifetime that is wrong in the dangerous direction, because every order gets a fresh generous alpha and the false-positive rate compounds quietly. Correcting by 8 when sixty have been tried is how a noise result eventually gets recorded as PROVEN with a p-value to point at. The denominator is now distinct hypotheses ever tested, persisted, and it moved 19 -> 38 within this session alone, alpha 0.0026 -> 0.0013. Re-tests deliberately do not inflate it: re-asking the same question on more data is not a new shot on goal, and counting it would punish the discipline of waiting for sample. THE MEASUREMENT. The differential the theory predicted is present: defence correlates with the counter's residual at +0.130 for GHOST, the contact and speed archetype, and -0.018 for BOMBER, the power archetype. A GHOST's hits depend on whether anyone can range to the ball; a BOMBER's barrels clear the defence entirely. So a flat BOMBER result is the theory working rather than the test failing. It is not a result. GHOST is n=104 against a 500 bar, with p=0.188 against a corrected alpha of 0.0013 -- three orders of magnitude short. Both are recorded as CANDIDATE with their measured lift, tagged contact-skill, so the re-run at full sample compares against a recorded baseline. Nothing proved, so nothing was recalibrated and nothing shipped. 4,228 tests green (336 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-03 22:20:30 -04:00
parent 25e36c0257
commit ece2b9f5f9
6 changed files with 435 additions and 6 deletions
@@ -0,0 +1,110 @@
+'use strict';
+
+/**
+ * testLedger — THE CUMULATIVE MULTIPLE-COMPARISONS DENOMINATOR.
+ *
+ * Bonferroni has been applied PER SESSION throughout this programme: a run
+ * testing eight features corrected by eight. Across a programme's lifetime that
+ * is wrong, and wrong in the dangerous direction. Every order that tests a fresh
+ * batch gets a fresh, generous alpha, so the false-positive rate compounds
+ * quietly with each session — and by now this programme has tried dozens of
+ * hypotheses across batters, pitchers, archetypes and interactions. Correcting
+ * by 8 when 60 have been tried is precisely how a noise result eventually gets
+ * recorded as PROVEN, with a p-value to point at.
+ *
+ * So the denominator is the count of DISTINCT hypotheses ever tested, persisted.
+ *
+ * ── WHY RE-TESTS DO NOT COUNT ────────────────────────────────────────────
+ * Re-running the same hypothesis on more data is the SAME question asked again,
+ * not a new shot on goal. Counting it again would punish the discipline of
+ * waiting for sample — exactly the behaviour this programme depends on. So
+ * `times_tested` increments while the denominator does not.
+ *
+ * ── WHY THIS MAKES THE BAR HARDER, ON PURPOSE ────────────────────────────
+ * The corrected alpha only ever shrinks. That is the point: it means an
+ * interaction proved late in the programme has cleared a genuinely higher bar
+ * than one proved on day one, which is the correct ordering — by then we have
+ * had many more chances to get lucky.
+ *
+ * Storage is injectable so the unit suite never touches the network.
+ */
+
+/** Stable identity for a hypothesis. Same question → same key → not re-counted. */
+function testKey({ sport, stat, archetype, interaction, target }) {
+  return [
+    String(sport || '').toLowerCase(),
+    String(stat || '').toLowerCase(),
+    String(archetype || 'ALL').toUpperCase(),
+    String(interaction || ''),
+    String(target || 'counter_residual'),
+  ].join('|');
+}
+
+/** In-memory store — the default for tests and for any caller without Supabase. */
+function memoryStore() {
+  const rows = new Map();
+  return {
+    async record(entry) {
+      const key = testKey(entry);
+      const cur = rows.get(key);
+      if (cur) { cur.times_tested += 1; return { key, isNew: false }; }
+      rows.set(key, { ...entry, test_key: key, times_tested: 1 });
+      return { key, isNew: true };
+    },
+    async distinctCount() { return rows.size; },
+    async all() { return [...rows.values()]; },
+  };
+}
+
+/** Supabase-backed store. */
+function supabaseStore(sb) {
+  return {
+    async record(entry) {
+      const key = testKey(entry);
+      const { data } = await sb.from('mc_test_ledger').select('test_key, times_tested').eq('test_key', key).limit(1);
+      if (data && data.length) {
+        await sb.from('mc_test_ledger')
+          .update({ times_tested: data[0].times_tested + 1, last_tested_at: new Date().toISOString() })
+          .eq('test_key', key);
+        return { key, isNew: false };
+      }
+      await sb.from('mc_test_ledger').insert({
+        test_key: key,
+        sport: String(entry.sport || '').toLowerCase(),
+        stat: entry.stat ? String(entry.stat).toLowerCase() : null,
+        archetype: entry.archetype ? String(entry.archetype).toUpperCase() : null,
+        interaction: String(entry.interaction || ''),
+        target: String(entry.target || 'counter_residual'),
+      });
+      return { key, isNew: true };
+    },
+    async distinctCount() {
+      const { count } = await sb.from('mc_test_ledger').select('test_key', { count: 'exact', head: true });
+      return count || 0;
+    },
+    async all() {
+      const { data } = await sb.from('mc_test_ledger').select('*');
+      return data || [];
+    },
+  };
+}
+
+/**
+ * Record a batch of hypotheses and return the CUMULATIVE distinct count to use
+ * as the Bonferroni denominator for this run.
+ *
+ * The count is taken AFTER recording, so the current batch is included — a run
+ * must be corrected for the tests it is itself performing, not only for history.
+ */
+async function recordAndCount(store, entries) {
+  const results = [];
+  for (const e of entries) results.push(await store.record(e));
+  const total = await store.distinctCount();
+  return {
+    cumulative_tests: total,
+    new_this_run: results.filter((r) => r.isNew).length,
+    repeat_this_run: results.filter((r) => !r.isNew).length,
+  };
+}
+
+module.exports = { testKey, memoryStore, supabaseStore, recordAndCount };