diff --git a/BUILD-STATE.md b/BUILD-STATE.md index efd565f..6f49981 100755 --- a/BUILD-STATE.md +++ b/BUILD-STATE.md @@ -3,6 +3,28 @@ ## Last Updated 2026-08-03 +## Session 83 (2026-08-03) — Batter cluster measured; the proven set is EMPTY ✅ +Spec: `specs/batter-cluster-prove.md`. 4,204 tests / 334 suites green, build exit +0. skillProjection byte-identical (TB frozen, verified by diff); counter untouched. +- **PREMISE CORRECTED: total_bases has NOT passed BAR 1.** It is inconclusive at + parity (CI includes zero) and contaminated. Installing it as the "proven + reference" would make the cluster's bar "be inconclusive at parity". +- **HITS CLOSED — well-powered negative.** n=803 CLEARS the gate sample bar, so + features were tested not refused: max |r| 0.053, interactions ≈0, head-to-head + −0.096 CI [−0.165,−0.029]. +- **Others n-blocked:** TB 383, rbi 391, HR 228, runs 188. Leads: home_runs + barrel r=−0.135 (negative = a correction, not a predictor); runs K×K + incremental +0.132 (largest in cluster). +- **RBI is half-unmodellable** — power × opportunity, and baserunner state is not + ingested at all. +- **statcast_history retention LIVE + verified in prod** (1,387 rows, as_of + 2026-08-03). First run failed on a drifted hand-written schema; table now + mirrors the source via LIKE. Usable point-in-time window starts 2026-08-04. +- **Stage B has nothing to calibrate.** Proven set is empty. +- **Next (waiting, not building):** let history accrue a week + TB/rbi reach + n>=500, then re-run `scripts/cluster-prove.js`. Ranked: TB → rbi (needs + baserunner state) → HR → runs. Do not re-run hits. + ## Session 82 (2026-08-03) — TB solo+interactions; point-in-time validation unblocked ✅ Spec: `specs/tb-solo-and-interactions.md`. 4,204 tests / 334 suites green, build exit 0, counter byte-identical. diff --git a/CLAUDE.md b/CLAUDE.md index df4371e..43218cf 100755 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1375,6 +1375,36 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section). deterministic bases-per-hit made P(TB>=2) EXACTLY P(hits>=1); non-degeneracy is locked by test. +## Batter cluster + the bar (Session 83 — non-obvious) +- **total_bases has NOT passed BAR 1.** Its head-to-head is INCONCLUSIVE at + parity (delta +0.004..+0.007, CI includes zero) and CONTAMINATED. It is + frozen, but frozen as an *inconclusive* model — do NOT install it as "the + proven reference standard", because then the bar other stats must clear + becomes "be inconclusive at parity", which admits everything on a null result. + **The proven set is EMPTY.** +- **HITS IS CLOSED — a well-powered negative.** At n=803 it CLEARS the gate's + sample bar, so its features were properly TESTED, not refused: max marginal + |r| = 0.053 vs the 0.15 bar, every interaction's incremental ≈ 0, and the + model loses head-to-head −0.096 with CI [−0.165,−0.029]. Don't re-run hits. +- **Everything else is n-blocked:** TB 383, rbi 391, HR 228, runs 188 (gate needs + 500). Two leads worth carrying: `home_runs · barrel_pct` marginal r = **−0.135** + (NEGATIVE — higher barrel goes with the counter OVER-predicting, i.e. a + correction not a predictor), and `runs · batterK×pitcherK` incremental **+0.132** + (largest in the cluster; mechanism = strikeouts destroy PA, and a PA that never + happens cannot score). +- **RBI is half-unmodellable today:** it is power × OPPORTUNITY and we ingest NO + baserunner state. A weak RBI result is evidence we model half the stat, not + that skill inputs fail for RBI. +- **`statcast_history` retention is LIVE and verified in prod** (1,387 rows, + as_of 2026-08-03). Two gotchas: the first run failed on a drifted hand-written + schema (`swing_pct` missing) — the table is now `create ... (like + statcast_aggregates)` and the writer passes rows through whole; and the + refresh still succeeded during that failure, confirming the best-effort guard. + A usable point-in-time WINDOW starts 2026-08-04 (as_of < game_date). +- **`scripts/cluster-prove.js`** runs the whole both-ways program for any stat via + `CLUSTER_STAT=`. Per-stat interaction sets are the TB map RE-WEIGHTED, never + copied — reuse speeds the search and grants no pass. + ## Active Skills - vyndr-voice (all user-facing output) - prop-analysis (grading methodology) diff --git a/scripts/cluster-prove.js b/scripts/cluster-prove.js new file mode 100644 index 0000000..01f914f --- /dev/null +++ b/scripts/cluster-prove.js @@ -0,0 +1,448 @@ +#!/usr/bin/env node +'use strict'; + +/** + * tb-solo-and-interactions — PROVE BOTH, with the solo pass as the control. + * + * A feature can carry signal alone, only in combination, or both. Testing only + * interactions misses solo-real features AND cannot tell whether an interaction + * ADDS anything or merely re-encodes its own parts. So the solo result is the + * baseline every interaction has to beat. + * + * ── HOW "ADDS OVER ITS PARTS" IS MEASURED ──────────────────────────────── + * Not by comparing two correlations by eye. The interaction's incremental + * signal is the PARTIAL correlation of the interaction term with the counter's + * residual, CONTROLLING FOR both component features: + * + * resid_I = I − OLS(I ~ A, B) + * resid_Y = Y − OLS(Y ~ A, B) + * incremental r = corr(resid_I, resid_Y) + * + * If the interaction is just barrel-rate wearing a different hat, regressing out + * barrel rate removes it and the incremental r collapses to ~0. That is exactly + * the redundancy the order is guarding against, and it is the difference between + * PASSES-AND-ADDS and PASSES-BUT-REDUNDANT. + * + * ── WHY THE COUNTER'S RESIDUAL IS THE TARGET ───────────────────────────── + * Correlating with the raw outcome rewards a feature for knowing what the + * counter already knows. Only the part the counter MISSES is new information, + * and only new information can improve the product. Both are reported; the + * residual one is the one that decides. + * + * ── THEORY FIRST ───────────────────────────────────────────────────────── + * Every interaction below is declared with a MECHANISM before it is measured. + * No blind pairwise search — with 8 features there are 28 pairs, and at α=.05 + * roughly one in twenty returns "significant" from noise alone. + * + * SUPABASE_URL=... node scripts/tb-solo-and-interactions.js + */ + +require('dotenv').config(); +const { createClient } = require('@supabase/supabase-js'); +const cv = require('../src/services/model/correlateValidator'); +const sk = require('../src/services/model/skillProjection'); +const reg = require('../src/services/model/featureRegistry'); +const mlb = require('../src/services/adapters/mlbStatsAdapter'); +const { knownRate, knownNumber } = require('../src/utils/known'); + +const SB_URL = process.env.SUPABASE_URL; +const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY; +const PAGE = 1000; +const GAMES_SO_FAR = Number(process.env.STAGEA_GAMES_SO_FAR || 103); +const STAT = process.env.CLUSTER_STAT || 'total_bases'; + +const r4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000); +const mean = (a) => (a.length ? a.reduce((x, y) => x + y, 0) / a.length : null); +const brier = (ps, ys) => (ps.length ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null); + +/** OLS residuals of y on the given predictor columns (with intercept). */ +function olsResiduals(y, Xcols) { + const n = y.length; + const p = Xcols.length + 1; + const X = []; + for (let i = 0; i < n; i += 1) { + const row = [1]; + for (const c of Xcols) row.push(c[i]); + X.push(row); + } + // Normal equations (X'X) b = X'y, solved by Gauss-Jordan. p is 2-4 here. + const XtX = Array.from({ length: p }, () => new Array(p).fill(0)); + const Xty = new Array(p).fill(0); + for (let i = 0; i < n; i += 1) { + for (let a = 0; a < p; a += 1) { + Xty[a] += X[i][a] * y[i]; + for (let b = 0; b < p; b += 1) XtX[a][b] += X[i][a] * X[i][b]; + } + } + const M = XtX.map((row, i) => [...row, Xty[i]]); + for (let col = 0; col < p; col += 1) { + let piv = col; + for (let r = col + 1; r < p; r += 1) if (Math.abs(M[r][col]) > Math.abs(M[piv][col])) piv = r; + if (Math.abs(M[piv][col]) < 1e-12) return null; // singular → cannot control honestly + [M[col], M[piv]] = [M[piv], M[col]]; + const d = M[col][col]; + for (let k = col; k <= p; k += 1) M[col][k] /= d; + for (let r = 0; r < p; r += 1) { + if (r === col) continue; + const f = M[r][col]; + for (let k = col; k <= p; k += 1) M[r][k] -= f * M[col][k]; + } + } + const beta = M.map((row) => row[p]); + return y.map((v, i) => v - X[i].reduce((s, xv, j) => s + xv * beta[j], 0)); +} + +/** + * Partial correlation of a with b, controlling for the columns in ctrl. + * + * COLLINEARITY IS CHECKED FIRST, and this is not pedantry — it caught a real + * error in this very script. The archetype-power proxy was defined as + * `barrel_pct / LEAGUE.barrel_pct`, an exact linear function of barrel_pct, so + * "control for both components" was rank-deficient and the partial correlation + * it produced (-0.132, the only one that looked like an incremental finding) was + * an artifact of a singular design matrix. The Gauss-Jordan pivot test missed it + * because the two columns differ by a scale factor, which keeps the pivot well + * above an absolute epsilon. Scale-free pairwise correlation catches it. + */ +function partialCorr(a, b, ctrl) { + for (let i = 0; i < ctrl.length; i += 1) { + for (let j = i + 1; j < ctrl.length; j += 1) { + const rr = cv.pearson(ctrl[i], ctrl[j]).r; + if (rr !== null && Math.abs(rr) > 0.999) return null; // same variable twice + } + } + const ra = olsResiduals(a, ctrl); + const rb = olsResiduals(b, ctrl); + if (!ra || !rb) return null; + return cv.pearson(ra, rb).r; +} + +/** Rows where every named key is known — the honest common sample. */ +function completeRows(rows, keys) { + return rows.filter((r) => keys.every((k) => knownNumber(r[k]) !== null)); +} + +function makeRnd(seed) { + let s = seed >>> 0; + return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; }; +} +function bootstrapDiff(rows, keyA, keyB, iters = 4000, seed = 20260804) { + if (rows.length < 30) return null; + const rnd = makeRnd(seed); + const n = rows.length; + const diffs = []; + for (let it = 0; it < iters; it += 1) { + const ys = []; const a = []; const b = []; + for (let i = 0; i < n; i += 1) { + const r = rows[Math.floor(rnd() * n)]; + ys.push(r.won); a.push(r[keyA]); b.push(r[keyB]); + } + const ca = cv.pearson(a, ys).r; const cb = cv.pearson(b, ys).r; + if (ca == null || cb == null) continue; + diffs.push(ca - cb); + } + if (diffs.length < 100) return null; + diffs.sort((x, y) => x - y); + const q = (pp) => r4(diffs[Math.floor(pp * (diffs.length - 1))]); + const ci = [q(0.025), q(0.975)]; + return { + point: r4(cv.pearson(rows.map((r) => r[keyA]), rows.map((r) => r.won)).r + - cv.pearson(rows.map((r) => r[keyB]), rows.map((r) => r.won)).r), + ci95: ci, ci_excludes_zero: ci[0] > 0 || ci[1] < 0, + }; +} + +async function page(sb, table, select, apply) { + const out = []; + for (let from = 0; ; from += PAGE) { + const { data, error } = await apply(sb.from(table).select(select)).range(from, from + PAGE - 1); + if (error) throw error; + if (!data || data.length === 0) break; + out.push(...data); + if (data.length < PAGE) break; + } + return out; +} + +async function opposingStarters(dates) { + const m = new Map(); + for (const d of dates) { + let games = []; + try { games = await mlb.getScheduleWithPitchers(d); } catch { games = []; } + for (const g of games) { + if (!g.home || !g.away) continue; + if (g.home.probablePitcher) m.set(`${d}|OPP:${g.home.team}`, g.home.probablePitcher.id); + if (g.away.probablePitcher) m.set(`${d}|OPP:${g.away.team}`, g.away.probablePitcher.id); + } + } + return m; +} +async function opponentByPlayerDate(players) { + const map = new Map(); + for (const [key, name] of players) { + try { + const found = await mlb.searchPlayer(name); + if (!found || !found.id) continue; + const log = await mlb.getPlayerGameLog(found.id); + for (const g of log || []) { + if (g && g.date && g.opponent) map.set(`${key}|${String(g.date).slice(0, 10)}`, g.opponent); + } + } catch { /* no log → no pitcher */ } + } + return map; +} + +const SOLO = ['batter_barrel_pct', 'batter_hard_hit_pct', 'batter_exit_velo', + 'batter_launch_angle', 'batter_k_pct', 'batter_bb_pct', + 'pitcher_k_pct', 'pitcher_hard_hit_allowed']; + +/** + * PER-STAT INTERACTION SETS — the total_bases conditioning map RE-WEIGHTED, not + * copied. Reuse speeds the search; it grants nothing. Each stat's features must + * independently earn their place FOR THAT STAT, and the mechanisms genuinely + * differ: barrel rate drives home runs through one channel (does the ball leave) + * and RBI through another (does anyone happen to be on base when it does). + */ +const STAT_INTERACTIONS = { + total_bases: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'barrel_x_power_archetype', 'batterK_x_pitcherK'], + hits: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'batterK_x_pitcherK'], + // HOME RUNS are the purest barrel stat: the ball must be hit hard AND at the + // right angle, and the pitcher must be the kind who allows that combination. + home_runs: ['launch_x_exit_velo', 'barrel_x_power_archetype', 'exitvelo_x_pitcher_suppression'], + // RBI is a POWER x OPPORTUNITY stat — a solo home run drives in one, the same + // swing with two on drives in three. We do not ingest baserunner state, so the + // opportunity half is genuinely missing and that is reported, not papered over. + rbi: ['barrel_x_power_archetype', 'exitvelo_x_pitcher_suppression', 'batterK_x_pitcherK'], + // RUNS scored is ON-BASE x what happens AFTER — mostly teammate-driven, which + // is the least self-contained stat in the cluster. + runs: ['batterK_x_pitcherK', 'exitvelo_x_pitcher_suppression'], +}; + +/** STEP 2 — theory first. Every interaction declares its mechanism. */ +const INTERACTIONS = [ + { + key: 'launch_x_exit_velo', + components: ['batter_launch_angle', 'batter_exit_velo'], + mechanism: 'Extra bases need BOTH conditions: hit hard AND hit in the air. A 105-mph ground ball is an out; a 25-degree popup is an out. Neither factor alone predicts bases, which is precisely why each may fail solo and the product may not.', + build: (r) => r.batter_launch_angle * r.batter_exit_velo, + }, + { + key: 'exitvelo_x_pitcher_suppression', + components: ['batter_exit_velo', 'pitcher_hard_hit_allowed'], + mechanism: 'A hitter only realises his contact quality against a pitcher who permits contact quality. Elite suppression should attenuate a power bat; a contact-permitting arm should amplify it. The effect is conditional by construction.', + build: (r) => r.batter_exit_velo * r.pitcher_hard_hit_allowed, + }, + { + key: 'barrel_x_power_archetype', + components: ['batter_barrel_pct', 'archetype_power'], + mechanism: 'ARCHETYPE-CONDITIONAL. Barrels convert to extra bases for hitters whose lane is power; for a speed/contact profile the same barrel rate is a rarer event on a swing built for something else. This is Discipline 2 stated as a testable interaction. NOTE: it is currently UNTESTABLE — statcast rows carry no archetype label, and the barrel-relative proxy is an exact linear function of barrel_pct, so controlling for both components is rank-deficient. It needs a real archetype classification joined in.', + build: (r) => r.batter_barrel_pct * r.archetype_power, + }, + { + key: 'batterK_x_pitcherK', + components: ['batter_k_pct', 'pitcher_k_pct'], + mechanism: 'Strikeout risk compounds multiplicatively (log5 is exactly this shape). A high-K bat against a high-K arm loses plate appearances to strikeouts, and a PA lost is a base opportunity that never happens — so it suppresses total bases through OPPORTUNITY, not contact quality.', + build: (r) => r.batter_k_pct * r.pitcher_k_pct, + }, +]; + +/** Latest settled game date in the pull — used to detect that the profile + * freeze now sits AFTER the data, i.e. no clean out-of-sample window exists. */ +function clean0Max(rows) { + return (rows || []).reduce((mx, r) => (String(r.game_date) > mx ? String(r.game_date) : mx), ''); +} + +async function main() { + if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required'); + const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } }); + + const statcast = await page(sb, 'statcast_aggregates', '*', (q) => q.eq('sport', 'mlb')); + const freezeDate = statcast.reduce((mx, r) => (String(r.updated_at) > mx ? String(r.updated_at) : mx), '').slice(0, 10); + const batters = new Map(); const pitchersById = new Map(); + for (const r of statcast) { + if (r.role === 'pitcher' && r.source_id != null) pitchersById.set(Number(r.source_id), sk.fromStatcastRow(r)); + if (r.player_key && r.role === 'batter') { + const prev = batters.get(r.player_key); + if (!prev || Number(r.sample_pa || 0) > Number(prev.rawPa || 0)) { + batters.set(r.player_key, Object.assign(sk.fromStatcastRow(r), { rawPa: Number(r.sample_pa || 0) })); + } + } + } + + // REAL ARCHETYPE LABELS. The barrel-relative proxy was a clipped monotone + // transform of barrel_pct, so `barrel x proxy` measured NONLINEARITY IN BARREL, + // not an archetype interaction — it could never have tested Discipline 2. + // model_snapshots carries the actual classification per prop, so the + // conditioning variable is now a genuine BOMBER indicator, which is + // categorical and therefore not a transform of barrel at all. + const snaps = await page(sb, 'model_snapshots', 'player_key, game_date, archetype', + (q) => q.eq('sport', 'mlb').eq('stat', STAT).not('archetype', 'is', null)); + const archetypeBy = new Map(); + for (const r of snaps) if (r.player_key && r.game_date) archetypeBy.set(`${r.player_key}|${r.game_date}`, r.archetype); + + const led = await page(sb, 'ledger_entries', + 'player_key, player_name, stat, line, side, outcome, game_date, p_win, quarantine_reason', + (q) => q.eq('sport', 'mlb').is('user_id', null).eq('stat', STAT) + .in('outcome', ['hit', 'miss']).not('p_win', 'is', null)); + // POINT-IN-TIME IS NO LONGER AVAILABLE FROM THIS TABLE. + // + // `statcast_aggregates` is upserted in place and keeps one as-of date. The + // first skill backtest was honest only by accident: the nightly refresh was + // unreachable code, so the table sat frozen at 2026-07-21 — BEFORE the settled + // window. Repairing that cron (correct for production) refreshed it to today, + // and every prior version is gone. + // + // So scoring a 2026-07-25 game now uses a season aggregate that CONTAINS that + // game. `statcast_history` (added this session) fixes it going forward; it has + // one day of data, which is not yet a window. Until it fills, results here are + // DIRECTIONAL AND CONTAMINATED, labelled as such, and are NOT gate verdicts. + const contaminated = String(freezeDate) >= String(clean0Max(led)); + const clean = led.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book') + && (contaminated ? true : String(r.game_date) > freezeDate)); + + const dates = [...new Set(clean.map((r) => r.game_date))].sort(); + const starters = await opposingStarters(dates); + const players = new Map(); + for (const r of clean) if (!players.has(r.player_key)) players.set(r.player_key, r.player_name); + const oppByPlayerDate = await opponentByPlayerDate(players); + + if (process.env.TB_DEBUG === '1') { + console.error(`[debug] statcast rows=${statcast.length} freeze=${freezeDate} batters=${batters.size} pitchers=${pitchersById.size}`); + console.error(`[debug] ledger tb rows=${led.length} clean(after freeze)=${clean.length}`); + const sampleKeys = clean.slice(0, 5).map((r) => r.player_key); + console.error(`[debug] sample ledger player_keys=${JSON.stringify(sampleKeys)}`); + console.error(`[debug] sample statcast keys=${JSON.stringify([...batters.keys()].slice(0, 5))}`); + console.error(`[debug] matches in sample=${sampleKeys.filter((k) => batters.has(k)).length}/5`); + } + const allowed = reg.candidateFeaturesForStat('mlb', STAT); + const rows = []; + for (const r of clean) { + const bat = batters.get(r.player_key); + if (!bat) continue; + const faced = oppByPlayerDate.get(`${r.player_key}|${r.game_date}`) || null; + const pit = faced ? pitchersById.get(Number(starters.get(`${r.game_date}|OPP:${faced}`))) || null : null; + const paRate = bat.rawPa > 0 ? Math.min(5.2, Math.max(2.0, bat.rawPa / GAMES_SO_FAR)) : null; + const under = String(r.side).toLowerCase() === 'under'; + const won = r.outcome === 'hit' ? 1 : 0; + const champ = Number(r.p_win); + + const proj = sk.projectSkill({ + batter: bat, pitcher: pit, park: 1, archetype: null, + statType: STAT, line: Number(r.line), expectedPa: paRate, allowed, + }); + + // The REAL archetype for this prop — a 0/1 power indicator, categorical and + // independent of barrel_pct by construction. + const arch = archetypeBy.get(`${r.player_key}|${r.game_date}`) || null; + const archetypePower = arch == null ? null : (String(arch).toUpperCase() === 'BOMBER' ? 1 : 0); + + rows.push({ + won, champ, residual: won - champ, + skill: proj ? (under ? 1 - proj.p_over_line : proj.p_over_line) : null, + had_pitcher: !!pit, + batter_barrel_pct: knownRate(bat.barrel_pct), + batter_hard_hit_pct: knownRate(bat.hard_hit_pct), + batter_exit_velo: knownRate(bat.avg_exit_velo), + batter_launch_angle: knownRate(bat.avg_launch_angle), + batter_k_pct: knownRate(bat.k_pct), + batter_bb_pct: knownRate(bat.bb_pct), + pitcher_k_pct: pit ? knownRate(pit.k_pct) : null, + pitcher_hard_hit_allowed: pit ? knownRate(pit.hard_hit_pct) : null, + archetype_power: archetypePower, + archetype: arch, + }); + } + + // Bonferroni denominator = every test in this family (solo + interaction). + const TESTS = SOLO.length + INTERACTIONS.length; + + // ── STEP 1 — SOLO PASS (the control) ──────────────────────────────────── + const solo = {}; + for (const f of SOLO) { + const rs = completeRows(rows, [f]); + solo[f] = { + n: rs.length, + vs_outcome: cv.validateFactor(rs.map((r) => r[f]), rs.map((r) => r.won), TESTS), + vs_counter_residual: cv.validateFactor(rs.map((r) => r[f]), rs.map((r) => r.residual), TESTS), + }; + } + + // ── STEP 3 — INTERACTIONS, each against its own solo baseline ─────────── + const interactions = {}; + const chosen = new Set(STAT_INTERACTIONS[STAT] || []); + for (const ix of INTERACTIONS.filter((x) => chosen.has(x.key))) { + const keys = [...ix.components]; + const rs = completeRows(rows, keys); + if (rs.length < 30) { interactions[ix.key] = { mechanism: ix.mechanism, n: rs.length, verdict: 'UNTESTABLE — no common sample' }; continue; } + const I = rs.map(ix.build); + const Y = rs.map((r) => r.residual); + const ctrl = keys.map((k) => rs.map((r) => r[k])); + + const gate = cv.validateFactor(I, Y, TESTS); + const incremental = partialCorr(I, Y, ctrl); + // The best solo |r| among its own components, on the SAME rows. + const componentSolo = keys.map((k) => ({ + feature: k, r: r4(cv.pearson(rs.map((r) => r[k]), Y).r), + })); + const bestComponent = Math.max(...componentSolo.map((c) => Math.abs(c.r ?? 0))); + + let verdict; + if (incremental === null) verdict = 'UNTESTABLE — controls are collinear'; + else if (gate.validated && Math.abs(incremental) >= cv.VALIDATION_REQUIREMENTS.min_pearson_r) verdict = 'PASSES-AND-ADDS'; + else if (gate.validated) verdict = 'PASSES-BUT-REDUNDANT'; + else if (rs.length < cv.VALIDATION_REQUIREMENTS.min_historical_instances) verdict = 'UNDERPOWERED — n below the gate'; + else verdict = 'FAILS'; + + interactions[ix.key] = { + mechanism: ix.mechanism, + components: keys, + n: rs.length, + raw_r_vs_residual: gate.pearson_r, + gate: { validated: gate.validated, reason: gate.reason, p_value: gate.p_value, corrected_alpha: gate.corrected_alpha, underpowered: !!gate.underpowered }, + component_solo_r_same_rows: componentSolo, + best_component_abs_r: r4(bestComponent), + INCREMENTAL_partial_r: r4(incremental), + adds_over_components: incremental !== null && Math.abs(incremental) > bestComponent, + verdict, + }; + } + + // ── STEP 4 — COMBINED vs COUNTER (valid at this n; the gate is not) ───── + const h2h = rows.filter((r) => r.skill != null); + const ys = h2h.map((r) => r.won); + const bs = bootstrapDiff(h2h, 'skill', 'champ'); + + console.log(JSON.stringify({ + stat: STAT, + VALIDITY: contaminated + ? 'CONTAMINATED / DIRECTIONAL ONLY — statcast_aggregates now carries a single as-of date (' + freezeDate + ') that is AFTER the settled games, so season profiles contain the games being predicted. These are NOT gate verdicts. statcast_history (new) makes point-in-time possible from tomorrow.' + : `CLEAN out-of-sample: profiles frozen ${freezeDate}; only game_date > ${freezeDate} scored`, + contaminated, + rows_scored: rows.length, + gate_spec: cv.VALIDATION_REQUIREMENTS, + bonferroni_tests: TESTS, + n_gap_note: `the gate needs ${cv.VALIDATION_REQUIREMENTS.min_historical_instances} rows; this run has ${rows.length}`, + archetype_coverage: { + labelled: rows.filter((r) => r.archetype).length, + bomber: rows.filter((r) => r.archetype_power === 1).length, + other: rows.filter((r) => r.archetype_power === 0).length, + }, + step1_solo_baseline: solo, + step3_interactions: interactions, + step4_combined_vs_counter: { + n: h2h.length, + pitcher_coverage: r4(mean(h2h.map((r) => (r.had_pitcher ? 1 : 0)))), + base_rate: r4(mean(ys)), + resolution: { skill_tb: r4(cv.pearson(h2h.map((r) => r.skill), ys).r), counter: r4(cv.pearson(h2h.map((r) => r.champ), ys).r) }, + brier: { skill_tb: r4(brier(h2h.map((r) => r.skill), ys)), counter: r4(brier(h2h.map((r) => r.champ), ys)) }, + delta: bs, + verdict: !bs ? 'N-BLOCKED' + : (bs.ci_excludes_zero && bs.point > 0) ? 'SKILL TB BEATS THE COUNTER' + : (bs.ci_excludes_zero && bs.point < 0) ? 'LOSES to the counter' + : 'INCONCLUSIVE', + }, + }, null, 2)); + process.exit(0); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/specs/batter-cluster-prove.md b/specs/batter-cluster-prove.md new file mode 100644 index 0000000..9da2cf7 --- /dev/null +++ b/specs/batter-cluster-prove.md @@ -0,0 +1,126 @@ +# THE BATTER CLUSTER — both-ways results, and what the bar actually is + +**2026-08-03.** Challenger-only. Counter byte-identical. `skillProjection` +byte-identical (frozen) — verified by diff against the prior commit. + +> **PREMISE CORRECTION, and it decides the whole order: total_bases has NOT +> passed BAR 1.** Yesterday's result was **delta +0.0071, CI95 +> [−0.0648, +0.0789] — inconclusive, at parity, and contaminated**. No feature +> passed the gate. It was described as "the first challenger that did not LOSE", +> which is not the same as proven. +> +> This matters structurally: if total_bases is installed as "the frozen proven +> reference" and every other stat is held to "the identical bar total_bases +> cleared", **the bar becomes "be inconclusive at parity"** — a standard that +> proves nothing and would admit the entire cluster on a null result. +> +> **The proven set is EMPTY.** That is the honest Stage-B input. + +--- + +## 0. Two orders running, one pattern + +This is the second consecutive order whose premise promoted a null result to a +pass (the previous one had "barrel rate PASSED solo" when every TB feature was +refused on sample size). Flagging the pattern once, without labouring it: the +measurements are being read at their most favourable interpretation somewhere +between sessions. The numbers below are stated so that cannot happen again. + +## 1. Validity: still contaminated today, but the clock has started + +`statcast_history` was **empty** — the retention shipped *after* yesterday's +refresh had already run. Two things fixed this session: + +1. The first prod run failed with `Could not find the 'swing_pct' column` — the + hand-enumerated schema had drifted from the table it was copying. **The + refresh still succeeded and wrote all 1,387 aggregate rows**, which verified + the best-effort guard in production: a retention failure does not fail the + refresh. +2. The table is now created `LIKE statcast_aggregates` and the writer passes the + row through whole, so there is no drift surface left. + +**Verified live: `history_retained: 1387, as_of 2026-08-03`.** The point-in-time +clock is running. It does not yet give a *window* — `as_of 2026-08-03` can only +score games from 2026-08-04 — so **everything below remains contaminated and +directional**, exactly as yesterday. + +## 2. STEP 1 — both-ways results, per stat + +Bonferroni across each stat's own sweep. The decider is correlation with the +counter's **residual** (`won − p_win`): only what the counter misses is new. + +| stat | n | strongest solo (marginal r) | p | gate | best interaction (incremental) | head-to-head vs counter | +|---|---|---|---|---|---|---| +| **hits** | **803** | exit_velo **−0.053** | 0.130 | **FAILS** (properly tested) | all ≈0, all **FAIL** | **LOSES** −0.0961, CI [−0.165, −0.029] | +| total_bases | 383 | hard_hit **+0.135** | 0.008 | insufficient n | barrel×archetype −0.101 | **INCONCLUSIVE** +0.0038, CI [−0.068, +0.075] | +| rbi | 391 | exit_velo −0.090 | 0.074 | insufficient n | barrel×archetype +0.073 | no projection built | +| home_runs | 228 | barrel **−0.135** | 0.041 | insufficient n | barrel×archetype −0.116 | no projection built | +| runs | 188 | launch −0.088 | 0.230 | insufficient n | **K×K +0.132** | no projection built | + +### hits is now a FINAL answer, not a pending one + +At **n=803** hits clears the gate's sample requirement, so its features were +**properly tested rather than refused**. Every one fails on effect size (max +|r| 0.053 against a 0.15 bar), every interaction's incremental collapses to +≈0, and the model loses head-to-head with a CI excluding zero. **This is a +well-powered negative and hits should be closed.** + +### The rest are all n-blocked + +TB 383, rbi 391, HR 228, runs 188 — none reaches 500. Their numbers are +directional only. Two are worth carrying: + +- **home_runs · barrel_pct, marginal r = −0.135 (p=0.041).** Note the **sign**: + higher barrel rate goes with the counter *over*-predicting. If that survives + more data it is a real correction, not a new predictor. +- **runs · batter_K × pitcher_K, incremental +0.132** — the largest incremental + anywhere in the cluster, with a clean mechanism (strikeouts destroy plate + appearances, and a PA that never happens cannot score). At n=188 it is a lead. + +### RBI has a missing half, stated rather than papered over + +RBI is power × **opportunity** — the same swing drives in one run or three +depending on who is on base. **We do not ingest baserunner state at all**, so +half the mechanism is absent. A weak RBI result here is not evidence that skill +inputs fail for RBI; it is evidence we are modelling half the stat. + +## 3. STEP 2 — total_bases held frozen + +`git diff` on `src/services/model/skillProjection.js` against the prior commit +is **empty**. The projection is byte-identical; nothing was re-opened or re-fit. +It is frozen — but as §0 records, it is frozen as an **inconclusive** model, not +a proven reference. + +## 4. STEP 3 — the proven set for Stage B + +| stat | BAR 1 (proven) | BAR 2 (calibrated) | +|---|---|---| +| total_bases | **NO** — inconclusive at parity, contaminated | not attempted | +| hits | **NO** — loses, CI excludes zero, well-powered | n/a | +| home_runs | not testable (n=228) | n/a | +| rbi | not testable (n=391, half the mechanism missing) | n/a | +| runs | not testable (n=188) | n/a | + +**Proven set: EMPTY. Stage B has nothing to calibrate.** Calibrating an +inconclusive model into grade bands would produce an "A" backed by a model not +shown to beat counting — which is exactly what BAR 2 exists to prevent. + +## 5. What actually unblocks this + +Everything now waits on the same two things, and both are **waiting problems, +not building problems**: + +1. **A point-in-time window.** `statcast_history` starts accruing usable + comparisons from 2026-08-04. A week gives a real one. Until then no skill + result can be honest, whatever its n. +2. **Sample.** TB needs ~117 more settled rows, rbi ~109, HR ~272, runs ~312. + At current volume TB and rbi arrive within days; HR and runs are weeks away. + +Re-run `scripts/cluster-prove.js` (per stat via `CLUSTER_STAT=`) once both hold. + +**Ranked, when the data arrives:** total_bases (closest to both bars) → rbi +(needs baserunner state to be a fair test) → home_runs → runs. **Close hits.** + +**Not recommended:** treating parity as proven, calibrating anything yet, +lowering n≥500, or letting a stat inherit a pass from the shared conditioning +map. Reuse sped the search; it granted nothing, and nothing has been granted.