Measure the batter cluster: the proven set is empty, and hits is closed
PREMISE CORRECTION FIRST, because it defines the bar. total_bases has not
passed BAR 1. Its head-to-head is inconclusive at parity -- delta +0.004 to
+0.007 with a CI spanning zero -- and it is contaminated, and no feature of
its passed the gate. It was described last session as the first challenger
that did not LOSE, which is not the same as proven. If it is installed as the
frozen proven reference and every other stat is held to "the identical bar
total_bases cleared", the bar becomes "be inconclusive at parity" and the
whole cluster passes on a null result. The proven set is EMPTY.
HITS IS NOW A FINAL ANSWER. At n=803 it clears the gate's sample requirement,
so its features were properly TESTED rather than refused: every one fails on
effect size (max marginal |r| 0.053 against a 0.15 bar), every interaction's
incremental contribution collapses to about zero, and the model loses
head-to-head by 0.096 with a CI excluding zero. That is a well-powered
negative and hits should be closed rather than retried.
The rest are n-blocked: total_bases 383, rbi 391, home_runs 228, runs 188,
against a bar of 500. Two leads are worth carrying. home_runs barrel rate has
a marginal r of -0.135, and the sign matters -- higher barrel rate goes with
the counter OVER-predicting, which would be a correction rather than a new
predictor. And runs batterK x pitcherK has the largest incremental in the
cluster at +0.132, with a clean mechanism: strikeouts destroy plate
appearances, and a PA that never happens cannot score.
RBI deserves a caveat rather than a verdict. It is power times OPPORTUNITY,
and we ingest no baserunner state at all, so half its mechanism is missing. A
weak RBI result is evidence that we are modelling half the stat.
total_bases was held frozen: git diff on skillProjection against the prior
commit is empty. The counter is untouched.
Also fixed and verified in production: the point-in-time retention shipped
after yesterday's refresh had already run, so statcast_history was empty, and
its first run then failed on a hand-enumerated schema that had already drifted
from its source ("could not find the 'swing_pct' column"). The refresh itself
still succeeded and wrote all 1,387 aggregate rows, which confirmed the
best-effort guard in prod. The table now mirrors the source via LIKE and the
writer passes rows through whole. Verified live: 1,387 rows retained at as_of
2026-08-03. A usable point-in-time window starts 2026-08-04.
Stage B has nothing to calibrate. Everything now waits on a point-in-time
window and on sample -- both waiting problems, not building problems.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -3,6 +3,28 @@
|
||||
## Last Updated
|
||||
2026-08-03
|
||||
|
||||
## Session 83 (2026-08-03) — Batter cluster measured; the proven set is EMPTY ✅
|
||||
Spec: `specs/batter-cluster-prove.md`. 4,204 tests / 334 suites green, build exit
|
||||
0. skillProjection byte-identical (TB frozen, verified by diff); counter untouched.
|
||||
- **PREMISE CORRECTED: total_bases has NOT passed BAR 1.** It is inconclusive at
|
||||
parity (CI includes zero) and contaminated. Installing it as the "proven
|
||||
reference" would make the cluster's bar "be inconclusive at parity".
|
||||
- **HITS CLOSED — well-powered negative.** n=803 CLEARS the gate sample bar, so
|
||||
features were tested not refused: max |r| 0.053, interactions ≈0, head-to-head
|
||||
−0.096 CI [−0.165,−0.029].
|
||||
- **Others n-blocked:** TB 383, rbi 391, HR 228, runs 188. Leads: home_runs
|
||||
barrel r=−0.135 (negative = a correction, not a predictor); runs K×K
|
||||
incremental +0.132 (largest in cluster).
|
||||
- **RBI is half-unmodellable** — power × opportunity, and baserunner state is not
|
||||
ingested at all.
|
||||
- **statcast_history retention LIVE + verified in prod** (1,387 rows, as_of
|
||||
2026-08-03). First run failed on a drifted hand-written schema; table now
|
||||
mirrors the source via LIKE. Usable point-in-time window starts 2026-08-04.
|
||||
- **Stage B has nothing to calibrate.** Proven set is empty.
|
||||
- **Next (waiting, not building):** let history accrue a week + TB/rbi reach
|
||||
n>=500, then re-run `scripts/cluster-prove.js`. Ranked: TB → rbi (needs
|
||||
baserunner state) → HR → runs. Do not re-run hits.
|
||||
|
||||
## Session 82 (2026-08-03) — TB solo+interactions; point-in-time validation unblocked ✅
|
||||
Spec: `specs/tb-solo-and-interactions.md`. 4,204 tests / 334 suites green, build
|
||||
exit 0, counter byte-identical.
|
||||
|
||||
@@ -1375,6 +1375,36 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
|
||||
deterministic bases-per-hit made P(TB>=2) EXACTLY P(hits>=1); non-degeneracy is
|
||||
locked by test.
|
||||
|
||||
## Batter cluster + the bar (Session 83 — non-obvious)
|
||||
- **total_bases has NOT passed BAR 1.** Its head-to-head is INCONCLUSIVE at
|
||||
parity (delta +0.004..+0.007, CI includes zero) and CONTAMINATED. It is
|
||||
frozen, but frozen as an *inconclusive* model — do NOT install it as "the
|
||||
proven reference standard", because then the bar other stats must clear
|
||||
becomes "be inconclusive at parity", which admits everything on a null result.
|
||||
**The proven set is EMPTY.**
|
||||
- **HITS IS CLOSED — a well-powered negative.** At n=803 it CLEARS the gate's
|
||||
sample bar, so its features were properly TESTED, not refused: max marginal
|
||||
|r| = 0.053 vs the 0.15 bar, every interaction's incremental ≈ 0, and the
|
||||
model loses head-to-head −0.096 with CI [−0.165,−0.029]. Don't re-run hits.
|
||||
- **Everything else is n-blocked:** TB 383, rbi 391, HR 228, runs 188 (gate needs
|
||||
500). Two leads worth carrying: `home_runs · barrel_pct` marginal r = **−0.135**
|
||||
(NEGATIVE — higher barrel goes with the counter OVER-predicting, i.e. a
|
||||
correction not a predictor), and `runs · batterK×pitcherK` incremental **+0.132**
|
||||
(largest in the cluster; mechanism = strikeouts destroy PA, and a PA that never
|
||||
happens cannot score).
|
||||
- **RBI is half-unmodellable today:** it is power × OPPORTUNITY and we ingest NO
|
||||
baserunner state. A weak RBI result is evidence we model half the stat, not
|
||||
that skill inputs fail for RBI.
|
||||
- **`statcast_history` retention is LIVE and verified in prod** (1,387 rows,
|
||||
as_of 2026-08-03). Two gotchas: the first run failed on a drifted hand-written
|
||||
schema (`swing_pct` missing) — the table is now `create ... (like
|
||||
statcast_aggregates)` and the writer passes rows through whole; and the
|
||||
refresh still succeeded during that failure, confirming the best-effort guard.
|
||||
A usable point-in-time WINDOW starts 2026-08-04 (as_of < game_date).
|
||||
- **`scripts/cluster-prove.js`** runs the whole both-ways program for any stat via
|
||||
`CLUSTER_STAT=`. Per-stat interaction sets are the TB map RE-WEIGHTED, never
|
||||
copied — reuse speeds the search and grants no pass.
|
||||
|
||||
## Active Skills
|
||||
- vyndr-voice (all user-facing output)
|
||||
- prop-analysis (grading methodology)
|
||||
|
||||
@@ -0,0 +1,448 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* tb-solo-and-interactions — PROVE BOTH, with the solo pass as the control.
|
||||
*
|
||||
* A feature can carry signal alone, only in combination, or both. Testing only
|
||||
* interactions misses solo-real features AND cannot tell whether an interaction
|
||||
* ADDS anything or merely re-encodes its own parts. So the solo result is the
|
||||
* baseline every interaction has to beat.
|
||||
*
|
||||
* ── HOW "ADDS OVER ITS PARTS" IS MEASURED ────────────────────────────────
|
||||
* Not by comparing two correlations by eye. The interaction's incremental
|
||||
* signal is the PARTIAL correlation of the interaction term with the counter's
|
||||
* residual, CONTROLLING FOR both component features:
|
||||
*
|
||||
* resid_I = I − OLS(I ~ A, B)
|
||||
* resid_Y = Y − OLS(Y ~ A, B)
|
||||
* incremental r = corr(resid_I, resid_Y)
|
||||
*
|
||||
* If the interaction is just barrel-rate wearing a different hat, regressing out
|
||||
* barrel rate removes it and the incremental r collapses to ~0. That is exactly
|
||||
* the redundancy the order is guarding against, and it is the difference between
|
||||
* PASSES-AND-ADDS and PASSES-BUT-REDUNDANT.
|
||||
*
|
||||
* ── WHY THE COUNTER'S RESIDUAL IS THE TARGET ─────────────────────────────
|
||||
* Correlating with the raw outcome rewards a feature for knowing what the
|
||||
* counter already knows. Only the part the counter MISSES is new information,
|
||||
* and only new information can improve the product. Both are reported; the
|
||||
* residual one is the one that decides.
|
||||
*
|
||||
* ── THEORY FIRST ─────────────────────────────────────────────────────────
|
||||
* Every interaction below is declared with a MECHANISM before it is measured.
|
||||
* No blind pairwise search — with 8 features there are 28 pairs, and at α=.05
|
||||
* roughly one in twenty returns "significant" from noise alone.
|
||||
*
|
||||
* SUPABASE_URL=... node scripts/tb-solo-and-interactions.js
|
||||
*/
|
||||
|
||||
require('dotenv').config();
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const cv = require('../src/services/model/correlateValidator');
|
||||
const sk = require('../src/services/model/skillProjection');
|
||||
const reg = require('../src/services/model/featureRegistry');
|
||||
const mlb = require('../src/services/adapters/mlbStatsAdapter');
|
||||
const { knownRate, knownNumber } = require('../src/utils/known');
|
||||
|
||||
const SB_URL = process.env.SUPABASE_URL;
|
||||
const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
|
||||
const PAGE = 1000;
|
||||
const GAMES_SO_FAR = Number(process.env.STAGEA_GAMES_SO_FAR || 103);
|
||||
const STAT = process.env.CLUSTER_STAT || 'total_bases';
|
||||
|
||||
const r4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
const mean = (a) => (a.length ? a.reduce((x, y) => x + y, 0) / a.length : null);
|
||||
const brier = (ps, ys) => (ps.length ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null);
|
||||
|
||||
/** OLS residuals of y on the given predictor columns (with intercept). */
|
||||
function olsResiduals(y, Xcols) {
|
||||
const n = y.length;
|
||||
const p = Xcols.length + 1;
|
||||
const X = [];
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
const row = [1];
|
||||
for (const c of Xcols) row.push(c[i]);
|
||||
X.push(row);
|
||||
}
|
||||
// Normal equations (X'X) b = X'y, solved by Gauss-Jordan. p is 2-4 here.
|
||||
const XtX = Array.from({ length: p }, () => new Array(p).fill(0));
|
||||
const Xty = new Array(p).fill(0);
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
for (let a = 0; a < p; a += 1) {
|
||||
Xty[a] += X[i][a] * y[i];
|
||||
for (let b = 0; b < p; b += 1) XtX[a][b] += X[i][a] * X[i][b];
|
||||
}
|
||||
}
|
||||
const M = XtX.map((row, i) => [...row, Xty[i]]);
|
||||
for (let col = 0; col < p; col += 1) {
|
||||
let piv = col;
|
||||
for (let r = col + 1; r < p; r += 1) if (Math.abs(M[r][col]) > Math.abs(M[piv][col])) piv = r;
|
||||
if (Math.abs(M[piv][col]) < 1e-12) return null; // singular → cannot control honestly
|
||||
[M[col], M[piv]] = [M[piv], M[col]];
|
||||
const d = M[col][col];
|
||||
for (let k = col; k <= p; k += 1) M[col][k] /= d;
|
||||
for (let r = 0; r < p; r += 1) {
|
||||
if (r === col) continue;
|
||||
const f = M[r][col];
|
||||
for (let k = col; k <= p; k += 1) M[r][k] -= f * M[col][k];
|
||||
}
|
||||
}
|
||||
const beta = M.map((row) => row[p]);
|
||||
return y.map((v, i) => v - X[i].reduce((s, xv, j) => s + xv * beta[j], 0));
|
||||
}
|
||||
|
||||
/**
|
||||
* Partial correlation of a with b, controlling for the columns in ctrl.
|
||||
*
|
||||
* COLLINEARITY IS CHECKED FIRST, and this is not pedantry — it caught a real
|
||||
* error in this very script. The archetype-power proxy was defined as
|
||||
* `barrel_pct / LEAGUE.barrel_pct`, an exact linear function of barrel_pct, so
|
||||
* "control for both components" was rank-deficient and the partial correlation
|
||||
* it produced (-0.132, the only one that looked like an incremental finding) was
|
||||
* an artifact of a singular design matrix. The Gauss-Jordan pivot test missed it
|
||||
* because the two columns differ by a scale factor, which keeps the pivot well
|
||||
* above an absolute epsilon. Scale-free pairwise correlation catches it.
|
||||
*/
|
||||
function partialCorr(a, b, ctrl) {
|
||||
for (let i = 0; i < ctrl.length; i += 1) {
|
||||
for (let j = i + 1; j < ctrl.length; j += 1) {
|
||||
const rr = cv.pearson(ctrl[i], ctrl[j]).r;
|
||||
if (rr !== null && Math.abs(rr) > 0.999) return null; // same variable twice
|
||||
}
|
||||
}
|
||||
const ra = olsResiduals(a, ctrl);
|
||||
const rb = olsResiduals(b, ctrl);
|
||||
if (!ra || !rb) return null;
|
||||
return cv.pearson(ra, rb).r;
|
||||
}
|
||||
|
||||
/** Rows where every named key is known — the honest common sample. */
|
||||
function completeRows(rows, keys) {
|
||||
return rows.filter((r) => keys.every((k) => knownNumber(r[k]) !== null));
|
||||
}
|
||||
|
||||
function makeRnd(seed) {
|
||||
let s = seed >>> 0;
|
||||
return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
|
||||
}
|
||||
function bootstrapDiff(rows, keyA, keyB, iters = 4000, seed = 20260804) {
|
||||
if (rows.length < 30) return null;
|
||||
const rnd = makeRnd(seed);
|
||||
const n = rows.length;
|
||||
const diffs = [];
|
||||
for (let it = 0; it < iters; it += 1) {
|
||||
const ys = []; const a = []; const b = [];
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
const r = rows[Math.floor(rnd() * n)];
|
||||
ys.push(r.won); a.push(r[keyA]); b.push(r[keyB]);
|
||||
}
|
||||
const ca = cv.pearson(a, ys).r; const cb = cv.pearson(b, ys).r;
|
||||
if (ca == null || cb == null) continue;
|
||||
diffs.push(ca - cb);
|
||||
}
|
||||
if (diffs.length < 100) return null;
|
||||
diffs.sort((x, y) => x - y);
|
||||
const q = (pp) => r4(diffs[Math.floor(pp * (diffs.length - 1))]);
|
||||
const ci = [q(0.025), q(0.975)];
|
||||
return {
|
||||
point: r4(cv.pearson(rows.map((r) => r[keyA]), rows.map((r) => r.won)).r
|
||||
- cv.pearson(rows.map((r) => r[keyB]), rows.map((r) => r.won)).r),
|
||||
ci95: ci, ci_excludes_zero: ci[0] > 0 || ci[1] < 0,
|
||||
};
|
||||
}
|
||||
|
||||
async function page(sb, table, select, apply) {
|
||||
const out = [];
|
||||
for (let from = 0; ; from += PAGE) {
|
||||
const { data, error } = await apply(sb.from(table).select(select)).range(from, from + PAGE - 1);
|
||||
if (error) throw error;
|
||||
if (!data || data.length === 0) break;
|
||||
out.push(...data);
|
||||
if (data.length < PAGE) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function opposingStarters(dates) {
|
||||
const m = new Map();
|
||||
for (const d of dates) {
|
||||
let games = [];
|
||||
try { games = await mlb.getScheduleWithPitchers(d); } catch { games = []; }
|
||||
for (const g of games) {
|
||||
if (!g.home || !g.away) continue;
|
||||
if (g.home.probablePitcher) m.set(`${d}|OPP:${g.home.team}`, g.home.probablePitcher.id);
|
||||
if (g.away.probablePitcher) m.set(`${d}|OPP:${g.away.team}`, g.away.probablePitcher.id);
|
||||
}
|
||||
}
|
||||
return m;
|
||||
}
|
||||
async function opponentByPlayerDate(players) {
|
||||
const map = new Map();
|
||||
for (const [key, name] of players) {
|
||||
try {
|
||||
const found = await mlb.searchPlayer(name);
|
||||
if (!found || !found.id) continue;
|
||||
const log = await mlb.getPlayerGameLog(found.id);
|
||||
for (const g of log || []) {
|
||||
if (g && g.date && g.opponent) map.set(`${key}|${String(g.date).slice(0, 10)}`, g.opponent);
|
||||
}
|
||||
} catch { /* no log → no pitcher */ }
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
const SOLO = ['batter_barrel_pct', 'batter_hard_hit_pct', 'batter_exit_velo',
|
||||
'batter_launch_angle', 'batter_k_pct', 'batter_bb_pct',
|
||||
'pitcher_k_pct', 'pitcher_hard_hit_allowed'];
|
||||
|
||||
/**
|
||||
* PER-STAT INTERACTION SETS — the total_bases conditioning map RE-WEIGHTED, not
|
||||
* copied. Reuse speeds the search; it grants nothing. Each stat's features must
|
||||
* independently earn their place FOR THAT STAT, and the mechanisms genuinely
|
||||
* differ: barrel rate drives home runs through one channel (does the ball leave)
|
||||
* and RBI through another (does anyone happen to be on base when it does).
|
||||
*/
|
||||
const STAT_INTERACTIONS = {
|
||||
total_bases: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'barrel_x_power_archetype', 'batterK_x_pitcherK'],
|
||||
hits: ['launch_x_exit_velo', 'exitvelo_x_pitcher_suppression', 'batterK_x_pitcherK'],
|
||||
// HOME RUNS are the purest barrel stat: the ball must be hit hard AND at the
|
||||
// right angle, and the pitcher must be the kind who allows that combination.
|
||||
home_runs: ['launch_x_exit_velo', 'barrel_x_power_archetype', 'exitvelo_x_pitcher_suppression'],
|
||||
// RBI is a POWER x OPPORTUNITY stat — a solo home run drives in one, the same
|
||||
// swing with two on drives in three. We do not ingest baserunner state, so the
|
||||
// opportunity half is genuinely missing and that is reported, not papered over.
|
||||
rbi: ['barrel_x_power_archetype', 'exitvelo_x_pitcher_suppression', 'batterK_x_pitcherK'],
|
||||
// RUNS scored is ON-BASE x what happens AFTER — mostly teammate-driven, which
|
||||
// is the least self-contained stat in the cluster.
|
||||
runs: ['batterK_x_pitcherK', 'exitvelo_x_pitcher_suppression'],
|
||||
};
|
||||
|
||||
/** STEP 2 — theory first. Every interaction declares its mechanism. */
|
||||
const INTERACTIONS = [
|
||||
{
|
||||
key: 'launch_x_exit_velo',
|
||||
components: ['batter_launch_angle', 'batter_exit_velo'],
|
||||
mechanism: 'Extra bases need BOTH conditions: hit hard AND hit in the air. A 105-mph ground ball is an out; a 25-degree popup is an out. Neither factor alone predicts bases, which is precisely why each may fail solo and the product may not.',
|
||||
build: (r) => r.batter_launch_angle * r.batter_exit_velo,
|
||||
},
|
||||
{
|
||||
key: 'exitvelo_x_pitcher_suppression',
|
||||
components: ['batter_exit_velo', 'pitcher_hard_hit_allowed'],
|
||||
mechanism: 'A hitter only realises his contact quality against a pitcher who permits contact quality. Elite suppression should attenuate a power bat; a contact-permitting arm should amplify it. The effect is conditional by construction.',
|
||||
build: (r) => r.batter_exit_velo * r.pitcher_hard_hit_allowed,
|
||||
},
|
||||
{
|
||||
key: 'barrel_x_power_archetype',
|
||||
components: ['batter_barrel_pct', 'archetype_power'],
|
||||
mechanism: 'ARCHETYPE-CONDITIONAL. Barrels convert to extra bases for hitters whose lane is power; for a speed/contact profile the same barrel rate is a rarer event on a swing built for something else. This is Discipline 2 stated as a testable interaction. NOTE: it is currently UNTESTABLE — statcast rows carry no archetype label, and the barrel-relative proxy is an exact linear function of barrel_pct, so controlling for both components is rank-deficient. It needs a real archetype classification joined in.',
|
||||
build: (r) => r.batter_barrel_pct * r.archetype_power,
|
||||
},
|
||||
{
|
||||
key: 'batterK_x_pitcherK',
|
||||
components: ['batter_k_pct', 'pitcher_k_pct'],
|
||||
mechanism: 'Strikeout risk compounds multiplicatively (log5 is exactly this shape). A high-K bat against a high-K arm loses plate appearances to strikeouts, and a PA lost is a base opportunity that never happens — so it suppresses total bases through OPPORTUNITY, not contact quality.',
|
||||
build: (r) => r.batter_k_pct * r.pitcher_k_pct,
|
||||
},
|
||||
];
|
||||
|
||||
/** Latest settled game date in the pull — used to detect that the profile
|
||||
* freeze now sits AFTER the data, i.e. no clean out-of-sample window exists. */
|
||||
function clean0Max(rows) {
|
||||
return (rows || []).reduce((mx, r) => (String(r.game_date) > mx ? String(r.game_date) : mx), '');
|
||||
}
|
||||
|
||||
async function main() {
|
||||
if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required');
|
||||
const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
|
||||
|
||||
const statcast = await page(sb, 'statcast_aggregates', '*', (q) => q.eq('sport', 'mlb'));
|
||||
const freezeDate = statcast.reduce((mx, r) => (String(r.updated_at) > mx ? String(r.updated_at) : mx), '').slice(0, 10);
|
||||
const batters = new Map(); const pitchersById = new Map();
|
||||
for (const r of statcast) {
|
||||
if (r.role === 'pitcher' && r.source_id != null) pitchersById.set(Number(r.source_id), sk.fromStatcastRow(r));
|
||||
if (r.player_key && r.role === 'batter') {
|
||||
const prev = batters.get(r.player_key);
|
||||
if (!prev || Number(r.sample_pa || 0) > Number(prev.rawPa || 0)) {
|
||||
batters.set(r.player_key, Object.assign(sk.fromStatcastRow(r), { rawPa: Number(r.sample_pa || 0) }));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// REAL ARCHETYPE LABELS. The barrel-relative proxy was a clipped monotone
|
||||
// transform of barrel_pct, so `barrel x proxy` measured NONLINEARITY IN BARREL,
|
||||
// not an archetype interaction — it could never have tested Discipline 2.
|
||||
// model_snapshots carries the actual classification per prop, so the
|
||||
// conditioning variable is now a genuine BOMBER indicator, which is
|
||||
// categorical and therefore not a transform of barrel at all.
|
||||
const snaps = await page(sb, 'model_snapshots', 'player_key, game_date, archetype',
|
||||
(q) => q.eq('sport', 'mlb').eq('stat', STAT).not('archetype', 'is', null));
|
||||
const archetypeBy = new Map();
|
||||
for (const r of snaps) if (r.player_key && r.game_date) archetypeBy.set(`${r.player_key}|${r.game_date}`, r.archetype);
|
||||
|
||||
const led = await page(sb, 'ledger_entries',
|
||||
'player_key, player_name, stat, line, side, outcome, game_date, p_win, quarantine_reason',
|
||||
(q) => q.eq('sport', 'mlb').is('user_id', null).eq('stat', STAT)
|
||||
.in('outcome', ['hit', 'miss']).not('p_win', 'is', null));
|
||||
// POINT-IN-TIME IS NO LONGER AVAILABLE FROM THIS TABLE.
|
||||
//
|
||||
// `statcast_aggregates` is upserted in place and keeps one as-of date. The
|
||||
// first skill backtest was honest only by accident: the nightly refresh was
|
||||
// unreachable code, so the table sat frozen at 2026-07-21 — BEFORE the settled
|
||||
// window. Repairing that cron (correct for production) refreshed it to today,
|
||||
// and every prior version is gone.
|
||||
//
|
||||
// So scoring a 2026-07-25 game now uses a season aggregate that CONTAINS that
|
||||
// game. `statcast_history` (added this session) fixes it going forward; it has
|
||||
// one day of data, which is not yet a window. Until it fills, results here are
|
||||
// DIRECTIONAL AND CONTAMINATED, labelled as such, and are NOT gate verdicts.
|
||||
const contaminated = String(freezeDate) >= String(clean0Max(led));
|
||||
const clean = led.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book')
|
||||
&& (contaminated ? true : String(r.game_date) > freezeDate));
|
||||
|
||||
const dates = [...new Set(clean.map((r) => r.game_date))].sort();
|
||||
const starters = await opposingStarters(dates);
|
||||
const players = new Map();
|
||||
for (const r of clean) if (!players.has(r.player_key)) players.set(r.player_key, r.player_name);
|
||||
const oppByPlayerDate = await opponentByPlayerDate(players);
|
||||
|
||||
if (process.env.TB_DEBUG === '1') {
|
||||
console.error(`[debug] statcast rows=${statcast.length} freeze=${freezeDate} batters=${batters.size} pitchers=${pitchersById.size}`);
|
||||
console.error(`[debug] ledger tb rows=${led.length} clean(after freeze)=${clean.length}`);
|
||||
const sampleKeys = clean.slice(0, 5).map((r) => r.player_key);
|
||||
console.error(`[debug] sample ledger player_keys=${JSON.stringify(sampleKeys)}`);
|
||||
console.error(`[debug] sample statcast keys=${JSON.stringify([...batters.keys()].slice(0, 5))}`);
|
||||
console.error(`[debug] matches in sample=${sampleKeys.filter((k) => batters.has(k)).length}/5`);
|
||||
}
|
||||
const allowed = reg.candidateFeaturesForStat('mlb', STAT);
|
||||
const rows = [];
|
||||
for (const r of clean) {
|
||||
const bat = batters.get(r.player_key);
|
||||
if (!bat) continue;
|
||||
const faced = oppByPlayerDate.get(`${r.player_key}|${r.game_date}`) || null;
|
||||
const pit = faced ? pitchersById.get(Number(starters.get(`${r.game_date}|OPP:${faced}`))) || null : null;
|
||||
const paRate = bat.rawPa > 0 ? Math.min(5.2, Math.max(2.0, bat.rawPa / GAMES_SO_FAR)) : null;
|
||||
const under = String(r.side).toLowerCase() === 'under';
|
||||
const won = r.outcome === 'hit' ? 1 : 0;
|
||||
const champ = Number(r.p_win);
|
||||
|
||||
const proj = sk.projectSkill({
|
||||
batter: bat, pitcher: pit, park: 1, archetype: null,
|
||||
statType: STAT, line: Number(r.line), expectedPa: paRate, allowed,
|
||||
});
|
||||
|
||||
// The REAL archetype for this prop — a 0/1 power indicator, categorical and
|
||||
// independent of barrel_pct by construction.
|
||||
const arch = archetypeBy.get(`${r.player_key}|${r.game_date}`) || null;
|
||||
const archetypePower = arch == null ? null : (String(arch).toUpperCase() === 'BOMBER' ? 1 : 0);
|
||||
|
||||
rows.push({
|
||||
won, champ, residual: won - champ,
|
||||
skill: proj ? (under ? 1 - proj.p_over_line : proj.p_over_line) : null,
|
||||
had_pitcher: !!pit,
|
||||
batter_barrel_pct: knownRate(bat.barrel_pct),
|
||||
batter_hard_hit_pct: knownRate(bat.hard_hit_pct),
|
||||
batter_exit_velo: knownRate(bat.avg_exit_velo),
|
||||
batter_launch_angle: knownRate(bat.avg_launch_angle),
|
||||
batter_k_pct: knownRate(bat.k_pct),
|
||||
batter_bb_pct: knownRate(bat.bb_pct),
|
||||
pitcher_k_pct: pit ? knownRate(pit.k_pct) : null,
|
||||
pitcher_hard_hit_allowed: pit ? knownRate(pit.hard_hit_pct) : null,
|
||||
archetype_power: archetypePower,
|
||||
archetype: arch,
|
||||
});
|
||||
}
|
||||
|
||||
// Bonferroni denominator = every test in this family (solo + interaction).
|
||||
const TESTS = SOLO.length + INTERACTIONS.length;
|
||||
|
||||
// ── STEP 1 — SOLO PASS (the control) ────────────────────────────────────
|
||||
const solo = {};
|
||||
for (const f of SOLO) {
|
||||
const rs = completeRows(rows, [f]);
|
||||
solo[f] = {
|
||||
n: rs.length,
|
||||
vs_outcome: cv.validateFactor(rs.map((r) => r[f]), rs.map((r) => r.won), TESTS),
|
||||
vs_counter_residual: cv.validateFactor(rs.map((r) => r[f]), rs.map((r) => r.residual), TESTS),
|
||||
};
|
||||
}
|
||||
|
||||
// ── STEP 3 — INTERACTIONS, each against its own solo baseline ───────────
|
||||
const interactions = {};
|
||||
const chosen = new Set(STAT_INTERACTIONS[STAT] || []);
|
||||
for (const ix of INTERACTIONS.filter((x) => chosen.has(x.key))) {
|
||||
const keys = [...ix.components];
|
||||
const rs = completeRows(rows, keys);
|
||||
if (rs.length < 30) { interactions[ix.key] = { mechanism: ix.mechanism, n: rs.length, verdict: 'UNTESTABLE — no common sample' }; continue; }
|
||||
const I = rs.map(ix.build);
|
||||
const Y = rs.map((r) => r.residual);
|
||||
const ctrl = keys.map((k) => rs.map((r) => r[k]));
|
||||
|
||||
const gate = cv.validateFactor(I, Y, TESTS);
|
||||
const incremental = partialCorr(I, Y, ctrl);
|
||||
// The best solo |r| among its own components, on the SAME rows.
|
||||
const componentSolo = keys.map((k) => ({
|
||||
feature: k, r: r4(cv.pearson(rs.map((r) => r[k]), Y).r),
|
||||
}));
|
||||
const bestComponent = Math.max(...componentSolo.map((c) => Math.abs(c.r ?? 0)));
|
||||
|
||||
let verdict;
|
||||
if (incremental === null) verdict = 'UNTESTABLE — controls are collinear';
|
||||
else if (gate.validated && Math.abs(incremental) >= cv.VALIDATION_REQUIREMENTS.min_pearson_r) verdict = 'PASSES-AND-ADDS';
|
||||
else if (gate.validated) verdict = 'PASSES-BUT-REDUNDANT';
|
||||
else if (rs.length < cv.VALIDATION_REQUIREMENTS.min_historical_instances) verdict = 'UNDERPOWERED — n below the gate';
|
||||
else verdict = 'FAILS';
|
||||
|
||||
interactions[ix.key] = {
|
||||
mechanism: ix.mechanism,
|
||||
components: keys,
|
||||
n: rs.length,
|
||||
raw_r_vs_residual: gate.pearson_r,
|
||||
gate: { validated: gate.validated, reason: gate.reason, p_value: gate.p_value, corrected_alpha: gate.corrected_alpha, underpowered: !!gate.underpowered },
|
||||
component_solo_r_same_rows: componentSolo,
|
||||
best_component_abs_r: r4(bestComponent),
|
||||
INCREMENTAL_partial_r: r4(incremental),
|
||||
adds_over_components: incremental !== null && Math.abs(incremental) > bestComponent,
|
||||
verdict,
|
||||
};
|
||||
}
|
||||
|
||||
// ── STEP 4 — COMBINED vs COUNTER (valid at this n; the gate is not) ─────
|
||||
const h2h = rows.filter((r) => r.skill != null);
|
||||
const ys = h2h.map((r) => r.won);
|
||||
const bs = bootstrapDiff(h2h, 'skill', 'champ');
|
||||
|
||||
console.log(JSON.stringify({
|
||||
stat: STAT,
|
||||
VALIDITY: contaminated
|
||||
? 'CONTAMINATED / DIRECTIONAL ONLY — statcast_aggregates now carries a single as-of date (' + freezeDate + ') that is AFTER the settled games, so season profiles contain the games being predicted. These are NOT gate verdicts. statcast_history (new) makes point-in-time possible from tomorrow.'
|
||||
: `CLEAN out-of-sample: profiles frozen ${freezeDate}; only game_date > ${freezeDate} scored`,
|
||||
contaminated,
|
||||
rows_scored: rows.length,
|
||||
gate_spec: cv.VALIDATION_REQUIREMENTS,
|
||||
bonferroni_tests: TESTS,
|
||||
n_gap_note: `the gate needs ${cv.VALIDATION_REQUIREMENTS.min_historical_instances} rows; this run has ${rows.length}`,
|
||||
archetype_coverage: {
|
||||
labelled: rows.filter((r) => r.archetype).length,
|
||||
bomber: rows.filter((r) => r.archetype_power === 1).length,
|
||||
other: rows.filter((r) => r.archetype_power === 0).length,
|
||||
},
|
||||
step1_solo_baseline: solo,
|
||||
step3_interactions: interactions,
|
||||
step4_combined_vs_counter: {
|
||||
n: h2h.length,
|
||||
pitcher_coverage: r4(mean(h2h.map((r) => (r.had_pitcher ? 1 : 0)))),
|
||||
base_rate: r4(mean(ys)),
|
||||
resolution: { skill_tb: r4(cv.pearson(h2h.map((r) => r.skill), ys).r), counter: r4(cv.pearson(h2h.map((r) => r.champ), ys).r) },
|
||||
brier: { skill_tb: r4(brier(h2h.map((r) => r.skill), ys)), counter: r4(brier(h2h.map((r) => r.champ), ys)) },
|
||||
delta: bs,
|
||||
verdict: !bs ? 'N-BLOCKED'
|
||||
: (bs.ci_excludes_zero && bs.point > 0) ? 'SKILL TB BEATS THE COUNTER'
|
||||
: (bs.ci_excludes_zero && bs.point < 0) ? 'LOSES to the counter'
|
||||
: 'INCONCLUSIVE',
|
||||
},
|
||||
}, null, 2));
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error(e); process.exit(1); });
|
||||
@@ -0,0 +1,126 @@
|
||||
# THE BATTER CLUSTER — both-ways results, and what the bar actually is
|
||||
|
||||
**2026-08-03.** Challenger-only. Counter byte-identical. `skillProjection`
|
||||
byte-identical (frozen) — verified by diff against the prior commit.
|
||||
|
||||
> **PREMISE CORRECTION, and it decides the whole order: total_bases has NOT
|
||||
> passed BAR 1.** Yesterday's result was **delta +0.0071, CI95
|
||||
> [−0.0648, +0.0789] — inconclusive, at parity, and contaminated**. No feature
|
||||
> passed the gate. It was described as "the first challenger that did not LOSE",
|
||||
> which is not the same as proven.
|
||||
>
|
||||
> This matters structurally: if total_bases is installed as "the frozen proven
|
||||
> reference" and every other stat is held to "the identical bar total_bases
|
||||
> cleared", **the bar becomes "be inconclusive at parity"** — a standard that
|
||||
> proves nothing and would admit the entire cluster on a null result.
|
||||
>
|
||||
> **The proven set is EMPTY.** That is the honest Stage-B input.
|
||||
|
||||
---
|
||||
|
||||
## 0. Two orders running, one pattern
|
||||
|
||||
This is the second consecutive order whose premise promoted a null result to a
|
||||
pass (the previous one had "barrel rate PASSED solo" when every TB feature was
|
||||
refused on sample size). Flagging the pattern once, without labouring it: the
|
||||
measurements are being read at their most favourable interpretation somewhere
|
||||
between sessions. The numbers below are stated so that cannot happen again.
|
||||
|
||||
## 1. Validity: still contaminated today, but the clock has started
|
||||
|
||||
`statcast_history` was **empty** — the retention shipped *after* yesterday's
|
||||
refresh had already run. Two things fixed this session:
|
||||
|
||||
1. The first prod run failed with `Could not find the 'swing_pct' column` — the
|
||||
hand-enumerated schema had drifted from the table it was copying. **The
|
||||
refresh still succeeded and wrote all 1,387 aggregate rows**, which verified
|
||||
the best-effort guard in production: a retention failure does not fail the
|
||||
refresh.
|
||||
2. The table is now created `LIKE statcast_aggregates` and the writer passes the
|
||||
row through whole, so there is no drift surface left.
|
||||
|
||||
**Verified live: `history_retained: 1387, as_of 2026-08-03`.** The point-in-time
|
||||
clock is running. It does not yet give a *window* — `as_of 2026-08-03` can only
|
||||
score games from 2026-08-04 — so **everything below remains contaminated and
|
||||
directional**, exactly as yesterday.
|
||||
|
||||
## 2. STEP 1 — both-ways results, per stat
|
||||
|
||||
Bonferroni across each stat's own sweep. The decider is correlation with the
|
||||
counter's **residual** (`won − p_win`): only what the counter misses is new.
|
||||
|
||||
| stat | n | strongest solo (marginal r) | p | gate | best interaction (incremental) | head-to-head vs counter |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **hits** | **803** | exit_velo **−0.053** | 0.130 | **FAILS** (properly tested) | all ≈0, all **FAIL** | **LOSES** −0.0961, CI [−0.165, −0.029] |
|
||||
| total_bases | 383 | hard_hit **+0.135** | 0.008 | insufficient n | barrel×archetype −0.101 | **INCONCLUSIVE** +0.0038, CI [−0.068, +0.075] |
|
||||
| rbi | 391 | exit_velo −0.090 | 0.074 | insufficient n | barrel×archetype +0.073 | no projection built |
|
||||
| home_runs | 228 | barrel **−0.135** | 0.041 | insufficient n | barrel×archetype −0.116 | no projection built |
|
||||
| runs | 188 | launch −0.088 | 0.230 | insufficient n | **K×K +0.132** | no projection built |
|
||||
|
||||
### hits is now a FINAL answer, not a pending one
|
||||
|
||||
At **n=803** hits clears the gate's sample requirement, so its features were
|
||||
**properly tested rather than refused**. Every one fails on effect size (max
|
||||
|r| 0.053 against a 0.15 bar), every interaction's incremental collapses to
|
||||
≈0, and the model loses head-to-head with a CI excluding zero. **This is a
|
||||
well-powered negative and hits should be closed.**
|
||||
|
||||
### The rest are all n-blocked
|
||||
|
||||
TB 383, rbi 391, HR 228, runs 188 — none reaches 500. Their numbers are
|
||||
directional only. Two are worth carrying:
|
||||
|
||||
- **home_runs · barrel_pct, marginal r = −0.135 (p=0.041).** Note the **sign**:
|
||||
higher barrel rate goes with the counter *over*-predicting. If that survives
|
||||
more data it is a real correction, not a new predictor.
|
||||
- **runs · batter_K × pitcher_K, incremental +0.132** — the largest incremental
|
||||
anywhere in the cluster, with a clean mechanism (strikeouts destroy plate
|
||||
appearances, and a PA that never happens cannot score). At n=188 it is a lead.
|
||||
|
||||
### RBI has a missing half, stated rather than papered over
|
||||
|
||||
RBI is power × **opportunity** — the same swing drives in one run or three
|
||||
depending on who is on base. **We do not ingest baserunner state at all**, so
|
||||
half the mechanism is absent. A weak RBI result here is not evidence that skill
|
||||
inputs fail for RBI; it is evidence we are modelling half the stat.
|
||||
|
||||
## 3. STEP 2 — total_bases held frozen
|
||||
|
||||
`git diff` on `src/services/model/skillProjection.js` against the prior commit
|
||||
is **empty**. The projection is byte-identical; nothing was re-opened or re-fit.
|
||||
It is frozen — but as §0 records, it is frozen as an **inconclusive** model, not
|
||||
a proven reference.
|
||||
|
||||
## 4. STEP 3 — the proven set for Stage B
|
||||
|
||||
| stat | BAR 1 (proven) | BAR 2 (calibrated) |
|
||||
|---|---|---|
|
||||
| total_bases | **NO** — inconclusive at parity, contaminated | not attempted |
|
||||
| hits | **NO** — loses, CI excludes zero, well-powered | n/a |
|
||||
| home_runs | not testable (n=228) | n/a |
|
||||
| rbi | not testable (n=391, half the mechanism missing) | n/a |
|
||||
| runs | not testable (n=188) | n/a |
|
||||
|
||||
**Proven set: EMPTY. Stage B has nothing to calibrate.** Calibrating an
|
||||
inconclusive model into grade bands would produce an "A" backed by a model not
|
||||
shown to beat counting — which is exactly what BAR 2 exists to prevent.
|
||||
|
||||
## 5. What actually unblocks this
|
||||
|
||||
Everything now waits on the same two things, and both are **waiting problems,
|
||||
not building problems**:
|
||||
|
||||
1. **A point-in-time window.** `statcast_history` starts accruing usable
|
||||
comparisons from 2026-08-04. A week gives a real one. Until then no skill
|
||||
result can be honest, whatever its n.
|
||||
2. **Sample.** TB needs ~117 more settled rows, rbi ~109, HR ~272, runs ~312.
|
||||
At current volume TB and rbi arrive within days; HR and runs are weeks away.
|
||||
|
||||
Re-run `scripts/cluster-prove.js` (per stat via `CLUSTER_STAT=`) once both hold.
|
||||
|
||||
**Ranked, when the data arrives:** total_bases (closest to both bars) → rbi
|
||||
(needs baserunner state to be a fair test) → home_runs → runs. **Close hits.**
|
||||
|
||||
**Not recommended:** treating parity as proven, calibrating anything yet,
|
||||
lowering n≥500, or letting a stat inherit a pass from the shared conditioning
|
||||
map. Reuse sped the search; it granted nothing, and nothing has been granted.
|
||||
Reference in New Issue
Block a user