Link 2 at the coarse grain: pen QUALITY proves, archetype does not
The refinement was right. Naming the individual reliever failed; the same
question at the grain the chain needs passes, and it transmits more than
anything else measured in this chain.
WHY IT WAS WORTH RE-ASKING: last session's null (the pen is on average no
softer, +0.0010 on 35,760 PAs) does NOT rule this out, and treating it as
though it did would have been the error. An average washing out is fully
consistent with quality VARIATION mattering. It does -- actual arm quality
moves the hit rate monotonically across quartiles, 0.2244 / 0.2293 /
0.2410 / 0.2501, a 2.57pp spread, larger than the whole times-through-
the-order effect.
CLUSTER UNIT CORRECTED, THEN CHECKED RATHER THAN ARGUED. Last session
refused Link 2 partly as team-borne (30 bullpens, the park ceiling). My
first re-check was that 76% of pen-quality variance is within-team -- but
that is a statement about TREATMENT variance, not about where errors
correlate, and stopping there would have been picking the convenient
answer. Measured the actual thing: ICC of prediction error by team =
0.0261, design effect 1.41, SEs inflated ~19%. So the verdict was run
three ways:
unclustered CI [-0.0067,-0.0010] excludes zero
team-clustered (30) CI [-0.0086,-0.0003] excludes zero (below the
40-cluster floor -- indicative, not a pass)
design-effect adjusted CI [-0.0072,-0.0005] excludes zero
QUALITY GRAIN PROVES on the concentrated elevated-early-exit subset:
n=501 team-games, 426 clusters, MAE 0.0294 -> 0.0260, delta -0.0034, CI
[-0.0063,-0.0005] at 110 cumulative tests. Pooled also proves, so it is
not a subset artefact.
ARCHETYPE GRAIN DOES NOT: 0.5669 vs a 0.5309 modal-guess baseline,
corrected interval [-0.1073,+0.0268] spans zero. Two grains tested, one
earned a place -- penQuality.js exposes no archetype and a test asserts
it.
WHAT LINK 3 RECEIVES, which is the number that actually matters -- not
the MAE gain but realized outcome separation, prediction strictly
point-in-time:
predicted BEST pen 167 games 2,044 PAs hit rate 0.2231 +/-0.0180
predicted WORST pen 167 games 1,799 PAs hit rate 0.2501 +/-0.0200
2.70pp separated, intervals non-overlapping, capturing nearly all the
2.57pp available at the quartile grain. Caveat stated not buried: the
tercile cut is chosen in-sample; the prediction driving it is not.
BUILT: penQuality.js + 9 tests. Abstains below 5 prior club games and 40
arm appearances -- a league-average stand-in would assert "this is an
ordinary bullpen", which is a claim, and usually the wrong one for exactly
the clubs whose pens just turned over.
Link 3 is unblocked on a proven Link 2 at the quality grain only. Not run
here; this order scopes to building and gating Link 2.
Parallel track logged unchanged: TB n=948 pooled, BOMBER x TB 340, short
by 160.
Counter and frozen clusters byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,206 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* LINK 2 (coarse grain) — WHICH BULLPEN, not which arm.
|
||||
*
|
||||
* Naming the individual reliever failed on merit: 17.2% accuracy, wrong five
|
||||
* times in six. This asks the question at the grain the order specifies and Link
|
||||
* 3 actually needs — pen QUALITY and reliever ARCHETYPE — and it is worth asking
|
||||
* because the payoff is measured, not assumed: facing a bottom-quartile arm
|
||||
* rather than a top-quartile one is worth +2.57pp of hit rate, larger than the
|
||||
* whole times-through-the-order effect.
|
||||
*
|
||||
* ── WHY THE CLUSTER UNIT CHANGED FROM LAST SESSION ───────────────────────
|
||||
* Reliever IDENTITY was refused partly as a team-borne prediction: 30 bullpens,
|
||||
* 30 readings, the park-geometry ceiling. Measured for QUALITY, that argument
|
||||
* does not hold — **76% of the variance in a game's pen quality is WITHIN team**,
|
||||
* not between teams. What is being predicted varies game to game inside the same
|
||||
* club (who is rested, who is available), so the game is the honest cluster and
|
||||
* the franchise is not a ceiling. Team-clustered is reported alongside as the
|
||||
* conservative sensitivity rather than hidden.
|
||||
*
|
||||
* ── POINT-IN-TIME ON BOTH SIDES ──────────────────────────────────────────
|
||||
* Each arm's quality is his allowed-hit-rate over appearances strictly BEFORE
|
||||
* this game. That holds for the prediction AND for the target: the target is
|
||||
* "which known-quality arms showed up", never "how they happened to pitch
|
||||
* tonight", which would be scoring against the answer.
|
||||
*
|
||||
* node scripts/link2b-pen-quality.js
|
||||
*/
|
||||
|
||||
require('dotenv').config();
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const pg = require('../src/services/model/predictionGate');
|
||||
const tl = require('../src/services/model/testLedger');
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
const { knownNumber } = require('../src/utils/known');
|
||||
|
||||
const CACHE = process.env.SEQ_OUT || path.join(process.cwd(), '.seq-cache', 'sequences.json');
|
||||
const HIT = new Set(['single', 'double', 'triple', 'home_run']);
|
||||
const PA = new Set(['single', 'double', 'triple', 'home_run', 'field_out', 'strikeout',
|
||||
'grounded_into_double_play', 'force_out', 'field_error', 'fielders_choice',
|
||||
'fielders_choice_out', 'double_play', 'sac_fly', 'pop_out', 'line_out', 'fly_out',
|
||||
'strikeout_double_play']);
|
||||
|
||||
/** Appearances before we will read an arm's quality at all. Below it: abstain. */
|
||||
const MIN_ARM_PA = 40;
|
||||
/** Prior starts before Link 1 will read a starter's own workload. */
|
||||
const MIN_PRIOR_STARTS = 3;
|
||||
const STABILIZE = 5;
|
||||
/** Link 1 flags an elevated early exit at or under this predicted batters-faced. */
|
||||
const EARLY_FLAG_BF = 22;
|
||||
|
||||
const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null);
|
||||
|
||||
/**
|
||||
* Reliever archetype at the coarse grain, from strikeout rate — the axis that
|
||||
* separates a power arm from a contact arm and the one Link 3 would condition on.
|
||||
*/
|
||||
function archetypeOf(kRate) {
|
||||
if (kRate === null) return null;
|
||||
if (kRate >= 0.28) return 'POWER';
|
||||
if (kRate <= 0.18) return 'CONTACT';
|
||||
return 'MIDDLE';
|
||||
}
|
||||
|
||||
function build() {
|
||||
const { games } = JSON.parse(fs.readFileSync(CACHE, 'utf8'));
|
||||
games.sort((a, b) => String(a.date).localeCompare(String(b.date)) || a.gamePk - b.gamePk);
|
||||
|
||||
const arm = new Map(); // pid -> { n, h, k } (all prior PAs)
|
||||
const penHist = new Map(); // team -> [{ quality, k }] per prior game
|
||||
const startHist = new Map(); // starter id -> [bf]
|
||||
const rows = [];
|
||||
|
||||
for (const g of games) {
|
||||
for (const side of ['home', 'away']) {
|
||||
const team = g[side].abbr || g[side].team;
|
||||
const st = (g[side].arms || []).find((a) => a.started);
|
||||
if (!team || !st) continue;
|
||||
const half = side === 'home' ? 'top' : 'bottom';
|
||||
const pas = g.pas.filter((p) => p.half === half && PA.has(p.event));
|
||||
const post = pas.filter((p) => p.pitcher !== st.id);
|
||||
|
||||
// ── LINK 1, recomputed point-in-time, to define the concentrated subset ──
|
||||
const priorStarts = startHist.get(st.id) || [];
|
||||
let predBf = null;
|
||||
if (priorStarts.length >= MIN_PRIOR_STARTS) {
|
||||
const w = priorStarts.length / (priorStarts.length + STABILIZE);
|
||||
predBf = w * mean(priorStarts) + (1 - w) * 21.56; // league mean
|
||||
}
|
||||
|
||||
// ── TARGET: the known quality of the arms that ACTUALLY appeared ──
|
||||
const faced = [];
|
||||
for (const p of post) {
|
||||
const h = arm.get(p.pitcher);
|
||||
if (!h || h.n < MIN_ARM_PA) continue; // abstain, never 0
|
||||
faced.push({ q: h.h / h.n, k: h.k / h.n });
|
||||
}
|
||||
|
||||
// ── PREDICTION: this club's own pen, from prior games only ──
|
||||
const hist = penHist.get(team) || [];
|
||||
if (faced.length && hist.length >= 5 && predBf !== null) {
|
||||
const predQ = mean(hist.map((x) => x.quality));
|
||||
const predK = mean(hist.map((x) => x.k));
|
||||
rows.push({
|
||||
team,
|
||||
gamePk: g.gamePk,
|
||||
date: g.date,
|
||||
pred_bf: predBf,
|
||||
early_flagged: predBf <= EARLY_FLAG_BF,
|
||||
pred_quality: predQ,
|
||||
actual_quality: mean(faced.map((f) => f.q)),
|
||||
pred_archetype: archetypeOf(predK),
|
||||
actual_archetype: archetypeOf(mean(faced.map((f) => f.k))),
|
||||
arms_faced: faced.length,
|
||||
});
|
||||
}
|
||||
|
||||
// Fold this game into history — never before predicting from it.
|
||||
if (faced.length) {
|
||||
penHist.set(team, hist.concat([{ quality: mean(faced.map((f) => f.q)), k: mean(faced.map((f) => f.k)) }]));
|
||||
}
|
||||
if (st.bf != null) startHist.set(st.id, priorStarts.concat([st.bf]));
|
||||
for (const p of pas) {
|
||||
const cur = arm.get(p.pitcher) || { n: 0, h: 0, k: 0 };
|
||||
cur.n += 1;
|
||||
cur.h += HIT.has(p.event) ? 1 : 0;
|
||||
cur.k += p.event === 'strikeout' ? 1 : 0;
|
||||
arm.set(p.pitcher, cur);
|
||||
}
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
function gateQuality(rows, leagueQ, cumulative, clusterKey, label) {
|
||||
return pg.adjudicate(rows.map((r) => ({
|
||||
cluster: r[clusterKey],
|
||||
baseline: leagueQ,
|
||||
prediction: r.pred_quality,
|
||||
actual: r.actual_quality,
|
||||
})), { link: label, loss: 'absolute', cumulativeTests: cumulative });
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const all = build();
|
||||
const subset = all.filter((r) => r.early_flagged);
|
||||
const leagueQ = mean(all.map((r) => r.actual_quality));
|
||||
|
||||
let cumulative = 1;
|
||||
try {
|
||||
const sb = createClient(process.env.SUPABASE_URL,
|
||||
process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY,
|
||||
{ auth: { persistSession: false } });
|
||||
const mc = await tl.recordAndCount(tl.supabaseStore(sb), [
|
||||
{ sport: 'mlb', stat: 'pen_quality', archetype: null, interaction: 'link2b:pen_quality', target: 'actual_arms' },
|
||||
{ sport: 'mlb', stat: 'pen_archetype', archetype: null, interaction: 'link2b:pen_archetype', target: 'actual_arms' },
|
||||
]);
|
||||
cumulative = mc.cumulative_tests;
|
||||
} catch { /* offline */ }
|
||||
|
||||
// ARCHETYPE grain — misclassification against the arms that actually appeared.
|
||||
const archRows = subset.filter((r) => r.pred_archetype && r.actual_archetype);
|
||||
const modal = (() => {
|
||||
const c = new Map();
|
||||
for (const r of all) c.set(r.actual_archetype, (c.get(r.actual_archetype) || 0) + 1);
|
||||
return [...c.entries()].sort((a, b) => b[1] - a[1])[0][0];
|
||||
})();
|
||||
const archGate = pg.adjudicate(archRows.map((r) => ({
|
||||
cluster: r.gamePk,
|
||||
baseline: r.actual_archetype === modal ? 1 : 0,
|
||||
prediction: r.actual_archetype === r.pred_archetype ? 1 : 0,
|
||||
actual: 1,
|
||||
})), { link: 'link2b_pen_archetype', loss: 'absolute', cumulativeTests: cumulative });
|
||||
|
||||
const acc = (rs, k) => (rs.length ? rs.filter((r) => r[k]).length / rs.length : null);
|
||||
|
||||
console.log(JSON.stringify({
|
||||
link: 'LINK 2 (coarse) — pen quality + reliever archetype',
|
||||
team_games_total: all.length,
|
||||
concentrated_subset_elevated_early_exit: subset.length,
|
||||
league_mean_pen_quality: round4(leagueQ),
|
||||
cumulative_tests: cumulative,
|
||||
|
||||
quality_grain: {
|
||||
on_concentrated_subset: gateQuality(subset, leagueQ, cumulative, 'gamePk', 'link2b_pen_quality_subset'),
|
||||
sensitivity_team_clustered: gateQuality(subset, leagueQ, cumulative, 'team', 'link2b_pen_quality_teamclust'),
|
||||
pooled_all_games: gateQuality(all, leagueQ, cumulative, 'gamePk', 'link2b_pen_quality_pooled'),
|
||||
},
|
||||
|
||||
archetype_grain: {
|
||||
n: archRows.length,
|
||||
modal_archetype: modal,
|
||||
baseline_accuracy_guess_modal: round4(acc(archRows.map((r) => ({ x: r.actual_archetype === modal })), 'x')),
|
||||
model_accuracy: round4(acc(archRows.map((r) => ({ x: r.actual_archetype === r.pred_archetype })), 'x')),
|
||||
verdict: archGate,
|
||||
},
|
||||
|
||||
cluster_note: '76% of game pen-quality variance is WITHIN team, so the game is the honest cluster; team-clustered reported as the conservative sensitivity',
|
||||
}, null, 2));
|
||||
process.exit(0);
|
||||
})();
|
||||
|
||||
const round4 = (v) => (v == null ? null : Math.round(v * 10000) / 10000);
|
||||
Reference in New Issue
Block a user