Reliever chain: Link 1 proves, Link 2 does not, and the premise inverts
The causal insight is right -- the game is a sequence and the matchup does shift mid-game. The direction is backwards, measured on 93,663 plate appearances from 1,238 games pulled free from statsapi. LINK 1 PROVES. Starter batters-faced, point-in-time from his own prior starts only, clustered on the pitcher: MAE 3.2226 -> 2.7990, delta -0.4236, CI [-0.6006,-0.2731] at 0.9995 corrected for 107 tests, 1,706 starts across 204 pitchers. It finds the tail the chain needed -- early exits are a 23.2% base rate, model-flagged starts are 34.0% early, lift +10.8pp. Scope correction inside Link 1: the order specifies fatigue x GAME SCRIPT, but game script is not available at grade time -- whether he gets hit tonight is the thing being projected, not an input to it. Only the workload half is measured; the in-game half is recorded as a live feature, out of scope, rather than quietly folded in. LINK 2 DOES NOT PROVE, twice over. Model accuracy 17.2% vs an 8.6% baseline -- doubling it sounds good and is not, since naming a specific arm is wrong five times in six. And structurally the entity is the BULLPEN: 39,629 post-starter plate appearances across 30 clubs is 30 readings, below the 40-cluster floor, the same permanent ceiling as park geometry and team defence. LINK 3 NOT RUN, per the order's own rule. THE PREMISE IS REFUTED, and this chains on nothing so it was safe to measure: vs STARTER n=48,492 hit rate 0.2444 +/-0.0038 vs BULLPEN n=35,760 hit rate 0.2373 +/-0.0044 The pen is 0.7pp HARDER. The specific effect the chain exists to exploit -- early exit making later at-bats softer -- is +0.0010 on 35,760 PAs. A well-powered null, not a sample problem. What IS real is times through the order: TTO1 0.2351 -> TTO2 0.2515 -> TTO3 0.2518. A starter does decay as the lineup sees him again, but that advantage is SURRENDERED when he leaves, not extended -- the pen is harder than his second and third time through. A modern bullpen is a queue of fresh specialists throwing one inning each; there is no tiring arm to punish. So the insight survives inverted, and Link 1 stays valuable for the opposite reason it was built: a likely early hook predicts the hitter LOSES his third-time-through look (0.2518 -> 0.2373 on that PA). The mispricing is on hitters who get an EXTRA look at a starter going deep. BUILT: predictionGate.js + tests -- the two-part gate for a continuous prediction. factorGate binarises outcomes for Brier, which would destroy a target like batters faced. Same discipline, same THEATER verdict, real scale. PRE-REGISTERED NOT RUN: Link 2' using a PA-weighted bullpen AGGREGATE rather than a named arm. Recorded rather than substituted in -- running Link 3 on a swapped-in Link 2 is the assumed-link failure the order forbids. Given the premise result its expected value is now low. PARALLEL TRACK logged: total_bases n=948 pooled, BOMBER x TB 340, short by 160. Sample-readiness only, not a verdict. Counter and frozen clusters byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,100 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* The two-part gate for a CONTINUOUS prediction.
|
||||
*
|
||||
* Same discipline as factorGate, different units — and the same dangerous
|
||||
* failure: a link that moves off the naive baseline while predicting nothing
|
||||
* makes the projection LOOK like it read the game script.
|
||||
*/
|
||||
|
||||
const pg = require('../../src/services/model/predictionGate');
|
||||
|
||||
/** n rows where the prediction tracks truth to a given degree. */
|
||||
function rows(n, { skill = 1, clusters = 60, seed = 5 } = {}) {
|
||||
let s = seed;
|
||||
const rnd = () => (s = (s * 1103515245 + 12345) % 2147483648) / 2147483648;
|
||||
const out = [];
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
const actual = 20 + (rnd() - 0.5) * 12;
|
||||
const noise = (rnd() - 0.5) * 12;
|
||||
out.push({
|
||||
cluster: `c${i % clusters}`,
|
||||
baseline: 20,
|
||||
prediction: 20 + skill * (actual - 20) + (1 - skill) * noise,
|
||||
actual,
|
||||
});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('a link that genuinely predicts', () => {
|
||||
it('PROVES when it beats the naive baseline out-of-sample', () => {
|
||||
const v = pg.adjudicate(rows(1200, { skill: 0.8 }), { link: 'good' });
|
||||
expect(v.verdict).toBe('PROVES');
|
||||
expect(v.improvement.loss_delta).toBeLessThan(0);
|
||||
expect(v.improvement.ci[1]).toBeLessThan(0);
|
||||
});
|
||||
|
||||
it('does NOT binarise the target — that is why factorGate cannot do this job', () => {
|
||||
// Brier collapses the outcome to 0/1. A target like "batters faced" would be
|
||||
// destroyed by that, so the loss here stays on the real scale.
|
||||
const v = pg.adjudicate(rows(1200, { skill: 0.9 }), { link: 'scale' });
|
||||
expect(v.improvement.loss_baseline).toBeGreaterThan(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the failures it must name', () => {
|
||||
it('THEATER — moves off the baseline and predicts nothing', () => {
|
||||
const v = pg.adjudicate(rows(1200, { skill: 0 }), { link: 'noise' });
|
||||
expect(v.verdict).toBe('THEATER');
|
||||
expect(v.movement.mean_abs_shift).toBeGreaterThan(0);
|
||||
expect(v.consequence).toMatch(/LOOK like it read/);
|
||||
});
|
||||
|
||||
it('INERT — never departs from the baseline at all', () => {
|
||||
const flat = rows(1200, { skill: 0 }).map((r) => ({ ...r, prediction: r.baseline }));
|
||||
const v = pg.adjudicate(flat, { link: 'flat', minMovement: 0.01 });
|
||||
expect(v.verdict).toBe('INERT');
|
||||
});
|
||||
|
||||
it('thin sample is PENDING, never a verdict', () => {
|
||||
const v = pg.adjudicate(rows(100, { skill: 0.9 }), { link: 'thin' });
|
||||
expect(v.verdict).toBe('PENDING_SAMPLE');
|
||||
expect(v.rows_needed).toBe(400);
|
||||
});
|
||||
});
|
||||
|
||||
describe('replication is counted in arms, not in starts', () => {
|
||||
it('refuses when the entity it rides on has too few clusters', () => {
|
||||
// 39,629 post-starter plate appearances across 30 bullpens is 30 readings.
|
||||
const v = pg.adjudicate(rows(5000, { skill: 0.9, clusters: 30 }), { link: 'bullpen' });
|
||||
expect(v.verdict).toBe('PENDING_SAMPLE');
|
||||
expect(v.reason).toMatch(/30 independent clusters < 40/);
|
||||
expect(v.clusters_needed).toBe(10);
|
||||
});
|
||||
|
||||
it('the clustered interval is wider than the unclustered one', () => {
|
||||
const r = rows(1500, { skill: 0.5, clusters: 45 });
|
||||
const clustered = pg.adjudicate(r, { link: 'a' });
|
||||
const flat = pg.adjudicate(r.map(({ cluster, ...x }) => x), { link: 'b' });
|
||||
const w = (v) => v.improvement.ci[1] - v.improvement.ci[0];
|
||||
expect(w(clustered)).toBeGreaterThan(w(flat));
|
||||
});
|
||||
|
||||
it('the cumulative correction widens the interval', () => {
|
||||
const r = rows(1500, { skill: 0.6 });
|
||||
const one = pg.adjudicate(r, { cumulativeTests: 1 });
|
||||
const many = pg.adjudicate(r, { cumulativeTests: 108 });
|
||||
expect(many.improvement.ci_level).toBeGreaterThan(one.improvement.ci_level);
|
||||
});
|
||||
});
|
||||
|
||||
describe('honesty', () => {
|
||||
it('an unreadable row is dropped, never zero-filled', () => {
|
||||
const r = rows(600, { skill: 0.8 });
|
||||
r[0].prediction = null; r[1].actual = null; r[2].baseline = null;
|
||||
const v = pg.adjudicate(r, { minN: 100 });
|
||||
expect(v.improvement.n).toBe(597);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user