Files
vyndr/tests/unit/skillModel.test.js
T
builtbykev 4aab18096f Prove both on total bases -- and find that my own fix destroyed the backtest
Nothing passed. Nothing promoted. Counter byte-identical.

THE BLOCKER, which is the real finding. statcast_aggregates is upserted in
place and holds exactly one as-of date. Yesterday's skill backtest was honest
only by accident: the nightly refresh was unreachable code, so the profiles
sat frozen at 2026-07-21 -- before the settled window. Repairing that cron was
right for production and it refreshed them to today, destroying every prior
version. Scoring a 2026-07-25 game now uses a season aggregate that contains
that game. Point-in-time validation is structurally impossible from that
table, so every number in this run is contaminated and directional, and none
of it is a gate verdict.

Fixed forward: statcast_history retains a dated snapshot on every refresh, so
point-in-time becomes "as_of_date < game_date, most recent". Retention is
best-effort and cannot fail the refresh; both properties are unit-tested. It
has one day of data, which is not yet a window.

SOLO BASELINE, n=383, Bonferroni across 12 tests (alpha 0.00417): nothing
passes. hard_hit_pct is closest at marginal r 0.135 with p 0.0080, failing
both the 0.15 effect bar and the corrected alpha. And it drifted DOWN from
0.153 at n=295 -- an estimate regressing as noise averages out, not an effect
firming up. I called that number encouraging yesterday; on 88 more rows it is
fading, and it should not keep being quoted at its best value.

INTERACTIONS, each scored by partial correlation against the counter residual
controlling for both of its own components: none pass. Only barrel x power
archetype has an incremental exceeding its parts (-0.101 against 0.019) at
n=260 -- the shape Discipline 2 predicts, but a lead, not a finding.

A methodological catch worth keeping. The archetype conditioner was first
built as barrel_pct over league barrel -- a monotone transform of one of its
own components -- so the "interaction" was barrel squared, measuring
nonlinearity in barrel rate rather than any archetype effect, and it produced
this run's only positive result. A Gauss-Jordan pivot test does not catch that,
because the two columns differ by a scale factor. Fixed with a scale-free
collinearity check plus real archetype labels joined from model_snapshots.
Without it this document would have reported a fabricated interaction as the
session's finding.

COMBINED vs COUNTER on total bases: 0.2718 against 0.2647, delta +0.0071, CI
[-0.065, +0.079] -- inconclusive, and the first time a challenger has not
lost. The same engine on hits was -0.116 with a CI excluding zero. That
contrast is the whole argument for total bases, and it is what the physics
said: contact quality governs extra bases, not whether a grounder finds a hole.

Also built: the compound TB projection. skillProjection no longer refuses
total bases -- a deterministic bases-per-hit multiplier had made P(TB>=2)
exactly P(hits>=1), a relabelled hits curve. It is now a convolution over
per-PA base outcomes with hit-type shares shifted by skill. Non-degeneracy is
locked by test.

4,204 tests green (334 suites); web build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-03 16:27:11 -04:00

246 lines
13 KiB
JavaScript

'use strict';
/**
* THE FIVE DISCIPLINES, asserted as behaviour.
*
* These are not coverage tests. Each block locks one discipline so it cannot be
* quietly abandoned later — which is exactly how the previous engine drifted
* into a frequency counter with six wired-but-dead features hanging off it.
*/
const reg = require('../../src/services/model/featureRegistry');
const sk = require('../../src/services/model/skillProjection');
const BOMBER = { k_pct: 0.28, bb_pct: 0.11, barrel_pct: 0.16, hard_hit_pct: 0.52, avg_exit_velo: 93.5, avg_launch_angle: 19 };
const GHOST = { k_pct: 0.14, bb_pct: 0.07, barrel_pct: 0.02, hard_hit_pct: 0.30, avg_exit_velo: 85.0, avg_launch_angle: 6 };
const ACE = { k_pct: 0.32, bb_pct: 0.05, hard_hit_pct: 0.30, whiff_pct: 0.33 };
const BATTING_PRACTICE = { k_pct: 0.15, bb_pct: 0.10, hard_hit_pct: 0.46, whiff_pct: 0.18 };
afterEach(() => reg.__reset());
// ── DISCIPLINE 3 ──────────────────────────────────────────────────────────
describe('D3 — earn its place or it is out', () => {
it('starts with an essentially EMPTY proven set — nothing is assumed to work', () => {
const live = reg.liveFeatures('mlb');
// Only the incumbent counter is PROVEN, and only because it was measured.
expect([...live]).toEqual(['recent_frequency_prior']);
});
it('refuses promotion without a real sample, positive lift, and a CI above zero', () => {
expect(reg.promote('mlb', 'batter_barrel_pct', null).ok).toBe(false);
expect(reg.promote('mlb', 'batter_barrel_pct', { n: 10, lift: 0.05, ci95: [0.01, 0.09] }).ok).toBe(false); // n too small
expect(reg.promote('mlb', 'batter_barrel_pct', { n: 500, lift: -0.02, ci95: [-0.05, -0.01] }).ok).toBe(false); // negative lift
expect(reg.promote('mlb', 'batter_barrel_pct', { n: 500, lift: 0.03, ci95: [-0.01, 0.07] }).ok).toBe(false); // CI straddles zero
expect(reg.statusOf('mlb', 'batter_barrel_pct')).toBe(reg.STATUS.CANDIDATE);
});
it('promotes on sufficient evidence, and records WHY', () => {
const ev = { n: 600, lift: 0.04, ci95: [0.012, 0.068], measured_at: '2026-09-01' };
expect(reg.promote('mlb', 'batter_barrel_pct', ev).ok).toBe(true);
expect(reg.isLive('mlb', 'batter_barrel_pct')).toBe(true);
expect(reg.liveFeatures('mlb').has('batter_barrel_pct')).toBe(true);
});
it('keeps DEAD features BY NAME so they are not silently rebuilt', () => {
const s = reg.summary('mlb');
const deadKeys = s.dead.map((d) => d.key);
expect(deadKeys).toEqual(expect.arrayContaining([
'champion_opp_rank_adj', 'champion_home_away_adj', 'champion_consistency_pull',
]));
// and each carries the reason it died
expect(s.dead.find((d) => d.key === 'champion_home_away_adj').why).toMatch(/harmful|IMPROVED/i);
});
it('per-sport: a feature proven for MLB says nothing about another sport', () => {
reg.promote('mlb', 'batter_barrel_pct', { n: 600, lift: 0.04, ci95: [0.01, 0.07] });
expect(reg.isLive('nba', 'batter_barrel_pct')).toBe(false);
expect(reg.statusOf('nba', 'batter_barrel_pct')).toBeNull();
});
it('THE GATE: with only PROVEN features allowed, the skill model REFUSES', () => {
// The proven set carries no contact-quality feature yet, so there is no
// forward read to make. Refusing is the correct output — this is what stops
// an unproven model reaching a user by accident.
const out = sk.projectSkill({
batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5,
expectedPa: 4.2, allowed: reg.liveFeatures('mlb'),
});
expect(out).toBeNull();
});
it('with CANDIDATES allowed (the challenger), it produces a read', () => {
const out = sk.projectSkill({
batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5,
expectedPa: 4.2, allowed: reg.candidateFeatures('mlb'),
});
expect(out).not.toBeNull();
expect(out.p_over_line).toBeGreaterThan(0);
});
});
describe('D3 — verdicts are PER STAT, because the physics differ', () => {
it('a feature can be DEAD for one stat and still CANDIDATE for another', () => {
// The first real gate run: every skill feature failed for `hits` (n=570,
// max |r| 0.062) while hard-hit rate showed the strongest marginal
// correlation yet seen for `total_bases` (r=0.153). A sport-wide DEAD mark
// would have killed, on hits evidence, the features most alive on TB.
reg.recordStatVerdict('mlb', 'hits', 'batter_barrel_pct', reg.STATUS.DEAD,
{ n: 570, r: -0.037, note: 'failed the gate for hits' });
expect(reg.statusForStat('mlb', 'hits', 'batter_barrel_pct')).toBe(reg.STATUS.DEAD);
expect(reg.statusForStat('mlb', 'total_bases', 'batter_barrel_pct')).toBe(reg.STATUS.CANDIDATE);
expect(reg.candidateFeaturesForStat('mlb', 'hits').has('batter_barrel_pct')).toBe(false);
expect(reg.candidateFeaturesForStat('mlb', 'total_bases').has('batter_barrel_pct')).toBe(true);
});
it('a stat with no verdict inherits the sport-level status', () => {
expect(reg.statusForStat('mlb', 'runs', 'batter_exit_velo')).toBe(reg.STATUS.CANDIDATE);
expect(reg.statusForStat('mlb', 'runs', 'champion_home_away_adj')).toBe(reg.STATUS.DEAD);
});
it('the live gate is per stat too, and still PROVEN-only', () => {
expect(reg.liveFeaturesForStat('mlb', 'hits').has('batter_barrel_pct')).toBe(false);
expect([...reg.liveFeaturesForStat('mlb', 'hits')]).toEqual(['recent_frequency_prior']);
});
});
// ── DISCIPLINE 1 ──────────────────────────────────────────────────────────
describe('D1 — skill, not results: the pitcher actually moves the read', () => {
const run = (pitcher) => sk.projectSkill({
batter: BOMBER, pitcher, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2,
});
it('a better pitcher LOWERS the projection for the same hitter', () => {
const vsAce = run(ACE);
const vsBP = run(BATTING_PRACTICE);
expect(vsAce.p_over_line).toBeLessThan(vsBP.p_over_line);
expect(vsAce.projected_value).toBeLessThan(vsBP.projected_value);
});
it('this is the thing the counter cannot do — no pitcher at all is a different read', () => {
const noPitcher = run(null);
expect(noPitcher).not.toBeNull();
expect(noPitcher.pitcher_applied).toBe(false);
expect(run(ACE).pitcher_applied).toBe(true);
});
it('the odds ratio returns league when both sides are league (the identity)', () => {
expect(sk.oddsRatio(0.222, 0.222, 0.222)).toBeCloseTo(0.222, 6);
});
it('an average pitcher leaves the batter rate untouched', () => {
expect(sk.oddsRatio(0.30, 0.222, 0.222)).toBeCloseTo(0.30, 6);
});
it('shrinks a thin sample toward its anchor, and leaves a full one alone', () => {
expect(sk.shrink(0.60, 10, 200, 0.39)).toBeLessThan(0.45); // 10-PA callup
expect(sk.shrink(0.60, 5000, 200, 0.39)).toBeGreaterThan(0.59); // established
});
});
// ── DISCIPLINE 2 ──────────────────────────────────────────────────────────
describe('D2 — archetype SELECTS features, it does not nudge', () => {
it('the SAME hitter read through two archetypes gets materially different reads', () => {
const asBomber = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2 });
const asGhost = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'GHOST', statType: 'hits', line: 0.5, expectedPa: 4.2 });
// Not a rounding difference — a different feature map entirely.
expect(Math.abs(asBomber.p_over_line - asGhost.p_over_line)).toBeGreaterThan(0.15);
});
it("a BOMBER's read is driven by barrels; a GHOST's is nearly blind to them", () => {
expect(sk.featureMapFor('BOMBER').hitWeights.barrel).toBeGreaterThan(0.4);
expect(sk.featureMapFor('GHOST').hitWeights.barrel).toBeLessThan(0.1);
expect(sk.featureMapFor('GHOST').hitWeights.gb_speed).toBeGreaterThan(0.5);
expect(sk.featureMapFor('BOMBER').hitWeights.gb_speed).toBe(0);
});
it('barrel rate moves a BOMBER and barely moves a GHOST — features silent where they do not apply', () => {
const move = (arch) => {
const lo = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: 0.04 }, pitcher: ACE, park: 1, archetype: arch });
const hi = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: 0.20 }, pitcher: ACE, park: 1, archetype: arch });
return hi - lo;
};
expect(move('BOMBER')).toBeGreaterThan(move('GHOST') * 3);
});
it('an unknown archetype falls back to a balanced map, not a refusal', () => {
const out = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'NOT_A_REAL_ONE', statType: 'hits', line: 0.5, expectedPa: 4.2 });
expect(out).not.toBeNull();
expect(out.archetype).toBe('NOT_A_REAL_ONE');
});
});
// ── HONESTY ───────────────────────────────────────────────────────────────
describe('honesty — unknown is not zero, and absent beats invented', () => {
it('no hitter profile → NO forward read (null), never a league-average guess', () => {
expect(sk.projectSkill({ batter: null, pitcher: ACE, statType: 'hits', line: 0.5 })).toBeNull();
expect(sk.projectSkill({ batter: {}, pitcher: ACE, statType: 'hits', line: 0.5 })).toBeNull();
});
it('a missing skill input is SILENT — it does not act as a measured zero', () => {
const full = sk.hitOnContact({ batter: BOMBER, pitcher: ACE, park: 1, archetype: 'BOMBER' });
const noBarrel = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: null }, pitcher: ACE, park: 1, archetype: 'BOMBER' });
const zeroBarrel = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: 0 }, pitcher: ACE, park: 1, archetype: 'BOMBER' });
expect(noBarrel).not.toBeNull();
// A REAL zero is a fact and must read far lower than an ABSENT one.
expect(zeroBarrel).toBeLessThan(noBarrel);
expect(Math.abs(noBarrel - full)).toBeLessThan(Math.abs(zeroBarrel - full));
});
it('total_bases is COMPOUND, not a relabelled hits curve', () => {
// The first cut multiplied hits by a constant bases-per-hit, which made
// P(TB>=2) EXACTLY equal to P(hits>=1) — a relabel carrying no information a
// hits model did not already have. It was refused rather than shipped. It is
// now a real convolution over per-PA base outcomes, and this test locks the
// property that distinguishes the two.
const tb = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'total_bases', line: 1.5, expectedPa: 4.2 });
const hits = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2 });
expect(tb).not.toBeNull();
expect(tb.family).toBe('pa_compound_bases_convolution');
expect(tb.p_over_line).not.toBeCloseTo(hits.p_over_line, 3); // NOT degenerate
expect(tb.distribution.reduce((a, b) => a + b, 0)).toBeCloseTo(1, 2);
});
it('hit-type shares respond to power skill — a slugger homers more per hit', () => {
const slugger = sk.hitTypeShares({ batter: BOMBER, archetype: 'BOMBER' });
const slap = sk.hitTypeShares({ batter: GHOST, archetype: 'GHOST' });
expect(slugger.homer).toBeGreaterThan(slap.homer * 3);
expect(slap.single).toBeGreaterThan(slugger.single);
// Shares are a distribution over hit types.
for (const s of [slugger, slap]) {
expect(s.single + s.double + s.triple + s.homer).toBeCloseTo(1, 6);
}
});
it('an absent power read leaves the shares at league — never a guessed lean', () => {
const noSkill = sk.hitTypeShares({ batter: { k_pct: 0.2 }, archetype: 'DEFAULT' });
expect(noSkill).toEqual(sk.LEAGUE_HIT_SHARES);
});
it('the distribution is a real distribution and P(>=k) is monotone', () => {
const out = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2 });
expect(out.distribution.reduce((a, b) => a + b, 0)).toBeCloseTo(1, 2);
let prev = 1;
for (let k = 1; k <= 5; k += 1) {
const p = sk.atLeast(out.distribution, k);
expect(p).toBeLessThanOrEqual(prev + 1e-9);
prev = p;
}
});
it('a hitter exactly at league on every axis lands on league BABIP — no invented lean', () => {
const leagueBat = {
k_pct: sk.LEAGUE.k_pct, bb_pct: sk.LEAGUE.bb_pct,
barrel_pct: sk.LEAGUE.barrel_pct, hard_hit_pct: sk.LEAGUE.hard_hit_pct,
avg_exit_velo: sk.LEAGUE.avg_exit_velo, avg_launch_angle: 12,
};
const boc = sk.hitOnContact({ batter: leagueBat, pitcher: { hard_hit_pct: sk.LEAGUE.hard_hit_pct }, park: 1, archetype: 'DEFAULT' });
expect(boc).toBeCloseTo(sk.LEAGUE.babip, 2);
});
it('opportunity matters: more projected PA means a higher chance of one hit', () => {
const few = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 2 });
const many = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 5 });
expect(many.p_over_line).toBeGreaterThan(few.p_over_line);
});
});