The skill engine: built, gated by construction, and Stage A honestly lost
Built src/services/model/ -- the forward, archetype-selected, skill-based projection, as a challenger. The champion is untouched. featureRegistry makes "earn its place or it's out" structural rather than aspirational: CANDIDATE / PROVEN / DEAD per feature per sport, liveFeatures() returns PROVEN only, promotion requires n>=200 with positive lift and a CI excluding zero, and there is deliberately no override argument. It ships with exactly ONE proven feature -- the incumbent counter, because it is the only one with a measurement. A test asserts that with only PROVEN features allowed the projection returns null, so an unproven model cannot reach a user by accident. The three champion adjustment layers are registered DEAD with their reasons so they cannot be silently rebuilt. skillProjection is a PA outcome tree: K and BB combined by log5 odds-ratio against league (both identities unit-tested), then archetype-weighted contact quality against contact allowed, then Binomial(PA, p_hit) mixed over a PA distribution. Archetype is a FEATURE SELECTOR, not a nudge -- BOMBER reads barrels at 0.50 and ground-ball speed at 0.00, GHOST inverts it -- and a test locks that the same hitter read two ways moves more than 0.15. STAGE A: IT LOSES. Out-of-sample on 570 settled hits props with 91.9% opposing-pitcher coverage, resolution 0.0499 against the champion's 0.166, delta -0.116 with CI [-0.189, -0.043]. It is not selective either: its eight most confident picks hit 50%, a lift of -0.065. Not promoted. The gate did its job on its first real test, which is the point of having built it that way. Two false starts, both recorded because they nearly produced a wrong verdict: statcast_aggregates stores PERCENTAGES, so raw rows made bip = 1-29.6-17.1 and refused 568 of 576 -- the honest-absent guards made a units bug loud instead of silent, and the conversion now lives at one chokepoint. And the first run resolved an opposing pitcher for 1 of 570 rows, because ledger team/opponent are NULL, so it would have reported "skill-v1 loses" while measuring a batter-only model with no matchup in it at all. The verdict above is from the corrected run. The loss is real but partial: park was passed as 1.0, handedness and opportunity_drift never fired, PA is season-PA over a constant, and the skill profiles carry no recency at all while the champion has a last-5 term. Also fixed: the Statcast nightly refresh was unreachable code. It sat inside tick() below "if (!HOURS_UTC.includes(h)) return" while testing h === 11, so it had never run once; the aggregates were 13 days stale and both of its alerts were in the same dead branch. It now runs on its own tick, and the test that passed happily throughout -- it only checked the string existed -- is replaced by one that asserts it is not behind the guard. 4,182 tests green (333 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -0,0 +1,197 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* THE FIVE DISCIPLINES, asserted as behaviour.
|
||||
*
|
||||
* These are not coverage tests. Each block locks one discipline so it cannot be
|
||||
* quietly abandoned later — which is exactly how the previous engine drifted
|
||||
* into a frequency counter with six wired-but-dead features hanging off it.
|
||||
*/
|
||||
|
||||
const reg = require('../../src/services/model/featureRegistry');
|
||||
const sk = require('../../src/services/model/skillProjection');
|
||||
|
||||
const BOMBER = { k_pct: 0.28, bb_pct: 0.11, barrel_pct: 0.16, hard_hit_pct: 0.52, avg_exit_velo: 93.5, avg_launch_angle: 19 };
|
||||
const GHOST = { k_pct: 0.14, bb_pct: 0.07, barrel_pct: 0.02, hard_hit_pct: 0.30, avg_exit_velo: 85.0, avg_launch_angle: 6 };
|
||||
const ACE = { k_pct: 0.32, bb_pct: 0.05, hard_hit_pct: 0.30, whiff_pct: 0.33 };
|
||||
const BATTING_PRACTICE = { k_pct: 0.15, bb_pct: 0.10, hard_hit_pct: 0.46, whiff_pct: 0.18 };
|
||||
|
||||
afterEach(() => reg.__reset());
|
||||
|
||||
// ── DISCIPLINE 3 ──────────────────────────────────────────────────────────
|
||||
describe('D3 — earn its place or it is out', () => {
|
||||
it('starts with an essentially EMPTY proven set — nothing is assumed to work', () => {
|
||||
const live = reg.liveFeatures('mlb');
|
||||
// Only the incumbent counter is PROVEN, and only because it was measured.
|
||||
expect([...live]).toEqual(['recent_frequency_prior']);
|
||||
});
|
||||
|
||||
it('refuses promotion without a real sample, positive lift, and a CI above zero', () => {
|
||||
expect(reg.promote('mlb', 'batter_barrel_pct', null).ok).toBe(false);
|
||||
expect(reg.promote('mlb', 'batter_barrel_pct', { n: 10, lift: 0.05, ci95: [0.01, 0.09] }).ok).toBe(false); // n too small
|
||||
expect(reg.promote('mlb', 'batter_barrel_pct', { n: 500, lift: -0.02, ci95: [-0.05, -0.01] }).ok).toBe(false); // negative lift
|
||||
expect(reg.promote('mlb', 'batter_barrel_pct', { n: 500, lift: 0.03, ci95: [-0.01, 0.07] }).ok).toBe(false); // CI straddles zero
|
||||
expect(reg.statusOf('mlb', 'batter_barrel_pct')).toBe(reg.STATUS.CANDIDATE);
|
||||
});
|
||||
|
||||
it('promotes on sufficient evidence, and records WHY', () => {
|
||||
const ev = { n: 600, lift: 0.04, ci95: [0.012, 0.068], measured_at: '2026-09-01' };
|
||||
expect(reg.promote('mlb', 'batter_barrel_pct', ev).ok).toBe(true);
|
||||
expect(reg.isLive('mlb', 'batter_barrel_pct')).toBe(true);
|
||||
expect(reg.liveFeatures('mlb').has('batter_barrel_pct')).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps DEAD features BY NAME so they are not silently rebuilt', () => {
|
||||
const s = reg.summary('mlb');
|
||||
const deadKeys = s.dead.map((d) => d.key);
|
||||
expect(deadKeys).toEqual(expect.arrayContaining([
|
||||
'champion_opp_rank_adj', 'champion_home_away_adj', 'champion_consistency_pull',
|
||||
]));
|
||||
// and each carries the reason it died
|
||||
expect(s.dead.find((d) => d.key === 'champion_home_away_adj').why).toMatch(/harmful|IMPROVED/i);
|
||||
});
|
||||
|
||||
it('per-sport: a feature proven for MLB says nothing about another sport', () => {
|
||||
reg.promote('mlb', 'batter_barrel_pct', { n: 600, lift: 0.04, ci95: [0.01, 0.07] });
|
||||
expect(reg.isLive('nba', 'batter_barrel_pct')).toBe(false);
|
||||
expect(reg.statusOf('nba', 'batter_barrel_pct')).toBeNull();
|
||||
});
|
||||
|
||||
it('THE GATE: with only PROVEN features allowed, the skill model REFUSES', () => {
|
||||
// The proven set carries no contact-quality feature yet, so there is no
|
||||
// forward read to make. Refusing is the correct output — this is what stops
|
||||
// an unproven model reaching a user by accident.
|
||||
const out = sk.projectSkill({
|
||||
batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5,
|
||||
expectedPa: 4.2, allowed: reg.liveFeatures('mlb'),
|
||||
});
|
||||
expect(out).toBeNull();
|
||||
});
|
||||
|
||||
it('with CANDIDATES allowed (the challenger), it produces a read', () => {
|
||||
const out = sk.projectSkill({
|
||||
batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5,
|
||||
expectedPa: 4.2, allowed: reg.candidateFeatures('mlb'),
|
||||
});
|
||||
expect(out).not.toBeNull();
|
||||
expect(out.p_over_line).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ── DISCIPLINE 1 ──────────────────────────────────────────────────────────
|
||||
describe('D1 — skill, not results: the pitcher actually moves the read', () => {
|
||||
const run = (pitcher) => sk.projectSkill({
|
||||
batter: BOMBER, pitcher, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2,
|
||||
});
|
||||
|
||||
it('a better pitcher LOWERS the projection for the same hitter', () => {
|
||||
const vsAce = run(ACE);
|
||||
const vsBP = run(BATTING_PRACTICE);
|
||||
expect(vsAce.p_over_line).toBeLessThan(vsBP.p_over_line);
|
||||
expect(vsAce.projected_value).toBeLessThan(vsBP.projected_value);
|
||||
});
|
||||
|
||||
it('this is the thing the counter cannot do — no pitcher at all is a different read', () => {
|
||||
const noPitcher = run(null);
|
||||
expect(noPitcher).not.toBeNull();
|
||||
expect(noPitcher.pitcher_applied).toBe(false);
|
||||
expect(run(ACE).pitcher_applied).toBe(true);
|
||||
});
|
||||
|
||||
it('the odds ratio returns league when both sides are league (the identity)', () => {
|
||||
expect(sk.oddsRatio(0.222, 0.222, 0.222)).toBeCloseTo(0.222, 6);
|
||||
});
|
||||
|
||||
it('an average pitcher leaves the batter rate untouched', () => {
|
||||
expect(sk.oddsRatio(0.30, 0.222, 0.222)).toBeCloseTo(0.30, 6);
|
||||
});
|
||||
|
||||
it('shrinks a thin sample toward its anchor, and leaves a full one alone', () => {
|
||||
expect(sk.shrink(0.60, 10, 200, 0.39)).toBeLessThan(0.45); // 10-PA callup
|
||||
expect(sk.shrink(0.60, 5000, 200, 0.39)).toBeGreaterThan(0.59); // established
|
||||
});
|
||||
});
|
||||
|
||||
// ── DISCIPLINE 2 ──────────────────────────────────────────────────────────
|
||||
describe('D2 — archetype SELECTS features, it does not nudge', () => {
|
||||
it('the SAME hitter read through two archetypes gets materially different reads', () => {
|
||||
const asBomber = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2 });
|
||||
const asGhost = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'GHOST', statType: 'hits', line: 0.5, expectedPa: 4.2 });
|
||||
// Not a rounding difference — a different feature map entirely.
|
||||
expect(Math.abs(asBomber.p_over_line - asGhost.p_over_line)).toBeGreaterThan(0.15);
|
||||
});
|
||||
|
||||
it("a BOMBER's read is driven by barrels; a GHOST's is nearly blind to them", () => {
|
||||
expect(sk.featureMapFor('BOMBER').hitWeights.barrel).toBeGreaterThan(0.4);
|
||||
expect(sk.featureMapFor('GHOST').hitWeights.barrel).toBeLessThan(0.1);
|
||||
expect(sk.featureMapFor('GHOST').hitWeights.gb_speed).toBeGreaterThan(0.5);
|
||||
expect(sk.featureMapFor('BOMBER').hitWeights.gb_speed).toBe(0);
|
||||
});
|
||||
|
||||
it('barrel rate moves a BOMBER and barely moves a GHOST — features silent where they do not apply', () => {
|
||||
const move = (arch) => {
|
||||
const lo = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: 0.04 }, pitcher: ACE, park: 1, archetype: arch });
|
||||
const hi = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: 0.20 }, pitcher: ACE, park: 1, archetype: arch });
|
||||
return hi - lo;
|
||||
};
|
||||
expect(move('BOMBER')).toBeGreaterThan(move('GHOST') * 3);
|
||||
});
|
||||
|
||||
it('an unknown archetype falls back to a balanced map, not a refusal', () => {
|
||||
const out = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'NOT_A_REAL_ONE', statType: 'hits', line: 0.5, expectedPa: 4.2 });
|
||||
expect(out).not.toBeNull();
|
||||
expect(out.archetype).toBe('NOT_A_REAL_ONE');
|
||||
});
|
||||
});
|
||||
|
||||
// ── HONESTY ───────────────────────────────────────────────────────────────
|
||||
describe('honesty — unknown is not zero, and absent beats invented', () => {
|
||||
it('no hitter profile → NO forward read (null), never a league-average guess', () => {
|
||||
expect(sk.projectSkill({ batter: null, pitcher: ACE, statType: 'hits', line: 0.5 })).toBeNull();
|
||||
expect(sk.projectSkill({ batter: {}, pitcher: ACE, statType: 'hits', line: 0.5 })).toBeNull();
|
||||
});
|
||||
|
||||
it('a missing skill input is SILENT — it does not act as a measured zero', () => {
|
||||
const full = sk.hitOnContact({ batter: BOMBER, pitcher: ACE, park: 1, archetype: 'BOMBER' });
|
||||
const noBarrel = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: null }, pitcher: ACE, park: 1, archetype: 'BOMBER' });
|
||||
const zeroBarrel = sk.hitOnContact({ batter: { ...BOMBER, barrel_pct: 0 }, pitcher: ACE, park: 1, archetype: 'BOMBER' });
|
||||
expect(noBarrel).not.toBeNull();
|
||||
// A REAL zero is a fact and must read far lower than an ABSENT one.
|
||||
expect(zeroBarrel).toBeLessThan(noBarrel);
|
||||
expect(Math.abs(noBarrel - full)).toBeLessThan(Math.abs(zeroBarrel - full));
|
||||
});
|
||||
|
||||
it('total_bases is REFUSED rather than shipped as a relabelled hits curve', () => {
|
||||
// A deterministic bases-per-hit multiplier made P(TB>=2) identical to
|
||||
// P(hits>=1). Refusing is correct until a real per-hit bases distribution
|
||||
// exists (tb-v1's compound shape).
|
||||
expect(sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'total_bases', line: 1.5 })).toBeNull();
|
||||
});
|
||||
|
||||
it('the distribution is a real distribution and P(>=k) is monotone', () => {
|
||||
const out = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2 });
|
||||
expect(out.distribution.reduce((a, b) => a + b, 0)).toBeCloseTo(1, 2);
|
||||
let prev = 1;
|
||||
for (let k = 1; k <= 5; k += 1) {
|
||||
const p = sk.atLeast(out.distribution, k);
|
||||
expect(p).toBeLessThanOrEqual(prev + 1e-9);
|
||||
prev = p;
|
||||
}
|
||||
});
|
||||
|
||||
it('a hitter exactly at league on every axis lands on league BABIP — no invented lean', () => {
|
||||
const leagueBat = {
|
||||
k_pct: sk.LEAGUE.k_pct, bb_pct: sk.LEAGUE.bb_pct,
|
||||
barrel_pct: sk.LEAGUE.barrel_pct, hard_hit_pct: sk.LEAGUE.hard_hit_pct,
|
||||
avg_exit_velo: sk.LEAGUE.avg_exit_velo, avg_launch_angle: 12,
|
||||
};
|
||||
const boc = sk.hitOnContact({ batter: leagueBat, pitcher: { hard_hit_pct: sk.LEAGUE.hard_hit_pct }, park: 1, archetype: 'DEFAULT' });
|
||||
expect(boc).toBeCloseTo(sk.LEAGUE.babip, 2);
|
||||
});
|
||||
|
||||
it('opportunity matters: more projected PA means a higher chance of one hit', () => {
|
||||
const few = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 2 });
|
||||
const many = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 5 });
|
||||
expect(many.p_over_line).toBeGreaterThan(few.p_over_line);
|
||||
});
|
||||
});
|
||||
@@ -212,10 +212,31 @@ describe('scheduler + route wiring', () => {
|
||||
it('is server-scheduled with a kill switch', () => {
|
||||
const s = read('src/snapshotScheduler.js');
|
||||
expect(s).toContain('STATCAST_HOUR_UTC');
|
||||
expect(s).toMatch(/process\.env\.STATCAST !== '0'/);
|
||||
expect(s).toMatch(/process\.env\.STATCAST === '0'/); // kill switch, early-return form
|
||||
expect(s).toContain('refreshSeason');
|
||||
});
|
||||
|
||||
it('RUNS ON ITS OWN TICK — not behind the snapshot-hours guard', () => {
|
||||
// THE REGRESSION THIS LOCKS (found 2026-08-03): the refresh used to live
|
||||
// inside `tick()`, BELOW `if (!HOURS_UTC.includes(h)) return`. HOURS_UTC is
|
||||
// 14,19,22,1,3 and the block tests h === STATCAST_HOUR_UTC (default 11), so
|
||||
// the guard could never admit the hour it waited for. It was unreachable
|
||||
// code that had never run once, and the aggregates sat 13 days stale while
|
||||
// every consumer served them as current. The previous assertion here passed
|
||||
// the entire time, because it only checked that the STRING existed.
|
||||
const s = read('src/snapshotScheduler.js');
|
||||
const statcastTick = s.slice(s.indexOf('const statcastTick'));
|
||||
expect(statcastTick.length).toBeGreaterThan(0);
|
||||
// Its own tick, registered on the interval alongside the others.
|
||||
expect(s).toMatch(/void statcastTick\(\)/);
|
||||
// And it must NOT be reachable only via the snapshot-hours guard.
|
||||
// Scope to the snapshot tick's OWN body (it ends where refreshTick begins),
|
||||
// so the statcastTick doc comment above it cannot satisfy this by accident.
|
||||
const tickBody = s.slice(s.indexOf('const tick = async'), s.indexOf('const refreshTick'));
|
||||
expect(tickBody).toContain('HOURS_UTC.includes(h)'); // the guard is still there
|
||||
expect(tickBody).not.toContain('STATCAST_HOUR_UTC'); // and statcast is NOT behind it
|
||||
});
|
||||
|
||||
it('pages on BOTH a failed run and silent staleness', () => {
|
||||
const s = read('src/snapshotScheduler.js');
|
||||
expect(s).toMatch(/Statcast refresh failed/);
|
||||
|
||||
Reference in New Issue
Block a user