Prove both on total bases -- and find that my own fix destroyed the backtest
Nothing passed. Nothing promoted. Counter byte-identical. THE BLOCKER, which is the real finding. statcast_aggregates is upserted in place and holds exactly one as-of date. Yesterday's skill backtest was honest only by accident: the nightly refresh was unreachable code, so the profiles sat frozen at 2026-07-21 -- before the settled window. Repairing that cron was right for production and it refreshed them to today, destroying every prior version. Scoring a 2026-07-25 game now uses a season aggregate that contains that game. Point-in-time validation is structurally impossible from that table, so every number in this run is contaminated and directional, and none of it is a gate verdict. Fixed forward: statcast_history retains a dated snapshot on every refresh, so point-in-time becomes "as_of_date < game_date, most recent". Retention is best-effort and cannot fail the refresh; both properties are unit-tested. It has one day of data, which is not yet a window. SOLO BASELINE, n=383, Bonferroni across 12 tests (alpha 0.00417): nothing passes. hard_hit_pct is closest at marginal r 0.135 with p 0.0080, failing both the 0.15 effect bar and the corrected alpha. And it drifted DOWN from 0.153 at n=295 -- an estimate regressing as noise averages out, not an effect firming up. I called that number encouraging yesterday; on 88 more rows it is fading, and it should not keep being quoted at its best value. INTERACTIONS, each scored by partial correlation against the counter residual controlling for both of its own components: none pass. Only barrel x power archetype has an incremental exceeding its parts (-0.101 against 0.019) at n=260 -- the shape Discipline 2 predicts, but a lead, not a finding. A methodological catch worth keeping. The archetype conditioner was first built as barrel_pct over league barrel -- a monotone transform of one of its own components -- so the "interaction" was barrel squared, measuring nonlinearity in barrel rate rather than any archetype effect, and it produced this run's only positive result. A Gauss-Jordan pivot test does not catch that, because the two columns differ by a scale factor. Fixed with a scale-free collinearity check plus real archetype labels joined from model_snapshots. Without it this document would have reported a fabricated interaction as the session's finding. COMBINED vs COUNTER on total bases: 0.2718 against 0.2647, delta +0.0071, CI [-0.065, +0.079] -- inconclusive, and the first time a challenger has not lost. The same engine on hits was -0.116 with a CI excluding zero. That contrast is the whole argument for total bases, and it is what the physics said: contact quality governs extra bases, not whether a grounder finds a hole. Also built: the compound TB projection. skillProjection no longer refuses total bases -- a deterministic bases-per-hit multiplier had made P(TB>=2) exactly P(hits>=1), a relabelled hits curve. It is now a convolution over per-PA base outcomes with hit-type shares shifted by skill. Non-degeneracy is locked by test. 4,204 tests green (334 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -186,11 +186,34 @@ describe('honesty — unknown is not zero, and absent beats invented', () => {
|
||||
expect(Math.abs(noBarrel - full)).toBeLessThan(Math.abs(zeroBarrel - full));
|
||||
});
|
||||
|
||||
it('total_bases is REFUSED rather than shipped as a relabelled hits curve', () => {
|
||||
// A deterministic bases-per-hit multiplier made P(TB>=2) identical to
|
||||
// P(hits>=1). Refusing is correct until a real per-hit bases distribution
|
||||
// exists (tb-v1's compound shape).
|
||||
expect(sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'total_bases', line: 1.5 })).toBeNull();
|
||||
it('total_bases is COMPOUND, not a relabelled hits curve', () => {
|
||||
// The first cut multiplied hits by a constant bases-per-hit, which made
|
||||
// P(TB>=2) EXACTLY equal to P(hits>=1) — a relabel carrying no information a
|
||||
// hits model did not already have. It was refused rather than shipped. It is
|
||||
// now a real convolution over per-PA base outcomes, and this test locks the
|
||||
// property that distinguishes the two.
|
||||
const tb = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'total_bases', line: 1.5, expectedPa: 4.2 });
|
||||
const hits = sk.projectSkill({ batter: BOMBER, pitcher: ACE, archetype: 'BOMBER', statType: 'hits', line: 0.5, expectedPa: 4.2 });
|
||||
expect(tb).not.toBeNull();
|
||||
expect(tb.family).toBe('pa_compound_bases_convolution');
|
||||
expect(tb.p_over_line).not.toBeCloseTo(hits.p_over_line, 3); // NOT degenerate
|
||||
expect(tb.distribution.reduce((a, b) => a + b, 0)).toBeCloseTo(1, 2);
|
||||
});
|
||||
|
||||
it('hit-type shares respond to power skill — a slugger homers more per hit', () => {
|
||||
const slugger = sk.hitTypeShares({ batter: BOMBER, archetype: 'BOMBER' });
|
||||
const slap = sk.hitTypeShares({ batter: GHOST, archetype: 'GHOST' });
|
||||
expect(slugger.homer).toBeGreaterThan(slap.homer * 3);
|
||||
expect(slap.single).toBeGreaterThan(slugger.single);
|
||||
// Shares are a distribution over hit types.
|
||||
for (const s of [slugger, slap]) {
|
||||
expect(s.single + s.double + s.triple + s.homer).toBeCloseTo(1, 6);
|
||||
}
|
||||
});
|
||||
|
||||
it('an absent power read leaves the shares at league — never a guessed lean', () => {
|
||||
const noSkill = sk.hitTypeShares({ batter: { k_pct: 0.2 }, archetype: 'DEFAULT' });
|
||||
expect(noSkill).toEqual(sk.LEAGUE_HIT_SHARES);
|
||||
});
|
||||
|
||||
it('the distribution is a real distribution and P(>=k) is monotone', () => {
|
||||
|
||||
@@ -144,10 +144,40 @@ describe('refreshSeason — the job', () => {
|
||||
});
|
||||
|
||||
it('upserts on the natural key so a re-run never duplicates', async () => {
|
||||
let opts = null;
|
||||
const sb = { from: () => ({ upsert: async (_b, o) => { opts = o; return { error: null }; } }) };
|
||||
// Track PER TABLE — the refresh now writes the aggregate AND its dated
|
||||
// history snapshot, and the two have deliberately different natural keys.
|
||||
const seen = {};
|
||||
const sb = { from: (t) => ({ upsert: async (_b, o) => { seen[t] = o; return { error: null }; } }) };
|
||||
await svc.refreshSeason({ supabase: sb, fetchSeason: async () => feeds({ batter: [BELL] }) });
|
||||
expect(opts.onConflict).toBe('sport,season,source_id,role');
|
||||
expect(seen.statcast_aggregates.onConflict).toBe('sport,season,source_id,role');
|
||||
});
|
||||
|
||||
it('RETAINS a dated point-in-time snapshot alongside the live aggregate', async () => {
|
||||
// statcast_aggregates is upserted IN PLACE, so it holds one as-of date and
|
||||
// destroys every earlier version — which silently makes any backtest score a
|
||||
// game with a profile that already contains it. The history table is the
|
||||
// only thing that makes point-in-time validation possible at all.
|
||||
const byTable = {};
|
||||
const sb = { from: (t) => ({ upsert: async (b, o) => { (byTable[t] = byTable[t] || []).push({ b, o }); return { error: null }; } }) };
|
||||
const out = await svc.refreshSeason({
|
||||
supabase: sb, fetchSeason: async () => feeds({ batter: [BELL] }), now: '2026-08-03T11:00:00.000Z',
|
||||
});
|
||||
expect(out.history_retained).toBe(1);
|
||||
expect(out.history_as_of).toBe('2026-08-03');
|
||||
expect(byTable.statcast_history[0].o.onConflict).toBe('as_of_date,sport,season,source_id,role');
|
||||
expect(byTable.statcast_history[0].b[0].as_of_date).toBe('2026-08-03');
|
||||
});
|
||||
|
||||
it('a retention failure NEVER fails the refresh — stale-but-current beats nothing', async () => {
|
||||
const sb = { from: (t) => ({
|
||||
upsert: async () => (t === 'statcast_history'
|
||||
? { error: { message: 'history table missing' } }
|
||||
: { error: null }),
|
||||
}) };
|
||||
const out = await svc.refreshSeason({ supabase: sb, fetchSeason: async () => feeds({ batter: [BELL] }) });
|
||||
expect(out.ok).toBe(true); // the refresh still succeeded
|
||||
expect(out.written).toBe(1);
|
||||
expect(out.history_error).toMatch(/history table missing/);
|
||||
});
|
||||
|
||||
it('REFUSES to write when every feed is empty — a bad night cannot blank a good table', async () => {
|
||||
|
||||
Reference in New Issue
Block a user