Decompose the champion: its whole edge is a hit-rate counter
READ-ONLY. src/ and web/ untouched; 4,159 tests still green. WHAT THE CHAMPION IS. probabilityEstimator is five lines of arithmetic: the empirical frequency of (stat > THIS line) over the game log, blended 0.6/0.4 with the last-5 frequency, then +/-0.03 opponent, +/-0.015 home/away, a cv>0.40 pull toward 0.50, and a clamp to [0.10, 0.95]. It reads three features. featureCache retains a dozen more that p_win never touches. THE ABLATION IS EXACT, NOT A REFIT. Every adjustment is closed-form from stored features and the consistency step is linear, so each layer subtracts algebraically out of the stored p_win -- no re-estimation, no re-fetch, no lookahead possible. Per stat, paired bootstrap: removing ALL THREE adjustments changes resolution by NOTHING on every stat hits -0.0059 total_bases -0.0015 rbi +0.0106 runs +0.0130 walks +0.0008 and rbi's home/away is mildly HARMFUL (+0.0053, CI excludes zero). So ~100% of the champion's resolution is base+recency: how often this player has cleared this number lately. Everything else is decoration. A CORRECTION. Pooled, the champion resolves 0.46; per stat it is 0.196 (hits) to 0.499 (rbi). Pooling stats with different base rates inflates correlation, so 0.46 should not be quoted as the champion's resolution. Last session's paired differences remain valid; only the absolute level was inflated. THE BIGGEST LOSS IS NOT A MISSING FEATURE -- IT IS THE CLAMP. 358 of 1,741 settled rows (20.6%) sit on the boundary, so the model emits a constant there and cannot rank a fifth of the book at all. And that constant hides two opposite failures: 0.900 covers home_runs-under truly winning 99.5% (9.5pts under-confident) next to hits-under truly winning 51.9% (38.1pts over- confident). PROB_CEIL=0.95 makes the 99.5% case inexpressible. Global over-prediction is +3.5pts, +7.6 on total_bases. None of this needs new data. ONE REAL MISSING-WEIGHTING LEAD: opportunity_drift, residual corr +0.156 on hits and +0.145 on total_bases -- it REPEATS across independent stats, unlike the weather hits on TB which sit inside the expected false-positive count (70 tests at alpha .05 expects 3-4). And we already compute it: arch-v1's opportunity axis uses it and extracts nothing (delta +0.0001). Wrong implementation, not a missing feature -- opportunity must scale the rate, not nudge the probability. ARCHETYPE IS UNMEASURABLE, NOT REFUTED. Only 2 of 41 labels (BOMBER, GHOST) reach n>=40 settled rows and every mean residual straddles zero. That is "we have not measured it", and it does not license acting in either direction. Why every challenger has failed is now legible: the ladder and hits-v1 REPLACE the frequency question with a fitted distribution; the environment axis adds inputs the champion ignores. Asking the frequency question at the traded line is the thing that works. Flagged, not fixed: model_snapshots.outcome is NULL on all 22,032 rows -- the retention table built for exactly this replay was never settled, so labels had to be joined from ledger_entries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -3,6 +3,34 @@
|
||||
## Last Updated
|
||||
2026-08-03
|
||||
|
||||
## Session 78 (2026-08-03) — Champion decomposed: the edge is a hit-rate counter ✅
|
||||
Spec: `specs/champion-input-diagnosis.md`. READ-ONLY (src/web untouched);
|
||||
4,159 tests green.
|
||||
- **The champion is 5 lines.** base = empirical frequency of (stat > THIS line),
|
||||
0.6/0.4 blend with last-5, ±0.03 opponent, ±0.015 home/away, cv>0.40 pull,
|
||||
clamp [0.10,0.95]. It reads 3 features; featureCache retains a dozen more that
|
||||
p_win never touches.
|
||||
- **Exact analytic ablation, per stat, paired bootstrap.** Removing ALL THREE
|
||||
adjustments changes resolution by nothing everywhere (hits −0.0059, TB −0.0015,
|
||||
rbi +0.0106, runs +0.0130, walks +0.0008) — and rbi's home/away is mildly
|
||||
HARMFUL (+0.0053, CI excludes 0). ~100% of the edge is base+recency.
|
||||
- **Pooled 0.46 is an artifact** — per stat 0.196 (hits) … 0.499 (rbi). Corrected
|
||||
last session's reading; paired differences unaffected.
|
||||
- **BIGGEST LOSS = the clamp.** 20.6% of settled rows pinned to a constant (no
|
||||
ranking possible there), and `0.900` covers home_runs-under truly 99.5% AND
|
||||
hits-under truly 51.9%. Global over-prediction +3.5pt (TB +7.6). No new data
|
||||
needed to fix.
|
||||
- **One real lead: `opportunity_drift`** (residual +0.156 hits, +0.145 TB —
|
||||
repeats across stats, unlike the weather hits which sit inside the expected
|
||||
false-positive count). We ALREADY compute it; arch-v1's opportunity axis
|
||||
extracts nothing from it. Wrong implementation, not a missing feature.
|
||||
- **Archetype: UNMEASURABLE** — 2 of 41 labels have testable n. Not refuted.
|
||||
- **Next order priority:** (1) clamp + calibration, (2) opportunity as a rate
|
||||
scaler, (3) prune the diluting axes, (4) get archetype coverage. Explicitly NOT
|
||||
another projection variant.
|
||||
- Flagged: `model_snapshots.outcome` NULL on all 22,032 rows — retention is
|
||||
never settled, so replays must join the ledger for labels.
|
||||
|
||||
## Session 77 (2026-08-03) — Settlement was dead for two days; scoreboard now readable ✅
|
||||
Specs: `specs/challenger-scoreboard.md`, `specs/odds-429-diagnosis.md`.
|
||||
4,159 tests / 332 suites green, web build exit 0.
|
||||
|
||||
@@ -1183,6 +1183,47 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
|
||||
DFS pick'em is excluded from PRICING/consensus (`EXCLUDED_FROM_PRICING`) — a
|
||||
fixed-payout shaded number is not a market price. Never "clean up" breadth.
|
||||
|
||||
## Champion decomposition — the edge is a hit-rate counter (Session 78 — non-obvious)
|
||||
- **`specs/champion-input-diagnosis.md`.** `probabilityEstimator` IS the champion
|
||||
and it is five lines: `base` = empirical frequency of (stat > THIS line) over
|
||||
the game log, blended 0.6/0.4 with the last-5 frequency, then oppAdj(±0.03) +
|
||||
homeAdj(±0.015) + a cv>0.40 pull toward 0.50, then clamp [0.10, 0.95].
|
||||
- **EXACT ANALYTIC ABLATION (no refit):** every adjustment is closed-form from
|
||||
stored features and the consistency step is linear (`f(x)=0.9x+0.05` ⟹
|
||||
`f(a+b)=f(a)+0.9b`), so layers subtract algebraically out of the stored p_win.
|
||||
Result: **removing ALL THREE adjustments changes resolution by nothing on every
|
||||
stat**, and on rbi/runs it IMPROVES it (rbi home/away removal +0.0053, CI
|
||||
excludes zero = mildly HARMFUL). ~100% of the edge is base+recency.
|
||||
- **POOLED RESOLUTION IS INFLATED — do not quote 0.46.** Per stat the champion is
|
||||
0.196 (hits) to 0.499 (rbi); pooling stats with different base rates adds
|
||||
correlation because p_win tracks the base rate across stats. Paired DIFFERENCES
|
||||
(the scoreboard) stay valid; the absolute level does not. Always per-stat.
|
||||
- **THE CLAMP IS THE BIGGEST LOSS, not a missing feature.** 358/1,741 settled rows
|
||||
(20.6%) sit ON the boundary, so the model emits a CONSTANT there and cannot rank
|
||||
within a fifth of the book. And `0.900` hides home_runs-under truly 99.5%
|
||||
(−9.5pt under-confident) next to hits-under truly 51.9% (+38.1pt over-confident).
|
||||
`PROB_CEIL=0.95` makes the 99.5% case inexpressible. Global over-prediction
|
||||
+3.5pt (total_bases +7.6). Fixable with NO new data.
|
||||
- **`opportunity_drift` is the ONE real missing-weighting lead** — residual corr
|
||||
+0.156 (hits) and +0.145 (total_bases), i.e. it REPEATS across independent
|
||||
stats. Discipline: 14 features × 5 stats = 70 tests, so 3–4 CI-excludes-zero
|
||||
results are expected BY CHANCE; a single hit (weather on TB) is noise. And we
|
||||
already compute it — arch-v1's opportunity axis uses it and extracts NOTHING
|
||||
(delta +0.0001). Wrong implementation, not a missing feature: opportunity must
|
||||
scale the RATE, not nudge the probability.
|
||||
- **ARCHETYPE VERDICT: unmeasurable, not refuted.** Only 2 of 41 archetypes
|
||||
(BOMBER, GHOST) reach n≥40 settled rows; all mean residuals straddle zero. That
|
||||
is "we have not measured it", NOT "archetypes carry no signal". Don't act
|
||||
either way. (Their uniformly negative residuals are the global over-prediction,
|
||||
not an archetype effect.)
|
||||
- **Why every challenger has failed:** the ladder/hits-v1 REPLACE the frequency
|
||||
question with a fitted distribution; arch-v1's env axis adds park/weather the
|
||||
champion ignores. Asking "how often has he cleared THIS number" directly is the
|
||||
thing that works — improve its inputs, never substitute it.
|
||||
- **`model_snapshots.outcome` is NULL on all 22,032 rows.** The retention table
|
||||
built for exactly this kind of replay was never settled, so ablations must join
|
||||
outcomes from `ledger_entries` on (player_key, stat, line, side, game_date).
|
||||
|
||||
## Active Skills
|
||||
- vyndr-voice (all user-facing output)
|
||||
- prop-analysis (grading methodology)
|
||||
|
||||
@@ -0,0 +1,374 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* champion-ablation — WHERE DOES THE CHAMPION'S RESOLUTION ACTUALLY COME FROM?
|
||||
*
|
||||
* Convergent evidence says the problem is INPUTS, not shape: hits-v1 refuted,
|
||||
* the ladder reliably worse (−0.030), arch-v1 moving 76% of rows to exactly zero
|
||||
* effect, contact/environment/opportunity all CI-includes-zero. Every one of
|
||||
* those changed the DISTRIBUTION or added a NUDGE. None changed the information.
|
||||
* So before building a sixth thing, decompose the champion.
|
||||
*
|
||||
* THE CHAMPION IS FIVE LINES OF ARITHMETIC (probabilityEstimator):
|
||||
*
|
||||
* base = empirical frequency of stat > line over the game log
|
||||
* weighted = 0.6·base + 0.4·(same frequency over the last 5)
|
||||
* p = weighted + oppAdj(±0.03) + homeAdj(±0.015)
|
||||
* if cv>0.40: p = 0.9·p + 0.05 (volatile → pull toward 0.50)
|
||||
* p_over = clamp(p, 0.10, 0.95); p_win = side==='under' ? 1−p_over : p_over
|
||||
*
|
||||
* THE ABLATION IS EXACT, NOT A REFIT. Every adjustment is a closed-form function
|
||||
* of stored features, and the consistency step is linear, so each layer can be
|
||||
* removed analytically from the stored p_win:
|
||||
*
|
||||
* f(x) = 0.9x + 0.05 ⟹ f(a+b) = f(a) + 0.9b
|
||||
*
|
||||
* so subtracting an adjustment is subtracting k·adj with k = 0.9 when the
|
||||
* consistency pull fired and 1 when it did not. Nothing is re-estimated, no
|
||||
* model is refit, and no game log is re-fetched — which also means no lookahead
|
||||
* is even possible here.
|
||||
*
|
||||
* WHAT CANNOT BE ABLATED SEPARATELY, STATED PLAINLY: `base` and `recency` are
|
||||
* recoverable only as their blend (`weighted`), because the stored feature
|
||||
* vector holds AVERAGES (l5_avg/l20_avg), not frequencies-over-the-line. So the
|
||||
* base/recency split is reported as ONE block. That is a real limit of this
|
||||
* measurement, not an oversight.
|
||||
*
|
||||
* CLAMPED ROWS ARE EXCLUDED from the ablation: at p_over ∈ {0.10, 0.95} the
|
||||
* inversion is ambiguous, and guessing the pre-clamp value would be fabrication.
|
||||
* Their count is reported.
|
||||
*
|
||||
* SECOND MEASUREMENT — THE MISSING-FEATURE TEST. featureCache computes and
|
||||
* RETAINS far more than the champion reads (park_*, weather_*, rest_days,
|
||||
* opportunity_drift, ab_per_game, l5/l10/l20 avgs). If any of those correlates
|
||||
* with the champion's RESIDUAL (won − p_win), that is signal sitting unused on
|
||||
* disk — a MISSING FEATURE. If none do, that is evidence for AT CEILING with
|
||||
* respect to everything we currently compute.
|
||||
*
|
||||
* SUPABASE_URL=... node scripts/champion-ablation.js
|
||||
*/
|
||||
|
||||
require('dotenv').config();
|
||||
const { createClient } = require('@supabase/supabase-js');
|
||||
|
||||
const SB_URL = process.env.SUPABASE_URL;
|
||||
const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
|
||||
const PAGE = 1000;
|
||||
|
||||
const CV_VOLATILE_THRESHOLD = 0.40;
|
||||
const PROB_FLOOR = 0.10;
|
||||
const PROB_CEIL = 0.95;
|
||||
const clamp = (p) => Math.max(PROB_FLOOR, Math.min(PROB_CEIL, p));
|
||||
|
||||
function corr(xs, ys) {
|
||||
const n = xs.length;
|
||||
if (n < 3) return null;
|
||||
const mx = xs.reduce((a, b) => a + b, 0) / n;
|
||||
const my = ys.reduce((a, b) => a + b, 0) / n;
|
||||
let sxy = 0; let sxx = 0; let syy = 0;
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
const dx = xs[i] - mx; const dy = ys[i] - my;
|
||||
sxy += dx * dy; sxx += dx * dx; syy += dy * dy;
|
||||
}
|
||||
if (sxx <= 0 || syy <= 0) return null;
|
||||
return sxy / Math.sqrt(sxx * syy);
|
||||
}
|
||||
const r4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
const num = (v) => {
|
||||
if (v == null || v === '' || typeof v === 'boolean' || typeof v === 'object') return null;
|
||||
const n = Number(v);
|
||||
return Number.isFinite(n) ? n : null;
|
||||
};
|
||||
|
||||
/** Deterministic xorshift32 — a measurement that changes between runs is not one. */
|
||||
function makeRnd(seed) {
|
||||
let s = seed >>> 0;
|
||||
return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
|
||||
}
|
||||
|
||||
/** Paired bootstrap on a DIFFERENCE of resolutions (same rows → same resample). */
|
||||
function bootstrapDiff(rows, keyA, keyB, iters = 3000, seed = 20260803) {
|
||||
if (rows.length < 30) return null;
|
||||
const rnd = makeRnd(seed);
|
||||
const n = rows.length;
|
||||
const diffs = [];
|
||||
for (let it = 0; it < iters; it += 1) {
|
||||
const ys = []; const a = []; const b = [];
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
const r = rows[Math.floor(rnd() * n)];
|
||||
ys.push(r.won); a.push(r[keyA]); b.push(r[keyB]);
|
||||
}
|
||||
const ca = corr(a, ys); const cb = corr(b, ys);
|
||||
if (ca == null || cb == null) continue;
|
||||
diffs.push(ca - cb);
|
||||
}
|
||||
if (diffs.length < 100) return null;
|
||||
diffs.sort((x, y) => x - y);
|
||||
const q = (p) => r4(diffs[Math.floor(p * (diffs.length - 1))]);
|
||||
const ci = [q(0.025), q(0.975)];
|
||||
return {
|
||||
point: r4(corr(rows.map((r) => r[keyA]), rows.map((r) => r.won))
|
||||
- corr(rows.map((r) => r[keyB]), rows.map((r) => r.won))),
|
||||
ci95: ci,
|
||||
ci_excludes_zero: ci[0] > 0 || ci[1] < 0,
|
||||
};
|
||||
}
|
||||
|
||||
/** Bootstrap CI on a single correlation (for the residual-signal test). */
|
||||
function bootstrapCorr(rows, key, seed = 20260804, iters = 3000) {
|
||||
const usable = rows.filter((r) => r[key] != null);
|
||||
if (usable.length < 40) return { n: usable.length, corr: null, ci95: null, ci_excludes_zero: false };
|
||||
const rnd = makeRnd(seed);
|
||||
const n = usable.length;
|
||||
const vals = [];
|
||||
for (let it = 0; it < iters; it += 1) {
|
||||
const xs = []; const ys = [];
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
const r = usable[Math.floor(rnd() * n)];
|
||||
xs.push(r[key]); ys.push(r.residual);
|
||||
}
|
||||
const c = corr(xs, ys);
|
||||
if (c != null) vals.push(c);
|
||||
}
|
||||
if (vals.length < 100) return { n, corr: null, ci95: null, ci_excludes_zero: false };
|
||||
vals.sort((a, b) => a - b);
|
||||
const q = (p) => r4(vals[Math.floor(p * (vals.length - 1))]);
|
||||
const ci = [q(0.025), q(0.975)];
|
||||
return {
|
||||
n,
|
||||
corr: r4(corr(usable.map((r) => r[key]), usable.map((r) => r.residual))),
|
||||
ci95: ci,
|
||||
ci_excludes_zero: ci[0] > 0 || ci[1] < 0,
|
||||
};
|
||||
}
|
||||
|
||||
async function page(sb, table, select, apply) {
|
||||
const out = [];
|
||||
for (let from = 0; ; from += PAGE) {
|
||||
let q = sb.from(table).select(select);
|
||||
q = apply(q).range(from, from + PAGE - 1);
|
||||
const { data, error } = await q;
|
||||
if (error) throw error;
|
||||
if (!data || data.length === 0) break;
|
||||
out.push(...data);
|
||||
if (data.length < PAGE) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const propKey = (r) => `${r.player_key}|${r.stat}|${Number(r.line)}|${String(r.side).toLowerCase()}|${r.game_date}`;
|
||||
|
||||
/**
|
||||
* OUTCOMES COME FROM THE LEDGER, NOT FROM RETENTION.
|
||||
*
|
||||
* `model_snapshots.outcome` is NULL on all 22,032 rows — the retention table
|
||||
* that exists so a different model can be replayed against the same conditions
|
||||
* stores the features but was never settled. So the labels are joined from
|
||||
* `ledger_entries` on (player_key, stat, line, side, game_date), which is the
|
||||
* same identity the ledger's own dedupe constraint uses. Flagged, not fixed —
|
||||
* this run is read-only.
|
||||
*/
|
||||
async function fetchAll(sb) {
|
||||
const snaps = await page(sb, 'model_snapshots',
|
||||
'player_key, stat, line, side, game_date, p_win, features, quarantine_reason, captured_at, archetype',
|
||||
(q) => q.eq('sport', 'mlb').not('p_win', 'is', null).not('features', 'is', null));
|
||||
const led = await page(sb, 'ledger_entries', 'player_key, stat, line, side, game_date, outcome, quarantine_reason',
|
||||
(q) => q.eq('sport', 'mlb').is('user_id', null).in('outcome', ['hit', 'miss']));
|
||||
|
||||
const outcomeBy = new Map();
|
||||
for (const r of led) {
|
||||
if ((r.quarantine_reason || '').startsWith('nontakeable_book')) continue;
|
||||
outcomeBy.set(propKey(r), r.outcome);
|
||||
}
|
||||
return snaps
|
||||
.map((r) => ({ ...r, outcome: outcomeBy.get(propKey(r)) || null }))
|
||||
.filter((r) => r.outcome === 'hit' || r.outcome === 'miss');
|
||||
}
|
||||
|
||||
function build(rows) {
|
||||
// One row per prop — the EARLIEST capture is the lock. Multiple snapshot
|
||||
// cycles per day would otherwise weight a prop by how often it was re-graded.
|
||||
const byProp = new Map();
|
||||
for (const r of rows) {
|
||||
if ((r.quarantine_reason || '').startsWith('nontakeable_book')) continue;
|
||||
const k = `${r.player_key}|${r.stat}|${r.line}|${r.side}|${r.game_date}`;
|
||||
const prev = byProp.get(k);
|
||||
if (!prev || String(r.captured_at) < String(prev.captured_at)) byProp.set(k, r);
|
||||
}
|
||||
|
||||
const out = [];
|
||||
let clamped = 0;
|
||||
for (const r of byProp.values()) {
|
||||
const f = r.features || {};
|
||||
const pWin = num(r.p_win);
|
||||
if (pWin == null) continue;
|
||||
const under = String(r.side || '').toLowerCase() === 'under';
|
||||
const pOver = under ? 1 - pWin : pWin;
|
||||
|
||||
// Clamped → the pre-clamp value is unrecoverable. Excluded, counted.
|
||||
if (pOver <= PROB_FLOOR + 1e-9 || pOver >= PROB_CEIL - 1e-9) { clamped += 1; continue; }
|
||||
|
||||
const rank = num(f.opp_rank_stat);
|
||||
const oppAdj = rank == null ? 0 : (rank >= 0.70 ? 0.03 : rank <= 0.30 ? -0.03 : 0);
|
||||
const ha = num(f.home_away);
|
||||
const homeAdj = ha === 1 ? 0.015 : ha === 0 ? -0.015 : 0;
|
||||
const sd = num(f.l10_stddev); const l20 = num(f.l20_avg);
|
||||
const cv = (sd != null && sd > 0 && l20 != null && l20 > 0) ? sd / l20 : null;
|
||||
const consistencyFired = cv != null && cv > CV_VOLATILE_THRESHOLD;
|
||||
const k = consistencyFired ? 0.9 : 1;
|
||||
|
||||
// p_over (unclamped) = f(weighted + oppAdj + homeAdj); f linear ⟹ exact removal.
|
||||
const noOpp = pOver - k * oppAdj;
|
||||
const noHome = pOver - k * homeAdj;
|
||||
const noAdj = pOver - k * oppAdj - k * homeAdj; // = f(weighted)
|
||||
// Removing the consistency pull: invert f on the whole thing.
|
||||
const noCons = consistencyFired ? (pOver - 0.05) / 0.9 : pOver;
|
||||
// base+recency block alone, with every adjustment off.
|
||||
const weighted = consistencyFired ? (noAdj - 0.05) / 0.9 : noAdj;
|
||||
|
||||
const flip = (p) => (under ? 1 - clamp(p) : clamp(p));
|
||||
const won = r.outcome === 'hit' ? 1 : 0;
|
||||
out.push({
|
||||
stat: r.stat,
|
||||
won,
|
||||
full: pWin,
|
||||
no_opp: flip(noOpp),
|
||||
no_home: flip(noHome),
|
||||
no_consistency: flip(noCons),
|
||||
no_adjustments: flip(noAdj),
|
||||
weighted_only: flip(weighted),
|
||||
residual: won - pWin,
|
||||
archetype: r.archetype || null,
|
||||
// Features the champion NEVER reads — the missing-feature candidates.
|
||||
l5_avg: num(f.l5_avg), l10_avg: num(f.l10_avg), l20_avg: num(f.l20_avg),
|
||||
ab_per_game: num(f.ab_per_game), recent_ab_per_game: num(f.recent_ab_per_game),
|
||||
opportunity_drift: num(f.opportunity_drift), rest_days: num(f.rest_days),
|
||||
game_count_in_7d: num(f.game_count_in_7d),
|
||||
park_h: num(f.park_h), park_hr: num(f.park_hr), park_r: num(f.park_r),
|
||||
weather_temp_f: num(f.weather_temp_f), weather_wind_mph: num(f.weather_wind_mph),
|
||||
weather_precip: num(f.weather_precip),
|
||||
// Features it DOES read — controls for the same test.
|
||||
opp_rank_stat: num(f.opp_rank_stat), home_away: num(f.home_away),
|
||||
l10_stddev: num(f.l10_stddev),
|
||||
});
|
||||
}
|
||||
return { rows: out, clamped };
|
||||
}
|
||||
|
||||
const ABLATIONS = [
|
||||
['no_opp', 'opponent (opp_rank_stat, ±0.03)'],
|
||||
['no_home', 'home/away (±0.015)'],
|
||||
['no_consistency', 'consistency pull (cv>0.40 → toward 0.50)'],
|
||||
['no_adjustments', 'ALL THREE adjustments (leaves base+recency)'],
|
||||
];
|
||||
|
||||
const UNUSED = ['l5_avg', 'l10_avg', 'l20_avg', 'ab_per_game', 'recent_ab_per_game',
|
||||
'opportunity_drift', 'rest_days', 'game_count_in_7d', 'park_h', 'park_hr', 'park_r',
|
||||
'weather_temp_f', 'weather_wind_mph', 'weather_precip'];
|
||||
const USED = ['opp_rank_stat', 'home_away', 'l10_stddev'];
|
||||
|
||||
function ablateStat(rows, label) {
|
||||
const ys = rows.map((r) => r.won);
|
||||
const full = r4(corr(rows.map((r) => r.full), ys));
|
||||
const abl = {};
|
||||
for (const [key, name] of ABLATIONS) {
|
||||
const bs = bootstrapDiff(rows, key, 'full');
|
||||
abl[name] = {
|
||||
resolution_without: r4(corr(rows.map((r) => r[key]), ys)),
|
||||
// NEGATIVE delta = removing it HURT = the feature carries signal.
|
||||
delta_from_removal: bs ? bs.point : null,
|
||||
ci95: bs ? bs.ci95 : null,
|
||||
carries_signal: bs ? (bs.ci_excludes_zero && bs.point < 0) : null,
|
||||
};
|
||||
}
|
||||
return { stat: label, n: rows.length, base_rate: r4(ys.reduce((a, b) => a + b, 0) / ys.length), resolution_full: full, ablations: abl };
|
||||
}
|
||||
|
||||
function residualStat(rows, label) {
|
||||
const scan = (keys, seedBase) => {
|
||||
const out = {};
|
||||
keys.forEach((k, i) => {
|
||||
const res = bootstrapCorr(rows, k, 20260804 + i + seedBase);
|
||||
out[k] = res;
|
||||
});
|
||||
return out;
|
||||
};
|
||||
return {
|
||||
stat: label,
|
||||
n: rows.length,
|
||||
unused_features: scan(UNUSED, 0),
|
||||
used_features_control: scan(USED, 500),
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required');
|
||||
const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
|
||||
const { rows, clamped } = build(await fetchAll(sb));
|
||||
|
||||
const counts = {};
|
||||
for (const r of rows) counts[r.stat] = (counts[r.stat] || 0) + 1;
|
||||
const stats = Object.entries(counts).filter(([, n]) => n >= 60).map(([s]) => s)
|
||||
.sort((a, b) => counts[b] - counts[a]);
|
||||
|
||||
const perStat = stats.map((s) => ablateStat(rows.filter((r) => r.stat === s), s));
|
||||
const residual = stats.map((s) => residualStat(rows.filter((r) => r.stat === s), s));
|
||||
|
||||
// ── ARCHETYPE ON TRIAL ────────────────────────────────────────────────
|
||||
// The champion reads NO archetype feature at all, so it cannot be ablated out
|
||||
// of it. The fair test is whether archetype explains what the champion GETS
|
||||
// WRONG: if a given archetype's rows are systematically mispriced, archetype
|
||||
// carries prop signal the model is missing (wrong IMPLEMENTATION). If every
|
||||
// archetype's mean residual straddles zero, archetype carries no prop signal.
|
||||
const archetypeTest = stats.map((st) => {
|
||||
const rs = rows.filter((r) => r.stat === st && r.archetype);
|
||||
const groups = {};
|
||||
for (const r of rs) (groups[r.archetype] = groups[r.archetype] || []).push(r.residual);
|
||||
const out = {};
|
||||
for (const [name, vals] of Object.entries(groups)) {
|
||||
if (vals.length < 40) continue;
|
||||
const rnd = makeRnd(20260805);
|
||||
const means = [];
|
||||
for (let it = 0; it < 3000; it += 1) {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < vals.length; i += 1) sum += vals[Math.floor(rnd() * vals.length)];
|
||||
means.push(sum / vals.length);
|
||||
}
|
||||
means.sort((a, b) => a - b);
|
||||
const ci = [r4(means[Math.floor(0.025 * (means.length - 1))]), r4(means[Math.floor(0.975 * (means.length - 1))])];
|
||||
out[name] = {
|
||||
n: vals.length,
|
||||
mean_residual: r4(vals.reduce((a, b) => a + b, 0) / vals.length),
|
||||
ci95: ci,
|
||||
systematically_mispriced: ci[0] > 0 || ci[1] < 0,
|
||||
};
|
||||
}
|
||||
return { stat: st, archetypes: out };
|
||||
});
|
||||
|
||||
// Any unused feature with a CI excluding zero, anywhere → a missing-feature lead.
|
||||
const leads = [];
|
||||
for (const rs of residual) {
|
||||
for (const [k, v] of Object.entries(rs.unused_features)) {
|
||||
if (v.ci_excludes_zero) leads.push({ stat: rs.stat, feature: k, corr: v.corr, ci95: v.ci95, n: v.n });
|
||||
}
|
||||
}
|
||||
|
||||
console.log(JSON.stringify({
|
||||
measurement: 'EXACT ANALYTIC ABLATION of the champion, on the REPAIRED settled set. No refit, no re-fetch, no lookahead.',
|
||||
limits: {
|
||||
base_recency_not_separable: 'stored features hold AVERAGES, not frequencies-over-line; reported as one block',
|
||||
clamped_rows_excluded: clamped,
|
||||
},
|
||||
total_rows: rows.length,
|
||||
per_stat_ablation: perStat,
|
||||
residual_signal_test: residual,
|
||||
archetype_test: archetypeTest,
|
||||
multiple_comparisons_note: 'The residual scan runs 14 unused features x 5 stats = 70 tests at alpha .05, so ~3-4 CI-excludes-zero results are EXPECTED BY CHANCE. Treat a single hit as noise; only a feature repeating across independent stats is evidence.',
|
||||
missing_feature_leads: leads,
|
||||
}, null, 2));
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error(e); process.exit(1); });
|
||||
@@ -0,0 +1,220 @@
|
||||
# DECOMPOSING THE CHAMPION — where its edge actually comes from
|
||||
|
||||
**Read-only diagnosis, 2026-08-03.** Nothing built, nothing touched. Run on the
|
||||
**repaired** settled set (n=1,741 after the settlement outage fix), not the frozen
|
||||
pre-fix set.
|
||||
|
||||
> **VERDICT, one line:** the champion's entire edge is a **hit-rate counter**, its
|
||||
> three adjustment layers contribute **nothing** (two are mildly harmful), and the
|
||||
> largest recoverable loss is **not a missing feature — it is the `[0.10, 0.95]`
|
||||
> clamp, which pins 20.6% of settled props to a constant** and hides outcomes
|
||||
> ranging from 52% to 99.5% behind the same number `0.900`.
|
||||
|
||||
---
|
||||
|
||||
## 1. What the champion actually uses (STEP 1)
|
||||
|
||||
`src/services/intelligence/probabilityEstimator.js` is **five lines of arithmetic**:
|
||||
|
||||
```
|
||||
base = empirical frequency of (stat > THIS line) over the game log
|
||||
weighted = 0.6·base + 0.4·(same frequency over the last 5 games)
|
||||
p = weighted + oppAdj(±0.03) + homeAdj(±0.015)
|
||||
if cv > 0.40: p = 0.9·p + 0.05 (volatile → pull toward 0.50)
|
||||
p_over = clamp(p, 0.10, 0.95)
|
||||
p_win = side === 'under' ? 1 − p_over : p_over
|
||||
```
|
||||
|
||||
It reads exactly **three** features: `opp_rank_stat`, `home_away`, and
|
||||
`l10_stddev`/`l20_avg` (for cv). `featureCache` computes and retains a dozen more
|
||||
— `park_h/hr/r`, `weather_temp_f/wind_mph/precip`, `rest_days`,
|
||||
`opportunity_drift`, `ab_per_game`, `recent_ab_per_game`, `l5/l10/l20_avg`,
|
||||
`game_count_in_7d` — and **p_win reads none of them.**
|
||||
|
||||
## 2. Per-stat ablation (STEP 2)
|
||||
|
||||
**Exact and analytic, not a refit.** Each adjustment is a closed-form function of
|
||||
stored features and the consistency step is linear (`f(x)=0.9x+0.05` ⟹
|
||||
`f(a+b)=f(a)+0.9b`), so every layer is removed algebraically from the stored
|
||||
`p_win`. Nothing re-estimated, nothing re-fetched, no lookahead possible.
|
||||
Paired bootstrap, 3,000 resamples, deterministic seed.
|
||||
|
||||
**A negative delta means removing the layer HURT — i.e. it carried signal.**
|
||||
|
||||
| stat | n | resolution (full) | −opponent | −home/away | −consistency | **−ALL THREE** |
|
||||
|---|---|---|---|---|---|---|
|
||||
| hits | 578 | 0.1964 | −0.0049 | −0.0006 | −0.0004 | **−0.0059** [−0.0168,+0.0056] |
|
||||
| total_bases | 284 | 0.2370 | −0.0056 | +0.0042 | +0.0002 | **−0.0015** [−0.0139,+0.0115] |
|
||||
| rbi | 273 | 0.4986 | +0.0059 | **+0.0053** [+0.0002,+0.0103] | +0.0005 | **+0.0106** [−0.0003,+0.0220] |
|
||||
| runs | 115 | 0.4062 | +0.0060 | +0.0087 | −0.0006 | **+0.0130** [−0.0106,+0.0366] |
|
||||
| walks | 66 | 0.4776 | −0.0035 | +0.0055 | +0.0018 | **+0.0008** [−0.0268,+0.0281] |
|
||||
|
||||
**Removing all three adjustments changes resolution by nothing on every stat, and
|
||||
on rbi/runs it IMPROVES it.** Exactly one ablation anywhere has a CI excluding
|
||||
zero — rbi home/away, and its sign says removing it makes the model **better**.
|
||||
|
||||
**So ~100% of the champion's resolution is `base + recency`: how often this
|
||||
player has cleared THIS number lately.** That is the whole model. Everything else
|
||||
is decoration.
|
||||
|
||||
### A correction to how we read last session's scoreboard
|
||||
|
||||
Pooled across stats the champion resolves **0.46**; per stat it is **0.196
|
||||
(hits)** to **0.499 (rbi)**. Pooling stats with different base rates *inflates*
|
||||
correlation, because p_win varies across stats in the same direction as the true
|
||||
base rate. **0.46 is a pooling artifact and should not be quoted as the
|
||||
champion's resolution.** The paired *differences* in the scoreboard remain valid
|
||||
(champion and challenger were pooled identically); only the absolute level was
|
||||
inflated.
|
||||
|
||||
## 3. Do the challengers have it, or dilute it? (STEP 3)
|
||||
|
||||
| challenger | uses the base-frequency signal? | verdict |
|
||||
|---|---|---|
|
||||
| **proj-v1.1 ladder** | **No — it replaces it.** Fits a rate + NB distribution instead of counting frequency at THIS line | **DILUTING.** Measured reliably worse (−0.0301, CI excludes 0). It discards the one thing that works in favour of a lossier route to the same question |
|
||||
| **hits-v1** | No — same substitution, binomial instead of NB | **DILUTING.** Refuted (−0.022, CI excludes 0) |
|
||||
| **arch-v1 · environment** | Adds park/weather, which the champion ignores | **DILUTING.** n=871, −0.0028, CI includes 0 — movement without information |
|
||||
| **arch-v1 · opportunity** | Uses `opportunity_drift` — **the one feature with repeated residual signal** | **HAS THE FEATURE, WRONG IMPLEMENTATION** (see §4) |
|
||||
| **arch-v1 · matchup** | — | STILL PENDING (rows settle after ET midnight) |
|
||||
| **contact-v1** | Statcast contact quality; not in the retained vector | No evidence either way (n=1,055, CI includes 0) |
|
||||
|
||||
## 4. The missing-feature test — one real lead, already in our hands
|
||||
|
||||
Correlation of each **unused** feature with the champion's residual (`won −
|
||||
p_win`), per stat, bootstrap CI.
|
||||
|
||||
**Multiple-comparisons discipline first:** 14 features × 5 stats = 70 tests at
|
||||
α=.05, so **3–4 CI-excludes-zero results are expected by chance.** Six appeared.
|
||||
A single hit is noise. **Only a feature that repeats across independent stats is
|
||||
evidence** — and exactly one does:
|
||||
|
||||
| feature | hits | total_bases | walks |
|
||||
|---|---|---|---|
|
||||
| **`opportunity_drift`** | **+0.156** [+0.007,+0.292] | **+0.145** [+0.005,+0.278] | −0.261 [−0.463,−0.017] |
|
||||
| `recent_ab_per_game` (same quantity) | +0.052 | **+0.136** [+0.002,+0.272] | −0.157 |
|
||||
|
||||
`opportunity_drift` = recent at-bats ÷ season at-bats-per-game. It is the one
|
||||
axis a frequency counter is **structurally blind to**: `base` knows how often he
|
||||
cleared the number, not that he has moved from 8th in the order to leadoff, or
|
||||
back from injury on a bench role. Sign flips on walks (n=47, and walks scale with
|
||||
plate appearances differently) — so this is stat-specific, which is doctrine-
|
||||
consistent, not a contradiction.
|
||||
|
||||
**But we already compute it, retain it, and built an axis on it — and that axis
|
||||
extracts nothing** (opportunity axis: n=539, delta +0.0001, CI [−0.0091,+0.0090]).
|
||||
So this is **not** "go get a new feature." It is **"the feature has signal and our
|
||||
implementation of it is wrong"** — arch-v1 applies it as a small multiplicative
|
||||
nudge to `p_win`, which is not how you use an opportunity term. Opportunity should
|
||||
scale the *rate*, before the frequency question is asked.
|
||||
|
||||
Weather on total_bases (`wind_mph` −0.164, `precip` −0.154) appears on **one stat
|
||||
only** and sits inside the expected false-positive count. Recorded as a
|
||||
non-lead unless it repeats.
|
||||
|
||||
## 5. Archetype on trial (STEP 4 item) — NO EVIDENCE, and the test is underpowered
|
||||
|
||||
The champion reads **no archetype feature at all**, so archetype cannot be ablated
|
||||
out of it. The fair test is whether archetype explains what the champion gets
|
||||
*wrong*: if an archetype's rows are systematically mispriced, archetype carries
|
||||
prop signal we're missing.
|
||||
|
||||
| stat | archetype | n | mean residual | CI95 | mispriced? |
|
||||
|---|---|---|---|---|---|
|
||||
| hits | BOMBER | 195 | −0.038 | [−0.105, +0.030] | no |
|
||||
| hits | GHOST | 85 | −0.076 | [−0.184, +0.033] | no |
|
||||
| total_bases | BOMBER | 95 | −0.046 | [−0.141, +0.058] | no |
|
||||
| total_bases | GHOST | 46 | −0.041 | [−0.187, +0.109] | no |
|
||||
| rbi | BOMBER | 88 | −0.068 | [−0.155, +0.019] | no |
|
||||
| runs | BOMBER | 41 | −0.012 | [−0.150, +0.121] | no |
|
||||
|
||||
**Verdict: (a) no-signal is UNPROVEN and (b) wrong-implementation is UNPROVEN —
|
||||
the test cannot separate them yet.** Only **2 of 41 archetypes** (BOMBER, GHOST)
|
||||
reach n≥40 settled rows. That is not "archetypes don't work"; it is "we have not
|
||||
measured them." Distinguishing (a) from (b) needs archetype coverage across more
|
||||
than two labels. **Do not act on archetype in either direction on this evidence.**
|
||||
|
||||
Worth noting: every archetype's mean residual is **negative**, which is not an
|
||||
archetype effect — it is the global over-prediction in §6.
|
||||
|
||||
## 6. THE BIGGEST FINDING — the clamp, not a feature
|
||||
|
||||
**358 of 1,741 settled props (20.6%) sit ON the clamp boundary** (353 at the
|
||||
floor). Within that fifth of the book the model emits a **constant**, so it cannot
|
||||
rank those props at all — resolution there is zero by construction.
|
||||
|
||||
And the constant is hiding two *opposite* failures at once:
|
||||
|
||||
| stat · side | n | model says | actually wins | miscalibration |
|
||||
|---|---|---|---|---|
|
||||
| home_runs · under | 222 | 0.900 | **0.995** | **−9.5 pts** (badly UNDER-confident) |
|
||||
| hits · under | 27 | 0.900 | **0.519** | **+38.1 pts** (a coin flip sold as 90%) |
|
||||
| total_bases · under | 12 | 0.900 | 0.583 | +31.7 pts |
|
||||
| rbi · over | 35 | 0.100 | 0.229 | −12.9 pts |
|
||||
|
||||
**The same output `0.900` covers true probabilities from 52% to 99.5%.** A
|
||||
near-lock and a coin flip are indistinguishable in the product. `PROB_CEIL = 0.95`
|
||||
also makes it *impossible* to express the 99.5% case honestly.
|
||||
|
||||
Overall calibration, all settled MLB rows:
|
||||
|
||||
| stat | n | mean p_win | actual | over-prediction |
|
||||
|---|---|---|---|---|
|
||||
| **ALL POOLED** | 1,741 | 0.585 | 0.550 | **+3.5 pts** |
|
||||
| total_bases | 297 | 0.534 | 0.458 | **+7.6** |
|
||||
| hits | 589 | 0.606 | 0.562 | +4.4 |
|
||||
| rbi | 320 | 0.399 | 0.356 | +4.3 |
|
||||
| walks | 83 | 0.546 | 0.506 | +4.0 |
|
||||
| runs | 124 | 0.596 | 0.605 | −0.9 (well calibrated) |
|
||||
| home_runs | 228 | 0.899 | 0.996 | **−9.6** |
|
||||
|
||||
Per the product doctrine, calibration is **half** the success criterion — "does
|
||||
60% mean 60%?" Right now 58.5% means 55.0%, and on total_bases 53.4% means 45.8%.
|
||||
**This is fixable with no new data at all.**
|
||||
|
||||
## 7. VERDICT PER STAT (STEP 4)
|
||||
|
||||
| stat | n | resolution | verdict |
|
||||
|---|---|---|---|
|
||||
| **hits** | 578 | 0.196 | **DILUTION + CALIBRATION.** Adjustments contribute nothing; one real lead (`opportunity_drift`) that we already compute and implement wrongly; +4.4pt over-prediction |
|
||||
| **total_bases** | 284 | 0.237 | **DILUTION + CALIBRATION (worst).** Same lead; +7.6pt over-prediction |
|
||||
| **rbi** | 273 | 0.499 | **DILUTION.** Removing home/away *improves* it (CI excludes zero). Prune |
|
||||
| **runs** | 115 | 0.406 | **AT CEILING.** Adjustments neutral-to-harmful, no residual leads. The model is good here |
|
||||
| **walks** | 66 | 0.478 | **AT CEILING** (underpowered, n=66) |
|
||||
| **home_runs** | 228 | n/a — entirely clamped | **CALIBRATION.** −9.6pts, structurally uncorrectable while `PROB_CEIL=0.95` |
|
||||
|
||||
**AT CEILING is a real result here, not a shrug**: on runs and walks the champion
|
||||
already resolves ~0.41–0.48 and nothing we compute explains its residual.
|
||||
|
||||
## 8. What this says about the next order
|
||||
|
||||
The convergent evidence was read correctly — the problem *is* inputs, not shape.
|
||||
But the decomposition sharpens it, and the ranking is not what we assumed:
|
||||
|
||||
1. **The clamp + calibration (biggest, cheapest, no new data).** 20.6% of the
|
||||
book pinned to a constant, a documented +3.5pt global over-prediction, and one
|
||||
number covering 52%–99.5%. This costs both halves of the success criterion.
|
||||
2. **Opportunity, implemented properly** — the one feature with repeated
|
||||
cross-stat residual signal. Not a new feature: a correct use of one we have.
|
||||
Scale the rate, don't nudge the probability.
|
||||
3. **Stop diluting** — the ladder and the environment axis add movement with no
|
||||
information, and are measurably worse or flat.
|
||||
4. **Archetype: measure before judging.** 2 of 41 labels have testable n.
|
||||
|
||||
**Explicitly NOT recommended:** another projection variant. That is the sixth
|
||||
thing, and this diagnosis is why it would fail — the champion's edge is asking
|
||||
the frequency question at the traded line, and every challenger so far has
|
||||
replaced that question rather than improved its inputs.
|
||||
|
||||
## 9. Incidental finding, flagged not fixed
|
||||
|
||||
**`model_snapshots.outcome` is NULL on all 22,032 rows.** The retention table
|
||||
built expressly so "a different model can be replayed against the same
|
||||
conditions" stores features but was **never settled**, so it cannot answer the
|
||||
question it exists for. This diagnosis worked around it by joining outcomes from
|
||||
`ledger_entries` on (player_key, stat, line, side, game_date). Settling retention
|
||||
would make every future ablation a single-table query — and would make refusals
|
||||
(which the ledger drops) measurable for the first time.
|
||||
|
||||
## 10. Reproduce
|
||||
|
||||
`SUPABASE_URL=... node scripts/champion-ablation.js`
|
||||
Reference in New Issue
Block a user