diff --git a/BUILD-STATE.md b/BUILD-STATE.md index 11b7f52..96b4cb1 100755 --- a/BUILD-STATE.md +++ b/BUILD-STATE.md @@ -3,6 +3,28 @@ ## Last Updated 2026-08-03 +## Session 81 (2026-08-03) — The gate, built and run: hits is dead, total bases is the stat ✅ +Spec: `specs/stagea-gate-result.md`. 4,200 tests / 334 suites green, build exit 0. +Counter byte-identical (zero diff on probabilityEstimator/analyzeViaEngine1). +- **PREMISE CORRECTED:** statModel.js and correlateValidator.js do NOT exist in + this repo. The spec lived only in an offline Python blueprint, and + supplementSystems.test.js inlines its own validateFactor (requires just + fs/path). Nothing to connect — so the gate was BUILT to spec. +- **`correlateValidator.js`** — n>=500, |r|>=0.15, p<0.05, Bonferroni. Exact + p-value (incomplete beta), unit-verified against known values. +- **GATE RUN, hits (n=570, Bonferroni-8): EVERYTHING FAILS.** Max marginal |r| + 0.062 vs the 0.15 bar — an effect-size failure at a well-powered n. Head-to-head + also loses: 0.0499 vs counter 0.166, delta −0.116 CI [−0.189,−0.043]. +- **GATE RUN, total_bases (n=295): CANNOT TEST — and that is the finding.** + hard_hit_pct marginal r=0.153 (above threshold), exit_velo 0.124; refused only + on n. ~205 more settled rows needed. Matches the physics: contact quality + drives extra bases, not singles. +- **Architecture change the run forced:** per-STAT feature verdicts, so a feature + dead for hits stays alive for TB. Gate now reports r/p when underpowered. +- **Next:** build the compound TB value projection (per-hit bases distribution + from launch/barrel — skillProjection still refuses TB by design), accrue to + n>=500, re-run the gate. Leave hits alone. Do not lower the bar. + ## Session 80 (2026-08-03) — The skill engine: built, gated, and Stage A honestly lost ✅ Spec: `specs/skill-engine-architecture.md`. 4,182 tests / 333 suites green, build exit 0. - **BUILT `src/services/model/`:** `featureRegistry` (CANDIDATE/PROVEN/DEAD per diff --git a/CLAUDE.md b/CLAUDE.md index ece6be1..176c941 100755 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1301,6 +1301,42 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section). branch. Now its own `statcastTick`. The old test only checked the string existed — the new one asserts it is not behind the snapshot-hours guard. +## The validation gate + the stat that was wrong (Session 81 — non-obvious) +- **`statModel.js` and `correlateValidator.js` NEVER EXISTED** in this repo. The + spec's only prior form was `src/services/python/blueprints/unconventional.py` + (Flask, in the OFFLINE python service, scoring NBA factors against an empty + warehouse), and `tests/unit/supplementSystems.test.js` INLINES its own + `validateFactor` (line 368; only `fs`/`path` are required). So those tests + passed for months with no implementation to connect — that is the real reason + every challenger was measured ungated. +- **`src/services/model/correlateValidator.js` is the gate now** — n>=500, + |r|>=0.15, p<0.05, Bonferroni. The p-value is EXACT (t-transform via a + regularized incomplete beta, Lentz CF) and unit-verified against known values; + scipy isn't available in Node so don't reach for an approximation. Pairs with + an unknown side are DROPPED — zero-filling a correlation invents a point at + the origin. +- **HITS IS A CLEAN NEGATIVE — stop modelling it.** Gate run at n=570, + Bonferroni-8: EVERY skill feature fails, max marginal |r| = 0.062 vs a 0.15 + bar. Not a power problem — an effect-size problem. And the value engine loses + head-to-head (0.0499 vs 0.166, CI [−0.189,−0.043]). At the 0.5 hits line there + is very little for skill inputs to know. +- **TOTAL BASES IS WHERE THE SIGNAL IS, and it is n-blocked.** Same features: + `hard_hit_pct` marginal r = **0.153** (above threshold), `exit_velo` 0.124, + raw r 0.196/0.167 — refused ONLY because n=295 < 500. Needs ~205 more settled + rows. This is what the physics predicts: contact quality drives EXTRA BASES, + not whether a grounder finds a hole. +- **The gate reports r and p even when underpowered** (`underpowered: true`, + `rows_needed`). "Not enough data yet" and "nothing here" need OPPOSITE + decisions — collapsing them into a bare refusal hid the best signal on the board. +- **Feature verdicts are PER STAT** (`recordStatVerdict` / `statusForStat` / + `candidateFeaturesForStat`). Marking these DEAD sport-wide on hits evidence + would have killed the features most alive on TB. Per-sport doctrine one level + deeper: physics differ per stat. +- **Next is the compound TB projection** — `skillProjection` still REFUSES + total_bases (a deterministic bases-per-hit made P(TB>=2) == P(hits>=1)). Build + the per-hit extra-base distribution off launch/barrel (tb-v1's shape, fed by + skill inputs), accrue to n>=500, re-run this gate. Do NOT lower the bar. + ## Active Skills - vyndr-voice (all user-facing output) - prop-analysis (grading methodology) diff --git a/scripts/stagea-gate-run.js b/scripts/stagea-gate-run.js new file mode 100644 index 0000000..6cf5d0f --- /dev/null +++ b/scripts/stagea-gate-run.js @@ -0,0 +1,247 @@ +#!/usr/bin/env node +'use strict'; + +/** + * stagea-gate-run — RUN THE SKILL FEATURES THROUGH THE GATE, THEN THE COUNTER. + * + * The original sin was never that the challengers were badly built. It was that + * every one of them was measured WITHOUT a validation gate, so "it didn't work" + * and "it was never allowed to prove it works" were indistinguishable. This runs + * the gate that spec'd for exactly this (n>=500, |r|>=0.15, p<0.05, Bonferroni) + * over the real skill features, and only then does the head-to-head. + * + * THREE MEASUREMENTS, in the order that makes each one meaningful: + * + * 1. RAW SIGNAL — corr(feature, outcome). Does this skill input relate to + * whether the prop hit at all? + * 2. MARGINAL CONTRIBUTION — corr(feature, counter residual). This is the one + * that matters: a feature can correlate with the outcome purely because the + * counter already knows it. Only the part the counter MISSES is new + * information, and that is what earns a place. Both go through the gate. + * 3. HEAD-TO-HEAD — the value projection vs the live counter on listed-line + * accuracy, paired bootstrap, out-of-sample. + * + * OUT-OF-SAMPLE: skill profiles are the frozen 2026-07-21 aggregate; only games + * AFTER that date are scored, so no profile contains the game it predicts. + * + * BONFERRONI DENOMINATOR is the number of features tested in this sweep — not 1. + * Testing many and reporting the best without correction is how the S78 residual + * scan produced six "findings" when chance alone predicts three or four. + * + * SUPABASE_URL=... node scripts/stagea-gate-run.js + */ + +require('dotenv').config(); +const { createClient } = require('@supabase/supabase-js'); +const cv = require('../src/services/model/correlateValidator'); +const sk = require('../src/services/model/skillProjection'); +const reg = require('../src/services/model/featureRegistry'); +const mlb = require('../src/services/adapters/mlbStatsAdapter'); +const { knownRate, knownNumber } = require('../src/utils/known'); + +const SB_URL = process.env.SUPABASE_URL; +const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY; +const PAGE = 1000; +const GAMES_SO_FAR = Number(process.env.STAGEA_GAMES_SO_FAR || 103); + +const r4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000); +const mean = (a) => (a.length ? a.reduce((x, y) => x + y, 0) / a.length : null); +const brier = (ps, ys) => (ps.length ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null); + +function makeRnd(seed) { + let s = seed >>> 0; + return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; }; +} +function corrOf(xs, ys) { return cv.pearson(xs, ys).r; } + +function bootstrapDiff(rows, keyA, keyB, iters = 4000, seed = 20260803) { + if (rows.length < 30) return null; + const rnd = makeRnd(seed); + const n = rows.length; + const diffs = []; + for (let it = 0; it < iters; it += 1) { + const ys = []; const a = []; const b = []; + for (let i = 0; i < n; i += 1) { + const r = rows[Math.floor(rnd() * n)]; + ys.push(r.won); a.push(r[keyA]); b.push(r[keyB]); + } + const ca = corrOf(a, ys); const cb = corrOf(b, ys); + if (ca == null || cb == null) continue; + diffs.push(ca - cb); + } + if (diffs.length < 100) return null; + diffs.sort((x, y) => x - y); + const q = (p) => r4(diffs[Math.floor(p * (diffs.length - 1))]); + const ci = [q(0.025), q(0.975)]; + return { + point: r4(corrOf(rows.map((r) => r[keyA]), rows.map((r) => r.won)) + - corrOf(rows.map((r) => r[keyB]), rows.map((r) => r.won))), + ci95: ci, ci_excludes_zero: ci[0] > 0 || ci[1] < 0, + }; +} + +async function page(sb, table, select, apply) { + const out = []; + for (let from = 0; ; from += PAGE) { + const { data, error } = await apply(sb.from(table).select(select)).range(from, from + PAGE - 1); + if (error) throw error; + if (!data || data.length === 0) break; + out.push(...data); + if (data.length < PAGE) break; + } + return out; +} + +async function opposingStarters(dates) { + const m = new Map(); + for (const d of dates) { + let games = []; + try { games = await mlb.getScheduleWithPitchers(d); } catch { games = []; } + for (const g of games) { + if (!g.home || !g.away) continue; + if (g.home.probablePitcher) m.set(`${d}|OPP:${g.home.team}`, g.home.probablePitcher.id); + if (g.away.probablePitcher) m.set(`${d}|OPP:${g.away.team}`, g.away.probablePitcher.id); + } + } + return m; +} + +/** `playerKey|date` → opponent faced. The ledger's team/opponent are NULL. */ +async function opponentByPlayerDate(players) { + const map = new Map(); + for (const [key, name] of players) { + try { + const found = await mlb.searchPlayer(name); + if (!found || !found.id) continue; + const log = await mlb.getPlayerGameLog(found.id); + for (const g of log || []) { + if (g && g.date && g.opponent) map.set(`${key}|${String(g.date).slice(0, 10)}`, g.opponent); + } + } catch { /* no log → no pitcher for those rows */ } + } + return map; +} + +async function main() { + if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required'); + const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } }); + + const statcast = await page(sb, 'statcast_aggregates', '*', (q) => q.eq('sport', 'mlb')); + const freezeDate = statcast.reduce((mx, r) => (String(r.updated_at) > mx ? String(r.updated_at) : mx), '').slice(0, 10); + const batters = new Map(); const pitchersById = new Map(); + for (const r of statcast) { + if (r.role === 'pitcher' && r.source_id != null) pitchersById.set(Number(r.source_id), sk.fromStatcastRow(r)); + if (r.player_key && r.role === 'batter') { + const prev = batters.get(r.player_key); + if (!prev || Number(r.sample_pa || 0) > Number(prev.rawPa || 0)) { + batters.set(r.player_key, Object.assign(sk.fromStatcastRow(r), { rawPa: Number(r.sample_pa || 0) })); + } + } + } + + const led = await page(sb, 'ledger_entries', + 'player_key, player_name, stat, line, side, outcome, game_date, p_win, quarantine_reason', + (q) => q.eq('sport', 'mlb').is('user_id', null) + .in('stat', ['hits', 'total_bases']) + .in('outcome', ['hit', 'miss']).not('p_win', 'is', null)); + + const clean = led.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book') + && String(r.game_date) > freezeDate); + + const dates = [...new Set(clean.map((r) => r.game_date))].sort(); + const starters = await opposingStarters(dates); + const players = new Map(); + for (const r of clean) if (!players.has(r.player_key)) players.set(r.player_key, r.player_name); + const oppByPlayerDate = await opponentByPlayerDate(players); + + const allowed = reg.candidateFeatures('mlb'); + const rows = []; + for (const r of clean) { + const bat = batters.get(r.player_key); + if (!bat) continue; + const faced = oppByPlayerDate.get(`${r.player_key}|${r.game_date}`) || null; + const pit = faced ? pitchersById.get(Number(starters.get(`${r.game_date}|OPP:${faced}`))) || null : null; + const paRate = bat.rawPa > 0 ? Math.min(5.2, Math.max(2.0, bat.rawPa / GAMES_SO_FAR)) : null; + const under = String(r.side).toLowerCase() === 'under'; + const won = r.outcome === 'hit' ? 1 : 0; + const champ = Number(r.p_win); + + // The value projection (hits only — TB is refused by design, see skillProjection). + const proj = r.stat === 'hits' + ? sk.projectSkill({ batter: bat, pitcher: pit, park: 1, archetype: null, + statType: 'hits', line: Number(r.line), expectedPa: paRate, allowed }) + : null; + + rows.push({ + stat: r.stat, won, champ, + skill: proj ? (under ? 1 - proj.p_over_line : proj.p_over_line) : null, + residual: won - champ, + had_pitcher: !!pit, + // Candidate skill features, archetype-relevant, in probability space. + batter_barrel_pct: knownRate(bat.barrel_pct), + batter_hard_hit_pct: knownRate(bat.hard_hit_pct), + batter_exit_velo: knownRate(bat.avg_exit_velo), + batter_launch_angle: knownRate(bat.avg_launch_angle), + batter_k_pct: knownRate(bat.k_pct), + batter_bb_pct: knownRate(bat.bb_pct), + pitcher_k_pct: pit ? knownRate(pit.k_pct) : null, + pitcher_hard_hit_allowed: pit ? knownRate(pit.hard_hit_pct) : null, + }); + } + + const FEATURES = ['batter_barrel_pct', 'batter_hard_hit_pct', 'batter_exit_velo', + 'batter_launch_angle', 'batter_k_pct', 'batter_bb_pct', + 'pitcher_k_pct', 'pitcher_hard_hit_allowed']; + + const perStat = {}; + for (const stat of ['hits', 'total_bases']) { + const rs = rows.filter((r) => r.stat === stat); + if (rs.length === 0) continue; + const tests = FEATURES.length; // the Bonferroni denominator for THIS sweep + const gate = {}; + for (const f of FEATURES) { + const xs = rs.map((r) => r[f]); + gate[f] = { + // 1. does it relate to the outcome at all? + raw_vs_outcome: cv.validateFactor(xs, rs.map((r) => r.won), tests), + // 2. THE ONE THAT COUNTS — is any of it NEW, i.e. missed by the counter? + marginal_vs_counter_residual: cv.validateFactor(xs, rs.map((r) => r.residual), tests), + }; + } + const passed = FEATURES.filter((f) => gate[f].marginal_vs_counter_residual.validated); + perStat[stat] = { + n: rs.length, + base_rate: r4(mean(rs.map((r) => r.won))), + bonferroni_tests: tests, + features_passing_gate_on_marginal: passed, + gate, + }; + } + + // HEAD-TO-HEAD — hits only (the value engine covers hits). + const h2h = rows.filter((r) => r.stat === 'hits' && r.skill != null); + const ys = h2h.map((r) => r.won); + const bs = bootstrapDiff(h2h, 'skill', 'champ'); + + console.log(JSON.stringify({ + premise_correction: 'statModel.js and correlateValidator.js do not exist in this repo. The gate was implemented to the spec in src/services/python/blueprints/unconventional.py (VALIDATION_REQUIREMENTS); supplementSystems.test.js inlines its own validateFactor and imports no implementation.', + out_of_sample: `skill profiles frozen ${freezeDate}; only game_date > ${freezeDate} scored`, + gate_spec: cv.VALIDATION_REQUIREMENTS, + per_stat_gate: perStat, + head_to_head_hits: { + n: h2h.length, + pitcher_coverage: r4(mean(h2h.map((r) => (r.had_pitcher ? 1 : 0)))), + base_rate: r4(mean(ys)), + resolution: { value_engine: r4(corrOf(h2h.map((r) => r.skill), ys)), counter: r4(corrOf(h2h.map((r) => r.champ), ys)) }, + brier: { value_engine: r4(brier(h2h.map((r) => r.skill), ys)), counter: r4(brier(h2h.map((r) => r.champ), ys)) }, + delta: bs, + verdict: !bs ? 'N-BLOCKED' + : (bs.ci_excludes_zero && bs.point > 0) ? 'VALUE ENGINE BEATS THE COUNTER' + : (bs.ci_excludes_zero && bs.point < 0) ? 'LOSES to the counter — iterate, do not promote' + : 'INCONCLUSIVE — do not promote', + }, + }, null, 2)); + process.exit(0); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/specs/stagea-gate-result.md b/specs/stagea-gate-result.md new file mode 100644 index 0000000..c38f2f5 --- /dev/null +++ b/specs/stagea-gate-result.md @@ -0,0 +1,158 @@ +# STAGE A THROUGH THE GATE — and the stat we should have picked + +**2026-08-03.** Challenger-only. The counter is byte-identical (zero diff on +`probabilityEstimator.js` / `analyzeViaEngine1.js`). + +> **RESULT: no skill feature passes the gate for HITS, and the value engine +> loses to the counter on hits. But the same features show the strongest +> correlations yet measured for TOTAL BASES — and total bases is 205 settled rows +> short of being testable at all.** We have been proving things on the wrong stat. + +--- + +## 0. Premise correction, with evidence + +The order describes `statModel.js` and `correlateValidator.js` as *built, tested, +disconnected*. **Neither file exists in this repository.** + +``` +find . -name "statModel*" → nothing (outside .claude/worktrees) +find . -name "correlateValidator*" → nothing +``` + +The validation spec's only prior existence is +`src/services/python/blueprints/unconventional.py` — a Flask blueprint in the +**Python service that is offline in production**, scoring NBA "unconventional +factors" (altitude, contract year, referee crew) against a data warehouse that +was never populated. And `tests/unit/supplementSystems.test.js` has exactly +**two `require()` calls — `fs` and `path`** — and defines its own `validateFactor` +inline at line 368. **Those tests assert a re-implementation of the thresholds, +not an implementation of them**, which is precisely why they passed for months +while nothing existed to connect. + +So the diagnosis behind the order is right — **every challenger was measured +without a gate** — but the cause is simpler than "someone forgot to import it": +**there was no gate on the Node side to import.** It is built now, to the exact +spec, and this document is its first real run. + +## 1. The gate (`src/services/model/correlateValidator.js`) + +`n ≥ 500 · |Pearson r| ≥ 0.15 · p < 0.05 · Bonferroni across the sweep`, +ported from `VALIDATION_REQUIREMENTS` in the Python blueprint. + +The p-value is exact, not approximated — scipy is unavailable in Node, so it is +computed from the t transform via a regularized incomplete beta (Lentz continued +fraction) and **verified against known values** in the unit suite (r=0.5,n=10 → +p≈0.1411; r=0.8,n=10 → p≈0.00546; r=0.15,n=500 → p≈0.00076). Pairs with an +unknown side are **dropped, never zero-filled** — a zero-fill inside a +correlation doesn't add noise, it invents a point at the origin. + +**Bonferroni is not decoration here.** The S78 residual scan ran 70 tests at +α=.05 and produced six "significant" hits — almost exactly the 3–4 chance +predicts. The correction is what makes PROVEN mean anything. + +## 2. THE GATE RUN — hits (n=570, clears the bar) + +Bonferroni denominator = 8 (the features in this sweep), corrected α = 0.00625. +Two questions per feature: does it relate to the outcome at all, and — the one +that matters — is any of it **new**, i.e. correlated with what the counter +*misses* (`won − p_win`)? + +| feature | r vs outcome | **r vs counter residual** | p | verdict | +|---|---|---|---|---| +| batter_bb_pct | −0.047 | **−0.062** | 0.138 | weak + not significant | +| batter_exit_velo | −0.031 | −0.056 | 0.180 | weak + not significant | +| batter_hard_hit_pct | −0.007 | −0.039 | 0.355 | weak + not significant | +| batter_barrel_pct | −0.020 | −0.037 | 0.378 | weak + not significant | +| batter_k_pct | −0.014 | −0.015 | 0.722 | weak + not significant | +| batter_launch_angle | −0.010 | −0.005 | 0.910 | weak + not significant | +| pitcher_k_pct | +0.018 | +0.012 | 0.777 | weak + not significant | +| pitcher_hard_hit_allowed | +0.016 | +0.011 | 0.806 | weak + not significant | + +**Nothing passes. Nothing is close.** The strongest marginal |r| is 0.062 against +a 0.15 bar — not "failed on significance", failed on effect size, at a sample +that would have detected a real effect easily. + +**This is a clean, well-powered negative result for hits**, and it explains every +earlier failure at a stroke: the skill features carry essentially no information +about whether a hitter gets one hit. The counter isn't beating them because it's +clever; it's beating them because at the 0.5 hits line there is very little for +them to know. + +## 3. THE GATE RUN — total bases (n=295, CANNOT be tested) + +| feature | r vs outcome | **r vs counter residual** | status | +|---|---|---|---| +| **batter_hard_hit_pct** | **+0.196** | **+0.153** | insufficient data (needs 205 more rows) | +| **batter_exit_velo** | **+0.167** | +0.124 | insufficient data | +| batter_barrel_pct | +0.121 | +0.088 | insufficient data | +| pitcher_k_pct | −0.049 | −0.077 | insufficient data | + +**`hard_hit_pct` marginal r = 0.153 — above the 0.15 threshold — and it is +refused solely because n = 295 < 500.** That is the gate working correctly, and +it is also the most encouraging number this line of work has produced. + +The gate now reports `r` and `p` even when underpowered, with +`underpowered: true` and `rows_needed`. **"Not enough data yet" and "nothing +here" require opposite decisions** — one waits, the other stops — and collapsing +them into a bare refusal was hiding the single most useful signal on the board. + +### And it is exactly what the physics predicts + +Exit velocity and hard-hit rate govern whether contact becomes **extra bases**. +They barely govern whether a ground ball finds a hole for a single. The archetype +doctrine says the same thing: contact quality is the BOMBER lane, and total bases +is the power stat. **We tested skill inputs on the one stat where skill inputs +shouldn't matter much.** + +## 4. Head-to-head — hits (the stat we could test) + +Out-of-sample: profiles frozen 2026-07-21, only later games scored. 570 rows, +91.9% opposing-pitcher coverage. + +``` +resolution value engine 0.0499 counter 0.1660 +brier value engine 0.2839 counter 0.2506 +delta −0.1161 CI95 [−0.1892, −0.0426] ← excludes zero +``` + +**LOSES. Not promoted.** Consistent with §2: a model built from features that +carry no hits signal cannot beat a counter on hits. + +## 5. What changed in the architecture because of this run + +**Verdicts are now PER STAT** (`recordStatVerdict` / `statusForStat` / +`candidateFeaturesForStat`). Marking these features DEAD sport-wide on hits +evidence would have killed, for total bases, the exact features that look most +alive there. That is the per-sport doctrine one level deeper: **a feature earns +or loses its place per stat, because the physics differ per stat.** Locked by +test. + +## 6. Verdict and the next order + +**Stage A on hits: the windshield loses, and the gate says why — there is no +skill signal there to find.** Do not iterate the hits model; that is polishing a +read of a stat whose outcome our best inputs are blind to. + +**Stage A should move to TOTAL BASES**, where: +- the leading marginal correlation (0.153) is already at the threshold, +- the gate needs **205 more settled TB rows** — days, now that settlement works, +- and `skillProjection` currently **refuses** total bases by design, because a + deterministic bases-per-hit multiplier made P(TB≥2) identical to P(hits≥1). + **That refusal is now the thing to build:** a real per-hit extra-base + distribution off the launch/barrel profile — which is tb-v1's compound shape, + fed by skill inputs instead of raw counts. + +Ranked next steps: + +1. **Build the compound TB value projection** (per-hit bases distribution from + launch/barrel), so TB can be projected at all. +2. **Accrue to n ≥ 500 on TB** and re-run this exact gate. +3. **Then** head-to-head on TB. If `hard_hit_pct` holds its 0.153 through the + Bonferroni-corrected gate at full n, that is the first feature in this + codebase to genuinely earn its place. +4. Leave hits alone. + +**Not recommended:** promoting anything now, lowering the n≥500 bar to make TB +testable today, or re-running the hits model with more features. The bar being +inconvenient is what makes it worth having. diff --git a/src/services/model/correlateValidator.js b/src/services/model/correlateValidator.js new file mode 100644 index 0000000..a198587 --- /dev/null +++ b/src/services/model/correlateValidator.js @@ -0,0 +1,208 @@ +'use strict'; + +/** + * correlateValidator — THE GATE A FEATURE MUST PASS TO EARN ITS PLACE. + * + * Kev's spec, exactly: n ≥ 500 · |Pearson r| ≥ 0.15 · p < 0.05 · Bonferroni- + * corrected across the number of factors being tested at once. + * + * PROVENANCE, because it matters. This is a JS implementation of the thresholds + * declared in `src/services/python/blueprints/unconventional.py` + * (`VALIDATION_REQUIREMENTS`), which is the spec's only prior existence — a + * Flask blueprint in the Python service that is offline in production, scoring + * NBA "unconventional factors" against a data warehouse that was never + * populated. `tests/unit/supplementSystems.test.js` asserts the same thresholds + * but INLINES its own `validateFactor` (line 368) rather than importing one, so + * those tests passed for months without any implementation existing in the Node + * path at all. That is the whole reason every challenger to date was measured + * without a gate: there was no gate on this side of the codebase to call. + * + * WHY BONFERRONI IS NOT OPTIONAL HERE. The S78 residual scan ran 14 features × + * 5 stats = 70 tests at α=.05 and produced six "significant" results — almost + * exactly the 3–4 the null hypothesis predicts by chance. Without a correction, + * a wide feature sweep manufactures discoveries. The correction is what makes + * "PROVEN" mean something. + * + * THE P-VALUE IS REAL, NOT APPROXIMATED. scipy's `pearsonr` is not available in + * Node, so the two-sided p is computed from the exact t transform + * + * t = r · sqrt((n−2) / (1−r²)), p = I_{v/(v+t²)}(v/2, 1/2), v = n−2 + * + * using a regularized incomplete beta (Lentz continued fraction). Verified in + * the unit suite against known values rather than trusted. + * + * HONESTY: pairs where either side is unknown are DROPPED, never zero-filled — + * `Number(null) === 0` inside a correlation does not merely add noise, it + * invents a data point at the origin and drags r toward whatever the means are. + */ + +const { knownNumber } = require('../../utils/known'); + +const VALIDATION_REQUIREMENTS = Object.freeze({ + min_historical_instances: 500, + min_pearson_r: 0.15, + max_p_value: 0.05, // BEFORE Bonferroni + bonferroni_correction: true, +}); + +/** log-gamma (Lanczos) — for the incomplete beta. */ +function logGamma(x) { + const c = [76.18009172947146, -86.50532032941677, 24.01409824083091, + -1.231739572450155, 0.1208650973866179e-2, -0.5395239384953e-5]; + let y = x; + let tmp = x + 5.5; + tmp -= (x + 0.5) * Math.log(tmp); + let ser = 1.000000000190015; + for (let j = 0; j < 6; j += 1) { y += 1; ser += c[j] / y; } + return -tmp + Math.log((2.5066282746310005 * ser) / x); +} + +/** Continued fraction for the incomplete beta (modified Lentz). */ +function betaCf(a, b, x) { + const MAXIT = 200; + const EPS = 3e-14; + const FPMIN = 1e-300; + const qab = a + b; + const qap = a + 1; + const qam = a - 1; + let c = 1; + let d = 1 - (qab * x) / qap; + if (Math.abs(d) < FPMIN) d = FPMIN; + d = 1 / d; + let h = d; + for (let m = 1; m <= MAXIT; m += 1) { + const m2 = 2 * m; + let aa = (m * (b - m) * x) / ((qam + m2) * (a + m2)); + d = 1 + aa * d; if (Math.abs(d) < FPMIN) d = FPMIN; + c = 1 + aa / c; if (Math.abs(c) < FPMIN) c = FPMIN; + d = 1 / d; + h *= d * c; + aa = (-(a + m) * (qab + m) * x) / ((a + m2) * (qap + m2)); + d = 1 + aa * d; if (Math.abs(d) < FPMIN) d = FPMIN; + c = 1 + aa / c; if (Math.abs(c) < FPMIN) c = FPMIN; + d = 1 / d; + const del = d * c; + h *= del; + if (Math.abs(del - 1) < EPS) break; + } + return h; +} + +/** Regularized incomplete beta I_x(a,b). */ +function incompleteBeta(a, b, x) { + if (x <= 0) return 0; + if (x >= 1) return 1; + const bt = Math.exp(logGamma(a + b) - logGamma(a) - logGamma(b) + + a * Math.log(x) + b * Math.log(1 - x)); + if (x < (a + 1) / (a + b + 2)) return (bt * betaCf(a, b, x)) / a; + return 1 - (bt * betaCf(b, a, 1 - x)) / b; +} + +/** + * Two-sided p-value for a Pearson r on n paired observations. + * Returns null when it is not defined (n < 3), never a comforting 1.0. + */ +function pearsonPValue(r, n) { + const rr = knownNumber(r); + const nn = knownNumber(n); + if (rr === null || nn === null || nn < 3) return null; + const v = nn - 2; + const r2 = Math.min(1, rr * rr); + if (r2 >= 1) return 0; // perfect correlation + const t2 = (r2 * v) / (1 - r2); + return incompleteBeta(v / 2, 0.5, v / (v + t2)); +} + +/** + * Pearson r over PAIRED observations, dropping any pair with an unknown side. + * Returns { r, n } so the caller always knows how many pairs actually counted — + * a correlation quoted without its surviving n hides exactly this kind of loss. + */ +function pearson(xs, ys) { + const X = []; const Y = []; + const len = Math.min((xs || []).length, (ys || []).length); + for (let i = 0; i < len; i += 1) { + const a = knownNumber(xs[i]); + const b = knownNumber(ys[i]); + if (a === null || b === null) continue; // DROP the pair, never zero-fill + X.push(a); Y.push(b); + } + const n = X.length; + if (n < 3) return { r: null, n }; + const mx = X.reduce((s, v) => s + v, 0) / n; + const my = Y.reduce((s, v) => s + v, 0) / n; + let sxy = 0; let sxx = 0; let syy = 0; + for (let i = 0; i < n; i += 1) { + const dx = X[i] - mx; const dy = Y[i] - my; + sxy += dx * dy; sxx += dx * dx; syy += dy * dy; + } + if (sxx <= 0 || syy <= 0) return { r: null, n }; // a constant has no correlation + return { r: sxy / Math.sqrt(sxx * syy), n }; +} + +/** + * THE GATE. + * + * @param {number[]} factorValues the feature, per observation + * @param {number[]} outcomeValues what actually happened, per observation + * @param {number} numActiveTests how many factors are being tested together + * (the Bonferroni denominator). 1 means no correction. + * @returns {object} a verdict that always states WHY, so a failure is readable + * without re-running anything. + */ +function validateFactor(factorValues, outcomeValues, numActiveTests = 1, opts = {}) { + const req = { ...VALIDATION_REQUIREMENTS, ...(opts.requirements || {}) }; + const { r, n } = pearson(factorValues, outcomeValues); + + if (n < req.min_historical_instances) { + // REPORT r AND p ANYWAY. `validated` stays false — the bar is the bar — but + // hiding the numbers turns "not enough data yet" into "nothing here", and + // those need different decisions: one waits, the other stops. The caller can + // see the trend without being allowed to act on it. + const pUnder = r === null ? null : pearsonPValue(r, n); + return { + validated: false, + reason: 'insufficient_data', + detail: `${n} < ${req.min_historical_instances} required instances`, + sample_size: n, + rows_needed: req.min_historical_instances - n, + pearson_r: r === null ? null : round6(r), + p_value: pUnder === null ? null : round8(pUnder), + underpowered: true, + }; + } + if (r === null) { + return { validated: false, reason: 'no_variation', sample_size: n, pearson_r: null }; + } + + const p = pearsonPValue(r, n); + const tests = Math.max(1, Math.round(knownNumber(numActiveTests) ?? 1)); + const correctedAlpha = req.bonferroni_correction ? req.max_p_value / tests : req.max_p_value; + + const strongEnough = Math.abs(r) >= req.min_pearson_r; + const significant = p !== null && p < correctedAlpha; + + let reason = null; + if (!strongEnough && !significant) reason = 'weak_correlation_and_not_significant'; + else if (!strongEnough) reason = 'weak_correlation'; + else if (!significant) reason = 'not_significant_after_bonferroni'; + + return { + validated: strongEnough && significant, + reason, + pearson_r: round6(r), + p_value: p === null ? null : round8(p), + corrected_alpha: round6(correctedAlpha), + bonferroni_tests: tests, + sample_size: n, + thresholds: { min_n: req.min_historical_instances, min_abs_r: req.min_pearson_r, base_alpha: req.max_p_value }, + }; +} + +const round6 = (v) => Math.round(v * 1e6) / 1e6; +const round8 = (v) => Math.round(v * 1e8) / 1e8; + +module.exports = { + VALIDATION_REQUIREMENTS, + validateFactor, pearson, pearsonPValue, incompleteBeta, +}; diff --git a/src/services/model/featureRegistry.js b/src/services/model/featureRegistry.js index f3a5a69..080b142 100644 --- a/src/services/model/featureRegistry.js +++ b/src/services/model/featureRegistry.js @@ -130,6 +130,53 @@ for (const f of FEATURES) byKey.set(`${f.sport}|${f.key}`, { ...f, evidence: f.e const idOf = (sport, key) => `${String(sport || '').toLowerCase()}|${key}`; +/** + * PER-STAT VERDICTS — added 2026-08-03, because the first real gate run demanded + * it. Every skill feature FAILED the gate for `hits` at n=570 (max |r| 0.062), + * while the same features show the strongest correlations yet seen for + * `total_bases` (hard-hit marginal r = 0.153). Marking them DEAD sport-wide + * would have killed, on hits evidence, the exact features that look most alive + * on total bases — which is the per-sport doctrine one level deeper: a feature + * earns or loses its place PER STAT, because the physics differ. Exit velocity + * governs extra bases; it barely governs whether a single finds a hole. + * + * A stat-level verdict OVERRIDES the sport-level status for that stat only. + */ +const statVerdicts = new Map(); // `${sport}|${stat}|${key}` -> {status, evidence} +const statIdOf = (sport, stat, key) => `${String(sport || '').toLowerCase()}|${String(stat || '').toLowerCase()}|${key}`; + +/** Record a measured verdict for ONE feature on ONE stat. */ +function recordStatVerdict(sport, stat, key, status, evidence) { + if (!byKey.has(idOf(sport, key))) return { ok: false, reason: 'unknown_feature' }; + if (!Object.values(STATUS).includes(status)) return { ok: false, reason: 'bad_status' }; + statVerdicts.set(statIdOf(sport, stat, key), { status, evidence: evidence || null }); + return { ok: true, status }; +} + +/** Status for a feature on a specific stat — falls back to the sport-level one. */ +function statusForStat(sport, stat, key) { + const v = statVerdicts.get(statIdOf(sport, stat, key)); + if (v) return v.status; + return statusOf(sport, key); +} + +/** The live gate for ONE stat: PROVEN at the stat level, or proven sport-wide and not stat-DEAD. */ +function liveFeaturesForStat(sport, stat) { + return new Set(allFeatures(sport) + .filter((f) => statusForStat(sport, stat, f.key) === STATUS.PROVEN) + .map((f) => f.key)); +} + +/** Candidates still worth measuring for ONE stat — a stat-DEAD feature is not. */ +function candidateFeaturesForStat(sport, stat) { + return new Set(allFeatures(sport) + .filter((f) => { + const st = statusForStat(sport, stat, f.key); + return st === STATUS.CANDIDATE || st === STATUS.PROVEN; + }) + .map((f) => f.key)); +} + /** Every registered feature for a sport (any status). */ function allFeatures(sport) { const sp = String(sport || '').toLowerCase(); @@ -223,6 +270,7 @@ function summary(sport) { /** Test-only: restore the declared statuses so suites cannot leak into each other. */ function __reset() { + statVerdicts.clear(); byKey.clear(); for (const f of FEATURES) byKey.set(`${f.sport}|${f.key}`, { ...f, evidence: f.evidence || null, history: [] }); } @@ -230,5 +278,6 @@ function __reset() { module.exports = { STATUS, MIN_PROMOTION_N, allFeatures, liveFeatures, candidateFeatures, statusOf, isLive, + recordStatVerdict, statusForStat, liveFeaturesForStat, candidateFeaturesForStat, isSufficient, promote, demote, summary, __reset, }; diff --git a/tests/unit/correlateValidator.test.js b/tests/unit/correlateValidator.test.js new file mode 100644 index 0000000..168726e --- /dev/null +++ b/tests/unit/correlateValidator.test.js @@ -0,0 +1,138 @@ +'use strict'; + +/** + * THE GATE — Kev's spec, verified rather than asserted. + * + * `tests/unit/supplementSystems.test.js` checks the same thresholds but inlines + * its own `validateFactor`, so it could never have caught the absence of an + * implementation. These tests import the real module, and the p-value is checked + * against values computable by hand rather than trusted. + */ + +const cv = require('../../src/services/model/correlateValidator'); + +/** Deterministic generator: y correlated with x at roughly the requested level. */ +function correlated(n, rho, seed = 7) { + let s = seed >>> 0; + const rnd = () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; }; + const gauss = () => { + const u = Math.max(1e-12, rnd()); + return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * rnd()); + }; + const xs = []; const ys = []; + for (let i = 0; i < n; i += 1) { + const x = gauss(); + xs.push(x); + ys.push(rho * x + Math.sqrt(Math.max(0, 1 - rho * rho)) * gauss()); + } + return { xs, ys }; +} + +describe('the spec is exactly Kev\'s', () => { + it('n>=500, |r|>=0.15, p<0.05, Bonferroni on', () => { + expect(cv.VALIDATION_REQUIREMENTS).toEqual({ + min_historical_instances: 500, + min_pearson_r: 0.15, + max_p_value: 0.05, + bonferroni_correction: true, + }); + }); +}); + +describe('Pearson r', () => { + it('is exactly 1 and -1 on perfect lines', () => { + expect(cv.pearson([1, 2, 3, 4], [2, 4, 6, 8]).r).toBeCloseTo(1, 12); + expect(cv.pearson([1, 2, 3, 4], [8, 6, 4, 2]).r).toBeCloseTo(-1, 12); + }); + + it('matches a hand-computable case', () => { + // x=[1,2,3,4,5], y=[2,4,5,4,5]; means 3 and 4. + // Sxy = 4+0+0+0+2 = 6 · Sxx = 10 · Syy = 6 → r = 6/sqrt(60) = 0.774597 + expect(cv.pearson([1, 2, 3, 4, 5], [2, 4, 5, 4, 5]).r).toBeCloseTo(0.774597, 6); + }); + + it('DROPS a pair with an unknown side — never zero-fills it', () => { + const clean = cv.pearson([1, 2, 3, 4], [2, 4, 6, 8]); + const withNull = cv.pearson([1, 2, 3, 4, 5], [2, 4, 6, 8, null]); + expect(withNull.n).toBe(4); + expect(withNull.r).toBeCloseTo(clean.r, 12); + // Zero-filling would have added a (5,0) point and wrecked it. + }); + + it('a constant series has no correlation, not a zero one', () => { + expect(cv.pearson([1, 1, 1, 1], [1, 2, 3, 4]).r).toBeNull(); + }); +}); + +describe('the p-value is real', () => { + it('matches known two-sided values for r and n', () => { + // r=0.5, n=10 -> t=1.6330, df=8 -> p ≈ 0.1411 + expect(cv.pearsonPValue(0.5, 10)).toBeCloseTo(0.1411, 3); + // r=0.8, n=10 -> t=3.7712, df=8 -> p ≈ 0.00546 + expect(cv.pearsonPValue(0.8, 10)).toBeCloseTo(0.00546, 4); + // r=0.15, n=500 -> p ≈ 0.000759 + expect(cv.pearsonPValue(0.15, 500)).toBeCloseTo(0.00076, 4); + }); + + it('r = 0 is p = 1, and the sign of r does not change p', () => { + expect(cv.pearsonPValue(0, 500)).toBeCloseTo(1, 6); + expect(cv.pearsonPValue(-0.3, 200)).toBeCloseTo(cv.pearsonPValue(0.3, 200), 12); + }); + + it('is UNDEFINED (null) below 3 points, never a comforting 1.0', () => { + expect(cv.pearsonPValue(0.9, 2)).toBeNull(); + }); + + it('the regularized incomplete beta is symmetric where it must be', () => { + expect(cv.incompleteBeta(2, 3, 0.5) + cv.incompleteBeta(3, 2, 0.5)).toBeCloseTo(1, 10); + }); +}); + +describe('the gate', () => { + it('REFUSES below n=500 no matter how strong the correlation', () => { + const { xs, ys } = correlated(300, 0.9); + const v = cv.validateFactor(xs, ys, 1); + expect(v.validated).toBe(false); + expect(v.reason).toBe('insufficient_data'); + expect(v.sample_size).toBe(300); + }); + + it('REFUSES a weak correlation even at huge n and tiny p', () => { + const { xs, ys } = correlated(4000, 0.08); + const v = cv.validateFactor(xs, ys, 1); + expect(Math.abs(v.pearson_r)).toBeLessThan(0.15); + expect(v.validated).toBe(false); + expect(v.reason).toBe('weak_correlation'); + }); + + it('PASSES a real effect at sufficient n', () => { + const { xs, ys } = correlated(600, 0.30); + const v = cv.validateFactor(xs, ys, 1); + expect(v.validated).toBe(true); + expect(Math.abs(v.pearson_r)).toBeGreaterThanOrEqual(0.15); + expect(v.p_value).toBeLessThan(0.05); + }); + + it('BONFERRONI actually bites — the same data can pass alone and fail in a sweep', () => { + // Chosen so p sits between 0.05/40 and 0.05: significant alone, not in a sweep. + const alone = cv.validateFactor(...Object.values(correlated(520, 0.155)), 1); + const sweep = cv.validateFactor(...Object.values(correlated(520, 0.155)), 40); + expect(sweep.corrected_alpha).toBeCloseTo(0.05 / 40, 12); + expect(sweep.bonferroni_tests).toBe(40); + // Whatever the draw, the corrected bar is strictly harder. + expect(sweep.corrected_alpha).toBeLessThan(alone.corrected_alpha); + if (alone.validated && alone.p_value > 0.05 / 40) expect(sweep.validated).toBe(false); + }); + + it('reports corrected alpha = 0.05/tests, and 0.05 with one test', () => { + const { xs, ys } = correlated(600, 0.3); + expect(cv.validateFactor(xs, ys, 4).corrected_alpha).toBeCloseTo(0.0125, 12); + expect(cv.validateFactor(xs, ys, 1).corrected_alpha).toBeCloseTo(0.05, 12); + }); + + it('always says WHY it failed', () => { + const weak = cv.validateFactor(...Object.values(correlated(600, 0.02)), 1); + expect(weak.reason).toBeTruthy(); + expect(weak.thresholds).toEqual({ min_n: 500, min_abs_r: 0.15, base_alpha: 0.05 }); + }); +}); diff --git a/tests/unit/skillModel.test.js b/tests/unit/skillModel.test.js index ae0c38b..652cf1f 100644 --- a/tests/unit/skillModel.test.js +++ b/tests/unit/skillModel.test.js @@ -78,6 +78,31 @@ describe('D3 — earn its place or it is out', () => { }); }); +describe('D3 — verdicts are PER STAT, because the physics differ', () => { + it('a feature can be DEAD for one stat and still CANDIDATE for another', () => { + // The first real gate run: every skill feature failed for `hits` (n=570, + // max |r| 0.062) while hard-hit rate showed the strongest marginal + // correlation yet seen for `total_bases` (r=0.153). A sport-wide DEAD mark + // would have killed, on hits evidence, the features most alive on TB. + reg.recordStatVerdict('mlb', 'hits', 'batter_barrel_pct', reg.STATUS.DEAD, + { n: 570, r: -0.037, note: 'failed the gate for hits' }); + expect(reg.statusForStat('mlb', 'hits', 'batter_barrel_pct')).toBe(reg.STATUS.DEAD); + expect(reg.statusForStat('mlb', 'total_bases', 'batter_barrel_pct')).toBe(reg.STATUS.CANDIDATE); + expect(reg.candidateFeaturesForStat('mlb', 'hits').has('batter_barrel_pct')).toBe(false); + expect(reg.candidateFeaturesForStat('mlb', 'total_bases').has('batter_barrel_pct')).toBe(true); + }); + + it('a stat with no verdict inherits the sport-level status', () => { + expect(reg.statusForStat('mlb', 'runs', 'batter_exit_velo')).toBe(reg.STATUS.CANDIDATE); + expect(reg.statusForStat('mlb', 'runs', 'champion_home_away_adj')).toBe(reg.STATUS.DEAD); + }); + + it('the live gate is per stat too, and still PROVEN-only', () => { + expect(reg.liveFeaturesForStat('mlb', 'hits').has('batter_barrel_pct')).toBe(false); + expect([...reg.liveFeaturesForStat('mlb', 'hits')]).toEqual(['recent_frequency_prior']); + }); +}); + // ── DISCIPLINE 1 ────────────────────────────────────────────────────────── describe('D1 — skill, not results: the pitcher actually moves the read', () => { const run = (pitcher) => sk.projectSkill({