Build the gate, run it, and find we were proving things on the wrong stat

PREMISE CORRECTION FIRST. statModel.js and correlateValidator.js do not exist
in this repository. The validation spec's only prior form is
src/services/python/blueprints/unconventional.py -- a Flask blueprint in the
Python service that is offline in production, scoring NBA factors against a
warehouse that was never populated -- and tests/unit/supplementSystems.test.js
requires only fs and path while defining its own validateFactor inline at line
368. Those tests assert a re-implementation of the thresholds, not an
implementation, which is exactly why they passed for months while nothing was
connected. The diagnosis behind the order is right -- every challenger was
measured without a gate -- but the cause is that there was no gate on the Node
side to import. So it is built, to the exact spec.

correlateValidator: n>=500, |r|>=0.15, p<0.05, Bonferroni across the sweep.
The p-value is exact rather than approximated (t-transform through a
regularized incomplete beta) and is verified in the suite against known
values, because scipy is not available here. Pairs with an unknown side are
dropped, never zero-filled -- a zero-fill inside a correlation does not add
noise, it invents a point at the origin.

THE RUN, hits, n=570, Bonferroni-8: every skill feature fails, and not
narrowly. The strongest marginal correlation against the counter's residual is
0.062 against a 0.15 bar. That is an effect-size failure at a sample that
would have found a real effect comfortably -- a clean, well-powered negative.
The head-to-head agrees: value engine 0.0499 against the counter's 0.166,
delta -0.116 with CI [-0.189, -0.043]. Not promoted.

THE RUN, total bases, n=295: cannot be tested, and that is the finding.
hard_hit_pct shows a marginal r of 0.153 -- above the threshold -- and exit
velo 0.124, refused solely because n is 205 short of 500. It is the most
encouraging number this work has produced, and it is what the physics
predicts: contact quality governs extra bases, not whether a grounder finds a
hole. We have been testing skill inputs on the one stat where they should not
matter much.

Two things the run forced. Feature verdicts are now PER STAT, because marking
these DEAD sport-wide on hits evidence would have killed, for total bases, the
features that look most alive there -- per-sport doctrine one level deeper.
And the gate now reports r and p even when underpowered, because "not enough
data yet" and "nothing here" demand opposite decisions and a bare refusal was
hiding the best signal on the board.

Next: build the compound TB projection (skillProjection still refuses total
bases by design, since a deterministic bases-per-hit made P(TB>=2) identical
to P(hits>=1)), accrue to n>=500, re-run this gate. Leave hits alone.

4,200 tests green (334 suites); web build exit 0; counter byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-03 02:34:02 -04:00
parent 258d8a6655
commit c7cc8f5e52
8 changed files with 883 additions and 0 deletions
+22
View File
@@ -3,6 +3,28 @@
## Last Updated
2026-08-03
## Session 81 (2026-08-03) — The gate, built and run: hits is dead, total bases is the stat ✅
Spec: `specs/stagea-gate-result.md`. 4,200 tests / 334 suites green, build exit 0.
Counter byte-identical (zero diff on probabilityEstimator/analyzeViaEngine1).
- **PREMISE CORRECTED:** statModel.js and correlateValidator.js do NOT exist in
this repo. The spec lived only in an offline Python blueprint, and
supplementSystems.test.js inlines its own validateFactor (requires just
fs/path). Nothing to connect — so the gate was BUILT to spec.
- **`correlateValidator.js`** — n>=500, |r|>=0.15, p<0.05, Bonferroni. Exact
p-value (incomplete beta), unit-verified against known values.
- **GATE RUN, hits (n=570, Bonferroni-8): EVERYTHING FAILS.** Max marginal |r|
0.062 vs the 0.15 bar — an effect-size failure at a well-powered n. Head-to-head
also loses: 0.0499 vs counter 0.166, delta 0.116 CI [0.189,0.043].
- **GATE RUN, total_bases (n=295): CANNOT TEST — and that is the finding.**
hard_hit_pct marginal r=0.153 (above threshold), exit_velo 0.124; refused only
on n. ~205 more settled rows needed. Matches the physics: contact quality
drives extra bases, not singles.
- **Architecture change the run forced:** per-STAT feature verdicts, so a feature
dead for hits stays alive for TB. Gate now reports r/p when underpowered.
- **Next:** build the compound TB value projection (per-hit bases distribution
from launch/barrel — skillProjection still refuses TB by design), accrue to
n>=500, re-run the gate. Leave hits alone. Do not lower the bar.
## Session 80 (2026-08-03) — The skill engine: built, gated, and Stage A honestly lost ✅
Spec: `specs/skill-engine-architecture.md`. 4,182 tests / 333 suites green, build exit 0.
- **BUILT `src/services/model/`:** `featureRegistry` (CANDIDATE/PROVEN/DEAD per
+36
View File
@@ -1301,6 +1301,42 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
branch. Now its own `statcastTick`. The old test only checked the string
existed — the new one asserts it is not behind the snapshot-hours guard.
## The validation gate + the stat that was wrong (Session 81 — non-obvious)
- **`statModel.js` and `correlateValidator.js` NEVER EXISTED** in this repo. The
spec's only prior form was `src/services/python/blueprints/unconventional.py`
(Flask, in the OFFLINE python service, scoring NBA factors against an empty
warehouse), and `tests/unit/supplementSystems.test.js` INLINES its own
`validateFactor` (line 368; only `fs`/`path` are required). So those tests
passed for months with no implementation to connect — that is the real reason
every challenger was measured ungated.
- **`src/services/model/correlateValidator.js` is the gate now** — n>=500,
|r|>=0.15, p<0.05, Bonferroni. The p-value is EXACT (t-transform via a
regularized incomplete beta, Lentz CF) and unit-verified against known values;
scipy isn't available in Node so don't reach for an approximation. Pairs with
an unknown side are DROPPED — zero-filling a correlation invents a point at
the origin.
- **HITS IS A CLEAN NEGATIVE — stop modelling it.** Gate run at n=570,
Bonferroni-8: EVERY skill feature fails, max marginal |r| = 0.062 vs a 0.15
bar. Not a power problem — an effect-size problem. And the value engine loses
head-to-head (0.0499 vs 0.166, CI [0.189,0.043]). At the 0.5 hits line there
is very little for skill inputs to know.
- **TOTAL BASES IS WHERE THE SIGNAL IS, and it is n-blocked.** Same features:
`hard_hit_pct` marginal r = **0.153** (above threshold), `exit_velo` 0.124,
raw r 0.196/0.167 — refused ONLY because n=295 < 500. Needs ~205 more settled
rows. This is what the physics predicts: contact quality drives EXTRA BASES,
not whether a grounder finds a hole.
- **The gate reports r and p even when underpowered** (`underpowered: true`,
`rows_needed`). "Not enough data yet" and "nothing here" need OPPOSITE
decisions — collapsing them into a bare refusal hid the best signal on the board.
- **Feature verdicts are PER STAT** (`recordStatVerdict` / `statusForStat` /
`candidateFeaturesForStat`). Marking these DEAD sport-wide on hits evidence
would have killed the features most alive on TB. Per-sport doctrine one level
deeper: physics differ per stat.
- **Next is the compound TB projection** — `skillProjection` still REFUSES
total_bases (a deterministic bases-per-hit made P(TB>=2) == P(hits>=1)). Build
the per-hit extra-base distribution off launch/barrel (tb-v1's shape, fed by
skill inputs), accrue to n>=500, re-run this gate. Do NOT lower the bar.
## Active Skills
- vyndr-voice (all user-facing output)
- prop-analysis (grading methodology)
+247
View File
@@ -0,0 +1,247 @@
#!/usr/bin/env node
'use strict';
/**
* stagea-gate-run — RUN THE SKILL FEATURES THROUGH THE GATE, THEN THE COUNTER.
*
* The original sin was never that the challengers were badly built. It was that
* every one of them was measured WITHOUT a validation gate, so "it didn't work"
* and "it was never allowed to prove it works" were indistinguishable. This runs
* the gate that spec'd for exactly this (n>=500, |r|>=0.15, p<0.05, Bonferroni)
* over the real skill features, and only then does the head-to-head.
*
* THREE MEASUREMENTS, in the order that makes each one meaningful:
*
* 1. RAW SIGNAL — corr(feature, outcome). Does this skill input relate to
* whether the prop hit at all?
* 2. MARGINAL CONTRIBUTION — corr(feature, counter residual). This is the one
* that matters: a feature can correlate with the outcome purely because the
* counter already knows it. Only the part the counter MISSES is new
* information, and that is what earns a place. Both go through the gate.
* 3. HEAD-TO-HEAD — the value projection vs the live counter on listed-line
* accuracy, paired bootstrap, out-of-sample.
*
* OUT-OF-SAMPLE: skill profiles are the frozen 2026-07-21 aggregate; only games
* AFTER that date are scored, so no profile contains the game it predicts.
*
* BONFERRONI DENOMINATOR is the number of features tested in this sweep — not 1.
* Testing many and reporting the best without correction is how the S78 residual
* scan produced six "findings" when chance alone predicts three or four.
*
* SUPABASE_URL=... node scripts/stagea-gate-run.js
*/
require('dotenv').config();
const { createClient } = require('@supabase/supabase-js');
const cv = require('../src/services/model/correlateValidator');
const sk = require('../src/services/model/skillProjection');
const reg = require('../src/services/model/featureRegistry');
const mlb = require('../src/services/adapters/mlbStatsAdapter');
const { knownRate, knownNumber } = require('../src/utils/known');
const SB_URL = process.env.SUPABASE_URL;
const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY;
const PAGE = 1000;
const GAMES_SO_FAR = Number(process.env.STAGEA_GAMES_SO_FAR || 103);
const r4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
const mean = (a) => (a.length ? a.reduce((x, y) => x + y, 0) / a.length : null);
const brier = (ps, ys) => (ps.length ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null);
function makeRnd(seed) {
let s = seed >>> 0;
return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
}
function corrOf(xs, ys) { return cv.pearson(xs, ys).r; }
function bootstrapDiff(rows, keyA, keyB, iters = 4000, seed = 20260803) {
if (rows.length < 30) return null;
const rnd = makeRnd(seed);
const n = rows.length;
const diffs = [];
for (let it = 0; it < iters; it += 1) {
const ys = []; const a = []; const b = [];
for (let i = 0; i < n; i += 1) {
const r = rows[Math.floor(rnd() * n)];
ys.push(r.won); a.push(r[keyA]); b.push(r[keyB]);
}
const ca = corrOf(a, ys); const cb = corrOf(b, ys);
if (ca == null || cb == null) continue;
diffs.push(ca - cb);
}
if (diffs.length < 100) return null;
diffs.sort((x, y) => x - y);
const q = (p) => r4(diffs[Math.floor(p * (diffs.length - 1))]);
const ci = [q(0.025), q(0.975)];
return {
point: r4(corrOf(rows.map((r) => r[keyA]), rows.map((r) => r.won))
- corrOf(rows.map((r) => r[keyB]), rows.map((r) => r.won))),
ci95: ci, ci_excludes_zero: ci[0] > 0 || ci[1] < 0,
};
}
async function page(sb, table, select, apply) {
const out = [];
for (let from = 0; ; from += PAGE) {
const { data, error } = await apply(sb.from(table).select(select)).range(from, from + PAGE - 1);
if (error) throw error;
if (!data || data.length === 0) break;
out.push(...data);
if (data.length < PAGE) break;
}
return out;
}
async function opposingStarters(dates) {
const m = new Map();
for (const d of dates) {
let games = [];
try { games = await mlb.getScheduleWithPitchers(d); } catch { games = []; }
for (const g of games) {
if (!g.home || !g.away) continue;
if (g.home.probablePitcher) m.set(`${d}|OPP:${g.home.team}`, g.home.probablePitcher.id);
if (g.away.probablePitcher) m.set(`${d}|OPP:${g.away.team}`, g.away.probablePitcher.id);
}
}
return m;
}
/** `playerKey|date` → opponent faced. The ledger's team/opponent are NULL. */
async function opponentByPlayerDate(players) {
const map = new Map();
for (const [key, name] of players) {
try {
const found = await mlb.searchPlayer(name);
if (!found || !found.id) continue;
const log = await mlb.getPlayerGameLog(found.id);
for (const g of log || []) {
if (g && g.date && g.opponent) map.set(`${key}|${String(g.date).slice(0, 10)}`, g.opponent);
}
} catch { /* no log → no pitcher for those rows */ }
}
return map;
}
async function main() {
if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required');
const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } });
const statcast = await page(sb, 'statcast_aggregates', '*', (q) => q.eq('sport', 'mlb'));
const freezeDate = statcast.reduce((mx, r) => (String(r.updated_at) > mx ? String(r.updated_at) : mx), '').slice(0, 10);
const batters = new Map(); const pitchersById = new Map();
for (const r of statcast) {
if (r.role === 'pitcher' && r.source_id != null) pitchersById.set(Number(r.source_id), sk.fromStatcastRow(r));
if (r.player_key && r.role === 'batter') {
const prev = batters.get(r.player_key);
if (!prev || Number(r.sample_pa || 0) > Number(prev.rawPa || 0)) {
batters.set(r.player_key, Object.assign(sk.fromStatcastRow(r), { rawPa: Number(r.sample_pa || 0) }));
}
}
}
const led = await page(sb, 'ledger_entries',
'player_key, player_name, stat, line, side, outcome, game_date, p_win, quarantine_reason',
(q) => q.eq('sport', 'mlb').is('user_id', null)
.in('stat', ['hits', 'total_bases'])
.in('outcome', ['hit', 'miss']).not('p_win', 'is', null));
const clean = led.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book')
&& String(r.game_date) > freezeDate);
const dates = [...new Set(clean.map((r) => r.game_date))].sort();
const starters = await opposingStarters(dates);
const players = new Map();
for (const r of clean) if (!players.has(r.player_key)) players.set(r.player_key, r.player_name);
const oppByPlayerDate = await opponentByPlayerDate(players);
const allowed = reg.candidateFeatures('mlb');
const rows = [];
for (const r of clean) {
const bat = batters.get(r.player_key);
if (!bat) continue;
const faced = oppByPlayerDate.get(`${r.player_key}|${r.game_date}`) || null;
const pit = faced ? pitchersById.get(Number(starters.get(`${r.game_date}|OPP:${faced}`))) || null : null;
const paRate = bat.rawPa > 0 ? Math.min(5.2, Math.max(2.0, bat.rawPa / GAMES_SO_FAR)) : null;
const under = String(r.side).toLowerCase() === 'under';
const won = r.outcome === 'hit' ? 1 : 0;
const champ = Number(r.p_win);
// The value projection (hits only — TB is refused by design, see skillProjection).
const proj = r.stat === 'hits'
? sk.projectSkill({ batter: bat, pitcher: pit, park: 1, archetype: null,
statType: 'hits', line: Number(r.line), expectedPa: paRate, allowed })
: null;
rows.push({
stat: r.stat, won, champ,
skill: proj ? (under ? 1 - proj.p_over_line : proj.p_over_line) : null,
residual: won - champ,
had_pitcher: !!pit,
// Candidate skill features, archetype-relevant, in probability space.
batter_barrel_pct: knownRate(bat.barrel_pct),
batter_hard_hit_pct: knownRate(bat.hard_hit_pct),
batter_exit_velo: knownRate(bat.avg_exit_velo),
batter_launch_angle: knownRate(bat.avg_launch_angle),
batter_k_pct: knownRate(bat.k_pct),
batter_bb_pct: knownRate(bat.bb_pct),
pitcher_k_pct: pit ? knownRate(pit.k_pct) : null,
pitcher_hard_hit_allowed: pit ? knownRate(pit.hard_hit_pct) : null,
});
}
const FEATURES = ['batter_barrel_pct', 'batter_hard_hit_pct', 'batter_exit_velo',
'batter_launch_angle', 'batter_k_pct', 'batter_bb_pct',
'pitcher_k_pct', 'pitcher_hard_hit_allowed'];
const perStat = {};
for (const stat of ['hits', 'total_bases']) {
const rs = rows.filter((r) => r.stat === stat);
if (rs.length === 0) continue;
const tests = FEATURES.length; // the Bonferroni denominator for THIS sweep
const gate = {};
for (const f of FEATURES) {
const xs = rs.map((r) => r[f]);
gate[f] = {
// 1. does it relate to the outcome at all?
raw_vs_outcome: cv.validateFactor(xs, rs.map((r) => r.won), tests),
// 2. THE ONE THAT COUNTS — is any of it NEW, i.e. missed by the counter?
marginal_vs_counter_residual: cv.validateFactor(xs, rs.map((r) => r.residual), tests),
};
}
const passed = FEATURES.filter((f) => gate[f].marginal_vs_counter_residual.validated);
perStat[stat] = {
n: rs.length,
base_rate: r4(mean(rs.map((r) => r.won))),
bonferroni_tests: tests,
features_passing_gate_on_marginal: passed,
gate,
};
}
// HEAD-TO-HEAD — hits only (the value engine covers hits).
const h2h = rows.filter((r) => r.stat === 'hits' && r.skill != null);
const ys = h2h.map((r) => r.won);
const bs = bootstrapDiff(h2h, 'skill', 'champ');
console.log(JSON.stringify({
premise_correction: 'statModel.js and correlateValidator.js do not exist in this repo. The gate was implemented to the spec in src/services/python/blueprints/unconventional.py (VALIDATION_REQUIREMENTS); supplementSystems.test.js inlines its own validateFactor and imports no implementation.',
out_of_sample: `skill profiles frozen ${freezeDate}; only game_date > ${freezeDate} scored`,
gate_spec: cv.VALIDATION_REQUIREMENTS,
per_stat_gate: perStat,
head_to_head_hits: {
n: h2h.length,
pitcher_coverage: r4(mean(h2h.map((r) => (r.had_pitcher ? 1 : 0)))),
base_rate: r4(mean(ys)),
resolution: { value_engine: r4(corrOf(h2h.map((r) => r.skill), ys)), counter: r4(corrOf(h2h.map((r) => r.champ), ys)) },
brier: { value_engine: r4(brier(h2h.map((r) => r.skill), ys)), counter: r4(brier(h2h.map((r) => r.champ), ys)) },
delta: bs,
verdict: !bs ? 'N-BLOCKED'
: (bs.ci_excludes_zero && bs.point > 0) ? 'VALUE ENGINE BEATS THE COUNTER'
: (bs.ci_excludes_zero && bs.point < 0) ? 'LOSES to the counter — iterate, do not promote'
: 'INCONCLUSIVE — do not promote',
},
}, null, 2));
process.exit(0);
}
main().catch((e) => { console.error(e); process.exit(1); });
+158
View File
@@ -0,0 +1,158 @@
# STAGE A THROUGH THE GATE — and the stat we should have picked
**2026-08-03.** Challenger-only. The counter is byte-identical (zero diff on
`probabilityEstimator.js` / `analyzeViaEngine1.js`).
> **RESULT: no skill feature passes the gate for HITS, and the value engine
> loses to the counter on hits. But the same features show the strongest
> correlations yet measured for TOTAL BASES — and total bases is 205 settled rows
> short of being testable at all.** We have been proving things on the wrong stat.
---
## 0. Premise correction, with evidence
The order describes `statModel.js` and `correlateValidator.js` as *built, tested,
disconnected*. **Neither file exists in this repository.**
```
find . -name "statModel*" → nothing (outside .claude/worktrees)
find . -name "correlateValidator*" → nothing
```
The validation spec's only prior existence is
`src/services/python/blueprints/unconventional.py` — a Flask blueprint in the
**Python service that is offline in production**, scoring NBA "unconventional
factors" (altitude, contract year, referee crew) against a data warehouse that
was never populated. And `tests/unit/supplementSystems.test.js` has exactly
**two `require()` calls — `fs` and `path`** — and defines its own `validateFactor`
inline at line 368. **Those tests assert a re-implementation of the thresholds,
not an implementation of them**, which is precisely why they passed for months
while nothing existed to connect.
So the diagnosis behind the order is right — **every challenger was measured
without a gate** — but the cause is simpler than "someone forgot to import it":
**there was no gate on the Node side to import.** It is built now, to the exact
spec, and this document is its first real run.
## 1. The gate (`src/services/model/correlateValidator.js`)
`n ≥ 500 · |Pearson r| ≥ 0.15 · p < 0.05 · Bonferroni across the sweep`,
ported from `VALIDATION_REQUIREMENTS` in the Python blueprint.
The p-value is exact, not approximated — scipy is unavailable in Node, so it is
computed from the t transform via a regularized incomplete beta (Lentz continued
fraction) and **verified against known values** in the unit suite (r=0.5,n=10 →
p≈0.1411; r=0.8,n=10 → p≈0.00546; r=0.15,n=500 → p≈0.00076). Pairs with an
unknown side are **dropped, never zero-filled** — a zero-fill inside a
correlation doesn't add noise, it invents a point at the origin.
**Bonferroni is not decoration here.** The S78 residual scan ran 70 tests at
α=.05 and produced six "significant" hits — almost exactly the 34 chance
predicts. The correction is what makes PROVEN mean anything.
## 2. THE GATE RUN — hits (n=570, clears the bar)
Bonferroni denominator = 8 (the features in this sweep), corrected α = 0.00625.
Two questions per feature: does it relate to the outcome at all, and — the one
that matters — is any of it **new**, i.e. correlated with what the counter
*misses* (`won p_win`)?
| feature | r vs outcome | **r vs counter residual** | p | verdict |
|---|---|---|---|---|
| batter_bb_pct | 0.047 | **0.062** | 0.138 | weak + not significant |
| batter_exit_velo | 0.031 | 0.056 | 0.180 | weak + not significant |
| batter_hard_hit_pct | 0.007 | 0.039 | 0.355 | weak + not significant |
| batter_barrel_pct | 0.020 | 0.037 | 0.378 | weak + not significant |
| batter_k_pct | 0.014 | 0.015 | 0.722 | weak + not significant |
| batter_launch_angle | 0.010 | 0.005 | 0.910 | weak + not significant |
| pitcher_k_pct | +0.018 | +0.012 | 0.777 | weak + not significant |
| pitcher_hard_hit_allowed | +0.016 | +0.011 | 0.806 | weak + not significant |
**Nothing passes. Nothing is close.** The strongest marginal |r| is 0.062 against
a 0.15 bar — not "failed on significance", failed on effect size, at a sample
that would have detected a real effect easily.
**This is a clean, well-powered negative result for hits**, and it explains every
earlier failure at a stroke: the skill features carry essentially no information
about whether a hitter gets one hit. The counter isn't beating them because it's
clever; it's beating them because at the 0.5 hits line there is very little for
them to know.
## 3. THE GATE RUN — total bases (n=295, CANNOT be tested)
| feature | r vs outcome | **r vs counter residual** | status |
|---|---|---|---|
| **batter_hard_hit_pct** | **+0.196** | **+0.153** | insufficient data (needs 205 more rows) |
| **batter_exit_velo** | **+0.167** | +0.124 | insufficient data |
| batter_barrel_pct | +0.121 | +0.088 | insufficient data |
| pitcher_k_pct | 0.049 | 0.077 | insufficient data |
**`hard_hit_pct` marginal r = 0.153 — above the 0.15 threshold — and it is
refused solely because n = 295 < 500.** That is the gate working correctly, and
it is also the most encouraging number this line of work has produced.
The gate now reports `r` and `p` even when underpowered, with
`underpowered: true` and `rows_needed`. **"Not enough data yet" and "nothing
here" require opposite decisions** — one waits, the other stops — and collapsing
them into a bare refusal was hiding the single most useful signal on the board.
### And it is exactly what the physics predicts
Exit velocity and hard-hit rate govern whether contact becomes **extra bases**.
They barely govern whether a ground ball finds a hole for a single. The archetype
doctrine says the same thing: contact quality is the BOMBER lane, and total bases
is the power stat. **We tested skill inputs on the one stat where skill inputs
shouldn't matter much.**
## 4. Head-to-head — hits (the stat we could test)
Out-of-sample: profiles frozen 2026-07-21, only later games scored. 570 rows,
91.9% opposing-pitcher coverage.
```
resolution value engine 0.0499 counter 0.1660
brier value engine 0.2839 counter 0.2506
delta 0.1161 CI95 [0.1892, 0.0426] ← excludes zero
```
**LOSES. Not promoted.** Consistent with §2: a model built from features that
carry no hits signal cannot beat a counter on hits.
## 5. What changed in the architecture because of this run
**Verdicts are now PER STAT** (`recordStatVerdict` / `statusForStat` /
`candidateFeaturesForStat`). Marking these features DEAD sport-wide on hits
evidence would have killed, for total bases, the exact features that look most
alive there. That is the per-sport doctrine one level deeper: **a feature earns
or loses its place per stat, because the physics differ per stat.** Locked by
test.
## 6. Verdict and the next order
**Stage A on hits: the windshield loses, and the gate says why — there is no
skill signal there to find.** Do not iterate the hits model; that is polishing a
read of a stat whose outcome our best inputs are blind to.
**Stage A should move to TOTAL BASES**, where:
- the leading marginal correlation (0.153) is already at the threshold,
- the gate needs **205 more settled TB rows** — days, now that settlement works,
- and `skillProjection` currently **refuses** total bases by design, because a
deterministic bases-per-hit multiplier made P(TB≥2) identical to P(hits≥1).
**That refusal is now the thing to build:** a real per-hit extra-base
distribution off the launch/barrel profile — which is tb-v1's compound shape,
fed by skill inputs instead of raw counts.
Ranked next steps:
1. **Build the compound TB value projection** (per-hit bases distribution from
launch/barrel), so TB can be projected at all.
2. **Accrue to n ≥ 500 on TB** and re-run this exact gate.
3. **Then** head-to-head on TB. If `hard_hit_pct` holds its 0.153 through the
Bonferroni-corrected gate at full n, that is the first feature in this
codebase to genuinely earn its place.
4. Leave hits alone.
**Not recommended:** promoting anything now, lowering the n≥500 bar to make TB
testable today, or re-running the hits model with more features. The bar being
inconvenient is what makes it worth having.
+208
View File
@@ -0,0 +1,208 @@
'use strict';
/**
* correlateValidator THE GATE A FEATURE MUST PASS TO EARN ITS PLACE.
*
* Kev's spec, exactly: n 500 · |Pearson r| 0.15 · p < 0.05 · Bonferroni-
* corrected across the number of factors being tested at once.
*
* PROVENANCE, because it matters. This is a JS implementation of the thresholds
* declared in `src/services/python/blueprints/unconventional.py`
* (`VALIDATION_REQUIREMENTS`), which is the spec's only prior existence a
* Flask blueprint in the Python service that is offline in production, scoring
* NBA "unconventional factors" against a data warehouse that was never
* populated. `tests/unit/supplementSystems.test.js` asserts the same thresholds
* but INLINES its own `validateFactor` (line 368) rather than importing one, so
* those tests passed for months without any implementation existing in the Node
* path at all. That is the whole reason every challenger to date was measured
* without a gate: there was no gate on this side of the codebase to call.
*
* WHY BONFERRONI IS NOT OPTIONAL HERE. The S78 residual scan ran 14 features ×
* 5 stats = 70 tests at α=.05 and produced six "significant" results almost
* exactly the 34 the null hypothesis predicts by chance. Without a correction,
* a wide feature sweep manufactures discoveries. The correction is what makes
* "PROVEN" mean something.
*
* THE P-VALUE IS REAL, NOT APPROXIMATED. scipy's `pearsonr` is not available in
* Node, so the two-sided p is computed from the exact t transform
*
* t = r · sqrt((n2) / (1)), p = I_{v/(v+)}(v/2, 1/2), v = n2
*
* using a regularized incomplete beta (Lentz continued fraction). Verified in
* the unit suite against known values rather than trusted.
*
* HONESTY: pairs where either side is unknown are DROPPED, never zero-filled
* `Number(null) === 0` inside a correlation does not merely add noise, it
* invents a data point at the origin and drags r toward whatever the means are.
*/
const { knownNumber } = require('../../utils/known');
const VALIDATION_REQUIREMENTS = Object.freeze({
min_historical_instances: 500,
min_pearson_r: 0.15,
max_p_value: 0.05, // BEFORE Bonferroni
bonferroni_correction: true,
});
/** log-gamma (Lanczos) — for the incomplete beta. */
function logGamma(x) {
const c = [76.18009172947146, -86.50532032941677, 24.01409824083091,
-1.231739572450155, 0.1208650973866179e-2, -0.5395239384953e-5];
let y = x;
let tmp = x + 5.5;
tmp -= (x + 0.5) * Math.log(tmp);
let ser = 1.000000000190015;
for (let j = 0; j < 6; j += 1) { y += 1; ser += c[j] / y; }
return -tmp + Math.log((2.5066282746310005 * ser) / x);
}
/** Continued fraction for the incomplete beta (modified Lentz). */
function betaCf(a, b, x) {
const MAXIT = 200;
const EPS = 3e-14;
const FPMIN = 1e-300;
const qab = a + b;
const qap = a + 1;
const qam = a - 1;
let c = 1;
let d = 1 - (qab * x) / qap;
if (Math.abs(d) < FPMIN) d = FPMIN;
d = 1 / d;
let h = d;
for (let m = 1; m <= MAXIT; m += 1) {
const m2 = 2 * m;
let aa = (m * (b - m) * x) / ((qam + m2) * (a + m2));
d = 1 + aa * d; if (Math.abs(d) < FPMIN) d = FPMIN;
c = 1 + aa / c; if (Math.abs(c) < FPMIN) c = FPMIN;
d = 1 / d;
h *= d * c;
aa = (-(a + m) * (qab + m) * x) / ((a + m2) * (qap + m2));
d = 1 + aa * d; if (Math.abs(d) < FPMIN) d = FPMIN;
c = 1 + aa / c; if (Math.abs(c) < FPMIN) c = FPMIN;
d = 1 / d;
const del = d * c;
h *= del;
if (Math.abs(del - 1) < EPS) break;
}
return h;
}
/** Regularized incomplete beta I_x(a,b). */
function incompleteBeta(a, b, x) {
if (x <= 0) return 0;
if (x >= 1) return 1;
const bt = Math.exp(logGamma(a + b) - logGamma(a) - logGamma(b)
+ a * Math.log(x) + b * Math.log(1 - x));
if (x < (a + 1) / (a + b + 2)) return (bt * betaCf(a, b, x)) / a;
return 1 - (bt * betaCf(b, a, 1 - x)) / b;
}
/**
* Two-sided p-value for a Pearson r on n paired observations.
* Returns null when it is not defined (n < 3), never a comforting 1.0.
*/
function pearsonPValue(r, n) {
const rr = knownNumber(r);
const nn = knownNumber(n);
if (rr === null || nn === null || nn < 3) return null;
const v = nn - 2;
const r2 = Math.min(1, rr * rr);
if (r2 >= 1) return 0; // perfect correlation
const t2 = (r2 * v) / (1 - r2);
return incompleteBeta(v / 2, 0.5, v / (v + t2));
}
/**
* Pearson r over PAIRED observations, dropping any pair with an unknown side.
* Returns { r, n } so the caller always knows how many pairs actually counted
* a correlation quoted without its surviving n hides exactly this kind of loss.
*/
function pearson(xs, ys) {
const X = []; const Y = [];
const len = Math.min((xs || []).length, (ys || []).length);
for (let i = 0; i < len; i += 1) {
const a = knownNumber(xs[i]);
const b = knownNumber(ys[i]);
if (a === null || b === null) continue; // DROP the pair, never zero-fill
X.push(a); Y.push(b);
}
const n = X.length;
if (n < 3) return { r: null, n };
const mx = X.reduce((s, v) => s + v, 0) / n;
const my = Y.reduce((s, v) => s + v, 0) / n;
let sxy = 0; let sxx = 0; let syy = 0;
for (let i = 0; i < n; i += 1) {
const dx = X[i] - mx; const dy = Y[i] - my;
sxy += dx * dy; sxx += dx * dx; syy += dy * dy;
}
if (sxx <= 0 || syy <= 0) return { r: null, n }; // a constant has no correlation
return { r: sxy / Math.sqrt(sxx * syy), n };
}
/**
* THE GATE.
*
* @param {number[]} factorValues the feature, per observation
* @param {number[]} outcomeValues what actually happened, per observation
* @param {number} numActiveTests how many factors are being tested together
* (the Bonferroni denominator). 1 means no correction.
* @returns {object} a verdict that always states WHY, so a failure is readable
* without re-running anything.
*/
function validateFactor(factorValues, outcomeValues, numActiveTests = 1, opts = {}) {
const req = { ...VALIDATION_REQUIREMENTS, ...(opts.requirements || {}) };
const { r, n } = pearson(factorValues, outcomeValues);
if (n < req.min_historical_instances) {
// REPORT r AND p ANYWAY. `validated` stays false — the bar is the bar — but
// hiding the numbers turns "not enough data yet" into "nothing here", and
// those need different decisions: one waits, the other stops. The caller can
// see the trend without being allowed to act on it.
const pUnder = r === null ? null : pearsonPValue(r, n);
return {
validated: false,
reason: 'insufficient_data',
detail: `${n} < ${req.min_historical_instances} required instances`,
sample_size: n,
rows_needed: req.min_historical_instances - n,
pearson_r: r === null ? null : round6(r),
p_value: pUnder === null ? null : round8(pUnder),
underpowered: true,
};
}
if (r === null) {
return { validated: false, reason: 'no_variation', sample_size: n, pearson_r: null };
}
const p = pearsonPValue(r, n);
const tests = Math.max(1, Math.round(knownNumber(numActiveTests) ?? 1));
const correctedAlpha = req.bonferroni_correction ? req.max_p_value / tests : req.max_p_value;
const strongEnough = Math.abs(r) >= req.min_pearson_r;
const significant = p !== null && p < correctedAlpha;
let reason = null;
if (!strongEnough && !significant) reason = 'weak_correlation_and_not_significant';
else if (!strongEnough) reason = 'weak_correlation';
else if (!significant) reason = 'not_significant_after_bonferroni';
return {
validated: strongEnough && significant,
reason,
pearson_r: round6(r),
p_value: p === null ? null : round8(p),
corrected_alpha: round6(correctedAlpha),
bonferroni_tests: tests,
sample_size: n,
thresholds: { min_n: req.min_historical_instances, min_abs_r: req.min_pearson_r, base_alpha: req.max_p_value },
};
}
const round6 = (v) => Math.round(v * 1e6) / 1e6;
const round8 = (v) => Math.round(v * 1e8) / 1e8;
module.exports = {
VALIDATION_REQUIREMENTS,
validateFactor, pearson, pearsonPValue, incompleteBeta,
};
+49
View File
@@ -130,6 +130,53 @@ for (const f of FEATURES) byKey.set(`${f.sport}|${f.key}`, { ...f, evidence: f.e
const idOf = (sport, key) => `${String(sport || '').toLowerCase()}|${key}`;
/**
* PER-STAT VERDICTS added 2026-08-03, because the first real gate run demanded
* it. Every skill feature FAILED the gate for `hits` at n=570 (max |r| 0.062),
* while the same features show the strongest correlations yet seen for
* `total_bases` (hard-hit marginal r = 0.153). Marking them DEAD sport-wide
* would have killed, on hits evidence, the exact features that look most alive
* on total bases which is the per-sport doctrine one level deeper: a feature
* earns or loses its place PER STAT, because the physics differ. Exit velocity
* governs extra bases; it barely governs whether a single finds a hole.
*
* A stat-level verdict OVERRIDES the sport-level status for that stat only.
*/
const statVerdicts = new Map(); // `${sport}|${stat}|${key}` -> {status, evidence}
const statIdOf = (sport, stat, key) => `${String(sport || '').toLowerCase()}|${String(stat || '').toLowerCase()}|${key}`;
/** Record a measured verdict for ONE feature on ONE stat. */
function recordStatVerdict(sport, stat, key, status, evidence) {
if (!byKey.has(idOf(sport, key))) return { ok: false, reason: 'unknown_feature' };
if (!Object.values(STATUS).includes(status)) return { ok: false, reason: 'bad_status' };
statVerdicts.set(statIdOf(sport, stat, key), { status, evidence: evidence || null });
return { ok: true, status };
}
/** Status for a feature on a specific stat — falls back to the sport-level one. */
function statusForStat(sport, stat, key) {
const v = statVerdicts.get(statIdOf(sport, stat, key));
if (v) return v.status;
return statusOf(sport, key);
}
/** The live gate for ONE stat: PROVEN at the stat level, or proven sport-wide and not stat-DEAD. */
function liveFeaturesForStat(sport, stat) {
return new Set(allFeatures(sport)
.filter((f) => statusForStat(sport, stat, f.key) === STATUS.PROVEN)
.map((f) => f.key));
}
/** Candidates still worth measuring for ONE stat — a stat-DEAD feature is not. */
function candidateFeaturesForStat(sport, stat) {
return new Set(allFeatures(sport)
.filter((f) => {
const st = statusForStat(sport, stat, f.key);
return st === STATUS.CANDIDATE || st === STATUS.PROVEN;
})
.map((f) => f.key));
}
/** Every registered feature for a sport (any status). */
function allFeatures(sport) {
const sp = String(sport || '').toLowerCase();
@@ -223,6 +270,7 @@ function summary(sport) {
/** Test-only: restore the declared statuses so suites cannot leak into each other. */
function __reset() {
statVerdicts.clear();
byKey.clear();
for (const f of FEATURES) byKey.set(`${f.sport}|${f.key}`, { ...f, evidence: f.evidence || null, history: [] });
}
@@ -230,5 +278,6 @@ function __reset() {
module.exports = {
STATUS, MIN_PROMOTION_N,
allFeatures, liveFeatures, candidateFeatures, statusOf, isLive,
recordStatVerdict, statusForStat, liveFeaturesForStat, candidateFeaturesForStat,
isSufficient, promote, demote, summary, __reset,
};
+138
View File
@@ -0,0 +1,138 @@
'use strict';
/**
* THE GATE Kev's spec, verified rather than asserted.
*
* `tests/unit/supplementSystems.test.js` checks the same thresholds but inlines
* its own `validateFactor`, so it could never have caught the absence of an
* implementation. These tests import the real module, and the p-value is checked
* against values computable by hand rather than trusted.
*/
const cv = require('../../src/services/model/correlateValidator');
/** Deterministic generator: y correlated with x at roughly the requested level. */
function correlated(n, rho, seed = 7) {
let s = seed >>> 0;
const rnd = () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; };
const gauss = () => {
const u = Math.max(1e-12, rnd());
return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * rnd());
};
const xs = []; const ys = [];
for (let i = 0; i < n; i += 1) {
const x = gauss();
xs.push(x);
ys.push(rho * x + Math.sqrt(Math.max(0, 1 - rho * rho)) * gauss());
}
return { xs, ys };
}
describe('the spec is exactly Kev\'s', () => {
it('n>=500, |r|>=0.15, p<0.05, Bonferroni on', () => {
expect(cv.VALIDATION_REQUIREMENTS).toEqual({
min_historical_instances: 500,
min_pearson_r: 0.15,
max_p_value: 0.05,
bonferroni_correction: true,
});
});
});
describe('Pearson r', () => {
it('is exactly 1 and -1 on perfect lines', () => {
expect(cv.pearson([1, 2, 3, 4], [2, 4, 6, 8]).r).toBeCloseTo(1, 12);
expect(cv.pearson([1, 2, 3, 4], [8, 6, 4, 2]).r).toBeCloseTo(-1, 12);
});
it('matches a hand-computable case', () => {
// x=[1,2,3,4,5], y=[2,4,5,4,5]; means 3 and 4.
// Sxy = 4+0+0+0+2 = 6 · Sxx = 10 · Syy = 6 → r = 6/sqrt(60) = 0.774597
expect(cv.pearson([1, 2, 3, 4, 5], [2, 4, 5, 4, 5]).r).toBeCloseTo(0.774597, 6);
});
it('DROPS a pair with an unknown side — never zero-fills it', () => {
const clean = cv.pearson([1, 2, 3, 4], [2, 4, 6, 8]);
const withNull = cv.pearson([1, 2, 3, 4, 5], [2, 4, 6, 8, null]);
expect(withNull.n).toBe(4);
expect(withNull.r).toBeCloseTo(clean.r, 12);
// Zero-filling would have added a (5,0) point and wrecked it.
});
it('a constant series has no correlation, not a zero one', () => {
expect(cv.pearson([1, 1, 1, 1], [1, 2, 3, 4]).r).toBeNull();
});
});
describe('the p-value is real', () => {
it('matches known two-sided values for r and n', () => {
// r=0.5, n=10 -> t=1.6330, df=8 -> p ≈ 0.1411
expect(cv.pearsonPValue(0.5, 10)).toBeCloseTo(0.1411, 3);
// r=0.8, n=10 -> t=3.7712, df=8 -> p ≈ 0.00546
expect(cv.pearsonPValue(0.8, 10)).toBeCloseTo(0.00546, 4);
// r=0.15, n=500 -> p ≈ 0.000759
expect(cv.pearsonPValue(0.15, 500)).toBeCloseTo(0.00076, 4);
});
it('r = 0 is p = 1, and the sign of r does not change p', () => {
expect(cv.pearsonPValue(0, 500)).toBeCloseTo(1, 6);
expect(cv.pearsonPValue(-0.3, 200)).toBeCloseTo(cv.pearsonPValue(0.3, 200), 12);
});
it('is UNDEFINED (null) below 3 points, never a comforting 1.0', () => {
expect(cv.pearsonPValue(0.9, 2)).toBeNull();
});
it('the regularized incomplete beta is symmetric where it must be', () => {
expect(cv.incompleteBeta(2, 3, 0.5) + cv.incompleteBeta(3, 2, 0.5)).toBeCloseTo(1, 10);
});
});
describe('the gate', () => {
it('REFUSES below n=500 no matter how strong the correlation', () => {
const { xs, ys } = correlated(300, 0.9);
const v = cv.validateFactor(xs, ys, 1);
expect(v.validated).toBe(false);
expect(v.reason).toBe('insufficient_data');
expect(v.sample_size).toBe(300);
});
it('REFUSES a weak correlation even at huge n and tiny p', () => {
const { xs, ys } = correlated(4000, 0.08);
const v = cv.validateFactor(xs, ys, 1);
expect(Math.abs(v.pearson_r)).toBeLessThan(0.15);
expect(v.validated).toBe(false);
expect(v.reason).toBe('weak_correlation');
});
it('PASSES a real effect at sufficient n', () => {
const { xs, ys } = correlated(600, 0.30);
const v = cv.validateFactor(xs, ys, 1);
expect(v.validated).toBe(true);
expect(Math.abs(v.pearson_r)).toBeGreaterThanOrEqual(0.15);
expect(v.p_value).toBeLessThan(0.05);
});
it('BONFERRONI actually bites — the same data can pass alone and fail in a sweep', () => {
// Chosen so p sits between 0.05/40 and 0.05: significant alone, not in a sweep.
const alone = cv.validateFactor(...Object.values(correlated(520, 0.155)), 1);
const sweep = cv.validateFactor(...Object.values(correlated(520, 0.155)), 40);
expect(sweep.corrected_alpha).toBeCloseTo(0.05 / 40, 12);
expect(sweep.bonferroni_tests).toBe(40);
// Whatever the draw, the corrected bar is strictly harder.
expect(sweep.corrected_alpha).toBeLessThan(alone.corrected_alpha);
if (alone.validated && alone.p_value > 0.05 / 40) expect(sweep.validated).toBe(false);
});
it('reports corrected alpha = 0.05/tests, and 0.05 with one test', () => {
const { xs, ys } = correlated(600, 0.3);
expect(cv.validateFactor(xs, ys, 4).corrected_alpha).toBeCloseTo(0.0125, 12);
expect(cv.validateFactor(xs, ys, 1).corrected_alpha).toBeCloseTo(0.05, 12);
});
it('always says WHY it failed', () => {
const weak = cv.validateFactor(...Object.values(correlated(600, 0.02)), 1);
expect(weak.reason).toBeTruthy();
expect(weak.thresholds).toEqual({ min_n: 500, min_abs_r: 0.15, base_alpha: 0.05 });
});
});
+25
View File
@@ -78,6 +78,31 @@ describe('D3 — earn its place or it is out', () => {
});
});
describe('D3 — verdicts are PER STAT, because the physics differ', () => {
it('a feature can be DEAD for one stat and still CANDIDATE for another', () => {
// The first real gate run: every skill feature failed for `hits` (n=570,
// max |r| 0.062) while hard-hit rate showed the strongest marginal
// correlation yet seen for `total_bases` (r=0.153). A sport-wide DEAD mark
// would have killed, on hits evidence, the features most alive on TB.
reg.recordStatVerdict('mlb', 'hits', 'batter_barrel_pct', reg.STATUS.DEAD,
{ n: 570, r: -0.037, note: 'failed the gate for hits' });
expect(reg.statusForStat('mlb', 'hits', 'batter_barrel_pct')).toBe(reg.STATUS.DEAD);
expect(reg.statusForStat('mlb', 'total_bases', 'batter_barrel_pct')).toBe(reg.STATUS.CANDIDATE);
expect(reg.candidateFeaturesForStat('mlb', 'hits').has('batter_barrel_pct')).toBe(false);
expect(reg.candidateFeaturesForStat('mlb', 'total_bases').has('batter_barrel_pct')).toBe(true);
});
it('a stat with no verdict inherits the sport-level status', () => {
expect(reg.statusForStat('mlb', 'runs', 'batter_exit_velo')).toBe(reg.STATUS.CANDIDATE);
expect(reg.statusForStat('mlb', 'runs', 'champion_home_away_adj')).toBe(reg.STATUS.DEAD);
});
it('the live gate is per stat too, and still PROVEN-only', () => {
expect(reg.liveFeaturesForStat('mlb', 'hits').has('batter_barrel_pct')).toBe(false);
expect([...reg.liveFeaturesForStat('mlb', 'hits')]).toEqual(['recent_frequency_prior']);
});
});
// ── DISCIPLINE 1 ──────────────────────────────────────────────────────────
describe('D1 — skill, not results: the pitcher actually moves the read', () => {
const run = (pitcher) => sk.projectSkill({