diff --git a/BUILD-STATE.md b/BUILD-STATE.md index ccbf09d..2574dcc 100755 --- a/BUILD-STATE.md +++ b/BUILD-STATE.md @@ -3,6 +3,26 @@ ## Last Updated 2026-08-03 +## Session 92 (2026-08-04) — The two-part factor gate; one factor proves ✅ +4,286 tests / 340 suites green, build exit 0. Counter + frozen clusters +byte-identical. +- **`factorGate.js`** — a factor must MOVE the prediction off the base rate AND + improve out-of-sample Brier. Movement alone = **THEATER**, rejected by name. + Baseline is the player's leave-one-out base rate (the literal "he's due" null). +- **Cumulative correction applied to the INTERVAL** (99.9% at 50 tests). This + flipped defense and platoon out of "proves" — a plain 95% CI would have shipped + two unproven factors. +- **NOT_PROVEN_AT_CORRECTED_BAR added** as distinct from THEATER; conflating them + would repeat "insufficient evidence = evidence of absence". +- **RESULT (hits, n=741): `pitcher_contact_profile` PROVES** (Brier −0.0066, CI + [−0.0114,−0.0016]). defense (−0.0043) and platoon (−0.0039) NOT_PROVEN at the + corrected bar. park_hits sample-blocked (n=405). **Zero theater.** +- Per-archetype all sample-blocked (BOMBER 252–294, GHOST 67–125). +- **Spec gaps found:** approach identities (SPRAY/DAMAGE-DEALER/COUNT-WORKER) + don't exist; `parkFactors` has no hits-specific factor (hits → run_base). +- **Grade rescale NOT run** — it was gated on factors proving, and one pooled + factor with a −0.0066 Brier gain is not a factor-informed distribution. + ## Session 91 (2026-08-04) — Hits calibrated point-in-time; parlay partially unblocked ✅ 4,275 tests / 339 suites green, build exit 0. Counter + frozen clusters byte-identical (`p_win` untouched). diff --git a/CLAUDE.md b/CLAUDE.md index 4645dba..17ecccc 100755 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1655,6 +1655,40 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section). correlate with date, so a time-split trains on wins and certifies on losses — the generator creating the exact leakage the split prevents. Interleave. +## The two-part factor gate (Session 92 — non-obvious) +- **`src/services/model/factorGate.js` asks a question correlation cannot.** A + factor must (a) MOVE the prediction off the player's base rate AND (b) improve + out-of-sample BRIER. Movement alone is **THEATER** — the grade LOOKS like it + read tonight's game while reading nothing, and neither a user nor a + correlation test can see it. arch-v1 was exactly this: moved 76% of rows by + 2.5pts, changed resolution by 0.0000, live for months. +- **BRIER, not correlation.** Correlation asks whether the ORDERING improved; + this asks whether the NUMBER got closer to what happened. For a graded + probability the number IS the product, and a factor can improve ordering while + degrading the number. +- **Baseline = the player's LEAVE-ONE-OUT base rate** — literally the "he's due" + null. A factor earns its place only by beating that. Leave-one-out matters: a + row must never contribute to its own baseline. +- **CUMULATIVE CORRECTION APPLIES TO THE INTERVAL ITSELF.** A plain 95% CI is + right for ONE test; at 50 cumulative tests ~2-3 of them exclude zero by chance. + The bootstrap interval now widens to 1 − 0.05/tests (currently **99.9%**). + Applying it flipped defense and platoon from "proves" to not-proven — a 95% CI + would have shipped two unproven factors. +- **NOT_PROVEN_AT_CORRECTED_BAR ≠ THEATER, and conflating them is the same error + as "insufficient evidence = evidence of absence".** THEATER is reserved for + brier_delta >= 0 (moves, reads nothing). A favourable point estimate whose + corrected CI spans zero is a real candidate held to a rising bar. +- **RESULT for hits (n=741):** `pitcher_contact_profile` **PROVES** + (Brier −0.0066, CI [−0.0114,−0.0016] at 99.9%). `defense` (−0.0043) and + `platoon` (−0.0039) are NOT_PROVEN at the corrected bar; `park_hits` is + sample-blocked (n=405). **Zero theater.** Per-archetype all sample-blocked + (BOMBER 252–294, GHOST 67–125). +- **Two spec gaps found:** approach identities (SPRAY / DAMAGE-DEALER / + COUNT-WORKER) **do not exist** in the registry — MLB batter archetypes are + BOMBER/GHOST/TORCH/BRUSH/DRIVER/FLEX/ALPHA/HYBRID/CATALYST. And + `parkFactors.STAT_BASE` maps `hits → run_base`, so there is **no hits-specific + park factor**: a park that turns outs into hits without scoring is invisible. + ## Active Skills - vyndr-voice (all user-facing output) - prop-analysis (grading methodology) diff --git a/scripts/prove-hit-factors.js b/scripts/prove-hit-factors.js new file mode 100644 index 0000000..8a159dd --- /dev/null +++ b/scripts/prove-hit-factors.js @@ -0,0 +1,207 @@ +#!/usr/bin/env node +'use strict'; + +/** + * prove-hit-factors — does the hit grade read tonight's game, or say "he's due"? + * + * Each factor is conditioned against the player's OWN base rate and put through + * the two-part gate: it must MOVE the prediction and the moved prediction must + * be MORE ACCURATE out-of-sample. Movement alone is THEATER — a grade that + * swings on park and platoon looks like it read the matchup, and a user cannot + * tell the difference from outside. + * + * The baseline is deliberately the honest null this order describes: the + * player's base rate, i.e. "he's due" with no reading of tonight at all. A + * factor earns its place only by beating that. + * + * SUPABASE_URL=... node scripts/prove-hit-factors.js + */ + +require('dotenv').config(); +const { createClient } = require('@supabase/supabase-js'); +const fg = require('../src/services/model/factorGate'); +const sk = require('../src/services/model/skillProjection'); +const tl = require('../src/services/model/testLedger'); +const mlb = require('../src/services/adapters/mlbStatsAdapter'); +const { knownNumber, knownRate } = require('../src/utils/known'); +const { nameKey } = require('../src/utils/playerName'); + +const SB_URL = process.env.SUPABASE_URL; +const SB_KEY = process.env.SUPABASE_SERVICE_ROLE_KEY || process.env.SUPABASE_SERVICE_KEY; +const PAGE = 1000; +const ARCHS = (process.env.HF_ARCHETYPES || 'BOMBER,GHOST,ALL').split(','); + +async function page(sb, table, select, apply) { + const out = []; + for (let from = 0; ; from += PAGE) { + const { data, error } = await apply(sb.from(table).select(select)).range(from, from + PAGE - 1); + if (error) throw error; + if (!data || data.length === 0) break; + out.push(...data); + if (data.length < PAGE) break; + } + return out; +} + +/** + * THE FACTORS. Each returns a MULTIPLIER on the base rate, or null when the + * input is absent — an absent factor must leave the baseline untouched rather + * than nudge it toward some default. + */ +const FACTORS = [ + { + key: 'defense', + needs: ['team_defense'], + mechanism: 'A ball in play becomes a hit or an out partly by who is standing behind the pitcher. Should matter most where contact stays in the park.', + // More outs converted above average -> fewer hits. + apply: (r) => 1 - Math.max(-0.12, Math.min(0.12, r.team_defense / 250)), + }, + { + key: 'pitcher_contact_profile', + needs: ['pitcher_hard_hit_allowed'], + mechanism: 'A contact-allowing arm concedes better contact than a bat-misser; hit probability should follow the quality of contact he permits.', + apply: (r) => 1 + Math.max(-0.15, Math.min(0.15, (r.pitcher_hard_hit_allowed - 0.389) * 1.2)), + }, + { + key: 'park_hits', + needs: ['park_factor'], + mechanism: 'Some parks turn outs into hits without producing runs — big outfields, high walls, deep gaps.', + apply: (r) => r.park_factor, + caveat: 'STAT_BASE maps hits -> run_base, so this is a RUN factor standing in for a HITS factor. A park that converts outs to hits without scoring is invisible to it.', + }, + { + key: 'platoon', + needs: ['platoon_edge'], + mechanism: 'Handedness advantage — a hitter facing the opposite hand sees the ball better and hits it harder.', + apply: (r) => (r.platoon_edge > 0 ? 1.06 : 0.96), + }, +]; + +async function main() { + if (!SB_URL || !SB_KEY) throw new Error('SUPABASE_URL / service key required'); + const sb = createClient(SB_URL, SB_KEY, { auth: { persistSession: false } }); + + const statcast = await page(sb, 'statcast_aggregates', '*', (q) => q.eq('sport', 'mlb')); + const batters = new Map(); const pitchersById = new Map(); + for (const r of statcast) { + const prof = sk.fromStatcastRow(r); + if (r.role === 'pitcher' && r.source_id != null) pitchersById.set(Number(r.source_id), prof); + if (r.role === 'batter' && r.player_key) batters.set(r.player_key, prof); + } + const defRows = await page(sb, 'team_defense', '*', (q) => q.eq('sport', 'mlb')); + const defByTeam = new Map(); + for (const d of defRows) defByTeam.set(d.team, d); + + const snaps = await page(sb, 'model_snapshots', 'player_key, game_date, archetype, stat', + (q) => q.eq('sport', 'mlb').eq('stat', 'hits').not('archetype', 'is', null)); + const archOf = new Map(); + for (const s of snaps) archOf.set(`${s.player_key}|${s.game_date}`, s.archetype); + + const led = await page(sb, 'ledger_entries', + 'id, player_key, player_name, line, side, outcome, game_date, p_win, quarantine_reason, env_park_base', + (q) => q.eq('sport', 'mlb').is('user_id', null).eq('stat', 'hits') + .in('outcome', ['hit', 'miss']).not('p_win', 'is', null)); + const clean = led.filter((r) => !(r.quarantine_reason || '').startsWith('nontakeable_book')); + + // Opponent faced, from each hitter's own game log. + const names = new Map(); + for (const r of clean) if (!names.has(r.player_key)) names.set(r.player_key, r.player_name); + const oppBy = new Map(); const startersBy = new Map(); + const dates = [...new Set(clean.map((r) => r.game_date))].sort(); + for (const d of dates) { + try { + const games = await mlb.getScheduleWithPitchers(d); + for (const g of games) { + if (!g.home || !g.away) continue; + if (g.home.probablePitcher) startersBy.set(`${d}|OPP:${g.home.team}`, g.home.probablePitcher.id); + if (g.away.probablePitcher) startersBy.set(`${d}|OPP:${g.away.team}`, g.away.probablePitcher.id); + } + } catch { /* absent slate */ } + } + for (const [key, name] of names) { + try { + const found = await mlb.searchPlayer(name); + if (!found || !found.id) continue; + const log = await mlb.getPlayerGameLog(found.id); + for (const g of log || []) if (g && g.date && g.opponent) oppBy.set(`${key}|${String(g.date).slice(0, 10)}`, g.opponent); + } catch { /* no log */ } + } + + // Per-player base rate — the honest null: "he's due", no reading of tonight. + const byPlayer = new Map(); + for (const r of clean) { + const cur = byPlayer.get(r.player_key) || { n: 0, w: 0 }; + cur.n += 1; cur.w += r.outcome === 'hit' ? 1 : 0; + byPlayer.set(r.player_key, cur); + } + + const rows = []; + for (const r of clean) { + const bat = batters.get(r.player_key); + const bp = byPlayer.get(r.player_key); + if (!bp || bp.n < 3) continue; + // Leave-one-out so a row never contributes to its own baseline. + const baseline = (bp.w - (r.outcome === 'hit' ? 1 : 0)) / (bp.n - 1); + const faced = oppBy.get(`${r.player_key}|${r.game_date}`) || null; + const nick = faced ? String(faced).split(' ').pop() : null; + const def = faced ? (defByTeam.get(faced) || defByTeam.get(nick)) : null; + const starterId = faced ? startersBy.get(`${r.game_date}|OPP:${faced}`) : null; + const pit = starterId != null ? pitchersById.get(Number(starterId)) : null; + rows.push({ + id: r.id, + archetype: archOf.get(`${r.player_key}|${r.game_date}`) || null, + won: r.outcome === 'hit' ? 1 : 0, + baseline, + team_defense: def ? knownNumber(def.oaa_sum) : null, + pitcher_hard_hit_allowed: pit ? knownRate(pit.hard_hit_pct) : null, + park_factor: knownNumber(r.env_park_base), + platoon_edge: (bat && pit && bat.bats && pit.throws) + ? (String(bat.bats)[0] !== String(pit.throws)[0] ? 1 : -1) : null, + }); + } + + // Cumulative Bonferroni across the programme lifetime. + const store = tl.supabaseStore(sb); + const mc = await tl.recordAndCount(store, FACTORS.flatMap((f) => + ARCHS.map((a) => ({ sport: 'mlb', stat: 'hits', archetype: a === 'ALL' ? null : a, interaction: `factor:${f.key}`, target: 'outcome' })))); + + const results = []; + for (const arch of ARCHS) { + const slot = arch === 'ALL' ? rows : rows.filter((r) => String(r.archetype || '').toUpperCase() === arch); + for (const f of FACTORS) { + const usable = slot.filter((r) => f.needs.every((k) => knownNumber(r[k]) !== null)); + const paired = usable.map((r) => { + const mult = f.apply(r); + const cond = mult === null ? null : Math.min(0.99, Math.max(0.01, r.baseline * mult)); + return { baseline: r.baseline, conditioned: cond, won: r.won }; + }); + const v = fg.adjudicate(paired, { + factor: f.key, archetype: arch, stat: 'hits', + cumulativeTests: mc.cumulative_tests, // native cumulative correction + }); + results.push({ + archetype: arch, factor: f.key, n: v.movement.n, + mean_abs_shift: v.movement.mean_abs_shift, + brier_delta: v.improvement ? v.improvement.brier_delta : null, + ci: v.improvement ? v.improvement.ci : null, + ci_level: v.improvement ? v.improvement.ci_level : null, + verdict: v.verdict, + reason: v.reason, + ...(f.caveat ? { input_caveat: f.caveat } : {}), + }); + } + } + + console.log(JSON.stringify({ + baseline: "each row scored against the player's OWN leave-one-out base rate — the honest 'he's due' null", + total_rows: rows.length, + cumulative_bonferroni: mc, + gate: 'a factor must MOVE the prediction AND improve out-of-sample Brier; movement alone is THEATER', + results, + proven: results.filter((r) => r.verdict === 'PROVES'), + theater: results.filter((r) => r.verdict === 'THEATER'), + }, null, 2)); + process.exit(0); +} + +main().catch((e) => { console.error(e); process.exit(1); }); diff --git a/src/services/model/factorGate.js b/src/services/model/factorGate.js new file mode 100644 index 0000000..00e09a6 --- /dev/null +++ b/src/services/model/factorGate.js @@ -0,0 +1,179 @@ +'use strict'; + +/** + * factorGate — DOES THIS FACTOR READ TONIGHT'S GAME, OR JUST MOVE THE NUMBER? + * + * Every gate in this codebase so far asks one question: is there a correlation. + * That is necessary and it is not sufficient, because it cannot tell apart the + * two ways a factor can look alive: + * + * PROVES the factor moves the prediction off the player's base rate AND the + * moved prediction is MORE ACCURATE out-of-sample. It is reading the + * game. + * THEATER the factor moves the prediction — sometimes a lot — and accuracy + * does not improve, or gets worse. The number looks responsive. It is + * responding to nothing. + * + * THEATER IS THE DANGEROUS ONE, and it is what a product ships by accident. A + * grade that swings on park and platoon LOOKS like it read tonight's matchup; + * a user cannot tell the difference from the outside, and neither can a + * correlation test. arch-v1 was exactly this: it moved 76% of rows by 2.5 points + * and changed resolution by 0.0000. It was live for months. + * + * So a factor must clear BOTH: + * + * (a) movement mean |Δp| against the base-rate baseline is real + * (b) improvement paired bootstrap on Brier score, CI excluding zero + * + * (a) alone is rejected BY NAME as THEATER rather than filed as "inconclusive", + * because the distinction is the whole point: an inconclusive factor might work + * with more data, and a theatrical one is actively misleading the user now. + * + * ── WHY BRIER AND NOT CORRELATION ──────────────────────────────────────── + * Correlation asks whether the ORDERING improved. This asks whether the NUMBER + * got closer to what happened, which is what a probability claims. A factor can + * improve ordering while degrading the number, and for a graded probability the + * number is the product. + */ + +const { knownNumber } = require('../../utils/known'); + +/** Minimum mean |Δp| for a factor to count as having moved anything at all. */ +const MIN_MOVEMENT = 0.01; +const MIN_N = 500; + +const brier = (ps, ys) => (ps.length + ? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null); + +function makeRnd(seed) { + let s = seed >>> 0; + return () => { s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; }; +} + +/** + * How far does the factor move the prediction off the baseline? + * + * Reported as the MEAN ABSOLUTE shift and its spread. A factor that shifts every + * prediction by the same amount is not reading the game either — it is a + * constant — so the spread matters as much as the mean. + */ +function movement(rows) { + const deltas = []; + for (const r of rows || []) { + const b = knownNumber(r && r.baseline); + const c = knownNumber(r && r.conditioned); + if (b === null || c === null) continue; // absent, never assumed equal + deltas.push(c - b); + } + if (deltas.length === 0) return { n: 0, mean_abs_shift: null, sd_shift: null, max_abs_shift: null }; + const meanAbs = deltas.reduce((s, d) => s + Math.abs(d), 0) / deltas.length; + const mean = deltas.reduce((s, d) => s + d, 0) / deltas.length; + const sd = deltas.length > 1 + ? Math.sqrt(deltas.reduce((s, d) => s + (d - mean) ** 2, 0) / (deltas.length - 1)) : 0; + return { + n: deltas.length, + mean_abs_shift: round4(meanAbs), + mean_signed_shift: round4(mean), + sd_shift: round4(sd), + max_abs_shift: round4(Math.max(...deltas.map(Math.abs))), + }; +} + +/** + * Did the moved prediction get CLOSER to what happened? + * + * Paired bootstrap on the Brier difference — the same rows score both models, so + * treating their errors as independent would overstate certainty. NEGATIVE delta + * means the conditioned model has lower Brier, i.e. it improved. + */ +function improvement(rows, iters = 4000, seed = 20260805, cumulativeTests = 1) { + const usable = (rows || []).filter((r) => + knownNumber(r.baseline) !== null && knownNumber(r.conditioned) !== null && knownNumber(r.won) !== null); + if (usable.length < 30) return null; + const rnd = makeRnd(seed); + const diffs = []; + for (let it = 0; it < iters; it += 1) { + const b = []; const c = []; const y = []; + for (let i = 0; i < usable.length; i += 1) { + const r = usable[Math.floor(rnd() * usable.length)]; + b.push(r.baseline); c.push(r.conditioned); y.push(r.won > 0 ? 1 : 0); + } + diffs.push(brier(c, y) - brier(b, y)); + } + diffs.sort((x, y) => x - y); + // CUMULATIVE CORRECTION APPLIED TO THE INTERVAL ITSELF. A plain 95% CI is the + // right bar for ONE test and far too lenient for a programme that has run + // dozens: at 50 cumulative tests, roughly two or three 95% intervals exclude + // zero by chance alone. So the interval widens to 1 − 0.05/tests, which is the + // same discipline the p-value gate applies, expressed as an interval. + const tests = Math.max(1, Math.round(knownNumber(cumulativeTests) ?? 1)); + const alpha = 0.05 / tests; + const q = (p) => round4(diffs[Math.floor(Math.min(diffs.length - 1, Math.max(0, p * (diffs.length - 1))))]); + const ci = [q(alpha / 2), q(1 - alpha / 2)]; + const ys = usable.map((r) => (r.won > 0 ? 1 : 0)); + return { + n: usable.length, + brier_baseline: round4(brier(usable.map((r) => r.baseline), ys)), + brier_conditioned: round4(brier(usable.map((r) => r.conditioned), ys)), + brier_delta: round4(brier(usable.map((r) => r.conditioned), ys) - brier(usable.map((r) => r.baseline), ys)), + ci: ci, + ci_level: round4(1 - alpha), + bonferroni_tests: tests, + improves: ci[1] < 0, // whole interval below zero = genuinely better + degrades: ci[0] > 0, + }; +} + +/** + * THE VERDICT. Both conditions, named outcomes. + * + * `cumulativeTests` is the programme-lifetime Bonferroni denominator; it tightens + * the improvement requirement the same way it does everywhere else. + */ +function adjudicate(rows, opts = {}) { + const minN = opts.minN ?? MIN_N; + const minMove = opts.minMovement ?? MIN_MOVEMENT; + const mv = movement(rows); + const imp = improvement(rows, opts.iters, opts.seed, opts.cumulativeTests); + + const base = { factor: opts.factor || null, archetype: opts.archetype || null, stat: opts.stat || 'hits', movement: mv, improvement: imp }; + + if (mv.n < minN) { + return { ...base, verdict: 'CANDIDATE_PENDING_SAMPLE', reason: `n ${mv.n} < ${minN}`, rows_needed: minN - mv.n }; + } + if (mv.mean_abs_shift === null || mv.mean_abs_shift < minMove) { + // It never moved the number, so it cannot be reading anything. + return { ...base, verdict: 'INERT', reason: `mean |shift| ${mv.mean_abs_shift} < ${minMove}` }; + } + if (!imp) return { ...base, verdict: 'CANDIDATE_PENDING_SAMPLE', reason: 'too few paired rows to bootstrap' }; + + // NOT PROVEN is not the same as THEATER, and collapsing them would repeat the + // error this codebase keeps having to correct: insufficient evidence is not + // evidence of absence. A factor whose POINT ESTIMATE improves accuracy but + // whose corrected interval still spans zero has not earned its place — and it + // is not decorative either. It is a real candidate held to a bar that rises + // with every hypothesis the programme tests. Saying so keeps THEATER meaning + // the one thing it must mean: moves the number, reads nothing. + if (!imp.improves && imp.brier_delta < 0) { + return { + ...base, + verdict: 'NOT_PROVEN_AT_CORRECTED_BAR', + reason: `moves ${mv.mean_abs_shift} and the point estimate improves Brier by ${-imp.brier_delta}, but the interval corrected for ${imp.bonferroni_tests} tests still spans zero (${JSON.stringify(imp.ci)} at level ${imp.ci_level})`, + note: 'a real candidate, not theatre — it improves on the point estimate and needs more sample, or a tighter bar than the programme can currently afford it', + }; + } + if (imp.improves) { + return { ...base, verdict: 'PROVES', reason: `moves ${mv.mean_abs_shift} and improves Brier by ${-imp.brier_delta} (CI ${JSON.stringify(imp.ci)} at level ${imp.ci_level}, corrected for ${imp.bonferroni_tests} tests)` }; + } + // MOVED BUT DID NOT IMPROVE. Named, not softened. + return { + ...base, + verdict: 'THEATER', + reason: `moves the prediction by ${mv.mean_abs_shift} on average (max ${mv.max_abs_shift}) while accuracy does NOT improve (Brier delta ${imp.brier_delta}, CI ${JSON.stringify(imp.ci)} at level ${imp.ci_level})`, + consequence: "wiring this would make the grade LOOK like it read tonight's game while reading nothing — the failure mode a user cannot detect from outside", + }; +} + +const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000); + +module.exports = { movement, improvement, adjudicate, MIN_MOVEMENT, MIN_N }; diff --git a/tests/unit/factorGate.test.js b/tests/unit/factorGate.test.js new file mode 100644 index 0000000..edd5afd --- /dev/null +++ b/tests/unit/factorGate.test.js @@ -0,0 +1,155 @@ +'use strict'; + +/** + * The two-part factor gate. + * + * The case these tests exist for is THEATER: a factor that moves the number + * convincingly and improves nothing. A correlation test cannot see it, and + * neither can a user — arch-v1 moved 76% of rows by 2.5 points, changed + * resolution by 0.0000, and stayed live for months. + */ + +const fg = require('../../src/services/model/factorGate'); + +/** n rows at a fixed baseline, with the conditioned value shifted by `shift(i)` + * and the outcome determined by `trueP(i)` — so a factor can be made genuinely + * informative or purely decorative on demand. */ +function rows(n, baseline, shift, trueP) { + const out = []; + for (let i = 0; i < n; i += 1) { + const p = typeof trueP === 'function' ? trueP(i) : trueP; + // Interleaved outcomes, never front-loaded — front-loading correlates the + // outcome with position and quietly rigs any split. + const won = Math.floor((i + 1) * p) > Math.floor(i * p) ? 1 : 0; + out.push({ + baseline, + conditioned: Math.min(0.99, Math.max(0.01, baseline + (typeof shift === 'function' ? shift(i) : shift))), + won, + }); + } + return out; +} + +describe('THEATER — moves the number, reads nothing', () => { + it('is REJECTED BY NAME, not filed as inconclusive', () => { + // Truth is a flat 0.5. The factor swings the prediction ±0.15 at random + // relative to the outcome, so it looks responsive and knows nothing. + const r = rows(800, 0.5, (i) => (i % 2 === 0 ? 0.15 : -0.15), 0.5); + const v = fg.adjudicate(r, { factor: 'decorative' }); + expect(v.verdict).toBe('THEATER'); + expect(v.movement.mean_abs_shift).toBeCloseTo(0.15, 2); + expect(v.improvement.improves).toBe(false); + expect(v.consequence).toMatch(/LOOK like it read/); + }); + + it('an unproven-but-favourable factor is NOT called theatre', () => { + // Point estimate improves, corrected interval spans zero. That is a real + // candidate held to a rising bar — collapsing it into THEATER would repeat + // the "insufficient evidence = evidence of absence" error. + const r = []; + for (let i = 0; i < 800; i += 1) { + const hot = i % 2 === 0; const p = hot ? 0.56 : 0.44; + const won = Math.floor((i + 1) * p) > Math.floor(i * p) ? 1 : 0; + r.push({ baseline: 0.5, conditioned: hot ? 0.53 : 0.47, won }); + } + const v = fg.adjudicate(r, { factor: 'weak-but-real', cumulativeTests: 200 }); + expect(['NOT_PROVEN_AT_CORRECTED_BAR', 'PROVES']).toContain(v.verdict); + if (v.verdict === 'NOT_PROVEN_AT_CORRECTED_BAR') { + expect(v.improvement.brier_delta).toBeLessThan(0); + expect(v.note).toMatch(/not theatre/); + } + }); + + it('a factor that moves a LOT is not thereby better — that is the trap', () => { + const big = fg.adjudicate(rows(800, 0.5, (i) => (i % 2 === 0 ? 0.3 : -0.3), 0.5), { factor: 'loud' }); + const small = fg.adjudicate(rows(800, 0.5, (i) => (i % 2 === 0 ? 0.02 : -0.02), 0.5), { factor: 'quiet' }); + expect(big.verdict).toBe('THEATER'); + expect(small.verdict).toBe('THEATER'); + expect(big.movement.mean_abs_shift).toBeGreaterThan(small.movement.mean_abs_shift * 5); + }); +}); + +describe('PROVES — moves the number AND gets closer to the truth', () => { + it('passes a factor that genuinely splits the population', () => { + // Truth alternates 0.8 / 0.2; the factor moves the prediction the right way. + const r = []; + for (let i = 0; i < 800; i += 1) { + const hot = i % 2 === 0; + const p = hot ? 0.8 : 0.2; + const won = Math.floor((i + 1) * p) > Math.floor(i * p) ? 1 : 0; + r.push({ baseline: 0.5, conditioned: hot ? 0.78 : 0.22, won }); + } + const v = fg.adjudicate(r, { factor: 'real' }); + expect(v.verdict).toBe('PROVES'); + expect(v.improvement.brier_delta).toBeLessThan(0); + expect(v.improvement.ci[1]).toBeLessThan(0); + }); +}); + +describe('INERT and PENDING are distinct from THEATER', () => { + it('a factor that never moves the number is INERT, not theatre', () => { + const v = fg.adjudicate(rows(800, 0.5, 0.0005, 0.5), { factor: 'flat' }); + expect(v.verdict).toBe('INERT'); + // Nothing was claimed, so nothing is misleading — a different problem. + }); + + it('thin sample is CANDIDATE_PENDING_SAMPLE with the rows still needed', () => { + const v = fg.adjudicate(rows(120, 0.5, 0.1, 0.5), { factor: 'thin' }); + expect(v.verdict).toBe('CANDIDATE_PENDING_SAMPLE'); + expect(v.rows_needed).toBe(380); + // Crucially NOT 'THEATER' — it might work; we simply cannot tell yet. + }); +}); + +describe('the measurements themselves', () => { + it('movement reports spread, because a constant shift reads nothing either', () => { + const constant = fg.movement(rows(200, 0.5, 0.1, 0.5)); + const varied = fg.movement(rows(200, 0.5, (i) => (i % 2 ? 0.1 : -0.1), 0.5)); + expect(constant.sd_shift).toBeCloseTo(0, 6); + expect(varied.sd_shift).toBeGreaterThan(0.09); + }); + + it('an unreadable side is DROPPED, never treated as no-change', () => { + const m = fg.movement([ + { baseline: 0.5, conditioned: 0.6 }, + { baseline: null, conditioned: 0.9 }, + { baseline: 0.5, conditioned: null }, + ]); + expect(m.n).toBe(1); + }); + + it('improvement uses a PAIRED bootstrap — same rows score both models', () => { + const r = rows(400, 0.5, 0.0, 0.5); + const imp = fg.improvement(r); + // Identical models must show no difference and a CI spanning zero. + expect(imp.brier_delta).toBeCloseTo(0, 6); + expect(imp.improves).toBe(false); + expect(imp.degrades).toBe(false); + }); + + it('the interval WIDENS with the cumulative test count — the bar rises', () => { + const r = []; + for (let i = 0; i < 800; i += 1) { + const hot = i % 2 === 0; const p = hot ? 0.8 : 0.2; + const won = Math.floor((i + 1) * p) > Math.floor(i * p) ? 1 : 0; + r.push({ baseline: 0.5, conditioned: hot ? 0.78 : 0.22, won }); + } + const one = fg.improvement(r, 3000, 1, 1); + const fifty = fg.improvement(r, 3000, 1, 50); + expect(fifty.ci_level).toBeGreaterThan(one.ci_level); + // A wider interval can only ever make PROVES harder, never easier. + expect(fifty.ci[1]).toBeGreaterThanOrEqual(one.ci[1]); + }); + + it('a factor that makes the number WORSE is flagged degrading', () => { + const r = []; + for (let i = 0; i < 600; i += 1) { + const p = 0.8; + const won = Math.floor((i + 1) * p) > Math.floor(i * p) ? 1 : 0; + r.push({ baseline: 0.8, conditioned: 0.2, won }); // confidently backwards + } + const imp = fg.improvement(r); + expect(imp.degrades).toBe(true); + expect(fg.adjudicate(r, { factor: 'backwards' }).verdict).toBe('THEATER'); + }); +});