Layer 3 Step 3: Tier-1 mappings live; Tier-2 nomination harness

PHASE 0 GATE — historical out-of-sample testing is NOT available, and the reason matters. statcast_aggregates is overwritten nightly by design (Layer 1 is a full re-pull upsert), so it holds season-TO-DATE numbers with no point-in-time history. Classifying a player for a 15 July game using today's aggregate would feed the model games from 15-21 July — look-ahead leakage, and the resulting "out-of-sample" verdict would be worthless. The harness therefore reads the archetype vector RETAINED at grade time (Session 70's instrument) and runs FORWARD-ACCRUAL, not historical. Reported rather than worked around. CANONICAL NAMES ASSERTED. Every mapping references the axis keys the classifier actually emits, and a test walks both maps against BATTER_AXES / PITCHER_AXES. A key that does not exist would look active and never fire — a mapping that appears wired while silently doing nothing is the exact failure this guards. TIER 1 IS LIVE, tautological and directional: PUNCHOUT/WHIFF raises strikeouts; SINKER/SEAM lowers home runs allowed and FLY BALL/ELEVATOR raises them (a ball on the ground cannot leave the park); SURGEON ARM/PINPOINT lowers walks allowed; SLUGGER/BOMBER raises total bases and home runs; TECHNICIAN/SURGEON raises hits and lowers strikeouts; GRINDER/SNIPER raises walks. Each adjusts only its named stat, mirrors exactly on the under side, and leaves an average player untouched. SPEED IS HONESTLY ABSENT. BURNER/stolen-bases has no axis to key on — SB is a statsapi field that never reached the aggregate store, so Layer 2 shelved it. The mapping is an empty object rather than an invented one. THE TIER-2 HARNESS tests MARGINAL CONTRIBUTION, not correlation. A ground-ball arm obviously correlates with fewer home runs; the question is whether the archetype explains the PROJECTION'S RESIDUAL (outcome minus p_win). If the projection already knows it, the residual carries no signal and the mapping is rejected as redundant — that hurdle is what catches double-counting. The split is by DATE, never random, because rows from one game share a pitcher, a park and a lineup and would leak across a random split. Direction is validated from the held-out data and a contradicted sign is REJECTED, never silently flipped to whatever the data says, which would be fitting noise. LIFECYCLE ENCODED — nominated, live, claimed. A mapping that survives runs live and is measured; only the quantified public claim waits for the ledger. Nothing sits dark. One fixture bug worth recording: my first synthetic generator aliased the carrier selector against the outcome draw and manufactured a 0.038 effect where the generator had put zero. The harness rejected it correctly — it just gave the sign reason instead of the redundancy reason, which is how I found it. The draw now uses a coprime modulus. Real candidate run end to end, GROUND-BALL to hits-allowed: INSUFFICIENT, 0 of 200 settled rows, because no settled row carries p_win yet (Session 70's instrument starts recording at the next new lock). That is the correct verdict and the expected one. Tests 3654 passed / 296 suites, web build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCNgGSt5qvcLxaeQqa7Zpj
2026-07-21 00:30:16 -04:00
parent 56fee267c9
commit 927e867a23
4 changed files with 374 additions and 1 deletions
@@ -59,10 +59,23 @@ const BATTER_MAP = Object.freeze({
  stolen_bases: {},
 });

+/**
+ * TIER 1 — TAUTOLOGICAL. The archetype IS the stat tendency, so these are safe
+ * in DIRECTION without an empirical test and run live from day one. They are
+ * still MEASURED like everything else; only a quantified accuracy claim waits.
+ * Keys below are the CANONICAL axis keys the classifier emits — a key that does
+ * not exist would be a silent no-op, so `challengerMappings.test.js` asserts
+ * every one against BATTER_AXES / PITCHER_AXES.
+ */
 const PITCHER_MAP = Object.freeze({
  strikeouts:       { strikeout: +1, chase: +1, velocity: +1, contact_allowed: -1 },
  pitcher_strikeouts: { strikeout: +1, chase: +1, velocity: +1, contact_allowed: -1 },
  hits_allowed:     { strikeout: -1, contact_allowed: +1, ground_ball: -1 },
+  // Tier 1: a ground-ball arm allows fewer home runs; a fly-ball arm more.
+  // The most tautological pair in the set — a ball on the ground cannot leave
+  // the park.
+  home_runs_allowed: { ground_ball: -1, fly_ball: +1, contact_allowed: +1 },
+  home_runs:         { ground_ball: -1, fly_ball: +1, contact_allowed: +1 },
  earned_runs:      { contact_allowed: +1, wild: +1, strikeout: -1 },
  outs_recorded:    { control: +1, ground_ball: +1, wild: -1 },
  innings_pitched:  { control: +1, ground_ball: +1, wild: -1 },
@@ -0,0 +1,188 @@
+'use strict';
+
+/**
+ * TIER-2 MAPPING NOMINATION HARNESS (Layer 3, Step 3).
+ *
+ * Decides whether a candidate archetype→stat mapping earns the right to run
+ * live. Three hurdles, all of which a plausible-but-worthless mapping fails:
+ *
+ *   1. MARGINAL CONTRIBUTION, not raw correlation. The question is never "does
+ *      this archetype correlate with this stat" — a ground-ball arm obviously
+ *      correlates with fewer home runs. The question is whether it adds signal
+ *      BEYOND what the projection already captures. We score the projection's
+ *      RESIDUAL (outcome − p_win) against the archetype: if the projection
+ *      already knows it, the residual carries no signal and the mapping is
+ *      REDUNDANT → rejected. This is the hurdle that catches double-counting.
+ *
+ *   2. OUT-OF-SAMPLE. Fit the effect on one period, measure it on a held-out
+ *      period the fit never touched. A mapping that only works in-sample is
+ *      overfit → rejected. The split is by DATE, never random: rows from one
+ *      game leak into each other, so a random split would leak.
+ *
+ *   3. DIRECTION VALIDATED FROM DATA. A correct pairing with a flipped sign is
+ *      worse than no mapping at all. If the held-out effect contradicts the
+ *      candidate's assumed sign, it is rejected — never silently flipped to
+ *      whatever the data says, because that is fitting noise.
+ *
+ * Under all three: MIN_ROWS. Below it the verdict is INSUFFICIENT, never a
+ * nomination — the same honest-absent rule the rest of the system runs on.
+ */
+
+const MIN_ROWS = Number(process.env.MAPPING_MIN_ROWS) || 200;
+const MIN_HELD_OUT = Number(process.env.MAPPING_MIN_HELD_OUT) || 60;
+/** The held-out residual effect must clear this to be a real lean, not drift. */
+const MIN_EFFECT = 0.02;
+
+const num = (v) => {
+  if (v == null || v === '') return null;
+  const n = typeof v === 'number' ? v : Number(v);
+  return Number.isFinite(n) ? n : null;
+};
+
+/**
+ * Split by DATE, never randomly. Rows from the same game share a pitcher, a
+ * park and a lineup; a random split puts correlated rows on both sides and
+ * leaks. `fitFrac` of the distinct dates (earliest first) fit; the rest test.
+ */
+function splitByDate(rows, fitFrac = 0.6) {
+  const dates = [...new Set(rows.map((r) => String(r.game_date)))].sort();
+  if (dates.length < 2) return { fit: [], test: [], dates, split: null };
+  const cut = Math.max(1, Math.floor(dates.length * fitFrac));
+  const fitDates = new Set(dates.slice(0, cut));
+  return {
+    fit: rows.filter((r) => fitDates.has(String(r.game_date))),
+    test: rows.filter((r) => !fitDates.has(String(r.game_date))),
+    dates,
+    split: dates[cut - 1],
+  };
+}
+
+/**
+ * meanResidual(rows) — average of (outcome − p_win).
+ * Positive = the projection UNDER-estimated these props; negative = over. This
+ * is the projection's own error, which is exactly what a new signal has to
+ * explain to be worth anything.
+ */
+function meanResidual(rows) {
+  const vals = rows
+    .map((r) => {
+      const p = num(r.p_win);
+      const y = r.outcome === 'hit' ? 1 : r.outcome === 'miss' ? 0 : null;
+      return p == null || y == null ? null : y - p;
+    })
+    .filter((v) => v != null);
+  if (!vals.length) return null;
+  return vals.reduce((a, b) => a + b, 0) / vals.length;
+}
+
+/**
+ * evaluateMapping(rows, candidate) — the verdict.
+ *
+ * rows: settled records carrying { p_win, outcome, game_date, stat, archetype_vector }
+ * candidate: { axis, stat, sign, role }  sign = +1 (raises the stat) / -1 (lowers)
+ *
+ * Returns { verdict, reason, ... } where verdict is one of:
+ *   NOMINATE      — marginal, out-of-sample, direction confirmed → runs live
+ *   REJECT        — redundant, or the held-out effect contradicts the sign
+ *   INSUFFICIENT  — not enough rows to say anything (honest-absent)
+ */
+function evaluateMapping(rows, candidate = {}) {
+  const { axis, stat, sign = 1 } = candidate;
+  const base = {
+    axis, stat, sign, min_rows: MIN_ROWS, evaluated_at: null,
+  };
+
+  const relevant = (rows || []).filter((r) => {
+    if (String(r.stat || '').toLowerCase() !== String(stat || '').toLowerCase()) return false;
+    if (num(r.p_win) == null) return false;
+    return r.outcome === 'hit' || r.outcome === 'miss';
+  });
+
+  if (relevant.length < MIN_ROWS) {
+    return { ...base, verdict: 'INSUFFICIENT', rows: relevant.length,
+      reason: `need ${MIN_ROWS} settled rows for ${stat}, have ${relevant.length}` };
+  }
+
+  // Split into HAS-the-archetype vs does not, using the vector retained at
+  // grade time — never a re-classification from today's stats, which would be
+  // look-ahead leakage.
+  const hasAxis = (r) => {
+    const v = r.archetype_vector;
+    if (!v || typeof v !== 'object') return false;
+    const vec = v.vector || v;
+    return Boolean(vec && vec[axis]);
+  };
+
+  const { fit, test, split, dates } = splitByDate(relevant);
+  if (test.length < MIN_HELD_OUT) {
+    return { ...base, verdict: 'INSUFFICIENT', rows: relevant.length, held_out: test.length,
+      reason: `held-out period too small (${test.length} < ${MIN_HELD_OUT})` };
+  }
+
+  const fitWith = fit.filter(hasAxis);
+  const fitWithout = fit.filter((r) => !hasAxis(r));
+  const testWith = test.filter(hasAxis);
+  const testWithout = test.filter((r) => !hasAxis(r));
+
+  if (fitWith.length < 20 || testWith.length < 20) {
+    return { ...base, verdict: 'INSUFFICIENT', rows: relevant.length,
+      with_axis_fit: fitWith.length, with_axis_test: testWith.length,
+      reason: 'too few rows carrying the archetype to separate its effect' };
+  }
+
+  // The EFFECT is the difference in the projection's residual between props
+  // whose player carries the archetype and props whose player does not. If the
+  // projection already accounts for the trait, both residuals sit at the same
+  // place and the effect is ~0 → redundant.
+  const fitEffect = meanResidual(fitWith) - meanResidual(fitWithout);
+  const testEffect = meanResidual(testWith) - meanResidual(testWithout);
+
+  const expectedSign = Math.sign(sign) || 1;
+  const directionHolds = Math.sign(testEffect) === expectedSign;
+  const bigEnough = Math.abs(testEffect) >= MIN_EFFECT;
+  // In-sample effect present but held-out effect gone = overfit.
+  const survivesOOS = bigEnough && directionHolds;
+
+  const out = {
+    ...base,
+    rows: relevant.length,
+    split_date: split, dates: dates.length,
+    fit_rows: fit.length, test_rows: test.length,
+    with_axis_fit: fitWith.length, with_axis_test: testWith.length,
+    fit_effect: Math.round(fitEffect * 1000) / 1000,
+    test_effect: Math.round(testEffect * 1000) / 1000,
+    direction_holds: directionHolds,
+  };
+
+  if (!bigEnough) {
+    return { ...out, verdict: 'REJECT',
+      reason: `held-out effect ${out.test_effect} is below ${MIN_EFFECT} — the projection already captures it (redundant)` };
+  }
+  if (!directionHolds) {
+    // Never silently flip: a sign that only reverses out-of-sample is noise.
+    return { ...out, verdict: 'REJECT',
+      reason: `held-out effect ${out.test_effect} contradicts the candidate sign ${expectedSign}` };
+  }
+  return { ...out, verdict: 'NOMINATE', survivesOOS,
+    reason: `adds ${out.test_effect} residual signal out-of-sample with the expected sign` };
+}
+
+/**
+ * LIFECYCLE — nothing sits dark.
+ *   nominated : survived the harness → RUNS LIVE as a challenger mapping,
+ *               adjusting real projections and measured by the instrument.
+ *   live      : running and accruing settled volume.
+ *   claimed   : ONLY after the live ledger shows it beats the market. This is
+ *               the only stage that is withheld, and only the CLAIM is.
+ */
+const LIFECYCLE = Object.freeze(['nominated', 'live', 'claimed']);
+
+module.exports = {
+  evaluateMapping,
+  splitByDate,
+  meanResidual,
+  LIFECYCLE,
+  MIN_ROWS,
+  MIN_HELD_OUT,
+  MIN_EFFECT,
+};