Layer 3 Step 3: Tier-1 mappings live; Tier-2 nomination harness

PHASE 0 GATE — historical out-of-sample testing is NOT available, and the reason
matters. statcast_aggregates is overwritten nightly by design (Layer 1 is a full
re-pull upsert), so it holds season-TO-DATE numbers with no point-in-time
history. Classifying a player for a 15 July game using today's aggregate would
feed the model games from 15-21 July — look-ahead leakage, and the resulting
"out-of-sample" verdict would be worthless. The harness therefore reads the
archetype vector RETAINED at grade time (Session 70's instrument) and runs
FORWARD-ACCRUAL, not historical. Reported rather than worked around.

CANONICAL NAMES ASSERTED. Every mapping references the axis keys the classifier
actually emits, and a test walks both maps against BATTER_AXES / PITCHER_AXES.
A key that does not exist would look active and never fire — a mapping that
appears wired while silently doing nothing is the exact failure this guards.

TIER 1 IS LIVE, tautological and directional: PUNCHOUT/WHIFF raises strikeouts;
SINKER/SEAM lowers home runs allowed and FLY BALL/ELEVATOR raises them (a ball
on the ground cannot leave the park); SURGEON ARM/PINPOINT lowers walks allowed;
SLUGGER/BOMBER raises total bases and home runs; TECHNICIAN/SURGEON raises hits
and lowers strikeouts; GRINDER/SNIPER raises walks. Each adjusts only its named
stat, mirrors exactly on the under side, and leaves an average player untouched.

SPEED IS HONESTLY ABSENT. BURNER/stolen-bases has no axis to key on — SB is a
statsapi field that never reached the aggregate store, so Layer 2 shelved it.
The mapping is an empty object rather than an invented one.

THE TIER-2 HARNESS tests MARGINAL CONTRIBUTION, not correlation. A ground-ball
arm obviously correlates with fewer home runs; the question is whether the
archetype explains the PROJECTION'S RESIDUAL (outcome minus p_win). If the
projection already knows it, the residual carries no signal and the mapping is
rejected as redundant — that hurdle is what catches double-counting. The split
is by DATE, never random, because rows from one game share a pitcher, a park and
a lineup and would leak across a random split. Direction is validated from the
held-out data and a contradicted sign is REJECTED, never silently flipped to
whatever the data says, which would be fitting noise.

LIFECYCLE ENCODED — nominated, live, claimed. A mapping that survives runs live
and is measured; only the quantified public claim waits for the ledger. Nothing
sits dark.

One fixture bug worth recording: my first synthetic generator aliased the
carrier selector against the outcome draw and manufactured a 0.038 effect where
the generator had put zero. The harness rejected it correctly — it just gave the
sign reason instead of the redundancy reason, which is how I found it. The draw
now uses a coprime modulus.

Real candidate run end to end, GROUND-BALL to hits-allowed: INSUFFICIENT, 0 of
200 settled rows, because no settled row carries p_win yet (Session 70's
instrument starts recording at the next new lock). That is the correct verdict
and the expected one.

Tests 3654 passed / 296 suites, web build exit 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCNgGSt5qvcLxaeQqa7Zpj

src/services/mappingHarness.js

@@ -0,0 +1,188 @@
+'use strict';
+
+/**
+ * TIER-2 MAPPING NOMINATION HARNESS (Layer 3, Step 3).
+ *
+ * Decides whether a candidate archetype→stat mapping earns the right to run
+ * live. Three hurdles, all of which a plausible-but-worthless mapping fails:
+ *
+ *   1. MARGINAL CONTRIBUTION, not raw correlation. The question is never "does
+ *      this archetype correlate with this stat" — a ground-ball arm obviously
+ *      correlates with fewer home runs. The question is whether it adds signal
+ *      BEYOND what the projection already captures. We score the projection's
+ *      RESIDUAL (outcome − p_win) against the archetype: if the projection
+ *      already knows it, the residual carries no signal and the mapping is
+ *      REDUNDANT → rejected. This is the hurdle that catches double-counting.
+ *
+ *   2. OUT-OF-SAMPLE. Fit the effect on one period, measure it on a held-out
+ *      period the fit never touched. A mapping that only works in-sample is
+ *      overfit → rejected. The split is by DATE, never random: rows from one
+ *      game leak into each other, so a random split would leak.
+ *
+ *   3. DIRECTION VALIDATED FROM DATA. A correct pairing with a flipped sign is
+ *      worse than no mapping at all. If the held-out effect contradicts the
+ *      candidate's assumed sign, it is rejected — never silently flipped to
+ *      whatever the data says, because that is fitting noise.
+ *
+ * Under all three: MIN_ROWS. Below it the verdict is INSUFFICIENT, never a
+ * nomination — the same honest-absent rule the rest of the system runs on.
+ */
+
+const MIN_ROWS = Number(process.env.MAPPING_MIN_ROWS) || 200;
+const MIN_HELD_OUT = Number(process.env.MAPPING_MIN_HELD_OUT) || 60;
+/** The held-out residual effect must clear this to be a real lean, not drift. */
+const MIN_EFFECT = 0.02;
+
+const num = (v) => {
+  if (v == null || v === '') return null;
+  const n = typeof v === 'number' ? v : Number(v);
+  return Number.isFinite(n) ? n : null;
+};
+
+/**
+ * Split by DATE, never randomly. Rows from the same game share a pitcher, a
+ * park and a lineup; a random split puts correlated rows on both sides and
+ * leaks. `fitFrac` of the distinct dates (earliest first) fit; the rest test.
+ */
+function splitByDate(rows, fitFrac = 0.6) {
+  const dates = [...new Set(rows.map((r) => String(r.game_date)))].sort();
+  if (dates.length < 2) return { fit: [], test: [], dates, split: null };
+  const cut = Math.max(1, Math.floor(dates.length * fitFrac));
+  const fitDates = new Set(dates.slice(0, cut));
+  return {
+    fit: rows.filter((r) => fitDates.has(String(r.game_date))),
+    test: rows.filter((r) => !fitDates.has(String(r.game_date))),
+    dates,
+    split: dates[cut - 1],
+  };
+}
+
+/**
+ * meanResidual(rows) — average of (outcome − p_win).
+ * Positive = the projection UNDER-estimated these props; negative = over. This
+ * is the projection's own error, which is exactly what a new signal has to
+ * explain to be worth anything.
+ */
+function meanResidual(rows) {
+  const vals = rows
+    .map((r) => {
+      const p = num(r.p_win);
+      const y = r.outcome === 'hit' ? 1 : r.outcome === 'miss' ? 0 : null;
+      return p == null || y == null ? null : y - p;
+    })
+    .filter((v) => v != null);
+  if (!vals.length) return null;
+  return vals.reduce((a, b) => a + b, 0) / vals.length;
+}
+
+/**
+ * evaluateMapping(rows, candidate) — the verdict.
+ *
+ * rows: settled records carrying { p_win, outcome, game_date, stat, archetype_vector }
+ * candidate: { axis, stat, sign, role }  sign = +1 (raises the stat) / -1 (lowers)
+ *
+ * Returns { verdict, reason, ... } where verdict is one of:
+ *   NOMINATE      — marginal, out-of-sample, direction confirmed → runs live
+ *   REJECT        — redundant, or the held-out effect contradicts the sign
+ *   INSUFFICIENT  — not enough rows to say anything (honest-absent)
+ */
+function evaluateMapping(rows, candidate = {}) {
+  const { axis, stat, sign = 1 } = candidate;
+  const base = {
+    axis, stat, sign, min_rows: MIN_ROWS, evaluated_at: null,
+  };
+
+  const relevant = (rows || []).filter((r) => {
+    if (String(r.stat || '').toLowerCase() !== String(stat || '').toLowerCase()) return false;
+    if (num(r.p_win) == null) return false;
+    return r.outcome === 'hit' || r.outcome === 'miss';
+  });
+
+  if (relevant.length < MIN_ROWS) {
+    return { ...base, verdict: 'INSUFFICIENT', rows: relevant.length,
+      reason: `need ${MIN_ROWS} settled rows for ${stat}, have ${relevant.length}` };
+  }
+
+  // Split into HAS-the-archetype vs does not, using the vector retained at
+  // grade time — never a re-classification from today's stats, which would be
+  // look-ahead leakage.
+  const hasAxis = (r) => {
+    const v = r.archetype_vector;
+    if (!v || typeof v !== 'object') return false;
+    const vec = v.vector || v;
+    return Boolean(vec && vec[axis]);
+  };
+
+  const { fit, test, split, dates } = splitByDate(relevant);
+  if (test.length < MIN_HELD_OUT) {
+    return { ...base, verdict: 'INSUFFICIENT', rows: relevant.length, held_out: test.length,
+      reason: `held-out period too small (${test.length} < ${MIN_HELD_OUT})` };
+  }
+
+  const fitWith = fit.filter(hasAxis);
+  const fitWithout = fit.filter((r) => !hasAxis(r));
+  const testWith = test.filter(hasAxis);
+  const testWithout = test.filter((r) => !hasAxis(r));
+
+  if (fitWith.length < 20 || testWith.length < 20) {
+    return { ...base, verdict: 'INSUFFICIENT', rows: relevant.length,
+      with_axis_fit: fitWith.length, with_axis_test: testWith.length,
+      reason: 'too few rows carrying the archetype to separate its effect' };
+  }
+
+  // The EFFECT is the difference in the projection's residual between props
+  // whose player carries the archetype and props whose player does not. If the
+  // projection already accounts for the trait, both residuals sit at the same
+  // place and the effect is ~0 → redundant.
+  const fitEffect = meanResidual(fitWith) - meanResidual(fitWithout);
+  const testEffect = meanResidual(testWith) - meanResidual(testWithout);
+
+  const expectedSign = Math.sign(sign) || 1;
+  const directionHolds = Math.sign(testEffect) === expectedSign;
+  const bigEnough = Math.abs(testEffect) >= MIN_EFFECT;
+  // In-sample effect present but held-out effect gone = overfit.
+  const survivesOOS = bigEnough && directionHolds;
+
+  const out = {
+    ...base,
+    rows: relevant.length,
+    split_date: split, dates: dates.length,
+    fit_rows: fit.length, test_rows: test.length,
+    with_axis_fit: fitWith.length, with_axis_test: testWith.length,
+    fit_effect: Math.round(fitEffect * 1000) / 1000,
+    test_effect: Math.round(testEffect * 1000) / 1000,
+    direction_holds: directionHolds,
+  };
+
+  if (!bigEnough) {
+    return { ...out, verdict: 'REJECT',
+      reason: `held-out effect ${out.test_effect} is below ${MIN_EFFECT} — the projection already captures it (redundant)` };
+  }
+  if (!directionHolds) {
+    // Never silently flip: a sign that only reverses out-of-sample is noise.
+    return { ...out, verdict: 'REJECT',
+      reason: `held-out effect ${out.test_effect} contradicts the candidate sign ${expectedSign}` };
+  }
+  return { ...out, verdict: 'NOMINATE', survivesOOS,
+    reason: `adds ${out.test_effect} residual signal out-of-sample with the expected sign` };
+}
+
+/**
+ * LIFECYCLE — nothing sits dark.
+ *   nominated : survived the harness → RUNS LIVE as a challenger mapping,
+ *               adjusting real projections and measured by the instrument.
+ *   live      : running and accruing settled volume.
+ *   claimed   : ONLY after the live ledger shows it beats the market. This is
+ *               the only stage that is withheld, and only the CLAIM is.
+ */
+const LIFECYCLE = Object.freeze(['nominated', 'live', 'claimed']);
+
+module.exports = {
+  evaluateMapping,
+  splitByDate,
+  meanResidual,
+  LIFECYCLE,
+  MIN_ROWS,
+  MIN_HELD_OUT,
+  MIN_EFFECT,
+};