Per-archetype grade bands: built, gated, and the rescale blocked twice

The premise does not hold. proven-status.js run fresh: PROVEN_SET is EMPTY, no archetype x stat reaches the gate. pitcher_contact_profile has a CI upper bound of exactly 0.0000 and platoon_severity is held on 4.5%-contaminated splits, so the proven set is one factor, pooled, not three archetype-conditioned ones. The specific pattern the order names -- defense strong for GHOST/BRUSH, null for BOMBER -- is the one I measured running the OTHER WAY yesterday, both noise-dominated. But the second blocker is new and matters more, because it would stop the rescale even if the factors had proved: the grade does not separate within any archetype. Every archetype collapses to ONE band at the corrected bar, because bands merge when their intervals overlap and publishing two letters we cannot tell apart is a distinction we have not measured. Uncorrected, so the ranking is visible rather than hidden by the bar, this INVERTS the order's design. The order gives contact types the factor-rich treatment and power types honest base-rate, reasoning that single-game hits are variance for a power profile. Measured: BOMBER n=466 corr(p_win,outcome) +0.207 quintiles 0.75 0.62 0.60 0.48 0.48 GHOST n=192 corr(p_win,outcome) -0.007 quintiles 0.47 0.63 0.74 0.58 0.45 BOMBER is the one archetype the model ranks, and it splits into a real A 0.660 / B 0.481 at 95%. GHOST is flat, and non-monotone -- its most confident reads hit 47% while its middle reads hit 74%. Shipping as specified would have given the factor-rich treatment to the archetype the model reads worst and left base-rate on the one it reads best. That is mechanically sensible in hindsight: a power hitter's hit tracks whether he can damage the arm, a contact hitter's depends on balls finding holes. BOMBER's split does not survive the cumulative correction at 106 tests. Exposing it by loosening the correction is the curve-to-make-A's the order forbids, so it stays one band. BUILT: gradeBands.js -- lift against the archetype's OWN base rate (the same 62% is lift for a 45% profile and a deficit for a 68% one), indistinguishable neighbours merged, thin bands PROVISIONAL not dropped, Wilson intervals widened by the cumulative correction. The two-bar rule is structural: proven-alone, calibrated-alone and neither all return base_rate with the reason stated, so with nothing proven no factor-informed band can be produced at all. reasoning() is built and tested but NOT wired to the card -- there is no per-archetype band being served, so attaching the copy now would ship product language for a rescale that does not exist. NOT BUILT: the specified power-type reason "the matchup edge is in total_bases". total_bases is recorded INCONCLUSIVE (+0.0038, CI [-0.068,+0.075]). Wiring it would assert an edge measured as indistinguishable from zero -- the exact fabricated-reason failure this module exists to prevent. BOMBER x hits is 29 rows short of the gate and is the archetype the model actually reads. That is the first slot to test, not GHOST. Counter and frozen clusters byte-identical. No letter was moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
2026-08-06 01:03:10 -04:00
parent 6b17f79367
commit 3081c92e00
5 changed files with 615 additions and 1 deletions
@@ -0,0 +1,235 @@
+'use strict';
+
+/**
+ * gradeBands — WHAT DOES A LETTER MEAN, PER ARCHETYPE, AND WHO SAYS SO?
+ *
+ * A grade is the product. So a letter has to be backed by a realized rate the
+ * ledger can stand behind — not by a threshold someone chose. This builds bands
+ * from settled outcomes, per archetype, and refuses to publish any band the
+ * evidence cannot support.
+ *
+ * ── LIFT, NOT RAW RATE ────────────────────────────────────────────────────
+ * An A must beat the archetype's OWN naive base rate. A power hitter and a
+ * contact hitter have different base rates for a hit, so the same 62% realized
+ * rate is a strong read for one and slightly below water for the other. Bands
+ * are therefore drawn where the realized rate SEPARATES from that archetype's
+ * base rate, and a band whose interval still contains the base rate has shown no
+ * lift — whatever its raw number looks like.
+ *
+ * ── THE TWO-BAR RULE IS STRUCTURAL, NOT A HABIT ───────────────────────────
+ * A band may be described as FACTOR-INFORMED only when that archetype's factors
+ * are both PROVEN and CALIBRATED. There is no argument that overrides it: the
+ * caller passes evidence, and without it every band comes back `basis:
+ * 'base_rate'` and says so. A test asserts that with nothing proven — which is
+ * the state today — no factor-informed band can be produced at all.
+ *
+ * This is the same shape as featureRegistry.liveFeatures(): the honest state is
+ * the DEFAULT, and the richer claim has to be earned past a gate, so it cannot
+ * be reached by forgetting.
+ *
+ * ── WHY A BAND CAN BE REFUSED ─────────────────────────────────────────────
+ * Thin bands are labelled PROVISIONAL rather than dropped, because "we are still
+ * counting" and "there is nothing here" are different claims. But a band whose
+ * corrected interval spans the base rate does NOT get a lift letter — it is
+ * reported as the base-rate read it is.
+ */
+
+const { knownNumber } = require('../../utils/known');
+
+/** Below this a band is published but flagged — counting, not concluded. */
+const PROVISIONAL_N = 30;
+/** Below this a band is not published at all; there is nothing to stand behind. */
+const MIN_BAND_N = 12;
+/** Letters, richest first. Assigned by measured lift, never by curve. */
+const LETTERS = ['A', 'B', 'C', 'D', 'F'];
+
+/**
+ * Wilson score interval, widened by the cumulative correction.
+ *
+ * Wilson rather than normal-approximation because these bands are small and
+ * rates sit near the edges, where the normal interval runs past 0 and 1 and
+ * quietly implies impossible rates.
+ */
+function wilson(hits, n, cumulativeTests = 1) {
+  const k = knownNumber(hits); const N = knownNumber(n);
+  if (k === null || N === null || N <= 0) return null;
+  const tests = Math.max(1, Math.round(knownNumber(cumulativeTests) ?? 1));
+  const z = zFor(1 - (0.05 / tests) / 2);
+  const p = k / N;
+  const d = 1 + (z * z) / N;
+  const centre = (p + (z * z) / (2 * N)) / d;
+  const half = (z * Math.sqrt((p * (1 - p)) / N + (z * z) / (4 * N * N))) / d;
+  return [round4(Math.max(0, centre - half)), round4(Math.min(1, centre + half))];
+}
+
+/** Inverse normal CDF (Acklam) — good to ~1e-9, enough for an interval bound. */
+function zFor(p) {
+  if (p <= 0 || p >= 1) return 0;
+  const a = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
+    1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00];
+  const b = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
+    6.680131188771972e+01, -1.328068155288572e+01];
+  const c = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
+    -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00];
+  const d = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
+    3.754408661907416e+00];
+  const pl = 0.02425;
+  let q; let r;
+  if (p < pl) {
+    q = Math.sqrt(-2 * Math.log(p));
+    return (((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5])
+      / ((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1);
+  }
+  if (p > 1 - pl) {
+    q = Math.sqrt(-2 * Math.log(1 - p));
+    return -(((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5])
+      / ((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1);
+  }
+  q = p - 0.5; r = q * q;
+  return (((((a[0] * r + a[1]) * r + a[2]) * r + a[3]) * r + a[4]) * r + a[5]) * q
+    / (((((b[0] * r + b[1]) * r + b[2]) * r + b[3]) * r + b[4]) * r + 1);
+}
+
+/**
+ * Build bands for ONE archetype.
+ *
+ * @param {Array}  rows  [{ p, won }] — p is the model's probability, won 0/1
+ * @param {object} opts
+ *   opts.archetype        label
+ *   opts.cumulativeTests  Bonferroni denominator (applied to every interval)
+ *   opts.proven           this archetype's factors passed the gate
+ *   opts.calibrated       this archetype's probabilities are certified calibrated
+ *   opts.targetBands      how many bands to attempt (default 5)
+ */
+function buildBands(rows, opts = {}) {
+  const usable = (rows || []).filter((r) =>
+    knownNumber(r && r.p) !== null && knownNumber(r && r.won) !== null);
+  const n = usable.length;
+  const tests = opts.cumulativeTests ?? 1;
+
+  // THE TWO-BAR RULE. Both, or the bands are a base-rate read and say so.
+  const factorInformed = Boolean(opts.proven) && Boolean(opts.calibrated);
+
+  if (n < MIN_BAND_N) {
+    return {
+      archetype: opts.archetype || null,
+      basis: factorInformed ? 'factor_informed' : 'base_rate',
+      n,
+      base_rate: null,
+      bands: [],
+      refused: 'insufficient settled outcomes to stand behind any band',
+    };
+  }
+
+  const wins = usable.reduce((s, r) => s + (r.won > 0 ? 1 : 0), 0);
+  const baseRate = wins / n;
+
+  // Candidate cuts by quantile of the model's own probability, then merged so
+  // every published band is distinguishable from its neighbour.
+  const sorted = [...usable].sort((a, b) => b.p - a.p);
+  const target = Math.max(2, Math.min(LETTERS.length, opts.targetBands || 5));
+  const per = Math.max(MIN_BAND_N, Math.floor(n / target));
+
+  let raw = [];
+  for (let i = 0; i < sorted.length; i += per) {
+    const slice = sorted.slice(i, i + per);
+    // A trailing remainder too small to stand alone joins the previous band
+    // rather than being published as its own thin claim.
+    if (slice.length < MIN_BAND_N && raw.length) raw[raw.length - 1].push(...slice);
+    else raw.push(slice);
+  }
+
+  // Merge neighbours whose corrected intervals overlap — if we cannot tell two
+  // bands apart, publishing them as different letters is a distinction we have
+  // not measured.
+  let merged = true;
+  while (merged && raw.length > 1) {
+    merged = false;
+    for (let i = 0; i < raw.length - 1; i += 1) {
+      const a = stats(raw[i], tests); const b = stats(raw[i + 1], tests);
+      if (a.ci && b.ci && a.ci[0] <= b.ci[1] && b.ci[0] <= a.ci[1]) {
+        raw.splice(i, 2, raw[i].concat(raw[i + 1]));
+        merged = true;
+        break;
+      }
+    }
+  }
+
+  const bands = raw.map((slice, i) => {
+    const s = stats(slice, tests);
+    // LIFT: does this band's interval clear the archetype's own base rate?
+    const clearsAbove = s.ci !== null && s.ci[0] > baseRate;
+    const clearsBelow = s.ci !== null && s.ci[1] < baseRate;
+    return {
+      letter: LETTERS[Math.min(i, LETTERS.length - 1)],
+      n: slice.length,
+      p_range: [round4(Math.min(...slice.map((r) => r.p))), round4(Math.max(...slice.map((r) => r.p)))],
+      realized_rate: s.rate,
+      ci: s.ci,
+      lift_vs_archetype_base: round4(s.rate - baseRate),
+      // The claim the letter is allowed to make.
+      shows_lift: clearsAbove,
+      shows_deficit: clearsBelow,
+      separation: clearsAbove ? 'above_base_rate' : (clearsBelow ? 'below_base_rate' : 'indistinguishable_from_base_rate'),
+      provisional: slice.length < PROVISIONAL_N,
+      basis: factorInformed ? 'factor_informed' : 'base_rate',
+    };
+  });
+
+  return {
+    archetype: opts.archetype || null,
+    basis: factorInformed ? 'factor_informed' : 'base_rate',
+    // Stated explicitly so a consumer never has to infer why it is a base-rate read.
+    two_bar: {
+      proven: Boolean(opts.proven),
+      calibrated: Boolean(opts.calibrated),
+      factor_informed_allowed: factorInformed,
+      ...(factorInformed ? {} : {
+        reason: !opts.proven
+          ? 'no factor has passed the gate for this archetype'
+          : 'factors proved but this archetype\'s probabilities are not certified calibrated',
+      }),
+    },
+    n,
+    base_rate: round4(baseRate),
+    bands,
+    bonferroni_tests: Math.max(1, Math.round(knownNumber(tests) ?? 1)),
+    bands_showing_lift: bands.filter((b) => b.shows_lift).length,
+  };
+}
+
+function stats(slice, tests) {
+  const n = slice.length;
+  const w = slice.reduce((s, r) => s + (r.won > 0 ? 1 : 0), 0);
+  return { rate: round4(n ? w / n : null), ci: wilson(w, n, tests) };
+}
+
+/**
+ * The sentence attached to a grade. TRUE or absent — never a fluent fallback.
+ *
+ * A factor-informed reason names the factors that proved FOR THIS ARCHETYPE. A
+ * base-rate reason says plainly that it is a base-rate read, because a user who
+ * is told "favourable matchup" when nothing about the matchup was read has been
+ * given a fabricated reason, and that is worse than being given none.
+ */
+function reasoning(band, opts = {}) {
+  if (!band) return null;
+  const arch = opts.archetype ? String(opts.archetype).toUpperCase() : null;
+  if (band.basis === 'factor_informed') {
+    const proved = (opts.provenFactors || []).filter(Boolean);
+    if (!proved.length) return null;          // cannot name what did not prove
+    return `${arch ? `${arch}: ` : ''}${proved.join(' + ')} — proved for this profile`;
+  }
+  const sep = band.separation === 'above_base_rate'
+    ? 'above this profile\'s base rate'
+    : (band.separation === 'below_base_rate' ? 'below this profile\'s base rate' : 'at this profile\'s base rate');
+  return `${arch ? `${arch}: ` : ''}base-rate read, ${sep}`
+    + ' — no matchup factor is proven for this profile yet';
+}
+
+const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
+
+module.exports = {
+  buildBands, reasoning, wilson,
+  PROVISIONAL_N, MIN_BAND_N, LETTERS,
+};