vyndr/specs/per-archetype-re-audit.md

Per-archetype re-audit — and the replication unit that decided it

The premise this order opened with does not hold, and the query it blames was never windowed. prove-hit-factors.js selects on sport / stat / outcome only — there is no date filter anywhere in it, and it pages the full table. Nothing was being clipped.

claimed	measured
full clean history 2,715 rows	1,266
platoon "tripled to 1,208 and PROVED"	1,056 rows; passes the gate, not promoted
three proven pooled factors	one — see below

platoon and platoon_severity were explicitly held last session, not promoted: they ride season-to-date splits containing the games they predict (4.5% median contamination, 12.4% p90) and passed with an upper bound of −0.0001. That still stands. pitcher_contact_profile was demoted last session. So the proven set going in was one factor, not three.

---

STEP 1 — Full-history sample audit per slot

Run against full clean settled history, deduped, non-quarantined:

factor	ALL	BOMBER	GHOST	BRUSH	DRIVER	CATALYST
pitcher_contact_profile	1,059	408	173	64	43	16
platoon	1,056	408	173	64	43	16
defense	912	357	150	57	37	12
defense_by_direction	782	319	117	49	32	12
platoon_severity	700	343	121	24	38	16
park_hits	619	239	96	36	22	9

No archetype slot reaches n≥500 on full history. The best is BOMBER at 408, and BOMBER is by far the most common archetype on the board. These are CONFIRMED genuinely short — not windowed-query artifacts.

---

STEP 2 — The replication unit, which changed every verdict

Errors are correlated within a game (shared starter, park, weather, game state), so the interval must be clustered. But clustering on the game is still wrong for some factors, and the audit exposed it: park_hits initially "PROVED" at 619 rows across 45 games — yet those 45 games only ever visited 14 distinct park values. A park effect is replicated across parks. Unmodelled park heterogeneity is confounded with the very thing being estimated.

So each factor is now clustered on the coarser of the game and the entity its treatment rides on:

factor	rows	games	treatment entities	clustered on	k	Brier Δ	CI (corrected, 99 tests)	verdict
defense_by_direction	782	84	442 hitter×team	game	84	−0.0031	[−0.0054, −0.0012]	PROVES
platoon_severity	700	85	145 hitters	game	85	−0.0038	[−0.0070, −0.0007]	PROVES*
platoon	1,056	85	228 hitters	game	85	−0.0033	[−0.0061, −0.0006]	PROVES*
pitcher_contact_profile	1,059	85	130 pitchers	game	85	−0.0034	[−0.0067, 0.0000]	NOT_PROVEN
defense	912	84	26 teams	entity	26	−0.0038	[−0.0075, −0.0006]	PENDING — k<40
park_hits	619	45	14 park values	entity	14	−0.0037	[−0.0093, +0.0029]	PENDING — k<40

* held, not promoted — contaminated inputs, unchanged from last session.

This is Kev's causal-correctness thesis confirmed from a new direction

The causally-correct atom is not merely more accurate — it is the only one that is measurable at all. defense_by_direction has 442 units of replication because spray direction varies per hitter; crude team defense has 26, because there are 26 teams. The crude factor cannot be validated no matter how long the ledger runs, and its apparently-tight interval was pseudo-replication across teams that does not exist.

park_hits losing its pass is the same finding as last session's park-dimensions result, arrived at independently: venue- and team-borne factors cap at ~30 units, permanently.

Gate change: two floors, not one transplanted bar

Last session I applied the 500 bar to clusters. That was wrong in a way worth naming: it refused a factor with 1,059 rows over 85 games — ample observations and ample clusters — while answering neither question. The floors are now separate, because they answer different things:

• rows ≥ 500 — is the point estimate stable?
• clusters ≥ 40 — can the interval around it be believed?

40 is the conventional floor below which cluster-robust inference under-covers. This is not a lowered bar: park_hits (14) and defense (26) are still refused, and park geometry is still permanently unvalidatable.

---

STEP 3 — What gets wired

Nothing new. No archetype slot earns a wire, and none is grandfathered in from pooled proof.

• defense_by_direction — PROVES pooled, stays POOLED-ONLY. Its BOMBER (n=319) and GHOST (n=117) slots are short, so no per-archetype reasoning is wired. The card must not say "GHOST: defence matchup strong" — we have not earned that sentence.
• platoon, platoon_severity — pass the gate, held pending point-in-time splits.
• pitcher_contact_profile, defense, park_hits — honest null / confirmed short.

The predicted fingerprint did NOT appear

The order expected defense_by_direction strong for GHOST and ~zero for BOMBER. Measured point estimates run the other way — BOMBER −0.0036, GHOST −0.0024 — and at n=319/117 both are noise-dominated. Recorded so it is not claimed later. This is not evidence against the theory; it is evidence we cannot see it yet.

---

STEP 4 — Rescale readiness: NOT READY

One proven factor, worth −0.0031 Brier, clustered-honest. Two more held behind a contaminated input. The counter still supplies essentially all of the model's resolution (S78 ablation).

Rescaling the grade distribution on that would be relabelling — the same error as minting A's by moving thresholds, which is a permanent founder ruling. The distribution is not factor-rich enough.

What would change the answer, in order of cost:

1. Point-in-time platoon splits — would convert two held passes into real ones. Cheapest, no waiting; needs per-game split reconstruction.
2. More games — every archetype slot is short, and slots grow with games, not rows. BOMBER needs ~92 more rows to reach 500.
3. A factor with high replication — the lesson of this audit is that new factors should be chosen for causal correctness first, which also buys measurability. Anything venue- or team-borne is dead on arrival.