Per-archetype re-audit: no slot reaches 500, and the replication unit
decided everything The premise does not hold. prove-hit-factors.js has no date filter anywhere in it and pages the full table -- there was never a window to widen. Full clean history is 1,266 rows, not 2,715. platoon was not "proved" last session, it was explicitly held on 4.5%-median-contaminated season-to-date splits, and pitcher_contact_profile was demoted. The proven set going in was one factor, not three. STEP 1: no archetype slot reaches n>=500 on full history. Best is BOMBER at 408, and BOMBER is the most common archetype on the board. GHOST 173, BRUSH 64, DRIVER 43, CATALYST 16. These are confirmed genuinely short, not artifacts. STEP 2 is where the real finding is. park_hits initially PROVED at 619 rows across 45 games -- but those games only ever visited 14 distinct park values. A park effect is replicated across parks, and unmodelled park heterogeneity is confounded with the thing being estimated. Each factor is now clustered on the coarser of the game and the entity its treatment rides on. That flipped two verdicts and confirms Kev's causal-correctness thesis from a new direction: defense_by_direction has 442 hitter-team units of replication where crude team defense has 26. The correct atom is not just more accurate, it is the only one measurable at all. park_hits (14) and defense (26) can never be validated however long the ledger runs -- the same ceiling as park dimensions, reached independently. Also fixed a bar I got wrong last session: I transplanted the 500-row floor onto clusters, which refused a factor with 1,059 rows over 85 games while answering neither question. Two floors now -- rows>=500 for a stable estimate, clusters>=40 for a trustworthy interval. Not a lowered bar: park_hits and defense are still refused. PROVEN: defense_by_direction only, pooled, [-0.0054,-0.0012] at 99 tests. It stays POOLED-ONLY -- no per-archetype reasoning wired, nothing grandfathered. The card must not say "GHOST: defence matchup strong" because we have not earned that sentence. The predicted fingerprint did not appear either: BOMBER -0.0036 vs GHOST -0.0024, the opposite direction, both noise-dominated. Recorded so it is not claimed later. RESCALE: NOT READY. One proven factor worth -0.0031 Brier. Rescaling on that is relabelling. Counter and frozen clusters byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -40,7 +40,24 @@ const { knownNumber } = require('../../utils/known');
|
||||
|
||||
/** Minimum mean |Δp| for a factor to count as having moved anything at all. */
|
||||
const MIN_MOVEMENT = 0.01;
|
||||
/** Observations needed for the effect ESTIMATE to be stable. */
|
||||
const MIN_N = 500;
|
||||
/**
|
||||
* Clusters needed for the cluster-robust INTERVAL to be trustworthy.
|
||||
*
|
||||
* These two floors answer different questions and must not be collapsed. Rows
|
||||
* govern whether the point estimate is stable; clusters govern whether the
|
||||
* interval around it means anything. Transplanting the 500-row bar onto clusters
|
||||
* refuses a factor measured over 1,059 rows and 85 games — which has ample
|
||||
* observations AND ample clusters — while telling us nothing about either.
|
||||
*
|
||||
* 40 is the conventional floor below which cluster-robust inference is known to
|
||||
* under-cover regardless of how many rows sit inside the clusters. It is a
|
||||
* statement about when the bootstrap can be believed, not a bar tuned to let
|
||||
* anything through: a venue-constant factor still caps at 30 ballparks and is
|
||||
* still refused, permanently.
|
||||
*/
|
||||
const MIN_CLUSTERS = 40;
|
||||
|
||||
const brier = (ps, ys) => (ps.length
|
||||
? ps.reduce((s, p, i) => s + (p - ys[i]) ** 2, 0) / ps.length : null);
|
||||
@@ -171,19 +188,20 @@ function adjudicate(rows, opts = {}) {
|
||||
|
||||
const base = { factor: opts.factor || null, archetype: opts.archetype || null, stat: opts.stat || 'hits', movement: mv, improvement: imp };
|
||||
|
||||
// Sample is judged in the unit the FACTOR varies over, not the unit the rows
|
||||
// happen to arrive in. A game-level factor with 928 rows across 53 games has
|
||||
// 53 readings, and calling that 928 is how a gate passes something on sample
|
||||
// it never had.
|
||||
const effN = imp && imp.effective_n != null ? imp.effective_n : mv.n;
|
||||
if (effN < minN) {
|
||||
const unit = imp && imp.cluster_unit === 'cluster' ? 'independent clusters' : 'rows';
|
||||
// TWO FLOORS, because they answer different questions. Rows decide whether the
|
||||
// point estimate is stable; clusters decide whether the interval around it can
|
||||
// be believed. A factor needs both.
|
||||
if (mv.n < minN) {
|
||||
return { ...base, verdict: 'CANDIDATE_PENDING_SAMPLE', reason: `n ${mv.n} < ${minN}`, rows_needed: minN - mv.n };
|
||||
}
|
||||
const minClusters = opts.minClusters ?? MIN_CLUSTERS;
|
||||
if (imp && imp.cluster_unit === 'cluster' && imp.effective_n < minClusters) {
|
||||
return {
|
||||
...base,
|
||||
verdict: 'CANDIDATE_PENDING_SAMPLE',
|
||||
reason: `${effN} ${unit} < ${minN}`
|
||||
+ (effN !== mv.n ? ` (${mv.n} rows, but the factor varies over ${effN} clusters — the rows are not independent readings)` : ''),
|
||||
rows_needed: minN - effN,
|
||||
reason: `${mv.n} rows but only ${imp.effective_n} independent clusters < ${minClusters}`
|
||||
+ ' — the rows are not independent readings and the interval cannot be trusted at this cluster count',
|
||||
clusters_needed: minClusters - imp.effective_n,
|
||||
};
|
||||
}
|
||||
if (mv.mean_abs_shift === null || mv.mean_abs_shift < minMove) {
|
||||
@@ -221,4 +239,5 @@ function adjudicate(rows, opts = {}) {
|
||||
|
||||
const round4 = (v) => (v == null || !Number.isFinite(v) ? null : Math.round(v * 10000) / 10000);
|
||||
|
||||
module.exports = { movement, improvement, adjudicate, MIN_MOVEMENT, MIN_N };
|
||||
module.exports = {
|
||||
MIN_CLUSTERS, movement, improvement, adjudicate, MIN_MOVEMENT, MIN_N };
|
||||
|
||||
Reference in New Issue
Block a user