builtbykev
b2e4c6c4fb
The refinement was right. Naming the individual reliever failed; the same
question at the grain the chain needs passes, and it transmits more than
anything else measured in this chain.
WHY IT WAS WORTH RE-ASKING: last session's null (the pen is on average no
softer, +0.0010 on 35,760 PAs) does NOT rule this out, and treating it as
though it did would have been the error. An average washing out is fully
consistent with quality VARIATION mattering. It does -- actual arm quality
moves the hit rate monotonically across quartiles, 0.2244 / 0.2293 /
0.2410 / 0.2501, a 2.57pp spread, larger than the whole times-through-
the-order effect.
CLUSTER UNIT CORRECTED, THEN CHECKED RATHER THAN ARGUED. Last session
refused Link 2 partly as team-borne (30 bullpens, the park ceiling). My
first re-check was that 76% of pen-quality variance is within-team -- but
that is a statement about TREATMENT variance, not about where errors
correlate, and stopping there would have been picking the convenient
answer. Measured the actual thing: ICC of prediction error by team =
0.0261, design effect 1.41, SEs inflated ~19%. So the verdict was run
three ways:
unclustered CI [-0.0067,-0.0010] excludes zero
team-clustered (30) CI [-0.0086,-0.0003] excludes zero (below the
40-cluster floor -- indicative, not a pass)
design-effect adjusted CI [-0.0072,-0.0005] excludes zero
QUALITY GRAIN PROVES on the concentrated elevated-early-exit subset:
n=501 team-games, 426 clusters, MAE 0.0294 -> 0.0260, delta -0.0034, CI
[-0.0063,-0.0005] at 110 cumulative tests. Pooled also proves, so it is
not a subset artefact.
ARCHETYPE GRAIN DOES NOT: 0.5669 vs a 0.5309 modal-guess baseline,
corrected interval [-0.1073,+0.0268] spans zero. Two grains tested, one
earned a place -- penQuality.js exposes no archetype and a test asserts
it.
WHAT LINK 3 RECEIVES, which is the number that actually matters -- not
the MAE gain but realized outcome separation, prediction strictly
point-in-time:
predicted BEST pen 167 games 2,044 PAs hit rate 0.2231 +/-0.0180
predicted WORST pen 167 games 1,799 PAs hit rate 0.2501 +/-0.0200
2.70pp separated, intervals non-overlapping, capturing nearly all the
2.57pp available at the quartile grain. Caveat stated not buried: the
tercile cut is chosen in-sample; the prediction driving it is not.
BUILT: penQuality.js + 9 tests. Abstains below 5 prior club games and 40
arm appearances -- a league-average stand-in would assert "this is an
ordinary bullpen", which is a claim, and usually the wrong one for exactly
the clubs whose pens just turned over.
Link 3 is unblocked on a proven Link 2 at the quality grain only. Not run
here; this order scopes to building and gating Link 2.
Parallel track logged unchanged: TB n=948 pooled, BOMBER x TB 340, short
by 160.
Counter and frozen clusters byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
131 lines
5.2 KiB
Markdown
131 lines
5.2 KiB
Markdown
# Link 2 at the coarse grain — pen QUALITY proves, archetype does not
|
||
|
||
**The refinement was right.** Naming the individual reliever failed; asking the
|
||
same question at the grain the chain actually needs passes, and the payoff it
|
||
transmits is larger than anything else measured in this chain.
|
||
|
||
---
|
||
|
||
## Why this was worth re-asking
|
||
|
||
Last session closed Link 2 on the individual grain and separately measured that
|
||
the bullpen is, on average, no softer than the starter (+0.0010 on 35,760 plate
|
||
appearances). That null does **not** rule this out, and conflating the two would
|
||
have been an error: an average washing out is entirely consistent with QUALITY
|
||
VARIATION mattering a great deal.
|
||
|
||
It does. Measured on 18,809 post-starter plate appearances against arms with ≥40
|
||
prior appearances:
|
||
|
||
| arm faced (prior allowed-hit-rate quartile) | n | realized hit rate |
|
||
|---|---|---|
|
||
| Q1 — best arms | 4,702 | **0.2244** ±0.0119 |
|
||
| Q2 | 4,702 | 0.2293 |
|
||
| Q3 | 4,702 | 0.2410 |
|
||
| Q4 — worst arms | 4,702 | **0.2501** ±0.0124 |
|
||
|
||
Monotone, spread **+2.57pp** — larger than the whole times-through-the-order
|
||
effect (+1.6pp) and far larger than the pen-vs-starter difference (−0.7pp).
|
||
|
||
---
|
||
|
||
## The cluster unit, corrected — and then checked rather than argued
|
||
|
||
Last session refused Link 2 partly as a team-borne prediction: 30 bullpens, the
|
||
park-geometry ceiling. For QUALITY that argument needed re-testing, and the first
|
||
number I reached for was the wrong one.
|
||
|
||
- **Treatment variance:** 76% of a game's pen-quality variance is WITHIN team.
|
||
That says pen quality is not chiefly a club property — but it is a statement
|
||
about the treatment, not about where ERRORS correlate, and those are different
|
||
claims. Stopping there would have been picking the convenient answer.
|
||
- **Measured directly:** ICC of the prediction ERROR by team = **0.0261**. With
|
||
16.7 rows per club that is a design effect of 1.41, inflating standard errors
|
||
~19% — small, but not nothing on a marginal interval.
|
||
|
||
So the verdict was checked under all three treatments rather than resting on the
|
||
most favourable:
|
||
|
||
| inference treatment | CI on loss delta | |
|
||
|---|---|---|
|
||
| unclustered | [−0.0067, −0.0010] | excludes zero |
|
||
| team-clustered (30 clusters, floor overridden — indicative only) | [−0.0086, −0.0003] | excludes zero |
|
||
| design-effect adjusted (deff 1.41) | [−0.0072, −0.0005] | excludes zero |
|
||
|
||
It survives all three. Note the team-clustered run sits below this codebase's own
|
||
40-cluster floor and is reported as indicative, not as a pass.
|
||
|
||
---
|
||
|
||
## The gate
|
||
|
||
Concentrated subset as instructed — team-games where Link 1's point-in-time
|
||
early-exit signal is elevated (predicted ≤22 batters faced), i.e. where the pen
|
||
actually enters for the later plate appearances.
|
||
|
||
### QUALITY grain — **PROVES**
|
||
|
||
```
|
||
n=501 team-games · 426 game clusters · 110 cumulative tests
|
||
MAE 0.0294 (league-average baseline) -> 0.0260 delta -0.0034
|
||
CI [-0.0063,-0.0005] at 0.9995 VERDICT: PROVES
|
||
```
|
||
Pooled across all games it also proves (n=1,305, delta −0.0030, CI [−0.0049,
|
||
−0.0015]), so the result is not an artefact of the subset.
|
||
|
||
### ARCHETYPE grain — **NOT PROVEN**
|
||
|
||
```
|
||
n=501 · modal-guess baseline 0.5309 -> model 0.5669
|
||
corrected interval [-0.1073, +0.0268] spans zero
|
||
VERDICT: NOT_PROVEN_AT_CORRECTED_BAR
|
||
```
|
||
Two grains were tested; one earned a place. `penQuality.js` deliberately exposes
|
||
no archetype, and a test asserts it.
|
||
|
||
---
|
||
|
||
## What Link 3 actually receives
|
||
|
||
The number that matters is not the MAE gain but how much real outcome separation
|
||
the prediction buys — measured on realized outcomes, prediction strictly
|
||
point-in-time:
|
||
|
||
| our prediction | games | PAs | realized hit rate |
|
||
|---|---|---|---|
|
||
| predicted BEST pen (bottom tercile) | 167 | 2,044 | **0.2231** ±0.0180 |
|
||
| predicted WORST pen (top tercile) | 167 | 1,799 | **0.2501** ±0.0200 |
|
||
|
||
**2.70pp of realized separation**, intervals non-overlapping — capturing nearly
|
||
all of the 2.57pp available at the quartile grain. corr(predicted, actual pen
|
||
quality) = 0.393.
|
||
|
||
Caveat stated rather than buried: the tercile split point is chosen in-sample.
|
||
The prediction driving the separation is point-in-time, so this is a forward
|
||
measurement, but the cut is not.
|
||
|
||
---
|
||
|
||
## Built
|
||
|
||
`src/services/model/penQuality.js` (+ 9 tests) — the proven half, ready for Link
|
||
3. `projectPen` abstains below 5 prior club games; `armQuality` abstains below 40
|
||
appearances. A league-average stand-in would assert "this is an ordinary
|
||
bullpen", which is a claim, and usually the wrong one for exactly the clubs whose
|
||
pens have just turned over.
|
||
|
||
`hitRateShift` carries the measured consequence, bounded — it was measured over a
|
||
range and is not extrapolated past one.
|
||
|
||
**Link 3 is now unblocked** on a proven Link 2 at the quality grain only. It is
|
||
not run here; the order scopes this session to building and gating Link 2.
|
||
|
||
## Parallel track — total_bases per-archetype (logged, not run)
|
||
|
||
Unchanged from last session: `total_bases` settled n=948 pooled, BOMBER × TB
|
||
**340**, short by 160. Sample-readiness only, not a verdict. The second blocker
|
||
from `specs/per-archetype-grade-bands.md` still applies — the grade does not yet
|
||
separate within any archetype.
|
||
|
||
Counter and frozen clusters byte-identical.
|