The skill engine: built, gated by construction, and Stage A honestly lost
Built src/services/model/ -- the forward, archetype-selected, skill-based projection, as a challenger. The champion is untouched. featureRegistry makes "earn its place or it's out" structural rather than aspirational: CANDIDATE / PROVEN / DEAD per feature per sport, liveFeatures() returns PROVEN only, promotion requires n>=200 with positive lift and a CI excluding zero, and there is deliberately no override argument. It ships with exactly ONE proven feature -- the incumbent counter, because it is the only one with a measurement. A test asserts that with only PROVEN features allowed the projection returns null, so an unproven model cannot reach a user by accident. The three champion adjustment layers are registered DEAD with their reasons so they cannot be silently rebuilt. skillProjection is a PA outcome tree: K and BB combined by log5 odds-ratio against league (both identities unit-tested), then archetype-weighted contact quality against contact allowed, then Binomial(PA, p_hit) mixed over a PA distribution. Archetype is a FEATURE SELECTOR, not a nudge -- BOMBER reads barrels at 0.50 and ground-ball speed at 0.00, GHOST inverts it -- and a test locks that the same hitter read two ways moves more than 0.15. STAGE A: IT LOSES. Out-of-sample on 570 settled hits props with 91.9% opposing-pitcher coverage, resolution 0.0499 against the champion's 0.166, delta -0.116 with CI [-0.189, -0.043]. It is not selective either: its eight most confident picks hit 50%, a lift of -0.065. Not promoted. The gate did its job on its first real test, which is the point of having built it that way. Two false starts, both recorded because they nearly produced a wrong verdict: statcast_aggregates stores PERCENTAGES, so raw rows made bip = 1-29.6-17.1 and refused 568 of 576 -- the honest-absent guards made a units bug loud instead of silent, and the conversion now lives at one chokepoint. And the first run resolved an opposing pitcher for 1 of 570 rows, because ledger team/opponent are NULL, so it would have reported "skill-v1 loses" while measuring a batter-only model with no matchup in it at all. The verdict above is from the corrected run. The loss is real but partial: park was passed as 1.0, handedness and opportunity_drift never fired, PA is season-PA over a constant, and the skill profiles carry no recency at all while the champion has a last-5 term. Also fixed: the Statcast nightly refresh was unreachable code. It sat inside tick() below "if (!HOURS_UTC.includes(h)) return" while testing h === 11, so it had never run once; the aggregates were 13 days stale and both of its alerts were in the same dead branch. It now runs on its own tick, and the test that passed happily throughout -- it only checked the string existed -- is replaced by one that asserts it is not behind the guard. 4,182 tests green (333 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
@@ -1262,6 +1262,45 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
|
||||
forward inputs. "AT CEILING" (runs/walks) is provisional: measured only against
|
||||
features the champion already reads.
|
||||
|
||||
## The skill engine + feature registry (Session 80 — non-obvious)
|
||||
- **`specs/skill-engine-architecture.md`.** `src/services/model/` is the forward
|
||||
engine: `featureRegistry.js` (CANDIDATE/PROVEN/DEAD per feature PER SPORT) and
|
||||
`skillProjection.js` (PA outcome tree: K/BB via log5 odds-ratio vs league, then
|
||||
archetype-weighted contact quality → Binomial(PA, p_hit) mixed over a PA
|
||||
distribution). Challenger-only; champion untouched.
|
||||
- **THE GATE IS STRUCTURAL, not a habit.** `liveFeatures()` returns PROVEN only,
|
||||
and the registry ships with exactly ONE proven feature (the incumbent counter).
|
||||
A test asserts that with only PROVEN allowed, `projectSkill` returns NULL — an
|
||||
unproven model cannot reach a user by accident. `promote()` requires n>=200,
|
||||
positive lift, CI excluding zero, and has NO override argument.
|
||||
- **STAGE A RESULT: skill-v1 LOSES, not promoted.** 570 rows, 91.9% pitcher
|
||||
coverage, resolution 0.0499 vs champion 0.166, delta −0.116 CI [−0.189,−0.043].
|
||||
Also NOT selective — its top-8 most confident picks hit 50% (lift −0.065).
|
||||
- **UNITS: `statcast_aggregates` stores PERCENTAGES (0–100), not fractions.**
|
||||
`k_pct: 29.6` means 29.6%. Feeding raw rows in made `bip = 1−29.6−17.1` negative
|
||||
and refused 568/576 rows. ALWAYS convert via `skillProjection.fromStatcastRow`
|
||||
(the one chokepoint); it nulls out-of-range values rather than clamping, and
|
||||
leaves mph/degrees fields alone.
|
||||
- **`ledger_entries.team`/`opponent` are NULL on ~all rows** — do NOT join a
|
||||
matchup on them. The first Stage A run resolved a pitcher for 1 of 570 rows and
|
||||
would have reported a verdict on a batter-only model. Resolve the opponent from
|
||||
the player's own statsapi game log (`getPlayerGameLog` → `{date, opponent}`),
|
||||
which is authoritative and point-in-time safe → 91.9% coverage.
|
||||
- **Archetype = FEATURE SELECTOR, not a nudge.** `ARCHETYPE_MAP` weights decide
|
||||
which skill inputs drive a hitter (BOMBER barrel 0.50 / gb_speed 0; GHOST
|
||||
barrel 0.05 / gb_speed 0.60). Locked by test: same hitter read through two
|
||||
archetypes moves >0.15. Weights are DOCUMENTED, not fitted — fitting on 1,741
|
||||
rows is curve-fitting; the registry exists so they get measured.
|
||||
- **total_bases is deliberately REFUSED by skillProjection.** A deterministic
|
||||
bases-per-hit multiplier made P(TB>=2) exactly equal P(hits>=1) — a relabelled
|
||||
hits curve carrying no new information. TB needs tb-v1's compound per-hit bases
|
||||
distribution; refusing beats shipping a relabel.
|
||||
- **STATCAST REFRESH WAS UNREACHABLE CODE** (fixed): it sat inside `tick()` below
|
||||
`if (!HOURS_UTC.includes(h)) return` (14,19,22,1,3) while testing `h === 11`.
|
||||
Never ran once; data 13 days stale; BOTH its alerts were in the same dead
|
||||
branch. Now its own `statcastTick`. The old test only checked the string
|
||||
existed — the new one asserts it is not behind the snapshot-hours guard.
|
||||
|
||||
## Active Skills
|
||||
- vyndr-voice (all user-facing output)
|
||||
- prop-analysis (grading methodology)
|
||||
|
||||
Reference in New Issue
Block a user