builtbykev/vyndr
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

The skill engine: built, gated by construction, and Stage A honestly lost

Browse Source 
Built src/services/model/ -- the forward, archetype-selected, skill-based
projection, as a challenger. The champion is untouched.

featureRegistry makes "earn its place or it's out" structural rather than
aspirational: CANDIDATE / PROVEN / DEAD per feature per sport, liveFeatures()
returns PROVEN only, promotion requires n>=200 with positive lift and a CI
excluding zero, and there is deliberately no override argument. It ships with
exactly ONE proven feature -- the incumbent counter, because it is the only
one with a measurement. A test asserts that with only PROVEN features allowed
the projection returns null, so an unproven model cannot reach a user by
accident. The three champion adjustment layers are registered DEAD with their
reasons so they cannot be silently rebuilt.

skillProjection is a PA outcome tree: K and BB combined by log5 odds-ratio
against league (both identities unit-tested), then archetype-weighted contact
quality against contact allowed, then Binomial(PA, p_hit) mixed over a PA
distribution. Archetype is a FEATURE SELECTOR, not a nudge -- BOMBER reads
barrels at 0.50 and ground-ball speed at 0.00, GHOST inverts it -- and a test
locks that the same hitter read two ways moves more than 0.15.

STAGE A: IT LOSES. Out-of-sample on 570 settled hits props with 91.9%
opposing-pitcher coverage, resolution 0.0499 against the champion's 0.166,
delta -0.116 with CI [-0.189, -0.043]. It is not selective either: its eight
most confident picks hit 50%, a lift of -0.065. Not promoted. The gate did its
job on its first real test, which is the point of having built it that way.

Two false starts, both recorded because they nearly produced a wrong verdict:
statcast_aggregates stores PERCENTAGES, so raw rows made bip = 1-29.6-17.1 and
refused 568 of 576 -- the honest-absent guards made a units bug loud instead of
silent, and the conversion now lives at one chokepoint. And the first run
resolved an opposing pitcher for 1 of 570 rows, because ledger team/opponent
are NULL, so it would have reported "skill-v1 loses" while measuring a
batter-only model with no matchup in it at all. The verdict above is from the
corrected run.

The loss is real but partial: park was passed as 1.0, handedness and
opportunity_drift never fired, PA is season-PA over a constant, and the skill
profiles carry no recency at all while the champion has a last-5 term.

Also fixed: the Statcast nightly refresh was unreachable code. It sat inside
tick() below "if (!HOURS_UTC.includes(h)) return" while testing h === 11, so
it had never run once; the aggregates were 13 days stale and both of its
alerts were in the same dead branch. It now runs on its own tick, and the test
that passed happily throughout -- it only checked the string existed -- is
replaced by one that asserts it is not behind the guard.

4,182 tests green (333 suites); web build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
This commit is contained in:
Kev
2026-08-03 02:20:40 -04:00
parent c551bf0340
commit 258d8a6655
9 changed files with 1439 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/skill-engine-architecture.md
+173
View File
@@ -0,0 +1,173 @@
# THE SKILL ENGINE — architecture, and Stage A's honest result
**Built 2026-08-03.** Baseball first. Challenger-only; the champion is untouched.
> **STAGE A VERDICT: skill-v1 LOSES to the frequency counter and is NOT promoted.**
> Out-of-sample on 570 settled hits props with 91.9% opposing-pitcher coverage:
> resolution **0.0499 vs the champion's 0.166**, delta **−0.116, CI
> [−0.189, −0.043]**. The architecture's own rule fired and kept it out of the
> live path. §5 is what was actually exercised and what wasn't — the loss is
> real, but it does not yet condemn the approach, and §6 says why.
---
## 1. What was built
| module | discipline | what it does |
|---|---|---|
| `src/services/model/featureRegistry.js` | **D3** — earn its place | CANDIDATE / PROVEN / DEAD per feature, per sport. `liveFeatures()` returns PROVEN only |
| `src/services/model/skillProjection.js` | **D1 + D2** | PA outcome tree from skill inputs; archetype selects the feature map |
| `tests/unit/skillModel.test.js` | all | 22 tests asserting the disciplines as behaviour |
| `scripts/skill-v1-stagea.js` | **D4** | the out-of-sample proof + selectivity measurement |
## 2. The model (D1 — skill, not results)
A hitter gets N plate appearances; each resolves through a tree:
```
PA ──> strikeout (batter K vs pitcher K, odds-ratio vs league)
──> walk (batter BB vs pitcher BB, odds-ratio vs league)
──> ball in play
└─> hit? (archetype-weighted contact quality vs contact allowed, park)
```
Rates combine by **log5 / odds ratio** against league — the standard way to ask
"this batter vs THIS pitcher" instead of "vs the average pitcher". Two identities
are unit-tested rather than assumed: league-vs-league returns league, and an
average pitcher leaves the batter's own rate untouched.
Hits then distribute as `Binomial(PA, p_hit_per_PA)`, mixed over a PA
distribution — because PA is not known in advance and pretending it is would be
a fabricated certainty.
## 3. Archetype selects features (D2)
Not a nudge, not a label — it decides **which skill inputs drive this hitter**:
| archetype | barrel | hard-hit | exit velo | GB/speed | power |
|---|---|---|---|---|---|
| BOMBER | **0.50** | 0.35 | 0.15 | **0.00** | 1.25 |
| GHOST | **0.05** | 0.20 | 0.15 | **0.60** | 0.70 |
| TORCH | 0.20 | 0.35 | 0.25 | 0.20 | 1.00 |
Locked by test: the same hitter read as BOMBER vs GHOST moves **>0.15** in
probability, and barrel rate moves a BOMBER more than 3× what it moves a GHOST.
Weights are documented by what the archetype physically does — **not fitted**,
because fitting them on 1,741 rows is curve-fitting, and the registry exists so
they get measured instead.
## 4. The gate is structural (D3)
The registry ships with **one PROVEN feature** — `recent_frequency_prior`, the
incumbent counter, and only because it has the measurement (S78 ablation). Every
skill input is CANDIDATE.
**Locked by test: with only PROVEN features allowed, `projectSkill` returns
null.** An unproven model cannot reach a user by accident — not by convention,
by construction. Promotion requires n ≥ 200, positive lift, and a CI excluding
zero; there is deliberately **no override argument**.
Three champion layers are already registered **DEAD** with their reasons, so they
cannot be silently rebuilt: `champion_opp_rank_adj`, `champion_home_away_adj`
("removing it IMPROVED rbi resolution — actively harmful"),
`champion_consistency_pull`.
## 5. STAGE A — the measurement, and what it actually tested
**Out-of-sample guarantee:** skill profiles were frozen 2026-07-21 (the nightly
refresh had never run — §7). Only rows with `game_date > 2026-07-21` were scored,
so no profile can contain the game it predicts. The bug that broke production
made this one measurement clean.
```
matched rows 570 pitcher coverage 91.9%
base rate 0.5649
resolution skill-v1 0.0499 champion 0.1660
brier skill-v1 0.2839 champion 0.2506
mean forecast skill-v1 0.5372 champion 0.6055
delta vs champion −0.1161 CI95 [−0.1892, −0.0426] ← excludes zero
```
**Selectivity (D4), measured not claimed:**
| top N by confidence | hit rate | lift over base |
|---|---|---|
| 8 | 0.500 | −0.065 |
| 15 | 0.533 | −0.032 |
| 25 | 0.480 | −0.085 |
| 100 | 0.550 | −0.015 |
**It is not selective yet either** — its most confident picks are not its best
ones. That is the honest read and it is the second reason not to promote.
### Two false starts caught in the build, both worth recording
1. **A units bug.** `statcast_aggregates` stores percentages (0–100), not
fractions. The first run made `bip = 1 − 29.6 − 17.1` deeply negative and
**refused 568 of 576 rows.** The honest-absent guards worked exactly as
designed — a model that refuses is visible, a model that fabricates is not.
The conversion now lives at one chokepoint (`fromStatcastRow`), and an
out-of-range percentage is treated as broken (null), never clamped.
2. **An invalid first verdict.** The initial run resolved an opposing pitcher for
**1 of 570 rows** (0.18%) because `ledger_entries.team`/`opponent` are NULL on
575 of 576 rows in this window. It would have reported "skill-v1 loses" while
silently measuring a **batter-profile-only** model with no matchup at all —
the exact false verdict this architecture exists to prevent. Fixed by
resolving the opponent from each player's own statsapi game log (91.9%
coverage). **The verdict in this document is from the corrected run.**
## 6. What the loss does and does not say
**Does say:** this implementation, on these inputs, is worse than the counter at
calling listed lines, and it does not go live.
**Does not say the approach is dead** — several candidate features never fired in
this run, so the test was of a partial model:
| feature | exercised? |
|---|---|
| batter K / BB / barrel / hard-hit / exit velo / launch | **yes** |
| pitcher K / BB / hard-hit allowed | **yes** (91.9% of rows) |
| `park_factor` | **no** — park was passed as 1.0 for every row |
| `handedness_platoon` | **no** — never wired into the tree |
| `opportunity_drift` | **no** — the one feature with measured residual signal |
| `projected_pa` | **crude** — season PA ÷ a constant 103 team games |
And one structural handicap worth naming: the skill profiles are **season
aggregates with no recency at all**, while the champion carries a last-5 term. So
skill-v1 was asked to beat a recency-aware model using a 13-day-old season
average. That is a fair loss to record and an unfair basis for concluding that
skill inputs don't work.
**Adding the pitcher moved resolution 0.0441 → 0.0499** — almost nothing. That is
the most interesting number here, and the honest next question: is the matchup
signal genuinely small at the 0.5 hits line, or is the log5 combine too blunt to
express it?
## 7. Fixed along the way — the Statcast refresh was unreachable code
`snapshotScheduler.tick()` returns for any hour not in `HOURS_UTC`
(14,19,22,1,3); the Statcast block then tested `h === STATCAST_HOUR_UTC`
(default **11**). That branch could never execute — the nightly refresh had never
run once, the aggregates sat frozen for 13 days while every consumer served them
as current, and **both of its alerts were inside the same dead branch**, so
nothing could report it. Same silently-guarded-out shape as the settlement
outage.
It now runs on its **own tick**, independent of the snapshot hours. The old test
asserted only that the string `STATCAST_HOUR_UTC` appeared in the file and passed
happily the entire time; it is replaced with one that asserts the refresh is
**not reachable only via the snapshot-hours guard**.
## 8. Next iteration — ranked by what the measurement actually showed
1. **Recency in the skill profile.** The champion's edge is recency and skill-v1
has none. Weight recent PA into the rates before concluding anything else.
2. **Wire the unexercised features** — park, handedness, and `opportunity_drift`
(the one feature with measured residual signal, S78).
3. **Real PA projection** from lineup slot, not season PA ÷ 103.
4. **Then re-run Stage A.** If it still loses with all of that firing, the
conclusion is much stronger and worth acting on.
**Not recommended:** promoting anything, tuning the archetype weights to fit
1,741 rows, or relaxing the bar. The gate did its job today.