b818626870cf3abbfb99c48433a15626aa110ce5
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 builtbykev
|
ff037e40c2 |
Re-adjudicate: nothing to demote, and close the hole that would have mattered
There is nothing to re-adjudicate. The proven set is empty and always has
been -- verified three ways: proven-status reports EMPTY, validatedSkills()
returns {} for every archetype, and zero conditioning entries have ever
reached PROVEN. The one PROVEN feature is recent_frequency_prior, which is the
incumbent counter itself, proven by the S78 ablation as ~100% of the
champion's resolution. It is the baseline every challenger is measured
against, not a conditioning interaction, and demoting it would leave the model
with nothing to grade from.
A correction to the premise: the cumulative gate did NOT catch a false
positive last session. It caught nothing, because there was nothing in the
proven set to catch. What it did was tighten alpha from 0.0026 to 0.0013
within one session, which demonstrated the mechanism working rather than a
demotion. So steps 3 and 4 -- demote, recalibrate -- are vacuous here, and
readjudicateAll says so plainly rather than glossing a no-op.
But the worry behind the order was well founded, and the audit found the real
exposure: promote() did not require the cumulative denominator. It checked n,
lift and CI, and nothing stopped a future session from testing eight
hypotheses, correcting by eight, and promoting on a p-value that would not
survive the programme's real denominator. That is precisely the hole that
makes a retroactive re-adjudication pass necessary later, so it is closed at
promotion time instead. isSufficient now refuses evidence carrying no
correction, evidence corrected against fewer tests than the cumulative count,
and any p-value that does not clear 0.05 over its own test count. The same
rule guards a PROVEN conditioning entry.
The second audit found two of four analysis scripts still correcting
per-session; pitcher-prove-k and tb-solo-and-interactions now use the
cumulative ledger, so the correction is native on every path.
reAblation.js is the standing second line: pure and injectable, so the
decision rule cannot drift from the gate's, and every verdict records both
p-values and both test counts so a demotion is re-derivable by anyone. A
feature promoted at alpha 0.05/20 can demote on the same p-value once the bar
is 0.05/60 -- correct, because the bar rose only after the programme had more
chances to get lucky. No fresh measurement is PENDING_RETEST and never a
demotion: absence of a re-test is not evidence, and demoting on it would
punish whichever stat happens to be off-season.
Net effect on the proven set is zero. No demotions, no recalibrations, and no
public ledger event -- announcing "recalibrated after re-adjudication" when
nothing changed would itself be a false signal of rigour.
4,238 tests green (337 suites); web build exit 0; counter byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9
|
||
|
builtbykev
|
4aab18096f |
Prove both on total bases -- and find that my own fix destroyed the backtest
Nothing passed. Nothing promoted. Counter byte-identical. THE BLOCKER, which is the real finding. statcast_aggregates is upserted in place and holds exactly one as-of date. Yesterday's skill backtest was honest only by accident: the nightly refresh was unreachable code, so the profiles sat frozen at 2026-07-21 -- before the settled window. Repairing that cron was right for production and it refreshed them to today, destroying every prior version. Scoring a 2026-07-25 game now uses a season aggregate that contains that game. Point-in-time validation is structurally impossible from that table, so every number in this run is contaminated and directional, and none of it is a gate verdict. Fixed forward: statcast_history retains a dated snapshot on every refresh, so point-in-time becomes "as_of_date < game_date, most recent". Retention is best-effort and cannot fail the refresh; both properties are unit-tested. It has one day of data, which is not yet a window. SOLO BASELINE, n=383, Bonferroni across 12 tests (alpha 0.00417): nothing passes. hard_hit_pct is closest at marginal r 0.135 with p 0.0080, failing both the 0.15 effect bar and the corrected alpha. And it drifted DOWN from 0.153 at n=295 -- an estimate regressing as noise averages out, not an effect firming up. I called that number encouraging yesterday; on 88 more rows it is fading, and it should not keep being quoted at its best value. INTERACTIONS, each scored by partial correlation against the counter residual controlling for both of its own components: none pass. Only barrel x power archetype has an incremental exceeding its parts (-0.101 against 0.019) at n=260 -- the shape Discipline 2 predicts, but a lead, not a finding. A methodological catch worth keeping. The archetype conditioner was first built as barrel_pct over league barrel -- a monotone transform of one of its own components -- so the "interaction" was barrel squared, measuring nonlinearity in barrel rate rather than any archetype effect, and it produced this run's only positive result. A Gauss-Jordan pivot test does not catch that, because the two columns differ by a scale factor. Fixed with a scale-free collinearity check plus real archetype labels joined from model_snapshots. Without it this document would have reported a fabricated interaction as the session's finding. COMBINED vs COUNTER on total bases: 0.2718 against 0.2647, delta +0.0071, CI [-0.065, +0.079] -- inconclusive, and the first time a challenger has not lost. The same engine on hits was -0.116 with a CI excluding zero. That contrast is the whole argument for total bases, and it is what the physics said: contact quality governs extra bases, not whether a grounder finds a hole. Also built: the compound TB projection. skillProjection no longer refuses total bases -- a deterministic bases-per-hit multiplier had made P(TB>=2) exactly P(hits>=1), a relabelled hits curve. It is now a convolution over per-PA base outcomes with hit-type shares shifted by skill. Non-degeneracy is locked by test. 4,204 tests green (334 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9 |
||
|
builtbykev
|
c7cc8f5e52 |
Build the gate, run it, and find we were proving things on the wrong stat
PREMISE CORRECTION FIRST. statModel.js and correlateValidator.js do not exist in this repository. The validation spec's only prior form is src/services/python/blueprints/unconventional.py -- a Flask blueprint in the Python service that is offline in production, scoring NBA factors against a warehouse that was never populated -- and tests/unit/supplementSystems.test.js requires only fs and path while defining its own validateFactor inline at line 368. Those tests assert a re-implementation of the thresholds, not an implementation, which is exactly why they passed for months while nothing was connected. The diagnosis behind the order is right -- every challenger was measured without a gate -- but the cause is that there was no gate on the Node side to import. So it is built, to the exact spec. correlateValidator: n>=500, |r|>=0.15, p<0.05, Bonferroni across the sweep. The p-value is exact rather than approximated (t-transform through a regularized incomplete beta) and is verified in the suite against known values, because scipy is not available here. Pairs with an unknown side are dropped, never zero-filled -- a zero-fill inside a correlation does not add noise, it invents a point at the origin. THE RUN, hits, n=570, Bonferroni-8: every skill feature fails, and not narrowly. The strongest marginal correlation against the counter's residual is 0.062 against a 0.15 bar. That is an effect-size failure at a sample that would have found a real effect comfortably -- a clean, well-powered negative. The head-to-head agrees: value engine 0.0499 against the counter's 0.166, delta -0.116 with CI [-0.189, -0.043]. Not promoted. THE RUN, total bases, n=295: cannot be tested, and that is the finding. hard_hit_pct shows a marginal r of 0.153 -- above the threshold -- and exit velo 0.124, refused solely because n is 205 short of 500. It is the most encouraging number this work has produced, and it is what the physics predicts: contact quality governs extra bases, not whether a grounder finds a hole. We have been testing skill inputs on the one stat where they should not matter much. Two things the run forced. Feature verdicts are now PER STAT, because marking these DEAD sport-wide on hits evidence would have killed, for total bases, the features that look most alive there -- per-sport doctrine one level deeper. And the gate now reports r and p even when underpowered, because "not enough data yet" and "nothing here" demand opposite decisions and a bare refusal was hiding the best signal on the board. Next: build the compound TB projection (skillProjection still refuses total bases by design, since a deterministic bases-per-hit made P(TB>=2) identical to P(hits>=1)), accrue to n>=500, re-run this gate. Leave hits alone. 4,200 tests green (334 suites); web build exit 0; counter byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9 |
||
|
builtbykev
|
258d8a6655 |
The skill engine: built, gated by construction, and Stage A honestly lost
Built src/services/model/ -- the forward, archetype-selected, skill-based projection, as a challenger. The champion is untouched. featureRegistry makes "earn its place or it's out" structural rather than aspirational: CANDIDATE / PROVEN / DEAD per feature per sport, liveFeatures() returns PROVEN only, promotion requires n>=200 with positive lift and a CI excluding zero, and there is deliberately no override argument. It ships with exactly ONE proven feature -- the incumbent counter, because it is the only one with a measurement. A test asserts that with only PROVEN features allowed the projection returns null, so an unproven model cannot reach a user by accident. The three champion adjustment layers are registered DEAD with their reasons so they cannot be silently rebuilt. skillProjection is a PA outcome tree: K and BB combined by log5 odds-ratio against league (both identities unit-tested), then archetype-weighted contact quality against contact allowed, then Binomial(PA, p_hit) mixed over a PA distribution. Archetype is a FEATURE SELECTOR, not a nudge -- BOMBER reads barrels at 0.50 and ground-ball speed at 0.00, GHOST inverts it -- and a test locks that the same hitter read two ways moves more than 0.15. STAGE A: IT LOSES. Out-of-sample on 570 settled hits props with 91.9% opposing-pitcher coverage, resolution 0.0499 against the champion's 0.166, delta -0.116 with CI [-0.189, -0.043]. It is not selective either: its eight most confident picks hit 50%, a lift of -0.065. Not promoted. The gate did its job on its first real test, which is the point of having built it that way. Two false starts, both recorded because they nearly produced a wrong verdict: statcast_aggregates stores PERCENTAGES, so raw rows made bip = 1-29.6-17.1 and refused 568 of 576 -- the honest-absent guards made a units bug loud instead of silent, and the conversion now lives at one chokepoint. And the first run resolved an opposing pitcher for 1 of 570 rows, because ledger team/opponent are NULL, so it would have reported "skill-v1 loses" while measuring a batter-only model with no matchup in it at all. The verdict above is from the corrected run. The loss is real but partial: park was passed as 1.0, handedness and opportunity_drift never fired, PA is season-PA over a constant, and the skill profiles carry no recency at all while the champion has a last-5 term. Also fixed: the Statcast nightly refresh was unreachable code. It sat inside tick() below "if (!HOURS_UTC.includes(h)) return" while testing h === 11, so it had never run once; the aggregates were 13 days stale and both of its alerts were in the same dead branch. It now runs on its own tick, and the test that passed happily throughout -- it only checked the string existed -- is replaced by one that asserts it is not behind the guard. 4,182 tests green (333 suites); web build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9 |