From c551bf03406e2ae46cd8570e340f1382d8b6b19e Mon Sep 17 00:00:00 2001 From: Kev Date: Mon, 3 Aug 2026 00:53:56 -0400 Subject: [PATCH] Reality assessment: the forward model exists, wired to the wrong side of the pipe READ-ONLY. src/ and web/ untouched. Inventoried every forward-model component against the real objective -- a forward matchup projection, not an edge number. The finding is that all of it already exists and is already loaded in production, and 100% of it sits DOWNSTREAM of the grade in challenger columns nothing serves. The served p_win reads three features and a game log; it has never seen a pitcher. Inputs are HAVE, not missing: statcast_aggregates carries 1,354 rows (750 pitchers, 604 batters) with exit velo, launch angle, barrel, hard-hit, whiff, chase, pitch mix, GB/FB, arm angle, and handedness complete on every row. Real gaps are team defense and catcher/umpire. So Stage A is a plumbing-and- modelling job, not a data-acquisition job. Found along the way: the Statcast nightly refresh is unreachable code. tick() returns for any hour not in HOURS_UTC (14,19,22,1,3) and the refresh block then tests h === 11, which that guard can never admit. The mechanism data has been frozen at its 2026-07-21 backfill for 13 days, and the block's own failure alert sits in the same dead branch -- the identical silently-guarded- out shape as the settlement outage. Design shows the counter: every factor label the SIGNAL BREAKDOWN renders is a restatement of recent frequency (l5_hot_vs_line, l20_over_line, back_to_back, home_game) plus several structurally-NBA labels (referees, coach pace, starters out) inside a baseball product. Not one names a pitcher, pitch type, handedness or park. The card's forward-read slots already exist and go unfilled -- the surface needs feeding, not redesigning. On what changes: the prior measurements were outcome-accuracy, not edge, so the metric was right and the question was narrow. proj-v1.1 and hits-v1 stay correctly refuted as DISTRIBUTION swaps on thin inputs -- neither tested a matchup-fed projection. arch-v1 is a market-relative nudge by construction and is the one component genuinely measured on the wrong axis. AT CEILING is provisional: measured only against features the champion already reads. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01W1sivYNqY2TS5ftykmHBU9 --- BUILD-STATE.md | 29 +++ CLAUDE.md | 38 ++++ specs/forward-model-reality-assessment.md | 232 ++++++++++++++++++++++ 3 files changed, 299 insertions(+) create mode 100644 specs/forward-model-reality-assessment.md diff --git a/BUILD-STATE.md b/BUILD-STATE.md index 2931fd3..c767a6e 100755 --- a/BUILD-STATE.md +++ b/BUILD-STATE.md @@ -3,6 +3,35 @@ ## Last Updated 2026-08-03 +## Session 79 (2026-08-03) — Reality assessment vs the FORWARD-PROJECTION objective ✅ +Spec: `specs/forward-model-reality-assessment.md`. READ-ONLY (src/web untouched). +- **Finding: the forward model's parts all EXIST and are all wired downstream of + the grade.** `probabilityEstimator` (the served p_win) reads 3 features + the + game log. Statcast/arsenal/park/weather/platoon/archetype load in + `snapshotService` AFTER grading, into challenger columns nothing serves. + `mlbContext` has zero consumers. +- **Statcast nightly refresh is DEAD CODE by guard** — tick() returns for hours + not in HOURS_UTC (14,19,22,1,3); the block tests h===11. Data frozen at + 2026-07-21 (13 days stale); its own failure alert is in the same dead branch. +- **Inputs are HAVE** — 1,354 statcast rows, handedness complete both sides, + pitch mix/velo/break, GB/FB, barrel, exit velo, launch. MISSING: team defense + (OAA/DRS), catcher framing/umpire. PARTIAL: batter GB/FB (in `metrics` JSONB), + lineup slot (role tables 0 rows). +- **Design shows the COUNTER.** Factor labels are all `l5_hot_vs_line`-family + plus NBA leftovers (refs, coach pace). The card's forward-read slots + (archetypeBlend "Why this grade", vyndrIntel.matchup, propDNA) exist and go + unfilled. Needs feeding, not redesign. +- **STAGED DISTANCE:** Stage A (forward baseball model) = ONE real build, ZERO + data acquisitions — assemble hitter profile × pitcher stuff × conditions as the + SPINE with frequency demoted to a prior; risk is sample, not feasibility. + Stage B (calibrated + scouting surface) = short once A exists (clamp/calibration + already diagnosed + swap the factor vocabulary). Stage C (per sport) = blocked + on mechanism data we do not have for NBA/WNBA (ESPN is box scores, Python + service offline) and soccer is odds-api quota-blocked. +- **Verdict re-checks:** proj-v1.1 + hits-v1 stay refuted AS DISTRIBUTION SWAPS + (neither tested a matchup-fed projection); arch-v1 is market-relative by + construction = the one measured on the wrong axis; "AT CEILING" is provisional. + ## Session 78 (2026-08-03) — Champion decomposed: the edge is a hit-rate counter ✅ Spec: `specs/champion-input-diagnosis.md`. READ-ONLY (src/web untouched); 4,159 tests green. diff --git a/CLAUDE.md b/CLAUDE.md index 664f297..33aabae 100755 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1224,6 +1224,44 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section). built for exactly this kind of replay was never settled, so ablations must join outcomes from `ledger_entries` on (player_key, stat, line, side, game_date). +## Forward-model reality check (Session 79 — non-obvious) +- **`specs/forward-model-reality-assessment.md`.** THE OBJECTIVE is a FORWARD + matchup projection (hitter profile × pitcher stuff × park/conditions, read + through archetype), not a market-edge number. Every component that needs + exists AND is loaded in prod — and ALL of it sits DOWNSTREAM of the grade. +- **The served grade sees NONE of it.** `probabilityEstimator` reads exactly + three features (`opp_rank_stat`, `home_away`, `l10_stddev`/`l20_avg`) plus the + game log. statcast rows, arsenal, park, weather, platoon and archetype are all + loaded in `snapshotService` AFTER grading and written to CHALLENGER columns. + `mlbContext` (platoon/handedness) has ZERO consumers — dead code. +- **STATCAST NIGHTLY REFRESH IS UNREACHABLE CODE.** `snapshotScheduler.tick()` + returns at `if (!HOURS_UTC.includes(h)) return` (14,19,22,1,3); the statcast + block then tests `h === STATCAST_HOUR_UTC` (default **11**), which that guard + can never admit. Data frozen at its 2026-07-21 backfill; its failure alert is + inside the same dead branch so it can't warn. Same shape as the settlement + outage — guarded-out code that reports nothing. Set STATCAST_HOUR_UTC to one of + HOURS_UTC or move the block above the guard. +- **Inputs are HAVE, not missing** — `statcast_aggregates` 1,354 rows (750 + pitchers / 604 batters): exit velo, launch, barrel, hard-hit, whiff, chase, + pitch_mix, GB/FB, arm angle, and **bats/throws complete on all 1,354**. Gaps + are team DEFENSE (only a coarse `opp_rank_stat`) and catcher framing/umpire. + PARTIAL: batter GB/FB land in the `metrics` JSONB not the typed columns; + lineup-slot tables (`player_role_profiles`, `lineup_role_profiles`) are 0 rows. +- **The card is designed for the forward read; the engine never fills it.** The + factor vocabulary the "SIGNAL BREAKDOWN" renders is entirely counter-restating + (`l5_hot_vs_line`, `l20_over_line`, `back_to_back`, `home_game`) with several + structurally-NBA labels (`ref_foul_high`, `coach_pace_delta`, + `opp_3plus_starters_out`). No signal names a pitcher, pitch type, handedness or + park. Surface needs FEEDING, not redesigning. +- **What the prior verdicts do and don't say.** Resolution = corr(forecast, + outcome) was never a market/edge test — the metric was right, the QUESTION was + narrow ("does challenger out-rank champion?"). proj-v1.1 and hits-v1 remain + correctly refuted AS DISTRIBUTION SWAPS on thin inputs; neither tested a + matchup-fed projection. arch-v1 IS market-relative by construction and is the + one component genuinely measured on the wrong axis — re-test its axes as + forward inputs. "AT CEILING" (runs/walks) is provisional: measured only against + features the champion already reads. + ## Active Skills - vyndr-voice (all user-facing output) - prop-analysis (grading methodology) diff --git a/specs/forward-model-reality-assessment.md b/specs/forward-model-reality-assessment.md new file mode 100644 index 0000000..379c96a --- /dev/null +++ b/specs/forward-model-reality-assessment.md @@ -0,0 +1,232 @@ +# REALITY ASSESSMENT — what exists vs the FORWARD PROJECTION objective + +**Read-only inventory, 2026-08-03.** Nothing built, nothing changed. + +> **THE HEADLINE:** every component the forward model needs **already exists and +> is already loaded in production** — Statcast batter contact quality, pitcher +> stuff and handedness, park, weather, platoon, arsenal classification, +> archetypes. **All of it sits DOWNSTREAM of the grade**, feeding challenger +> columns that are never served. The thing the user actually sees is a hit-rate +> counter that has never seen a pitcher. +> +> The distance to Stage A is **not data acquisition. It is re-plumbing** — plus +> one broken cron and one unreachable code branch found along the way. + +--- + +## 0. First, a correction — because it changes what needs re-checking + +The order says "we have measured EDGE for five sessions — the WRONG test." +**That is not what the measurements were, and the distinction matters.** + +Resolution as measured = `corr(forecast, outcome)` — *do higher forecasts hit +more often*. That is outcome accuracy at the listed line. It never referenced the +market. The market-relative objects are separate and few: `proj_book_implied`, +`fair_prob`, CLV/dCLV, and **arch-v1, which is by construction a "market-relative +p_win nudge."** + +So the metric was right. **What was wrong was the QUESTION.** Every session asked +*"does challenger X out-rank champion Y?"* and never *"can a forward matchup read +be produced at all, and does it call listed lines better than base rate?"* Those +have different answers, because: + +- **Resolution parity ≠ product parity.** A frequency counter and a forward model + could both resolve ~0.20 on hits, and only one of them can say *why*. +- Ranking is not the same as **accuracy above base rate**, which is what "accurate + listed-line calls" means. That was only measured once, at the very end of the + last session (the calibration table), and it failed: **+3.5pt global + over-prediction**. + +That correction narrows what has to be re-checked (§4) rather than voiding it. + +## 1. Component inventory — SPINE / NUDGE / DISCONNECTED (STEP 1) + +**The served grade is `gradeSlateService → analyzeViaEngine1 → probabilityEstimator`. +`p_win` reads exactly three features:** `opp_rank_stat`, `home_away`, +`l10_stddev`/`l20_avg`. Everything below is measured against that path. + +| component | exists? | what it computes today | role in the SERVED grade | +|---|---|---|---| +| **base + recency frequency** | yes | frequency of (stat > THIS line), 0.6/0.4 with last-5 | **THE SPINE — ~100% of resolution** (measured, `specs/champion-input-diagnosis.md`) | +| **archetypeService** (41 archetypes) | yes | full feature-scoring classifier, primary/secondary/blend | **DISCONNECTED.** Attached at *snapshot* time for display only; `analyzeViaEngine1` explicitly does not set it | +| **matchupRead** (arsenal vs hitter) | yes | sinker/breaking mix, whiff & hard-hit tendency → multiplier | **DISCONNECTED from the grade.** Only inside proj-v1.1, a challenger measured *worse* | +| **park** (`parkBase`, `parkFactors`) | yes | per-venue factors, exposure-relative baseline | **NUDGE, in challengers only** (arch-v1 env axis, proj-v1.1) | +| **weather** (`environmentContext`) | yes | temp / wind / precip → multiplier | **NUDGE, in challengers only** | +| **platoon** (`platoonSplits`, `mlbContext`) | yes | handedness advantage | **DISCONNECTED — `mlbContext` has ZERO consumers.** Dead code | +| **opportunity_drift** | yes | recent AB ÷ season AB/game | **NUDGE** (arch-v1 opportunity axis). The one feature with repeated residual signal — and the axis extracts none of it | +| **statcastAggregateService** | yes | 1,354 rows of real mechanism data | **DISCONNECTED from grading.** Required only by `snapshotScheduler` (writer) and `routes/internal` | +| **contact quality** (contact-v1) | yes | season contact profile | **NUDGE** (challenger column, n=1,055, inconclusive) | +| **pitcher stuff / tendencies** | yes | pitch mix, velo, break, arm angle, GB/FB, whiff | **DISCONNECTED.** Loaded per snapshot as `arsenalById`, used only by proj-v1.1 | + +**Not one forward-model component is in the spine. The spine is the counter.** + +## 2. Baseball input coverage — the real distance (STEP 2) + +`statcast_aggregates`: **1,354 rows** (750 pitchers, 604 batters), real Baseball +Savant, free and unlimited. + +| input the forward model needs | status | evidence | +|---|---|---| +| hitter exit velo (avg/max) | **HAVE** | 600/604 batters | +| hitter launch angle | **HAVE** | 600/604 | +| hitter barrel% / hard-hit% | **HAVE** | 600/604 | +| hitter whiff / chase / K / BB | **HAVE** | 602–603/604 | +| **hitter batted-ball type (GB/FB/LD)** | **PARTIAL** | `gb_pct`/`fb_pct` are **0/604 for batters** (populated for pitchers only). The batted-ball feed carries `fb_ld_pct`/`gb_pct_bb` — they land in the `metrics` JSONB, not the typed columns | +| **handedness (bats / throws)** | **HAVE** | 1,354/1,354 — both sides, complete | +| pitcher pitch mix + velo + break | **HAVE** | `pitch_mix` 729/750 | +| pitcher GB/FB/LD | **HAVE** | 713/750 | +| pitcher whiff / chase / barrel & hard-hit allowed | **HAVE** | 713/750 | +| pitcher arm angle | **HAVE** | 713/750 | +| park factors | **HAVE** | `parkBase` / `parkFactors`, per venue | +| weather (temp/wind/precip) | **HAVE** | `wx_forecast` on ledger rows; 5,855 feature rows | +| probable pitcher for tonight | **HAVE** | `probablePitchers` → `mlbStatsAdapter.getScheduleWithPitchers` | +| lineup slot / opportunity | **PARTIAL** | `opportunity_drift` + `ab_per_game` are proxies; `depthChartService` exists but `lineup_role_profiles` / `player_role_profiles` tables are **0 rows** | +| **team defense (OAA / DRS / range)** | **MISSING** | only `opp_rank_stat`, a coarse 0..1 team rank | +| **catcher framing, umpire zone** | **MISSING** | the `ref_*` features are NBA officiating, not MLB umpires | +| **batter-vs-pitcher history** | PARTIAL | `getBatterVsPitcher` exists in the adapter; not in the grade path | + +**Verdict on STEP 2: the inputs are overwhelmingly HAVE.** The forward model is +not blocked on data acquisition. Two real gaps (team defense, catcher/umpire) and +two partials (batter batted-ball columns, lineup slot) — none of which block a +first forward model. + +### But the mechanism data is 13 days stale, and here is why + +`statcast_aggregates.updated_at` max = **2026-07-21**. Today is 2026-08-03. + +**The nightly refresh is unreachable code.** In `src/snapshotScheduler.js`: + +``` +152 const tick = async () => { +158 if (!HOURS_UTC.includes(h)) return; // HOURS_UTC = 14,19,22,1,3 +... +323 const statcastHour = Number(process.env.STATCAST_HOUR_UTC || 11); +324 if (h === statcastHour && process.env.STATCAST !== '0') { +``` + +Line 324 is only reachable when `h ∈ {14,19,22,1,3}`; it tests `h === 11`. **That +branch can never execute at the default.** The data has sat at its Session-68 +backfill ever since. Its failure alert (line 330) is inside the same dead branch, +so it has never been able to warn either — the same *silently guarded-out* shape +as the settlement outage. (Caveat: if `STATCAST_HOUR_UTC` were set in prod to one +of the cron hours it would run; the 13-day staleness says it is not.) + +## 3. Does the design reflect the forward read? (STEP 3) + +**The card has the slots. The engine never fills the matchup ones.** + +`GradeResultCard` already supports `archetypeBlend` (with a literal "Why this +grade:" caption), `propDNA` (reliable/volatile), `statContext`, `vyndrIntel` +(form/usage/**matchup**/rest), `signals`, `killConditions`. The surface was +designed for a scouting report. + +**What actually reaches it** — the complete factor-label vocabulary the "SIGNAL +BREAKDOWN · N FACTORS" section renders: + +``` +l5_hot_vs_line l5_cold_vs_line l20_over_line l20_under_line +l20_contradicts_over/under l5_under_friendly consistency_boom_bust +back_to_back rested_2plus heavy_workload_7d home_game +top_opponent_defense weak_opponent_defense away_vs_top5_defense +ref_foul_high/low ref_pace_adjustment coach_pace_delta +opp_2_starters_out opp_3plus_starters_out rookie/veteran_in_playoffs +``` + +**Every one is a restatement of the counter** — he's been hot against this +number, he's rested, he's home. **Not one names a pitcher, a pitch type, a +handedness edge, a park, or a batted-ball profile.** Several are structurally NBA +(referees, coach pace, starters out, playoffs) inside a baseball product. + +**Design verdict: the product communicates the COUNTER, with the counter's own +inputs relabelled as "signals."** The "why" is currently "because he's been doing +it," which is the one thing a forward projection is supposed to replace. The good +news is that the surface does not need redesigning — it needs feeding. + +## 4. What prior work changes (STEP 4) + +**KEEP — serves the forward model regardless of objective:** + +| work | why it survives | +|---|---| +| Settlement fix (S77) | the measurement floor. Without it nothing is testable at all | +| Clamp + calibration diagnosis (S78) | 20.6% of props pinned to a constant, +3.5pt over-prediction. A forward model inherits this bug unless it is fixed — it is in the *output* stage, not the model | +| Champion decomposition (S78) | tells us precisely what the counter contributes (~all of it from base+recency) — that is the **baseline any forward model must beat** | +| Takeability / book-identity work | market-structure hygiene, objective-independent | +| Ledger + retention infrastructure | the proving ground | +| `opportunity_drift` residual finding | a genuine forward input, wrongly implemented | + +**RE-CHECK — verdicts that answered the narrow question:** + +| verdict | why it needs re-checking | +|---|---| +| **arch-v1 "inconclusive"** | it is explicitly a **market-relative** nudge — the one component genuinely measured on the wrong axis. Its axes (env/matchup/opportunity) should be re-tested as **forward inputs to a projection**, not as nudges to a market-relative probability | +| **proj-v1.1 "reliably worse"** | valid as measured, but it was a *distribution* swap on the same thin inputs (season rate + bounded multiplier). It does not establish that a **matchup-fed** projection is worse — that was never built | +| **hits-v1 "refuted"** | same. It changed the family, not the information. The refutation stands for *that* model and says nothing about a stuff-vs-profile read | +| **"AT CEILING" on runs/walks** | provisional. The ceiling was measured against *features the champion already reads*. It cannot rule out inputs never wired in | +| **Archetype "unmeasurable"** | unchanged — 2 of 41 labels have testable n. Still the honest answer | + +**Nothing gets thrown away. Two verdicts (proj-v1.1, hits-v1) are correctly +scoped to what they tested and simply do not answer the forward question.** + +## 5. STAGED DISTANCE TO PRODUCT (STEP 5) + +### Stage A — a forward baseball model that beats the counter on listed-line accuracy + +**BUILT (more than expected):** all five Statcast feeds ingested and typed; +handedness complete on both sides; arsenal classification (`matchupRead`) written +and unit-tested; park + weather + platoon multipliers written; probable pitchers +resolved per game; archetype classifier complete; the settled ledger that proves +or kills it; a champion baseline decomposed to the number. + +**NEEDED:** +1. Fix the Statcast cron (one guard) — a forward model on 13-day-old mechanism + data is not a forward model. +2. **The actual new thing: a projection whose SPINE is `hitter profile × pitcher + stuff × park/conditions`, with frequency demoted to a prior.** Every input + exists; none is assembled this way. This is the real build. +3. Wire it into the *grade path* (`analyzeViaEngine1`), not a challenger column — + though it should ride as a challenger first to be proven. +4. Fill the two partials: batter GB/FB into typed columns, lineup slot. + +**Honest distance: this is the one genuinely substantial build, and it is a +plumbing-and-modelling job, not a data job.** Main risk is not feasibility but +**sample**: 1,741 settled rows, ~578 on the biggest stat, is thin for validating a +richer model per stat. Expect the verdict to be n-blocked for weeks after it +ships — which is an argument for shipping it soon, not for waiting. + +### Stage B — calibrated + productized (the scouting-report surface) + +**BUILT:** the entire card (`archetypeBlend` + "Why this grade", `propDNA`, +`statContext`, `vyndrIntel.matchup`, signals, kill conditions), tier gating, the +row grammar, the honesty layer. + +**NEEDED:** (a) the clamp/calibration fix — already diagnosed, no new data; +(b) matchup-derived factor labels to replace the `l5_hot_vs_line` vocabulary; +(c) archetype attached at grade time (currently refused there on I/O-cost grounds +that a forward model changes). **Distance: short once Stage A exists** — this is +mostly deleting the counter's vocabulary and passing through what Stage A already +computes. + +### Stage C — same method per sport + +**BUILT:** the per-sport doctrine, the sport-agnostic Bayesian scaffolding, WNBA +settling end-to-end. + +**NEEDED:** a mechanism feed per sport equivalent to Statcast. **This does not +exist for NBA/WNBA in our stack** — `espnStatsAdapter` is box scores, and the +Python `nba_api` service is offline in production. Soccer is quota-blocked on +odds-api (478/500). **Distance: materially further than A or B, and gated on data +we do not currently have.** Baseball is the only sport where the forward model is +buildable today — which is an argument for making it excellent in one sport +first, exactly as the per-sport doctrine says. + +--- + +## 6. The one-sentence answer to "how far" + +**Stage A is one real build away and zero data acquisitions away**; Stage B is +mostly wiring plus an already-diagnosed calibration fix; **Stage C is blocked on +per-sport mechanism data that does not exist in the stack yet.** The reason five +sessions of challengers failed is now legible and it is not the modelling — it is +that every one of them was attached *downstream of the grade*, nudging a counter +instead of replacing it.