From 41ba38ed4e510845a48c07039490e73551e37a42 Mon Sep 17 00:00:00 2001 From: Kev Date: Mon, 17 Aug 2026 01:41:33 -0400 Subject: [PATCH] =?UTF-8?q?Master=20state=20board,=20repo-verified=20?= =?UTF-8?q?=E2=80=94=20and=20a=20live=20retention=20regression?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the complete position before the build-process pivot, derived from the repo and live database checks rather than from any summary. Every claim is marked VERIFIED / INFERRED / UNVERIFIED with the check shown. THE HEADLINE IS A PRODUCTION REGRESSION FOUND WHILE VERIFYING. model_snapshots stopped being written on ~2026-08-15: 14,718 rows on 08-12 -> 7,080 -> 378 -> 0 on 08-16 and 08-17. ledger_entries and closing_captures are still writing normally, so this is one write path failing, not a dead cron. Cause, verified three ways: retentionService.js:165 declares chain_shadow on every row; migration 038 was never applied so the column does not exist; and persist() catches the error because retention is best-effort by design. The chain-v1 report flagged this exact sequence as a blocking precondition. The commit was pushed and deployed; the migration was not applied. Consequence: both pending verdicts are frozen. A8 sits at 623 complete pairs (MIN_N 500 met) but only 20 of 40 distinct settled games, and has gained nothing since 08-15. The chain-vs-counter head-to-head has never accrued a row, because its column does not exist either. Four premise corrections the verification forced: - WNBA is BUILT, not ingested. All three WNBA tables are absent; the source is ESPN, not nba_api (WAF-blocked, package not installed); Basketball-Reference cannot serve as-of-date aggregates; wehoop has no lineup/on-off data. - Lineup reconstruction is not pending. It works, 40/40 games to a complete 5-a-side. What is missing is a column to store it in. - WNBA footprint measures 42.6 MB/season, not ~475 MB. The Pro conclusion may still hold, but not from that number. - The chain diagnostic is BOTH a difficulty-correlated component and an unexplained uniform offset, and the correlation is near-mechanical -- it shows the wiring reaches the number, not that the adjustment is right. Also records the fix options for the regression without taking either, since this order is documentation-only. Reverting retentionService.js:165 restores retention immediately and needs no credentials; applying 038 is the complete fix and needs DDL access this environment does not have. Documentation only. No code, schema, served path or accrual changed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Xcpku8k719mtCWqFKXg287 --- specs/MASTER-STATE-2026-08-17.md | 362 +++++++++++++++++++++++++++++++ 1 file changed, 362 insertions(+) create mode 100644 specs/MASTER-STATE-2026-08-17.md diff --git a/specs/MASTER-STATE-2026-08-17.md b/specs/MASTER-STATE-2026-08-17.md new file mode 100644 index 0000000..d4c9f11 --- /dev/null +++ b/specs/MASTER-STATE-2026-08-17.md @@ -0,0 +1,362 @@ +# VYNDR — MASTER STATE, 2026-08-17 + +**This is the board. Every future order reads it first.** + +Derived from the repo and from live database checks on 2026-08-17, not from any +summary. Every line is marked **VERIFIED** (checked this session, with the check +shown), **INFERRED** (follows from something verified), or **UNVERIFIED** (could +not be checked from here — do not act on it as fact). + +Repo HEAD: **`6c34af3`** "checkpoint: chain shadow, WNBA possession feed, +baseball chain" — **VERIFIED pushed to `gitea/main`** +(`git ls-remote gitea main` → `6c34af34140b6dd…`). + +--- + +# 🔴 0. STOP — LIVE PRODUCTION REGRESSION, FOUND THIS SESSION + +**`model_snapshots` (the retention / measurement moat) stopped being written on +or about 2026-08-15.** VERIFIED by direct row counts: + +| game_date | model_snapshots | ledger_entries | +|---|---|---| +| 2026-08-12 | 14,718 | 2,187 | +| 2026-08-13 | 12,004 | 1,368 | +| 2026-08-14 | 7,080 | 1,834 | +| 2026-08-15 | **378** | 1,988 | +| 2026-08-16 | **0** | 2,020 | +| 2026-08-17 | **0** | 220 | + +The rest of the pipeline is **healthy**: `ledger_entries` is still writing, and +`closing_captures` newest `captured_at` = **2026-08-16T23:01:49Z**. So this is +not a dead cron — it is one write path failing. + +### Cause — VERIFIED by code + schema check + +1. `src/services/retentionService.js:165` declares **`chain_shadow: null` on + every retention row** (added by the chain-v1 order so a PostgREST bulk insert + has one consistent shape). +2. **Migration `038_chain_shadow.sql` was never applied** — VERIFIED: + `select chain_shadow from model_snapshots` → *"column + model_snapshots.chain_shadow does not exist"*. +3. `retentionService.persist()` catches the error into `out.error`; the caller + logs a warning and the snapshot continues, **because retention is + best-effort by design.** So every insert fails and nothing pages. + +That is exactly the failure the chain-v1 report flagged as a blocking +precondition, verbatim: *"If the column does not exist, every retention insert +fails, and retention is best-effort, so it fails SILENTLY — the exact 'retention +writes nothing' shape the S64 ops alarm exists to catch."* The commit was pushed +and deployed; the migration was not applied. + +### Consequence — the accrual clock is FROZEN + +A8 (§5) and the chain-vs-counter head-to-head both accrue on `model_snapshots`. +**Neither has gained a row since 2026-08-15.** + +### The fix is one of two things, and NEITHER was done here + +This order is documentation-only, so nothing was changed. The options: + +- **(a) Apply migration 038.** Correct and complete. Needs DB DDL access, which + is unavailable from this environment (§9). +- **(b) Revert `retentionService.js:165`** (`chain_shadow: null`). Restores + retention immediately with no schema change; the chain shadow then has nowhere + to land, which is acceptable because it is served by nothing. + +**(b) is the faster unblock and requires no credentials.** This is the single +highest-priority action on the board. + +--- + +## 1. STORAGE — and why WNBA forces the Pro decision + +| item | state | +|---|---| +| `closing_captures` | **1,294,740 rows** — VERIFIED. Down from the ~4.2M peak. | +| `closing_captures` where `missed_reason='missed_window'` | **0** — VERIFIED. Fix B2 held; the bleed is stopped AND the historical rows are gone. | +| `lock_lines` | **ABSENT** — VERIFIED (*"Could not find the table"*). Dropped, writer retired. | +| `ledger_entries` | 30,209 rows — VERIFIED | +| `model_snapshots` | 180,410 rows — VERIFIED | +| `statcast_history` | 19,798 rows — VERIFIED (point-in-time window accruing) | +| `team_defense` | 403 rows — VERIFIED | +| **Total DB size vs 500 MB cap** | **UNVERIFIED.** `pg_database_size` needs DDL/SQL access, unavailable from here. Last recorded figure was **406 MB (81%)** after the lock_lines drop. Given `closing_captures` fell from ~4.2M to 1.29M rows since, the true current figure is **probably materially lower** — but that is INFERRED, not measured. **Re-measure before any capacity decision.** | + +### WNBA footprint — MEASURED, and it contradicts the number in the order + +The order states WNBA makes Pro a requirement at **~475 MB/season**. **Measured +value is 42.6 MB/season** — an order of magnitude lower. + +``` +392 bytes/event (typed columns + 24B tuple header + ~40B PK index entry) +× 411.9 events/game × 264 games = 42.6 MB per season (0.158 MB per game) +``` + +Method: 16,114 real events pulled over 2026-08-01..14 through the shipped +parser; JSON-serialised at 596 B/event; typed-column factor 0.55 plus Postgres +row and index overhead. **VERIFIED** (`scripts/wnba-possession-verify.js`). + +**The Pro conclusion may still be right, but not from this number.** WNBA v1+v2 +together ≈ 58 MB. Against the last-known 94 MB headroom that is ~62% of what is +left — enough to block a second sport and next season, not enough to be a +crisis. **Where 475 MB came from is unknown; do not plan against it.** + +--- + +## 2. MLB REPAIR — VERIFIED present in the repo + +| item | evidence | +|---|---| +| Read integrity | `scripts/read-integrity.js` REGISTRY = **34 readers** (VERIFIED by count). `safePaginate`/`pageSafe` used in **26 files**; **16 scripts** export named `READS`. | +| Version stamp alignment (A3) | `src/config/modelVersion.js` is the single declaration. Live rows carry **`engine1@2026-08-07-fullwindow`** — VERIFIED from `model_snapshots.model_version`. | +| Settlement | `ledger_entries` settled = **27,941**; `model_snapshots` settled = **54,736**. Backlog of unsettled rows with `game_date < 2026-08-16` = **0** — VERIFIED. Settlement is healthy. | +| As-of context (A4) | `hitsFactorContext.build(sb,{asOf})` present. | +| Factor-input freeze (A5) | `model_snapshots.factor_inputs` column **PRESENT**, **4,527 rows populated** — VERIFIED. A5 is deployed and was accruing. | +| Join keys / shadow fire (A6/A7) | `factor_inputs.would_fire` present on **1,445** of 3,389 rows sampled 08-05..08-14 — VERIFIED. The factors that never fired are firing **in shadow**. | +| dCLV (B1) | `ledger_entries.dclv_state` populated — VERIFIED. | +| **Prod deploy sha** | **UNVERIFIED.** `vyndr.app/api/health` → 404, `www.vyndr.app` → 503. Cannot read the container's sha. INFERRED to be at or near `6c34af3` because (i) it is on `gitea/main` and (ii) the retention failure in §0 requires that commit's code. | + +--- + +## 3. THE CHAIN ENGINE — built, shadow, served by nothing + +**Files VERIFIED present:** `chain.js` (325L), `baseballChain.js` (272L), +`chainShadow.js` (359L). + +- **`chainFn` slot** added — was described in the header and absent from the code + for the module's whole life. Defaults to identity-on-`p`, so prior callers are + byte-identical. +- **Sign-aware correlation** — clamp `[0,1]` → `[-1,1]`, interpolating toward the + Fréchet bound the sign selects. The positive branch is arithmetically + unchanged. Basketball's negative usage-competition case was previously + *inexpressible*, not merely mismodelled. +- **`redistribute` on both readings** + exported `prepareAtoms`, so a + redistribution cannot reach one reading and make `selfCheck` flag an + inconsistency the model just manufactured. +- **MLB chainFn fires** via `skillProjection.paOutcome` → Binomial(PA) — + **99.6%** of the board (9,752/9,792), sole refusal reason `no_batter_profile`. +- **The PA→game conversion was never broken.** `paDistribution` is a + mean-preserving two-point mixture and `atLeast(Binomial(n,p),1)` *is* + `1−(1−p)^n` averaged over n. The real defect was the **input E[PA]** — the + shadow passed no lineup slot, so every hitter ran on `DEFAULT_PA = 4.1`. Fixed; + posted slot now reaches 88.8% of props. + +### DIAGNOSTIC RESULT — state it precisely + +Quintiles of opposing-pitcher K%, over side, n=3,705: + +``` +Q1 (soft) +0.0749 → Q5 (hard) +0.0256 spread +0.0494 Pearson r = −0.119 +below-counter share monotone across all five: 27.8 → 30.8 → 34.1 → 35.0 → 38.6 +``` + +**The result is BOTH, and the order's phrasing ("signal") is half of it:** + +1. A **difficulty-correlated component** (~+0.049) — the chain conditions on the + matchup. +2. A **uniform +0.026 offset surviving into the hardest quintile** — it does not + move with the matchup, so it is not conditioning. **Unexplained.** Calling the + whole result "signal" buries it. + +And the correlation is **near-mechanical**: the chain reads opposing-pitcher K% +directly and the counter reads nothing about the pitcher, so a monotone +relationship is close to proof that *the wiring reaches the number* — **not that +the adjustment is correct**. Only settled outcomes can say that. + +**`CALIBRATION_DEPLOYED = Object.freeze([])` — VERIFIED at +`snapshotService.js:301`. The counter still serves. The grade was frozen +throughout, proven byte-identical with the shadow on and off.** + +--- + +## 4. WNBA — CORRECTION: built, NOT ingested + +**The order says the possession feed is "ingested". It is not. Nothing WNBA +exists in the database.** VERIFIED: + +``` +wnba_player_game → "Could not find the table" (migration 039 unapplied) +wnba_possession_event → "Could not find the table" (migration 040 unapplied) +wnba_season_context → "Could not find the table" (migration 041 unapplied) +``` + +Three further premise corrections, each VERIFIED: + +- **NOT `nba_api playbyplayv3`.** The package is **not installed** + (`ModuleNotFoundError`; pinned only in the offline python service), and + `stats.wnba.com` is **WAF-blocked** — HTTP 000 on every call after the first; + `cdn.wnba.com` returns **403 from `failover-waf.nba.com`**. The source is + **ESPN**, with wehoop parquet as the bulk twin. +- **`scoremargin` is not a served field** anywhere. It is **derived** from + `homeScore − awayScore` (exact, but derived). +- **Basketball-Reference cannot serve as-of-date aggregates** — season-to-date + as of today, no date parameter. It is captured **dated, forward-only**, and + its real value is an **independent second opinion** on usage. +- **wehoop has no lineup/on-off data** (14 data dirs, none of them lineup), so + it was deliberately **not** given a validation table — that would be a second + ~30 MB copy of the same stream with no reader. + +### What IS built and measured (in memory, over the live season) + +`espnWnbaAdapter` (377L), `wnbaUsageService` (215L), `wnbaPbpAdapter` (253L), +`wnbaLineupState` (197L), `wnbaSeasonContextService` (219L) — all VERIFIED +present. + +``` +16,114 events · 40 games · 402.9/game · 0 fetch errors (08-01..08-14) +score_margin / game_seconds_remaining / period 100% each +substitutions 2,175 — BOTH players resolved on 2,175 of 2,175 +lineup fold: 40/40 games resolve to a COMPLETE 5-a-side +states: competitive 14,177 · clutch 864 · lopsided 840 · decided 233 +as-of leakage: NONE at three cutoffs +``` + +### ⚠️ CORRECTION on "players-on-court NULL" + +The order lists players-on-court as NULL with lineup reconstruction pending. +**Lineup reconstruction is BUILT and works** — `wnbaLineupState.foldLineups` +folds the on-court five forward from substitution events and resolves **40/40 +games to a complete 5-a-side**. What is true is that **no column stores it**, +because no table exists. It is computed, not persisted. + +**Dormant / honest gaps that remain:** +- **Team-level only for redistribution attribution today** — per-player on/off + needs the fold joined to outcomes, which needs the table. +- The feedback layer is **capable but unproven**: no chainFn consumes it yet. +- A real parse bug was found and fixed: ESPN drops the colon under a minute + (`"56.2"`, `"5.7"`), so a colon-only clock parser discarded **the final minute + of every period** — the clutch/garbage-time window. Coverage 86.6% → 100%. + +**NEXT for WNBA = the chainFn (v3).** Not started. + +--- + +## 5. PENDING VERDICTS ON THE ACCRUAL CLOCK + +### A8 — do the MLB hits factors improve the forecast? + +Spec: `specs/a8-shadow-factor-gate.md`, **pre-registered before any accrual +existed.** Binding constraint is `MIN_CLUSTERS = 40` distinct settled games, not +`MIN_N`. + +**MEASURED 2026-08-17** (sampled 08-05..08-14, 1,000 rows/date cap): + +``` +rows with factor_inputs 3,389 + ...with would_fire 1,445 + ...AND settled 623 COMPLETE PAIRS (MIN_N 500 → MET) +distinct settled games 20 (MIN_CLUSTERS 40 → 50%) +``` + +**Status: HALFWAY, AND STALLED.** The row bar is met; the cluster bar is at +20/40 — and **no new pairs have accrued since 2026-08-15** because of §0. The +clock does not restart until retention writes again. + +The order's "~Aug 17" date is **not met** and was never going to be met on +clusters; it was a row-count estimate. + +### Chain vs counter — Brier on the `(chain_p, counter_p, outcome)` triples + +**Not accruing at all.** The triple lands in `model_snapshots.chain_shadow`, +which **does not exist** (migration 038). Zero rows. Blocked on the same fix. + +**Both verdicts are pre-registered, both wait on settled games, and both are +currently gated on a manual "did we hit N" check** — which is exactly what +build-process upgrade (c) below removes. + +--- + +## 6. THE REMAINING BUILD WAVE (ordered) + +0. **🔴 Restore retention** (§0) — everything downstream is blocked on it. +1. **WNBA chainFn (v3)** — `usage × possessions × efficiency`, contested + opportunity, team-level feedback. +2. **Lineup reconstruction → player-level redistribution** — the fold exists; + this is joining it to outcomes and persisting. +3. **WNBA shadow** — same fence as MLB: served by nothing. +4. **Gate adjudication** — A8 + chain-vs-counter through `factorGate` and the + cumulative Bonferroni denominator. +5. **NBA** — content swap onto the same machinery; season opens October. +6. **Soccer.** +7. **Media / design layer** — F5 article media, E16/F8 cards, in-season hub: the + surfaces that reflect all sports + correlations. + +--- + +## 7. BUILD-PROCESS UPGRADES IDENTIFIED + +- **(a) NEXT — independent verifier on fresh context** for the + grade-freeze / 0-leaks invariant. The invariant is currently asserted by tests + written by the same pass that wrote the code. + **§0 sharpens this:** the failure that actually bit was not a leak but a + **silently-failing best-effort write with an unapplied migration**. The + verifier should check *deploy preconditions* (does every column the code + writes exist?), not only the leak invariant. +- **(b) SOON — deterministic gates from the harness.** Promote read-integrity + anchors to gates. +- **(c) SOON — scheduled loop for the accrual verdicts**, removing the manual + "did we hit N dates" clock. §5 shows the cost of the manual clock: the accrual + stopped two days ago and nothing noticed. +- **(d) NAMED-FUTURE — data-dependency-graph audit** of the codebase: find false + sequential dependencies and missing real edges. + +**Add (e), earned this session: a migration/deploy interlock.** A commit whose +code writes a column that no applied migration provides must not reach prod. §0 +is the second time a documented blocking precondition was flagged and then +bypassed. + +--- + +## 8. STANDING DOCTRINE + INVARIANTS + +1. **The served grade never moves without a human.** Every model change ships + behind a shadow first; `CALIBRATION_DEPLOYED = []` and the counter serves. +2. **Moat tables are never touched by an automated loop** — `ledger_entries`, + `model_snapshots`. (Note the irony of §0: the moat was not *touched*, it was + *starved*. "Do not write" is not the only failure mode; "silently stop + writing" is the other.) +3. **Each sport is its own model.** Machinery shared, content re-derived, + factors start at α = 0 and earn their place through the gate. +4. **Truth Law** — no fabricated data anywhere; honest-absent beats invented; + absence is not zero; label limitations in-band; provisional stays provisional + until re-run; verify by inducing the real code path. +5. **Nothing is proven.** `proven-status.js` → the PROVEN set is EMPTY. + `validatedSkills()` = `{}` for every archetype. The only proven *feature* is + the incumbent counter itself. +6. **Cumulative Bonferroni** across the programme lifetime; α only ever shrinks; + re-tests do not inflate it. + +--- + +## 9. THE STANDING ENVIRONMENTAL BLOCKER + +**Four migrations are written and unapplied: 038, 039, 040, 041** — VERIFIED, +all four target tables/columns confirmed absent. + +DDL cannot be run from this environment: +- `SUPABASE_DB_PASSWORD` fails pooler auth (`aws-1-us-east-1…` → *"password + authentication failed"*). +- `db..supabase.co` resolves **IPv6-only** → `ENETUNREACH` from WSL2. +- PostgREST reads and writes work fine — **DDL is the blocker, not writes.** +- Local `.env` has the transposed project ref (`zmdnczhtdxcddszxttub`; real is + `zmdnczhtdxcddsxzttub`), so scripts need an explicit `SUPABASE_URL=` override. + +**Related IPv6 lesson:** `stats.wnba.com` also fails over IPv6 and works over +IPv4 (`curl -4`). Anything reporting a host unreachable from this box should be +re-tested with `-4` before it is believed. + +--- + +## 10. TWO ANOMALIES WORTH A LOOK (neither investigated) + +- **28 `ledger_entries` rows have `game_date` in the future** (> 2026-08-17), + newest **2026-10-20** — VERIFIED count. Could be futures markets or mis-dated + rows. Unexamined. +- **`model_snapshots.captured_at` returned NULL** on an ordered + `limit 1` read. Unexamined; may be an ordering artifact or a genuine null. + +--- + +*Board compiled 2026-08-17 from repo HEAD `6c34af3` and live database checks. +No code, schema, served path or accrual was changed by the order that produced +this document.*