Master state board, repo-verified — and a live retention regression
Captures the complete position before the build-process pivot, derived from the repo and live database checks rather than from any summary. Every claim is marked VERIFIED / INFERRED / UNVERIFIED with the check shown. THE HEADLINE IS A PRODUCTION REGRESSION FOUND WHILE VERIFYING. model_snapshots stopped being written on ~2026-08-15: 14,718 rows on 08-12 -> 7,080 -> 378 -> 0 on 08-16 and 08-17. ledger_entries and closing_captures are still writing normally, so this is one write path failing, not a dead cron. Cause, verified three ways: retentionService.js:165 declares chain_shadow on every row; migration 038 was never applied so the column does not exist; and persist() catches the error because retention is best-effort by design. The chain-v1 report flagged this exact sequence as a blocking precondition. The commit was pushed and deployed; the migration was not applied. Consequence: both pending verdicts are frozen. A8 sits at 623 complete pairs (MIN_N 500 met) but only 20 of 40 distinct settled games, and has gained nothing since 08-15. The chain-vs-counter head-to-head has never accrued a row, because its column does not exist either. Four premise corrections the verification forced: - WNBA is BUILT, not ingested. All three WNBA tables are absent; the source is ESPN, not nba_api (WAF-blocked, package not installed); Basketball-Reference cannot serve as-of-date aggregates; wehoop has no lineup/on-off data. - Lineup reconstruction is not pending. It works, 40/40 games to a complete 5-a-side. What is missing is a column to store it in. - WNBA footprint measures 42.6 MB/season, not ~475 MB. The Pro conclusion may still hold, but not from that number. - The chain diagnostic is BOTH a difficulty-correlated component and an unexplained uniform offset, and the correlation is near-mechanical -- it shows the wiring reaches the number, not that the adjustment is right. Also records the fix options for the regression without taking either, since this order is documentation-only. Reverting retentionService.js:165 restores retention immediately and needs no credentials; applying 038 is the complete fix and needs DDL access this environment does not have. Documentation only. No code, schema, served path or accrual changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xcpku8k719mtCWqFKXg287
This commit is contained in:
@@ -0,0 +1,362 @@
|
||||
# VYNDR — MASTER STATE, 2026-08-17
|
||||
|
||||
**This is the board. Every future order reads it first.**
|
||||
|
||||
Derived from the repo and from live database checks on 2026-08-17, not from any
|
||||
summary. Every line is marked **VERIFIED** (checked this session, with the check
|
||||
shown), **INFERRED** (follows from something verified), or **UNVERIFIED** (could
|
||||
not be checked from here — do not act on it as fact).
|
||||
|
||||
Repo HEAD: **`6c34af3`** "checkpoint: chain shadow, WNBA possession feed,
|
||||
baseball chain" — **VERIFIED pushed to `gitea/main`**
|
||||
(`git ls-remote gitea main` → `6c34af34140b6dd…`).
|
||||
|
||||
---
|
||||
|
||||
# 🔴 0. STOP — LIVE PRODUCTION REGRESSION, FOUND THIS SESSION
|
||||
|
||||
**`model_snapshots` (the retention / measurement moat) stopped being written on
|
||||
or about 2026-08-15.** VERIFIED by direct row counts:
|
||||
|
||||
| game_date | model_snapshots | ledger_entries |
|
||||
|---|---|---|
|
||||
| 2026-08-12 | 14,718 | 2,187 |
|
||||
| 2026-08-13 | 12,004 | 1,368 |
|
||||
| 2026-08-14 | 7,080 | 1,834 |
|
||||
| 2026-08-15 | **378** | 1,988 |
|
||||
| 2026-08-16 | **0** | 2,020 |
|
||||
| 2026-08-17 | **0** | 220 |
|
||||
|
||||
The rest of the pipeline is **healthy**: `ledger_entries` is still writing, and
|
||||
`closing_captures` newest `captured_at` = **2026-08-16T23:01:49Z**. So this is
|
||||
not a dead cron — it is one write path failing.
|
||||
|
||||
### Cause — VERIFIED by code + schema check
|
||||
|
||||
1. `src/services/retentionService.js:165` declares **`chain_shadow: null` on
|
||||
every retention row** (added by the chain-v1 order so a PostgREST bulk insert
|
||||
has one consistent shape).
|
||||
2. **Migration `038_chain_shadow.sql` was never applied** — VERIFIED:
|
||||
`select chain_shadow from model_snapshots` → *"column
|
||||
model_snapshots.chain_shadow does not exist"*.
|
||||
3. `retentionService.persist()` catches the error into `out.error`; the caller
|
||||
logs a warning and the snapshot continues, **because retention is
|
||||
best-effort by design.** So every insert fails and nothing pages.
|
||||
|
||||
That is exactly the failure the chain-v1 report flagged as a blocking
|
||||
precondition, verbatim: *"If the column does not exist, every retention insert
|
||||
fails, and retention is best-effort, so it fails SILENTLY — the exact 'retention
|
||||
writes nothing' shape the S64 ops alarm exists to catch."* The commit was pushed
|
||||
and deployed; the migration was not applied.
|
||||
|
||||
### Consequence — the accrual clock is FROZEN
|
||||
|
||||
A8 (§5) and the chain-vs-counter head-to-head both accrue on `model_snapshots`.
|
||||
**Neither has gained a row since 2026-08-15.**
|
||||
|
||||
### The fix is one of two things, and NEITHER was done here
|
||||
|
||||
This order is documentation-only, so nothing was changed. The options:
|
||||
|
||||
- **(a) Apply migration 038.** Correct and complete. Needs DB DDL access, which
|
||||
is unavailable from this environment (§9).
|
||||
- **(b) Revert `retentionService.js:165`** (`chain_shadow: null`). Restores
|
||||
retention immediately with no schema change; the chain shadow then has nowhere
|
||||
to land, which is acceptable because it is served by nothing.
|
||||
|
||||
**(b) is the faster unblock and requires no credentials.** This is the single
|
||||
highest-priority action on the board.
|
||||
|
||||
---
|
||||
|
||||
## 1. STORAGE — and why WNBA forces the Pro decision
|
||||
|
||||
| item | state |
|
||||
|---|---|
|
||||
| `closing_captures` | **1,294,740 rows** — VERIFIED. Down from the ~4.2M peak. |
|
||||
| `closing_captures` where `missed_reason='missed_window'` | **0** — VERIFIED. Fix B2 held; the bleed is stopped AND the historical rows are gone. |
|
||||
| `lock_lines` | **ABSENT** — VERIFIED (*"Could not find the table"*). Dropped, writer retired. |
|
||||
| `ledger_entries` | 30,209 rows — VERIFIED |
|
||||
| `model_snapshots` | 180,410 rows — VERIFIED |
|
||||
| `statcast_history` | 19,798 rows — VERIFIED (point-in-time window accruing) |
|
||||
| `team_defense` | 403 rows — VERIFIED |
|
||||
| **Total DB size vs 500 MB cap** | **UNVERIFIED.** `pg_database_size` needs DDL/SQL access, unavailable from here. Last recorded figure was **406 MB (81%)** after the lock_lines drop. Given `closing_captures` fell from ~4.2M to 1.29M rows since, the true current figure is **probably materially lower** — but that is INFERRED, not measured. **Re-measure before any capacity decision.** |
|
||||
|
||||
### WNBA footprint — MEASURED, and it contradicts the number in the order
|
||||
|
||||
The order states WNBA makes Pro a requirement at **~475 MB/season**. **Measured
|
||||
value is 42.6 MB/season** — an order of magnitude lower.
|
||||
|
||||
```
|
||||
392 bytes/event (typed columns + 24B tuple header + ~40B PK index entry)
|
||||
× 411.9 events/game × 264 games = 42.6 MB per season (0.158 MB per game)
|
||||
```
|
||||
|
||||
Method: 16,114 real events pulled over 2026-08-01..14 through the shipped
|
||||
parser; JSON-serialised at 596 B/event; typed-column factor 0.55 plus Postgres
|
||||
row and index overhead. **VERIFIED** (`scripts/wnba-possession-verify.js`).
|
||||
|
||||
**The Pro conclusion may still be right, but not from this number.** WNBA v1+v2
|
||||
together ≈ 58 MB. Against the last-known 94 MB headroom that is ~62% of what is
|
||||
left — enough to block a second sport and next season, not enough to be a
|
||||
crisis. **Where 475 MB came from is unknown; do not plan against it.**
|
||||
|
||||
---
|
||||
|
||||
## 2. MLB REPAIR — VERIFIED present in the repo
|
||||
|
||||
| item | evidence |
|
||||
|---|---|
|
||||
| Read integrity | `scripts/read-integrity.js` REGISTRY = **34 readers** (VERIFIED by count). `safePaginate`/`pageSafe` used in **26 files**; **16 scripts** export named `READS`. |
|
||||
| Version stamp alignment (A3) | `src/config/modelVersion.js` is the single declaration. Live rows carry **`engine1@2026-08-07-fullwindow`** — VERIFIED from `model_snapshots.model_version`. |
|
||||
| Settlement | `ledger_entries` settled = **27,941**; `model_snapshots` settled = **54,736**. Backlog of unsettled rows with `game_date < 2026-08-16` = **0** — VERIFIED. Settlement is healthy. |
|
||||
| As-of context (A4) | `hitsFactorContext.build(sb,{asOf})` present. |
|
||||
| Factor-input freeze (A5) | `model_snapshots.factor_inputs` column **PRESENT**, **4,527 rows populated** — VERIFIED. A5 is deployed and was accruing. |
|
||||
| Join keys / shadow fire (A6/A7) | `factor_inputs.would_fire` present on **1,445** of 3,389 rows sampled 08-05..08-14 — VERIFIED. The factors that never fired are firing **in shadow**. |
|
||||
| dCLV (B1) | `ledger_entries.dclv_state` populated — VERIFIED. |
|
||||
| **Prod deploy sha** | **UNVERIFIED.** `vyndr.app/api/health` → 404, `www.vyndr.app` → 503. Cannot read the container's sha. INFERRED to be at or near `6c34af3` because (i) it is on `gitea/main` and (ii) the retention failure in §0 requires that commit's code. |
|
||||
|
||||
---
|
||||
|
||||
## 3. THE CHAIN ENGINE — built, shadow, served by nothing
|
||||
|
||||
**Files VERIFIED present:** `chain.js` (325L), `baseballChain.js` (272L),
|
||||
`chainShadow.js` (359L).
|
||||
|
||||
- **`chainFn` slot** added — was described in the header and absent from the code
|
||||
for the module's whole life. Defaults to identity-on-`p`, so prior callers are
|
||||
byte-identical.
|
||||
- **Sign-aware correlation** — clamp `[0,1]` → `[-1,1]`, interpolating toward the
|
||||
Fréchet bound the sign selects. The positive branch is arithmetically
|
||||
unchanged. Basketball's negative usage-competition case was previously
|
||||
*inexpressible*, not merely mismodelled.
|
||||
- **`redistribute` on both readings** + exported `prepareAtoms`, so a
|
||||
redistribution cannot reach one reading and make `selfCheck` flag an
|
||||
inconsistency the model just manufactured.
|
||||
- **MLB chainFn fires** via `skillProjection.paOutcome` → Binomial(PA) —
|
||||
**99.6%** of the board (9,752/9,792), sole refusal reason `no_batter_profile`.
|
||||
- **The PA→game conversion was never broken.** `paDistribution` is a
|
||||
mean-preserving two-point mixture and `atLeast(Binomial(n,p),1)` *is*
|
||||
`1−(1−p)^n` averaged over n. The real defect was the **input E[PA]** — the
|
||||
shadow passed no lineup slot, so every hitter ran on `DEFAULT_PA = 4.1`. Fixed;
|
||||
posted slot now reaches 88.8% of props.
|
||||
|
||||
### DIAGNOSTIC RESULT — state it precisely
|
||||
|
||||
Quintiles of opposing-pitcher K%, over side, n=3,705:
|
||||
|
||||
```
|
||||
Q1 (soft) +0.0749 → Q5 (hard) +0.0256 spread +0.0494 Pearson r = −0.119
|
||||
below-counter share monotone across all five: 27.8 → 30.8 → 34.1 → 35.0 → 38.6
|
||||
```
|
||||
|
||||
**The result is BOTH, and the order's phrasing ("signal") is half of it:**
|
||||
|
||||
1. A **difficulty-correlated component** (~+0.049) — the chain conditions on the
|
||||
matchup.
|
||||
2. A **uniform +0.026 offset surviving into the hardest quintile** — it does not
|
||||
move with the matchup, so it is not conditioning. **Unexplained.** Calling the
|
||||
whole result "signal" buries it.
|
||||
|
||||
And the correlation is **near-mechanical**: the chain reads opposing-pitcher K%
|
||||
directly and the counter reads nothing about the pitcher, so a monotone
|
||||
relationship is close to proof that *the wiring reaches the number* — **not that
|
||||
the adjustment is correct**. Only settled outcomes can say that.
|
||||
|
||||
**`CALIBRATION_DEPLOYED = Object.freeze([])` — VERIFIED at
|
||||
`snapshotService.js:301`. The counter still serves. The grade was frozen
|
||||
throughout, proven byte-identical with the shadow on and off.**
|
||||
|
||||
---
|
||||
|
||||
## 4. WNBA — CORRECTION: built, NOT ingested
|
||||
|
||||
**The order says the possession feed is "ingested". It is not. Nothing WNBA
|
||||
exists in the database.** VERIFIED:
|
||||
|
||||
```
|
||||
wnba_player_game → "Could not find the table" (migration 039 unapplied)
|
||||
wnba_possession_event → "Could not find the table" (migration 040 unapplied)
|
||||
wnba_season_context → "Could not find the table" (migration 041 unapplied)
|
||||
```
|
||||
|
||||
Three further premise corrections, each VERIFIED:
|
||||
|
||||
- **NOT `nba_api playbyplayv3`.** The package is **not installed**
|
||||
(`ModuleNotFoundError`; pinned only in the offline python service), and
|
||||
`stats.wnba.com` is **WAF-blocked** — HTTP 000 on every call after the first;
|
||||
`cdn.wnba.com` returns **403 from `failover-waf.nba.com`**. The source is
|
||||
**ESPN**, with wehoop parquet as the bulk twin.
|
||||
- **`scoremargin` is not a served field** anywhere. It is **derived** from
|
||||
`homeScore − awayScore` (exact, but derived).
|
||||
- **Basketball-Reference cannot serve as-of-date aggregates** — season-to-date
|
||||
as of today, no date parameter. It is captured **dated, forward-only**, and
|
||||
its real value is an **independent second opinion** on usage.
|
||||
- **wehoop has no lineup/on-off data** (14 data dirs, none of them lineup), so
|
||||
it was deliberately **not** given a validation table — that would be a second
|
||||
~30 MB copy of the same stream with no reader.
|
||||
|
||||
### What IS built and measured (in memory, over the live season)
|
||||
|
||||
`espnWnbaAdapter` (377L), `wnbaUsageService` (215L), `wnbaPbpAdapter` (253L),
|
||||
`wnbaLineupState` (197L), `wnbaSeasonContextService` (219L) — all VERIFIED
|
||||
present.
|
||||
|
||||
```
|
||||
16,114 events · 40 games · 402.9/game · 0 fetch errors (08-01..08-14)
|
||||
score_margin / game_seconds_remaining / period 100% each
|
||||
substitutions 2,175 — BOTH players resolved on 2,175 of 2,175
|
||||
lineup fold: 40/40 games resolve to a COMPLETE 5-a-side
|
||||
states: competitive 14,177 · clutch 864 · lopsided 840 · decided 233
|
||||
as-of leakage: NONE at three cutoffs
|
||||
```
|
||||
|
||||
### ⚠️ CORRECTION on "players-on-court NULL"
|
||||
|
||||
The order lists players-on-court as NULL with lineup reconstruction pending.
|
||||
**Lineup reconstruction is BUILT and works** — `wnbaLineupState.foldLineups`
|
||||
folds the on-court five forward from substitution events and resolves **40/40
|
||||
games to a complete 5-a-side**. What is true is that **no column stores it**,
|
||||
because no table exists. It is computed, not persisted.
|
||||
|
||||
**Dormant / honest gaps that remain:**
|
||||
- **Team-level only for redistribution attribution today** — per-player on/off
|
||||
needs the fold joined to outcomes, which needs the table.
|
||||
- The feedback layer is **capable but unproven**: no chainFn consumes it yet.
|
||||
- A real parse bug was found and fixed: ESPN drops the colon under a minute
|
||||
(`"56.2"`, `"5.7"`), so a colon-only clock parser discarded **the final minute
|
||||
of every period** — the clutch/garbage-time window. Coverage 86.6% → 100%.
|
||||
|
||||
**NEXT for WNBA = the chainFn (v3).** Not started.
|
||||
|
||||
---
|
||||
|
||||
## 5. PENDING VERDICTS ON THE ACCRUAL CLOCK
|
||||
|
||||
### A8 — do the MLB hits factors improve the forecast?
|
||||
|
||||
Spec: `specs/a8-shadow-factor-gate.md`, **pre-registered before any accrual
|
||||
existed.** Binding constraint is `MIN_CLUSTERS = 40` distinct settled games, not
|
||||
`MIN_N`.
|
||||
|
||||
**MEASURED 2026-08-17** (sampled 08-05..08-14, 1,000 rows/date cap):
|
||||
|
||||
```
|
||||
rows with factor_inputs 3,389
|
||||
...with would_fire 1,445
|
||||
...AND settled 623 COMPLETE PAIRS (MIN_N 500 → MET)
|
||||
distinct settled games 20 (MIN_CLUSTERS 40 → 50%)
|
||||
```
|
||||
|
||||
**Status: HALFWAY, AND STALLED.** The row bar is met; the cluster bar is at
|
||||
20/40 — and **no new pairs have accrued since 2026-08-15** because of §0. The
|
||||
clock does not restart until retention writes again.
|
||||
|
||||
The order's "~Aug 17" date is **not met** and was never going to be met on
|
||||
clusters; it was a row-count estimate.
|
||||
|
||||
### Chain vs counter — Brier on the `(chain_p, counter_p, outcome)` triples
|
||||
|
||||
**Not accruing at all.** The triple lands in `model_snapshots.chain_shadow`,
|
||||
which **does not exist** (migration 038). Zero rows. Blocked on the same fix.
|
||||
|
||||
**Both verdicts are pre-registered, both wait on settled games, and both are
|
||||
currently gated on a manual "did we hit N" check** — which is exactly what
|
||||
build-process upgrade (c) below removes.
|
||||
|
||||
---
|
||||
|
||||
## 6. THE REMAINING BUILD WAVE (ordered)
|
||||
|
||||
0. **🔴 Restore retention** (§0) — everything downstream is blocked on it.
|
||||
1. **WNBA chainFn (v3)** — `usage × possessions × efficiency`, contested
|
||||
opportunity, team-level feedback.
|
||||
2. **Lineup reconstruction → player-level redistribution** — the fold exists;
|
||||
this is joining it to outcomes and persisting.
|
||||
3. **WNBA shadow** — same fence as MLB: served by nothing.
|
||||
4. **Gate adjudication** — A8 + chain-vs-counter through `factorGate` and the
|
||||
cumulative Bonferroni denominator.
|
||||
5. **NBA** — content swap onto the same machinery; season opens October.
|
||||
6. **Soccer.**
|
||||
7. **Media / design layer** — F5 article media, E16/F8 cards, in-season hub: the
|
||||
surfaces that reflect all sports + correlations.
|
||||
|
||||
---
|
||||
|
||||
## 7. BUILD-PROCESS UPGRADES IDENTIFIED
|
||||
|
||||
- **(a) NEXT — independent verifier on fresh context** for the
|
||||
grade-freeze / 0-leaks invariant. The invariant is currently asserted by tests
|
||||
written by the same pass that wrote the code.
|
||||
**§0 sharpens this:** the failure that actually bit was not a leak but a
|
||||
**silently-failing best-effort write with an unapplied migration**. The
|
||||
verifier should check *deploy preconditions* (does every column the code
|
||||
writes exist?), not only the leak invariant.
|
||||
- **(b) SOON — deterministic gates from the harness.** Promote read-integrity
|
||||
anchors to gates.
|
||||
- **(c) SOON — scheduled loop for the accrual verdicts**, removing the manual
|
||||
"did we hit N dates" clock. §5 shows the cost of the manual clock: the accrual
|
||||
stopped two days ago and nothing noticed.
|
||||
- **(d) NAMED-FUTURE — data-dependency-graph audit** of the codebase: find false
|
||||
sequential dependencies and missing real edges.
|
||||
|
||||
**Add (e), earned this session: a migration/deploy interlock.** A commit whose
|
||||
code writes a column that no applied migration provides must not reach prod. §0
|
||||
is the second time a documented blocking precondition was flagged and then
|
||||
bypassed.
|
||||
|
||||
---
|
||||
|
||||
## 8. STANDING DOCTRINE + INVARIANTS
|
||||
|
||||
1. **The served grade never moves without a human.** Every model change ships
|
||||
behind a shadow first; `CALIBRATION_DEPLOYED = []` and the counter serves.
|
||||
2. **Moat tables are never touched by an automated loop** — `ledger_entries`,
|
||||
`model_snapshots`. (Note the irony of §0: the moat was not *touched*, it was
|
||||
*starved*. "Do not write" is not the only failure mode; "silently stop
|
||||
writing" is the other.)
|
||||
3. **Each sport is its own model.** Machinery shared, content re-derived,
|
||||
factors start at α = 0 and earn their place through the gate.
|
||||
4. **Truth Law** — no fabricated data anywhere; honest-absent beats invented;
|
||||
absence is not zero; label limitations in-band; provisional stays provisional
|
||||
until re-run; verify by inducing the real code path.
|
||||
5. **Nothing is proven.** `proven-status.js` → the PROVEN set is EMPTY.
|
||||
`validatedSkills()` = `{}` for every archetype. The only proven *feature* is
|
||||
the incumbent counter itself.
|
||||
6. **Cumulative Bonferroni** across the programme lifetime; α only ever shrinks;
|
||||
re-tests do not inflate it.
|
||||
|
||||
---
|
||||
|
||||
## 9. THE STANDING ENVIRONMENTAL BLOCKER
|
||||
|
||||
**Four migrations are written and unapplied: 038, 039, 040, 041** — VERIFIED,
|
||||
all four target tables/columns confirmed absent.
|
||||
|
||||
DDL cannot be run from this environment:
|
||||
- `SUPABASE_DB_PASSWORD` fails pooler auth (`aws-1-us-east-1…` → *"password
|
||||
authentication failed"*).
|
||||
- `db.<ref>.supabase.co` resolves **IPv6-only** → `ENETUNREACH` from WSL2.
|
||||
- PostgREST reads and writes work fine — **DDL is the blocker, not writes.**
|
||||
- Local `.env` has the transposed project ref (`zmdnczhtdxcddszxttub`; real is
|
||||
`zmdnczhtdxcddsxzttub`), so scripts need an explicit `SUPABASE_URL=` override.
|
||||
|
||||
**Related IPv6 lesson:** `stats.wnba.com` also fails over IPv6 and works over
|
||||
IPv4 (`curl -4`). Anything reporting a host unreachable from this box should be
|
||||
re-tested with `-4` before it is believed.
|
||||
|
||||
---
|
||||
|
||||
## 10. TWO ANOMALIES WORTH A LOOK (neither investigated)
|
||||
|
||||
- **28 `ledger_entries` rows have `game_date` in the future** (> 2026-08-17),
|
||||
newest **2026-10-20** — VERIFIED count. Could be futures markets or mis-dated
|
||||
rows. Unexamined.
|
||||
- **`model_snapshots.captured_at` returned NULL** on an ordered
|
||||
`limit 1` read. Unexamined; may be an ordering artifact or a genuine null.
|
||||
|
||||
---
|
||||
|
||||
*Board compiled 2026-08-17 from repo HEAD `6c34af3` and live database checks.
|
||||
No code, schema, served path or accrual was changed by the order that produced
|
||||
this document.*
|
||||
Reference in New Issue
Block a user