11277a1b99359c8f6dc6ab10c5e0df90e6f9062a
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 builtbykev
|
11277a1b99 |
Materialization truth: a completed write is not a complete cohort
Transport truth says every intended write succeeded. Materialization truth says
every identity that should exist actually exists. The retention writer could
only report the first, and the gap is not theoretical.
THE CONFLICT IDENTITY, traced to the real index:
model_snapshots_cycle_prop_uniq UNIQUE (snapshot_id, player_key, stat, line, side)
written by `upsert(..., { onConflict: same five columns, ignoreDuplicates: true })`.
Measured in production: no key column is ever NULL (0 of 328,262 rows), so
NULLS DISTINCT never applies and the identity is plain column equality. `line`
is an unconstrained numeric, so identity normalises it — 0.5 in and "0.50" out
must not read as two identities for one stored row.
PRIOR-CYCLE COLLISION IS IMPOSSIBLE. `snapshot_id` is in the identity and is a
fresh UUID per cycle, so no row can be suppressed by an earlier cycle.
Append-only chronology across cycles is safe, and `captured_at` is not in the
identity, so a cohort cannot be split by timing.
INTRA-CYCLE COLLISION IS REAL, AND WE CAUSED IT. `canonical_event_id` is NOT in
the identity. A doubleheader — same hitter, same stat, same line, two genuinely
different games — is ONE identity. Demonstrated through the real collector: 4
outbound rows, 2 distinct identities, 2 rows discarded by ignoreDuplicates with
no error, `written` counting all 4 and the terminal status reading COMPLETE.
Before event-aware dedupe the second game was dropped before grading, so the
collision could not arise; that fix moved the loss downstream into retention.
The conflict identity is NOT changed here — that is a separate decision with its
own before/after. This makes the loss visible instead of silent.
EXPECTED vs ACTUAL. `expectedMaterialization(rows)` derives the identity set
from the FINAL outbound payload using the exact database identity — never from
`attempted`, which counts rows sent, not identities that can exist.
`reconcileMaterialization` compares SETS, not counts: two sets of equal size can
still differ, and a cohort that swapped one identity for another passes every
count test ever written. A collision passes set equality by construction (the
discarded row was never in the expected set) while real rows were lost, so
collision_count > 0 fails the cohort on its own.
A cohort is evidence-complete only when transport is COMPLETE, missing = 0,
extra = 0, and collisions = 0.
OBSERVABILITY stayed minimal. `last_retention` was already PER SPORT (a Map
keyed by sport), so no fix was needed there and the route is UNCHANGED — the new
fields ride the existing entry: outbound_rows, expected_materialized_count,
outbound_collision_count, expected_identity_digest. Counts and a digest only,
never the identities, which carry player names. The expected set is the one
materialization fact unrecoverable from the database afterwards, which is why it
is the only thing recorded at runtime.
A collision leaves transport COMPLETE, so the existing failure alert could never
see it. It now has its own high-severity alert naming the counts, the cycle and
the build, and says the cohort is not evidence-complete.
Seven teeth, each injection verified present, against a GREEN baseline of 63:
1 attempted===written as evidence completeness -> 1 fail
2 COUNT(*) equality instead of set equality -> 1 fail
3 snapshot_id dropped from expected identity -> 4 fail
4 unexpected collision allowed to qualify -> 1 fail
5 single global last_retention slot -> 2 fail
6 partial chunk failure treated as usable -> 3 fail
7 collision loses its announcement -> 1 fail
Restored byte-identically (retention 742f116473d97f49, snapshot 81129facbabeb280).
Three brittle assertions repaired, with the reason recorded: two windowed on a
byte count that a neighbouring block outgrew — a test failing because of its
neighbour, not its subject — now windowed to syntactic landmarks; and one
counted TERMINAL.COMPLETE occurrences, which a legitimate comparison
incremented. It now asserts one DECISION and one READ.
persist() and createCollector are BYTE-IDENTICAL. onConflict and
ignoreDuplicates appear in the diff only as prose. Model, event, ledger,
calibration, chain, lineage config, and the status route: UNCHANGED. Zero
lineage/publication files, zero cacheSet changes, zero web paths. Lineage OFF.
Schema contract unchanged: release 64, prod 67, prod-only 3 (debt, not
authorized), missing in prod 0.
384 suites / 5,144 tests pass. web tsc exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
|
||
|
builtbykev
|
9809626c99 |
Retention completion: a cohort is complete only when the writer says N of N
The previous bug made the recorder write nothing. The dangerous successor is a
recorder that writes half and looks healthy: persist() writes in chunks of 250
and STOPS AT THE FIRST FAILED CHUNK, so chunks committed before the failure are
already durable. Rows exist under the snapshot_id, captured_at is uniform, Redis
kept working — and the cohort is short.
So row presence was never completion evidence, and neither was a matching
timestamp. Completeness is now proven by the writer or not at all.
TERMINAL RETENTION STATES (retentionService.classifyPersist):
NOTHING_TO_PERSIST attempted 0 — a refusal-only slate is still a cycle
SKIPPED_NO_DATABASE no database configured; not a failure
COMPLETE attempted > 0, written === attempted, no error
FAILED_ZERO_WRITE written === 0 — first chunk failed
FAILED_PARTIAL 0 < written < attempted — a later chunk failed
FAILED_UNRESOLVED_ERROR counts look complete but an error is unresolved;
unreachable through today's loop, and kept because
the alternative is reporting COMPLETE holding an error
The invariant: any written < attempted with attempted > 0 is a FAILED cycle. A
partial cohort is never degraded success.
classifyPersist reads the EXACT persist() result and refuses anything else — it
never recomputes attempted or written, because a second calculation could
disagree with the writer and then the status would describe a cycle that did not
happen. persist() itself is byte-identical to
|
||
|
builtbykev
|
35da190f2c |
Retention hotfix: drop published_side, derive the schema contract, break the silence
`createCollector.onPublished` set `published_side` beside `published`.
`published_side` is not a model_snapshots column. supabase-js declares the
UNION of row keys in the `columns=` parameter, so one invalid key made
PostgREST reject the ENTIRE batch with a 400 — every sport, every cycle.
Retention is best-effort, so nothing surfaced. Confirmed in edge logs.
The field was redundant as well as invalid: `side` is already on the row.
Deleted rather than added to the schema — a column would preserve an
accidental artifact.
Three things missed it, and each is now closed:
1. WRONG SHAPE INSPECTED. The manual check sampled the collector after
onGraded only and never called onPublished, so the offending key was
not yet on the row. It read a pre-publication shape and reported the
final outbound shape as clean. The new test captures the array actually
handed to .upsert(), after the full production call order.
2. NO CONTRACT. Every retention test injects a permissive fake client that
accepts any column set, so 381 suites proved the logic and never once
compared a row against the database. The contract is now DERIVED — the
migration chain applied to a disposable postgres, read out of
information_schema (scripts/generate-schema-contract.js). A
hand-maintained list would be a second opinion about the schema, and a
second opinion is what let this through. scripts/verify-schema-contract.js
checks the contract still describes a live database.
3. SILENT FAILURE. A failed batch reached one console.log. It now emits a
high-severity structured event carrying sport, snapshot id, stage,
error, code_sha and timestamp. Best-effort semantics are unchanged —
the product continues and says so — but the failure is observable.
`skipped` (no database configured) is not a failure and does not alert.
Teeth, each with the injection verified present before the run:
- published_side back into the final payload -> 4 tests fail; restored
byte-identically (sha 6a0ced7c52134135 both sides)
- settled_at (a REAL contract column) -> accepted, so the guard
discriminates by contract membership, not by novelty
- alert block deleted -> 3 tests fail; restored byte-identically
Model and product behaviour untouched: analyzeViaEngine1,
probabilityEstimator, gradeSlateService, lineageCanaryConfig all unchanged.
Lineage stays OFF. Net source change is one behavioural line plus the alert.
382 suites / 5,094 tests pass. web tsc exit 0 (zero web paths touched).
Measurement blackout recorded, NOT backfilled: last good retention write
2026-08-27T19:08:32Z;
|
||
|
builtbykev
|
352016790a |
MLB canonical event identity, impossible-binding refusal, event-aware dedupe, publication commit
Release-isolated slice built from
|
||
|
builtbykev
|
6c34af3414 |
checkpoint: chain shadow, WNBA possession feed, baseball chain
Backup commit of uncommitted working-tree state found during Legion recon (Tony resurrection, STEP 0). This work existed only on the laptop disk. - chain shadow accrual + probe script (038_chain_shadow.sql) - WNBA possession feed: ESPN adapter, usage service, verify script (039_wnba_player_game.sql) - baseball chain - retention/snapshot service updates, tableKeys, matchupKeys - specs: chain-v1, wnba-possession-feed, wnba-source-survey - unit tests for the above Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QnvJAkC3h5QGmb6dipoiWn |
||
|
builtbykev
|
f61ec6b391 |
Read integrity, as-of context, and the shadow matchup resolve (A1-A7)
Seven orders of measurement-first repair. The served grade does not move. A0/A1 — the unordered page walk returned the right COUNT and the wrong ROWS: 410-617 of 2,490 duplicated with an equal number never returned, while rows.length matched the server exactly. safePaginate orders on a real unique key, verifies the tuple at runtime, and THROWS on a query error instead of treating it as end-of-data. Both hits PROVES are withdrawn: they were drawn through that reader, and defense_by_direction's distinct-n was likely below the gate floor all along. A2/A2b — rolled across every reader: 11 FAIL -> 0. Composite keys pulled from pg_index (the context tables are dated-composite and had no single unique column). The unordered helper is deleted, not parked. A3 — ledgerService and retentionService defaulted the SAME env var to DIFFERENT versions, so no ledger row ever carried the marker eligibility requires. One source now. model_snapshots settlement moved onto the cron: 15,484 -> 28,894 settled, repaired-champion 0 -> 7,556. A4 — hitsFactorContext takes an as-of cutoff. Refusal over reconstruction: no row at-or-before the date means the factor does not apply, never the nearest row. Live path unchanged, proven 400/400 on real rows. A5 — factor_inputs freezes what the factor READ, never the multiplier, so an audit can recompute and check. It also recorded the finding: the three hits factors have NEVER fired. prop.opponent and prop.opposing_pitcher are read by the resolver and written by nothing. A6/A7 — matchupKeys resolves those keys from the posted lineup plus the schedule's probable pitchers, and fires the factors into a SHADOW freeze: 248 fires on 308 props, 245 of which would move the grade. The served forecast is untouched. specs/a8-shadow-factor-gate.md pre-registers the test that decides whether they ever go live. Nothing is turned on. CALIBRATION_DEPLOYED stays []. Both verdicts stay withdrawn. 4,772 tests / 371 suites green, web build exit 0, read-integrity harness 34/34. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
builtbykev
|
494c83cf76 |
Hunt the window-bug class: three more paths, and the forward re-audit rule
in code PHASE 0 — getStatRows is the single base-rate path, so every branch is audited, plus the feature builders since l20_avg is the season reference projectionFor reads: getStatRows MLB -> estimator base fullLog CORRECT ( |
||
|
builtbykev
|
1bcdd8b305 |
Fix the ROOT: bind props to their real game, never date by the grade clock
Order 1.5 Phase 1. PropLine emits NO commence_time (grep-verified: zero hits in proplineAdapter), so ledgerService's `dateET(prop.game_time) || dateET(gradedTs)` always fell through to the GRADE timestamp — and a 01:00/03:00 UTC snapshot is 21:00/23:00 ET the PREVIOUS day. Tonight's props were filed under yesterday, settlement correctly found no game there, and Order 1's void logic turned that into 64 destroyed results. gameBinder.attachGameTimes() now matches every prop to a scheduled game by TEAMS across the plausible ET window (grade date, +1, -1) and attaches the GAME'S OWN time/date/id. It runs in snapshotService before grading and before the ledger write, so ledger, retention and settlement all inherit the correct date from one place. HARD CONTRACT: an unbindable prop returns NOTHING. ledgerService no longer has a grade-clock fallback — a row with no real game time is SKIPPED and counted, because a mis-dated row is fabricated data and the ledger holds real values or nothing. A slate that binds nothing pages. DOUBLEHEADERS are reported, never guessed: two games with the same teams on one date mark the binding `ambiguous` so settlement can decline rather than attribute a prop to the wrong game. (Real example already in the data: mlb:2026-07-11:MilwaukeeBrewers@PittsburghPirates(Game1).) Also fixes retention, which had the SAME bug from last night — I had dated model_snapshots rows with the snapshot clock. Rows now take the ET date of the bound game_time. Suite 282/3381 green, build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA |
||
|
builtbykev
|
5a5e37e32e |
Retention: fill enrichment fields + page on a zero-write slot
PHASE 1 — cron capture needed NO wiring. Verified in code: the scheduler tick calls runAll = snapshotService.runAllSnapshots, which loops runSnapshot per sport, which already carries the onGraded -> retention hook. The scheduled path and the manual path are the SAME function. The reason no cron cycle had been captured is simply that no slot has fired since retention deployed (slots are 14/19/22/1/3 UTC; retention landed ~02:55). Induced proof follows the deploy. PHASE 2 — archetype/team/opponent were permanently null because retention persisted at GRADE time, before enrichment attaches them. Retention still COLLECTS at grade time (the only moment the feature vector exists) but now PERSISTS after enrichment, merging those three fields via retentionService.mergeEnrichment. The merge is pure and fills ONLY those three fields — features and every model output are grade-time values and must never be rewritten by enrichment; a test asserts that. Unmatched rows (refusals not in the enriched slate) keep nulls rather than guesses. The empty-slate early return now persists too: a refusal-only slate is still history worth keeping. PHASE 3 — ZERO-WRITE ALARM. opsWatch.retentionZeroWriteAlarm pages at missed-snapshot severity when a slot GRADED props but retention wrote fewer rows than the slate (or nothing). runSnapshot now returns retentionRows so the scheduler can evaluate it. Retention is best-effort by design so it can never break a snapshot — which means a broken write is silent by construction. This is the counterweight. A slot that graded nothing never false-pages; an absent count reads as NOTHING and still pages, distinct from a reported 0. Suite 280/3349 green, build exit 0. Outcome stamping deliberately NOT implemented (depends on the settlement fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA |
||
|
builtbykev
|
d3ffa1b8c2 |
Retention: model_snapshots live + base64 SSH key support
RETENTION (Phase 2, priority zero). History starts compounding tonight. migration 025 model_snapshots — APPLIED to prod. Append-only, one row per graded prop PER SIDE PER CYCLE, with a unique index on (snapshot_id, player_key, stat, line, side) so a retried cycle cannot duplicate. RLS on, service-role writes only. What it captures that the ledger never did: - features jsonb — the model's INPUTS. Without these a backtest can only grade our own homework; with them any future model can be replayed against the exact conditions this one faced. - REFUSALS (refused + refusal_reason). The ledger drops them, so a gate refusing props that would have WON is invisible — unmeasurable lost edge. Captured via a new onGraded hook in gradeSlateService that fires with BOTH sides before any filtering. - grade_11, the pre-collapse grade. The 4-letter map throws away the entire live C-/C/C+/B- range. - model_version + code_sha on every row. ledger_entries mixes pre/post-fix grades with no marker and cannot be separated retroactively. - p_win / ev_pct / fair_odds / takeable / value — none of which any permanent store held. Wiring: analyzeViaEngine1 attaches _features/_grade_11 (underscore = internal); gradeSlateService fires onGraded then STRIPS them so they never reach a cache or API payload; snapshotService builds rows and persists best-effort. Retention reuses the LEDGER's dateET/gameIdFor helpers so rows share the ledger's natural key exactly — otherwise the settle pass could never join outcomes onto them. Rows are written BEFORE the empty- slate early return: a slate that refused everything is exactly the case worth recording. CONTRACT HELD: retention is injectable and every path is caught. persist() returns errors, never throws; a missing Supabase client is SKIPPED, not an error. A retention failure can never break a snapshot. BACKUP: backup-db.sh now accepts BACKUP_SSH_KEY as base64 (recommended — survives env-var newline mangling, which is how injected SSH keys usually break silently) OR raw PEM, detected by decoding and looking for the PEM header. Verified both forms detect correctly against a real generated key. Suite 279/3325 green, build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA |