builtbykev/vyndr
1
0
Fork 0
Code Issues 1 Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Event-aware retention identity: two games, two receipts

Browse Source 
One player prop in Game 1 and the same-looking prop in Game 2 are two different
historical claims. The retention conflict identity did not know that.

SEMANTIC IDENTITY FIRST. Two outbound rows are the same retention proposition
within one cycle when they share the cycle, the EVENT, the participant, the
stat, the line and the side. Book is deliberately absent — collapsing books is
dedupeProps's actual job and the price anchor is chosen later. The database
index is enforcement of that answer, never the definition of it.

THE EVENT COMPONENT NEVER FABRICATES. canonical_event_id where a sport has a
resolver — MLB's admission gate rejects unresolved/ambiguous/contradicted props
BEFORE grading, so every row that can reach retention has one — and game_id
otherwise, which is NOT NULL in the schema and is the only event label sports
without a resolver possess. Both are in the identity, so the weaker label still
discriminates where the stronger is absent.

NULLS NOT DISTINCT IS LOAD-BEARING, NOT STYLISTIC. canonical_event_id is NULL
for every non-MLB row. Measured on a disposable PG17: under PostgreSQL's default
semantics the same NBA proposition inserted twice produced TWO rows — every
retry duplicating for ever. With NULLS NOT DISTINCT the same test yields one.
That measurement is what rejected the plain composite option.

MIXED-FLEET BRIDGE. A rollout serves both builds at once (measured 11/12 new,
1 old). Old and new writers need different indexes and NO schema state satisfies
both: with the legacy index present a new writer fails 23505 on a doubleheader;
with it gone an old writer fails 42P10. A bare ON CONFLICT DO NOTHING would have
bridged this, and PostgREST does not emit one — `ignoreDuplicates` WITHOUT
`onConflict` was measured raising a real duplicate-key error, so that bridge does
not exist through this client.

So the writer bridges it. It targets the event-aware identity and, on exactly
the two errors meaning "the schema is not in the state I expect" (42P10, or
23505 NAMING the legacy index), retries the SAME chunk on the legacy target. A
failed chunk rolls back atomically — measured 0 rows — so the retry cannot
double-write. Correct in every schema state: legacy-only and both-present
degrade to legacy semantics with no outage; new-only keeps both games.

The bridge is deliberately narrow. A supersedes conflict is ALSO a 23505, and
swallowing it would destroy the forked-history guard, so the legacy index must
be named. All three model_snapshots writers (persist, commitPublication,
recoverFromFork) go through it; no hardcoded legacy target survives.

MEASURED, through the real supabase-js -> PostgREST -> Postgres path on
production-shaped PG17:
  * 1,000 REAL propositions from the verified 2026-08-17 STL@CIN doubleheader
    (1,738 retained rows under ONE game_id), replayed across both real gamePks:
    OLD index materialized 1,000 of 2,000 — 1,000 LOST. NEW index materialized
    2,000 of 2,000 — 0 lost.
  * retry idempotency, over/under, line, stat, player, non-MLB same-game and
    non-MLB different-game all behave correctly under the new index.
  * ORDINARY-SLATE PARITY over ALL 434 real cohorts / 328,262 retained rows:
    old identities 328,262, new identities 328,262, delta 0, cohorts changed 0.
    The index is therefore guaranteed creatable and nothing historical splits.

CONFLICT_IDENTITY is now DERIVED from RETENTION_CONFLICT rather than restated —
a test caught them silently disagreeing, which is exactly how the materialization
check could have expected an identity the database no longer enforced.

EXPAND/CONTRACT are separate files on purpose. 048 is additive and retires
nothing; 049 drops the legacy index and must not be applied until fleet
convergence is proven by sampling, never assumed from a fast rollout.

NO BACKFILL. Legacy rows keep NULL canonical_event_id and remain LEGACY
EVENT-AGNOSTIC RETENTION, which is what that NULL truthfully says.

The materialization defence is untouched and now reports the bridge honestly:
while the legacy index still collapses a doubleheader, expected 4 vs actual 2
yields MATERIALIZATION_MISSING and the cohort is refused.

Nine teeth, injections verified present, against a green baseline of 97:
1 event distinction removed (10) · 2 phases collapsed (2) · 3 bridge swallows
everything (6) · 4 NULLS NOT DISTINCT removed (1) · 5 old-container error as
success (3) · 6 semantic/DB identity disagree (8) · 7 collision detector removed
(2) · 8 partial transport usable (3) · 9 collision unannounced (1).
Restored byte-identically.

Model and product untouched: gradeSlateService (event-aware dedupe), event
identity, ledger, calibration, chain, lineage config and the status route all
UNCHANGED. Zero cacheSet changes, zero web paths, schema contract unchanged (no
new columns). Lineage stays OFF.

385 suites / 5,178 tests pass. web tsc exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
Kev
2026-08-27 20:30:45 -04:00
parent 11277a1b99
commit 048e4eaa3f
5 changed files with 481 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/retentionService.js
+103 -16
View File
@@ -450,9 +450,7 @@ async function recoverFromFork(chunk, deps = {}) {
// eslint-disable-next-line no-await-in-loop
await attachLineage(chunk, deps);
// eslint-disable-next-line no-await-in-loop
const { error } = await supabase
.from('model_snapshots')
.upsert(chunk, { onConflict: 'snapshot_id,player_key,stat,line,side' });
const { error } = await upsertSnapshotChunk(supabase, chunk);
if (!error) { out.recovered = true; out.written = chunk.length; return out; }
if (!isSupersedesConflict(error)) { out.error = error.message; return out; }
out.error = error.message;
@@ -521,9 +519,7 @@ async function commitPublication(spec, deps = {}) {
for (let i = 0; i < served.length; i += CHUNK) {
const chunk = served.slice(i, i + CHUNK);
// eslint-disable-next-line no-await-in-loop
const { error } = await supabase
.from('model_snapshots')
.upsert(chunk, { onConflict: 'snapshot_id,player_key,stat,line,side' });
const { error } = await upsertSnapshotChunk(supabase, chunk);
if (!error) { out.published += chunk.length; continue; }
// ── CONCURRENCY RECOVERY ────────────────────────────────────────────
@@ -606,12 +602,9 @@ async function persist(rows, deps = {}) {
const CHUNK = 250;
for (let i = 0; i < rows.length; i += CHUNK) {
const chunk = rows.slice(i, i + CHUNK);
const { error } = await supabase
.from('model_snapshots')
.upsert(chunk, {
onConflict: 'snapshot_id,player_key,stat,line,side',
ignoreDuplicates: true,
});
const res = await upsertSnapshotChunk(supabase, chunk, { ignoreDuplicates: true });
if (res.fellBack) out.legacy_conflict_fallback = true;
const { error } = res;
if (error) { out.error = error.message; break; }
out.written += chunk.length;
}
@@ -727,14 +720,98 @@ function newSnapshotId() {
return crypto.randomUUID();
}
/* ------------------------------------------------------------------ *
* RETENTION CONFLICT TARGET — event-aware, mixed-fleet safe
*
* SEMANTIC IDENTITY. Two outbound rows are the SAME retention proposition
* within one cycle when they share: the cycle, the EVENT, the participant, the
* stat, the line and the side. Provider/book is deliberately NOT part of it —
* collapsing books is `dedupeProps`'s actual job, and the price anchor is
* chosen later.
*
* The EVENT component uses the strongest truthful label available and never
* fabricates one: `canonical_event_id` where a sport has a resolver (MLB, where
* `admitForGrading` makes it non-null for every row that can reach retention),
* and `game_id` otherwise (NOT NULL in the schema, and the only event label
* sports without a resolver possess). Both columns are in the identity, so the
* weaker label still discriminates where the stronger one is absent.
*
* NULLS NOT DISTINCT is load-bearing. `canonical_event_id` is NULL for every
* non-MLB row, and under PostgreSQL's DEFAULT null semantics two NULLs are
* DISTINCT — measured: the same NBA proposition inserted twice produced TWO
* rows, i.e. every retry would duplicate for ever. With NULLS NOT DISTINCT the
* same test produces one row and retry idempotency holds.
*
* MIXED-FLEET SAFETY. A rollout serves old and new containers at once
* (measured: 11 of 12 probes new, 1 old). The two writers need different
* indexes, and no schema state satisfies both:
*
* old index present -> old writer works; new writer ERRORS 23505 on a
* doubleheader (the legacy index rejects a row the
* event-aware identity considers distinct)
* old index dropped -> new writer works; old writer ERRORS 42P10
*
* A bare `ON CONFLICT DO NOTHING` would have bridged this, but PostgREST does
* NOT emit one: `ignoreDuplicates` without `onConflict` was measured raising a
* real duplicate-key error, so that bridge does not exist through this client.
*
* So the writer bridges it instead. It targets the event-aware identity and, on
* exactly the two errors that mean "the schema is not in the state I expect",
* retries the SAME chunk on the legacy target. A failed chunk rolls back
* atomically (measured: 0 rows), so the retry cannot double-write. The result
* is a writer that is correct in every schema state:
*
* legacy only -> 42P10 -> legacy target -> legacy semantics, no outage
* both present -> 23505 -> legacy target -> legacy semantics, no outage
* new only -> primary succeeds -> doubleheaders kept
*
* The fallback is a BRIDGE, not a resting place: while it fires, doubleheader
* rows are still lost, and `outbound_collision_count` still reports it.
* ------------------------------------------------------------------ */
const LEGACY_CONFLICT_INDEX = 'model_snapshots_cycle_prop_uniq';
const LEGACY_CONFLICT = 'snapshot_id,player_key,stat,line,side';
const RETENTION_CONFLICT = 'snapshot_id,game_id,canonical_event_id,player_key,stat,line,side';
/**
* Is this error "the schema is not in the state the event-aware target
* expects"? Deliberately narrow: a supersedes conflict is also a 23505, and
* swallowing THAT would destroy the forked-history guard, so the legacy index
* must be named.
*/
function isLegacyConflictBlock(error) {
if (!error) return false;
const code = String(error.code || '');
const msg = String(error.message || '');
if (code === '42P10' || /no unique or exclusion constraint matching/i.test(msg)) return true;
return (code === '23505' || /duplicate key value/i.test(msg)) && msg.includes(LEGACY_CONFLICT_INDEX);
}
/**
* One chunk write. Returns the error (never throws) plus which target actually
* carried it, so a caller can report that the bridge is still in use.
*/
async function upsertSnapshotChunk(supabase, chunk, opts = {}) {
const first = await supabase.from('model_snapshots')
.upsert(chunk, { onConflict: RETENTION_CONFLICT, ...opts });
if (!first.error) return { error: null, target: RETENTION_CONFLICT, fellBack: false };
if (!isLegacyConflictBlock(first.error)) {
return { error: first.error, target: RETENTION_CONFLICT, fellBack: false };
}
const second = await supabase.from('model_snapshots')
.upsert(chunk, { onConflict: LEGACY_CONFLICT, ...opts });
return { error: second.error || null, target: LEGACY_CONFLICT, fellBack: true };
}
/* ------------------------------------------------------------------ *
* MATERIALIZATION IDENTITY
*
* TRANSPORT COMPLETE and MATERIALIZATION COMPLETE are different facts.
*
* The write is `upsert(..., { onConflict: 'snapshot_id,player_key,stat,line,side',
* ignoreDuplicates: true })`, and the production index behind it is
* The write is `upsert(..., { onConflict: RETENTION_CONFLICT, ignoreDuplicates:
* true })`. Until migration 049 retires it, the LEGACY index
* model_snapshots_cycle_prop_uniq UNIQUE (snapshot_id, player_key, stat, line, side)
* is still enforced and the writer falls back to it (see the bridge above)
* so two outbound rows sharing that tuple collapse to ONE stored row and the
* loser is discarded WITHOUT AN ERROR. `written` counts rows in committed
* chunks, so transport reports both as written and the status reads COMPLETE.
@@ -753,8 +830,13 @@ function newSnapshotId() {
* The conflict identity is NOT changed here. This only makes the loss visible.
* ------------------------------------------------------------------ */
/** The exact columns of model_snapshots_cycle_prop_uniq, in index order. */
const CONFLICT_IDENTITY = Object.freeze(['snapshot_id', 'player_key', 'stat', 'line', 'side']);
/**
* The exact columns the database conflict target names, in index order —
* DERIVED from RETENTION_CONFLICT rather than restated, so the identity the
* materialization check expects can never drift from the identity the database
* enforces. Restating it is how the two silently disagreed before.
*/
const CONFLICT_IDENTITY = Object.freeze(RETENTION_CONFLICT.split(','));
const IDENTITY_SEP = '\u001f';
const IDENTITY_NULL = '\u0000NULL';
@@ -956,6 +1038,11 @@ function resetTerminal() { lastTerminal.clear(); }
module.exports = {
MODEL_VERSION,
TERMINAL,
RETENTION_CONFLICT,
LEGACY_CONFLICT,
LEGACY_CONFLICT_INDEX,
isLegacyConflictBlock,
upsertSnapshotChunk,
MATERIALIZATION,
CONFLICT_IDENTITY,
rowIdentity,