builtbykev/vyndr
1
0
Fork 0
Code Issues 1 Pull Requests Actions 6 Packages Projects Releases Wiki Activity

Materialization truth: a completed write is not a complete cohort

Browse Source 
Transport truth says every intended write succeeded. Materialization truth says
every identity that should exist actually exists. The retention writer could
only report the first, and the gap is not theoretical.

THE CONFLICT IDENTITY, traced to the real index:
  model_snapshots_cycle_prop_uniq UNIQUE (snapshot_id, player_key, stat, line, side)
written by `upsert(..., { onConflict: same five columns, ignoreDuplicates: true })`.
Measured in production: no key column is ever NULL (0 of 328,262 rows), so
NULLS DISTINCT never applies and the identity is plain column equality. `line`
is an unconstrained numeric, so identity normalises it — 0.5 in and "0.50" out
must not read as two identities for one stored row.

PRIOR-CYCLE COLLISION IS IMPOSSIBLE. `snapshot_id` is in the identity and is a
fresh UUID per cycle, so no row can be suppressed by an earlier cycle.
Append-only chronology across cycles is safe, and `captured_at` is not in the
identity, so a cohort cannot be split by timing.

INTRA-CYCLE COLLISION IS REAL, AND WE CAUSED IT. `canonical_event_id` is NOT in
the identity. A doubleheader — same hitter, same stat, same line, two genuinely
different games — is ONE identity. Demonstrated through the real collector: 4
outbound rows, 2 distinct identities, 2 rows discarded by ignoreDuplicates with
no error, `written` counting all 4 and the terminal status reading COMPLETE.
Before event-aware dedupe the second game was dropped before grading, so the
collision could not arise; that fix moved the loss downstream into retention.

The conflict identity is NOT changed here — that is a separate decision with its
own before/after. This makes the loss visible instead of silent.

EXPECTED vs ACTUAL. `expectedMaterialization(rows)` derives the identity set
from the FINAL outbound payload using the exact database identity — never from
`attempted`, which counts rows sent, not identities that can exist.
`reconcileMaterialization` compares SETS, not counts: two sets of equal size can
still differ, and a cohort that swapped one identity for another passes every
count test ever written. A collision passes set equality by construction (the
discarded row was never in the expected set) while real rows were lost, so
collision_count > 0 fails the cohort on its own.

A cohort is evidence-complete only when transport is COMPLETE, missing = 0,
extra = 0, and collisions = 0.

OBSERVABILITY stayed minimal. `last_retention` was already PER SPORT (a Map
keyed by sport), so no fix was needed there and the route is UNCHANGED — the new
fields ride the existing entry: outbound_rows, expected_materialized_count,
outbound_collision_count, expected_identity_digest. Counts and a digest only,
never the identities, which carry player names. The expected set is the one
materialization fact unrecoverable from the database afterwards, which is why it
is the only thing recorded at runtime.

A collision leaves transport COMPLETE, so the existing failure alert could never
see it. It now has its own high-severity alert naming the counts, the cycle and
the build, and says the cohort is not evidence-complete.

Seven teeth, each injection verified present, against a GREEN baseline of 63:
  1 attempted===written as evidence completeness   -> 1 fail
  2 COUNT(*) equality instead of set equality       -> 1 fail
  3 snapshot_id dropped from expected identity      -> 4 fail
  4 unexpected collision allowed to qualify         -> 1 fail
  5 single global last_retention slot               -> 2 fail
  6 partial chunk failure treated as usable         -> 3 fail
  7 collision loses its announcement                -> 1 fail
Restored byte-identically (retention 742f116473d97f49, snapshot 81129facbabeb280).

Three brittle assertions repaired, with the reason recorded: two windowed on a
byte count that a neighbouring block outgrew — a test failing because of its
neighbour, not its subject — now windowed to syntactic landmarks; and one
counted TERMINAL.COMPLETE occurrences, which a legitimate comparison
incremented. It now asserts one DECISION and one READ.

persist() and createCollector are BYTE-IDENTICAL. onConflict and
ignoreDuplicates appear in the diff only as prose. Model, event, ledger,
calibration, chain, lineage config, and the status route: UNCHANGED. Zero
lineage/publication files, zero cacheSet changes, zero web paths. Lineage OFF.

Schema contract unchanged: release 64, prod 67, prod-only 3 (debt, not
authorized), missing in prod 0.

384 suites / 5,144 tests pass. web tsc exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
Kev
2026-08-27 19:44:56 -04:00
parent 9809626c99
commit 11277a1b99
5 changed files with 477 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/retentionService.js
+140 -1
View File
@@ -727,6 +727,130 @@ function newSnapshotId() {
return crypto.randomUUID();
}
/* ------------------------------------------------------------------ *
* MATERIALIZATION IDENTITY
*
* TRANSPORT COMPLETE and MATERIALIZATION COMPLETE are different facts.
*
* The write is `upsert(..., { onConflict: 'snapshot_id,player_key,stat,line,side',
* ignoreDuplicates: true })`, and the production index behind it is
* model_snapshots_cycle_prop_uniq UNIQUE (snapshot_id, player_key, stat, line, side)
* so two outbound rows sharing that tuple collapse to ONE stored row and the
* loser is discarded WITHOUT AN ERROR. `written` counts rows in committed
* chunks, so transport reports both as written and the status reads COMPLETE.
*
* `snapshot_id` IS in the identity and is a fresh UUID per cycle, so a row can
* never be suppressed by a PREVIOUS cycle — append-only chronology across
* cycles is safe. `canonical_event_id` is NOT in the identity, so two DISTINCT
* events inside ONE cycle (a doubleheader: same hitter, same stat, same line,
* both games) DO collide.
*
* No duplicate is intentional here: `rowsFromSides` emits one row per (base,
* side) and bases are already deduped by event::player::stat::line. Any
* duplicate identity in an outbound payload is therefore an UNEXPECTED
* COLLISION, and a cohort carrying one does not qualify as evidence.
*
* The conflict identity is NOT changed here. This only makes the loss visible.
* ------------------------------------------------------------------ */
/** The exact columns of model_snapshots_cycle_prop_uniq, in index order. */
const CONFLICT_IDENTITY = Object.freeze(['snapshot_id', 'player_key', 'stat', 'line', 'side']);
const IDENTITY_SEP = '\u001f';
const IDENTITY_NULL = '\u0000NULL';
/**
* The database identity of one row. `line` is an unconstrained `numeric`, so it
* is normalised through Number() — otherwise 0.5 on the way in and "0.50" on
* the way out would read as two identities for one stored row.
*
* Every key column is NOT NULL in production (measured: 0 nulls across 328,262
* rows), so PostgreSQL's NULLS DISTINCT behaviour never applies. A null still
* gets its own marker rather than collapsing to '' — an empty string is a real
* value and must not be confused with an absent one.
*/
function rowIdentity(row) {
return CONFLICT_IDENTITY.map((c) => {
const v = row ? row[c] : undefined;
if (v === null || v === undefined) return IDENTITY_NULL;
return c === 'line' ? String(Number(v)) : String(v);
}).join(IDENTITY_SEP);
}
function identityDigest(identities) {
return crypto.createHash('sha256').update([...identities].sort().join('\n')).digest('hex').slice(0, 32);
}
/**
* What SHOULD materialize, derived from the FINAL outbound payload using the
* exact database identity — never from `attempted`, which counts rows sent, not
* identities that can exist.
*/
function expectedMaterialization(rows) {
const list = Array.isArray(rows) ? rows : [];
const seen = new Set();
let collisions = 0;
for (const r of list) {
const id = rowIdentity(r);
if (seen.has(id)) collisions += 1;
else seen.add(id);
}
return {
outbound_rows: list.length,
expected_identities: seen.size,
collision_count: collisions,
digest: identityDigest(seen),
identities: seen,
};
}
const MATERIALIZATION = Object.freeze({
COMPLETE: 'MATERIALIZATION_COMPLETE',
MISSING: 'MATERIALIZATION_MISSING',
EXTRA: 'MATERIALIZATION_EXTRA',
COLLISION: 'MATERIALIZATION_UNEXPECTED_COLLISION',
TRANSPORT_FAILED: 'MATERIALIZATION_UNPROVEN_TRANSPORT_FAILED',
NOT_RECONCILED: 'MATERIALIZATION_NOT_RECONCILED',
});
/**
* Exact SET comparison, not a count comparison. Two sets of equal size can
* still differ, and a cohort that swapped one identity for another would pass
* every count test ever written.
*
* A cohort qualifies ONLY when transport completed, the sets are equal, AND the
* outbound payload held no colliding identity — a collision passes set equality
* by construction (the discarded row was never in the expected set) while real
* rows were lost.
*/
function reconcileMaterialization({ expected, actualIdentities, transportStatus }) {
if (!expected || typeof expected.expected_identities !== 'number') {
throw new TypeError('reconcileMaterialization requires an expectedMaterialization() result');
}
const actual = actualIdentities instanceof Set ? actualIdentities : new Set(actualIdentities || []);
const exp = expected.identities instanceof Set ? expected.identities : new Set();
const missing = [...exp].filter((i) => !actual.has(i));
const extra = [...actual].filter((i) => !exp.has(i));
const out = {
expected_materialized_count: expected.expected_identities,
actual_materialized_count: actual.size,
outbound_rows: expected.outbound_rows,
missing_identity_count: missing.length,
extra_identity_count: extra.length,
collision_count: expected.collision_count,
status: MATERIALIZATION.NOT_RECONCILED,
};
if (transportStatus && transportStatus !== TERMINAL.COMPLETE) {
out.status = MATERIALIZATION.TRANSPORT_FAILED;
return out;
}
if (missing.length) out.status = MATERIALIZATION.MISSING;
else if (extra.length) out.status = MATERIALIZATION.EXTRA;
else if (expected.collision_count > 0) out.status = MATERIALIZATION.COLLISION;
else out.status = MATERIALIZATION.COMPLETE;
return out;
}
/* ------------------------------------------------------------------ *
* TERMINAL RETENTION STATE
*
@@ -795,8 +919,12 @@ function classifyPersist(result) {
*/
const lastTerminal = new Map();
function recordTerminal({ sport, snapshotId, result, completedAt }) {
function recordTerminal({ sport, snapshotId, result, completedAt, rows }) {
const status = classifyPersist(result);
// Derived from the FINAL outbound payload, which does not survive the call.
// This is the one materialization fact unrecoverable from the database
// afterwards, which is why it is the only one recorded at runtime.
const expected = rows ? expectedMaterialization(rows) : null;
const entry = Object.freeze({
sport: sport || null,
snapshot_id: snapshotId || null,
@@ -807,6 +935,11 @@ function recordTerminal({ sport, snapshotId, result, completedAt }) {
code_sha: codeSha(),
// Message only — never row payloads.
error_summary: result.error ? String(result.error).slice(0, 300) : null,
// Counts and a digest only — never the identities, which carry player names.
outbound_rows: expected ? expected.outbound_rows : null,
expected_materialized_count: expected ? expected.expected_identities : null,
outbound_collision_count: expected ? expected.collision_count : null,
expected_identity_digest: expected ? expected.digest : null,
});
if (sport) lastTerminal.set(sport, entry);
return entry;
@@ -823,6 +956,12 @@ function resetTerminal() { lastTerminal.clear(); }
module.exports = {
MODEL_VERSION,
TERMINAL,
MATERIALIZATION,
CONFLICT_IDENTITY,
rowIdentity,
identityDigest,
expectedMaterialization,
reconcileMaterialization,
classifyPersist,
isRetentionFailure,
recordTerminal,
src/services/snapshotService.js
+24
View File
@@ -667,6 +667,9 @@ async function runSnapshot(sport, opts = {}) {
snapshotId: retentionCtx.snapshotId,
result: r,
completedAt: deps.now(),
// The outbound payload, so the expected materialized identity set is
// derived from what was actually sent. It does not survive this call.
rows,
});
console.log(`[snapshot] retention ${sp}: ${terminal.status} ${r.written}/${r.attempted} rows snapshot_id=${terminal.snapshot_id}${r.error ? ` ERROR: ${r.error}` : ''}`);
// RETENTION FAILURE MUST NOT BE SILENT — INCLUDING A PARTIAL ONE.
@@ -681,6 +684,27 @@ async function runSnapshot(sport, opts = {}) {
// healthy. It alerts exactly as loudly as a total failure.
//
// NOTHING_TO_PERSIST and SKIPPED_NO_DATABASE are not failures.
// AN UNEXPECTED COLLISION IS SILENT DATA LOSS WITH A HEALTHY TRANSPORT.
//
// Two outbound rows sharing (snapshot_id, player_key, stat, line, side)
// collapse to one stored row under ignoreDuplicates, with no error. The
// transport reports both as written and the status reads COMPLETE, so the
// failure alert below can never see it. The known cause is a doubleheader:
// canonical_event_id is not in the conflict identity, so the same hitter's
// same line in two real games is one identity.
//
// The cohort is NOT evidence-complete. The conflict identity is not
// changed here; this only refuses to lose the rows quietly.
if (terminal.outbound_collision_count > 0) {
await deps.notify(
`Retention UNEXPECTED COLLISION for ${sp.toUpperCase()} — ${terminal.outbound_collision_count} of `
+ `${terminal.outbound_rows} outbound rows share a conflict identity and were discarded by `
+ `ignoreDuplicates. transport=${terminal.status} expected_identities=${terminal.expected_materialized_count} `
+ `snapshot_id=${terminal.snapshot_id} code_sha=${terminal.code_sha || 'unknown'} at=${terminal.completed_at}. `
+ 'This retention cohort is NOT evidence-complete.',
{ title: 'VYNDR pipeline', priority: 'high', tags: ['rotating_light'] },
);
}
if (retention.isRetentionFailure(terminal.status)) {
await deps.notify(
`Retention ${terminal.status} for ${sp.toUpperCase()} — ${terminal.written}/${terminal.attempted} rows persisted. `