builtbykev/vyndr
1
0
Fork 0
Code Issues 1 Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Lineage family lookup: bound it to a slate, and say what an action is

Browse Source 
TWO DEFECTS, one lookup.

SCALE. The family lookup sent 100 natural keys as a PostgREST IN-list.
`read_natural_key` has NO pg_stats row at all -- the table's last autoanalyze
(2026-08-26) predates the column ever being populated -- so the planner used a
default per-value selectivity, estimated 172,409 rows and chose a sequential
scan of 344,818: 8.5s, then 57014. At 50 keys the same shape returned in
~357ms. The cliff is a statistics artifact, not a volume one, which is why the
repair does not depend on the estimate improving and is not CH=50.

`readNaturalKey` builds `sport|game_date|player_key|stat|side|line[|#event]`,
so SPORT AND GAME_DATE ARE COMPONENTS OF THE KEY. Two rows sharing a key
necessarily share both, and scoping the lookup to the (sport, game_date) pairs
present in the requested keys is LOSSLESS BY CONSTRUCTION. One index-backed
range per date, walked with safePaginate; cost is bounded by ONE SLATE however
long the chronology gets. Measured: 5,000 keys -> 1 scope, and the plan is
`Index Scan using model_snapshots_lineage_family_idx, cost 0.28..1.92`.

VALIDITY. A row carrying `read_natural_key` is not history: the key is stamped
on every candidate BEFORE the lookup, so a failure leaves it on a row that
never became an action. Proven this was not cosmetic -- fed the raw rows the
old lookup returned, the resolver produced a REVISION with a NULL read_id (an
orphaned chain node) and labelled a brand-new Read LEGACY_UNVERIFIED.
`isValidLineageAction` states what a completed action IS: all nine fields, in
the query and again in code.

ATOMICITY. A failed attempt now leaves NO lineage-specific state.
`publication_id`/`published_at` are untouched -- the slate really was
published, and erasing a true fact to tidy a false one is the wrong repair.

Replayed the exact failed 19:00Z cohort through the real resolver, side-effect
free: 119 NEW / 379 CHANGED / 621 UNCHANGED -> ORIGIN 119 / REVISION 379 /
RECAPTURE 621, 0 wrong parent, 0 wrong ordinal, 0 null read_id, 0 forks --
byte-identical with all 1,119 failed partial rows present. Clean-head parity
1,024/1,024.

Migration 050 is CONCURRENTLY + IF NOT EXISTS, drops nothing, rewrites nothing.
Lineage stays OFF.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
Kev
2026-08-28 19:56:56 -04:00
parent 889a96584a
commit 7efb04e280
4 changed files with 498 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/retentionService.js
+121 -10
View File
@@ -259,6 +259,71 @@ const LINEAGE_KEYS = Object.freeze([
'digest_algorithm_version',
]);
/**
* ── WHAT A COMPLETED LINEAGE ACTION IS ───────────────────────────────────
* A row carrying `read_natural_key` is NOT lineage history. The key is stamped
* on every candidate BEFORE the family lookup runs, so a failed resolution
* leaves it behind on a row that never became an action.
*
* A completed action is a row that answers all four questions the chronology
* asks of it: WHICH Read (`read_id`), WHAT it did (`lineage_action`), WHAT it
* claimed (`claim_digest` + the two version stamps that make the digest
* interpretable), and WHERE it sits (`revision_ordinal`). Miss any one and the
* row cannot serve as history, a head, a parent or a recapture predecessor.
*
* This is stated as what an action IS — not as a rule shaped to exclude one
* known batch of failed rows.
*/
const VALID_LINEAGE_ACTION_FIELDS = Object.freeze([
'read_id', 'read_natural_key', 'lineage_action', 'claim_digest',
'revision_ordinal', 'lineage_state', 'lineage_version',
'claim_schema_version', 'digest_algorithm_version',
]);
/** Does this physical row qualify as a completed lineage action? */
function isValidLineageAction(row) {
if (!row) return false;
for (const f of VALID_LINEAGE_ACTION_FIELDS) {
const v = row[f];
if (v === null || v === undefined || v === '') return false;
}
return true;
}
/**
* ── WHY THE FAMILY IS FETCHED BY (sport, game_date) ──────────────────────
* `readLineage.readNaturalKey` builds the key as
* sport | game_date | player_key | stat | side | line [| #event]
* so SPORT AND GAME_DATE ARE COMPONENTS OF THE KEY ITSELF. Two rows sharing a
* natural key necessarily share both. Restricting the lookup to the (sport,
* game_date) pairs present in the requested keys is therefore LOSSLESS BY
* CONSTRUCTION, not an optimisation that trades recall for speed. A test
* asserts the derivation against the key builder so the two cannot drift.
*
* That bound is what makes the read scale with THE SLATE rather than with
* history: one date's valid actions, however many years of chronology sit
* behind it.
*
* WHY NOT KEEP THE IN-LIST. Measured 2026-08-28: `read_natural_key` has NO
* pg_stats row at all (the last autoanalyze predates the column ever being
* populated), so the planner falls back to a default per-value selectivity.
* At 100 keys it estimated 172,409 rows and chose a sequential scan of 344,818
* — 8.5s, then 57014. The plan for this shape is
* `Index Scan using model_snapshots_lineage_family_idx, cost 0.28..1.92`,
* and it does not depend on an estimate being good.
*/
function familyScopesFrom(naturalKeys) {
const scopes = new Map();
for (const k of naturalKeys || []) {
const parts = String(k).split('|');
const sport = parts[0];
const gameDate = parts[1];
if (!sport || !gameDate) continue;
scopes.set(`${sport}|${gameDate}`, { sport, game_date: gameDate });
}
return [...scopes.values()];
}
/** Claim fields the resolver needs from EXISTING rows to classify a change. */
const LINEAGE_FETCH_CLAIM = Object.freeze([
'canonical_event_id',
@@ -335,26 +400,49 @@ async function attachLineage(rows, deps = {}) {
const getClient = deps.getClient || require('../utils/supabase').getSupabaseServiceClient;
const supabase = getClient();
if (!supabase) return null;
const { paginate } = require('../utils/safePaginate');
const columns = ['id', 'read_id', 'read_natural_key', 'game_id', 'claim_digest',
'revision_ordinal', 'lineage_action', 'supersedes_id', 'captured_at',
'lineage_state', 'lineage_version', 'claim_schema_version',
'digest_algorithm_version', ...LINEAGE_FETCH_CLAIM].join(', ');
const wanted = new Set(naturalKeys);
const found = [];
const CH = 100; // never send an unbounded id list — it becomes a URL
for (let i = 0; i < naturalKeys.length; i += CH) {
const { data, error } = await supabase
out.lookup = { scopes: 0, rows_scanned: 0, valid_actions: 0, invalid_excluded: 0 };
for (const scope of familyScopesFrom(naturalKeys)) {
// ONE index-backed range per (sport, game_date), walked with the
// repository's keyset paginator — never a hand-rolled second walk, and
// it THROWS rather than treating a failed page as end-of-data.
// eslint-disable-next-line no-await-in-loop
const rows = await paginate(() => supabase
.from('model_snapshots')
.select(['id', 'read_id', 'read_natural_key', 'game_id', 'claim_digest',
'revision_ordinal', 'lineage_action', 'supersedes_id', 'captured_at',
...LINEAGE_FETCH_CLAIM].join(', '))
.in('read_natural_key', naturalKeys.slice(i, i + CH));
if (error) throw new Error(error.message);
if (Array.isArray(data)) found.push(...data);
.select(columns)
.eq('sport', scope.sport)
.eq('game_date', scope.game_date)
.not('lineage_action', 'is', null), { key: 'id', label: 'attachLineage.fetchExisting' });
out.lookup.scopes += 1;
out.lookup.rows_scanned += rows.length;
for (const r of rows) {
if (!wanted.has(r.read_natural_key)) continue;
// The predicate is applied in code as well as in the query. The query
// narrows what travels; this decides what COUNTS, and a row that half
// resolved must never be mistaken for history by either.
if (!isValidLineageAction(r)) { out.lookup.invalid_excluded += 1; continue; }
out.lookup.valid_actions += 1;
found.push(r);
}
}
return found;
});
const existing = await fetchExisting(keys);
if (existing === null) return out; // no database configured — leave NULL
// Second line of defence: an injected or future fetcher must not be able to
// smuggle a half-resolved row into the chronology either.
const validExisting = existing.filter(isValidLineageAction);
out.invalid_rows_excluded = existing.length - validExisting.length;
const byKey = new Map();
for (const e of existing) {
for (const e of validExisting) {
// The prior claim rides alongside so `classifyChange` can say WHY the
// published state advanced, not merely that the digest differs.
const claim = {};
@@ -413,6 +501,26 @@ async function attachLineage(rows, deps = {}) {
} catch (e) {
out.error = e && e.message ? e.message : String(e);
}
// ── FAILURE ATOMICITY ───────────────────────────────────────────────────
// `read_natural_key` is stamped on every candidate BEFORE the family lookup,
// so a lookup that throws leaves it behind on a row that never became an
// action. That is the shape the 2026-08-28 scheduled canary wrote 1,119 times.
//
// It is LINEAGE FAMILY IDENTITY, not publication provenance, so a failed
// attempt has no claim to it. `publication_id` / `published_at` are NOT
// touched here: the slate really was published, and erasing that to make the
// failure look tidier would delete a true fact to hide a false one.
//
// The result: a failed row carries no lineage-specific state at all, so it
// cannot be mistaken for history by a future resolver, a query, or a reader.
let cleared = 0;
for (const r of list) {
if (isValidLineageAction(r)) continue;
if (LINEAGE_KEYS.some((k) => r[k] !== null && r[k] !== undefined)) cleared += 1;
Object.assign(r, blankLineage());
}
out.incomplete_cleared = cleared;
return out;
}
@@ -1063,6 +1171,9 @@ module.exports = {
persist,
attachLineage,
commitPublication,
isValidLineageAction,
VALID_LINEAGE_ACTION_FIELDS,
familyScopesFrom,
recoverFromFork,
isSupersedesConflict,
FORK_RETRY_LIMIT,