Files
vyndr/tests/unit/runtimeObservability.test.js
T
builtbykev 9809626c99 Retention completion: a cohort is complete only when the writer says N of N
The previous bug made the recorder write nothing. The dangerous successor is a
recorder that writes half and looks healthy: persist() writes in chunks of 250
and STOPS AT THE FIRST FAILED CHUNK, so chunks committed before the failure are
already durable. Rows exist under the snapshot_id, captured_at is uniform, Redis
kept working — and the cohort is short.

So row presence was never completion evidence, and neither was a matching
timestamp. Completeness is now proven by the writer or not at all.

TERMINAL RETENTION STATES (retentionService.classifyPersist):
  NOTHING_TO_PERSIST       attempted 0 — a refusal-only slate is still a cycle
  SKIPPED_NO_DATABASE      no database configured; not a failure
  COMPLETE                 attempted > 0, written === attempted, no error
  FAILED_ZERO_WRITE        written === 0 — first chunk failed
  FAILED_PARTIAL           0 < written < attempted — a later chunk failed
  FAILED_UNRESOLVED_ERROR  counts look complete but an error is unresolved;
                           unreachable through today's loop, and kept because
                           the alternative is reporting COMPLETE holding an error

The invariant: any written < attempted with attempted > 0 is a FAILED cycle. A
partial cohort is never degraded success.

classifyPersist reads the EXACT persist() result and refuses anything else — it
never recomputes attempted or written, because a second calculation could
disagree with the writer and then the status would describe a cycle that did not
happen. persist() itself is byte-identical to 35da190.

`written` counts rows in COMMITTED CHUNKS, not database inserts: the upsert uses
ignoreDuplicates, so a re-run legitimately inserts far fewer rows than it writes.
Comparing written to count(*) will disagree by design. Documented, because that
mismatch is exactly what would be misread as a partial write.

VISIBILITY. The 35da190 alert condition was
`r.error || (!r.skipped && r.attempted > 0 && r.written === 0)` — it could not
see a partial cohort as a distinct state. It is now driven by terminal status,
so FAILED_PARTIAL alerts as loudly as a total failure and is labelled INCOMPLETE
and unusable as evidence. Best-effort is unchanged: the product continues and
the alert says so.

OBSERVABILITY. A successful cycle previously left only a console.log with no
snapshot_id, no code_sha and no terminal status, so completion could not be
established after the fact. `GET /api/internal/snapshot/status` now returns
`last_retention` per sport — sport, snapshot_id, attempted, written, status,
completed_at, code_sha, error_summary — taken verbatim from the persistence
result. Existing internal auth, read-only, counts and status only, no payloads.
No new table, no new route.

RELEASE-AUTHORIZED INSERT CONTRACT. The migration-derived contract is the
release authority; production is not. A prod-only column is DRIFT / RECORDED
DEBT and never becomes permission by existing. Verifier classifies: release
column missing in prod -> HARD FAILURE; prod-only -> drift warning; outbound key
outside the contract -> contract failure (enforced against the real upsert
payload). It is read-only and never rewrites the contract from live schema.
Live: release 64, prod 67, prod-only 3, missing in prod 0.

Six teeth, each with the injection verified present, against a green baseline:
  1 written>0 as generic success        -> 6 fail
  2 later-chunk failure reports COMPLETE -> 5 fail
  3 FAILED_PARTIAL does not alert        -> 3 fail
  4 status reports a recalculated count  -> 1 fail
  5 row presence treated as completion   -> 1 fail
  6 invalid outbound column reintroduced -> 4 fail
Restored byte-identically (retention b341cf16c1baa992, snapshot 81ab1bd7730dee89).

Two stale assertions updated rather than deleted, with the mechanism change
recorded: the alert-shape tests described the superseded written===0 condition,
and the runtime probe test pinned an exact import list.

Model and product preserved: analyzeViaEngine1, probabilityEstimator,
gradeSlateService, lineageCanaryConfig, eventIdentity, ledgerService,
calibration and chain all UNCHANGED; zero lineage/publication files touched;
zero cacheSet changes; zero web paths. Lineage stays OFF.

383 suites / 5,118 tests pass. web tsc exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
2026-08-27 19:12:59 -04:00

213 lines
9.1 KiB
JavaScript

'use strict';
/**
* RUNTIME OBSERVABILITY — the status probe must report the reality the system
* acts on, not its own version of it.
*
* The rollout stalled at RUNTIME_UNVERIFIED because "which build is running?"
* and "is lineage effectively on?" were answerable only as a side effect of a
* scheduled snapshot writing a row. These tests lock the two properties that
* make the answer trustworthy: ONE build-identity resolver and ONE canary
* parser, shared with production.
*/
const path = require('path');
const fs = require('fs');
const ROOT = path.resolve(__dirname, '..', '..');
const ROUTE_RAW = fs.readFileSync(path.join(ROOT, 'src/routes/internal.js'), 'utf8');
/**
* Comments are stripped before any forbidden-string scan.
*
* The header of this route EXPLAINS that it must never use gitea/main and must
* never be named `deployed_at` — and a raw scan flagged that prose as the
* violation. A guard that cannot tell code from the comment describing it will
* eventually be silenced by deleting the explanation, which is the worst
* possible fix. Strip, then scan.
*/
const stripComments = (src) => src
.replace(/\/\*[\s\S]*?\*\//g, '')
.replace(/(^|[^:])\/\/.*$/gm, '$1');
const ROUTE_SRC = stripComments(ROUTE_RAW);
const loadConfig = (val) => {
const prev = process.env.LINEAGE_CANARY_SPORTS;
if (val === undefined) delete process.env.LINEAGE_CANARY_SPORTS;
else process.env.LINEAGE_CANARY_SPORTS = val;
jest.resetModules();
// eslint-disable-next-line global-require
const cfg = require('../../src/services/lineageCanaryConfig');
if (prev === undefined) delete process.env.LINEAGE_CANARY_SPORTS;
else process.env.LINEAGE_CANARY_SPORTS = prev;
return cfg;
};
describe('RUNTIME SHA — one build identity', () => {
test('the probe uses the production codeSha resolver, not git', () => {
// The destructure now also pulls `lastRetention` (terminal retention
// status), so match the resolver rather than the exact import list.
expect(ROUTE_SRC).toMatch(/const \{ codeSha[^}]*\} = require\('\.\.\/services\/retentionService'\)/);
expect(ROUTE_SRC).toMatch(/code_sha: codeSha\(\)/);
// Never repository state.
expect(ROUTE_SRC).not.toMatch(/rev-parse|child_process|execSync|gitea|refs\/heads/);
});
test('a known runtime SHA is returned exactly', () => {
const prev = process.env.SOURCE_COMMIT;
process.env.SOURCE_COMMIT = 'abc123def456';
jest.resetModules();
// eslint-disable-next-line global-require
const { codeSha } = require('../../src/services/retentionService');
expect(codeSha()).toBe('abc123def456');
if (prev === undefined) delete process.env.SOURCE_COMMIT; else process.env.SOURCE_COMMIT = prev;
});
test('an unavailable runtime SHA is null, never a substitute', () => {
const saved = {};
for (const k of ['SOURCE_COMMIT', 'GIT_SHA', 'COOLIFY_GIT_COMMIT_SHA']) {
saved[k] = process.env[k]; delete process.env[k];
}
jest.resetModules();
// eslint-disable-next-line global-require
const { codeSha } = require('../../src/services/retentionService');
expect(codeSha()).toBeNull();
for (const [k, v] of Object.entries(saved)) if (v !== undefined) process.env[k] = v;
});
});
describe('RUNTIME START — one process lifetime', () => {
test('started_at is computed ONCE at module load, not per request', () => {
// Recomputing per call would make it read as "now" and destroy its only
// use: marking a boundary.
expect(ROUTE_SRC).toMatch(/const PROCESS_STARTED_AT = new Date\(Date\.now\(\) - Math\.round\(process\.uptime\(\) \* 1000\)\)\.toISOString\(\)/);
expect(ROUTE_SRC).toMatch(/started_at: PROCESS_STARTED_AT/);
// Exactly one assignment — no shadowing recompute.
expect((ROUTE_SRC.match(/PROCESS_STARTED_AT\s*=/g) || []).length).toBe(1);
});
test('it is NOT named deployed_at — a restart moves it without a deploy', () => {
expect(ROUTE_SRC).not.toMatch(/deployed_at/);
});
test('the derived instant is in the past and plausible for this process', () => {
const started = Date.now() - Math.round(process.uptime() * 1000);
expect(started).toBeLessThanOrEqual(Date.now());
expect(Number.isFinite(started)).toBe(true);
});
});
describe('LINEAGE CONFIG — one parser', () => {
test('the write gate and the probe consume the SAME module', () => {
const snap = fs.readFileSync(path.join(ROOT, 'src/services/snapshotService.js'), 'utf8');
expect(snap).toMatch(/require\('\.\/lineageCanaryConfig'\)/);
expect(ROUTE_SRC).toMatch(/require\('\.\.\/services\/lineageCanaryConfig'\)/);
// Neither may re-parse the environment itself.
expect(ROUTE_SRC).not.toMatch(/process\.env\.LINEAGE_CANARY_SPORTS/);
expect(snap).not.toMatch(/process\.env\.LINEAGE_CANARY_SPORTS/);
});
test('the gate delegates to the shared resolver', () => {
jest.resetModules();
// eslint-disable-next-line global-require
const snap = require('../../src/services/snapshotService');
// eslint-disable-next-line global-require
const cfg = require('../../src/services/lineageCanaryConfig');
for (const sp of ['mlb', 'wnba', 'nba', 'soccer']) {
expect(snap.lineageCanaryEnabled(sp)).toBe(cfg.isEnabled(sp));
}
expect(snap.LINEAGE_CANARY_SPORTS).toBe(cfg.SPORTS);
});
test('unset reports disabled + [] + DEFAULT', () => {
const c = loadConfig(undefined);
expect(c.state()).toEqual({ enabled: false, sports: [], configuration_source: 'DEFAULT' });
});
test('mlb reports enabled + ["mlb"] + ENVIRONMENT', () => {
const c = loadConfig('mlb');
expect(c.state()).toEqual({ enabled: true, sports: ['mlb'], configuration_source: 'ENVIRONMENT' });
});
test('an EXPLICIT empty is ENVIRONMENT, not DEFAULT', () => {
// An operator deliberately blanking the value reads differently from never
// having set it, even though both are dark.
const c = loadConfig('');
expect(c.state()).toEqual({ enabled: false, sports: [], configuration_source: 'ENVIRONMENT' });
});
test('multiple sports normalise deterministically — sorted, deduped, trimmed, lowercased', () => {
expect(loadConfig('MLB, mlb ,wnba').state().sports).toEqual(['mlb', 'wnba']);
expect(loadConfig('wnba,mlb').state().sports).toEqual(['mlb', 'wnba']);
expect(loadConfig(' MLB ').state().sports).toEqual(['mlb']);
expect(loadConfig(',,mlb,,').state().sports).toEqual(['mlb']);
});
test('an unsupported value behaves exactly as the gate behaves', () => {
const c = loadConfig('cricket');
expect(c.state().sports).toEqual(['cricket']);
expect(c.isEnabled('cricket')).toBe(true); // the flag is scope, not validation
expect(c.isEnabled('mlb')).toBe(false);
});
test('the source-code default cannot hide an environment override', () => {
// The whole point: if Coolify sets mlb, the probe must say mlb.
expect(loadConfig('mlb').state().enabled).toBe(true);
expect(loadConfig(undefined).state().enabled).toBe(false);
});
});
describe('SECURITY — no raw config, no weakened access', () => {
test('the raw environment value is never returned', () => {
const c = loadConfig('MLB, wnba ');
const payload = JSON.stringify(c.state());
expect(payload).not.toContain('MLB, wnba');
expect(payload).not.toContain(' ');
});
test('the response exposes no secret-like config', () => {
const forbidden = [
'VYNDR_INTERNAL_KEY', 'SUPABASE_SERVICE', 'STRIPE', 'REDIS_URL',
'PROPLINE_API_KEY', 'ODDS_API_KEY', 'SUPABASE_DB_PASSWORD',
];
const handler = ROUTE_SRC.slice(ROUTE_SRC.indexOf("router.get('/snapshot/status'"));
const body = handler.slice(0, handler.indexOf('router.'));
for (const f of forbidden) expect(body).not.toContain(f);
});
test('router-wide internal auth is unchanged', () => {
expect(ROUTE_SRC).toMatch(/router\.use\(requireInternalAuth\(\{ loopbackOnly: false \}\)\)/);
// The status route must not opt itself out.
expect(ROUTE_SRC).not.toMatch(/\/snapshot\/status'[^)]*skipAuth/);
});
});
describe('READ ONLY — the probe observes and nothing else', () => {
test('the handler performs no writes of any kind', () => {
const start = ROUTE_SRC.indexOf("router.get('/snapshot/status'");
const body = ROUTE_SRC.slice(start, ROUTE_SRC.indexOf('\nrouter.', start + 10));
for (const verb of ['cacheSet', '.insert(', '.upsert(', '.update(', '.delete(', 'runSnapshot', 'commitPublication', 'attachLineage']) {
expect(body).not.toContain(verb);
}
// Reading Redis is the only side-effect-free access it needs.
expect(body).toContain('cacheGet');
});
test('it cannot enable lineage — it only reads the frozen state', () => {
const c = loadConfig(undefined);
const before = JSON.stringify(c.state());
c.state(); c.state();
expect(JSON.stringify(c.state())).toBe(before);
// No setter exists.
expect(Object.keys(c).filter((k) => /^(set|enable|disable|update)/i.test(k))).toHaveLength(0);
});
test('the reported state is a copy — a caller cannot mutate the gate', () => {
const c = loadConfig('mlb');
const s1 = c.state();
s1.sports.push('wnba');
expect(c.state().sports).toEqual(['mlb']);
expect(c.isEnabled('wnba')).toBe(false);
});
});