builtbykev/vyndr
1
0
Fork 0
Code Issues 1 Pull Requests Actions 6 Packages Projects Releases Wiki Activity

A canary is temporary: bound activation with a fail-closed lease

Browse Source 
`LINEAGE_CANARY_SPORTS=mlb` was parsed once at module load and frozen, so an
enabled canary stayed writable for the whole process lifetime. On 2026-08-29 it
was left set and the 22:00Z, 01:00Z and 03:00Z scheduled snapshots each wrote
lineage overnight with nobody watching. That history happened to be correct.
Lineage is append-only evidence, so a defective canary would have written
irreversible WRONG evidence exactly as quietly. Correct history was luck, not a
safety property.

NEW CONTRACT:  LINEAGE_CANARY_SPORTS=mlb@2026-08-30T18:00:00Z

Absolute UTC instant only -- no duration, no local timezone. A relative "4h"
would silently restart on every redeploy, which is the defect being removed.

LEGACY `mlb` NO LONGER ACTIVATES ANYTHING. It is INVALID_MISSING_EXPIRY. Leaving
it working would have left the defect in place behind a nicer-looking
alternative, so this is the load-bearing half of the repair.

EXPIRY IS EVALUATED AT THE WRITE GATE, not at startup. `isEnabled(sport, now)`
re-reads the clock on every call and `lineageCanaryEnabled` threads it through,
so a lease turns itself off with no operator, no restart, no Redis and no
network. A design where an expired canary keeps writing until someone restarts
is the same failure in a different hat.

MAX_LEASE is FOUR HOURS, and the bound is proven rather than chosen. MLB ticks
are [14,19,22,1,3] UTC; exhaustively over a minute grid across UTC date
boundaries, the shortest span enclosing THREE consecutive ticks is
22:00 -> 01:00 -> 03:00 = five hours. Four hours encloses at most two, with an
hour of margin. (An earlier note claimed six hours admitted two. It admits
three; that claim was false and the test now pins the arithmetic.) The bound is
a function of the scheduler, so a test reads the hours from sportCadence and
fails if a cadence change invalidates the proof.

Everything fails closed: absent, empty, bare sport, comma list, two leases,
duplicate @, non-leasable sport, missing timezone, numeric offset, malformed,
over-long, already-expired, expiry-equals-now, and an unreadable clock.
Configured-but-EXPIRED stays distinguishable from never-configured so automatic
containment is auditable.

No Redis, no Supabase, no counter, no network in the lease path -- an
unreachable dependency must never decide whether lineage may write.

Lineage algorithms, cache-date, participant and retention identity untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
Kev
2026-08-30 14:58:11 -04:00
parent 836b8c73d5
commit cc5bdf5797
5 changed files with 529 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/lineageCanaryConfig.js
+196 -55
View File
@@ -1,79 +1,220 @@
'use strict';
"use strict";
/**
* LINEAGE CANARY CONFIG — the ONE resolver.
* LINEAGE CANARY LEASE — the ONE resolver, now BOUNDED IN TIME.
*
* -- WHY THIS MODULE EXISTS ----------------------------------------------
* The rollout stalled at RUNTIME_UNVERIFIED because nothing could answer
* "is MLB lineage effectively enabled?" without waiting for a scheduled
* snapshot to write a row. Exposing the answer requires a status route to read
* the config — and a status route that parses the environment ITSELF would be a
* second version of the truth, free to drift from the gate it claims to report.
* -- WHY THIS CHANGED -----------------------------------------------------
* The previous contract parsed `LINEAGE_CANARY_SPORTS=mlb` once at module load
* and froze the answer. `isEnabled()` then returned the same boolean for the
* lifetime of the process. On 2026-08-29 the flag was left set; the 22:00Z,
* 01:00Z and 03:00Z scheduled snapshots each wrote lineage overnight with no
* operator present. The history they wrote happened to be correct. That is luck,
* not a safety property: lineage is append-only evidence, so a defective canary
* would have written irreversible wrong evidence just as silently.
*
* So the parse lives here, once. `snapshotService` consumes it for the actual
* lineage write gate, and the internal status probe consumes the SAME frozen
* state. Observability reports the reality the system acts on, or it is not
* observability.
* A canary is temporary BY DEFINITION. This encodes that into the activation
* contract instead of relying on somebody remembering.
*
* -- EFFECTIVE-TIME SEMANTICS --------------------------------------------
* The value is resolved ONCE, at module load, exactly as the previous inline
* constant was. It is therefore FIXED FOR THE LIFETIME OF THE PROCESS:
* -- THE SYNTAX -----------------------------------------------------------
* LINEAGE_CANARY_SPORTS=mlb@2026-08-30T18:00:00Z
* ^^^ ^^^^^^^^^^^^^^^^^^^^
* sport absolute UTC expiry
*
* read from process.env yes
* parsed once at module startup yes
* re-read per lineage decision no
* can change without a restart no
* Absolute instant only. No duration ("4h"), no local timezone, no implicit
* offset -- a relative duration would silently restart on every redeploy, which
* is the exact defect being removed.
*
* That is what makes the process start time a defensible lower bound for
* "this runtime has had this effective lineage state since at least then".
* Changing the variable in Coolify restarts the container, which yields a new
* process and therefore a new boundary.
* -- LEGACY `mlb` NO LONGER ACTIVATES ANYTHING ---------------------------
* The bare sport form is now INVALID_MISSING_EXPIRY, not a valid indefinite
* activation. This is load-bearing: leaving it working would leave the defect
* in place behind a nicer-looking alternative.
*
* -- EXPIRY IS EVALUATED AT THE WRITE GATE, NOT AT STARTUP ---------------
* Parsing the raw string once is fine; it cannot change without a restart.
* EFFECTIVE ENABLEMENT is not frozen. `isEnabled(sport, now)` re-reads the clock
* on every call, so expiry takes effect with no operator, no restart, no Redis
* and no network. A design where an expired canary keeps writing until someone
* restarts it is the same failure wearing a different hat.
*
* -- NO EXTERNAL DEPENDENCY ----------------------------------------------
* Lease truth is local parsed configuration plus the wall clock. An unreachable
* Redis or Supabase must never be what decides whether lineage may write.
*/
/** Where the effective value came from. Never the value itself. */
const CONFIGURATION_SOURCE = Object.freeze({
ENVIRONMENT: 'ENVIRONMENT',
DEFAULT: 'DEFAULT',
ENVIRONMENT: "ENVIRONMENT",
DEFAULT: "DEFAULT",
});
/** Lease states, in the language the status probe reports. */
const LEASE_STATE = Object.freeze({
OFF: "OFF", // nothing configured
ACTIVE: "ACTIVE", // configured, valid, and now < expires_at
EXPIRED: "EXPIRED", // configured and valid, but the clock passed expiry
INVALID: "INVALID", // configured and unusable — always fails closed
});
/**
* THE BOUND, and why it is four hours.
*
* MLB scheduled snapshot ticks are [14,19,22,1,3] UTC. Proven exhaustively over
* a minute grid across UTC date boundaries: the SHORTEST span enclosing three
* consecutive ticks is 22:00 -> 01:00 -> 03:00 = FIVE HOURS. A lease of four
* hours therefore encloses at most TWO scheduled ticks, with an hour of margin.
*
* (An earlier design note claimed six hours admitted two ticks. That was wrong:
* six hours spans 22->01->03 and admits three.)
*
* THIS BOUND IS A FUNCTION OF THE SCHEDULER. Changing SNAPSHOT_HOURS_UTC
* invalidates the proof and requires revalidating MAX_LEASE_MS. A test pins
* both the hours and the bound together so the two cannot drift apart.
*/
const MAX_LEASE_MS = 4 * 60 * 60 * 1000;
/** The sports this rollout may lease. Widening is a deliberate, separate act. */
const LEASABLE_SPORTS = Object.freeze(["mlb"]);
const RAW = process.env.LINEAGE_CANARY_SPORTS;
/**
* Normalised, deterministic, frozen. Sorted so two processes with the same
* effective configuration report byte-identical state regardless of the order
* it was written in.
*/
const SPORTS = Object.freeze(
String(RAW || '')
.split(',')
.map((x) => x.trim().toLowerCase())
.filter(Boolean)
.filter((x, i, a) => a.indexOf(x) === i)
.sort(),
);
/**
* DEFAULT means the variable was never set — the dark default. ENVIRONMENT
* means something set it, INCLUDING setting it to empty, because an explicit
* empty is an operator decision and reads differently from an absent one.
*/
const SOURCE = RAW === undefined ? CONFIGURATION_SOURCE.DEFAULT : CONFIGURATION_SOURCE.ENVIRONMENT;
/** THE gate. Lineage writes for a sport only if this says so. */
function isEnabled(sport) {
return SPORTS.includes(String(sport || '').toLowerCase());
/**
* A strict absolute-UTC ISO instant. Deliberately narrow: `Date.parse` accepts
* plenty of shapes whose meaning depends on the reader's timezone, and a lease
* that means different things on different containers is not a lease.
*/
const ISO_UTC = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?Z$/;
/**
* Parse the raw value ONCE. Returns a frozen description — never a decision.
* The decision needs a clock and is made in `evaluate`.
*/
function parseLease(raw) {
const bad = (reason) => Object.freeze({
configured: true, valid: false, invalid_reason: reason, sport: null, expires_at: null,
});
if (raw === undefined || raw === null) {
return Object.freeze({ configured: false, valid: false, invalid_reason: null, sport: null, expires_at: null });
}
const s = String(raw).trim();
if (s === "") {
// An explicit empty is an operator decision to be off, not a malformed one.
return Object.freeze({ configured: false, valid: false, invalid_reason: null, sport: null, expires_at: null });
}
// ONE token only. Two sports leased at once is how a canary quietly widens.
if (s.includes(",")) return bad("INVALID_MULTIPLE_TOKENS");
const at = s.indexOf("@");
if (at === -1) return bad("INVALID_MISSING_EXPIRY"); // the legacy bare `mlb`
if (s.indexOf("@", at + 1) !== -1) return bad("INVALID_SYNTAX");
const sport = s.slice(0, at).trim().toLowerCase();
const stamp = s.slice(at + 1).trim();
if (!sport) return bad("INVALID_MISSING_SPORT");
if (!LEASABLE_SPORTS.includes(sport)) return bad("INVALID_SPORT_NOT_LEASABLE");
if (!ISO_UTC.test(stamp)) return bad("INVALID_EXPIRY_FORMAT");
const ms = Date.parse(stamp);
if (!Number.isFinite(ms)) return bad("INVALID_EXPIRY_VALUE");
return Object.freeze({
configured: true, valid: true, invalid_reason: null,
sport, expires_at: new Date(ms).toISOString(), expires_ms: ms,
});
}
const LEASE = parseLease(RAW);
/** The clock, as a finite epoch-ms, or null. A clock we cannot read is not a
* reason to allow writing. */
function nowMs(now) {
const d = now === undefined ? new Date() : now;
const ms = d instanceof Date ? d.getTime() : Date.parse(d);
return Number.isFinite(ms) ? ms : null;
}
/**
* The reportable state. Contains no raw environment value by construction —
* `RAW` is never returned, only the normalised set derived from it.
* THE ONE EVALUATOR. The write gate, the status probe and the tests all come
* here, so a status that says EXPIRED can never sit beside a writer that still
* writes.
*/
function state() {
return {
enabled: SPORTS.length > 0,
sports: [...SPORTS],
function evaluate(now) {
const t = nowMs(now);
const base = {
configured: LEASE.configured,
configuration_source: SOURCE,
config_valid: LEASE.valid,
invalid_reason: LEASE.invalid_reason,
configured_sport: LEASE.sport,
configured_expires_at: LEASE.expires_at,
lease_state: LEASE_STATE.OFF,
effective_enabled: false,
effective_active_sports: [],
remaining_ms: null,
};
if (!LEASE.configured) return Object.freeze(base);
if (!LEASE.valid) return Object.freeze({ ...base, lease_state: LEASE_STATE.INVALID });
if (t === null) {
// Fail closed, and say WHY rather than reporting a bare OFF.
return Object.freeze({ ...base, lease_state: LEASE_STATE.INVALID, invalid_reason: "INVALID_CLOCK" });
}
const remaining = LEASE.expires_ms - t;
// A lease longer than the bound is refused outright — a week-long "canary" is
// the defect this module exists to remove.
if (remaining > MAX_LEASE_MS) {
return Object.freeze({ ...base, lease_state: LEASE_STATE.INVALID, invalid_reason: "INVALID_LEASE_TOO_LONG" });
}
// `<= 0` covers both expiry and an expiry set at or before now. EXPIRED is
// kept distinct from OFF so automatic containment stays auditable.
if (remaining <= 0) {
return Object.freeze({ ...base, lease_state: LEASE_STATE.EXPIRED, remaining_ms: 0 });
}
return Object.freeze({
...base,
lease_state: LEASE_STATE.ACTIVE,
effective_enabled: true,
effective_active_sports: [LEASE.sport],
remaining_ms: remaining,
});
}
/** THE gate. Lineage writes for a sport only if this says so, RIGHT NOW. */
function isEnabled(sport, now) {
const e = evaluate(now);
if (!e.effective_enabled) return false;
return e.effective_active_sports.includes(String(sport || "").toLowerCase());
}
/** The reportable state. Contains no raw environment value by construction. */
function state(now) {
const e = evaluate(now);
return {
// Back-compatible keys the probe and its tests already consume.
enabled: e.effective_enabled,
sports: [...e.effective_active_sports],
configuration_source: e.configuration_source,
// The lease surface: configured vs EFFECTIVE, so an expired lease is
// visibly different from one that was never set.
configured: e.configured,
config_valid: e.config_valid,
invalid_reason: e.invalid_reason,
configured_sport: e.configured_sport,
configured_expires_at: e.configured_expires_at,
lease_state: e.lease_state,
effective_enabled: e.effective_enabled,
effective_active_sports: [...e.effective_active_sports],
remaining_ms: e.remaining_ms,
max_lease_ms: MAX_LEASE_MS,
};
}
module.exports = { SPORTS, isEnabled, state, CONFIGURATION_SOURCE };
/**
* The CONFIGURED sport list. Deliberately NOT the gate: it says what was asked
* for, never what is permitted now.
*/
const SPORTS = Object.freeze(LEASE.valid && LEASE.sport ? [LEASE.sport] : []);
module.exports = {
SPORTS, isEnabled, state, evaluate, parseLease,
CONFIGURATION_SOURCE, LEASE_STATE, MAX_LEASE_MS, LEASABLE_SPORTS,
};
src/services/snapshotService.js
+5 -2
View File
@@ -319,8 +319,11 @@ const lineageCanaryConfig = require('./lineageCanaryConfig');
const LINEAGE_CANARY_SPORTS = lineageCanaryConfig.SPORTS;
function lineageCanaryEnabled(sport) {
return lineageCanaryConfig.isEnabled(sport);
// `now` is threaded so the LEASE is evaluated at WRITE TIME, not at startup.
// A cached boolean here would reintroduce the exact defect the lease removes:
// an expired canary that keeps writing until somebody restarts the process.
function lineageCanaryEnabled(sport, now) {
return lineageCanaryConfig.isEnabled(sport, now);
}
/**