S8 (a1): ops — the product watches itself

Settlement alarm (settle-pass THROW pages high; morning zero-settle alarm
keyed off the Postgres ledger settle results, once per ET date, never on an
empty yesterday), per-sport 3-consecutive-slot failure pager (pure
opsWatch.createFailureTracker, pages once per losing streak), odds-api >=80%
quota alert (once per day, Redis-deduped), systemHealth (statfs + os mem,
disk>85 / mem>90 pages), daily 9 AM ET pulse (ONE notification: ledger rows
yesterday via ledgerService.countRowsForDate, settles 24h, quota, disk/mem,
desk line), docs/OPS-RUNBOOK.md (Uptime Kuma monitors, Coolify
deploy-failure -> ntfy, phone subscription). All copy VOICE v1.1 — deadpan,
numbers, no exclamation points (tests lint for it). 2398 -> 2437 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Kev
2026-07-11 14:34:45 -04:00
parent e4d2e79f95
commit 044941d406
11 changed files with 1083 additions and 4 deletions
+26
View File
@@ -801,6 +801,32 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
amber / VALUE green / red = STALE-or-miss only), mono data surfaces, and
schedule-derived waiting copy. Keep them green.
## Ops Self-Watch (Session 8, A1 board — non-obvious)
- **Pure alarm logic lives in `src/services/opsWatch.js`** (failure tracker,
zero-settle signal, quota daily check, pulse assembly) + `src/services/
systemHealth.js` (statfs/os, injectable). `snapshotScheduler` only WIRES
them — keep new alarm logic in opsWatch so it stays unit-testable.
- **Zero-settle signal = the ledger settle's own return values** (Postgres
`game_date < today AND outcome IS NULL`), NOT `snapshot:{sport}:previous` —
the snapshot key expires in exactly the failure mode the alarm exists to
catch (the S60 SNAP_TTL bug). Scoped to `SETTLEABLE_SPORTS` (mlb);
NBA/WNBA rows legitimately stay pending and must not page daily. Morning
slot only (`morningHourUtc` = first configured hour ≥ 06 UTC).
- **Failure pager pages ONCE per losing streak** (exactly at count 3), not
once per failing slot; `status:'skipped', reason:'no grades'` is NOT a
failure (props arrived, grader refused) and resets the counter.
- **Daily pulse** fires at `PULSE_HOUR_UTC` (default 13 = 9 AM EDT), one
notification, dual dedupe (in-process date + Redis `ops:pulse:{date}`).
Missing data renders 'n/a' — a pulse never fabricates a zero (Data
Semantics Rule applies to ops copy). All alert copy: no exclamation
points, ever (VOICE v1.1) — tests lint for it.
- **Single-suite jest runs of scheduler tests can hang at exit** when redis
is down (ioredis reconnect timer keeps the process alive) — PRE-EXISTING
at baseline, full-suite runs are unaffected. Don't chase it as a leak in
new code; inject cacheGet/cacheSet in tests to avoid creating a client.
- Box-side ops (Uptime Kuma, Coolify deploy-failure webhook, ntfy phone
setup) = `docs/OPS-RUNBOOK.md`.
## Active Skills
- vyndr-voice (all user-facing output)
- prop-analysis (grading methodology)