vyndr/specs/session-8-ops-watch.md

Session 8 (A1 board) — Ops: the product watches itself

Problem

The pipeline alerts on snapshot success/failure and missed cron slots (S56), but the record can still die silently: a settlement pass that throws or quietly settles nothing, a sport erroring slot after slot, quota creeping to the block threshold, a full disk. Kev must be paged by the product, not discover it days later.

Scope (extends existing scheduler/services — nothing rebuilt)

1. Settlement alarm (snapshotScheduler.js)
   • Settle pass (outcomes or ledger) THROWS -> ntfy priority high.
   • Zero-settle alarm: at the MORNING slot only (first configured hour

     = 06 UTC; 14 UTC default = 10 AM ET, the first slot after overnight completions), alert when the ledger settle pass found rows from before today but settled none. Chosen signal (documented): the ledger settle pass's own return values (settleLedger fetches game_date < today AND outcome IS NULL from Postgres). It is the cheapest (zero extra reads) and the most reliable: Postgres survives the Redis TTL expiry that silently killed morning settles in Session 60 — a snapshot:{sport}:previous signal would vanish in exactly that failure mode. Scoped to SETTLEABLE sports (mlb) because NBA/WNBA/soccer rows legitimately stay pending until their settle feed exists. A genuinely empty yesterday fetches 0 rows -> no alarm. Deduped once per ET date via Redis ops:settle_zero:{date}.

2. Persistent snapshot failure (opsWatch.createFailureTracker, pure)
   • Per-sport consecutive count of bad outcomes (status==='error' or skipped with reason 'no props') across cron slots (in-process Map).
   • Pages ONCE (priority high) when a sport reaches 3 consecutive bad slots; stays quiet at 4+; any good outcome resets and re-arms.
3. Quota alert (opsWatch.checkQuotaDaily)
   • After each snapshot run: odds-api usage >= 80% -> one alert per day (Redis dedupe ops:quota_day:{provider}:{YYYY-MM-DD}). Complements the once-per-MONTH quotaTracker warn.
4. Box health (src/services/systemHealth.js, pure + injectable)
   • getSystemHealth({fsImpl, osImpl}) -> {disk_pct, mem_pct} via fs.promises.statfs('/') + os.freemem/totalmem.
   • healthIssues() flags disk > 85% / mem > 90%; checked in the daily pulse; issues alert at priority high.
5. Daily pulse (scheduler, 13:00 UTC = 9 AM EDT, dedupe by date in-process + Redis ops:pulse:{date}), ONE notification: ledger rows written yesterday (Supabase count via ledgerService.countRowsForDate, n/a when unconfigured), settles in the last 24h (from outcomes:{sport}:log), odds-api quota pct, disk/mem, desk pack: see /desk placeholder.
6. Runbook — docs/OPS-RUNBOOK.md: Uptime Kuma monitors, Coolify deploy-failure webhook -> ntfy, phone subscription steps.

Data shapes

• createFailureTracker(threshold=3).record(sport, result) -> { sport, count, shouldPage }
• zeroSettleAlarm(ledgerResults, settleable=['mlb']) -> { alarm, settled, pending }
• checkQuotaDaily(deps) -> { alerted, deduped?, pct }
• getSystemHealth() -> { disk_pct: 0-100|null, mem_pct: 0-100|null }
• buildPulseMessage(pieces) -> single multi-line string, VOICE v1.1 (deadpan, numbers, zero exclamation points).

Acceptance criteria

• Settle-pass throw -> one high-priority ntfy; zero-settle morning alarm fires only when Postgres held pre-today unsettled rows; empty yesterday never alerts.
• 3 consecutive bad slots page exactly once per losing streak; success resets.
• Quota >= 80% alerts at most once per day.
• Pulse is ONE notification, fields present, n/a when Supabase off, no ! anywhere in any alert copy.
• One real test alert POSTed to the live ntfy channel (HTTP result in the session report). Full jest green.

Test plan

• tests/unit/opsWatch.test.js — tracker (increment/page-once/reset, per-sport isolation), zero-settle signal, morning-hour resolution, quota dedupe (per-day, injected now), pulse assembly (fields + no-! lint), recent-settle counting.
• tests/unit/systemHealth.test.js — injected fs/os math, statfs failure degrades to null, threshold flags at 85/90 boundaries.
• tests/unit/opsScheduler.test.js — scheduler wiring: settle-throw alert, 3-slot page via ticks, pulse fires once at 13:00 with Redis dedupe.

Zero out-of-pocket

ntfy.sh (existing channel) + Uptime Kuma (already on the box) + node built-ins (fs.statfs, os). No new dependency, no paid service.