S8 (a1): ops — the product watches itself

Settlement alarm (settle-pass THROW pages high; morning zero-settle alarm
keyed off the Postgres ledger settle results, once per ET date, never on an
empty yesterday), per-sport 3-consecutive-slot failure pager (pure
opsWatch.createFailureTracker, pages once per losing streak), odds-api >=80%
quota alert (once per day, Redis-deduped), systemHealth (statfs + os mem,
disk>85 / mem>90 pages), daily 9 AM ET pulse (ONE notification: ledger rows
yesterday via ledgerService.countRowsForDate, settles 24h, quota, disk/mem,
desk line), docs/OPS-RUNBOOK.md (Uptime Kuma monitors, Coolify
deploy-failure -> ntfy, phone subscription). All copy VOICE v1.1 — deadpan,
numbers, no exclamation points (tests lint for it). 2398 -> 2437 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

specs/session-8-ops-watch.md

@@ -0,0 +1,83 @@
+# Session 8 (A1 board) — Ops: the product watches itself
+
+## Problem
+The pipeline alerts on snapshot success/failure and missed cron slots (S56),
+but the record can still die silently: a settlement pass that throws or
+quietly settles nothing, a sport erroring slot after slot, quota creeping to
+the block threshold, a full disk. Kev must be paged by the product, not
+discover it days later.
+
+## Scope (extends existing scheduler/services — nothing rebuilt)
+1. **Settlement alarm** (`snapshotScheduler.js`)
+   - Settle pass (outcomes or ledger) THROWS -> ntfy priority `high`.
+   - Zero-settle alarm: at the MORNING slot only (first configured hour
+     >= 06 UTC; 14 UTC default = 10 AM ET, the first slot after overnight
+     completions), alert when the ledger settle pass found rows from before
+     today but settled none. **Chosen signal (documented): the ledger settle
+     pass's own return values** (`settleLedger` fetches
+     `game_date < today AND outcome IS NULL` from Postgres). It is the
+     cheapest (zero extra reads) and the most reliable: Postgres survives the
+     Redis TTL expiry that silently killed morning settles in Session 60 —
+     a `snapshot:{sport}:previous` signal would vanish in exactly that
+     failure mode. Scoped to SETTLEABLE sports (mlb) because NBA/WNBA/soccer
+     rows legitimately stay pending until their settle feed exists.
+     A genuinely empty yesterday fetches 0 rows -> no alarm. Deduped once
+     per ET date via Redis `ops:settle_zero:{date}`.
+2. **Persistent snapshot failure** (`opsWatch.createFailureTracker`, pure)
+   - Per-sport consecutive count of bad outcomes (`status==='error'` or
+     `skipped` with reason `'no props'`) across cron slots (in-process Map).
+   - Pages ONCE (priority high) when a sport reaches 3 consecutive bad
+     slots; stays quiet at 4+; any good outcome resets and re-arms.
+3. **Quota alert** (`opsWatch.checkQuotaDaily`)
+   - After each snapshot run: odds-api usage >= 80% -> one alert per day
+     (Redis dedupe `ops:quota_day:{provider}:{YYYY-MM-DD}`). Complements the
+     once-per-MONTH quotaTracker warn.
+4. **Box health** (`src/services/systemHealth.js`, pure + injectable)
+   - `getSystemHealth({fsImpl, osImpl})` -> `{disk_pct, mem_pct}` via
+     `fs.promises.statfs('/')` + `os.freemem/totalmem`.
+   - `healthIssues()` flags disk > 85% / mem > 90%; checked in the daily
+     pulse; issues alert at priority high.
+5. **Daily pulse** (scheduler, 13:00 UTC = 9 AM EDT, dedupe by date
+   in-process + Redis `ops:pulse:{date}`), ONE notification:
+   ledger rows written yesterday (Supabase count via
+   `ledgerService.countRowsForDate`, `n/a` when unconfigured), settles in
+   the last 24h (from `outcomes:{sport}:log`), odds-api quota pct,
+   disk/mem, `desk pack: see /desk` placeholder.
+6. **Runbook** — `docs/OPS-RUNBOOK.md`: Uptime Kuma monitors, Coolify
+   deploy-failure webhook -> ntfy, phone subscription steps.
+
+## Data shapes
+- `createFailureTracker(threshold=3).record(sport, result)` ->
+  `{ sport, count, shouldPage }`
+- `zeroSettleAlarm(ledgerResults, settleable=['mlb'])` ->
+  `{ alarm, settled, pending }`
+- `checkQuotaDaily(deps)` -> `{ alerted, deduped?, pct }`
+- `getSystemHealth()` -> `{ disk_pct: 0-100|null, mem_pct: 0-100|null }`
+- `buildPulseMessage(pieces)` -> single multi-line string, VOICE v1.1
+  (deadpan, numbers, zero exclamation points).
+
+## Acceptance criteria
+- Settle-pass throw -> one high-priority ntfy; zero-settle morning alarm
+  fires only when Postgres held pre-today unsettled rows; empty yesterday
+  never alerts.
+- 3 consecutive bad slots page exactly once per losing streak; success
+  resets.
+- Quota >= 80% alerts at most once per day.
+- Pulse is ONE notification, fields present, `n/a` when Supabase off,
+  no `!` anywhere in any alert copy.
+- One real test alert POSTed to the live ntfy channel (HTTP result in the
+  session report). Full jest green.
+
+## Test plan
+- `tests/unit/opsWatch.test.js` — tracker (increment/page-once/reset,
+  per-sport isolation), zero-settle signal, morning-hour resolution, quota
+  dedupe (per-day, injected now), pulse assembly (fields + no-`!` lint),
+  recent-settle counting.
+- `tests/unit/systemHealth.test.js` — injected fs/os math, statfs failure
+  degrades to null, threshold flags at 85/90 boundaries.
+- `tests/unit/opsScheduler.test.js` — scheduler wiring: settle-throw alert,
+  3-slot page via ticks, pulse fires once at 13:00 with Redis dedupe.
+
+## Zero out-of-pocket
+ntfy.sh (existing channel) + Uptime Kuma (already on the box) + node
+built-ins (`fs.statfs`, `os`). No new dependency, no paid service.