# Session 8 (A1 board) — Ops: the product watches itself

## Problem
The pipeline alerts on snapshot success/failure and missed cron slots (S56),
but the record can still die silently: a settlement pass that throws or
quietly settles nothing, a sport erroring slot after slot, quota creeping to
the block threshold, a full disk. Kev must be paged by the product, not
discover it days later.

## Scope (extends existing scheduler/services — nothing rebuilt)
1. **Settlement alarm** (`snapshotScheduler.js`)
   - Settle pass (outcomes or ledger) THROWS -> ntfy priority `high`.
   - Zero-settle alarm: at the MORNING slot only (first configured hour
     >= 06 UTC; 14 UTC default = 10 AM ET, the first slot after overnight
     completions), alert when the ledger settle pass found rows from before
     today but settled none. **Chosen signal (documented): the ledger settle
     pass's own return values** (`settleLedger` fetches
     `game_date < today AND outcome IS NULL` from Postgres). It is the
     cheapest (zero extra reads) and the most reliable: Postgres survives the
     Redis TTL expiry that silently killed morning settles in Session 60 —
     a `snapshot:{sport}:previous` signal would vanish in exactly that
     failure mode. Scoped to SETTLEABLE sports (mlb) because NBA/WNBA/soccer
     rows legitimately stay pending until their settle feed exists.
     A genuinely empty yesterday fetches 0 rows -> no alarm. Deduped once
     per ET date via Redis `ops:settle_zero:{date}`.
2. **Persistent snapshot failure** (`opsWatch.createFailureTracker`, pure)
   - Per-sport consecutive count of bad outcomes (`status==='error'` or
     `skipped` with reason `'no props'`) across cron slots (in-process Map).
   - Pages ONCE (priority high) when a sport reaches 3 consecutive bad
     slots; stays quiet at 4+; any good outcome resets and re-arms.
3. **Quota alert** (`opsWatch.checkQuotaDaily`)
   - After each snapshot run: odds-api usage >= 80% -> one alert per day
     (Redis dedupe `ops:quota_day:{provider}:{YYYY-MM-DD}`). Complements the
     once-per-MONTH quotaTracker warn.
4. **Box health** (`src/services/systemHealth.js`, pure + injectable)
   - `getSystemHealth({fsImpl, osImpl})` -> `{disk_pct, mem_pct}` via
     `fs.promises.statfs('/')` + `os.freemem/totalmem`.
   - `healthIssues()` flags disk > 85% / mem > 90%; checked in the daily
     pulse; issues alert at priority high.
5. **Daily pulse** (scheduler, 13:00 UTC = 9 AM EDT, dedupe by date
   in-process + Redis `ops:pulse:{date}`), ONE notification:
   ledger rows written yesterday (Supabase count via
   `ledgerService.countRowsForDate`, `n/a` when unconfigured), settles in
   the last 24h (from `outcomes:{sport}:log`), odds-api quota pct,
   disk/mem, `desk pack: see /desk` placeholder.
6. **Runbook** — `docs/OPS-RUNBOOK.md`: Uptime Kuma monitors, Coolify
   deploy-failure webhook -> ntfy, phone subscription steps.

## Data shapes
- `createFailureTracker(threshold=3).record(sport, result)` ->
  `{ sport, count, shouldPage }`
- `zeroSettleAlarm(ledgerResults, settleable=['mlb'])` ->
  `{ alarm, settled, pending }`
- `checkQuotaDaily(deps)` -> `{ alerted, deduped?, pct }`
- `getSystemHealth()` -> `{ disk_pct: 0-100|null, mem_pct: 0-100|null }`
- `buildPulseMessage(pieces)` -> single multi-line string, VOICE v1.1
  (deadpan, numbers, zero exclamation points).

## Acceptance criteria
- Settle-pass throw -> one high-priority ntfy; zero-settle morning alarm
  fires only when Postgres held pre-today unsettled rows; empty yesterday
  never alerts.
- 3 consecutive bad slots page exactly once per losing streak; success
  resets.
- Quota >= 80% alerts at most once per day.
- Pulse is ONE notification, fields present, `n/a` when Supabase off,
  no `!` anywhere in any alert copy.
- One real test alert POSTed to the live ntfy channel (HTTP result in the
  session report). Full jest green.

## Test plan
- `tests/unit/opsWatch.test.js` — tracker (increment/page-once/reset,
  per-sport isolation), zero-settle signal, morning-hour resolution, quota
  dedupe (per-day, injected now), pulse assembly (fields + no-`!` lint),
  recent-settle counting.
- `tests/unit/systemHealth.test.js` — injected fs/os math, statfs failure
  degrades to null, threshold flags at 85/90 boundaries.
- `tests/unit/opsScheduler.test.js` — scheduler wiring: settle-throw alert,
  3-slot page via ticks, pulse fires once at 13:00 with Redis dedupe.

## Zero out-of-pocket
ntfy.sh (existing channel) + Uptime Kuma (already on the box) + node
built-ins (`fs.statfs`, `os`). No new dependency, no paid service.