Merge S8 (a1): ops — the product watches itself

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

# Conflicts:
#	BUILD-STATE.md
#	CLAUDE.md
#	src/snapshotScheduler.js
This commit is contained in:
Kev
2026-07-11 14:38:06 -04:00
11 changed files with 1081 additions and 3 deletions
+28
View File
@@ -84,6 +84,34 @@ work-order 1.6 closed (canonical player keys + slate join invariant),
Phase 2 slate UX, Phase 3 mobile P0. DEPLOY GATE: Coolify by Sat 10 AM ET;
first snapshot locks against freshly posted lines (manual internal trigger).
## Session 8 (A1 board, 2026-07-11) — SHIPPED ✅ OPS: THE PRODUCT WATCHES ITSELF
Branch off day1/a1-board tip. 2398 → 2437 tests (208 suites). Backend-only.
- **Settlement alarm** (`snapshotScheduler`): outcome/ledger settle pass
THROW → ntfy high. Morning zero-settle alarm: ledger settle found
pre-today Postgres rows but settled none → page once per ET date
(signal = settleLedger's own return values — survives the Redis-TTL
failure mode that silently killed morning settles in S60; documented in
`opsWatch.zeroSettleAlarm`). Genuinely empty yesterday never alerts.
- **Persistent-failure pager** (`opsWatch.createFailureTracker`, pure):
3 consecutive error / skipped-'no props' slots for a sport pages once
(high); any good slot resets and re-arms. Single-slot misses stay quiet.
- **Quota** (`opsWatch.checkQuotaDaily`): odds-api ≥80% after a snapshot
run → one alert/day, Redis-deduped (`ops:quota_day:{provider}:{date}`).
- **Box health** (`services/systemHealth`, pure + injectable): fs.statfs
+ os mem → `{disk_pct, mem_pct}`; pages high at disk>85 / mem>90.
- **Daily pulse** 13:00 UTC (9 AM EDT; `PULSE_HOUR_UTC` to move), ONE
notification: ledger rows yesterday (`ledgerService.countRowsForDate`,
'n/a' when Supabase off — never a fabricated 0), settles last 24h,
quota pct, disk/mem, 'desk pack: see /desk'. Dedupe: in-process date +
`ops:pulse:{date}`.
- **docs/OPS-RUNBOOK.md**: Uptime Kuma monitors (vyndr.app 200, api
/api/health keyword 'healthy', /api/snapshot/summary), Coolify
deploy-failure webhook → ntfy, ntfy phone-subscription steps.
- Live acceptance alert POSTed to ntfy.sh/vyndr-pipeline-kev2026.
- All alert copy VOICE v1.1 — deadpan, numbers, zero exclamation points
(tests lint for `!`). Spec: `specs/session-8-ops-watch.md`.
## Session 60 (night2/full-board, 2026-07-11) — BUILT ✅ THE WHOLE BOARD (awaiting "merge the train")
Branch `night2/full-board` off d10bb4c — ZERO pushes to main. 2352 →