S8 (a1): ops — the product watches itself
Settlement alarm (settle-pass THROW pages high; morning zero-settle alarm keyed off the Postgres ledger settle results, once per ET date, never on an empty yesterday), per-sport 3-consecutive-slot failure pager (pure opsWatch.createFailureTracker, pages once per losing streak), odds-api >=80% quota alert (once per day, Redis-deduped), systemHealth (statfs + os mem, disk>85 / mem>90 pages), daily 9 AM ET pulse (ONE notification: ledger rows yesterday via ledgerService.countRowsForDate, settles 24h, quota, disk/mem, desk line), docs/OPS-RUNBOOK.md (Uptime Kuma monitors, Coolify deploy-failure -> ntfy, phone subscription). All copy VOICE v1.1 — deadpan, numbers, no exclamation points (tests lint for it). 2398 -> 2437 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,106 @@
|
||||
# VYNDR OPS RUNBOOK — the product watches itself (Session 8, A1 board)
|
||||
|
||||
The in-app half (settle alarms, failure pager, quota check, box health, daily
|
||||
pulse) lives in `src/snapshotScheduler.js` + `src/services/opsWatch.js` +
|
||||
`src/services/systemHealth.js` and pushes to ntfy via `src/utils/opsNotify.js`.
|
||||
This runbook is the BOX-SIDE half: the monitors and hooks that catch the cases
|
||||
the app cannot report on itself — a dead container, a failed deploy, a dead box.
|
||||
|
||||
Channel: `https://ntfy.sh/vyndr-pipeline-kev2026`
|
||||
(override with `NTFY_URL` / `NTFY_TOPIC`; both default in `opsNotify.js`).
|
||||
Zero out-of-pocket: ntfy.sh is free, Uptime Kuma is already on the box.
|
||||
|
||||
---
|
||||
|
||||
## 1. Phone subscription (do this first — alerts nobody reads are logs)
|
||||
|
||||
Android / iOS:
|
||||
1. Install **ntfy** (Play Store, App Store, or F-Droid).
|
||||
2. Open the app → **+ Subscribe to topic**.
|
||||
3. Server: `https://ntfy.sh` (default). Topic: `vyndr-pipeline-kev2026`.
|
||||
4. Android: in the subscription's settings enable **Instant delivery**
|
||||
(foreground service) and exempt ntfy from battery optimization
|
||||
(Settings → Apps → ntfy → Battery → Unrestricted). Without this,
|
||||
Android can delay high-priority pages by hours.
|
||||
5. Send yourself a probe and confirm the phone buzzes:
|
||||
`curl -s -d "probe" https://ntfy.sh/vyndr-pipeline-kev2026`
|
||||
|
||||
Desktop (optional): open `https://ntfy.sh/vyndr-pipeline-kev2026` in a browser
|
||||
tab and allow notifications.
|
||||
|
||||
Note: the topic name is the only secret. Anyone who knows it can read and
|
||||
write the channel. If it ever leaks, rotate via `NTFY_TOPIC` on the container
|
||||
and re-subscribe the phone.
|
||||
|
||||
## 2. Uptime Kuma monitors (external heartbeat — catches a dead container)
|
||||
|
||||
Uptime Kuma already runs on the box. Add THREE monitors
|
||||
(**Add New Monitor** for each):
|
||||
|
||||
| # | Name | Type | URL | Pass condition | Interval |
|
||||
|---|------|------|-----|----------------|----------|
|
||||
| 1 | `vyndr web` | HTTP(s) | `https://vyndr.app` | status 200 | 60s, retries 3 |
|
||||
| 2 | `vyndr api health` | HTTP(s) — **Keyword** | `https://api.vyndr.app/api/health` | keyword `healthy` present | 60s, retries 3 |
|
||||
| 3 | `vyndr snapshot summary` | HTTP(s) | `https://api.vyndr.app/api/snapshot/summary` | status 200 | 300s, retries 3 |
|
||||
|
||||
Why keyword on #2: `/api/health` answers 200 `"status":"healthy"` only when
|
||||
Redis AND Supabase check out; degraded mode returns 503 `"degraded"`. The
|
||||
keyword check fails on BOTH a dead container and a degraded one.
|
||||
Monitor #3 is the public cache-only snapshot read — it proves the pipeline's
|
||||
output surface is serving, and it can never drain provider quota.
|
||||
|
||||
Wire Kuma to the pager:
|
||||
1. **Settings → Notifications → Setup Notification → ntfy**.
|
||||
2. Server URL `https://ntfy.sh`, topic `vyndr-pipeline-kev2026`,
|
||||
priority: default 5 (Kuma sends its own priority on down events).
|
||||
3. Check **Default enabled** so future monitors inherit it, then attach the
|
||||
notification to all three monitors above.
|
||||
|
||||
## 3. Coolify deploy-failure -> ntfy
|
||||
|
||||
Coolify owns deploys, so the app can't report its own failed deploy. Two
|
||||
options, in order of preference:
|
||||
|
||||
**A. Native notification channel (Coolify v4):**
|
||||
1. Coolify → **Notifications** (team level).
|
||||
2. If your Coolify version lists **ntfy**: server `https://ntfy.sh`, topic
|
||||
`vyndr-pipeline-kev2026`, enable only the **Deployment Failed** (and
|
||||
optionally **Deployment Success**) events.
|
||||
3. If it doesn't list ntfy, use the generic **Webhook** channel pointed at
|
||||
`https://ntfy.sh/vyndr-pipeline-kev2026`. ntfy accepts any POST body as
|
||||
the message text — a JSON payload arrives readable, just ugly.
|
||||
|
||||
**B. Per-app webhook (any Coolify version):**
|
||||
1. App → **Webhooks** → add `https://ntfy.sh/vyndr-pipeline-kev2026` for the
|
||||
deployment-failed event.
|
||||
2. Verify by triggering a deploy of a branch that fails its build; the phone
|
||||
should receive the payload within seconds.
|
||||
|
||||
Either way, send one manual probe to prove the path before trusting it:
|
||||
`curl -s -d "coolify webhook path test" https://ntfy.sh/vyndr-pipeline-kev2026`
|
||||
|
||||
## 4. What the app already pages on its own (for reference — do not duplicate)
|
||||
|
||||
| Alert | Source | Priority |
|
||||
|-------|--------|----------|
|
||||
| Snapshot success / 0-props / hard failure | `snapshotService` (S56) | default / low / high |
|
||||
| Missed cron slot (per-minute watchdog) | `snapshotScheduler` (S56) | high |
|
||||
| Settlement pass THREW (outcomes or ledger) | `snapshotScheduler` (S8) | high |
|
||||
| Morning settle closed with 0 settles while Postgres held pre-today pending rows | `snapshotScheduler` + `opsWatch.zeroSettleAlarm` (S8) | high |
|
||||
| Sport produced nothing 3 consecutive slots | `opsWatch.createFailureTracker` (S8) | high |
|
||||
| odds-api >= 80% quota (once per day) | `opsWatch.checkQuotaDaily` (S8) | high |
|
||||
| Disk > 85% or memory > 90% | `systemHealth` via daily pulse (S8) | high |
|
||||
| Daily pulse, 9 AM ET (13:00 UTC, `PULSE_HOUR_UTC` to move) | `snapshotScheduler` (S8) | default |
|
||||
|
||||
Kill switches: `PIPELINE_ALERTS=0` silences everything app-side;
|
||||
`NODE_ENV=test` is silent by design.
|
||||
|
||||
## 5. Weekly 5-minute sanity pass
|
||||
|
||||
1. Phone got yesterday's 9 AM pulse. If not: check `ops:pulse:*` in Redis and
|
||||
the container log for `[opsWatch] armed`.
|
||||
2. `GET /api/internal/snapshot/status` (header `x-internal-key`) shows
|
||||
`cron_armed: true`, fresh `last_snapshot`, `overdue: false`.
|
||||
3. Uptime Kuma dashboard: three greens.
|
||||
4. Disk line in the pulse trending up week over week means log rotation or
|
||||
Redis growth needs a look before the 85% page ever fires.
|
||||
Reference in New Issue
Block a user