# VYNDR OPS RUNBOOK — the product watches itself (Session 8, A1 board)

The in-app half (settle alarms, failure pager, quota check, box health, daily
pulse) lives in `src/snapshotScheduler.js` + `src/services/opsWatch.js` +
`src/services/systemHealth.js` and pushes to ntfy via `src/utils/opsNotify.js`.
This runbook is the BOX-SIDE half: the monitors and hooks that catch the cases
the app cannot report on itself — a dead container, a failed deploy, a dead box.

Channel: `https://ntfy.sh/vyndr-pipeline-kev2026`
(override with `NTFY_URL` / `NTFY_TOPIC`; both default in `opsNotify.js`).
Zero out-of-pocket: ntfy.sh is free, Uptime Kuma is already on the box.

---

## 1. Phone subscription (do this first — alerts nobody reads are logs)

Android / iOS:
1. Install **ntfy** (Play Store, App Store, or F-Droid).
2. Open the app → **+ Subscribe to topic**.
3. Server: `https://ntfy.sh` (default). Topic: `vyndr-pipeline-kev2026`.
4. Android: in the subscription's settings enable **Instant delivery**
   (foreground service) and exempt ntfy from battery optimization
   (Settings → Apps → ntfy → Battery → Unrestricted). Without this,
   Android can delay high-priority pages by hours.
5. Send yourself a probe and confirm the phone buzzes:
   `curl -s -d "probe" https://ntfy.sh/vyndr-pipeline-kev2026`

Desktop (optional): open `https://ntfy.sh/vyndr-pipeline-kev2026` in a browser
tab and allow notifications.

Note: the topic name is the only secret. Anyone who knows it can read and
write the channel. If it ever leaks, rotate via `NTFY_TOPIC` on the container
and re-subscribe the phone.

## 2. Uptime Kuma monitors (external heartbeat — catches a dead container)

Uptime Kuma already runs on the box. Add THREE monitors
(**Add New Monitor** for each):

| # | Name | Type | URL | Pass condition | Interval |
|---|------|------|-----|----------------|----------|
| 1 | `vyndr web` | HTTP(s) | `https://vyndr.app` | status 200 | 60s, retries 3 |
| 2 | `vyndr api health` | HTTP(s) — **Keyword** | `https://api.vyndr.app/api/health` | keyword `healthy` present | 60s, retries 3 |
| 3 | `vyndr snapshot summary` | HTTP(s) | `https://api.vyndr.app/api/snapshot/summary` | status 200 | 300s, retries 3 |

Why keyword on #2: `/api/health` answers 200 `"status":"healthy"` only when
Redis AND Supabase check out; degraded mode returns 503 `"degraded"`. The
keyword check fails on BOTH a dead container and a degraded one.
Monitor #3 is the public cache-only snapshot read — it proves the pipeline's
output surface is serving, and it can never drain provider quota.

Wire Kuma to the pager:
1. **Settings → Notifications → Setup Notification → ntfy**.
2. Server URL `https://ntfy.sh`, topic `vyndr-pipeline-kev2026`,
   priority: default 5 (Kuma sends its own priority on down events).
3. Check **Default enabled** so future monitors inherit it, then attach the
   notification to all three monitors above.

## 3. Coolify deploy-failure -> ntfy

Coolify owns deploys, so the app can't report its own failed deploy. Two
options, in order of preference:

**A. Native notification channel (Coolify v4):**
1. Coolify → **Notifications** (team level).
2. If your Coolify version lists **ntfy**: server `https://ntfy.sh`, topic
   `vyndr-pipeline-kev2026`, enable only the **Deployment Failed** (and
   optionally **Deployment Success**) events.
3. If it doesn't list ntfy, use the generic **Webhook** channel pointed at
   `https://ntfy.sh/vyndr-pipeline-kev2026`. ntfy accepts any POST body as
   the message text — a JSON payload arrives readable, just ugly.

**B. Per-app webhook (any Coolify version):**
1. App → **Webhooks** → add `https://ntfy.sh/vyndr-pipeline-kev2026` for the
   deployment-failed event.
2. Verify by triggering a deploy of a branch that fails its build; the phone
   should receive the payload within seconds.

Either way, send one manual probe to prove the path before trusting it:
`curl -s -d "coolify webhook path test" https://ntfy.sh/vyndr-pipeline-kev2026`

## 4. What the app already pages on its own (for reference — do not duplicate)

| Alert | Source | Priority |
|-------|--------|----------|
| Snapshot success / 0-props / hard failure | `snapshotService` (S56) | default / low / high |
| Missed cron slot (per-minute watchdog) | `snapshotScheduler` (S56) | high |
| Settlement pass THREW (outcomes or ledger) | `snapshotScheduler` (S8) | high |
| Morning settle closed with 0 settles while Postgres held pre-today pending rows | `snapshotScheduler` + `opsWatch.zeroSettleAlarm` (S8) | high |
| Sport produced nothing 3 consecutive slots | `opsWatch.createFailureTracker` (S8) | high |
| odds-api >= 80% quota (once per day) | `opsWatch.checkQuotaDaily` (S8) | high |
| Disk > 85% or memory > 90% | `systemHealth` via daily pulse (S8) | high |
| Daily pulse, 9 AM ET (13:00 UTC, `PULSE_HOUR_UTC` to move) | `snapshotScheduler` (S8) | default |

Kill switches: `PIPELINE_ALERTS=0` silences everything app-side;
`NODE_ENV=test` is silent by design.

## 5. Weekly 5-minute sanity pass

1. Phone got yesterday's 9 AM pulse. If not: check `ops:pulse:*` in Redis and
   the container log for `[opsWatch] armed`.
2. `GET /api/internal/snapshot/status` (header `x-internal-key`) shows
   `cron_armed: true`, fresh `last_snapshot`, `overdue: false`.
3. Uptime Kuma dashboard: three greens.
4. Disk line in the pulse trending up week over week means log rotation or
   Redis growth needs a look before the 85% page ever fires.