vyndr/docs/OPS-RUNBOOK.md

VYNDR OPS RUNBOOK — the product watches itself (Session 8, A1 board)

The in-app half (settle alarms, failure pager, quota check, box health, daily pulse) lives in src/snapshotScheduler.js + src/services/opsWatch.js + src/services/systemHealth.js and pushes to ntfy via src/utils/opsNotify.js. This runbook is the BOX-SIDE half: the monitors and hooks that catch the cases the app cannot report on itself — a dead container, a failed deploy, a dead box.

Channel: https://ntfy.sh/vyndr-pipeline-kev2026 (override with NTFY_URL / NTFY_TOPIC; both default in opsNotify.js). Zero out-of-pocket: ntfy.sh is free, Uptime Kuma is already on the box.

---

1. Phone subscription (do this first — alerts nobody reads are logs)

Android / iOS:

1. Install ntfy (Play Store, App Store, or F-Droid).
2. Open the app → + Subscribe to topic.
3. Server: https://ntfy.sh (default). Topic: vyndr-pipeline-kev2026.
4. Android: in the subscription's settings enable Instant delivery (foreground service) and exempt ntfy from battery optimization (Settings → Apps → ntfy → Battery → Unrestricted). Without this, Android can delay high-priority pages by hours.
5. Send yourself a probe and confirm the phone buzzes: curl -s -d "probe" https://ntfy.sh/vyndr-pipeline-kev2026

Desktop (optional): open https://ntfy.sh/vyndr-pipeline-kev2026 in a browser tab and allow notifications.

Note: the topic name is the only secret. Anyone who knows it can read and write the channel. If it ever leaks, rotate via NTFY_TOPIC on the container and re-subscribe the phone.

2. Uptime Kuma monitors (external heartbeat — catches a dead container)

Uptime Kuma already runs on the box. Add THREE monitors (Add New Monitor for each):

#	Name	Type	URL	Pass condition	Interval
1	vyndr web	HTTP(s)	https://vyndr.app	status 200	60s, retries 3
2	vyndr api health	HTTP(s) — Keyword	https://api.vyndr.app/api/health	keyword healthy present	60s, retries 3
3	vyndr snapshot summary	HTTP(s)	https://api.vyndr.app/api/snapshot/summary	status 200	300s, retries 3

Why keyword on #2: /api/health answers 200 "status":"healthy" only when Redis AND Supabase check out; degraded mode returns 503 "degraded". The keyword check fails on BOTH a dead container and a degraded one. Monitor #3 is the public cache-only snapshot read — it proves the pipeline's output surface is serving, and it can never drain provider quota.

Wire Kuma to the pager:

1. Settings → Notifications → Setup Notification → ntfy.
2. Server URL https://ntfy.sh, topic vyndr-pipeline-kev2026, priority: default 5 (Kuma sends its own priority on down events).
3. Check Default enabled so future monitors inherit it, then attach the notification to all three monitors above.

3. Coolify deploy-failure -> ntfy

Coolify owns deploys, so the app can't report its own failed deploy. Two options, in order of preference:

A. Native notification channel (Coolify v4):

1. Coolify → Notifications (team level).
2. If your Coolify version lists ntfy: server https://ntfy.sh, topic vyndr-pipeline-kev2026, enable only the Deployment Failed (and optionally Deployment Success) events.
3. If it doesn't list ntfy, use the generic Webhook channel pointed at https://ntfy.sh/vyndr-pipeline-kev2026. ntfy accepts any POST body as the message text — a JSON payload arrives readable, just ugly.

B. Per-app webhook (any Coolify version):

1. App → Webhooks → add https://ntfy.sh/vyndr-pipeline-kev2026 for the deployment-failed event.
2. Verify by triggering a deploy of a branch that fails its build; the phone should receive the payload within seconds.

Either way, send one manual probe to prove the path before trusting it: curl -s -d "coolify webhook path test" https://ntfy.sh/vyndr-pipeline-kev2026

4. What the app already pages on its own (for reference — do not duplicate)

Alert	Source	Priority
Snapshot success / 0-props / hard failure	snapshotService (S56)	default / low / high
Missed cron slot (per-minute watchdog)	snapshotScheduler (S56)	high
Settlement pass THREW (outcomes or ledger)	snapshotScheduler (S8)	high
Morning settle closed with 0 settles while Postgres held pre-today pending rows	snapshotScheduler + opsWatch.zeroSettleAlarm (S8)	high
Sport produced nothing 3 consecutive slots	opsWatch.createFailureTracker (S8)	high
odds-api >= 80% quota (once per day)	opsWatch.checkQuotaDaily (S8)	high
Disk > 85% or memory > 90%	systemHealth via daily pulse (S8)	high
Daily pulse, 9 AM ET (13:00 UTC, PULSE_HOUR_UTC to move)	snapshotScheduler (S8)	default

Kill switches: PIPELINE_ALERTS=0 silences everything app-side; NODE_ENV=test is silent by design.

5. Weekly 5-minute sanity pass

1. Phone got yesterday's 9 AM pulse. If not: check ops:pulse:* in Redis and the container log for [opsWatch] armed.
2. GET /api/internal/snapshot/status (header x-internal-key) shows cron_armed: true, fresh last_snapshot, overdue: false.
3. Uptime Kuma dashboard: three greens.
4. Disk line in the pulse trending up week over week means log rotation or Redis growth needs a look before the 85% page ever fires.