Merge S8 (a1): ops — the product watches itself
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> # Conflicts: # BUILD-STATE.md # CLAUDE.md # src/snapshotScheduler.js
This commit is contained in:
@@ -84,6 +84,34 @@ work-order 1.6 closed (canonical player keys + slate join invariant),
|
||||
Phase 2 slate UX, Phase 3 mobile P0. DEPLOY GATE: Coolify by Sat 10 AM ET;
|
||||
first snapshot locks against freshly posted lines (manual internal trigger).
|
||||
|
||||
## Session 8 (A1 board, 2026-07-11) — SHIPPED ✅ OPS: THE PRODUCT WATCHES ITSELF
|
||||
|
||||
Branch off day1/a1-board tip. 2398 → 2437 tests (208 suites). Backend-only.
|
||||
- **Settlement alarm** (`snapshotScheduler`): outcome/ledger settle pass
|
||||
THROW → ntfy high. Morning zero-settle alarm: ledger settle found
|
||||
pre-today Postgres rows but settled none → page once per ET date
|
||||
(signal = settleLedger's own return values — survives the Redis-TTL
|
||||
failure mode that silently killed morning settles in S60; documented in
|
||||
`opsWatch.zeroSettleAlarm`). Genuinely empty yesterday never alerts.
|
||||
- **Persistent-failure pager** (`opsWatch.createFailureTracker`, pure):
|
||||
3 consecutive error / skipped-'no props' slots for a sport pages once
|
||||
(high); any good slot resets and re-arms. Single-slot misses stay quiet.
|
||||
- **Quota** (`opsWatch.checkQuotaDaily`): odds-api ≥80% after a snapshot
|
||||
run → one alert/day, Redis-deduped (`ops:quota_day:{provider}:{date}`).
|
||||
- **Box health** (`services/systemHealth`, pure + injectable): fs.statfs
|
||||
+ os mem → `{disk_pct, mem_pct}`; pages high at disk>85 / mem>90.
|
||||
- **Daily pulse** 13:00 UTC (9 AM EDT; `PULSE_HOUR_UTC` to move), ONE
|
||||
notification: ledger rows yesterday (`ledgerService.countRowsForDate`,
|
||||
'n/a' when Supabase off — never a fabricated 0), settles last 24h,
|
||||
quota pct, disk/mem, 'desk pack: see /desk'. Dedupe: in-process date +
|
||||
`ops:pulse:{date}`.
|
||||
- **docs/OPS-RUNBOOK.md**: Uptime Kuma monitors (vyndr.app 200, api
|
||||
/api/health keyword 'healthy', /api/snapshot/summary), Coolify
|
||||
deploy-failure webhook → ntfy, ntfy phone-subscription steps.
|
||||
- Live acceptance alert POSTed to ntfy.sh/vyndr-pipeline-kev2026.
|
||||
- All alert copy VOICE v1.1 — deadpan, numbers, zero exclamation points
|
||||
(tests lint for `!`). Spec: `specs/session-8-ops-watch.md`.
|
||||
|
||||
## Session 60 (night2/full-board, 2026-07-11) — BUILT ✅ THE WHOLE BOARD (awaiting "merge the train")
|
||||
|
||||
Branch `night2/full-board` off d10bb4c — ZERO pushes to main. 2352 →
|
||||
|
||||
Reference in New Issue
Block a user