S8 (a1): ops — the product watches itself
Settlement alarm (settle-pass THROW pages high; morning zero-settle alarm keyed off the Postgres ledger settle results, once per ET date, never on an empty yesterday), per-sport 3-consecutive-slot failure pager (pure opsWatch.createFailureTracker, pages once per losing streak), odds-api >=80% quota alert (once per day, Redis-deduped), systemHealth (statfs + os mem, disk>85 / mem>90 pages), daily 9 AM ET pulse (ONE notification: ledger rows yesterday via ledgerService.countRowsForDate, settles 24h, quota, disk/mem, desk line), docs/OPS-RUNBOOK.md (Uptime Kuma monitors, Coolify deploy-failure -> ntfy, phone subscription). All copy VOICE v1.1 — deadpan, numbers, no exclamation points (tests lint for it). 2398 -> 2437 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -801,6 +801,32 @@ phased plan in the Session-57 conversation / BUILD-STATE Next section).
|
||||
amber / VALUE green / red = STALE-or-miss only), mono data surfaces, and
|
||||
schedule-derived waiting copy. Keep them green.
|
||||
|
||||
## Ops Self-Watch (Session 8, A1 board — non-obvious)
|
||||
- **Pure alarm logic lives in `src/services/opsWatch.js`** (failure tracker,
|
||||
zero-settle signal, quota daily check, pulse assembly) + `src/services/
|
||||
systemHealth.js` (statfs/os, injectable). `snapshotScheduler` only WIRES
|
||||
them — keep new alarm logic in opsWatch so it stays unit-testable.
|
||||
- **Zero-settle signal = the ledger settle's own return values** (Postgres
|
||||
`game_date < today AND outcome IS NULL`), NOT `snapshot:{sport}:previous` —
|
||||
the snapshot key expires in exactly the failure mode the alarm exists to
|
||||
catch (the S60 SNAP_TTL bug). Scoped to `SETTLEABLE_SPORTS` (mlb);
|
||||
NBA/WNBA rows legitimately stay pending and must not page daily. Morning
|
||||
slot only (`morningHourUtc` = first configured hour ≥ 06 UTC).
|
||||
- **Failure pager pages ONCE per losing streak** (exactly at count 3), not
|
||||
once per failing slot; `status:'skipped', reason:'no grades'` is NOT a
|
||||
failure (props arrived, grader refused) and resets the counter.
|
||||
- **Daily pulse** fires at `PULSE_HOUR_UTC` (default 13 = 9 AM EDT), one
|
||||
notification, dual dedupe (in-process date + Redis `ops:pulse:{date}`).
|
||||
Missing data renders 'n/a' — a pulse never fabricates a zero (Data
|
||||
Semantics Rule applies to ops copy). All alert copy: no exclamation
|
||||
points, ever (VOICE v1.1) — tests lint for it.
|
||||
- **Single-suite jest runs of scheduler tests can hang at exit** when redis
|
||||
is down (ioredis reconnect timer keeps the process alive) — PRE-EXISTING
|
||||
at baseline, full-suite runs are unaffected. Don't chase it as a leak in
|
||||
new code; inject cacheGet/cacheSet in tests to avoid creating a client.
|
||||
- Box-side ops (Uptime Kuma, Coolify deploy-failure webhook, ntfy phone
|
||||
setup) = `docs/OPS-RUNBOOK.md`.
|
||||
|
||||
## Active Skills
|
||||
- vyndr-voice (all user-facing output)
|
||||
- prop-analysis (grading methodology)
|
||||
|
||||
Reference in New Issue
Block a user