From cc8e47884d96d2dca5e9664ba3fbceea6e39c622 Mon Sep 17 00:00:00 2001 From: Kev Date: Sun, 19 Jul 2026 22:32:59 -0400 Subject: [PATCH] STATE.md: backup is durable on-box and verified by read-back (643==643) --- specs/STATE.md | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/specs/STATE.md b/specs/STATE.md index 1888b89..60a6a96 100644 --- a/specs/STATE.md +++ b/specs/STATE.md @@ -1,5 +1,41 @@ # VYNDR — STATE OF THE WORLD -### As of `a80868c` (main, DEPLOYED + fingerprinted live), 2026-07-19. This file opens every future session. Update it when a train ships. +### As of `HEADPLACEHOLDER` (main, DEPLOYED + fingerprinted live), 2026-07-20. This file opens every future session. Update it when a train ships. + +## 🟢 BACKUP — DURABLE ON-BOX, VERIFIED BY READ-BACK (2026-07-20, deployed) +The highest-severity open item is closed **on-box**. Off-box is a tracked follow-up. +- **Runs as CODE, not a cron somebody must install** (`src/backupScheduler.js`, + armed in `server.js`). Arming is opt-OUT: armed whenever `SUPABASE_DB_URL` + exists, `BACKUP_CRON=0` kills it, nightly `BACKUP_HOUR_UTC:BACKUP_MINUTE_UTC` + (default 03:10 UTC). The S62 design was opt-in and nobody ever opted in — that + is why the DB went unbacked for weeks. Deploy == installed now. +- **DURABLE:** `BACKUP_DIR=/app/backups` is a Coolify persistent volume, so dumps + survive redeploys. The container-ephemeral risk is closed. +- **VERIFIED END-TO-END (2026-07-20 02:31 UTC):** `POST /api/internal/backup/run` + → exit 0 in 31s → `vyndr-20260720-023136.dump`, **821,549 bytes**, 698 archive + objects, ledger_entries present. `GET /api/internal/backup/verify` read the + rows back OUT of the archive with `pg_restore`: **643 rows in the dump vs 643 + live — exact match**, DB 18 MB. + - The earlier figure of 604 was a moving target, not a discrepancy: the ledger + is append-only and 39 rows were written between the census and the dump + (2 manual regrades + the 01:00 UTC cron; newest lock 01:01:45 UTC). + 604 + 39 = 643 reconciles exactly. +- **Permission bug found and fixed by the real run:** the first attempt failed + with `pg_dump: Permission denied` — the container runs as non-root `vyndr` + (uid 100/gid 101) and the mounted volume was root-owned. The Dockerfile now + creates + chowns `/app/backups`; the verify endpoint reports `uid`/`gid`/ + `backup_dir_writable` so a mounted-but-unwritable volume is loud, not silent. +- **🔶 OFF-BOX IS DEFERRED, NOT DONE.** Storage Box SSH auth is not working yet, + so the rsync is gated behind `BACKUP_OFFBOX=1` (+ `BACKUP_REMOTE` + + `BACKUP_SSH_KEY`) and currently logs "DEFERRED". A failing push is a LOW + priority notice, never a failure — the durable on-box dump succeeded, and + crying wolf trains us to ignore backup alerts. **Remaining risk: a single + Hetzner-side loss takes the dumps with it.** To finish: install the generated + public key on the Storage Box, set `BACKUP_SSH_KEY` in Coolify (port 23 — + 22 is SFTP-only), flip `BACKUP_OFFBOX=1`. +- **Not yet done:** a full restore into a scratch Postgres SERVER. The read-back + proves the archive CONTAINS the data; a server restore would additionally + prove schema/constraint restorability. It needs the dump off the container, + which is blocked on the same Storage Box auth. ## MODEL TRAIN — ARC 2 PREP: DEAD PROBABILITY LAYER REVIVED (2026-07-19, on main, DEPLOYED + fingerprinted) Re-sequenced ahead of the gate work after the audit found the ranking signal did