STATE.md: backup is durable on-box and verified by read-back (643==643)

specs/STATE.md

@@ -1,5 +1,41 @@
 # VYNDR — STATE OF THE WORLD
-### As of `a80868c` (main, DEPLOYED + fingerprinted live), 2026-07-19. This file opens every future session. Update it when a train ships.
+### As of `HEADPLACEHOLDER` (main, DEPLOYED + fingerprinted live), 2026-07-20. This file opens every future session. Update it when a train ships.
+
+## 🟢 BACKUP — DURABLE ON-BOX, VERIFIED BY READ-BACK (2026-07-20, deployed)
+The highest-severity open item is closed **on-box**. Off-box is a tracked follow-up.
+- **Runs as CODE, not a cron somebody must install** (`src/backupScheduler.js`,
+  armed in `server.js`). Arming is opt-OUT: armed whenever `SUPABASE_DB_URL`
+  exists, `BACKUP_CRON=0` kills it, nightly `BACKUP_HOUR_UTC:BACKUP_MINUTE_UTC`
+  (default 03:10 UTC). The S62 design was opt-in and nobody ever opted in — that
+  is why the DB went unbacked for weeks. Deploy == installed now.
+- **DURABLE:** `BACKUP_DIR=/app/backups` is a Coolify persistent volume, so dumps
+  survive redeploys. The container-ephemeral risk is closed.
+- **VERIFIED END-TO-END (2026-07-20 02:31 UTC):** `POST /api/internal/backup/run`
+  → exit 0 in 31s → `vyndr-20260720-023136.dump`, **821,549 bytes**, 698 archive
+  objects, ledger_entries present. `GET /api/internal/backup/verify` read the
+  rows back OUT of the archive with `pg_restore`: **643 rows in the dump vs 643
+  live — exact match**, DB 18 MB.
+  - The earlier figure of 604 was a moving target, not a discrepancy: the ledger
+    is append-only and 39 rows were written between the census and the dump
+    (2 manual regrades + the 01:00 UTC cron; newest lock 01:01:45 UTC).
+    604 + 39 = 643 reconciles exactly.
+- **Permission bug found and fixed by the real run:** the first attempt failed
+  with `pg_dump: Permission denied` — the container runs as non-root `vyndr`
+  (uid 100/gid 101) and the mounted volume was root-owned. The Dockerfile now
+  creates + chowns `/app/backups`; the verify endpoint reports `uid`/`gid`/
+  `backup_dir_writable` so a mounted-but-unwritable volume is loud, not silent.
+- **🔶 OFF-BOX IS DEFERRED, NOT DONE.** Storage Box SSH auth is not working yet,
+  so the rsync is gated behind `BACKUP_OFFBOX=1` (+ `BACKUP_REMOTE` +
+  `BACKUP_SSH_KEY`) and currently logs "DEFERRED". A failing push is a LOW
+  priority notice, never a failure — the durable on-box dump succeeded, and
+  crying wolf trains us to ignore backup alerts. **Remaining risk: a single
+  Hetzner-side loss takes the dumps with it.** To finish: install the generated
+  public key on the Storage Box, set `BACKUP_SSH_KEY` in Coolify (port 23 —
+  22 is SFTP-only), flip `BACKUP_OFFBOX=1`.
+- **Not yet done:** a full restore into a scratch Postgres SERVER. The read-back
+  proves the archive CONTAINS the data; a server restore would additionally
+  prove schema/constraint restorability. It needs the dump off the container,
+  which is blocked on the same Storage Box auth.
 
 ## MODEL TRAIN — ARC 2 PREP: DEAD PROBABILITY LAYER REVIVED (2026-07-19, on main, DEPLOYED + fingerprinted)
 Re-sequenced ahead of the gate work after the audit found the ranking signal did