diff --git a/specs/STATE.md b/specs/STATE.md index 41fc1df..19945d2 100644 --- a/specs/STATE.md +++ b/specs/STATE.md @@ -31,7 +31,7 @@ was verified live unless explicitly marked UNVERIFIED.* picks the MOST overconfident read.** Do not surface EV as truth until calibration runs on the harness. -## 🟢 OFF-BOX BACKUP — WORKING + VERIFIED ON THE BOX (2026-07-20 05:12 UTC) +## 🟢 OFF-BOX BACKUP — CLOSED. FULL ROUND TRIP PROVEN (2026-07-20 05:25 UTC) **Root cause (Kev):** the vyndr-backup key was only in Hetzner's PROJECT ssh-key store, never in the Storage Box's own `.ssh/authorized_keys`. Installed there → @@ -62,19 +62,29 @@ list, which is what "SSH not enabled on this box" looks like). reports `offbox_ok:false`. Exit code still tracks ON-BOX durability so a good on-box dump never raises a false total-failure alarm. -### 🔶 ONE ACCEPTANCE ITEM OUTSTANDING — the round-trip restore -**Not done: pull the dump back FROM the box and restore it into scratch Postgres.** -The dev box **cannot authenticate to the Storage Box** — the authorized key is -Kev's `~/vyndr-backup-key`, not the in-session keypair -(`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`), and the container has no -Postgres server to restore into. -**Unblock (either):** add that in-session public key to the box's -`authorized_keys`, **or** supply the Storage Box password (password auth is now -offered). Then: pull → restore to scratch → assert `ledger_entries` **>= 645** -(live count at 2026-07-20 05:00 UTC; higher is a PASS, append-only). -*Evidence short of a full restore:* remote size is byte-identical to the -validated local dump (833,917), and that local dump passed -`pg_restore --list` with `ledger_entries` present. +### ✅ ROUND TRIP CLOSED — pulled back off-box and rebuilt into a live DB (05:25 UTC) +The full insurance chain is now PROVEN, not assumed: dump → validated → pushed +off-box → verified on the box → **pulled back down from the box → restored into a +live Postgres**. +- **Pulled FROM the Storage Box** (not the local copy) with the in-session key + through the pinned host key (`StrictHostKeyChecking=yes`, never bypassed): + `vyndr-20260720-051158.dump`, 833,917 bytes — byte-identical to what was pushed. +- **Restored into scratch Postgres 17** (docker): 715 archive objects, + **42 public tables**, `ledger_entries` with all **27 columns** and real rows + (spot-checked: Kelsey Mitchell points o22.5 B −105 hit). +- **ASSERTION PASSED — `ledger_entries` restored = 645, live = 645** (target was + ≥ 645). `model_snapshots` also restored 100/100, so the new retention store is + covered by backups from day one. +- Scratch DB torn down; the pulled copy deleted; **both dumps still on the box** + (nothing rotated out, cron untouched). + +**🔴 OPERATIONAL GOTCHA FOUND DURING THE RESTORE — record this in any DR runbook:** +the dump is written by **pg_dump 17** (Supabase is PG 17.6), so **pg_restore 16 +CANNOT read it** — it fails with `unsupported version (1.16) in file header`. +The first restore attempt failed for exactly this reason. **Restore with PG 17+ +tooling** (`postgres:17-alpine` works). Restoring into vanilla Postgres also logs +`errors ignored on restore: 12` — Supabase-specific roles/extensions that do not +exist locally; harmless, the data restores completely. ## Open items — with status