From c2f604140613d8744708778b088489a5e8a418cc Mon Sep 17 00:00:00 2001 From: Kev Date: Mon, 20 Jul 2026 01:28:29 -0400 Subject: [PATCH] =?UTF-8?q?STATE:=20off-box=20round=20trip=20CLOSED=20?= =?UTF-8?q?=E2=80=94=20645=20restored=20from=20the=20box=20copy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 complete. The insurance chain is proven end to end rather than assumed: dump -> validated -> pushed off-box -> verified on the box -> pulled back down -> rebuilt into a live database. Pulled vyndr-20260720-051158.dump FROM the Storage Box (not the local copy) with the in-session key through the pinned host key, never bypassing StrictHostKeyChecking. Restored into scratch Postgres 17: 715 archive objects, 42 public tables, ledger_entries with all 27 columns and real spot-checked rows. ASSERTION PASSED: ledger_entries restored 645 == live 645 (target >= 645). model_snapshots restored 100/100, so the retention store shipped yesterday is covered by backups from day one. Records the operational gotcha the restore surfaced: the dump is written by pg_dump 17 (Supabase 17.6) and pg_restore 16 CANNOT read it — 'unsupported version (1.16) in file header'. The first attempt failed on exactly this. Any DR runbook must use PG17+ tooling. Restoring into vanilla Postgres also logs 12 ignored errors (Supabase roles/extensions absent locally) which are harmless. Scratch DB torn down, pulled copy deleted, both dumps still on the box, nightly cron untouched. No private key material echoed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA --- specs/STATE.md | 38 ++++++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/specs/STATE.md b/specs/STATE.md index 41fc1df..19945d2 100644 --- a/specs/STATE.md +++ b/specs/STATE.md @@ -31,7 +31,7 @@ was verified live unless explicitly marked UNVERIFIED.* picks the MOST overconfident read.** Do not surface EV as truth until calibration runs on the harness. -## 🟢 OFF-BOX BACKUP — WORKING + VERIFIED ON THE BOX (2026-07-20 05:12 UTC) +## 🟢 OFF-BOX BACKUP — CLOSED. FULL ROUND TRIP PROVEN (2026-07-20 05:25 UTC) **Root cause (Kev):** the vyndr-backup key was only in Hetzner's PROJECT ssh-key store, never in the Storage Box's own `.ssh/authorized_keys`. Installed there → @@ -62,19 +62,29 @@ list, which is what "SSH not enabled on this box" looks like). reports `offbox_ok:false`. Exit code still tracks ON-BOX durability so a good on-box dump never raises a false total-failure alarm. -### 🔶 ONE ACCEPTANCE ITEM OUTSTANDING — the round-trip restore -**Not done: pull the dump back FROM the box and restore it into scratch Postgres.** -The dev box **cannot authenticate to the Storage Box** — the authorized key is -Kev's `~/vyndr-backup-key`, not the in-session keypair -(`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`), and the container has no -Postgres server to restore into. -**Unblock (either):** add that in-session public key to the box's -`authorized_keys`, **or** supply the Storage Box password (password auth is now -offered). Then: pull → restore to scratch → assert `ledger_entries` **>= 645** -(live count at 2026-07-20 05:00 UTC; higher is a PASS, append-only). -*Evidence short of a full restore:* remote size is byte-identical to the -validated local dump (833,917), and that local dump passed -`pg_restore --list` with `ledger_entries` present. +### ✅ ROUND TRIP CLOSED — pulled back off-box and rebuilt into a live DB (05:25 UTC) +The full insurance chain is now PROVEN, not assumed: dump → validated → pushed +off-box → verified on the box → **pulled back down from the box → restored into a +live Postgres**. +- **Pulled FROM the Storage Box** (not the local copy) with the in-session key + through the pinned host key (`StrictHostKeyChecking=yes`, never bypassed): + `vyndr-20260720-051158.dump`, 833,917 bytes — byte-identical to what was pushed. +- **Restored into scratch Postgres 17** (docker): 715 archive objects, + **42 public tables**, `ledger_entries` with all **27 columns** and real rows + (spot-checked: Kelsey Mitchell points o22.5 B −105 hit). +- **ASSERTION PASSED — `ledger_entries` restored = 645, live = 645** (target was + ≥ 645). `model_snapshots` also restored 100/100, so the new retention store is + covered by backups from day one. +- Scratch DB torn down; the pulled copy deleted; **both dumps still on the box** + (nothing rotated out, cron untouched). + +**🔴 OPERATIONAL GOTCHA FOUND DURING THE RESTORE — record this in any DR runbook:** +the dump is written by **pg_dump 17** (Supabase is PG 17.6), so **pg_restore 16 +CANNOT read it** — it fails with `unsupported version (1.16) in file header`. +The first restore attempt failed for exactly this reason. **Restore with PG 17+ +tooling** (`postgres:17-alpine` works). Restoring into vanilla Postgres also logs +`errors ignored on restore: 12` — Supabase-specific roles/extensions that do not +exist locally; harmless, the data restores completely. ## Open items — with status