STATE: off-box round trip CLOSED — 645 restored from the box copy
Phase 3 complete. The insurance chain is proven end to end rather than assumed: dump -> validated -> pushed off-box -> verified on the box -> pulled back down -> rebuilt into a live database. Pulled vyndr-20260720-051158.dump FROM the Storage Box (not the local copy) with the in-session key through the pinned host key, never bypassing StrictHostKeyChecking. Restored into scratch Postgres 17: 715 archive objects, 42 public tables, ledger_entries with all 27 columns and real spot-checked rows. ASSERTION PASSED: ledger_entries restored 645 == live 645 (target >= 645). model_snapshots restored 100/100, so the retention store shipped yesterday is covered by backups from day one. Records the operational gotcha the restore surfaced: the dump is written by pg_dump 17 (Supabase 17.6) and pg_restore 16 CANNOT read it — 'unsupported version (1.16) in file header'. The first attempt failed on exactly this. Any DR runbook must use PG17+ tooling. Restoring into vanilla Postgres also logs 12 ignored errors (Supabase roles/extensions absent locally) which are harmless. Scratch DB torn down, pulled copy deleted, both dumps still on the box, nightly cron untouched. No private key material echoed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
This commit is contained in:
+24
-14
@@ -31,7 +31,7 @@ was verified live unless explicitly marked UNVERIFIED.*
|
||||
picks the MOST overconfident read.** Do not surface EV as truth until
|
||||
calibration runs on the harness.
|
||||
|
||||
## 🟢 OFF-BOX BACKUP — WORKING + VERIFIED ON THE BOX (2026-07-20 05:12 UTC)
|
||||
## 🟢 OFF-BOX BACKUP — CLOSED. FULL ROUND TRIP PROVEN (2026-07-20 05:25 UTC)
|
||||
|
||||
**Root cause (Kev):** the vyndr-backup key was only in Hetzner's PROJECT ssh-key
|
||||
store, never in the Storage Box's own `.ssh/authorized_keys`. Installed there →
|
||||
@@ -62,19 +62,29 @@ list, which is what "SSH not enabled on this box" looks like).
|
||||
reports `offbox_ok:false`. Exit code still tracks ON-BOX durability so a good
|
||||
on-box dump never raises a false total-failure alarm.
|
||||
|
||||
### 🔶 ONE ACCEPTANCE ITEM OUTSTANDING — the round-trip restore
|
||||
**Not done: pull the dump back FROM the box and restore it into scratch Postgres.**
|
||||
The dev box **cannot authenticate to the Storage Box** — the authorized key is
|
||||
Kev's `~/vyndr-backup-key`, not the in-session keypair
|
||||
(`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`), and the container has no
|
||||
Postgres server to restore into.
|
||||
**Unblock (either):** add that in-session public key to the box's
|
||||
`authorized_keys`, **or** supply the Storage Box password (password auth is now
|
||||
offered). Then: pull → restore to scratch → assert `ledger_entries` **>= 645**
|
||||
(live count at 2026-07-20 05:00 UTC; higher is a PASS, append-only).
|
||||
*Evidence short of a full restore:* remote size is byte-identical to the
|
||||
validated local dump (833,917), and that local dump passed
|
||||
`pg_restore --list` with `ledger_entries` present.
|
||||
### ✅ ROUND TRIP CLOSED — pulled back off-box and rebuilt into a live DB (05:25 UTC)
|
||||
The full insurance chain is now PROVEN, not assumed: dump → validated → pushed
|
||||
off-box → verified on the box → **pulled back down from the box → restored into a
|
||||
live Postgres**.
|
||||
- **Pulled FROM the Storage Box** (not the local copy) with the in-session key
|
||||
through the pinned host key (`StrictHostKeyChecking=yes`, never bypassed):
|
||||
`vyndr-20260720-051158.dump`, 833,917 bytes — byte-identical to what was pushed.
|
||||
- **Restored into scratch Postgres 17** (docker): 715 archive objects,
|
||||
**42 public tables**, `ledger_entries` with all **27 columns** and real rows
|
||||
(spot-checked: Kelsey Mitchell points o22.5 B −105 hit).
|
||||
- **ASSERTION PASSED — `ledger_entries` restored = 645, live = 645** (target was
|
||||
≥ 645). `model_snapshots` also restored 100/100, so the new retention store is
|
||||
covered by backups from day one.
|
||||
- Scratch DB torn down; the pulled copy deleted; **both dumps still on the box**
|
||||
(nothing rotated out, cron untouched).
|
||||
|
||||
**🔴 OPERATIONAL GOTCHA FOUND DURING THE RESTORE — record this in any DR runbook:**
|
||||
the dump is written by **pg_dump 17** (Supabase is PG 17.6), so **pg_restore 16
|
||||
CANNOT read it** — it fails with `unsupported version (1.16) in file header`.
|
||||
The first restore attempt failed for exactly this reason. **Restore with PG 17+
|
||||
tooling** (`postgres:17-alpine` works). Restoring into vanilla Postgres also logs
|
||||
`errors ignored on restore: 12` — Supabase-specific roles/extensions that do not
|
||||
exist locally; harmless, the data restores completely.
|
||||
|
||||
## Open items — with status
|
||||
|
||||
|
||||
Reference in New Issue
Block a user