STATE: off-box round trip CLOSED — 645 restored from the box copy

Phase 3 complete. The insurance chain is proven end to end rather than
assumed: dump -> validated -> pushed off-box -> verified on the box ->
pulled back down -> rebuilt into a live database.

Pulled vyndr-20260720-051158.dump FROM the Storage Box (not the local
copy) with the in-session key through the pinned host key, never
bypassing StrictHostKeyChecking. Restored into scratch Postgres 17: 715
archive objects, 42 public tables, ledger_entries with all 27 columns and
real spot-checked rows.

ASSERTION PASSED: ledger_entries restored 645 == live 645 (target >= 645).
model_snapshots restored 100/100, so the retention store shipped yesterday
is covered by backups from day one.

Records the operational gotcha the restore surfaced: the dump is written
by pg_dump 17 (Supabase 17.6) and pg_restore 16 CANNOT read it —
'unsupported version (1.16) in file header'. The first attempt failed on
exactly this. Any DR runbook must use PG17+ tooling. Restoring into
vanilla Postgres also logs 12 ignored errors (Supabase roles/extensions
absent locally) which are harmless.

Scratch DB torn down, pulled copy deleted, both dumps still on the box,
nightly cron untouched. No private key material echoed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
This commit is contained in:
Kev
2026-07-20 01:28:29 -04:00
parent 491e636b7f
commit c2f6041406
+24 -14
View File
@@ -31,7 +31,7 @@ was verified live unless explicitly marked UNVERIFIED.*
picks the MOST overconfident read.** Do not surface EV as truth until
calibration runs on the harness.
## 🟢 OFF-BOX BACKUP — WORKING + VERIFIED ON THE BOX (2026-07-20 05:12 UTC)
## 🟢 OFF-BOX BACKUP — CLOSED. FULL ROUND TRIP PROVEN (2026-07-20 05:25 UTC)
**Root cause (Kev):** the vyndr-backup key was only in Hetzner's PROJECT ssh-key
store, never in the Storage Box's own `.ssh/authorized_keys`. Installed there →
@@ -62,19 +62,29 @@ list, which is what "SSH not enabled on this box" looks like).
reports `offbox_ok:false`. Exit code still tracks ON-BOX durability so a good
on-box dump never raises a false total-failure alarm.
### 🔶 ONE ACCEPTANCE ITEM OUTSTANDING — the round-trip restore
**Not done: pull the dump back FROM the box and restore it into scratch Postgres.**
The dev box **cannot authenticate to the Storage Box** — the authorized key is
Kev's `~/vyndr-backup-key`, not the in-session keypair
(`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`), and the container has no
Postgres server to restore into.
**Unblock (either):** add that in-session public key to the box's
`authorized_keys`, **or** supply the Storage Box password (password auth is now
offered). Then: pull → restore to scratch → assert `ledger_entries` **>= 645**
(live count at 2026-07-20 05:00 UTC; higher is a PASS, append-only).
*Evidence short of a full restore:* remote size is byte-identical to the
validated local dump (833,917), and that local dump passed
`pg_restore --list` with `ledger_entries` present.
### ✅ ROUND TRIP CLOSED — pulled back off-box and rebuilt into a live DB (05:25 UTC)
The full insurance chain is now PROVEN, not assumed: dump → validated → pushed
off-box → verified on the box → **pulled back down from the box → restored into a
live Postgres**.
- **Pulled FROM the Storage Box** (not the local copy) with the in-session key
through the pinned host key (`StrictHostKeyChecking=yes`, never bypassed):
`vyndr-20260720-051158.dump`, 833,917 bytes — byte-identical to what was pushed.
- **Restored into scratch Postgres 17** (docker): 715 archive objects,
**42 public tables**, `ledger_entries` with all **27 columns** and real rows
(spot-checked: Kelsey Mitchell points o22.5 B 105 hit).
- **ASSERTION PASSED — `ledger_entries` restored = 645, live = 645** (target was
≥ 645). `model_snapshots` also restored 100/100, so the new retention store is
covered by backups from day one.
- Scratch DB torn down; the pulled copy deleted; **both dumps still on the box**
(nothing rotated out, cron untouched).
**🔴 OPERATIONAL GOTCHA FOUND DURING THE RESTORE — record this in any DR runbook:**
the dump is written by **pg_dump 17** (Supabase is PG 17.6), so **pg_restore 16
CANNOT read it** — it fails with `unsupported version (1.16) in file header`.
The first restore attempt failed for exactly this reason. **Restore with PG 17+
tooling** (`postgres:17-alpine` works). Restoring into vanilla Postgres also logs
`errors ignored on restore: 12` — Supabase-specific roles/extensions that do not
exist locally; harmless, the data restores completely.
## Open items — with status