From 491e636b7f99523735b8b8e9cd155ab22a7e6c13 Mon Sep 17 00:00:00 2001 From: Kev Date: Mon, 20 Jul 2026 01:16:30 -0400 Subject: [PATCH] STATE: off-box backup working + verified on the box MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records off-box as WORKING with the root cause (key was only in Hetzner's project store, never in the box's authorized_keys — the box previously offered an EMPTY auth list) and the proof: offbox_ok:true, and the file independently VERIFIED on the box via rsync --list-only through the pinned host key (vyndr-20260720-051158.dump, 833,917 bytes, 05:12:28 UTC, byte-identical to the local dump). Env truth captured from the run output: key is correctly base64-decoded, destination has no leading-slash bug. Hardening recorded: host key statically pinned (accept-new gone, missing pin refuses the push), remote dir guaranteed, failed required push now pages at urgent with offbox_ok:false while the exit code still tracks on-box durability. Flags the ONE outstanding acceptance item honestly: the round-trip restore is NOT done, because the dev box cannot authenticate to the Storage Box (the authorized key is Kev's, not the in-session keypair) and the container has no Postgres server. Lists both unblocks and the assertion target (>= 645). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA --- specs/STATE.md | 72 ++++++++++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 32 deletions(-) diff --git a/specs/STATE.md b/specs/STATE.md index da69df8..41fc1df 100644 --- a/specs/STATE.md +++ b/specs/STATE.md @@ -31,42 +31,50 @@ was verified live unless explicitly marked UNVERIFIED.* picks the MOST overconfident read.** Do not surface EV as truth until calibration runs on the harness. -## 🔴 OFF-BOX BACKUP — NOT WORKING. Deferred. (real state) +## 🟢 OFF-BOX BACKUP — WORKING + VERIFIED ON THE BOX (2026-07-20 05:12 UTC) -**On-box is durable and proven. Off-box has never succeeded once.** -Every dump currently lives only on the Hetzner persistent volume — **a single -box-side loss takes the backups with it.** +**Root cause (Kev):** the vyndr-backup key was only in Hetzner's PROJECT ssh-key +store, never in the Storage Box's own `.ssh/authorized_keys`. Installed there → +the box now offers `publickey,password` (it previously offered an EMPTY auth +list, which is what "SSH not enabled on this box" looks like). -Fixed tonight (both real blockers, both now gone): -1. `BACKUP_SSH_KEY` was not base64-decoded → now auto-detects base64 vs raw PEM. -2. Container had `rsync` but **no `ssh` binary** → `openssh-client` added to the - Dockerfile. (`Failed to exec ssh` is gone.) +**Proven end-to-end:** +- `POST /api/internal/backup/run` → `ok:true`, **`offbox_ok:true`**, exit 0, 31s. +- Output confirms both env questions: **`ssh key: base64-decoded`** (the key is + correct base64, not a mangled raw paste) and destination + **`u635423@...:vyndr-backups/`** — no leading-slash bug. +- **File VERIFIED on the box, not inferred from exit 0:** + `GET /api/internal/backup/offbox` (rsync `--list-only` through the pinned host + key) returned `vyndr-20260720-051158.dump`, **833,917 bytes, 05:12:28 UTC** — + byte-identical size to the local dump, timestamp from that run. -**Remaining blocker — SSH auth is rejected at the Storage Box.** rsync exits 255. -Verified from the dev box with the private key directly: -``` -debug1: Offering public key: ... SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo -debug1: Authentications that can continue: <-- EMPTY -Permission denied (). -``` -**The server offers NO auth methods at all.** That is not a wrong-key signature — -it is the account refusing all authentication. Port 23 answers with an OpenSSH -banner regardless, which is why SSH can look enabled when it isn't. +**Hardening shipped with it:** +- **Host key STATICALLY PINNED.** `ssh-keyscan` matched the out-of-band + fingerprint `SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM`; + `scripts/storagebox_known_hosts` ships to the container and the script uses + `StrictHostKeyChecking=yes` + `UserKnownHostsFile=`. `accept-new` + (trust-on-first-use) is GONE; a missing pin file REFUSES the push. Tests + assert the weakenings never return. +- **Remote dir guaranteed** — `--mkpath`, else remote `mkdir -p`. Without it a + push into a missing parent can write the dump AS the directory name: one file, + overwritten nightly, reading as "backups exist" while retaining exactly one. +- **A failed required push now PAGES** at urgent (was "low/deferred") and the run + reports `offbox_ok:false`. Exit code still tracks ON-BOX durability so a good + on-box dump never raises a false total-failure alarm. -**UNVERIFIED — I did NOT run a Chrome/UI diagnostic and have no data on:** the -Hetzner "SSH support" toggle state, an "External Reachability" setting, whether -the key was added at project level vs the box itself, or any Hetzner incident. -Those are the untested hypotheses, listed in likelihood order: -1. SSH support disabled on the Storage Box (separate checkbox from adding a key). -2. Key added to the wrong scope (project/sub-account vs this box). -3. External reachability disabled for the box. -4. Hetzner-side outage — no evidence either way; not checked. - -**To close:** confirm the panel shows fingerprint -`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`, enable SSH, then -`POST /api/internal/backup/run` and look for `off-box push OK`. **The full -scratch-Postgres restore proof is still OWED and is blocked on this** (needs the -dump off the container; docker is available locally to do it immediately after). +### 🔶 ONE ACCEPTANCE ITEM OUTSTANDING — the round-trip restore +**Not done: pull the dump back FROM the box and restore it into scratch Postgres.** +The dev box **cannot authenticate to the Storage Box** — the authorized key is +Kev's `~/vyndr-backup-key`, not the in-session keypair +(`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`), and the container has no +Postgres server to restore into. +**Unblock (either):** add that in-session public key to the box's +`authorized_keys`, **or** supply the Storage Box password (password auth is now +offered). Then: pull → restore to scratch → assert `ledger_entries` **>= 645** +(live count at 2026-07-20 05:00 UTC; higher is a PASS, append-only). +*Evidence short of a full restore:* remote size is byte-identical to the +validated local dump (833,917), and that local dump passed +`pg_restore --list` with `ledger_entries` present. ## Open items — with status