STATE: off-box backup working + verified on the box
Records off-box as WORKING with the root cause (key was only in Hetzner's project store, never in the box's authorized_keys — the box previously offered an EMPTY auth list) and the proof: offbox_ok:true, and the file independently VERIFIED on the box via rsync --list-only through the pinned host key (vyndr-20260720-051158.dump, 833,917 bytes, 05:12:28 UTC, byte-identical to the local dump). Env truth captured from the run output: key is correctly base64-decoded, destination has no leading-slash bug. Hardening recorded: host key statically pinned (accept-new gone, missing pin refuses the push), remote dir guaranteed, failed required push now pages at urgent with offbox_ok:false while the exit code still tracks on-box durability. Flags the ONE outstanding acceptance item honestly: the round-trip restore is NOT done, because the dev box cannot authenticate to the Storage Box (the authorized key is Kev's, not the in-session keypair) and the container has no Postgres server. Lists both unblocks and the assertion target (>= 645). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
This commit is contained in:
+40
-32
@@ -31,42 +31,50 @@ was verified live unless explicitly marked UNVERIFIED.*
|
||||
picks the MOST overconfident read.** Do not surface EV as truth until
|
||||
calibration runs on the harness.
|
||||
|
||||
## 🔴 OFF-BOX BACKUP — NOT WORKING. Deferred. (real state)
|
||||
## 🟢 OFF-BOX BACKUP — WORKING + VERIFIED ON THE BOX (2026-07-20 05:12 UTC)
|
||||
|
||||
**On-box is durable and proven. Off-box has never succeeded once.**
|
||||
Every dump currently lives only on the Hetzner persistent volume — **a single
|
||||
box-side loss takes the backups with it.**
|
||||
**Root cause (Kev):** the vyndr-backup key was only in Hetzner's PROJECT ssh-key
|
||||
store, never in the Storage Box's own `.ssh/authorized_keys`. Installed there →
|
||||
the box now offers `publickey,password` (it previously offered an EMPTY auth
|
||||
list, which is what "SSH not enabled on this box" looks like).
|
||||
|
||||
Fixed tonight (both real blockers, both now gone):
|
||||
1. `BACKUP_SSH_KEY` was not base64-decoded → now auto-detects base64 vs raw PEM.
|
||||
2. Container had `rsync` but **no `ssh` binary** → `openssh-client` added to the
|
||||
Dockerfile. (`Failed to exec ssh` is gone.)
|
||||
**Proven end-to-end:**
|
||||
- `POST /api/internal/backup/run` → `ok:true`, **`offbox_ok:true`**, exit 0, 31s.
|
||||
- Output confirms both env questions: **`ssh key: base64-decoded`** (the key is
|
||||
correct base64, not a mangled raw paste) and destination
|
||||
**`u635423@...:vyndr-backups/`** — no leading-slash bug.
|
||||
- **File VERIFIED on the box, not inferred from exit 0:**
|
||||
`GET /api/internal/backup/offbox` (rsync `--list-only` through the pinned host
|
||||
key) returned `vyndr-20260720-051158.dump`, **833,917 bytes, 05:12:28 UTC** —
|
||||
byte-identical size to the local dump, timestamp from that run.
|
||||
|
||||
**Remaining blocker — SSH auth is rejected at the Storage Box.** rsync exits 255.
|
||||
Verified from the dev box with the private key directly:
|
||||
```
|
||||
debug1: Offering public key: ... SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo
|
||||
debug1: Authentications that can continue: <-- EMPTY
|
||||
Permission denied ().
|
||||
```
|
||||
**The server offers NO auth methods at all.** That is not a wrong-key signature —
|
||||
it is the account refusing all authentication. Port 23 answers with an OpenSSH
|
||||
banner regardless, which is why SSH can look enabled when it isn't.
|
||||
**Hardening shipped with it:**
|
||||
- **Host key STATICALLY PINNED.** `ssh-keyscan` matched the out-of-band
|
||||
fingerprint `SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM`;
|
||||
`scripts/storagebox_known_hosts` ships to the container and the script uses
|
||||
`StrictHostKeyChecking=yes` + `UserKnownHostsFile=<pin>`. `accept-new`
|
||||
(trust-on-first-use) is GONE; a missing pin file REFUSES the push. Tests
|
||||
assert the weakenings never return.
|
||||
- **Remote dir guaranteed** — `--mkpath`, else remote `mkdir -p`. Without it a
|
||||
push into a missing parent can write the dump AS the directory name: one file,
|
||||
overwritten nightly, reading as "backups exist" while retaining exactly one.
|
||||
- **A failed required push now PAGES** at urgent (was "low/deferred") and the run
|
||||
reports `offbox_ok:false`. Exit code still tracks ON-BOX durability so a good
|
||||
on-box dump never raises a false total-failure alarm.
|
||||
|
||||
**UNVERIFIED — I did NOT run a Chrome/UI diagnostic and have no data on:** the
|
||||
Hetzner "SSH support" toggle state, an "External Reachability" setting, whether
|
||||
the key was added at project level vs the box itself, or any Hetzner incident.
|
||||
Those are the untested hypotheses, listed in likelihood order:
|
||||
1. SSH support disabled on the Storage Box (separate checkbox from adding a key).
|
||||
2. Key added to the wrong scope (project/sub-account vs this box).
|
||||
3. External reachability disabled for the box.
|
||||
4. Hetzner-side outage — no evidence either way; not checked.
|
||||
|
||||
**To close:** confirm the panel shows fingerprint
|
||||
`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`, enable SSH, then
|
||||
`POST /api/internal/backup/run` and look for `off-box push OK`. **The full
|
||||
scratch-Postgres restore proof is still OWED and is blocked on this** (needs the
|
||||
dump off the container; docker is available locally to do it immediately after).
|
||||
### 🔶 ONE ACCEPTANCE ITEM OUTSTANDING — the round-trip restore
|
||||
**Not done: pull the dump back FROM the box and restore it into scratch Postgres.**
|
||||
The dev box **cannot authenticate to the Storage Box** — the authorized key is
|
||||
Kev's `~/vyndr-backup-key`, not the in-session keypair
|
||||
(`SHA256:ZO3j7en9J7wN7EKcbYHfpyNcxTPw266hJxECUWXoMTo`), and the container has no
|
||||
Postgres server to restore into.
|
||||
**Unblock (either):** add that in-session public key to the box's
|
||||
`authorized_keys`, **or** supply the Storage Box password (password auth is now
|
||||
offered). Then: pull → restore to scratch → assert `ledger_entries` **>= 645**
|
||||
(live count at 2026-07-20 05:00 UTC; higher is a PASS, append-only).
|
||||
*Evidence short of a full restore:* remote size is byte-identical to the
|
||||
validated local dump (833,917), and that local dump passed
|
||||
`pg_restore --list` with `ledger_entries` present.
|
||||
|
||||
## Open items — with status
|
||||
|
||||
|
||||
Reference in New Issue
Block a user