Off-box backup: pin the host key, guarantee the remote dir, page on failure

PHASE 1 — HOST KEY STATICALLY PINNED. ssh-keyscan -p 23 returned an
ED25519 key whose fingerprint EQUALS the out-of-band value
SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM, so it is safe to pin.
scripts/storagebox_known_hosts now carries that verified line and ships to
the container (Dockerfile already COPYs scripts/). backup-db.sh uses
StrictHostKeyChecking=yes + UserKnownHostsFile=<pin> instead of
accept-new, which was trust-on-first-use and would have accepted an
impostor on the very first run. A missing pin file REFUSES the push rather
than silently falling back. Never weakened to accept-new/=no//dev/null —
a test asserts that on executable lines.

PHASE 1b — REMOTE DIR GUARANTEED. The box has only .ssh/, and rsyncing a
file into a missing parent either fails or silently writes the dump AS the
directory name — one file, overwritten nightly, reading as "backups exist"
while retaining exactly one. Uses rsync --mkpath when available, else an
explicit remote mkdir -p ahead of the push.

PHASE 2b — FAILED OFF-BOX PUSH IS NOW LOUD. Off-box is required, so the
failed-push path pages at "urgent" (was "low"/deferred) and the script
emits a machine-readable OFFBOX_OK=1/0/deferred that
POST /api/internal/backup/run surfaces as a distinct offbox_ok field.
Exit code deliberately still reflects ON-BOX durability — a good on-box
dump must not raise a false total-failure alarm. Surfacing the truth, not
manufacturing a failure.

No key material is echoed anywhere; only the PUBLIC host key is committed.

Suite 280/3338 green, build exit 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA

scripts/backup-db.sh

@@ -72,7 +72,15 @@ find "${BACKUP_DIR}" -name 'vyndr-*.dump' -type f -mtime "+${KEEP_DAYS}" -delete
 SSH_KEY_FILE=""
 cleanup_key() { [ -n "${SSH_KEY_FILE}" ] && rm -f "${SSH_KEY_FILE}" || true; }
 trap cleanup_key EXIT
-RSYNC_SSH="ssh -p ${BACKUP_SSH_PORT:-23} -o StrictHostKeyChecking=accept-new -o BatchMode=yes"
+# HOST KEY IS STATICALLY PINNED (Session 64). This used to be
+# StrictHostKeyChecking=accept-new — trust-on-first-use, which accepts whatever
+# host key it meets first and would happily trust an impostor on the first run.
+# scripts/storagebox_known_hosts carries the ED25519 line verified out-of-band
+# (SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM). A mismatch is now a HARD
+# FAIL — which is the point. Never weaken this to accept-new/=no//dev/null.
+KNOWN_HOSTS="${BACKUP_KNOWN_HOSTS:-$(dirname "$0")/storagebox_known_hosts}"
+[ -f "${KNOWN_HOSTS}" ] || fail "pinned known_hosts missing at ${KNOWN_HOSTS} — refusing to push without host-key verification"
+RSYNC_SSH="ssh -p ${BACKUP_SSH_PORT:-23} -o StrictHostKeyChecking=yes -o UserKnownHostsFile=${KNOWN_HOSTS} -o BatchMode=yes"
 if [ -n "${BACKUP_SSH_KEY:-}" ]; then
   SSH_KEY_FILE="$(mktemp)"
   chmod 600 "${SSH_KEY_FILE}"
@@ -106,16 +114,38 @@ fi
 #    Set BACKUP_OFFBOX=1 (with BACKUP_SSH_KEY) to re-enable. Until then we log
 #    and page at LOW priority, and we never call a deferred push a failure.
 if [ "${BACKUP_OFFBOX:-0}" = "1" ] && [ -n "${BACKUP_REMOTE:-}" ] && [ -n "${BACKUP_SSH_KEY:-}" ]; then
-  if rsync -az --timeout=120 -e "${RSYNC_SSH}" "${DUMP}" "${BACKUP_REMOTE}"; then
-    echo "off-box push OK -> ${BACKUP_REMOTE%%:*}"
+  # Phase 1b — GUARANTEE THE REMOTE DIRECTORY EXISTS. The box starts with only
+  # .ssh/, and rsync of a file into a missing parent either fails or silently
+  # writes the dump AS the directory name (one file, overwritten nightly, which
+  # would read as "backups exist" while retaining exactly one). Prefer rsync's
+  # own --mkpath; fall back to an explicit ssh mkdir -p for older rsync.
+  REMOTE_HOST="${BACKUP_REMOTE%%:*}"
+  REMOTE_PATH="${BACKUP_REMOTE#*:}"
+  MKPATH_FLAG=""
+  if rsync --help 2>&1 | grep -q -- '--mkpath'; then
+    MKPATH_FLAG="--mkpath"
+  else
+    ${RSYNC_SSH} "${REMOTE_HOST}" "mkdir -p '${REMOTE_PATH}'" \
+      || echo "warn: remote mkdir -p failed; relying on an existing directory"
+  fi
+
+  # OFF-BOX IS REQUIRED NOW (Session 64, Phase 2b). A failed push must never
+  # again read as success: it PAGES, and the run reports offbox_ok:false.
+  # Exit code deliberately still reflects ON-BOX durability — a good on-box dump
+  # must not raise a false total-failure alarm. Surface the truth; don't
+  # manufacture a failure.
+  if rsync -az --timeout=120 ${MKPATH_FLAG} -e "${RSYNC_SSH}" "${DUMP}" "${BACKUP_REMOTE}"; then
+    echo "off-box push OK -> ${BACKUP_REMOTE}"
+    echo "OFFBOX_OK=1"
     notify "VYNDR backup OK (+off-box)" "default" "Nightly dump ${STAMP} (${SIZE} bytes) pushed off-box."
   else
-    # NOT a failure: the durable on-box dump succeeded.
-    echo "off-box push FAILED (deferred; on-box dump is durable)"
-    notify "VYNDR off-box push deferred" "low" "Dump ${STAMP} (${SIZE} bytes) is durable on the persistent volume; the off-box rsync failed and is deferred."
+    echo "off-box push FAILED (on-box dump is durable, but OFF-BOX IS REQUIRED)"
+    echo "OFFBOX_OK=0"
+    notify "VYNDR OFF-BOX PUSH FAILED" "urgent" "Dump ${STAMP} (${SIZE} bytes) is on the persistent volume but did NOT reach the Storage Box. The database has no off-box copy tonight."
   fi
 else
   echo "off-box push DEFERRED (BACKUP_OFFBOX!=1 or remote/key unset) — on-box dump is durable at ${DUMP}"
+  echo "OFFBOX_OK=deferred"
 fi
 
 echo "backup ok: ${DUMP} (${SIZE} bytes)"