From 40aba37f83679ce36d7360e22404aeac9db39603 Mon Sep 17 00:00:00 2001 From: Kev Date: Sun, 19 Jul 2026 21:42:27 -0400 Subject: [PATCH] Backup: env-injected SSH key, nightly off-box push, triggerable run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closing the backup for real. Three changes, each fixing something that would have made the Storage Box target fail or silently rot. 1. SSH KEY COMES FROM ENV, not from the container. Generating a keypair inside the API container was the obvious move and it is wrong: the container filesystem is ephemeral, so the key dies on the next redeploy and the off-box push starts failing silently. backup-db.sh now reads BACKUP_SSH_KEY (a Coolify secret), writes it to a 0600 temp file per run, and removes it on exit via trap. 2. PORT 23, verified live. Hetzner Storage Box runs full OpenSSH on 23; port 22 answers with mod_sftp (SFTP only). Banner-checked both against u635423.your-storagebox.de. rsync now uses -e "ssh -p ${BACKUP_SSH_PORT:-23} ... -i "; the old invocation had no -e at all and would have gone to 22. 3. OFF-BOX PUSH IS NIGHTLY, not Sundays-only. A weekly push meant up to six days of dumps existed ONLY inside an ephemeral container, which is the same as not existing. Alert copy updated to say exactly that when the push fails or is skipped. Also adds POST /api/internal/backup/run (internal-key gated) so a real backup can be TRIGGERED and OBSERVED — it returns exit code, duration, output tail, and whether the remote + ssh key are configured. The backup can only run where SUPABASE_DB_URL and the Supabase route live (this container), and there was no way to fire or inspect it without a shell. Connectivity established this session: Storage Box reachable from the dev box on 22/23; Supabase :5432 NOT reachable from WSL2 (so the dump must run in-container, as designed); docker IS available locally, so the restore-verify can run against a scratch Postgres using the real dump. Suite 278/3305 green, build exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA --- scripts/backup-db.sh | 33 ++++++++++++++++++++++++++------- src/routes/internal.js | 31 +++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 7 deletions(-) diff --git a/scripts/backup-db.sh b/scripts/backup-db.sh index 6ea51a6..02e787b 100644 --- a/scripts/backup-db.sh +++ b/scripts/backup-db.sh @@ -63,17 +63,36 @@ echo "backup validated: ${DUMP} (${SIZE} bytes, ${OBJECTS} archive objects, ledg # 3. Rotate: drop local dumps older than KEEP_DAYS. find "${BACKUP_DIR}" -name 'vyndr-*.dump' -type f -mtime "+${KEEP_DAYS}" -delete || true -# 4. Weekly off-box copy (Sundays). A single failure of the off-box push is a -# WARN, not a hard failure — the local dump still succeeded. -if [ "$(date -u +%u)" = "7" ]; then +# 3b. SSH key for the off-box push (Session 64). +# The container filesystem is EPHEMERAL — a keypair generated inside it dies +# on the next redeploy and the off-box push would silently start failing. So +# the PRIVATE key is injected as an env var (Coolify secret) and written to a +# 0600 temp file per run. Hetzner Storage Box speaks full OpenSSH on PORT 23 +# (port 22 is SFTP-only, mod_sftp) — verified live; rsync must target 23. +SSH_KEY_FILE="" +cleanup_key() { [ -n "${SSH_KEY_FILE}" ] && rm -f "${SSH_KEY_FILE}" || true; } +trap cleanup_key EXIT +RSYNC_SSH="ssh -p ${BACKUP_SSH_PORT:-23} -o StrictHostKeyChecking=accept-new -o BatchMode=yes" +if [ -n "${BACKUP_SSH_KEY:-}" ]; then + SSH_KEY_FILE="$(mktemp)" + chmod 600 "${SSH_KEY_FILE}" + # Accept the key with literal \n escapes (how env vars usually carry it). + printf '%b\n' "${BACKUP_SSH_KEY}" | sed -e 's/[[:space:]]*$//' > "${SSH_KEY_FILE}" + RSYNC_SSH="${RSYNC_SSH} -i ${SSH_KEY_FILE}" +fi + +# 4. OFF-BOX COPY — every night, not only Sundays (Session 64). +# A weekly push meant up to 6 days of dumps existed ONLY inside an ephemeral +# container, which is the same as not existing. Off-box is the real backup. +if true; then if [ -n "${BACKUP_REMOTE:-}" ]; then - if rsync -az --timeout=120 "${DUMP}" "${BACKUP_REMOTE}"; then - notify "VYNDR backup OK (+off-box)" "default" "Nightly dump ${STAMP} (${SIZE} bytes) + weekly off-box copy pushed." + if rsync -az --timeout=120 -e "${RSYNC_SSH}" "${DUMP}" "${BACKUP_REMOTE}"; then + notify "VYNDR backup OK (+off-box)" "default" "Nightly dump ${STAMP} (${SIZE} bytes) pushed off-box to ${BACKUP_REMOTE%%:*}." else - notify "VYNDR off-box push FAILED" "high" "Local dump ${STAMP} is fine (${SIZE} bytes) but the weekly off-box rsync failed." + notify "VYNDR off-box push FAILED" "high" "Local dump ${STAMP} is fine (${SIZE} bytes) but the off-box rsync FAILED — the dump exists only in an ephemeral container." fi else - notify "VYNDR off-box push SKIPPED" "high" "Local dump ${STAMP} OK but BACKUP_REMOTE is unset — no off-box copy this week." + notify "VYNDR off-box push SKIPPED" "high" "Local dump ${STAMP} OK but BACKUP_REMOTE is unset — the dump exists only in an ephemeral container." fi else echo "backup ok: ${DUMP} (${SIZE} bytes)" diff --git a/src/routes/internal.js b/src/routes/internal.js index 93db799..91204b7 100644 --- a/src/routes/internal.js +++ b/src/routes/internal.js @@ -240,6 +240,37 @@ router.post('/refresh/all', async (req, res) => { } }); +/** + * POST /api/internal/backup/run (Session 64) — fire ONE real database backup + * now: pg_dump → validate (`pg_restore --list` must contain ledger_entries) → + * off-box rsync to the Storage Box. + * + * Exists because the backup can only run where SUPABASE_DB_URL and the Supabase + * network route live — inside this container — and there was previously no way + * to trigger or observe it without a shell on the box. Returns the script's + * exit code and output tail so a real run can be VERIFIED, not assumed. + */ +router.post('/backup/run', async (req, res) => { + const { runBackup, durabilityWarning } = require('../backupScheduler'); + try { + const started = Date.now(); + const result = await runBackup(); + return res.json({ + ok: result.ok, + exit_code: result.code, + duration_ms: Date.now() - started, + durability_warning: durabilityWarning() || null, + remote_configured: !!process.env.BACKUP_REMOTE, + ssh_key_configured: !!process.env.BACKUP_SSH_KEY, + output_tail: result.tail || result.error || null, + }); + } catch (err) { + const message = err && err.message ? err.message : String(err); + console.error('[internal/backup/run] failed:', message); + return res.status(500).json({ ok: false, error: message }); + } +}); + /** * POST /api/internal/ledger/settle (Session 58, Phase 1) — settle the * persistent ledger (outcome + actual_value + CLV) across every sport.