Backup: env-injected SSH key, nightly off-box push, triggerable run

Closing the backup for real. Three changes, each fixing something that
would have made the Storage Box target fail or silently rot.

1. SSH KEY COMES FROM ENV, not from the container. Generating a keypair
   inside the API container was the obvious move and it is wrong: the
   container filesystem is ephemeral, so the key dies on the next
   redeploy and the off-box push starts failing silently. backup-db.sh
   now reads BACKUP_SSH_KEY (a Coolify secret), writes it to a 0600 temp
   file per run, and removes it on exit via trap.

2. PORT 23, verified live. Hetzner Storage Box runs full OpenSSH on 23;
   port 22 answers with mod_sftp (SFTP only). Banner-checked both against
   u635423.your-storagebox.de. rsync now uses
   -e "ssh -p ${BACKUP_SSH_PORT:-23} ... -i <key>"; the old invocation had
   no -e at all and would have gone to 22.

3. OFF-BOX PUSH IS NIGHTLY, not Sundays-only. A weekly push meant up to
   six days of dumps existed ONLY inside an ephemeral container, which is
   the same as not existing. Alert copy updated to say exactly that when
   the push fails or is skipped.

Also adds POST /api/internal/backup/run (internal-key gated) so a real
backup can be TRIGGERED and OBSERVED — it returns exit code, duration,
output tail, and whether the remote + ssh key are configured. The backup
can only run where SUPABASE_DB_URL and the Supabase route live (this
container), and there was no way to fire or inspect it without a shell.

Connectivity established this session: Storage Box reachable from the dev
box on 22/23; Supabase :5432 NOT reachable from WSL2 (so the dump must
run in-container, as designed); docker IS available locally, so the
restore-verify can run against a scratch Postgres using the real dump.

Suite 278/3305 green, build exit 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA

scripts/backup-db.sh

@@ -63,17 +63,36 @@ echo "backup validated: ${DUMP} (${SIZE} bytes, ${OBJECTS} archive objects, ledg
 # 3. Rotate: drop local dumps older than KEEP_DAYS.
 find "${BACKUP_DIR}" -name 'vyndr-*.dump' -type f -mtime "+${KEEP_DAYS}" -delete || true
 
-# 4. Weekly off-box copy (Sundays). A single failure of the off-box push is a
-#    WARN, not a hard failure — the local dump still succeeded.
-if [ "$(date -u +%u)" = "7" ]; then
+# 3b. SSH key for the off-box push (Session 64).
+#     The container filesystem is EPHEMERAL — a keypair generated inside it dies
+#     on the next redeploy and the off-box push would silently start failing. So
+#     the PRIVATE key is injected as an env var (Coolify secret) and written to a
+#     0600 temp file per run. Hetzner Storage Box speaks full OpenSSH on PORT 23
+#     (port 22 is SFTP-only, mod_sftp) — verified live; rsync must target 23.
+SSH_KEY_FILE=""
+cleanup_key() { [ -n "${SSH_KEY_FILE}" ] && rm -f "${SSH_KEY_FILE}" || true; }
+trap cleanup_key EXIT
+RSYNC_SSH="ssh -p ${BACKUP_SSH_PORT:-23} -o StrictHostKeyChecking=accept-new -o BatchMode=yes"
+if [ -n "${BACKUP_SSH_KEY:-}" ]; then
+  SSH_KEY_FILE="$(mktemp)"
+  chmod 600 "${SSH_KEY_FILE}"
+  # Accept the key with literal \n escapes (how env vars usually carry it).
+  printf '%b\n' "${BACKUP_SSH_KEY}" | sed -e 's/[[:space:]]*$//' > "${SSH_KEY_FILE}"
+  RSYNC_SSH="${RSYNC_SSH} -i ${SSH_KEY_FILE}"
+fi
+
+# 4. OFF-BOX COPY — every night, not only Sundays (Session 64).
+#    A weekly push meant up to 6 days of dumps existed ONLY inside an ephemeral
+#    container, which is the same as not existing. Off-box is the real backup.
+if true; then
   if [ -n "${BACKUP_REMOTE:-}" ]; then
-    if rsync -az --timeout=120 "${DUMP}" "${BACKUP_REMOTE}"; then
-      notify "VYNDR backup OK (+off-box)" "default" "Nightly dump ${STAMP} (${SIZE} bytes) + weekly off-box copy pushed."
+    if rsync -az --timeout=120 -e "${RSYNC_SSH}" "${DUMP}" "${BACKUP_REMOTE}"; then
+      notify "VYNDR backup OK (+off-box)" "default" "Nightly dump ${STAMP} (${SIZE} bytes) pushed off-box to ${BACKUP_REMOTE%%:*}."
     else
-      notify "VYNDR off-box push FAILED" "high" "Local dump ${STAMP} is fine (${SIZE} bytes) but the weekly off-box rsync failed."
+      notify "VYNDR off-box push FAILED" "high" "Local dump ${STAMP} is fine (${SIZE} bytes) but the off-box rsync FAILED — the dump exists only in an ephemeral container."
     fi
   else
-    notify "VYNDR off-box push SKIPPED" "high" "Local dump ${STAMP} OK but BACKUP_REMOTE is unset — no off-box copy this week."
+    notify "VYNDR off-box push SKIPPED" "high" "Local dump ${STAMP} OK but BACKUP_REMOTE is unset — the dump exists only in an ephemeral container."
   fi
 else
   echo "backup ok: ${DUMP} (${SIZE} bytes)"

src/routes/internal.js

@@ -240,6 +240,37 @@ router.post('/refresh/all', async (req, res) => {
   }
 });
 
+/**
+ * POST /api/internal/backup/run (Session 64) — fire ONE real database backup
+ * now: pg_dump → validate (`pg_restore --list` must contain ledger_entries) →
+ * off-box rsync to the Storage Box.
+ *
+ * Exists because the backup can only run where SUPABASE_DB_URL and the Supabase
+ * network route live — inside this container — and there was previously no way
+ * to trigger or observe it without a shell on the box. Returns the script's
+ * exit code and output tail so a real run can be VERIFIED, not assumed.
+ */
+router.post('/backup/run', async (req, res) => {
+  const { runBackup, durabilityWarning } = require('../backupScheduler');
+  try {
+    const started = Date.now();
+    const result = await runBackup();
+    return res.json({
+      ok: result.ok,
+      exit_code: result.code,
+      duration_ms: Date.now() - started,
+      durability_warning: durabilityWarning() || null,
+      remote_configured: !!process.env.BACKUP_REMOTE,
+      ssh_key_configured: !!process.env.BACKUP_SSH_KEY,
+      output_tail: result.tail || result.error || null,
+    });
+  } catch (err) {
+    const message = err && err.message ? err.message : String(err);
+    console.error('[internal/backup/run] failed:', message);
+    return res.status(500).json({ ok: false, error: message });
+  }
+});
+
 /**
  * POST /api/internal/ledger/settle (Session 58, Phase 1) — settle the
  * persistent ledger (outcome + actual_value + CLV) across every sport.