Closing the backup for real. Three changes, each fixing something that
would have made the Storage Box target fail or silently rot.
1. SSH KEY COMES FROM ENV, not from the container. Generating a keypair
inside the API container was the obvious move and it is wrong: the
container filesystem is ephemeral, so the key dies on the next
redeploy and the off-box push starts failing silently. backup-db.sh
now reads BACKUP_SSH_KEY (a Coolify secret), writes it to a 0600 temp
file per run, and removes it on exit via trap.
2. PORT 23, verified live. Hetzner Storage Box runs full OpenSSH on 23;
port 22 answers with mod_sftp (SFTP only). Banner-checked both against
u635423.your-storagebox.de. rsync now uses
-e "ssh -p ${BACKUP_SSH_PORT:-23} ... -i <key>"; the old invocation had
no -e at all and would have gone to 22.
3. OFF-BOX PUSH IS NIGHTLY, not Sundays-only. A weekly push meant up to
six days of dumps existed ONLY inside an ephemeral container, which is
the same as not existing. Alert copy updated to say exactly that when
the push fails or is skipped.
Also adds POST /api/internal/backup/run (internal-key gated) so a real
backup can be TRIGGERED and OBSERVED — it returns exit code, duration,
output tail, and whether the remote + ssh key are configured. The backup
can only run where SUPABASE_DB_URL and the Supabase route live (this
container), and there was no way to fire or inspect it without a shell.
Connectivity established this session: Storage Box reachable from the dev
box on 22/23; Supabase :5432 NOT reachable from WSL2 (so the dump must
run in-container, as designed); docker IS available locally, so the
restore-verify can run against a scratch Postgres using the real dump.
Suite 278/3305 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
SUPABASE_DB_URL is set in Coolify on the API service, and this WSL2 box can't
reach db.<ref>.supabase.co — so the backup runs INSIDE the API container, which
has the env + Supabase network. Made that real:
- Dockerfile: install postgresql-client (pg_dump/pg_restore) + rsync + bash in
the runner image.
- backup-db.sh: added an integrity fingerprint on every run — pg_restore --list
must parse the archive AND find ledger_entries, else the run FAILS + pages
(stronger than the size check; catches a corrupt/structureless dump).
- BACKUP-RUNBOOK.md: rewritten for the container-exec reality — host cron does
`docker exec <api> sh /app/scripts/backup-db.sh` (inherits env + network +
pg_dump), or a Coolify Scheduled Task. Full restore-fingerprint steps included.
MECHANISM FINGERPRINT (run locally, docker + pg16): seeded a ledger_entries
table (137 rows) → ran backup-db.sh (dump + validate: 22 archive objects,
ledger_entries present) → pg_restore into a scratch DB → 137 rows restored,
exact match. The dump/validate/restore path is proven end-to-end; it's the same
pg_dump/pg_restore that run in the container against Supabase.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/backup-db.sh: nightly full-DB pg_dump via the direct connection string,
14-day local rotation, weekly off-box rsync copy, ntfy alert on any failure +
an undersized-dump guard (an empty dump is a silent failure). docs/BACKUP-
RUNBOOK.md: the ONE env var Kev must set (SUPABASE_DB_URL — the direct
db.<ref>.supabase.co:5432 URI, not the pooler), the cron line, the off-box
target (Hetzner Storage Box via rsync, simplest for a Hetzner box), and the
restore FINGERPRINT procedure (pg_restore into a scratch DB + count
ledger_entries — proves it's a real, restorable backup).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
builtbykev