diff --git a/docs/BACKUP-RUNBOOK.md b/docs/BACKUP-RUNBOOK.md new file mode 100644 index 0000000..83b97ac --- /dev/null +++ b/docs/BACKUP-RUNBOOK.md @@ -0,0 +1,61 @@ +# VYNDR Backup Runbook (security follow-up item 2) + +Supabase free tier has **zero** backups (no scheduled, no PITR). `scripts/backup-db.sh` +is the safety net: a nightly full-database `pg_dump`, 14 days kept locally, a +weekly copy pushed off-box, ntfy alert on any failure. + +## What Kev needs to set (one env var) + +**`SUPABASE_DB_URL`** — the Supabase **direct** connection string (session mode). +Supabase → Project → Settings → Database → **Connection string → URI**, the +`db..supabase.co:5432` one (NOT the `:6543` transaction pooler — `pg_dump` +needs a real session). Paste it in Coolify as an env var; never commit it. + +Optional: `BACKUP_REMOTE` (off-box rsync target for the weekly copy — see below). + +## Install the cron (on the Hetzner box) + +```bash +# 1. Ensure the client is present +sudo apt-get install -y postgresql-client rsync + +# 2. Nightly at 03:10 UTC. Pass the env the script needs (or source an env file). +sudo crontab -e +# add: +10 3 * * * SUPABASE_DB_URL='postgresql://...' BACKUP_REMOTE='u123456@u123456.your-storagebox.de:vyndr-backups/' /path/to/vyndr/scripts/backup-db.sh >> /var/log/vyndr-backup.log 2>&1 +``` + +(If the cron runs inside the Coolify container instead, the env vars are already +present — just schedule `scripts/backup-db.sh`.) + +## Off-box target (simplest reliable pick) + +**Hetzner Storage Box** over `rsync`/SSH — you're already on Hetzner, it's ~€3/mo +for 1TB, and needs no extra tooling. Create one, add the box's SSH key to it, set +`BACKUP_REMOTE=u@u.your-storagebox.de:vyndr-backups/`. The script pushes +the latest dump every Sunday. (Alternative: Backblaze B2 via `rclone` if you'd +rather keep it off Hetzner entirely — swap the `rsync` line for `rclone copy`.) + +## Restore / FINGERPRINT (proves it's a real backup, not just a file) + +A dump only counts once a restore of it succeeds. Load one into a scratch DB: + +```bash +# spin a throwaway local postgres, restore the newest dump, count a known table +docker run -d --name vyndr-restore-test -e POSTGRES_PASSWORD=x -p 55432:5432 postgres:15 +sleep 5 +newest=$(ls -t /var/backups/vyndr/vyndr-*.dump | head -1) +pg_restore --no-owner --no-privileges -d "postgresql://postgres:x@localhost:55432/postgres" "$newest" +psql "postgresql://postgres:x@localhost:55432/postgres" -c "select count(*) from public.ledger_entries;" +docker rm -f vyndr-restore-test +``` + +A non-zero `ledger_entries` count from the restored dump = the backup is real and +restorable. Record the date + row count as the fingerprint. + +## Alerting + +Any hard failure (missing env, `pg_dump` error, empty/undersized dump) pages +`ntfy` topic `vyndr-backups-kev2026` at urgent priority. The weekly off-box push +failing (or `BACKUP_REMOTE` unset) pages at high priority but does not fail the +run — the local dump still succeeded. Subscribe the phone to that topic. diff --git a/scripts/backup-db.sh b/scripts/backup-db.sh new file mode 100644 index 0000000..983dca5 --- /dev/null +++ b/scripts/backup-db.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# +# VYNDR nightly database backup (security follow-up item 2). +# +# Supabase free tier has ZERO backups (no scheduled, no PITR) — the ledger and +# everything else have no safety net. This dumps the WHOLE database nightly via +# the direct connection string, keeps 14 days locally, pushes a weekly copy +# off-box, and pages ntfy on ANY failure. Runs on the Hetzner box via cron. +# +# REQUIRED env (set on the box / in the container that runs the cron): +# SUPABASE_DB_URL the Supabase DIRECT connection string (session mode, the +# db..supabase.co:5432 URL — NOT the :6543 pooler; +# pg_dump needs a real session). Kev pastes this in Coolify. +# OPTIONAL env: +# BACKUP_DIR local dump dir (default /var/backups/vyndr) +# BACKUP_KEEP_DAYS local retention (default 14) +# BACKUP_REMOTE off-box rsync target for the weekly copy, e.g. +# u123456@u123456.your-storagebox.de:vyndr-backups/ +# (empty = skip the off-box push; a WARN is paged) +# NTFY_URL (default https://ntfy.sh) +# NTFY_TOPIC (default vyndr-backups-kev2026) +# +set -Eeuo pipefail + +BACKUP_DIR="${BACKUP_DIR:-/var/backups/vyndr}" +KEEP_DAYS="${BACKUP_KEEP_DAYS:-14}" +NTFY_URL="${NTFY_URL:-https://ntfy.sh}" +NTFY_TOPIC="${NTFY_TOPIC:-vyndr-backups-kev2026}" +STAMP="$(date -u +%Y%m%d-%H%M%S)" +DUMP="${BACKUP_DIR}/vyndr-${STAMP}.dump" +MIN_BYTES="${BACKUP_MIN_BYTES:-50000}" # a real dump of this DB is far bigger; guards an empty/failed dump + +notify() { # notify <priority> <message> + curl -fsS --max-time 15 \ + -H "Title: ${1}" -H "Priority: ${2}" -H "Tags: floppy_disk" \ + -d "${3}" "${NTFY_URL}/${NTFY_TOPIC}" >/dev/null 2>&1 || true +} + +fail() { notify "VYNDR backup FAILED" "urgent" "${1}"; echo "ERROR: ${1}" >&2; exit 1; } +trap 'fail "backup script errored near line ${LINENO}"' ERR + +[ -n "${SUPABASE_DB_URL:-}" ] || fail "SUPABASE_DB_URL is not set — cannot back up" +command -v pg_dump >/dev/null 2>&1 || fail "pg_dump not installed (apt-get install postgresql-client)" +mkdir -p "${BACKUP_DIR}" + +# 1. Dump the whole DB in custom format (-Fc: compressed, restorable with pg_restore). +pg_dump "${SUPABASE_DB_URL}" -Fc --no-owner --no-privileges -f "${DUMP}" \ + || fail "pg_dump failed" + +# 2. Sanity: a real dump is not tiny. An empty/near-empty file is a silent failure. +SIZE="$(stat -c%s "${DUMP}" 2>/dev/null || echo 0)" +[ "${SIZE}" -ge "${MIN_BYTES}" ] || fail "dump is only ${SIZE} bytes (< ${MIN_BYTES}) — treating as a failed backup" + +# 3. Rotate: drop local dumps older than KEEP_DAYS. +find "${BACKUP_DIR}" -name 'vyndr-*.dump' -type f -mtime "+${KEEP_DAYS}" -delete || true + +# 4. Weekly off-box copy (Sundays). A single failure of the off-box push is a +# WARN, not a hard failure — the local dump still succeeded. +if [ "$(date -u +%u)" = "7" ]; then + if [ -n "${BACKUP_REMOTE:-}" ]; then + if rsync -az --timeout=120 "${DUMP}" "${BACKUP_REMOTE}"; then + notify "VYNDR backup OK (+off-box)" "default" "Nightly dump ${STAMP} (${SIZE} bytes) + weekly off-box copy pushed." + else + notify "VYNDR off-box push FAILED" "high" "Local dump ${STAMP} is fine (${SIZE} bytes) but the weekly off-box rsync failed." + fi + else + notify "VYNDR off-box push SKIPPED" "high" "Local dump ${STAMP} OK but BACKUP_REMOTE is unset — no off-box copy this week." + fi +else + echo "backup ok: ${DUMP} (${SIZE} bytes)" +fi + +exit 0