The off-box push failed with 'rsync: Failed to exec ssh: No such file or
directory (2)'. The container had rsync and pg_dump from S62 but no ssh
binary, and rsync shells out to ssh for every remote transport. The dump
itself succeeded, so this failed AFTER a good backup and reads like a
network/auth problem when it is a missing package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
The real backup run failed: pg_dump could not write to /app/backups —
'Permission denied'. Cause: the container runs as the non-root 'vyndr'
user (Dockerfile USER vyndr) and the Coolify-mounted volume is
root-owned, so the mount is present but unwritable.
- Dockerfile now creates AND chowns /app/backups to vyndr alongside the
existing /app/data + /app/.pm2 line. Docker seeds ownership into a
NAMED volume on first creation, so this fixes it for a fresh volume;
a host bind-mount still needs a host-side chown, which is why the
next change exists.
- GET /api/internal/backup/verify now reports process uid/gid,
backup_dir_writable and the access errno, so the exact chown target is
observable instead of guessed. A mounted-but-unwritable volume reads as
'configured' everywhere else — this makes it loud.
Suite 278/3310 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
SUPABASE_DB_URL is set in Coolify on the API service, and this WSL2 box can't
reach db.<ref>.supabase.co — so the backup runs INSIDE the API container, which
has the env + Supabase network. Made that real:
- Dockerfile: install postgresql-client (pg_dump/pg_restore) + rsync + bash in
the runner image.
- backup-db.sh: added an integrity fingerprint on every run — pg_restore --list
must parse the archive AND find ledger_entries, else the run FAILS + pages
(stronger than the size check; catches a corrupt/structureless dump).
- BACKUP-RUNBOOK.md: rewritten for the container-exec reality — host cron does
`docker exec <api> sh /app/scripts/backup-db.sh` (inherits env + network +
pg_dump), or a Coolify Scheduled Task. Full restore-fingerprint steps included.
MECHANISM FINGERPRINT (run locally, docker + pg16): seeded a ledger_entries
table (137 rows) → ran backup-db.sh (dump + validate: 22 archive objects,
ledger_entries present) → pg_restore into a scratch DB → 137 rows restored,
exact match. The dump/validate/restore path is proven end-to-end; it's the same
pg_dump/pg_restore that run in the container against Supabase.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ROOT CAUSE: the Dockerfile copied src/poller/scripts/supabase but NOT
content/. mediaEngine.js read content/stark-lines.json with an unguarded
module-load readFileSync; ENOENT in the image threw at require time, and
via app.js → routes/desk → deskService → mediaEngine that crashed the
ENTIRE API at boot. The Coolify healthcheck rolled back to the last
healthy image (4d2b27d), so every deploy since 219167e silently served a
14-hour-old build — S11 live tracking, S6 API code, the settlement boot
line, and SNAPSHOT_EXPECTED_INTERVAL were all merged but NOT running.
FIX (one train):
1. Dockerfile COPYs content/ into the runner image.
2. mediaEngine: stark-lines.json is OPTIONAL (garnish, never load-bearing)
— loadStark() try/catch → {} → posts render without the Stark kicker,
never a crash. Belt AND suspenders with #1.
3. src/preflight.js (§A4): boot prints '[preflight] OK' or 'DEGRADED'
naming exactly what content/env is missing — before the healthcheck
can fail silently. Run first in server.js.
4. Full fragility sweep: mediaEngine was the ONLY unguarded module-load
file read; coachSignals (config/coaches.json) was already lazy +
try/catch + copied. No others.
Verified: requiring app.js + deskService + mediaEngine with
stark-lines.json ABSENT now boots clean (reproduced the exact prod
failure). 2757 -> 2763 tests (tests/unit/bootResilience.test.js).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
builtbykev