Exit 0 from the backup script is deliberately tied to ON-BOX durability,
so it is not proof the off-box copy landed. GET /api/internal/backup/offbox
runs rsync --list-only against BACKUP_REMOTE using the SAME pinned
known_hosts as the push (checking never disabled) and returns the dumps
actually present, with size and timestamp — so off-box presence is a
verified fact rather than an inference from an exit code.
Needed because the dev box cannot authenticate to the Storage Box: the
authorized key installed there is Kev's ~/vyndr-backup-key, not the
keypair generated in-session, so independent verification has to run from
the container that does hold working credentials.
Suite 280/3338 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
BACKUP_DIR is now a persistent volume (/app/backups), so the dump already
survives redeploys — the container-ephemeral risk that made this urgent is
closed. Storage Box SSH auth is not sorted yet, so the off-box push is
explicitly DEFERRED rather than failing:
- gated on BACKUP_OFFBOX=1 (plus BACKUP_REMOTE and BACKUP_SSH_KEY); until
then the script logs "off-box push DEFERRED" and exits clean.
- if an enabled push DOES fail, it is a LOW-priority "deferred" notice, not
a failure — the durable on-box dump succeeded, and calling that an
incident would train us to ignore backup alerts.
Adds the read-back check, because a backup nobody has read is a hope:
countRowsInDump() runs `pg_restore --data-only --table=X -f -` and counts
the rows between `FROM stdin;` and the terminating `\.`, proving the
archive CONTAINS the data rather than merely parsing. Needs no Postgres
server, so it runs inside the API container. Validated against a real
pg_dump from a scratch Postgres: counted exactly 604 rows.
GET /api/internal/backup/verify exposes it (newest dump in BACKUP_DIR,
size, table, rows_in_dump). Unit tests inject spawn/fs so CI needs neither
docker nor pg_restore.
Suite 278/3310 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
FOUNDATION-FIRST re-order, phase 1 (tooling + safety).
BACKUP (highest-severity open item) — INSTALLED, not re-proven.
src/backupScheduler.js runs scripts/backup-db.sh nightly from inside the
API container, armed at boot in server.js. The container already has
SUPABASE_DB_URL, pg_dump and the Supabase route, so deploy == installed:
no host crontab, no Coolify click. Arming is deliberately opt-OUT (armed
whenever SUPABASE_DB_URL exists; BACKUP_CRON=0 kills it) because the S62
design was opt-in and nobody ever opted in — the DB went unbacked every
night for weeks. A failed run pages high-priority ntfy; silence is the
danger with backups.
Durability is the one part still needing a human: the container FS is
ephemeral, so a dump dies on redeploy unless BACKUP_REMOTE (off-box
rsync) or BACKUP_DIR (persistent volume) is set. The scheduler detects
that and pages a WARNING at boot rather than letting an undurable backup
read as "backed up". Runbook rewritten to lead with the code path.
MANUAL REGRADE TRIGGER — scripts/run-snapshot.js, runnable via
docker exec with no VYNDR_INTERNAL_KEY and no new HTTP surface. Runs the
SAME snapshotService.runSnapshot the cron runs (including the team-stats
refresh that powers opp_rank_stat), supports `all` and `--settle`, and
prints the grade/confidence distribution plus p_win/ev_pct presence —
which is the thing you actually want when verifying a grading change.
ACCESS BLOCKER, logged honestly in specs/model-train.md: there is no
VYNDR_INTERNAL_KEY in the local .env and SSH to the box times out from
WSL2, so I can neither curl the internal endpoints (which already exist
from S45) nor docker exec. The trigger is built and correct but only Kev
can run it until a key or SSH access exists. This is the highest-leverage
unblock for phases 2 and 3, which both need on-demand regrade+settle to
verify anything.
Also logged the standing cautions: CLV ledger stays private until
backtest-proven; "self-improving model" is unsupported marketing until
the loop closes; the engine is MLB/WNBA-calibrated and NFL/NBA/soccer
need their own calibration before the hub grades them (scaling gate).
Suite 277/3300 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
builtbykev