Exit 0 from the backup script is deliberately tied to ON-BOX durability,
so it is not proof the off-box copy landed. GET /api/internal/backup/offbox
runs rsync --list-only against BACKUP_REMOTE using the SAME pinned
known_hosts as the push (checking never disabled) and returns the dumps
actually present, with size and timestamp — so off-box presence is a
verified fact rather than an inference from an exit code.
Needed because the dev box cannot authenticate to the Storage Box: the
authorized key installed there is Kev's ~/vyndr-backup-key, not the
keypair generated in-session, so independent verification has to run from
the container that does hold working credentials.
Suite 280/3338 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
PHASE 1 — HOST KEY STATICALLY PINNED. ssh-keyscan -p 23 returned an
ED25519 key whose fingerprint EQUALS the out-of-band value
SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM, so it is safe to pin.
scripts/storagebox_known_hosts now carries that verified line and ships to
the container (Dockerfile already COPYs scripts/). backup-db.sh uses
StrictHostKeyChecking=yes + UserKnownHostsFile=<pin> instead of
accept-new, which was trust-on-first-use and would have accepted an
impostor on the very first run. A missing pin file REFUSES the push rather
than silently falling back. Never weakened to accept-new/=no//dev/null —
a test asserts that on executable lines.
PHASE 1b — REMOTE DIR GUARANTEED. The box has only .ssh/, and rsyncing a
file into a missing parent either fails or silently writes the dump AS the
directory name — one file, overwritten nightly, reading as "backups exist"
while retaining exactly one. Uses rsync --mkpath when available, else an
explicit remote mkdir -p ahead of the push.
PHASE 2b — FAILED OFF-BOX PUSH IS NOW LOUD. Off-box is required, so the
failed-push path pages at "urgent" (was "low"/deferred) and the script
emits a machine-readable OFFBOX_OK=1/0/deferred that
POST /api/internal/backup/run surfaces as a distinct offbox_ok field.
Exit code deliberately still reflects ON-BOX durability — a good on-box
dump must not raise a false total-failure alarm. Surfacing the truth, not
manufacturing a failure.
No key material is echoed anywhere; only the PUBLIC host key is committed.
Suite 280/3338 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
The real backup run failed: pg_dump could not write to /app/backups —
'Permission denied'. Cause: the container runs as the non-root 'vyndr'
user (Dockerfile USER vyndr) and the Coolify-mounted volume is
root-owned, so the mount is present but unwritable.
- Dockerfile now creates AND chowns /app/backups to vyndr alongside the
existing /app/data + /app/.pm2 line. Docker seeds ownership into a
NAMED volume on first creation, so this fixes it for a fresh volume;
a host bind-mount still needs a host-side chown, which is why the
next change exists.
- GET /api/internal/backup/verify now reports process uid/gid,
backup_dir_writable and the access errno, so the exact chown target is
observable instead of guessed. A mounted-but-unwritable volume reads as
'configured' everywhere else — this makes it loud.
Suite 278/3310 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
BACKUP_DIR is now a persistent volume (/app/backups), so the dump already
survives redeploys — the container-ephemeral risk that made this urgent is
closed. Storage Box SSH auth is not sorted yet, so the off-box push is
explicitly DEFERRED rather than failing:
- gated on BACKUP_OFFBOX=1 (plus BACKUP_REMOTE and BACKUP_SSH_KEY); until
then the script logs "off-box push DEFERRED" and exits clean.
- if an enabled push DOES fail, it is a LOW-priority "deferred" notice, not
a failure — the durable on-box dump succeeded, and calling that an
incident would train us to ignore backup alerts.
Adds the read-back check, because a backup nobody has read is a hope:
countRowsInDump() runs `pg_restore --data-only --table=X -f -` and counts
the rows between `FROM stdin;` and the terminating `\.`, proving the
archive CONTAINS the data rather than merely parsing. Needs no Postgres
server, so it runs inside the API container. Validated against a real
pg_dump from a scratch Postgres: counted exactly 604 rows.
GET /api/internal/backup/verify exposes it (newest dump in BACKUP_DIR,
size, table, rows_in_dump). Unit tests inject spawn/fs so CI needs neither
docker nor pg_restore.
Suite 278/3310 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
Closing the backup for real. Three changes, each fixing something that
would have made the Storage Box target fail or silently rot.
1. SSH KEY COMES FROM ENV, not from the container. Generating a keypair
inside the API container was the obvious move and it is wrong: the
container filesystem is ephemeral, so the key dies on the next
redeploy and the off-box push starts failing silently. backup-db.sh
now reads BACKUP_SSH_KEY (a Coolify secret), writes it to a 0600 temp
file per run, and removes it on exit via trap.
2. PORT 23, verified live. Hetzner Storage Box runs full OpenSSH on 23;
port 22 answers with mod_sftp (SFTP only). Banner-checked both against
u635423.your-storagebox.de. rsync now uses
-e "ssh -p ${BACKUP_SSH_PORT:-23} ... -i <key>"; the old invocation had
no -e at all and would have gone to 22.
3. OFF-BOX PUSH IS NIGHTLY, not Sundays-only. A weekly push meant up to
six days of dumps existed ONLY inside an ephemeral container, which is
the same as not existing. Alert copy updated to say exactly that when
the push fails or is skipped.
Also adds POST /api/internal/backup/run (internal-key gated) so a real
backup can be TRIGGERED and OBSERVED — it returns exit code, duration,
output tail, and whether the remote + ssh key are configured. The backup
can only run where SUPABASE_DB_URL and the Supabase route live (this
container), and there was no way to fire or inspect it without a shell.
Connectivity established this session: Storage Box reachable from the dev
box on 22/23; Supabase :5432 NOT reachable from WSL2 (so the dump must
run in-container, as designed); docker IS available locally, so the
restore-verify can run against a scratch Postgres using the real dump.
Suite 278/3305 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
Diagnosis (why 500/500 went unpaged): the only regular odds-api burner was
futuresService, which called axios DIRECTLY — bypassing the gateway, so it
never hit recordCall (the ONE place the WARN/BLOCK pager fires) and never
respected the 95% block. It only syncFromHeaders, which updated the counter's
number SILENTLY. oddsService (which does go through the gateway) only touches
odds-api when PropLine fails, so recordCall for odds-api effectively never ran.
Result: the counter could reach 100% with neither pager firing.
Fixes (a silent drain is now impossible, not just guarded):
- futuresService routes through gateway.fetch('odds-api', …) → counted, blocked
at 95%, and reserve-gated. Closes the raw-axios bypass.
- Reserve floor in the gateway: a DISCRETIONARY call (futures/soccer) passes
reserve=ODDS_API_RESERVE (default 50) and is refused while remaining <= reserve.
The ESSENTIAL MLB prop-backup passes no reserve and may spend to the 95% block.
→ a futures/soccer drain can NEVER starve MLB's backup path.
- quotaTracker.syncFromHeaders (the AUTHORITATIVE number) now fires the same
once-per-period WARN/BLOCK alert on a crossing — extracted fireThresholdAlert
shared with recordCall. The header-only drain now pages.
- POST /api/internal/quota/test-alert (internal-key) test-fires the pager
end-to-end so ntfy delivery is verifiable on demand.
Also (reality-corrected cadence): WNBA restored to the full grid. 2026-07-15
had two AFTERNOON WNBA games finished before the 22 UTC slot — 14 UTC (10am ET)
is the only slot early enough for a 1pm ET game's props, and on PropLine the
extra slots cost a rounding error. Soccer stays the only trimmed sport (the
real odds-api discipline). Assumption corrected by observed data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4a VOICE v1.1 committed (board start); lint is EXECUTABLE — banned list +
no-exclamation law enforced in the engine (throws in test, drops in
prod) and locked by tests. Curly-apostrophe variants covered.
4b mediaEngine: deterministic templates (MORNING WIRE, SIGNAL, STREAK
WATCH, THE SETTLE, RECEIPTS, ARCHETYPE WATCH, LINE DISPATCH) filled
ONLY from snapshot/ledger/streaks JSON. Record percentages never render
under n>=20 (counts + 'Record building' below). Stark layer = curated
committed library (content/stark-lines.json), day-rotated selection —
selected, never generated.
4c /desk (founder-only: requireAuth + DESK_OWNERS email allowlist,
deny-by-default): all formats as text + <=280-char pre-segmented tweets
with per-tweet copy buttons + char counts, wire/numbers-only variants,
DATA BRIEF block (structured day numbers) with copy-for-claude.ai.
ntfy ping after the day's first snapshot: 'Desk pack ready'.
4d ghostPublisher: DRAFTS ONLY (status:'draft' test-locked), env-gated
no-op, HS256 JWT via node crypto (zero new deps). POST
/api/internal/ghost/drafts saves slate preview + settle drafts.
Nothing anywhere auto-posts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Email capture + daily report assembly + operator-triggered Listmonk send.
- NewsletterCapture (dark terminal, mono) on landing (below FAQ) + /welcome
(the real signup success surface); double-opt-in note; 'Signups open soon'
when Listmonk env is unset.
- POST /api/newsletter/subscribe: public, 10/min IP limit, honeypot,
server-side email validation, forwards to Listmonk subscribers API with
preconfirm_subscriptions:false (Listmonk sends the confirmation).
No env -> calm 200 { ok:false, reason:'not configured' }. Next proxy
web/src/app/api/newsletter/subscribe/route.ts (S25 rule).
- newsletterService.buildDailyReport: signals from snapshot:{sport}:latest,
STREAK WATCH via rosterLogs -> streaksService -> streakLens, THE RECORD via
ledgerService.getModelAggregate (percentage only when hit_pct != null —
n>=20 gate — else 'RECORD BUILDING · N pending'). RG footer (21+,
1-800-GAMBLER, Listmonk-native {{ UnsubscribeURL }}) in html + text.
VOICE v1.1 lint locked by tests: no '!', no banned vocabulary, numbers
only from injected pipeline data.
- sendDailyReport: creates + starts a Listmonk campaign; env-gated no-op;
refuses an empty report. Deliberately UNSCHEDULED — only
POST /api/internal/newsletter/send (internal key) triggers it.
- docs/NEWSLETTER.md: box-side Listmonk runbook (install, double-opt-in
list, API user, Coolify env, test-send).
- Spec: specs/feature-a1-s7-newsletter.md.
Tests 2398 -> 2429 (207 suites, all green); next build exit 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Odds-only refresh every 20 min during slate hours (noon–midnight ET),
in-process (INTRADAY_REFRESH=0 kill switch; skips full-snapshot slots):
- signed delta RELATIVE TO THE GRADED SIDE; direction is the signal.
- WITH the grade → STEAM badge, never a re-grade.
- AGAINST >=1.0 → re-grade THAT PROP ONLY at the real current line:
holds/refuses → VALUE (better entry, same read); drops → PUBLIC revision
(grade updates, revised_from_grade preserves the ORIGINAL forever, UI
strikethrough on slate strips + ledger cards).
- Every run recaptures closing_line/odds → the close is now refresh-
fidelity; SYNC goes live by dropping SNAPSHOT_EXPECTED_INTERVAL to 1200.
- Ticker MOVE events feed from the refresh (>=1.0 moves).
- QUOTA MATH: 36 runs/day/sport x 4 sports <= 144 PropLine calls/day vs
9,000/day free capacity (3 keys x 3,000). Re-grades are internal compute.
- POST /api/internal/refresh/all for manual runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ledger_entries is live (migration 019 applied to prod, RLS + NULLS NOT
DISTINCT dedupe verified against the real database). Every grade now
persists, settles against the real result, and carries closing-line value.
- ledgerService: pipeline pre-grade upserts (public model record, user_id
null, idempotent), closing capture on every snapshot (last write before
game start = the close), settlement with SIGNED CLV (over = locked -
closing; beat/faded/flat), 30d model aggregate with the hard n>=20 rule.
- Write paths: snapshotService -> ledger (priority path); Next /api/scan ->
ledger for authenticated users only (anon never touches the public
record). Refused reads write nothing and don't burn a scan.
- Honest refusal (work-order 1.5): no projection => insufficient_data,
grade null, "INSUFFICIENT DATA - no read" UI. The web gradeAdapter no
longer displays the line as the model projection (the audit's
model==line / +0% edge degenerate); the card renders absent states.
projectionFor is sport-aware (l5 -> l20 -> {stat}_per_90 -> xG).
- /ledger: MY READS | MODEL tabs; model header shows hit% + beat-close%
only at n>=20, else RECORD BUILDING + live pending count. ModelRecord
deferred-render strip on landing + player hero. CLV + outcome chips,
revised_from_grade strikethrough (Phase 2.5 ready).
- SYNC (Task 5): thresholds vs SNAPSHOT_EXPECTED_INTERVAL (normal <1.5x,
amber >=1.5x, STALE red >=3x) via /api/snapshot/summary.
- Phase 2.5 logged in specs/vyndr-roadmap.md (build after Phase 3).
- Data-semantics hardening: strict null-safe numeric parsing everywhere a
market value is handled (Number(null)===0 would have fabricated lines).
Backend 2309 -> 2327 tests (201 suites), web build exit 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Product overhaul core — the two transformative, differentiated systems:
Self-learning loop (Phase 2): outcomeService settles locked snapshot grades
against real MLB Stats API results → hit/miss/push, rolling accuracy by grade
tier (30d window). Idempotent, injectable, unit-tested. New GET /api/accuracy +
/api/ledger/accuracy + internal settle triggers + cron hook. AccuracyBadge
(dashboard/scan/landing) is honest — "LEARNING" below MIN_SAMPLE, never a fake
number. Settled HIT/MISS chips overlay the live slate.
Real-time layer (Phase 1): Slate silent 60s auto-refresh (no flash, no wipe on
transient blips) + "SIGNAL LIVE · UPDATED Xs ago" freshness strip; Ticker LIVE
badge that flashes on fresh events.
Landing (Phase 3): TopSignals shows tonight's real top-3 A-rated grades + live
accuracy — the product shown, not described.
Founder pricing: FOUNDER_CODE_EXPIRY default 2026-06-30 → 2026-12-31 (had
lapsed, disabling every founder code + the ClaimMeter pitch). That expiry — not
a tier change — was the real cause of the 4 stripe test failures.
Backend 2255 (4 failing) → 2274 (all green; +19 new, +4 fixed). Web build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The on-demand "Read" grade model is RETIRED. A scheduled pipeline pre-grades the
slate, locks grades to the line, tracks movement; the dashboard shows them already
there. Orchestrates existing services — nothing rebuilt.
- snapshotService.runSnapshot(sport): getOdds → gradeAndCacheSlate → classify
archetype per player → lock gradedAt → line deltas vs previous snapshot → write
snapshot:{sport}:latest/previous + grades:{sport} → ticker events. Fully
injectable, zero-network unit tests. runAllSnapshots = cron entrypoint.
- Internal trigger POST /api/internal/snapshot/:sport + /all (requireInternalAuth).
In-process cron (SNAPSHOT_CRON=1, UTC 14,19,22,1,3) in server.js, no new dep.
- Public reads: GET /api/snapshot/:sport (cache-only) + GET /api/ticker (merges
TICKER_MANUAL pins) + Next proxies.
- GameCard swap: live Slate renders vyndr/GameCard (legacy kept for types only),
overlays locked grades onto game props → player name once + archetype badge +
"Graded Xh ago at -115 · Current 2.5 · ▲ TOWARD +1.0". Ungraded → "Awaiting next
scan", NO Read button. On-demand onGrade flow deleted.
- Ticker polls /api/ticker every 30s, graceful fallback to hardcoded items.
- NBA/WNBA: espnStatsAdapter free fallback (defensive parse → found:false on shape
mismatch) wired into resolvePlayerStats after the offline Python service.
Env: PROPLINE_API_KEY_1/2/3, VYNDR_INTERNAL_KEY, SNAPSHOT_CRON=1, TICKER_MANUAL.
Backend 2061 -> 2100 tests (+39), 173 suites. Web build clean (exit 0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
builtbykev