Phase 0 gate PASSED: the join is clean. No FK exists; the natural key
(sport, player_key, stat, line, side, game_date) yields 283 clean 1:1
joins with ZERO ambiguity. game_id is NOT usable — 400/550 snapshot rows
carry UNK@UNK because home/away names weren't threaded into the grader
until Order 1.6. Non-joining rows are EXPECTED, not errors: retention
stores both sides plus refusals; the ledger keeps only the graded side.
Outcomes are NOT denormalized — ledger_entries stays the source of truth.
BUILT TEST-FIRST, and the first property proven is the REFUSAL, not the
math. Below threshold the harness emits INSUFFICIENT with n and the
shortfall and NO rate anywhere in the payload, so a downstream renderer
cannot surface one by accident. A test asserts the payload contains no
hit_rate number at all.
- Wilson intervals (correct at the n we actually have, unlike the normal
approximation which emits negative lower bounds).
- Strata NEVER mix sport or model_version.
- Denominator excludes quarantined, void, unrecoverable, pending, push —
asserted by test.
- Monotonicity refuses to RANK buckets whose intervals overlap; it reports
"not distinguishable on this sample".
- Probability calibration (Brier + reliability) also respects the
threshold: a thin sample returns status INSUFFICIENT and a NULL score.
- Replay seam reads the STORED feature vector only. A row whose input was
never retained is UN-BACKTESTABLE, never scored with substituted current
data. Identity replay reproduces the live prediction exactly.
The tests caught a real bug in my own code: `Number(null) === 0` let a
null p_win through as a confident 0% forecast — this codebase's signature
fabrication bug, inside the harness whose entire purpose is refusing
invented numbers. Fixed with a strict null guard.
FIRST LIVE RUN — the correct, passing output:
VERDICT: INSUFFICIENT_HISTORY (can_validate=false)
283 joined -> 35 scored (120 quarantined, 124 pending, 4 terminal)
C n=18 (short by 2), B n=17 (short by 3)
strata: mlb 7, wnba 28 — never mixed
migration 028 adds harness_results (append-only trend log; INSUFFICIENT
rows are expected and correct) and opsWatch.harnessStaleAlarm pages if the
harness stops running — a validator that isn't running looks exactly like
one that keeps passing.
Suite 283/3403 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
Order 2 Phases 2 + 4. Pre-heal rollback point secured first:
vyndr-20260720-093821.dump (856,890 bytes) VERIFIED ON THE BOX, not just
exit 0.
MIGRATION 027 — two DISTINCT exclusion scopes, deliberately separate:
- quarantine_reason: the row's GRADE is untrustworthy (wrong_opponent_grade).
The row REMAINS a real public settled result — the bet happened, the
outcome is real — but it must never train or validate, so
getModelAggregate now excludes it from the denominator alongside
void/unrecoverable.
- analysis_flags: the row is VALID for settlement and the record but
unattributable for PER-GAME analysis (doubleheader dates). Explicitly NOT
filtered from aggregates.
Collapsing these would either wrongly drop 166 doubleheader rows from the
record or wrongly keep 25 wrong-opponent grades inside model validation.
Tests assert both directions, including that analysis_flags is NOT filtered.
Also adds re_settled_at + settlement_source to model_snapshots.
DNP VOIDING RE-ENABLED — reversing my own Order 1.5 disable, with scrutiny,
because its premise was FALSE. Order 1.5 assumed a missing player row meant
the row's DATE was wrong. The Phase 0 dry-run disproved it: across every
bindable row the stored date matched a real game (MIS-DATED: 0), and the
players I had cited as counter-evidence were genuine DNPs on their true
dates (Freeman 07-18; Kwan/Hedges/Davis 07-17 — their teams played, they
did not). The evidence is positive: games FINAL + no line in a full-season
log = no bet existed.
I got this wrong twice tonight in opposite directions; the dry-run is what
caught it. Recording the reasoning in the code so the next reader sees why
the flag flipped back.
Suite 282/3386 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
PHASE 1 — HOST KEY STATICALLY PINNED. ssh-keyscan -p 23 returned an
ED25519 key whose fingerprint EQUALS the out-of-band value
SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM, so it is safe to pin.
scripts/storagebox_known_hosts now carries that verified line and ships to
the container (Dockerfile already COPYs scripts/). backup-db.sh uses
StrictHostKeyChecking=yes + UserKnownHostsFile=<pin> instead of
accept-new, which was trust-on-first-use and would have accepted an
impostor on the very first run. A missing pin file REFUSES the push rather
than silently falling back. Never weakened to accept-new/=no//dev/null —
a test asserts that on executable lines.
PHASE 1b — REMOTE DIR GUARANTEED. The box has only .ssh/, and rsyncing a
file into a missing parent either fails or silently writes the dump AS the
directory name — one file, overwritten nightly, reading as "backups exist"
while retaining exactly one. Uses rsync --mkpath when available, else an
explicit remote mkdir -p ahead of the push.
PHASE 2b — FAILED OFF-BOX PUSH IS NOW LOUD. Off-box is required, so the
failed-push path pages at "urgent" (was "low"/deferred) and the script
emits a machine-readable OFFBOX_OK=1/0/deferred that
POST /api/internal/backup/run surfaces as a distinct offbox_ok field.
Exit code deliberately still reflects ON-BOX durability — a good on-box
dump must not raise a false total-failure alarm. Surfacing the truth, not
manufacturing a failure.
No key material is echoed anywhere; only the PUBLIC host key is committed.
Suite 280/3338 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
RETENTION (Phase 2, priority zero). History starts compounding tonight.
migration 025 model_snapshots — APPLIED to prod. Append-only, one row per
graded prop PER SIDE PER CYCLE, with a unique index on
(snapshot_id, player_key, stat, line, side) so a retried cycle cannot
duplicate. RLS on, service-role writes only.
What it captures that the ledger never did:
- features jsonb — the model's INPUTS. Without these a backtest can only
grade our own homework; with them any future model can be replayed
against the exact conditions this one faced.
- REFUSALS (refused + refusal_reason). The ledger drops them, so a gate
refusing props that would have WON is invisible — unmeasurable lost
edge. Captured via a new onGraded hook in gradeSlateService that fires
with BOTH sides before any filtering.
- grade_11, the pre-collapse grade. The 4-letter map throws away the
entire live C-/C/C+/B- range.
- model_version + code_sha on every row. ledger_entries mixes pre/post-fix
grades with no marker and cannot be separated retroactively.
- p_win / ev_pct / fair_odds / takeable / value — none of which any
permanent store held.
Wiring: analyzeViaEngine1 attaches _features/_grade_11 (underscore =
internal); gradeSlateService fires onGraded then STRIPS them so they never
reach a cache or API payload; snapshotService builds rows and persists
best-effort. Retention reuses the LEDGER's dateET/gameIdFor helpers so
rows share the ledger's natural key exactly — otherwise the settle pass
could never join outcomes onto them. Rows are written BEFORE the empty-
slate early return: a slate that refused everything is exactly the case
worth recording.
CONTRACT HELD: retention is injectable and every path is caught. persist()
returns errors, never throws; a missing Supabase client is SKIPPED, not an
error. A retention failure can never break a snapshot.
BACKUP: backup-db.sh now accepts BACKUP_SSH_KEY as base64 (recommended —
survives env-var newline mangling, which is how injected SSH keys usually
break silently) OR raw PEM, detected by decoding and looking for the PEM
header. Verified both forms detect correctly against a real generated key.
Suite 279/3325 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
BACKUP_DIR is now a persistent volume (/app/backups), so the dump already
survives redeploys — the container-ephemeral risk that made this urgent is
closed. Storage Box SSH auth is not sorted yet, so the off-box push is
explicitly DEFERRED rather than failing:
- gated on BACKUP_OFFBOX=1 (plus BACKUP_REMOTE and BACKUP_SSH_KEY); until
then the script logs "off-box push DEFERRED" and exits clean.
- if an enabled push DOES fail, it is a LOW-priority "deferred" notice, not
a failure — the durable on-box dump succeeded, and calling that an
incident would train us to ignore backup alerts.
Adds the read-back check, because a backup nobody has read is a hope:
countRowsInDump() runs `pg_restore --data-only --table=X -f -` and counts
the rows between `FROM stdin;` and the terminating `\.`, proving the
archive CONTAINS the data rather than merely parsing. Needs no Postgres
server, so it runs inside the API container. Validated against a real
pg_dump from a scratch Postgres: counted exactly 604 rows.
GET /api/internal/backup/verify exposes it (newest dump in BACKUP_DIR,
size, table, rows_in_dump). Unit tests inject spawn/fs so CI needs neither
docker nor pg_restore.
Suite 278/3310 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
Closing the backup for real. Three changes, each fixing something that
would have made the Storage Box target fail or silently rot.
1. SSH KEY COMES FROM ENV, not from the container. Generating a keypair
inside the API container was the obvious move and it is wrong: the
container filesystem is ephemeral, so the key dies on the next
redeploy and the off-box push starts failing silently. backup-db.sh
now reads BACKUP_SSH_KEY (a Coolify secret), writes it to a 0600 temp
file per run, and removes it on exit via trap.
2. PORT 23, verified live. Hetzner Storage Box runs full OpenSSH on 23;
port 22 answers with mod_sftp (SFTP only). Banner-checked both against
u635423.your-storagebox.de. rsync now uses
-e "ssh -p ${BACKUP_SSH_PORT:-23} ... -i <key>"; the old invocation had
no -e at all and would have gone to 22.
3. OFF-BOX PUSH IS NIGHTLY, not Sundays-only. A weekly push meant up to
six days of dumps existed ONLY inside an ephemeral container, which is
the same as not existing. Alert copy updated to say exactly that when
the push fails or is skipped.
Also adds POST /api/internal/backup/run (internal-key gated) so a real
backup can be TRIGGERED and OBSERVED — it returns exit code, duration,
output tail, and whether the remote + ssh key are configured. The backup
can only run where SUPABASE_DB_URL and the Supabase route live (this
container), and there was no way to fire or inspect it without a shell.
Connectivity established this session: Storage Box reachable from the dev
box on 22/23; Supabase :5432 NOT reachable from WSL2 (so the dump must
run in-container, as designed); docker IS available locally, so the
restore-verify can run against a scratch Postgres using the real dump.
Suite 278/3305 green, build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
FOUNDATION-FIRST re-order, phase 1 (tooling + safety).
BACKUP (highest-severity open item) — INSTALLED, not re-proven.
src/backupScheduler.js runs scripts/backup-db.sh nightly from inside the
API container, armed at boot in server.js. The container already has
SUPABASE_DB_URL, pg_dump and the Supabase route, so deploy == installed:
no host crontab, no Coolify click. Arming is deliberately opt-OUT (armed
whenever SUPABASE_DB_URL exists; BACKUP_CRON=0 kills it) because the S62
design was opt-in and nobody ever opted in — the DB went unbacked every
night for weeks. A failed run pages high-priority ntfy; silence is the
danger with backups.
Durability is the one part still needing a human: the container FS is
ephemeral, so a dump dies on redeploy unless BACKUP_REMOTE (off-box
rsync) or BACKUP_DIR (persistent volume) is set. The scheduler detects
that and pages a WARNING at boot rather than letting an undurable backup
read as "backed up". Runbook rewritten to lead with the code path.
MANUAL REGRADE TRIGGER — scripts/run-snapshot.js, runnable via
docker exec with no VYNDR_INTERNAL_KEY and no new HTTP surface. Runs the
SAME snapshotService.runSnapshot the cron runs (including the team-stats
refresh that powers opp_rank_stat), supports `all` and `--settle`, and
prints the grade/confidence distribution plus p_win/ev_pct presence —
which is the thing you actually want when verifying a grading change.
ACCESS BLOCKER, logged honestly in specs/model-train.md: there is no
VYNDR_INTERNAL_KEY in the local .env and SSH to the box times out from
WSL2, so I can neither curl the internal endpoints (which already exist
from S45) nor docker exec. The trigger is built and correct but only Kev
can run it until a key or SSH access exists. This is the highest-leverage
unblock for phases 2 and 3, which both need on-demand regrade+settle to
verify anything.
Also logged the standing cautions: CLV ledger stays private until
backtest-proven; "self-improving model" is unsupported marketing until
the loop closes; the engine is MLB/WNBA-calibrated and NFL/NBA/soccer
need their own calibration before the hub grades them (scaling gate).
Suite 277/3300 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
Folds re-sequenced steps 1+2 into one change (Kev's call): same bug
family — features wired to sources that return null.
THE PROBABILITY LAYER WAS DEAD IN PRODUCTION. p_win/ev_pct/kelly/
model_odds/value were absent on 0/8 live grades because
gameLogService.getGameLogs returns null for MLB by construction and
depends on the offline Python service for NBA/WNBA, so meta.gameLogs was
[] for every sport. This was the S46 bug in a second location — that fix
gave featureCache an MLB branch (why grades still worked) but never the
estimator. featureCache.getStatRows now supplies normalized rows
([{date,[statType]:v}], most-recent-first) for every sport, feeding the
estimator AND consistency AND game_count_in_7d from one fetch.
VERIFIED on real props: p_win 25/25 WNBA, 8/8 MLB (was 0).
GRADE RANGE, ON MERIT — never by rescaling (permanent founder ruling:
minting A's without new information is a relabelled B sold as an A and
corrupts an append-only ledger).
- refreshTeamStats wired into runSnapshot — it had ZERO production
callers, so opp_rank_stat was permanently null and a +/-1.0 factor
could never fire. Test-env no-op (opsNotify precedent).
- L20 made SYMMETRIC: both branches were delta +1.0, so the season
baseline could only ever ADD. No negative path was a structural reason
D was unreachable. New l20_contradicts_* carries -1.0.
- game_count_in_7d derived from real logged dates (heavy_workload_7d).
- NOT wired, deliberately, with reasons inline: teamId (no team_id
column; getFeatures reads it top-level; factor also needs a starter-id
list) and season_type (ESPN 2 = REGULAR season; threading it raw would
fire veteran_in_playoffs in July). Dead code dressed as a fix is the
thing we are removing, not adding.
CALIBRATION GUARD (found by verifying, not assuming): consistency CV is
NBA-tuned; for a Poisson-ish stat cv ~ 1/sqrt(mean), so any stat with
mean < 4 auto-classifies boom_bust. First verification run showed 8/8 MLB
props boom_bust — a blanket -1.0 that dropped the board to all-C. Floored
at CONSISTENCY_MIN_MEAN=4 -> 'unknown' below. Absent beats wrong. MLB
low-count stats therefore still get no consistency factor: honest, not
fixed. Scale-free index-of-dispersion classifier is the open follow-up.
CONFIDENCE IS NOT A PROBABILITY: payloads carry confidence_basis:
'grade_band'. Corrected mlb-grade-degradation.md — its "25/25
grade<->confidence agreement" is a TAUTOLOGY (confidence is derived FROM
the letter, so it would report 25/25 even if every grade were wrong), not
a validation. Removed dead mlbGrader.js (referenced only by its own test)
and the stale computeFeatures comment claiming a penalty that never ran.
VERIFICATION (scripts/verify-grade-range.js, real props/logs/engine):
WNBA 25 props B 68%->32%, C 32%->64%, D 0->1 (4%); 11-step spread went
from 2 steps to 5 (C/C+/B-/D). The D is earned: Angel Reese assists o2.5,
p_win 0.365. Nothing flooded — grades got HARDER. A did not emit locally
because opp_rank_stat needs the Redis cache only prod populates (local
ceiling +3.0 vs the +4.5 A needs); reachability is proven arithmetically
and locked in tests. Prod A-emission is the outstanding fingerprint.
MARKETING HOLD: "A-RATED" (AccuracyBadge, TopSignals) is unsupported
until that fingerprint. Confirmed honest fallbacks render today —
/api/ledger/accuracy has B and C buckets only, so the badge shows
"MODEL · 63% HIT" and TopSignals self-hides. Nothing fabricated ships.
Suite 276/3286 green, web build exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmNjJAwEnqHPtXbvSZR8kA
SUPABASE_DB_URL is set in Coolify on the API service, and this WSL2 box can't
reach db.<ref>.supabase.co — so the backup runs INSIDE the API container, which
has the env + Supabase network. Made that real:
- Dockerfile: install postgresql-client (pg_dump/pg_restore) + rsync + bash in
the runner image.
- backup-db.sh: added an integrity fingerprint on every run — pg_restore --list
must parse the archive AND find ledger_entries, else the run FAILS + pages
(stronger than the size check; catches a corrupt/structureless dump).
- BACKUP-RUNBOOK.md: rewritten for the container-exec reality — host cron does
`docker exec <api> sh /app/scripts/backup-db.sh` (inherits env + network +
pg_dump), or a Coolify Scheduled Task. Full restore-fingerprint steps included.
MECHANISM FINGERPRINT (run locally, docker + pg16): seeded a ledger_entries
table (137 rows) → ran backup-db.sh (dump + validate: 22 archive objects,
ledger_entries present) → pg_restore into a scratch DB → 137 rows restored,
exact match. The dump/validate/restore path is proven end-to-end; it's the same
pg_dump/pg_restore that run in the container against Supabase.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/backup-db.sh: nightly full-DB pg_dump via the direct connection string,
14-day local rotation, weekly off-box rsync copy, ntfy alert on any failure +
an undersized-dump guard (an empty dump is a silent failure). docs/BACKUP-
RUNBOOK.md: the ONE env var Kev must set (SUPABASE_DB_URL — the direct
db.<ref>.supabase.co:5432 URI, not the pooler), the cron line, the off-box
target (Hetzner Storage Box via rsync, simplest for a Hetzner box), and the
restore FINGERPRINT procedure (pg_restore into a scratch DB + count
ledger_entries — proves it's a real, restorable backup).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/validate-grade-fix.js checks the live MLB snapshot for the three
degradation signatures (projection=0, edge=100 cluster, grade/conf disagreement)
— run after the next 14:00 UTC regrade to fingerprint the fix. The finding doc
now records root causes, fixes (commits 888d103/9fc4edf), the blast-radius SQL
(box can't reach Supabase directly), and the shared-path note for NBA/WNBA.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
builtbykev