5cad851922f31ff844ef009e80f4fa90b5129686
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 builtbykev
|
a8de676756 |
A probability is served because evidence supports it, not because nothing else answered
The band gate was blocked for its `else` branch. It read:
candidate = F(raw)
served = inCertifiedBand(candidate) ? candidate : RAW
and above raw 0.60 the model is measured overconfident — holdout raw 0.80-0.90
predicts 0.843 and realizes 0.639. So "the calibrator is not supported here" was
being answered with a number already proven wrong. Unsupported calibration does
not make raw true.
Four candidates were adjudicated on ONE split — fit on the earliest 60% of
train, decide support on the last 40%, evaluate on a holdout that saw neither:
A low-param 80.2% coverage 0.24374 REFUTED — its extra region
(raw 0.80-0.90) certified on cert (err +0.040, n=55) and
refuted on holdout (served 0.754 vs observed 0.639), and it
leaves a hole at 0.70-0.80 while serving the island above it
B isotonic 91.3% coverage 0.24337 CERTIFIED, contiguous raw [0.50,0.80)
C empirical band 91.3% coverage 0.24335 REFUTED — refitted point-in-time on
current-model hits the realized rates INVERT in grade order
(B+ 0.593 < B 0.614 < C+ 0.623), so the served function steps
down at raw 0.78. Its shipped constants come from 3,417 props
pooled across four batter stats and do not reproduce here
D raw identity 43.1% coverage 0.24866 certifies raw 0.50-0.60 and only there
Raw is candidate D, not a fallback. It earns exactly one region (holdout error
+0.010 on n=1,316), which is why the law is "raw must earn its region" rather
than "raw is never true". B already covers that region, so no hybrid is built.
Above raw 0.80 nothing is certified and nothing is served. That is the region
where raw is most wrong, isotonic over-corrects (cert err -0.093) and its LODO
mapping at 0.95 has spread 0.180. 8.7% of holdout rows land there.
The registry did not need changing. `serves(stat, p)` already tested certified
bands against the RAW p_win — support in the input domain, the correct question —
and returned {serve:false, reason}. It never said "serve raw". The output-space
gate and the raw fallback were both invented downstream in calibrationService.
ACTIVATION IS OFF. PROBABILITY_CONTRACT_SHADOW defaults to 0, CALIBRATION_DEPLOYED
stays frozen empty, and every served field is byte-identical. This releases the
support first, which is the required order. The shadow records raw belief, the
candidate served value, the state, the estimator identity, and what EV/Kelly/VALUE
would be under the actionability law — into its own column, read by nothing.
Migration 051 was applied to production BEFORE retentionService named the column.
PostgREST builds a bulk insert from the first row's shape, so a key whose column
does not exist 400s the whole batch silently — that is how migration 038 took
retention down for three days.
The user-facing contradiction is NOT fixed here. A B+ still says "realized about
66%" beside a confidence of 84. Fixing that is activation, and activation costs
32% of VALUE flags and 46% of Kelly recommendations on the holdout.
Suite 401/401, 5,580 passed, 4 skipped, deterministic across three runs.
Teeth 23/23, each independently injected and restored byte-identically.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
|
||
|
builtbykev
|
9809626c99 |
Retention completion: a cohort is complete only when the writer says N of N
The previous bug made the recorder write nothing. The dangerous successor is a
recorder that writes half and looks healthy: persist() writes in chunks of 250
and STOPS AT THE FIRST FAILED CHUNK, so chunks committed before the failure are
already durable. Rows exist under the snapshot_id, captured_at is uniform, Redis
kept working — and the cohort is short.
So row presence was never completion evidence, and neither was a matching
timestamp. Completeness is now proven by the writer or not at all.
TERMINAL RETENTION STATES (retentionService.classifyPersist):
NOTHING_TO_PERSIST attempted 0 — a refusal-only slate is still a cycle
SKIPPED_NO_DATABASE no database configured; not a failure
COMPLETE attempted > 0, written === attempted, no error
FAILED_ZERO_WRITE written === 0 — first chunk failed
FAILED_PARTIAL 0 < written < attempted — a later chunk failed
FAILED_UNRESOLVED_ERROR counts look complete but an error is unresolved;
unreachable through today's loop, and kept because
the alternative is reporting COMPLETE holding an error
The invariant: any written < attempted with attempted > 0 is a FAILED cycle. A
partial cohort is never degraded success.
classifyPersist reads the EXACT persist() result and refuses anything else — it
never recomputes attempted or written, because a second calculation could
disagree with the writer and then the status would describe a cycle that did not
happen. persist() itself is byte-identical to
|
||
|
builtbykev
|
35da190f2c |
Retention hotfix: drop published_side, derive the schema contract, break the silence
`createCollector.onPublished` set `published_side` beside `published`.
`published_side` is not a model_snapshots column. supabase-js declares the
UNION of row keys in the `columns=` parameter, so one invalid key made
PostgREST reject the ENTIRE batch with a 400 — every sport, every cycle.
Retention is best-effort, so nothing surfaced. Confirmed in edge logs.
The field was redundant as well as invalid: `side` is already on the row.
Deleted rather than added to the schema — a column would preserve an
accidental artifact.
Three things missed it, and each is now closed:
1. WRONG SHAPE INSPECTED. The manual check sampled the collector after
onGraded only and never called onPublished, so the offending key was
not yet on the row. It read a pre-publication shape and reported the
final outbound shape as clean. The new test captures the array actually
handed to .upsert(), after the full production call order.
2. NO CONTRACT. Every retention test injects a permissive fake client that
accepts any column set, so 381 suites proved the logic and never once
compared a row against the database. The contract is now DERIVED — the
migration chain applied to a disposable postgres, read out of
information_schema (scripts/generate-schema-contract.js). A
hand-maintained list would be a second opinion about the schema, and a
second opinion is what let this through. scripts/verify-schema-contract.js
checks the contract still describes a live database.
3. SILENT FAILURE. A failed batch reached one console.log. It now emits a
high-severity structured event carrying sport, snapshot id, stage,
error, code_sha and timestamp. Best-effort semantics are unchanged —
the product continues and says so — but the failure is observable.
`skipped` (no database configured) is not a failure and does not alert.
Teeth, each with the injection verified present before the run:
- published_side back into the final payload -> 4 tests fail; restored
byte-identically (sha 6a0ced7c52134135 both sides)
- settled_at (a REAL contract column) -> accepted, so the guard
discriminates by contract membership, not by novelty
- alert block deleted -> 3 tests fail; restored byte-identically
Model and product behaviour untouched: analyzeViaEngine1,
probabilityEstimator, gradeSlateService, lineageCanaryConfig all unchanged.
Lineage stays OFF. Net source change is one behavioural line plus the alert.
382 suites / 5,094 tests pass. web tsc exit 0 (zero web paths touched).
Measurement blackout recorded, NOT backfilled: last good retention write
2026-08-27T19:08:32Z;
|