builtbykev/vyndr
1
0
Fork 0
Code Issues 1 Pull Requests Actions 6 Packages Projects Releases Wiki Activity

A probability is served because evidence supports it, not because nothing else answered

Browse Source 
The band gate was blocked for its `else` branch. It read:

    candidate = F(raw)
    served    = inCertifiedBand(candidate) ? candidate : RAW

and above raw 0.60 the model is measured overconfident — holdout raw 0.80-0.90
predicts 0.843 and realizes 0.639. So "the calibrator is not supported here" was
being answered with a number already proven wrong. Unsupported calibration does
not make raw true.

Four candidates were adjudicated on ONE split — fit on the earliest 60% of
train, decide support on the last 40%, evaluate on a holdout that saw neither:

  A low-param      80.2% coverage  0.24374  REFUTED — its extra region
                   (raw 0.80-0.90) certified on cert (err +0.040, n=55) and
                   refuted on holdout (served 0.754 vs observed 0.639), and it
                   leaves a hole at 0.70-0.80 while serving the island above it
  B isotonic       91.3% coverage  0.24337  CERTIFIED, contiguous raw [0.50,0.80)
  C empirical band 91.3% coverage  0.24335  REFUTED — refitted point-in-time on
                   current-model hits the realized rates INVERT in grade order
                   (B+ 0.593 < B 0.614 < C+ 0.623), so the served function steps
                   down at raw 0.78. Its shipped constants come from 3,417 props
                   pooled across four batter stats and do not reproduce here
  D raw identity   43.1% coverage  0.24866  certifies raw 0.50-0.60 and only there

Raw is candidate D, not a fallback. It earns exactly one region (holdout error
+0.010 on n=1,316), which is why the law is "raw must earn its region" rather
than "raw is never true". B already covers that region, so no hybrid is built.

Above raw 0.80 nothing is certified and nothing is served. That is the region
where raw is most wrong, isotonic over-corrects (cert err -0.093) and its LODO
mapping at 0.95 has spread 0.180. 8.7% of holdout rows land there.

The registry did not need changing. `serves(stat, p)` already tested certified
bands against the RAW p_win — support in the input domain, the correct question —
and returned {serve:false, reason}. It never said "serve raw". The output-space
gate and the raw fallback were both invented downstream in calibrationService.

ACTIVATION IS OFF. PROBABILITY_CONTRACT_SHADOW defaults to 0, CALIBRATION_DEPLOYED
stays frozen empty, and every served field is byte-identical. This releases the
support first, which is the required order. The shadow records raw belief, the
candidate served value, the state, the estimator identity, and what EV/Kelly/VALUE
would be under the actionability law — into its own column, read by nothing.

Migration 051 was applied to production BEFORE retentionService named the column.
PostgREST builds a bulk insert from the first row's shape, so a key whose column
does not exist 400s the whole batch silently — that is how migration 038 took
retention down for three days.

The user-facing contradiction is NOT fixed here. A B+ still says "realized about
66%" beside a confidence of 84. Fixing that is activation, and activation costs
32% of VALUE flags and 46% of Kelly recommendations on the holdout.

Suite 401/401, 5,580 passed, 4 skipped, deterministic across three runs.
Teeth 23/23, each independently injected and restored byte-identically.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
This commit is contained in:
Kev
2026-09-02 21:03:16 -04:00
parent 9dda9df132
commit a8de676756
10 changed files with 1359 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/model/probabilityContract.js
+246
View File
@@ -0,0 +1,246 @@
'use strict';
/**
* probabilityContract — WHAT NUMBER MAY BE SERVED, AND WHY.
*
* ── THE LAW THIS ENFORCES ────────────────────────────────────────────────
* RAW MODEL BELIEF IS NOT THE AUTOMATIC FALLBACK FOR SERVED PROBABILITY.
*
* The blocked contract was:
*
* candidate = F(raw)
* served = inCertifiedBand(candidate) ? candidate : RAW
*
* Its `else` branch is the defect. Above raw 0.60 the model is measured
* overconfident (holdout raw 0.80-0.90: predicted 0.843, observed 0.639), so
* "the calibrator is not supported here" was being answered with a number we
* had already proven wrong. Unsupported calibration does not make RAW true.
*
* Here there is NO fallback. A region is served because evidence supports the
* estimator that produced it, or it is not served at all.
*
* ── RAW IS A CANDIDATE, NOT A DEFAULT ────────────────────────────────────
* Raw identity is CANDIDATE D and it certifies in exactly one region
* (raw 0.50-0.60, holdout error +0.010 on n=1,316). That is a real result and
* it is why the law is "raw must earn its region", not "raw is never true".
*
* ── WHY THE REGISTRY DID NOT NEED CHANGING ───────────────────────────────
* `calibrationRegistry.serves(stat, p)` already tests certified bands against
* the RAW p_win — support in the INPUT domain, which is the correct question:
* what evidence makes THIS prediction eligible? And it returns
* `{serve:false, reason}`; it never said "serve raw". The output-space gate and
* the raw fallback were both invented downstream in calibrationService.
*
* ── THE PROBABILITY OBJECT ───────────────────────────────────────────────
* P(selected side succeeds). Proven, not assumed: 493/493 two-sided stored
* pairs satisfy p(over)+p(under)=1 within 3dp rounding. Side selection happens
* upstream in gradeBestSide and this module never touches it.
*/
const { knownNumber } = require('../../utils/known');
/** Step 17 states. NO_SERVED_PROBABILITY is the ABSENCE of a number, which is
* expressed as `served_probability === null`, not as a seventh state name. */
const STATE = Object.freeze({
CERTIFIED_CALIBRATED: 'CERTIFIED_CALIBRATED',
CERTIFIED_RAW: 'CERTIFIED_RAW',
CERTIFIED_EMPIRICAL_BAND: 'CERTIFIED_EMPIRICAL_BAND',
UNCERTIFIED: 'UNCERTIFIED',
UNSUPPORTED: 'UNSUPPORTED',
VERSION_MISMATCH: 'VERSION_MISMATCH',
INVALID: 'INVALID',
});
const CERTIFIED_STATES = Object.freeze([
STATE.CERTIFIED_CALIBRATED, STATE.CERTIFIED_RAW, STATE.CERTIFIED_EMPIRICAL_BAND,
]);
const ESTIMATOR = Object.freeze({
ISOTONIC: 'isotonic',
LOW_PARAM: 'low_param',
EMPIRICAL_BAND: 'empirical_band',
RAW_IDENTITY: 'raw_identity',
});
/**
* THE CERTIFIED ARTIFACT — MLB hits, engine1@2026-08-07-fullwindow.
*
* Adjudicated 2026-09-02 on 6,050 picked-side settled rows over 22 dates, one
* split shared by all four candidates:
* FIT earliest 60% of TRAIN (n=1,798, through 08-17) -> derive
* CERT latest 40% of TRAIN (n=1,199, 08-17..08-21) -> decide support
* HOLD >= 2026-08-22 (n=3,053, 11 dates) -> evaluate, untouched
*
* candidate coverage holdout brier vs raw on covered verdict
* A low-param 80.2% 0.24374 -0.00273 [-.0039,-.0014] REFUTED
* B isotonic 91.3% 0.24337 -0.00347 [-.0061,-.0010] CERTIFIED
* C empirical band 91.3% 0.24335 -0.00348 [-.0061,-.0011] REFUTED
* D raw identity 43.1% 0.24866 n/a (is raw) narrower
*
* A is REFUTED despite more coverage: its extra region (raw 0.80-0.90) was
* certified on CERT (err +0.040, n=55) and refuted on HOLD (served 0.754 vs
* observed 0.639, CI [0.566,0.705], err +0.115). It also leaves a hole at
* 0.70-0.80 while serving 0.80-0.90 — a discontiguous contract whose upper
* island is the wrong one.
*
* C is REFUTED as a scalar estimator: refitted point-in-time on current-model
* hits-only rows, the realized rates INVERT in grade order (B+ 0.593 <
* B 0.614 < C+ 0.623), so the served function moves DOWNWARD at raw 0.78.
* Its shipped constants come from 3,417 props POOLED ACROSS FOUR BATTER STATS
* and do not reproduce here. Two of its seven bands (D, F) have n=0.
*
* D certifies only raw 0.50-0.60 and B covers that region already, so no hybrid
* is built (Step 14: prefer one estimator; do not patchwork for coverage).
*
* B's certified support is CONTIGUOUS raw [0.50, 0.80). Holdout band errors:
* 0.50-0.60 served 0.530 observed 0.540 [0.513,0.567] -0.011 z -0.73
* 0.60-0.70 served 0.578 observed 0.613 [0.582,0.643] -0.035 z -2.22
* 0.70-0.80 served 0.616 observed 0.604 [0.561,0.645] +0.012 z +0.56
* The middle band is the weakest: p 0.027 uncorrected, which does NOT clear the
* programme's cumulative Bonferroni bar (alpha ~= 0.001). It is inside the
* repository's own 0.05 tolerance and is recorded here rather than smoothed.
*
* ABOVE raw 0.80 NOTHING is certified — isotonic over-corrects there (CERT err
* -0.093) and its LODO mapping at 0.95 has spread 0.180. That region is exactly
* where raw is most wrong, so it receives NO SERVED PROBABILITY.
*/
const MLB_HITS = Object.freeze({
sport: 'mlb',
stat: 'hits',
model_version: 'engine1@2026-08-07-fullwindow',
estimator_type: ESTIMATOR.ISOTONIC,
estimator_version: 'mlb-hits-isotonic@2026-09-02',
certification_version: 'adjudication@2026-09-02',
/** Support in the RAW INPUT domain — half-open [lo, hi). */
certified_bands: Object.freeze([Object.freeze([0.50, 0.80])]),
state_when_served: STATE.CERTIFIED_CALIBRATED,
/** Recorded so a later reader can re-derive the decision, not just trust it. */
evidence: Object.freeze({
n: 6050, dates: 22, fit_n: 1798, cert_n: 1199, holdout_n: 3053, holdout_dates: 11,
holdout_brier: 0.24337, holdout_brier_raw_same_rows: 0.24684,
delta_vs_raw: -0.00347, ci95: Object.freeze([-0.00609, -0.00099]),
coverage_pct: 0.913, tolerance: 0.05, min_bin: 40,
}),
});
const CONTRACTS = Object.freeze({ 'mlb:hits': MLB_HITS });
const contractKey = (sport, stat) => `${String(sport || '').toLowerCase()}:${String(stat || '').toLowerCase()}`;
const contractFor = (sport, stat) => CONTRACTS[contractKey(sport, stat)] || null;
const inCertifiedRawBand = (bands, p) =>
Array.isArray(bands) && bands.some(([lo, hi]) => p >= lo && p < hi);
/**
* Resolve the served probability for one Read.
*
* `raw_model_probability` is ALWAYS preserved — it is model evidence and is not
* deleted because another estimator answered, or declined to.
*
* @param {object} read {sport, stat, model_version, p_win}
* @param {object} deps {estimate(p) -> number|null} the fitted estimator
*/
function resolve(read = {}, deps = {}) {
const raw = knownNumber(read.p_win);
const contract = contractFor(read.sport, read.stat);
const base = {
raw_model_probability: raw,
served_probability: null,
probability_state: null,
estimator_type: contract ? contract.estimator_type : null,
estimator_version: contract ? contract.estimator_version : null,
certification_version: contract ? contract.certification_version : null,
model_version: read.model_version ?? null,
contract_model_version: contract ? contract.model_version : null,
reason: null,
};
if (!contract) {
return { ...base, probability_state: STATE.UNSUPPORTED, reason: 'no certified contract for this sport/stat' };
}
if (read.model_version !== contract.model_version) {
// A calibration artifact is only meaningful against the forecast it was
// fitted to. A different model era is a different forecaster.
return { ...base, probability_state: STATE.VERSION_MISMATCH, reason: 'model version differs from the certified era' };
}
if (raw === null || raw < 0 || raw > 1) {
return { ...base, probability_state: STATE.INVALID, reason: 'raw probability absent or out of range' };
}
if (!inCertifiedRawBand(contract.certified_bands, raw)) {
// NO RAW FALLBACK. This is the whole point of the module.
return { ...base, probability_state: STATE.UNCERTIFIED, reason: 'raw value lies outside certified estimator support' };
}
const estimate = typeof deps.estimate === 'function' ? knownNumber(deps.estimate(raw)) : null;
if (estimate === null || estimate < 0 || estimate > 1) {
// The estimator was supposed to answer here and did not. Refuse; never
// substitute raw, which is what made the previous contract untruthful.
return { ...base, probability_state: STATE.UNCERTIFIED, reason: 'estimator produced no value inside its own support' };
}
return {
...base,
served_probability: Math.round(estimate * 1000) / 1000,
probability_state: contract.state_when_served,
reason: null,
};
}
const isCertified = (res) => !!res && CERTIFIED_STATES.includes(res.probability_state)
&& knownNumber(res.served_probability) !== null;
/**
* THE ACTIONABILITY LAW. Probability-derived claims require a certified served
* probability. They are NOT computed from raw when the contract declines —
* doing that in secret is the same lie as displaying raw, one layer down.
*
* Market data is untouched: odds are a fact about the book, not a model output.
*/
function derivedClaims(res, odds, deps = {}) {
const unavailable = (reason) => ({
available: false, ev_pct: null, kelly: null, value: null, reason,
});
if (!isCertified(res)) return unavailable(res ? res.probability_state : 'no resolution');
const evPct = deps.evPct || require('../../utils/devig').evPct;
const isValue = deps.isValue || require('../../config/valueEngine').isValue;
const quarterKelly = deps.quarterKelly || require('../../utils/kelly').quarterKelly;
const p = res.served_probability;
const ev = evPct(p, odds);
return {
available: true,
ev_pct: ev,
kelly: quarterKelly(p, odds),
value: isValue(odds, ev),
reason: null,
computed_from: 'served_probability',
};
}
/**
* What the surface may say. An uncertified Read keeps its grade and its market
* data; what it loses is the claim to an exact number.
*/
function confidenceDisplay(res) {
if (isCertified(res)) {
return {
exact_probability: res.served_probability,
exact_pct: Math.round(res.served_probability * 1000) / 10,
state: res.probability_state,
calibrated: res.probability_state === STATE.CERTIFIED_CALIBRATED,
};
}
return {
exact_probability: null,
exact_pct: null,
state: res ? res.probability_state : STATE.UNSUPPORTED,
calibrated: false,
label: 'Confidence not calibrated',
};
}
module.exports = {
STATE, CERTIFIED_STATES, ESTIMATOR, CONTRACTS, MLB_HITS,
contractFor, inCertifiedRawBand, resolve, isCertified, derivedClaims, confidenceDisplay,
};
src/services/model/probabilityContractService.js
+47
View File
@@ -0,0 +1,47 @@
'use strict';
/**
* probabilityContractService — fit the certified estimator, point in time.
*
* REUSES the existing fitter and its "fit on settled history strictly before
* today" discipline. What it does NOT reuse is `calibrationService.calibrate()`,
* whose gate is evaluated in CALIBRATED-OUTPUT space and whose else-branch
* serves RAW. Both of those are the blocked contract; only the MAP is taken.
*
* SUPPORT COMES FROM THE CERTIFIED ARTIFACT, NOT FROM TONIGHT'S FIT. The bands
* in `probabilityContract.MLB_HITS` were adjudicated on a three-way split and
* are a fixed property of that adjudication. Letting a nightly refit widen its
* own support is how an estimator certifies itself.
*/
const cal = require('./calibration');
const pc = require('./probabilityContract');
/**
* @returns {null|{estimate, fit_n, fitted_through, contract}} null when there
* is not enough settled history — and null means NOTHING is served, never
* "pass raw through".
*/
async function build(sb, { sport = 'mlb', stat = 'hits', before = null, ...opts } = {}) {
const contract = pc.contractFor(sport, stat);
if (!contract || !sb) return null;
const svc = opts.calibrationService || require('./calibrationService');
const fitted = await svc.fromLedger(sb, { sport, stat, before, ...opts });
if (!fitted || !fitted.map) return null;
return {
contract,
fit_n: fitted.fit_n,
fitted_through: fitted.fitted_through,
cutoff: fitted.cutoff,
/** The estimator, and only the estimator. No gate, no fallback. */
estimate: (p) => cal.applyIsotonic(fitted.map, p),
/** Resolve one grade through the full contract. */
resolve(read) {
return pc.resolve({ ...read, sport, stat }, { estimate: (p) => cal.applyIsotonic(fitted.map, p) });
},
};
}
module.exports = { build };
src/services/retentionService.js
+62
View File
@@ -193,6 +193,12 @@ function rowsFromSides(base, sides, ctx = {}) {
// rows is silently dropped for the whole batch. Filled by
// `mergeChainShadow` after enrichment; never read by anything served.
chain_shadow: null,
// The PROBABILITY CONTRACT shadow — what the certified serving contract
// WOULD serve, beside what was actually served. Declared always, for the
// same first-row-shape reason as chain_shadow. Filled by
// `mergeProbabilityContract`; SHADOW ONLY, read by nothing served.
probability_contract: null,
});
}
return rows;
@@ -825,6 +831,61 @@ function mergeEnrichment(rows, enrichedGrades) {
* keeping, and a fabricated third of a triple would poison the adjudication this
* column exists to enable.
*/
/**
* PROBABILITY CONTRACT SHADOW.
*
* Records, per row: the raw model belief, what the certified contract would
* serve, the state, the estimator identity, and what EV/Kelly/VALUE would be
* under the actionability law. It writes to its own column and mutates nothing
* a user sees — `p_win`, `confidence`, `grade`, `ev_pct`, `value` and
* `takeable` on the row are untouched.
*
* The raw probability is ALWAYS carried, including on refusals: it is model
* evidence, and a row that records only the refusal cannot be re-adjudicated.
*/
function mergeProbabilityContract(rows, contract) {
if (!Array.isArray(rows) || !rows.length) return rows || [];
if (!contract || typeof contract.resolve !== 'function') return rows;
const pc = require('./model/probabilityContract');
return rows.map((r) => {
let res;
try {
res = contract.resolve({ model_version: r.model_version, p_win: numOrNull(r.p_win) });
} catch { return r; }
if (!res) return r;
let derived;
// The SIDE'S price. A retention row carries book_odds plus both sides'
// prices; using the wrong side would price the opposite bet.
const sideOdds = r.book_odds != null ? r.book_odds
: (String(r.side) === 'under' ? r.under_odds : r.over_odds);
try { derived = pc.derivedClaims(res, sideOdds); } catch { derived = null; }
return {
...r,
probability_contract: {
raw_model_probability: res.raw_model_probability,
served_probability: res.served_probability,
probability_state: res.probability_state,
estimator_type: res.estimator_type,
estimator_version: res.estimator_version,
certification_version: res.certification_version,
model_version: res.model_version,
reason: res.reason,
fitted_through: contract.fitted_through || null,
fit_n: contract.fit_n || null,
derived: derived ? {
available: derived.available,
ev_pct: derived.ev_pct,
kelly_pct: derived.kelly ? derived.kelly.pct : null,
value: derived.value,
} : null,
// SHADOW. Nothing here has been served to anyone.
servable: false,
},
};
});
}
function mergeChainShadow(rows, shadow) {
if (!Array.isArray(rows) || !rows.length) return rows || [];
const byKey = shadow && shadow.byKey;
@@ -1183,6 +1244,7 @@ module.exports = {
createCollector,
mergeEnrichment,
mergeChainShadow,
mergeProbabilityContract,
persist,
attachLineage,
commitPublication,
src/services/snapshotService.js
+27
View File
@@ -852,6 +852,26 @@ async function runSnapshot(sport, opts = {}) {
// read_revision_id is deliberately NOT resolved here: the upsert does not
// return row ids, and issuing a second query in the hot path to obtain one
// would be a real cost for a column nothing reads yet.
// ── PROBABILITY CONTRACT SHADOW (default OFF) ──────────────────────────
// Builds the certified estimator once per snapshot so the shadow costs one
// ledger fit, not one per row. OFF unless PROBABILITY_CONTRACT_SHADOW=1:
// this releases the support with activation off, which is the required order.
// A failure here must never cost the snapshot — it is a measurement layer.
let probContract = null;
if (String(process.env.PROBABILITY_CONTRACT_SHADOW || '') === '1' && sp === 'mlb') {
try {
const pcs = deps.probabilityContractService || require('./model/probabilityContractService');
const sbc = deps.supabase || require('../utils/supabase').getSupabaseServiceClient();
probContract = sbc ? await pcs.build(sbc, { sport: 'mlb', stat: 'hits' }) : null;
console.log(probContract
? `[probability-contract] shadow armed — isotonic, fit n=${probContract.fit_n} through ${probContract.fitted_through}, certified raw [0.50,0.80)`
: '[probability-contract] shadow armed but NO estimator (thin history) — nothing would be served');
} catch (e) {
probContract = null;
console.warn('[probability-contract] shadow build failed (snapshot continues):', e.message);
}
}
let lineageIndex = null;
let persistedRows = null;
const persistRetention = async (enrichedGrades, chainShadow = null) => {
@@ -868,6 +888,13 @@ async function runSnapshot(sport, opts = {}) {
console.warn('[chain-shadow] merge skipped (retention continues):', e.message);
}
}
// PROBABILITY CONTRACT SHADOW — its own guard, for the same reason: the
// retention write is the record and a measurement layer may not cost it.
if (probContract && retention.mergeProbabilityContract) {
try { rows = retention.mergeProbabilityContract(rows, probContract); } catch (e) {
console.warn('[probability-contract] merge skipped (retention continues):', e.message);
}
}
const r = await retention.persist(rows);
retentionRows = r.written || 0;
// Held for the publication commit, which happens only after the