A digest names the curve; it does not vouch for the procedure that makes the next one

Artifact identity was the previous tranche's answer. It is not certification. `knot_digest` says WHICH mapping ran and nothing about whether tomorrow's refit deserves the same trust — that one would get its own digest and be equally "identified" while fitted on anything at all. So the object being certified is named: VYNDR certifies a PROCEDURE, not a frozen curve. A frozen curve goes stale against a live model and has to be replaced by hand on no schedule; "fit past, apply forward" is procedural by construction. `fitPolicy.POLICY_V1` declares it — data selection, horizon, algorithm and version, minimum rows, model-version restriction, sport, stat, and a support contract that a refit may NOT widen. Each artifact still carries its own digest. `fitPolicy.validate` is the Step-22 gate: an artifact does not become servable because the algorithm ran. It refuses a widened support, a wrong era, a wrong estimator, a thin fit, a missing identity or a missing training cutoff, and `servable` is false whenever any violation stands, with no override argument. The statistical bars stay where they already live in calibrationRegistry — this is not a second governance system. MEASURED, AND THE REASON LIVE SERVING STAYS BLOCKED: production does not match the declaration. `loadSettledRows` applies no model_version filter, so at fit_as_of 2026-09-02 the fit drew 6,084 rows from 9,361 settled — all 3,292 from the superseded engine1@2026-07-20 plus 2,792 current-era. 54.1% of the served map is fitted on a forecaster it was never certified for, while the artifact declares the current era. That is a provenance contradiction, not a performance claim: era-filtered scores 0.24065 against pooled 0.24068 on 1,120 out-of-sample rows and both intervals span zero. It is blocked because nothing prevents the next era change from repeating it, and because the freshness lag grows. `era_restricted` is answered STRUCTURALLY, not by an extra read — the query is in this service and applies no filter, so the artifact records ERA_NOT_RESTRICTED rather than claiming a restriction that did not hold. An unverified restriction is recorded as a violation, because "we did not check" is exactly the state production is in. The violation does NOT distort the shadow. Flipping every row to UNCERTIFIED would make the shadow measure the violation instead of the contract, so the policy state rides beside the resolution and a test asserts the shadow still reads CERTIFIED_CALIBRATED at 0.65 and UNCERTIFIED at 0.91. Nothing serves. CALIBRATION_DEPLOYED still []. Shadow still OFF (the production variable remains absent — the probe reads configuration_source "default"). Suite 402/402, 5,607 passed, 4 skipped. Teeth 10/10 + 23/23. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CQJeAG8vcDoL5zkiaJyVb8
2026-09-02 23:43:31 -04:00
parent 556d186ff1
commit 5cad851922
4 changed files with 308 additions and 4 deletions
@@ -0,0 +1,111 @@
+'use strict';
+
+/**
+ * ARTIFACT IDENTITY IS NOT FIT-POLICY CERTIFICATION.
+ *
+ * A digest says WHICH mapping ran. It says nothing about whether the procedure
+ * that produced it should be trusted to produce tomorrow's.
+ */
+const fp = require('../../src/services/model/fitPolicy');
+const svc = require('../../src/services/model/probabilityContractService');
+const cal = require('../../src/services/model/calibration');
+
+const ERA = 'engine1@2026-08-07-fullwindow';
+const artifact = (over = {}) => ({
+  estimator_type: 'isotonic', model_version: ERA, fit_n: 6084,
+  training_cutoff: '2026-08-21', knot_digest: 'd9d571d728ba76de',
+  served_curve_digest: '9d731cd88ba073d7',
+  certified_bands: [[0.50, 0.80]], ...over,
+});
+
+describe('the declared procedure', () => {
+  it('pins every decision that would otherwise be implicit', () => {
+    // If any of these change, they change DELIBERATELY and this test says so.
+    expect(fp.POLICY_V1.policy_version).toBe('mlb-hits-isotonic-refit@v1');
+    expect(fp.POLICY_V1.sport).toBe('mlb');
+    expect(fp.POLICY_V1.stat).toBe('hits');
+    expect(fp.POLICY_V1.model_version).toBe(ERA);
+    expect(fp.POLICY_V1.fit_algorithm).toBe('isotonic-pav');
+    expect(fp.POLICY_V1.min_fit_rows).toBe(200);
+    expect(fp.POLICY_V1.data_selection.horizon).toBe('game_date < fit_as_of');
+    expect(fp.POLICY_V1.support_contract.certified_bands).toEqual([[0.50, 0.80]]);
+    expect(fp.POLICY_V1.support_contract.may_be_widened_by_refit).toBe(false);
+  });
+
+  it('a refit may never widen the region it is trusted in', () => {
+    const r = fp.validate(artifact({ certified_bands: [[0.50, 0.95]] }),
+      { era_counts: { [ERA]: 6084 } });
+    expect(r.valid).toBe(false);
+    expect(r.violations).toContain(fp.VIOLATION.SUPPORT_WIDENED);
+    expect(r.servable).toBe(false);
+  });
+});
+
+describe('the minimum automatic gates', () => {
+  it('passes only an era-restricted fit', () => {
+    const r = fp.validate(artifact(), { era_counts: { [ERA]: 6084 } });
+    expect(r.valid).toBe(true);
+    expect(r.servable).toBe(true);
+  });
+
+  it('REFUSES the production era mix — the measured state today', () => {
+    const r = fp.validate(artifact(), { era_counts: { [ERA]: 2792, 'engine1@2026-07-20': 3292 } });
+    expect(r.valid).toBe(false);
+    expect(r.violations).toContain(fp.VIOLATION.ERA_NOT_RESTRICTED);
+    expect(r.servable).toBe(false);
+  });
+
+  it('"we did not check" is a violation, not a pass', () => {
+    // An unverified restriction is exactly the state production is in.
+    const r = fp.validate(artifact(), {});
+    expect(r.valid).toBe(false);
+    expect(r.violations).toContain(fp.VIOLATION.ERA_NOT_RESTRICTED);
+  });
+
+  it('catches a missing identity, a wrong era, a wrong estimator and a thin fit', () => {
+    const ev = { era_counts: { [ERA]: 6084 } };
+    expect(fp.validate(artifact({ knot_digest: null }), ev).violations).toContain(fp.VIOLATION.NO_ARTIFACT_IDENTITY);
+    expect(fp.validate(artifact({ training_cutoff: null }), ev).violations).toContain(fp.VIOLATION.NO_TRAINING_CUTOFF);
+    expect(fp.validate(artifact({ model_version: 'engine1@2026-07-20' }), ev).violations).toContain(fp.VIOLATION.ERA_MISMATCH);
+    expect(fp.validate(artifact({ estimator_type: 'low_param' }), ev).violations).toContain(fp.VIOLATION.ESTIMATOR_MISMATCH);
+    expect(fp.validate(artifact({ fit_n: 12 }), ev).violations).toContain(fp.VIOLATION.INSUFFICIENT_FIT_ROWS);
+  });
+
+  it('a null artifact is refused, never treated as vacuously valid', () => {
+    expect(fp.validate(null).valid).toBe(false);
+    expect(fp.validate(null).servable).toBe(false);
+  });
+
+  it('servable is never true while any violation stands — there is no override', () => {
+    for (const bad of [{ fit_n: 1 }, { model_version: 'x' }, { estimator_type: 'y' }, { knot_digest: null }]) {
+      expect(fp.validate(artifact(bad), { era_counts: { [ERA]: 6084 } }).servable).toBe(false);
+    }
+    expect(fp.validate.length).toBeLessThanOrEqual(3);   // no force/override argument
+  });
+});
+
+describe('the built artifact carries its policy state', () => {
+  const map = cal.fitIsotonic(Array.from({ length: 900 }, (_, i) => {
+    const p = Math.round((0.35 + (i % 60) / 100) * 1000) / 1000;
+    return { p, won: ((i * 2654435761) % 1000) / 1000 < (0.5 + 0.45 * (p - 0.5)) ? 1 : 0, date: `d${i % 14}` };
+  }), { minTotal: 200 });
+  const build = () => svc.build({}, { calibrationService: { fromLedger: async () => ({
+    map, fit_n: 900, fitted_through: '2026-08-21', cutoff: '2026-09-02' }) } });
+
+  it('records the policy version, the violation and servable:false', async () => {
+    const b = await build();
+    expect(b.artifact.fit_policy_version).toBe('mlb-hits-isotonic-refit@v1');
+    expect(b.artifact.fit_policy_valid).toBe(false);
+    expect(b.artifact.fit_policy_violations).toContain(fp.VIOLATION.ERA_NOT_RESTRICTED);
+    expect(b.artifact.servable).toBe(false);
+  });
+
+  it('the policy violation does NOT distort what the shadow measures', async () => {
+    // The shadow's job is to measure the certified contract on real rows. If a
+    // policy violation flipped every row to UNCERTIFIED the shadow would
+    // measure the violation instead, and the receipt would be worthless.
+    const b = await build();
+    expect(b.resolve({ model_version: ERA, p_win: 0.65 }).probability_state).toBe('CERTIFIED_CALIBRATED');
+    expect(b.resolve({ model_version: ERA, p_win: 0.91 }).probability_state).toBe('UNCERTIFIED');
+  });
+});