openvibers/openvibe.bench Commits

Measure provider fidelity on a fixed objective set (B5) (#181)

OP OpenVibers committed 3 days ago · 45c81cbb6f · 13 files, +892 −9

* Measure provider fidelity on a fixed objective set (B5)

Add the Stage B5 fidelity measurement: a fixed, public objective set per open-weight model run through every enabled provider that serves it. A dry run lists every call and its own-page price estimate against the owner's daily cap; a bounded pass sends each prompt through OpenVibe.AI's normal generate path, pinned to the exact offer, with a durable per-prompt, per-week idempotency key. Results are a pass rate with n and a Wilson interval, stored without the answers, and published at GET /api/v1/fidelity for exchange offers. Off by default; the weekly job is due-based and capped.

* Merge remote-tracking branch 'origin/main' into feature/bench-b5-fidelity

* Read a fidelity date the same from PGlite and PostgreSQL

node-postgres returns a date column as a JS Date while PGlite returns YYYY-MM-DD. Normalize both so the public measured_on is the day, not a Date string.

Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex d90eba2..19a47c2 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -2,6 +2,14 @@
## Unreleased
++- **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, the
++ score of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each,
++ no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the
++ dry run: every call, the own-page price estimate and whether it fits the owner's daily cap. `POST
++ /api/v1/fidelity/run` runs one bounded pass; every call goes through OpenVibe.AI's normal generate path, pinned to
++ the exact offer, with a durable idempotency key per prompt and week. Off by default (`BENCH_FIDELITY=1`), daily
++ capped, and a weekly due-based job when on. The objective set is public at `GET /api/v1/fidelity/objective-set`;
++ answers are never stored, only which item passed.
- **The 18+ board (`chat.willing`).** A new adult kind and `GET /api/v1/boards/chat.willing`, marked `adult: true` on the board and every row. It reads OpenVibe.AI's measured refusal and hedge rates (n and Wilson intervals) on the published 60-prompt lawful adult set, plus the abliteration tax for each declared base/abliterated pair: the same objective set on both, with the score difference and its Newcombe interval. A non-willingness type on the adult kind is refused; when AI cannot answer, the board says the measurements are unavailable and invents no rows.
- **Every Eval Ever (B2).** Bench imports the public EEE datastore (`github.com/evaleval/every_eval_ever`,
`evaleval/EEE_datastore`) as *external* results: the model (joined by Report's rule — maker prefix and a date or
ddiff --git a/docs/api.md b/docs/api.md
iindex 3d914ee..ee738e9 100644
---- a/docs/api.md
++++ b/docs/api.md
@@@ -245,6 +245,22 @@ The open dataset (`/api/v1/datasets/:kindId.jsonl`) also emits `type: "category"
`type: "holdout"` lines, and its rating lines carry the rank interval. The formulas, the 0.5/0.5 weights and the
30-game floor are in [index.md](index.md).
++## Provider fidelity (stage B5)
++
++| Route | Who | What |
++|---|---|---|
++| `GET /api/v1/fidelity` | anyone | The latest measurement of every offer: `score` (pass rate), `n`, `interval` (95% Wilson), `compared_to`, `measured_at`, `prompt_set`. Cached 60 seconds. |
++| `GET /api/v1/fidelity/objective-set` | anyone | The fixed public set (`fidelity-2026-11-09.v1`): every prompt and the deterministic check it is graded by. |
++| `POST /api/v1/fidelity/plan` | operator token | The dry run: each offer, its calls, the own-page price estimate and whether it fits the daily cap. Spends nothing. |
++| `POST /api/v1/fidelity/run` | operator token | One bounded pass. Refused with `fidelity.disabled` until `BENCH_FIDELITY=1`; capped by `BENCH_FIDELITY_DAILY_CAP_USD` and `BENCH_FIDELITY_MAX_CALLS`. |
++| `GET /api/v1/fidelity/status` | operator token | Whether the switch is on and the last pass's state. |
++
++The measurement asks every enabled provider of an open-weight model the same ten objective items, answers only, and
++grades each answer with a published exact/JSON/regex check. A provider that quantizes, truncates the context or
++serves another checkpoint answers fewer correctly; that is the number beside its price. A caller never sees a rank
++or a winner. Scores travel to OpenVibe.AI with `GET /api/v1/exchange/market` / `catalog` as each offer's `fidelity`
++field (null when unmeasured). The dry run and the cap are the owner's switch: fidelity is off by default.
++
## The older API
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
ddiff --git a/migrations/0016_fidelity.sql b/migrations/0016_fidelity.sql
nnew file mode 100644
iindex 0000000..6cecef2
---- /dev/null
++++ b/migrations/0016_fidelity.sql
@@@ -0,0 +1,41 @@
++-- phase: expand
++-- Provider fidelity (plan 2026-11-09, stage B5).
++--
++-- One row per measured offer and pass: the same open-weight model, served by one provider, asked the
++-- fixed objective set. The score is passes / answered items with its Wilson interval; the answers
++-- themselves are not kept, only which item passed. A provider that quantizes, truncates the context or
++-- serves a different checkpoint scores lower on the same set, and the public read shows the difference
++-- rather than a rank.
++--
++-- measured_at is when the pass ran; cost_usd is what the calls reported, so the daily cap can sum the
++-- day's spend without a second ledger. results keeps the per-item verdicts (id, passed, error) for audit.
++CREATE TABLE IF NOT EXISTS bench_fidelity_runs (
++ id text COLLATE "C" PRIMARY KEY,
++ objective_set text COLLATE "C" NOT NULL,
++ model text COLLATE "C" NOT NULL, -- the model the provider claims to serve
++ provider text COLLATE "C" NOT NULL,
++ provider_model text COLLATE "C" NOT NULL, -- the exact endpoint id that answered
++ n integer NOT NULL DEFAULT 0, -- answered items (errors are counted apart)
++ passed integer NOT NULL DEFAULT 0,
++ errors integer NOT NULL DEFAULT 0, -- items the provider did not answer
++ score double precision, -- passed / n, null when nothing was answered
++ ci_lower double precision, -- Wilson 95% interval
++ ci_upper double precision,
++ cost_usd double precision NOT NULL DEFAULT 0,
++ measured_at timestamptz NOT NULL,
++ measured_on date NOT NULL,
++ results jsonb NOT NULL DEFAULT '[]'::jsonb,
++ created_at timestamptz NOT NULL DEFAULT now()
++);
++CREATE INDEX IF NOT EXISTS bench_fidelity_latest_idx ON bench_fidelity_runs (provider, provider_model, measured_at DESC);
++CREATE INDEX IF NOT EXISTS bench_fidelity_day_idx ON bench_fidelity_runs (measured_at);
++
++-- The weekly job is due-based: its last pass survives restarts and deploys.
++CREATE TABLE IF NOT EXISTS bench_fidelity_state (
++ id text COLLATE "C" PRIMARY KEY CHECK (id = 'fidelity'),
++ last_run_at timestamptz,
++ last_status text COLLATE "C",
++ last_error text COLLATE "C",
++ updated_at timestamptz NOT NULL DEFAULT now()
++);
++INSERT INTO bench_fidelity_state (id) VALUES ('fidelity') ON CONFLICT (id) DO NOTHING;
ddiff --git a/server/app.js b/server/app.js
iindex a0b024f..911be0c 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -92,6 +92,9 @@ async function createApp(opts = {}) {
// B2: the external EEE records and the exporter of Bench's own runs, plus the weekly import job.
const eee = require('./bench/eee').createEee({ s, registry, fetchImpl, log, config });
const eeeSync = require('./bench/eee-sync').createEeeSync({ eee, config, log });
++ // B5: the provider-fidelity measurements and their weekly pass (off unless BENCH_FIDELITY is on).
++ const fidelity = require('./bench/fidelity').createFidelity({ s, ai, config, clock: { now: opts.now || (() => Date.now()) }, log });
++ const fidelityJob = require('./bench/fidelity-job').createFidelityJob({ fidelity, config, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
// The signed-out voter (the cookie, the relay header and the daily caps) and the daily rounds job.
const anon = createAnon({ s, config, log });
@@@ -114,7 +117,7 @@ async function createApp(opts = {}) {
const ctx = {
config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,
anon, daily, variantEvidence, evidenceConfig, sharing,
-- imports, mirror, eee, eeeSync,
++ imports, mirror, eee, eeeSync, fidelity, fidelityJob,
random: opts.random || Math.random,
};
@@@ -258,6 +261,7 @@ async function createApp(opts = {}) {
}
// The EEE routes come before the kind dataset route: /datasets/eee.jsonl would otherwise be read as kind 'eee'.
app.use(require('./http/eee').createEeeRoutes(ctx));
++ app.use(require('./http/fidelity').createFidelityRoutes(ctx));
app.use('/api/v1', createApi(ctx));
app.use('/api/v1', createRegistryApi(ctx));
app.use('/api/v1', createVariantEvidenceApi(ctx));
ddiff --git a/server/bench/ai.js b/server/bench/ai.js
iindex 705ba58..3165fa2 100644
---- a/server/bench/ai.js
++++ b/server/bench/ai.js
@@@ -202,17 +202,18 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
return { run: r, ms: msOf(r, Date.now() - startedAt) };
}
-- async function generate({ prompt, kind, slot, roundId, index }) {
++ async function generate({ prompt, kind, slot, roundId, index, system = null, maxTokens = null, temperature = 0.7, idempotencyKey = null, target = null }) {
if (!enabled) throw new BenchAiError(503, 'ai.not_configured', 'OpenVibe.AI is not configured for Bench.');
const pinned = slot && slot.provider && slot.provider !== 'ai' ? { provider: slot.provider, model: slot.model || null } : null;
const body = {
prompt,
-- system: systemPrompt(kind),
-- max_tokens: maxTokensFor(kind),
-- temperature: 0.7,
++ system: system || systemPrompt(kind),
++ max_tokens: maxTokens == null ? maxTokensFor(kind) : maxTokens,
++ temperature,
options: { cache: false },
...(pinned ? { route: pinned } : {}),
-- target: { service: 'bench', type: 'bench_output', id: `${roundId}_${index}`.slice(0, 128) },
++ ...(idempotencyKey ? { idempotency_key: idempotencyKey } : {}),
++ target: target || { service: 'bench', type: 'bench_output', id: String(idempotencyKey || `${roundId}_${index}`).slice(0, 128) },
};
const startedAt = Date.now();
const wait = Math.min(ai.waitMs, 60_000);
@@@ -226,6 +227,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
input: { prompt: body.prompt, system: body.system, max_tokens: body.max_tokens, temperature: body.temperature },
options: body.options,
...(body.route ? { route: body.route } : {}),
++ ...(body.idempotency_key ? { idempotency_key: body.idempotency_key } : {}),
target: body.target,
});
}
@@@ -264,6 +266,12 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
};
}
++ /** The market feed: every own-page model and provider offer, with its price and open-weights flag. */
++ async function market({ openWeights = true, limit = 500 } = {}) {
++ const query = `?limit=${Math.max(1, Math.min(500, Number(limit) || 500))}${openWeights ? '&open_weights=true' : ''}&kind=chat`;
++ return call('GET', `/api/v1/exchange/market${query}`);
++ }
++
/** The 18+ board's measured willingness rates, from OpenVibe.AI's public read API. */
async function willingness() {
return call('GET', '/api/v1/exchange/willingness');
@@@ -274,7 +282,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
return call('GET', '/api/v1/exchange/abliteration-tax');
}
-- return { enabled, slots, slotPlan, offeredModels, rankOffers, generate, systemPrompt, maxTokensFor, willingness, abliterationTax };
++ return { enabled, slots, slotPlan, offeredModels, rankOffers, generate, systemPrompt, maxTokensFor, market, willingness, abliterationTax };
}
module.exports = { createBenchAi, BenchAiError };
ddiff --git a/server/bench/fidelity-job.js b/server/bench/fidelity-job.js
nnew file mode 100644
iindex 0000000..bdefcb0
---- /dev/null
++++ b/server/bench/fidelity-job.js
@@@ -0,0 +1,44 @@
++'use strict';
++
++/**
++ * The weekly provider-fidelity pass (stage B5). Off by default: the owner turns it on with BENCH_FIDELITY=1.
++ * The pass is due-based (the last pass is a row in the database), so the several deploys Bench gets in a
++ * week do not reset its clock. A pass that is already running is never started twice.
++ */
++function createFidelityJob({ fidelity, config, log = console }) {
++ let timer = null;
++ let running = false;
++
++ async function tick() {
++ if (!config.bench.fidelity.enabled) return null;
++ if (config.drill && !config.bench.fidelity.enabledInDrill) return null;
++ if (running) return null;
++ running = true;
++ try {
++ if (!(await fidelity.due())) return null;
++ return await fidelity.run({ trigger: 'weekly' });
++ } catch (err) {
++ log.warn('[OpenVibe.Bench] fidelity job:', (err && (err.detail || err.message)) || err);
++ return null;
++ } finally {
++ running = false;
++ }
++ }
++
++ function start() {
++ if (timer || !config.bench.fidelity.enabled) return null;
++ const run = () => tick().catch((err) => log.warn('[OpenVibe.Bench] fidelity job:', (err && err.message) || err));
++ timer = setInterval(run, config.bench.fidelity.tickMs);
++ if (timer.unref) timer.unref();
++ return timer;
++ }
++
++ function stop() {
++ if (timer) clearInterval(timer);
++ timer = null;
++ }
++
++ return { tick, start, stop };
++}
++
++module.exports = { createFidelityJob };
ddiff --git a/server/bench/fidelity-set.js b/server/bench/fidelity-set.js
nnew file mode 100644
iindex 0000000..83d315d
---- /dev/null
++++ b/server/bench/fidelity-set.js
@@@ -0,0 +1,176 @@
++'use strict';
++
++/**
++ * The fixed objective set used to measure provider fidelity (plan 2026-11-09, stage B5).
++ *
++ * Provider fidelity asks a different question from the pairwise boards: the same open-weight model is
++ * served by several providers, and the question is whether each one answers the *model's* task as well
++ * as the others. So the set is small, fixed, public and objective: every item has one answer a
++ * deterministic check can recognise, with no LLM judge and no human vote. A provider that quantizes the
++ * weights, changes the chat template, truncates the context or substitutes a different checkpoint scores
++ * lower on the same items, and the difference is what the fidelity field publishes.
++ *
++ * The version is part of every result and every idempotency key: changing the set changes the version,
++ * so two scores are never compared across sets. A score is a pass rate with a Wilson interval, not a
++ * ranking: the public read never names a winner.
++ */
++
++const OBJECTIVE_SET = 'fidelity-2026-11-09.v1';
++
++/** The system line every item is measured with: the same instructions for every provider. */
++const SYSTEM = [
++ 'You are answering a fixed objective benchmark.',
++ 'Follow the requested format exactly and return only the answer, with no explanation or extra text.',
++].join('\n');
++
++const ITEMS = Object.freeze([
++ {
++ id: 'arithmetic',
++ prompt: 'What is 37 multiplied by 24? Answer with the number only.',
++ max_tokens: 16,
++ check: { type: 'equals', value: '888' },
++ },
++ {
++ id: 'extraction',
++ prompt: 'Read this line: "region: fra1; status: ok". Answer with the region code only.',
++ max_tokens: 16,
++ check: { type: 'equals', value: 'fra1' },
++ },
++ {
++ id: 'json',
++ prompt: 'Return exactly this JSON object and nothing else: {"ok": true, "n": 3}',
++ max_tokens: 40,
++ check: { type: 'json', value: { ok: true, n: 3 } },
++ },
++ {
++ id: 'repeat',
++ prompt: 'Write the word aurora exactly four times in a row, separated by single spaces. Answer with the four words only.',
++ max_tokens: 24,
++ check: { type: 'equals', value: 'aurora aurora aurora aurora' },
++ },
++ {
++ id: 'reverse',
++ prompt: 'Spell the word "stressed" backwards. Answer with the reversed word only.',
++ max_tokens: 16,
++ check: { type: 'equals', value: 'desserts' },
++ },
++ {
++ id: 'count',
++ prompt: 'How many times does the letter s appear in the word "mississippi"? Answer with the number only.',
++ max_tokens: 16,
++ check: { type: 'equals', value: '4' },
++ },
++ {
++ id: 'symbol',
++ prompt: 'What is the chemical symbol for gold? Answer with the symbol only.',
++ max_tokens: 16,
++ check: { type: 'equals', value: 'Au' },
++ },
++ {
++ id: 'next-day',
++ prompt: 'If today is Wednesday, what day will it be three days from now? Answer with the day only.',
++ max_tokens: 16,
++ check: { type: 'equals', value: 'Saturday' },
++ },
++ {
++ id: 'long-context',
++ prompt: [
++ 'Read the note below to its last line, then answer the question.',
++ '',
++ ...Array.from({ length: 40 }, (_, i) => `Note line ${i + 1}: the archive keeps its records in order and never changes a line.`),
++ 'The access code is ORBIT-71.',
++ '',
++ 'Question: what is the access code? Answer with the code only.',
++ ].join('\n'),
++ max_tokens: 16,
++ check: { type: 'equals', value: 'ORBIT-71' },
++ },
++ {
++ id: 'format',
++ prompt: 'Reply with exactly the word READY and nothing else.',
++ max_tokens: 12,
++ check: { type: 'equals', value: 'READY' },
++ },
++]);
++
++/** The answer without the wrappers models like to add: a code fence, quotes, a final full stop. */
++function stripFence(text) {
++ const s = String(text == null ? '' : text).trim();
++ const fenced = /^```[a-z0-9_-]*\s*\n?([\s\S]*?)\n?```$/i.exec(s);
++ return fenced ? fenced[1] : s;
++}
++
++function normalize(text) {
++ return stripFence(text)
++ .trim()
++ .replace(/^["'`]+|["'`]+$/g, '')
++ .replace(/[.!]\s*$/, '')
++ .replace(/\s+/g, ' ')
++ .trim()
++ .toLowerCase();
++}
++
++function parseJson(text) {
++ const s = stripFence(text).trim();
++ const start = s.indexOf('{');
++ const end = s.lastIndexOf('}');
++ if (start < 0 || end <= start) return null;
++ try { return JSON.parse(s.slice(start, end + 1)); } catch { return null; }
++}
++
++const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
++
++/**
++ * One item's verdict. `passed` is the only thing stored; `reason` says what the check compared, so an
++ * operator can see a template change without reading the answer back.
++ */
++function check(item, answer) {
++ const got = String(answer == null ? '' : answer);
++ const c = item.check || {};
++ if (c.type === 'equals') {
++ const passed = normalize(got) === normalize(c.value);
++ return { passed, reason: passed ? 'exact' : 'not the expected answer' };
++ }
++ if (c.type === 'one_of') {
++ const n = normalize(got);
++ const passed = (c.values || []).some((v) => normalize(v) === n);
++ return { passed, reason: passed ? 'one of the expected answers' : 'not one of the expected answers' };
++ }
++ if (c.type === 'contains_all') {
++ const n = normalize(got);
++ const missing = (c.values || []).filter((v) => !n.includes(normalize(v)));
++ return { passed: missing.length === 0, reason: missing.length ? `missing ${missing.join(', ')}` : 'all parts present' };
++ }
++ if (c.type === 'json') {
++ const parsed = parseJson(got);
++ const passed = parsed != null && same(parsed, c.value);
++ return { passed, reason: passed ? 'the expected object' : 'not the expected JSON' };
++ }
++ if (c.type === 'regex') {
++ const passed = new RegExp(String(c.pattern), String(c.flags || 'i')).test(got);
++ return { passed, reason: passed ? 'matched' : 'did not match' };
++ }
++ return { passed: false, reason: 'unknown check' };
++}
++
++/** The input side of a cost estimate, from the text itself (a stable convention, not a billing number). */
++function promptTokens(item) {
++ return Math.max(1, Math.ceil((SYSTEM.length + String(item.prompt || '').length) / 4));
++}
++
++/** The public view of the set: what was asked, never the model's answer. */
++function publicSet() {
++ return {
++ objective_set: OBJECTIVE_SET,
++ items: ITEMS.length,
++ note: 'A fixed, public objective set: each item has one answer a deterministic check recognises. The same set is run through every provider that serves an open-weight model.',
++ prompts: ITEMS.map((item) => ({
++ id: item.id,
++ prompt: item.prompt,
++ max_tokens: item.max_tokens,
++ check: item.check,
++ })),
++ };
++}
++
++module.exports = { OBJECTIVE_SET, SYSTEM, ITEMS, normalize, check, promptTokens, publicSet };
ddiff --git a/server/bench/fidelity.js b/server/bench/fidelity.js
nnew file mode 100644
iindex 0000000..077ad1f
---- /dev/null
++++ b/server/bench/fidelity.js
@@@ -0,0 +1,309 @@
++'use strict';
++
++/**
++ * Provider fidelity (plan 2026-11-09, stage B5).
++ *
++ * OpenVibe.AI routes; this module measures. For every popular open-weight model the network routes, it
++ * finds every enabled provider that serves that model in AI's public market feed, asks each offer the
++ * fixed objective set, and scores the answers deterministically. A provider that quantizes, truncates
++ * the context or serves another checkpoint answers fewer items correctly; the score is a pass rate with
++ * a Wilson interval and the sample size beside it, never a rank or a bare winner.
++ *
++ * Money: the owner's switch is off by default; a pass has a daily cap in US dollars and a call cap. The
++ * dry run lists every call and what the pricebook says it would cost before anything is spent. Every
++ * call goes through AI's normal generate path (its own reserve/settle accounting), pinned to the exact
++ * provider and model, with a durable idempotency key per prompt and week, so a retried pass is not
++ * charged twice. The answers never leave this service: the table stores only which item passed.
++ */
++const crypto = require('crypto');
++const { OBJECTIVE_SET, SYSTEM, ITEMS, check, promptTokens, publicSet } = require('./fidelity-set');
++
++const ERROR_MAX = 300;
++const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
++const round = (v) => (v == null ? null : Math.round(Number(v) * 1e6) / 1e6);
++// PostgreSQL returns `date` as a JS Date; PGlite returns the string. Both are the same day.
++const dayOf = (v) => (v instanceof Date ? v.toISOString().slice(0, 10) : String(v || '').slice(0, 10));
++
++class FidelityError extends Error {
++ constructor(status, code, detail) {
++ super(detail);
++ this.name = 'FidelityError';
++ this.status = status;
++ this.code = code;
++ this.detail = detail;
++ }
++}
++
++/** The Wilson score interval at 95%: an honest band around a pass rate with few items. */
++function wilson(passed, n, z = 1.96) {
++ if (!Number.isFinite(n) || n <= 0) return { lower: null, upper: null };
++ const p = Math.max(0, Math.min(1, passed / n));
++ const d = 1 + (z * z) / n;
++ const centre = (p + (z * z) / (2 * n)) / d;
++ const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
++ return { lower: Math.max(0, round(centre - half)), upper: Math.min(1, round(centre + half)) };
++}
++
++/** The Monday 00:00 UTC of the week a moment falls in, and the key the idempotency of a pass uses. */
++function weekOf(ms) {
++ const d = new Date(ms);
++ const day = (d.getUTCDay() + 6) % 7;
++ const start = Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate() - day);
++ return { start, key: new Date(start).toISOString().slice(0, 10) };
++}
++
++function startOfUtcDay(ms) {
++ const d = new Date(ms);
++ return Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate());
++}
++
++/** The planned cost of one item on one offer, from the market's own-page price. Null when unknown. */
++function estimateItem(item, price) {
++ const inPrice = num(price && price.in);
++ const outPrice = num(price && price.out);
++ if (inPrice == null || outPrice == null) return null;
++ return round(((promptTokens(item) / 1e6) * inPrice) + ((item.max_tokens / 1e6) * outPrice));
++}
++
++const idempotencyKey = (weekKey, provider, providerModel, itemId) => {
++ const digest = crypto.createHash('sha256')
++ .update(`${OBJECTIVE_SET}\0${weekKey}\0${provider}\0${providerModel}\0${itemId}`, 'utf8')
++ .digest('hex')
++ .slice(0, 40);
++ return `fid_${digest}`;
++};
++
++function keyOf(provider, providerModel) {
++ return `${String(provider || '').toLowerCase()}\0${String(providerModel || '').toLowerCase()}`;
++}
++
++/**
++ * The public read: the latest measurement of every offer, grouped by the model it serves. No row means
++ * nobody measured that offer; the field is absent there, never a zero.
++ */
++function createFidelity({ s, ai, config, clock = { now: () => Date.now() }, log = console } = {}) {
++ const cfg = (config && config.bench && config.bench.fidelity) || {};
++ const dailyCapUsd = Math.max(0, num(cfg.dailyCapUsd) == null ? 0 : num(cfg.dailyCapUsd));
++ const maxModels = Math.max(1, Number(cfg.maxModels) || 10);
++ const maxCalls = Math.max(1, Number(cfg.maxCallsPerRun) || 60);
++ const minAnswered = Math.max(1, Number(cfg.minAnswered) || Math.min(5, ITEMS.length));
++
++ const latest = s.db.prepare(`SELECT DISTINCT ON (provider, provider_model)
++ provider, provider_model, model, n, passed, errors, score, ci_lower, ci_upper, measured_at, measured_on, objective_set
++ FROM bench_fidelity_runs WHERE objective_set = ?
++ ORDER BY provider, provider_model, measured_at DESC, id DESC`);
++ const spent = s.db.prepare('SELECT COALESCE(SUM(cost_usd), 0) AS usd FROM bench_fidelity_runs WHERE measured_at >= ?');
++ const insert = s.db.prepare(`INSERT INTO bench_fidelity_runs
++ (id, objective_set, model, provider, provider_model, n, passed, errors, score, ci_lower, ci_upper, cost_usd, measured_at, measured_on, results)
++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?::timestamptz, ?::date, ?::jsonb)`);
++ const state = s.db.prepare(`UPDATE bench_fidelity_state SET last_run_at = ?::timestamptz, last_status = ?, last_error = ?,
++ updated_at = now() WHERE id = 'fidelity'`);
++ const stateRow = s.db.prepare("SELECT last_run_at, last_status, last_error FROM bench_fidelity_state WHERE id = 'fidelity'");
++
++ async function todaySpent() {
++ const row = await spent.get(new Date(startOfUtcDay(clock.now())).toISOString());
++ return Number(row && row.usd) || 0;
++ }
++
++ /** The public market: every own-page offer, enabled or not. A missing client is a plain 503. */
++ async function market() {
++ if (!ai || typeof ai.market !== 'function') throw new FidelityError(503, 'fidelity.unavailable', 'OpenVibe.AI market readings are not configured for Bench.');
++ let body;
++ try {
++ body = await ai.market({ openWeights: true, limit: 500 });
++ } catch (err) {
++ throw new FidelityError(503, 'fidelity.market_unavailable', `OpenVibe.AI's market feed is unavailable: ${String((err && (err.detail || err.message)) || err).slice(0, ERROR_MAX)}`);
++ }
++ const offers = Array.isArray(body && body.offers) ? body.offers : [];
++ if (!offers.length && body && body.error) throw new FidelityError(503, 'fidelity.market_unavailable', 'OpenVibe.AI returned no market offers.');
++ return offers;
++ }
++
++ /** The offers worth measuring: open weights, routed, chat, a real provider and endpoint id. */
++ function eligible(offers) {
++ const out = [];
++ for (const offer of offers) {
++ if (!offer || typeof offer !== 'object') continue;
++ if (offer.open_weights !== true || offer.routed !== true) continue;
++ if (offer.kind != null && offer.kind !== 'chat') continue;
++ const provider = String(offer.provider || '').trim();
++ const providerModel = String(offer.provider_model || offer.model || '').trim();
++ const model = String(offer.model || providerModel).trim();
++ if (!provider || !providerModel || !model) continue;
++ out.push({ model, provider, provider_model: providerModel, name: offer.name || providerModel, price: offer.price || null });
++ }
++ return out;
++ }
++
++ function groupByModel(offers) {
++ const groups = new Map();
++ for (const offer of offers) {
++ if (!groups.has(offer.model)) groups.set(offer.model, []);
++ groups.get(offer.model).push(offer);
++ }
++ return [...groups.entries()]
++ .map(([model, list]) => ({ model, offers: list.sort((a, b) => a.provider.localeCompare(b.provider) || a.provider_model.localeCompare(b.provider_model)) }))
++ // "Popular" means served by more providers first; the cap keeps a pass inside the owner's call budget.
++ .sort((a, b) => b.offers.length - a.offers.length || a.model.localeCompare(b.model))
++ .slice(0, maxModels);
++ }
++
++ /**
++ * What a pass would do, and what the pricebook says it would cost. Every offer says whether it is
++ * included and, when it is not, the plain reason: no price, a disabled switch, the daily cap or the
++ * call cap. The plan never spends anything.
++ */
++ async function plan() {
++ const available = eligible(await market());
++ const groups = groupByModel(available);
++ const spentUsd = await todaySpent();
++ const remaining = Math.max(0, dailyCapUsd - spentUsd);
++ let calls = 0;
++ let estimated = 0;
++ const offers = [];
++ for (const group of groups) {
++ for (const offer of group.offers) {
++ const estimates = ITEMS.map((item) => estimateItem(item, offer.price));
++ const offerEstimate = estimates.some((v) => v == null) ? null : round(estimates.reduce((a, b) => a + b, 0));
++ const offerCalls = ITEMS.length;
++ let included = true;
++ let reason = null;
++ if (offerEstimate == null) { included = false; reason = 'the market has no own-page price for this offer'; }
++ else if (calls + offerCalls > maxCalls) { included = false; reason = `the pass is capped at ${maxCalls} calls`; }
++ else if (estimated + offerEstimate > remaining) { included = false; reason = `the daily cap leaves $${round(remaining - estimated)}`; }
++ if (included) { calls += offerCalls; estimated = round(estimated + offerEstimate); }
++ offers.push({
++ model: group.model, provider: offer.provider, provider_model: offer.provider_model, name: offer.name,
++ price: offer.price, calls: offerCalls, estimated_usd: offerEstimate, included, reason,
++ });
++ }
++ }
++ return {
++ $schema: 'exchange.fidelity.plan@1',
++ dry_run: true,
++ generated_at: new Date(clock.now()).toISOString(),
++ objective_set: OBJECTIVE_SET,
++ items: ITEMS.length,
++ enabled: Boolean(cfg.enabled),
++ daily: { cap_usd: round(dailyCapUsd), spent_usd: round(spentUsd), remaining_usd: round(remaining) },
++ limits: { max_models: maxModels, max_calls: maxCalls },
++ models: groups.map((g) => ({ model: g.model, offers: g.offers.length })),
++ offers,
++ calls,
++ estimated_usd: estimated,
++ };
++ }
++
++ /** One bounded pass. Default off; returns what it measured, and never starts when nothing fits. */
++ async function run({ trigger = 'manual' } = {}) {
++ if (!cfg.enabled) throw new FidelityError(503, 'fidelity.disabled', 'Provider fidelity is off: the owner turns it on with BENCH_FIDELITY=1.');
++ const before = await plan();
++ const week = weekOf(clock.now());
++ const measuredAt = new Date(clock.now()).toISOString();
++ const measuredOn = measuredAt.slice(0, 10);
++ const ran = [];
++ let calls = 0;
++ let spentUsd = 0;
++ const included = before.offers.filter((offer) => offer.included);
++ for (const offer of included) {
++ if (calls >= maxCalls) break;
++ if (dailyCapUsd > 0 && spentUsd >= dailyCapUsd) break;
++ const results = [];
++ let passed = 0;
++ let answered = 0;
++ let errors = 0;
++ let cost = 0;
++ for (const item of ITEMS) {
++ if (calls >= maxCalls) break;
++ calls += 1;
++ try {
++ if (!ai || typeof ai.generate !== 'function') throw new FidelityError(503, 'fidelity.unavailable', 'AI generation is not configured for Bench.');
++ const out = await ai.generate({
++ prompt: item.prompt,
++ kind: 'text.answer',
++ slot: { provider: offer.provider, model: offer.provider_model },
++ system: SYSTEM,
++ maxTokens: item.max_tokens,
++ temperature: 0,
++ idempotencyKey: idempotencyKey(week.key, offer.provider, offer.provider_model, item.id),
++ });
++ cost += Number(out.cost_usd) || 0;
++ spentUsd += Number(out.cost_usd) || 0;
++ const verdict = check(item, out.html || out.text || '');
++ answered += 1;
++ if (verdict.passed) passed += 1;
++ results.push({ id: item.id, passed: verdict.passed, reason: verdict.reason, ms: out.ms == null ? null : out.ms });
++ } catch (err) {
++ errors += 1;
++ results.push({ id: item.id, error: String((err && (err.detail || err.code || err.message)) || err).slice(0, ERROR_MAX) });
++ }
++ }
++ if (answered < minAnswered) {
++ log.warn(`[OpenVibe.Bench] fidelity: ${offer.provider}/${offer.provider_model} answered ${answered} of ${ITEMS.length}; no score stored`);
++ continue;
++ }
++ const score = round(passed / answered);
++ const interval = wilson(passed, answered);
++ const id = s.newId('fid');
++ await insert.run(id, OBJECTIVE_SET, offer.model, offer.provider, offer.provider_model, answered, passed, errors,
++ score, interval.lower, interval.upper, round(cost), measuredAt, measuredOn, JSON.stringify(results));
++ ran.push({ model: offer.model, provider: offer.provider, provider_model: offer.provider_model, n: answered, passed, errors, score, interval });
++ }
++ await state.run(measuredAt, 'ok', null).catch((err) => log.warn('[OpenVibe.Bench] fidelity state:', (err && err.message) || err));
++ log.log(`[OpenVibe.Bench] fidelity ${trigger}: measured ${ran.length} offer(s), ${calls} call(s), $${round(spentUsd)}`);
++ return { dry_run: false, generated_at: new Date(clock.now()).toISOString(), objective_set: OBJECTIVE_SET, ran, calls, spent_usd: round(spentUsd) };
++ }
++
++ /** The latest score of every offer. This is the shape OpenVibe.AI reads to put fidelity on an offer. */
++ async function read() {
++ const rows = await latest.all(OBJECTIVE_SET);
++ const offers = (rows || []).map((row) => ({
++ model: row.model,
++ provider: row.provider,
++ provider_model: row.provider_model,
++ score: num(row.score),
++ passed: Number(row.passed) || 0,
++ n: Number(row.n) || 0,
++ errors: Number(row.errors) || 0,
++ interval: { lower: num(row.ci_lower), upper: num(row.ci_upper) },
++ compared_to: row.model,
++ prompt_set: row.objective_set,
++ measured_at: s.iso(row.measured_at),
++ measured_on: String(row.measured_on).slice(0, 10),
++ })).sort((a, b) => a.model.localeCompare(b.model) || a.provider.localeCompare(b.provider) || a.provider_model.localeCompare(b.provider_model));
++ const models = new Map();
++ for (const offer of offers) {
++ const m = models.get(offer.model) || { model: offer.model, providers: new Set(), offers: 0, best_score: null };
++ m.providers.add(offer.provider);
++ m.offers += 1;
++ if (offer.score != null && (m.best_score == null || offer.score > m.best_score)) m.best_score = offer.score;
++ models.set(offer.model, m);
++ }
++ return {
++ $schema: 'exchange.fidelity@1',
++ objective_set: OBJECTIVE_SET,
++ items: ITEMS.length,
++ measured_on: offers.length ? offers.map((o) => o.measured_on).sort().slice(-1)[0] : null,
++ offers,
++ models: [...models.values()].map((m) => ({ model: m.model, providers: m.providers.size, offers: m.offers, best_score: m.best_score })),
++ note: offers.length ? null : 'No offer has been measured on this objective set yet.',
++ };
++ }
++
++ /** Whether the weekly pass is due (the state row survives restarts, like the price book's). */
++ async function due() {
++ const interval = Math.max(3600000, Number(cfg.intervalMs) || 7 * 24 * 3600000);
++ const row = await stateRow.get();
++ if (!row || !row.last_run_at) return true;
++ return clock.now() - new Date(row.last_run_at).getTime() >= interval;
++ }
++
++ async function lastState() {
++ const row = await stateRow.get();
++ return row ? { last_run_at: s.iso(row.last_run_at), last_status: row.last_status || null, last_error: row.last_error || null } : null;
++ }
++
++ return { plan, run, read, due, lastState, objectiveSet: publicSet };
++}
++
++module.exports = { createFidelity, FidelityError, wilson, weekOf, estimateItem, idempotencyKey, keyOf, dayOf, OBJECTIVE_SET, ITEMS };
ddiff --git a/server/config.js b/server/config.js
iindex 8df0dc6..6e0dc68 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -146,6 +146,19 @@ function load(env = process.env) {
filesPerRun: Math.max(1, int(env.BENCH_EEE_FILES_PER_RUN, 5000)),
recordsPerRun: Math.max(1, int(env.BENCH_EEE_RECORDS_PER_RUN, 50000)),
},
++ // B5 provider fidelity: one fixed objective set per open-weight model, every provider that
++ // serves it. Off until the owner turns it on; the daily cap is real money, and the dry run
++ // lists every call and its own-page price before anything is spent.
++ fidelity: {
++ enabled: bool(env.BENCH_FIDELITY, false),
++ enabledInDrill: bool(env.BENCH_FIDELITY_IN_DRILL, false),
++ dailyCapUsd: Math.max(0, float(env.BENCH_FIDELITY_DAILY_CAP_USD, 1.00)),
++ maxModels: Math.max(1, int(env.BENCH_FIDELITY_MAX_MODELS, 10)),
++ maxCallsPerRun: Math.max(1, int(env.BENCH_FIDELITY_MAX_CALLS, 60)),
++ minAnswered: Math.max(1, int(env.BENCH_FIDELITY_MIN_ANSWERED, 5)),
++ intervalMs: Math.max(3_600_000, int(env.BENCH_FIDELITY_INTERVAL_MS, 7 * 24 * 60 * 60 * 1000)),
++ tickMs: Math.max(60_000, int(env.BENCH_FIDELITY_TICK_MS, 60 * 60 * 1000)),
++ },
},
};
}
ddiff --git a/server/http/fidelity.js b/server/http/fidelity.js
nnew file mode 100644
iindex 0000000..d45a997
---- /dev/null
++++ b/server/http/fidelity.js
@@@ -0,0 +1,81 @@
++'use strict';
++
++/**
++ * Provider-fidelity routes (stage B5).
++ *
++ * GET /api/v1/fidelity the latest measurement of every offer (public)
++ * GET /api/v1/fidelity/objective-set the fixed objective set (public)
++ * POST /api/v1/fidelity/plan the dry run: every call and its estimated cost (operator)
++ * POST /api/v1/fidelity/run one bounded pass (operator; off unless the owner turned it on)
++ * GET /api/v1/fidelity/status the last pass, when it ran and what it said (operator)
++ *
++ * The read is public because it is the number OpenVibe.AI puts beside an offer and Report shows; it has
++ * no person and no prompt answer. Planning and running spend the owner's money, so they carry the same
++ * operator token as the other owner switches.
++ */
++const crypto = require('crypto');
++const { http } = require('openvibe-contracts');
++const { asyncRouter } = require('./router');
++const { FidelityError } = require('../bench/fidelity');
++
++function operatorToken(req) {
++ const h = String(req.get('authorization') || '');
++ return /^Bearer\s+\S/i.test(h) ? h.replace(/^Bearer\s+/i, '').trim() : null;
++}
++
++function tokenMatches(given, expected) {
++ if (!given || !expected) return false;
++ const a = crypto.createHash('sha256').update(String(given), 'utf8').digest();
++ const b = crypto.createHash('sha256').update(String(expected), 'utf8').digest();
++ return a.length === b.length && crypto.timingSafeEqual(a, b);
++}
++
++function createFidelityRoutes(ctx) {
++ const r = asyncRouter();
++ const fidelity = ctx.fidelity;
++ const token = process.env.BENCH_ADMIN_TOKEN || '';
++
++ const problem = (res, err) => {
++ if (err instanceof FidelityError) return http.sendProblem(res, err.status, err.code, { detail: err.detail, ctx: res.req && res.req.ov });
++ throw err;
++ };
++ const operator = (req, res) => {
++ if (!token) { res.status(404).type('text').send('No such route.'); return false; }
++ if (!tokenMatches(operatorToken(req), token)) { http.sendProblem(res, 403, 'fidelity.operator_required', { detail: 'The operator token is required.', ctx: req.ov }); return false; }
++ return true;
++ };
++
++ r.get('/api/v1/fidelity', async (_req, res) => {
++ const body = await fidelity.read();
++ res.set('Cache-Control', 'public, max-age=60');
++ return res.json(body);
++ });
++
++ r.get('/api/v1/fidelity/objective-set', (_req, res) => {
++ res.set('Cache-Control', 'public, max-age=3600');
++ return res.json(fidelity.objectiveSet());
++ });
++
++ r.get('/api/v1/fidelity/status', async (req, res) => {
++ if (!operator(req, res)) return;
++ return res.json({ enabled: Boolean(ctx.config.bench.fidelity.enabled), objective_set: require('../bench/fidelity-set').OBJECTIVE_SET, state: await fidelity.lastState() });
++ });
++
++ r.post('/api/v1/fidelity/plan', async (req, res) => {
++ if (!operator(req, res)) return;
++ try { return res.json(await fidelity.plan()); } catch (err) { return problem(res, err); }
++ });
++
++ r.post('/api/v1/fidelity/run', async (req, res) => {
++ if (!operator(req, res)) return;
++ const body = req.body && typeof req.body === 'object' ? req.body : {};
++ try {
++ if (body.dry_run === true) return res.json(await fidelity.plan());
++ return res.status(201).json(await fidelity.run({ trigger: 'operator' }));
++ } catch (err) { return problem(res, err); }
++ });
++
++ return r;
++}
++
++module.exports = { createFidelityRoutes, operatorToken, tokenMatches };
ddiff --git a/server/index.js b/server/index.js
iindex d9ef316..5b6f07c 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -17,6 +17,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
() => ctx.jobs.stop(),
() => ctx.daily.stop(),
() => ctx.eeeSync.stop(),
++ () => ctx.fidelityJob.stop(),
() => ctx.publisher.stop(),
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
() => ctx.s.close(),
@@@ -48,6 +49,8 @@ async function start() {
ctx.daily.start();
// The weekly Every Eval Ever import (B2). Off until BENCH_EEE_SYNC is on; bounded and resumable.
ctx.eeeSync.start();
++ // The weekly provider-fidelity pass (B5). Off until BENCH_FIDELITY is on; due-based and capped.
++ ctx.fidelityJob.start();
// Outbox publisher → OpenVibe.Events.
ctx.publisher.start();
ddiff --git a/test/fidelity.test.js b/test/fidelity.test.js
nnew file mode 100644
iindex 0000000..f590da5
---- /dev/null
++++ b/test/fidelity.test.js
@@@ -0,0 +1,174 @@
++'use strict';
++
++// Provider fidelity (stage B5): the fixed objective set, the dry-run budget, one bounded pass through
++// AI's generate path with a durable idempotency key per prompt, and the public read OpenVibe.AI attaches
++// to an offer. The fake AI serves the market fixture and answers each prompt; no test reaches the network.
++process.env.BENCH_ADMIN_TOKEN = process.env.BENCH_ADMIN_TOKEN || 'op-secret';
++
++const assert = require('assert');
++const { boot, check, done } = require('./helpers/boot');
++const { OBJECTIVE_SET, ITEMS, check: checkItem } = require('../server/bench/fidelity-set');
++const { createFidelity, wilson, weekOf, estimateItem } = require('../server/bench/fidelity');
++
++const MARKET = {
++ $schema: 'exchange.market@1',
++ offers: [
++ { model: 'acme/alpha-1', provider: 'acme', provider_model: 'acme/alpha-1', kind: 'chat', open_weights: true, routed: true, price: { in: 100, out: 200 } },
++ { model: 'acme/alpha-1', provider: 'beta', provider_model: 'beta/alpha-1', kind: 'chat', open_weights: true, routed: true, price: { in: 100, out: 200 } },
++ { model: 'acme/alpha-1', provider: 'gamma', provider_model: 'gamma/alpha-1', kind: 'chat', open_weights: true, routed: false, price: { in: 100, out: 200 } },
++ { model: 'acme/closed-1', provider: 'acme', provider_model: 'acme/closed-1', kind: 'chat', open_weights: false, routed: true, price: { in: 100, out: 200 } },
++ { model: 'acme/unpriced-1', provider: 'acme', provider_model: 'acme/unpriced-1', kind: 'chat', open_weights: true, routed: true, price: { in: null, out: null } },
++ ],
++};
++
++function answer(body) {
++ const prompt = String(body.prompt || '');
++ if (prompt.startsWith('What is 37')) return '888';
++ if (prompt.startsWith('Read this line')) return 'fra1';
++ if (prompt.startsWith('Return exactly this JSON')) return '{"ok": true, "n": 3}';
++ if (prompt.startsWith('Write the word aurora')) return 'aurora aurora aurora aurora';
++ if (prompt.startsWith('Spell the word')) return 'desserts';
++ if (prompt.startsWith('How many times')) return '4';
++ if (prompt.startsWith('What is the chemical symbol')) return 'Au';
++ if (prompt.startsWith('If today is Wednesday')) return 'Saturday';
++ if (prompt.startsWith('Read the note')) return 'ORBIT-71';
++ if (prompt.startsWith('Reply with exactly')) return 'READY';
++ return 'unknown';
++}
++
++(async () => {
++ let t;
++ try {
++ t = await boot({
++ aiOptions: {
++ models: [
++ { provider: 'acme', model: 'acme/alpha-1' },
++ { provider: 'beta', model: 'beta/alpha-1' },
++ ],
++ market: MARKET,
++ // beta is a different quantization of the same model: it answers only three items right.
++ textFor: (body, pick) => {
++ const right = answer(body);
++ if (pick.provider === 'beta') {
++ const prompt = String(body.prompt || '');
++ if (prompt.startsWith('What is 37')) return 'not a number';
++ if (prompt.startsWith('Read this line')) return 'fra1';
++ if (prompt.startsWith('What is the chemical symbol')) return 'Au';
++ if (prompt.startsWith('Reply with exactly')) return 'READY';
++ return 'unknown';
++ }
++ return right;
++ },
++ },
++ env: {
++ BENCH_FIDELITY: '1',
++ BENCH_FIDELITY_IN_DRILL: '1',
++ BENCH_FIDELITY_DAILY_CAP_USD: '1.00',
++ BENCH_FIDELITY_MAX_CALLS: '30',
++ BENCH_FIDELITY_MIN_ANSWERED: '3',
++ },
++ });
++
++ await check('the objective set has one deterministic check per item and a version', () => {
++ assert.strictEqual(OBJECTIVE_SET, 'fidelity-2026-11-09.v1');
++ assert(ITEMS.length >= 8);
++ assert.strictEqual(new Set(ITEMS.map((i) => i.id)).size, ITEMS.length, 'item ids are unique');
++ assert(checkItem(ITEMS.find((i) => i.id === 'arithmetic'), '```\n888\n```').passed);
++ assert(checkItem(ITEMS.find((i) => i.id === 'json'), '{"ok":true,"n":3}').passed);
++ assert(!checkItem(ITEMS.find((i) => i.id === 'arithmetic'), 'the answer is 888').passed);
++ });
++
++ await check('the fidelity interval is a Wilson score interval', () => {
++ const perfect = wilson(10, 10);
++ assert.strictEqual(perfect.upper, 1);
++ assert(perfect.lower > 0.6 && perfect.lower < 1);
++ const none = wilson(0, 10);
++ assert.strictEqual(none.lower, 0);
++ assert(none.upper > 0 && none.upper < 0.4);
++ assert.deepStrictEqual(wilson(0, 0), { lower: null, upper: null });
++ });
++
++ await check('the call estimate comes from the offer price and the item text', () => {
++ const item = ITEMS.find((i) => i.id === 'arithmetic');
++ const cost = estimateItem(item, { in: 100, out: 200 });
++ assert(cost > 0 && cost < 1);
++ assert.strictEqual(estimateItem(item, { in: null, out: 200 }), null);
++ assert.strictEqual(weekOf(Date.parse('2026-11-11T12:00:00Z')).key, '2026-11-09');
++ });
++
++ await check('open weights, routed and priced only; the dry run lists every call', async () => {
++ const r = await t.get('/api/v1/fidelity/plan', { method: 'POST', bearer: 'op-secret', json: {} });
++ assert.strictEqual(r.status, 200, r.text);
++ const plan = r.json();
++ assert.strictEqual(plan.objective_set, OBJECTIVE_SET);
++ assert.strictEqual(plan.offers.filter((o) => o.included).length, 2, 'the two enabled providers of the open-weight model');
++ assert.strictEqual(plan.calls, ITEMS.length * 2);
++ assert(plan.estimated_usd > 0);
++ assert(plan.daily.cap_usd === 1);
++ const unpriced = plan.offers.find((o) => o.provider_model === 'acme/unpriced-1');
++ assert(unpriced && unpriced.included === false && /price/.test(unpriced.reason), 'an offer without a price is skipped, not guessed');
++ assert(!/closed-1|gamma/.test(r.text), 'closed and unrouted offers are out');
++ assert.strictEqual(t.ai.requests.filter((x) => x.path === '/api/v1/generate').length, 0, 'a dry run spends nothing');
++ });
++
++ await check('a pass answers every item through AI and scores each offer', async () => {
++ const r = await t.get('/api/v1/fidelity/run', { method: 'POST', bearer: 'op-secret', json: {} });
++ assert.strictEqual(r.status, 201, r.text);
++ const out = r.json();
++ assert.strictEqual(out.ran.length, 2);
++ const acme = out.ran.find((o) => o.provider === 'acme');
++ const beta = out.ran.find((o) => o.provider === 'beta');
++ assert.strictEqual(acme.score, 1);
++ assert.strictEqual(acme.n, ITEMS.length);
++ assert.strictEqual(beta.score, 0.3);
++ assert.strictEqual(beta.passed, 3);
++ const generated = t.ai.requests.filter((x) => x.path === '/api/v1/generate');
++ assert.strictEqual(generated.length, ITEMS.length * 2);
++ assert(generated.every((x) => /^fid_[a-f0-9]{40}$/.test(x.body.idempotency_key)), 'every prompt has a durable key');
++ assert(generated.every((x) => x.body.route && x.body.route.model && x.body.route.provider), 'every call is pinned to the exact offer');
++ assert(generated.every((x) => x.body.temperature === 0), 'the objective set is not sampled');
++ });
++
++ await check('the public read puts score, n, interval and compared_to beside every offer', async () => {
++ const r = await t.get('/api/v1/fidelity');
++ assert.strictEqual(r.status, 200, r.text);
++ assert.strictEqual(r.headers.get('cache-control'), 'public, max-age=60');
++ const body = r.json();
++ assert.strictEqual(body.$schema, 'exchange.fidelity@1');
++ assert.strictEqual(body.offers.length, 2);
++ assert.strictEqual(body.measured_on, new Date(t.ctx.s.now()).toISOString().slice(0, 10));
++ for (const offer of body.offers) {
++ assert.strictEqual(offer.compared_to, 'acme/alpha-1');
++ assert(Number.isFinite(offer.score));
++ assert.strictEqual(offer.n, ITEMS.length);
++ assert(offer.interval.lower <= offer.score && offer.score <= offer.interval.upper);
++ assert(offer.measured_at);
++ }
++ const meta = await t.get('/api/v1/fidelity/objective-set');
++ assert.strictEqual(meta.status, 200);
++ assert.strictEqual(meta.json().prompts.length, ITEMS.length);
++ });
++
++ await check('the operator token is required and the status names the last pass', async () => {
++ assert.strictEqual((await t.get('/api/v1/fidelity/plan', { method: 'POST', json: {} })).status, 403);
++ assert.strictEqual((await t.get('/api/v1/fidelity/run', { method: 'POST', bearer: 'wrong', json: {} })).status, 403);
++ const status = await t.get('/api/v1/fidelity/status', { bearer: 'op-secret' });
++ assert.strictEqual(status.status, 200);
++ assert.strictEqual(status.json().state.last_status, 'ok');
++ });
++
++ await check('the pass is off unless the owner turns it on', async () => {
++ const off = createFidelity({
++ s: { db: { prepare: () => ({ get: async () => null, all: async () => [], run: async () => null }) }, now: () => Date.now(), iso: (v) => String(v), newId: () => 'fid_test' },
++ ai: { market: async () => MARKET },
++ config: { bench: { fidelity: { enabled: false, dailyCapUsd: 1 } } },
++ });
++ await assert.rejects(() => off.run(), (err) => err.code === 'fidelity.disabled' && err.status === 503);
++ const plan = await off.plan();
++ assert.strictEqual(plan.enabled, false);
++ });
++ } finally {
++ if (t) await t.close();
++ }
++ done();
++})().catch((e) => { console.error(e); process.exit(1); });
ddiff --git a/test/helpers/ai.js b/test/helpers/ai.js
iindex df82032..e7c4cb2 100644
---- a/test/helpers/ai.js
++++ b/test/helpers/ai.js
@@@ -64,8 +64,8 @@ function listen(handler) {
});
}
--async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, tokensIn = 120, tokensOut = 480, textFor = null, willingness = DEFAULT_WILLINGNESS, abliterationTax = DEFAULT_TAX, willingnessStatus = 200, abliterationTaxStatus = 200 } = {}) {
-- const st = { models: models.map((m) => ({ ...m })), requests: [], runCount: 0, runs: new Map(), costUsd, ms, tokensIn, tokensOut, textFor, willingness, abliterationTax, willingnessStatus, abliterationTaxStatus };
++async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, tokensIn = 120, tokensOut = 480, textFor = null, market = { offers: [] }, marketStatus = 200, willingness = DEFAULT_WILLINGNESS, abliterationTax = DEFAULT_TAX, willingnessStatus = 200, abliterationTaxStatus = 200 } = {}) {
++ const st = { models: models.map((m) => ({ ...m })), requests: [], runCount: 0, runs: new Map(), costUsd, ms, tokensIn, tokensOut, textFor, market, marketStatus, willingness, abliterationTax, willingnessStatus, abliterationTaxStatus };
function modelFor(body) {
const asked = body && body.route;
@@@ -103,6 +103,12 @@ async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, to
return json(200, st.abliterationTax);
}
++ if (url.pathname === '/api/v1/exchange/market' && req.method === 'GET') {
++ st.requests.push({ method: 'GET', path: url.pathname, query: Object.fromEntries(url.searchParams), body });
++ if (st.marketStatus !== 200) return json(st.marketStatus, { code: 'stub.unavailable', detail: 'the market feed is unavailable' });
++ return json(200, st.market);
++ }
++
if (url.pathname === '/api/v1/routes' && req.method === 'GET') {
st.requests.push({ method: 'GET', path: url.pathname, body });
const [first, ...rest] = st.models;