openvibers/openvibe.bench Commits
Measure provider fidelity on a fixed objective set (B5) (#181)
Changes
@@@ -2,6 +2,14 @@## Unreleased++- **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, the++ score of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each,++ no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the++ dry run: every call, the own-page price estimate and whether it fits the owner's daily cap. `POST++ /api/v1/fidelity/run` runs one bounded pass; every call goes through OpenVibe.AI's normal generate path, pinned to++ the exact offer, with a durable idempotency key per prompt and week. Off by default (`BENCH_FIDELITY=1`), daily++ capped, and a weekly due-based job when on. The objective set is public at `GET /api/v1/fidelity/objective-set`;++ answers are never stored, only which item passed.- **The 18+ board (`chat.willing`).** A new adult kind and `GET /api/v1/boards/chat.willing`, marked `adult: true` on the board and every row. It reads OpenVibe.AI's measured refusal and hedge rates (n and Wilson intervals) on the published 60-prompt lawful adult set, plus the abliteration tax for each declared base/abliterated pair: the same objective set on both, with the score difference and its Newcombe interval. A non-willingness type on the adult kind is refused; when AI cannot answer, the board says the measurements are unavailable and invents no rows.- **Every Eval Ever (B2).** Bench imports the public EEE datastore (`github.com/evaleval/every_eval_ever`,`evaleval/EEE_datastore`) as *external* results: the model (joined by Report's rule — maker prefix and a date or@@@ -245,6 +245,22 @@ The open dataset (`/api/v1/datasets/:kindId.jsonl`) also emits `type: "category"`type: "holdout"` lines, and its rating lines carry the rank interval. The formulas, the 0.5/0.5 weights and the30-game floor are in [index.md](index.md).++## Provider fidelity (stage B5)++++| Route | Who | What |++|---|---|---|++| `GET /api/v1/fidelity` | anyone | The latest measurement of every offer: `score` (pass rate), `n`, `interval` (95% Wilson), `compared_to`, `measured_at`, `prompt_set`. Cached 60 seconds. |++| `GET /api/v1/fidelity/objective-set` | anyone | The fixed public set (`fidelity-2026-11-09.v1`): every prompt and the deterministic check it is graded by. |++| `POST /api/v1/fidelity/plan` | operator token | The dry run: each offer, its calls, the own-page price estimate and whether it fits the daily cap. Spends nothing. |++| `POST /api/v1/fidelity/run` | operator token | One bounded pass. Refused with `fidelity.disabled` until `BENCH_FIDELITY=1`; capped by `BENCH_FIDELITY_DAILY_CAP_USD` and `BENCH_FIDELITY_MAX_CALLS`. |++| `GET /api/v1/fidelity/status` | operator token | Whether the switch is on and the last pass's state. |++++The measurement asks every enabled provider of an open-weight model the same ten objective items, answers only, and++grades each answer with a published exact/JSON/regex check. A provider that quantizes, truncates the context or++serves another checkpoint answers fewer correctly; that is the number beside its price. A caller never sees a rank++or a winner. Scores travel to OpenVibe.AI with `GET /api/v1/exchange/market` / `catalog` as each offer's `fidelity`++field (null when unmeasured). The dry run and the cap are the owner's switch: fidelity is off by default.++## The older APIBench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the privatennew file mode 100644@@@ -0,0 +1,41 @@++-- phase: expand++-- Provider fidelity (plan 2026-11-09, stage B5).++--++-- One row per measured offer and pass: the same open-weight model, served by one provider, asked the++-- fixed objective set. The score is passes / answered items with its Wilson interval; the answers++-- themselves are not kept, only which item passed. A provider that quantizes, truncates the context or++-- serves a different checkpoint scores lower on the same set, and the public read shows the difference++-- rather than a rank.++--++-- measured_at is when the pass ran; cost_usd is what the calls reported, so the daily cap can sum the++-- day's spend without a second ledger. results keeps the per-item verdicts (id, passed, error) for audit.++CREATE TABLE IF NOT EXISTS bench_fidelity_runs (++ id text COLLATE "C" PRIMARY KEY,++ objective_set text COLLATE "C" NOT NULL,++ model text COLLATE "C" NOT NULL, -- the model the provider claims to serve++ provider text COLLATE "C" NOT NULL,++ provider_model text COLLATE "C" NOT NULL, -- the exact endpoint id that answered++ n integer NOT NULL DEFAULT 0, -- answered items (errors are counted apart)++ passed integer NOT NULL DEFAULT 0,++ errors integer NOT NULL DEFAULT 0, -- items the provider did not answer++ score double precision, -- passed / n, null when nothing was answered++ ci_lower double precision, -- Wilson 95% interval++ ci_upper double precision,++ cost_usd double precision NOT NULL DEFAULT 0,++ measured_at timestamptz NOT NULL,++ measured_on date NOT NULL,++ results jsonb NOT NULL DEFAULT '[]'::jsonb,++ created_at timestamptz NOT NULL DEFAULT now()++);++CREATE INDEX IF NOT EXISTS bench_fidelity_latest_idx ON bench_fidelity_runs (provider, provider_model, measured_at DESC);++CREATE INDEX IF NOT EXISTS bench_fidelity_day_idx ON bench_fidelity_runs (measured_at);++++-- The weekly job is due-based: its last pass survives restarts and deploys.++CREATE TABLE IF NOT EXISTS bench_fidelity_state (++ id text COLLATE "C" PRIMARY KEY CHECK (id = 'fidelity'),++ last_run_at timestamptz,++ last_status text COLLATE "C",++ last_error text COLLATE "C",++ updated_at timestamptz NOT NULL DEFAULT now()++);++INSERT INTO bench_fidelity_state (id) VALUES ('fidelity') ON CONFLICT (id) DO NOTHING;@@@ -92,6 +92,9 @@ async function createApp(opts = {}) {// B2: the external EEE records and the exporter of Bench's own runs, plus the weekly import job.const eee = require('./bench/eee').createEee({ s, registry, fetchImpl, log, config });const eeeSync = require('./bench/eee-sync').createEeeSync({ eee, config, log });++ // B5: the provider-fidelity measurements and their weekly pass (off unless BENCH_FIDELITY is on).++ const fidelity = require('./bench/fidelity').createFidelity({ s, ai, config, clock: { now: opts.now || (() => Date.now()) }, log });++ const fidelityJob = require('./bench/fidelity-job').createFidelityJob({ fidelity, config, log });const jobs = createBenchJobs({ s, bench, ai, config, log });// The signed-out voter (the cookie, the relay header and the daily caps) and the daily rounds job.const anon = createAnon({ s, config, log });@@@ -114,7 +117,7 @@ async function createApp(opts = {}) {const ctx = {config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,anon, daily, variantEvidence, evidenceConfig, sharing,-- imports, mirror, eee, eeeSync,++ imports, mirror, eee, eeeSync, fidelity, fidelityJob,random: opts.random || Math.random,};@@@ -258,6 +261,7 @@ async function createApp(opts = {}) {}// The EEE routes come before the kind dataset route: /datasets/eee.jsonl would otherwise be read as kind 'eee'.app.use(require('./http/eee').createEeeRoutes(ctx));++ app.use(require('./http/fidelity').createFidelityRoutes(ctx));app.use('/api/v1', createApi(ctx));app.use('/api/v1', createRegistryApi(ctx));app.use('/api/v1', createVariantEvidenceApi(ctx));@@@ -202,17 +202,18 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })return { run: r, ms: msOf(r, Date.now() - startedAt) };}-- async function generate({ prompt, kind, slot, roundId, index }) {++ async function generate({ prompt, kind, slot, roundId, index, system = null, maxTokens = null, temperature = 0.7, idempotencyKey = null, target = null }) {if (!enabled) throw new BenchAiError(503, 'ai.not_configured', 'OpenVibe.AI is not configured for Bench.');const pinned = slot && slot.provider && slot.provider !== 'ai' ? { provider: slot.provider, model: slot.model || null } : null;const body = {prompt,-- system: systemPrompt(kind),-- max_tokens: maxTokensFor(kind),-- temperature: 0.7,++ system: system || systemPrompt(kind),++ max_tokens: maxTokens == null ? maxTokensFor(kind) : maxTokens,++ temperature,options: { cache: false },...(pinned ? { route: pinned } : {}),-- target: { service: 'bench', type: 'bench_output', id: `${roundId}_${index}`.slice(0, 128) },++ ...(idempotencyKey ? { idempotency_key: idempotencyKey } : {}),++ target: target || { service: 'bench', type: 'bench_output', id: String(idempotencyKey || `${roundId}_${index}`).slice(0, 128) },};const startedAt = Date.now();const wait = Math.min(ai.waitMs, 60_000);@@@ -226,6 +227,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })input: { prompt: body.prompt, system: body.system, max_tokens: body.max_tokens, temperature: body.temperature },options: body.options,...(body.route ? { route: body.route } : {}),++ ...(body.idempotency_key ? { idempotency_key: body.idempotency_key } : {}),target: body.target,});}@@@ -264,6 +266,12 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })};}++ /** The market feed: every own-page model and provider offer, with its price and open-weights flag. */++ async function market({ openWeights = true, limit = 500 } = {}) {++ const query = `?limit=${Math.max(1, Math.min(500, Number(limit) || 500))}${openWeights ? '&open_weights=true' : ''}&kind=chat`;++ return call('GET', `/api/v1/exchange/market${query}`);++ }++/** The 18+ board's measured willingness rates, from OpenVibe.AI's public read API. */async function willingness() {return call('GET', '/api/v1/exchange/willingness');@@@ -274,7 +282,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })return call('GET', '/api/v1/exchange/abliteration-tax');}-- return { enabled, slots, slotPlan, offeredModels, rankOffers, generate, systemPrompt, maxTokensFor, willingness, abliterationTax };++ return { enabled, slots, slotPlan, offeredModels, rankOffers, generate, systemPrompt, maxTokensFor, market, willingness, abliterationTax };}module.exports = { createBenchAi, BenchAiError };nnew file mode 100644@@@ -0,0 +1,44 @@++'use strict';++++/**++ * The weekly provider-fidelity pass (stage B5). Off by default: the owner turns it on with BENCH_FIDELITY=1.++ * The pass is due-based (the last pass is a row in the database), so the several deploys Bench gets in a++ * week do not reset its clock. A pass that is already running is never started twice.++ */++function createFidelityJob({ fidelity, config, log = console }) {++ let timer = null;++ let running = false;++++ async function tick() {++ if (!config.bench.fidelity.enabled) return null;++ if (config.drill && !config.bench.fidelity.enabledInDrill) return null;++ if (running) return null;++ running = true;++ try {++ if (!(await fidelity.due())) return null;++ return await fidelity.run({ trigger: 'weekly' });++ } catch (err) {++ log.warn('[OpenVibe.Bench] fidelity job:', (err && (err.detail || err.message)) || err);++ return null;++ } finally {++ running = false;++ }++ }++++ function start() {++ if (timer || !config.bench.fidelity.enabled) return null;++ const run = () => tick().catch((err) => log.warn('[OpenVibe.Bench] fidelity job:', (err && err.message) || err));++ timer = setInterval(run, config.bench.fidelity.tickMs);++ if (timer.unref) timer.unref();++ return timer;++ }++++ function stop() {++ if (timer) clearInterval(timer);++ timer = null;++ }++++ return { tick, start, stop };++}++++module.exports = { createFidelityJob };nnew file mode 100644@@@ -0,0 +1,176 @@++'use strict';++++/**++ * The fixed objective set used to measure provider fidelity (plan 2026-11-09, stage B5).++ *++ * Provider fidelity asks a different question from the pairwise boards: the same open-weight model is++ * served by several providers, and the question is whether each one answers the *model's* task as well++ * as the others. So the set is small, fixed, public and objective: every item has one answer a++ * deterministic check can recognise, with no LLM judge and no human vote. A provider that quantizes the++ * weights, changes the chat template, truncates the context or substitutes a different checkpoint scores++ * lower on the same items, and the difference is what the fidelity field publishes.++ *++ * The version is part of every result and every idempotency key: changing the set changes the version,++ * so two scores are never compared across sets. A score is a pass rate with a Wilson interval, not a++ * ranking: the public read never names a winner.++ */++++const OBJECTIVE_SET = 'fidelity-2026-11-09.v1';++++/** The system line every item is measured with: the same instructions for every provider. */++const SYSTEM = [++ 'You are answering a fixed objective benchmark.',++ 'Follow the requested format exactly and return only the answer, with no explanation or extra text.',++].join('\n');++++const ITEMS = Object.freeze([++ {++ id: 'arithmetic',++ prompt: 'What is 37 multiplied by 24? Answer with the number only.',++ max_tokens: 16,++ check: { type: 'equals', value: '888' },++ },++ {++ id: 'extraction',++ prompt: 'Read this line: "region: fra1; status: ok". Answer with the region code only.',++ max_tokens: 16,++ check: { type: 'equals', value: 'fra1' },++ },++ {++ id: 'json',++ prompt: 'Return exactly this JSON object and nothing else: {"ok": true, "n": 3}',++ max_tokens: 40,++ check: { type: 'json', value: { ok: true, n: 3 } },++ },++ {++ id: 'repeat',++ prompt: 'Write the word aurora exactly four times in a row, separated by single spaces. Answer with the four words only.',++ max_tokens: 24,++ check: { type: 'equals', value: 'aurora aurora aurora aurora' },++ },++ {++ id: 'reverse',++ prompt: 'Spell the word "stressed" backwards. Answer with the reversed word only.',++ max_tokens: 16,++ check: { type: 'equals', value: 'desserts' },++ },++ {++ id: 'count',++ prompt: 'How many times does the letter s appear in the word "mississippi"? Answer with the number only.',++ max_tokens: 16,++ check: { type: 'equals', value: '4' },++ },++ {++ id: 'symbol',++ prompt: 'What is the chemical symbol for gold? Answer with the symbol only.',++ max_tokens: 16,++ check: { type: 'equals', value: 'Au' },++ },++ {++ id: 'next-day',++ prompt: 'If today is Wednesday, what day will it be three days from now? Answer with the day only.',++ max_tokens: 16,++ check: { type: 'equals', value: 'Saturday' },++ },++ {++ id: 'long-context',++ prompt: [++ 'Read the note below to its last line, then answer the question.',++ '',++ ...Array.from({ length: 40 }, (_, i) => `Note line ${i + 1}: the archive keeps its records in order and never changes a line.`),++ 'The access code is ORBIT-71.',++ '',++ 'Question: what is the access code? Answer with the code only.',++ ].join('\n'),++ max_tokens: 16,++ check: { type: 'equals', value: 'ORBIT-71' },++ },++ {++ id: 'format',++ prompt: 'Reply with exactly the word READY and nothing else.',++ max_tokens: 12,++ check: { type: 'equals', value: 'READY' },++ },++]);++++/** The answer without the wrappers models like to add: a code fence, quotes, a final full stop. */++function stripFence(text) {++ const s = String(text == null ? '' : text).trim();++ const fenced = /^```[a-z0-9_-]*\s*\n?([\s\S]*?)\n?```$/i.exec(s);++ return fenced ? fenced[1] : s;++}++++function normalize(text) {++ return stripFence(text)++ .trim()++ .replace(/^["'`]+|["'`]+$/g, '')++ .replace(/[.!]\s*$/, '')++ .replace(/\s+/g, ' ')++ .trim()++ .toLowerCase();++}++++function parseJson(text) {++ const s = stripFence(text).trim();++ const start = s.indexOf('{');++ const end = s.lastIndexOf('}');++ if (start < 0 || end <= start) return null;++ try { return JSON.parse(s.slice(start, end + 1)); } catch { return null; }++}++++const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);++++/**++ * One item's verdict. `passed` is the only thing stored; `reason` says what the check compared, so an++ * operator can see a template change without reading the answer back.++ */++function check(item, answer) {++ const got = String(answer == null ? '' : answer);++ const c = item.check || {};++ if (c.type === 'equals') {++ const passed = normalize(got) === normalize(c.value);++ return { passed, reason: passed ? 'exact' : 'not the expected answer' };++ }++ if (c.type === 'one_of') {++ const n = normalize(got);++ const passed = (c.values || []).some((v) => normalize(v) === n);++ return { passed, reason: passed ? 'one of the expected answers' : 'not one of the expected answers' };++ }++ if (c.type === 'contains_all') {++ const n = normalize(got);++ const missing = (c.values || []).filter((v) => !n.includes(normalize(v)));++ return { passed: missing.length === 0, reason: missing.length ? `missing ${missing.join(', ')}` : 'all parts present' };++ }++ if (c.type === 'json') {++ const parsed = parseJson(got);++ const passed = parsed != null && same(parsed, c.value);++ return { passed, reason: passed ? 'the expected object' : 'not the expected JSON' };++ }++ if (c.type === 'regex') {++ const passed = new RegExp(String(c.pattern), String(c.flags || 'i')).test(got);++ return { passed, reason: passed ? 'matched' : 'did not match' };++ }++ return { passed: false, reason: 'unknown check' };++}++++/** The input side of a cost estimate, from the text itself (a stable convention, not a billing number). */++function promptTokens(item) {++ return Math.max(1, Math.ceil((SYSTEM.length + String(item.prompt || '').length) / 4));++}++++/** The public view of the set: what was asked, never the model's answer. */++function publicSet() {++ return {++ objective_set: OBJECTIVE_SET,++ items: ITEMS.length,++ note: 'A fixed, public objective set: each item has one answer a deterministic check recognises. The same set is run through every provider that serves an open-weight model.',++ prompts: ITEMS.map((item) => ({++ id: item.id,++ prompt: item.prompt,++ max_tokens: item.max_tokens,++ check: item.check,++ })),++ };++}++++module.exports = { OBJECTIVE_SET, SYSTEM, ITEMS, normalize, check, promptTokens, publicSet };nnew file mode 100644@@@ -0,0 +1,309 @@++'use strict';++++/**++ * Provider fidelity (plan 2026-11-09, stage B5).++ *++ * OpenVibe.AI routes; this module measures. For every popular open-weight model the network routes, it++ * finds every enabled provider that serves that model in AI's public market feed, asks each offer the++ * fixed objective set, and scores the answers deterministically. A provider that quantizes, truncates++ * the context or serves another checkpoint answers fewer items correctly; the score is a pass rate with++ * a Wilson interval and the sample size beside it, never a rank or a bare winner.++ *++ * Money: the owner's switch is off by default; a pass has a daily cap in US dollars and a call cap. The++ * dry run lists every call and what the pricebook says it would cost before anything is spent. Every++ * call goes through AI's normal generate path (its own reserve/settle accounting), pinned to the exact++ * provider and model, with a durable idempotency key per prompt and week, so a retried pass is not++ * charged twice. The answers never leave this service: the table stores only which item passed.++ */++const crypto = require('crypto');++const { OBJECTIVE_SET, SYSTEM, ITEMS, check, promptTokens, publicSet } = require('./fidelity-set');++++const ERROR_MAX = 300;++const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));++const round = (v) => (v == null ? null : Math.round(Number(v) * 1e6) / 1e6);++// PostgreSQL returns `date` as a JS Date; PGlite returns the string. Both are the same day.++const dayOf = (v) => (v instanceof Date ? v.toISOString().slice(0, 10) : String(v || '').slice(0, 10));++++class FidelityError extends Error {++ constructor(status, code, detail) {++ super(detail);++ this.name = 'FidelityError';++ this.status = status;++ this.code = code;++ this.detail = detail;++ }++}++++/** The Wilson score interval at 95%: an honest band around a pass rate with few items. */++function wilson(passed, n, z = 1.96) {++ if (!Number.isFinite(n) || n <= 0) return { lower: null, upper: null };++ const p = Math.max(0, Math.min(1, passed / n));++ const d = 1 + (z * z) / n;++ const centre = (p + (z * z) / (2 * n)) / d;++ const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;++ return { lower: Math.max(0, round(centre - half)), upper: Math.min(1, round(centre + half)) };++}++++/** The Monday 00:00 UTC of the week a moment falls in, and the key the idempotency of a pass uses. */++function weekOf(ms) {++ const d = new Date(ms);++ const day = (d.getUTCDay() + 6) % 7;++ const start = Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate() - day);++ return { start, key: new Date(start).toISOString().slice(0, 10) };++}++++function startOfUtcDay(ms) {++ const d = new Date(ms);++ return Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate());++}++++/** The planned cost of one item on one offer, from the market's own-page price. Null when unknown. */++function estimateItem(item, price) {++ const inPrice = num(price && price.in);++ const outPrice = num(price && price.out);++ if (inPrice == null || outPrice == null) return null;++ return round(((promptTokens(item) / 1e6) * inPrice) + ((item.max_tokens / 1e6) * outPrice));++}++++const idempotencyKey = (weekKey, provider, providerModel, itemId) => {++ const digest = crypto.createHash('sha256')++ .update(`${OBJECTIVE_SET}\0${weekKey}\0${provider}\0${providerModel}\0${itemId}`, 'utf8')++ .digest('hex')++ .slice(0, 40);++ return `fid_${digest}`;++};++++function keyOf(provider, providerModel) {++ return `${String(provider || '').toLowerCase()}\0${String(providerModel || '').toLowerCase()}`;++}++++/**++ * The public read: the latest measurement of every offer, grouped by the model it serves. No row means++ * nobody measured that offer; the field is absent there, never a zero.++ */++function createFidelity({ s, ai, config, clock = { now: () => Date.now() }, log = console } = {}) {++ const cfg = (config && config.bench && config.bench.fidelity) || {};++ const dailyCapUsd = Math.max(0, num(cfg.dailyCapUsd) == null ? 0 : num(cfg.dailyCapUsd));++ const maxModels = Math.max(1, Number(cfg.maxModels) || 10);++ const maxCalls = Math.max(1, Number(cfg.maxCallsPerRun) || 60);++ const minAnswered = Math.max(1, Number(cfg.minAnswered) || Math.min(5, ITEMS.length));++++ const latest = s.db.prepare(`SELECT DISTINCT ON (provider, provider_model)++ provider, provider_model, model, n, passed, errors, score, ci_lower, ci_upper, measured_at, measured_on, objective_set++ FROM bench_fidelity_runs WHERE objective_set = ?++ ORDER BY provider, provider_model, measured_at DESC, id DESC`);++ const spent = s.db.prepare('SELECT COALESCE(SUM(cost_usd), 0) AS usd FROM bench_fidelity_runs WHERE measured_at >= ?');++ const insert = s.db.prepare(`INSERT INTO bench_fidelity_runs++ (id, objective_set, model, provider, provider_model, n, passed, errors, score, ci_lower, ci_upper, cost_usd, measured_at, measured_on, results)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?::timestamptz, ?::date, ?::jsonb)`);++ const state = s.db.prepare(`UPDATE bench_fidelity_state SET last_run_at = ?::timestamptz, last_status = ?, last_error = ?,++ updated_at = now() WHERE id = 'fidelity'`);++ const stateRow = s.db.prepare("SELECT last_run_at, last_status, last_error FROM bench_fidelity_state WHERE id = 'fidelity'");++++ async function todaySpent() {++ const row = await spent.get(new Date(startOfUtcDay(clock.now())).toISOString());++ return Number(row && row.usd) || 0;++ }++++ /** The public market: every own-page offer, enabled or not. A missing client is a plain 503. */++ async function market() {++ if (!ai || typeof ai.market !== 'function') throw new FidelityError(503, 'fidelity.unavailable', 'OpenVibe.AI market readings are not configured for Bench.');++ let body;++ try {++ body = await ai.market({ openWeights: true, limit: 500 });++ } catch (err) {++ throw new FidelityError(503, 'fidelity.market_unavailable', `OpenVibe.AI's market feed is unavailable: ${String((err && (err.detail || err.message)) || err).slice(0, ERROR_MAX)}`);++ }++ const offers = Array.isArray(body && body.offers) ? body.offers : [];++ if (!offers.length && body && body.error) throw new FidelityError(503, 'fidelity.market_unavailable', 'OpenVibe.AI returned no market offers.');++ return offers;++ }++++ /** The offers worth measuring: open weights, routed, chat, a real provider and endpoint id. */++ function eligible(offers) {++ const out = [];++ for (const offer of offers) {++ if (!offer || typeof offer !== 'object') continue;++ if (offer.open_weights !== true || offer.routed !== true) continue;++ if (offer.kind != null && offer.kind !== 'chat') continue;++ const provider = String(offer.provider || '').trim();++ const providerModel = String(offer.provider_model || offer.model || '').trim();++ const model = String(offer.model || providerModel).trim();++ if (!provider || !providerModel || !model) continue;++ out.push({ model, provider, provider_model: providerModel, name: offer.name || providerModel, price: offer.price || null });++ }++ return out;++ }++++ function groupByModel(offers) {++ const groups = new Map();++ for (const offer of offers) {++ if (!groups.has(offer.model)) groups.set(offer.model, []);++ groups.get(offer.model).push(offer);++ }++ return [...groups.entries()]++ .map(([model, list]) => ({ model, offers: list.sort((a, b) => a.provider.localeCompare(b.provider) || a.provider_model.localeCompare(b.provider_model)) }))++ // "Popular" means served by more providers first; the cap keeps a pass inside the owner's call budget.++ .sort((a, b) => b.offers.length - a.offers.length || a.model.localeCompare(b.model))++ .slice(0, maxModels);++ }++++ /**++ * What a pass would do, and what the pricebook says it would cost. Every offer says whether it is++ * included and, when it is not, the plain reason: no price, a disabled switch, the daily cap or the++ * call cap. The plan never spends anything.++ */++ async function plan() {++ const available = eligible(await market());++ const groups = groupByModel(available);++ const spentUsd = await todaySpent();++ const remaining = Math.max(0, dailyCapUsd - spentUsd);++ let calls = 0;++ let estimated = 0;++ const offers = [];++ for (const group of groups) {++ for (const offer of group.offers) {++ const estimates = ITEMS.map((item) => estimateItem(item, offer.price));++ const offerEstimate = estimates.some((v) => v == null) ? null : round(estimates.reduce((a, b) => a + b, 0));++ const offerCalls = ITEMS.length;++ let included = true;++ let reason = null;++ if (offerEstimate == null) { included = false; reason = 'the market has no own-page price for this offer'; }++ else if (calls + offerCalls > maxCalls) { included = false; reason = `the pass is capped at ${maxCalls} calls`; }++ else if (estimated + offerEstimate > remaining) { included = false; reason = `the daily cap leaves $${round(remaining - estimated)}`; }++ if (included) { calls += offerCalls; estimated = round(estimated + offerEstimate); }++ offers.push({++ model: group.model, provider: offer.provider, provider_model: offer.provider_model, name: offer.name,++ price: offer.price, calls: offerCalls, estimated_usd: offerEstimate, included, reason,++ });++ }++ }++ return {++ $schema: 'exchange.fidelity.plan@1',++ dry_run: true,++ generated_at: new Date(clock.now()).toISOString(),++ objective_set: OBJECTIVE_SET,++ items: ITEMS.length,++ enabled: Boolean(cfg.enabled),++ daily: { cap_usd: round(dailyCapUsd), spent_usd: round(spentUsd), remaining_usd: round(remaining) },++ limits: { max_models: maxModels, max_calls: maxCalls },++ models: groups.map((g) => ({ model: g.model, offers: g.offers.length })),++ offers,++ calls,++ estimated_usd: estimated,++ };++ }++++ /** One bounded pass. Default off; returns what it measured, and never starts when nothing fits. */++ async function run({ trigger = 'manual' } = {}) {++ if (!cfg.enabled) throw new FidelityError(503, 'fidelity.disabled', 'Provider fidelity is off: the owner turns it on with BENCH_FIDELITY=1.');++ const before = await plan();++ const week = weekOf(clock.now());++ const measuredAt = new Date(clock.now()).toISOString();++ const measuredOn = measuredAt.slice(0, 10);++ const ran = [];++ let calls = 0;++ let spentUsd = 0;++ const included = before.offers.filter((offer) => offer.included);++ for (const offer of included) {++ if (calls >= maxCalls) break;++ if (dailyCapUsd > 0 && spentUsd >= dailyCapUsd) break;++ const results = [];++ let passed = 0;++ let answered = 0;++ let errors = 0;++ let cost = 0;++ for (const item of ITEMS) {++ if (calls >= maxCalls) break;++ calls += 1;++ try {++ if (!ai || typeof ai.generate !== 'function') throw new FidelityError(503, 'fidelity.unavailable', 'AI generation is not configured for Bench.');++ const out = await ai.generate({++ prompt: item.prompt,++ kind: 'text.answer',++ slot: { provider: offer.provider, model: offer.provider_model },++ system: SYSTEM,++ maxTokens: item.max_tokens,++ temperature: 0,++ idempotencyKey: idempotencyKey(week.key, offer.provider, offer.provider_model, item.id),++ });++ cost += Number(out.cost_usd) || 0;++ spentUsd += Number(out.cost_usd) || 0;++ const verdict = check(item, out.html || out.text || '');++ answered += 1;++ if (verdict.passed) passed += 1;++ results.push({ id: item.id, passed: verdict.passed, reason: verdict.reason, ms: out.ms == null ? null : out.ms });++ } catch (err) {++ errors += 1;++ results.push({ id: item.id, error: String((err && (err.detail || err.code || err.message)) || err).slice(0, ERROR_MAX) });++ }++ }++ if (answered < minAnswered) {++ log.warn(`[OpenVibe.Bench] fidelity: ${offer.provider}/${offer.provider_model} answered ${answered} of ${ITEMS.length}; no score stored`);++ continue;++ }++ const score = round(passed / answered);++ const interval = wilson(passed, answered);++ const id = s.newId('fid');++ await insert.run(id, OBJECTIVE_SET, offer.model, offer.provider, offer.provider_model, answered, passed, errors,++ score, interval.lower, interval.upper, round(cost), measuredAt, measuredOn, JSON.stringify(results));++ ran.push({ model: offer.model, provider: offer.provider, provider_model: offer.provider_model, n: answered, passed, errors, score, interval });++ }++ await state.run(measuredAt, 'ok', null).catch((err) => log.warn('[OpenVibe.Bench] fidelity state:', (err && err.message) || err));++ log.log(`[OpenVibe.Bench] fidelity ${trigger}: measured ${ran.length} offer(s), ${calls} call(s), $${round(spentUsd)}`);++ return { dry_run: false, generated_at: new Date(clock.now()).toISOString(), objective_set: OBJECTIVE_SET, ran, calls, spent_usd: round(spentUsd) };++ }++++ /** The latest score of every offer. This is the shape OpenVibe.AI reads to put fidelity on an offer. */++ async function read() {++ const rows = await latest.all(OBJECTIVE_SET);++ const offers = (rows || []).map((row) => ({++ model: row.model,++ provider: row.provider,++ provider_model: row.provider_model,++ score: num(row.score),++ passed: Number(row.passed) || 0,++ n: Number(row.n) || 0,++ errors: Number(row.errors) || 0,++ interval: { lower: num(row.ci_lower), upper: num(row.ci_upper) },++ compared_to: row.model,++ prompt_set: row.objective_set,++ measured_at: s.iso(row.measured_at),++ measured_on: String(row.measured_on).slice(0, 10),++ })).sort((a, b) => a.model.localeCompare(b.model) || a.provider.localeCompare(b.provider) || a.provider_model.localeCompare(b.provider_model));++ const models = new Map();++ for (const offer of offers) {++ const m = models.get(offer.model) || { model: offer.model, providers: new Set(), offers: 0, best_score: null };++ m.providers.add(offer.provider);++ m.offers += 1;++ if (offer.score != null && (m.best_score == null || offer.score > m.best_score)) m.best_score = offer.score;++ models.set(offer.model, m);++ }++ return {++ $schema: 'exchange.fidelity@1',++ objective_set: OBJECTIVE_SET,++ items: ITEMS.length,++ measured_on: offers.length ? offers.map((o) => o.measured_on).sort().slice(-1)[0] : null,++ offers,++ models: [...models.values()].map((m) => ({ model: m.model, providers: m.providers.size, offers: m.offers, best_score: m.best_score })),++ note: offers.length ? null : 'No offer has been measured on this objective set yet.',++ };++ }++++ /** Whether the weekly pass is due (the state row survives restarts, like the price book's). */++ async function due() {++ const interval = Math.max(3600000, Number(cfg.intervalMs) || 7 * 24 * 3600000);++ const row = await stateRow.get();++ if (!row || !row.last_run_at) return true;++ return clock.now() - new Date(row.last_run_at).getTime() >= interval;++ }++++ async function lastState() {++ const row = await stateRow.get();++ return row ? { last_run_at: s.iso(row.last_run_at), last_status: row.last_status || null, last_error: row.last_error || null } : null;++ }++++ return { plan, run, read, due, lastState, objectiveSet: publicSet };++}++++module.exports = { createFidelity, FidelityError, wilson, weekOf, estimateItem, idempotencyKey, keyOf, dayOf, OBJECTIVE_SET, ITEMS };@@@ -146,6 +146,19 @@ function load(env = process.env) {filesPerRun: Math.max(1, int(env.BENCH_EEE_FILES_PER_RUN, 5000)),recordsPerRun: Math.max(1, int(env.BENCH_EEE_RECORDS_PER_RUN, 50000)),},++ // B5 provider fidelity: one fixed objective set per open-weight model, every provider that++ // serves it. Off until the owner turns it on; the daily cap is real money, and the dry run++ // lists every call and its own-page price before anything is spent.++ fidelity: {++ enabled: bool(env.BENCH_FIDELITY, false),++ enabledInDrill: bool(env.BENCH_FIDELITY_IN_DRILL, false),++ dailyCapUsd: Math.max(0, float(env.BENCH_FIDELITY_DAILY_CAP_USD, 1.00)),++ maxModels: Math.max(1, int(env.BENCH_FIDELITY_MAX_MODELS, 10)),++ maxCallsPerRun: Math.max(1, int(env.BENCH_FIDELITY_MAX_CALLS, 60)),++ minAnswered: Math.max(1, int(env.BENCH_FIDELITY_MIN_ANSWERED, 5)),++ intervalMs: Math.max(3_600_000, int(env.BENCH_FIDELITY_INTERVAL_MS, 7 * 24 * 60 * 60 * 1000)),++ tickMs: Math.max(60_000, int(env.BENCH_FIDELITY_TICK_MS, 60 * 60 * 1000)),++ },},};}nnew file mode 100644@@@ -0,0 +1,81 @@++'use strict';++++/**++ * Provider-fidelity routes (stage B5).++ *++ * GET /api/v1/fidelity the latest measurement of every offer (public)++ * GET /api/v1/fidelity/objective-set the fixed objective set (public)++ * POST /api/v1/fidelity/plan the dry run: every call and its estimated cost (operator)++ * POST /api/v1/fidelity/run one bounded pass (operator; off unless the owner turned it on)++ * GET /api/v1/fidelity/status the last pass, when it ran and what it said (operator)++ *++ * The read is public because it is the number OpenVibe.AI puts beside an offer and Report shows; it has++ * no person and no prompt answer. Planning and running spend the owner's money, so they carry the same++ * operator token as the other owner switches.++ */++const crypto = require('crypto');++const { http } = require('openvibe-contracts');++const { asyncRouter } = require('./router');++const { FidelityError } = require('../bench/fidelity');++++function operatorToken(req) {++ const h = String(req.get('authorization') || '');++ return /^Bearer\s+\S/i.test(h) ? h.replace(/^Bearer\s+/i, '').trim() : null;++}++++function tokenMatches(given, expected) {++ if (!given || !expected) return false;++ const a = crypto.createHash('sha256').update(String(given), 'utf8').digest();++ const b = crypto.createHash('sha256').update(String(expected), 'utf8').digest();++ return a.length === b.length && crypto.timingSafeEqual(a, b);++}++++function createFidelityRoutes(ctx) {++ const r = asyncRouter();++ const fidelity = ctx.fidelity;++ const token = process.env.BENCH_ADMIN_TOKEN || '';++++ const problem = (res, err) => {++ if (err instanceof FidelityError) return http.sendProblem(res, err.status, err.code, { detail: err.detail, ctx: res.req && res.req.ov });++ throw err;++ };++ const operator = (req, res) => {++ if (!token) { res.status(404).type('text').send('No such route.'); return false; }++ if (!tokenMatches(operatorToken(req), token)) { http.sendProblem(res, 403, 'fidelity.operator_required', { detail: 'The operator token is required.', ctx: req.ov }); return false; }++ return true;++ };++++ r.get('/api/v1/fidelity', async (_req, res) => {++ const body = await fidelity.read();++ res.set('Cache-Control', 'public, max-age=60');++ return res.json(body);++ });++++ r.get('/api/v1/fidelity/objective-set', (_req, res) => {++ res.set('Cache-Control', 'public, max-age=3600');++ return res.json(fidelity.objectiveSet());++ });++++ r.get('/api/v1/fidelity/status', async (req, res) => {++ if (!operator(req, res)) return;++ return res.json({ enabled: Boolean(ctx.config.bench.fidelity.enabled), objective_set: require('../bench/fidelity-set').OBJECTIVE_SET, state: await fidelity.lastState() });++ });++++ r.post('/api/v1/fidelity/plan', async (req, res) => {++ if (!operator(req, res)) return;++ try { return res.json(await fidelity.plan()); } catch (err) { return problem(res, err); }++ });++++ r.post('/api/v1/fidelity/run', async (req, res) => {++ if (!operator(req, res)) return;++ const body = req.body && typeof req.body === 'object' ? req.body : {};++ try {++ if (body.dry_run === true) return res.json(await fidelity.plan());++ return res.status(201).json(await fidelity.run({ trigger: 'operator' }));++ } catch (err) { return problem(res, err); }++ });++++ return r;++}++++module.exports = { createFidelityRoutes, operatorToken, tokenMatches };@@@ -17,6 +17,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }() => ctx.jobs.stop(),() => ctx.daily.stop(),() => ctx.eeeSync.stop(),++ () => ctx.fidelityJob.stop(),() => ctx.publisher.stop(),async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },() => ctx.s.close(),@@@ -48,6 +49,8 @@ async function start() {ctx.daily.start();// The weekly Every Eval Ever import (B2). Off until BENCH_EEE_SYNC is on; bounded and resumable.ctx.eeeSync.start();++ // The weekly provider-fidelity pass (B5). Off until BENCH_FIDELITY is on; due-based and capped.++ ctx.fidelityJob.start();// Outbox publisher → OpenVibe.Events.ctx.publisher.start();nnew file mode 100644@@@ -0,0 +1,174 @@++'use strict';++++// Provider fidelity (stage B5): the fixed objective set, the dry-run budget, one bounded pass through++// AI's generate path with a durable idempotency key per prompt, and the public read OpenVibe.AI attaches++// to an offer. The fake AI serves the market fixture and answers each prompt; no test reaches the network.++process.env.BENCH_ADMIN_TOKEN = process.env.BENCH_ADMIN_TOKEN || 'op-secret';++++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const { OBJECTIVE_SET, ITEMS, check: checkItem } = require('../server/bench/fidelity-set');++const { createFidelity, wilson, weekOf, estimateItem } = require('../server/bench/fidelity');++++const MARKET = {++ $schema: 'exchange.market@1',++ offers: [++ { model: 'acme/alpha-1', provider: 'acme', provider_model: 'acme/alpha-1', kind: 'chat', open_weights: true, routed: true, price: { in: 100, out: 200 } },++ { model: 'acme/alpha-1', provider: 'beta', provider_model: 'beta/alpha-1', kind: 'chat', open_weights: true, routed: true, price: { in: 100, out: 200 } },++ { model: 'acme/alpha-1', provider: 'gamma', provider_model: 'gamma/alpha-1', kind: 'chat', open_weights: true, routed: false, price: { in: 100, out: 200 } },++ { model: 'acme/closed-1', provider: 'acme', provider_model: 'acme/closed-1', kind: 'chat', open_weights: false, routed: true, price: { in: 100, out: 200 } },++ { model: 'acme/unpriced-1', provider: 'acme', provider_model: 'acme/unpriced-1', kind: 'chat', open_weights: true, routed: true, price: { in: null, out: null } },++ ],++};++++function answer(body) {++ const prompt = String(body.prompt || '');++ if (prompt.startsWith('What is 37')) return '888';++ if (prompt.startsWith('Read this line')) return 'fra1';++ if (prompt.startsWith('Return exactly this JSON')) return '{"ok": true, "n": 3}';++ if (prompt.startsWith('Write the word aurora')) return 'aurora aurora aurora aurora';++ if (prompt.startsWith('Spell the word')) return 'desserts';++ if (prompt.startsWith('How many times')) return '4';++ if (prompt.startsWith('What is the chemical symbol')) return 'Au';++ if (prompt.startsWith('If today is Wednesday')) return 'Saturday';++ if (prompt.startsWith('Read the note')) return 'ORBIT-71';++ if (prompt.startsWith('Reply with exactly')) return 'READY';++ return 'unknown';++}++++(async () => {++ let t;++ try {++ t = await boot({++ aiOptions: {++ models: [++ { provider: 'acme', model: 'acme/alpha-1' },++ { provider: 'beta', model: 'beta/alpha-1' },++ ],++ market: MARKET,++ // beta is a different quantization of the same model: it answers only three items right.++ textFor: (body, pick) => {++ const right = answer(body);++ if (pick.provider === 'beta') {++ const prompt = String(body.prompt || '');++ if (prompt.startsWith('What is 37')) return 'not a number';++ if (prompt.startsWith('Read this line')) return 'fra1';++ if (prompt.startsWith('What is the chemical symbol')) return 'Au';++ if (prompt.startsWith('Reply with exactly')) return 'READY';++ return 'unknown';++ }++ return right;++ },++ },++ env: {++ BENCH_FIDELITY: '1',++ BENCH_FIDELITY_IN_DRILL: '1',++ BENCH_FIDELITY_DAILY_CAP_USD: '1.00',++ BENCH_FIDELITY_MAX_CALLS: '30',++ BENCH_FIDELITY_MIN_ANSWERED: '3',++ },++ });++++ await check('the objective set has one deterministic check per item and a version', () => {++ assert.strictEqual(OBJECTIVE_SET, 'fidelity-2026-11-09.v1');++ assert(ITEMS.length >= 8);++ assert.strictEqual(new Set(ITEMS.map((i) => i.id)).size, ITEMS.length, 'item ids are unique');++ assert(checkItem(ITEMS.find((i) => i.id === 'arithmetic'), '```\n888\n```').passed);++ assert(checkItem(ITEMS.find((i) => i.id === 'json'), '{"ok":true,"n":3}').passed);++ assert(!checkItem(ITEMS.find((i) => i.id === 'arithmetic'), 'the answer is 888').passed);++ });++++ await check('the fidelity interval is a Wilson score interval', () => {++ const perfect = wilson(10, 10);++ assert.strictEqual(perfect.upper, 1);++ assert(perfect.lower > 0.6 && perfect.lower < 1);++ const none = wilson(0, 10);++ assert.strictEqual(none.lower, 0);++ assert(none.upper > 0 && none.upper < 0.4);++ assert.deepStrictEqual(wilson(0, 0), { lower: null, upper: null });++ });++++ await check('the call estimate comes from the offer price and the item text', () => {++ const item = ITEMS.find((i) => i.id === 'arithmetic');++ const cost = estimateItem(item, { in: 100, out: 200 });++ assert(cost > 0 && cost < 1);++ assert.strictEqual(estimateItem(item, { in: null, out: 200 }), null);++ assert.strictEqual(weekOf(Date.parse('2026-11-11T12:00:00Z')).key, '2026-11-09');++ });++++ await check('open weights, routed and priced only; the dry run lists every call', async () => {++ const r = await t.get('/api/v1/fidelity/plan', { method: 'POST', bearer: 'op-secret', json: {} });++ assert.strictEqual(r.status, 200, r.text);++ const plan = r.json();++ assert.strictEqual(plan.objective_set, OBJECTIVE_SET);++ assert.strictEqual(plan.offers.filter((o) => o.included).length, 2, 'the two enabled providers of the open-weight model');++ assert.strictEqual(plan.calls, ITEMS.length * 2);++ assert(plan.estimated_usd > 0);++ assert(plan.daily.cap_usd === 1);++ const unpriced = plan.offers.find((o) => o.provider_model === 'acme/unpriced-1');++ assert(unpriced && unpriced.included === false && /price/.test(unpriced.reason), 'an offer without a price is skipped, not guessed');++ assert(!/closed-1|gamma/.test(r.text), 'closed and unrouted offers are out');++ assert.strictEqual(t.ai.requests.filter((x) => x.path === '/api/v1/generate').length, 0, 'a dry run spends nothing');++ });++++ await check('a pass answers every item through AI and scores each offer', async () => {++ const r = await t.get('/api/v1/fidelity/run', { method: 'POST', bearer: 'op-secret', json: {} });++ assert.strictEqual(r.status, 201, r.text);++ const out = r.json();++ assert.strictEqual(out.ran.length, 2);++ const acme = out.ran.find((o) => o.provider === 'acme');++ const beta = out.ran.find((o) => o.provider === 'beta');++ assert.strictEqual(acme.score, 1);++ assert.strictEqual(acme.n, ITEMS.length);++ assert.strictEqual(beta.score, 0.3);++ assert.strictEqual(beta.passed, 3);++ const generated = t.ai.requests.filter((x) => x.path === '/api/v1/generate');++ assert.strictEqual(generated.length, ITEMS.length * 2);++ assert(generated.every((x) => /^fid_[a-f0-9]{40}$/.test(x.body.idempotency_key)), 'every prompt has a durable key');++ assert(generated.every((x) => x.body.route && x.body.route.model && x.body.route.provider), 'every call is pinned to the exact offer');++ assert(generated.every((x) => x.body.temperature === 0), 'the objective set is not sampled');++ });++++ await check('the public read puts score, n, interval and compared_to beside every offer', async () => {++ const r = await t.get('/api/v1/fidelity');++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.headers.get('cache-control'), 'public, max-age=60');++ const body = r.json();++ assert.strictEqual(body.$schema, 'exchange.fidelity@1');++ assert.strictEqual(body.offers.length, 2);++ assert.strictEqual(body.measured_on, new Date(t.ctx.s.now()).toISOString().slice(0, 10));++ for (const offer of body.offers) {++ assert.strictEqual(offer.compared_to, 'acme/alpha-1');++ assert(Number.isFinite(offer.score));++ assert.strictEqual(offer.n, ITEMS.length);++ assert(offer.interval.lower <= offer.score && offer.score <= offer.interval.upper);++ assert(offer.measured_at);++ }++ const meta = await t.get('/api/v1/fidelity/objective-set');++ assert.strictEqual(meta.status, 200);++ assert.strictEqual(meta.json().prompts.length, ITEMS.length);++ });++++ await check('the operator token is required and the status names the last pass', async () => {++ assert.strictEqual((await t.get('/api/v1/fidelity/plan', { method: 'POST', json: {} })).status, 403);++ assert.strictEqual((await t.get('/api/v1/fidelity/run', { method: 'POST', bearer: 'wrong', json: {} })).status, 403);++ const status = await t.get('/api/v1/fidelity/status', { bearer: 'op-secret' });++ assert.strictEqual(status.status, 200);++ assert.strictEqual(status.json().state.last_status, 'ok');++ });++++ await check('the pass is off unless the owner turns it on', async () => {++ const off = createFidelity({++ s: { db: { prepare: () => ({ get: async () => null, all: async () => [], run: async () => null }) }, now: () => Date.now(), iso: (v) => String(v), newId: () => 'fid_test' },++ ai: { market: async () => MARKET },++ config: { bench: { fidelity: { enabled: false, dailyCapUsd: 1 } } },++ });++ await assert.rejects(() => off.run(), (err) => err.code === 'fidelity.disabled' && err.status === 503);++ const plan = await off.plan();++ assert.strictEqual(plan.enabled, false);++ });++ } finally {++ if (t) await t.close();++ }++ done();++})().catch((e) => { console.error(e); process.exit(1); });@@@ -64,8 +64,8 @@ function listen(handler) {});}--async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, tokensIn = 120, tokensOut = 480, textFor = null, willingness = DEFAULT_WILLINGNESS, abliterationTax = DEFAULT_TAX, willingnessStatus = 200, abliterationTaxStatus = 200 } = {}) {-- const st = { models: models.map((m) => ({ ...m })), requests: [], runCount: 0, runs: new Map(), costUsd, ms, tokensIn, tokensOut, textFor, willingness, abliterationTax, willingnessStatus, abliterationTaxStatus };++async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, tokensIn = 120, tokensOut = 480, textFor = null, market = { offers: [] }, marketStatus = 200, willingness = DEFAULT_WILLINGNESS, abliterationTax = DEFAULT_TAX, willingnessStatus = 200, abliterationTaxStatus = 200 } = {}) {++ const st = { models: models.map((m) => ({ ...m })), requests: [], runCount: 0, runs: new Map(), costUsd, ms, tokensIn, tokensOut, textFor, market, marketStatus, willingness, abliterationTax, willingnessStatus, abliterationTaxStatus };function modelFor(body) {const asked = body && body.route;@@@ -103,6 +103,12 @@ async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, toreturn json(200, st.abliterationTax);}++ if (url.pathname === '/api/v1/exchange/market' && req.method === 'GET') {++ st.requests.push({ method: 'GET', path: url.pathname, query: Object.fromEntries(url.searchParams), body });++ if (st.marketStatus !== 200) return json(st.marketStatus, { code: 'stub.unavailable', detail: 'the market feed is unavailable' });++ return json(200, st.market);++ }++if (url.pathname === '/api/v1/routes' && req.method === 'GET') {st.requests.push({ method: 'GET', path: url.pathname, body });const [first, ...rest] = st.models;
@@ -2,6 +2,14 @@
@@ -2,6 +2,14 @@
## Unreleased
## Unreleased
+- **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, the
+ score of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each,
+ no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the
+ dry run: every call, the own-page price estimate and whether it fits the owner's daily cap. `POST
+ /api/v1/fidelity/run` runs one bounded pass; every call goes through OpenVibe.AI's normal generate path, pinned to
+ the exact offer, with a durable idempotency key per prompt and week. Off by default (`BENCH_FIDELITY=1`), daily
+ capped, and a weekly due-based job when on. The objective set is public at `GET /api/v1/fidelity/objective-set`;
+ answers are never stored, only which item passed.
- **The 18+ board (`chat.willing`).** A new adult kind and `GET /api/v1/boards/chat.willing`, marked `adult: true` on the board and every row. It reads OpenVibe.AI's measured refusal and hedge rates (n and Wilson intervals) on the published 60-prompt lawful adult set, plus the abliteration tax for each declared base/abliterated pair: the same objective set on both, with the score difference and its Newcombe interval. A non-willingness type on the adult kind is refused; when AI cannot answer, the board says the measurements are unavailable and invents no rows.
- **The 18+ board (`chat.willing`).** A new adult kind and `GET /api/v1/boards/chat.willing`, marked `adult: true` on the board and every row. It reads OpenVibe.AI's measured refusal and hedge rates (n and Wilson intervals) on the published 60-prompt lawful adult set, plus the abliteration tax for each declared base/abliterated pair: the same objective set on both, with the score difference and its Newcombe interval. A non-willingness type on the adult kind is refused; when AI cannot answer, the board says the measurements are unavailable and invents no rows.
- **Every Eval Ever (B2).** Bench imports the public EEE datastore (`github.com/evaleval/every_eval_ever`,
- **Every Eval Ever (B2).** Bench imports the public EEE datastore (`github.com/evaleval/every_eval_ever`,
`evaleval/EEE_datastore`) as *external* results: the model (joined by Report's rule — maker prefix and a date or
`evaleval/EEE_datastore`) as *external* results: the model (joined by Report's rule — maker prefix and a date or
@@ -245,6 +245,22 @@ The open dataset (`/api/v1/datasets/:kindId.jsonl`) also emits `type: "category"
@@ -245,6 +245,22 @@ The open dataset (`/api/v1/datasets/:kindId.jsonl`) also emits `type: "category"
`type: "holdout"` lines, and its rating lines carry the rank interval. The formulas, the 0.5/0.5 weights and the
`type: "holdout"` lines, and its rating lines carry the rank interval. The formulas, the 0.5/0.5 weights and the
30-game floor are in [index.md](index.md).
30-game floor are in [index.md](index.md).
+## Provider fidelity (stage B5)
+
+| Route | Who | What |
+|---|---|---|
+| `GET /api/v1/fidelity` | anyone | The latest measurement of every offer: `score` (pass rate), `n`, `interval` (95% Wilson), `compared_to`, `measured_at`, `prompt_set`. Cached 60 seconds. |
+| `GET /api/v1/fidelity/objective-set` | anyone | The fixed public set (`fidelity-2026-11-09.v1`): every prompt and the deterministic check it is graded by. |
+| `POST /api/v1/fidelity/plan` | operator token | The dry run: each offer, its calls, the own-page price estimate and whether it fits the daily cap. Spends nothing. |
+| `POST /api/v1/fidelity/run` | operator token | One bounded pass. Refused with `fidelity.disabled` until `BENCH_FIDELITY=1`; capped by `BENCH_FIDELITY_DAILY_CAP_USD` and `BENCH_FIDELITY_MAX_CALLS`. |
+| `GET /api/v1/fidelity/status` | operator token | Whether the switch is on and the last pass's state. |
+
+The measurement asks every enabled provider of an open-weight model the same ten objective items, answers only, and
+grades each answer with a published exact/JSON/regex check. A provider that quantizes, truncates the context or
+serves another checkpoint answers fewer correctly; that is the number beside its price. A caller never sees a rank
+or a winner. Scores travel to OpenVibe.AI with `GET /api/v1/exchange/market` / `catalog` as each offer's `fidelity`
+field (null when unmeasured). The dry run and the cap are the owner's switch: fidelity is off by default.
+
## The older API
## The older API
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
new file mode 100644
new file mode 100644
@@ -0,0 +1,41 @@
@@ -0,0 +1,41 @@
+-- phase: expand
+-- Provider fidelity (plan 2026-11-09, stage B5).
+--
+-- One row per measured offer and pass: the same open-weight model, served by one provider, asked the
+-- fixed objective set. The score is passes / answered items with its Wilson interval; the answers
+-- themselves are not kept, only which item passed. A provider that quantizes, truncates the context or
+-- serves a different checkpoint scores lower on the same set, and the public read shows the difference
+-- rather than a rank.
+--
+-- measured_at is when the pass ran; cost_usd is what the calls reported, so the daily cap can sum the
+-- day's spend without a second ledger. results keeps the per-item verdicts (id, passed, error) for audit.
+CREATE TABLE IF NOT EXISTS bench_fidelity_runs (
+ id text COLLATE "C" PRIMARY KEY,
+ objective_set text COLLATE "C" NOT NULL,
+ model text COLLATE "C" NOT NULL, -- the model the provider claims to serve
+ provider text COLLATE "C" NOT NULL,
+ provider_model text COLLATE "C" NOT NULL, -- the exact endpoint id that answered
+ n integer NOT NULL DEFAULT 0, -- answered items (errors are counted apart)
+ passed integer NOT NULL DEFAULT 0,
+ errors integer NOT NULL DEFAULT 0, -- items the provider did not answer
+ score double precision, -- passed / n, null when nothing was answered
+ ci_lower double precision, -- Wilson 95% interval
+ ci_upper double precision,
+ cost_usd double precision NOT NULL DEFAULT 0,
+ measured_at timestamptz NOT NULL,
+ measured_on date NOT NULL,
+ results jsonb NOT NULL DEFAULT '[]'::jsonb,
+ created_at timestamptz NOT NULL DEFAULT now()
+);
+CREATE INDEX IF NOT EXISTS bench_fidelity_latest_idx ON bench_fidelity_runs (provider, provider_model, measured_at DESC);
+CREATE INDEX IF NOT EXISTS bench_fidelity_day_idx ON bench_fidelity_runs (measured_at);
+
+-- The weekly job is due-based: its last pass survives restarts and deploys.
+CREATE TABLE IF NOT EXISTS bench_fidelity_state (
+ id text COLLATE "C" PRIMARY KEY CHECK (id = 'fidelity'),
+ last_run_at timestamptz,
+ last_status text COLLATE "C",
+ last_error text COLLATE "C",
+ updated_at timestamptz NOT NULL DEFAULT now()
+);
+INSERT INTO bench_fidelity_state (id) VALUES ('fidelity') ON CONFLICT (id) DO NOTHING;
@@ -92,6 +92,9 @@ async function createApp(opts = {}) {
@@ -92,6 +92,9 @@ async function createApp(opts = {}) {
// B2: the external EEE records and the exporter of Bench's own runs, plus the weekly import job.
// B2: the external EEE records and the exporter of Bench's own runs, plus the weekly import job.
const eee = require('./bench/eee').createEee({ s, registry, fetchImpl, log, config });
const eee = require('./bench/eee').createEee({ s, registry, fetchImpl, log, config });
const eeeSync = require('./bench/eee-sync').createEeeSync({ eee, config, log });
const eeeSync = require('./bench/eee-sync').createEeeSync({ eee, config, log });
+ // B5: the provider-fidelity measurements and their weekly pass (off unless BENCH_FIDELITY is on).
+ const fidelity = require('./bench/fidelity').createFidelity({ s, ai, config, clock: { now: opts.now || (() => Date.now()) }, log });
+ const fidelityJob = require('./bench/fidelity-job').createFidelityJob({ fidelity, config, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
// The signed-out voter (the cookie, the relay header and the daily caps) and the daily rounds job.
// The signed-out voter (the cookie, the relay header and the daily caps) and the daily rounds job.
const anon = createAnon({ s, config, log });
const anon = createAnon({ s, config, log });
@@ -114,7 +117,7 @@ async function createApp(opts = {}) {
@@ -114,7 +117,7 @@ async function createApp(opts = {}) {
const ctx = {
const ctx = {
config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,
config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,
anon, daily, variantEvidence, evidenceConfig, sharing,
anon, daily, variantEvidence, evidenceConfig, sharing,
- imports, mirror, eee, eeeSync,
+ imports, mirror, eee, eeeSync, fidelity, fidelityJob,
random: opts.random || Math.random,
random: opts.random || Math.random,
};
};
@@ -258,6 +261,7 @@ async function createApp(opts = {}) {
@@ -258,6 +261,7 @@ async function createApp(opts = {}) {
}
}
// The EEE routes come before the kind dataset route: /datasets/eee.jsonl would otherwise be read as kind 'eee'.
// The EEE routes come before the kind dataset route: /datasets/eee.jsonl would otherwise be read as kind 'eee'.
app.use(require('./http/eee').createEeeRoutes(ctx));
app.use(require('./http/eee').createEeeRoutes(ctx));
+ app.use(require('./http/fidelity').createFidelityRoutes(ctx));
app.use('/api/v1', createApi(ctx));
app.use('/api/v1', createApi(ctx));
app.use('/api/v1', createRegistryApi(ctx));
app.use('/api/v1', createRegistryApi(ctx));
app.use('/api/v1', createVariantEvidenceApi(ctx));
app.use('/api/v1', createVariantEvidenceApi(ctx));
@@ -202,17 +202,18 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
@@ -202,17 +202,18 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
return { run: r, ms: msOf(r, Date.now() - startedAt) };
return { run: r, ms: msOf(r, Date.now() - startedAt) };
}
}
- async function generate({ prompt, kind, slot, roundId, index }) {
+ async function generate({ prompt, kind, slot, roundId, index, system = null, maxTokens = null, temperature = 0.7, idempotencyKey = null, target = null }) {
if (!enabled) throw new BenchAiError(503, 'ai.not_configured', 'OpenVibe.AI is not configured for Bench.');
if (!enabled) throw new BenchAiError(503, 'ai.not_configured', 'OpenVibe.AI is not configured for Bench.');
const pinned = slot && slot.provider && slot.provider !== 'ai' ? { provider: slot.provider, model: slot.model || null } : null;
const pinned = slot && slot.provider && slot.provider !== 'ai' ? { provider: slot.provider, model: slot.model || null } : null;
const body = {
const body = {
prompt,
prompt,
- system: systemPrompt(kind),
- max_tokens: maxTokensFor(kind),
- temperature: 0.7,
+ system: system || systemPrompt(kind),
+ max_tokens: maxTokens == null ? maxTokensFor(kind) : maxTokens,
+ temperature,
options: { cache: false },
options: { cache: false },
...(pinned ? { route: pinned } : {}),
...(pinned ? { route: pinned } : {}),
- target: { service: 'bench', type: 'bench_output', id: `${roundId}_${index}`.slice(0, 128) },
+ ...(idempotencyKey ? { idempotency_key: idempotencyKey } : {}),
+ target: target || { service: 'bench', type: 'bench_output', id: String(idempotencyKey || `${roundId}_${index}`).slice(0, 128) },
};
};
const startedAt = Date.now();
const startedAt = Date.now();
const wait = Math.min(ai.waitMs, 60_000);
const wait = Math.min(ai.waitMs, 60_000);
@@ -226,6 +227,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
@@ -226,6 +227,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
input: { prompt: body.prompt, system: body.system, max_tokens: body.max_tokens, temperature: body.temperature },
input: { prompt: body.prompt, system: body.system, max_tokens: body.max_tokens, temperature: body.temperature },
options: body.options,
options: body.options,
...(body.route ? { route: body.route } : {}),
...(body.route ? { route: body.route } : {}),
+ ...(body.idempotency_key ? { idempotency_key: body.idempotency_key } : {}),
target: body.target,
target: body.target,
});
});
}
}
@@ -264,6 +266,12 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
@@ -264,6 +266,12 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
};
};
}
}
+ /** The market feed: every own-page model and provider offer, with its price and open-weights flag. */
+ async function market({ openWeights = true, limit = 500 } = {}) {
+ const query = `?limit=${Math.max(1, Math.min(500, Number(limit) || 500))}${openWeights ? '&open_weights=true' : ''}&kind=chat`;
+ return call('GET', `/api/v1/exchange/market${query}`);
+ }
+
/** The 18+ board's measured willingness rates, from OpenVibe.AI's public read API. */
/** The 18+ board's measured willingness rates, from OpenVibe.AI's public read API. */
async function willingness() {
async function willingness() {
return call('GET', '/api/v1/exchange/willingness');
return call('GET', '/api/v1/exchange/willingness');
@@ -274,7 +282,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
@@ -274,7 +282,7 @@ function createBenchAi({ config, fetchImpl = globalThis.fetch, log = console })
return call('GET', '/api/v1/exchange/abliteration-tax');
return call('GET', '/api/v1/exchange/abliteration-tax');
}
}
- return { enabled, slots, slotPlan, offeredModels, rankOffers, generate, systemPrompt, maxTokensFor, willingness, abliterationTax };
+ return { enabled, slots, slotPlan, offeredModels, rankOffers, generate, systemPrompt, maxTokensFor, market, willingness, abliterationTax };
}
}
module.exports = { createBenchAi, BenchAiError };
module.exports = { createBenchAi, BenchAiError };
new file mode 100644
new file mode 100644
@@ -0,0 +1,44 @@
@@ -0,0 +1,44 @@
+'use strict';
+
+/**
+ * The weekly provider-fidelity pass (stage B5). Off by default: the owner turns it on with BENCH_FIDELITY=1.
+ * The pass is due-based (the last pass is a row in the database), so the several deploys Bench gets in a
+ * week do not reset its clock. A pass that is already running is never started twice.
+ */
+function createFidelityJob({ fidelity, config, log = console }) {
+ let timer = null;
+ let running = false;
+
+ async function tick() {
+ if (!config.bench.fidelity.enabled) return null;
+ if (config.drill && !config.bench.fidelity.enabledInDrill) return null;
+ if (running) return null;
+ running = true;
+ try {
+ if (!(await fidelity.due())) return null;
+ return await fidelity.run({ trigger: 'weekly' });
+ } catch (err) {
+ log.warn('[OpenVibe.Bench] fidelity job:', (err && (err.detail || err.message)) || err);
+ return null;
+ } finally {
+ running = false;
+ }
+ }
+
+ function start() {
+ if (timer || !config.bench.fidelity.enabled) return null;
+ const run = () => tick().catch((err) => log.warn('[OpenVibe.Bench] fidelity job:', (err && err.message) || err));
+ timer = setInterval(run, config.bench.fidelity.tickMs);
+ if (timer.unref) timer.unref();
+ return timer;
+ }
+
+ function stop() {
+ if (timer) clearInterval(timer);
+ timer = null;
+ }
+
+ return { tick, start, stop };
+}
+
+module.exports = { createFidelityJob };
new file mode 100644
new file mode 100644
@@ -0,0 +1,176 @@
@@ -0,0 +1,176 @@
+'use strict';
+
+/**
+ * The fixed objective set used to measure provider fidelity (plan 2026-11-09, stage B5).
+ *
+ * Provider fidelity asks a different question from the pairwise boards: the same open-weight model is
+ * served by several providers, and the question is whether each one answers the *model's* task as well
+ * as the others. So the set is small, fixed, public and objective: every item has one answer a
+ * deterministic check can recognise, with no LLM judge and no human vote. A provider that quantizes the
+ * weights, changes the chat template, truncates the context or substitutes a different checkpoint scores
+ * lower on the same items, and the difference is what the fidelity field publishes.
+ *
+ * The version is part of every result and every idempotency key: changing the set changes the version,
+ * so two scores are never compared across sets. A score is a pass rate with a Wilson interval, not a
+ * ranking: the public read never names a winner.
+ */
+
+const OBJECTIVE_SET = 'fidelity-2026-11-09.v1';
+
+/** The system line every item is measured with: the same instructions for every provider. */
+const SYSTEM = [
+ 'You are answering a fixed objective benchmark.',
+ 'Follow the requested format exactly and return only the answer, with no explanation or extra text.',
+].join('\n');
+
+const ITEMS = Object.freeze([
+ {
+ id: 'arithmetic',
+ prompt: 'What is 37 multiplied by 24? Answer with the number only.',
+ max_tokens: 16,
+ check: { type: 'equals', value: '888' },
+ },
+ {
+ id: 'extraction',
+ prompt: 'Read this line: "region: fra1; status: ok". Answer with the region code only.',
+ max_tokens: 16,
+ check: { type: 'equals', value: 'fra1' },
+ },
+ {
+ id: 'json',
+ prompt: 'Return exactly this JSON object and nothing else: {"ok": true, "n": 3}',
+ max_tokens: 40,
+ check: { type: 'json', value: { ok: true, n: 3 } },
+ },
+ {
+ id: 'repeat',
+ prompt: 'Write the word aurora exactly four times in a row, separated by single spaces. Answer with the four words only.',
+ max_tokens: 24,
+ check: { type: 'equals', value: 'aurora aurora aurora aurora' },
+ },
+ {
+ id: 'reverse',
+ prompt: 'Spell the word "stressed" backwards. Answer with the reversed word only.',
+ max_tokens: 16,
+ check: { type: 'equals', value: 'desserts' },
+ },
+ {
+ id: 'count',
+ prompt: 'How many times does the letter s appear in the word "mississippi"? Answer with the number only.',
+ max_tokens: 16,
+ check: { type: 'equals', value: '4' },
+ },
+ {
+ id: 'symbol',
+ prompt: 'What is the chemical symbol for gold? Answer with the symbol only.',
+ max_tokens: 16,
+ check: { type: 'equals', value: 'Au' },
+ },
+ {
+ id: 'next-day',
+ prompt: 'If today is Wednesday, what day will it be three days from now? Answer with the day only.',
+ max_tokens: 16,
+ check: { type: 'equals', value: 'Saturday' },
+ },
+ {
+ id: 'long-context',
+ prompt: [
+ 'Read the note below to its last line, then answer the question.',
+ '',
+ ...Array.from({ length: 40 }, (_, i) => `Note line ${i + 1}: the archive keeps its records in order and never changes a line.`),
+ 'The access code is ORBIT-71.',
+ '',
+ 'Question: what is the access code? Answer with the code only.',
+ ].join('\n'),
+ max_tokens: 16,
+ check: { type: 'equals', value: 'ORBIT-71' },
+ },
+ {
+ id: 'format',
+ prompt: 'Reply with exactly the word READY and nothing else.',
+ max_tokens: 12,
+ check: { type: 'equals', value: 'READY' },
+ },
+]);
+
+/** The answer without the wrappers models like to add: a code fence, quotes, a final full stop. */
+function stripFence(text) {
+ const s = String(text == null ? '' : text).trim();
+ const fenced = /^```[a-z0-9_-]*\s*\n?([\s\S]*?)\n?```$/i.exec(s);
+ return fenced ? fenced[1] : s;
+}
+
+function normalize(text) {
+ return stripFence(text)
+ .trim()
+ .replace(/^["'`]+|["'`]+$/g, '')
+ .replace(/[.!]\s*$/, '')
+ .replace(/\s+/g, ' ')
+ .trim()
+ .toLowerCase();
+}
+
+function parseJson(text) {
+ const s = stripFence(text).trim();
+ const start = s.indexOf('{');
+ const end = s.lastIndexOf('}');
+ if (start < 0 || end <= start) return null;
+ try { return JSON.parse(s.slice(start, end + 1)); } catch { return null; }
+}
+
+const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
+
+/**
+ * One item's verdict. `passed` is the only thing stored; `reason` says what the check compared, so an
+ * operator can see a template change without reading the answer back.
+ */
+function check(item, answer) {
+ const got = String(answer == null ? '' : answer);
+ const c = item.check || {};
+ if (c.type === 'equals') {
+ const passed = normalize(got) === normalize(c.value);
+ return { passed, reason: passed ? 'exact' : 'not the expected answer' };
+ }
+ if (c.type === 'one_of') {
+ const n = normalize(got);
+ const passed = (c.values || []).some((v) => normalize(v) === n);
+ return { passed, reason: passed ? 'one of the expected answers' : 'not one of the expected answers' };
+ }
+ if (c.type === 'contains_all') {
+ const n = normalize(got);
+ const missing = (c.values || []).filter((v) => !n.includes(normalize(v)));
+ return { passed: missing.length === 0, reason: missing.length ? `missing ${missing.join(', ')}` : 'all parts present' };
+ }
+ if (c.type === 'json') {
+ const parsed = parseJson(got);
+ const passed = parsed != null && same(parsed, c.value);
+ return { passed, reason: passed ? 'the expected object' : 'not the expected JSON' };
+ }
+ if (c.type === 'regex') {
+ const passed = new RegExp(String(c.pattern), String(c.flags || 'i')).test(got);
+ return { passed, reason: passed ? 'matched' : 'did not match' };
+ }
+ return { passed: false, reason: 'unknown check' };
+}
+
+/** The input side of a cost estimate, from the text itself (a stable convention, not a billing number). */
+function promptTokens(item) {
+ return Math.max(1, Math.ceil((SYSTEM.length + String(item.prompt || '').length) / 4));
+}
+
+/** The public view of the set: what was asked, never the model's answer. */
+function publicSet() {
+ return {
+ objective_set: OBJECTIVE_SET,
+ items: ITEMS.length,
+ note: 'A fixed, public objective set: each item has one answer a deterministic check recognises. The same set is run through every provider that serves an open-weight model.',
+ prompts: ITEMS.map((item) => ({
+ id: item.id,
+ prompt: item.prompt,
+ max_tokens: item.max_tokens,
+ check: item.check,
+ })),
+ };
+}
+
+module.exports = { OBJECTIVE_SET, SYSTEM, ITEMS, normalize, check, promptTokens, publicSet };
new file mode 100644
new file mode 100644
@@ -0,0 +1,309 @@
@@ -0,0 +1,309 @@
+'use strict';
+
+/**
+ * Provider fidelity (plan 2026-11-09, stage B5).
+ *
+ * OpenVibe.AI routes; this module measures. For every popular open-weight model the network routes, it
+ * finds every enabled provider that serves that model in AI's public market feed, asks each offer the
+ * fixed objective set, and scores the answers deterministically. A provider that quantizes, truncates
+ * the context or serves another checkpoint answers fewer items correctly; the score is a pass rate with
+ * a Wilson interval and the sample size beside it, never a rank or a bare winner.
+ *
+ * Money: the owner's switch is off by default; a pass has a daily cap in US dollars and a call cap. The
+ * dry run lists every call and what the pricebook says it would cost before anything is spent. Every
+ * call goes through AI's normal generate path (its own reserve/settle accounting), pinned to the exact
+ * provider and model, with a durable idempotency key per prompt and week, so a retried pass is not
+ * charged twice. The answers never leave this service: the table stores only which item passed.
+ */
+const crypto = require('crypto');
+const { OBJECTIVE_SET, SYSTEM, ITEMS, check, promptTokens, publicSet } = require('./fidelity-set');
+
+const ERROR_MAX = 300;
+const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
+const round = (v) => (v == null ? null : Math.round(Number(v) * 1e6) / 1e6);
+// PostgreSQL returns `date` as a JS Date; PGlite returns the string. Both are the same day.
+const dayOf = (v) => (v instanceof Date ? v.toISOString().slice(0, 10) : String(v || '').slice(0, 10));
+
+class FidelityError extends Error {
+ constructor(status, code, detail) {
+ super(detail);
+ this.name = 'FidelityError';
+ this.status = status;
+ this.code = code;
+ this.detail = detail;
+ }
+}
+
+/** The Wilson score interval at 95%: an honest band around a pass rate with few items. */
+function wilson(passed, n, z = 1.96) {
+ if (!Number.isFinite(n) || n <= 0) return { lower: null, upper: null };
+ const p = Math.max(0, Math.min(1, passed / n));
+ const d = 1 + (z * z) / n;
+ const centre = (p + (z * z) / (2 * n)) / d;
+ const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
+ return { lower: Math.max(0, round(centre - half)), upper: Math.min(1, round(centre + half)) };
+}
+
+/** The Monday 00:00 UTC of the week a moment falls in, and the key the idempotency of a pass uses. */
+function weekOf(ms) {
+ const d = new Date(ms);
+ const day = (d.getUTCDay() + 6) % 7;
+ const start = Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate() - day);
+ return { start, key: new Date(start).toISOString().slice(0, 10) };
+}
+
+function startOfUtcDay(ms) {
+ const d = new Date(ms);
+ return Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate());
+}
+
+/** The planned cost of one item on one offer, from the market's own-page price. Null when unknown. */
+function estimateItem(item, price) {
+ const inPrice = num(price && price.in);
+ const outPrice = num(price && price.out);
+ if (inPrice == null || outPrice == null) return null;
+ return round(((promptTokens(item) / 1e6) * inPrice) + ((item.max_tokens / 1e6) * outPrice));
+}
+
+const idempotencyKey = (weekKey, provider, providerModel, itemId) => {
+ const digest = crypto.createHash('sha256')
+ .update(`${OBJECTIVE_SET}\0${weekKey}\0${provider}\0${providerModel}\0${itemId}`, 'utf8')
+ .digest('hex')
+ .slice(0, 40);
+ return `fid_${digest}`;
+};
+
+function keyOf(provider, providerModel) {
+ return `${String(provider || '').toLowerCase()}\0${String(providerModel || '').toLowerCase()}`;
+}
+
+/**
+ * The public read: the latest measurement of every offer, grouped by the model it serves. No row means
+ * nobody measured that offer; the field is absent there, never a zero.
+ */
+function createFidelity({ s, ai, config, clock = { now: () => Date.now() }, log = console } = {}) {
+ const cfg = (config && config.bench && config.bench.fidelity) || {};
+ const dailyCapUsd = Math.max(0, num(cfg.dailyCapUsd) == null ? 0 : num(cfg.dailyCapUsd));
+ const maxModels = Math.max(1, Number(cfg.maxModels) || 10);
+ const maxCalls = Math.max(1, Number(cfg.maxCallsPerRun) || 60);
+ const minAnswered = Math.max(1, Number(cfg.minAnswered) || Math.min(5, ITEMS.length));
+
+ const latest = s.db.prepare(`SELECT DISTINCT ON (provider, provider_model)
+ provider, provider_model, model, n, passed, errors, score, ci_lower, ci_upper, measured_at, measured_on, objective_set
+ FROM bench_fidelity_runs WHERE objective_set = ?
+ ORDER BY provider, provider_model, measured_at DESC, id DESC`);
+ const spent = s.db.prepare('SELECT COALESCE(SUM(cost_usd), 0) AS usd FROM bench_fidelity_runs WHERE measured_at >= ?');
+ const insert = s.db.prepare(`INSERT INTO bench_fidelity_runs
+ (id, objective_set, model, provider, provider_model, n, passed, errors, score, ci_lower, ci_upper, cost_usd, measured_at, measured_on, results)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?::timestamptz, ?::date, ?::jsonb)`);
+ const state = s.db.prepare(`UPDATE bench_fidelity_state SET last_run_at = ?::timestamptz, last_status = ?, last_error = ?,
+ updated_at = now() WHERE id = 'fidelity'`);
+ const stateRow = s.db.prepare("SELECT last_run_at, last_status, last_error FROM bench_fidelity_state WHERE id = 'fidelity'");
+
+ async function todaySpent() {
+ const row = await spent.get(new Date(startOfUtcDay(clock.now())).toISOString());
+ return Number(row && row.usd) || 0;
+ }
+
+ /** The public market: every own-page offer, enabled or not. A missing client is a plain 503. */
+ async function market() {
+ if (!ai || typeof ai.market !== 'function') throw new FidelityError(503, 'fidelity.unavailable', 'OpenVibe.AI market readings are not configured for Bench.');
+ let body;
+ try {
+ body = await ai.market({ openWeights: true, limit: 500 });
+ } catch (err) {
+ throw new FidelityError(503, 'fidelity.market_unavailable', `OpenVibe.AI's market feed is unavailable: ${String((err && (err.detail || err.message)) || err).slice(0, ERROR_MAX)}`);
+ }
+ const offers = Array.isArray(body && body.offers) ? body.offers : [];
+ if (!offers.length && body && body.error) throw new FidelityError(503, 'fidelity.market_unavailable', 'OpenVibe.AI returned no market offers.');
+ return offers;
+ }
+
+ /** The offers worth measuring: open weights, routed, chat, a real provider and endpoint id. */
+ function eligible(offers) {
+ const out = [];
+ for (const offer of offers) {
+ if (!offer || typeof offer !== 'object') continue;
+ if (offer.open_weights !== true || offer.routed !== true) continue;
+ if (offer.kind != null && offer.kind !== 'chat') continue;
+ const provider = String(offer.provider || '').trim();
+ const providerModel = String(offer.provider_model || offer.model || '').trim();
+ const model = String(offer.model || providerModel).trim();
+ if (!provider || !providerModel || !model) continue;
+ out.push({ model, provider, provider_model: providerModel, name: offer.name || providerModel, price: offer.price || null });
+ }
+ return out;
+ }
+
+ function groupByModel(offers) {
+ const groups = new Map();
+ for (const offer of offers) {
+ if (!groups.has(offer.model)) groups.set(offer.model, []);
+ groups.get(offer.model).push(offer);
+ }
+ return [...groups.entries()]
+ .map(([model, list]) => ({ model, offers: list.sort((a, b) => a.provider.localeCompare(b.provider) || a.provider_model.localeCompare(b.provider_model)) }))
+ // "Popular" means served by more providers first; the cap keeps a pass inside the owner's call budget.
+ .sort((a, b) => b.offers.length - a.offers.length || a.model.localeCompare(b.model))
+ .slice(0, maxModels);
+ }
+
+ /**
+ * What a pass would do, and what the pricebook says it would cost. Every offer says whether it is
+ * included and, when it is not, the plain reason: no price, a disabled switch, the daily cap or the
+ * call cap. The plan never spends anything.
+ */
+ async function plan() {
+ const available = eligible(await market());
+ const groups = groupByModel(available);
+ const spentUsd = await todaySpent();
+ const remaining = Math.max(0, dailyCapUsd - spentUsd);
+ let calls = 0;
+ let estimated = 0;
+ const offers = [];
+ for (const group of groups) {
+ for (const offer of group.offers) {
+ const estimates = ITEMS.map((item) => estimateItem(item, offer.price));
+ const offerEstimate = estimates.some((v) => v == null) ? null : round(estimates.reduce((a, b) => a + b, 0));
+ const offerCalls = ITEMS.length;
+ let included = true;
+ let reason = null;
+ if (offerEstimate == null) { included = false; reason = 'the market has no own-page price for this offer'; }
+ else if (calls + offerCalls > maxCalls) { included = false; reason = `the pass is capped at ${maxCalls} calls`; }
+ else if (estimated + offerEstimate > remaining) { included = false; reason = `the daily cap leaves $${round(remaining - estimated)}`; }
+ if (included) { calls += offerCalls; estimated = round(estimated + offerEstimate); }
+ offers.push({
+ model: group.model, provider: offer.provider, provider_model: offer.provider_model, name: offer.name,
+ price: offer.price, calls: offerCalls, estimated_usd: offerEstimate, included, reason,
+ });
+ }
+ }
+ return {
+ $schema: 'exchange.fidelity.plan@1',
+ dry_run: true,
+ generated_at: new Date(clock.now()).toISOString(),
+ objective_set: OBJECTIVE_SET,
+ items: ITEMS.length,
+ enabled: Boolean(cfg.enabled),
+ daily: { cap_usd: round(dailyCapUsd), spent_usd: round(spentUsd), remaining_usd: round(remaining) },
+ limits: { max_models: maxModels, max_calls: maxCalls },
+ models: groups.map((g) => ({ model: g.model, offers: g.offers.length })),
+ offers,
+ calls,
+ estimated_usd: estimated,
+ };
+ }
+
+ /** One bounded pass. Default off; returns what it measured, and never starts when nothing fits. */
+ async function run({ trigger = 'manual' } = {}) {
+ if (!cfg.enabled) throw new FidelityError(503, 'fidelity.disabled', 'Provider fidelity is off: the owner turns it on with BENCH_FIDELITY=1.');
+ const before = await plan();
+ const week = weekOf(clock.now());
+ const measuredAt = new Date(clock.now()).toISOString();
+ const measuredOn = measuredAt.slice(0, 10);
+ const ran = [];
+ let calls = 0;
+ let spentUsd = 0;
+ const included = before.offers.filter((offer) => offer.included);
+ for (const offer of included) {
+ if (calls >= maxCalls) break;
+ if (dailyCapUsd > 0 && spentUsd >= dailyCapUsd) break;
+ const results = [];
+ let passed = 0;
+ let answered = 0;
+ let errors = 0;
+ let cost = 0;
+ for (const item of ITEMS) {
+ if (calls >= maxCalls) break;
+ calls += 1;
+ try {
+ if (!ai || typeof ai.generate !== 'function') throw new FidelityError(503, 'fidelity.unavailable', 'AI generation is not configured for Bench.');
+ const out = await ai.generate({
+ prompt: item.prompt,
+ kind: 'text.answer',
+ slot: { provider: offer.provider, model: offer.provider_model },
+ system: SYSTEM,
+ maxTokens: item.max_tokens,
+ temperature: 0,
+ idempotencyKey: idempotencyKey(week.key, offer.provider, offer.provider_model, item.id),
+ });
+ cost += Number(out.cost_usd) || 0;
+ spentUsd += Number(out.cost_usd) || 0;
+ const verdict = check(item, out.html || out.text || '');
+ answered += 1;
+ if (verdict.passed) passed += 1;
+ results.push({ id: item.id, passed: verdict.passed, reason: verdict.reason, ms: out.ms == null ? null : out.ms });
+ } catch (err) {
+ errors += 1;
+ results.push({ id: item.id, error: String((err && (err.detail || err.code || err.message)) || err).slice(0, ERROR_MAX) });
+ }
+ }
+ if (answered < minAnswered) {
+ log.warn(`[OpenVibe.Bench] fidelity: ${offer.provider}/${offer.provider_model} answered ${answered} of ${ITEMS.length}; no score stored`);
+ continue;
+ }
+ const score = round(passed / answered);
+ const interval = wilson(passed, answered);
+ const id = s.newId('fid');
+ await insert.run(id, OBJECTIVE_SET, offer.model, offer.provider, offer.provider_model, answered, passed, errors,
+ score, interval.lower, interval.upper, round(cost), measuredAt, measuredOn, JSON.stringify(results));
+ ran.push({ model: offer.model, provider: offer.provider, provider_model: offer.provider_model, n: answered, passed, errors, score, interval });
+ }
+ await state.run(measuredAt, 'ok', null).catch((err) => log.warn('[OpenVibe.Bench] fidelity state:', (err && err.message) || err));
+ log.log(`[OpenVibe.Bench] fidelity ${trigger}: measured ${ran.length} offer(s), ${calls} call(s), $${round(spentUsd)}`);
+ return { dry_run: false, generated_at: new Date(clock.now()).toISOString(), objective_set: OBJECTIVE_SET, ran, calls, spent_usd: round(spentUsd) };
+ }
+
+ /** The latest score of every offer. This is the shape OpenVibe.AI reads to put fidelity on an offer. */
+ async function read() {
+ const rows = await latest.all(OBJECTIVE_SET);
+ const offers = (rows || []).map((row) => ({
+ model: row.model,
+ provider: row.provider,
+ provider_model: row.provider_model,
+ score: num(row.score),
+ passed: Number(row.passed) || 0,
+ n: Number(row.n) || 0,
+ errors: Number(row.errors) || 0,
+ interval: { lower: num(row.ci_lower), upper: num(row.ci_upper) },
+ compared_to: row.model,
+ prompt_set: row.objective_set,
+ measured_at: s.iso(row.measured_at),
+ measured_on: String(row.measured_on).slice(0, 10),
+ })).sort((a, b) => a.model.localeCompare(b.model) || a.provider.localeCompare(b.provider) || a.provider_model.localeCompare(b.provider_model));
+ const models = new Map();
+ for (const offer of offers) {
+ const m = models.get(offer.model) || { model: offer.model, providers: new Set(), offers: 0, best_score: null };
+ m.providers.add(offer.provider);
+ m.offers += 1;
+ if (offer.score != null && (m.best_score == null || offer.score > m.best_score)) m.best_score = offer.score;
+ models.set(offer.model, m);
+ }
+ return {
+ $schema: 'exchange.fidelity@1',
+ objective_set: OBJECTIVE_SET,
+ items: ITEMS.length,
+ measured_on: offers.length ? offers.map((o) => o.measured_on).sort().slice(-1)[0] : null,
+ offers,
+ models: [...models.values()].map((m) => ({ model: m.model, providers: m.providers.size, offers: m.offers, best_score: m.best_score })),
+ note: offers.length ? null : 'No offer has been measured on this objective set yet.',
+ };
+ }
+
+ /** Whether the weekly pass is due (the state row survives restarts, like the price book's). */
+ async function due() {
+ const interval = Math.max(3600000, Number(cfg.intervalMs) || 7 * 24 * 3600000);
+ const row = await stateRow.get();
+ if (!row || !row.last_run_at) return true;
+ return clock.now() - new Date(row.last_run_at).getTime() >= interval;
+ }
+
+ async function lastState() {
+ const row = await stateRow.get();
+ return row ? { last_run_at: s.iso(row.last_run_at), last_status: row.last_status || null, last_error: row.last_error || null } : null;
+ }
+
+ return { plan, run, read, due, lastState, objectiveSet: publicSet };
+}
+
+module.exports = { createFidelity, FidelityError, wilson, weekOf, estimateItem, idempotencyKey, keyOf, dayOf, OBJECTIVE_SET, ITEMS };
@@ -146,6 +146,19 @@ function load(env = process.env) {
@@ -146,6 +146,19 @@ function load(env = process.env) {
filesPerRun: Math.max(1, int(env.BENCH_EEE_FILES_PER_RUN, 5000)),
filesPerRun: Math.max(1, int(env.BENCH_EEE_FILES_PER_RUN, 5000)),
recordsPerRun: Math.max(1, int(env.BENCH_EEE_RECORDS_PER_RUN, 50000)),
recordsPerRun: Math.max(1, int(env.BENCH_EEE_RECORDS_PER_RUN, 50000)),
},
},
+ // B5 provider fidelity: one fixed objective set per open-weight model, every provider that
+ // serves it. Off until the owner turns it on; the daily cap is real money, and the dry run
+ // lists every call and its own-page price before anything is spent.
+ fidelity: {
+ enabled: bool(env.BENCH_FIDELITY, false),
+ enabledInDrill: bool(env.BENCH_FIDELITY_IN_DRILL, false),
+ dailyCapUsd: Math.max(0, float(env.BENCH_FIDELITY_DAILY_CAP_USD, 1.00)),
+ maxModels: Math.max(1, int(env.BENCH_FIDELITY_MAX_MODELS, 10)),
+ maxCallsPerRun: Math.max(1, int(env.BENCH_FIDELITY_MAX_CALLS, 60)),
+ minAnswered: Math.max(1, int(env.BENCH_FIDELITY_MIN_ANSWERED, 5)),
+ intervalMs: Math.max(3_600_000, int(env.BENCH_FIDELITY_INTERVAL_MS, 7 * 24 * 60 * 60 * 1000)),
+ tickMs: Math.max(60_000, int(env.BENCH_FIDELITY_TICK_MS, 60 * 60 * 1000)),
+ },
},
},
};
};
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,81 @@
@@ -0,0 +1,81 @@
+'use strict';
+
+/**
+ * Provider-fidelity routes (stage B5).
+ *
+ * GET /api/v1/fidelity the latest measurement of every offer (public)
+ * GET /api/v1/fidelity/objective-set the fixed objective set (public)
+ * POST /api/v1/fidelity/plan the dry run: every call and its estimated cost (operator)
+ * POST /api/v1/fidelity/run one bounded pass (operator; off unless the owner turned it on)
+ * GET /api/v1/fidelity/status the last pass, when it ran and what it said (operator)
+ *
+ * The read is public because it is the number OpenVibe.AI puts beside an offer and Report shows; it has
+ * no person and no prompt answer. Planning and running spend the owner's money, so they carry the same
+ * operator token as the other owner switches.
+ */
+const crypto = require('crypto');
+const { http } = require('openvibe-contracts');
+const { asyncRouter } = require('./router');
+const { FidelityError } = require('../bench/fidelity');
+
+function operatorToken(req) {
+ const h = String(req.get('authorization') || '');
+ return /^Bearer\s+\S/i.test(h) ? h.replace(/^Bearer\s+/i, '').trim() : null;
+}
+
+function tokenMatches(given, expected) {
+ if (!given || !expected) return false;
+ const a = crypto.createHash('sha256').update(String(given), 'utf8').digest();
+ const b = crypto.createHash('sha256').update(String(expected), 'utf8').digest();
+ return a.length === b.length && crypto.timingSafeEqual(a, b);
+}
+
+function createFidelityRoutes(ctx) {
+ const r = asyncRouter();
+ const fidelity = ctx.fidelity;
+ const token = process.env.BENCH_ADMIN_TOKEN || '';
+
+ const problem = (res, err) => {
+ if (err instanceof FidelityError) return http.sendProblem(res, err.status, err.code, { detail: err.detail, ctx: res.req && res.req.ov });
+ throw err;
+ };
+ const operator = (req, res) => {
+ if (!token) { res.status(404).type('text').send('No such route.'); return false; }
+ if (!tokenMatches(operatorToken(req), token)) { http.sendProblem(res, 403, 'fidelity.operator_required', { detail: 'The operator token is required.', ctx: req.ov }); return false; }
+ return true;
+ };
+
+ r.get('/api/v1/fidelity', async (_req, res) => {
+ const body = await fidelity.read();
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json(body);
+ });
+
+ r.get('/api/v1/fidelity/objective-set', (_req, res) => {
+ res.set('Cache-Control', 'public, max-age=3600');
+ return res.json(fidelity.objectiveSet());
+ });
+
+ r.get('/api/v1/fidelity/status', async (req, res) => {
+ if (!operator(req, res)) return;
+ return res.json({ enabled: Boolean(ctx.config.bench.fidelity.enabled), objective_set: require('../bench/fidelity-set').OBJECTIVE_SET, state: await fidelity.lastState() });
+ });
+
+ r.post('/api/v1/fidelity/plan', async (req, res) => {
+ if (!operator(req, res)) return;
+ try { return res.json(await fidelity.plan()); } catch (err) { return problem(res, err); }
+ });
+
+ r.post('/api/v1/fidelity/run', async (req, res) => {
+ if (!operator(req, res)) return;
+ const body = req.body && typeof req.body === 'object' ? req.body : {};
+ try {
+ if (body.dry_run === true) return res.json(await fidelity.plan());
+ return res.status(201).json(await fidelity.run({ trigger: 'operator' }));
+ } catch (err) { return problem(res, err); }
+ });
+
+ return r;
+}
+
+module.exports = { createFidelityRoutes, operatorToken, tokenMatches };
@@ -17,6 +17,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
@@ -17,6 +17,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
() => ctx.jobs.stop(),
() => ctx.jobs.stop(),
() => ctx.daily.stop(),
() => ctx.daily.stop(),
() => ctx.eeeSync.stop(),
() => ctx.eeeSync.stop(),
+ () => ctx.fidelityJob.stop(),
() => ctx.publisher.stop(),
() => ctx.publisher.stop(),
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
() => ctx.s.close(),
() => ctx.s.close(),
@@ -48,6 +49,8 @@ async function start() {
@@ -48,6 +49,8 @@ async function start() {
ctx.daily.start();
ctx.daily.start();
// The weekly Every Eval Ever import (B2). Off until BENCH_EEE_SYNC is on; bounded and resumable.
// The weekly Every Eval Ever import (B2). Off until BENCH_EEE_SYNC is on; bounded and resumable.
ctx.eeeSync.start();
ctx.eeeSync.start();
+ // The weekly provider-fidelity pass (B5). Off until BENCH_FIDELITY is on; due-based and capped.
+ ctx.fidelityJob.start();
// Outbox publisher → OpenVibe.Events.
// Outbox publisher → OpenVibe.Events.
ctx.publisher.start();
ctx.publisher.start();
new file mode 100644
new file mode 100644
@@ -0,0 +1,174 @@
@@ -0,0 +1,174 @@
+'use strict';
+
+// Provider fidelity (stage B5): the fixed objective set, the dry-run budget, one bounded pass through
+// AI's generate path with a durable idempotency key per prompt, and the public read OpenVibe.AI attaches
+// to an offer. The fake AI serves the market fixture and answers each prompt; no test reaches the network.
+process.env.BENCH_ADMIN_TOKEN = process.env.BENCH_ADMIN_TOKEN || 'op-secret';
+
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const { OBJECTIVE_SET, ITEMS, check: checkItem } = require('../server/bench/fidelity-set');
+const { createFidelity, wilson, weekOf, estimateItem } = require('../server/bench/fidelity');
+
+const MARKET = {
+ $schema: 'exchange.market@1',
+ offers: [
+ { model: 'acme/alpha-1', provider: 'acme', provider_model: 'acme/alpha-1', kind: 'chat', open_weights: true, routed: true, price: { in: 100, out: 200 } },
+ { model: 'acme/alpha-1', provider: 'beta', provider_model: 'beta/alpha-1', kind: 'chat', open_weights: true, routed: true, price: { in: 100, out: 200 } },
+ { model: 'acme/alpha-1', provider: 'gamma', provider_model: 'gamma/alpha-1', kind: 'chat', open_weights: true, routed: false, price: { in: 100, out: 200 } },
+ { model: 'acme/closed-1', provider: 'acme', provider_model: 'acme/closed-1', kind: 'chat', open_weights: false, routed: true, price: { in: 100, out: 200 } },
+ { model: 'acme/unpriced-1', provider: 'acme', provider_model: 'acme/unpriced-1', kind: 'chat', open_weights: true, routed: true, price: { in: null, out: null } },
+ ],
+};
+
+function answer(body) {
+ const prompt = String(body.prompt || '');
+ if (prompt.startsWith('What is 37')) return '888';
+ if (prompt.startsWith('Read this line')) return 'fra1';
+ if (prompt.startsWith('Return exactly this JSON')) return '{"ok": true, "n": 3}';
+ if (prompt.startsWith('Write the word aurora')) return 'aurora aurora aurora aurora';
+ if (prompt.startsWith('Spell the word')) return 'desserts';
+ if (prompt.startsWith('How many times')) return '4';
+ if (prompt.startsWith('What is the chemical symbol')) return 'Au';
+ if (prompt.startsWith('If today is Wednesday')) return 'Saturday';
+ if (prompt.startsWith('Read the note')) return 'ORBIT-71';
+ if (prompt.startsWith('Reply with exactly')) return 'READY';
+ return 'unknown';
+}
+
+(async () => {
+ let t;
+ try {
+ t = await boot({
+ aiOptions: {
+ models: [
+ { provider: 'acme', model: 'acme/alpha-1' },
+ { provider: 'beta', model: 'beta/alpha-1' },
+ ],
+ market: MARKET,
+ // beta is a different quantization of the same model: it answers only three items right.
+ textFor: (body, pick) => {
+ const right = answer(body);
+ if (pick.provider === 'beta') {
+ const prompt = String(body.prompt || '');
+ if (prompt.startsWith('What is 37')) return 'not a number';
+ if (prompt.startsWith('Read this line')) return 'fra1';
+ if (prompt.startsWith('What is the chemical symbol')) return 'Au';
+ if (prompt.startsWith('Reply with exactly')) return 'READY';
+ return 'unknown';
+ }
+ return right;
+ },
+ },
+ env: {
+ BENCH_FIDELITY: '1',
+ BENCH_FIDELITY_IN_DRILL: '1',
+ BENCH_FIDELITY_DAILY_CAP_USD: '1.00',
+ BENCH_FIDELITY_MAX_CALLS: '30',
+ BENCH_FIDELITY_MIN_ANSWERED: '3',
+ },
+ });
+
+ await check('the objective set has one deterministic check per item and a version', () => {
+ assert.strictEqual(OBJECTIVE_SET, 'fidelity-2026-11-09.v1');
+ assert(ITEMS.length >= 8);
+ assert.strictEqual(new Set(ITEMS.map((i) => i.id)).size, ITEMS.length, 'item ids are unique');
+ assert(checkItem(ITEMS.find((i) => i.id === 'arithmetic'), '```\n888\n```').passed);
+ assert(checkItem(ITEMS.find((i) => i.id === 'json'), '{"ok":true,"n":3}').passed);
+ assert(!checkItem(ITEMS.find((i) => i.id === 'arithmetic'), 'the answer is 888').passed);
+ });
+
+ await check('the fidelity interval is a Wilson score interval', () => {
+ const perfect = wilson(10, 10);
+ assert.strictEqual(perfect.upper, 1);
+ assert(perfect.lower > 0.6 && perfect.lower < 1);
+ const none = wilson(0, 10);
+ assert.strictEqual(none.lower, 0);
+ assert(none.upper > 0 && none.upper < 0.4);
+ assert.deepStrictEqual(wilson(0, 0), { lower: null, upper: null });
+ });
+
+ await check('the call estimate comes from the offer price and the item text', () => {
+ const item = ITEMS.find((i) => i.id === 'arithmetic');
+ const cost = estimateItem(item, { in: 100, out: 200 });
+ assert(cost > 0 && cost < 1);
+ assert.strictEqual(estimateItem(item, { in: null, out: 200 }), null);
+ assert.strictEqual(weekOf(Date.parse('2026-11-11T12:00:00Z')).key, '2026-11-09');
+ });
+
+ await check('open weights, routed and priced only; the dry run lists every call', async () => {
+ const r = await t.get('/api/v1/fidelity/plan', { method: 'POST', bearer: 'op-secret', json: {} });
+ assert.strictEqual(r.status, 200, r.text);
+ const plan = r.json();
+ assert.strictEqual(plan.objective_set, OBJECTIVE_SET);
+ assert.strictEqual(plan.offers.filter((o) => o.included).length, 2, 'the two enabled providers of the open-weight model');
+ assert.strictEqual(plan.calls, ITEMS.length * 2);
+ assert(plan.estimated_usd > 0);
+ assert(plan.daily.cap_usd === 1);
+ const unpriced = plan.offers.find((o) => o.provider_model === 'acme/unpriced-1');
+ assert(unpriced && unpriced.included === false && /price/.test(unpriced.reason), 'an offer without a price is skipped, not guessed');
+ assert(!/closed-1|gamma/.test(r.text), 'closed and unrouted offers are out');
+ assert.strictEqual(t.ai.requests.filter((x) => x.path === '/api/v1/generate').length, 0, 'a dry run spends nothing');
+ });
+
+ await check('a pass answers every item through AI and scores each offer', async () => {
+ const r = await t.get('/api/v1/fidelity/run', { method: 'POST', bearer: 'op-secret', json: {} });
+ assert.strictEqual(r.status, 201, r.text);
+ const out = r.json();
+ assert.strictEqual(out.ran.length, 2);
+ const acme = out.ran.find((o) => o.provider === 'acme');
+ const beta = out.ran.find((o) => o.provider === 'beta');
+ assert.strictEqual(acme.score, 1);
+ assert.strictEqual(acme.n, ITEMS.length);
+ assert.strictEqual(beta.score, 0.3);
+ assert.strictEqual(beta.passed, 3);
+ const generated = t.ai.requests.filter((x) => x.path === '/api/v1/generate');
+ assert.strictEqual(generated.length, ITEMS.length * 2);
+ assert(generated.every((x) => /^fid_[a-f0-9]{40}$/.test(x.body.idempotency_key)), 'every prompt has a durable key');
+ assert(generated.every((x) => x.body.route && x.body.route.model && x.body.route.provider), 'every call is pinned to the exact offer');
+ assert(generated.every((x) => x.body.temperature === 0), 'the objective set is not sampled');
+ });
+
+ await check('the public read puts score, n, interval and compared_to beside every offer', async () => {
+ const r = await t.get('/api/v1/fidelity');
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.headers.get('cache-control'), 'public, max-age=60');
+ const body = r.json();
+ assert.strictEqual(body.$schema, 'exchange.fidelity@1');
+ assert.strictEqual(body.offers.length, 2);
+ assert.strictEqual(body.measured_on, new Date(t.ctx.s.now()).toISOString().slice(0, 10));
+ for (const offer of body.offers) {
+ assert.strictEqual(offer.compared_to, 'acme/alpha-1');
+ assert(Number.isFinite(offer.score));
+ assert.strictEqual(offer.n, ITEMS.length);
+ assert(offer.interval.lower <= offer.score && offer.score <= offer.interval.upper);
+ assert(offer.measured_at);
+ }
+ const meta = await t.get('/api/v1/fidelity/objective-set');
+ assert.strictEqual(meta.status, 200);
+ assert.strictEqual(meta.json().prompts.length, ITEMS.length);
+ });
+
+ await check('the operator token is required and the status names the last pass', async () => {
+ assert.strictEqual((await t.get('/api/v1/fidelity/plan', { method: 'POST', json: {} })).status, 403);
+ assert.strictEqual((await t.get('/api/v1/fidelity/run', { method: 'POST', bearer: 'wrong', json: {} })).status, 403);
+ const status = await t.get('/api/v1/fidelity/status', { bearer: 'op-secret' });
+ assert.strictEqual(status.status, 200);
+ assert.strictEqual(status.json().state.last_status, 'ok');
+ });
+
+ await check('the pass is off unless the owner turns it on', async () => {
+ const off = createFidelity({
+ s: { db: { prepare: () => ({ get: async () => null, all: async () => [], run: async () => null }) }, now: () => Date.now(), iso: (v) => String(v), newId: () => 'fid_test' },
+ ai: { market: async () => MARKET },
+ config: { bench: { fidelity: { enabled: false, dailyCapUsd: 1 } } },
+ });
+ await assert.rejects(() => off.run(), (err) => err.code === 'fidelity.disabled' && err.status === 503);
+ const plan = await off.plan();
+ assert.strictEqual(plan.enabled, false);
+ });
+ } finally {
+ if (t) await t.close();
+ }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
@@ -64,8 +64,8 @@ function listen(handler) {
@@ -64,8 +64,8 @@ function listen(handler) {
});
});
}
}
-async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, tokensIn = 120, tokensOut = 480, textFor = null, willingness = DEFAULT_WILLINGNESS, abliterationTax = DEFAULT_TAX, willingnessStatus = 200, abliterationTaxStatus = 200 } = {}) {
- const st = { models: models.map((m) => ({ ...m })), requests: [], runCount: 0, runs: new Map(), costUsd, ms, tokensIn, tokensOut, textFor, willingness, abliterationTax, willingnessStatus, abliterationTaxStatus };
+async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, tokensIn = 120, tokensOut = 480, textFor = null, market = { offers: [] }, marketStatus = 200, willingness = DEFAULT_WILLINGNESS, abliterationTax = DEFAULT_TAX, willingnessStatus = 200, abliterationTaxStatus = 200 } = {}) {
+ const st = { models: models.map((m) => ({ ...m })), requests: [], runCount: 0, runs: new Map(), costUsd, ms, tokensIn, tokensOut, textFor, market, marketStatus, willingness, abliterationTax, willingnessStatus, abliterationTaxStatus };
function modelFor(body) {
function modelFor(body) {
const asked = body && body.route;
const asked = body && body.route;
@@ -103,6 +103,12 @@ async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, to
@@ -103,6 +103,12 @@ async function startAi({ models = DEFAULT_MODELS, costUsd = 0.0012, ms = 321, to
return json(200, st.abliterationTax);
return json(200, st.abliterationTax);
}
}
+ if (url.pathname === '/api/v1/exchange/market' && req.method === 'GET') {
+ st.requests.push({ method: 'GET', path: url.pathname, query: Object.fromEntries(url.searchParams), body });
+ if (st.marketStatus !== 200) return json(st.marketStatus, { code: 'stub.unavailable', detail: 'the market feed is unavailable' });
+ return json(200, st.market);
+ }
+
if (url.pathname === '/api/v1/routes' && req.method === 'GET') {
if (url.pathname === '/api/v1/routes' && req.method === 'GET') {
st.requests.push({ method: 'GET', path: url.pathname, body });
st.requests.push({ method: 'GET', path: url.pathname, body });
const [first, ...rest] = st.models;
const [first, ...rest] = st.models;