openvibers/openvibe.bench Commits
D11 owner settings: bench.variant_evidence (#121)
Changes
@@@ -137,6 +137,98 @@ granted is refused with `bench.clip_source_denied` and nothing is stored.(`GET /api/v1/judgements/next?kind=live.clip`) and on the battle page, and never once `expires_at` has passed: anexpired clip is shown as expired and its link is not served.++## Model-variant evidence (P6)++++P6 records measured throughput, TTFT and cold-start evidence per **exact model id**, grouped into variant families.++It is Bench's evaluation evidence for a placement: Bench still never runs inference, and nothing here starts a run.++The whole surface is behind `BENCH_VARIANT_EVIDENCE` and is **off** by default; with the flag off these routes do not++exist (a 404 with `route.not_found`).++++An exact id is the D3 form `<org>/<model>[:<quant>]@<revision>`, and the quant and the revision are optional:++++```++qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9++qwen/qwen3-9b-abliterated:q4_k_m++qwen/qwen3-9b-abliterated@main++qwen/qwen3-9b-abliterated++```++++The **family** is `<org>/<model>`, so the four ids above are one family with two quants and two revisions.++++| Route | Who | What |++|---|---|---|++| `GET /api/v1/variant-families` | anyone | Every family: its exact ids (variants), quants, revisions and sample counts. |++| `GET /api/v1/variant-families/:family` | anyone | One family; `:family` is URL-encoded (`qwen%2Fqwen3-9b-abliterated`). |++| `GET /api/v1/variant-evidence` | anyone | Every exact id's evidence; `?family=` filters one family. |++| `GET /api/v1/variant-evidence/:exactId` | anyone | One exact id; `:exactId` is URL-encoded. |++| `POST /api/v1/variant-evidence` | a service token | Record one observation from a pinned round or an authenticated run. |++++A write is accepted only from a service token, never from a person, an anonymous caller or a model's own agent:++++- **A pinned round** (`source.kind: "round"`, `source.id: "brd_…"`) needs `bench.entry.create`. The round must be++ `finished` and carry its receipt; a queued or failed round is refused.++- **A run** (`source.kind: "run"`) needs `bench.run.publish`. A Bench run (`brn_…`) must be `verified`; a Run job++ (`job_…`) or AI run (`run_…`) is recorded as an authenticated run. A submitted run is refused.++- A token whose identity names the model (for example `agent:qwen/qwen3-9b-abliterated`) is refused with++ `bench.self_rating`: a model never rates itself.++++```json++{++ "exact_id": "qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9",++ "source": { "kind": "round", "id": "brd_0123456789ABCDEFGHJKMNPQRS" },++ "measured_at": "2026-11-04T10:00:00.000Z",++ "sample_count": 120,++ "success_count": 118,++ "timeout_count": 2,++ "ttft_p50_ms": 210,++ "ttft_p95_ms": 480,++ "prefill_tokens_per_s": 1420,++ "decode_tokens_per_s": 86.5,++ "cold_start_ms": 5300++}++```++++A rate may be sent instead of its count (`success_rate`, `timeout_rate`, each 0 to 1); it is converted with the++sample count. `sample_count` is required. Every other number is optional, and a missing one stays `null` — it is never++guessed or filled in from the family or another revision.++++The read returns the observations and their totals. The newest observation's metrics are also on the variant for++convenience, and `observations` keeps each source's own numbers:++++```json++{++ "evidence": {++ "exact_id": "qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9",++ "family": "qwen/qwen3-9b-abliterated",++ "quant": "q4_k_m",++ "revision": "3f2a1c9",++ "sample_count": 120,++ "success_rate": 0.983333,++ "timeout_rate": 0.016667,++ "confidence": {++ "level": "high",++ "sparse": false,++ "samples": 120,++ "minimum_samples": 10,++ "success_rate": { "estimate": 0.983333, "lower": 0.9412, "upper": 0.9965 }++ },++ "metrics": { "ttft_p50_ms": 210, "ttft_p95_ms": 480, "decode_tokens_per_s": 86.5 },++ "ttft_p50_ms": 210,++ "ttft_p95_ms": 480,++ "provenance": { "kind": "round", "id": "brd_…", "receipt": "…" },++ "observations": [ { "sample_count": 120, "metrics": { "…": "…" }, "provenance": { "…": "…" } } ]++ }++}++```++++`confidence.level` is `sparse` under 10 samples, `low` from 10, `moderate` from 30 and `high` from 100;++`sparse: true` is always explicit. The success and timeout intervals are Wilson bounds. A rate across several++observations is returned only when every observation counts that outcome; a missing count is not a zero.++++**Private sets stay private.** The evidence table has no prompt, output, payload or transcript column, the write++ignores any such fields, and the read routes never join a task, benchmark or entry. The only text in a row is an++exact id, a family, and the id of the round or run that produced it.++## The older APIBench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private@@@ -119,3 +119,15 @@ GET /api/v1/percentile?benchmark=&model=&score=`value` is the score per dollar, `speed` the score per minute, and `consistency` the spread of a model's scoresinside one session (the lower, the steadier). The percentile says where a score sits among that model's runs.++++## Variants and measured evidence (P6, flag off)++++A model is not one id. A **variant family** groups the exact ids of one `<org>/<model>` across quants and revisions++(`qwen/qwen3-9b-abliterated:q4_k_m@abc` and `qwen/qwen3-9b-abliterated:q8_0@def` are one family, two variants). Bench++records typed measurements per exact id — TTFT p50/p95, prefill and decode tokens/s, cold start, success and timeout++counts — and returns each with its sample count and confidence. Sparse evidence says so; it is never filled in from a++sibling revision or the family.++++The evidence is written only by a **finished pinned round** or an authenticated **run**, never by a model about++itself, and it carries no prompt or output text. The API is in [api.md](api.md#model-variant-evidence-p6). The whole++surface is behind `BENCH_VARIANT_EVIDENCE` and is off by default.nnew file mode 100644@@@ -0,0 +1,48 @@++-- phase: expand++-- P6 (plan 2026-11-04, G-P6 and ObservedPerformance): measured model-variant evidence.++--++-- One row is one observation batch for one exact id from one source. The exact id is the D3 form++-- <org>/<model>[:<quant>]@<revision>; the family is <org>/<model>, so the variants of one model group across++-- quants and revisions. The source is a pinned Bench round or an authenticated Run job / verified Bench run.++-- Nothing here is prompt or output text: only identifiers, counts and measurements are stored, so a private++-- benchmark's prompt can never leak through this table or its read API.++++CREATE TABLE bench_variant_evidence (++ id text COLLATE "C" PRIMARY KEY, -- bve_ + ULID++ exact_id text COLLATE "C" NOT NULL, -- org/model[:quant]@revision++ family text COLLATE "C" NOT NULL, -- org/model++ quant text COLLATE "C",++ revision text COLLATE "C",++ source_kind text COLLATE "C" NOT NULL, -- round | run++ source_id text COLLATE "C" NOT NULL, -- brd_…, brn_…, job_… or run_…++ provenance jsonb NOT NULL DEFAULT '{}'::jsonb,++ measured_at timestamptz NOT NULL,++ sample_count integer NOT NULL,++ success_count integer,++ timeout_count integer,++ ttft_p50_ms integer,++ ttft_p95_ms integer,++ prefill_tokens_per_s numeric(14, 3),++ decode_tokens_per_s numeric(14, 3),++ cold_start_ms integer,++ recorded_by text COLLATE "C" NOT NULL,++ created_at timestamptz NOT NULL DEFAULT now(),++ CONSTRAINT bench_variant_evidence_id CHECK (id ~ '^bve_[0-9A-HJKMNP-TV-Z]{26}$'),++ CONSTRAINT bench_variant_evidence_exact CHECK (length(exact_id) BETWEEN 3 AND 200),++ CONSTRAINT bench_variant_evidence_family CHECK (length(family) BETWEEN 3 AND 200),++ CONSTRAINT bench_variant_evidence_source CHECK (source_kind IN ('round', 'run')),++ CONSTRAINT bench_variant_evidence_source_id CHECK (length(source_id) BETWEEN 3 AND 200),++ CONSTRAINT bench_variant_evidence_samples CHECK (sample_count > 0),++ CONSTRAINT bench_variant_evidence_success CHECK (success_count IS NULL OR (success_count >= 0 AND success_count <= sample_count)),++ CONSTRAINT bench_variant_evidence_timeout CHECK (timeout_count IS NULL OR (timeout_count >= 0 AND timeout_count <= sample_count)),++ CONSTRAINT bench_variant_evidence_counts CHECK (success_count IS NULL OR timeout_count IS NULL OR success_count + timeout_count <= sample_count),++ CONSTRAINT bench_variant_evidence_ttft_p50 CHECK (ttft_p50_ms IS NULL OR ttft_p50_ms >= 0),++ CONSTRAINT bench_variant_evidence_ttft_p95 CHECK (ttft_p95_ms IS NULL OR ttft_p95_ms >= 0),++ CONSTRAINT bench_variant_evidence_ttft_order CHECK (ttft_p50_ms IS NULL OR ttft_p95_ms IS NULL OR ttft_p95_ms >= ttft_p50_ms),++ CONSTRAINT bench_variant_evidence_prefill CHECK (prefill_tokens_per_s IS NULL OR prefill_tokens_per_s >= 0),++ CONSTRAINT bench_variant_evidence_decode CHECK (decode_tokens_per_s IS NULL OR decode_tokens_per_s >= 0),++ CONSTRAINT bench_variant_evidence_cold CHECK (cold_start_ms IS NULL OR cold_start_ms >= 0),++ UNIQUE (exact_id, source_kind, source_id)++);++CREATE INDEX bench_variant_evidence_family ON bench_variant_evidence (family, measured_at DESC);++CREATE INDEX bench_variant_evidence_exact ON bench_variant_evidence (exact_id, measured_at DESC);nnew file mode 100644@@@ -0,0 +1,28 @@++-- phase: expand++-- D11 owner settings: openvibe-shared/config (config_snapshots, config_keys). Applied migrations are immutable.++CREATE TABLE IF NOT EXISTS config_snapshots (++ namespace text COLLATE "C" NOT NULL,++ revision bigint NOT NULL,++ service text COLLATE "C" NOT NULL,++ previous_revision bigint,++ state text COLLATE "C" NOT NULL CHECK (state IN ('proposed', 'active', 'superseded', 'rejected', 'rolled_back')),++ values_json text COLLATE "C" NOT NULL, -- the full values; secret-class values are envelopes (never their plaintext)++ classification_json text COLLATE "C" NOT NULL,++ values_checksum text COLLATE "C" NOT NULL, -- sha256 of the values as served (secrets as markers)++ created_at text COLLATE "C" NOT NULL,++ created_by text COLLATE "C" NOT NULL,++ activated_at text COLLATE "C",++ activated_by text COLLATE "C",++ reason text COLLATE "C",++ error text COLLATE "C",++ copied_from bigint,++ good bigint NOT NULL DEFAULT 0, -- 1 once it activated successfully (last-known-good)++ PRIMARY KEY (namespace, revision)++);++CREATE UNIQUE INDEX IF NOT EXISTS config_snapshots_one_active ON config_snapshots (namespace) WHERE state = 'active';++-- The fingerprint key of each namespace. Never served, never logged.++CREATE TABLE IF NOT EXISTS config_keys (++ namespace text COLLATE "C" PRIMARY KEY,++ hmac_key bytea NOT NULL, -- the 32-byte fingerprint key, or that key wrapped under the current KEK++ created_at text COLLATE "C" NOT NULL++);@@@ -15,9 +15,9 @@"express-rate-limit": "^7.5.0","helmet": "^8.1.0","iovalkey": "^0.4.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.247.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.288.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.110.0","pg": "^8.23.0","three": "0.170.0"},@@@ -1151,9 +1151,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.247.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.247.0",-- "integrity": "sha512-ZBPPRdIygPqwENt99cMaJ4K2/yN4go9wQ6WTzjRVmdejT5qWij2CNLr3aSRRt+C83Do0qgGAO0469F6ONRa/Ng==",++ "version": "0.288.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.288.0",++ "integrity": "sha512-AKRP1yXACYNu7okpf4kCDq4ds0PhsoNvD92VlvBWBCjcSlNimSTQhLWIvSa3D31/kQYz91fodo06ZnwuTanOqg==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -1195,9 +1195,9 @@}},"node_modules/openvibe-shared": {-- "version": "4.109.2",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",-- "integrity": "sha512-t9K1yHr6GOJsfamqmgERRTTtiWYWAVmjWBEm4ykTcYjEenbTiqQTwX6CJsZW4lYnqS1IlqEvn+0jG4vXmHSJ3Q==",++ "version": "4.110.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.110.0",++ "integrity": "sha512-ZNOSraSgE89jhLTI//lFJALvKCiBJIBwf+V0aCkiIk0VCrPPAytsfsgOgCINU8kXDVdCEnuWWfcA9hZUc54QVA==","license": "MIT","dependencies": {"jsonwebtoken": "^9.0.0",@@@ -22,9 +22,9 @@"express-rate-limit": "^7.5.0","helmet": "^8.1.0","iovalkey": "^0.4.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.247.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.288.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.110.0","pg": "^8.23.0","three": "0.170.0"},@@@ -24,8 +24,10 @@ const { openStore } = require('./db');const { createKeyStore } = require('./auth/keys');const { createSso } = require('./auth/sso');const { createPrincipal } = require('./http/principal');++const createVariantEvidenceConfig = require('./variant-evidence-config').create;const { createApi } = require('./http/api');const { createRegistryApi } = require('./http/registry');++const { createVariantEvidenceApi } = require('./http/variant-evidence');const { createPageRoutes } = require('./http/pages');const { createShowcaseRoutes, createShowcasePages } = require('./http/showcase');const { createGalleryRoutes, createGalleryPages } = require('./http/gallery');@@@ -74,6 +76,9 @@ async function createApp(opts = {}) {const keys = createKeyStore({ config, fetchImpl, log });const sso = createSso({ config, keys, fetchImpl, now: opts.now || (() => Date.now()), log });const principal = createPrincipal({ config, keys });++ // D11 owner settings (bench.variant_evidence): the switch is read through the store, env pins winning.++ const evidenceConfig = await createVariantEvidenceConfig({ db: s.db, principal, config, env: opts.env || process.env, log });++ Object.defineProperty(config.bench, 'variantEvidence', { enumerable: true, configurable: true, get: () => evidenceConfig.get('enabled') });const bench = opts.bench || createStore(s, { config, log, random: opts.random });const registry = opts.registry || createRegistry(s, { log });@@@ -97,9 +102,11 @@ async function createApp(opts = {}) {: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Bench cannot answer account events'); });const launcher = createLauncher({ config, registry, fetchImpl: fetchImpl || globalThis.fetch, log });++ // P6: model-variant evidence. Constructed always so the owner switch applies live; its API returns 404 while off.++ const variantEvidence = require('./bench/variant-evidence').createVariantEvidence({ s, log });const ctx = {config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,-- anon, daily,++ anon, daily, variantEvidence, evidenceConfig,imports, mirror,random: opts.random || Math.random,};@@@ -148,6 +155,8 @@ async function createApp(opts = {}) {const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);ctx.valkey = valkey;++ evidenceConfig.mount(app);++app.use(contracts.http.middleware());app.use(helmet({contentSecurityPolicy: {@@@ -230,6 +239,7 @@ async function createApp(opts = {}) {app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.BENCH_API_RATE_LIMIT_PER_MIN) || 240, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));app.use('/api/v1', createApi(ctx));app.use('/api/v1', createRegistryApi(ctx));++ app.use('/api/v1', createVariantEvidenceApi(ctx));app.use('/api/v1/showcase', createShowcaseRoutes(ctx));app.use('/api/v1/gallery', createGalleryRoutes(ctx));app.use('/api/v1', createPickRoutes(ctx));nnew file mode 100644@@@ -0,0 +1,516 @@++'use strict';++++/**++ * P6 (plan 2026-11-04, G-P6 and ObservedPerformance): model-variant evidence.++ *++ * Bench is the evaluation authority, never an inference gateway. This module keeps the measured, typed evidence a++ * placement can read: one exact id (D3: org/model[:quant]@revision) per row, grouped by its family org/model across++ * quants and revisions. A row is written only when it names a finished pinned Bench round or an authenticated run,++ * and a model's own agent identity may never write evidence for itself.++ *++ * The numbers are stored as counts and integers so the read API cannot invent precision:++ * - sample_count, success_count, timeout_count;++ * - ttft_p50_ms, ttft_p95_ms, cold_start_ms (whole milliseconds);++ * - prefill_tokens_per_s, decode_tokens_per_s (three decimal places).++ * No prompt, output, payload or transcript is accepted or stored. The read API never joins a task, benchmark or++ * entry, so private sets stay private.++ */++++const ratingLib = require('../rating/lib');++++const MAX_EXACT_ID = 200;++const MIN_SAMPLES = 10;++const MODERATE_SAMPLES = 30;++const HIGH_SAMPLES = 100;++const MAX_OBSERVATIONS = 5000;++const MAX_MS = 3_600_000;++const MAX_TPS = 1e7;++++/** The D3 exact id: <org>/<model>[:<quant>]@<revision>, with the quant and the revision optional. */++const EXACT_ID_RE = /^([A-Za-z0-9][A-Za-z0-9._-]*)\/([A-Za-z0-9][A-Za-z0-9._-]*)(?::([A-Za-z0-9][A-Za-z0-9._-]*))?(?:@([A-Za-z0-9][A-Za-z0-9._-]*))?$/;++const IDENTITY_PREFIX = /^(?:svc|service|app|mod|agent):/i;++++const SELF_RATING = 'bench.self_rating';++++class VariantEvidenceError extends Error {++ constructor(status, code, detail) {++ super(detail);++ this.name = 'VariantEvidenceError';++ this.status = status;++ this.code = code;++ this.detail = detail;++ }++}++++const text = (v) => (v == null ? '' : String(v).trim());++const objectOf = (v) => (v && typeof v === 'object' && !Array.isArray(v) ? v : {});++const firstDefined = (...values) => values.find((v) => v !== undefined && v !== null && v !== '');++const round3 = (v) => (v == null ? null : Math.round(Number(v) * 1000) / 1000);++const round6 = (v) => (v == null ? null : Math.round(Number(v) * 1e6) / 1e6);++const numberOrNull = (v) => (v == null || v === '' ? null : Number(v));++++function parseExactId(value) {++ const raw = text(value);++ if (!raw) return { ok: false, error: 'exact_id is required' };++ if (raw.length > MAX_EXACT_ID) return { ok: false, error: `exact_id must be at most ${MAX_EXACT_ID} characters` };++ const match = EXACT_ID_RE.exec(raw);++ if (!match) return { ok: false, error: 'exact_id must be org/model[:quant]@revision, with the quant and the revision optional' };++ const [, org, model, quant, revision] = match;++ return {++ ok: true,++ value: {++ exact_id: raw,++ org,++ model,++ family: `${org}/${model}`,++ quant: quant || null,++ revision: revision || null,++ },++ };++}++++/** A family is exactly org/model: no quant and no revision. */++function parseFamily(value) {++ const raw = text(value);++ if (!raw || raw.length > MAX_EXACT_ID) return null;++ if (raw.includes(':') || raw.includes('@')) return null;++ const parsed = parseExactId(raw);++ if (!parsed.ok) return null;++ return parsed.value;++}++++/**++ * How much weight the sample count carries. Sparse evidence is never hidden or filled in: it is returned with the++ * count that makes it sparse. The 10-sample line is the same provisional line the rest of Bench uses.++ */++function confidenceFor(samples) {++ const n = Number.isSafeInteger(Number(samples)) && Number(samples) >= 0 ? Number(samples) : 0;++ let level = 'none';++ if (n >= HIGH_SAMPLES) level = 'high';++ else if (n >= MODERATE_SAMPLES) level = 'moderate';++ else if (n >= MIN_SAMPLES) level = 'low';++ else if (n > 0) level = 'sparse';++ return { level, sparse: n < MIN_SAMPLES, samples: n, minimum_samples: MIN_SAMPLES };++}++++function rateOf(count, sample) {++ if (count == null || !sample) return null;++ return round6(Number(count) / Number(sample));++}++++function confidenceInterval(successes, samples) {++ if (successes == null || !samples) return null;++ const w = ratingLib.wilson(Number(successes), Number(samples));++ return { estimate: round6(w.p), lower: round6(w.lower), upper: round6(w.upper) };++}++++function readInteger(errors, label, value, { min = 0, max = null } = {}) {++ if (value === undefined || value === null || value === '') return null;++ if (typeof value !== 'number' && typeof value !== 'string') {++ errors.push(`${label} must be a whole number`);++ return null;++ }++ const n = Number(value);++ if (!Number.isFinite(n) || !Number.isInteger(n) || n < min || (max != null && n > max)) {++ errors.push(`${label} must be a whole number${max == null ? ` at least ${min}` : ` from ${min} to ${max}`}`);++ return null;++ }++ return n;++}++++function readDecimal(errors, label, value, { min = 0, max = MAX_TPS } = {}) {++ if (value === undefined || value === null || value === '') return null;++ if (typeof value !== 'number' && typeof value !== 'string') {++ errors.push(`${label} must be a number`);++ return null;++ }++ const n = Number(value);++ if (!Number.isFinite(n) || n < min || n > max) {++ errors.push(`${label} must be a number from ${min} to ${max}`);++ return null;++ }++ return n;++}++++function readRate(errors, label, value) {++ if (value === undefined || value === null || value === '') return null;++ if (typeof value !== 'number' && typeof value !== 'string') {++ errors.push(`${label} must be a fraction from 0 to 1`);++ return null;++ }++ const n = Number(value);++ if (!Number.isFinite(n) || n < 0 || n > 1) {++ errors.push(`${label} must be a fraction from 0 to 1`);++ return null;++ }++ return n;++}++++function readSource(body) {++ const raw = body.source;++ const source = typeof raw === 'string' ? { id: raw } : objectOf(firstDefined(raw, body.provenance));++ let kind = text(firstDefined(source.kind, source.type)).toLowerCase();++ if (kind === 'pinned_round') kind = 'round';++ if (kind === 'run_job') kind = 'run';++ const roundId = firstDefined(source.round_id, body.round_id, body.pinned_round);++ const runId = firstDefined(source.run_id, body.run_id, body.job_id, body.run_job_id);++ const id = text(firstDefined(source.id, roundId, runId));++ if (!kind) kind = roundId ? 'round' : (runId ? 'run' : '');++ return { kind, id };++}++++/**++ * Validate and normalize one observation. Unknown fields are ignored, never stored: a caller cannot smuggle prompt++ * or output text into an evidence row. Returns { ok: true, value } or { ok: false, errors }.++ */++function validateObservation(body, { now = new Date().toISOString() } = {}) {++ const input = objectOf(body);++ const errors = [];++++ const exact = parseExactId(firstDefined(input.exact_id, input.exactId, input.model_id));++ if (!exact.ok) errors.push(exact.error);++++ const source = readSource(input);++ if (!source.kind) errors.push('source.kind must be round or run');++ else if (source.kind !== 'round' && source.kind !== 'run') errors.push('source.kind must be round or run');++ if (!source.id) errors.push('source.id is required');++ else if (source.id.length > MAX_EXACT_ID) errors.push(`source.id must be at most ${MAX_EXACT_ID} characters`);++ if (source.kind === 'round' && source.id && !/^brd_[0-9A-HJKMNP-TV-Z]{26}$/.test(source.id)) {++ errors.push('a round source id must be a Bench round id (brd_…)');++ }++ if (source.kind === 'run' && source.id && !/^(?:brn|job)_[0-9A-HJKMNP-TV-Z]{26}$/.test(source.id) && !/^run_[A-Za-z0-9_-]{1,120}$/.test(source.id)) {++ errors.push('a run source id must be a Bench run id (brn_…), a Run job id (job_…) or an AI run id (run_…)');++ }++++ const sampleCount = readInteger(errors, 'sample_count', firstDefined(input.sample_count, input.samples, input.n), { min: 1, max: 10_000_000 });++++ const ttft = objectOf(firstDefined(input.ttft, input.ttft_ms));++ const ttftP50 = readInteger(errors, 'ttft_p50_ms', firstDefined(input.ttft_p50_ms, input.ttft_ms_p50, input.ttft_p50, ttft.p50_ms, ttft.p50), { min: 0, max: MAX_MS });++ const ttftP95 = readInteger(errors, 'ttft_p95_ms', firstDefined(input.ttft_p95_ms, input.ttft_ms_p95, input.ttft_p95, ttft.p95_ms, ttft.p95), { min: 0, max: MAX_MS });++ if (ttftP50 != null && ttftP95 != null && ttftP95 < ttftP50) errors.push('ttft_p95_ms must be at least ttft_p50_ms');++++ const throughput = objectOf(input.throughput);++ const prefill = readDecimal(errors, 'prefill_tokens_per_s', firstDefined(++ input.prefill_tokens_per_s, input.prefill_tps, input.prefill_tokens_s,++ throughput.prefill_tokens_per_s, throughput.prefill_tps,++ ));++ const decode = readDecimal(errors, 'decode_tokens_per_s', firstDefined(++ input.decode_tokens_per_s, input.decode_tps, input.decode_tokens_s,++ throughput.decode_tokens_per_s, throughput.decode_tps,++ ));++ const coldStart = readInteger(errors, 'cold_start_ms', firstDefined(++ input.cold_start_ms, input.coldStartMs, typeof input.cold_start === 'number' ? input.cold_start : undefined,++ objectOf(input.cold_start).ms, objectOf(input.cold_start).cold_start_ms,++ ), { min: 0, max: MAX_MS });++++ const successInfo = objectOf(input.success);++ const timeoutInfo = objectOf(input.timeout);++ const successCount = readInteger(errors, 'success_count', firstDefined(++ input.success_count, input.successes, successInfo.count, typeof input.success === 'number' ? input.success : undefined,++ ), { min: 0, max: sampleCount == null ? undefined : sampleCount });++ const timeoutCount = readInteger(errors, 'timeout_count', firstDefined(++ input.timeout_count, input.timeouts, timeoutInfo.count, typeof input.timeout === 'number' ? input.timeout : undefined,++ ), { min: 0, max: sampleCount == null ? undefined : sampleCount });++ const successRate = readRate(errors, 'success_rate', firstDefined(input.success_rate, successInfo.rate));++ const timeoutRate = readRate(errors, 'timeout_rate', firstDefined(input.timeout_rate, timeoutInfo.rate));++++ let successes = successCount;++ let timeouts = timeoutCount;++ if (sampleCount != null) {++ if (successes == null && successRate != null) successes = Math.round(successRate * sampleCount);++ if (timeouts == null && timeoutRate != null) timeouts = Math.round(timeoutRate * sampleCount);++ if (successes != null && timeouts != null && successes + timeouts > sampleCount) {++ errors.push('success_count and timeout_count cannot add up to more than sample_count');++ }++ }++++ let measuredAt = text(now) || new Date().toISOString();++ const measuredInput = firstDefined(input.measured_at, input.measuredAt, input.at);++ if (measuredInput != null) {++ const ms = Date.parse(measuredInput);++ if (!Number.isFinite(ms)) errors.push('measured_at must be an ISO timestamp');++ else measuredAt = new Date(ms).toISOString();++ }++++ if (errors.length) return { ok: false, errors };++ return {++ ok: true,++ value: {++ exact_id: exact.value.exact_id,++ family: exact.value.family,++ org: exact.value.org,++ model: exact.value.model,++ quant: exact.value.quant,++ revision: exact.value.revision,++ source_kind: source.kind,++ source_id: source.id,++ sample_count: sampleCount,++ success_count: successes,++ timeout_count: timeouts,++ ttft_p50_ms: ttftP50,++ ttft_p95_ms: ttftP95,++ prefill_tokens_per_s: prefill == null ? null : round3(prefill),++ decode_tokens_per_s: decode == null ? null : round3(decode),++ cold_start_ms: coldStart,++ measured_at: measuredAt,++ },++ };++}++++/** True when the caller is the model it is trying to measure. A model never rates itself. */++function isSelfRating(principal, value) {++ if (!principal || !value) return false;++ const claims = objectOf(principal.claims);++ const names = [value.exact_id, value.family, value.model, `${value.org}/${value.model}`]++ .filter(Boolean).map((v) => String(v).toLowerCase());++ const identities = [++ principal.requester,++ claims.sub,++ claims.model,++ claims.model_id,++ claims.exact_id,++ claims.model_key,++ ].filter(Boolean).map((v) => String(v).trim().toLowerCase());++ for (const identity of identities) {++ const id = identity.replace(IDENTITY_PREFIX, '');++ if (!id) continue;++ if (names.includes(id)) return true;++ const family = String(value.family || '').toLowerCase();++ if (family && id.startsWith(family)) {++ const rest = id.slice(family.length);++ if (!rest || rest.startsWith(':') || rest.startsWith('@')) return true;++ }++ }++ return false;++}++++function wireObservation(row) {++ const samples = numberOrNull(row.sample_count) || 0;++ const successes = row.success_count == null ? null : numberOrNull(row.success_count);++ const timeouts = row.timeout_count == null ? null : numberOrNull(row.timeout_count);++ const provenance = typeof row.provenance === 'string' ? JSON.parse(row.provenance) : (row.provenance || {});++ return {++ id: row.id,++ exact_id: row.exact_id,++ family: row.family,++ quant: row.quant || null,++ revision: row.revision || null,++ sample_count: samples,++ success_count: successes,++ success_rate: rateOf(successes, samples),++ timeout_count: timeouts,++ timeout_rate: rateOf(timeouts, samples),++ metrics: {++ ttft_p50_ms: row.ttft_p50_ms == null ? null : numberOrNull(row.ttft_p50_ms),++ ttft_p95_ms: row.ttft_p95_ms == null ? null : numberOrNull(row.ttft_p95_ms),++ prefill_tokens_per_s: round3(numberOrNull(row.prefill_tokens_per_s)),++ decode_tokens_per_s: round3(numberOrNull(row.decode_tokens_per_s)),++ cold_start_ms: row.cold_start_ms == null ? null : numberOrNull(row.cold_start_ms),++ },++ measured_at: row.measured_at == null ? null : (row.measured_at instanceof Date ? row.measured_at.toISOString() : String(row.measured_at)),++ provenance,++ recorded_by: row.recorded_by,++ created_at: row.created_at == null ? null : (row.created_at instanceof Date ? row.created_at.toISOString() : String(row.created_at)),++ };++}++++/** One exact id's evidence: every observation, plus the totals and the confidence they carry. */++function aggregateRows(exactId, rows) {++ const list = (rows || []).filter(Boolean);++ if (!list.length) return null;++ const parsed = parseExactId(exactId);++ const first = list[0];++ const observations = list.map(wireObservation).sort((a, b) => String(b.measured_at || '').localeCompare(String(a.measured_at || '')) || String(b.id).localeCompare(String(a.id)));++ const latest = observations[0] || null;++ const samples = observations.reduce((sum, o) => sum + (o.sample_count || 0), 0);++ // A rate is known only when every observation counts that outcome. A missing count is not a zero.++ const completeSuccess = observations.every((o) => o.success_count != null);++ const completeTimeout = observations.every((o) => o.timeout_count != null);++ const successes = completeSuccess ? observations.reduce((sum, o) => sum + (o.success_count || 0), 0) : null;++ const timeouts = completeTimeout ? observations.reduce((sum, o) => sum + (o.timeout_count || 0), 0) : null;++ const confidence = confidenceFor(samples);++ const successInterval = confidenceInterval(successes, samples);++ const timeoutInterval = confidenceInterval(timeouts, samples);++ if (successInterval) confidence.success_rate = successInterval;++ if (timeoutInterval) confidence.timeout_rate = timeoutInterval;++ const metrics = latest ? latest.metrics : {};++ return {++ exact_id: parsed.ok ? parsed.value.exact_id : String(exactId || ''),++ family: parsed.ok ? parsed.value.family : first.family,++ org: parsed.ok ? parsed.value.org : null,++ model: parsed.ok ? parsed.value.model : null,++ quant: parsed.ok ? parsed.value.quant : (first.quant || null),++ revision: parsed.ok ? parsed.value.revision : (first.revision || null),++ sample_count: samples,++ success_count: successes,++ success_rate: rateOf(successes, samples),++ timeout_count: timeouts,++ timeout_rate: rateOf(timeouts, samples),++ confidence,++ sparse: confidence.sparse,++ metrics,++ ...metrics,++ measured_at: latest ? latest.measured_at : null,++ provenance: latest ? latest.provenance : null,++ observations,++ };++}++++/** Group evidence rows by exact id and then by family org/model. Pure, no I/O. */++function groupFamilies(rows) {++ const byFamily = new Map();++ for (const row of (rows || [])) {++ if (!row) continue;++ const parsed = parseExactId(row.exact_id);++ if (!parsed.ok) continue;++ const family = parsed.value.family;++ if (!byFamily.has(family)) byFamily.set(family, { org: parsed.value.org, model: parsed.value.model, rows: [] });++ byFamily.get(family).rows.push(row);++ }++ const families = [];++ for (const [family, group] of byFamily) {++ const byExact = new Map();++ for (const row of group.rows) {++ if (!byExact.has(row.exact_id)) byExact.set(row.exact_id, []);++ byExact.get(row.exact_id).push(row);++ }++ const variants = [...byExact.entries()].map(([exact, list]) => aggregateRows(exact, list)).filter(Boolean);++ variants.sort((a, b) => String(a.exact_id).localeCompare(String(b.exact_id)));++ families.push({++ family,++ org: group.org,++ model: group.model,++ variants,++ quants: [...new Set(variants.map((v) => v.quant).filter(Boolean))].sort(),++ revisions: [...new Set(variants.map((v) => v.revision).filter(Boolean))].sort(),++ sample_count: variants.reduce((sum, v) => sum + (v.sample_count || 0), 0),++ });++ }++ families.sort((a, b) => a.family.localeCompare(b.family));++ return families;++}++++function createVariantEvidence({ s, log = console }) {++ async function sourceProvenance(sourceKind, sourceId) {++ if (sourceKind === 'round') {++ const row = await s.db.maybe('SELECT id, status, receipt FROM bench_rounds WHERE id = $1', [sourceId]);++ if (!row) throw new VariantEvidenceError(404, 'bench.round_not_found', 'No such Bench round.');++ if (row.status !== 'finished' || !row.receipt) {++ throw new VariantEvidenceError(409, 'bench.round_not_pinned', 'Evidence can only come from a finished, pinned round with a receipt.');++ }++ return { kind: 'round', id: row.id, receipt: row.receipt };++ }++ if (sourceId.startsWith('brn_')) {++ const row = await s.db.maybe('SELECT id, provenance FROM bench_runs WHERE id = $1', [sourceId]);++ if (!row) throw new VariantEvidenceError(404, 'bench.run_not_found', 'No such Bench run.');++ if (row.provenance !== 'verified') {++ throw new VariantEvidenceError(409, 'bench.run_not_verified', 'Evidence can only come from a verified run.');++ }++ return { kind: 'run', id: row.id, verified: true, bench_run_id: row.id };++ }++ const viaRun = await s.db.maybe(++ `SELECT id FROM bench_runs++ WHERE provenance = 'verified' AND executor->>'kind' = 'run' AND executor->>'id' = $1++ ORDER BY created_at DESC, id LIMIT 1`,++ [sourceId],++ ).catch(() => null);++ return {++ kind: 'run',++ id: sourceId,++ verified: Boolean(viaRun),++ executor: sourceId.startsWith('job_') ? 'run' : 'ai',++ bench_run_id: viaRun ? viaRun.id : null,++ };++ }++++ async function record(value, { createdBy }) {++ const provenance = await sourceProvenance(value.source_kind, value.source_id);++ const row = await s.db.maybe(++ `INSERT INTO bench_variant_evidence++ (id, exact_id, family, quant, revision, source_kind, source_id, provenance, measured_at,++ sample_count, success_count, timeout_count, ttft_p50_ms, ttft_p95_ms, prefill_tokens_per_s,++ decode_tokens_per_s, cold_start_ms, recorded_by, created_at)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9::timestamptz,++ $10, $11, $12, $13, $14, $15, $16, $17, $18, $19::timestamptz)++ ON CONFLICT (exact_id, source_kind, source_id) DO UPDATE SET++ family = EXCLUDED.family, quant = EXCLUDED.quant, revision = EXCLUDED.revision,++ provenance = EXCLUDED.provenance, measured_at = EXCLUDED.measured_at,++ sample_count = EXCLUDED.sample_count, success_count = EXCLUDED.success_count,++ timeout_count = EXCLUDED.timeout_count, ttft_p50_ms = EXCLUDED.ttft_p50_ms,++ ttft_p95_ms = EXCLUDED.ttft_p95_ms, prefill_tokens_per_s = EXCLUDED.prefill_tokens_per_s,++ decode_tokens_per_s = EXCLUDED.decode_tokens_per_s, cold_start_ms = EXCLUDED.cold_start_ms,++ recorded_by = EXCLUDED.recorded_by++ RETURNING *`,++ [s.newId('bve'), value.exact_id, value.family, value.quant, value.revision, value.source_kind, value.source_id,++ JSON.stringify(provenance), value.measured_at, value.sample_count, value.success_count, value.timeout_count,++ value.ttft_p50_ms, value.ttft_p95_ms, value.prefill_tokens_per_s, value.decode_tokens_per_s,++ value.cold_start_ms, String(createdBy || 'service:unknown').slice(0, 200), s.iso()],++ );++ return aggregateRows(value.exact_id, [row]);++ }++++ async function rowsFor({ family = null, exactId = null } = {}) {++ const clauses = [];++ const params = [];++ if (family) { params.push(String(family)); clauses.push(`family = $${params.length}`); }++ if (exactId) { params.push(String(exactId)); clauses.push(`exact_id = $${params.length}`); }++ const where = clauses.length ? `WHERE ${clauses.join(' AND ')}` : '';++ return await s.db.many(++ `SELECT * FROM bench_variant_evidence ${where}++ ORDER BY measured_at DESC, created_at DESC, id DESC LIMIT ${MAX_OBSERVATIONS}`,++ params,++ );++ }++++ async function families() {++ return groupFamilies(await rowsFor());++ }++++ async function family(familyValue) {++ const parsed = parseFamily(familyValue);++ if (!parsed) return null;++ const rows = await rowsFor({ family: parsed.family });++ if (!rows.length) return null;++ return groupFamilies(rows)[0] || null;++ }++++ async function list({ family: familyValue = null, exactId = null } = {}) {++ if (exactId) {++ const one = await get(exactId);++ return one ? [one] : [];++ }++ const rows = await rowsFor({ family: familyValue ? text(familyValue) : null });++ const byExact = new Map();++ for (const row of rows) {++ if (!byExact.has(row.exact_id)) byExact.set(row.exact_id, []);++ byExact.get(row.exact_id).push(row);++ }++ return [...byExact.entries()].map(([exact, listOfRows]) => aggregateRows(exact, listOfRows)).filter(Boolean)++ .sort((a, b) => a.family.localeCompare(b.family) || String(a.exact_id).localeCompare(String(b.exact_id)));++ }++++ async function get(exactId) {++ const parsed = parseExactId(exactId);++ if (!parsed.ok) return null;++ const rows = await rowsFor({ exactId: parsed.value.exact_id });++ return aggregateRows(parsed.value.exact_id, rows);++ }++++ return { record, families, family, list, get, sourceProvenance };++}++++module.exports = {++ parseExactId,++ parseFamily,++ confidenceFor,++ validateObservation,++ isSelfRating,++ aggregateRows,++ groupFamilies,++ wireObservation,++ createVariantEvidence,++ VariantEvidenceError,++ MIN_SAMPLES,++ MAX_EXACT_ID,++ SELF_RATING,++};@@@ -79,6 +79,9 @@ function load(env = process.env) {// Bench budgets and jobs.bench: {++ // P6 model-variant evidence (exact ids, quants, revisions and their measured numbers). Off until the++ // owner turns it on; off, none of its routes exist and nothing writes its table.++ variantEvidence: bool(env.BENCH_VARIANT_EVIDENCE, false),roundsPerDay: Math.max(1, int(env.BENCH_ROUNDS_PER_DAY, 3)),roundsDaily: Math.max(1, int(env.BENCH_ROUNDS_DAILY, 20)),intervalMs: Math.max(250, int(env.BENCH_JOB_INTERVAL_MS, 1500)),nnew file mode 100644@@@ -0,0 +1,101 @@++'use strict';++++/**++ * P6: the model-variant evidence API (plan 2026-11-04, G-P6). Behind BENCH_VARIANT_EVIDENCE, off by default.++ *++ * GET /api/v1/variant-families every family, grouped by org/model++ * GET /api/v1/variant-families/:family one family (URL-encoded org/model)++ * GET /api/v1/variant-evidence every exact id's evidence++ * GET /api/v1/variant-evidence/:exactId one exact id (URL-encoded)++ * POST /api/v1/variant-evidence record one observation++ *++ * The write is trusted provenance, not an open form. A round source needs a service token with bench.entry.create++ * and a finished round with a receipt; a run source needs a service token with bench.run.publish. A model's own++ * agent token is refused. The response carries only ids, counts and measurements: no prompt, output or payload.++ */++++const express = require('express');++const { http } = require('openvibe-contracts');++const { asyncRouter } = require('./router');++const { isSelfRating, validateObservation, VariantEvidenceError, SELF_RATING } = require('../bench/variant-evidence');++++function createVariantEvidenceApi(ctx) {++ const { principal, variantEvidence, config, log } = ctx;++ const r = asyncRouter();++ // D11 owner settings: the switch is read live. Off, these paths are the API's ordinary 404 and nothing is read.++ r.use((_req, _res, next) => (config.bench.variantEvidence ? next() : next('router')));++++ r.use(express.json({ limit: '64kb' }));++ r.use((req, res, next) => (req.principal ? next() : principal.middleware(req, res, next)));++ r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });++++ const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });++ const invalid = (res, detail) => problem(res, 400, 'bench.invalid', detail);++ const failed = (res, err) => {++ if (err instanceof VariantEvidenceError) return problem(res, err.status, err.code, err.detail);++ log.warn('[OpenVibe.Bench] variant evidence failed:', (err && err.message) || err);++ return problem(res, 500, 'internal.error', 'Bench could not answer. Try again in a moment.');++ };++++ const encoded = (id) => encodeURIComponent(String(id));++++ r.get('/variant-families', async (_req, res) => {++ const families = await variantEvidence.families();++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ count: families.length, families });++ });++++ r.get('/variant-families/:family', async (req, res) => {++ const family = await variantEvidence.family(String(req.params.family || ''));++ if (!family) return problem(res, 404, 'bench.variant_family_not_found', 'No such variant family.');++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ family });++ });++++ r.get('/variant-evidence', async (req, res) => {++ const exactId = typeof req.query.exact_id === 'string' && req.query.exact_id ? req.query.exact_id : null;++ const family = typeof req.query.family === 'string' && req.query.family ? req.query.family : null;++ const variants = await variantEvidence.list({ family, exactId });++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ count: variants.length, variants });++ });++++ r.get('/variant-evidence/:exactId', async (req, res) => {++ const evidence = await variantEvidence.get(String(req.params.exactId || ''));++ if (!evidence) return problem(res, 404, 'bench.variant_not_found', 'No evidence for that exact id.');++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ evidence });++ });++++ r.post('/variant-evidence', async (req, res) => {++ const body = req.body && typeof req.body === 'object' ? req.body : {};++ const checked = validateObservation(body, { now: ctx.s.iso() });++ if (!checked.ok) return invalid(res, checked.errors.join('; '));++ const value = checked.value;++++ const p = req.principal || { kind: 'anonymous' };++ const cap = value.source_kind === 'run' ? 'bench.run.publish' : 'bench.entry.create';++ if (p.kind === 'anonymous') {++ return problem(res, 401, 'token.required', 'Send a service token with bench.entry.create (a pinned round) or bench.run.publish (a run).');++ }++ if (p.kind !== 'app') {++ return problem(res, 403, 'bench.capability_required', `This route needs the ${cap} capability on a service token.`);++ }++ if (!p.hasCapability(cap)) {++ return problem(res, 403, 'bench.capability_denied', `A service token needs ${cap} to write this evidence.`);++ }++ if (isSelfRating(p, value)) {++ return problem(res, 403, SELF_RATING, 'A model may not write evidence about itself: use a service that ran it.');++ }++++ try {++ const evidence = await variantEvidence.record(value, { createdBy: p.requester });++ res.set('Location', `/api/v1/variant-evidence/${encoded(value.exact_id)}`);++ return res.status(201).json({ evidence });++ } catch (err) { return failed(res, err); }++ });++++ return r;++}++++module.exports = { createVariantEvidenceApi };nnew file mode 100644@@@ -0,0 +1,82 @@++'use strict';++/**++ * The `bench.variant_evidence` namespace (D11 owner settings): the switch the owner changes from++ * openvibe.network's AI tab. BENCH_VARIANT_EVIDENCE stays the pin: while it is set it wins, is read-only in the++ * summary and a write answers 409 config.pinned_in_env.++ *++ * /internal/config is openvibe-shared/config's adminRoutes, guarded by bench.config.manage (Network's service++ * token only). actor is the owner Network names in X-OV-Subject; X-OV-Fresh: 1 gates a secret or x-danger write++ * (this namespace has neither). Every activation writes one audit line with the actor and the keys, never a value.++ */++const express = require('express');++const shared = require('openvibe-shared/config');++const { capabilities, http, ids } = require('openvibe-contracts');++++const SERVICE = 'bench';++const NAMESPACE = 'bench.variant_evidence';++const CAPABILITY = 'bench.config.manage';++++const SCHEMA = {++ $schema: 'https://json-schema.org/draft/2020-12/schema',++ title: 'Model-variant evidence',++ description: 'The exact model ids, quants and revisions Bench has measured, and the evidence API a service publishes them through.',++ type: 'object',++ additionalProperties: false,++ required: ['enabled'],++ properties: {++ enabled: {++ type: 'boolean', title: 'Variant evidence', 'x-group': 'Bench', 'x-env': 'BENCH_VARIANT_EVIDENCE',++ description: 'Serve /api/v1/variant-families and /api/v1/variant-evidence and accept a verified round or run as provenance.',++ },++ },++};++const DEFAULTS = Object.freeze({ enabled: false });++++function actor(req) {++ const named = String((req.headers && req.headers['x-ov-subject']) || '').trim();++ return ids.isSubjectId('user', named) ? { type: 'user', id: named } : null;++}++++const requireFresh = (req) => String((req.headers && req.headers['x-ov-fresh']) || '') === '1';++++/** principal.middleware resolves the token; this finish refuses anything that is not a service token with the capability. */++function capabilityGuard({ principal, config, log }) {++ const resolve = principal.middleware;++ return function guard(req, res, next) {++ resolve(req, res, (err) => {++ if (err) return next(err);++ const p = req.principal || {};++ if (p.kind !== 'app') return http.sendProblem(res, 403, 'capability.denied', { detail: `${CAPABILITY} is first-party; an app may not change Bench configuration`, ctx: req.ov });++ const decision = capabilities.check(p.claims, CAPABILITY);++ if (!decision.allowed) return http.sendProblem(res, 403, decision.code || 'capability.denied', { detail: decision.reason || `${CAPABILITY} not granted`, ctx: req.ov });++ req.configClaims = p.claims;++ return next();++ });++ };++}++++async function create({ db, principal, config, env = process.env, log = console, onActivate = null }) {++ // BENCH_VARIANT_EVIDENCE has always accepted 1/true/yes/on; the shared store's boolean pin reads true/1.++ const storeEnv = { ...env };++ if (['1', 'true', 'yes', 'on'].includes(String(storeEnv.BENCH_VARIANT_EVIDENCE || '').trim().toLowerCase())) storeEnv.BENCH_VARIANT_EVIDENCE = '1';++ const store = await shared.createConfigStore({++ db, service: SERVICE, namespace: NAMESPACE, schema: SCHEMA, defaults: DEFAULTS, env: storeEnv, log,++ onActivate: async (values) => { if (onActivate) await onActivate(values); },++ onChange: ({ namespace, keys, actor: who, revision }) => {++ log.info(`[config] ${SERVICE} ${namespace} revision ${revision} by ${who && who.id ? `${who.type}:${who.id}` : 'unknown'}: ${keys.join(', ')}`);++ },++ });++ if (env.OV_CONFIG_KEKS) {++ try { await store.reseal(); } catch (err) { log.warn(`[config] ${NAMESPACE}: reseal failed: ${(err && err.message) || err}`); }++ }++ const router = shared.adminRoutes([store], {++ basePath: '/internal/config', requireAdmin: capabilityGuard({ principal, config, log }), actor, requireFresh, log,++ });++ return {++ store, router,++ get: (key) => store.get(key),++ mount(app) { app.use('/internal/config', express.json({ limit: '128kb' })); router.mount(app); },++ };++}++++module.exports = { create, SCHEMA, NAMESPACE, CAPABILITY };@@@ -42,6 +42,7 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;exempt: {account_data_events: 'the ledger of the requests themselves: which export and deletion Bench applied',bench_ratings: 'the ratings are an aggregate over the models: the subject column names a model, not a person',++ config_snapshots: 'D11 configuration revisions: created_by names the owner who changed a setting, not a person\'s account data',},});assert.deepStrictEqual(missing, [], `a table with a person column is neither covered nor exempted: ${missing.join(', ')}`);@@@ -40,7 +40,7 @@ async function boot(opts = {}) {const config = configLib.load(env);const testdb = opts.store ? null : await require('./db').testDb({ store: opts.testStore });const store = opts.store || createStore(testdb.db, { now: opts.now });-- const built = await createApp({ config, store, benchAi: opts.benchAi, log, now: opts.now, accountSend: opts.accountSend, random: opts.random, telemetrySink: opts.telemetrySink });++ const built = await createApp({ config, env, store, benchAi: opts.benchAi, log, now: opts.now, accountSend: opts.accountSend, random: opts.random, telemetrySink: opts.telemetrySink });await built.ctx.keys.ensure();const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });const base = `http://127.0.0.1:${server.address().port}`;@@@ -323,6 +323,7 @@ function multipart(filename, bytes) {exempt: {account_data_events: 'the ledger of the requests themselves',bench_ratings: 'the ratings are an aggregate over the models',++ config_snapshots: 'D11 configuration revisions: created_by names the owner who changed a setting, not a person\'s account data',},});assert.deepStrictEqual(missing, [], 'a table with a person column is neither covered nor exempted: ' + missing.join(', '));nnew file mode 100644@@@ -0,0 +1,90 @@++'use strict';++/**++ * D11 owner settings: the bench.variant_evidence namespace and /internal/config.++ * Env pins win; an owner change applies live; bench.config.manage is required; no KEK still works;++ * a secret-class value never leaves a summary, snapshot or history.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++++const CAP = ['bench.config.manage'];++const OWNER = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';++const SECRET = 'fixture-bench-d11-not-a-credential';++++(async () => {++ const t = await boot({ env: { BENCH_VARIANT_EVIDENCE: '1' } });++ const admin = () => t.network.serviceToken({ sub: 'svc:network', audience: 'openvibe.bench', cap: CAP });++ try {++ await check('the summary: schema, the env pin, sealed:false without a KEK', async () => {++ const r = await t.get('/internal/config/bench.variant_evidence', { bearer: admin() });++ assert.strictEqual(r.status, 200, r.text.slice(0, 300));++ assert.strictEqual(r.json().namespace, 'bench.variant_evidence');++ assert.strictEqual(r.json().sealed, false);++ assert.strictEqual(r.json().values.enabled, true);++ assert.deepStrictEqual(r.json().pins.enabled, { env: 'BENCH_VARIANT_EVIDENCE', pinned: true });++ assert.ok(!r.text.includes(SECRET));++ });++++ await check('a service token without bench.config.manage is refused', async () => {++ const token = t.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.round.read'] });++ const r = await t.get('/internal/config/bench.variant_evidence', { bearer: token });++ assert.strictEqual(r.status, 403);++ assert.strictEqual(r.json().code, 'capability.denied');++ });++++ await check('an env-pinned key is read-only', async () => {++ const r = await t.get('/internal/config/bench.variant_evidence', {++ method: 'POST', bearer: admin(), headers: { 'x-ov-subject': OWNER }, json: { values: { enabled: false }, reason: 'off' },++ });++ assert.strictEqual(r.status, 409, r.text.slice(0, 300));++ assert.strictEqual(r.json().code, 'config.pinned_in_env');++ });++ } finally { await t.close(); }++++ const live = await boot();++ const liveAdmin = () => live.network.serviceToken({ sub: 'svc:network', audience: 'openvibe.bench', cap: CAP });++ try {++ await check('an owner change applies live: the evidence API appears', async () => {++ const reader = live.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.round.read'] });++ assert.strictEqual((await live.get('/api/v1/variant-families', { bearer: reader })).status, 404);++ const r = await live.get('/internal/config/bench.variant_evidence', {++ method: 'POST', bearer: liveAdmin(), headers: { 'x-ov-subject': OWNER }, json: { values: { enabled: true }, reason: 'owner turned it on' },++ });++ assert.strictEqual(r.status, 200, r.text.slice(0, 300));++ assert.strictEqual(r.json().revision, 2);++ assert.strictEqual(live.config.bench.variantEvidence, true);++ assert.strictEqual((await live.get('/api/v1/variant-families', { bearer: reader })).status, 200);++ assert.ok(live.logs().includes('by user:' + OWNER));++ assert.ok(!live.logs().includes(SECRET));++ });++++ await check('with a KEK the store is sealed and boot reseals', async () => {++ const sealed = await boot({ env: { OV_CONFIG_KEKS: 'k1:MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=', OV_CONFIG_KEK_CURRENT: 'k1' } });++ try {++ const token = sealed.network.serviceToken({ sub: 'svc:network', audience: 'openvibe.bench', cap: CAP });++ const r = await sealed.get('/internal/config/bench.variant_evidence', { bearer: token });++ assert.strictEqual(r.status, 200, r.text.slice(0, 300));++ assert.strictEqual(r.json().sealed, true);++ assert.strictEqual(r.json().kek, 'k1');++ } finally { await sealed.close(); }++ });++++ await check('a secret is write-only and never appears in a summary, snapshot or history', async () => {++ const shared = require('openvibe-shared/config');++ const extra = await shared.createConfigStore({++ db: live.store.db, service: 'bench', namespace: 'bench.test_secrets',++ schema: { type: 'object', additionalProperties: false, properties: { api_key: { type: 'string' } } },++ classify: () => 'secret',++ });++ const marker = await extra.setSecret('api_key', SECRET, { actor: { type: 'user', id: OWNER }, reason: 'test' });++ assert.strictEqual(marker.redacted, true);++ assert.ok(!JSON.stringify(marker).includes(SECRET));++ const summary = await extra.summary();++ assert.strictEqual(summary.values.api_key.redacted, true);++ assert.ok(!JSON.stringify(summary).includes(SECRET));++ assert.ok(!JSON.stringify(await extra.history()).includes(SECRET));++ assert.ok(!JSON.stringify(await extra.snapshot(1)).includes(SECRET));++ });++ } finally { await live.close(); }++ done();++})().catch((e) => { console.error(e); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,351 @@++'use strict';++++/**++ * P6 model-variant evidence (plan 2026-11-04, G-P6): exact ids group into a family across quants and revisions;++ * the quant and revision are optional; sparse evidence stays sparse; a write needs a pinned round or an++ * authenticated run; a model never writes its own evidence; the flag off leaves no route and no write.++ */++const assert = require('assert');++const crypto = require('crypto');++const { boot, check, done } = require('./helpers/boot');++const configLib = require('../server/config');++const { parseExactId, groupFamilies } = require('../server/bench/variant-evidence');++const { createVariantEvidenceApi } = require('../server/http/variant-evidence');++++const EXACT = 'qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9';++const FAMILY = 'qwen/qwen3-9b-abliterated';++++function row(over = {}) {++ return {++ id: 'bve_00000000000000000000000000',++ exact_id: EXACT,++ family: FAMILY,++ quant: 'q4_k_m',++ revision: '3f2a1c9',++ source_kind: 'round',++ source_id: 'brd_00000000000000000000000000',++ provenance: { kind: 'round', id: 'brd_00000000000000000000000000', receipt: 'a'.repeat(64) },++ measured_at: '2026-11-04T10:00:00.000Z',++ sample_count: 4,++ success_count: 3,++ timeout_count: 1,++ ttft_p50_ms: 210,++ ttft_p95_ms: 480,++ prefill_tokens_per_s: 1420,++ decode_tokens_per_s: 86.5,++ cold_start_ms: 5300,++ created_by: 'service:codes',++ created_at: '2026-11-04T10:00:01.000Z',++ ...over,++ };++}++++(async () => {++ await check('the flag is off by default and the route factory exists only for the on path', async () => {++ assert.strictEqual(configLib.load({ NODE_ENV: 'test' }).bench.variantEvidence, false);++ assert.strictEqual(configLib.load({ NODE_ENV: 'test', BENCH_VARIANT_EVIDENCE: '1' }).bench.variantEvidence, true);++ assert.strictEqual(typeof createVariantEvidenceApi, 'function');++ });++++ await check('the exact id parses the family, quant and revision, each optional', async () => {++ const full = parseExactId(EXACT);++ assert.strictEqual(full.ok, true);++ assert.strictEqual(full.value.family, FAMILY);++ assert.strictEqual(full.value.quant, 'q4_k_m');++ assert.strictEqual(full.value.revision, '3f2a1c9');++ const bare = parseExactId('qwen/qwen3-9b');++ assert.strictEqual(bare.ok, true);++ assert.strictEqual(bare.value.family, 'qwen/qwen3-9b');++ assert.strictEqual(bare.value.quant, null);++ assert.strictEqual(bare.value.revision, null);++ const quantOnly = parseExactId('qwen/qwen3-9b:q8_0');++ assert.strictEqual(quantOnly.value.quant, 'q8_0');++ assert.strictEqual(quantOnly.value.revision, null);++ const revisionOnly = parseExactId('qwen/qwen3-9b@main');++ assert.strictEqual(revisionOnly.value.quant, null);++ assert.strictEqual(revisionOnly.value.revision, 'main');++ for (const bad of ['qwen', 'qwen/', '/qwen3', 'qwen/qwen3@', 'qwen/qwen3:q4:extra', 'qwen/qwen3@a@b', 'qwen/qwen 3']) {++ assert.strictEqual(parseExactId(bad).ok, false, bad);++ }++ });++++ await check('the family groups every quant and revision of one org/model, with its own counts', async () => {++ const families = groupFamilies([++ row(),++ row({ id: 'bve_11111111111111111111111111', exact_id: 'qwen/qwen3-9b-abliterated:q8_0@rev2', quant: 'q8_0', revision: 'rev2', sample_count: 20, success_count: 20, timeout_count: 0 }),++ row({ id: 'bve_22222222222222222222222222', exact_id: 'other/model:q4@main', family: 'other/model', quant: 'q4', revision: 'main', sample_count: 1, success_count: null }),++ ]);++ assert.strictEqual(families.length, 2);++ const qwen = families.find((f) => f.family === FAMILY);++ assert.ok(qwen);++ assert.strictEqual(qwen.variants.length, 2);++ assert.deepStrictEqual(qwen.quants, ['q4_k_m', 'q8_0']);++ assert.deepStrictEqual(qwen.revisions, ['3f2a1c9', 'rev2']);++ assert.strictEqual(qwen.sample_count, 24);++ const q4 = qwen.variants.find((v) => v.quant === 'q4_k_m');++ assert.strictEqual(q4.sample_count, 4);++ assert.strictEqual(q4.sparse, true);++ assert.strictEqual(q4.success_rate, 0.75);++ assert.strictEqual(q4.metrics.ttft_p95_ms, 480);++ });++++ const t = await boot({ env: { BENCH_VARIANT_EVIDENCE: '1' } });++ const alice = t.network.addUser('alice');++ const roundWriter = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.entry.create'] });++ const publishers = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.entry.create', 'bench.run.publish', 'bench.benchmark.write'] });++ const wrongCap = t.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.round.read'] });++ const secret = 'ZZ-PRIVATE-PROMPT-DO-NOT-STORE';++ const secretOutput = 'ZZ-PRIVATE-OUTPUT-DO-NOT-STORE';++++ let finishedRound;++ let benchmarkId;++ let verifiedRunId;++++ await check('a person starts a round and the job pins it with a receipt', async () => {++ const task = await t.get('/api/v1/kinds/design.page/tasks', { as: alice, json: { prompt: `A landing page for a tiny robot. ${secret}`, public: true } });++ assert.strictEqual(task.status, 201, task.text);++ const queued = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });++ assert.strictEqual(queued.status, 201, queued.text);++ await t.tick();++ const round = await t.get(`/api/v1/rounds/${queued.json().round.id}`, { as: alice });++ assert.strictEqual(round.json().round.status, 'finished');++ assert.ok(round.json().round.receipt);++ finishedRound = round.json().round;++ });++++ await check('a pinned-round write records the typed numbers', async () => {++ const written = await t.get('/api/v1/variant-evidence', {++ bearer: roundWriter,++ json: {++ exact_id: EXACT,++ source: { kind: 'round', id: finishedRound.id },++ sample_count: 4,++ ttft_p50_ms: 210,++ ttft_p95_ms: 480,++ prefill_tokens_per_s: 1420,++ decode_tokens_per_s: 86.5,++ cold_start_ms: 5300,++ success_count: 3,++ timeout_count: 1,++ measured_at: '2026-11-04T10:00:00.000Z',++ // A caller may send these; they are ignored and never stored.++ prompt: secret,++ output: secretOutput,++ },++ });++ assert.strictEqual(written.status, 201, written.text);++ const ev = written.json().evidence;++ assert.strictEqual(ev.exact_id, EXACT);++ assert.strictEqual(ev.family, FAMILY);++ assert.strictEqual(ev.sample_count, 4);++ assert.strictEqual(ev.success_rate, 0.75);++ assert.strictEqual(ev.timeout_rate, 0.25);++ assert.strictEqual(ev.metrics.ttft_p50_ms, 210);++ assert.strictEqual(ev.provenance.kind, 'round');++ assert.strictEqual(ev.provenance.receipt, finishedRound.receipt);++ assert.strictEqual(ev.sparse, true);++ assert.strictEqual(ev.confidence.level, 'sparse');++ assert.strictEqual(ev.confidence.success_rate.estimate, 0.75);++ });++++ await check('a sparse exact id stays sparse and is never filled from its family', async () => {++ const dense = await t.get('/api/v1/variant-evidence', {++ bearer: roundWriter,++ json: {++ exact_id: 'qwen/qwen3-9b-abliterated:q8_0@rev2',++ source: { kind: 'round', id: finishedRound.id },++ sample_count: 50,++ success_count: 49,++ timeout_count: 1,++ ttft_p50_ms: 150,++ ttft_p95_ms: 300,++ prefill_tokens_per_s: 2000,++ decode_tokens_per_s: 120,++ },++ });++ assert.strictEqual(dense.status, 201, dense.text);++ const family = await t.get(`/api/v1/variant-families/${encodeURIComponent(FAMILY)}`);++ assert.strictEqual(family.status, 200, family.text);++ const variants = family.json().family.variants;++ assert.strictEqual(variants.length, 2);++ const q4 = variants.find((v) => v.exact_id === EXACT);++ assert.strictEqual(q4.sample_count, 4);++ assert.strictEqual(q4.sparse, true);++ assert.strictEqual(q4.metrics.decode_tokens_per_s, 86.5);++ const q8 = variants.find((v) => v.exact_id.includes('q8_0'));++ assert.strictEqual(q8.sample_count, 50);++ assert.strictEqual(q8.sparse, false);++ assert.strictEqual(q8.confidence.level, 'moderate');++++ const one = await t.get(`/api/v1/variant-evidence/${encodeURIComponent(EXACT)}`);++ assert.strictEqual(one.status, 200, one.text);++ assert.strictEqual(one.json().evidence.sample_count, 4);++ assert.strictEqual(one.json().evidence.metrics.decode_tokens_per_s, 86.5);++ });++++ await check('the family index groups one org/model and hides no sparse variant', async () => {++ const list = await t.get('/api/v1/variant-families');++ assert.strictEqual(list.status, 200, list.text);++ const qwen = list.json().families.find((f) => f.family === FAMILY);++ assert.ok(qwen);++ assert.strictEqual(qwen.variants.length, 2);++ assert.strictEqual(qwen.sample_count, 54);++ const all = await t.get(`/api/v1/variant-evidence?family=${encodeURIComponent(FAMILY)}`);++ assert.strictEqual(all.status, 200, all.text);++ assert.strictEqual(all.json().count, 2);++ });++++ await check('nothing in the evidence rows or the read API carries the private prompt or output', async () => {++ const rows = await t.ctx.s.db.many('SELECT * FROM bench_variant_evidence');++ const stored = JSON.stringify(rows);++ assert.ok(!stored.includes(secret), 'the prompt was stored in the evidence rows');++ assert.ok(!stored.includes(secretOutput), 'the output was stored in the evidence rows');++ const read = await t.get(`/api/v1/variant-evidence/${encodeURIComponent(EXACT)}`);++ assert.ok(!read.text.includes(secret));++ assert.ok(!read.text.includes(secretOutput));++ for (const key of ['prompt', 'output', 'payload', 'transcript']) assert.ok(!(key in read.json().evidence), key);++ });++++ await check('an unauthenticated write is refused and a person is not a provenance writer', async () => {++ const anon = await t.get('/api/v1/variant-evidence', { json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 1 } });++ assert.strictEqual(anon.status, 401, anon.text);++ const person = await t.get('/api/v1/variant-evidence', { as: alice, json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 1 } });++ assert.strictEqual(person.status, 403, person.text);++ const noCap = await t.get('/api/v1/variant-evidence', { bearer: wrongCap, json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 1 } });++ assert.strictEqual(noCap.status, 403, noCap.text);++ });++++ await check('a queued round is not pinned provenance', async () => {++ const task = await t.get('/api/v1/kinds/design.page/tasks', { as: alice, json: { prompt: 'A second page.', public: true } });++ const queued = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });++ const written = await t.get('/api/v1/variant-evidence', {++ bearer: roundWriter,++ json: { exact_id: 'qwen/qwen3-9b@queued', source: { kind: 'round', id: queued.json().round.id }, sample_count: 1 },++ });++ assert.strictEqual(written.status, 409, written.text);++ assert.strictEqual(written.json().code, 'bench.round_not_pinned');++ });++++ await check('a model agent may not write evidence about itself', async () => {++ // A service token whose identity claims the model it would measure: the sub is a valid service sub, and the++ // model claim is what names the model. Bench refuses it even though the capability is present.++ const now = Math.floor(Date.now() / 1000);++ const self = t.network.signJwt({++ iss: t.network.issuer(), sub: 'svc:model-runner', actor_type: 'service', aud: ['openvibe.bench'],++ cap: ['bench.entry.create'], ns: [], model: FAMILY, iat: now, exp: now + 300, jti: crypto.randomUUID(),++ });++ const written = await t.get('/api/v1/variant-evidence', {++ bearer: self,++ json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 2, success_count: 2 },++ });++ assert.strictEqual(written.status, 403, written.text);++ assert.strictEqual(written.json().code, 'bench.self_rating');++ });++++ await check('an authenticated verified run is provenance too', async () => {++ const definition = {++ title: 'Variant evidence run', description: 'A run for the P6 evidence path.', kind: 'games.play',++ tags: ['evidence'], visibility: 'public',++ tasks: [{ id: 't1', prompt: 'Play the game.', files: [], expected: null, holdout: false }],++ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },++ limits: { ms: 120000, tokens: 200000, usd: 0.5 },++ matrix: { models: ['acme/model'], harnesses: ['run'] },++ };++ const created = await t.get('/api/v1/benchmarks', { bearer: publishers, json: definition });++ assert.strictEqual(created.status, 201, created.text);++ benchmarkId = created.json().benchmark.id;++ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: publishers, json: {} });++ assert.strictEqual(frozen.status, 201, frozen.text);++ const run = await t.get('/api/v1/runs', {++ bearer: publishers,++ json: {++ benchmark: `${benchmarkId}@1`,++ model: 'acme/model:q4@rev9',++ route: { provider: 'acme' }, provider: 'acme',++ harness: { name: 'run', version: '1.0' },++ settings: {}, executor: { kind: 'run', id: 'job_00000000000000000000000000' },++ tasks: [{ task_id: 't1', score: 90, cost_usd: 0.01, ms: 1000, tokens: 100, artifacts: [] }],++ score: 90, cost_usd: 0.01, ms: 1000, tokens: 100,++ },++ });++ assert.strictEqual(run.status, 201, run.text);++ verifiedRunId = run.json().run.id;++ const written = await t.get('/api/v1/variant-evidence', {++ bearer: publishers,++ json: {++ exact_id: 'acme/model:q4@rev9',++ source: { kind: 'run', id: verifiedRunId },++ sample_count: 12, success_count: 12, timeout_count: 0,++ ttft_p50_ms: 90, ttft_p95_ms: 140, prefill_tokens_per_s: 900, decode_tokens_per_s: 70,++ measured_at: '2026-11-04T11:00:00.000Z',++ },++ });++ assert.strictEqual(written.status, 201, written.text);++ assert.strictEqual(written.json().evidence.provenance.kind, 'run');++ assert.strictEqual(written.json().evidence.provenance.verified, true);++ assert.strictEqual(written.json().evidence.provenance.bench_run_id, verifiedRunId);++ assert.strictEqual(written.json().evidence.confidence.level, 'low');++ });++++ await check('an authenticated Run job id is recorded as run provenance', async () => {++ const jobId = 'job_01JZ0000000000000000000000';++ const written = await t.get('/api/v1/variant-evidence', {++ bearer: publishers,++ json: {++ exact_id: 'acme/model:q8_0@rev10',++ source: { kind: 'run', id: jobId },++ sample_count: 30, success_count: 30, timeout_count: 0,++ ttft_p50_ms: 80, ttft_p95_ms: 120,++ },++ });++ assert.strictEqual(written.status, 201, written.text);++ assert.strictEqual(written.json().evidence.provenance.kind, 'run');++ assert.strictEqual(written.json().evidence.provenance.id, jobId);++ assert.strictEqual(written.json().evidence.provenance.executor, 'run');++ assert.strictEqual(written.json().evidence.provenance.verified, false);++ });++++ await check('a submitted run is not evidence, and a missing source is a 404', async () => {++ const submitted = await t.get('/api/v1/runs', {++ as: alice,++ json: {++ benchmark: `${benchmarkId}@1`, model: 'acme/model:q4@rev9',++ route: { provider: 'acme' }, provider: 'acme', harness: { name: 'run', version: '1.0' },++ settings: {}, tasks: [{ task_id: 't1', score: 80, cost_usd: 0.01, ms: 1000, tokens: 100, artifacts: [] }],++ score: 80, cost_usd: 0.01, ms: 1000, tokens: 100,++ },++ });++ assert.strictEqual(submitted.status, 201, submitted.text);++ const refused = await t.get('/api/v1/variant-evidence', {++ bearer: publishers,++ json: { exact_id: 'acme/model:q4@rev9', source: { kind: 'run', id: submitted.json().run.id }, sample_count: 5 },++ });++ assert.strictEqual(refused.status, 409, refused.text);++ assert.strictEqual(refused.json().code, 'bench.run_not_verified');++ const missing = await t.get('/api/v1/variant-evidence', {++ bearer: roundWriter,++ json: { exact_id: 'acme/model:q4@rev9', source: { kind: 'round', id: 'brd_00000000000000000000000000' }, sample_count: 5 },++ });++ assert.strictEqual(missing.status, 404, missing.text);++ assert.strictEqual(missing.json().code, 'bench.round_not_found');++ });++++ await t.close();++++ const off = await boot();++ await check('the flag off leaves the routes absent and the rest of Bench unchanged', async () => {++ assert.strictEqual(off.config.bench.variantEvidence, false);++ const families = await off.get('/api/v1/variant-families');++ assert.strictEqual(families.status, 404, families.text);++ assert.strictEqual(families.json().code, 'route.not_found');++ const writer = off.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.entry.create'] });++ const write = await off.get('/api/v1/variant-evidence', { bearer: writer, json: { exact_id: EXACT, source: { kind: 'round', id: 'brd_00000000000000000000000000' }, sample_count: 1 } });++ assert.strictEqual(write.status, 404, write.text);++ const kinds = await off.get('/api/v1/kinds');++ assert.strictEqual(kinds.status, 200, kinds.text);++ });++ await off.close();++++ done();++})();
@@ -137,6 +137,98 @@ granted is refused with `bench.clip_source_denied` and nothing is stored.
@@ -137,6 +137,98 @@ granted is refused with `bench.clip_source_denied` and nothing is stored.
(`GET /api/v1/judgements/next?kind=live.clip`) and on the battle page, and never once `expires_at` has passed: an
(`GET /api/v1/judgements/next?kind=live.clip`) and on the battle page, and never once `expires_at` has passed: an
expired clip is shown as expired and its link is not served.
expired clip is shown as expired and its link is not served.
+## Model-variant evidence (P6)
+
+P6 records measured throughput, TTFT and cold-start evidence per **exact model id**, grouped into variant families.
+It is Bench's evaluation evidence for a placement: Bench still never runs inference, and nothing here starts a run.
+The whole surface is behind `BENCH_VARIANT_EVIDENCE` and is **off** by default; with the flag off these routes do not
+exist (a 404 with `route.not_found`).
+
+An exact id is the D3 form `<org>/<model>[:<quant>]@<revision>`, and the quant and the revision are optional:
+
+```
+qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9
+qwen/qwen3-9b-abliterated:q4_k_m
+qwen/qwen3-9b-abliterated@main
+qwen/qwen3-9b-abliterated
+```
+
+The **family** is `<org>/<model>`, so the four ids above are one family with two quants and two revisions.
+
+| Route | Who | What |
+|---|---|---|
+| `GET /api/v1/variant-families` | anyone | Every family: its exact ids (variants), quants, revisions and sample counts. |
+| `GET /api/v1/variant-families/:family` | anyone | One family; `:family` is URL-encoded (`qwen%2Fqwen3-9b-abliterated`). |
+| `GET /api/v1/variant-evidence` | anyone | Every exact id's evidence; `?family=` filters one family. |
+| `GET /api/v1/variant-evidence/:exactId` | anyone | One exact id; `:exactId` is URL-encoded. |
+| `POST /api/v1/variant-evidence` | a service token | Record one observation from a pinned round or an authenticated run. |
+
+A write is accepted only from a service token, never from a person, an anonymous caller or a model's own agent:
+
+- **A pinned round** (`source.kind: "round"`, `source.id: "brd_…"`) needs `bench.entry.create`. The round must be
+ `finished` and carry its receipt; a queued or failed round is refused.
+- **A run** (`source.kind: "run"`) needs `bench.run.publish`. A Bench run (`brn_…`) must be `verified`; a Run job
+ (`job_…`) or AI run (`run_…`) is recorded as an authenticated run. A submitted run is refused.
+- A token whose identity names the model (for example `agent:qwen/qwen3-9b-abliterated`) is refused with
+ `bench.self_rating`: a model never rates itself.
+
+```json
+{
+ "exact_id": "qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9",
+ "source": { "kind": "round", "id": "brd_0123456789ABCDEFGHJKMNPQRS" },
+ "measured_at": "2026-11-04T10:00:00.000Z",
+ "sample_count": 120,
+ "success_count": 118,
+ "timeout_count": 2,
+ "ttft_p50_ms": 210,
+ "ttft_p95_ms": 480,
+ "prefill_tokens_per_s": 1420,
+ "decode_tokens_per_s": 86.5,
+ "cold_start_ms": 5300
+}
+```
+
+A rate may be sent instead of its count (`success_rate`, `timeout_rate`, each 0 to 1); it is converted with the
+sample count. `sample_count` is required. Every other number is optional, and a missing one stays `null` — it is never
+guessed or filled in from the family or another revision.
+
+The read returns the observations and their totals. The newest observation's metrics are also on the variant for
+convenience, and `observations` keeps each source's own numbers:
+
+```json
+{
+ "evidence": {
+ "exact_id": "qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9",
+ "family": "qwen/qwen3-9b-abliterated",
+ "quant": "q4_k_m",
+ "revision": "3f2a1c9",
+ "sample_count": 120,
+ "success_rate": 0.983333,
+ "timeout_rate": 0.016667,
+ "confidence": {
+ "level": "high",
+ "sparse": false,
+ "samples": 120,
+ "minimum_samples": 10,
+ "success_rate": { "estimate": 0.983333, "lower": 0.9412, "upper": 0.9965 }
+ },
+ "metrics": { "ttft_p50_ms": 210, "ttft_p95_ms": 480, "decode_tokens_per_s": 86.5 },
+ "ttft_p50_ms": 210,
+ "ttft_p95_ms": 480,
+ "provenance": { "kind": "round", "id": "brd_…", "receipt": "…" },
+ "observations": [ { "sample_count": 120, "metrics": { "…": "…" }, "provenance": { "…": "…" } } ]
+ }
+}
+```
+
+`confidence.level` is `sparse` under 10 samples, `low` from 10, `moderate` from 30 and `high` from 100;
+`sparse: true` is always explicit. The success and timeout intervals are Wilson bounds. A rate across several
+observations is returned only when every observation counts that outcome; a missing count is not a zero.
+
+**Private sets stay private.** The evidence table has no prompt, output, payload or transcript column, the write
+ignores any such fields, and the read routes never join a task, benchmark or entry. The only text in a row is an
+exact id, a family, and the id of the round or run that produced it.
+
## The older API
## The older API
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
@@ -119,3 +119,15 @@ GET /api/v1/percentile?benchmark=&model=&score=
@@ -119,3 +119,15 @@ GET /api/v1/percentile?benchmark=&model=&score=
`value` is the score per dollar, `speed` the score per minute, and `consistency` the spread of a model's scores
`value` is the score per dollar, `speed` the score per minute, and `consistency` the spread of a model's scores
inside one session (the lower, the steadier). The percentile says where a score sits among that model's runs.
inside one session (the lower, the steadier). The percentile says where a score sits among that model's runs.
+
+## Variants and measured evidence (P6, flag off)
+
+A model is not one id. A **variant family** groups the exact ids of one `<org>/<model>` across quants and revisions
+(`qwen/qwen3-9b-abliterated:q4_k_m@abc` and `qwen/qwen3-9b-abliterated:q8_0@def` are one family, two variants). Bench
+records typed measurements per exact id — TTFT p50/p95, prefill and decode tokens/s, cold start, success and timeout
+counts — and returns each with its sample count and confidence. Sparse evidence says so; it is never filled in from a
+sibling revision or the family.
+
+The evidence is written only by a **finished pinned round** or an authenticated **run**, never by a model about
+itself, and it carries no prompt or output text. The API is in [api.md](api.md#model-variant-evidence-p6). The whole
+surface is behind `BENCH_VARIANT_EVIDENCE` and is off by default.
new file mode 100644
new file mode 100644
@@ -0,0 +1,48 @@
@@ -0,0 +1,48 @@
+-- phase: expand
+-- P6 (plan 2026-11-04, G-P6 and ObservedPerformance): measured model-variant evidence.
+--
+-- One row is one observation batch for one exact id from one source. The exact id is the D3 form
+-- <org>/<model>[:<quant>]@<revision>; the family is <org>/<model>, so the variants of one model group across
+-- quants and revisions. The source is a pinned Bench round or an authenticated Run job / verified Bench run.
+-- Nothing here is prompt or output text: only identifiers, counts and measurements are stored, so a private
+-- benchmark's prompt can never leak through this table or its read API.
+
+CREATE TABLE bench_variant_evidence (
+ id text COLLATE "C" PRIMARY KEY, -- bve_ + ULID
+ exact_id text COLLATE "C" NOT NULL, -- org/model[:quant]@revision
+ family text COLLATE "C" NOT NULL, -- org/model
+ quant text COLLATE "C",
+ revision text COLLATE "C",
+ source_kind text COLLATE "C" NOT NULL, -- round | run
+ source_id text COLLATE "C" NOT NULL, -- brd_…, brn_…, job_… or run_…
+ provenance jsonb NOT NULL DEFAULT '{}'::jsonb,
+ measured_at timestamptz NOT NULL,
+ sample_count integer NOT NULL,
+ success_count integer,
+ timeout_count integer,
+ ttft_p50_ms integer,
+ ttft_p95_ms integer,
+ prefill_tokens_per_s numeric(14, 3),
+ decode_tokens_per_s numeric(14, 3),
+ cold_start_ms integer,
+ recorded_by text COLLATE "C" NOT NULL,
+ created_at timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT bench_variant_evidence_id CHECK (id ~ '^bve_[0-9A-HJKMNP-TV-Z]{26}$'),
+ CONSTRAINT bench_variant_evidence_exact CHECK (length(exact_id) BETWEEN 3 AND 200),
+ CONSTRAINT bench_variant_evidence_family CHECK (length(family) BETWEEN 3 AND 200),
+ CONSTRAINT bench_variant_evidence_source CHECK (source_kind IN ('round', 'run')),
+ CONSTRAINT bench_variant_evidence_source_id CHECK (length(source_id) BETWEEN 3 AND 200),
+ CONSTRAINT bench_variant_evidence_samples CHECK (sample_count > 0),
+ CONSTRAINT bench_variant_evidence_success CHECK (success_count IS NULL OR (success_count >= 0 AND success_count <= sample_count)),
+ CONSTRAINT bench_variant_evidence_timeout CHECK (timeout_count IS NULL OR (timeout_count >= 0 AND timeout_count <= sample_count)),
+ CONSTRAINT bench_variant_evidence_counts CHECK (success_count IS NULL OR timeout_count IS NULL OR success_count + timeout_count <= sample_count),
+ CONSTRAINT bench_variant_evidence_ttft_p50 CHECK (ttft_p50_ms IS NULL OR ttft_p50_ms >= 0),
+ CONSTRAINT bench_variant_evidence_ttft_p95 CHECK (ttft_p95_ms IS NULL OR ttft_p95_ms >= 0),
+ CONSTRAINT bench_variant_evidence_ttft_order CHECK (ttft_p50_ms IS NULL OR ttft_p95_ms IS NULL OR ttft_p95_ms >= ttft_p50_ms),
+ CONSTRAINT bench_variant_evidence_prefill CHECK (prefill_tokens_per_s IS NULL OR prefill_tokens_per_s >= 0),
+ CONSTRAINT bench_variant_evidence_decode CHECK (decode_tokens_per_s IS NULL OR decode_tokens_per_s >= 0),
+ CONSTRAINT bench_variant_evidence_cold CHECK (cold_start_ms IS NULL OR cold_start_ms >= 0),
+ UNIQUE (exact_id, source_kind, source_id)
+);
+CREATE INDEX bench_variant_evidence_family ON bench_variant_evidence (family, measured_at DESC);
+CREATE INDEX bench_variant_evidence_exact ON bench_variant_evidence (exact_id, measured_at DESC);
new file mode 100644
new file mode 100644
@@ -0,0 +1,28 @@
@@ -0,0 +1,28 @@
+-- phase: expand
+-- D11 owner settings: openvibe-shared/config (config_snapshots, config_keys). Applied migrations are immutable.
+CREATE TABLE IF NOT EXISTS config_snapshots (
+ namespace text COLLATE "C" NOT NULL,
+ revision bigint NOT NULL,
+ service text COLLATE "C" NOT NULL,
+ previous_revision bigint,
+ state text COLLATE "C" NOT NULL CHECK (state IN ('proposed', 'active', 'superseded', 'rejected', 'rolled_back')),
+ values_json text COLLATE "C" NOT NULL, -- the full values; secret-class values are envelopes (never their plaintext)
+ classification_json text COLLATE "C" NOT NULL,
+ values_checksum text COLLATE "C" NOT NULL, -- sha256 of the values as served (secrets as markers)
+ created_at text COLLATE "C" NOT NULL,
+ created_by text COLLATE "C" NOT NULL,
+ activated_at text COLLATE "C",
+ activated_by text COLLATE "C",
+ reason text COLLATE "C",
+ error text COLLATE "C",
+ copied_from bigint,
+ good bigint NOT NULL DEFAULT 0, -- 1 once it activated successfully (last-known-good)
+ PRIMARY KEY (namespace, revision)
+);
+CREATE UNIQUE INDEX IF NOT EXISTS config_snapshots_one_active ON config_snapshots (namespace) WHERE state = 'active';
+-- The fingerprint key of each namespace. Never served, never logged.
+CREATE TABLE IF NOT EXISTS config_keys (
+ namespace text COLLATE "C" PRIMARY KEY,
+ hmac_key bytea NOT NULL, -- the 32-byte fingerprint key, or that key wrapped under the current KEK
+ created_at text COLLATE "C" NOT NULL
+);
@@ -15,9 +15,9 @@
@@ -15,9 +15,9 @@
"express-rate-limit": "^7.5.0",
"express-rate-limit": "^7.5.0",
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.247.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.288.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.110.0",
"pg": "^8.23.0",
"pg": "^8.23.0",
"three": "0.170.0"
"three": "0.170.0"
},
},
@@ -1151,9 +1151,9 @@
@@ -1151,9 +1151,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.247.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.247.0",
- "integrity": "sha512-ZBPPRdIygPqwENt99cMaJ4K2/yN4go9wQ6WTzjRVmdejT5qWij2CNLr3aSRRt+C83Do0qgGAO0469F6ONRa/Ng==",
+ "version": "0.288.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.288.0",
+ "integrity": "sha512-AKRP1yXACYNu7okpf4kCDq4ds0PhsoNvD92VlvBWBCjcSlNimSTQhLWIvSa3D31/kQYz91fodo06ZnwuTanOqg==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -1195,9 +1195,9 @@
@@ -1195,9 +1195,9 @@
}
}
},
},
"node_modules/openvibe-shared": {
"node_modules/openvibe-shared": {
- "version": "4.109.2",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",
- "integrity": "sha512-t9K1yHr6GOJsfamqmgERRTTtiWYWAVmjWBEm4ykTcYjEenbTiqQTwX6CJsZW4lYnqS1IlqEvn+0jG4vXmHSJ3Q==",
+ "version": "4.110.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.110.0",
+ "integrity": "sha512-ZNOSraSgE89jhLTI//lFJALvKCiBJIBwf+V0aCkiIk0VCrPPAytsfsgOgCINU8kXDVdCEnuWWfcA9hZUc54QVA==",
"license": "MIT",
"license": "MIT",
"dependencies": {
"dependencies": {
"jsonwebtoken": "^9.0.0",
"jsonwebtoken": "^9.0.0",
@@ -22,9 +22,9 @@
@@ -22,9 +22,9 @@
"express-rate-limit": "^7.5.0",
"express-rate-limit": "^7.5.0",
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.247.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.288.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.110.0",
"pg": "^8.23.0",
"pg": "^8.23.0",
"three": "0.170.0"
"three": "0.170.0"
},
},
@@ -24,8 +24,10 @@ const { openStore } = require('./db');
@@ -24,8 +24,10 @@ const { openStore } = require('./db');
const { createKeyStore } = require('./auth/keys');
const { createKeyStore } = require('./auth/keys');
const { createSso } = require('./auth/sso');
const { createSso } = require('./auth/sso');
const { createPrincipal } = require('./http/principal');
const { createPrincipal } = require('./http/principal');
+const createVariantEvidenceConfig = require('./variant-evidence-config').create;
const { createApi } = require('./http/api');
const { createApi } = require('./http/api');
const { createRegistryApi } = require('./http/registry');
const { createRegistryApi } = require('./http/registry');
+const { createVariantEvidenceApi } = require('./http/variant-evidence');
const { createPageRoutes } = require('./http/pages');
const { createPageRoutes } = require('./http/pages');
const { createShowcaseRoutes, createShowcasePages } = require('./http/showcase');
const { createShowcaseRoutes, createShowcasePages } = require('./http/showcase');
const { createGalleryRoutes, createGalleryPages } = require('./http/gallery');
const { createGalleryRoutes, createGalleryPages } = require('./http/gallery');
@@ -74,6 +76,9 @@ async function createApp(opts = {}) {
@@ -74,6 +76,9 @@ async function createApp(opts = {}) {
const keys = createKeyStore({ config, fetchImpl, log });
const keys = createKeyStore({ config, fetchImpl, log });
const sso = createSso({ config, keys, fetchImpl, now: opts.now || (() => Date.now()), log });
const sso = createSso({ config, keys, fetchImpl, now: opts.now || (() => Date.now()), log });
const principal = createPrincipal({ config, keys });
const principal = createPrincipal({ config, keys });
+ // D11 owner settings (bench.variant_evidence): the switch is read through the store, env pins winning.
+ const evidenceConfig = await createVariantEvidenceConfig({ db: s.db, principal, config, env: opts.env || process.env, log });
+ Object.defineProperty(config.bench, 'variantEvidence', { enumerable: true, configurable: true, get: () => evidenceConfig.get('enabled') });
const bench = opts.bench || createStore(s, { config, log, random: opts.random });
const bench = opts.bench || createStore(s, { config, log, random: opts.random });
const registry = opts.registry || createRegistry(s, { log });
const registry = opts.registry || createRegistry(s, { log });
@@ -97,9 +102,11 @@ async function createApp(opts = {}) {
@@ -97,9 +102,11 @@ async function createApp(opts = {}) {
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Bench cannot answer account events'); });
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Bench cannot answer account events'); });
const launcher = createLauncher({ config, registry, fetchImpl: fetchImpl || globalThis.fetch, log });
const launcher = createLauncher({ config, registry, fetchImpl: fetchImpl || globalThis.fetch, log });
+ // P6: model-variant evidence. Constructed always so the owner switch applies live; its API returns 404 while off.
+ const variantEvidence = require('./bench/variant-evidence').createVariantEvidence({ s, log });
const ctx = {
const ctx = {
config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,
config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,
- anon, daily,
+ anon, daily, variantEvidence, evidenceConfig,
imports, mirror,
imports, mirror,
random: opts.random || Math.random,
random: opts.random || Math.random,
};
};
@@ -148,6 +155,8 @@ async function createApp(opts = {}) {
@@ -148,6 +155,8 @@ async function createApp(opts = {}) {
const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);
const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);
ctx.valkey = valkey;
ctx.valkey = valkey;
+ evidenceConfig.mount(app);
+
app.use(contracts.http.middleware());
app.use(contracts.http.middleware());
app.use(helmet({
app.use(helmet({
contentSecurityPolicy: {
contentSecurityPolicy: {
@@ -230,6 +239,7 @@ async function createApp(opts = {}) {
@@ -230,6 +239,7 @@ async function createApp(opts = {}) {
app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.BENCH_API_RATE_LIMIT_PER_MIN) || 240, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));
app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.BENCH_API_RATE_LIMIT_PER_MIN) || 240, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));
app.use('/api/v1', createApi(ctx));
app.use('/api/v1', createApi(ctx));
app.use('/api/v1', createRegistryApi(ctx));
app.use('/api/v1', createRegistryApi(ctx));
+ app.use('/api/v1', createVariantEvidenceApi(ctx));
app.use('/api/v1/showcase', createShowcaseRoutes(ctx));
app.use('/api/v1/showcase', createShowcaseRoutes(ctx));
app.use('/api/v1/gallery', createGalleryRoutes(ctx));
app.use('/api/v1/gallery', createGalleryRoutes(ctx));
app.use('/api/v1', createPickRoutes(ctx));
app.use('/api/v1', createPickRoutes(ctx));
new file mode 100644
new file mode 100644
@@ -0,0 +1,516 @@
@@ -0,0 +1,516 @@
+'use strict';
+
+/**
+ * P6 (plan 2026-11-04, G-P6 and ObservedPerformance): model-variant evidence.
+ *
+ * Bench is the evaluation authority, never an inference gateway. This module keeps the measured, typed evidence a
+ * placement can read: one exact id (D3: org/model[:quant]@revision) per row, grouped by its family org/model across
+ * quants and revisions. A row is written only when it names a finished pinned Bench round or an authenticated run,
+ * and a model's own agent identity may never write evidence for itself.
+ *
+ * The numbers are stored as counts and integers so the read API cannot invent precision:
+ * - sample_count, success_count, timeout_count;
+ * - ttft_p50_ms, ttft_p95_ms, cold_start_ms (whole milliseconds);
+ * - prefill_tokens_per_s, decode_tokens_per_s (three decimal places).
+ * No prompt, output, payload or transcript is accepted or stored. The read API never joins a task, benchmark or
+ * entry, so private sets stay private.
+ */
+
+const ratingLib = require('../rating/lib');
+
+const MAX_EXACT_ID = 200;
+const MIN_SAMPLES = 10;
+const MODERATE_SAMPLES = 30;
+const HIGH_SAMPLES = 100;
+const MAX_OBSERVATIONS = 5000;
+const MAX_MS = 3_600_000;
+const MAX_TPS = 1e7;
+
+/** The D3 exact id: <org>/<model>[:<quant>]@<revision>, with the quant and the revision optional. */
+const EXACT_ID_RE = /^([A-Za-z0-9][A-Za-z0-9._-]*)\/([A-Za-z0-9][A-Za-z0-9._-]*)(?::([A-Za-z0-9][A-Za-z0-9._-]*))?(?:@([A-Za-z0-9][A-Za-z0-9._-]*))?$/;
+const IDENTITY_PREFIX = /^(?:svc|service|app|mod|agent):/i;
+
+const SELF_RATING = 'bench.self_rating';
+
+class VariantEvidenceError extends Error {
+ constructor(status, code, detail) {
+ super(detail);
+ this.name = 'VariantEvidenceError';
+ this.status = status;
+ this.code = code;
+ this.detail = detail;
+ }
+}
+
+const text = (v) => (v == null ? '' : String(v).trim());
+const objectOf = (v) => (v && typeof v === 'object' && !Array.isArray(v) ? v : {});
+const firstDefined = (...values) => values.find((v) => v !== undefined && v !== null && v !== '');
+const round3 = (v) => (v == null ? null : Math.round(Number(v) * 1000) / 1000);
+const round6 = (v) => (v == null ? null : Math.round(Number(v) * 1e6) / 1e6);
+const numberOrNull = (v) => (v == null || v === '' ? null : Number(v));
+
+function parseExactId(value) {
+ const raw = text(value);
+ if (!raw) return { ok: false, error: 'exact_id is required' };
+ if (raw.length > MAX_EXACT_ID) return { ok: false, error: `exact_id must be at most ${MAX_EXACT_ID} characters` };
+ const match = EXACT_ID_RE.exec(raw);
+ if (!match) return { ok: false, error: 'exact_id must be org/model[:quant]@revision, with the quant and the revision optional' };
+ const [, org, model, quant, revision] = match;
+ return {
+ ok: true,
+ value: {
+ exact_id: raw,
+ org,
+ model,
+ family: `${org}/${model}`,
+ quant: quant || null,
+ revision: revision || null,
+ },
+ };
+}
+
+/** A family is exactly org/model: no quant and no revision. */
+function parseFamily(value) {
+ const raw = text(value);
+ if (!raw || raw.length > MAX_EXACT_ID) return null;
+ if (raw.includes(':') || raw.includes('@')) return null;
+ const parsed = parseExactId(raw);
+ if (!parsed.ok) return null;
+ return parsed.value;
+}
+
+/**
+ * How much weight the sample count carries. Sparse evidence is never hidden or filled in: it is returned with the
+ * count that makes it sparse. The 10-sample line is the same provisional line the rest of Bench uses.
+ */
+function confidenceFor(samples) {
+ const n = Number.isSafeInteger(Number(samples)) && Number(samples) >= 0 ? Number(samples) : 0;
+ let level = 'none';
+ if (n >= HIGH_SAMPLES) level = 'high';
+ else if (n >= MODERATE_SAMPLES) level = 'moderate';
+ else if (n >= MIN_SAMPLES) level = 'low';
+ else if (n > 0) level = 'sparse';
+ return { level, sparse: n < MIN_SAMPLES, samples: n, minimum_samples: MIN_SAMPLES };
+}
+
+function rateOf(count, sample) {
+ if (count == null || !sample) return null;
+ return round6(Number(count) / Number(sample));
+}
+
+function confidenceInterval(successes, samples) {
+ if (successes == null || !samples) return null;
+ const w = ratingLib.wilson(Number(successes), Number(samples));
+ return { estimate: round6(w.p), lower: round6(w.lower), upper: round6(w.upper) };
+}
+
+function readInteger(errors, label, value, { min = 0, max = null } = {}) {
+ if (value === undefined || value === null || value === '') return null;
+ if (typeof value !== 'number' && typeof value !== 'string') {
+ errors.push(`${label} must be a whole number`);
+ return null;
+ }
+ const n = Number(value);
+ if (!Number.isFinite(n) || !Number.isInteger(n) || n < min || (max != null && n > max)) {
+ errors.push(`${label} must be a whole number${max == null ? ` at least ${min}` : ` from ${min} to ${max}`}`);
+ return null;
+ }
+ return n;
+}
+
+function readDecimal(errors, label, value, { min = 0, max = MAX_TPS } = {}) {
+ if (value === undefined || value === null || value === '') return null;
+ if (typeof value !== 'number' && typeof value !== 'string') {
+ errors.push(`${label} must be a number`);
+ return null;
+ }
+ const n = Number(value);
+ if (!Number.isFinite(n) || n < min || n > max) {
+ errors.push(`${label} must be a number from ${min} to ${max}`);
+ return null;
+ }
+ return n;
+}
+
+function readRate(errors, label, value) {
+ if (value === undefined || value === null || value === '') return null;
+ if (typeof value !== 'number' && typeof value !== 'string') {
+ errors.push(`${label} must be a fraction from 0 to 1`);
+ return null;
+ }
+ const n = Number(value);
+ if (!Number.isFinite(n) || n < 0 || n > 1) {
+ errors.push(`${label} must be a fraction from 0 to 1`);
+ return null;
+ }
+ return n;
+}
+
+function readSource(body) {
+ const raw = body.source;
+ const source = typeof raw === 'string' ? { id: raw } : objectOf(firstDefined(raw, body.provenance));
+ let kind = text(firstDefined(source.kind, source.type)).toLowerCase();
+ if (kind === 'pinned_round') kind = 'round';
+ if (kind === 'run_job') kind = 'run';
+ const roundId = firstDefined(source.round_id, body.round_id, body.pinned_round);
+ const runId = firstDefined(source.run_id, body.run_id, body.job_id, body.run_job_id);
+ const id = text(firstDefined(source.id, roundId, runId));
+ if (!kind) kind = roundId ? 'round' : (runId ? 'run' : '');
+ return { kind, id };
+}
+
+/**
+ * Validate and normalize one observation. Unknown fields are ignored, never stored: a caller cannot smuggle prompt
+ * or output text into an evidence row. Returns { ok: true, value } or { ok: false, errors }.
+ */
+function validateObservation(body, { now = new Date().toISOString() } = {}) {
+ const input = objectOf(body);
+ const errors = [];
+
+ const exact = parseExactId(firstDefined(input.exact_id, input.exactId, input.model_id));
+ if (!exact.ok) errors.push(exact.error);
+
+ const source = readSource(input);
+ if (!source.kind) errors.push('source.kind must be round or run');
+ else if (source.kind !== 'round' && source.kind !== 'run') errors.push('source.kind must be round or run');
+ if (!source.id) errors.push('source.id is required');
+ else if (source.id.length > MAX_EXACT_ID) errors.push(`source.id must be at most ${MAX_EXACT_ID} characters`);
+ if (source.kind === 'round' && source.id && !/^brd_[0-9A-HJKMNP-TV-Z]{26}$/.test(source.id)) {
+ errors.push('a round source id must be a Bench round id (brd_…)');
+ }
+ if (source.kind === 'run' && source.id && !/^(?:brn|job)_[0-9A-HJKMNP-TV-Z]{26}$/.test(source.id) && !/^run_[A-Za-z0-9_-]{1,120}$/.test(source.id)) {
+ errors.push('a run source id must be a Bench run id (brn_…), a Run job id (job_…) or an AI run id (run_…)');
+ }
+
+ const sampleCount = readInteger(errors, 'sample_count', firstDefined(input.sample_count, input.samples, input.n), { min: 1, max: 10_000_000 });
+
+ const ttft = objectOf(firstDefined(input.ttft, input.ttft_ms));
+ const ttftP50 = readInteger(errors, 'ttft_p50_ms', firstDefined(input.ttft_p50_ms, input.ttft_ms_p50, input.ttft_p50, ttft.p50_ms, ttft.p50), { min: 0, max: MAX_MS });
+ const ttftP95 = readInteger(errors, 'ttft_p95_ms', firstDefined(input.ttft_p95_ms, input.ttft_ms_p95, input.ttft_p95, ttft.p95_ms, ttft.p95), { min: 0, max: MAX_MS });
+ if (ttftP50 != null && ttftP95 != null && ttftP95 < ttftP50) errors.push('ttft_p95_ms must be at least ttft_p50_ms');
+
+ const throughput = objectOf(input.throughput);
+ const prefill = readDecimal(errors, 'prefill_tokens_per_s', firstDefined(
+ input.prefill_tokens_per_s, input.prefill_tps, input.prefill_tokens_s,
+ throughput.prefill_tokens_per_s, throughput.prefill_tps,
+ ));
+ const decode = readDecimal(errors, 'decode_tokens_per_s', firstDefined(
+ input.decode_tokens_per_s, input.decode_tps, input.decode_tokens_s,
+ throughput.decode_tokens_per_s, throughput.decode_tps,
+ ));
+ const coldStart = readInteger(errors, 'cold_start_ms', firstDefined(
+ input.cold_start_ms, input.coldStartMs, typeof input.cold_start === 'number' ? input.cold_start : undefined,
+ objectOf(input.cold_start).ms, objectOf(input.cold_start).cold_start_ms,
+ ), { min: 0, max: MAX_MS });
+
+ const successInfo = objectOf(input.success);
+ const timeoutInfo = objectOf(input.timeout);
+ const successCount = readInteger(errors, 'success_count', firstDefined(
+ input.success_count, input.successes, successInfo.count, typeof input.success === 'number' ? input.success : undefined,
+ ), { min: 0, max: sampleCount == null ? undefined : sampleCount });
+ const timeoutCount = readInteger(errors, 'timeout_count', firstDefined(
+ input.timeout_count, input.timeouts, timeoutInfo.count, typeof input.timeout === 'number' ? input.timeout : undefined,
+ ), { min: 0, max: sampleCount == null ? undefined : sampleCount });
+ const successRate = readRate(errors, 'success_rate', firstDefined(input.success_rate, successInfo.rate));
+ const timeoutRate = readRate(errors, 'timeout_rate', firstDefined(input.timeout_rate, timeoutInfo.rate));
+
+ let successes = successCount;
+ let timeouts = timeoutCount;
+ if (sampleCount != null) {
+ if (successes == null && successRate != null) successes = Math.round(successRate * sampleCount);
+ if (timeouts == null && timeoutRate != null) timeouts = Math.round(timeoutRate * sampleCount);
+ if (successes != null && timeouts != null && successes + timeouts > sampleCount) {
+ errors.push('success_count and timeout_count cannot add up to more than sample_count');
+ }
+ }
+
+ let measuredAt = text(now) || new Date().toISOString();
+ const measuredInput = firstDefined(input.measured_at, input.measuredAt, input.at);
+ if (measuredInput != null) {
+ const ms = Date.parse(measuredInput);
+ if (!Number.isFinite(ms)) errors.push('measured_at must be an ISO timestamp');
+ else measuredAt = new Date(ms).toISOString();
+ }
+
+ if (errors.length) return { ok: false, errors };
+ return {
+ ok: true,
+ value: {
+ exact_id: exact.value.exact_id,
+ family: exact.value.family,
+ org: exact.value.org,
+ model: exact.value.model,
+ quant: exact.value.quant,
+ revision: exact.value.revision,
+ source_kind: source.kind,
+ source_id: source.id,
+ sample_count: sampleCount,
+ success_count: successes,
+ timeout_count: timeouts,
+ ttft_p50_ms: ttftP50,
+ ttft_p95_ms: ttftP95,
+ prefill_tokens_per_s: prefill == null ? null : round3(prefill),
+ decode_tokens_per_s: decode == null ? null : round3(decode),
+ cold_start_ms: coldStart,
+ measured_at: measuredAt,
+ },
+ };
+}
+
+/** True when the caller is the model it is trying to measure. A model never rates itself. */
+function isSelfRating(principal, value) {
+ if (!principal || !value) return false;
+ const claims = objectOf(principal.claims);
+ const names = [value.exact_id, value.family, value.model, `${value.org}/${value.model}`]
+ .filter(Boolean).map((v) => String(v).toLowerCase());
+ const identities = [
+ principal.requester,
+ claims.sub,
+ claims.model,
+ claims.model_id,
+ claims.exact_id,
+ claims.model_key,
+ ].filter(Boolean).map((v) => String(v).trim().toLowerCase());
+ for (const identity of identities) {
+ const id = identity.replace(IDENTITY_PREFIX, '');
+ if (!id) continue;
+ if (names.includes(id)) return true;
+ const family = String(value.family || '').toLowerCase();
+ if (family && id.startsWith(family)) {
+ const rest = id.slice(family.length);
+ if (!rest || rest.startsWith(':') || rest.startsWith('@')) return true;
+ }
+ }
+ return false;
+}
+
+function wireObservation(row) {
+ const samples = numberOrNull(row.sample_count) || 0;
+ const successes = row.success_count == null ? null : numberOrNull(row.success_count);
+ const timeouts = row.timeout_count == null ? null : numberOrNull(row.timeout_count);
+ const provenance = typeof row.provenance === 'string' ? JSON.parse(row.provenance) : (row.provenance || {});
+ return {
+ id: row.id,
+ exact_id: row.exact_id,
+ family: row.family,
+ quant: row.quant || null,
+ revision: row.revision || null,
+ sample_count: samples,
+ success_count: successes,
+ success_rate: rateOf(successes, samples),
+ timeout_count: timeouts,
+ timeout_rate: rateOf(timeouts, samples),
+ metrics: {
+ ttft_p50_ms: row.ttft_p50_ms == null ? null : numberOrNull(row.ttft_p50_ms),
+ ttft_p95_ms: row.ttft_p95_ms == null ? null : numberOrNull(row.ttft_p95_ms),
+ prefill_tokens_per_s: round3(numberOrNull(row.prefill_tokens_per_s)),
+ decode_tokens_per_s: round3(numberOrNull(row.decode_tokens_per_s)),
+ cold_start_ms: row.cold_start_ms == null ? null : numberOrNull(row.cold_start_ms),
+ },
+ measured_at: row.measured_at == null ? null : (row.measured_at instanceof Date ? row.measured_at.toISOString() : String(row.measured_at)),
+ provenance,
+ recorded_by: row.recorded_by,
+ created_at: row.created_at == null ? null : (row.created_at instanceof Date ? row.created_at.toISOString() : String(row.created_at)),
+ };
+}
+
+/** One exact id's evidence: every observation, plus the totals and the confidence they carry. */
+function aggregateRows(exactId, rows) {
+ const list = (rows || []).filter(Boolean);
+ if (!list.length) return null;
+ const parsed = parseExactId(exactId);
+ const first = list[0];
+ const observations = list.map(wireObservation).sort((a, b) => String(b.measured_at || '').localeCompare(String(a.measured_at || '')) || String(b.id).localeCompare(String(a.id)));
+ const latest = observations[0] || null;
+ const samples = observations.reduce((sum, o) => sum + (o.sample_count || 0), 0);
+ // A rate is known only when every observation counts that outcome. A missing count is not a zero.
+ const completeSuccess = observations.every((o) => o.success_count != null);
+ const completeTimeout = observations.every((o) => o.timeout_count != null);
+ const successes = completeSuccess ? observations.reduce((sum, o) => sum + (o.success_count || 0), 0) : null;
+ const timeouts = completeTimeout ? observations.reduce((sum, o) => sum + (o.timeout_count || 0), 0) : null;
+ const confidence = confidenceFor(samples);
+ const successInterval = confidenceInterval(successes, samples);
+ const timeoutInterval = confidenceInterval(timeouts, samples);
+ if (successInterval) confidence.success_rate = successInterval;
+ if (timeoutInterval) confidence.timeout_rate = timeoutInterval;
+ const metrics = latest ? latest.metrics : {};
+ return {
+ exact_id: parsed.ok ? parsed.value.exact_id : String(exactId || ''),
+ family: parsed.ok ? parsed.value.family : first.family,
+ org: parsed.ok ? parsed.value.org : null,
+ model: parsed.ok ? parsed.value.model : null,
+ quant: parsed.ok ? parsed.value.quant : (first.quant || null),
+ revision: parsed.ok ? parsed.value.revision : (first.revision || null),
+ sample_count: samples,
+ success_count: successes,
+ success_rate: rateOf(successes, samples),
+ timeout_count: timeouts,
+ timeout_rate: rateOf(timeouts, samples),
+ confidence,
+ sparse: confidence.sparse,
+ metrics,
+ ...metrics,
+ measured_at: latest ? latest.measured_at : null,
+ provenance: latest ? latest.provenance : null,
+ observations,
+ };
+}
+
+/** Group evidence rows by exact id and then by family org/model. Pure, no I/O. */
+function groupFamilies(rows) {
+ const byFamily = new Map();
+ for (const row of (rows || [])) {
+ if (!row) continue;
+ const parsed = parseExactId(row.exact_id);
+ if (!parsed.ok) continue;
+ const family = parsed.value.family;
+ if (!byFamily.has(family)) byFamily.set(family, { org: parsed.value.org, model: parsed.value.model, rows: [] });
+ byFamily.get(family).rows.push(row);
+ }
+ const families = [];
+ for (const [family, group] of byFamily) {
+ const byExact = new Map();
+ for (const row of group.rows) {
+ if (!byExact.has(row.exact_id)) byExact.set(row.exact_id, []);
+ byExact.get(row.exact_id).push(row);
+ }
+ const variants = [...byExact.entries()].map(([exact, list]) => aggregateRows(exact, list)).filter(Boolean);
+ variants.sort((a, b) => String(a.exact_id).localeCompare(String(b.exact_id)));
+ families.push({
+ family,
+ org: group.org,
+ model: group.model,
+ variants,
+ quants: [...new Set(variants.map((v) => v.quant).filter(Boolean))].sort(),
+ revisions: [...new Set(variants.map((v) => v.revision).filter(Boolean))].sort(),
+ sample_count: variants.reduce((sum, v) => sum + (v.sample_count || 0), 0),
+ });
+ }
+ families.sort((a, b) => a.family.localeCompare(b.family));
+ return families;
+}
+
+function createVariantEvidence({ s, log = console }) {
+ async function sourceProvenance(sourceKind, sourceId) {
+ if (sourceKind === 'round') {
+ const row = await s.db.maybe('SELECT id, status, receipt FROM bench_rounds WHERE id = $1', [sourceId]);
+ if (!row) throw new VariantEvidenceError(404, 'bench.round_not_found', 'No such Bench round.');
+ if (row.status !== 'finished' || !row.receipt) {
+ throw new VariantEvidenceError(409, 'bench.round_not_pinned', 'Evidence can only come from a finished, pinned round with a receipt.');
+ }
+ return { kind: 'round', id: row.id, receipt: row.receipt };
+ }
+ if (sourceId.startsWith('brn_')) {
+ const row = await s.db.maybe('SELECT id, provenance FROM bench_runs WHERE id = $1', [sourceId]);
+ if (!row) throw new VariantEvidenceError(404, 'bench.run_not_found', 'No such Bench run.');
+ if (row.provenance !== 'verified') {
+ throw new VariantEvidenceError(409, 'bench.run_not_verified', 'Evidence can only come from a verified run.');
+ }
+ return { kind: 'run', id: row.id, verified: true, bench_run_id: row.id };
+ }
+ const viaRun = await s.db.maybe(
+ `SELECT id FROM bench_runs
+ WHERE provenance = 'verified' AND executor->>'kind' = 'run' AND executor->>'id' = $1
+ ORDER BY created_at DESC, id LIMIT 1`,
+ [sourceId],
+ ).catch(() => null);
+ return {
+ kind: 'run',
+ id: sourceId,
+ verified: Boolean(viaRun),
+ executor: sourceId.startsWith('job_') ? 'run' : 'ai',
+ bench_run_id: viaRun ? viaRun.id : null,
+ };
+ }
+
+ async function record(value, { createdBy }) {
+ const provenance = await sourceProvenance(value.source_kind, value.source_id);
+ const row = await s.db.maybe(
+ `INSERT INTO bench_variant_evidence
+ (id, exact_id, family, quant, revision, source_kind, source_id, provenance, measured_at,
+ sample_count, success_count, timeout_count, ttft_p50_ms, ttft_p95_ms, prefill_tokens_per_s,
+ decode_tokens_per_s, cold_start_ms, recorded_by, created_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9::timestamptz,
+ $10, $11, $12, $13, $14, $15, $16, $17, $18, $19::timestamptz)
+ ON CONFLICT (exact_id, source_kind, source_id) DO UPDATE SET
+ family = EXCLUDED.family, quant = EXCLUDED.quant, revision = EXCLUDED.revision,
+ provenance = EXCLUDED.provenance, measured_at = EXCLUDED.measured_at,
+ sample_count = EXCLUDED.sample_count, success_count = EXCLUDED.success_count,
+ timeout_count = EXCLUDED.timeout_count, ttft_p50_ms = EXCLUDED.ttft_p50_ms,
+ ttft_p95_ms = EXCLUDED.ttft_p95_ms, prefill_tokens_per_s = EXCLUDED.prefill_tokens_per_s,
+ decode_tokens_per_s = EXCLUDED.decode_tokens_per_s, cold_start_ms = EXCLUDED.cold_start_ms,
+ recorded_by = EXCLUDED.recorded_by
+ RETURNING *`,
+ [s.newId('bve'), value.exact_id, value.family, value.quant, value.revision, value.source_kind, value.source_id,
+ JSON.stringify(provenance), value.measured_at, value.sample_count, value.success_count, value.timeout_count,
+ value.ttft_p50_ms, value.ttft_p95_ms, value.prefill_tokens_per_s, value.decode_tokens_per_s,
+ value.cold_start_ms, String(createdBy || 'service:unknown').slice(0, 200), s.iso()],
+ );
+ return aggregateRows(value.exact_id, [row]);
+ }
+
+ async function rowsFor({ family = null, exactId = null } = {}) {
+ const clauses = [];
+ const params = [];
+ if (family) { params.push(String(family)); clauses.push(`family = $${params.length}`); }
+ if (exactId) { params.push(String(exactId)); clauses.push(`exact_id = $${params.length}`); }
+ const where = clauses.length ? `WHERE ${clauses.join(' AND ')}` : '';
+ return await s.db.many(
+ `SELECT * FROM bench_variant_evidence ${where}
+ ORDER BY measured_at DESC, created_at DESC, id DESC LIMIT ${MAX_OBSERVATIONS}`,
+ params,
+ );
+ }
+
+ async function families() {
+ return groupFamilies(await rowsFor());
+ }
+
+ async function family(familyValue) {
+ const parsed = parseFamily(familyValue);
+ if (!parsed) return null;
+ const rows = await rowsFor({ family: parsed.family });
+ if (!rows.length) return null;
+ return groupFamilies(rows)[0] || null;
+ }
+
+ async function list({ family: familyValue = null, exactId = null } = {}) {
+ if (exactId) {
+ const one = await get(exactId);
+ return one ? [one] : [];
+ }
+ const rows = await rowsFor({ family: familyValue ? text(familyValue) : null });
+ const byExact = new Map();
+ for (const row of rows) {
+ if (!byExact.has(row.exact_id)) byExact.set(row.exact_id, []);
+ byExact.get(row.exact_id).push(row);
+ }
+ return [...byExact.entries()].map(([exact, listOfRows]) => aggregateRows(exact, listOfRows)).filter(Boolean)
+ .sort((a, b) => a.family.localeCompare(b.family) || String(a.exact_id).localeCompare(String(b.exact_id)));
+ }
+
+ async function get(exactId) {
+ const parsed = parseExactId(exactId);
+ if (!parsed.ok) return null;
+ const rows = await rowsFor({ exactId: parsed.value.exact_id });
+ return aggregateRows(parsed.value.exact_id, rows);
+ }
+
+ return { record, families, family, list, get, sourceProvenance };
+}
+
+module.exports = {
+ parseExactId,
+ parseFamily,
+ confidenceFor,
+ validateObservation,
+ isSelfRating,
+ aggregateRows,
+ groupFamilies,
+ wireObservation,
+ createVariantEvidence,
+ VariantEvidenceError,
+ MIN_SAMPLES,
+ MAX_EXACT_ID,
+ SELF_RATING,
+};
@@ -79,6 +79,9 @@ function load(env = process.env) {
@@ -79,6 +79,9 @@ function load(env = process.env) {
// Bench budgets and jobs.
// Bench budgets and jobs.
bench: {
bench: {
+ // P6 model-variant evidence (exact ids, quants, revisions and their measured numbers). Off until the
+ // owner turns it on; off, none of its routes exist and nothing writes its table.
+ variantEvidence: bool(env.BENCH_VARIANT_EVIDENCE, false),
roundsPerDay: Math.max(1, int(env.BENCH_ROUNDS_PER_DAY, 3)),
roundsPerDay: Math.max(1, int(env.BENCH_ROUNDS_PER_DAY, 3)),
roundsDaily: Math.max(1, int(env.BENCH_ROUNDS_DAILY, 20)),
roundsDaily: Math.max(1, int(env.BENCH_ROUNDS_DAILY, 20)),
intervalMs: Math.max(250, int(env.BENCH_JOB_INTERVAL_MS, 1500)),
intervalMs: Math.max(250, int(env.BENCH_JOB_INTERVAL_MS, 1500)),
new file mode 100644
new file mode 100644
@@ -0,0 +1,101 @@
@@ -0,0 +1,101 @@
+'use strict';
+
+/**
+ * P6: the model-variant evidence API (plan 2026-11-04, G-P6). Behind BENCH_VARIANT_EVIDENCE, off by default.
+ *
+ * GET /api/v1/variant-families every family, grouped by org/model
+ * GET /api/v1/variant-families/:family one family (URL-encoded org/model)
+ * GET /api/v1/variant-evidence every exact id's evidence
+ * GET /api/v1/variant-evidence/:exactId one exact id (URL-encoded)
+ * POST /api/v1/variant-evidence record one observation
+ *
+ * The write is trusted provenance, not an open form. A round source needs a service token with bench.entry.create
+ * and a finished round with a receipt; a run source needs a service token with bench.run.publish. A model's own
+ * agent token is refused. The response carries only ids, counts and measurements: no prompt, output or payload.
+ */
+
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { asyncRouter } = require('./router');
+const { isSelfRating, validateObservation, VariantEvidenceError, SELF_RATING } = require('../bench/variant-evidence');
+
+function createVariantEvidenceApi(ctx) {
+ const { principal, variantEvidence, config, log } = ctx;
+ const r = asyncRouter();
+ // D11 owner settings: the switch is read live. Off, these paths are the API's ordinary 404 and nothing is read.
+ r.use((_req, _res, next) => (config.bench.variantEvidence ? next() : next('router')));
+
+ r.use(express.json({ limit: '64kb' }));
+ r.use((req, res, next) => (req.principal ? next() : principal.middleware(req, res, next)));
+ r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });
+
+ const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });
+ const invalid = (res, detail) => problem(res, 400, 'bench.invalid', detail);
+ const failed = (res, err) => {
+ if (err instanceof VariantEvidenceError) return problem(res, err.status, err.code, err.detail);
+ log.warn('[OpenVibe.Bench] variant evidence failed:', (err && err.message) || err);
+ return problem(res, 500, 'internal.error', 'Bench could not answer. Try again in a moment.');
+ };
+
+ const encoded = (id) => encodeURIComponent(String(id));
+
+ r.get('/variant-families', async (_req, res) => {
+ const families = await variantEvidence.families();
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ count: families.length, families });
+ });
+
+ r.get('/variant-families/:family', async (req, res) => {
+ const family = await variantEvidence.family(String(req.params.family || ''));
+ if (!family) return problem(res, 404, 'bench.variant_family_not_found', 'No such variant family.');
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ family });
+ });
+
+ r.get('/variant-evidence', async (req, res) => {
+ const exactId = typeof req.query.exact_id === 'string' && req.query.exact_id ? req.query.exact_id : null;
+ const family = typeof req.query.family === 'string' && req.query.family ? req.query.family : null;
+ const variants = await variantEvidence.list({ family, exactId });
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ count: variants.length, variants });
+ });
+
+ r.get('/variant-evidence/:exactId', async (req, res) => {
+ const evidence = await variantEvidence.get(String(req.params.exactId || ''));
+ if (!evidence) return problem(res, 404, 'bench.variant_not_found', 'No evidence for that exact id.');
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ evidence });
+ });
+
+ r.post('/variant-evidence', async (req, res) => {
+ const body = req.body && typeof req.body === 'object' ? req.body : {};
+ const checked = validateObservation(body, { now: ctx.s.iso() });
+ if (!checked.ok) return invalid(res, checked.errors.join('; '));
+ const value = checked.value;
+
+ const p = req.principal || { kind: 'anonymous' };
+ const cap = value.source_kind === 'run' ? 'bench.run.publish' : 'bench.entry.create';
+ if (p.kind === 'anonymous') {
+ return problem(res, 401, 'token.required', 'Send a service token with bench.entry.create (a pinned round) or bench.run.publish (a run).');
+ }
+ if (p.kind !== 'app') {
+ return problem(res, 403, 'bench.capability_required', `This route needs the ${cap} capability on a service token.`);
+ }
+ if (!p.hasCapability(cap)) {
+ return problem(res, 403, 'bench.capability_denied', `A service token needs ${cap} to write this evidence.`);
+ }
+ if (isSelfRating(p, value)) {
+ return problem(res, 403, SELF_RATING, 'A model may not write evidence about itself: use a service that ran it.');
+ }
+
+ try {
+ const evidence = await variantEvidence.record(value, { createdBy: p.requester });
+ res.set('Location', `/api/v1/variant-evidence/${encoded(value.exact_id)}`);
+ return res.status(201).json({ evidence });
+ } catch (err) { return failed(res, err); }
+ });
+
+ return r;
+}
+
+module.exports = { createVariantEvidenceApi };
new file mode 100644
new file mode 100644
@@ -0,0 +1,82 @@
@@ -0,0 +1,82 @@
+'use strict';
+/**
+ * The `bench.variant_evidence` namespace (D11 owner settings): the switch the owner changes from
+ * openvibe.network's AI tab. BENCH_VARIANT_EVIDENCE stays the pin: while it is set it wins, is read-only in the
+ * summary and a write answers 409 config.pinned_in_env.
+ *
+ * /internal/config is openvibe-shared/config's adminRoutes, guarded by bench.config.manage (Network's service
+ * token only). actor is the owner Network names in X-OV-Subject; X-OV-Fresh: 1 gates a secret or x-danger write
+ * (this namespace has neither). Every activation writes one audit line with the actor and the keys, never a value.
+ */
+const express = require('express');
+const shared = require('openvibe-shared/config');
+const { capabilities, http, ids } = require('openvibe-contracts');
+
+const SERVICE = 'bench';
+const NAMESPACE = 'bench.variant_evidence';
+const CAPABILITY = 'bench.config.manage';
+
+const SCHEMA = {
+ $schema: 'https://json-schema.org/draft/2020-12/schema',
+ title: 'Model-variant evidence',
+ description: 'The exact model ids, quants and revisions Bench has measured, and the evidence API a service publishes them through.',
+ type: 'object',
+ additionalProperties: false,
+ required: ['enabled'],
+ properties: {
+ enabled: {
+ type: 'boolean', title: 'Variant evidence', 'x-group': 'Bench', 'x-env': 'BENCH_VARIANT_EVIDENCE',
+ description: 'Serve /api/v1/variant-families and /api/v1/variant-evidence and accept a verified round or run as provenance.',
+ },
+ },
+};
+const DEFAULTS = Object.freeze({ enabled: false });
+
+function actor(req) {
+ const named = String((req.headers && req.headers['x-ov-subject']) || '').trim();
+ return ids.isSubjectId('user', named) ? { type: 'user', id: named } : null;
+}
+
+const requireFresh = (req) => String((req.headers && req.headers['x-ov-fresh']) || '') === '1';
+
+/** principal.middleware resolves the token; this finish refuses anything that is not a service token with the capability. */
+function capabilityGuard({ principal, config, log }) {
+ const resolve = principal.middleware;
+ return function guard(req, res, next) {
+ resolve(req, res, (err) => {
+ if (err) return next(err);
+ const p = req.principal || {};
+ if (p.kind !== 'app') return http.sendProblem(res, 403, 'capability.denied', { detail: `${CAPABILITY} is first-party; an app may not change Bench configuration`, ctx: req.ov });
+ const decision = capabilities.check(p.claims, CAPABILITY);
+ if (!decision.allowed) return http.sendProblem(res, 403, decision.code || 'capability.denied', { detail: decision.reason || `${CAPABILITY} not granted`, ctx: req.ov });
+ req.configClaims = p.claims;
+ return next();
+ });
+ };
+}
+
+async function create({ db, principal, config, env = process.env, log = console, onActivate = null }) {
+ // BENCH_VARIANT_EVIDENCE has always accepted 1/true/yes/on; the shared store's boolean pin reads true/1.
+ const storeEnv = { ...env };
+ if (['1', 'true', 'yes', 'on'].includes(String(storeEnv.BENCH_VARIANT_EVIDENCE || '').trim().toLowerCase())) storeEnv.BENCH_VARIANT_EVIDENCE = '1';
+ const store = await shared.createConfigStore({
+ db, service: SERVICE, namespace: NAMESPACE, schema: SCHEMA, defaults: DEFAULTS, env: storeEnv, log,
+ onActivate: async (values) => { if (onActivate) await onActivate(values); },
+ onChange: ({ namespace, keys, actor: who, revision }) => {
+ log.info(`[config] ${SERVICE} ${namespace} revision ${revision} by ${who && who.id ? `${who.type}:${who.id}` : 'unknown'}: ${keys.join(', ')}`);
+ },
+ });
+ if (env.OV_CONFIG_KEKS) {
+ try { await store.reseal(); } catch (err) { log.warn(`[config] ${NAMESPACE}: reseal failed: ${(err && err.message) || err}`); }
+ }
+ const router = shared.adminRoutes([store], {
+ basePath: '/internal/config', requireAdmin: capabilityGuard({ principal, config, log }), actor, requireFresh, log,
+ });
+ return {
+ store, router,
+ get: (key) => store.get(key),
+ mount(app) { app.use('/internal/config', express.json({ limit: '128kb' })); router.mount(app); },
+ };
+}
+
+module.exports = { create, SCHEMA, NAMESPACE, CAPABILITY };
@@ -42,6 +42,7 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
@@ -42,6 +42,7 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
exempt: {
exempt: {
account_data_events: 'the ledger of the requests themselves: which export and deletion Bench applied',
account_data_events: 'the ledger of the requests themselves: which export and deletion Bench applied',
bench_ratings: 'the ratings are an aggregate over the models: the subject column names a model, not a person',
bench_ratings: 'the ratings are an aggregate over the models: the subject column names a model, not a person',
+ config_snapshots: 'D11 configuration revisions: created_by names the owner who changed a setting, not a person\'s account data',
},
},
});
});
assert.deepStrictEqual(missing, [], `a table with a person column is neither covered nor exempted: ${missing.join(', ')}`);
assert.deepStrictEqual(missing, [], `a table with a person column is neither covered nor exempted: ${missing.join(', ')}`);
@@ -40,7 +40,7 @@ async function boot(opts = {}) {
@@ -40,7 +40,7 @@ async function boot(opts = {}) {
const config = configLib.load(env);
const config = configLib.load(env);
const testdb = opts.store ? null : await require('./db').testDb({ store: opts.testStore });
const testdb = opts.store ? null : await require('./db').testDb({ store: opts.testStore });
const store = opts.store || createStore(testdb.db, { now: opts.now });
const store = opts.store || createStore(testdb.db, { now: opts.now });
- const built = await createApp({ config, store, benchAi: opts.benchAi, log, now: opts.now, accountSend: opts.accountSend, random: opts.random, telemetrySink: opts.telemetrySink });
+ const built = await createApp({ config, env, store, benchAi: opts.benchAi, log, now: opts.now, accountSend: opts.accountSend, random: opts.random, telemetrySink: opts.telemetrySink });
await built.ctx.keys.ensure();
await built.ctx.keys.ensure();
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
@@ -323,6 +323,7 @@ function multipart(filename, bytes) {
@@ -323,6 +323,7 @@ function multipart(filename, bytes) {
exempt: {
exempt: {
account_data_events: 'the ledger of the requests themselves',
account_data_events: 'the ledger of the requests themselves',
bench_ratings: 'the ratings are an aggregate over the models',
bench_ratings: 'the ratings are an aggregate over the models',
+ config_snapshots: 'D11 configuration revisions: created_by names the owner who changed a setting, not a person\'s account data',
},
},
});
});
assert.deepStrictEqual(missing, [], 'a table with a person column is neither covered nor exempted: ' + missing.join(', '));
assert.deepStrictEqual(missing, [], 'a table with a person column is neither covered nor exempted: ' + missing.join(', '));
new file mode 100644
new file mode 100644
@@ -0,0 +1,90 @@
@@ -0,0 +1,90 @@
+'use strict';
+/**
+ * D11 owner settings: the bench.variant_evidence namespace and /internal/config.
+ * Env pins win; an owner change applies live; bench.config.manage is required; no KEK still works;
+ * a secret-class value never leaves a summary, snapshot or history.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+
+const CAP = ['bench.config.manage'];
+const OWNER = 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ';
+const SECRET = 'fixture-bench-d11-not-a-credential';
+
+(async () => {
+ const t = await boot({ env: { BENCH_VARIANT_EVIDENCE: '1' } });
+ const admin = () => t.network.serviceToken({ sub: 'svc:network', audience: 'openvibe.bench', cap: CAP });
+ try {
+ await check('the summary: schema, the env pin, sealed:false without a KEK', async () => {
+ const r = await t.get('/internal/config/bench.variant_evidence', { bearer: admin() });
+ assert.strictEqual(r.status, 200, r.text.slice(0, 300));
+ assert.strictEqual(r.json().namespace, 'bench.variant_evidence');
+ assert.strictEqual(r.json().sealed, false);
+ assert.strictEqual(r.json().values.enabled, true);
+ assert.deepStrictEqual(r.json().pins.enabled, { env: 'BENCH_VARIANT_EVIDENCE', pinned: true });
+ assert.ok(!r.text.includes(SECRET));
+ });
+
+ await check('a service token without bench.config.manage is refused', async () => {
+ const token = t.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.round.read'] });
+ const r = await t.get('/internal/config/bench.variant_evidence', { bearer: token });
+ assert.strictEqual(r.status, 403);
+ assert.strictEqual(r.json().code, 'capability.denied');
+ });
+
+ await check('an env-pinned key is read-only', async () => {
+ const r = await t.get('/internal/config/bench.variant_evidence', {
+ method: 'POST', bearer: admin(), headers: { 'x-ov-subject': OWNER }, json: { values: { enabled: false }, reason: 'off' },
+ });
+ assert.strictEqual(r.status, 409, r.text.slice(0, 300));
+ assert.strictEqual(r.json().code, 'config.pinned_in_env');
+ });
+ } finally { await t.close(); }
+
+ const live = await boot();
+ const liveAdmin = () => live.network.serviceToken({ sub: 'svc:network', audience: 'openvibe.bench', cap: CAP });
+ try {
+ await check('an owner change applies live: the evidence API appears', async () => {
+ const reader = live.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.round.read'] });
+ assert.strictEqual((await live.get('/api/v1/variant-families', { bearer: reader })).status, 404);
+ const r = await live.get('/internal/config/bench.variant_evidence', {
+ method: 'POST', bearer: liveAdmin(), headers: { 'x-ov-subject': OWNER }, json: { values: { enabled: true }, reason: 'owner turned it on' },
+ });
+ assert.strictEqual(r.status, 200, r.text.slice(0, 300));
+ assert.strictEqual(r.json().revision, 2);
+ assert.strictEqual(live.config.bench.variantEvidence, true);
+ assert.strictEqual((await live.get('/api/v1/variant-families', { bearer: reader })).status, 200);
+ assert.ok(live.logs().includes('by user:' + OWNER));
+ assert.ok(!live.logs().includes(SECRET));
+ });
+
+ await check('with a KEK the store is sealed and boot reseals', async () => {
+ const sealed = await boot({ env: { OV_CONFIG_KEKS: 'k1:MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=', OV_CONFIG_KEK_CURRENT: 'k1' } });
+ try {
+ const token = sealed.network.serviceToken({ sub: 'svc:network', audience: 'openvibe.bench', cap: CAP });
+ const r = await sealed.get('/internal/config/bench.variant_evidence', { bearer: token });
+ assert.strictEqual(r.status, 200, r.text.slice(0, 300));
+ assert.strictEqual(r.json().sealed, true);
+ assert.strictEqual(r.json().kek, 'k1');
+ } finally { await sealed.close(); }
+ });
+
+ await check('a secret is write-only and never appears in a summary, snapshot or history', async () => {
+ const shared = require('openvibe-shared/config');
+ const extra = await shared.createConfigStore({
+ db: live.store.db, service: 'bench', namespace: 'bench.test_secrets',
+ schema: { type: 'object', additionalProperties: false, properties: { api_key: { type: 'string' } } },
+ classify: () => 'secret',
+ });
+ const marker = await extra.setSecret('api_key', SECRET, { actor: { type: 'user', id: OWNER }, reason: 'test' });
+ assert.strictEqual(marker.redacted, true);
+ assert.ok(!JSON.stringify(marker).includes(SECRET));
+ const summary = await extra.summary();
+ assert.strictEqual(summary.values.api_key.redacted, true);
+ assert.ok(!JSON.stringify(summary).includes(SECRET));
+ assert.ok(!JSON.stringify(await extra.history()).includes(SECRET));
+ assert.ok(!JSON.stringify(await extra.snapshot(1)).includes(SECRET));
+ });
+ } finally { await live.close(); }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,351 @@
@@ -0,0 +1,351 @@
+'use strict';
+
+/**
+ * P6 model-variant evidence (plan 2026-11-04, G-P6): exact ids group into a family across quants and revisions;
+ * the quant and revision are optional; sparse evidence stays sparse; a write needs a pinned round or an
+ * authenticated run; a model never writes its own evidence; the flag off leaves no route and no write.
+ */
+const assert = require('assert');
+const crypto = require('crypto');
+const { boot, check, done } = require('./helpers/boot');
+const configLib = require('../server/config');
+const { parseExactId, groupFamilies } = require('../server/bench/variant-evidence');
+const { createVariantEvidenceApi } = require('../server/http/variant-evidence');
+
+const EXACT = 'qwen/qwen3-9b-abliterated:q4_k_m@3f2a1c9';
+const FAMILY = 'qwen/qwen3-9b-abliterated';
+
+function row(over = {}) {
+ return {
+ id: 'bve_00000000000000000000000000',
+ exact_id: EXACT,
+ family: FAMILY,
+ quant: 'q4_k_m',
+ revision: '3f2a1c9',
+ source_kind: 'round',
+ source_id: 'brd_00000000000000000000000000',
+ provenance: { kind: 'round', id: 'brd_00000000000000000000000000', receipt: 'a'.repeat(64) },
+ measured_at: '2026-11-04T10:00:00.000Z',
+ sample_count: 4,
+ success_count: 3,
+ timeout_count: 1,
+ ttft_p50_ms: 210,
+ ttft_p95_ms: 480,
+ prefill_tokens_per_s: 1420,
+ decode_tokens_per_s: 86.5,
+ cold_start_ms: 5300,
+ created_by: 'service:codes',
+ created_at: '2026-11-04T10:00:01.000Z',
+ ...over,
+ };
+}
+
+(async () => {
+ await check('the flag is off by default and the route factory exists only for the on path', async () => {
+ assert.strictEqual(configLib.load({ NODE_ENV: 'test' }).bench.variantEvidence, false);
+ assert.strictEqual(configLib.load({ NODE_ENV: 'test', BENCH_VARIANT_EVIDENCE: '1' }).bench.variantEvidence, true);
+ assert.strictEqual(typeof createVariantEvidenceApi, 'function');
+ });
+
+ await check('the exact id parses the family, quant and revision, each optional', async () => {
+ const full = parseExactId(EXACT);
+ assert.strictEqual(full.ok, true);
+ assert.strictEqual(full.value.family, FAMILY);
+ assert.strictEqual(full.value.quant, 'q4_k_m');
+ assert.strictEqual(full.value.revision, '3f2a1c9');
+ const bare = parseExactId('qwen/qwen3-9b');
+ assert.strictEqual(bare.ok, true);
+ assert.strictEqual(bare.value.family, 'qwen/qwen3-9b');
+ assert.strictEqual(bare.value.quant, null);
+ assert.strictEqual(bare.value.revision, null);
+ const quantOnly = parseExactId('qwen/qwen3-9b:q8_0');
+ assert.strictEqual(quantOnly.value.quant, 'q8_0');
+ assert.strictEqual(quantOnly.value.revision, null);
+ const revisionOnly = parseExactId('qwen/qwen3-9b@main');
+ assert.strictEqual(revisionOnly.value.quant, null);
+ assert.strictEqual(revisionOnly.value.revision, 'main');
+ for (const bad of ['qwen', 'qwen/', '/qwen3', 'qwen/qwen3@', 'qwen/qwen3:q4:extra', 'qwen/qwen3@a@b', 'qwen/qwen 3']) {
+ assert.strictEqual(parseExactId(bad).ok, false, bad);
+ }
+ });
+
+ await check('the family groups every quant and revision of one org/model, with its own counts', async () => {
+ const families = groupFamilies([
+ row(),
+ row({ id: 'bve_11111111111111111111111111', exact_id: 'qwen/qwen3-9b-abliterated:q8_0@rev2', quant: 'q8_0', revision: 'rev2', sample_count: 20, success_count: 20, timeout_count: 0 }),
+ row({ id: 'bve_22222222222222222222222222', exact_id: 'other/model:q4@main', family: 'other/model', quant: 'q4', revision: 'main', sample_count: 1, success_count: null }),
+ ]);
+ assert.strictEqual(families.length, 2);
+ const qwen = families.find((f) => f.family === FAMILY);
+ assert.ok(qwen);
+ assert.strictEqual(qwen.variants.length, 2);
+ assert.deepStrictEqual(qwen.quants, ['q4_k_m', 'q8_0']);
+ assert.deepStrictEqual(qwen.revisions, ['3f2a1c9', 'rev2']);
+ assert.strictEqual(qwen.sample_count, 24);
+ const q4 = qwen.variants.find((v) => v.quant === 'q4_k_m');
+ assert.strictEqual(q4.sample_count, 4);
+ assert.strictEqual(q4.sparse, true);
+ assert.strictEqual(q4.success_rate, 0.75);
+ assert.strictEqual(q4.metrics.ttft_p95_ms, 480);
+ });
+
+ const t = await boot({ env: { BENCH_VARIANT_EVIDENCE: '1' } });
+ const alice = t.network.addUser('alice');
+ const roundWriter = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.entry.create'] });
+ const publishers = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.entry.create', 'bench.run.publish', 'bench.benchmark.write'] });
+ const wrongCap = t.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.round.read'] });
+ const secret = 'ZZ-PRIVATE-PROMPT-DO-NOT-STORE';
+ const secretOutput = 'ZZ-PRIVATE-OUTPUT-DO-NOT-STORE';
+
+ let finishedRound;
+ let benchmarkId;
+ let verifiedRunId;
+
+ await check('a person starts a round and the job pins it with a receipt', async () => {
+ const task = await t.get('/api/v1/kinds/design.page/tasks', { as: alice, json: { prompt: `A landing page for a tiny robot. ${secret}`, public: true } });
+ assert.strictEqual(task.status, 201, task.text);
+ const queued = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });
+ assert.strictEqual(queued.status, 201, queued.text);
+ await t.tick();
+ const round = await t.get(`/api/v1/rounds/${queued.json().round.id}`, { as: alice });
+ assert.strictEqual(round.json().round.status, 'finished');
+ assert.ok(round.json().round.receipt);
+ finishedRound = round.json().round;
+ });
+
+ await check('a pinned-round write records the typed numbers', async () => {
+ const written = await t.get('/api/v1/variant-evidence', {
+ bearer: roundWriter,
+ json: {
+ exact_id: EXACT,
+ source: { kind: 'round', id: finishedRound.id },
+ sample_count: 4,
+ ttft_p50_ms: 210,
+ ttft_p95_ms: 480,
+ prefill_tokens_per_s: 1420,
+ decode_tokens_per_s: 86.5,
+ cold_start_ms: 5300,
+ success_count: 3,
+ timeout_count: 1,
+ measured_at: '2026-11-04T10:00:00.000Z',
+ // A caller may send these; they are ignored and never stored.
+ prompt: secret,
+ output: secretOutput,
+ },
+ });
+ assert.strictEqual(written.status, 201, written.text);
+ const ev = written.json().evidence;
+ assert.strictEqual(ev.exact_id, EXACT);
+ assert.strictEqual(ev.family, FAMILY);
+ assert.strictEqual(ev.sample_count, 4);
+ assert.strictEqual(ev.success_rate, 0.75);
+ assert.strictEqual(ev.timeout_rate, 0.25);
+ assert.strictEqual(ev.metrics.ttft_p50_ms, 210);
+ assert.strictEqual(ev.provenance.kind, 'round');
+ assert.strictEqual(ev.provenance.receipt, finishedRound.receipt);
+ assert.strictEqual(ev.sparse, true);
+ assert.strictEqual(ev.confidence.level, 'sparse');
+ assert.strictEqual(ev.confidence.success_rate.estimate, 0.75);
+ });
+
+ await check('a sparse exact id stays sparse and is never filled from its family', async () => {
+ const dense = await t.get('/api/v1/variant-evidence', {
+ bearer: roundWriter,
+ json: {
+ exact_id: 'qwen/qwen3-9b-abliterated:q8_0@rev2',
+ source: { kind: 'round', id: finishedRound.id },
+ sample_count: 50,
+ success_count: 49,
+ timeout_count: 1,
+ ttft_p50_ms: 150,
+ ttft_p95_ms: 300,
+ prefill_tokens_per_s: 2000,
+ decode_tokens_per_s: 120,
+ },
+ });
+ assert.strictEqual(dense.status, 201, dense.text);
+ const family = await t.get(`/api/v1/variant-families/${encodeURIComponent(FAMILY)}`);
+ assert.strictEqual(family.status, 200, family.text);
+ const variants = family.json().family.variants;
+ assert.strictEqual(variants.length, 2);
+ const q4 = variants.find((v) => v.exact_id === EXACT);
+ assert.strictEqual(q4.sample_count, 4);
+ assert.strictEqual(q4.sparse, true);
+ assert.strictEqual(q4.metrics.decode_tokens_per_s, 86.5);
+ const q8 = variants.find((v) => v.exact_id.includes('q8_0'));
+ assert.strictEqual(q8.sample_count, 50);
+ assert.strictEqual(q8.sparse, false);
+ assert.strictEqual(q8.confidence.level, 'moderate');
+
+ const one = await t.get(`/api/v1/variant-evidence/${encodeURIComponent(EXACT)}`);
+ assert.strictEqual(one.status, 200, one.text);
+ assert.strictEqual(one.json().evidence.sample_count, 4);
+ assert.strictEqual(one.json().evidence.metrics.decode_tokens_per_s, 86.5);
+ });
+
+ await check('the family index groups one org/model and hides no sparse variant', async () => {
+ const list = await t.get('/api/v1/variant-families');
+ assert.strictEqual(list.status, 200, list.text);
+ const qwen = list.json().families.find((f) => f.family === FAMILY);
+ assert.ok(qwen);
+ assert.strictEqual(qwen.variants.length, 2);
+ assert.strictEqual(qwen.sample_count, 54);
+ const all = await t.get(`/api/v1/variant-evidence?family=${encodeURIComponent(FAMILY)}`);
+ assert.strictEqual(all.status, 200, all.text);
+ assert.strictEqual(all.json().count, 2);
+ });
+
+ await check('nothing in the evidence rows or the read API carries the private prompt or output', async () => {
+ const rows = await t.ctx.s.db.many('SELECT * FROM bench_variant_evidence');
+ const stored = JSON.stringify(rows);
+ assert.ok(!stored.includes(secret), 'the prompt was stored in the evidence rows');
+ assert.ok(!stored.includes(secretOutput), 'the output was stored in the evidence rows');
+ const read = await t.get(`/api/v1/variant-evidence/${encodeURIComponent(EXACT)}`);
+ assert.ok(!read.text.includes(secret));
+ assert.ok(!read.text.includes(secretOutput));
+ for (const key of ['prompt', 'output', 'payload', 'transcript']) assert.ok(!(key in read.json().evidence), key);
+ });
+
+ await check('an unauthenticated write is refused and a person is not a provenance writer', async () => {
+ const anon = await t.get('/api/v1/variant-evidence', { json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 1 } });
+ assert.strictEqual(anon.status, 401, anon.text);
+ const person = await t.get('/api/v1/variant-evidence', { as: alice, json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 1 } });
+ assert.strictEqual(person.status, 403, person.text);
+ const noCap = await t.get('/api/v1/variant-evidence', { bearer: wrongCap, json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 1 } });
+ assert.strictEqual(noCap.status, 403, noCap.text);
+ });
+
+ await check('a queued round is not pinned provenance', async () => {
+ const task = await t.get('/api/v1/kinds/design.page/tasks', { as: alice, json: { prompt: 'A second page.', public: true } });
+ const queued = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });
+ const written = await t.get('/api/v1/variant-evidence', {
+ bearer: roundWriter,
+ json: { exact_id: 'qwen/qwen3-9b@queued', source: { kind: 'round', id: queued.json().round.id }, sample_count: 1 },
+ });
+ assert.strictEqual(written.status, 409, written.text);
+ assert.strictEqual(written.json().code, 'bench.round_not_pinned');
+ });
+
+ await check('a model agent may not write evidence about itself', async () => {
+ // A service token whose identity claims the model it would measure: the sub is a valid service sub, and the
+ // model claim is what names the model. Bench refuses it even though the capability is present.
+ const now = Math.floor(Date.now() / 1000);
+ const self = t.network.signJwt({
+ iss: t.network.issuer(), sub: 'svc:model-runner', actor_type: 'service', aud: ['openvibe.bench'],
+ cap: ['bench.entry.create'], ns: [], model: FAMILY, iat: now, exp: now + 300, jti: crypto.randomUUID(),
+ });
+ const written = await t.get('/api/v1/variant-evidence', {
+ bearer: self,
+ json: { exact_id: EXACT, source: { kind: 'round', id: finishedRound.id }, sample_count: 2, success_count: 2 },
+ });
+ assert.strictEqual(written.status, 403, written.text);
+ assert.strictEqual(written.json().code, 'bench.self_rating');
+ });
+
+ await check('an authenticated verified run is provenance too', async () => {
+ const definition = {
+ title: 'Variant evidence run', description: 'A run for the P6 evidence path.', kind: 'games.play',
+ tags: ['evidence'], visibility: 'public',
+ tasks: [{ id: 't1', prompt: 'Play the game.', files: [], expected: null, holdout: false }],
+ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],
+ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
+ limits: { ms: 120000, tokens: 200000, usd: 0.5 },
+ matrix: { models: ['acme/model'], harnesses: ['run'] },
+ };
+ const created = await t.get('/api/v1/benchmarks', { bearer: publishers, json: definition });
+ assert.strictEqual(created.status, 201, created.text);
+ benchmarkId = created.json().benchmark.id;
+ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: publishers, json: {} });
+ assert.strictEqual(frozen.status, 201, frozen.text);
+ const run = await t.get('/api/v1/runs', {
+ bearer: publishers,
+ json: {
+ benchmark: `${benchmarkId}@1`,
+ model: 'acme/model:q4@rev9',
+ route: { provider: 'acme' }, provider: 'acme',
+ harness: { name: 'run', version: '1.0' },
+ settings: {}, executor: { kind: 'run', id: 'job_00000000000000000000000000' },
+ tasks: [{ task_id: 't1', score: 90, cost_usd: 0.01, ms: 1000, tokens: 100, artifacts: [] }],
+ score: 90, cost_usd: 0.01, ms: 1000, tokens: 100,
+ },
+ });
+ assert.strictEqual(run.status, 201, run.text);
+ verifiedRunId = run.json().run.id;
+ const written = await t.get('/api/v1/variant-evidence', {
+ bearer: publishers,
+ json: {
+ exact_id: 'acme/model:q4@rev9',
+ source: { kind: 'run', id: verifiedRunId },
+ sample_count: 12, success_count: 12, timeout_count: 0,
+ ttft_p50_ms: 90, ttft_p95_ms: 140, prefill_tokens_per_s: 900, decode_tokens_per_s: 70,
+ measured_at: '2026-11-04T11:00:00.000Z',
+ },
+ });
+ assert.strictEqual(written.status, 201, written.text);
+ assert.strictEqual(written.json().evidence.provenance.kind, 'run');
+ assert.strictEqual(written.json().evidence.provenance.verified, true);
+ assert.strictEqual(written.json().evidence.provenance.bench_run_id, verifiedRunId);
+ assert.strictEqual(written.json().evidence.confidence.level, 'low');
+ });
+
+ await check('an authenticated Run job id is recorded as run provenance', async () => {
+ const jobId = 'job_01JZ0000000000000000000000';
+ const written = await t.get('/api/v1/variant-evidence', {
+ bearer: publishers,
+ json: {
+ exact_id: 'acme/model:q8_0@rev10',
+ source: { kind: 'run', id: jobId },
+ sample_count: 30, success_count: 30, timeout_count: 0,
+ ttft_p50_ms: 80, ttft_p95_ms: 120,
+ },
+ });
+ assert.strictEqual(written.status, 201, written.text);
+ assert.strictEqual(written.json().evidence.provenance.kind, 'run');
+ assert.strictEqual(written.json().evidence.provenance.id, jobId);
+ assert.strictEqual(written.json().evidence.provenance.executor, 'run');
+ assert.strictEqual(written.json().evidence.provenance.verified, false);
+ });
+
+ await check('a submitted run is not evidence, and a missing source is a 404', async () => {
+ const submitted = await t.get('/api/v1/runs', {
+ as: alice,
+ json: {
+ benchmark: `${benchmarkId}@1`, model: 'acme/model:q4@rev9',
+ route: { provider: 'acme' }, provider: 'acme', harness: { name: 'run', version: '1.0' },
+ settings: {}, tasks: [{ task_id: 't1', score: 80, cost_usd: 0.01, ms: 1000, tokens: 100, artifacts: [] }],
+ score: 80, cost_usd: 0.01, ms: 1000, tokens: 100,
+ },
+ });
+ assert.strictEqual(submitted.status, 201, submitted.text);
+ const refused = await t.get('/api/v1/variant-evidence', {
+ bearer: publishers,
+ json: { exact_id: 'acme/model:q4@rev9', source: { kind: 'run', id: submitted.json().run.id }, sample_count: 5 },
+ });
+ assert.strictEqual(refused.status, 409, refused.text);
+ assert.strictEqual(refused.json().code, 'bench.run_not_verified');
+ const missing = await t.get('/api/v1/variant-evidence', {
+ bearer: roundWriter,
+ json: { exact_id: 'acme/model:q4@rev9', source: { kind: 'round', id: 'brd_00000000000000000000000000' }, sample_count: 5 },
+ });
+ assert.strictEqual(missing.status, 404, missing.text);
+ assert.strictEqual(missing.json().code, 'bench.round_not_found');
+ });
+
+ await t.close();
+
+ const off = await boot();
+ await check('the flag off leaves the routes absent and the rest of Bench unchanged', async () => {
+ assert.strictEqual(off.config.bench.variantEvidence, false);
+ const families = await off.get('/api/v1/variant-families');
+ assert.strictEqual(families.status, 404, families.text);
+ assert.strictEqual(families.json().code, 'route.not_found');
+ const writer = off.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.entry.create'] });
+ const write = await off.get('/api/v1/variant-evidence', { bearer: writer, json: { exact_id: EXACT, source: { kind: 'round', id: 'brd_00000000000000000000000000' }, sample_count: 1 } });
+ assert.strictEqual(write.status, 404, write.text);
+ const kinds = await off.get('/api/v1/kinds');
+ assert.strictEqual(kinds.status, 200, kinds.text);
+ });
+ await off.close();
+
+ done();
+})();