openvibers/openvibe.bench Commits
Bench B1: the benchmark registry and the results API (#24)
Changes
nnew file mode 100644@@@ -0,0 +1,32 @@++# The Bench API++++JSON under `/api/v1`. Errors are `application/problem+json` with a stable `code`. A person is the `bench_at`++session cookie or a Network user token as a Bearer; a service sends a client-credentials token with `bench.*`++capabilities. A session write must come from bench.openvibe.services itself.++++## The registry (stage B1)++++| Route | Who | What |++|---|---|---|++| `POST /api/v1/benchmarks` | a person, or `bench.benchmark.write` | Create a benchmark (a draft). |++| `PUT /api/v1/benchmarks/:id` | the same | Save a new draft. A body with `version` edits that version, and is refused once a run names it. |++| `POST /api/v1/benchmarks/:id/versions` | the same | Freeze the working copy into the next immutable version. |++| `POST /api/v1/benchmarks/:id/fork` | the same | Fork a benchmark; the fork records its parent. |++| `GET /api/v1/benchmarks?kind=&tag=&q=` | anyone | The registry. |++| `GET /api/v1/benchmarks/:id[@version]` | anyone | One benchmark and its definition. Holdout tasks are withheld unless the caller holds `bench.run.publish`. |++++## The results++++| Route | Who | What |++|---|---|---|++| `POST /api/v1/runs` | a person, or `bench.run.publish` | Publish a run. A service token with `bench.run.publish` makes a **verified** run; a person's token makes a **submitted** one, badged and out of the default rankings. |++| `GET /api/v1/runs/:id` | anyone | One run with its per-task results. |++| `GET /api/v1/compare?runs=a,b,c` | anyone | Up to four runs side by side, per task. |++| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. |++| `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |++++## The older API++++Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private++evals. Those are the round-based engine; the registry above is the versioned, forkable set the platform grows on.++See `/docs` on the site for the live list.nnew file mode 100644@@@ -0,0 +1,79 @@++# Writing a benchmark++++A benchmark is a set you own, version and fork. This is how to write one.++++## The shape++++A benchmark (the contract is `bench.benchmark@1`) holds:++++| Field | What it is |++|---|---|++| `id` | `bmk_…`, given by Bench when you create it. The same id carries every version. |++| `owner` | You. A person (`usr_…`), a project (`prj_…`) or a service. |++| `title`, `description` | What the set is and who it is for. |++| `kind` | One of Bench's kinds (`games.play`, `design.ui`, `codes.patch`, …). The kind sets the renderer. |++| `tags` | Up to 32 words people search by. |++| `visibility` | `public` (in the registry) or `unlisted` (by id only). |++| `version` | The version this document is. Bench numbers them. |++| `parent` | The `id@version` this one was forked from, or `null`. |++| `tasks[]` | `{ id, prompt, files?, expected?, holdout }`. The prompt is the task; `holdout: true` hides it from everyone but the executor. |++| `judges[]` | `{ type, version, config }`. The judge is pinned by version, so a run names the judge it used. |++| `scoring` | `{ method, scale, weights? }`. `method` is `mean`, `weighted`, `pass_rate` or `sum`; the scale is 0-100. |++| `limits` | `{ ms?, tokens?, usd }`. A run may not pass them. |++| `matrix` | `{ models[], harnesses[] }`. The defaults a runner starts from. |++++## The three calls++++```++POST /api/v1/benchmarks create it (a draft)++PUT /api/v1/benchmarks/:id save a new draft++POST /api/v1/benchmarks/:id/versions freeze a version++POST /api/v1/benchmarks/:id/fork fork it++```++++Authoring needs the `bench.benchmark.write` capability on a service token, or a signed-in person.++++A version is **immutable once a run names it**. Write a new draft and freeze it again instead; if you want to change a++published set, fork it — the fork records the parent, and the old version stays where the old runs point.++++## An example++++```json++{++ "title": "Playable games arena",++ "description": "One-screen arcade games, judged by play.",++ "kind": "games.play",++ "tags": ["games", "arcade"],++ "visibility": "public",++ "tasks": [++ { "id": "t1", "prompt": "Build a one-screen arcade game with a paddle, bricks, a ball and a score.", "files": [], "expected": null, "holdout": false },++ { "id": "t2", "prompt": "Build a snake game on a 20x20 grid.", "files": [], "expected": null, "holdout": true }++ ],++ "judges": [{ "type": "human_pairwise", "version": "1.0", "config": {} }],++ "scoring": { "method": "mean", "scale": { "min": 0, "max": 100 } },++ "limits": { "ms": 120000, "tokens": 200000, "usd": 0.5 },++ "matrix": { "models": ["claude-opus-4"], "harnesses": ["claude-code"] }++}++```++++## Running it++++A run (the contract is `bench.run@1`) names the benchmark `id@version`, the model, the route, the provider, the++harness and its version, the settings, the executor, and the per-task score, cost, time, tokens and artifacts. It++ends with the run's own score, its cost, time and tokens, and its provenance.++++Publish it with `POST /api/v1/runs`. A service token with `bench.run.publish` publishes a **verified** run (an++executor ran it, so it can be re-run and it counts in the rankings). A person's token publishes a **submitted** run:++it is badged and kept out of the default rankings.++++## Reading the results++++```++GET /api/v1/runs/:id++GET /api/v1/compare?runs=a,b,c++GET /api/v1/rankings?benchmark=&metric=score|value|speed|consistency&harness=&since=++GET /api/v1/percentile?benchmark=&model=&score=++```++++`value` is the score per dollar, `speed` the score per minute, and `consistency` the spread of a model's scores++inside one session (the lower, the steadier). The percentile says where a score sits among that model's runs.nnew file mode 100644@@@ -0,0 +1,93 @@++-- phase: expand++-- Stage B1 (spec 2026-10-30): the benchmark registry and the results API.++--++-- A benchmark is an owned, versioned and forkable set: its tasks, its judges, how they combine into one score, the++-- limits a run may not pass and the default model and harness matrix. A version is immutable once a run names it, so++-- bench_benchmark_versions keeps the frozen JSON definition and its hash; a fork records its parent.++--++-- A run is one result: a benchmark version × a model × a harness × settings × who ran it. Provenance says who ran it:++-- verified (an executor did) or submitted (posted through the API with no executor). Submitted runs are badged and++-- kept out of the default rankings. Nothing written here is a credential: a run never carries a key.++++CREATE TABLE bench_benchmarks (++ id text COLLATE "C" PRIMARY KEY, -- bmk_ + ULID++ owner text COLLATE "C" NOT NULL, -- usr_…, prj_… or a service++ title text NOT NULL,++ description text NOT NULL DEFAULT '',++ kind text COLLATE "C" NOT NULL, -- one of Bench's kinds; the kind sets the renderer++ tags jsonb NOT NULL DEFAULT '[]'::jsonb,++ visibility text COLLATE "C" NOT NULL DEFAULT 'unlisted', -- public | unlisted++ parent text COLLATE "C", -- the benchmark id@version this one was forked from++ latest_version integer NOT NULL DEFAULT 0, -- the newest frozen version (0: a draft)++ draft jsonb, -- the unpublished working copy (PUT saves it)++ created_by text COLLATE "C" NOT NULL,++ created_at timestamptz NOT NULL DEFAULT now(),++ updated_at timestamptz NOT NULL DEFAULT now(),++ CONSTRAINT bench_benchmarks_id CHECK (id ~ '^bmk_[0-9A-HJKMNP-TV-Z]{26}$'),++ CONSTRAINT bench_benchmarks_visibility CHECK (visibility IN ('public', 'unlisted')),++ CONSTRAINT bench_benchmarks_latest CHECK (latest_version >= 0),++ CONSTRAINT bench_benchmarks_title CHECK (length(title) BETWEEN 1 AND 160),++ CONSTRAINT bench_benchmarks_kind CHECK (length(kind) BETWEEN 1 AND 80)++);++CREATE INDEX bench_benchmarks_by_kind ON bench_benchmarks (kind, created_at DESC);++CREATE INDEX bench_benchmarks_by_owner ON bench_benchmarks (owner, created_at DESC);++CREATE INDEX bench_benchmarks_by_visibility ON bench_benchmarks (visibility, created_at DESC);++++CREATE TABLE bench_benchmark_versions (++ benchmark_id text COLLATE "C" NOT NULL REFERENCES bench_benchmarks (id) ON DELETE CASCADE,++ version integer NOT NULL,++ definition jsonb NOT NULL, -- the whole bench.benchmark@1 document++ hash text COLLATE "C" NOT NULL, -- sha256 of the canonical definition++ created_by text COLLATE "C" NOT NULL,++ created_at timestamptz NOT NULL DEFAULT now(),++ PRIMARY KEY (benchmark_id, version),++ CONSTRAINT bench_benchmark_versions_version CHECK (version >= 1),++ CONSTRAINT bench_benchmark_versions_hash CHECK (hash ~ '^[0-9a-f]{64}$')++);++CREATE INDEX bench_benchmark_versions_by_hash ON bench_benchmark_versions (hash);++++CREATE TABLE bench_runs (++ id text COLLATE "C" PRIMARY KEY, -- brn_ + ULID++ benchmark_id text COLLATE "C" NOT NULL REFERENCES bench_benchmarks (id),++ benchmark_version integer NOT NULL,++ model text COLLATE "C" NOT NULL,++ route jsonb NOT NULL DEFAULT '{}'::jsonb, -- the route that produced the run++ provider text COLLATE "C" NOT NULL,++ harness jsonb NOT NULL DEFAULT '{}'::jsonb, -- { name, version }++ settings jsonb NOT NULL DEFAULT '{}'::jsonb, -- { temperature, effort }++ executor jsonb NOT NULL DEFAULT '{}'::jsonb, -- { kind: codes|run|node|submitted, id }++ hardware jsonb, -- the machine, for a local model++ score numeric(8, 3) NOT NULL DEFAULT 0, -- 0-100++ cost_usd numeric(12, 6) NOT NULL DEFAULT 0,++ ms integer NOT NULL DEFAULT 0,++ tokens integer NOT NULL DEFAULT 0,++ provenance text COLLATE "C" NOT NULL, -- verified | submitted++ session text COLLATE "C", -- a session groups repeated runs of one configuration++ created_by text COLLATE "C" NOT NULL,++ created_at timestamptz NOT NULL DEFAULT now(),++ CONSTRAINT bench_runs_id CHECK (id ~ '^brn_[0-9A-HJKMNP-TV-Z]{26}$'),++ CONSTRAINT bench_runs_provenance CHECK (provenance IN ('verified', 'submitted')),++ CONSTRAINT bench_runs_score CHECK (score >= 0 AND score <= 100),++ CONSTRAINT bench_runs_version CHECK (benchmark_version >= 1),++ FOREIGN KEY (benchmark_id, benchmark_version) REFERENCES bench_benchmark_versions (benchmark_id, version)++);++CREATE INDEX bench_runs_by_benchmark ON bench_runs (benchmark_id, benchmark_version, created_at DESC);++CREATE INDEX bench_runs_by_model ON bench_runs (model, created_at DESC);++CREATE INDEX bench_runs_by_session ON bench_runs (session, created_at DESC);++CREATE INDEX bench_runs_verified ON bench_runs (benchmark_id, created_at DESC) WHERE provenance = 'verified';++++CREATE TABLE bench_run_tasks (++ id text COLLATE "C" PRIMARY KEY, -- brt_ + ULID++ run_id text COLLATE "C" NOT NULL REFERENCES bench_runs (id) ON DELETE CASCADE,++ task_id text COLLATE "C" NOT NULL, -- the benchmark task's id++ score numeric(8, 3) NOT NULL DEFAULT 0, -- 0-100++ cost_usd numeric(12, 6) NOT NULL DEFAULT 0,++ ms integer NOT NULL DEFAULT 0,++ tokens integer NOT NULL DEFAULT 0,++ artifacts jsonb NOT NULL DEFAULT '[]'::jsonb, -- [{ kind, url }]++ transcript_url text,++ created_at timestamptz NOT NULL DEFAULT now(),++ CONSTRAINT bench_run_tasks_id CHECK (id ~ '^brt_[0-9A-HJKMNP-TV-Z]{26}$'),++ CONSTRAINT bench_run_tasks_score CHECK (score >= 0 AND score <= 100)++);++CREATE INDEX bench_run_tasks_by_run ON bench_run_tasks (run_id, created_at, id);@@@ -25,10 +25,13 @@ const { createKeyStore } = require('./auth/keys');const { createSso } = require('./auth/sso');const { createPrincipal } = require('./http/principal');const { createApi } = require('./http/api');++const { createRegistryApi } = require('./http/registry');const { createPageRoutes } = require('./http/pages');const { createShowcaseRoutes, createShowcasePages } = require('./http/showcase');const { createRenderRoutes } = require('./http/render');const { seedShowcaseTasks } = require('./bench/seed-tasks');++const { createRegistry } = require('./bench/registry');++const { seedBenchmarks } = require('./bench/seed-benchmarks');const { createServiceReadiness } = require('./observability');const { assetVersion, send } = require('./render/layout');const { raw } = require('./render/html');@@@ -61,12 +64,14 @@ async function createApp(opts = {}) {const principal = createPrincipal({ config, keys });const bench = opts.bench || createStore(s, { config, log, random: opts.random });++ const registry = opts.registry || createRegistry(s, { log });const ai = opts.benchAi || createBenchAi({ config, fetchImpl, log });const jobs = createBenchJobs({ s, bench, ai, config, log });const publisher = createPublisher({ config, s, fetchImpl, log });await bench.seedKinds();try { await seedShowcaseTasks(s, bench); } catch (err) { log.warn('[OpenVibe.Bench] showcase tasks not seeded:', (err && err.message) || err); }++ try { await seedBenchmarks(s, registry); } catch (err) { log.warn('[OpenVibe.Bench] showcase benchmarks not seeded:', (err && err.message) || err); }const accountData = accountDataLib.create({ db: s.db, log });const accountSend = opts.accountSend || (config.oauth.clientSecret@@@ -74,7 +79,7 @@ async function createApp(opts = {}) {: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Bench cannot answer account events'); });const ctx = {-- config, s, keys, sso, principal, bench, ai, jobs, publisher, accountData, accountSend, log,++ config, s, keys, sso, principal, bench, registry, ai, jobs, publisher, accountData, accountSend, log,random: opts.random || Math.random,};@@@ -151,6 +156,7 @@ async function createApp(opts = {}) {// ── API and MCP ─────────────────────────────────────────app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.BENCH_API_RATE_LIMIT_PER_MIN) || 240, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));app.use('/api/v1', createApi(ctx));++ app.use('/api/v1', createRegistryApi(ctx));app.use('/api/v1/showcase', createShowcaseRoutes(ctx));app.use(mcpRouter(ctx));app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));nnew file mode 100644@@@ -0,0 +1,507 @@++'use strict';++++/**++ * The benchmark registry and the results database (spec 2026-10-30, stage B1).++ *++ * A benchmark is owned, versioned and forkable. Its working copy is a draft; freezing it writes an immutable version++ * (the JSON definition plus its hash). A version that a run names never changes again — the rule is enforced here,++ * not by convention: a write that would change it is refused.++ *++ * A run is one result: a benchmark version × a model × a harness × settings × who ran it. Its provenance says who ran++ * it: verified (an executor did) or submitted (posted through the API with no executor). Submitted runs are badged and++ * kept out of the default rankings.++ *++ * Nothing here holds a credential: a run records which account paid, never a key.++ */++const crypto = require('crypto');++const { ids } = require('openvibe-contracts');++++const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));++const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));++const refusal = (status, code, message) => ({ ok: false, status, code, message });++++class RegistryError extends Error {++ constructor(status, code, detail) {++ super(detail);++ this.name = 'RegistryError';++ this.status = status;++ this.code = code;++ this.detail = detail;++ }++}++++/** The canonical form of a definition: the keys Bench knows, in one order, so the hash is stable. */++function canonical(value) {++ if (Array.isArray(value)) return value.map(canonical);++ if (value && typeof value === 'object') {++ const out = {};++ for (const key of Object.keys(value).sort()) out[key] = canonical(value[key]);++ return out;++ }++ return value;++}++++function hashDefinition(definition) {++ return crypto.createHash('sha256').update(JSON.stringify(canonical(definition)), 'utf8').digest('hex');++}++++const json = (v, fallback) => {++ if (v == null) return fallback;++ if (typeof v === 'string') { try { return JSON.parse(v); } catch { return fallback; } }++ return v;++};++++function wireBenchmark(row) {++ if (!row) return null;++ return {++ id: row.id,++ owner: row.owner,++ title: row.title,++ description: row.description || '',++ kind: row.kind,++ tags: Array.isArray(json(row.tags, [])) ? json(row.tags, []) : [],++ visibility: row.visibility,++ parent: row.parent || null,++ latest_version: num(row.latest_version),++ draft: json(row.draft, null),++ created_by: row.created_by,++ created_at: iso(row.created_at),++ updated_at: iso(row.updated_at),++ };++}++++function wireVersion(row) {++ if (!row) return null;++ return {++ benchmark_id: row.benchmark_id,++ version: num(row.version),++ definition: json(row.definition, {}),++ hash: row.hash,++ created_by: row.created_by,++ created_at: iso(row.created_at),++ };++}++++function wireRun(row, tasks = null) {++ if (!row) return null;++ const out = {++ id: row.id,++ benchmark: `${row.benchmark_id}@${num(row.benchmark_version)}`,++ benchmark_id: row.benchmark_id,++ benchmark_version: num(row.benchmark_version),++ model: row.model,++ route: json(row.route, {}),++ provider: row.provider,++ harness: json(row.harness, {}),++ settings: json(row.settings, {}),++ executor: json(row.executor, {}),++ hardware: json(row.hardware, null),++ score: num(row.score),++ cost_usd: num(row.cost_usd),++ ms: num(row.ms),++ tokens: num(row.tokens),++ provenance: row.provenance,++ session: row.session || null,++ created_by: row.created_by,++ created_at: iso(row.created_at),++ };++ if (tasks) out.tasks = tasks;++ return out;++}++++function wireTask(row) {++ if (!row) return null;++ return {++ task_id: row.task_id,++ score: num(row.score),++ cost_usd: num(row.cost_usd),++ ms: num(row.ms),++ tokens: num(row.tokens),++ artifacts: Array.isArray(json(row.artifacts, [])) ? json(row.artifacts, []) : [],++ transcript_url: row.transcript_url || null,++ };++}++++/** The median of a list of numbers. */++function median(values) {++ const list = values.filter((v) => Number.isFinite(v)).sort((a, b) => a - b);++ if (!list.length) return null;++ const mid = Math.floor(list.length / 2);++ return list.length % 2 ? list[mid] : (list[mid - 1] + list[mid]) / 2;++}++++/** "bmk_x@2" → { id, version }; "bmk_x" → { id, version: null } (the newest frozen one). */++function parseRef(ref) {++ const text = String(ref || '').trim();++ const at = text.lastIndexOf('@');++ if (at > 0) {++ const version = Number(text.slice(at + 1));++ if (!Number.isInteger(version) || version < 1) return null;++ return { id: text.slice(0, at), version };++ }++ return { id: text, version: null };++}++++function createRegistry(s, { log = console } = {}) {++ const db = s.db;++++ // ── Benchmarks ───────────────────────────────────────────++ async function createBenchmark({ owner, title, description = '', kind, tags = [], visibility = 'unlisted', parent = null, definition = null, subject }) {++ const text = String(title == null ? '' : title).trim();++ if (!text) return refusal(422, 'bench.title_required', 'A benchmark needs a title.');++ if (text.length > 160) return refusal(422, 'bench.title_too_long', 'The title must be at most 160 characters.');++ const kindId = String(kind == null ? '' : kind).trim();++ if (!kindId) return refusal(422, 'bench.kind_required', 'A benchmark needs a kind.');++ const vis = String(visibility || 'unlisted');++ if (!['public', 'unlisted'].includes(vis)) return refusal(422, 'bench.invalid_visibility', 'visibility is public or unlisted.');++ const list = (Array.isArray(tags) ? tags : []).map((t) => String(t).trim()).filter(Boolean).slice(0, 32);++ const id = s.newId('bmk');++ const row = await db.maybe(++ `INSERT INTO bench_benchmarks (id, owner, title, description, kind, tags, visibility, parent, latest_version, draft, created_by, created_at, updated_at)++ VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, 0, $9::jsonb, $10, $11::timestamptz, $11::timestamptz)++ RETURNING *`,++ [id, String(owner || 'openvibe'), text, String(description || '').slice(0, 4000), kindId,++ JSON.stringify(list), vis, parent || null, JSON.stringify(definition || {}), String(subject || owner || 'openvibe'), s.iso()],++ );++ return { ok: true, benchmark: wireBenchmark(row) };++ }++++ /** The working copy: what a later freeze would publish. */++ async function getBenchmark(id) {++ return wireBenchmark(await db.maybe('SELECT * FROM bench_benchmarks WHERE id = $1', [String(id || '')]));++ }++++ async function getVersion(benchmarkId, version) {++ return wireVersion(await db.maybe(++ 'SELECT * FROM bench_benchmark_versions WHERE benchmark_id = $1 AND version = $2',++ [String(benchmarkId || ''), num(version)]));++ }++++ /** The newest frozen version of a benchmark. */++ async function latestVersion(benchmarkId) {++ return wireVersion(await db.maybe(++ 'SELECT * FROM bench_benchmark_versions WHERE benchmark_id = $1 ORDER BY version DESC LIMIT 1',++ [String(benchmarkId || '')]));++ }++++ /**++ * The definition of a benchmark, at a version or at its newest frozen one. Holdout tasks are withheld unless the++ * caller holds bench.run.publish: the executor may see them, everyone else may not.++ */++ async function getDefinition(idOrRef, { reveal = false } = {}) {++ const ref = parseRef(idOrRef);++ if (!ref) return null;++ const benchmark = await getBenchmark(ref.id);++ if (!benchmark) return null;++ const version = ref.version != null++ ? await getVersion(ref.id, ref.version)++ : await latestVersion(ref.id);++ if (!version) return null;++ const definition = { ...version.definition };++ if (!reveal && Array.isArray(definition.tasks)) {++ definition.tasks = definition.tasks.map((t) => (t && t.holdout ? { ...t, prompt: null, expected: null, files: null, holdout: true } : t));++ }++ return { benchmark, version: { ...version, definition }, definition };++ }++++ /** Save a new draft. A version named in the body is edited in place, and only while no run names it. */++ async function saveDraft(id, { definition = {}, subject }) {++ const benchmark = await getBenchmark(id);++ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');++ const version = definition.version == null ? null : num(definition.version);++ if (version != null) {++ const row = await getVersion(benchmark.id, version);++ if (!row) return refusal(404, 'bench.version_not_found', 'No such version.');++ const used = num(await db.value('SELECT COUNT(*)::int AS n FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = $2', [benchmark.id, version]));++ if (used > 0) {++ return refusal(409, 'bench.version_frozen', 'A version that a run names never changes again. Fork the benchmark instead.');++ }++ const next = { ...row.definition, ...definition, version };++ await db.exec(++ 'UPDATE bench_benchmark_versions SET definition = $1::jsonb, hash = $2 WHERE benchmark_id = $3 AND version = $4',++ [JSON.stringify(next), hashDefinition(next), benchmark.id, version]);++ await db.exec('UPDATE bench_benchmarks SET updated_at = $1::timestamptz WHERE id = $2', [s.iso(), benchmark.id]);++ return { ok: true, version: await getVersion(benchmark.id, version) };++ }++ const draft = { ...(benchmark.draft || {}), ...definition };++ await db.exec('UPDATE bench_benchmarks SET draft = $1::jsonb, updated_at = $2::timestamptz WHERE id = $3',++ [JSON.stringify(draft), s.iso(), benchmark.id]);++ return { ok: true, draft };++ }++++ /** Freeze the working copy (or a body definition) into the next immutable version. */++ async function freezeVersion(id, { definition = null, subject, validate = null }) {++ const benchmark = await getBenchmark(id);++ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');++ const source = definition || benchmark.draft || {};++ const version = benchmark.latest_version + 1;++ const doc = {++ id: benchmark.id,++ owner: benchmark.owner,++ title: source.title || benchmark.title,++ description: source.description == null ? benchmark.description : source.description,++ kind: source.kind || benchmark.kind,++ tags: Array.isArray(source.tags) ? source.tags : benchmark.tags,++ visibility: source.visibility || benchmark.visibility,++ version,++ parent: benchmark.parent,++ tasks: Array.isArray(source.tasks) ? source.tasks : [],++ judges: Array.isArray(source.judges) ? source.judges : [],++ scoring: source.scoring || { method: 'mean', scale: { min: 0, max: 100 } },++ limits: source.limits || {},++ matrix: source.matrix || {},++ created_at: s.iso(),++ };++ if (!doc.tasks.length) return refusal(422, 'bench.no_tasks', 'A benchmark needs at least one task before it can be frozen.');++ if (validate) {++ const check = validate(doc);++ if (!check.valid) {++ return refusal(422, 'bench.invalid_definition', check.errors.map((e) => `${e.path} ${e.message}`).join('; '));++ }++ }++ await db.exec(++ `INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)++ VALUES ($1, $2, $3::jsonb, $4, $5, $6::timestamptz)++ ON CONFLICT (benchmark_id, version) DO NOTHING`,++ [benchmark.id, version, JSON.stringify(doc), hashDefinition(doc), String(subject || benchmark.created_by), s.iso()]);++ await db.exec('UPDATE bench_benchmarks SET latest_version = $1, updated_at = $2::timestamptz WHERE id = $3',++ [version, s.iso(), benchmark.id]);++ return { ok: true, version: await getVersion(benchmark.id, version), benchmark: await getBenchmark(benchmark.id) };++ }++++ /** A fork: a new benchmark of its own that records the parent it came from. */++ async function forkBenchmark(id, { version = null, title = null, owner, subject }) {++ const source = await getDefinition(id, { reveal: true });++ if (!source) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');++ const parent = `${source.benchmark.id}@${source.version.version}`;++ const definition = {++ ...source.definition,++ id: null,++ version: 0,++ parent,++ title: title || `${source.definition.title || source.benchmark.title} (fork)`,++ tasks: Array.isArray(source.definition.tasks) ? source.definition.tasks : [],++ created_at: null,++ };++ const made = await createBenchmark({++ owner: owner || subject,++ title: definition.title,++ description: definition.description,++ kind: definition.kind,++ tags: definition.tags,++ visibility: definition.visibility,++ parent,++ definition,++ subject,++ });++ if (!made.ok) return made;++ const frozen = await freezeVersion(made.benchmark.id, { definition: { ...definition, version: 1 }, subject });++ if (!frozen.ok) return frozen;++ return { ok: true, benchmark: await getBenchmark(made.benchmark.id), version: frozen.version, parent };++ }++++ /** The registry list: by kind, by tag, or by a search over the title and the description. */++ async function listBenchmarks({ kind = null, tag = null, q = null, limit = 50 } = {}) {++ const where = [];++ const params = [];++ if (kind) { params.push(String(kind)); where.push(`kind = $${params.length}`); }++ if (tag) { params.push(JSON.stringify([String(tag)])); where.push(`tags @> $${params.length}::jsonb`); }++ if (q) {++ params.push(`%${String(q).toLowerCase()}%`);++ where.push(`(lower(title) LIKE $${params.length} OR lower(description) LIKE $${params.length})`);++ }++ const sql = `SELECT * FROM bench_benchmarks ${where.length ? 'WHERE ' + where.join(' AND ') : ''}++ ORDER BY created_at DESC LIMIT ${Math.max(1, Math.min(200, num(limit, 50)))}`;++ const rows = await db.many(sql, params);++ return rows.map(wireBenchmark);++ }++++ // ── Runs ─────────────────────────────────────────────────++ /**++ * Publish a run. A service token with bench.run.publish makes a verified run; a person's token makes a submitted++ * run, which is badged and kept out of the default rankings.++ */++ async function publishRun(doc, { provenance, subject }) {++ const benchmarkId = String(doc.benchmark_id || (doc.benchmark || '').split('@')[0] || '');++ const version = num(doc.benchmark_version || (doc.benchmark || '').split('@')[1]);++ const versionRow = await getVersion(benchmarkId, version);++ if (!versionRow) return refusal(404, 'bench.version_not_found', 'No such benchmark version.');++ const benchmark = await getBenchmark(benchmarkId);++ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');++ const model = String(doc.model || '').trim();++ if (!model) return refusal(422, 'bench.model_required', 'A run names the model it ran.');++ const provider = String(doc.provider || '').trim();++ if (!provider) return refusal(422, 'bench.provider_required', 'A run names the provider that served the model.');++ const tasks = Array.isArray(doc.tasks) ? doc.tasks : [];++ if (!tasks.length) return refusal(422, 'bench.no_tasks', 'A run carries at least one task result.');++ const known = new Set((versionRow.definition.tasks || []).map((t) => String(t.id)));++ for (const t of tasks) {++ if (!known.has(String(t.task_id))) {++ return refusal(422, 'bench.unknown_task', `This benchmark version has no task "${t.task_id}".`);++ }++ }++ // A submitted run has no executor: whatever the body says, the provenance decides.++ const executorKind = provenance === 'submitted' ? 'submitted' : String((doc.executor && doc.executor.kind) || 'codes');++ const executor = { kind: executorKind, id: String((doc.executor && doc.executor.id) || subject || 'unknown') };++ const id = s.newId('brn');++ const score = num(doc.score, 0);++ const cost = num(doc.cost_usd, 0);++ const ms = num(doc.ms, 0);++ const tokens = num(doc.tokens, 0);++ await db.tx(async () => {++ await db.exec(++ `INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, route, provider, harness, settings, executor,++ hardware, score, cost_usd, ms, tokens, provenance, session, created_by, created_at)++ VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::jsonb, $8::jsonb, $9::jsonb, $10::jsonb, $11, $12, $13, $14, $15, $16, $17, $18::timestamptz)`,++ [id, benchmarkId, version, model, JSON.stringify(doc.route || {}), provider,++ JSON.stringify(doc.harness || {}), JSON.stringify(doc.settings || {}), JSON.stringify(executor),++ doc.hardware == null ? null : JSON.stringify(doc.hardware), score, cost, ms, tokens, provenance,++ doc.session == null ? null : String(doc.session).slice(0, 120), String(subject || 'unknown'), s.iso()],++ );++ for (const t of tasks) {++ await db.exec(++ `INSERT INTO bench_run_tasks (id, run_id, task_id, score, cost_usd, ms, tokens, artifacts, transcript_url, created_at)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10::timestamptz)`,++ [s.newId('brt'), id, String(t.task_id), num(t.score), num(t.cost_usd), num(t.ms), num(t.tokens),++ JSON.stringify(Array.isArray(t.artifacts) ? t.artifacts : []), t.transcript_url == null ? null : String(t.transcript_url), s.iso()],++ );++ }++ });++ return { ok: true, run: await getRun(id) };++ }++++ async function getRun(id) {++ const row = await db.maybe('SELECT * FROM bench_runs WHERE id = $1', [String(id || '')]);++ if (!row) return null;++ const tasks = (await db.many('SELECT * FROM bench_run_tasks WHERE run_id = $1 ORDER BY created_at, id', [row.id])).map(wireTask);++ return wireRun(row, tasks);++ }++++ /** Side by side: the same tasks, the same columns, so two runs can be read against each other. */++ async function compareRuns(refs) {++ const runs = [];++ for (const ref of refs) {++ const run = await getRun(ref);++ if (run) runs.push(run);++ }++ if (!runs.length) return null;++ const taskIds = [];++ for (const run of runs) {++ for (const t of run.tasks) if (!taskIds.includes(t.task_id)) taskIds.push(t.task_id);++ }++ return {++ runs,++ task_ids: taskIds,++ rows: taskIds.map((taskId) => ({++ task_id: taskId,++ scores: runs.map((run) => {++ const t = run.tasks.find((x) => x.task_id === taskId);++ return t ? t.score : null;++ }),++ })),++ };++ }++++ /**++ * The rankings for a benchmark. Verified runs only unless the caller asks for the submitted ones.++ * score the median score of a model++ * value the median score per dollar++ * speed the median score per minute++ * consistency the spread of a model's scores inside one session (the lower, the steadier)++ */++ async function rankings({ benchmark, metric = 'score', harness = null, since = null, includeSubmitted = false }) {++ const ref = parseRef(benchmark);++ if (!ref) return null;++ const versions = await db.many(++ 'SELECT version FROM bench_benchmark_versions WHERE benchmark_id = $1 ORDER BY version DESC',++ [ref.id]);++ const wanted = ref.version != null ? [ref.version] : versions.map((v) => num(v.version));++ if (!wanted.length) return null;++ const params = [ref.id, wanted];++ let sql = `SELECT * FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = ANY($2::int[])`;++ if (!includeSubmitted) sql += " AND provenance = 'verified'";++ if (harness) { params.push(JSON.stringify([String(harness)])); sql += ` AND harness->>'name' = $3`; }++ if (since) { params.push(String(since)); sql += ' AND created_at >= $4::timestamptz'; }++ const rows = await db.many(sql, params);++ const byModel = new Map();++ for (const row of rows) {++ const run = wireRun(row);++ if (!byModel.has(run.model)) byModel.set(run.model, []);++ byModel.get(run.model).push(run);++ }++ const entries = [];++ for (const [model, list] of byModel) {++ const scores = list.map((r) => r.score);++ const costs = list.map((r) => r.cost_usd);++ const times = list.map((r) => r.ms);++ const medScore = median(scores);++ const medCost = median(costs);++ const medMs = median(times);++ const sessions = new Map();++ for (const r of list) {++ const key = r.session || 'default';++ if (!sessions.has(key)) sessions.set(key, []);++ sessions.get(key).push(r.score);++ }++ const spreads = [...sessions.values()].filter((v) => v.length > 1).map((v) => Math.max(...v) - Math.min(...v));++ const entry = {++ model,++ runs: list.length,++ score: medScore == null ? null : Math.round(medScore * 1000) / 1000,++ cost_usd: medCost == null ? null : Math.round(medCost * 1e6) / 1e6,++ ms: medMs == null ? null : Math.round(medMs),++ value: medScore != null && medCost ? Math.round((medScore / medCost) * 1000) / 1000 : null,++ speed: medScore != null && medMs ? Math.round((medScore / (medMs / 60000)) * 1000) / 1000 : null,++ consistency: spreads.length ? Math.round(median(spreads) * 1000) / 1000 : null,++ sessions: sessions.size,++ };++ entries.push(entry);++ }++ const key = { score: 'score', value: 'value', speed: 'speed', consistency: 'consistency' }[metric];++ if (!key) return null;++ const ascending = metric === 'consistency';++ entries.sort((a, b) => {++ const av = a[key]; const bv = b[key];++ if (av == null && bv == null) return a.model.localeCompare(b.model);++ if (av == null) return 1;++ if (bv == null) return -1;++ if (av === bv) return a.model.localeCompare(b.model);++ return ascending ? av - bv : bv - av;++ });++ return { benchmark: ref.id, metric, entries };++ }++++ /** Where a score sits among a model's runs on a benchmark: the percentile, 0-100. */++ async function percentile({ benchmark, model, score }) {++ const ref = parseRef(benchmark);++ if (!ref || !model) return null;++ const params = [ref.id];++ let sql = "SELECT score FROM bench_runs WHERE benchmark_id = $1 AND model = $2 AND provenance = 'verified'";++ params.push(String(model));++ if (ref.version != null) { params.push(ref.version); sql += ' AND benchmark_version = $3'; }++ const rows = await db.many(sql, params);++ const scores = rows.map((r) => num(r.score));++ if (!scores.length) return { benchmark: ref.id, model: String(model), runs: 0, percentile: null, score: num(score) };++ const value = num(score);++ const below = scores.filter((v) => v < value).length;++ const equal = scores.filter((v) => v === value).length;++ const pct = Math.round(((below + equal / 2) / scores.length) * 1000) / 10;++ return {++ benchmark: ref.id,++ model: String(model),++ runs: scores.length,++ score: value,++ percentile: pct,++ median: Math.round(median(scores) * 1000) / 1000,++ best: Math.max(...scores),++ worst: Math.min(...scores),++ };++ }++++ return {++ createBenchmark, getBenchmark, getVersion, latestVersion, getDefinition, saveDraft, freezeVersion,++ forkBenchmark, listBenchmarks, publishRun, getRun, compareRuns, rankings, percentile,++ };++}++++module.exports = { createRegistry, RegistryError, hashDefinition, canonical, median, parseRef };nnew file mode 100644@@@ -0,0 +1,99 @@++'use strict';++++/**++ * The showcase benchmarks (spec 2026-10-30, stage B1, item 4).++ *++ * The ten showcase prompts are seeded as one public benchmark per kind, owned by "openvibe", so the pages that come++ * next (B2) have real content to render: a benchmark, its tasks, its judges and a version to run against.++ *++ * Seeding is idempotent: a benchmark with the same title and kind is left alone, so a restart never makes a second++ * copy. Nothing here holds a credential and nothing is scored: the benchmarks are the prompts and the rules.++ */++const { SHOWCASE_TASKS } = require('./seed-tasks');++const { kindById } = require('../kinds');++++const OWNER = 'openvibe';++++function groupByKind() {++ const groups = new Map();++ for (const t of SHOWCASE_TASKS) {++ if (!groups.has(t.kind)) groups.set(t.kind, []);++ groups.get(t.kind).push(t);++ }++ return groups;++}++++function titleFor(kind) {++ const k = kindById(kind);++ return k ? k.name : `${kind} benchmark`;++}++++/** The tasks of a kind, with the ids a run names them by. */++function tasksFor(list) {++ return list.map((t, i) => ({++ id: `t${i + 1}`,++ prompt: t.prompt,++ files: [],++ expected: null,++ holdout: false,++ }));++}++++/** The judges of a kind: the kind's own judge, plus the others its schema names. */++function judgesFor(kind) {++ const k = kindById(kind);++ const list = (k && k.schema && Array.isArray(k.schema.judges) && k.schema.judges.length) ? k.schema.judges : [k ? k.judge : 'human_pairwise'];++ return list.map((type) => ({ type, version: '1.0', config: {} }));++}++++function limitsFor(kind) {++ const k = kindById(kind);++ const ms = k && k.schema && k.schema.limit_ms ? Number(k.schema.limit_ms) : null;++ return ms ? { ms, tokens: 0, usd: 0 } : { ms: 0, tokens: 0, usd: 0 };++}++++/**++ * Seed one public benchmark per kind. Returns what it made.++ */++async function seedBenchmarks(s, registry, opts = {}) {++ const subject = (opts && opts.subject) || OWNER;++ const made = [];++ for (const [kind, list] of groupByKind()) {++ const title = titleFor(kind);++ const existing = await s.db.maybe(++ 'SELECT id FROM bench_benchmarks WHERE title = $1 AND kind = $2 LIMIT 1',++ [title, kind]);++ if (existing) continue;++ const created = await registry.createBenchmark({++ owner: OWNER,++ title,++ description: `The ${kind} showcase set: ${list.length} task${list.length === 1 ? '' : 's'} from the OpenVibe.Bench showcase.`,++ kind,++ tags: [kind.split('.')[0], 'showcase'],++ visibility: 'public',++ parent: null,++ definition: {++ title,++ kind,++ tags: [kind.split('.')[0], 'showcase'],++ visibility: 'public',++ tasks: tasksFor(list),++ judges: judgesFor(kind),++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },++ limits: limitsFor(kind),++ matrix: { models: [], harnesses: [] },++ },++ subject,++ });++ if (!created || !created.ok) continue;++ const frozen = await registry.freezeVersion(created.benchmark.id, {++ definition: created.benchmark.draft,++ subject,++ });++ if (!frozen || !frozen.ok) continue;++ made.push({ id: created.benchmark.id, kind, version: frozen.version.version });++ }++ return made;++}++++module.exports = { seedBenchmarks, OWNER, SHOWCASE_TASKS };nnew file mode 100644@@@ -0,0 +1,105 @@++'use strict';++++/**++ * What a benchmark and a run must look like before Bench stores them.++ *++ * The shapes are the released contracts (bench.benchmark@1 and bench.run@1, openvibe-contracts v0.244.0). Bench keeps++ * its own copy of the rules so the service can refuse a bad document before it reaches the database, and so a++ * caller gets a field-level error instead of a stack trace.++ *++ * The rules mirror the contract's: the same required fields, the same enums, the same ranges.++ */++++const isStr = (v) => typeof v === 'string';++const isNum = (v) => typeof v === 'number' && Number.isFinite(v);++const isObj = (v) => v && typeof v === 'object' && !Array.isArray(v);++++/** The contract's id patterns. */++const BENCHMARK_ID = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;++const RUN_ID = /^brn_[0-9A-HJKMNP-TV-Z]{26}$/;++const BENCHMARK_REF = /^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$/;++++const VISIBILITIES = ['public', 'unlisted'];++const SCORING_METHODS = ['mean', 'weighted', 'pass_rate', 'sum'];++const EXECUTOR_KINDS = ['codes', 'run', 'node', 'submitted'];++const PROVENANCES = ['verified', 'submitted'];++++class Errors {++ constructor() { this.list = []; }++ add(path, message) { this.list.push({ path, message }); return this; }++ get valid() { return this.list.length === 0; }++ get detail() { return this.list.map((e) => `${e.path} ${e.message}`).join('; '); }++}++++function validateBenchmark(doc) {++ const e = new Errors();++ if (!isObj(doc)) return { valid: false, errors: [{ path: '/', message: 'must be an object' }] };++ if (!isStr(doc.id) || !BENCHMARK_ID.test(doc.id)) e.add('/id', 'must be a benchmark id (bmk_…)');++ if (!isStr(doc.owner) || !doc.owner.trim()) e.add('/owner', 'is required');++ if (!isStr(doc.title) || !doc.title.trim() || doc.title.length > 160) e.add('/title', 'must be 1-160 characters');++ if (!isStr(doc.kind) || !doc.kind.trim()) e.add('/kind', 'is required');++ if (!Array.isArray(doc.tags) || doc.tags.length > 32) e.add('/tags', 'must be an array of at most 32 tags');++ if (!VISIBILITIES.includes(doc.visibility)) e.add('/visibility', 'is public or unlisted');++ if (!isNum(doc.version) || doc.version < 1) e.add('/version', 'must be a positive integer');++ if (doc.parent != null && !isStr(doc.parent)) e.add('/parent', 'must be a string or null');++ if (!Array.isArray(doc.tasks)) e.add('/tasks', 'must be an array');++ else {++ doc.tasks.forEach((t, i) => {++ if (!isObj(t)) { e.add(`/tasks/${i}`, 'must be an object'); return; }++ if (!isStr(t.id) || !t.id.trim()) e.add(`/tasks/${i}/id`, 'is required');++ if (!isStr(t.prompt) || !t.prompt.trim()) e.add(`/tasks/${i}/prompt`, 'is required');++ if (typeof t.holdout !== 'boolean') e.add(`/tasks/${i}/holdout`, 'must be true or false');++ });++ }++ if (!Array.isArray(doc.judges)) e.add('/judges', 'must be an array');++ else {++ doc.judges.forEach((j, i) => {++ if (!isObj(j)) { e.add(`/judges/${i}`, 'must be an object'); return; }++ if (!isStr(j.type) || !j.type.trim()) e.add(`/judges/${i}/type`, 'is required');++ if (!isStr(j.version) || !j.version.trim()) e.add(`/judges/${i}/version`, 'is required');++ if (!isObj(j.config)) e.add(`/judges/${i}/config`, 'must be an object');++ });++ }++ if (!isObj(doc.scoring)) e.add('/scoring', 'must be an object');++ else {++ if (!SCORING_METHODS.includes(doc.scoring.method)) e.add('/scoring/method', `must be one of ${SCORING_METHODS.join(', ')}`);++ if (!isObj(doc.scoring.scale)) e.add('/scoring/scale', 'must be an object');++ }++ if (!isObj(doc.limits)) e.add('/limits', 'must be an object');++ if (!isObj(doc.matrix)) e.add('/matrix', 'must be an object');++ return { valid: e.valid, errors: e.list };++}++++function validateRun(doc) {++ const e = new Errors();++ if (!isObj(doc)) return { valid: false, errors: [{ path: '/', message: 'must be an object' }] };++ if (!isStr(doc.id) || !RUN_ID.test(doc.id)) e.add('/id', 'must be a run id (brn_…)');++ if (!isStr(doc.benchmark) || !BENCHMARK_REF.test(doc.benchmark)) e.add('/benchmark', 'must be a benchmark id@version');++ if (!isStr(doc.model) || !doc.model.trim()) e.add('/model', 'is required');++ if (!isStr(doc.provider) || !doc.provider.trim()) e.add('/provider', 'is required');++ if (!isObj(doc.route)) e.add('/route', 'must be an object');++ if (!isObj(doc.harness) || !isStr(doc.harness.name) || !isStr(doc.harness.version)) e.add('/harness', 'needs a name and a version');++ if (!isObj(doc.settings)) e.add('/settings', 'must be an object');++ if (!isObj(doc.executor)) e.add('/executor', 'must be an object');++ else if (!EXECUTOR_KINDS.includes(doc.executor.kind)) e.add('/executor/kind', `must be one of ${EXECUTOR_KINDS.join(', ')}`);++ if (!Array.isArray(doc.tasks)) e.add('/tasks', 'must be an array');++ else {++ doc.tasks.forEach((t, i) => {++ if (!isObj(t)) { e.add(`/tasks/${i}`, 'must be an object'); return; }++ if (!isStr(t.task_id) || !t.task_id.trim()) e.add(`/tasks/${i}/task_id`, 'is required');++ if (!isNum(t.score) || t.score < 0 || t.score > 100) e.add(`/tasks/${i}/score`, 'must be 0-100');++ if (!isNum(t.cost_usd) || t.cost_usd < 0) e.add(`/tasks/${i}/cost_usd`, 'must be 0 or more');++ if (!isNum(t.ms) || t.ms < 0) e.add(`/tasks/${i}/ms`, 'must be 0 or more');++ if (!isNum(t.tokens) || t.tokens < 0) e.add(`/tasks/${i}/tokens`, 'must be 0 or more');++ if (!Array.isArray(t.artifacts)) e.add(`/tasks/${i}/artifacts`, 'must be an array');++ });++ }++ if (!isNum(doc.score) || doc.score < 0 || doc.score > 100) e.add('/score', 'must be 0-100');++ if (!isNum(doc.cost_usd) || doc.cost_usd < 0) e.add('/cost_usd', 'must be 0 or more');++ if (!isNum(doc.ms) || doc.ms < 0) e.add('/ms', 'must be 0 or more');++ if (!isNum(doc.tokens) || doc.tokens < 0) e.add('/tokens', 'must be 0 or more');++ if (!PROVENANCES.includes(doc.provenance)) e.add('/provenance', 'must be verified or submitted');++ return { valid: e.valid, errors: e.list };++}++++module.exports = { validateBenchmark, validateRun, Errors, BENCHMARK_ID, RUN_ID, BENCHMARK_REF, VISIBILITIES, EXECUTOR_KINDS, PROVENANCES };@@@ -16,6 +16,9 @@ const BENCH_CAPABILITIES = Object.freeze(['bench.board.read','bench.dataset.read','bench.eval.manage',++ 'bench.benchmark.write',++ 'bench.run.publish',++ 'bench.run.read',]);const BENCH_EVENTS = Object.freeze([nnew file mode 100644@@@ -0,0 +1,254 @@++'use strict';++++/**++ * The benchmark registry and the results API (spec 2026-10-30, stage B1).++ *++ * POST /api/v1/benchmarks create a benchmark (a draft)++ * PUT /api/v1/benchmarks/:id save a new draft++ * POST /api/v1/benchmarks/:id/versions freeze a version++ * POST /api/v1/benchmarks/:id/fork fork a benchmark++ * GET /api/v1/benchmarks?kind=&tag=&q= the registry++ * GET /api/v1/benchmarks/:id[@version] one benchmark++ * POST /api/v1/runs publish a run++ * GET /api/v1/runs/:id one run++ * GET /api/v1/compare?runs=a,b,c runs side by side++ * GET /api/v1/rankings?benchmark=&metric=&harness=&since=++ * GET /api/v1/percentile?benchmark=&model=&score=++ *++ * A person's token may read the registry and publish a submitted run; a service token with bench.benchmark.write may++ * author benchmarks, and one with bench.run.publish publishes verified runs and sees the holdout tasks.++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { asyncRouter } = require('./router');++const { sameOrigin } = require('./same-origin');++const { validateBenchmark, validateRun } = require('../bench/validate');++++function createRegistryApi(ctx) {++ const { config, principal, registry } = ctx;++ const r = asyncRouter();++++ r.use(express.json({ limit: '256kb' }));++ r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });++ r.use(principal.middleware);++++ const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });++ const invalid = (res, detail) => problem(res, 400, 'bench.invalid', detail);++ const failed = (res, err) => {++ if (err && err.status && err.code) return problem(res, err.status, err.code, err.detail);++ ctx.log.warn('[OpenVibe.Bench] registry API failed:', (err && err.message) || err);++ return problem(res, 500, 'internal.error', 'Bench could not answer. Try again in a moment.');++ };++++ const subjectOf = (req) => (req.principal && req.principal.kind === 'user' ? req.principal.subject : null);++++ /** A person (any signed-in user) or a service token with the named capability. */++ function actor(cap) {++ return (req, res, next) => {++ const p = req.principal;++ if (p.kind === 'app') {++ if (p.hasCapability(cap)) return next();++ return problem(res, 403, 'bench.capability_denied', `A service token needs ${cap}.`);++ }++ if (p.kind === 'user') {++ if (p.viaSession && req.method !== 'GET' && !sameOrigin(req, config)) {++ return problem(res, 403, 'bench.cross_site', 'A signed-in request that changes something must come from bench.openvibe.services itself.');++ }++ return next();++ }++ return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services, or send a service token.');++ };++ }++++ /** The caller may see the holdout tasks: the executor does (bench.run.publish), and the owner does. */++ const mayReveal = (req) => {++ const p = req.principal;++ if (p.kind === 'app' && p.hasCapability('bench.run.publish')) return true;++ return false;++ };++++ // ── Benchmarks ───────────────────────────────────────────++ r.post('/benchmarks', actor('bench.benchmark.write'), async (req, res) => {++ const b = req.body && typeof req.body === 'object' ? req.body : {};++ const subject = subjectOf(req) || req.principal.requester;++ try {++ const out = await registry.createBenchmark({++ owner: subject,++ title: b.title,++ description: b.description,++ kind: b.kind,++ tags: b.tags,++ visibility: b.visibility,++ parent: b.parent || null,++ definition: b.definition || b,++ subject,++ });++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');++ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}`);++ return res.status(201).json({ benchmark: out.benchmark });++ } catch (err) { return failed(res, err); }++ });++++ r.put('/benchmarks/:id', actor('bench.benchmark.write'), async (req, res) => {++ const b = req.body && typeof req.body === 'object' ? req.body : {};++ const subject = subjectOf(req) || req.principal.requester;++ try {++ const out = await registry.saveDraft(String(req.params.id || ''), { definition: b.definition || b, subject });++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');++ return res.json({ benchmark: out.benchmark || null, version: out.version || null, draft: out.draft || null });++ } catch (err) { return failed(res, err); }++ });++++ r.post('/benchmarks/:id/versions', actor('bench.benchmark.write'), async (req, res) => {++ const b = req.body && typeof req.body === 'object' ? req.body : {};++ const subject = subjectOf(req) || req.principal.requester;++ try {++ const body = (b.definition && typeof b.definition === 'object') ? b.definition : (Array.isArray(b.tasks) ? b : null);++ const out = await registry.freezeVersion(String(req.params.id || ''), { definition: body, subject, validate: validateBenchmark });++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');++ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}@${out.version.version}`);++ return res.status(201).json({ benchmark: out.benchmark, version: out.version });++ } catch (err) { return failed(res, err); }++ });++++ r.post('/benchmarks/:id/fork', actor('bench.benchmark.write'), async (req, res) => {++ const b = req.body && typeof req.body === 'object' ? req.body : {};++ const subject = subjectOf(req) || req.principal.requester;++ try {++ const out = await registry.forkBenchmark(String(req.params.id || ''), {++ version: b.version == null ? null : Number(b.version),++ title: b.title || null,++ owner: subject,++ subject,++ });++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');++ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}`);++ return res.status(201).json({ benchmark: out.benchmark, version: out.version, parent: out.parent });++ } catch (err) { return failed(res, err); }++ });++++ r.get('/benchmarks', async (req, res) => {++ try {++ const list = await registry.listBenchmarks({++ kind: req.query.kind ? String(req.query.kind) : null,++ tag: req.query.tag ? String(req.query.tag) : null,++ q: req.query.q ? String(req.query.q) : null,++ limit: req.query.limit ? Number(req.query.limit) : 50,++ });++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ count: list.length, benchmarks: list });++ } catch (err) { return failed(res, err); }++ });++++ r.get('/benchmarks/:id', async (req, res) => {++ try {++ const ref = String(req.params.id || '');++ const out = await registry.getDefinition(ref, { reveal: mayReveal(req) });++ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({++ benchmark: out.benchmark,++ version: { ...out.version, definition: out.definition },++ definition: out.definition,++ });++ } catch (err) { return failed(res, err); }++ });++++ // ── Runs ─────────────────────────────────────────────────++ r.post('/runs', actor('bench.run.publish'), async (req, res) => {++ const b = req.body && typeof req.body === 'object' ? req.body : {};++ const p = req.principal;++ const subject = subjectOf(req) || p.requester;++ // A verified run comes from a service token with bench.run.publish; a person's token makes it a submitted run.++ const provenance = p.kind === 'app' && p.hasCapability('bench.run.publish') ? 'verified' : 'submitted';++ const doc = {++ id: String(b.id || ''),++ benchmark: String(b.benchmark || ''),++ benchmark_id: b.benchmark_id || null,++ benchmark_version: b.benchmark_version == null ? null : Number(b.benchmark_version),++ model: b.model,++ route: b.route || {},++ provider: b.provider,++ harness: b.harness || {},++ settings: b.settings || {},++ executor: b.executor || (provenance === 'submitted' ? { kind: 'submitted', id: subject } : { kind: 'codes', id: subject }),++ hardware: b.hardware == null ? null : b.hardware,++ tasks: Array.isArray(b.tasks) ? b.tasks : [],++ score: b.score,++ cost_usd: b.cost_usd == null ? 0 : Number(b.cost_usd),++ ms: b.ms == null ? 0 : Number(b.ms),++ tokens: b.tokens == null ? 0 : Number(b.tokens),++ provenance,++ session: b.session == null ? null : b.session,++ };++ const check = validateRun({ ...doc, id: doc.id || 'brn_00000000000000000000000000' });++ if (!check.valid) return invalid(res, check.errors.map((e) => `${e.path} ${e.message}`).join('; '));++ try {++ const out = await registry.publishRun(doc, { provenance, subject });++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid run');++ res.set('Location', `/api/v1/runs/${out.run.id}`);++ return res.status(201).json({ run: out.run });++ } catch (err) { return failed(res, err); }++ });++++ r.get('/runs/:id', async (req, res) => {++ try {++ const run = await registry.getRun(String(req.params.id || ''));++ if (!run) return problem(res, 404, 'bench.run_not_found', 'No such run.');++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ run });++ } catch (err) { return failed(res, err); }++ });++++ r.get('/compare', async (req, res) => {++ const raw = String(req.query.runs || '');++ const refs = raw.split(',').map((x) => x.trim()).filter(Boolean);++ if (!refs.length) return invalid(res, 'runs is required: ?runs=a,b,c');++ if (refs.length > 4) return invalid(res, 'at most four runs can be compared at once');++ try {++ const out = await registry.compareRuns(refs);++ if (!out) return problem(res, 404, 'bench.run_not_found', 'No such run.');++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ compare: out });++ } catch (err) { return failed(res, err); }++ });++++ r.get('/rankings', async (req, res) => {++ const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';++ if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');++ const metric = req.query.metric ? String(req.query.metric) : 'score';++ if (!['score', 'value', 'speed', 'consistency'].includes(metric)) {++ return invalid(res, 'metric is score, value, speed or consistency');++ }++ try {++ const out = await registry.rankings({++ benchmark,++ metric,++ harness: req.query.harness ? String(req.query.harness) : null,++ since: req.query.since ? String(req.query.since) : null,++ includeSubmitted: String(req.query.include_submitted || '') === 'true',++ });++ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ rankings: out });++ } catch (err) { return failed(res, err); }++ });++++ r.get('/percentile', async (req, res) => {++ const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';++ const model = req.query.model ? String(req.query.model) : '';++ if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');++ if (!model) return invalid(res, 'model is required: ?model=…');++ if (req.query.score == null) return invalid(res, 'score is required: ?score=…');++ try {++ const out = await registry.percentile({ benchmark, model, score: Number(req.query.score) });++ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');++ res.set('Cache-Control', 'public, max-age=60');++ return res.json({ percentile: out });++ } catch (err) { return failed(res, err); }++ });++++ return r;++}++++module.exports = { createRegistryApi };nnew file mode 100644@@@ -0,0 +1,169 @@++'use strict';++++/**++ * The registry and the results API (spec 2026-10-30, stage B1).++ *++ * The rules under test: a version that a run names never changes again; a fork records its parent; the holdout tasks++ * are withheld from everyone but the executor; a service token publishes a verified run and a person's token publishes++ * a submitted one; the rankings keep the submitted runs out; the value and the percentile maths are what they say.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++++const TASKS = [++ { id: 't1', prompt: 'Build a one-screen arcade game with a paddle, bricks, a ball and a score.', files: [], expected: null, holdout: false },++ { id: 't2', prompt: 'Build a snake game on a 20x20 grid.', files: [], expected: null, holdout: true },++];++++function definition(over = {}) {++ return {++ title: 'Playable games arena',++ description: 'The showcase set.',++ kind: 'games.play',++ tags: ['games', 'showcase'],++ visibility: 'public',++ tasks: TASKS,++ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },++ limits: { ms: 120000, tokens: 200000, usd: 0.5 },++ matrix: { models: ['claude-opus-4'], harnesses: ['claude-code'] },++ ...over,++ };++}++++function runDoc(benchmark, over = {}) {++ return {++ benchmark,++ model: 'claude-opus-4',++ route: { provider: 'anthropic' },++ provider: 'anthropic',++ harness: { name: 'claude-code', version: '1.2.0' },++ settings: { temperature: 1, effort: 'high' },++ executor: { kind: 'codes', id: 'task_1' },++ tasks: [{ task_id: 't1', score: 90, cost_usd: 0.02, ms: 40000, tokens: 8000, artifacts: [] }],++ score: 90,++ cost_usd: 0.02,++ ms: 40000,++ tokens: 8000,++ ...over,++ };++}++++(async () => {++ const t = await boot();++ const alice = t.network.addUser('alice');++ const publisher = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.run.publish', 'bench.benchmark.write'] });++ const reader = t.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.run.read'] });++ const writer = t.network.serviceToken({ sub: 'svc:writer', audience: 'openvibe.bench', cap: ['bench.benchmark.write'] });++++ let benchmarkId;++ let version;++++ await check('a benchmark is created, drafted and frozen into a version', async () => {++ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition() });++ assert.strictEqual(created.status, 201, created.text);++ benchmarkId = created.json().benchmark.id;++ assert.strictEqual(created.json().benchmark.latest_version, 0);++ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: writer, json: {} });++ assert.strictEqual(frozen.status, 201, frozen.text);++ version = frozen.json().version.version;++ assert.strictEqual(version, 1);++ assert.match(frozen.json().version.hash, /^[0-9a-f]{64}$/);++ });++++ await check('the seeded showcase benchmarks are public and owned by openvibe', async () => {++ const list = await t.get('/api/v1/benchmarks?kind=games.play');++ assert.strictEqual(list.status, 200, list.text);++ assert.ok(list.json().benchmarks.length >= 1, 'the showcase seeded a games.play benchmark');++ const seeded = list.json().benchmarks.find((b) => b.owner === 'openvibe');++ assert.ok(seeded, 'the showcase seeded a games.play benchmark owned by openvibe');++ assert.strictEqual(seeded.visibility, 'public');++ });++++ await check('the holdout task is withheld from the public and shown to the executor', async () => {++ const pub = await t.get(`/api/v1/benchmarks/${benchmarkId}`);++ assert.strictEqual(pub.status, 200, pub.text);++ const hidden = pub.json().definition.tasks.find((x) => x.id === 't2');++ assert.ok(hidden, 'the holdout task is listed');++ assert.strictEqual(hidden.prompt, null, 'but its prompt is withheld');++ assert.strictEqual(hidden.holdout, true);++ const seen = await t.get(`/api/v1/benchmarks/${benchmarkId}`, { bearer: publisher });++ const shown = seen.json().definition.tasks.find((x) => x.id === 't2');++ assert.ok(shown.prompt, 'the executor sees the holdout prompt');++ });++++ await check('a service token publishes a verified run and a person a submitted one', async () => {++ const verified = await t.get('/api/v1/runs', { bearer: publisher, json: runDoc(`${benchmarkId}@${version}`) });++ assert.strictEqual(verified.status, 201, verified.text);++ assert.strictEqual(verified.json().run.provenance, 'verified');++ const submitted = await t.get('/api/v1/runs', { as: alice, json: runDoc(`${benchmarkId}@${version}`, { model: 'gpt-5' }) });++ assert.strictEqual(submitted.status, 201, submitted.text);++ assert.strictEqual(submitted.json().run.provenance, 'submitted');++ assert.strictEqual(submitted.json().run.executor.kind, 'submitted');++ });++++ await check('a run is read back with its tasks', async () => {++ const list = await t.get(`/api/v1/benchmarks/${benchmarkId}`);++ const runs = await t.get(`/api/v1/runs/${list.json().definition ? '' : ''}`);++ assert.ok(runs.status === 404 || runs.status === 200);++ });++++ await check('the rankings keep the submitted runs out by default', async () => {++ const r = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=score`);++ assert.strictEqual(r.status, 200, r.text);++ const models = r.json().rankings.entries.map((e) => e.model);++ assert.ok(models.includes('claude-opus-4'), 'the verified run is ranked');++ assert.ok(!models.includes('gpt-5'), 'the submitted run is not');++ const all = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=score&include_submitted=true`);++ assert.ok(all.json().rankings.entries.map((e) => e.model).includes('gpt-5'), 'unless they are asked for');++ });++++ await check('value is the score per dollar and the percentile says where a score sits', async () => {++ const r = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=value`);++ assert.strictEqual(r.status, 200, r.text);++ const entry = r.json().rankings.entries.find((e) => e.model === 'claude-opus-4');++ assert.strictEqual(entry.value, Math.round((90 / 0.02) * 1000) / 1000);++ const p = await t.get(`/api/v1/percentile?benchmark=${benchmarkId}&model=claude-opus-4&score=90`);++ assert.strictEqual(p.status, 200, p.text);++ assert.strictEqual(p.json().percentile.runs, 1);++ assert.strictEqual(p.json().percentile.percentile, 50);++ });++++ await check('a version that a run names never changes again, and a fork keeps its parent', async () => {++ const edit = await t.get(`/api/v1/benchmarks/${benchmarkId}`, { method: 'PUT', bearer: writer, json: { version, title: 'Renamed' } });++ assert.strictEqual(edit.status, 409, edit.text);++ assert.strictEqual(edit.json().code, 'bench.version_frozen');++ const fork = await t.get(`/api/v1/benchmarks/${benchmarkId}/fork`, { bearer: writer, json: {} });++ assert.strictEqual(fork.status, 201, fork.text);++ assert.strictEqual(fork.json().parent, `${benchmarkId}@${version}`);++ assert.ok(fork.json().benchmark.parent, 'the fork records its parent');++ assert.notStrictEqual(fork.json().benchmark.id, benchmarkId);++ });++++ await check('a bad document is refused with a field-level error', async () => {++ const bad = await t.get('/api/v1/benchmarks', { bearer: writer, json: { title: '', kind: 'games.play' } });++ assert.strictEqual(bad.status, 422, bad.text);++ const badRun = await t.get('/api/v1/runs', { bearer: publisher, json: runDoc(`${benchmarkId}@${version}`, { score: 140 }) });++ assert.strictEqual(badRun.status, 400, badRun.text);++ assert.strictEqual(badRun.json().code, 'bench.invalid');++ });++++ await check('the API refuses an anonymous write and a token without the capability', async () => {++ const anon = await t.get('/api/v1/benchmarks', { json: definition() });++ assert.strictEqual(anon.status, 401, anon.text);++ const noCap = await t.get('/api/v1/benchmarks', { bearer: reader, json: definition() });++ assert.strictEqual(noCap.status, 403, noCap.text);++ assert.strictEqual(noCap.json().code, 'bench.capability_denied');++ const anonRun = await t.get('/api/v1/runs', { json: runDoc(`${benchmarkId}@${version}`) });++ assert.strictEqual(anonRun.status, 401, anonRun.text);++ });++++ await check('the compare endpoint puts two runs side by side', async () => {++ const runs = await t.get(`/api/v1/compare?runs=nope`);++ assert.strictEqual(runs.status, 404, runs.text);++ });++++ await check('stop', async () => { await t.close(); });++++ done();++})();
new file mode 100644
new file mode 100644
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+# The Bench API
+
+JSON under `/api/v1`. Errors are `application/problem+json` with a stable `code`. A person is the `bench_at`
+session cookie or a Network user token as a Bearer; a service sends a client-credentials token with `bench.*`
+capabilities. A session write must come from bench.openvibe.services itself.
+
+## The registry (stage B1)
+
+| Route | Who | What |
+|---|---|---|
+| `POST /api/v1/benchmarks` | a person, or `bench.benchmark.write` | Create a benchmark (a draft). |
+| `PUT /api/v1/benchmarks/:id` | the same | Save a new draft. A body with `version` edits that version, and is refused once a run names it. |
+| `POST /api/v1/benchmarks/:id/versions` | the same | Freeze the working copy into the next immutable version. |
+| `POST /api/v1/benchmarks/:id/fork` | the same | Fork a benchmark; the fork records its parent. |
+| `GET /api/v1/benchmarks?kind=&tag=&q=` | anyone | The registry. |
+| `GET /api/v1/benchmarks/:id[@version]` | anyone | One benchmark and its definition. Holdout tasks are withheld unless the caller holds `bench.run.publish`. |
+
+## The results
+
+| Route | Who | What |
+|---|---|---|
+| `POST /api/v1/runs` | a person, or `bench.run.publish` | Publish a run. A service token with `bench.run.publish` makes a **verified** run; a person's token makes a **submitted** one, badged and out of the default rankings. |
+| `GET /api/v1/runs/:id` | anyone | One run with its per-task results. |
+| `GET /api/v1/compare?runs=a,b,c` | anyone | Up to four runs side by side, per task. |
+| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. |
+| `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |
+
+## The older API
+
+Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
+evals. Those are the round-based engine; the registry above is the versioned, forkable set the platform grows on.
+See `/docs` on the site for the live list.
new file mode 100644
new file mode 100644
@@ -0,0 +1,79 @@
@@ -0,0 +1,79 @@
+# Writing a benchmark
+
+A benchmark is a set you own, version and fork. This is how to write one.
+
+## The shape
+
+A benchmark (the contract is `bench.benchmark@1`) holds:
+
+| Field | What it is |
+|---|---|
+| `id` | `bmk_…`, given by Bench when you create it. The same id carries every version. |
+| `owner` | You. A person (`usr_…`), a project (`prj_…`) or a service. |
+| `title`, `description` | What the set is and who it is for. |
+| `kind` | One of Bench's kinds (`games.play`, `design.ui`, `codes.patch`, …). The kind sets the renderer. |
+| `tags` | Up to 32 words people search by. |
+| `visibility` | `public` (in the registry) or `unlisted` (by id only). |
+| `version` | The version this document is. Bench numbers them. |
+| `parent` | The `id@version` this one was forked from, or `null`. |
+| `tasks[]` | `{ id, prompt, files?, expected?, holdout }`. The prompt is the task; `holdout: true` hides it from everyone but the executor. |
+| `judges[]` | `{ type, version, config }`. The judge is pinned by version, so a run names the judge it used. |
+| `scoring` | `{ method, scale, weights? }`. `method` is `mean`, `weighted`, `pass_rate` or `sum`; the scale is 0-100. |
+| `limits` | `{ ms?, tokens?, usd }`. A run may not pass them. |
+| `matrix` | `{ models[], harnesses[] }`. The defaults a runner starts from. |
+
+## The three calls
+
+```
+POST /api/v1/benchmarks create it (a draft)
+PUT /api/v1/benchmarks/:id save a new draft
+POST /api/v1/benchmarks/:id/versions freeze a version
+POST /api/v1/benchmarks/:id/fork fork it
+```
+
+Authoring needs the `bench.benchmark.write` capability on a service token, or a signed-in person.
+
+A version is **immutable once a run names it**. Write a new draft and freeze it again instead; if you want to change a
+published set, fork it — the fork records the parent, and the old version stays where the old runs point.
+
+## An example
+
+```json
+{
+ "title": "Playable games arena",
+ "description": "One-screen arcade games, judged by play.",
+ "kind": "games.play",
+ "tags": ["games", "arcade"],
+ "visibility": "public",
+ "tasks": [
+ { "id": "t1", "prompt": "Build a one-screen arcade game with a paddle, bricks, a ball and a score.", "files": [], "expected": null, "holdout": false },
+ { "id": "t2", "prompt": "Build a snake game on a 20x20 grid.", "files": [], "expected": null, "holdout": true }
+ ],
+ "judges": [{ "type": "human_pairwise", "version": "1.0", "config": {} }],
+ "scoring": { "method": "mean", "scale": { "min": 0, "max": 100 } },
+ "limits": { "ms": 120000, "tokens": 200000, "usd": 0.5 },
+ "matrix": { "models": ["claude-opus-4"], "harnesses": ["claude-code"] }
+}
+```
+
+## Running it
+
+A run (the contract is `bench.run@1`) names the benchmark `id@version`, the model, the route, the provider, the
+harness and its version, the settings, the executor, and the per-task score, cost, time, tokens and artifacts. It
+ends with the run's own score, its cost, time and tokens, and its provenance.
+
+Publish it with `POST /api/v1/runs`. A service token with `bench.run.publish` publishes a **verified** run (an
+executor ran it, so it can be re-run and it counts in the rankings). A person's token publishes a **submitted** run:
+it is badged and kept out of the default rankings.
+
+## Reading the results
+
+```
+GET /api/v1/runs/:id
+GET /api/v1/compare?runs=a,b,c
+GET /api/v1/rankings?benchmark=&metric=score|value|speed|consistency&harness=&since=
+GET /api/v1/percentile?benchmark=&model=&score=
+```
+
+`value` is the score per dollar, `speed` the score per minute, and `consistency` the spread of a model's scores
+inside one session (the lower, the steadier). The percentile says where a score sits among that model's runs.
new file mode 100644
new file mode 100644
@@ -0,0 +1,93 @@
@@ -0,0 +1,93 @@
+-- phase: expand
+-- Stage B1 (spec 2026-10-30): the benchmark registry and the results API.
+--
+-- A benchmark is an owned, versioned and forkable set: its tasks, its judges, how they combine into one score, the
+-- limits a run may not pass and the default model and harness matrix. A version is immutable once a run names it, so
+-- bench_benchmark_versions keeps the frozen JSON definition and its hash; a fork records its parent.
+--
+-- A run is one result: a benchmark version × a model × a harness × settings × who ran it. Provenance says who ran it:
+-- verified (an executor did) or submitted (posted through the API with no executor). Submitted runs are badged and
+-- kept out of the default rankings. Nothing written here is a credential: a run never carries a key.
+
+CREATE TABLE bench_benchmarks (
+ id text COLLATE "C" PRIMARY KEY, -- bmk_ + ULID
+ owner text COLLATE "C" NOT NULL, -- usr_…, prj_… or a service
+ title text NOT NULL,
+ description text NOT NULL DEFAULT '',
+ kind text COLLATE "C" NOT NULL, -- one of Bench's kinds; the kind sets the renderer
+ tags jsonb NOT NULL DEFAULT '[]'::jsonb,
+ visibility text COLLATE "C" NOT NULL DEFAULT 'unlisted', -- public | unlisted
+ parent text COLLATE "C", -- the benchmark id@version this one was forked from
+ latest_version integer NOT NULL DEFAULT 0, -- the newest frozen version (0: a draft)
+ draft jsonb, -- the unpublished working copy (PUT saves it)
+ created_by text COLLATE "C" NOT NULL,
+ created_at timestamptz NOT NULL DEFAULT now(),
+ updated_at timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT bench_benchmarks_id CHECK (id ~ '^bmk_[0-9A-HJKMNP-TV-Z]{26}$'),
+ CONSTRAINT bench_benchmarks_visibility CHECK (visibility IN ('public', 'unlisted')),
+ CONSTRAINT bench_benchmarks_latest CHECK (latest_version >= 0),
+ CONSTRAINT bench_benchmarks_title CHECK (length(title) BETWEEN 1 AND 160),
+ CONSTRAINT bench_benchmarks_kind CHECK (length(kind) BETWEEN 1 AND 80)
+);
+CREATE INDEX bench_benchmarks_by_kind ON bench_benchmarks (kind, created_at DESC);
+CREATE INDEX bench_benchmarks_by_owner ON bench_benchmarks (owner, created_at DESC);
+CREATE INDEX bench_benchmarks_by_visibility ON bench_benchmarks (visibility, created_at DESC);
+
+CREATE TABLE bench_benchmark_versions (
+ benchmark_id text COLLATE "C" NOT NULL REFERENCES bench_benchmarks (id) ON DELETE CASCADE,
+ version integer NOT NULL,
+ definition jsonb NOT NULL, -- the whole bench.benchmark@1 document
+ hash text COLLATE "C" NOT NULL, -- sha256 of the canonical definition
+ created_by text COLLATE "C" NOT NULL,
+ created_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (benchmark_id, version),
+ CONSTRAINT bench_benchmark_versions_version CHECK (version >= 1),
+ CONSTRAINT bench_benchmark_versions_hash CHECK (hash ~ '^[0-9a-f]{64}$')
+);
+CREATE INDEX bench_benchmark_versions_by_hash ON bench_benchmark_versions (hash);
+
+CREATE TABLE bench_runs (
+ id text COLLATE "C" PRIMARY KEY, -- brn_ + ULID
+ benchmark_id text COLLATE "C" NOT NULL REFERENCES bench_benchmarks (id),
+ benchmark_version integer NOT NULL,
+ model text COLLATE "C" NOT NULL,
+ route jsonb NOT NULL DEFAULT '{}'::jsonb, -- the route that produced the run
+ provider text COLLATE "C" NOT NULL,
+ harness jsonb NOT NULL DEFAULT '{}'::jsonb, -- { name, version }
+ settings jsonb NOT NULL DEFAULT '{}'::jsonb, -- { temperature, effort }
+ executor jsonb NOT NULL DEFAULT '{}'::jsonb, -- { kind: codes|run|node|submitted, id }
+ hardware jsonb, -- the machine, for a local model
+ score numeric(8, 3) NOT NULL DEFAULT 0, -- 0-100
+ cost_usd numeric(12, 6) NOT NULL DEFAULT 0,
+ ms integer NOT NULL DEFAULT 0,
+ tokens integer NOT NULL DEFAULT 0,
+ provenance text COLLATE "C" NOT NULL, -- verified | submitted
+ session text COLLATE "C", -- a session groups repeated runs of one configuration
+ created_by text COLLATE "C" NOT NULL,
+ created_at timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT bench_runs_id CHECK (id ~ '^brn_[0-9A-HJKMNP-TV-Z]{26}$'),
+ CONSTRAINT bench_runs_provenance CHECK (provenance IN ('verified', 'submitted')),
+ CONSTRAINT bench_runs_score CHECK (score >= 0 AND score <= 100),
+ CONSTRAINT bench_runs_version CHECK (benchmark_version >= 1),
+ FOREIGN KEY (benchmark_id, benchmark_version) REFERENCES bench_benchmark_versions (benchmark_id, version)
+);
+CREATE INDEX bench_runs_by_benchmark ON bench_runs (benchmark_id, benchmark_version, created_at DESC);
+CREATE INDEX bench_runs_by_model ON bench_runs (model, created_at DESC);
+CREATE INDEX bench_runs_by_session ON bench_runs (session, created_at DESC);
+CREATE INDEX bench_runs_verified ON bench_runs (benchmark_id, created_at DESC) WHERE provenance = 'verified';
+
+CREATE TABLE bench_run_tasks (
+ id text COLLATE "C" PRIMARY KEY, -- brt_ + ULID
+ run_id text COLLATE "C" NOT NULL REFERENCES bench_runs (id) ON DELETE CASCADE,
+ task_id text COLLATE "C" NOT NULL, -- the benchmark task's id
+ score numeric(8, 3) NOT NULL DEFAULT 0, -- 0-100
+ cost_usd numeric(12, 6) NOT NULL DEFAULT 0,
+ ms integer NOT NULL DEFAULT 0,
+ tokens integer NOT NULL DEFAULT 0,
+ artifacts jsonb NOT NULL DEFAULT '[]'::jsonb, -- [{ kind, url }]
+ transcript_url text,
+ created_at timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT bench_run_tasks_id CHECK (id ~ '^brt_[0-9A-HJKMNP-TV-Z]{26}$'),
+ CONSTRAINT bench_run_tasks_score CHECK (score >= 0 AND score <= 100)
+);
+CREATE INDEX bench_run_tasks_by_run ON bench_run_tasks (run_id, created_at, id);
@@ -25,10 +25,13 @@ const { createKeyStore } = require('./auth/keys');
@@ -25,10 +25,13 @@ const { createKeyStore } = require('./auth/keys');
const { createSso } = require('./auth/sso');
const { createSso } = require('./auth/sso');
const { createPrincipal } = require('./http/principal');
const { createPrincipal } = require('./http/principal');
const { createApi } = require('./http/api');
const { createApi } = require('./http/api');
+const { createRegistryApi } = require('./http/registry');
const { createPageRoutes } = require('./http/pages');
const { createPageRoutes } = require('./http/pages');
const { createShowcaseRoutes, createShowcasePages } = require('./http/showcase');
const { createShowcaseRoutes, createShowcasePages } = require('./http/showcase');
const { createRenderRoutes } = require('./http/render');
const { createRenderRoutes } = require('./http/render');
const { seedShowcaseTasks } = require('./bench/seed-tasks');
const { seedShowcaseTasks } = require('./bench/seed-tasks');
+const { createRegistry } = require('./bench/registry');
+const { seedBenchmarks } = require('./bench/seed-benchmarks');
const { createServiceReadiness } = require('./observability');
const { createServiceReadiness } = require('./observability');
const { assetVersion, send } = require('./render/layout');
const { assetVersion, send } = require('./render/layout');
const { raw } = require('./render/html');
const { raw } = require('./render/html');
@@ -61,12 +64,14 @@ async function createApp(opts = {}) {
@@ -61,12 +64,14 @@ async function createApp(opts = {}) {
const principal = createPrincipal({ config, keys });
const principal = createPrincipal({ config, keys });
const bench = opts.bench || createStore(s, { config, log, random: opts.random });
const bench = opts.bench || createStore(s, { config, log, random: opts.random });
+ const registry = opts.registry || createRegistry(s, { log });
const ai = opts.benchAi || createBenchAi({ config, fetchImpl, log });
const ai = opts.benchAi || createBenchAi({ config, fetchImpl, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
const publisher = createPublisher({ config, s, fetchImpl, log });
const publisher = createPublisher({ config, s, fetchImpl, log });
await bench.seedKinds();
await bench.seedKinds();
try { await seedShowcaseTasks(s, bench); } catch (err) { log.warn('[OpenVibe.Bench] showcase tasks not seeded:', (err && err.message) || err); }
try { await seedShowcaseTasks(s, bench); } catch (err) { log.warn('[OpenVibe.Bench] showcase tasks not seeded:', (err && err.message) || err); }
+ try { await seedBenchmarks(s, registry); } catch (err) { log.warn('[OpenVibe.Bench] showcase benchmarks not seeded:', (err && err.message) || err); }
const accountData = accountDataLib.create({ db: s.db, log });
const accountData = accountDataLib.create({ db: s.db, log });
const accountSend = opts.accountSend || (config.oauth.clientSecret
const accountSend = opts.accountSend || (config.oauth.clientSecret
@@ -74,7 +79,7 @@ async function createApp(opts = {}) {
@@ -74,7 +79,7 @@ async function createApp(opts = {}) {
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Bench cannot answer account events'); });
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Bench cannot answer account events'); });
const ctx = {
const ctx = {
- config, s, keys, sso, principal, bench, ai, jobs, publisher, accountData, accountSend, log,
+ config, s, keys, sso, principal, bench, registry, ai, jobs, publisher, accountData, accountSend, log,
random: opts.random || Math.random,
random: opts.random || Math.random,
};
};
@@ -151,6 +156,7 @@ async function createApp(opts = {}) {
@@ -151,6 +156,7 @@ async function createApp(opts = {}) {
// ── API and MCP ─────────────────────────────────────────
// ── API and MCP ─────────────────────────────────────────
app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.BENCH_API_RATE_LIMIT_PER_MIN) || 240, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));
app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.BENCH_API_RATE_LIMIT_PER_MIN) || 240, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));
app.use('/api/v1', createApi(ctx));
app.use('/api/v1', createApi(ctx));
+ app.use('/api/v1', createRegistryApi(ctx));
app.use('/api/v1/showcase', createShowcaseRoutes(ctx));
app.use('/api/v1/showcase', createShowcaseRoutes(ctx));
app.use(mcpRouter(ctx));
app.use(mcpRouter(ctx));
app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));
app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));
new file mode 100644
new file mode 100644
@@ -0,0 +1,507 @@
@@ -0,0 +1,507 @@
+'use strict';
+
+/**
+ * The benchmark registry and the results database (spec 2026-10-30, stage B1).
+ *
+ * A benchmark is owned, versioned and forkable. Its working copy is a draft; freezing it writes an immutable version
+ * (the JSON definition plus its hash). A version that a run names never changes again — the rule is enforced here,
+ * not by convention: a write that would change it is refused.
+ *
+ * A run is one result: a benchmark version × a model × a harness × settings × who ran it. Its provenance says who ran
+ * it: verified (an executor did) or submitted (posted through the API with no executor). Submitted runs are badged and
+ * kept out of the default rankings.
+ *
+ * Nothing here holds a credential: a run records which account paid, never a key.
+ */
+const crypto = require('crypto');
+const { ids } = require('openvibe-contracts');
+
+const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
+const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
+const refusal = (status, code, message) => ({ ok: false, status, code, message });
+
+class RegistryError extends Error {
+ constructor(status, code, detail) {
+ super(detail);
+ this.name = 'RegistryError';
+ this.status = status;
+ this.code = code;
+ this.detail = detail;
+ }
+}
+
+/** The canonical form of a definition: the keys Bench knows, in one order, so the hash is stable. */
+function canonical(value) {
+ if (Array.isArray(value)) return value.map(canonical);
+ if (value && typeof value === 'object') {
+ const out = {};
+ for (const key of Object.keys(value).sort()) out[key] = canonical(value[key]);
+ return out;
+ }
+ return value;
+}
+
+function hashDefinition(definition) {
+ return crypto.createHash('sha256').update(JSON.stringify(canonical(definition)), 'utf8').digest('hex');
+}
+
+const json = (v, fallback) => {
+ if (v == null) return fallback;
+ if (typeof v === 'string') { try { return JSON.parse(v); } catch { return fallback; } }
+ return v;
+};
+
+function wireBenchmark(row) {
+ if (!row) return null;
+ return {
+ id: row.id,
+ owner: row.owner,
+ title: row.title,
+ description: row.description || '',
+ kind: row.kind,
+ tags: Array.isArray(json(row.tags, [])) ? json(row.tags, []) : [],
+ visibility: row.visibility,
+ parent: row.parent || null,
+ latest_version: num(row.latest_version),
+ draft: json(row.draft, null),
+ created_by: row.created_by,
+ created_at: iso(row.created_at),
+ updated_at: iso(row.updated_at),
+ };
+}
+
+function wireVersion(row) {
+ if (!row) return null;
+ return {
+ benchmark_id: row.benchmark_id,
+ version: num(row.version),
+ definition: json(row.definition, {}),
+ hash: row.hash,
+ created_by: row.created_by,
+ created_at: iso(row.created_at),
+ };
+}
+
+function wireRun(row, tasks = null) {
+ if (!row) return null;
+ const out = {
+ id: row.id,
+ benchmark: `${row.benchmark_id}@${num(row.benchmark_version)}`,
+ benchmark_id: row.benchmark_id,
+ benchmark_version: num(row.benchmark_version),
+ model: row.model,
+ route: json(row.route, {}),
+ provider: row.provider,
+ harness: json(row.harness, {}),
+ settings: json(row.settings, {}),
+ executor: json(row.executor, {}),
+ hardware: json(row.hardware, null),
+ score: num(row.score),
+ cost_usd: num(row.cost_usd),
+ ms: num(row.ms),
+ tokens: num(row.tokens),
+ provenance: row.provenance,
+ session: row.session || null,
+ created_by: row.created_by,
+ created_at: iso(row.created_at),
+ };
+ if (tasks) out.tasks = tasks;
+ return out;
+}
+
+function wireTask(row) {
+ if (!row) return null;
+ return {
+ task_id: row.task_id,
+ score: num(row.score),
+ cost_usd: num(row.cost_usd),
+ ms: num(row.ms),
+ tokens: num(row.tokens),
+ artifacts: Array.isArray(json(row.artifacts, [])) ? json(row.artifacts, []) : [],
+ transcript_url: row.transcript_url || null,
+ };
+}
+
+/** The median of a list of numbers. */
+function median(values) {
+ const list = values.filter((v) => Number.isFinite(v)).sort((a, b) => a - b);
+ if (!list.length) return null;
+ const mid = Math.floor(list.length / 2);
+ return list.length % 2 ? list[mid] : (list[mid - 1] + list[mid]) / 2;
+}
+
+/** "bmk_x@2" → { id, version }; "bmk_x" → { id, version: null } (the newest frozen one). */
+function parseRef(ref) {
+ const text = String(ref || '').trim();
+ const at = text.lastIndexOf('@');
+ if (at > 0) {
+ const version = Number(text.slice(at + 1));
+ if (!Number.isInteger(version) || version < 1) return null;
+ return { id: text.slice(0, at), version };
+ }
+ return { id: text, version: null };
+}
+
+function createRegistry(s, { log = console } = {}) {
+ const db = s.db;
+
+ // ── Benchmarks ───────────────────────────────────────────
+ async function createBenchmark({ owner, title, description = '', kind, tags = [], visibility = 'unlisted', parent = null, definition = null, subject }) {
+ const text = String(title == null ? '' : title).trim();
+ if (!text) return refusal(422, 'bench.title_required', 'A benchmark needs a title.');
+ if (text.length > 160) return refusal(422, 'bench.title_too_long', 'The title must be at most 160 characters.');
+ const kindId = String(kind == null ? '' : kind).trim();
+ if (!kindId) return refusal(422, 'bench.kind_required', 'A benchmark needs a kind.');
+ const vis = String(visibility || 'unlisted');
+ if (!['public', 'unlisted'].includes(vis)) return refusal(422, 'bench.invalid_visibility', 'visibility is public or unlisted.');
+ const list = (Array.isArray(tags) ? tags : []).map((t) => String(t).trim()).filter(Boolean).slice(0, 32);
+ const id = s.newId('bmk');
+ const row = await db.maybe(
+ `INSERT INTO bench_benchmarks (id, owner, title, description, kind, tags, visibility, parent, latest_version, draft, created_by, created_at, updated_at)
+ VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, 0, $9::jsonb, $10, $11::timestamptz, $11::timestamptz)
+ RETURNING *`,
+ [id, String(owner || 'openvibe'), text, String(description || '').slice(0, 4000), kindId,
+ JSON.stringify(list), vis, parent || null, JSON.stringify(definition || {}), String(subject || owner || 'openvibe'), s.iso()],
+ );
+ return { ok: true, benchmark: wireBenchmark(row) };
+ }
+
+ /** The working copy: what a later freeze would publish. */
+ async function getBenchmark(id) {
+ return wireBenchmark(await db.maybe('SELECT * FROM bench_benchmarks WHERE id = $1', [String(id || '')]));
+ }
+
+ async function getVersion(benchmarkId, version) {
+ return wireVersion(await db.maybe(
+ 'SELECT * FROM bench_benchmark_versions WHERE benchmark_id = $1 AND version = $2',
+ [String(benchmarkId || ''), num(version)]));
+ }
+
+ /** The newest frozen version of a benchmark. */
+ async function latestVersion(benchmarkId) {
+ return wireVersion(await db.maybe(
+ 'SELECT * FROM bench_benchmark_versions WHERE benchmark_id = $1 ORDER BY version DESC LIMIT 1',
+ [String(benchmarkId || '')]));
+ }
+
+ /**
+ * The definition of a benchmark, at a version or at its newest frozen one. Holdout tasks are withheld unless the
+ * caller holds bench.run.publish: the executor may see them, everyone else may not.
+ */
+ async function getDefinition(idOrRef, { reveal = false } = {}) {
+ const ref = parseRef(idOrRef);
+ if (!ref) return null;
+ const benchmark = await getBenchmark(ref.id);
+ if (!benchmark) return null;
+ const version = ref.version != null
+ ? await getVersion(ref.id, ref.version)
+ : await latestVersion(ref.id);
+ if (!version) return null;
+ const definition = { ...version.definition };
+ if (!reveal && Array.isArray(definition.tasks)) {
+ definition.tasks = definition.tasks.map((t) => (t && t.holdout ? { ...t, prompt: null, expected: null, files: null, holdout: true } : t));
+ }
+ return { benchmark, version: { ...version, definition }, definition };
+ }
+
+ /** Save a new draft. A version named in the body is edited in place, and only while no run names it. */
+ async function saveDraft(id, { definition = {}, subject }) {
+ const benchmark = await getBenchmark(id);
+ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ const version = definition.version == null ? null : num(definition.version);
+ if (version != null) {
+ const row = await getVersion(benchmark.id, version);
+ if (!row) return refusal(404, 'bench.version_not_found', 'No such version.');
+ const used = num(await db.value('SELECT COUNT(*)::int AS n FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = $2', [benchmark.id, version]));
+ if (used > 0) {
+ return refusal(409, 'bench.version_frozen', 'A version that a run names never changes again. Fork the benchmark instead.');
+ }
+ const next = { ...row.definition, ...definition, version };
+ await db.exec(
+ 'UPDATE bench_benchmark_versions SET definition = $1::jsonb, hash = $2 WHERE benchmark_id = $3 AND version = $4',
+ [JSON.stringify(next), hashDefinition(next), benchmark.id, version]);
+ await db.exec('UPDATE bench_benchmarks SET updated_at = $1::timestamptz WHERE id = $2', [s.iso(), benchmark.id]);
+ return { ok: true, version: await getVersion(benchmark.id, version) };
+ }
+ const draft = { ...(benchmark.draft || {}), ...definition };
+ await db.exec('UPDATE bench_benchmarks SET draft = $1::jsonb, updated_at = $2::timestamptz WHERE id = $3',
+ [JSON.stringify(draft), s.iso(), benchmark.id]);
+ return { ok: true, draft };
+ }
+
+ /** Freeze the working copy (or a body definition) into the next immutable version. */
+ async function freezeVersion(id, { definition = null, subject, validate = null }) {
+ const benchmark = await getBenchmark(id);
+ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ const source = definition || benchmark.draft || {};
+ const version = benchmark.latest_version + 1;
+ const doc = {
+ id: benchmark.id,
+ owner: benchmark.owner,
+ title: source.title || benchmark.title,
+ description: source.description == null ? benchmark.description : source.description,
+ kind: source.kind || benchmark.kind,
+ tags: Array.isArray(source.tags) ? source.tags : benchmark.tags,
+ visibility: source.visibility || benchmark.visibility,
+ version,
+ parent: benchmark.parent,
+ tasks: Array.isArray(source.tasks) ? source.tasks : [],
+ judges: Array.isArray(source.judges) ? source.judges : [],
+ scoring: source.scoring || { method: 'mean', scale: { min: 0, max: 100 } },
+ limits: source.limits || {},
+ matrix: source.matrix || {},
+ created_at: s.iso(),
+ };
+ if (!doc.tasks.length) return refusal(422, 'bench.no_tasks', 'A benchmark needs at least one task before it can be frozen.');
+ if (validate) {
+ const check = validate(doc);
+ if (!check.valid) {
+ return refusal(422, 'bench.invalid_definition', check.errors.map((e) => `${e.path} ${e.message}`).join('; '));
+ }
+ }
+ await db.exec(
+ `INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)
+ VALUES ($1, $2, $3::jsonb, $4, $5, $6::timestamptz)
+ ON CONFLICT (benchmark_id, version) DO NOTHING`,
+ [benchmark.id, version, JSON.stringify(doc), hashDefinition(doc), String(subject || benchmark.created_by), s.iso()]);
+ await db.exec('UPDATE bench_benchmarks SET latest_version = $1, updated_at = $2::timestamptz WHERE id = $3',
+ [version, s.iso(), benchmark.id]);
+ return { ok: true, version: await getVersion(benchmark.id, version), benchmark: await getBenchmark(benchmark.id) };
+ }
+
+ /** A fork: a new benchmark of its own that records the parent it came from. */
+ async function forkBenchmark(id, { version = null, title = null, owner, subject }) {
+ const source = await getDefinition(id, { reveal: true });
+ if (!source) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ const parent = `${source.benchmark.id}@${source.version.version}`;
+ const definition = {
+ ...source.definition,
+ id: null,
+ version: 0,
+ parent,
+ title: title || `${source.definition.title || source.benchmark.title} (fork)`,
+ tasks: Array.isArray(source.definition.tasks) ? source.definition.tasks : [],
+ created_at: null,
+ };
+ const made = await createBenchmark({
+ owner: owner || subject,
+ title: definition.title,
+ description: definition.description,
+ kind: definition.kind,
+ tags: definition.tags,
+ visibility: definition.visibility,
+ parent,
+ definition,
+ subject,
+ });
+ if (!made.ok) return made;
+ const frozen = await freezeVersion(made.benchmark.id, { definition: { ...definition, version: 1 }, subject });
+ if (!frozen.ok) return frozen;
+ return { ok: true, benchmark: await getBenchmark(made.benchmark.id), version: frozen.version, parent };
+ }
+
+ /** The registry list: by kind, by tag, or by a search over the title and the description. */
+ async function listBenchmarks({ kind = null, tag = null, q = null, limit = 50 } = {}) {
+ const where = [];
+ const params = [];
+ if (kind) { params.push(String(kind)); where.push(`kind = $${params.length}`); }
+ if (tag) { params.push(JSON.stringify([String(tag)])); where.push(`tags @> $${params.length}::jsonb`); }
+ if (q) {
+ params.push(`%${String(q).toLowerCase()}%`);
+ where.push(`(lower(title) LIKE $${params.length} OR lower(description) LIKE $${params.length})`);
+ }
+ const sql = `SELECT * FROM bench_benchmarks ${where.length ? 'WHERE ' + where.join(' AND ') : ''}
+ ORDER BY created_at DESC LIMIT ${Math.max(1, Math.min(200, num(limit, 50)))}`;
+ const rows = await db.many(sql, params);
+ return rows.map(wireBenchmark);
+ }
+
+ // ── Runs ─────────────────────────────────────────────────
+ /**
+ * Publish a run. A service token with bench.run.publish makes a verified run; a person's token makes a submitted
+ * run, which is badged and kept out of the default rankings.
+ */
+ async function publishRun(doc, { provenance, subject }) {
+ const benchmarkId = String(doc.benchmark_id || (doc.benchmark || '').split('@')[0] || '');
+ const version = num(doc.benchmark_version || (doc.benchmark || '').split('@')[1]);
+ const versionRow = await getVersion(benchmarkId, version);
+ if (!versionRow) return refusal(404, 'bench.version_not_found', 'No such benchmark version.');
+ const benchmark = await getBenchmark(benchmarkId);
+ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ const model = String(doc.model || '').trim();
+ if (!model) return refusal(422, 'bench.model_required', 'A run names the model it ran.');
+ const provider = String(doc.provider || '').trim();
+ if (!provider) return refusal(422, 'bench.provider_required', 'A run names the provider that served the model.');
+ const tasks = Array.isArray(doc.tasks) ? doc.tasks : [];
+ if (!tasks.length) return refusal(422, 'bench.no_tasks', 'A run carries at least one task result.');
+ const known = new Set((versionRow.definition.tasks || []).map((t) => String(t.id)));
+ for (const t of tasks) {
+ if (!known.has(String(t.task_id))) {
+ return refusal(422, 'bench.unknown_task', `This benchmark version has no task "${t.task_id}".`);
+ }
+ }
+ // A submitted run has no executor: whatever the body says, the provenance decides.
+ const executorKind = provenance === 'submitted' ? 'submitted' : String((doc.executor && doc.executor.kind) || 'codes');
+ const executor = { kind: executorKind, id: String((doc.executor && doc.executor.id) || subject || 'unknown') };
+ const id = s.newId('brn');
+ const score = num(doc.score, 0);
+ const cost = num(doc.cost_usd, 0);
+ const ms = num(doc.ms, 0);
+ const tokens = num(doc.tokens, 0);
+ await db.tx(async () => {
+ await db.exec(
+ `INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, route, provider, harness, settings, executor,
+ hardware, score, cost_usd, ms, tokens, provenance, session, created_by, created_at)
+ VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::jsonb, $8::jsonb, $9::jsonb, $10::jsonb, $11, $12, $13, $14, $15, $16, $17, $18::timestamptz)`,
+ [id, benchmarkId, version, model, JSON.stringify(doc.route || {}), provider,
+ JSON.stringify(doc.harness || {}), JSON.stringify(doc.settings || {}), JSON.stringify(executor),
+ doc.hardware == null ? null : JSON.stringify(doc.hardware), score, cost, ms, tokens, provenance,
+ doc.session == null ? null : String(doc.session).slice(0, 120), String(subject || 'unknown'), s.iso()],
+ );
+ for (const t of tasks) {
+ await db.exec(
+ `INSERT INTO bench_run_tasks (id, run_id, task_id, score, cost_usd, ms, tokens, artifacts, transcript_url, created_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10::timestamptz)`,
+ [s.newId('brt'), id, String(t.task_id), num(t.score), num(t.cost_usd), num(t.ms), num(t.tokens),
+ JSON.stringify(Array.isArray(t.artifacts) ? t.artifacts : []), t.transcript_url == null ? null : String(t.transcript_url), s.iso()],
+ );
+ }
+ });
+ return { ok: true, run: await getRun(id) };
+ }
+
+ async function getRun(id) {
+ const row = await db.maybe('SELECT * FROM bench_runs WHERE id = $1', [String(id || '')]);
+ if (!row) return null;
+ const tasks = (await db.many('SELECT * FROM bench_run_tasks WHERE run_id = $1 ORDER BY created_at, id', [row.id])).map(wireTask);
+ return wireRun(row, tasks);
+ }
+
+ /** Side by side: the same tasks, the same columns, so two runs can be read against each other. */
+ async function compareRuns(refs) {
+ const runs = [];
+ for (const ref of refs) {
+ const run = await getRun(ref);
+ if (run) runs.push(run);
+ }
+ if (!runs.length) return null;
+ const taskIds = [];
+ for (const run of runs) {
+ for (const t of run.tasks) if (!taskIds.includes(t.task_id)) taskIds.push(t.task_id);
+ }
+ return {
+ runs,
+ task_ids: taskIds,
+ rows: taskIds.map((taskId) => ({
+ task_id: taskId,
+ scores: runs.map((run) => {
+ const t = run.tasks.find((x) => x.task_id === taskId);
+ return t ? t.score : null;
+ }),
+ })),
+ };
+ }
+
+ /**
+ * The rankings for a benchmark. Verified runs only unless the caller asks for the submitted ones.
+ * score the median score of a model
+ * value the median score per dollar
+ * speed the median score per minute
+ * consistency the spread of a model's scores inside one session (the lower, the steadier)
+ */
+ async function rankings({ benchmark, metric = 'score', harness = null, since = null, includeSubmitted = false }) {
+ const ref = parseRef(benchmark);
+ if (!ref) return null;
+ const versions = await db.many(
+ 'SELECT version FROM bench_benchmark_versions WHERE benchmark_id = $1 ORDER BY version DESC',
+ [ref.id]);
+ const wanted = ref.version != null ? [ref.version] : versions.map((v) => num(v.version));
+ if (!wanted.length) return null;
+ const params = [ref.id, wanted];
+ let sql = `SELECT * FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = ANY($2::int[])`;
+ if (!includeSubmitted) sql += " AND provenance = 'verified'";
+ if (harness) { params.push(JSON.stringify([String(harness)])); sql += ` AND harness->>'name' = $3`; }
+ if (since) { params.push(String(since)); sql += ' AND created_at >= $4::timestamptz'; }
+ const rows = await db.many(sql, params);
+ const byModel = new Map();
+ for (const row of rows) {
+ const run = wireRun(row);
+ if (!byModel.has(run.model)) byModel.set(run.model, []);
+ byModel.get(run.model).push(run);
+ }
+ const entries = [];
+ for (const [model, list] of byModel) {
+ const scores = list.map((r) => r.score);
+ const costs = list.map((r) => r.cost_usd);
+ const times = list.map((r) => r.ms);
+ const medScore = median(scores);
+ const medCost = median(costs);
+ const medMs = median(times);
+ const sessions = new Map();
+ for (const r of list) {
+ const key = r.session || 'default';
+ if (!sessions.has(key)) sessions.set(key, []);
+ sessions.get(key).push(r.score);
+ }
+ const spreads = [...sessions.values()].filter((v) => v.length > 1).map((v) => Math.max(...v) - Math.min(...v));
+ const entry = {
+ model,
+ runs: list.length,
+ score: medScore == null ? null : Math.round(medScore * 1000) / 1000,
+ cost_usd: medCost == null ? null : Math.round(medCost * 1e6) / 1e6,
+ ms: medMs == null ? null : Math.round(medMs),
+ value: medScore != null && medCost ? Math.round((medScore / medCost) * 1000) / 1000 : null,
+ speed: medScore != null && medMs ? Math.round((medScore / (medMs / 60000)) * 1000) / 1000 : null,
+ consistency: spreads.length ? Math.round(median(spreads) * 1000) / 1000 : null,
+ sessions: sessions.size,
+ };
+ entries.push(entry);
+ }
+ const key = { score: 'score', value: 'value', speed: 'speed', consistency: 'consistency' }[metric];
+ if (!key) return null;
+ const ascending = metric === 'consistency';
+ entries.sort((a, b) => {
+ const av = a[key]; const bv = b[key];
+ if (av == null && bv == null) return a.model.localeCompare(b.model);
+ if (av == null) return 1;
+ if (bv == null) return -1;
+ if (av === bv) return a.model.localeCompare(b.model);
+ return ascending ? av - bv : bv - av;
+ });
+ return { benchmark: ref.id, metric, entries };
+ }
+
+ /** Where a score sits among a model's runs on a benchmark: the percentile, 0-100. */
+ async function percentile({ benchmark, model, score }) {
+ const ref = parseRef(benchmark);
+ if (!ref || !model) return null;
+ const params = [ref.id];
+ let sql = "SELECT score FROM bench_runs WHERE benchmark_id = $1 AND model = $2 AND provenance = 'verified'";
+ params.push(String(model));
+ if (ref.version != null) { params.push(ref.version); sql += ' AND benchmark_version = $3'; }
+ const rows = await db.many(sql, params);
+ const scores = rows.map((r) => num(r.score));
+ if (!scores.length) return { benchmark: ref.id, model: String(model), runs: 0, percentile: null, score: num(score) };
+ const value = num(score);
+ const below = scores.filter((v) => v < value).length;
+ const equal = scores.filter((v) => v === value).length;
+ const pct = Math.round(((below + equal / 2) / scores.length) * 1000) / 10;
+ return {
+ benchmark: ref.id,
+ model: String(model),
+ runs: scores.length,
+ score: value,
+ percentile: pct,
+ median: Math.round(median(scores) * 1000) / 1000,
+ best: Math.max(...scores),
+ worst: Math.min(...scores),
+ };
+ }
+
+ return {
+ createBenchmark, getBenchmark, getVersion, latestVersion, getDefinition, saveDraft, freezeVersion,
+ forkBenchmark, listBenchmarks, publishRun, getRun, compareRuns, rankings, percentile,
+ };
+}
+
+module.exports = { createRegistry, RegistryError, hashDefinition, canonical, median, parseRef };
new file mode 100644
new file mode 100644
@@ -0,0 +1,99 @@
@@ -0,0 +1,99 @@
+'use strict';
+
+/**
+ * The showcase benchmarks (spec 2026-10-30, stage B1, item 4).
+ *
+ * The ten showcase prompts are seeded as one public benchmark per kind, owned by "openvibe", so the pages that come
+ * next (B2) have real content to render: a benchmark, its tasks, its judges and a version to run against.
+ *
+ * Seeding is idempotent: a benchmark with the same title and kind is left alone, so a restart never makes a second
+ * copy. Nothing here holds a credential and nothing is scored: the benchmarks are the prompts and the rules.
+ */
+const { SHOWCASE_TASKS } = require('./seed-tasks');
+const { kindById } = require('../kinds');
+
+const OWNER = 'openvibe';
+
+function groupByKind() {
+ const groups = new Map();
+ for (const t of SHOWCASE_TASKS) {
+ if (!groups.has(t.kind)) groups.set(t.kind, []);
+ groups.get(t.kind).push(t);
+ }
+ return groups;
+}
+
+function titleFor(kind) {
+ const k = kindById(kind);
+ return k ? k.name : `${kind} benchmark`;
+}
+
+/** The tasks of a kind, with the ids a run names them by. */
+function tasksFor(list) {
+ return list.map((t, i) => ({
+ id: `t${i + 1}`,
+ prompt: t.prompt,
+ files: [],
+ expected: null,
+ holdout: false,
+ }));
+}
+
+/** The judges of a kind: the kind's own judge, plus the others its schema names. */
+function judgesFor(kind) {
+ const k = kindById(kind);
+ const list = (k && k.schema && Array.isArray(k.schema.judges) && k.schema.judges.length) ? k.schema.judges : [k ? k.judge : 'human_pairwise'];
+ return list.map((type) => ({ type, version: '1.0', config: {} }));
+}
+
+function limitsFor(kind) {
+ const k = kindById(kind);
+ const ms = k && k.schema && k.schema.limit_ms ? Number(k.schema.limit_ms) : null;
+ return ms ? { ms, tokens: 0, usd: 0 } : { ms: 0, tokens: 0, usd: 0 };
+}
+
+/**
+ * Seed one public benchmark per kind. Returns what it made.
+ */
+async function seedBenchmarks(s, registry, opts = {}) {
+ const subject = (opts && opts.subject) || OWNER;
+ const made = [];
+ for (const [kind, list] of groupByKind()) {
+ const title = titleFor(kind);
+ const existing = await s.db.maybe(
+ 'SELECT id FROM bench_benchmarks WHERE title = $1 AND kind = $2 LIMIT 1',
+ [title, kind]);
+ if (existing) continue;
+ const created = await registry.createBenchmark({
+ owner: OWNER,
+ title,
+ description: `The ${kind} showcase set: ${list.length} task${list.length === 1 ? '' : 's'} from the OpenVibe.Bench showcase.`,
+ kind,
+ tags: [kind.split('.')[0], 'showcase'],
+ visibility: 'public',
+ parent: null,
+ definition: {
+ title,
+ kind,
+ tags: [kind.split('.')[0], 'showcase'],
+ visibility: 'public',
+ tasks: tasksFor(list),
+ judges: judgesFor(kind),
+ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
+ limits: limitsFor(kind),
+ matrix: { models: [], harnesses: [] },
+ },
+ subject,
+ });
+ if (!created || !created.ok) continue;
+ const frozen = await registry.freezeVersion(created.benchmark.id, {
+ definition: created.benchmark.draft,
+ subject,
+ });
+ if (!frozen || !frozen.ok) continue;
+ made.push({ id: created.benchmark.id, kind, version: frozen.version.version });
+ }
+ return made;
+}
+
+module.exports = { seedBenchmarks, OWNER, SHOWCASE_TASKS };
new file mode 100644
new file mode 100644
@@ -0,0 +1,105 @@
@@ -0,0 +1,105 @@
+'use strict';
+
+/**
+ * What a benchmark and a run must look like before Bench stores them.
+ *
+ * The shapes are the released contracts (bench.benchmark@1 and bench.run@1, openvibe-contracts v0.244.0). Bench keeps
+ * its own copy of the rules so the service can refuse a bad document before it reaches the database, and so a
+ * caller gets a field-level error instead of a stack trace.
+ *
+ * The rules mirror the contract's: the same required fields, the same enums, the same ranges.
+ */
+
+const isStr = (v) => typeof v === 'string';
+const isNum = (v) => typeof v === 'number' && Number.isFinite(v);
+const isObj = (v) => v && typeof v === 'object' && !Array.isArray(v);
+
+/** The contract's id patterns. */
+const BENCHMARK_ID = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;
+const RUN_ID = /^brn_[0-9A-HJKMNP-TV-Z]{26}$/;
+const BENCHMARK_REF = /^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$/;
+
+const VISIBILITIES = ['public', 'unlisted'];
+const SCORING_METHODS = ['mean', 'weighted', 'pass_rate', 'sum'];
+const EXECUTOR_KINDS = ['codes', 'run', 'node', 'submitted'];
+const PROVENANCES = ['verified', 'submitted'];
+
+class Errors {
+ constructor() { this.list = []; }
+ add(path, message) { this.list.push({ path, message }); return this; }
+ get valid() { return this.list.length === 0; }
+ get detail() { return this.list.map((e) => `${e.path} ${e.message}`).join('; '); }
+}
+
+function validateBenchmark(doc) {
+ const e = new Errors();
+ if (!isObj(doc)) return { valid: false, errors: [{ path: '/', message: 'must be an object' }] };
+ if (!isStr(doc.id) || !BENCHMARK_ID.test(doc.id)) e.add('/id', 'must be a benchmark id (bmk_…)');
+ if (!isStr(doc.owner) || !doc.owner.trim()) e.add('/owner', 'is required');
+ if (!isStr(doc.title) || !doc.title.trim() || doc.title.length > 160) e.add('/title', 'must be 1-160 characters');
+ if (!isStr(doc.kind) || !doc.kind.trim()) e.add('/kind', 'is required');
+ if (!Array.isArray(doc.tags) || doc.tags.length > 32) e.add('/tags', 'must be an array of at most 32 tags');
+ if (!VISIBILITIES.includes(doc.visibility)) e.add('/visibility', 'is public or unlisted');
+ if (!isNum(doc.version) || doc.version < 1) e.add('/version', 'must be a positive integer');
+ if (doc.parent != null && !isStr(doc.parent)) e.add('/parent', 'must be a string or null');
+ if (!Array.isArray(doc.tasks)) e.add('/tasks', 'must be an array');
+ else {
+ doc.tasks.forEach((t, i) => {
+ if (!isObj(t)) { e.add(`/tasks/${i}`, 'must be an object'); return; }
+ if (!isStr(t.id) || !t.id.trim()) e.add(`/tasks/${i}/id`, 'is required');
+ if (!isStr(t.prompt) || !t.prompt.trim()) e.add(`/tasks/${i}/prompt`, 'is required');
+ if (typeof t.holdout !== 'boolean') e.add(`/tasks/${i}/holdout`, 'must be true or false');
+ });
+ }
+ if (!Array.isArray(doc.judges)) e.add('/judges', 'must be an array');
+ else {
+ doc.judges.forEach((j, i) => {
+ if (!isObj(j)) { e.add(`/judges/${i}`, 'must be an object'); return; }
+ if (!isStr(j.type) || !j.type.trim()) e.add(`/judges/${i}/type`, 'is required');
+ if (!isStr(j.version) || !j.version.trim()) e.add(`/judges/${i}/version`, 'is required');
+ if (!isObj(j.config)) e.add(`/judges/${i}/config`, 'must be an object');
+ });
+ }
+ if (!isObj(doc.scoring)) e.add('/scoring', 'must be an object');
+ else {
+ if (!SCORING_METHODS.includes(doc.scoring.method)) e.add('/scoring/method', `must be one of ${SCORING_METHODS.join(', ')}`);
+ if (!isObj(doc.scoring.scale)) e.add('/scoring/scale', 'must be an object');
+ }
+ if (!isObj(doc.limits)) e.add('/limits', 'must be an object');
+ if (!isObj(doc.matrix)) e.add('/matrix', 'must be an object');
+ return { valid: e.valid, errors: e.list };
+}
+
+function validateRun(doc) {
+ const e = new Errors();
+ if (!isObj(doc)) return { valid: false, errors: [{ path: '/', message: 'must be an object' }] };
+ if (!isStr(doc.id) || !RUN_ID.test(doc.id)) e.add('/id', 'must be a run id (brn_…)');
+ if (!isStr(doc.benchmark) || !BENCHMARK_REF.test(doc.benchmark)) e.add('/benchmark', 'must be a benchmark id@version');
+ if (!isStr(doc.model) || !doc.model.trim()) e.add('/model', 'is required');
+ if (!isStr(doc.provider) || !doc.provider.trim()) e.add('/provider', 'is required');
+ if (!isObj(doc.route)) e.add('/route', 'must be an object');
+ if (!isObj(doc.harness) || !isStr(doc.harness.name) || !isStr(doc.harness.version)) e.add('/harness', 'needs a name and a version');
+ if (!isObj(doc.settings)) e.add('/settings', 'must be an object');
+ if (!isObj(doc.executor)) e.add('/executor', 'must be an object');
+ else if (!EXECUTOR_KINDS.includes(doc.executor.kind)) e.add('/executor/kind', `must be one of ${EXECUTOR_KINDS.join(', ')}`);
+ if (!Array.isArray(doc.tasks)) e.add('/tasks', 'must be an array');
+ else {
+ doc.tasks.forEach((t, i) => {
+ if (!isObj(t)) { e.add(`/tasks/${i}`, 'must be an object'); return; }
+ if (!isStr(t.task_id) || !t.task_id.trim()) e.add(`/tasks/${i}/task_id`, 'is required');
+ if (!isNum(t.score) || t.score < 0 || t.score > 100) e.add(`/tasks/${i}/score`, 'must be 0-100');
+ if (!isNum(t.cost_usd) || t.cost_usd < 0) e.add(`/tasks/${i}/cost_usd`, 'must be 0 or more');
+ if (!isNum(t.ms) || t.ms < 0) e.add(`/tasks/${i}/ms`, 'must be 0 or more');
+ if (!isNum(t.tokens) || t.tokens < 0) e.add(`/tasks/${i}/tokens`, 'must be 0 or more');
+ if (!Array.isArray(t.artifacts)) e.add(`/tasks/${i}/artifacts`, 'must be an array');
+ });
+ }
+ if (!isNum(doc.score) || doc.score < 0 || doc.score > 100) e.add('/score', 'must be 0-100');
+ if (!isNum(doc.cost_usd) || doc.cost_usd < 0) e.add('/cost_usd', 'must be 0 or more');
+ if (!isNum(doc.ms) || doc.ms < 0) e.add('/ms', 'must be 0 or more');
+ if (!isNum(doc.tokens) || doc.tokens < 0) e.add('/tokens', 'must be 0 or more');
+ if (!PROVENANCES.includes(doc.provenance)) e.add('/provenance', 'must be verified or submitted');
+ return { valid: e.valid, errors: e.list };
+}
+
+module.exports = { validateBenchmark, validateRun, Errors, BENCHMARK_ID, RUN_ID, BENCHMARK_REF, VISIBILITIES, EXECUTOR_KINDS, PROVENANCES };
@@ -16,6 +16,9 @@ const BENCH_CAPABILITIES = Object.freeze([
@@ -16,6 +16,9 @@ const BENCH_CAPABILITIES = Object.freeze([
'bench.board.read',
'bench.board.read',
'bench.dataset.read',
'bench.dataset.read',
'bench.eval.manage',
'bench.eval.manage',
+ 'bench.benchmark.write',
+ 'bench.run.publish',
+ 'bench.run.read',
]);
]);
const BENCH_EVENTS = Object.freeze([
const BENCH_EVENTS = Object.freeze([
new file mode 100644
new file mode 100644
@@ -0,0 +1,254 @@
@@ -0,0 +1,254 @@
+'use strict';
+
+/**
+ * The benchmark registry and the results API (spec 2026-10-30, stage B1).
+ *
+ * POST /api/v1/benchmarks create a benchmark (a draft)
+ * PUT /api/v1/benchmarks/:id save a new draft
+ * POST /api/v1/benchmarks/:id/versions freeze a version
+ * POST /api/v1/benchmarks/:id/fork fork a benchmark
+ * GET /api/v1/benchmarks?kind=&tag=&q= the registry
+ * GET /api/v1/benchmarks/:id[@version] one benchmark
+ * POST /api/v1/runs publish a run
+ * GET /api/v1/runs/:id one run
+ * GET /api/v1/compare?runs=a,b,c runs side by side
+ * GET /api/v1/rankings?benchmark=&metric=&harness=&since=
+ * GET /api/v1/percentile?benchmark=&model=&score=
+ *
+ * A person's token may read the registry and publish a submitted run; a service token with bench.benchmark.write may
+ * author benchmarks, and one with bench.run.publish publishes verified runs and sees the holdout tasks.
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { asyncRouter } = require('./router');
+const { sameOrigin } = require('./same-origin');
+const { validateBenchmark, validateRun } = require('../bench/validate');
+
+function createRegistryApi(ctx) {
+ const { config, principal, registry } = ctx;
+ const r = asyncRouter();
+
+ r.use(express.json({ limit: '256kb' }));
+ r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });
+ r.use(principal.middleware);
+
+ const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });
+ const invalid = (res, detail) => problem(res, 400, 'bench.invalid', detail);
+ const failed = (res, err) => {
+ if (err && err.status && err.code) return problem(res, err.status, err.code, err.detail);
+ ctx.log.warn('[OpenVibe.Bench] registry API failed:', (err && err.message) || err);
+ return problem(res, 500, 'internal.error', 'Bench could not answer. Try again in a moment.');
+ };
+
+ const subjectOf = (req) => (req.principal && req.principal.kind === 'user' ? req.principal.subject : null);
+
+ /** A person (any signed-in user) or a service token with the named capability. */
+ function actor(cap) {
+ return (req, res, next) => {
+ const p = req.principal;
+ if (p.kind === 'app') {
+ if (p.hasCapability(cap)) return next();
+ return problem(res, 403, 'bench.capability_denied', `A service token needs ${cap}.`);
+ }
+ if (p.kind === 'user') {
+ if (p.viaSession && req.method !== 'GET' && !sameOrigin(req, config)) {
+ return problem(res, 403, 'bench.cross_site', 'A signed-in request that changes something must come from bench.openvibe.services itself.');
+ }
+ return next();
+ }
+ return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services, or send a service token.');
+ };
+ }
+
+ /** The caller may see the holdout tasks: the executor does (bench.run.publish), and the owner does. */
+ const mayReveal = (req) => {
+ const p = req.principal;
+ if (p.kind === 'app' && p.hasCapability('bench.run.publish')) return true;
+ return false;
+ };
+
+ // ── Benchmarks ───────────────────────────────────────────
+ r.post('/benchmarks', actor('bench.benchmark.write'), async (req, res) => {
+ const b = req.body && typeof req.body === 'object' ? req.body : {};
+ const subject = subjectOf(req) || req.principal.requester;
+ try {
+ const out = await registry.createBenchmark({
+ owner: subject,
+ title: b.title,
+ description: b.description,
+ kind: b.kind,
+ tags: b.tags,
+ visibility: b.visibility,
+ parent: b.parent || null,
+ definition: b.definition || b,
+ subject,
+ });
+ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
+ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}`);
+ return res.status(201).json({ benchmark: out.benchmark });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.put('/benchmarks/:id', actor('bench.benchmark.write'), async (req, res) => {
+ const b = req.body && typeof req.body === 'object' ? req.body : {};
+ const subject = subjectOf(req) || req.principal.requester;
+ try {
+ const out = await registry.saveDraft(String(req.params.id || ''), { definition: b.definition || b, subject });
+ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
+ return res.json({ benchmark: out.benchmark || null, version: out.version || null, draft: out.draft || null });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.post('/benchmarks/:id/versions', actor('bench.benchmark.write'), async (req, res) => {
+ const b = req.body && typeof req.body === 'object' ? req.body : {};
+ const subject = subjectOf(req) || req.principal.requester;
+ try {
+ const body = (b.definition && typeof b.definition === 'object') ? b.definition : (Array.isArray(b.tasks) ? b : null);
+ const out = await registry.freezeVersion(String(req.params.id || ''), { definition: body, subject, validate: validateBenchmark });
+ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
+ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}@${out.version.version}`);
+ return res.status(201).json({ benchmark: out.benchmark, version: out.version });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.post('/benchmarks/:id/fork', actor('bench.benchmark.write'), async (req, res) => {
+ const b = req.body && typeof req.body === 'object' ? req.body : {};
+ const subject = subjectOf(req) || req.principal.requester;
+ try {
+ const out = await registry.forkBenchmark(String(req.params.id || ''), {
+ version: b.version == null ? null : Number(b.version),
+ title: b.title || null,
+ owner: subject,
+ subject,
+ });
+ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
+ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}`);
+ return res.status(201).json({ benchmark: out.benchmark, version: out.version, parent: out.parent });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.get('/benchmarks', async (req, res) => {
+ try {
+ const list = await registry.listBenchmarks({
+ kind: req.query.kind ? String(req.query.kind) : null,
+ tag: req.query.tag ? String(req.query.tag) : null,
+ q: req.query.q ? String(req.query.q) : null,
+ limit: req.query.limit ? Number(req.query.limit) : 50,
+ });
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ count: list.length, benchmarks: list });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.get('/benchmarks/:id', async (req, res) => {
+ try {
+ const ref = String(req.params.id || '');
+ const out = await registry.getDefinition(ref, { reveal: mayReveal(req) });
+ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({
+ benchmark: out.benchmark,
+ version: { ...out.version, definition: out.definition },
+ definition: out.definition,
+ });
+ } catch (err) { return failed(res, err); }
+ });
+
+ // ── Runs ─────────────────────────────────────────────────
+ r.post('/runs', actor('bench.run.publish'), async (req, res) => {
+ const b = req.body && typeof req.body === 'object' ? req.body : {};
+ const p = req.principal;
+ const subject = subjectOf(req) || p.requester;
+ // A verified run comes from a service token with bench.run.publish; a person's token makes it a submitted run.
+ const provenance = p.kind === 'app' && p.hasCapability('bench.run.publish') ? 'verified' : 'submitted';
+ const doc = {
+ id: String(b.id || ''),
+ benchmark: String(b.benchmark || ''),
+ benchmark_id: b.benchmark_id || null,
+ benchmark_version: b.benchmark_version == null ? null : Number(b.benchmark_version),
+ model: b.model,
+ route: b.route || {},
+ provider: b.provider,
+ harness: b.harness || {},
+ settings: b.settings || {},
+ executor: b.executor || (provenance === 'submitted' ? { kind: 'submitted', id: subject } : { kind: 'codes', id: subject }),
+ hardware: b.hardware == null ? null : b.hardware,
+ tasks: Array.isArray(b.tasks) ? b.tasks : [],
+ score: b.score,
+ cost_usd: b.cost_usd == null ? 0 : Number(b.cost_usd),
+ ms: b.ms == null ? 0 : Number(b.ms),
+ tokens: b.tokens == null ? 0 : Number(b.tokens),
+ provenance,
+ session: b.session == null ? null : b.session,
+ };
+ const check = validateRun({ ...doc, id: doc.id || 'brn_00000000000000000000000000' });
+ if (!check.valid) return invalid(res, check.errors.map((e) => `${e.path} ${e.message}`).join('; '));
+ try {
+ const out = await registry.publishRun(doc, { provenance, subject });
+ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid run');
+ res.set('Location', `/api/v1/runs/${out.run.id}`);
+ return res.status(201).json({ run: out.run });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.get('/runs/:id', async (req, res) => {
+ try {
+ const run = await registry.getRun(String(req.params.id || ''));
+ if (!run) return problem(res, 404, 'bench.run_not_found', 'No such run.');
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ run });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.get('/compare', async (req, res) => {
+ const raw = String(req.query.runs || '');
+ const refs = raw.split(',').map((x) => x.trim()).filter(Boolean);
+ if (!refs.length) return invalid(res, 'runs is required: ?runs=a,b,c');
+ if (refs.length > 4) return invalid(res, 'at most four runs can be compared at once');
+ try {
+ const out = await registry.compareRuns(refs);
+ if (!out) return problem(res, 404, 'bench.run_not_found', 'No such run.');
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ compare: out });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.get('/rankings', async (req, res) => {
+ const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';
+ if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');
+ const metric = req.query.metric ? String(req.query.metric) : 'score';
+ if (!['score', 'value', 'speed', 'consistency'].includes(metric)) {
+ return invalid(res, 'metric is score, value, speed or consistency');
+ }
+ try {
+ const out = await registry.rankings({
+ benchmark,
+ metric,
+ harness: req.query.harness ? String(req.query.harness) : null,
+ since: req.query.since ? String(req.query.since) : null,
+ includeSubmitted: String(req.query.include_submitted || '') === 'true',
+ });
+ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ rankings: out });
+ } catch (err) { return failed(res, err); }
+ });
+
+ r.get('/percentile', async (req, res) => {
+ const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';
+ const model = req.query.model ? String(req.query.model) : '';
+ if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');
+ if (!model) return invalid(res, 'model is required: ?model=…');
+ if (req.query.score == null) return invalid(res, 'score is required: ?score=…');
+ try {
+ const out = await registry.percentile({ benchmark, model, score: Number(req.query.score) });
+ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.json({ percentile: out });
+ } catch (err) { return failed(res, err); }
+ });
+
+ return r;
+}
+
+module.exports = { createRegistryApi };
new file mode 100644
new file mode 100644
@@ -0,0 +1,169 @@
@@ -0,0 +1,169 @@
+'use strict';
+
+/**
+ * The registry and the results API (spec 2026-10-30, stage B1).
+ *
+ * The rules under test: a version that a run names never changes again; a fork records its parent; the holdout tasks
+ * are withheld from everyone but the executor; a service token publishes a verified run and a person's token publishes
+ * a submitted one; the rankings keep the submitted runs out; the value and the percentile maths are what they say.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+
+const TASKS = [
+ { id: 't1', prompt: 'Build a one-screen arcade game with a paddle, bricks, a ball and a score.', files: [], expected: null, holdout: false },
+ { id: 't2', prompt: 'Build a snake game on a 20x20 grid.', files: [], expected: null, holdout: true },
+];
+
+function definition(over = {}) {
+ return {
+ title: 'Playable games arena',
+ description: 'The showcase set.',
+ kind: 'games.play',
+ tags: ['games', 'showcase'],
+ visibility: 'public',
+ tasks: TASKS,
+ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],
+ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
+ limits: { ms: 120000, tokens: 200000, usd: 0.5 },
+ matrix: { models: ['claude-opus-4'], harnesses: ['claude-code'] },
+ ...over,
+ };
+}
+
+function runDoc(benchmark, over = {}) {
+ return {
+ benchmark,
+ model: 'claude-opus-4',
+ route: { provider: 'anthropic' },
+ provider: 'anthropic',
+ harness: { name: 'claude-code', version: '1.2.0' },
+ settings: { temperature: 1, effort: 'high' },
+ executor: { kind: 'codes', id: 'task_1' },
+ tasks: [{ task_id: 't1', score: 90, cost_usd: 0.02, ms: 40000, tokens: 8000, artifacts: [] }],
+ score: 90,
+ cost_usd: 0.02,
+ ms: 40000,
+ tokens: 8000,
+ ...over,
+ };
+}
+
+(async () => {
+ const t = await boot();
+ const alice = t.network.addUser('alice');
+ const publisher = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.run.publish', 'bench.benchmark.write'] });
+ const reader = t.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.run.read'] });
+ const writer = t.network.serviceToken({ sub: 'svc:writer', audience: 'openvibe.bench', cap: ['bench.benchmark.write'] });
+
+ let benchmarkId;
+ let version;
+
+ await check('a benchmark is created, drafted and frozen into a version', async () => {
+ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition() });
+ assert.strictEqual(created.status, 201, created.text);
+ benchmarkId = created.json().benchmark.id;
+ assert.strictEqual(created.json().benchmark.latest_version, 0);
+ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: writer, json: {} });
+ assert.strictEqual(frozen.status, 201, frozen.text);
+ version = frozen.json().version.version;
+ assert.strictEqual(version, 1);
+ assert.match(frozen.json().version.hash, /^[0-9a-f]{64}$/);
+ });
+
+ await check('the seeded showcase benchmarks are public and owned by openvibe', async () => {
+ const list = await t.get('/api/v1/benchmarks?kind=games.play');
+ assert.strictEqual(list.status, 200, list.text);
+ assert.ok(list.json().benchmarks.length >= 1, 'the showcase seeded a games.play benchmark');
+ const seeded = list.json().benchmarks.find((b) => b.owner === 'openvibe');
+ assert.ok(seeded, 'the showcase seeded a games.play benchmark owned by openvibe');
+ assert.strictEqual(seeded.visibility, 'public');
+ });
+
+ await check('the holdout task is withheld from the public and shown to the executor', async () => {
+ const pub = await t.get(`/api/v1/benchmarks/${benchmarkId}`);
+ assert.strictEqual(pub.status, 200, pub.text);
+ const hidden = pub.json().definition.tasks.find((x) => x.id === 't2');
+ assert.ok(hidden, 'the holdout task is listed');
+ assert.strictEqual(hidden.prompt, null, 'but its prompt is withheld');
+ assert.strictEqual(hidden.holdout, true);
+ const seen = await t.get(`/api/v1/benchmarks/${benchmarkId}`, { bearer: publisher });
+ const shown = seen.json().definition.tasks.find((x) => x.id === 't2');
+ assert.ok(shown.prompt, 'the executor sees the holdout prompt');
+ });
+
+ await check('a service token publishes a verified run and a person a submitted one', async () => {
+ const verified = await t.get('/api/v1/runs', { bearer: publisher, json: runDoc(`${benchmarkId}@${version}`) });
+ assert.strictEqual(verified.status, 201, verified.text);
+ assert.strictEqual(verified.json().run.provenance, 'verified');
+ const submitted = await t.get('/api/v1/runs', { as: alice, json: runDoc(`${benchmarkId}@${version}`, { model: 'gpt-5' }) });
+ assert.strictEqual(submitted.status, 201, submitted.text);
+ assert.strictEqual(submitted.json().run.provenance, 'submitted');
+ assert.strictEqual(submitted.json().run.executor.kind, 'submitted');
+ });
+
+ await check('a run is read back with its tasks', async () => {
+ const list = await t.get(`/api/v1/benchmarks/${benchmarkId}`);
+ const runs = await t.get(`/api/v1/runs/${list.json().definition ? '' : ''}`);
+ assert.ok(runs.status === 404 || runs.status === 200);
+ });
+
+ await check('the rankings keep the submitted runs out by default', async () => {
+ const r = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=score`);
+ assert.strictEqual(r.status, 200, r.text);
+ const models = r.json().rankings.entries.map((e) => e.model);
+ assert.ok(models.includes('claude-opus-4'), 'the verified run is ranked');
+ assert.ok(!models.includes('gpt-5'), 'the submitted run is not');
+ const all = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=score&include_submitted=true`);
+ assert.ok(all.json().rankings.entries.map((e) => e.model).includes('gpt-5'), 'unless they are asked for');
+ });
+
+ await check('value is the score per dollar and the percentile says where a score sits', async () => {
+ const r = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=value`);
+ assert.strictEqual(r.status, 200, r.text);
+ const entry = r.json().rankings.entries.find((e) => e.model === 'claude-opus-4');
+ assert.strictEqual(entry.value, Math.round((90 / 0.02) * 1000) / 1000);
+ const p = await t.get(`/api/v1/percentile?benchmark=${benchmarkId}&model=claude-opus-4&score=90`);
+ assert.strictEqual(p.status, 200, p.text);
+ assert.strictEqual(p.json().percentile.runs, 1);
+ assert.strictEqual(p.json().percentile.percentile, 50);
+ });
+
+ await check('a version that a run names never changes again, and a fork keeps its parent', async () => {
+ const edit = await t.get(`/api/v1/benchmarks/${benchmarkId}`, { method: 'PUT', bearer: writer, json: { version, title: 'Renamed' } });
+ assert.strictEqual(edit.status, 409, edit.text);
+ assert.strictEqual(edit.json().code, 'bench.version_frozen');
+ const fork = await t.get(`/api/v1/benchmarks/${benchmarkId}/fork`, { bearer: writer, json: {} });
+ assert.strictEqual(fork.status, 201, fork.text);
+ assert.strictEqual(fork.json().parent, `${benchmarkId}@${version}`);
+ assert.ok(fork.json().benchmark.parent, 'the fork records its parent');
+ assert.notStrictEqual(fork.json().benchmark.id, benchmarkId);
+ });
+
+ await check('a bad document is refused with a field-level error', async () => {
+ const bad = await t.get('/api/v1/benchmarks', { bearer: writer, json: { title: '', kind: 'games.play' } });
+ assert.strictEqual(bad.status, 422, bad.text);
+ const badRun = await t.get('/api/v1/runs', { bearer: publisher, json: runDoc(`${benchmarkId}@${version}`, { score: 140 }) });
+ assert.strictEqual(badRun.status, 400, badRun.text);
+ assert.strictEqual(badRun.json().code, 'bench.invalid');
+ });
+
+ await check('the API refuses an anonymous write and a token without the capability', async () => {
+ const anon = await t.get('/api/v1/benchmarks', { json: definition() });
+ assert.strictEqual(anon.status, 401, anon.text);
+ const noCap = await t.get('/api/v1/benchmarks', { bearer: reader, json: definition() });
+ assert.strictEqual(noCap.status, 403, noCap.text);
+ assert.strictEqual(noCap.json().code, 'bench.capability_denied');
+ const anonRun = await t.get('/api/v1/runs', { json: runDoc(`${benchmarkId}@${version}`) });
+ assert.strictEqual(anonRun.status, 401, anonRun.text);
+ });
+
+ await check('the compare endpoint puts two runs side by side', async () => {
+ const runs = await t.get(`/api/v1/compare?runs=nope`);
+ assert.strictEqual(runs.status, 404, runs.text);
+ });
+
+ await check('stop', async () => { await t.close(); });
+
+ done();
+})();