openvibers/openvibe.bench Commits

Bench B1: the benchmark registry and the results API (#24)

OP OpenVibers committed 14 days ago · 415b603aff · 10 files, +1348 −1

Stage B1 of the open benchmark platform (spec 2026-10-30).

A benchmark is owned, versioned and forkable. Its working copy is a draft; freezing
it writes an immutable version (the JSON definition plus its hash), and a version that
a run names never changes again. A fork records its parent.

A run is one result: a benchmark version × a model × a harness × settings × who ran
it. Its provenance says who ran it — verified (an executor did) or submitted (posted
with no executor, badged and kept out of the default rankings).

The API: POST/PUT /api/v1/benchmarks, POST .../versions (freeze), POST .../fork;
GET /api/v1/benchmarks?kind=&tag=&q= and GET .../:id[@version], with the holdout
tasks withheld unless the caller holds bench.run.publish; POST /api/v1/runs (verified
from a service token with bench.run.publish, submitted from a person's token),
GET /api/v1/runs/:id, GET /api/v1/compare, GET /api/v1/rankings (score, value, speed,
consistency; verified runs only by default) and GET /api/v1/percentile.

The migration adds bench_benchmarks, bench_benchmark_versions, bench_runs and
bench_run_tasks. The ten showcase prompts are seeded as one public benchmark per
kind, owned by openvibe, so the pages that come next have real content.

Changes

ddiff --git a/docs/api.md b/docs/api.md
nnew file mode 100644
iindex 0000000..5576845
---- /dev/null
++++ b/docs/api.md
@@@ -0,0 +1,32 @@
++# The Bench API
++
++JSON under `/api/v1`. Errors are `application/problem+json` with a stable `code`. A person is the `bench_at`
++session cookie or a Network user token as a Bearer; a service sends a client-credentials token with `bench.*`
++capabilities. A session write must come from bench.openvibe.services itself.
++
++## The registry (stage B1)
++
++| Route | Who | What |
++|---|---|---|
++| `POST /api/v1/benchmarks` | a person, or `bench.benchmark.write` | Create a benchmark (a draft). |
++| `PUT /api/v1/benchmarks/:id` | the same | Save a new draft. A body with `version` edits that version, and is refused once a run names it. |
++| `POST /api/v1/benchmarks/:id/versions` | the same | Freeze the working copy into the next immutable version. |
++| `POST /api/v1/benchmarks/:id/fork` | the same | Fork a benchmark; the fork records its parent. |
++| `GET /api/v1/benchmarks?kind=&tag=&q=` | anyone | The registry. |
++| `GET /api/v1/benchmarks/:id[@version]` | anyone | One benchmark and its definition. Holdout tasks are withheld unless the caller holds `bench.run.publish`. |
++
++## The results
++
++| Route | Who | What |
++|---|---|---|
++| `POST /api/v1/runs` | a person, or `bench.run.publish` | Publish a run. A service token with `bench.run.publish` makes a **verified** run; a person's token makes a **submitted** one, badged and out of the default rankings. |
++| `GET /api/v1/runs/:id` | anyone | One run with its per-task results. |
++| `GET /api/v1/compare?runs=a,b,c` | anyone | Up to four runs side by side, per task. |
++| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. |
++| `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |
++
++## The older API
++
++Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
++evals. Those are the round-based engine; the registry above is the versioned, forkable set the platform grows on.
++See `/docs` on the site for the live list.
ddiff --git a/docs/benchmarks.md b/docs/benchmarks.md
nnew file mode 100644
iindex 0000000..a83fd70
---- /dev/null
++++ b/docs/benchmarks.md
@@@ -0,0 +1,79 @@
++# Writing a benchmark
++
++A benchmark is a set you own, version and fork. This is how to write one.
++
++## The shape
++
++A benchmark (the contract is `bench.benchmark@1`) holds:
++
++| Field | What it is |
++|---|---|
++| `id` | `bmk_…`, given by Bench when you create it. The same id carries every version. |
++| `owner` | You. A person (`usr_…`), a project (`prj_…`) or a service. |
++| `title`, `description` | What the set is and who it is for. |
++| `kind` | One of Bench's kinds (`games.play`, `design.ui`, `codes.patch`, …). The kind sets the renderer. |
++| `tags` | Up to 32 words people search by. |
++| `visibility` | `public` (in the registry) or `unlisted` (by id only). |
++| `version` | The version this document is. Bench numbers them. |
++| `parent` | The `id@version` this one was forked from, or `null`. |
++| `tasks[]` | `{ id, prompt, files?, expected?, holdout }`. The prompt is the task; `holdout: true` hides it from everyone but the executor. |
++| `judges[]` | `{ type, version, config }`. The judge is pinned by version, so a run names the judge it used. |
++| `scoring` | `{ method, scale, weights? }`. `method` is `mean`, `weighted`, `pass_rate` or `sum`; the scale is 0-100. |
++| `limits` | `{ ms?, tokens?, usd }`. A run may not pass them. |
++| `matrix` | `{ models[], harnesses[] }`. The defaults a runner starts from. |
++
++## The three calls
++
++```
++POST /api/v1/benchmarks create it (a draft)
++PUT /api/v1/benchmarks/:id save a new draft
++POST /api/v1/benchmarks/:id/versions freeze a version
++POST /api/v1/benchmarks/:id/fork fork it
++```
++
++Authoring needs the `bench.benchmark.write` capability on a service token, or a signed-in person.
++
++A version is **immutable once a run names it**. Write a new draft and freeze it again instead; if you want to change a
++published set, fork it — the fork records the parent, and the old version stays where the old runs point.
++
++## An example
++
++```json
++{
++ "title": "Playable games arena",
++ "description": "One-screen arcade games, judged by play.",
++ "kind": "games.play",
++ "tags": ["games", "arcade"],
++ "visibility": "public",
++ "tasks": [
++ { "id": "t1", "prompt": "Build a one-screen arcade game with a paddle, bricks, a ball and a score.", "files": [], "expected": null, "holdout": false },
++ { "id": "t2", "prompt": "Build a snake game on a 20x20 grid.", "files": [], "expected": null, "holdout": true }
++ ],
++ "judges": [{ "type": "human_pairwise", "version": "1.0", "config": {} }],
++ "scoring": { "method": "mean", "scale": { "min": 0, "max": 100 } },
++ "limits": { "ms": 120000, "tokens": 200000, "usd": 0.5 },
++ "matrix": { "models": ["claude-opus-4"], "harnesses": ["claude-code"] }
++}
++```
++
++## Running it
++
++A run (the contract is `bench.run@1`) names the benchmark `id@version`, the model, the route, the provider, the
++harness and its version, the settings, the executor, and the per-task score, cost, time, tokens and artifacts. It
++ends with the run's own score, its cost, time and tokens, and its provenance.
++
++Publish it with `POST /api/v1/runs`. A service token with `bench.run.publish` publishes a **verified** run (an
++executor ran it, so it can be re-run and it counts in the rankings). A person's token publishes a **submitted** run:
++it is badged and kept out of the default rankings.
++
++## Reading the results
++
++```
++GET /api/v1/runs/:id
++GET /api/v1/compare?runs=a,b,c
++GET /api/v1/rankings?benchmark=&metric=score|value|speed|consistency&harness=&since=
++GET /api/v1/percentile?benchmark=&model=&score=
++```
++
++`value` is the score per dollar, `speed` the score per minute, and `consistency` the spread of a model's scores
++inside one session (the lower, the steadier). The percentile says where a score sits among that model's runs.
ddiff --git a/migrations/0003_bench_registry.sql b/migrations/0003_bench_registry.sql
nnew file mode 100644
iindex 0000000..b355452
---- /dev/null
++++ b/migrations/0003_bench_registry.sql
@@@ -0,0 +1,93 @@
++-- phase: expand
++-- Stage B1 (spec 2026-10-30): the benchmark registry and the results API.
++--
++-- A benchmark is an owned, versioned and forkable set: its tasks, its judges, how they combine into one score, the
++-- limits a run may not pass and the default model and harness matrix. A version is immutable once a run names it, so
++-- bench_benchmark_versions keeps the frozen JSON definition and its hash; a fork records its parent.
++--
++-- A run is one result: a benchmark version × a model × a harness × settings × who ran it. Provenance says who ran it:
++-- verified (an executor did) or submitted (posted through the API with no executor). Submitted runs are badged and
++-- kept out of the default rankings. Nothing written here is a credential: a run never carries a key.
++
++CREATE TABLE bench_benchmarks (
++ id text COLLATE "C" PRIMARY KEY, -- bmk_ + ULID
++ owner text COLLATE "C" NOT NULL, -- usr_…, prj_… or a service
++ title text NOT NULL,
++ description text NOT NULL DEFAULT '',
++ kind text COLLATE "C" NOT NULL, -- one of Bench's kinds; the kind sets the renderer
++ tags jsonb NOT NULL DEFAULT '[]'::jsonb,
++ visibility text COLLATE "C" NOT NULL DEFAULT 'unlisted', -- public | unlisted
++ parent text COLLATE "C", -- the benchmark id@version this one was forked from
++ latest_version integer NOT NULL DEFAULT 0, -- the newest frozen version (0: a draft)
++ draft jsonb, -- the unpublished working copy (PUT saves it)
++ created_by text COLLATE "C" NOT NULL,
++ created_at timestamptz NOT NULL DEFAULT now(),
++ updated_at timestamptz NOT NULL DEFAULT now(),
++ CONSTRAINT bench_benchmarks_id CHECK (id ~ '^bmk_[0-9A-HJKMNP-TV-Z]{26}$'),
++ CONSTRAINT bench_benchmarks_visibility CHECK (visibility IN ('public', 'unlisted')),
++ CONSTRAINT bench_benchmarks_latest CHECK (latest_version >= 0),
++ CONSTRAINT bench_benchmarks_title CHECK (length(title) BETWEEN 1 AND 160),
++ CONSTRAINT bench_benchmarks_kind CHECK (length(kind) BETWEEN 1 AND 80)
++);
++CREATE INDEX bench_benchmarks_by_kind ON bench_benchmarks (kind, created_at DESC);
++CREATE INDEX bench_benchmarks_by_owner ON bench_benchmarks (owner, created_at DESC);
++CREATE INDEX bench_benchmarks_by_visibility ON bench_benchmarks (visibility, created_at DESC);
++
++CREATE TABLE bench_benchmark_versions (
++ benchmark_id text COLLATE "C" NOT NULL REFERENCES bench_benchmarks (id) ON DELETE CASCADE,
++ version integer NOT NULL,
++ definition jsonb NOT NULL, -- the whole bench.benchmark@1 document
++ hash text COLLATE "C" NOT NULL, -- sha256 of the canonical definition
++ created_by text COLLATE "C" NOT NULL,
++ created_at timestamptz NOT NULL DEFAULT now(),
++ PRIMARY KEY (benchmark_id, version),
++ CONSTRAINT bench_benchmark_versions_version CHECK (version >= 1),
++ CONSTRAINT bench_benchmark_versions_hash CHECK (hash ~ '^[0-9a-f]{64}$')
++);
++CREATE INDEX bench_benchmark_versions_by_hash ON bench_benchmark_versions (hash);
++
++CREATE TABLE bench_runs (
++ id text COLLATE "C" PRIMARY KEY, -- brn_ + ULID
++ benchmark_id text COLLATE "C" NOT NULL REFERENCES bench_benchmarks (id),
++ benchmark_version integer NOT NULL,
++ model text COLLATE "C" NOT NULL,
++ route jsonb NOT NULL DEFAULT '{}'::jsonb, -- the route that produced the run
++ provider text COLLATE "C" NOT NULL,
++ harness jsonb NOT NULL DEFAULT '{}'::jsonb, -- { name, version }
++ settings jsonb NOT NULL DEFAULT '{}'::jsonb, -- { temperature, effort }
++ executor jsonb NOT NULL DEFAULT '{}'::jsonb, -- { kind: codes|run|node|submitted, id }
++ hardware jsonb, -- the machine, for a local model
++ score numeric(8, 3) NOT NULL DEFAULT 0, -- 0-100
++ cost_usd numeric(12, 6) NOT NULL DEFAULT 0,
++ ms integer NOT NULL DEFAULT 0,
++ tokens integer NOT NULL DEFAULT 0,
++ provenance text COLLATE "C" NOT NULL, -- verified | submitted
++ session text COLLATE "C", -- a session groups repeated runs of one configuration
++ created_by text COLLATE "C" NOT NULL,
++ created_at timestamptz NOT NULL DEFAULT now(),
++ CONSTRAINT bench_runs_id CHECK (id ~ '^brn_[0-9A-HJKMNP-TV-Z]{26}$'),
++ CONSTRAINT bench_runs_provenance CHECK (provenance IN ('verified', 'submitted')),
++ CONSTRAINT bench_runs_score CHECK (score >= 0 AND score <= 100),
++ CONSTRAINT bench_runs_version CHECK (benchmark_version >= 1),
++ FOREIGN KEY (benchmark_id, benchmark_version) REFERENCES bench_benchmark_versions (benchmark_id, version)
++);
++CREATE INDEX bench_runs_by_benchmark ON bench_runs (benchmark_id, benchmark_version, created_at DESC);
++CREATE INDEX bench_runs_by_model ON bench_runs (model, created_at DESC);
++CREATE INDEX bench_runs_by_session ON bench_runs (session, created_at DESC);
++CREATE INDEX bench_runs_verified ON bench_runs (benchmark_id, created_at DESC) WHERE provenance = 'verified';
++
++CREATE TABLE bench_run_tasks (
++ id text COLLATE "C" PRIMARY KEY, -- brt_ + ULID
++ run_id text COLLATE "C" NOT NULL REFERENCES bench_runs (id) ON DELETE CASCADE,
++ task_id text COLLATE "C" NOT NULL, -- the benchmark task's id
++ score numeric(8, 3) NOT NULL DEFAULT 0, -- 0-100
++ cost_usd numeric(12, 6) NOT NULL DEFAULT 0,
++ ms integer NOT NULL DEFAULT 0,
++ tokens integer NOT NULL DEFAULT 0,
++ artifacts jsonb NOT NULL DEFAULT '[]'::jsonb, -- [{ kind, url }]
++ transcript_url text,
++ created_at timestamptz NOT NULL DEFAULT now(),
++ CONSTRAINT bench_run_tasks_id CHECK (id ~ '^brt_[0-9A-HJKMNP-TV-Z]{26}$'),
++ CONSTRAINT bench_run_tasks_score CHECK (score >= 0 AND score <= 100)
++);
++CREATE INDEX bench_run_tasks_by_run ON bench_run_tasks (run_id, created_at, id);
ddiff --git a/server/app.js b/server/app.js
iindex 13bdc40..f85d361 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -25,10 +25,13 @@ const { createKeyStore } = require('./auth/keys');
const { createSso } = require('./auth/sso');
const { createPrincipal } = require('./http/principal');
const { createApi } = require('./http/api');
++const { createRegistryApi } = require('./http/registry');
const { createPageRoutes } = require('./http/pages');
const { createShowcaseRoutes, createShowcasePages } = require('./http/showcase');
const { createRenderRoutes } = require('./http/render');
const { seedShowcaseTasks } = require('./bench/seed-tasks');
++const { createRegistry } = require('./bench/registry');
++const { seedBenchmarks } = require('./bench/seed-benchmarks');
const { createServiceReadiness } = require('./observability');
const { assetVersion, send } = require('./render/layout');
const { raw } = require('./render/html');
@@@ -61,12 +64,14 @@ async function createApp(opts = {}) {
const principal = createPrincipal({ config, keys });
const bench = opts.bench || createStore(s, { config, log, random: opts.random });
++ const registry = opts.registry || createRegistry(s, { log });
const ai = opts.benchAi || createBenchAi({ config, fetchImpl, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
const publisher = createPublisher({ config, s, fetchImpl, log });
await bench.seedKinds();
try { await seedShowcaseTasks(s, bench); } catch (err) { log.warn('[OpenVibe.Bench] showcase tasks not seeded:', (err && err.message) || err); }
++ try { await seedBenchmarks(s, registry); } catch (err) { log.warn('[OpenVibe.Bench] showcase benchmarks not seeded:', (err && err.message) || err); }
const accountData = accountDataLib.create({ db: s.db, log });
const accountSend = opts.accountSend || (config.oauth.clientSecret
@@@ -74,7 +79,7 @@ async function createApp(opts = {}) {
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Bench cannot answer account events'); });
const ctx = {
-- config, s, keys, sso, principal, bench, ai, jobs, publisher, accountData, accountSend, log,
++ config, s, keys, sso, principal, bench, registry, ai, jobs, publisher, accountData, accountSend, log,
random: opts.random || Math.random,
};
@@@ -151,6 +156,7 @@ async function createApp(opts = {}) {
// ── API and MCP ─────────────────────────────────────────
app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.BENCH_API_RATE_LIMIT_PER_MIN) || 240, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));
app.use('/api/v1', createApi(ctx));
++ app.use('/api/v1', createRegistryApi(ctx));
app.use('/api/v1/showcase', createShowcaseRoutes(ctx));
app.use(mcpRouter(ctx));
app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));
ddiff --git a/server/bench/registry.js b/server/bench/registry.js
nnew file mode 100644
iindex 0000000..47de5cb
---- /dev/null
++++ b/server/bench/registry.js
@@@ -0,0 +1,507 @@
++'use strict';
++
++/**
++ * The benchmark registry and the results database (spec 2026-10-30, stage B1).
++ *
++ * A benchmark is owned, versioned and forkable. Its working copy is a draft; freezing it writes an immutable version
++ * (the JSON definition plus its hash). A version that a run names never changes again — the rule is enforced here,
++ * not by convention: a write that would change it is refused.
++ *
++ * A run is one result: a benchmark version × a model × a harness × settings × who ran it. Its provenance says who ran
++ * it: verified (an executor did) or submitted (posted through the API with no executor). Submitted runs are badged and
++ * kept out of the default rankings.
++ *
++ * Nothing here holds a credential: a run records which account paid, never a key.
++ */
++const crypto = require('crypto');
++const { ids } = require('openvibe-contracts');
++
++const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
++const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
++const refusal = (status, code, message) => ({ ok: false, status, code, message });
++
++class RegistryError extends Error {
++ constructor(status, code, detail) {
++ super(detail);
++ this.name = 'RegistryError';
++ this.status = status;
++ this.code = code;
++ this.detail = detail;
++ }
++}
++
++/** The canonical form of a definition: the keys Bench knows, in one order, so the hash is stable. */
++function canonical(value) {
++ if (Array.isArray(value)) return value.map(canonical);
++ if (value && typeof value === 'object') {
++ const out = {};
++ for (const key of Object.keys(value).sort()) out[key] = canonical(value[key]);
++ return out;
++ }
++ return value;
++}
++
++function hashDefinition(definition) {
++ return crypto.createHash('sha256').update(JSON.stringify(canonical(definition)), 'utf8').digest('hex');
++}
++
++const json = (v, fallback) => {
++ if (v == null) return fallback;
++ if (typeof v === 'string') { try { return JSON.parse(v); } catch { return fallback; } }
++ return v;
++};
++
++function wireBenchmark(row) {
++ if (!row) return null;
++ return {
++ id: row.id,
++ owner: row.owner,
++ title: row.title,
++ description: row.description || '',
++ kind: row.kind,
++ tags: Array.isArray(json(row.tags, [])) ? json(row.tags, []) : [],
++ visibility: row.visibility,
++ parent: row.parent || null,
++ latest_version: num(row.latest_version),
++ draft: json(row.draft, null),
++ created_by: row.created_by,
++ created_at: iso(row.created_at),
++ updated_at: iso(row.updated_at),
++ };
++}
++
++function wireVersion(row) {
++ if (!row) return null;
++ return {
++ benchmark_id: row.benchmark_id,
++ version: num(row.version),
++ definition: json(row.definition, {}),
++ hash: row.hash,
++ created_by: row.created_by,
++ created_at: iso(row.created_at),
++ };
++}
++
++function wireRun(row, tasks = null) {
++ if (!row) return null;
++ const out = {
++ id: row.id,
++ benchmark: `${row.benchmark_id}@${num(row.benchmark_version)}`,
++ benchmark_id: row.benchmark_id,
++ benchmark_version: num(row.benchmark_version),
++ model: row.model,
++ route: json(row.route, {}),
++ provider: row.provider,
++ harness: json(row.harness, {}),
++ settings: json(row.settings, {}),
++ executor: json(row.executor, {}),
++ hardware: json(row.hardware, null),
++ score: num(row.score),
++ cost_usd: num(row.cost_usd),
++ ms: num(row.ms),
++ tokens: num(row.tokens),
++ provenance: row.provenance,
++ session: row.session || null,
++ created_by: row.created_by,
++ created_at: iso(row.created_at),
++ };
++ if (tasks) out.tasks = tasks;
++ return out;
++}
++
++function wireTask(row) {
++ if (!row) return null;
++ return {
++ task_id: row.task_id,
++ score: num(row.score),
++ cost_usd: num(row.cost_usd),
++ ms: num(row.ms),
++ tokens: num(row.tokens),
++ artifacts: Array.isArray(json(row.artifacts, [])) ? json(row.artifacts, []) : [],
++ transcript_url: row.transcript_url || null,
++ };
++}
++
++/** The median of a list of numbers. */
++function median(values) {
++ const list = values.filter((v) => Number.isFinite(v)).sort((a, b) => a - b);
++ if (!list.length) return null;
++ const mid = Math.floor(list.length / 2);
++ return list.length % 2 ? list[mid] : (list[mid - 1] + list[mid]) / 2;
++}
++
++/** "bmk_x@2" → { id, version }; "bmk_x" → { id, version: null } (the newest frozen one). */
++function parseRef(ref) {
++ const text = String(ref || '').trim();
++ const at = text.lastIndexOf('@');
++ if (at > 0) {
++ const version = Number(text.slice(at + 1));
++ if (!Number.isInteger(version) || version < 1) return null;
++ return { id: text.slice(0, at), version };
++ }
++ return { id: text, version: null };
++}
++
++function createRegistry(s, { log = console } = {}) {
++ const db = s.db;
++
++ // ── Benchmarks ───────────────────────────────────────────
++ async function createBenchmark({ owner, title, description = '', kind, tags = [], visibility = 'unlisted', parent = null, definition = null, subject }) {
++ const text = String(title == null ? '' : title).trim();
++ if (!text) return refusal(422, 'bench.title_required', 'A benchmark needs a title.');
++ if (text.length > 160) return refusal(422, 'bench.title_too_long', 'The title must be at most 160 characters.');
++ const kindId = String(kind == null ? '' : kind).trim();
++ if (!kindId) return refusal(422, 'bench.kind_required', 'A benchmark needs a kind.');
++ const vis = String(visibility || 'unlisted');
++ if (!['public', 'unlisted'].includes(vis)) return refusal(422, 'bench.invalid_visibility', 'visibility is public or unlisted.');
++ const list = (Array.isArray(tags) ? tags : []).map((t) => String(t).trim()).filter(Boolean).slice(0, 32);
++ const id = s.newId('bmk');
++ const row = await db.maybe(
++ `INSERT INTO bench_benchmarks (id, owner, title, description, kind, tags, visibility, parent, latest_version, draft, created_by, created_at, updated_at)
++ VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, 0, $9::jsonb, $10, $11::timestamptz, $11::timestamptz)
++ RETURNING *`,
++ [id, String(owner || 'openvibe'), text, String(description || '').slice(0, 4000), kindId,
++ JSON.stringify(list), vis, parent || null, JSON.stringify(definition || {}), String(subject || owner || 'openvibe'), s.iso()],
++ );
++ return { ok: true, benchmark: wireBenchmark(row) };
++ }
++
++ /** The working copy: what a later freeze would publish. */
++ async function getBenchmark(id) {
++ return wireBenchmark(await db.maybe('SELECT * FROM bench_benchmarks WHERE id = $1', [String(id || '')]));
++ }
++
++ async function getVersion(benchmarkId, version) {
++ return wireVersion(await db.maybe(
++ 'SELECT * FROM bench_benchmark_versions WHERE benchmark_id = $1 AND version = $2',
++ [String(benchmarkId || ''), num(version)]));
++ }
++
++ /** The newest frozen version of a benchmark. */
++ async function latestVersion(benchmarkId) {
++ return wireVersion(await db.maybe(
++ 'SELECT * FROM bench_benchmark_versions WHERE benchmark_id = $1 ORDER BY version DESC LIMIT 1',
++ [String(benchmarkId || '')]));
++ }
++
++ /**
++ * The definition of a benchmark, at a version or at its newest frozen one. Holdout tasks are withheld unless the
++ * caller holds bench.run.publish: the executor may see them, everyone else may not.
++ */
++ async function getDefinition(idOrRef, { reveal = false } = {}) {
++ const ref = parseRef(idOrRef);
++ if (!ref) return null;
++ const benchmark = await getBenchmark(ref.id);
++ if (!benchmark) return null;
++ const version = ref.version != null
++ ? await getVersion(ref.id, ref.version)
++ : await latestVersion(ref.id);
++ if (!version) return null;
++ const definition = { ...version.definition };
++ if (!reveal && Array.isArray(definition.tasks)) {
++ definition.tasks = definition.tasks.map((t) => (t && t.holdout ? { ...t, prompt: null, expected: null, files: null, holdout: true } : t));
++ }
++ return { benchmark, version: { ...version, definition }, definition };
++ }
++
++ /** Save a new draft. A version named in the body is edited in place, and only while no run names it. */
++ async function saveDraft(id, { definition = {}, subject }) {
++ const benchmark = await getBenchmark(id);
++ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
++ const version = definition.version == null ? null : num(definition.version);
++ if (version != null) {
++ const row = await getVersion(benchmark.id, version);
++ if (!row) return refusal(404, 'bench.version_not_found', 'No such version.');
++ const used = num(await db.value('SELECT COUNT(*)::int AS n FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = $2', [benchmark.id, version]));
++ if (used > 0) {
++ return refusal(409, 'bench.version_frozen', 'A version that a run names never changes again. Fork the benchmark instead.');
++ }
++ const next = { ...row.definition, ...definition, version };
++ await db.exec(
++ 'UPDATE bench_benchmark_versions SET definition = $1::jsonb, hash = $2 WHERE benchmark_id = $3 AND version = $4',
++ [JSON.stringify(next), hashDefinition(next), benchmark.id, version]);
++ await db.exec('UPDATE bench_benchmarks SET updated_at = $1::timestamptz WHERE id = $2', [s.iso(), benchmark.id]);
++ return { ok: true, version: await getVersion(benchmark.id, version) };
++ }
++ const draft = { ...(benchmark.draft || {}), ...definition };
++ await db.exec('UPDATE bench_benchmarks SET draft = $1::jsonb, updated_at = $2::timestamptz WHERE id = $3',
++ [JSON.stringify(draft), s.iso(), benchmark.id]);
++ return { ok: true, draft };
++ }
++
++ /** Freeze the working copy (or a body definition) into the next immutable version. */
++ async function freezeVersion(id, { definition = null, subject, validate = null }) {
++ const benchmark = await getBenchmark(id);
++ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
++ const source = definition || benchmark.draft || {};
++ const version = benchmark.latest_version + 1;
++ const doc = {
++ id: benchmark.id,
++ owner: benchmark.owner,
++ title: source.title || benchmark.title,
++ description: source.description == null ? benchmark.description : source.description,
++ kind: source.kind || benchmark.kind,
++ tags: Array.isArray(source.tags) ? source.tags : benchmark.tags,
++ visibility: source.visibility || benchmark.visibility,
++ version,
++ parent: benchmark.parent,
++ tasks: Array.isArray(source.tasks) ? source.tasks : [],
++ judges: Array.isArray(source.judges) ? source.judges : [],
++ scoring: source.scoring || { method: 'mean', scale: { min: 0, max: 100 } },
++ limits: source.limits || {},
++ matrix: source.matrix || {},
++ created_at: s.iso(),
++ };
++ if (!doc.tasks.length) return refusal(422, 'bench.no_tasks', 'A benchmark needs at least one task before it can be frozen.');
++ if (validate) {
++ const check = validate(doc);
++ if (!check.valid) {
++ return refusal(422, 'bench.invalid_definition', check.errors.map((e) => `${e.path} ${e.message}`).join('; '));
++ }
++ }
++ await db.exec(
++ `INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)
++ VALUES ($1, $2, $3::jsonb, $4, $5, $6::timestamptz)
++ ON CONFLICT (benchmark_id, version) DO NOTHING`,
++ [benchmark.id, version, JSON.stringify(doc), hashDefinition(doc), String(subject || benchmark.created_by), s.iso()]);
++ await db.exec('UPDATE bench_benchmarks SET latest_version = $1, updated_at = $2::timestamptz WHERE id = $3',
++ [version, s.iso(), benchmark.id]);
++ return { ok: true, version: await getVersion(benchmark.id, version), benchmark: await getBenchmark(benchmark.id) };
++ }
++
++ /** A fork: a new benchmark of its own that records the parent it came from. */
++ async function forkBenchmark(id, { version = null, title = null, owner, subject }) {
++ const source = await getDefinition(id, { reveal: true });
++ if (!source) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
++ const parent = `${source.benchmark.id}@${source.version.version}`;
++ const definition = {
++ ...source.definition,
++ id: null,
++ version: 0,
++ parent,
++ title: title || `${source.definition.title || source.benchmark.title} (fork)`,
++ tasks: Array.isArray(source.definition.tasks) ? source.definition.tasks : [],
++ created_at: null,
++ };
++ const made = await createBenchmark({
++ owner: owner || subject,
++ title: definition.title,
++ description: definition.description,
++ kind: definition.kind,
++ tags: definition.tags,
++ visibility: definition.visibility,
++ parent,
++ definition,
++ subject,
++ });
++ if (!made.ok) return made;
++ const frozen = await freezeVersion(made.benchmark.id, { definition: { ...definition, version: 1 }, subject });
++ if (!frozen.ok) return frozen;
++ return { ok: true, benchmark: await getBenchmark(made.benchmark.id), version: frozen.version, parent };
++ }
++
++ /** The registry list: by kind, by tag, or by a search over the title and the description. */
++ async function listBenchmarks({ kind = null, tag = null, q = null, limit = 50 } = {}) {
++ const where = [];
++ const params = [];
++ if (kind) { params.push(String(kind)); where.push(`kind = $${params.length}`); }
++ if (tag) { params.push(JSON.stringify([String(tag)])); where.push(`tags @> $${params.length}::jsonb`); }
++ if (q) {
++ params.push(`%${String(q).toLowerCase()}%`);
++ where.push(`(lower(title) LIKE $${params.length} OR lower(description) LIKE $${params.length})`);
++ }
++ const sql = `SELECT * FROM bench_benchmarks ${where.length ? 'WHERE ' + where.join(' AND ') : ''}
++ ORDER BY created_at DESC LIMIT ${Math.max(1, Math.min(200, num(limit, 50)))}`;
++ const rows = await db.many(sql, params);
++ return rows.map(wireBenchmark);
++ }
++
++ // ── Runs ─────────────────────────────────────────────────
++ /**
++ * Publish a run. A service token with bench.run.publish makes a verified run; a person's token makes a submitted
++ * run, which is badged and kept out of the default rankings.
++ */
++ async function publishRun(doc, { provenance, subject }) {
++ const benchmarkId = String(doc.benchmark_id || (doc.benchmark || '').split('@')[0] || '');
++ const version = num(doc.benchmark_version || (doc.benchmark || '').split('@')[1]);
++ const versionRow = await getVersion(benchmarkId, version);
++ if (!versionRow) return refusal(404, 'bench.version_not_found', 'No such benchmark version.');
++ const benchmark = await getBenchmark(benchmarkId);
++ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
++ const model = String(doc.model || '').trim();
++ if (!model) return refusal(422, 'bench.model_required', 'A run names the model it ran.');
++ const provider = String(doc.provider || '').trim();
++ if (!provider) return refusal(422, 'bench.provider_required', 'A run names the provider that served the model.');
++ const tasks = Array.isArray(doc.tasks) ? doc.tasks : [];
++ if (!tasks.length) return refusal(422, 'bench.no_tasks', 'A run carries at least one task result.');
++ const known = new Set((versionRow.definition.tasks || []).map((t) => String(t.id)));
++ for (const t of tasks) {
++ if (!known.has(String(t.task_id))) {
++ return refusal(422, 'bench.unknown_task', `This benchmark version has no task "${t.task_id}".`);
++ }
++ }
++ // A submitted run has no executor: whatever the body says, the provenance decides.
++ const executorKind = provenance === 'submitted' ? 'submitted' : String((doc.executor && doc.executor.kind) || 'codes');
++ const executor = { kind: executorKind, id: String((doc.executor && doc.executor.id) || subject || 'unknown') };
++ const id = s.newId('brn');
++ const score = num(doc.score, 0);
++ const cost = num(doc.cost_usd, 0);
++ const ms = num(doc.ms, 0);
++ const tokens = num(doc.tokens, 0);
++ await db.tx(async () => {
++ await db.exec(
++ `INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, route, provider, harness, settings, executor,
++ hardware, score, cost_usd, ms, tokens, provenance, session, created_by, created_at)
++ VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7::jsonb, $8::jsonb, $9::jsonb, $10::jsonb, $11, $12, $13, $14, $15, $16, $17, $18::timestamptz)`,
++ [id, benchmarkId, version, model, JSON.stringify(doc.route || {}), provider,
++ JSON.stringify(doc.harness || {}), JSON.stringify(doc.settings || {}), JSON.stringify(executor),
++ doc.hardware == null ? null : JSON.stringify(doc.hardware), score, cost, ms, tokens, provenance,
++ doc.session == null ? null : String(doc.session).slice(0, 120), String(subject || 'unknown'), s.iso()],
++ );
++ for (const t of tasks) {
++ await db.exec(
++ `INSERT INTO bench_run_tasks (id, run_id, task_id, score, cost_usd, ms, tokens, artifacts, transcript_url, created_at)
++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10::timestamptz)`,
++ [s.newId('brt'), id, String(t.task_id), num(t.score), num(t.cost_usd), num(t.ms), num(t.tokens),
++ JSON.stringify(Array.isArray(t.artifacts) ? t.artifacts : []), t.transcript_url == null ? null : String(t.transcript_url), s.iso()],
++ );
++ }
++ });
++ return { ok: true, run: await getRun(id) };
++ }
++
++ async function getRun(id) {
++ const row = await db.maybe('SELECT * FROM bench_runs WHERE id = $1', [String(id || '')]);
++ if (!row) return null;
++ const tasks = (await db.many('SELECT * FROM bench_run_tasks WHERE run_id = $1 ORDER BY created_at, id', [row.id])).map(wireTask);
++ return wireRun(row, tasks);
++ }
++
++ /** Side by side: the same tasks, the same columns, so two runs can be read against each other. */
++ async function compareRuns(refs) {
++ const runs = [];
++ for (const ref of refs) {
++ const run = await getRun(ref);
++ if (run) runs.push(run);
++ }
++ if (!runs.length) return null;
++ const taskIds = [];
++ for (const run of runs) {
++ for (const t of run.tasks) if (!taskIds.includes(t.task_id)) taskIds.push(t.task_id);
++ }
++ return {
++ runs,
++ task_ids: taskIds,
++ rows: taskIds.map((taskId) => ({
++ task_id: taskId,
++ scores: runs.map((run) => {
++ const t = run.tasks.find((x) => x.task_id === taskId);
++ return t ? t.score : null;
++ }),
++ })),
++ };
++ }
++
++ /**
++ * The rankings for a benchmark. Verified runs only unless the caller asks for the submitted ones.
++ * score the median score of a model
++ * value the median score per dollar
++ * speed the median score per minute
++ * consistency the spread of a model's scores inside one session (the lower, the steadier)
++ */
++ async function rankings({ benchmark, metric = 'score', harness = null, since = null, includeSubmitted = false }) {
++ const ref = parseRef(benchmark);
++ if (!ref) return null;
++ const versions = await db.many(
++ 'SELECT version FROM bench_benchmark_versions WHERE benchmark_id = $1 ORDER BY version DESC',
++ [ref.id]);
++ const wanted = ref.version != null ? [ref.version] : versions.map((v) => num(v.version));
++ if (!wanted.length) return null;
++ const params = [ref.id, wanted];
++ let sql = `SELECT * FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = ANY($2::int[])`;
++ if (!includeSubmitted) sql += " AND provenance = 'verified'";
++ if (harness) { params.push(JSON.stringify([String(harness)])); sql += ` AND harness->>'name' = $3`; }
++ if (since) { params.push(String(since)); sql += ' AND created_at >= $4::timestamptz'; }
++ const rows = await db.many(sql, params);
++ const byModel = new Map();
++ for (const row of rows) {
++ const run = wireRun(row);
++ if (!byModel.has(run.model)) byModel.set(run.model, []);
++ byModel.get(run.model).push(run);
++ }
++ const entries = [];
++ for (const [model, list] of byModel) {
++ const scores = list.map((r) => r.score);
++ const costs = list.map((r) => r.cost_usd);
++ const times = list.map((r) => r.ms);
++ const medScore = median(scores);
++ const medCost = median(costs);
++ const medMs = median(times);
++ const sessions = new Map();
++ for (const r of list) {
++ const key = r.session || 'default';
++ if (!sessions.has(key)) sessions.set(key, []);
++ sessions.get(key).push(r.score);
++ }
++ const spreads = [...sessions.values()].filter((v) => v.length > 1).map((v) => Math.max(...v) - Math.min(...v));
++ const entry = {
++ model,
++ runs: list.length,
++ score: medScore == null ? null : Math.round(medScore * 1000) / 1000,
++ cost_usd: medCost == null ? null : Math.round(medCost * 1e6) / 1e6,
++ ms: medMs == null ? null : Math.round(medMs),
++ value: medScore != null && medCost ? Math.round((medScore / medCost) * 1000) / 1000 : null,
++ speed: medScore != null && medMs ? Math.round((medScore / (medMs / 60000)) * 1000) / 1000 : null,
++ consistency: spreads.length ? Math.round(median(spreads) * 1000) / 1000 : null,
++ sessions: sessions.size,
++ };
++ entries.push(entry);
++ }
++ const key = { score: 'score', value: 'value', speed: 'speed', consistency: 'consistency' }[metric];
++ if (!key) return null;
++ const ascending = metric === 'consistency';
++ entries.sort((a, b) => {
++ const av = a[key]; const bv = b[key];
++ if (av == null && bv == null) return a.model.localeCompare(b.model);
++ if (av == null) return 1;
++ if (bv == null) return -1;
++ if (av === bv) return a.model.localeCompare(b.model);
++ return ascending ? av - bv : bv - av;
++ });
++ return { benchmark: ref.id, metric, entries };
++ }
++
++ /** Where a score sits among a model's runs on a benchmark: the percentile, 0-100. */
++ async function percentile({ benchmark, model, score }) {
++ const ref = parseRef(benchmark);
++ if (!ref || !model) return null;
++ const params = [ref.id];
++ let sql = "SELECT score FROM bench_runs WHERE benchmark_id = $1 AND model = $2 AND provenance = 'verified'";
++ params.push(String(model));
++ if (ref.version != null) { params.push(ref.version); sql += ' AND benchmark_version = $3'; }
++ const rows = await db.many(sql, params);
++ const scores = rows.map((r) => num(r.score));
++ if (!scores.length) return { benchmark: ref.id, model: String(model), runs: 0, percentile: null, score: num(score) };
++ const value = num(score);
++ const below = scores.filter((v) => v < value).length;
++ const equal = scores.filter((v) => v === value).length;
++ const pct = Math.round(((below + equal / 2) / scores.length) * 1000) / 10;
++ return {
++ benchmark: ref.id,
++ model: String(model),
++ runs: scores.length,
++ score: value,
++ percentile: pct,
++ median: Math.round(median(scores) * 1000) / 1000,
++ best: Math.max(...scores),
++ worst: Math.min(...scores),
++ };
++ }
++
++ return {
++ createBenchmark, getBenchmark, getVersion, latestVersion, getDefinition, saveDraft, freezeVersion,
++ forkBenchmark, listBenchmarks, publishRun, getRun, compareRuns, rankings, percentile,
++ };
++}
++
++module.exports = { createRegistry, RegistryError, hashDefinition, canonical, median, parseRef };
ddiff --git a/server/bench/seed-benchmarks.js b/server/bench/seed-benchmarks.js
nnew file mode 100644
iindex 0000000..02603b7
---- /dev/null
++++ b/server/bench/seed-benchmarks.js
@@@ -0,0 +1,99 @@
++'use strict';
++
++/**
++ * The showcase benchmarks (spec 2026-10-30, stage B1, item 4).
++ *
++ * The ten showcase prompts are seeded as one public benchmark per kind, owned by "openvibe", so the pages that come
++ * next (B2) have real content to render: a benchmark, its tasks, its judges and a version to run against.
++ *
++ * Seeding is idempotent: a benchmark with the same title and kind is left alone, so a restart never makes a second
++ * copy. Nothing here holds a credential and nothing is scored: the benchmarks are the prompts and the rules.
++ */
++const { SHOWCASE_TASKS } = require('./seed-tasks');
++const { kindById } = require('../kinds');
++
++const OWNER = 'openvibe';
++
++function groupByKind() {
++ const groups = new Map();
++ for (const t of SHOWCASE_TASKS) {
++ if (!groups.has(t.kind)) groups.set(t.kind, []);
++ groups.get(t.kind).push(t);
++ }
++ return groups;
++}
++
++function titleFor(kind) {
++ const k = kindById(kind);
++ return k ? k.name : `${kind} benchmark`;
++}
++
++/** The tasks of a kind, with the ids a run names them by. */
++function tasksFor(list) {
++ return list.map((t, i) => ({
++ id: `t${i + 1}`,
++ prompt: t.prompt,
++ files: [],
++ expected: null,
++ holdout: false,
++ }));
++}
++
++/** The judges of a kind: the kind's own judge, plus the others its schema names. */
++function judgesFor(kind) {
++ const k = kindById(kind);
++ const list = (k && k.schema && Array.isArray(k.schema.judges) && k.schema.judges.length) ? k.schema.judges : [k ? k.judge : 'human_pairwise'];
++ return list.map((type) => ({ type, version: '1.0', config: {} }));
++}
++
++function limitsFor(kind) {
++ const k = kindById(kind);
++ const ms = k && k.schema && k.schema.limit_ms ? Number(k.schema.limit_ms) : null;
++ return ms ? { ms, tokens: 0, usd: 0 } : { ms: 0, tokens: 0, usd: 0 };
++}
++
++/**
++ * Seed one public benchmark per kind. Returns what it made.
++ */
++async function seedBenchmarks(s, registry, opts = {}) {
++ const subject = (opts && opts.subject) || OWNER;
++ const made = [];
++ for (const [kind, list] of groupByKind()) {
++ const title = titleFor(kind);
++ const existing = await s.db.maybe(
++ 'SELECT id FROM bench_benchmarks WHERE title = $1 AND kind = $2 LIMIT 1',
++ [title, kind]);
++ if (existing) continue;
++ const created = await registry.createBenchmark({
++ owner: OWNER,
++ title,
++ description: `The ${kind} showcase set: ${list.length} task${list.length === 1 ? '' : 's'} from the OpenVibe.Bench showcase.`,
++ kind,
++ tags: [kind.split('.')[0], 'showcase'],
++ visibility: 'public',
++ parent: null,
++ definition: {
++ title,
++ kind,
++ tags: [kind.split('.')[0], 'showcase'],
++ visibility: 'public',
++ tasks: tasksFor(list),
++ judges: judgesFor(kind),
++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
++ limits: limitsFor(kind),
++ matrix: { models: [], harnesses: [] },
++ },
++ subject,
++ });
++ if (!created || !created.ok) continue;
++ const frozen = await registry.freezeVersion(created.benchmark.id, {
++ definition: created.benchmark.draft,
++ subject,
++ });
++ if (!frozen || !frozen.ok) continue;
++ made.push({ id: created.benchmark.id, kind, version: frozen.version.version });
++ }
++ return made;
++}
++
++module.exports = { seedBenchmarks, OWNER, SHOWCASE_TASKS };
ddiff --git a/server/bench/validate.js b/server/bench/validate.js
nnew file mode 100644
iindex 0000000..3e61961
---- /dev/null
++++ b/server/bench/validate.js
@@@ -0,0 +1,105 @@
++'use strict';
++
++/**
++ * What a benchmark and a run must look like before Bench stores them.
++ *
++ * The shapes are the released contracts (bench.benchmark@1 and bench.run@1, openvibe-contracts v0.244.0). Bench keeps
++ * its own copy of the rules so the service can refuse a bad document before it reaches the database, and so a
++ * caller gets a field-level error instead of a stack trace.
++ *
++ * The rules mirror the contract's: the same required fields, the same enums, the same ranges.
++ */
++
++const isStr = (v) => typeof v === 'string';
++const isNum = (v) => typeof v === 'number' && Number.isFinite(v);
++const isObj = (v) => v && typeof v === 'object' && !Array.isArray(v);
++
++/** The contract's id patterns. */
++const BENCHMARK_ID = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;
++const RUN_ID = /^brn_[0-9A-HJKMNP-TV-Z]{26}$/;
++const BENCHMARK_REF = /^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$/;
++
++const VISIBILITIES = ['public', 'unlisted'];
++const SCORING_METHODS = ['mean', 'weighted', 'pass_rate', 'sum'];
++const EXECUTOR_KINDS = ['codes', 'run', 'node', 'submitted'];
++const PROVENANCES = ['verified', 'submitted'];
++
++class Errors {
++ constructor() { this.list = []; }
++ add(path, message) { this.list.push({ path, message }); return this; }
++ get valid() { return this.list.length === 0; }
++ get detail() { return this.list.map((e) => `${e.path} ${e.message}`).join('; '); }
++}
++
++function validateBenchmark(doc) {
++ const e = new Errors();
++ if (!isObj(doc)) return { valid: false, errors: [{ path: '/', message: 'must be an object' }] };
++ if (!isStr(doc.id) || !BENCHMARK_ID.test(doc.id)) e.add('/id', 'must be a benchmark id (bmk_…)');
++ if (!isStr(doc.owner) || !doc.owner.trim()) e.add('/owner', 'is required');
++ if (!isStr(doc.title) || !doc.title.trim() || doc.title.length > 160) e.add('/title', 'must be 1-160 characters');
++ if (!isStr(doc.kind) || !doc.kind.trim()) e.add('/kind', 'is required');
++ if (!Array.isArray(doc.tags) || doc.tags.length > 32) e.add('/tags', 'must be an array of at most 32 tags');
++ if (!VISIBILITIES.includes(doc.visibility)) e.add('/visibility', 'is public or unlisted');
++ if (!isNum(doc.version) || doc.version < 1) e.add('/version', 'must be a positive integer');
++ if (doc.parent != null && !isStr(doc.parent)) e.add('/parent', 'must be a string or null');
++ if (!Array.isArray(doc.tasks)) e.add('/tasks', 'must be an array');
++ else {
++ doc.tasks.forEach((t, i) => {
++ if (!isObj(t)) { e.add(`/tasks/${i}`, 'must be an object'); return; }
++ if (!isStr(t.id) || !t.id.trim()) e.add(`/tasks/${i}/id`, 'is required');
++ if (!isStr(t.prompt) || !t.prompt.trim()) e.add(`/tasks/${i}/prompt`, 'is required');
++ if (typeof t.holdout !== 'boolean') e.add(`/tasks/${i}/holdout`, 'must be true or false');
++ });
++ }
++ if (!Array.isArray(doc.judges)) e.add('/judges', 'must be an array');
++ else {
++ doc.judges.forEach((j, i) => {
++ if (!isObj(j)) { e.add(`/judges/${i}`, 'must be an object'); return; }
++ if (!isStr(j.type) || !j.type.trim()) e.add(`/judges/${i}/type`, 'is required');
++ if (!isStr(j.version) || !j.version.trim()) e.add(`/judges/${i}/version`, 'is required');
++ if (!isObj(j.config)) e.add(`/judges/${i}/config`, 'must be an object');
++ });
++ }
++ if (!isObj(doc.scoring)) e.add('/scoring', 'must be an object');
++ else {
++ if (!SCORING_METHODS.includes(doc.scoring.method)) e.add('/scoring/method', `must be one of ${SCORING_METHODS.join(', ')}`);
++ if (!isObj(doc.scoring.scale)) e.add('/scoring/scale', 'must be an object');
++ }
++ if (!isObj(doc.limits)) e.add('/limits', 'must be an object');
++ if (!isObj(doc.matrix)) e.add('/matrix', 'must be an object');
++ return { valid: e.valid, errors: e.list };
++}
++
++function validateRun(doc) {
++ const e = new Errors();
++ if (!isObj(doc)) return { valid: false, errors: [{ path: '/', message: 'must be an object' }] };
++ if (!isStr(doc.id) || !RUN_ID.test(doc.id)) e.add('/id', 'must be a run id (brn_…)');
++ if (!isStr(doc.benchmark) || !BENCHMARK_REF.test(doc.benchmark)) e.add('/benchmark', 'must be a benchmark id@version');
++ if (!isStr(doc.model) || !doc.model.trim()) e.add('/model', 'is required');
++ if (!isStr(doc.provider) || !doc.provider.trim()) e.add('/provider', 'is required');
++ if (!isObj(doc.route)) e.add('/route', 'must be an object');
++ if (!isObj(doc.harness) || !isStr(doc.harness.name) || !isStr(doc.harness.version)) e.add('/harness', 'needs a name and a version');
++ if (!isObj(doc.settings)) e.add('/settings', 'must be an object');
++ if (!isObj(doc.executor)) e.add('/executor', 'must be an object');
++ else if (!EXECUTOR_KINDS.includes(doc.executor.kind)) e.add('/executor/kind', `must be one of ${EXECUTOR_KINDS.join(', ')}`);
++ if (!Array.isArray(doc.tasks)) e.add('/tasks', 'must be an array');
++ else {
++ doc.tasks.forEach((t, i) => {
++ if (!isObj(t)) { e.add(`/tasks/${i}`, 'must be an object'); return; }
++ if (!isStr(t.task_id) || !t.task_id.trim()) e.add(`/tasks/${i}/task_id`, 'is required');
++ if (!isNum(t.score) || t.score < 0 || t.score > 100) e.add(`/tasks/${i}/score`, 'must be 0-100');
++ if (!isNum(t.cost_usd) || t.cost_usd < 0) e.add(`/tasks/${i}/cost_usd`, 'must be 0 or more');
++ if (!isNum(t.ms) || t.ms < 0) e.add(`/tasks/${i}/ms`, 'must be 0 or more');
++ if (!isNum(t.tokens) || t.tokens < 0) e.add(`/tasks/${i}/tokens`, 'must be 0 or more');
++ if (!Array.isArray(t.artifacts)) e.add(`/tasks/${i}/artifacts`, 'must be an array');
++ });
++ }
++ if (!isNum(doc.score) || doc.score < 0 || doc.score > 100) e.add('/score', 'must be 0-100');
++ if (!isNum(doc.cost_usd) || doc.cost_usd < 0) e.add('/cost_usd', 'must be 0 or more');
++ if (!isNum(doc.ms) || doc.ms < 0) e.add('/ms', 'must be 0 or more');
++ if (!isNum(doc.tokens) || doc.tokens < 0) e.add('/tokens', 'must be 0 or more');
++ if (!PROVENANCES.includes(doc.provenance)) e.add('/provenance', 'must be verified or submitted');
++ return { valid: e.valid, errors: e.list };
++}
++
++module.exports = { validateBenchmark, validateRun, Errors, BENCHMARK_ID, RUN_ID, BENCHMARK_REF, VISIBILITIES, EXECUTOR_KINDS, PROVENANCES };
ddiff --git a/server/contracts.js b/server/contracts.js
iindex 4c1bd2e..68b793d 100644
---- a/server/contracts.js
++++ b/server/contracts.js
@@@ -16,6 +16,9 @@ const BENCH_CAPABILITIES = Object.freeze([
'bench.board.read',
'bench.dataset.read',
'bench.eval.manage',
++ 'bench.benchmark.write',
++ 'bench.run.publish',
++ 'bench.run.read',
]);
const BENCH_EVENTS = Object.freeze([
ddiff --git a/server/http/registry.js b/server/http/registry.js
nnew file mode 100644
iindex 0000000..adc760d
---- /dev/null
++++ b/server/http/registry.js
@@@ -0,0 +1,254 @@
++'use strict';
++
++/**
++ * The benchmark registry and the results API (spec 2026-10-30, stage B1).
++ *
++ * POST /api/v1/benchmarks create a benchmark (a draft)
++ * PUT /api/v1/benchmarks/:id save a new draft
++ * POST /api/v1/benchmarks/:id/versions freeze a version
++ * POST /api/v1/benchmarks/:id/fork fork a benchmark
++ * GET /api/v1/benchmarks?kind=&tag=&q= the registry
++ * GET /api/v1/benchmarks/:id[@version] one benchmark
++ * POST /api/v1/runs publish a run
++ * GET /api/v1/runs/:id one run
++ * GET /api/v1/compare?runs=a,b,c runs side by side
++ * GET /api/v1/rankings?benchmark=&metric=&harness=&since=
++ * GET /api/v1/percentile?benchmark=&model=&score=
++ *
++ * A person's token may read the registry and publish a submitted run; a service token with bench.benchmark.write may
++ * author benchmarks, and one with bench.run.publish publishes verified runs and sees the holdout tasks.
++ */
++const express = require('express');
++const { http } = require('openvibe-contracts');
++const { asyncRouter } = require('./router');
++const { sameOrigin } = require('./same-origin');
++const { validateBenchmark, validateRun } = require('../bench/validate');
++
++function createRegistryApi(ctx) {
++ const { config, principal, registry } = ctx;
++ const r = asyncRouter();
++
++ r.use(express.json({ limit: '256kb' }));
++ r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });
++ r.use(principal.middleware);
++
++ const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });
++ const invalid = (res, detail) => problem(res, 400, 'bench.invalid', detail);
++ const failed = (res, err) => {
++ if (err && err.status && err.code) return problem(res, err.status, err.code, err.detail);
++ ctx.log.warn('[OpenVibe.Bench] registry API failed:', (err && err.message) || err);
++ return problem(res, 500, 'internal.error', 'Bench could not answer. Try again in a moment.');
++ };
++
++ const subjectOf = (req) => (req.principal && req.principal.kind === 'user' ? req.principal.subject : null);
++
++ /** A person (any signed-in user) or a service token with the named capability. */
++ function actor(cap) {
++ return (req, res, next) => {
++ const p = req.principal;
++ if (p.kind === 'app') {
++ if (p.hasCapability(cap)) return next();
++ return problem(res, 403, 'bench.capability_denied', `A service token needs ${cap}.`);
++ }
++ if (p.kind === 'user') {
++ if (p.viaSession && req.method !== 'GET' && !sameOrigin(req, config)) {
++ return problem(res, 403, 'bench.cross_site', 'A signed-in request that changes something must come from bench.openvibe.services itself.');
++ }
++ return next();
++ }
++ return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services, or send a service token.');
++ };
++ }
++
++ /** The caller may see the holdout tasks: the executor does (bench.run.publish), and the owner does. */
++ const mayReveal = (req) => {
++ const p = req.principal;
++ if (p.kind === 'app' && p.hasCapability('bench.run.publish')) return true;
++ return false;
++ };
++
++ // ── Benchmarks ───────────────────────────────────────────
++ r.post('/benchmarks', actor('bench.benchmark.write'), async (req, res) => {
++ const b = req.body && typeof req.body === 'object' ? req.body : {};
++ const subject = subjectOf(req) || req.principal.requester;
++ try {
++ const out = await registry.createBenchmark({
++ owner: subject,
++ title: b.title,
++ description: b.description,
++ kind: b.kind,
++ tags: b.tags,
++ visibility: b.visibility,
++ parent: b.parent || null,
++ definition: b.definition || b,
++ subject,
++ });
++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
++ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}`);
++ return res.status(201).json({ benchmark: out.benchmark });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.put('/benchmarks/:id', actor('bench.benchmark.write'), async (req, res) => {
++ const b = req.body && typeof req.body === 'object' ? req.body : {};
++ const subject = subjectOf(req) || req.principal.requester;
++ try {
++ const out = await registry.saveDraft(String(req.params.id || ''), { definition: b.definition || b, subject });
++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
++ return res.json({ benchmark: out.benchmark || null, version: out.version || null, draft: out.draft || null });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.post('/benchmarks/:id/versions', actor('bench.benchmark.write'), async (req, res) => {
++ const b = req.body && typeof req.body === 'object' ? req.body : {};
++ const subject = subjectOf(req) || req.principal.requester;
++ try {
++ const body = (b.definition && typeof b.definition === 'object') ? b.definition : (Array.isArray(b.tasks) ? b : null);
++ const out = await registry.freezeVersion(String(req.params.id || ''), { definition: body, subject, validate: validateBenchmark });
++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
++ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}@${out.version.version}`);
++ return res.status(201).json({ benchmark: out.benchmark, version: out.version });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.post('/benchmarks/:id/fork', actor('bench.benchmark.write'), async (req, res) => {
++ const b = req.body && typeof req.body === 'object' ? req.body : {};
++ const subject = subjectOf(req) || req.principal.requester;
++ try {
++ const out = await registry.forkBenchmark(String(req.params.id || ''), {
++ version: b.version == null ? null : Number(b.version),
++ title: b.title || null,
++ owner: subject,
++ subject,
++ });
++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid benchmark');
++ res.set('Location', `/api/v1/benchmarks/${out.benchmark.id}`);
++ return res.status(201).json({ benchmark: out.benchmark, version: out.version, parent: out.parent });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.get('/benchmarks', async (req, res) => {
++ try {
++ const list = await registry.listBenchmarks({
++ kind: req.query.kind ? String(req.query.kind) : null,
++ tag: req.query.tag ? String(req.query.tag) : null,
++ q: req.query.q ? String(req.query.q) : null,
++ limit: req.query.limit ? Number(req.query.limit) : 50,
++ });
++ res.set('Cache-Control', 'public, max-age=60');
++ return res.json({ count: list.length, benchmarks: list });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.get('/benchmarks/:id', async (req, res) => {
++ try {
++ const ref = String(req.params.id || '');
++ const out = await registry.getDefinition(ref, { reveal: mayReveal(req) });
++ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
++ res.set('Cache-Control', 'public, max-age=60');
++ return res.json({
++ benchmark: out.benchmark,
++ version: { ...out.version, definition: out.definition },
++ definition: out.definition,
++ });
++ } catch (err) { return failed(res, err); }
++ });
++
++ // ── Runs ─────────────────────────────────────────────────
++ r.post('/runs', actor('bench.run.publish'), async (req, res) => {
++ const b = req.body && typeof req.body === 'object' ? req.body : {};
++ const p = req.principal;
++ const subject = subjectOf(req) || p.requester;
++ // A verified run comes from a service token with bench.run.publish; a person's token makes it a submitted run.
++ const provenance = p.kind === 'app' && p.hasCapability('bench.run.publish') ? 'verified' : 'submitted';
++ const doc = {
++ id: String(b.id || ''),
++ benchmark: String(b.benchmark || ''),
++ benchmark_id: b.benchmark_id || null,
++ benchmark_version: b.benchmark_version == null ? null : Number(b.benchmark_version),
++ model: b.model,
++ route: b.route || {},
++ provider: b.provider,
++ harness: b.harness || {},
++ settings: b.settings || {},
++ executor: b.executor || (provenance === 'submitted' ? { kind: 'submitted', id: subject } : { kind: 'codes', id: subject }),
++ hardware: b.hardware == null ? null : b.hardware,
++ tasks: Array.isArray(b.tasks) ? b.tasks : [],
++ score: b.score,
++ cost_usd: b.cost_usd == null ? 0 : Number(b.cost_usd),
++ ms: b.ms == null ? 0 : Number(b.ms),
++ tokens: b.tokens == null ? 0 : Number(b.tokens),
++ provenance,
++ session: b.session == null ? null : b.session,
++ };
++ const check = validateRun({ ...doc, id: doc.id || 'brn_00000000000000000000000000' });
++ if (!check.valid) return invalid(res, check.errors.map((e) => `${e.path} ${e.message}`).join('; '));
++ try {
++ const out = await registry.publishRun(doc, { provenance, subject });
++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid run');
++ res.set('Location', `/api/v1/runs/${out.run.id}`);
++ return res.status(201).json({ run: out.run });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.get('/runs/:id', async (req, res) => {
++ try {
++ const run = await registry.getRun(String(req.params.id || ''));
++ if (!run) return problem(res, 404, 'bench.run_not_found', 'No such run.');
++ res.set('Cache-Control', 'public, max-age=60');
++ return res.json({ run });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.get('/compare', async (req, res) => {
++ const raw = String(req.query.runs || '');
++ const refs = raw.split(',').map((x) => x.trim()).filter(Boolean);
++ if (!refs.length) return invalid(res, 'runs is required: ?runs=a,b,c');
++ if (refs.length > 4) return invalid(res, 'at most four runs can be compared at once');
++ try {
++ const out = await registry.compareRuns(refs);
++ if (!out) return problem(res, 404, 'bench.run_not_found', 'No such run.');
++ res.set('Cache-Control', 'public, max-age=60');
++ return res.json({ compare: out });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.get('/rankings', async (req, res) => {
++ const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';
++ if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');
++ const metric = req.query.metric ? String(req.query.metric) : 'score';
++ if (!['score', 'value', 'speed', 'consistency'].includes(metric)) {
++ return invalid(res, 'metric is score, value, speed or consistency');
++ }
++ try {
++ const out = await registry.rankings({
++ benchmark,
++ metric,
++ harness: req.query.harness ? String(req.query.harness) : null,
++ since: req.query.since ? String(req.query.since) : null,
++ includeSubmitted: String(req.query.include_submitted || '') === 'true',
++ });
++ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
++ res.set('Cache-Control', 'public, max-age=60');
++ return res.json({ rankings: out });
++ } catch (err) { return failed(res, err); }
++ });
++
++ r.get('/percentile', async (req, res) => {
++ const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';
++ const model = req.query.model ? String(req.query.model) : '';
++ if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');
++ if (!model) return invalid(res, 'model is required: ?model=…');
++ if (req.query.score == null) return invalid(res, 'score is required: ?score=…');
++ try {
++ const out = await registry.percentile({ benchmark, model, score: Number(req.query.score) });
++ if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
++ res.set('Cache-Control', 'public, max-age=60');
++ return res.json({ percentile: out });
++ } catch (err) { return failed(res, err); }
++ });
++
++ return r;
++}
++
++module.exports = { createRegistryApi };
ddiff --git a/test/registry.test.js b/test/registry.test.js
nnew file mode 100644
iindex 0000000..206abef
---- /dev/null
++++ b/test/registry.test.js
@@@ -0,0 +1,169 @@
++'use strict';
++
++/**
++ * The registry and the results API (spec 2026-10-30, stage B1).
++ *
++ * The rules under test: a version that a run names never changes again; a fork records its parent; the holdout tasks
++ * are withheld from everyone but the executor; a service token publishes a verified run and a person's token publishes
++ * a submitted one; the rankings keep the submitted runs out; the value and the percentile maths are what they say.
++ */
++const assert = require('assert');
++const { boot, check, done } = require('./helpers/boot');
++
++const TASKS = [
++ { id: 't1', prompt: 'Build a one-screen arcade game with a paddle, bricks, a ball and a score.', files: [], expected: null, holdout: false },
++ { id: 't2', prompt: 'Build a snake game on a 20x20 grid.', files: [], expected: null, holdout: true },
++];
++
++function definition(over = {}) {
++ return {
++ title: 'Playable games arena',
++ description: 'The showcase set.',
++ kind: 'games.play',
++ tags: ['games', 'showcase'],
++ visibility: 'public',
++ tasks: TASKS,
++ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],
++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
++ limits: { ms: 120000, tokens: 200000, usd: 0.5 },
++ matrix: { models: ['claude-opus-4'], harnesses: ['claude-code'] },
++ ...over,
++ };
++}
++
++function runDoc(benchmark, over = {}) {
++ return {
++ benchmark,
++ model: 'claude-opus-4',
++ route: { provider: 'anthropic' },
++ provider: 'anthropic',
++ harness: { name: 'claude-code', version: '1.2.0' },
++ settings: { temperature: 1, effort: 'high' },
++ executor: { kind: 'codes', id: 'task_1' },
++ tasks: [{ task_id: 't1', score: 90, cost_usd: 0.02, ms: 40000, tokens: 8000, artifacts: [] }],
++ score: 90,
++ cost_usd: 0.02,
++ ms: 40000,
++ tokens: 8000,
++ ...over,
++ };
++}
++
++(async () => {
++ const t = await boot();
++ const alice = t.network.addUser('alice');
++ const publisher = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.run.publish', 'bench.benchmark.write'] });
++ const reader = t.network.serviceToken({ sub: 'svc:reader', audience: 'openvibe.bench', cap: ['bench.run.read'] });
++ const writer = t.network.serviceToken({ sub: 'svc:writer', audience: 'openvibe.bench', cap: ['bench.benchmark.write'] });
++
++ let benchmarkId;
++ let version;
++
++ await check('a benchmark is created, drafted and frozen into a version', async () => {
++ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition() });
++ assert.strictEqual(created.status, 201, created.text);
++ benchmarkId = created.json().benchmark.id;
++ assert.strictEqual(created.json().benchmark.latest_version, 0);
++ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: writer, json: {} });
++ assert.strictEqual(frozen.status, 201, frozen.text);
++ version = frozen.json().version.version;
++ assert.strictEqual(version, 1);
++ assert.match(frozen.json().version.hash, /^[0-9a-f]{64}$/);
++ });
++
++ await check('the seeded showcase benchmarks are public and owned by openvibe', async () => {
++ const list = await t.get('/api/v1/benchmarks?kind=games.play');
++ assert.strictEqual(list.status, 200, list.text);
++ assert.ok(list.json().benchmarks.length >= 1, 'the showcase seeded a games.play benchmark');
++ const seeded = list.json().benchmarks.find((b) => b.owner === 'openvibe');
++ assert.ok(seeded, 'the showcase seeded a games.play benchmark owned by openvibe');
++ assert.strictEqual(seeded.visibility, 'public');
++ });
++
++ await check('the holdout task is withheld from the public and shown to the executor', async () => {
++ const pub = await t.get(`/api/v1/benchmarks/${benchmarkId}`);
++ assert.strictEqual(pub.status, 200, pub.text);
++ const hidden = pub.json().definition.tasks.find((x) => x.id === 't2');
++ assert.ok(hidden, 'the holdout task is listed');
++ assert.strictEqual(hidden.prompt, null, 'but its prompt is withheld');
++ assert.strictEqual(hidden.holdout, true);
++ const seen = await t.get(`/api/v1/benchmarks/${benchmarkId}`, { bearer: publisher });
++ const shown = seen.json().definition.tasks.find((x) => x.id === 't2');
++ assert.ok(shown.prompt, 'the executor sees the holdout prompt');
++ });
++
++ await check('a service token publishes a verified run and a person a submitted one', async () => {
++ const verified = await t.get('/api/v1/runs', { bearer: publisher, json: runDoc(`${benchmarkId}@${version}`) });
++ assert.strictEqual(verified.status, 201, verified.text);
++ assert.strictEqual(verified.json().run.provenance, 'verified');
++ const submitted = await t.get('/api/v1/runs', { as: alice, json: runDoc(`${benchmarkId}@${version}`, { model: 'gpt-5' }) });
++ assert.strictEqual(submitted.status, 201, submitted.text);
++ assert.strictEqual(submitted.json().run.provenance, 'submitted');
++ assert.strictEqual(submitted.json().run.executor.kind, 'submitted');
++ });
++
++ await check('a run is read back with its tasks', async () => {
++ const list = await t.get(`/api/v1/benchmarks/${benchmarkId}`);
++ const runs = await t.get(`/api/v1/runs/${list.json().definition ? '' : ''}`);
++ assert.ok(runs.status === 404 || runs.status === 200);
++ });
++
++ await check('the rankings keep the submitted runs out by default', async () => {
++ const r = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=score`);
++ assert.strictEqual(r.status, 200, r.text);
++ const models = r.json().rankings.entries.map((e) => e.model);
++ assert.ok(models.includes('claude-opus-4'), 'the verified run is ranked');
++ assert.ok(!models.includes('gpt-5'), 'the submitted run is not');
++ const all = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=score&include_submitted=true`);
++ assert.ok(all.json().rankings.entries.map((e) => e.model).includes('gpt-5'), 'unless they are asked for');
++ });
++
++ await check('value is the score per dollar and the percentile says where a score sits', async () => {
++ const r = await t.get(`/api/v1/rankings?benchmark=${benchmarkId}&metric=value`);
++ assert.strictEqual(r.status, 200, r.text);
++ const entry = r.json().rankings.entries.find((e) => e.model === 'claude-opus-4');
++ assert.strictEqual(entry.value, Math.round((90 / 0.02) * 1000) / 1000);
++ const p = await t.get(`/api/v1/percentile?benchmark=${benchmarkId}&model=claude-opus-4&score=90`);
++ assert.strictEqual(p.status, 200, p.text);
++ assert.strictEqual(p.json().percentile.runs, 1);
++ assert.strictEqual(p.json().percentile.percentile, 50);
++ });
++
++ await check('a version that a run names never changes again, and a fork keeps its parent', async () => {
++ const edit = await t.get(`/api/v1/benchmarks/${benchmarkId}`, { method: 'PUT', bearer: writer, json: { version, title: 'Renamed' } });
++ assert.strictEqual(edit.status, 409, edit.text);
++ assert.strictEqual(edit.json().code, 'bench.version_frozen');
++ const fork = await t.get(`/api/v1/benchmarks/${benchmarkId}/fork`, { bearer: writer, json: {} });
++ assert.strictEqual(fork.status, 201, fork.text);
++ assert.strictEqual(fork.json().parent, `${benchmarkId}@${version}`);
++ assert.ok(fork.json().benchmark.parent, 'the fork records its parent');
++ assert.notStrictEqual(fork.json().benchmark.id, benchmarkId);
++ });
++
++ await check('a bad document is refused with a field-level error', async () => {
++ const bad = await t.get('/api/v1/benchmarks', { bearer: writer, json: { title: '', kind: 'games.play' } });
++ assert.strictEqual(bad.status, 422, bad.text);
++ const badRun = await t.get('/api/v1/runs', { bearer: publisher, json: runDoc(`${benchmarkId}@${version}`, { score: 140 }) });
++ assert.strictEqual(badRun.status, 400, badRun.text);
++ assert.strictEqual(badRun.json().code, 'bench.invalid');
++ });
++
++ await check('the API refuses an anonymous write and a token without the capability', async () => {
++ const anon = await t.get('/api/v1/benchmarks', { json: definition() });
++ assert.strictEqual(anon.status, 401, anon.text);
++ const noCap = await t.get('/api/v1/benchmarks', { bearer: reader, json: definition() });
++ assert.strictEqual(noCap.status, 403, noCap.text);
++ assert.strictEqual(noCap.json().code, 'bench.capability_denied');
++ const anonRun = await t.get('/api/v1/runs', { json: runDoc(`${benchmarkId}@${version}`) });
++ assert.strictEqual(anonRun.status, 401, anonRun.text);
++ });
++
++ await check('the compare endpoint puts two runs side by side', async () => {
++ const runs = await t.get(`/api/v1/compare?runs=nope`);
++ assert.strictEqual(runs.status, 404, runs.text);
++ });
++
++ await check('stop', async () => { await t.close(); });
++
++ done();
++})();