openvibers/openvibe.bench Commits
Your own prompts: a private benchmark from a paste, run it, share it (#41)
Changes
@@@ -12,11 +12,24 @@ capabilities. A session write must come from bench.openvibe.services itself.| `PUT /api/v1/benchmarks/:id` | the same | Save a new draft. A body with `version` edits that version, and is refused once a run names it. || `POST /api/v1/benchmarks/:id/versions` | the same | Freeze the working copy into the next immutable version. || `POST /api/v1/benchmarks/:id/fork` | the same | Fork a benchmark; the fork records its parent. |--| `GET /api/v1/benchmarks?kind=&tag=&q=` | anyone | The registry. |--| `GET /api/v1/benchmarks/:id[@version]` | anyone | One benchmark and its definition. Holdout tasks are withheld unless the caller holds `bench.run.publish` or owns the benchmark. |++| `POST /api/v1/benchmarks/:id/share` | its owner | `{ visibility: "private" | "unlisted" | "public" }`. `unlisted` returns a new share token once; the old link stops. |++| `GET /api/v1/benchmarks?kind=&tag=&q=&mine=1` | anyone | The registry; `mine=1` adds the caller's own unlisted and private benchmarks. |++| `GET /api/v1/benchmarks/:id[@version]` | anyone who may read it | One benchmark and its definition. A private benchmark is a 404 except to its owner and the executor; a shared unlisted one needs its link. Holdout tasks are withheld unless the caller holds `bench.run.publish` or owns the benchmark. |--The documents are the published contracts, from `openvibe-contracts` v0.244.0: `bench.benchmark@1` for a--benchmark version, `bench.run@1` for a run. Every document is checked against them before it is stored.++The documents are the published contracts, from `openvibe-contracts`: `bench.benchmark@1` for a benchmark++version, `bench.run@1` for a run. Every contract document is checked against them before it is stored. `private`++is Bench's own stricter form of unlisted: the contract document is frozen as `unlisted`, and the benchmark row's++visibility is what every access check reads.++++### Your own prompts++++`POST /benchmarks/new` on the site (or the API above) makes a benchmark from 5 to 50 pasted prompts: one per line,++or CSV with an optional expected answer. The default kind is `text.answer`; any existing kind may be chosen. It is++private by default, exports and deletes with the account, and its runs go through the existing openvibe.codes launcher++(the same estimate and confirmation).++++The open data: `GET /api/v1/datasets/benchmarks.jsonl` carries every public benchmark, its visible tasks and every++run under it, with no person id. Nothing private or unlisted is ever read there.## The results@@@ -13,7 +13,7 @@ A benchmark (the contract is `bench.benchmark@1`) holds:| `title`, `description` | What the set is and who it is for. || `kind` | One of Bench's kinds (`games.play`, `design.ui`, `codes.patch`, …). The kind sets the renderer. || `tags` | Up to 32 words people search by. |--| `visibility` | `public` (in the registry) or `unlisted` (by id only). |++| `visibility` | `public` (in the registry), `unlisted` (reachable by its id; a share link when one is made), or `private` (its owner only). A pasted set of your own prompts starts private. || `version` | The version this document is. Bench numbers them. || `parent` | The `id@version` this one was forked from, or `null`. || `tasks[]` | `{ id, prompt, files?, expected?, holdout }`. The prompt is the task; `holdout: true` hides it from everyone but the executor. |@@@ -56,6 +56,16 @@ published set, fork it — the fork records the parent, and the old version stay}```++## Your own prompts++++A person can paste 5 to 50 of their own prompts at `/benchmarks/new`: one per line, or CSV with an optional++expected answer in the second column. They become the tasks of a private benchmark (`text.answer`, or any existing++kind), frozen as version 1. It is private by default: the owner alone sees the prompts, and another person gets a++404. The owner may share an unlisted link (a 256-bit token, stored as a hash and checked in constant time), make it++public (then it is in the lists, the sitemap and `/api/v1/datasets/benchmarks.jsonl`), or make it private again (the++link stops working). Runs still go through the openvibe.codes launcher: the same estimate, the same "ask before it++starts".++## The contractsBench speaks the published shapes, from `openvibe-contracts` **v0.244.0**:nnew file mode 100644@@@ -0,0 +1,19 @@++-- phase: expand++-- B11: a person's own prompts as a private benchmark (spec "Your own prompts", 2026-10-30).++--++-- A benchmark is private until its owner shares it. `visibility` gains `private`; the check is widened, never++-- narrowed, so an older writer that only knows public and unlisted keeps working.++--++-- An unlisted share carries a long random token in its link. Only the token's SHA-256 is stored: a database read++-- never yields a working link. The token is checked in constant time (server/bench/registry.js) and cleared when++-- the benchmark becomes private or public, so an old link stops working.++ALTER TABLE bench_benchmarks DROP CONSTRAINT bench_benchmarks_visibility;++ALTER TABLE bench_benchmarks ADD CONSTRAINT bench_benchmarks_visibility++ CHECK (visibility IN ('public', 'unlisted', 'private'));++++ALTER TABLE bench_benchmarks ADD COLUMN share_token_hash text;++ALTER TABLE bench_benchmarks ADD CONSTRAINT bench_benchmarks_share_token_hash++ CHECK (share_token_hash IS NULL OR share_token_hash ~ '^[0-9a-f]{64}$');++-- At most one benchmark per token, so a hash collision cannot make a link ambiguous.++CREATE UNIQUE INDEX bench_benchmarks_share_token ON bench_benchmarks (share_token_hash)++ WHERE share_token_hash IS NOT NULL;@@@ -420,3 +420,76 @@ html, body { margin: 0; background: var(--bg-primary, #0a0f1c); color: var(--tex.br-usethis-row { display: grid; gap: 6px; min-width: 0; }.br-usethis-row textarea { width: 100%; min-height: 44px; padding: 10px 12px; border-radius: 12px; border: 1px solid var(--border); background: var(--bg-primary); color: var(--text-secondary); font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: .8rem; line-height: 1.45; resize: vertical; }.br-follow { margin: 0; }++++/* ---- Your own prompts (B11): the paste form, the share box and the two leaderboards ---- */++.br-new-form textarea, .br-new-form .br-field textarea {++ width: 100%;++ min-height: 44px;++ padding: 10px 12px;++ border-radius: 12px;++ border: 1px solid var(--border);++ background: var(--bg-input, var(--bg-primary));++ color: var(--text-primary);++ font: inherit;++ line-height: 1.5;++ resize: vertical;++}++.br-new-form textarea:focus-visible { outline: 2px solid var(--room-accent); outline-offset: 1px; }++.br-new-form .br-prompts { min-height: 15rem; font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: .9rem; }++.br-new-form .br-paste { min-width: 0; }++.br-new-form .br-field { min-width: 0; }++.br-vis { text-transform: lowercase; }++.br-mine { margin-top: clamp(24px, 4vw, 36px); }++.br-mine .bm-head-row, .br-share .bm-head-row { flex-wrap: wrap; }++.br-mine-hint { margin: 14px 0 0; }++.br-private { border-color: color-mix(in srgb, var(--room-accent) 40%, transparent); }++.br-share { padding: 16px; border: 1px solid var(--border); border-radius: 18px; background: var(--bg-card); }++.br-share-form { display: flex; flex-wrap: wrap; gap: 10px; margin: 14px 0 8px; }++.br-share-now { display: grid; gap: 10px; }++.br-share-now p { margin: 0; }++.br-share-link { width: 100%; min-width: 0; min-height: 44px; padding: 0 12px; border-radius: 12px; border: 1px solid var(--border); background: var(--bg-input, var(--bg-primary)); color: var(--text-primary); font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: .82rem; }++.br-duo { display: grid; gap: 18px; grid-template-columns: 1fr; align-items: start; }++.br-duo-pane { min-width: 0; padding: 16px; border: 1px solid var(--border); border-radius: 18px; background: var(--bg-card); }++.br-duo-pane .br-table-wrap { background: transparent; }++.br-duo-pane .br-chart { padding: 10px; }++.br-duo-pane .bm-empty { background: transparent; }++.br-prompt-table .rt-col-model { width: auto; }++.br-spread { display: block; margin-top: 4px; white-space: nowrap; }++.br-prompt-table .rt-col-raw { width: 9em; }++.br-prompt-table .rt-col-band { width: 6.5em; }++@media (min-width: 900px) {++ .br-duo { grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); }++}++@media (max-width: 560px) {++ /* The shared ratings fold: score and cost/solved on the second line, as a band + model + score + cost + runs. */++ .br-prompt-table.rt-table tr { grid-template-columns: minmax(0, 1fr) minmax(0, 1fr) 3.2em; }++ .br-prompt-table .rt-col-controlled { grid-area: 2 / 1 / 3 / 2; }++ .br-prompt-table .rt-col-raw { grid-area: 2 / 2 / 3 / 3; width: auto; }++ .br-prompt-table td.rt-col-votes { grid-area: 2 / 3 / 3 / 4; text-align: right; }++ .br-prompt-table .br-spread { white-space: normal; }++ .br-share { padding: 14px; }++}++.bm-page-h > .bm-head-row { flex-wrap: wrap; align-items: flex-end; }++.br-new-form .br-field select, .br-new-form .br-field input[type="text"] { width: 100%; min-width: 0; }++/* The rankings table folds into two lines on a phone: band + model + score, then the numbers with their names. */++@media (max-width: 560px) {++ .br-rank-table, .br-rank-table tbody { display: block; width: 100%; }++ .br-rank-table thead { display: none; }++ .br-rank-table tr { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 12px; padding: 11px 14px; border-bottom: 1px solid var(--border); }++ .br-rank-table tbody tr:last-child { border-bottom: 0; }++ .br-rank-table td { display: block; padding: 0; border: 0; min-width: 0; width: auto; white-space: normal; }++ .br-rank-table td.br-rank { display: none; }++ .br-rank-table td.br-c-band { flex: none; order: 0; }++ .br-rank-table td.br-c-model { flex: 1 1 auto; order: 0; overflow-wrap: anywhere; }++ .br-rank-table td.br-c-score { flex: none; order: 0; text-align: right; }++ .br-rank-table tr::after { content: ""; flex-basis: 100%; height: 0; order: 1; }++ .br-rank-table td.br-c-metric, .br-rank-table td.br-c-cost, .br-rank-table td.br-c-time, .br-rank-table td.br-c-runs { flex: none; order: 2; font-size: .82rem; color: var(--text-secondary); text-align: left; }++ .br-rank-table td[data-label]::before { content: attr(data-label) " "; color: var(--text-muted); }++}++.br-prompt-table .rt-model { white-space: nowrap; }++@media (max-width: 560px) { .br-prompt-table .rt-model { white-space: normal; overflow-wrap: anywhere; } }++.br-duo-pane .br-prompt-table { font-size: .86rem; }++.br-duo-pane .br-prompt-table th, .br-duo-pane .br-prompt-table td { padding: 8px 8px; }++.br-prompt-table .rt-col-band { width: 5em; }++.br-prompt-table .rt-col-raw { width: 7em; }++.br-prompt-table .rt-col-votes { width: 3.2em; }@@@ -7,6 +7,7 @@* something the API refuses.*/const { createServiceTokenClient } = require('openvibe-sdk/auth');++const { canViewBenchmark } = require('./registry');const CONFIRM_ABOVE_USD = 1;const HARNESS_TTL_MS = 10 * 60 * 1000;@@@ -43,6 +44,15 @@ function createLauncher({ config, registry, fetchImpl = globalThis.fetch, log =}const definition = await registry.getDefinition(String(id), { reveal: false });if (!definition) return fail(404, 'bench.benchmark_not_found', 'No such benchmark.');++ // A private benchmark is its owner's: another signed-in person cannot start it, even by knowing its id.++ if (!canViewBenchmark(definition.benchmark, { subject: principal.subject })) {++ return fail(404, 'bench.benchmark_not_found', 'No such benchmark.');++ }++ // A set made from a person's own prompts is compared on 3 to 6 models, as the paste form says.++ if (Array.isArray(definition.benchmark.tags) && definition.benchmark.tags.includes('your-prompts')++ && (models.length < 3 || models.length > 6)) {++ return fail(400, 'bench.prompt_models', 'Pick 3 to 6 models for a benchmark made from your own prompts.');++ }const cost = estimate(definition.definition, { models, harnesses, repeat });if ((cost == null || cost > CONFIRM_ABOVE_USD) && confirm !== true) {return { status: 409, body: { code: 'bench.confirm_cost', cost_estimate_usd: cost, confirm_required: true, ...(cost == null ? { detail: 'This benchmark sets no dollar limit per task, so its cost depends on the models.' } : {}) } };@@@ -15,7 +15,8 @@*/const crypto = require('crypto');const { ids } = require('openvibe-contracts');-- const { contractRun, checkContract } = require('./validate');++const { contractRun, checkContract, VISIBILITIES } = require('./validate');++const { rankBands } = require('../rating/style');const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));@@@ -52,6 +53,38 @@ const json = (v, fallback) => {return v;};++/** A share token: 256 random bits in base64url (43 characters), never guessable. */++function newShareToken() {++ return crypto.randomBytes(32).toString('base64url');++}++++/** Only the SHA-256 of a share token is stored, so a database read never yields a working link. */++function shareTokenHash(token) {++ return crypto.createHash('sha256').update(String(token || ''), 'utf8').digest('hex');++}++++/** Constant-time check of a token against the stored hash: lengths first, then timingSafeEqual. */++function shareTokenMatches(token, storedHash) {++ if (!storedHash) return false;++ const got = Buffer.from(shareTokenHash(token), 'hex');++ const want = Buffer.from(String(storedHash), 'hex');++ return got.length === want.length && crypto.timingSafeEqual(got, want);++}++++/**++ * Who may read a benchmark. Public is for everyone; private is its owner (and the executor, which needs the tasks++ * to run them); unlisted is reachable by id as it always was, but a shared unlisted benchmark is token-only: its++ * id alone no longer opens it.++ */++function canViewBenchmark(benchmark, { subject = null, executor = false } = {}) {++ if (!benchmark) return false;++ const owner = subject != null && String(benchmark.owner) === String(subject);++ if (benchmark.visibility === 'public') return true;++ if (benchmark.visibility === 'private') return owner || executor;++ if (benchmark.visibility === 'unlisted' && benchmark.shared) return owner || executor;++ return true;++}++function wireBenchmark(row) {if (!row) return null;return {@@@ -62,6 +95,8 @@ function wireBenchmark(row) {kind: row.kind,tags: Array.isArray(json(row.tags, [])) ? json(row.tags, []) : [],visibility: row.visibility,++ // Whether an unlisted share link exists. The token itself is never in a wire object or a page source.++ shared: Boolean(row.share_token_hash),parent: row.parent || null,latest_version: num(row.latest_version),draft: json(row.draft, null),@@@ -172,7 +207,7 @@ function createRegistry(s, { log = console } = {}) {const kindId = String(kind == null ? '' : kind).trim();if (!kindId) return refusal(422, 'bench.kind_required', 'A benchmark needs a kind.');const vis = String(visibility || 'unlisted');-- if (!['public', 'unlisted'].includes(vis)) return refusal(422, 'bench.invalid_visibility', 'visibility is public or unlisted.');++ if (!VISIBILITIES.includes(vis)) return refusal(422, 'bench.invalid_visibility', `visibility is ${VISIBILITIES.join(', ')}.`);const list = (Array.isArray(tags) ? tags : []).map((t) => String(t).trim()).filter(Boolean).slice(0, 32);const id = s.newId('bmk');const row = await db.maybe(@@@ -254,6 +289,9 @@ function createRegistry(s, { log = console } = {}) {if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');const source = definition || benchmark.draft || {};const version = benchmark.latest_version + 1;++ // The published contract knows public and unlisted only. A private benchmark is frozen as unlisted (the++ // stricter form); the row's own visibility is what every access check reads, so its prompts stay private.++ const frozenVisibility = benchmark.visibility === 'public' ? 'public' : 'unlisted';const doc = {id: benchmark.id,owner: benchmark.owner,@@@ -261,7 +299,7 @@ function createRegistry(s, { log = console } = {}) {description: source.description == null ? benchmark.description : source.description,kind: source.kind || benchmark.kind,tags: Array.isArray(source.tags) ? source.tags : benchmark.tags,-- visibility: source.visibility || benchmark.visibility,++ visibility: frozenVisibility,version,parent: benchmark.parent,tasks: Array.isArray(source.tasks) ? source.tasks : [],@@@ -319,6 +357,85 @@ function createRegistry(s, { log = console } = {}) {return { ok: true, benchmark: await getBenchmark(made.benchmark.id), version: frozen.version, parent };}++ /**++ * Share or unshare a benchmark. The owner chooses:++ * private only they see it; no link works++ * unlisted a new random link works, and nothing lists it++ * public it is in the lists, the sitemap and the open data++ * The raw token is returned once, on the call that makes the link; only its hash is stored. Every change replaces++ * the old token, so a link from a previous share stops working.++ */++ async function setVisibility(id, { visibility, subject }) {++ const benchmark = await getBenchmark(id);++ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');++ if (String(benchmark.owner) !== String(subject)) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');++ const vis = String(visibility || '').trim();++ if (!VISIBILITIES.includes(vis)) return refusal(422, 'bench.invalid_visibility', `visibility is ${VISIBILITIES.join(', ')}.`);++ const token = vis === 'unlisted' ? newShareToken() : null;++ const row = await db.maybe(++ `UPDATE bench_benchmarks SET visibility = $1, share_token_hash = $2, updated_at = $3::timestamptz++ WHERE id = $4 RETURNING *`,++ [vis, token ? shareTokenHash(token) : null, s.iso(), benchmark.id]);++ return {++ ok: true,++ benchmark: wireBenchmark(row),++ token,++ share_path: token ? `/b/${token}` : null,++ };++ }++++ /** The benchmark an unlisted link names, checked in constant time. A private or public row has no token. */++ async function findShared(token) {++ const text = String(token || '');++ if (!/^[A-Za-z0-9_-]{20,120}$/.test(text)) return null;++ const hash = shareTokenHash(text);++ const row = await db.maybe(++ "SELECT * FROM bench_benchmarks WHERE share_token_hash = $1 AND visibility = 'unlisted'",++ [hash]);++ if (!row || !shareTokenMatches(text, row.share_token_hash)) return null;++ return wireBenchmark(row);++ }++++ /**++ * The open data for the registry: every public benchmark, its visible tasks, and every run published under it.++ * Nothing private or unlisted is ever read here; a person id (created_by, the executor's id) is left out, and a++ * holdout task keeps its prompt to the executor.++ */++ async function publicDatasetLines() {++ const benchmarks = await listBenchmarks({ limit: 200 });++ const lines = [];++ for (const benchmark of benchmarks) {++ if (benchmark.visibility !== 'public') continue;++ const version = await latestVersion(benchmark.id);++ if (!version) continue;++ const definition = version.definition || {};++ lines.push(JSON.stringify({++ type: 'benchmark', id: benchmark.id, title: benchmark.title, description: benchmark.description,++ kind: benchmark.kind, tags: benchmark.tags, version: version.version, parent: benchmark.parent,++ created_at: benchmark.created_at,++ }));++ for (const task of (definition.tasks || [])) {++ if (!task || task.holdout) continue;++ const item = { type: 'benchmark_task', benchmark: `${benchmark.id}@${version.version}`, id: task.id, prompt: task.prompt };++ if (task.expected != null) item.expected = task.expected;++ lines.push(JSON.stringify(item));++ }++ const runs = await db.many(++ 'SELECT * FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = $2 ORDER BY created_at, id',++ [benchmark.id, version.version]);++ for (const row of runs) {++ const run = wireRun(row);++ lines.push(JSON.stringify({++ type: 'benchmark_run', benchmark: run.benchmark, model: run.model, route: run.route,++ provider: run.provider, harness: run.harness, settings: run.settings, hardware: run.hardware,++ score: run.score, cost_usd: run.cost_usd, ms: run.ms, tokens: run.tokens,++ provenance: run.provenance, session: run.session, created_at: run.created_at,++ }));++ }++ }++ return lines.length ? `${lines.join('\n')}\n` : '';++ }++/** The registry list: by kind, by tag, or by a search over the title and the description. *//*** Benchmarks for a listing: the public ones, plus every one the `owner` owns when one is named. Unlisted and@@@ -485,6 +602,7 @@ function createRegistry(s, { log = console } = {}) {const entries = [];for (const [model, list] of byModel) {const scores = list.map((r) => r.score);++ const cleanScores = scores.filter((v) => v != null);const costs = list.map((r) => r.cost_usd);const times = list.map((r) => r.ms);const medScore = median(scores);@@@ -508,6 +626,12 @@ function createRegistry(s, { log = console } = {}) {speed: medScore != null && medMs ? Math.round((medScore / (medMs / 60000)) * 1000) / 1000 : null,consistency: spreads.length ? Math.round(median(spreads) * 1000) / 1000 : null,sessions: sessions.size,++ // What the runs actually did, and the value metric of the rules: cost per solved task = the cost++ // of an attempt divided by the success rate (the median score / 100).++ min_score: cleanScores.length ? Math.min(...cleanScores) : null,++ max_score: cleanScores.length ? Math.max(...cleanScores) : null,++ spread: cleanScores.length ? Math.round((Math.max(...cleanScores) - Math.min(...cleanScores)) * 1000) / 1000 : null,++ cost_per_solved: medScore > 0 && medCost > 0 ? Math.round((medCost / (medScore / 100)) * 1e6) / 1e6 : null,};entries.push(entry);}@@@ -522,6 +646,24 @@ function createRegistry(s, { log = console } = {}) {if (av === bv) return a.model.localeCompare(b.model);return ascending ? av - bv : bv - av;});++ // Ranks are bands, not places: models whose score intervals overlap share a band. The interval is the++ // run-to-run spread here (a private set has too few runs for a bootstrap to say more).++ if (metric === 'score') {++ const bands = rankBands(entries.filter((e) => e.score != null).map((e) => ({++ model: e.model,++ rating: e.score,++ ci_lower: e.min_score == null ? e.score : e.min_score,++ ci_upper: e.max_score == null ? e.score : e.max_score,++ })));++ const bandOf = new Map();++ for (const band of bands) for (const model of band.models) bandOf.set(model, band);++ for (const e of entries) {++ const band = bandOf.get(e.model);++ e.band = band ? band.label : null;++ e.band_from = band ? band.from : null;++ e.band_to = band ? band.to : null;++ }++ }return { benchmark: ref.id, metric, entries };}@@@ -621,8 +763,12 @@ function createRegistry(s, { log = console } = {}) {return {createBenchmark, getBenchmark, getVersion, latestVersion, getDefinition, saveDraft, freezeVersion,-- forkBenchmark, listBenchmarks, publishRun, getRun, compareRuns, rankings, percentile, listRuns, modelSummary, runCounts,++ forkBenchmark, setVisibility, findShared, publicDatasetLines,++ listBenchmarks, publishRun, getRun, compareRuns, rankings, percentile, listRuns, modelSummary, runCounts,};}--module.exports = { createRegistry, RegistryError, hashDefinition, canonical, median, parseRef };++module.exports = {++ createRegistry, RegistryError, hashDefinition, canonical, median, parseRef,++ canViewBenchmark, shareTokenHash, shareTokenMatches, newShareToken,++};@@@ -3,11 +3,13 @@/*** What a benchmark and a run must look like before Bench stores them.*-- * The shapes are the released contracts (bench.benchmark@1 and bench.run@1, openvibe-contracts v0.244.0). Bench keeps-- * its own copy of the rules so the service can refuse a bad document before it reaches the database, and so a-- * caller gets a field-level error instead of a stack trace.++ * The shapes are the released contracts (bench.benchmark@1 and bench.run@1, openvibe-contracts). Bench keeps its++ * own copy of the rules so the service can refuse a bad document before it reaches the database, and so a caller++ * gets a field-level error instead of a stack trace.*-- * The rules mirror the contract's: the same required fields, the same enums, the same ranges.++ * The rules mirror the contract's: the same required fields, the same enums, the same ranges. The one addition is++ * private visibility (B11), Bench's stricter form of unlisted: the benchmark row is private while the frozen++ * contract document stays unlisted, so every stored contract document remains a valid bench.benchmark@1.*/const { assertValid } = require('openvibe-contracts');@@@ -22,7 +24,9 @@ const BENCHMARK_ID = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;const RUN_ID = /^brn_[0-9A-HJKMNP-TV-Z]{26}$/;const BENCHMARK_REF = /^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$/;--const VISIBILITIES = ['public', 'unlisted'];++// private is Bench's own stricter form of unlisted (B11): the row's access is private; the frozen contract++// document says unlisted, because the published shape knows only public and unlisted.++const VISIBILITIES = ['public', 'unlisted', 'private'];const SCORING_METHODS = ['mean', 'weighted', 'pass_rate', 'sum'];const EXECUTOR_KINDS = ['codes', 'run', 'node', 'submitted'];const PROVENANCES = ['verified', 'submitted'];nnew file mode 100644@@@ -0,0 +1,166 @@++'use strict';++++/**++ * B11: a person's own prompts, pasted as a private benchmark.++ *++ * The paste is either one prompt per line, or CSV with an optional expected answer in the second column. The parser++ * is deliberately small and local: nothing here calls a model, so a prompt never leaves the request that made it.++ * Bounds: 5–50 prompts (the spec's "paste 5–50 of your real prompts").++ *++ * A CSV may carry a header (`prompt` or `prompt,expected`); without a header the form's format choice says which++ * form it is. Quotes follow RFC 4180 closely enough for a spreadsheet export: a field may be quoted, a `""` inside++ * quotes is one quote, and a quoted field may hold commas and newlines.++ */++const { kindById } = require('../kinds');++++const MIN_PROMPTS = 5;++const MAX_PROMPTS = 50;++const MAX_PROMPT = 8000;++const MAX_EXPECTED = 4000;++const MAX_PASTE = 512 * 1024;++const DEFAULT_TITLE = 'Your prompts';++const TAG = 'your-prompts';++++const clean = (v) => String(v == null ? '' : v).replace(/^\uFEFF/, '').replace(/\r\n?/g, '\n').trim();++++/** Parse a CSV body into rows of strings. */++function csvRows(text) {++ const rows = [];++ let row = [];++ let field = '';++ let quoted = false;++ for (let i = 0; i < text.length; i++) {++ const c = text[i];++ if (quoted) {++ if (c === '"' && text[i + 1] === '"') { field += '"'; i++; }++ else if (c === '"') quoted = false;++ else field += c;++ } else if (c === '"' && field === '') quoted = true;++ else if (c === ',') { row.push(field); field = ''; }++ else if (c === '\n') { row.push(field); rows.push(row); row = []; field = ''; }++ else field += c;++ }++ if (field !== '' || row.length) { row.push(field); rows.push(row); }++ return rows;++}++++const HEADER_PROMPT = new Set(['prompt', 'prompts', 'task', 'input']);++const HEADER_EXPECTED = new Set(['expected', 'expected answer', 'answer', 'expected_answer', 'output']);++++/** The first row is a header when it names a prompt column (and at most an expected-answer column). */++function headerKind(row) {++ if (!row || !row.length) return null;++ const cells = row.map((c) => String(c || '').trim().toLowerCase());++ if (!HEADER_PROMPT.has(cells[0])) return null;++ if (cells.length === 1) return { expected: false };++ if (cells.length === 2 && HEADER_EXPECTED.has(cells[1])) return { expected: true };++ return null;++}++++function promptRows(rows) {++ const out = [];++ for (const row of rows) {++ if (!row || !row.length) continue;++ const prompt = String(row[0] == null ? '' : row[0]).trim();++ if (!prompt && row.every((c) => !String(c || '').trim())) continue;++ if (!prompt) return { ok: false, code: 'bench.prompt_empty', detail: `An empty prompt is not a task (row ${out.length + 1}).` };++ const answer = row.length > 1 ? String(row[1] == null ? '' : row[1]).trim() : '';++ if (row.length > 2 && row.slice(2).some((c) => String(c || '').trim())) {++ return { ok: false, code: 'bench.csv_columns', detail: `Row ${out.length + 1} has more than two columns; use prompt,expected.` };++ }++ out.push({ prompt, expected: answer || null });++ }++ return { ok: true, prompts: out };++}++++function bounds(prompts, format) {++ if (prompts.length < MIN_PROMPTS) {++ return { ok: false, code: 'bench.too_few_prompts', detail: `Paste at least ${MIN_PROMPTS} prompts; this has ${prompts.length}.` };++ }++ if (prompts.length > MAX_PROMPTS) {++ return { ok: false, code: 'bench.too_many_prompts', detail: `Paste at most ${MAX_PROMPTS} prompts; this has ${prompts.length}.` };++ }++ for (let i = 0; i < prompts.length; i++) {++ const p = prompts[i];++ if (p.prompt.length > MAX_PROMPT) return { ok: false, code: 'bench.prompt_too_long', detail: `Prompt ${i + 1} is longer than ${MAX_PROMPT} characters.` };++ if (p.expected && p.expected.length > MAX_EXPECTED) return { ok: false, code: 'bench.expected_too_long', detail: `The expected answer for prompt ${i + 1} is longer than ${MAX_EXPECTED} characters.` };++ }++ return { ok: true, prompts, format };++}++++/**++ * Parse a paste.++ *++ * format 'lines' one prompt per line (a comma is part of the prompt)++ * format 'csv' comma-separated, optional expected answer in the second column++ * format 'auto' CSV only when the first line is a prompt/expected header, else lines++ */++function parsePrompts(raw, { format = 'auto' } = {}) {++ const text = clean(raw);++ if (!text) return { ok: false, code: 'bench.prompts_required', detail: 'Paste 5 to 50 prompts.' };++ if (text.length > MAX_PASTE) return { ok: false, code: 'bench.prompts_too_large', detail: 'That paste is too large; keep it under 512 KB.' };++ let mode = String(format || 'auto').toLowerCase();++ if (!['auto', 'lines', 'csv'].includes(mode)) mode = 'auto';++ const lines = text.split('\n').map((line) => line.trim()).filter(Boolean);++ if (mode === 'auto') {++ const head = csvRows(lines[0] || '');++ mode = head.length && headerKind(head[0]) ? 'csv' : 'lines';++ }++ if (mode === 'csv') {++ const rows = csvRows(text);++ const header = rows.length ? headerKind(rows[0]) : null;++ const parsed = promptRows(header ? rows.slice(1) : rows);++ if (!parsed.ok) return parsed;++ return bounds(parsed.prompts, 'csv');++ }++ // Lines: one prompt per line. A line may still be a CSV header by mistake; say so rather than making it a task.++ const head = lines.length ? csvRows(lines[0]) : [];++ if (head.length && headerKind(head[0])) {++ return { ok: false, code: 'bench.prompt_header', detail: 'That looks like a CSV header. Choose the CSV format, or remove the header line.' };++ }++ return bounds(lines.map((prompt) => ({ prompt, expected: null })), 'lines');++}++++/** The judges of a kind: its schema's list, else its own judge. */++function judgesFor(kind) {++ const list = kind && Array.isArray(kind.schema && kind.schema.judges) && kind.schema.judges.length++ ? kind.schema.judges++ : [kind ? kind.judge : 'automatic_grader'];++ return list.map((type) => ({ type, version: '1.0', config: {} }));++}++++/**++ * The definition a paste becomes: the tasks (one per prompt), the kind's judges, mean scoring, a time limit from++ * the kind and no dollar limit. No dollar limit means the existing launcher always asks before it starts, and no++ * cap is added or raised here.++ */++function definitionFor({ prompts, kind, title, description }) {++ const k = kindById(kind) || kindById('text.answer');++ const name = String(title == null ? '' : title).replace(/\s+/g, ' ').trim().slice(0, 160) || DEFAULT_TITLE;++ const text = String(description == null ? '' : description).trim().slice(0, 4000);++ return {++ title: name,++ description: text,++ kind: k.id,++ tags: [TAG, String(k.id).split('.')[0]],++ // Stored as unlisted in the contract document (the published shape knows public and unlisted); the++ // benchmark row's own visibility is private, and that row is what access checks read.++ visibility: 'unlisted',++ tasks: prompts.map((p, i) => ({++ id: `t${i + 1}`,++ prompt: p.prompt,++ files: [],++ expected: p.expected ? { answer: p.expected } : null,++ holdout: false,++ })),++ judges: judgesFor(k),++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },++ limits: { ms: Number(k.schema && k.schema.limit_ms) || 120000, tokens: 0, usd: 0 },++ matrix: { models: [], harnesses: [] },++ };++}++++module.exports = {++ parsePrompts, csvRows, definitionFor, judgesFor, headerKind,++ MIN_PROMPTS, MAX_PROMPTS, MAX_PROMPT, MAX_EXPECTED, MAX_PASTE, DEFAULT_TITLE, TAG,++};@@@ -32,7 +32,7 @@ const { BenchError } = require('../store');const boardLib = require('../rating/board');function createApi(ctx) {-- const { config, s, principal, bench } = ctx;++ const { config, s, principal, bench, registry } = ctx;const r = asyncRouter();r.use(express.json({ limit: '64kb' }));@@@ -226,6 +226,17 @@ function createApi(ctx) {} catch (err) { return failed(res, err); }});++ // The open data for the benchmark registry (B11): public benchmarks, their visible tasks and every run under++ // them. Private and unlisted rows are never read here, and no person id appears in a line.++ r.get('/datasets/benchmarks.jsonl', async (req, res) => {++ try {++ const text = await registry.publicDatasetLines();++ res.set('Cache-Control', 'public, max-age=60');++ res.set('Content-Disposition', 'attachment; filename="bench-benchmarks.jsonl"');++ return res.type('application/x-ndjson').send(text);++ } catch (err) { return failed(res, err); }++ });++// The open vote dump (stage B9): every human vote, no voter ids. It comes before the kind dataset route,// because '/datasets/votes.jsonl' would otherwise be read as the kind 'votes'.r.get('/datasets/votes.jsonl', async (req, res) => {@@@ -51,7 +51,7 @@ function createDiscoveryRoutes(ctx) {res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({name: SITE_NAME,summary: DESCRIPTION,-- details: 'GET /api/v1/kinds lists the benchmark kinds. GET /api/v1/boards/:kind?type=elo|pareto|cost renders a board as JSON. GET /api/v1/datasets/:kind.jsonl is the open dataset without voter ids. POST /mcp is the Model Context Protocol endpoint.',++ details: 'GET /api/v1/kinds lists the benchmark kinds. GET /api/v1/boards/:kind?type=elo|pareto|cost renders a board as JSON. GET /api/v1/datasets/:kind.jsonl is the open dataset without voter ids. GET /api/v1/datasets/benchmarks.jsonl is every public benchmark and its runs. POST /mcp is the Model Context Protocol endpoint.',sections: [{ title: 'Start here', links: [{ title: SITE_NAME, url: abs('/') }, { title: 'The API', url: abs('/docs') }] },{ title: 'Machine-readable', links: [{ title: 'Kinds (JSON)', url: abs('/api/v1/kinds') }, { title: 'Sitemap', url: abs('/sitemap.xml') }, { title: 'Release metadata (JSON)', url: abs('/release.json') }] },@@@ -69,9 +69,17 @@ function createDiscoveryRoutes(ctx) {}));});-- r.get('/sitemap.xml', (_req, res) => {++ r.get('/sitemap.xml', async (_req, res) => {const lastmod = siteUpdated();-- const urls = PAGES.map((p) => ({ loc: abs(p.path), ...(lastmod ? { lastmod } : {}), changefreq: p.changefreq, priority: p.priority }));++ // Public pages only: a private or unlisted benchmark never enters the sitemap (the registry lists public++ // ones only).++ const pages = [...PAGES, { path: '/benchmarks', changefreq: 'daily', priority: 0.8, title: 'Benchmarks', text: 'Versioned AI benchmarks anyone can run again.' }];++ const benchmarks = await ctx.registry.listBenchmarks({ limit: 200 });++ for (const b of benchmarks) {++ if (b.visibility !== 'public') continue;++ pages.push({ path: `/benchmarks/${encodeURIComponent(b.id)}`, changefreq: 'weekly', priority: 0.7, title: b.title, text: b.description || b.title });++ }++ const urls = pages.map((p) => ({ loc: abs(p.path), ...(lastmod ? { lastmod } : {}), changefreq: p.changefreq, priority: p.priority }));res.type('application/xml').set('Cache-Control', TEXT).send(seo.sitemapXml(urls));});@@@ -13,6 +13,10 @@ const { raw } = require('../render/html');const { send } = require('../render/layout');const views = require('../views');const results = require('../views/results');++const { canViewBenchmark } = require('../bench/registry');++const { parsePrompts, definitionFor, TAG: PROMPT_TAG } = require('../bench/your-prompts');++const { validateBenchmark } = require('../bench/validate');++const { kindById } = require('../kinds');const PUBLIC_CACHE = 'public, max-age=60';@@@ -69,6 +73,58 @@ function createPageRoutes(ctx) {const notFound = (req, res, what) => page(req, res, { title: `No such ${what}`, body: raw(views.notFound()) }, 404);const METRIC_IDS = results.METRICS.map((m) => m.id);++ /** The benchmark a page may show, or null: a private one belongs to its owner, a shared unlisted one to its link. */++ const readableBenchmark = async (req, ref) => {++ const out = await registry.getDefinition(String(ref || ''), { reveal: false });++ if (!out) return null;++ return canViewBenchmark(out.benchmark, { subject: subjectOf(req) }) ? out : null;++ };++++ /** The public board of a kind: the first public benchmark with verified runs, else the first public one. */++ const publicBoardForKind = async (kind, { excludeId = null } = {}) => {++ const list = await registry.listBenchmarks({ kind: String(kind || ''), limit: 50 });++ const publicList = list.filter((b) => b.visibility === 'public' && b.id !== excludeId);++ for (const b of publicList) {++ const rankings = await registry.rankings({ benchmark: b.id, metric: 'score' });++ if (rankings && rankings.entries.length) return { benchmark: b, entries: rankings.entries };++ }++ return publicList.length ? { benchmark: publicList[0], entries: [] } : null;++ };++++ /** One benchmark page, whether its owner, a public reader or someone following a share link is looking. */++ const renderBenchmark = async (req, res, out, { share = null, shared = false, status = 200 } = {}) => {++ const metric = METRIC_IDS.includes(String(req.query.metric)) ? String(req.query.metric) : 'score';++ const id = out.benchmark.id;++ const owner = subjectOf(req) != null && String(out.benchmark.owner) === String(subjectOf(req));++ const [rankings, runs] = await Promise.all([++ registry.rankings({ benchmark: id, metric }),++ registry.listRuns({ benchmark: id, limit: 8 }),++ ]);++ const isPublic = out.benchmark.visibility === 'public';++ const isPromptBenchmark = Array.isArray(out.benchmark.tags) && out.benchmark.tags.includes(PROMPT_TAG);++ const ownBoards = owner && isPromptBenchmark++ ? {++ ownEntries: (rankings && rankings.entries) || [],++ ownRuns: ((rankings && rankings.entries) || []).reduce((n, e) => n + e.runs, 0),++ publicBoard: await publicBoardForKind(out.benchmark.kind, { excludeId: out.benchmark.id }),++ visibility: out.benchmark.visibility,++ }++ : null;++ return page(req, res, {++ index: isPublic && metric === 'score' && !shared, cache: isPublic && !shared ? PUBLIC_CACHE : null,++ title: out.benchmark.title,++ description: out.benchmark.description || `${out.benchmark.title}: rankings, value and every run.`,++ jsonLd: isPublic && !shared++ ? [seo.jsonLd.webPage({ name: out.benchmark.title, url: `${base()}/benchmarks/${encodeURIComponent(id)}`, description: out.benchmark.description || out.benchmark.title, siteUrl: base() })]++ : [],++ body: raw(String(results.benchmarkPage({++ benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version,++ rankings, metric, runs, codesRunUrl: config.codesRunUrl || null, viewer: req.viewer, baseUrl: base(),++ ownBoards, share, shared, owner,++ }))),++ }, status);++ };++r.get('/benchmarks', async (req, res) => {const q = String(req.query.q || '').slice(0, 80);const family = String(req.query.kind || '').replace(/[^a-z0-9._-]/gi, '').slice(0, 40) || null;@@@ -76,50 +132,115 @@ function createPageRoutes(ctx) {const kinds = [...new Set(all.map((b) => b.kind).filter(Boolean))];const list = family ? all.filter((b) => b.kind === family || String(b.kind).startsWith(`${family}.`)) : all;const counts = await registry.runCounts();++ const subject = subjectOf(req);++ const mine = subject ? (await registry.listBenchmarks({ owner: subject, limit: 100 })).filter((b) => b.visibility !== 'public') : [];for (const b of list) b.runs = counts.get(b.id) || 0;++ for (const b of mine) b.runs = counts.get(b.id) || 0;return page(req, res, {index: !q && !family, cache: PUBLIC_CACHE,title: 'Benchmarks',description: 'Versioned AI benchmarks anyone can run again: tasks, judges and limits, with every run published.',jsonLd: [seo.jsonLd.webPage({ name: 'Benchmarks', url: `${base()}/benchmarks`, description: 'Versioned AI benchmarks anyone can run again.', siteUrl: base() })],-- body: raw(String(results.benchmarksIndex({ benchmarks: list, kinds, kind: family, q }))),++ body: raw(String(results.benchmarksIndex({ benchmarks: list, kinds, kind: family, q, mine, signedIn: Boolean(subject) }))),});});-- r.get('/benchmarks/:ref', async (req, res) => {-- const ref = String(req.params.ref || '');-- const out = await registry.getDefinition(ref, { reveal: false });++ // ---- a person's own prompts (B11) ---------------------------------------------------------------------++ // The form is a plain form post, same-origin, and nothing here logs the prompts.++ r.get('/benchmarks/new', async (req, res) => {++ const kinds = await bench.listKinds();const subject = subjectOf(req);-- if (!out || (out.benchmark.visibility === 'private' && out.benchmark.owner !== subject)) return notFound(req, res, 'benchmark');-- const metric = METRIC_IDS.includes(String(req.query.metric)) ? String(req.query.metric) : 'score';-- const id = out.benchmark.id;-- const [rankings, runs] = await Promise.all([-- registry.rankings({ benchmark: id, metric }),-- registry.listRuns({ benchmark: id, limit: 8 }),-- ]);-- const isPublic = out.benchmark.visibility === 'public';return page(req, res, {-- index: isPublic && metric === 'score', cache: isPublic ? PUBLIC_CACHE : null,-- title: out.benchmark.title,-- description: out.benchmark.description || `${out.benchmark.title}: rankings, value and every run.`,-- jsonLd: [seo.jsonLd.webPage({ name: out.benchmark.title, url: `${base()}/benchmarks/${encodeURIComponent(id)}`, description: out.benchmark.description || out.benchmark.title, siteUrl: base() })],-- body: raw(String(results.benchmarkPage({-- benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version,-- rankings, metric, runs, codesRunUrl: config.codesRunUrl || null, viewer: req.viewer, baseUrl: base(),++ index: false, title: 'Make your own benchmark',++ description: 'Paste 5 to 50 of your own prompts as a private benchmark, then run them on 3 to 6 models.',++ body: raw(String(results.newBenchmarkPage({++ kinds,++ values: { kind: String(req.query.kind || 'text.answer') },++ signedIn: Boolean(subject),++ loginUrl: `/auth/login?next=${encodeURIComponent('/benchmarks/new')}`,}))),});});++ r.post('/benchmarks/new', express.urlencoded({ extended: false, limit: '512kb' }), ctx.principal.middleware, async (req, res) => {++ if (!sameOrigin(req, config)) return res.status(403).type('text').send('This form only works from Bench itself.');++ const subject = subjectOf(req);++ const kinds = await bench.listKinds();++ const b = req.body && typeof req.body === 'object' ? req.body : {};++ const values = {++ title: String(b.title || '').slice(0, 160),++ description: String(b.description || '').slice(0, 4000),++ kind: String(b.kind || 'text.answer'),++ format: String(b.format || 'auto'),++ prompts: String(b.prompts || ''),++ };++ const fail = (status, error) => page(req, res, {++ index: false, title: 'Make your own benchmark',++ description: 'Paste 5 to 50 of your own prompts as a private benchmark.',++ body: raw(String(results.newBenchmarkPage({ kinds, values, error, signedIn: Boolean(subject), loginUrl: `/auth/login?next=${encodeURIComponent('/benchmarks/new')}` }))),++ }, status);++ if (!subject) return fail(401, 'Sign in to make a benchmark of your own.');++ const kind = kindById(values.kind);++ if (!kind) return fail(422, 'Pick one of Bench’s kinds.');++ const parsed = parsePrompts(values.prompts, { format: values.format });++ if (!parsed.ok) return fail(422, parsed.detail);++ const definition = definitionFor({ prompts: parsed.prompts, kind: kind.id, title: values.title, description: values.description });++ try {++ const made = await registry.createBenchmark({++ owner: subject, title: definition.title, description: definition.description, kind: definition.kind,++ tags: definition.tags, visibility: 'private', definition, subject,++ });++ if (!made.ok) return fail(made.status || 422, made.message || 'The benchmark could not be made.');++ const frozen = await registry.freezeVersion(made.benchmark.id, { definition, subject, validate: validateBenchmark });++ if (!frozen.ok) return fail(frozen.status || 422, frozen.message || 'The benchmark could not be frozen.');++ return res.redirect(303, `/benchmarks/${encodeURIComponent(made.benchmark.id)}`);++ } catch (err) {++ ctx.log.warn('[OpenVibe.Bench] making a prompt benchmark failed:', (err && err.message) || err);++ return fail(500, 'The benchmark could not be made. Try again in a moment.');++ }++ });++++ // A share link: only an unlisted benchmark has one, and only its hash is stored.++ r.get('/b/:token', async (req, res) => {++ const benchmark = await registry.findShared(String(req.params.token || ''));++ if (!benchmark) return notFound(req, res, 'share link');++ const out = await registry.getDefinition(benchmark.id, { reveal: false });++ if (!out) return notFound(req, res, 'share link');++ return renderBenchmark(req, res, out, { shared: true });++ });++++ // The owner shares, unshares or publishes: a plain same-origin form post.++ r.post('/benchmarks/:ref/share', express.urlencoded({ extended: false, limit: '16kb' }), ctx.principal.middleware, async (req, res) => {++ if (!sameOrigin(req, config)) return res.status(403).type('text').send('This form only works from Bench itself.');++ const subject = subjectOf(req);++ const out = await registry.getDefinition(String(req.params.ref || ''), { reveal: false });++ if (!out || !subject || String(out.benchmark.owner) !== String(subject)) return notFound(req, res, 'benchmark');++ const changed = await registry.setVisibility(out.benchmark.id, { visibility: String((req.body || {}).visibility || ''), subject });++ if (!changed.ok) {++ return page(req, res, { index: false, title: out.benchmark.title, body: raw(String(results.benchmarkPage({ benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version, rankings: await registry.rankings({ benchmark: out.benchmark.id, metric: 'score' }), runs: await registry.listRuns({ benchmark: out.benchmark.id, limit: 8 }), codesRunUrl: config.codesRunUrl || null, viewer: req.viewer, baseUrl: base(), owner: true, share: { error: changed.message } }))) }, 422);++ }++ const fresh = await registry.getDefinition(out.benchmark.id, { reveal: false });++ return renderBenchmark(req, res, fresh, {++ share: changed.token ? { shareUrl: `${base()}${changed.share_path}`, token: changed.token } : null,++ });++ });++++ r.get('/benchmarks/:ref', async (req, res) => {++ const out = await readableBenchmark(req, req.params.ref);++ if (!out) return notFound(req, res, 'benchmark');++ return renderBenchmark(req, res, out);++ });++// The run form (B3): GET shows it, POST starts the run through the same launcher as the API.const runFormPage = async (req, res, extra = {}) => {const ref = String(req.params.ref || '');-- const out = await registry.getDefinition(ref, { reveal: false });-- if (!out || out.benchmark.visibility === 'private') return notFound(req, res, 'benchmark');++ const out = await readableBenchmark(req, ref);++ if (!out) return notFound(req, res, 'benchmark');if (!config.codesRunUrl) return res.redirect(303, `/benchmarks/${encodeURIComponent(out.benchmark.id)}`);const signedIn = req.viewer && req.viewer.kind === 'user';return page(req, res, {index: false, title: `Run ${out.benchmark.title}`,-- description: `Run ${out.benchmark.title} on openvibe.codes with your own keys.`,++ description: `Run ${out.benchmark.title} on openvibe.codes with your own keys or the free pool.`,body: raw(String(results.runForm({benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version, signedIn,harnesses: signedIn ? await ctx.launcher.harnessList() : [],@@@ -139,6 +260,7 @@ function createPageRoutes(ctx) {const picked = { models, harnesses, repeat };const out = await ctx.launcher.launch({ principal: req.principal, id: req.params.ref, models, harnesses, repeat, confirm: b.confirm === '1' });if (out.status === 202) return res.redirect(303, out.body.tasks_url || config.codesRunUrl);++ if (out.status === 404) return notFound(req, res, 'benchmark');if (out.status === 409) return runFormPage(req, res, { picked, confirm: out.body.cost_estimate_usd == null ? 'unknown' : out.body.cost_estimate_usd });return runFormPage(req, res, { picked, error: out.body.detail || 'The run could not start.', status: out.status === 400 ? 400 : 200 });});@@@ -148,7 +270,7 @@ function createPageRoutes(ctx) {if (!run) return notFound(req, res, 'run');const def = await registry.getDefinition(`${run.benchmark_id}@${run.benchmark_version}`, { reveal: false });const benchmark = def ? def.benchmark : null;-- if (benchmark && benchmark.visibility === 'private' && benchmark.owner !== subjectOf(req)) return notFound(req, res, 'run');++ if (benchmark && !canViewBenchmark(benchmark, { subject: subjectOf(req) })) return notFound(req, res, 'run');const pct = run.score != null ? await registry.percentile({ benchmark: run.benchmark_id, model: run.model, score: run.score }) : null;const isPublic = !benchmark || benchmark.visibility === 'public';return page(req, res, {@@@ -163,19 +285,21 @@ function createPageRoutes(ctx) {const refs = String(req.query.runs || '').split(',').map((x) => x.trim()).filter(Boolean).slice(0, 4);let cmp = refs.length ? await registry.compareRuns(refs) : null;const titles = {};++ let allPublic = true;if (cmp) {const subject = subjectOf(req);const keep = [];for (const run of cmp.runs) {const b = await registry.getBenchmark(run.benchmark_id);-- if (b && b.visibility === 'private' && b.owner !== subject) continue;-- if (b) titles[b.id] = b.title;++ if (!b || !canViewBenchmark(b, { subject })) continue;++ if (b.visibility !== 'public') allPublic = false;++ titles[b.id] = b.title;keep.push(run);}cmp = keep.length ? { ...cmp, runs: keep, rows: cmp.rows.map((row) => ({ ...row, scores: keep.map((run) => { const t = run.tasks.find((x) => x.task_id === row.task_id); return t ? t.score : null; }) })) } : null;}return page(req, res, {-- index: false, cache: PUBLIC_CACHE,++ index: false, cache: allPublic ? PUBLIC_CACHE : null,title: cmp ? `Compare: ${cmp.runs.map((x) => x.model).join(' vs ')}` : 'Compare runs',description: 'Runs side by side: the same tasks, their scores, cost, time and outputs.',body: raw(String(results.comparePage({ cmp, titles }))),@@@ -23,6 +23,7 @@ const { http } = require('openvibe-contracts');const { asyncRouter } = require('./router');const { sameOrigin } = require('./same-origin');const { validateBenchmark, validateRun } = require('../bench/validate');++const { canViewBenchmark } = require('../bench/registry');function createRegistryApi(ctx) {const { config, principal, registry } = ctx;@@@ -67,6 +68,13 @@ function createRegistryApi(ctx) {return false;};++ /** The caller's view of a benchmark: its owner, or the executor that must read the tasks. */++ const viewerOf = (req) => ({++ subject: subjectOf(req),++ executor: req.principal.kind === 'app' && req.principal.hasCapability('bench.run.publish'),++ });++ const mayRead = (req, benchmark) => canViewBenchmark(benchmark, viewerOf(req));++// ── Benchmarks ───────────────────────────────────────────r.post('/benchmarks', actor('bench.benchmark.write'), async (req, res) => {const b = req.body && typeof req.body === 'object' ? req.body : {};@@@ -127,6 +135,17 @@ function createRegistryApi(ctx) {} catch (err) { return failed(res, err); }});++ r.post('/benchmarks/:id/share', actor('bench.benchmark.write'), async (req, res) => {++ const b = req.body && typeof req.body === 'object' ? req.body : {};++ const subject = subjectOf(req) || req.principal.requester;++ try {++ const out = await registry.setVisibility(String(req.params.id || ''), { visibility: b.visibility, subject });++ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid visibility');++ res.set('Cache-Control', 'no-store');++ return res.json({ benchmark: out.benchmark, token: out.token, share_path: out.share_path });++ } catch (err) { return failed(res, err); }++ });++r.get('/benchmarks', async (req, res) => {try {// ?mine=1 adds the caller's own unlisted and private benchmarks; nobody else ever sees those in a list.@@@ -138,7 +157,7 @@ function createRegistryApi(ctx) {q: req.query.q ? String(req.query.q) : null,limit: req.query.limit ? Number(req.query.limit) : 50,});-- res.set('Cache-Control', 'public, max-age=60');++ res.set('Cache-Control', mine ? 'private, no-store' : 'public, max-age=60');return res.json({ count: list.length, benchmarks: list });} catch (err) { return failed(res, err); }});@@@ -146,16 +165,18 @@ function createRegistryApi(ctx) {r.get('/benchmarks/:id', async (req, res) => {try {const ref = String(req.params.id || '');-- // The holdout goes to the executor and to the benchmark's owner; to nobody else.-- let reveal = mayReveal(req);-- if (!reveal && req.principal.kind === 'user') {-- const owner = await registry.getBenchmark(ref.split('@')[0]);-- reveal = !!(owner && owner.owner === req.principal.subject);-- }-- const out = await registry.getDefinition(ref, { reveal });++ const out = await registry.getDefinition(ref, { reveal: false });if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');-- // A private benchmark answers only its owner and the executor; to everyone else it does not exist.-- if (out.benchmark.visibility === 'private' && !reveal) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');++ // A private benchmark answers only its owner and the executor; a shared unlisted one only its owner by++ // id (the token link is the way in). Everyone else gets a 404, not a "forbidden".++ if (!mayRead(req, out.benchmark)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');++ const view = viewerOf(req);++ const owner = view.subject != null && String(out.benchmark.owner) === String(view.subject);++ const reveal = mayReveal(req) || owner;++ if (reveal) {++ const full = await registry.getDefinition(ref, { reveal: true });++ if (full) { out.definition = full.definition; out.version = full.version; }++ }res.set('Cache-Control', out.benchmark.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');return res.json({benchmark: out.benchmark,@@@ -205,6 +226,10 @@ function createRegistryApi(ctx) {const check = validateRun({ ...doc, id: doc.id || 'brn_00000000000000000000000000' });if (!check.valid) return invalid(res, check.errors.map((e) => `${e.path} ${e.message}`).join('; '));try {++ // A run may only be published on a benchmark its author may read: another person's private one is a 404.++ const bmId = String(doc.benchmark_id || doc.benchmark || '').split('@')[0];++ const bm = bmId ? await registry.getBenchmark(bmId) : null;++ if (bm && !mayRead(req, bm)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');const out = await registry.publishRun(doc, { provenance, subject });if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid run');res.set('Location', `/api/v1/runs/${out.run.id}`);@@@ -216,7 +241,9 @@ function createRegistryApi(ctx) {try {const run = await registry.getRun(String(req.params.id || ''));if (!run) return problem(res, 404, 'bench.run_not_found', 'No such run.');-- res.set('Cache-Control', 'public, max-age=60');++ const bm = await registry.getBenchmark(run.benchmark_id);++ if (bm && !mayRead(req, bm)) return problem(res, 404, 'bench.run_not_found', 'No such run.');++ res.set('Cache-Control', bm && bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');return res.json({ run });} catch (err) { return failed(res, err); }});@@@ -229,8 +256,30 @@ function createRegistryApi(ctx) {try {const out = await registry.compareRuns(refs);if (!out) return problem(res, 404, 'bench.run_not_found', 'No such run.');-- res.set('Cache-Control', 'public, max-age=60');-- return res.json({ compare: out });++ // One unreadable run hides itself; if none is readable the comparison does not exist.++ const allowed = [];++ let allPublic = true;++ for (const run of out.runs) {++ const bm = await registry.getBenchmark(run.benchmark_id);++ if (!bm || mayRead(req, bm)) allowed.push(run);++ if (!bm || bm.visibility !== 'public') allPublic = false;++ }++ if (!allowed.length) return problem(res, 404, 'bench.run_not_found', 'No such run.');++ const taskIds = out.task_ids;++ allowed.forEach((run) => { for (const t of run.tasks) if (!taskIds.includes(t.task_id)) taskIds.push(t.task_id); });++ const compare = {++ runs: allowed,++ task_ids: taskIds,++ rows: taskIds.map((taskId) => ({++ task_id: taskId,++ scores: allowed.map((run) => {++ const t = run.tasks.find((x) => x.task_id === taskId);++ return t ? t.score : null;++ }),++ })),++ };++ res.set('Cache-Control', allPublic ? 'public, max-age=60' : 'private, no-store');++ return res.json({ compare });} catch (err) { return failed(res, err); }});@@@ -242,6 +291,8 @@ function createRegistryApi(ctx) {return invalid(res, 'metric is score, value, speed or consistency');}try {++ const bm = await registry.getBenchmark(benchmark.split('@')[0]);++ if (!bm || !mayRead(req, bm)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');const out = await registry.rankings({benchmark,metric,@@@ -250,7 +301,7 @@ function createRegistryApi(ctx) {includeSubmitted: String(req.query.include_submitted || '') === 'true',});if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');-- res.set('Cache-Control', 'public, max-age=60');++ res.set('Cache-Control', bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');return res.json({ rankings: out });} catch (err) { return failed(res, err); }});@@@ -262,9 +313,11 @@ function createRegistryApi(ctx) {if (!model) return invalid(res, 'model is required: ?model=…');if (req.query.score == null) return invalid(res, 'score is required: ?score=…');try {++ const bm = await registry.getBenchmark(benchmark.split('@')[0]);++ if (!bm || !mayRead(req, bm)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');const out = await registry.percentile({ benchmark, model, score: Number(req.query.score) });if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');-- res.set('Cache-Control', 'public, max-age=60');++ res.set('Cache-Control', bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');return res.json({ percentile: out });} catch (err) { return failed(res, err); }});@@@ -4,7 +4,8 @@* Account export and deletion → Bench (ADR-033; openvibe-sdk/account-data). What Bench holds about a person:** theirs, and gone the tasks and rounds they started, the private evals they own, the runs they-- * submitted, the pairs they have seen and the benchmarks they watch++ * submitted, a private benchmark with its prompts and every run on it, the pairs they++ * have seen and the benchmarks they watch* theirs, and kept a judgement they cast stays (it sits under somebody else's round, so the round and* the ratings keep their place) and loses the person; a benchmark other people ran* stays as a tombstone, so their runs keep theirs@@@ -12,8 +13,10 @@* person), the entries (a model's output) and the receipts of the requests themselves** A benchmark and its frozen versions are the person's, but other people's runs hang under them, so what happens-- * to them depends on what else is there: a benchmark nobody else ran goes with its versions, one others ran stays-- * as a tombstone. extraErase decides that, and GUARDED names the tables the guard needs to know about.++ * to them depends on what else is there: a private benchmark (a person's own prompts) goes with its versions and++ * every run on it, wherever it was shared; a public or unlisted benchmark nobody else ran goes too; one others ran++ * stays as a tombstone, so their runs keep their place. extraErase decides that, and GUARDED names the tables the++ * guard needs to know about.*/const { createAccountData, TOPICS } = require('openvibe-sdk/account-data');@@@ -38,7 +41,7 @@ const TABLES = [// under them. extraErase decides; the guard needs the names.const GUARDED = ['bench_benchmarks', 'bench_benchmark_versions'];--/** The benchmarks of the person: what they own and what they wrote, with their versions. */++/** The benchmarks of the person: what they own and what they wrote, with their versions and their private runs. */async function extraExport(db, subject) {const files = [];const benchmarks = await db.many(`SELECT id, owner, title, description, kind, tags, visibility, parent, latest_version, draft, created_by, created_at, updated_at@@@ -47,21 +50,42 @@ async function extraExport(db, subject) {const versions = await db.many(`SELECT benchmark_id, version, definition, hash, created_by, created_at FROM bench_benchmark_versionsWHERE created_by = $1 OR benchmark_id IN (SELECT id FROM bench_benchmarks WHERE owner = $1) ORDER BY created_at DESC LIMIT 5000`, [subject]);if (versions.length) files.push({ name: 'benchmark_versions.json', content: versions });++ // A run on the person's own prompt benchmark is part of their data even when a service executed and published++ // it, and even if the set was shared.++ const runs = await db.many(`SELECT id, benchmark_id, benchmark_version, model, route, provider, harness, settings, hardware,++ score, cost_usd, ms, tokens, provenance, session, created_at++ FROM bench_runs WHERE benchmark_id IN (++ SELECT id FROM bench_benchmarks WHERE owner = $1++ AND (visibility = 'private' OR tags @> '["your-prompts"]'::jsonb))++ ORDER BY created_at DESC LIMIT 5000`, [subject]);++ if (runs.length) files.push({ name: 'benchmark_runs.json', content: runs });return files;}async function extraErase(t, subjects, counts) {-- // The benchmarks the person owns. One nobody else ran goes with its versions; one others ran stays as a-- // tombstone, so their runs keep their place.-- const ids = (await t.many('SELECT id FROM bench_benchmarks WHERE owner = ANY($1::text[])', [subjects])).map((r) => r.id);++ // A private benchmark, or one made from the person's own pasted prompts (tagged your-prompts), is nothing but++ // their text: it goes with its versions and every run on it, wherever it was shared. A public or unlisted++ // benchmark another person ran stays as a tombstone, so their runs keep their place.++ const owned = await t.many('SELECT id, visibility, tags FROM bench_benchmarks WHERE owner = ANY($1::text[])', [subjects]);++ const tagsOf = (row) => {++ let tags = row.tags;++ if (typeof tags === 'string') { try { tags = JSON.parse(tags); } catch { tags = []; } }++ return Array.isArray(tags) ? tags : [];++ };++ const deleteIds = owned.filter((r) => r.visibility === 'private' || tagsOf(r).includes('your-prompts')).map((r) => r.id);++ if (deleteIds.length) {++ counts.add(counts.erased, 'bench_runs', await t.exec('DELETE FROM bench_runs WHERE benchmark_id = ANY($1::text[])', [deleteIds]));++ counts.add(counts.erased, 'bench_benchmarks', await t.exec('DELETE FROM bench_benchmarks WHERE id = ANY($1::text[])', [deleteIds]));++ }const kept = [];-- for (const id of ids) {-- const runs = Number(await t.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [id])) || 0;-- if (runs) { kept.push(id); continue; }-- counts.add(counts.erased, 'bench_benchmarks', await t.exec('DELETE FROM bench_benchmarks WHERE id = $1', [id]));++ for (const row of owned) {++ if (deleteIds.includes(row.id)) continue;++ const runs = Number(await t.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [row.id])) || 0;++ if (runs) { kept.push(row.id); continue; }++ counts.add(counts.erased, 'bench_benchmarks', await t.exec('DELETE FROM bench_benchmarks WHERE id = $1', [row.id]));}if (kept.length) {-- counts.add(counts.retained, 'tombstones', await t.exec('UPDATE bench_benchmarks SET owner = $2, created_by = $2 WHERE id = ANY($1::text[])', [kept, TOMBSTONE]));++ counts.add(counts.retained, 'tombstones', await t.exec('UPDATE bench_benchmarks SET owner = $2, created_by = $2, share_token_hash = NULL WHERE id = ANY($1::text[])', [kept, TOMBSTONE]));}// The benchmarks of somebody else that the person wrote: the definition stays, the person leaves it.counts.add(counts.retained, 'tombstones', await t.exec('UPDATE bench_benchmarks SET created_by = $2 WHERE created_by = ANY($1::text[])', [subjects, TOMBSTONE]));@@@ -76,7 +100,7 @@ function create({ db, log = console } = {}) {tables: TABLES,extraExport,extraErase,-- note: 'Bench exports your tasks, rounds, judgements, private evals, runs, watched benchmarks and the pairs you have seen; the public dataset is anonymous and a rating never names a person.',++ note: 'Bench exports your tasks, rounds, judgements, private evals, benchmarks and their prompts, the runs on your private benchmarks, watched benchmarks and the pairs you have seen; the public dataset is anonymous and a rating never names a person.',log,});}@@@ -27,6 +27,9 @@ const KINDS = Object.freeze([{ id: 'speech.asr', room: 'speech', entry: 'transcript', judge: 'automatic_grader', name: 'Speech recognition', metrics: ['word_error_rate', 'cost_usd'], schema: { entry: 'transcript', payload: { transcript: 'string' } } },{ id: 'translate.ja', room: 'translate', entry: 'translation', judge: 'human_pairwise', name: 'Japanese translation', metrics: ['elo', 'cost_usd'], schema: { entry: 'translation', payload: { translation: 'string' }, prompt: { source_text: 'string', source_language: 'string', audience: 'string' }, rubric: ['Preserve the source meaning', 'Use natural Japanese for the audience', 'Preserve names, numbers and uncertainty'], seeds: [{ source_text: 'The meeting starts at 3 p.m. Please bring the draft.', source_language: 'en', audience: 'coworker' }] } },{ id: 'search.answer', room: 'search', entry: 'cited_answer', judge: 'human_pairwise', name: 'Cited answers', metrics: ['elo', 'cost_usd'], schema: { entry: 'cited_answer', payload: { answer: 'string', citations: 'array' }, prompt: { question: 'string', source_urls: 'array' }, rubric: ['Answer from the supplied sources', 'Cite claims with supporting URLs', 'State when the sources do not establish an answer'], seeds: [{ question: 'What year was the first release announced?', source_urls: ['https://example.org/releases'] }] } },++ // B11: a plain text answer is the default kind for a person's own prompts. No renderer: the answer is text, and++ // the expected answer (when the pasted CSV carries one) tells the automatic grader what correct looks like.++ { id: 'text.answer', room: 'bench', entry: 'model_answer', judge: 'automatic_grader', name: 'Text answers', metrics: ['pass_rate', 'cost_usd', 'ms'], schema: { entry: 'model_answer', payload: { text: 'string' }, judges: ['automatic_grader'], rubric: ['Answer the prompt directly', 'Respect any format the prompt asks for', 'Do not add unrequested commentary'] } },// The nine showcase kinds (spec 2, 2026-10-30). Script-bearing kinds render on the isolated render origin// (server/bench/render.js): the inert in-page sandbox allows no scripts, so a game, a canvas loop, three.js or// a shader cannot run there.@@@ -16,7 +16,7 @@ function home({ kinds = [], counts = { kinds: 0 }, benchmarks = [] } = {}) {return `<div class="bm-wrap bm-home" id="bm-home"><header class="bm-hero"><p class="bm-kicker">OpenVibe.Bench</p><h1>See what AI really makes</h1><p class="bm-lede">Every model on the same tasks: real pages, games, scenes and fixes, shown live side by side. Every run is published with what it cost and how long it took, and anyone can run it again.</p>--<div class="bm-actions"><a class="bm-btn" href="/benchmarks">Browse benchmarks</a><a class="bm-btn bm-btn-quiet" href="/showcase">Vote in the showcase</a></div></header>++<div class="bm-actions"><a class="bm-btn" href="/benchmarks">Browse benchmarks</a><a class="bm-btn bm-btn-quiet" href="/benchmarks/new">Make your own prompts</a><a class="bm-btn bm-btn-quiet" href="/showcase">Vote in the showcase</a></div></header><ul class="bm-why" aria-label="What Bench promises"><li><b>Outputs, not just scores</b><span>The pages, games and drawings themselves, live in sandboxed frames, next to the numbers.</span></li><li><b>Every run published</b><span>Whatever it scored. No private variants, no best of ten: the service that ran it publishes it.</span></li>@@@ -44,6 +44,7 @@ ${ep('GET', '/api/v1/rounds/:id', 'One round with entries; entries stay blind un${ep('GET', '/api/v1/boards/:kindId?type=elo|pareto|cost', 'A rendered board.')}${ep('GET', '/api/v1/models/:model', 'One model across every kind.')}${ep('GET', '/api/v1/datasets/:kindId.jsonl', 'The open dataset (prompts, entries, judgements without voter ids, ratings).')}++${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its visible tasks and every run, with no person id.')}</dl></section><section><h2>Write</h2><dl class="bm-eps">${ep('POST', '/api/v1/kinds/:kindId/tasks', '<code>{ prompt, task_class?, public? }</code>. A person or a service with bench.task.create.')}@@@ -55,10 +56,11 @@ ${ep('POST', '/api/v1/evals', 'Create a private task suite (never in the public${ep('POST', '/api/v1/evals/:id/run', 'Run a private eval against chosen offers, under the free daily cap.')}${ep('GET', '/api/v1/evals/:id/results', 'A private eval and its runs.')}</dl></section>--<section id="br-publish"><h2>Benchmarks and runs</h2><p class="bm-fine">A benchmark is a versioned set of tasks, judges and limits; a frozen version never changes once a run names it. A run published by a service (openvibe.codes, Run, a Node worker) is <b>verified</b>; one published with a person's token is <b>submitted</b>, labelled, and never ranked by default.</p><dl class="bm-eps">--${ep('GET', '/api/v1/benchmarks?kind=&tag=&q=&mine=1', 'The public benchmarks; <code>mine=1</code> adds your own unlisted ones.')}--${ep('GET', '/api/v1/benchmarks/:id[@version]', 'A definition. Held-out tasks keep their prompt for the executor and the owner.')}++<section id="br-publish"><h2>Benchmarks and runs</h2><p class="bm-fine">A benchmark is a versioned set of tasks, judges and limits; a frozen version never changes once a run names it. A run published by a service (openvibe.codes, Run, a Node worker) is <b>verified</b>; one published with a person's token is <b>submitted</b>, labelled, and never ranked by default. Your own prompts can be pasted at <a href="/benchmarks/new">/benchmarks/new</a>: 5 to 50 of them, private by default, with an unlisted share link or a public listing when you choose.</p><dl class="bm-eps">++${ep('GET', '/api/v1/benchmarks?kind=&tag=&q=&mine=1', 'The public benchmarks; <code>mine=1</code> adds your own unlisted and private ones.')}++${ep('GET', '/api/v1/benchmarks/:id[@version]', 'A definition. Held-out tasks keep their prompt for the executor and the owner; a private benchmark is a 404 to everyone else.')}${ep('POST', '/api/v1/benchmarks', 'Create one (<code>bench.benchmark.write</code>); <code>PUT</code> saves a draft, <code>POST …/versions</code> freezes it, <code>POST …/fork</code> forks it.')}++${ep('POST', '/api/v1/benchmarks/:id/share', '<code>{ visibility: "private" | "unlisted" | "public" }</code>. The owner only. An unlisted call returns a new share token once; the old link stops working.')}${ep('POST', '/api/v1/runs', 'Publish a run: <code>{ benchmark: "bmk_…@N", model, provider, harness: { name, version }, executor, tasks: [{ task_id, score, cost_usd, ms, tokens, artifacts }], score, cost_usd, ms, tokens }</code>.')}${ep('GET', '/api/v1/runs/:id', 'A run with its tasks; <code>/api/v1/compare?runs=a,b</code> puts up to four side by side.')}${ep('GET', '/api/v1/rankings?benchmark=&metric=score|value|speed|consistency&harness=&since=', 'Medians per model over verified runs; value is score per dollar, speed is score per minute.')}@@@ -154,7 +154,44 @@ function metricTabs(base, current) {function rankingsTable(entries, metric) {if (!entries.length) return '';const col = { value: (e) => (e.value == null ? '—' : `${fmtScore(e.value)} / $`), speed: (e) => (e.speed == null ? '—' : `${fmtScore(e.speed)} / min`), consistency: (e) => (e.consistency == null ? '—' : `±${fmtScore(e.consistency)}`) }[metric];-- return html`<div class="br-table-wrap"><table class="br-table"><thead><tr><th scope="col">#</th><th scope="col">Model</th><th scope="col">Score</th>${col ? html`<th scope="col">${METRICS.find((m) => m.id === metric).label}</th>` : ''}<th scope="col" class="br-num">Cost / run</th><th scope="col" class="br-num">Time</th><th scope="col" class="br-num">Runs</th></tr></thead><tbody>${entries.map((e, i) => html`<tr><td class="br-rank">${i + 1}</td><td><a class="br-model" href="${modelHref(e.model)}">${e.model}</a></td><td>${scoreBar(e.score)}</td>${col ? html`<td class="br-num">${col(e)}</td>` : ''}<td class="br-num">${fmtUsd(e.cost_usd)}</td><td class="br-num">${fmtMs(e.ms)}</td><td class="br-num">${e.runs}</td></tr>`)}</tbody></table></div>`;++ const bands = metric === 'score' && entries.some((e) => e.band);++ return html`<div class="br-table-wrap"><table class="br-table br-rank-table"><thead><tr><th scope="col">#</th>${bands ? html`<th scope="col">Band</th>` : ''}<th scope="col">Model</th><th scope="col">Score</th>${col ? html`<th scope="col">${METRICS.find((m) => m.id === metric).label}</th>` : ''}<th scope="col" class="br-num">Cost / run</th><th scope="col" class="br-num">Time</th><th scope="col" class="br-num">Runs</th></tr></thead><tbody>${entries.map((e, i) => html`<tr><td class="br-rank">${i + 1}</td>${bands ? html`<td class="br-c-band"><span class="rt-band">${e.band || '—'}</span></td>` : ''}<td class="br-c-model"><a class="br-model" href="${modelHref(e.model)}">${e.model}</a></td><td class="br-c-score">${scoreBar(e.score)}</td>${col ? html`<td class="br-num br-c-metric" data-label="${METRICS.find((m) => m.id === metric).label}">${col(e)}</td>` : ''}<td class="br-num br-c-cost" data-label="cost/run">${fmtUsd(e.cost_usd)}</td><td class="br-num br-c-time" data-label="time">${fmtMs(e.ms)}</td><td class="br-num br-c-runs" data-label="runs">${e.runs}</td></tr>`)}</tbody></table></div>`;++}++++/**++ * One leaderboard in the same `br-` table language for a person's own prompts and the public board beside it.++ * Bands are the overlapping score intervals (2nd–4th, not a fake third place); the cost is per solved task, the++ * rules' value metric (the median cost of an attempt divided by the success rate).++ */++function promptLeaderboard(entries, { emptyText = 'No runs yet.', caption = 'Models in rank bands, with their score, cost per solved task and run count' } = {}) {++ if (!entries.length) return html`<div class="bm-empty"><p>${emptyText}</p></div>`;++ return html`<div class="br-table-wrap"><table class="br-table rt-table br-prompt-table">++<caption class="br-sr">${caption}</caption>++<thead><tr><th scope="col">Band</th><th scope="col">Model</th><th scope="col">Score</th><th scope="col" class="br-num">Cost / solved task</th><th scope="col" class="br-num">Runs</th></tr></thead>++<tbody>${entries.map((e) => html`<tr data-model="${e.model}"><td class="rt-col-band"><span class="rt-band">${e.band || '—'}</span></td><td class="rt-col-model"><a class="rt-model br-model" href="${modelHref(e.model)}">${e.model}</a></td><td class="rt-col-controlled">${scoreBar(e.score)}${e.spread ? html`<span class="br-fine br-spread">${fmtScore(e.min_score)}–${fmtScore(e.max_score)} over ${plural(e.runs, 'run')}</span>` : ''}</td><td class="rt-col-raw br-num rt-prompt-cost">${e.cost_per_solved == null ? '—' : fmtUsd(e.cost_per_solved)}</td><td class="rt-col-votes br-num">${e.runs}</td></tr>`)}</tbody></table></div>`;++}++++/** The owner's leaderboard beside the public one for the same kind; the two are never mixed. */++function promptBoards({ ownEntries = [], publicBoard = null, ownRuns = 0, visibility = 'private' }) {++ const mineLabel = `Your prompts, ${ownRuns} ${ownRuns === 1 ? 'run' : 'runs'}`;++ const publicEntries = (publicBoard && publicBoard.entries) || [];++ return html`<section class="bm-section br-your-prompts" id="br-rank" aria-labelledby="br-yours-h">++<div class="br-duo">++<section class="br-duo-pane" aria-labelledby="br-yours-h">++<div class="bm-head-row"><h2 id="br-yours-h">${mineLabel}</h2><span class="br-chip br-vis">${visibility}</span></div>++<p class="br-fine">Your own tasks, your runs. This board is never mixed into the public ranks.</p>++${ownEntries.length ? valueChart(ownEntries) : ''}++${promptLeaderboard(ownEntries, { emptyText: 'No runs yet. Start it on 3 to 6 models; every run lands here, whatever it scores.' })}++</section>++<section class="br-duo-pane" aria-labelledby="br-public-h">++<div class="bm-head-row"><h2 id="br-public-h">The public board for this kind</h2>${publicBoard && publicBoard.benchmark ? html`<a class="bm-more" href="${benchmarkHref(publicBoard.benchmark.id)}">${publicBoard.benchmark.title}</a>` : ''}</div>++<p class="br-fine">Other people's runs of the public set, ranked the same way. Your prompts are not in it.</p>++${publicEntries.length++ ? promptLeaderboard(publicEntries, { emptyText: '' })++ : empty('No public board yet', 'No public benchmark of this kind has verified runs yet, so there is nothing to stand beside. Your own board still counts your runs.')}++</section>++</div>++</section>`;}// ---- runs list -----------------------------------------------------------------------------------------------@@@ -181,28 +218,92 @@ function artifact(a, label) {// ---- pages -----------------------------------------------------------------------------------------------------function benchmarksIndex({ benchmarks = [], kinds = [], kind = null, q = '' } = {}) {++function benchmarksIndex({ benchmarks = [], kinds = [], kind = null, q = '', mine = [], signedIn = false } = {}) {const families = [...new Set(kinds.map((k) => String(k).split('.')[0]))].sort();const chip = (label, value) => {const on = (value || null) === (kind || null);const href = value ? `/benchmarks?kind=${encodeURIComponent(value)}` : '/benchmarks';return html`<a class="br-filter" href="${href}"${on ? raw(' aria-current="page"') : ''}>${label}</a>`;};++ const own = mine.length++ ? html`<section class="bm-section br-mine" aria-labelledby="br-mine-h"><div class="bm-head-row"><h2 id="br-mine-h">Your benchmarks</h2><span class="bm-count">${mine.length}</span></div>++<p class="br-fine">Private ones are yours alone. An unlisted one opens with its share link; a public one is listed below.</p>++<ul class="br-cards">${mine.map(benchmarkCard)}</ul></section>`++ : '';return html`<div class="bm-wrap br" id="br-benchmarks">--<header class="bm-page-h"><p class="bm-kicker">Benchmarks</p><h1>Benchmarks anyone can run again</h1>--<p class="bm-lede">Each one is a versioned set of tasks with its judges and limits. A version never changes once a run names it, so every score here can be repeated.</p></header>++<header class="bm-page-h"><div class="bm-head-row"><div><p class="bm-kicker">Benchmarks</p><h1>Benchmarks anyone can run again</h1>++<p class="bm-lede">Each one is a versioned set of tasks with its judges and limits. A version never changes once a run names it, so every score here can be repeated.</p></div><a class="bm-btn" href="/benchmarks/new">Make your own</a></div></header><form class="br-search" method="get" action="/benchmarks" role="search"><label class="br-sr" for="br-q">Search benchmarks</label><input id="br-q" name="q" type="search" value="${q}" placeholder="Search by title, tag or kind" autocomplete="off">${kind ? html`<input type="hidden" name="kind" value="${kind}">` : ''}<button class="bm-btn bm-btn-small" type="submit">Search</button></form><nav class="br-filters" aria-label="Filter by family">${chip('All', null)}${families.map((f) => chip(f, f))}</nav>++${own}++${signedIn ? '' : html`<p class="br-fine br-mine-hint">Signed in, you can paste 5 to 50 of your own prompts as a private benchmark: <a href="/benchmarks/new">make your own</a>.</p>`}${benchmarks.length ? html`<ul class="br-cards">${benchmarks.map(benchmarkCard)}</ul>` : empty('No benchmark matches', q ? 'Try a shorter search, or clear the filter.' : 'None are published yet.', html`<a class="bm-btn bm-btn-quiet bm-btn-small" href="/benchmarks">Show all</a>`)}</div>`;}function benchmarkCard(b) {const tasks = ((b.draft && b.draft.tasks) || []).length;-- return html`<li class="br-card"><a class="br-card-link" href="${benchmarkHref(b.id)}"><span class="br-card-top">${kindChip(b.kind)}<span class="br-card-v">v${b.latest_version || 0}</span></span><span class="br-card-title">${b.title}</span><span class="br-card-desc">${b.description}</span><span class="br-card-foot">${plural(tasks, 'task')}${b.runs != null ? html` · ${plural(b.runs, 'run')}` : ''}${b.parent ? ' · a fork' : ''}</span></a></li>`;++ const vis = b.visibility && b.visibility !== 'public'++ ? html`<span class="br-chip br-vis">${b.visibility === 'private' ? 'private' : 'unlisted'}</span>`++ : '';++ return html`<li class="br-card"><a class="br-card-link" href="${benchmarkHref(b.id)}"><span class="br-card-top">${kindChip(b.kind)}<span class="br-card-v">v${b.latest_version || 0}</span>${vis}</span><span class="br-card-title">${b.title}</span><span class="br-card-desc">${b.description}</span><span class="br-card-foot">${plural(tasks, 'task')}${b.runs != null ? html` · ${plural(b.runs, 'run')}` : ''}${b.parent ? ' · a fork' : ''}</span></a></li>`;++}++++/** The owner's sharing controls: private, an unlisted link, or public. Plain form posts, no JavaScript. */++function shareBox({ benchmark, shareUrl = null, error = null }) {++ const visibility = benchmark.visibility || 'private';++ const note = visibility === 'private'++ ? 'Private: only you can see this benchmark and its prompts. No link works.'++ : visibility === 'unlisted'++ ? 'Unlisted: anyone with the link can see it. It is not in any list, search, sitemap or the open data.'++ : 'Public: it is in the benchmark list, the sitemap and the open data, with every run published.';++ return html`<section class="bm-section br-share" id="br-share" aria-labelledby="br-share-h">++<div class="bm-head-row"><h2 id="br-share-h">Sharing</h2><span class="br-chip br-vis">${visibility}</span></div>++<p class="br-fine">${note}</p>++${error ? html`<div class="bm-note is-bad" role="alert">${error}</div>` : ''}++${shareUrl ? html`<div class="bm-note br-share-now" role="status"><p>Your unlisted link is ready. It is shown once; copy it now. Share it again for a new link; the old one stops working.</p><label class="br-field"><span>The link</span><input class="br-share-link" type="text" readonly value="${shareUrl}" onfocus="this.select()"></label></div>` : ''}++<form class="br-share-form" method="post" action="/benchmarks/${encodeURIComponent(benchmark.id)}/share">++<button class="bm-btn" type="submit" name="visibility" value="unlisted">${visibility === 'unlisted' ? 'New unlisted link' : 'Share an unlisted link'}</button>++<button class="bm-btn bm-btn-quiet" type="submit" name="visibility" value="public">Publish public</button>++<button class="bm-btn bm-btn-quiet" type="submit" name="visibility" value="private">Make private</button>++</form>++<p class="br-fine">A person with the unlisted link can read the prompts and the runs. Only you can start a run on it.</p>++</section>`;++}++++/** The new-benchmark form: 5–50 pasted prompts, one per line or CSV with an optional expected answer. */++function newBenchmarkPage({ kinds = [], values = {}, error = null, signedIn = false, loginUrl = '/auth/login' }) {++ const head = html`<nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><span>Make your own</span></nav>++<header class="bm-page-h"><p class="bm-kicker">Your prompts</p><h1>Make a benchmark from your own prompts</h1>++<p class="bm-lede">Paste 5 to 50 prompts you really use. They become the tasks of a private benchmark: no model is called and nothing is charged until you run it on openvibe.codes.</p></header>`;++ if (!signedIn) {++ return html`<div class="bm-wrap br bm-narrow" id="br-new">${head}${empty('Sign in first', 'A benchmark of your own is attached to your account, so the prompts can be exported or deleted with you.', html`<a class="bm-btn bm-btn-small" href="${loginUrl}">Sign in with OpenVibe</a>`)}</div>`;++ }++ const kindList = [...kinds].sort((a, b) => (a.id === 'text.answer' ? -1 : b.id === 'text.answer' ? 1 : String(a.name || a.id).localeCompare(String(b.name || b.id))));++ const kind = values.kind || 'text.answer';++ const format = values.format || 'auto';++ return html`<div class="bm-wrap br bm-narrow" id="br-new">${head}++${error ? html`<div class="bm-note is-bad" role="alert">${error}</div>` : ''}++<form class="br-form br-new-form" method="post" action="/benchmarks/new">++<label class="br-field"><span>Name (optional)</span><input name="title" type="text" maxlength="160" value="${values.title || ''}" placeholder="Your prompts, e.g. the support replies I really send" autocomplete="off"></label>++<label class="br-field"><span>Description (optional)</span><textarea name="description" rows="3" maxlength="4000" placeholder="What these prompts are for.">${values.description || ''}</textarea></label>++<label class="br-field"><span>Kind</span><select name="kind">${kindList.map((k) => html`<option value="${k.id}"${k.id === kind ? raw(' selected') : ''}>${k.name || k.id}${k.id === 'text.answer' ? ' (a plain text answer)' : ''}</option>`)}</select></label>++<fieldset class="br-paste"><legend>Your prompts</legend>++<label class="br-field"><span>Format</span><select name="format">++<option value="auto"${format === 'auto' ? raw(' selected') : ''}>Auto-detect</option>++<option value="lines"${format === 'lines' ? raw(' selected') : ''}>One prompt per line</option>++<option value="csv"${format === 'csv' ? raw(' selected') : ''}>CSV: prompt, expected answer (optional)</option>++</select></label>++<label class="br-field"><span>Paste 5 to 50 prompts</span><textarea class="br-prompts" name="prompts" rows="12" required placeholder="Write a haiku about rain.\nExplain a database index to a new developer.\nWhat is 17 × 24?">${values.prompts || ''}</textarea></label>++<p class="br-fine">One per line, or CSV with an optional expected answer in the second column. A quoted field may hold commas. The benchmark is private by default; you can share a link or publish it after it exists.</p>++</fieldset>++<div class="bm-actions"><button class="bm-btn" type="submit">Make my benchmark</button><a class="bm-btn bm-btn-quiet" href="/benchmarks">Cancel</a></div>++</form>++<p class="br-fine">The prompts are never logged. They are exported and deleted with your account, like the rest of your data, and they are only in a public page if you publish the benchmark.</p>++</div>`;}--function benchmarkPage({ benchmark, definition, version, rankings, metric = 'score', runs = [], codesRunUrl = null, viewer = null, baseUrl = '' }) {++function benchmarkPage({ benchmark, definition, version, rankings, metric = 'score', runs = [], codesRunUrl = null, viewer = null, baseUrl = '', ownBoards = null, share = null, shared = false, owner = false }) {const d = definition || {};const tasks = Array.isArray(d.tasks) ? d.tasks : [];const shown = tasks.filter((t) => !t.holdout && t.prompt);@@@ -211,21 +312,34 @@ function benchmarkPage({ benchmark, definition, version, rankings, metric = 'scoconst limits = d.limits || {};const entries = (rankings && rankings.entries) || [];const base = benchmarkHref(benchmark.id);-- const runYourself = codesRunUrl-- ? html`<a class="bm-btn" href="${benchmarkHref(benchmark.id)}/run">Run it yourself</a>`-- : html`<a class="bm-btn bm-btn-quiet" href="/docs#br-publish">Publish a run</a>`;++ const canRun = Boolean(codesRunUrl) && !shared;++ const runYourself = shared++ ? ''++ : canRun++ ? html`<a class="bm-btn" href="${benchmarkHref(benchmark.id)}/run">Run it yourself</a>`++ : html`<a class="bm-btn bm-btn-quiet" href="/docs#br-publish">Publish a run</a>`;++ const byline = owner ? 'by you' : shared ? 'by its owner' : `by ${benchmark.owner}`;++ const privateNote = benchmark.visibility === 'private'++ ? html`<div class="bm-note br-private" role="status"><b>Private.</b> Only you can see this benchmark and its prompts. It is not in any list, search or the open data.</div>`++ : shared++ ? html`<div class="bm-note br-shared" role="status"><b>Shared by its owner.</b> This page is not in any list or search engine. The link can stop working if the owner makes it private.</div>`++ : '';return html`<div class="bm-wrap br" id="br-benchmark"><nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><span>${benchmark.title}</span></nav>--<header class="br-head"><div class="br-head-main"><p class="br-head-meta">${kindChip(benchmark.kind)}<span>v${version || benchmark.latest_version || 0}</span><span>by ${benchmark.owner}</span>${benchmark.parent ? html`<a href="${benchmarkHref(benchmark.parent)}">forked</a>` : ''}</p>++<header class="br-head"><div class="br-head-main"><p class="br-head-meta">${kindChip(benchmark.kind)}<span>v${version || benchmark.latest_version || 0}</span><span>${byline}</span>${benchmark.visibility !== 'public' ? html`<span class="br-chip br-vis">${benchmark.visibility}</span>` : ''}${benchmark.parent ? html`<a href="${benchmarkHref(benchmark.parent)}">forked</a>` : ''}</p><h1>${benchmark.title}</h1>${benchmark.description ? html`<p class="bm-lede">${benchmark.description}</p>` : ''}</div>--<div class="br-head-act">${runYourself}<p class="br-fine">${codesRunUrl ? 'On openvibe.codes, with your own keys. The result is published here.' : 'Running it on openvibe.codes with your own keys is coming. Today a run is published through the API.'}</p></div></header>++<div class="br-head-act">${runYourself}<p class="br-fine">${canRun ? 'On openvibe.codes, with your own keys or the free pool through OpenVibe.AI. The result is published here.' : codesRunUrl ? 'Only the owner starts runs on this benchmark. Its results are readable here.' : 'Running it on openvibe.codes with your own keys is coming. Today a run is published through the API.'}</p></div></header>++${privateNote}<dl class="br-facts"><div><dt>Tasks</dt><dd>${tasks.length}${hidden ? html` <span class="br-fine">(${hidden} hidden)</span>` : ''}</dd></div><div><dt>Judged by</dt><dd>${judges.length ? judges.map((j) => String(j.type).replace(/_/g, ' ')).join(', ') : '—'}</dd></div><div><dt>Limits</dt><dd>${[limits.ms ? fmtMs(limits.ms) : null, limits.tokens ? `${fmtTokens(limits.tokens)} tokens` : null, limits.usd ? fmtUsd(limits.usd) : null].filter(Boolean).join(' · ') || '—'}</dd></div><div><dt>Verified runs</dt><dd>${entries.reduce((n, e) => n + e.runs, 0)}</dd></div></dl>--<section class="bm-section" id="br-rank" aria-labelledby="br-rank-h"><div class="bm-head-row"><h2 id="br-rank-h">Rankings</h2>${metricTabs(base, metric)}</div>++${owner ? shareBox({ benchmark, shareUrl: share && share.shareUrl, error: share && share.error }) : ''}++${ownBoards++ ? promptBoards(ownBoards)++ : html`<section class="bm-section" id="br-rank" aria-labelledby="br-rank-h"><div class="bm-head-row"><h2 id="br-rank-h">Rankings</h2>${metricTabs(base, metric)}</div>${valueChart(entries)}${entries.length ? rankingsTable(entries, metric) : empty('No verified runs yet', 'A run counts here once openvibe.codes, Run or a Node worker ran it and published it. Submitted runs are listed below but never ranked.')}--</section>++</section>`}<section class="bm-section" aria-labelledby="br-tasks-h"><div class="bm-head-row"><h2 id="br-tasks-h">Tasks</h2><span class="bm-count">${tasks.length}</span></div>--<ol class="br-tasks">${shown.map((t) => html`<li class="br-task"><span class="br-task-id br-mono">${t.id}</span><p>${t.prompt}</p></li>`)}${hidden ? html`<li class="br-task is-hidden"><span class="br-task-id br-mono">${plural(hidden, 'task')}</span><p>Held out: only the executor sees ${hidden === 1 ? 'it' : 'them'}, so nobody can tune for the answer.</p></li>` : ''}</ol></section>++<ol class="br-tasks">${shown.map((t) => html`<li class="br-task"><span class="br-task-id br-mono">${t.id}</span><p>${t.prompt}</p>${t.expected && t.expected.answer ? html`<p class="br-fine">Expected: ${String(t.expected.answer).slice(0, 300)}</p>` : ''}</li>`)}${hidden ? html`<li class="br-task is-hidden"><span class="br-task-id br-mono">${plural(hidden, 'task')}</span><p>Held out: only the executor sees ${hidden === 1 ? 'it' : 'them'}, so nobody can tune for the answer.</p></li>` : ''}</ol></section><section class="bm-section" aria-labelledby="br-latest-h"><div class="bm-head-row"><h2 id="br-latest-h">Latest runs</h2>${runs.length > 1 ? html`<a class="bm-more" href="${compareHref(runs.slice(0, 2).map((r) => r.id))}">Compare the newest two</a>` : ''}</div>${runs.length ? runsList(runs) : empty('No runs yet', 'The first run shows up here, verified or submitted.')}</section>${benchmarkUseThis({ baseUrl, benchmark, entries, viewer })}@@@ -250,17 +364,17 @@ function runForm({ benchmark, definition, version, signedIn, harnesses = [], sugfor (const id of p.harnesses) if (!harnessList.some((h) => h.id === id)) harnessList.push({ id, label: id });const base = benchmarkHref(benchmark.id);const head = html`<nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><a href="${base}">${benchmark.title}</a><span aria-hidden="true">/</span><span>Run it</span></nav>--<header class="bm-page-h"><p class="bm-kicker">Run it yourself</p><h1>${benchmark.title}</h1><p class="bm-lede">openvibe.codes runs every task on each model and harness you pick, with your own keys, and publishes each run here as verified, whatever it scores.</p></header>`;++<header class="bm-page-h"><p class="bm-kicker">Run it yourself</p><h1>${benchmark.title}</h1><p class="bm-lede">Pick 3 to 6 models. openvibe.codes runs every task on each model and harness, with your own keys through OpenVibe.AI or the free pool, and publishes each run here as verified, whatever it scores.</p></header>`;if (!signedIn) {-- return html`<div class="bm-wrap br bm-narrow" id="br-run-form">${head}${empty('Sign in first', 'A run uses your own provider keys on openvibe.codes, so it starts as you.', html`<a class="bm-btn bm-btn-small" href="${loginUrl}">Sign in with OpenVibe</a>`)}</div>`;++ return html`<div class="bm-wrap br bm-narrow" id="br-run-form">${head}${empty('Sign in first', 'A run uses your own provider keys, or the free pool, through OpenVibe.AI on openvibe.codes, so it starts as you.', html`<a class="bm-btn bm-btn-small" href="${loginUrl}">Sign in with OpenVibe</a>`)}</div>`;}const check = (name, value, label, on) => html`<label class="br-check"><input type="checkbox" name="${name}" value="${value}"${on ? raw(' checked') : ''}><span>${label}</span></label>`;return html`<div class="bm-wrap br bm-narrow" id="br-run-form">${head}${error ? html`<div class="bm-note is-bad" role="alert">${error}</div>` : ''}--${confirm === 'unknown' ? html`<div class="bm-note" role="status">This benchmark sets no dollar limit per task, so what it costs depends on the models you pick. It runs on your own keys. Check the choices below, then confirm.</div>` : confirm != null ? html`<div class="bm-note" role="status">This run may cost up to <b>${fmtUsd(confirm)}</b> on your own keys. Check the choices below, then confirm.</div>` : ''}++${confirm === 'unknown' ? html`<div class="bm-note" role="status">This benchmark sets no dollar limit per task, so what it costs depends on the models you pick. It runs on your own keys or the free pool, through OpenVibe.AI. Check the choices below, then confirm.</div>` : confirm != null ? html`<div class="bm-note" role="status">This run may cost up to <b>${fmtUsd(confirm)}</b> on your own keys. Check the choices below, then confirm.</div>` : ''}<form class="br-form" method="post" action="${base}/run" data-tasks="${tasks}" data-usd="${usd}"><fieldset><legend>Models</legend>${modelList.length ? html`<div class="br-checks">${modelList.map((m) => check('models', m, m, p.models.includes(m)))}</div>` : ''}--<label class="br-field"><span>${modelList.length ? 'More models, separated by commas' : 'Models, separated by commas'}</span><input name="more_models" type="text" autocomplete="off" placeholder="deepseek-v4.1-flash, kimi-k3"></label></fieldset>++<label class="br-field"><span>${modelList.length ? 'More models, separated by commas' : 'Models, separated by commas (3 to 6)'}</span><input name="more_models" type="text" autocomplete="off" placeholder="deepseek-v4.1-flash, kimi-k3"></label></fieldset><fieldset><legend>Harnesses</legend><div class="br-checks">${harnessList.map((h) => check('harnesses', h.id, h.label, p.harnesses.includes(h.id)))}</div></fieldset><label class="br-field"><span>Runs of each</span><select name="repeat">${[1, 2, 3, 5].map((n) => html`<option value="${n}"${n === Number(p.repeat) ? raw(' selected') : ''}>${n === 1 ? 'Once' : `${n} times (shows how steady it is)`}</option>`)}</select></label><p class="br-fine" data-estimate>${usd > 0 ? `${tasks} ${tasks === 1 ? 'task' : 'tasks'} × your picks × runs, at up to ${fmtUsd(usd)} a task. Over $1 asks you once more.` : `${tasks} ${tasks === 1 ? 'task' : 'tasks'} × your picks × runs. This benchmark sets no dollar limit per task, so you confirm before it starts.`}</p>@@@ -337,6 +451,7 @@ ${modelUseThis({ baseUrl, model, rows, viewer })}}module.exports = {-- benchmarksIndex, benchmarkPage, runForm, runPage, comparePage, modelPage, valueChart, METRICS,++ benchmarksIndex, benchmarkPage, newBenchmarkPage, promptLeaderboard, promptBoards, shareBox,++ runForm, runPage, comparePage, modelPage, valueChart, METRICS,fmtScore, fmtUsd, fmtMs, fmtTokens, ordinal, benchmarkCard, render,};@@@ -28,6 +28,8 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;let judgementId;let sharedBenchmark;let lonelyBenchmark;++ let privateBenchmark;++ let sharedPromptBenchmark;const signedEvent = (event) => {const raw = JSON.stringify({ event });@@@ -73,6 +75,22 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;await db.exec(`INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)VALUES ($1, 1, '{"kind":"design.page"}', $2, $3, now())`, [id, 'a'.repeat(64), owner]);}++ // Her own prompts: a private benchmark with a service-run on it. Both go with her.++ privateBenchmark = ulid('bmk', 'AAC');++ await db.exec(`INSERT INTO bench_benchmarks (id, owner, title, kind, visibility, share_token_hash, created_by, created_at, updated_at)++ VALUES ($1, $2, 'Private prompts', 'text.answer', 'private', $3, $2, now(), now())`, [privateBenchmark, alice.subject, 'b'.repeat(64)]);++ await db.exec(`INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)++ VALUES ($1, 1, $2::jsonb, $3, $4, now())`, [privateBenchmark, JSON.stringify({ kind: 'text.answer', tasks: [{ id: 't1', prompt: 'A very private prompt ZEBRA', holdout: false }] }), 'c'.repeat(64), alice.subject]);++ await db.exec(`INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, provider, provenance, created_by, created_at)++ VALUES ($1, $2, 1, 'model-private', 'acme', 'verified', 'svc:private-exec', now())`, [ulid('brn', 'AAC'), privateBenchmark]);++ // A prompt benchmark she shared: it still goes with her, because the prompts are hers.++ sharedPromptBenchmark = ulid('bmk', 'AAD');++ await db.exec(`INSERT INTO bench_benchmarks (id, owner, title, kind, visibility, tags, share_token_hash, created_by, created_at, updated_at)++ VALUES ($1, $2, 'Shared prompts', 'text.answer', 'unlisted', '["your-prompts"]'::jsonb, $3, $2, now(), now())`, [sharedPromptBenchmark, alice.subject, 'd'.repeat(64)]);++ await db.exec(`INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)++ VALUES ($1, 1, $2::jsonb, $3, $4, now())`, [sharedPromptBenchmark, JSON.stringify({ kind: 'text.answer', tasks: [{ id: 't1', prompt: 'A shared private prompt ZEBRA', holdout: false }] }), 'e'.repeat(64), alice.subject]);++ await db.exec(`INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, provider, provenance, created_by, created_at)++ VALUES ($1, $2, 1, 'model-shared', 'acme', 'verified', $3, now())`, [ulid('brn', 'AAD'), sharedPromptBenchmark, carol.subject]);// Her run on the benchmark, and Carol's run on the same one.await db.exec(`INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, provider, provenance, created_by, created_at)VALUES ($1, $2, 1, 'model-a', 'openai', 'submitted', $3, now())`, [ulid('brn', 'AAA'), sharedBenchmark, alice.subject]);@@@ -101,9 +119,15 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;assert.ok(part.body.files.some((f) => f.name === 'bench_runs.json' && f.content.length === 1));assert.ok(part.body.files.some((f) => f.name === 'bench_battle_seen.json' && f.content.length === 1));const benchmarks = part.body.files.find((f) => f.name === 'benchmarks.json');-- assert.ok(benchmarks && benchmarks.content.length === 2, 'her benchmarks are in the part');++ assert.ok(benchmarks && benchmarks.content.length === 4, 'her benchmarks are in the part');const versions = part.body.files.find((f) => f.name === 'benchmark_versions.json');-- assert.ok(versions && versions.content.length === 2, 'the versions of her benchmarks are in the part');++ assert.ok(versions && versions.content.length === 4, 'the versions of her benchmarks are in the part');++ assert.ok(JSON.stringify(versions).includes('A very private prompt ZEBRA'), 'her private prompts are in her export');++ assert.ok(JSON.stringify(versions).includes('A shared private prompt ZEBRA'), 'her shared prompts are in her export');++ const privateRuns = part.body.files.find((f) => f.name === 'benchmark_runs.json');++ assert.ok(privateRuns && privateRuns.content.length === 2, 'the runs on her prompt benchmarks are in her export');++ assert.ok(privateRuns.content.some((r) => r.model === 'model-private'));++ assert.ok(privateRuns.content.some((r) => r.model === 'model-shared'));assert.ok(!JSON.stringify(part.body).includes(carol.subject), 'nobody else\'s rows');});@@@ -117,8 +141,8 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;assert.strictEqual(confirm.body.subject, alice.subject);assert.strictEqual(confirm.body.erased.bench_tasks, 1);assert.strictEqual(confirm.body.erased.bench_rounds, 1);-- assert.strictEqual(confirm.body.erased.bench_runs, 1, 'her run is gone');-- assert.strictEqual(confirm.body.erased.bench_benchmarks, 1, 'the benchmark nobody else ran is gone');++ assert.strictEqual(confirm.body.erased.bench_runs, 3, 'her run and the runs on her prompt benchmarks are gone');++ assert.strictEqual(confirm.body.erased.bench_benchmarks, 3, 'the benchmark nobody else ran and both prompt benchmarks are gone');assert.strictEqual(confirm.body.erased.bench_battle_seen, 1, 'the pairs she had seen are gone');// Three rows stay without her: her judgement, the benchmark other people ran and its frozen version.assert.strictEqual(confirm.body.retained.tombstones, 3, 'her judgement, the benchmark others ran and its version stay');@@@ -135,6 +159,12 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;assert.strictEqual(kept.created_by, 'deleted');assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [sharedBenchmark])), 1, 'Carol\'s run is still there');assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmarks WHERE id = $1', [lonelyBenchmark])), 0, 'the benchmark nobody ran is gone');++ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmarks WHERE id = $1', [privateBenchmark])), 0, 'the private benchmark is gone');++ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmark_versions WHERE benchmark_id = $1', [privateBenchmark])), 0, 'its prompts go with it');++ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [privateBenchmark])), 0, 'the run on it is gone too');++ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmarks WHERE id = $1', [sharedPromptBenchmark])), 0, 'a shared prompt benchmark goes too');++ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmark_versions WHERE benchmark_id = $1', [sharedPromptBenchmark])), 0, 'its shared prompts go with it');++ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [sharedPromptBenchmark])), 0, 'its run goes too');if (await db.value("SELECT to_regclass('bench_watches') IS NOT NULL")) {assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_watches WHERE subject = $1', [alice.subject])), 0, 'her watch is gone');}@@@ -17,7 +17,7 @@ const { boot, check, done } = require('./helpers/boot');await check('kinds are seeded from code (the 15 of §4.12 plus the nine showcase kinds)', async () => {const r = await t.get('/api/v1/kinds');assert.strictEqual(r.status, 200, r.text);-- assert.strictEqual(r.json().kinds.length, 24);++ assert.strictEqual(r.json().kinds.length, 25);const ids = r.json().kinds.map((k) => k.id);for (const want of ['design.page', 'games.match', 'meme.caption', 'vision.answer', 'codes.patch', 'speech.tts', 'speech.asr', 'translate.ja', 'search.answer']) {assert.ok(ids.includes(want), `missing kind ${want}`);@@@ -58,7 +58,7 @@ const rpc = (method, params, id = 1) => ({ jsonrpc: '2.0', id, method, ...(paramawait check('bench.list_kinds lists every kind Bench runs', async () => {const r = await t.get('/mcp', { json: rpc('tools/call', { name: 'bench.list_kinds', arguments: {} }, 8) });assert.strictEqual(r.json().result.isError, false);-- assert.strictEqual(r.json().result.structuredContent.kinds.length, 24);++ assert.strictEqual(r.json().result.structuredContent.kinds.length, 25);});await check('bench.get_board renders a board without a person', async () => {@@@ -28,7 +28,7 @@ const HOSTILE = [const r = await t.get('/api/v1/kinds');assert.strictEqual(r.status, 200, r.text);const byId = new Map(r.json().kinds.map((k) => [k.id, k]));-- assert.strictEqual(r.json().kinds.length, 24);++ assert.strictEqual(r.json().kinds.length, 25);const want = ['games.play', 'games.three', 'design.svg', 'design.motion', 'design.ui', 'design.dataviz', 'design.model3d', 'design.shader', 'design.interface'];for (const id of want) {const k = byId.get(id);nnew file mode 100644@@@ -0,0 +1,298 @@++'use strict';++++/**++ * B11: a person's own prompts as a private benchmark.++ *++ * Under test: the paste parser (lines, CSV with an optional expected answer, 5–50 bounds); a private benchmark by++ * default; another person gets a 404 and nothing private is listed, in the sitemap or in the open data; the owner's++ * leaderboard sits beside the public board for the same kind (bands and cost per solved task, never mixed); the run++ * goes through the existing launcher and still asks before it starts; an unlisted share link with a long random++ * token, its "shared by its owner" page, noindex, and unlisting/unsharing/public/private; CSRF.++ */++const assert = require('assert');++const http = require('http');++const { boot, check, done } = require('./helpers/boot');++const { parsePrompts, MAX_PROMPTS } = require('../server/bench/your-prompts');++const { shareTokenMatches } = require('../server/bench/registry');++++function fakeCodes() {++ const calls = [];++ const server = http.createServer((req, res) => {++ let body = '';++ req.on('data', (c) => { body += c; });++ req.on('end', () => {++ calls.push({ method: req.method, url: req.url, headers: req.headers, body: body ? JSON.parse(body) : null });++ res.setHeader('content-type', 'application/json');++ if (req.url === '/api/v1/harnesses') return res.end(JSON.stringify({ harnesses: [{ id: 'codex', label: 'Codex' }, { id: 'claude-code', label: 'Claude Code' }] }));++ if (/^\/api\/v1\/benchmarks\/[^/]+\/run$/.test(req.url) && req.method === 'POST') {++ res.statusCode = 202;++ return res.end(JSON.stringify({ session: 'ses_b11', tasks: [{ id: 'tsk_b11' }], task: { id: 'tsk_b11' }, total: 1 }));++ }++ res.statusCode = 404; res.end('{}');++ });++ });++ return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ url: `http://127.0.0.1:${server.address().port}`, calls, close: () => server.close() })));++}++++(async () => {++ const codes = await fakeCodes();++ const t = await boot({ env: { CODES_BENCH_RUN_URL: codes.url } });++ const alice = t.network.addUser('alice');++ const bob = t.network.addUser('bob');++ const publisher = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.run.publish', 'bench.benchmark.write'] });++ const base = String(t.config.baseUrl).replace(/\/+$/, '');++ const MARK = 'ZEBRA-B11-PROMPT';++++ const form = (o) => new URLSearchParams(o).toString();++ const post = (path, o, opt = {}) => t.get(path, { as: alice, raw: form(o), headers: { 'content-type': 'application/x-www-form-urlencoded' }, ...opt });++ const share = (id, visibility) => post(`/benchmarks/${id}/share`, { visibility });++ const tokenOf = (res) => {++ const m = /\/b\/([A-Za-z0-9_-]{20,120})/.exec(res.text);++ assert.ok(m, `a share token in the page: ${res.text.slice(0, 300)}`);++ return m[1];++ };++ const publish = async (ref, model, scores, { cost = 0.03 } = {}) => {++ const tasks = scores.map((score, i) => ({ task_id: `t${i + 1}`, score, cost_usd: cost / scores.length, ms: 1000 + i, tokens: 100, artifacts: [] }));++ const body = {++ benchmark: ref, model, route: { provider: 'acme' }, provider: 'acme',++ harness: { name: 'codex', version: '1.0' }, settings: {}, executor: { kind: 'codes', id: 'tsk_b11' },++ tasks, score: scores.reduce((a, b) => a + b, 0) / scores.length, cost_usd: cost, ms: 2000, tokens: 600,++ };++ const res = await t.get('/api/v1/runs', { bearer: publisher, json: body });++ assert.strictEqual(res.status, 201, res.text);++ return res.json().run.id;++ };++++ let id;++ let version;++ let token;++ const prompts = Array.from({ length: 6 }, (_, i) => `${MARK} number ${i + 1}`);++++ await check('the parser takes one prompt per line and enforces 5–50', async () => {++ const good = parsePrompts(prompts.join('\n'), { format: 'lines' });++ assert.strictEqual(good.ok, true, good.detail);++ assert.strictEqual(good.prompts.length, 6);++ assert.ok(good.prompts.every((p) => p.expected === null));++ assert.strictEqual(parsePrompts(prompts.slice(0, 4).join('\n')).ok, false, 'four is too few');++ assert.strictEqual(parsePrompts(Array.from({ length: MAX_PROMPTS + 1 }, (_, i) => `p${i}`).join('\n')).ok, false, 'fifty-one is too many');++ });++++ await check('the parser reads CSV with an optional expected answer and quoted commas', async () => {++ const csv = 'prompt,expected\n"What is 2 + 2, exactly?","4"\n"Say hello, in French.",bonjour\nName one primary colour.,red\nWhat is 3 × 3?,9\nFinish the word: sun—,flower\n';++ const parsed = parsePrompts(csv);++ assert.strictEqual(parsed.ok, true, parsed.detail);++ assert.strictEqual(parsed.format, 'csv', 'a prompt/expected header is detected');++ assert.strictEqual(parsed.prompts.length, 5);++ assert.strictEqual(parsed.prompts[0].prompt, 'What is 2 + 2, exactly?');++ assert.strictEqual(parsed.prompts[0].expected, '4');++ assert.strictEqual(parsed.prompts[1].prompt, 'Say hello, in French.');++ assert.strictEqual(parsed.prompts[1].expected, 'bonjour');++ const noHeader = parsePrompts('one,two\nthree,four\nfive,six\nseven,eight\nnine,ten', { format: 'csv' });++ assert.strictEqual(noHeader.ok, true, noHeader.detail);++ assert.deepStrictEqual(noHeader.prompts[0], { prompt: 'one', expected: 'two' });++ });++++ await check('a share token is stored as a hash and checked in constant time', async () => {++ assert.strictEqual(shareTokenMatches('abc', 'a'.repeat(64)), false);++ assert.strictEqual(shareTokenMatches('abc', null), false);++ });++++ await check('the form makes a private benchmark from pasted prompts', async () => {++ const res = await post('/benchmarks/new', {++ title: 'My private prompts', description: 'Six real prompts.',++ kind: 'text.answer', format: 'lines', prompts: prompts.join('\n'),++ });++ assert.strictEqual(res.status, 303, res.text.slice(0, 400));++ id = res.headers.get('location').split('/').pop();++ const r = await t.get(`/api/v1/benchmarks/${id}`, { as: alice });++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.json().benchmark.visibility, 'private', 'private by default');++ assert.strictEqual(r.json().benchmark.title, 'My private prompts');++ assert.strictEqual(r.json().benchmark.latest_version, 1, 'frozen into a version');++ version = r.json().version.version;++ assert.strictEqual(r.json().definition.kind, 'text.answer');++ assert.strictEqual(r.json().definition.tasks.length, 6);++ assert.deepStrictEqual(r.json().definition.tasks.map((x) => x.id), ['t1', 't2', 't3', 't4', 't5', 't6']);++ assert.ok(r.json().definition.tasks.every((x) => x.prompt.startsWith(MARK)));++ assert.strictEqual(r.json().definition.tasks[0].expected, null);++ assert.strictEqual(r.json().definition.limits.usd, 0, 'no dollar cap is added or raised');++ assert.ok(!t.logs().includes(MARK), 'a prompt never reaches the log');++ });++++ await check('a CSV paste keeps the expected answer of each row', async () => {++ const csv = 'prompt,expected\nQ1,A1\nQ2,A2\nQ3,A3\nQ4,A4\nQ5,A5\n';++ const res = await post('/benchmarks/new', { title: 'CSV prompts', kind: 'design.svg', format: 'csv', prompts: csv });++ assert.strictEqual(res.status, 303, res.text.slice(0, 400));++ const cid = res.headers.get('location').split('/').pop();++ const r = await t.get(`/api/v1/benchmarks/${cid}`, { as: alice });++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.json().definition.kind, 'design.svg', 'an existing kind can be chosen');++ assert.deepStrictEqual(r.json().definition.tasks[0].expected, { answer: 'A1' });++ assert.strictEqual(r.json().definition.tasks.length, 5);++ });++++ await check('another person gets a 404 for the private benchmark everywhere', async () => {++ for (const [path, what] of [++ [`/benchmarks/${id}`, 'page'],++ [`/benchmarks/${id}/run`, 'run form'],++ [`/api/v1/benchmarks/${id}`, 'definition'],++ [`/api/v1/rankings?benchmark=${id}`, 'rankings'],++ [`/api/v1/percentile?benchmark=${id}&model=a&score=90`, 'percentile'],++ ]) {++ const res = await t.get(path, { as: bob });++ assert.strictEqual(res.status, 404, `${what}: ${res.status} ${res.text.slice(0, 200)}`);++ }++ const run = await post(`/benchmarks/${id}/run`, { models: 'a', harnesses: 'codex', repeat: '1' }, { as: bob });++ assert.strictEqual(run.status, 404, run.text.slice(0, 200));++ });++++ await check('a private benchmark is never in a list, the sitemap or the open data', async () => {++ const list = await t.get('/api/v1/benchmarks');++ assert.ok(!list.json().benchmarks.some((b) => b.id === id), 'not in the public list');++ const page = await t.get('/benchmarks');++ assert.ok(!page.text.includes('My private prompts'), 'not on the benchmarks page');++ const sitemap = await t.get('/sitemap.xml');++ assert.ok(!sitemap.text.includes(id), 'not in the sitemap');++ const dataset = await t.get('/api/v1/datasets/benchmarks.jsonl');++ assert.ok(!dataset.text.includes(MARK), 'no prompt in the open data');++ const mine = await t.get('/api/v1/benchmarks?mine=1', { as: alice });++ assert.ok(mine.json().benchmarks.some((b) => b.id === id), 'the owner lists it');++ assert.strictEqual(mine.headers.get('cache-control'), 'private, no-store', 'the owner list is not publicly cacheable');++ const theirs = await t.get('/api/v1/benchmarks?mine=1', { as: bob });++ assert.ok(!theirs.json().benchmarks.some((b) => b.id === id), 'nobody else does');++ });++++ await check('the owner sees the private page and its prompts', async () => {++ const res = await t.get(`/benchmarks/${id}`, { as: alice });++ assert.strictEqual(res.status, 200, res.text.slice(0, 200));++ assert.ok(res.text.includes('Private.'), 'the private notice');++ assert.ok(res.text.includes(prompts[0]), 'the owner sees the prompt');++ assert.ok(res.text.includes('noindex'), 'kept out of search');++ assert.ok(res.text.includes('Share an unlisted link'), 'the owner can share');++ assert.strictEqual(res.headers.get('cache-control'), 'private, no-store');++ });++++ await check('runs published for the owner make the two separate leaderboards', async () => {++ // A public benchmark of the same kind, with a run, is "the public one beside it".++ const pub = await t.get('/api/v1/benchmarks', { bearer: publisher, json: {++ title: 'Public text answers', description: 'The public board.', kind: 'text.answer', tags: [], visibility: 'public',++ tasks: [{ id: 't1', prompt: 'A public question.', files: [], expected: null, holdout: false }],++ judges: [{ type: 'automatic_grader', version: '1.0', config: {} }], scoring: { method: 'mean', scale: { min: 0, max: 100 } },++ limits: { ms: 60000, tokens: 0, usd: 0 }, matrix: { models: [], harnesses: [] },++ } });++ assert.strictEqual(pub.status, 201, pub.text);++ const pubId = pub.json().benchmark.id;++ assert.strictEqual((await t.get(`/api/v1/benchmarks/${pubId}/versions`, { bearer: publisher, json: {} })).status, 201);++ await publish(`${pubId}@1`, 'public-model', [90], { cost: 0.01 });++++ await publish(`${id}@${version}`, 'model-a', [80], { cost: 0.03 });++ await publish(`${id}@${version}`, 'model-a', [95], { cost: 0.03 });++ await publish(`${id}@${version}`, 'model-b', [85], { cost: 0.05 });++ await publish(`${id}@${version}`, 'model-b', [96], { cost: 0.05 });++ await publish(`${id}@${version}`, 'model-c', [50], { cost: 0.01 });++ await publish(`${id}@${version}`, 'model-c', [55], { cost: 0.01 });++++ const res = await t.get(`/benchmarks/${id}`, { as: alice });++ assert.strictEqual(res.status, 200, res.text.slice(0, 200));++ assert.ok(res.text.includes('Your prompts, 6 runs'), 'the owner leaderboard label');++ assert.ok(res.text.includes('The public board for this kind'), 'the public board is beside it');++ assert.ok(res.text.includes('Cost / solved task'), 'cost per solved task');++ assert.ok(/1st.{0,2}2nd/.test(res.text), 'overlapping intervals share a band');++ assert.ok(res.text.includes('model-a') && res.text.includes('model-b') && res.text.includes('model-c'), 'the own models');++ assert.ok(res.text.includes('public-model'), 'the public model');++ const start = res.text.indexOf('The public board for this kind');++ const end = res.text.indexOf('id="br-tasks-h"', start);++ const publicPart = res.text.slice(start, end);++ assert.ok(!publicPart.includes('model-a') && !publicPart.includes('model-b'), 'the private ranks are not mixed into the public board');++ });++++ await check('the run goes through the launcher and still asks before it starts', async () => {++ const before = codes.calls.filter((c) => c.method === 'POST').length;++ const tooFew = await post(`/benchmarks/${id}/run`, { models: 'model-a', harnesses: 'codex', repeat: '1' });++ assert.strictEqual(tooFew.status, 400, tooFew.text.slice(0, 200));++ assert.ok(tooFew.text.includes('3 to 6 models'), '3 to 6 models');++ assert.strictEqual(codes.calls.filter((c) => c.method === 'POST').length, before, 'nobody called Codes for one model');++ const ask = await post(`/benchmarks/${id}/run`, { models: 'model-a', more_models: 'model-b, model-c', harnesses: 'codex', repeat: '1' });++ assert.strictEqual(ask.status, 200, ask.text.slice(0, 200));++ assert.ok(ask.text.includes('sets no dollar limit'), 'the no-limit confirmation');++ assert.strictEqual(codes.calls.filter((c) => c.method === 'POST').length, before, 'Codes was not called yet');++ const go = await post(`/benchmarks/${id}/run`, { models: 'model-a', more_models: 'model-b, model-c', harnesses: 'codex', repeat: '1', confirm: '1' });++ assert.strictEqual(go.status, 303, go.text.slice(0, 200));++ const calls = codes.calls.filter((c) => c.method === 'POST').slice(before);++ assert.strictEqual(calls.length, 1);++ assert.deepStrictEqual(calls[0].body.models, ['model-a', 'model-b', 'model-c']);++ assert.ok(calls[0].headers['x-ov-subject'], 'as the person');++ assert.match(String(calls[0].headers.authorization), /^Bearer /, 'with Bench’s service token');++ const api = await t.get(`/api/v1/benchmarks/${id}/run`, { as: alice, json: { models: ['model-a', 'model-b', 'model-c'], harnesses: ['codex'], repeat: 1 } });++ assert.strictEqual(api.status, 409, api.text);++ assert.strictEqual(api.json().cost_estimate_usd, null, 'no estimate is invented');++ });++++ await check('an unlisted share link opens the page once, with a shared note and noindex', async () => {++ const res = await share(id, 'unlisted');++ assert.strictEqual(res.status, 200, res.text.slice(0, 300));++ token = tokenOf(res);++ assert.ok(token.length >= 40, 'a long random token');++ const page = await t.get(`/b/${token}`);++ assert.strictEqual(page.status, 200, page.text.slice(0, 300));++ assert.ok(page.text.includes('Shared by its owner'), 'the shared note');++ assert.ok(page.text.includes('noindex'), 'noindex');++ assert.ok(page.text.includes(MARK), 'the prompts are readable by the link');++ assert.ok(!page.text.includes('Share an unlisted link'), 'a visitor cannot re-share');++ assert.strictEqual((await t.get(`/benchmarks/${id}`)).status, 404, 'the id alone no longer opens it');++ assert.strictEqual((await t.get(`/api/v1/benchmarks/${id}`)).status, 404, 'nor its API definition');++ assert.strictEqual((await t.get('/b/not-a-real-token')).status, 404);++ assert.strictEqual((await t.get(`/benchmarks/${id}`, { as: bob })).status, 404);++ });++++ await check('making it private again stops the link', async () => {++ const res = await share(id, 'private');++ assert.strictEqual(res.status, 200, res.text.slice(0, 200));++ assert.ok(res.text.includes('Private.'), 'private again');++ assert.strictEqual((await t.get(`/b/${token}`)).status, 404, 'the old link is dead');++ assert.strictEqual((await t.get(`/benchmarks/${id}`, { as: alice })).status, 200);++ });++++ await check('sharing unlisted again makes a new link and invalidates the old one, then public lists it', async () => {++ const res = await share(id, 'unlisted');++ const second = tokenOf(res);++ assert.notStrictEqual(second, token, 'a new share is a new token');++ assert.strictEqual((await t.get(`/b/${token}`)).status, 404, 'the first link is dead');++ assert.strictEqual((await t.get(`/b/${second}`)).status, 200);++ const pub = await share(id, 'public');++ assert.strictEqual(pub.status, 200, pub.text.slice(0, 200));++ assert.strictEqual((await t.get(`/b/${second}`)).status, 404, 'the token is cleared when it becomes public');++ assert.strictEqual((await t.get(`/benchmarks/${id}`)).status, 200, 'public by id');++ const list = await t.get('/api/v1/benchmarks');++ assert.ok(list.json().benchmarks.some((b) => b.id === id), 'in the public list');++ const sitemap = await t.get('/sitemap.xml');++ assert.ok(sitemap.text.includes(id), 'in the sitemap');++ const dataset = await t.get('/api/v1/datasets/benchmarks.jsonl');++ assert.ok(dataset.text.includes(MARK), 'its prompts are in the open data');++ assert.ok(dataset.text.includes('benchmark_run'), 'with its runs');++ assert.ok(!dataset.text.includes(alice.subject), 'no person id in the open data');++ assert.ok(!dataset.text.includes('created_by'), 'no created_by field in the open data');++ });++++ await check('making it private again takes it out of every public surface', async () => {++ await share(id, 'private');++ const list = await t.get('/api/v1/benchmarks');++ assert.ok(!list.json().benchmarks.some((b) => b.id === id));++ const dataset = await t.get('/api/v1/datasets/benchmarks.jsonl');++ assert.ok(!dataset.text.includes(MARK), 'the prompts leave the open data');++ const sitemap = await t.get('/sitemap.xml');++ assert.ok(!sitemap.text.includes(id));++ });++++ await check('every form post is same-origin', async () => {++ const make = await post('/benchmarks/new', { kind: 'text.answer', format: 'lines', prompts: prompts.join('\n') }, { origin: 'https://evil.example' });++ assert.strictEqual(make.status, 403, make.text.slice(0, 200));++ const shared = await post(`/benchmarks/${id}/share`, { visibility: 'public' }, { origin: 'https://evil.example' });++ assert.strictEqual(shared.status, 403, shared.text.slice(0, 200));++ });++++ await check('stop', async () => { await t.close(); codes.close(); });++++ done();++})();++
@@ -12,11 +12,24 @@ capabilities. A session write must come from bench.openvibe.services itself.
@@ -12,11 +12,24 @@ capabilities. A session write must come from bench.openvibe.services itself.
| `PUT /api/v1/benchmarks/:id` | the same | Save a new draft. A body with `version` edits that version, and is refused once a run names it. |
| `PUT /api/v1/benchmarks/:id` | the same | Save a new draft. A body with `version` edits that version, and is refused once a run names it. |
| `POST /api/v1/benchmarks/:id/versions` | the same | Freeze the working copy into the next immutable version. |
| `POST /api/v1/benchmarks/:id/versions` | the same | Freeze the working copy into the next immutable version. |
| `POST /api/v1/benchmarks/:id/fork` | the same | Fork a benchmark; the fork records its parent. |
| `POST /api/v1/benchmarks/:id/fork` | the same | Fork a benchmark; the fork records its parent. |
-| `GET /api/v1/benchmarks?kind=&tag=&q=` | anyone | The registry. |
-| `GET /api/v1/benchmarks/:id[@version]` | anyone | One benchmark and its definition. Holdout tasks are withheld unless the caller holds `bench.run.publish` or owns the benchmark. |
+| `POST /api/v1/benchmarks/:id/share` | its owner | `{ visibility: "private" | "unlisted" | "public" }`. `unlisted` returns a new share token once; the old link stops. |
+| `GET /api/v1/benchmarks?kind=&tag=&q=&mine=1` | anyone | The registry; `mine=1` adds the caller's own unlisted and private benchmarks. |
+| `GET /api/v1/benchmarks/:id[@version]` | anyone who may read it | One benchmark and its definition. A private benchmark is a 404 except to its owner and the executor; a shared unlisted one needs its link. Holdout tasks are withheld unless the caller holds `bench.run.publish` or owns the benchmark. |
-The documents are the published contracts, from `openvibe-contracts` v0.244.0: `bench.benchmark@1` for a
-benchmark version, `bench.run@1` for a run. Every document is checked against them before it is stored.
+The documents are the published contracts, from `openvibe-contracts`: `bench.benchmark@1` for a benchmark
+version, `bench.run@1` for a run. Every contract document is checked against them before it is stored. `private`
+is Bench's own stricter form of unlisted: the contract document is frozen as `unlisted`, and the benchmark row's
+visibility is what every access check reads.
+
+### Your own prompts
+
+`POST /benchmarks/new` on the site (or the API above) makes a benchmark from 5 to 50 pasted prompts: one per line,
+or CSV with an optional expected answer. The default kind is `text.answer`; any existing kind may be chosen. It is
+private by default, exports and deletes with the account, and its runs go through the existing openvibe.codes launcher
+(the same estimate and confirmation).
+
+The open data: `GET /api/v1/datasets/benchmarks.jsonl` carries every public benchmark, its visible tasks and every
+run under it, with no person id. Nothing private or unlisted is ever read there.
## The results
## The results
@@ -13,7 +13,7 @@ A benchmark (the contract is `bench.benchmark@1`) holds:
@@ -13,7 +13,7 @@ A benchmark (the contract is `bench.benchmark@1`) holds:
| `title`, `description` | What the set is and who it is for. |
| `title`, `description` | What the set is and who it is for. |
| `kind` | One of Bench's kinds (`games.play`, `design.ui`, `codes.patch`, …). The kind sets the renderer. |
| `kind` | One of Bench's kinds (`games.play`, `design.ui`, `codes.patch`, …). The kind sets the renderer. |
| `tags` | Up to 32 words people search by. |
| `tags` | Up to 32 words people search by. |
-| `visibility` | `public` (in the registry) or `unlisted` (by id only). |
+| `visibility` | `public` (in the registry), `unlisted` (reachable by its id; a share link when one is made), or `private` (its owner only). A pasted set of your own prompts starts private. |
| `version` | The version this document is. Bench numbers them. |
| `version` | The version this document is. Bench numbers them. |
| `parent` | The `id@version` this one was forked from, or `null`. |
| `parent` | The `id@version` this one was forked from, or `null`. |
| `tasks[]` | `{ id, prompt, files?, expected?, holdout }`. The prompt is the task; `holdout: true` hides it from everyone but the executor. |
| `tasks[]` | `{ id, prompt, files?, expected?, holdout }`. The prompt is the task; `holdout: true` hides it from everyone but the executor. |
@@ -56,6 +56,16 @@ published set, fork it — the fork records the parent, and the old version stay
@@ -56,6 +56,16 @@ published set, fork it — the fork records the parent, and the old version stay
}
}
```
```
+## Your own prompts
+
+A person can paste 5 to 50 of their own prompts at `/benchmarks/new`: one per line, or CSV with an optional
+expected answer in the second column. They become the tasks of a private benchmark (`text.answer`, or any existing
+kind), frozen as version 1. It is private by default: the owner alone sees the prompts, and another person gets a
+404. The owner may share an unlisted link (a 256-bit token, stored as a hash and checked in constant time), make it
+public (then it is in the lists, the sitemap and `/api/v1/datasets/benchmarks.jsonl`), or make it private again (the
+link stops working). Runs still go through the openvibe.codes launcher: the same estimate, the same "ask before it
+starts".
+
## The contracts
## The contracts
Bench speaks the published shapes, from `openvibe-contracts` **v0.244.0**:
Bench speaks the published shapes, from `openvibe-contracts` **v0.244.0**:
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+-- phase: expand
+-- B11: a person's own prompts as a private benchmark (spec "Your own prompts", 2026-10-30).
+--
+-- A benchmark is private until its owner shares it. `visibility` gains `private`; the check is widened, never
+-- narrowed, so an older writer that only knows public and unlisted keeps working.
+--
+-- An unlisted share carries a long random token in its link. Only the token's SHA-256 is stored: a database read
+-- never yields a working link. The token is checked in constant time (server/bench/registry.js) and cleared when
+-- the benchmark becomes private or public, so an old link stops working.
+ALTER TABLE bench_benchmarks DROP CONSTRAINT bench_benchmarks_visibility;
+ALTER TABLE bench_benchmarks ADD CONSTRAINT bench_benchmarks_visibility
+ CHECK (visibility IN ('public', 'unlisted', 'private'));
+
+ALTER TABLE bench_benchmarks ADD COLUMN share_token_hash text;
+ALTER TABLE bench_benchmarks ADD CONSTRAINT bench_benchmarks_share_token_hash
+ CHECK (share_token_hash IS NULL OR share_token_hash ~ '^[0-9a-f]{64}$');
+-- At most one benchmark per token, so a hash collision cannot make a link ambiguous.
+CREATE UNIQUE INDEX bench_benchmarks_share_token ON bench_benchmarks (share_token_hash)
+ WHERE share_token_hash IS NOT NULL;
@@ -420,3 +420,76 @@ html, body { margin: 0; background: var(--bg-primary, #0a0f1c); color: var(--tex
@@ -420,3 +420,76 @@ html, body { margin: 0; background: var(--bg-primary, #0a0f1c); color: var(--tex
.br-usethis-row { display: grid; gap: 6px; min-width: 0; }
.br-usethis-row { display: grid; gap: 6px; min-width: 0; }
.br-usethis-row textarea { width: 100%; min-height: 44px; padding: 10px 12px; border-radius: 12px; border: 1px solid var(--border); background: var(--bg-primary); color: var(--text-secondary); font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: .8rem; line-height: 1.45; resize: vertical; }
.br-usethis-row textarea { width: 100%; min-height: 44px; padding: 10px 12px; border-radius: 12px; border: 1px solid var(--border); background: var(--bg-primary); color: var(--text-secondary); font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: .8rem; line-height: 1.45; resize: vertical; }
.br-follow { margin: 0; }
.br-follow { margin: 0; }
+
+/* ---- Your own prompts (B11): the paste form, the share box and the two leaderboards ---- */
+.br-new-form textarea, .br-new-form .br-field textarea {
+ width: 100%;
+ min-height: 44px;
+ padding: 10px 12px;
+ border-radius: 12px;
+ border: 1px solid var(--border);
+ background: var(--bg-input, var(--bg-primary));
+ color: var(--text-primary);
+ font: inherit;
+ line-height: 1.5;
+ resize: vertical;
+}
+.br-new-form textarea:focus-visible { outline: 2px solid var(--room-accent); outline-offset: 1px; }
+.br-new-form .br-prompts { min-height: 15rem; font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: .9rem; }
+.br-new-form .br-paste { min-width: 0; }
+.br-new-form .br-field { min-width: 0; }
+.br-vis { text-transform: lowercase; }
+.br-mine { margin-top: clamp(24px, 4vw, 36px); }
+.br-mine .bm-head-row, .br-share .bm-head-row { flex-wrap: wrap; }
+.br-mine-hint { margin: 14px 0 0; }
+.br-private { border-color: color-mix(in srgb, var(--room-accent) 40%, transparent); }
+.br-share { padding: 16px; border: 1px solid var(--border); border-radius: 18px; background: var(--bg-card); }
+.br-share-form { display: flex; flex-wrap: wrap; gap: 10px; margin: 14px 0 8px; }
+.br-share-now { display: grid; gap: 10px; }
+.br-share-now p { margin: 0; }
+.br-share-link { width: 100%; min-width: 0; min-height: 44px; padding: 0 12px; border-radius: 12px; border: 1px solid var(--border); background: var(--bg-input, var(--bg-primary)); color: var(--text-primary); font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: .82rem; }
+.br-duo { display: grid; gap: 18px; grid-template-columns: 1fr; align-items: start; }
+.br-duo-pane { min-width: 0; padding: 16px; border: 1px solid var(--border); border-radius: 18px; background: var(--bg-card); }
+.br-duo-pane .br-table-wrap { background: transparent; }
+.br-duo-pane .br-chart { padding: 10px; }
+.br-duo-pane .bm-empty { background: transparent; }
+.br-prompt-table .rt-col-model { width: auto; }
+.br-spread { display: block; margin-top: 4px; white-space: nowrap; }
+.br-prompt-table .rt-col-raw { width: 9em; }
+.br-prompt-table .rt-col-band { width: 6.5em; }
+@media (min-width: 900px) {
+ .br-duo { grid-template-columns: minmax(0, 1fr) minmax(0, 1fr); }
+}
+@media (max-width: 560px) {
+ /* The shared ratings fold: score and cost/solved on the second line, as a band + model + score + cost + runs. */
+ .br-prompt-table.rt-table tr { grid-template-columns: minmax(0, 1fr) minmax(0, 1fr) 3.2em; }
+ .br-prompt-table .rt-col-controlled { grid-area: 2 / 1 / 3 / 2; }
+ .br-prompt-table .rt-col-raw { grid-area: 2 / 2 / 3 / 3; width: auto; }
+ .br-prompt-table td.rt-col-votes { grid-area: 2 / 3 / 3 / 4; text-align: right; }
+ .br-prompt-table .br-spread { white-space: normal; }
+ .br-share { padding: 14px; }
+}
+.bm-page-h > .bm-head-row { flex-wrap: wrap; align-items: flex-end; }
+.br-new-form .br-field select, .br-new-form .br-field input[type="text"] { width: 100%; min-width: 0; }
+/* The rankings table folds into two lines on a phone: band + model + score, then the numbers with their names. */
+@media (max-width: 560px) {
+ .br-rank-table, .br-rank-table tbody { display: block; width: 100%; }
+ .br-rank-table thead { display: none; }
+ .br-rank-table tr { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 12px; padding: 11px 14px; border-bottom: 1px solid var(--border); }
+ .br-rank-table tbody tr:last-child { border-bottom: 0; }
+ .br-rank-table td { display: block; padding: 0; border: 0; min-width: 0; width: auto; white-space: normal; }
+ .br-rank-table td.br-rank { display: none; }
+ .br-rank-table td.br-c-band { flex: none; order: 0; }
+ .br-rank-table td.br-c-model { flex: 1 1 auto; order: 0; overflow-wrap: anywhere; }
+ .br-rank-table td.br-c-score { flex: none; order: 0; text-align: right; }
+ .br-rank-table tr::after { content: ""; flex-basis: 100%; height: 0; order: 1; }
+ .br-rank-table td.br-c-metric, .br-rank-table td.br-c-cost, .br-rank-table td.br-c-time, .br-rank-table td.br-c-runs { flex: none; order: 2; font-size: .82rem; color: var(--text-secondary); text-align: left; }
+ .br-rank-table td[data-label]::before { content: attr(data-label) " "; color: var(--text-muted); }
+}
+.br-prompt-table .rt-model { white-space: nowrap; }
+@media (max-width: 560px) { .br-prompt-table .rt-model { white-space: normal; overflow-wrap: anywhere; } }
+.br-duo-pane .br-prompt-table { font-size: .86rem; }
+.br-duo-pane .br-prompt-table th, .br-duo-pane .br-prompt-table td { padding: 8px 8px; }
+.br-prompt-table .rt-col-band { width: 5em; }
+.br-prompt-table .rt-col-raw { width: 7em; }
+.br-prompt-table .rt-col-votes { width: 3.2em; }
@@ -7,6 +7,7 @@
@@ -7,6 +7,7 @@
* something the API refuses.
* something the API refuses.
*/
*/
const { createServiceTokenClient } = require('openvibe-sdk/auth');
const { createServiceTokenClient } = require('openvibe-sdk/auth');
+const { canViewBenchmark } = require('./registry');
const CONFIRM_ABOVE_USD = 1;
const CONFIRM_ABOVE_USD = 1;
const HARNESS_TTL_MS = 10 * 60 * 1000;
const HARNESS_TTL_MS = 10 * 60 * 1000;
@@ -43,6 +44,15 @@ function createLauncher({ config, registry, fetchImpl = globalThis.fetch, log =
@@ -43,6 +44,15 @@ function createLauncher({ config, registry, fetchImpl = globalThis.fetch, log =
}
}
const definition = await registry.getDefinition(String(id), { reveal: false });
const definition = await registry.getDefinition(String(id), { reveal: false });
if (!definition) return fail(404, 'bench.benchmark_not_found', 'No such benchmark.');
if (!definition) return fail(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ // A private benchmark is its owner's: another signed-in person cannot start it, even by knowing its id.
+ if (!canViewBenchmark(definition.benchmark, { subject: principal.subject })) {
+ return fail(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ }
+ // A set made from a person's own prompts is compared on 3 to 6 models, as the paste form says.
+ if (Array.isArray(definition.benchmark.tags) && definition.benchmark.tags.includes('your-prompts')
+ && (models.length < 3 || models.length > 6)) {
+ return fail(400, 'bench.prompt_models', 'Pick 3 to 6 models for a benchmark made from your own prompts.');
+ }
const cost = estimate(definition.definition, { models, harnesses, repeat });
const cost = estimate(definition.definition, { models, harnesses, repeat });
if ((cost == null || cost > CONFIRM_ABOVE_USD) && confirm !== true) {
if ((cost == null || cost > CONFIRM_ABOVE_USD) && confirm !== true) {
return { status: 409, body: { code: 'bench.confirm_cost', cost_estimate_usd: cost, confirm_required: true, ...(cost == null ? { detail: 'This benchmark sets no dollar limit per task, so its cost depends on the models.' } : {}) } };
return { status: 409, body: { code: 'bench.confirm_cost', cost_estimate_usd: cost, confirm_required: true, ...(cost == null ? { detail: 'This benchmark sets no dollar limit per task, so its cost depends on the models.' } : {}) } };
@@ -15,7 +15,8 @@
@@ -15,7 +15,8 @@
*/
*/
const crypto = require('crypto');
const crypto = require('crypto');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
- const { contractRun, checkContract } = require('./validate');
+const { contractRun, checkContract, VISIBILITIES } = require('./validate');
+const { rankBands } = require('../rating/style');
const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
@@ -52,6 +53,38 @@ const json = (v, fallback) => {
@@ -52,6 +53,38 @@ const json = (v, fallback) => {
return v;
return v;
};
};
+/** A share token: 256 random bits in base64url (43 characters), never guessable. */
+function newShareToken() {
+ return crypto.randomBytes(32).toString('base64url');
+}
+
+/** Only the SHA-256 of a share token is stored, so a database read never yields a working link. */
+function shareTokenHash(token) {
+ return crypto.createHash('sha256').update(String(token || ''), 'utf8').digest('hex');
+}
+
+/** Constant-time check of a token against the stored hash: lengths first, then timingSafeEqual. */
+function shareTokenMatches(token, storedHash) {
+ if (!storedHash) return false;
+ const got = Buffer.from(shareTokenHash(token), 'hex');
+ const want = Buffer.from(String(storedHash), 'hex');
+ return got.length === want.length && crypto.timingSafeEqual(got, want);
+}
+
+/**
+ * Who may read a benchmark. Public is for everyone; private is its owner (and the executor, which needs the tasks
+ * to run them); unlisted is reachable by id as it always was, but a shared unlisted benchmark is token-only: its
+ * id alone no longer opens it.
+ */
+function canViewBenchmark(benchmark, { subject = null, executor = false } = {}) {
+ if (!benchmark) return false;
+ const owner = subject != null && String(benchmark.owner) === String(subject);
+ if (benchmark.visibility === 'public') return true;
+ if (benchmark.visibility === 'private') return owner || executor;
+ if (benchmark.visibility === 'unlisted' && benchmark.shared) return owner || executor;
+ return true;
+}
+
function wireBenchmark(row) {
function wireBenchmark(row) {
if (!row) return null;
if (!row) return null;
return {
return {
@@ -62,6 +95,8 @@ function wireBenchmark(row) {
@@ -62,6 +95,8 @@ function wireBenchmark(row) {
kind: row.kind,
kind: row.kind,
tags: Array.isArray(json(row.tags, [])) ? json(row.tags, []) : [],
tags: Array.isArray(json(row.tags, [])) ? json(row.tags, []) : [],
visibility: row.visibility,
visibility: row.visibility,
+ // Whether an unlisted share link exists. The token itself is never in a wire object or a page source.
+ shared: Boolean(row.share_token_hash),
parent: row.parent || null,
parent: row.parent || null,
latest_version: num(row.latest_version),
latest_version: num(row.latest_version),
draft: json(row.draft, null),
draft: json(row.draft, null),
@@ -172,7 +207,7 @@ function createRegistry(s, { log = console } = {}) {
@@ -172,7 +207,7 @@ function createRegistry(s, { log = console } = {}) {
const kindId = String(kind == null ? '' : kind).trim();
const kindId = String(kind == null ? '' : kind).trim();
if (!kindId) return refusal(422, 'bench.kind_required', 'A benchmark needs a kind.');
if (!kindId) return refusal(422, 'bench.kind_required', 'A benchmark needs a kind.');
const vis = String(visibility || 'unlisted');
const vis = String(visibility || 'unlisted');
- if (!['public', 'unlisted'].includes(vis)) return refusal(422, 'bench.invalid_visibility', 'visibility is public or unlisted.');
+ if (!VISIBILITIES.includes(vis)) return refusal(422, 'bench.invalid_visibility', `visibility is ${VISIBILITIES.join(', ')}.`);
const list = (Array.isArray(tags) ? tags : []).map((t) => String(t).trim()).filter(Boolean).slice(0, 32);
const list = (Array.isArray(tags) ? tags : []).map((t) => String(t).trim()).filter(Boolean).slice(0, 32);
const id = s.newId('bmk');
const id = s.newId('bmk');
const row = await db.maybe(
const row = await db.maybe(
@@ -254,6 +289,9 @@ function createRegistry(s, { log = console } = {}) {
@@ -254,6 +289,9 @@ function createRegistry(s, { log = console } = {}) {
if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
const source = definition || benchmark.draft || {};
const source = definition || benchmark.draft || {};
const version = benchmark.latest_version + 1;
const version = benchmark.latest_version + 1;
+ // The published contract knows public and unlisted only. A private benchmark is frozen as unlisted (the
+ // stricter form); the row's own visibility is what every access check reads, so its prompts stay private.
+ const frozenVisibility = benchmark.visibility === 'public' ? 'public' : 'unlisted';
const doc = {
const doc = {
id: benchmark.id,
id: benchmark.id,
owner: benchmark.owner,
owner: benchmark.owner,
@@ -261,7 +299,7 @@ function createRegistry(s, { log = console } = {}) {
@@ -261,7 +299,7 @@ function createRegistry(s, { log = console } = {}) {
description: source.description == null ? benchmark.description : source.description,
description: source.description == null ? benchmark.description : source.description,
kind: source.kind || benchmark.kind,
kind: source.kind || benchmark.kind,
tags: Array.isArray(source.tags) ? source.tags : benchmark.tags,
tags: Array.isArray(source.tags) ? source.tags : benchmark.tags,
- visibility: source.visibility || benchmark.visibility,
+ visibility: frozenVisibility,
version,
version,
parent: benchmark.parent,
parent: benchmark.parent,
tasks: Array.isArray(source.tasks) ? source.tasks : [],
tasks: Array.isArray(source.tasks) ? source.tasks : [],
@@ -319,6 +357,85 @@ function createRegistry(s, { log = console } = {}) {
@@ -319,6 +357,85 @@ function createRegistry(s, { log = console } = {}) {
return { ok: true, benchmark: await getBenchmark(made.benchmark.id), version: frozen.version, parent };
return { ok: true, benchmark: await getBenchmark(made.benchmark.id), version: frozen.version, parent };
}
}
+ /**
+ * Share or unshare a benchmark. The owner chooses:
+ * private only they see it; no link works
+ * unlisted a new random link works, and nothing lists it
+ * public it is in the lists, the sitemap and the open data
+ * The raw token is returned once, on the call that makes the link; only its hash is stored. Every change replaces
+ * the old token, so a link from a previous share stops working.
+ */
+ async function setVisibility(id, { visibility, subject }) {
+ const benchmark = await getBenchmark(id);
+ if (!benchmark) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ if (String(benchmark.owner) !== String(subject)) return refusal(404, 'bench.benchmark_not_found', 'No such benchmark.');
+ const vis = String(visibility || '').trim();
+ if (!VISIBILITIES.includes(vis)) return refusal(422, 'bench.invalid_visibility', `visibility is ${VISIBILITIES.join(', ')}.`);
+ const token = vis === 'unlisted' ? newShareToken() : null;
+ const row = await db.maybe(
+ `UPDATE bench_benchmarks SET visibility = $1, share_token_hash = $2, updated_at = $3::timestamptz
+ WHERE id = $4 RETURNING *`,
+ [vis, token ? shareTokenHash(token) : null, s.iso(), benchmark.id]);
+ return {
+ ok: true,
+ benchmark: wireBenchmark(row),
+ token,
+ share_path: token ? `/b/${token}` : null,
+ };
+ }
+
+ /** The benchmark an unlisted link names, checked in constant time. A private or public row has no token. */
+ async function findShared(token) {
+ const text = String(token || '');
+ if (!/^[A-Za-z0-9_-]{20,120}$/.test(text)) return null;
+ const hash = shareTokenHash(text);
+ const row = await db.maybe(
+ "SELECT * FROM bench_benchmarks WHERE share_token_hash = $1 AND visibility = 'unlisted'",
+ [hash]);
+ if (!row || !shareTokenMatches(text, row.share_token_hash)) return null;
+ return wireBenchmark(row);
+ }
+
+ /**
+ * The open data for the registry: every public benchmark, its visible tasks, and every run published under it.
+ * Nothing private or unlisted is ever read here; a person id (created_by, the executor's id) is left out, and a
+ * holdout task keeps its prompt to the executor.
+ */
+ async function publicDatasetLines() {
+ const benchmarks = await listBenchmarks({ limit: 200 });
+ const lines = [];
+ for (const benchmark of benchmarks) {
+ if (benchmark.visibility !== 'public') continue;
+ const version = await latestVersion(benchmark.id);
+ if (!version) continue;
+ const definition = version.definition || {};
+ lines.push(JSON.stringify({
+ type: 'benchmark', id: benchmark.id, title: benchmark.title, description: benchmark.description,
+ kind: benchmark.kind, tags: benchmark.tags, version: version.version, parent: benchmark.parent,
+ created_at: benchmark.created_at,
+ }));
+ for (const task of (definition.tasks || [])) {
+ if (!task || task.holdout) continue;
+ const item = { type: 'benchmark_task', benchmark: `${benchmark.id}@${version.version}`, id: task.id, prompt: task.prompt };
+ if (task.expected != null) item.expected = task.expected;
+ lines.push(JSON.stringify(item));
+ }
+ const runs = await db.many(
+ 'SELECT * FROM bench_runs WHERE benchmark_id = $1 AND benchmark_version = $2 ORDER BY created_at, id',
+ [benchmark.id, version.version]);
+ for (const row of runs) {
+ const run = wireRun(row);
+ lines.push(JSON.stringify({
+ type: 'benchmark_run', benchmark: run.benchmark, model: run.model, route: run.route,
+ provider: run.provider, harness: run.harness, settings: run.settings, hardware: run.hardware,
+ score: run.score, cost_usd: run.cost_usd, ms: run.ms, tokens: run.tokens,
+ provenance: run.provenance, session: run.session, created_at: run.created_at,
+ }));
+ }
+ }
+ return lines.length ? `${lines.join('\n')}\n` : '';
+ }
+
/** The registry list: by kind, by tag, or by a search over the title and the description. */
/** The registry list: by kind, by tag, or by a search over the title and the description. */
/**
/**
* Benchmarks for a listing: the public ones, plus every one the `owner` owns when one is named. Unlisted and
* Benchmarks for a listing: the public ones, plus every one the `owner` owns when one is named. Unlisted and
@@ -485,6 +602,7 @@ function createRegistry(s, { log = console } = {}) {
@@ -485,6 +602,7 @@ function createRegistry(s, { log = console } = {}) {
const entries = [];
const entries = [];
for (const [model, list] of byModel) {
for (const [model, list] of byModel) {
const scores = list.map((r) => r.score);
const scores = list.map((r) => r.score);
+ const cleanScores = scores.filter((v) => v != null);
const costs = list.map((r) => r.cost_usd);
const costs = list.map((r) => r.cost_usd);
const times = list.map((r) => r.ms);
const times = list.map((r) => r.ms);
const medScore = median(scores);
const medScore = median(scores);
@@ -508,6 +626,12 @@ function createRegistry(s, { log = console } = {}) {
@@ -508,6 +626,12 @@ function createRegistry(s, { log = console } = {}) {
speed: medScore != null && medMs ? Math.round((medScore / (medMs / 60000)) * 1000) / 1000 : null,
speed: medScore != null && medMs ? Math.round((medScore / (medMs / 60000)) * 1000) / 1000 : null,
consistency: spreads.length ? Math.round(median(spreads) * 1000) / 1000 : null,
consistency: spreads.length ? Math.round(median(spreads) * 1000) / 1000 : null,
sessions: sessions.size,
sessions: sessions.size,
+ // What the runs actually did, and the value metric of the rules: cost per solved task = the cost
+ // of an attempt divided by the success rate (the median score / 100).
+ min_score: cleanScores.length ? Math.min(...cleanScores) : null,
+ max_score: cleanScores.length ? Math.max(...cleanScores) : null,
+ spread: cleanScores.length ? Math.round((Math.max(...cleanScores) - Math.min(...cleanScores)) * 1000) / 1000 : null,
+ cost_per_solved: medScore > 0 && medCost > 0 ? Math.round((medCost / (medScore / 100)) * 1e6) / 1e6 : null,
};
};
entries.push(entry);
entries.push(entry);
}
}
@@ -522,6 +646,24 @@ function createRegistry(s, { log = console } = {}) {
@@ -522,6 +646,24 @@ function createRegistry(s, { log = console } = {}) {
if (av === bv) return a.model.localeCompare(b.model);
if (av === bv) return a.model.localeCompare(b.model);
return ascending ? av - bv : bv - av;
return ascending ? av - bv : bv - av;
});
});
+ // Ranks are bands, not places: models whose score intervals overlap share a band. The interval is the
+ // run-to-run spread here (a private set has too few runs for a bootstrap to say more).
+ if (metric === 'score') {
+ const bands = rankBands(entries.filter((e) => e.score != null).map((e) => ({
+ model: e.model,
+ rating: e.score,
+ ci_lower: e.min_score == null ? e.score : e.min_score,
+ ci_upper: e.max_score == null ? e.score : e.max_score,
+ })));
+ const bandOf = new Map();
+ for (const band of bands) for (const model of band.models) bandOf.set(model, band);
+ for (const e of entries) {
+ const band = bandOf.get(e.model);
+ e.band = band ? band.label : null;
+ e.band_from = band ? band.from : null;
+ e.band_to = band ? band.to : null;
+ }
+ }
return { benchmark: ref.id, metric, entries };
return { benchmark: ref.id, metric, entries };
}
}
@@ -621,8 +763,12 @@ function createRegistry(s, { log = console } = {}) {
@@ -621,8 +763,12 @@ function createRegistry(s, { log = console } = {}) {
return {
return {
createBenchmark, getBenchmark, getVersion, latestVersion, getDefinition, saveDraft, freezeVersion,
createBenchmark, getBenchmark, getVersion, latestVersion, getDefinition, saveDraft, freezeVersion,
- forkBenchmark, listBenchmarks, publishRun, getRun, compareRuns, rankings, percentile, listRuns, modelSummary, runCounts,
+ forkBenchmark, setVisibility, findShared, publicDatasetLines,
+ listBenchmarks, publishRun, getRun, compareRuns, rankings, percentile, listRuns, modelSummary, runCounts,
};
};
}
}
-module.exports = { createRegistry, RegistryError, hashDefinition, canonical, median, parseRef };
+module.exports = {
+ createRegistry, RegistryError, hashDefinition, canonical, median, parseRef,
+ canViewBenchmark, shareTokenHash, shareTokenMatches, newShareToken,
+};
@@ -3,11 +3,13 @@
@@ -3,11 +3,13 @@
/**
/**
* What a benchmark and a run must look like before Bench stores them.
* What a benchmark and a run must look like before Bench stores them.
*
*
- * The shapes are the released contracts (bench.benchmark@1 and bench.run@1, openvibe-contracts v0.244.0). Bench keeps
- * its own copy of the rules so the service can refuse a bad document before it reaches the database, and so a
- * caller gets a field-level error instead of a stack trace.
+ * The shapes are the released contracts (bench.benchmark@1 and bench.run@1, openvibe-contracts). Bench keeps its
+ * own copy of the rules so the service can refuse a bad document before it reaches the database, and so a caller
+ * gets a field-level error instead of a stack trace.
*
*
- * The rules mirror the contract's: the same required fields, the same enums, the same ranges.
+ * The rules mirror the contract's: the same required fields, the same enums, the same ranges. The one addition is
+ * private visibility (B11), Bench's stricter form of unlisted: the benchmark row is private while the frozen
+ * contract document stays unlisted, so every stored contract document remains a valid bench.benchmark@1.
*/
*/
const { assertValid } = require('openvibe-contracts');
const { assertValid } = require('openvibe-contracts');
@@ -22,7 +24,9 @@ const BENCHMARK_ID = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;
@@ -22,7 +24,9 @@ const BENCHMARK_ID = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;
const RUN_ID = /^brn_[0-9A-HJKMNP-TV-Z]{26}$/;
const RUN_ID = /^brn_[0-9A-HJKMNP-TV-Z]{26}$/;
const BENCHMARK_REF = /^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$/;
const BENCHMARK_REF = /^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$/;
-const VISIBILITIES = ['public', 'unlisted'];
+// private is Bench's own stricter form of unlisted (B11): the row's access is private; the frozen contract
+// document says unlisted, because the published shape knows only public and unlisted.
+const VISIBILITIES = ['public', 'unlisted', 'private'];
const SCORING_METHODS = ['mean', 'weighted', 'pass_rate', 'sum'];
const SCORING_METHODS = ['mean', 'weighted', 'pass_rate', 'sum'];
const EXECUTOR_KINDS = ['codes', 'run', 'node', 'submitted'];
const EXECUTOR_KINDS = ['codes', 'run', 'node', 'submitted'];
const PROVENANCES = ['verified', 'submitted'];
const PROVENANCES = ['verified', 'submitted'];
new file mode 100644
new file mode 100644
@@ -0,0 +1,166 @@
@@ -0,0 +1,166 @@
+'use strict';
+
+/**
+ * B11: a person's own prompts, pasted as a private benchmark.
+ *
+ * The paste is either one prompt per line, or CSV with an optional expected answer in the second column. The parser
+ * is deliberately small and local: nothing here calls a model, so a prompt never leaves the request that made it.
+ * Bounds: 5–50 prompts (the spec's "paste 5–50 of your real prompts").
+ *
+ * A CSV may carry a header (`prompt` or `prompt,expected`); without a header the form's format choice says which
+ * form it is. Quotes follow RFC 4180 closely enough for a spreadsheet export: a field may be quoted, a `""` inside
+ * quotes is one quote, and a quoted field may hold commas and newlines.
+ */
+const { kindById } = require('../kinds');
+
+const MIN_PROMPTS = 5;
+const MAX_PROMPTS = 50;
+const MAX_PROMPT = 8000;
+const MAX_EXPECTED = 4000;
+const MAX_PASTE = 512 * 1024;
+const DEFAULT_TITLE = 'Your prompts';
+const TAG = 'your-prompts';
+
+const clean = (v) => String(v == null ? '' : v).replace(/^\uFEFF/, '').replace(/\r\n?/g, '\n').trim();
+
+/** Parse a CSV body into rows of strings. */
+function csvRows(text) {
+ const rows = [];
+ let row = [];
+ let field = '';
+ let quoted = false;
+ for (let i = 0; i < text.length; i++) {
+ const c = text[i];
+ if (quoted) {
+ if (c === '"' && text[i + 1] === '"') { field += '"'; i++; }
+ else if (c === '"') quoted = false;
+ else field += c;
+ } else if (c === '"' && field === '') quoted = true;
+ else if (c === ',') { row.push(field); field = ''; }
+ else if (c === '\n') { row.push(field); rows.push(row); row = []; field = ''; }
+ else field += c;
+ }
+ if (field !== '' || row.length) { row.push(field); rows.push(row); }
+ return rows;
+}
+
+const HEADER_PROMPT = new Set(['prompt', 'prompts', 'task', 'input']);
+const HEADER_EXPECTED = new Set(['expected', 'expected answer', 'answer', 'expected_answer', 'output']);
+
+/** The first row is a header when it names a prompt column (and at most an expected-answer column). */
+function headerKind(row) {
+ if (!row || !row.length) return null;
+ const cells = row.map((c) => String(c || '').trim().toLowerCase());
+ if (!HEADER_PROMPT.has(cells[0])) return null;
+ if (cells.length === 1) return { expected: false };
+ if (cells.length === 2 && HEADER_EXPECTED.has(cells[1])) return { expected: true };
+ return null;
+}
+
+function promptRows(rows) {
+ const out = [];
+ for (const row of rows) {
+ if (!row || !row.length) continue;
+ const prompt = String(row[0] == null ? '' : row[0]).trim();
+ if (!prompt && row.every((c) => !String(c || '').trim())) continue;
+ if (!prompt) return { ok: false, code: 'bench.prompt_empty', detail: `An empty prompt is not a task (row ${out.length + 1}).` };
+ const answer = row.length > 1 ? String(row[1] == null ? '' : row[1]).trim() : '';
+ if (row.length > 2 && row.slice(2).some((c) => String(c || '').trim())) {
+ return { ok: false, code: 'bench.csv_columns', detail: `Row ${out.length + 1} has more than two columns; use prompt,expected.` };
+ }
+ out.push({ prompt, expected: answer || null });
+ }
+ return { ok: true, prompts: out };
+}
+
+function bounds(prompts, format) {
+ if (prompts.length < MIN_PROMPTS) {
+ return { ok: false, code: 'bench.too_few_prompts', detail: `Paste at least ${MIN_PROMPTS} prompts; this has ${prompts.length}.` };
+ }
+ if (prompts.length > MAX_PROMPTS) {
+ return { ok: false, code: 'bench.too_many_prompts', detail: `Paste at most ${MAX_PROMPTS} prompts; this has ${prompts.length}.` };
+ }
+ for (let i = 0; i < prompts.length; i++) {
+ const p = prompts[i];
+ if (p.prompt.length > MAX_PROMPT) return { ok: false, code: 'bench.prompt_too_long', detail: `Prompt ${i + 1} is longer than ${MAX_PROMPT} characters.` };
+ if (p.expected && p.expected.length > MAX_EXPECTED) return { ok: false, code: 'bench.expected_too_long', detail: `The expected answer for prompt ${i + 1} is longer than ${MAX_EXPECTED} characters.` };
+ }
+ return { ok: true, prompts, format };
+}
+
+/**
+ * Parse a paste.
+ *
+ * format 'lines' one prompt per line (a comma is part of the prompt)
+ * format 'csv' comma-separated, optional expected answer in the second column
+ * format 'auto' CSV only when the first line is a prompt/expected header, else lines
+ */
+function parsePrompts(raw, { format = 'auto' } = {}) {
+ const text = clean(raw);
+ if (!text) return { ok: false, code: 'bench.prompts_required', detail: 'Paste 5 to 50 prompts.' };
+ if (text.length > MAX_PASTE) return { ok: false, code: 'bench.prompts_too_large', detail: 'That paste is too large; keep it under 512 KB.' };
+ let mode = String(format || 'auto').toLowerCase();
+ if (!['auto', 'lines', 'csv'].includes(mode)) mode = 'auto';
+ const lines = text.split('\n').map((line) => line.trim()).filter(Boolean);
+ if (mode === 'auto') {
+ const head = csvRows(lines[0] || '');
+ mode = head.length && headerKind(head[0]) ? 'csv' : 'lines';
+ }
+ if (mode === 'csv') {
+ const rows = csvRows(text);
+ const header = rows.length ? headerKind(rows[0]) : null;
+ const parsed = promptRows(header ? rows.slice(1) : rows);
+ if (!parsed.ok) return parsed;
+ return bounds(parsed.prompts, 'csv');
+ }
+ // Lines: one prompt per line. A line may still be a CSV header by mistake; say so rather than making it a task.
+ const head = lines.length ? csvRows(lines[0]) : [];
+ if (head.length && headerKind(head[0])) {
+ return { ok: false, code: 'bench.prompt_header', detail: 'That looks like a CSV header. Choose the CSV format, or remove the header line.' };
+ }
+ return bounds(lines.map((prompt) => ({ prompt, expected: null })), 'lines');
+}
+
+/** The judges of a kind: its schema's list, else its own judge. */
+function judgesFor(kind) {
+ const list = kind && Array.isArray(kind.schema && kind.schema.judges) && kind.schema.judges.length
+ ? kind.schema.judges
+ : [kind ? kind.judge : 'automatic_grader'];
+ return list.map((type) => ({ type, version: '1.0', config: {} }));
+}
+
+/**
+ * The definition a paste becomes: the tasks (one per prompt), the kind's judges, mean scoring, a time limit from
+ * the kind and no dollar limit. No dollar limit means the existing launcher always asks before it starts, and no
+ * cap is added or raised here.
+ */
+function definitionFor({ prompts, kind, title, description }) {
+ const k = kindById(kind) || kindById('text.answer');
+ const name = String(title == null ? '' : title).replace(/\s+/g, ' ').trim().slice(0, 160) || DEFAULT_TITLE;
+ const text = String(description == null ? '' : description).trim().slice(0, 4000);
+ return {
+ title: name,
+ description: text,
+ kind: k.id,
+ tags: [TAG, String(k.id).split('.')[0]],
+ // Stored as unlisted in the contract document (the published shape knows public and unlisted); the
+ // benchmark row's own visibility is private, and that row is what access checks read.
+ visibility: 'unlisted',
+ tasks: prompts.map((p, i) => ({
+ id: `t${i + 1}`,
+ prompt: p.prompt,
+ files: [],
+ expected: p.expected ? { answer: p.expected } : null,
+ holdout: false,
+ })),
+ judges: judgesFor(k),
+ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
+ limits: { ms: Number(k.schema && k.schema.limit_ms) || 120000, tokens: 0, usd: 0 },
+ matrix: { models: [], harnesses: [] },
+ };
+}
+
+module.exports = {
+ parsePrompts, csvRows, definitionFor, judgesFor, headerKind,
+ MIN_PROMPTS, MAX_PROMPTS, MAX_PROMPT, MAX_EXPECTED, MAX_PASTE, DEFAULT_TITLE, TAG,
+};
@@ -32,7 +32,7 @@ const { BenchError } = require('../store');
@@ -32,7 +32,7 @@ const { BenchError } = require('../store');
const boardLib = require('../rating/board');
const boardLib = require('../rating/board');
function createApi(ctx) {
function createApi(ctx) {
- const { config, s, principal, bench } = ctx;
+ const { config, s, principal, bench, registry } = ctx;
const r = asyncRouter();
const r = asyncRouter();
r.use(express.json({ limit: '64kb' }));
r.use(express.json({ limit: '64kb' }));
@@ -226,6 +226,17 @@ function createApi(ctx) {
@@ -226,6 +226,17 @@ function createApi(ctx) {
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
+ // The open data for the benchmark registry (B11): public benchmarks, their visible tasks and every run under
+ // them. Private and unlisted rows are never read here, and no person id appears in a line.
+ r.get('/datasets/benchmarks.jsonl', async (req, res) => {
+ try {
+ const text = await registry.publicDatasetLines();
+ res.set('Cache-Control', 'public, max-age=60');
+ res.set('Content-Disposition', 'attachment; filename="bench-benchmarks.jsonl"');
+ return res.type('application/x-ndjson').send(text);
+ } catch (err) { return failed(res, err); }
+ });
+
// The open vote dump (stage B9): every human vote, no voter ids. It comes before the kind dataset route,
// The open vote dump (stage B9): every human vote, no voter ids. It comes before the kind dataset route,
// because '/datasets/votes.jsonl' would otherwise be read as the kind 'votes'.
// because '/datasets/votes.jsonl' would otherwise be read as the kind 'votes'.
r.get('/datasets/votes.jsonl', async (req, res) => {
r.get('/datasets/votes.jsonl', async (req, res) => {
@@ -51,7 +51,7 @@ function createDiscoveryRoutes(ctx) {
@@ -51,7 +51,7 @@ function createDiscoveryRoutes(ctx) {
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({
name: SITE_NAME,
name: SITE_NAME,
summary: DESCRIPTION,
summary: DESCRIPTION,
- details: 'GET /api/v1/kinds lists the benchmark kinds. GET /api/v1/boards/:kind?type=elo|pareto|cost renders a board as JSON. GET /api/v1/datasets/:kind.jsonl is the open dataset without voter ids. POST /mcp is the Model Context Protocol endpoint.',
+ details: 'GET /api/v1/kinds lists the benchmark kinds. GET /api/v1/boards/:kind?type=elo|pareto|cost renders a board as JSON. GET /api/v1/datasets/:kind.jsonl is the open dataset without voter ids. GET /api/v1/datasets/benchmarks.jsonl is every public benchmark and its runs. POST /mcp is the Model Context Protocol endpoint.',
sections: [
sections: [
{ title: 'Start here', links: [{ title: SITE_NAME, url: abs('/') }, { title: 'The API', url: abs('/docs') }] },
{ title: 'Start here', links: [{ title: SITE_NAME, url: abs('/') }, { title: 'The API', url: abs('/docs') }] },
{ title: 'Machine-readable', links: [{ title: 'Kinds (JSON)', url: abs('/api/v1/kinds') }, { title: 'Sitemap', url: abs('/sitemap.xml') }, { title: 'Release metadata (JSON)', url: abs('/release.json') }] },
{ title: 'Machine-readable', links: [{ title: 'Kinds (JSON)', url: abs('/api/v1/kinds') }, { title: 'Sitemap', url: abs('/sitemap.xml') }, { title: 'Release metadata (JSON)', url: abs('/release.json') }] },
@@ -69,9 +69,17 @@ function createDiscoveryRoutes(ctx) {
@@ -69,9 +69,17 @@ function createDiscoveryRoutes(ctx) {
}));
}));
});
});
- r.get('/sitemap.xml', (_req, res) => {
+ r.get('/sitemap.xml', async (_req, res) => {
const lastmod = siteUpdated();
const lastmod = siteUpdated();
- const urls = PAGES.map((p) => ({ loc: abs(p.path), ...(lastmod ? { lastmod } : {}), changefreq: p.changefreq, priority: p.priority }));
+ // Public pages only: a private or unlisted benchmark never enters the sitemap (the registry lists public
+ // ones only).
+ const pages = [...PAGES, { path: '/benchmarks', changefreq: 'daily', priority: 0.8, title: 'Benchmarks', text: 'Versioned AI benchmarks anyone can run again.' }];
+ const benchmarks = await ctx.registry.listBenchmarks({ limit: 200 });
+ for (const b of benchmarks) {
+ if (b.visibility !== 'public') continue;
+ pages.push({ path: `/benchmarks/${encodeURIComponent(b.id)}`, changefreq: 'weekly', priority: 0.7, title: b.title, text: b.description || b.title });
+ }
+ const urls = pages.map((p) => ({ loc: abs(p.path), ...(lastmod ? { lastmod } : {}), changefreq: p.changefreq, priority: p.priority }));
res.type('application/xml').set('Cache-Control', TEXT).send(seo.sitemapXml(urls));
res.type('application/xml').set('Cache-Control', TEXT).send(seo.sitemapXml(urls));
});
});
@@ -13,6 +13,10 @@ const { raw } = require('../render/html');
@@ -13,6 +13,10 @@ const { raw } = require('../render/html');
const { send } = require('../render/layout');
const { send } = require('../render/layout');
const views = require('../views');
const views = require('../views');
const results = require('../views/results');
const results = require('../views/results');
+const { canViewBenchmark } = require('../bench/registry');
+const { parsePrompts, definitionFor, TAG: PROMPT_TAG } = require('../bench/your-prompts');
+const { validateBenchmark } = require('../bench/validate');
+const { kindById } = require('../kinds');
const PUBLIC_CACHE = 'public, max-age=60';
const PUBLIC_CACHE = 'public, max-age=60';
@@ -69,6 +73,58 @@ function createPageRoutes(ctx) {
@@ -69,6 +73,58 @@ function createPageRoutes(ctx) {
const notFound = (req, res, what) => page(req, res, { title: `No such ${what}`, body: raw(views.notFound()) }, 404);
const notFound = (req, res, what) => page(req, res, { title: `No such ${what}`, body: raw(views.notFound()) }, 404);
const METRIC_IDS = results.METRICS.map((m) => m.id);
const METRIC_IDS = results.METRICS.map((m) => m.id);
+ /** The benchmark a page may show, or null: a private one belongs to its owner, a shared unlisted one to its link. */
+ const readableBenchmark = async (req, ref) => {
+ const out = await registry.getDefinition(String(ref || ''), { reveal: false });
+ if (!out) return null;
+ return canViewBenchmark(out.benchmark, { subject: subjectOf(req) }) ? out : null;
+ };
+
+ /** The public board of a kind: the first public benchmark with verified runs, else the first public one. */
+ const publicBoardForKind = async (kind, { excludeId = null } = {}) => {
+ const list = await registry.listBenchmarks({ kind: String(kind || ''), limit: 50 });
+ const publicList = list.filter((b) => b.visibility === 'public' && b.id !== excludeId);
+ for (const b of publicList) {
+ const rankings = await registry.rankings({ benchmark: b.id, metric: 'score' });
+ if (rankings && rankings.entries.length) return { benchmark: b, entries: rankings.entries };
+ }
+ return publicList.length ? { benchmark: publicList[0], entries: [] } : null;
+ };
+
+ /** One benchmark page, whether its owner, a public reader or someone following a share link is looking. */
+ const renderBenchmark = async (req, res, out, { share = null, shared = false, status = 200 } = {}) => {
+ const metric = METRIC_IDS.includes(String(req.query.metric)) ? String(req.query.metric) : 'score';
+ const id = out.benchmark.id;
+ const owner = subjectOf(req) != null && String(out.benchmark.owner) === String(subjectOf(req));
+ const [rankings, runs] = await Promise.all([
+ registry.rankings({ benchmark: id, metric }),
+ registry.listRuns({ benchmark: id, limit: 8 }),
+ ]);
+ const isPublic = out.benchmark.visibility === 'public';
+ const isPromptBenchmark = Array.isArray(out.benchmark.tags) && out.benchmark.tags.includes(PROMPT_TAG);
+ const ownBoards = owner && isPromptBenchmark
+ ? {
+ ownEntries: (rankings && rankings.entries) || [],
+ ownRuns: ((rankings && rankings.entries) || []).reduce((n, e) => n + e.runs, 0),
+ publicBoard: await publicBoardForKind(out.benchmark.kind, { excludeId: out.benchmark.id }),
+ visibility: out.benchmark.visibility,
+ }
+ : null;
+ return page(req, res, {
+ index: isPublic && metric === 'score' && !shared, cache: isPublic && !shared ? PUBLIC_CACHE : null,
+ title: out.benchmark.title,
+ description: out.benchmark.description || `${out.benchmark.title}: rankings, value and every run.`,
+ jsonLd: isPublic && !shared
+ ? [seo.jsonLd.webPage({ name: out.benchmark.title, url: `${base()}/benchmarks/${encodeURIComponent(id)}`, description: out.benchmark.description || out.benchmark.title, siteUrl: base() })]
+ : [],
+ body: raw(String(results.benchmarkPage({
+ benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version,
+ rankings, metric, runs, codesRunUrl: config.codesRunUrl || null, viewer: req.viewer, baseUrl: base(),
+ ownBoards, share, shared, owner,
+ }))),
+ }, status);
+ };
+
r.get('/benchmarks', async (req, res) => {
r.get('/benchmarks', async (req, res) => {
const q = String(req.query.q || '').slice(0, 80);
const q = String(req.query.q || '').slice(0, 80);
const family = String(req.query.kind || '').replace(/[^a-z0-9._-]/gi, '').slice(0, 40) || null;
const family = String(req.query.kind || '').replace(/[^a-z0-9._-]/gi, '').slice(0, 40) || null;
@@ -76,50 +132,115 @@ function createPageRoutes(ctx) {
@@ -76,50 +132,115 @@ function createPageRoutes(ctx) {
const kinds = [...new Set(all.map((b) => b.kind).filter(Boolean))];
const kinds = [...new Set(all.map((b) => b.kind).filter(Boolean))];
const list = family ? all.filter((b) => b.kind === family || String(b.kind).startsWith(`${family}.`)) : all;
const list = family ? all.filter((b) => b.kind === family || String(b.kind).startsWith(`${family}.`)) : all;
const counts = await registry.runCounts();
const counts = await registry.runCounts();
+ const subject = subjectOf(req);
+ const mine = subject ? (await registry.listBenchmarks({ owner: subject, limit: 100 })).filter((b) => b.visibility !== 'public') : [];
for (const b of list) b.runs = counts.get(b.id) || 0;
for (const b of list) b.runs = counts.get(b.id) || 0;
+ for (const b of mine) b.runs = counts.get(b.id) || 0;
return page(req, res, {
return page(req, res, {
index: !q && !family, cache: PUBLIC_CACHE,
index: !q && !family, cache: PUBLIC_CACHE,
title: 'Benchmarks',
title: 'Benchmarks',
description: 'Versioned AI benchmarks anyone can run again: tasks, judges and limits, with every run published.',
description: 'Versioned AI benchmarks anyone can run again: tasks, judges and limits, with every run published.',
jsonLd: [seo.jsonLd.webPage({ name: 'Benchmarks', url: `${base()}/benchmarks`, description: 'Versioned AI benchmarks anyone can run again.', siteUrl: base() })],
jsonLd: [seo.jsonLd.webPage({ name: 'Benchmarks', url: `${base()}/benchmarks`, description: 'Versioned AI benchmarks anyone can run again.', siteUrl: base() })],
- body: raw(String(results.benchmarksIndex({ benchmarks: list, kinds, kind: family, q }))),
+ body: raw(String(results.benchmarksIndex({ benchmarks: list, kinds, kind: family, q, mine, signedIn: Boolean(subject) }))),
});
});
});
});
- r.get('/benchmarks/:ref', async (req, res) => {
- const ref = String(req.params.ref || '');
- const out = await registry.getDefinition(ref, { reveal: false });
+ // ---- a person's own prompts (B11) ---------------------------------------------------------------------
+ // The form is a plain form post, same-origin, and nothing here logs the prompts.
+ r.get('/benchmarks/new', async (req, res) => {
+ const kinds = await bench.listKinds();
const subject = subjectOf(req);
const subject = subjectOf(req);
- if (!out || (out.benchmark.visibility === 'private' && out.benchmark.owner !== subject)) return notFound(req, res, 'benchmark');
- const metric = METRIC_IDS.includes(String(req.query.metric)) ? String(req.query.metric) : 'score';
- const id = out.benchmark.id;
- const [rankings, runs] = await Promise.all([
- registry.rankings({ benchmark: id, metric }),
- registry.listRuns({ benchmark: id, limit: 8 }),
- ]);
- const isPublic = out.benchmark.visibility === 'public';
return page(req, res, {
return page(req, res, {
- index: isPublic && metric === 'score', cache: isPublic ? PUBLIC_CACHE : null,
- title: out.benchmark.title,
- description: out.benchmark.description || `${out.benchmark.title}: rankings, value and every run.`,
- jsonLd: [seo.jsonLd.webPage({ name: out.benchmark.title, url: `${base()}/benchmarks/${encodeURIComponent(id)}`, description: out.benchmark.description || out.benchmark.title, siteUrl: base() })],
- body: raw(String(results.benchmarkPage({
- benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version,
- rankings, metric, runs, codesRunUrl: config.codesRunUrl || null, viewer: req.viewer, baseUrl: base(),
+ index: false, title: 'Make your own benchmark',
+ description: 'Paste 5 to 50 of your own prompts as a private benchmark, then run them on 3 to 6 models.',
+ body: raw(String(results.newBenchmarkPage({
+ kinds,
+ values: { kind: String(req.query.kind || 'text.answer') },
+ signedIn: Boolean(subject),
+ loginUrl: `/auth/login?next=${encodeURIComponent('/benchmarks/new')}`,
}))),
}))),
});
});
});
});
+ r.post('/benchmarks/new', express.urlencoded({ extended: false, limit: '512kb' }), ctx.principal.middleware, async (req, res) => {
+ if (!sameOrigin(req, config)) return res.status(403).type('text').send('This form only works from Bench itself.');
+ const subject = subjectOf(req);
+ const kinds = await bench.listKinds();
+ const b = req.body && typeof req.body === 'object' ? req.body : {};
+ const values = {
+ title: String(b.title || '').slice(0, 160),
+ description: String(b.description || '').slice(0, 4000),
+ kind: String(b.kind || 'text.answer'),
+ format: String(b.format || 'auto'),
+ prompts: String(b.prompts || ''),
+ };
+ const fail = (status, error) => page(req, res, {
+ index: false, title: 'Make your own benchmark',
+ description: 'Paste 5 to 50 of your own prompts as a private benchmark.',
+ body: raw(String(results.newBenchmarkPage({ kinds, values, error, signedIn: Boolean(subject), loginUrl: `/auth/login?next=${encodeURIComponent('/benchmarks/new')}` }))),
+ }, status);
+ if (!subject) return fail(401, 'Sign in to make a benchmark of your own.');
+ const kind = kindById(values.kind);
+ if (!kind) return fail(422, 'Pick one of Bench’s kinds.');
+ const parsed = parsePrompts(values.prompts, { format: values.format });
+ if (!parsed.ok) return fail(422, parsed.detail);
+ const definition = definitionFor({ prompts: parsed.prompts, kind: kind.id, title: values.title, description: values.description });
+ try {
+ const made = await registry.createBenchmark({
+ owner: subject, title: definition.title, description: definition.description, kind: definition.kind,
+ tags: definition.tags, visibility: 'private', definition, subject,
+ });
+ if (!made.ok) return fail(made.status || 422, made.message || 'The benchmark could not be made.');
+ const frozen = await registry.freezeVersion(made.benchmark.id, { definition, subject, validate: validateBenchmark });
+ if (!frozen.ok) return fail(frozen.status || 422, frozen.message || 'The benchmark could not be frozen.');
+ return res.redirect(303, `/benchmarks/${encodeURIComponent(made.benchmark.id)}`);
+ } catch (err) {
+ ctx.log.warn('[OpenVibe.Bench] making a prompt benchmark failed:', (err && err.message) || err);
+ return fail(500, 'The benchmark could not be made. Try again in a moment.');
+ }
+ });
+
+ // A share link: only an unlisted benchmark has one, and only its hash is stored.
+ r.get('/b/:token', async (req, res) => {
+ const benchmark = await registry.findShared(String(req.params.token || ''));
+ if (!benchmark) return notFound(req, res, 'share link');
+ const out = await registry.getDefinition(benchmark.id, { reveal: false });
+ if (!out) return notFound(req, res, 'share link');
+ return renderBenchmark(req, res, out, { shared: true });
+ });
+
+ // The owner shares, unshares or publishes: a plain same-origin form post.
+ r.post('/benchmarks/:ref/share', express.urlencoded({ extended: false, limit: '16kb' }), ctx.principal.middleware, async (req, res) => {
+ if (!sameOrigin(req, config)) return res.status(403).type('text').send('This form only works from Bench itself.');
+ const subject = subjectOf(req);
+ const out = await registry.getDefinition(String(req.params.ref || ''), { reveal: false });
+ if (!out || !subject || String(out.benchmark.owner) !== String(subject)) return notFound(req, res, 'benchmark');
+ const changed = await registry.setVisibility(out.benchmark.id, { visibility: String((req.body || {}).visibility || ''), subject });
+ if (!changed.ok) {
+ return page(req, res, { index: false, title: out.benchmark.title, body: raw(String(results.benchmarkPage({ benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version, rankings: await registry.rankings({ benchmark: out.benchmark.id, metric: 'score' }), runs: await registry.listRuns({ benchmark: out.benchmark.id, limit: 8 }), codesRunUrl: config.codesRunUrl || null, viewer: req.viewer, baseUrl: base(), owner: true, share: { error: changed.message } }))) }, 422);
+ }
+ const fresh = await registry.getDefinition(out.benchmark.id, { reveal: false });
+ return renderBenchmark(req, res, fresh, {
+ share: changed.token ? { shareUrl: `${base()}${changed.share_path}`, token: changed.token } : null,
+ });
+ });
+
+ r.get('/benchmarks/:ref', async (req, res) => {
+ const out = await readableBenchmark(req, req.params.ref);
+ if (!out) return notFound(req, res, 'benchmark');
+ return renderBenchmark(req, res, out);
+ });
+
// The run form (B3): GET shows it, POST starts the run through the same launcher as the API.
// The run form (B3): GET shows it, POST starts the run through the same launcher as the API.
const runFormPage = async (req, res, extra = {}) => {
const runFormPage = async (req, res, extra = {}) => {
const ref = String(req.params.ref || '');
const ref = String(req.params.ref || '');
- const out = await registry.getDefinition(ref, { reveal: false });
- if (!out || out.benchmark.visibility === 'private') return notFound(req, res, 'benchmark');
+ const out = await readableBenchmark(req, ref);
+ if (!out) return notFound(req, res, 'benchmark');
if (!config.codesRunUrl) return res.redirect(303, `/benchmarks/${encodeURIComponent(out.benchmark.id)}`);
if (!config.codesRunUrl) return res.redirect(303, `/benchmarks/${encodeURIComponent(out.benchmark.id)}`);
const signedIn = req.viewer && req.viewer.kind === 'user';
const signedIn = req.viewer && req.viewer.kind === 'user';
return page(req, res, {
return page(req, res, {
index: false, title: `Run ${out.benchmark.title}`,
index: false, title: `Run ${out.benchmark.title}`,
- description: `Run ${out.benchmark.title} on openvibe.codes with your own keys.`,
+ description: `Run ${out.benchmark.title} on openvibe.codes with your own keys or the free pool.`,
body: raw(String(results.runForm({
body: raw(String(results.runForm({
benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version, signedIn,
benchmark: out.benchmark, definition: out.definition, version: out.version && out.version.version, signedIn,
harnesses: signedIn ? await ctx.launcher.harnessList() : [],
harnesses: signedIn ? await ctx.launcher.harnessList() : [],
@@ -139,6 +260,7 @@ function createPageRoutes(ctx) {
@@ -139,6 +260,7 @@ function createPageRoutes(ctx) {
const picked = { models, harnesses, repeat };
const picked = { models, harnesses, repeat };
const out = await ctx.launcher.launch({ principal: req.principal, id: req.params.ref, models, harnesses, repeat, confirm: b.confirm === '1' });
const out = await ctx.launcher.launch({ principal: req.principal, id: req.params.ref, models, harnesses, repeat, confirm: b.confirm === '1' });
if (out.status === 202) return res.redirect(303, out.body.tasks_url || config.codesRunUrl);
if (out.status === 202) return res.redirect(303, out.body.tasks_url || config.codesRunUrl);
+ if (out.status === 404) return notFound(req, res, 'benchmark');
if (out.status === 409) return runFormPage(req, res, { picked, confirm: out.body.cost_estimate_usd == null ? 'unknown' : out.body.cost_estimate_usd });
if (out.status === 409) return runFormPage(req, res, { picked, confirm: out.body.cost_estimate_usd == null ? 'unknown' : out.body.cost_estimate_usd });
return runFormPage(req, res, { picked, error: out.body.detail || 'The run could not start.', status: out.status === 400 ? 400 : 200 });
return runFormPage(req, res, { picked, error: out.body.detail || 'The run could not start.', status: out.status === 400 ? 400 : 200 });
});
});
@@ -148,7 +270,7 @@ function createPageRoutes(ctx) {
@@ -148,7 +270,7 @@ function createPageRoutes(ctx) {
if (!run) return notFound(req, res, 'run');
if (!run) return notFound(req, res, 'run');
const def = await registry.getDefinition(`${run.benchmark_id}@${run.benchmark_version}`, { reveal: false });
const def = await registry.getDefinition(`${run.benchmark_id}@${run.benchmark_version}`, { reveal: false });
const benchmark = def ? def.benchmark : null;
const benchmark = def ? def.benchmark : null;
- if (benchmark && benchmark.visibility === 'private' && benchmark.owner !== subjectOf(req)) return notFound(req, res, 'run');
+ if (benchmark && !canViewBenchmark(benchmark, { subject: subjectOf(req) })) return notFound(req, res, 'run');
const pct = run.score != null ? await registry.percentile({ benchmark: run.benchmark_id, model: run.model, score: run.score }) : null;
const pct = run.score != null ? await registry.percentile({ benchmark: run.benchmark_id, model: run.model, score: run.score }) : null;
const isPublic = !benchmark || benchmark.visibility === 'public';
const isPublic = !benchmark || benchmark.visibility === 'public';
return page(req, res, {
return page(req, res, {
@@ -163,19 +285,21 @@ function createPageRoutes(ctx) {
@@ -163,19 +285,21 @@ function createPageRoutes(ctx) {
const refs = String(req.query.runs || '').split(',').map((x) => x.trim()).filter(Boolean).slice(0, 4);
const refs = String(req.query.runs || '').split(',').map((x) => x.trim()).filter(Boolean).slice(0, 4);
let cmp = refs.length ? await registry.compareRuns(refs) : null;
let cmp = refs.length ? await registry.compareRuns(refs) : null;
const titles = {};
const titles = {};
+ let allPublic = true;
if (cmp) {
if (cmp) {
const subject = subjectOf(req);
const subject = subjectOf(req);
const keep = [];
const keep = [];
for (const run of cmp.runs) {
for (const run of cmp.runs) {
const b = await registry.getBenchmark(run.benchmark_id);
const b = await registry.getBenchmark(run.benchmark_id);
- if (b && b.visibility === 'private' && b.owner !== subject) continue;
- if (b) titles[b.id] = b.title;
+ if (!b || !canViewBenchmark(b, { subject })) continue;
+ if (b.visibility !== 'public') allPublic = false;
+ titles[b.id] = b.title;
keep.push(run);
keep.push(run);
}
}
cmp = keep.length ? { ...cmp, runs: keep, rows: cmp.rows.map((row) => ({ ...row, scores: keep.map((run) => { const t = run.tasks.find((x) => x.task_id === row.task_id); return t ? t.score : null; }) })) } : null;
cmp = keep.length ? { ...cmp, runs: keep, rows: cmp.rows.map((row) => ({ ...row, scores: keep.map((run) => { const t = run.tasks.find((x) => x.task_id === row.task_id); return t ? t.score : null; }) })) } : null;
}
}
return page(req, res, {
return page(req, res, {
- index: false, cache: PUBLIC_CACHE,
+ index: false, cache: allPublic ? PUBLIC_CACHE : null,
title: cmp ? `Compare: ${cmp.runs.map((x) => x.model).join(' vs ')}` : 'Compare runs',
title: cmp ? `Compare: ${cmp.runs.map((x) => x.model).join(' vs ')}` : 'Compare runs',
description: 'Runs side by side: the same tasks, their scores, cost, time and outputs.',
description: 'Runs side by side: the same tasks, their scores, cost, time and outputs.',
body: raw(String(results.comparePage({ cmp, titles }))),
body: raw(String(results.comparePage({ cmp, titles }))),
@@ -23,6 +23,7 @@ const { http } = require('openvibe-contracts');
@@ -23,6 +23,7 @@ const { http } = require('openvibe-contracts');
const { asyncRouter } = require('./router');
const { asyncRouter } = require('./router');
const { sameOrigin } = require('./same-origin');
const { sameOrigin } = require('./same-origin');
const { validateBenchmark, validateRun } = require('../bench/validate');
const { validateBenchmark, validateRun } = require('../bench/validate');
+const { canViewBenchmark } = require('../bench/registry');
function createRegistryApi(ctx) {
function createRegistryApi(ctx) {
const { config, principal, registry } = ctx;
const { config, principal, registry } = ctx;
@@ -67,6 +68,13 @@ function createRegistryApi(ctx) {
@@ -67,6 +68,13 @@ function createRegistryApi(ctx) {
return false;
return false;
};
};
+ /** The caller's view of a benchmark: its owner, or the executor that must read the tasks. */
+ const viewerOf = (req) => ({
+ subject: subjectOf(req),
+ executor: req.principal.kind === 'app' && req.principal.hasCapability('bench.run.publish'),
+ });
+ const mayRead = (req, benchmark) => canViewBenchmark(benchmark, viewerOf(req));
+
// ── Benchmarks ───────────────────────────────────────────
// ── Benchmarks ───────────────────────────────────────────
r.post('/benchmarks', actor('bench.benchmark.write'), async (req, res) => {
r.post('/benchmarks', actor('bench.benchmark.write'), async (req, res) => {
const b = req.body && typeof req.body === 'object' ? req.body : {};
const b = req.body && typeof req.body === 'object' ? req.body : {};
@@ -127,6 +135,17 @@ function createRegistryApi(ctx) {
@@ -127,6 +135,17 @@ function createRegistryApi(ctx) {
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
+ r.post('/benchmarks/:id/share', actor('bench.benchmark.write'), async (req, res) => {
+ const b = req.body && typeof req.body === 'object' ? req.body : {};
+ const subject = subjectOf(req) || req.principal.requester;
+ try {
+ const out = await registry.setVisibility(String(req.params.id || ''), { visibility: b.visibility, subject });
+ if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid visibility');
+ res.set('Cache-Control', 'no-store');
+ return res.json({ benchmark: out.benchmark, token: out.token, share_path: out.share_path });
+ } catch (err) { return failed(res, err); }
+ });
+
r.get('/benchmarks', async (req, res) => {
r.get('/benchmarks', async (req, res) => {
try {
try {
// ?mine=1 adds the caller's own unlisted and private benchmarks; nobody else ever sees those in a list.
// ?mine=1 adds the caller's own unlisted and private benchmarks; nobody else ever sees those in a list.
@@ -138,7 +157,7 @@ function createRegistryApi(ctx) {
@@ -138,7 +157,7 @@ function createRegistryApi(ctx) {
q: req.query.q ? String(req.query.q) : null,
q: req.query.q ? String(req.query.q) : null,
limit: req.query.limit ? Number(req.query.limit) : 50,
limit: req.query.limit ? Number(req.query.limit) : 50,
});
});
- res.set('Cache-Control', 'public, max-age=60');
+ res.set('Cache-Control', mine ? 'private, no-store' : 'public, max-age=60');
return res.json({ count: list.length, benchmarks: list });
return res.json({ count: list.length, benchmarks: list });
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
@@ -146,16 +165,18 @@ function createRegistryApi(ctx) {
@@ -146,16 +165,18 @@ function createRegistryApi(ctx) {
r.get('/benchmarks/:id', async (req, res) => {
r.get('/benchmarks/:id', async (req, res) => {
try {
try {
const ref = String(req.params.id || '');
const ref = String(req.params.id || '');
- // The holdout goes to the executor and to the benchmark's owner; to nobody else.
- let reveal = mayReveal(req);
- if (!reveal && req.principal.kind === 'user') {
- const owner = await registry.getBenchmark(ref.split('@')[0]);
- reveal = !!(owner && owner.owner === req.principal.subject);
- }
- const out = await registry.getDefinition(ref, { reveal });
+ const out = await registry.getDefinition(ref, { reveal: false });
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
- // A private benchmark answers only its owner and the executor; to everyone else it does not exist.
- if (out.benchmark.visibility === 'private' && !reveal) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
+ // A private benchmark answers only its owner and the executor; a shared unlisted one only its owner by
+ // id (the token link is the way in). Everyone else gets a 404, not a "forbidden".
+ if (!mayRead(req, out.benchmark)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
+ const view = viewerOf(req);
+ const owner = view.subject != null && String(out.benchmark.owner) === String(view.subject);
+ const reveal = mayReveal(req) || owner;
+ if (reveal) {
+ const full = await registry.getDefinition(ref, { reveal: true });
+ if (full) { out.definition = full.definition; out.version = full.version; }
+ }
res.set('Cache-Control', out.benchmark.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
res.set('Cache-Control', out.benchmark.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
return res.json({
return res.json({
benchmark: out.benchmark,
benchmark: out.benchmark,
@@ -205,6 +226,10 @@ function createRegistryApi(ctx) {
@@ -205,6 +226,10 @@ function createRegistryApi(ctx) {
const check = validateRun({ ...doc, id: doc.id || 'brn_00000000000000000000000000' });
const check = validateRun({ ...doc, id: doc.id || 'brn_00000000000000000000000000' });
if (!check.valid) return invalid(res, check.errors.map((e) => `${e.path} ${e.message}`).join('; '));
if (!check.valid) return invalid(res, check.errors.map((e) => `${e.path} ${e.message}`).join('; '));
try {
try {
+ // A run may only be published on a benchmark its author may read: another person's private one is a 404.
+ const bmId = String(doc.benchmark_id || doc.benchmark || '').split('@')[0];
+ const bm = bmId ? await registry.getBenchmark(bmId) : null;
+ if (bm && !mayRead(req, bm)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
const out = await registry.publishRun(doc, { provenance, subject });
const out = await registry.publishRun(doc, { provenance, subject });
if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid run');
if (!out.ok) return problem(res, out.status || 400, out.code || 'bench.invalid', out.message || 'invalid run');
res.set('Location', `/api/v1/runs/${out.run.id}`);
res.set('Location', `/api/v1/runs/${out.run.id}`);
@@ -216,7 +241,9 @@ function createRegistryApi(ctx) {
@@ -216,7 +241,9 @@ function createRegistryApi(ctx) {
try {
try {
const run = await registry.getRun(String(req.params.id || ''));
const run = await registry.getRun(String(req.params.id || ''));
if (!run) return problem(res, 404, 'bench.run_not_found', 'No such run.');
if (!run) return problem(res, 404, 'bench.run_not_found', 'No such run.');
- res.set('Cache-Control', 'public, max-age=60');
+ const bm = await registry.getBenchmark(run.benchmark_id);
+ if (bm && !mayRead(req, bm)) return problem(res, 404, 'bench.run_not_found', 'No such run.');
+ res.set('Cache-Control', bm && bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
return res.json({ run });
return res.json({ run });
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
@@ -229,8 +256,30 @@ function createRegistryApi(ctx) {
@@ -229,8 +256,30 @@ function createRegistryApi(ctx) {
try {
try {
const out = await registry.compareRuns(refs);
const out = await registry.compareRuns(refs);
if (!out) return problem(res, 404, 'bench.run_not_found', 'No such run.');
if (!out) return problem(res, 404, 'bench.run_not_found', 'No such run.');
- res.set('Cache-Control', 'public, max-age=60');
- return res.json({ compare: out });
+ // One unreadable run hides itself; if none is readable the comparison does not exist.
+ const allowed = [];
+ let allPublic = true;
+ for (const run of out.runs) {
+ const bm = await registry.getBenchmark(run.benchmark_id);
+ if (!bm || mayRead(req, bm)) allowed.push(run);
+ if (!bm || bm.visibility !== 'public') allPublic = false;
+ }
+ if (!allowed.length) return problem(res, 404, 'bench.run_not_found', 'No such run.');
+ const taskIds = out.task_ids;
+ allowed.forEach((run) => { for (const t of run.tasks) if (!taskIds.includes(t.task_id)) taskIds.push(t.task_id); });
+ const compare = {
+ runs: allowed,
+ task_ids: taskIds,
+ rows: taskIds.map((taskId) => ({
+ task_id: taskId,
+ scores: allowed.map((run) => {
+ const t = run.tasks.find((x) => x.task_id === taskId);
+ return t ? t.score : null;
+ }),
+ })),
+ };
+ res.set('Cache-Control', allPublic ? 'public, max-age=60' : 'private, no-store');
+ return res.json({ compare });
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
@@ -242,6 +291,8 @@ function createRegistryApi(ctx) {
@@ -242,6 +291,8 @@ function createRegistryApi(ctx) {
return invalid(res, 'metric is score, value, speed or consistency');
return invalid(res, 'metric is score, value, speed or consistency');
}
}
try {
try {
+ const bm = await registry.getBenchmark(benchmark.split('@')[0]);
+ if (!bm || !mayRead(req, bm)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
const out = await registry.rankings({
const out = await registry.rankings({
benchmark,
benchmark,
metric,
metric,
@@ -250,7 +301,7 @@ function createRegistryApi(ctx) {
@@ -250,7 +301,7 @@ function createRegistryApi(ctx) {
includeSubmitted: String(req.query.include_submitted || '') === 'true',
includeSubmitted: String(req.query.include_submitted || '') === 'true',
});
});
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
- res.set('Cache-Control', 'public, max-age=60');
+ res.set('Cache-Control', bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
return res.json({ rankings: out });
return res.json({ rankings: out });
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
@@ -262,9 +313,11 @@ function createRegistryApi(ctx) {
@@ -262,9 +313,11 @@ function createRegistryApi(ctx) {
if (!model) return invalid(res, 'model is required: ?model=…');
if (!model) return invalid(res, 'model is required: ?model=…');
if (req.query.score == null) return invalid(res, 'score is required: ?score=…');
if (req.query.score == null) return invalid(res, 'score is required: ?score=…');
try {
try {
+ const bm = await registry.getBenchmark(benchmark.split('@')[0]);
+ if (!bm || !mayRead(req, bm)) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
const out = await registry.percentile({ benchmark, model, score: Number(req.query.score) });
const out = await registry.percentile({ benchmark, model, score: Number(req.query.score) });
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
- res.set('Cache-Control', 'public, max-age=60');
+ res.set('Cache-Control', bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
return res.json({ percentile: out });
return res.json({ percentile: out });
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
@@ -4,7 +4,8 @@
@@ -4,7 +4,8 @@
* Account export and deletion → Bench (ADR-033; openvibe-sdk/account-data). What Bench holds about a person:
* Account export and deletion → Bench (ADR-033; openvibe-sdk/account-data). What Bench holds about a person:
*
*
* theirs, and gone the tasks and rounds they started, the private evals they own, the runs they
* theirs, and gone the tasks and rounds they started, the private evals they own, the runs they
- * submitted, the pairs they have seen and the benchmarks they watch
+ * submitted, a private benchmark with its prompts and every run on it, the pairs they
+ * have seen and the benchmarks they watch
* theirs, and kept a judgement they cast stays (it sits under somebody else's round, so the round and
* theirs, and kept a judgement they cast stays (it sits under somebody else's round, so the round and
* the ratings keep their place) and loses the person; a benchmark other people ran
* the ratings keep their place) and loses the person; a benchmark other people ran
* stays as a tombstone, so their runs keep theirs
* stays as a tombstone, so their runs keep theirs
@@ -12,8 +13,10 @@
@@ -12,8 +13,10 @@
* person), the entries (a model's output) and the receipts of the requests themselves
* person), the entries (a model's output) and the receipts of the requests themselves
*
*
* A benchmark and its frozen versions are the person's, but other people's runs hang under them, so what happens
* A benchmark and its frozen versions are the person's, but other people's runs hang under them, so what happens
- * to them depends on what else is there: a benchmark nobody else ran goes with its versions, one others ran stays
- * as a tombstone. extraErase decides that, and GUARDED names the tables the guard needs to know about.
+ * to them depends on what else is there: a private benchmark (a person's own prompts) goes with its versions and
+ * every run on it, wherever it was shared; a public or unlisted benchmark nobody else ran goes too; one others ran
+ * stays as a tombstone, so their runs keep their place. extraErase decides that, and GUARDED names the tables the
+ * guard needs to know about.
*/
*/
const { createAccountData, TOPICS } = require('openvibe-sdk/account-data');
const { createAccountData, TOPICS } = require('openvibe-sdk/account-data');
@@ -38,7 +41,7 @@ const TABLES = [
@@ -38,7 +41,7 @@ const TABLES = [
// under them. extraErase decides; the guard needs the names.
// under them. extraErase decides; the guard needs the names.
const GUARDED = ['bench_benchmarks', 'bench_benchmark_versions'];
const GUARDED = ['bench_benchmarks', 'bench_benchmark_versions'];
-/** The benchmarks of the person: what they own and what they wrote, with their versions. */
+/** The benchmarks of the person: what they own and what they wrote, with their versions and their private runs. */
async function extraExport(db, subject) {
async function extraExport(db, subject) {
const files = [];
const files = [];
const benchmarks = await db.many(`SELECT id, owner, title, description, kind, tags, visibility, parent, latest_version, draft, created_by, created_at, updated_at
const benchmarks = await db.many(`SELECT id, owner, title, description, kind, tags, visibility, parent, latest_version, draft, created_by, created_at, updated_at
@@ -47,21 +50,42 @@ async function extraExport(db, subject) {
@@ -47,21 +50,42 @@ async function extraExport(db, subject) {
const versions = await db.many(`SELECT benchmark_id, version, definition, hash, created_by, created_at FROM bench_benchmark_versions
const versions = await db.many(`SELECT benchmark_id, version, definition, hash, created_by, created_at FROM bench_benchmark_versions
WHERE created_by = $1 OR benchmark_id IN (SELECT id FROM bench_benchmarks WHERE owner = $1) ORDER BY created_at DESC LIMIT 5000`, [subject]);
WHERE created_by = $1 OR benchmark_id IN (SELECT id FROM bench_benchmarks WHERE owner = $1) ORDER BY created_at DESC LIMIT 5000`, [subject]);
if (versions.length) files.push({ name: 'benchmark_versions.json', content: versions });
if (versions.length) files.push({ name: 'benchmark_versions.json', content: versions });
+ // A run on the person's own prompt benchmark is part of their data even when a service executed and published
+ // it, and even if the set was shared.
+ const runs = await db.many(`SELECT id, benchmark_id, benchmark_version, model, route, provider, harness, settings, hardware,
+ score, cost_usd, ms, tokens, provenance, session, created_at
+ FROM bench_runs WHERE benchmark_id IN (
+ SELECT id FROM bench_benchmarks WHERE owner = $1
+ AND (visibility = 'private' OR tags @> '["your-prompts"]'::jsonb))
+ ORDER BY created_at DESC LIMIT 5000`, [subject]);
+ if (runs.length) files.push({ name: 'benchmark_runs.json', content: runs });
return files;
return files;
}
}
async function extraErase(t, subjects, counts) {
async function extraErase(t, subjects, counts) {
- // The benchmarks the person owns. One nobody else ran goes with its versions; one others ran stays as a
- // tombstone, so their runs keep their place.
- const ids = (await t.many('SELECT id FROM bench_benchmarks WHERE owner = ANY($1::text[])', [subjects])).map((r) => r.id);
+ // A private benchmark, or one made from the person's own pasted prompts (tagged your-prompts), is nothing but
+ // their text: it goes with its versions and every run on it, wherever it was shared. A public or unlisted
+ // benchmark another person ran stays as a tombstone, so their runs keep their place.
+ const owned = await t.many('SELECT id, visibility, tags FROM bench_benchmarks WHERE owner = ANY($1::text[])', [subjects]);
+ const tagsOf = (row) => {
+ let tags = row.tags;
+ if (typeof tags === 'string') { try { tags = JSON.parse(tags); } catch { tags = []; } }
+ return Array.isArray(tags) ? tags : [];
+ };
+ const deleteIds = owned.filter((r) => r.visibility === 'private' || tagsOf(r).includes('your-prompts')).map((r) => r.id);
+ if (deleteIds.length) {
+ counts.add(counts.erased, 'bench_runs', await t.exec('DELETE FROM bench_runs WHERE benchmark_id = ANY($1::text[])', [deleteIds]));
+ counts.add(counts.erased, 'bench_benchmarks', await t.exec('DELETE FROM bench_benchmarks WHERE id = ANY($1::text[])', [deleteIds]));
+ }
const kept = [];
const kept = [];
- for (const id of ids) {
- const runs = Number(await t.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [id])) || 0;
- if (runs) { kept.push(id); continue; }
- counts.add(counts.erased, 'bench_benchmarks', await t.exec('DELETE FROM bench_benchmarks WHERE id = $1', [id]));
+ for (const row of owned) {
+ if (deleteIds.includes(row.id)) continue;
+ const runs = Number(await t.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [row.id])) || 0;
+ if (runs) { kept.push(row.id); continue; }
+ counts.add(counts.erased, 'bench_benchmarks', await t.exec('DELETE FROM bench_benchmarks WHERE id = $1', [row.id]));
}
}
if (kept.length) {
if (kept.length) {
- counts.add(counts.retained, 'tombstones', await t.exec('UPDATE bench_benchmarks SET owner = $2, created_by = $2 WHERE id = ANY($1::text[])', [kept, TOMBSTONE]));
+ counts.add(counts.retained, 'tombstones', await t.exec('UPDATE bench_benchmarks SET owner = $2, created_by = $2, share_token_hash = NULL WHERE id = ANY($1::text[])', [kept, TOMBSTONE]));
}
}
// The benchmarks of somebody else that the person wrote: the definition stays, the person leaves it.
// The benchmarks of somebody else that the person wrote: the definition stays, the person leaves it.
counts.add(counts.retained, 'tombstones', await t.exec('UPDATE bench_benchmarks SET created_by = $2 WHERE created_by = ANY($1::text[])', [subjects, TOMBSTONE]));
counts.add(counts.retained, 'tombstones', await t.exec('UPDATE bench_benchmarks SET created_by = $2 WHERE created_by = ANY($1::text[])', [subjects, TOMBSTONE]));
@@ -76,7 +100,7 @@ function create({ db, log = console } = {}) {
@@ -76,7 +100,7 @@ function create({ db, log = console } = {}) {
tables: TABLES,
tables: TABLES,
extraExport,
extraExport,
extraErase,
extraErase,
- note: 'Bench exports your tasks, rounds, judgements, private evals, runs, watched benchmarks and the pairs you have seen; the public dataset is anonymous and a rating never names a person.',
+ note: 'Bench exports your tasks, rounds, judgements, private evals, benchmarks and their prompts, the runs on your private benchmarks, watched benchmarks and the pairs you have seen; the public dataset is anonymous and a rating never names a person.',
log,
log,
});
});
}
}
@@ -27,6 +27,9 @@ const KINDS = Object.freeze([
@@ -27,6 +27,9 @@ const KINDS = Object.freeze([
{ id: 'speech.asr', room: 'speech', entry: 'transcript', judge: 'automatic_grader', name: 'Speech recognition', metrics: ['word_error_rate', 'cost_usd'], schema: { entry: 'transcript', payload: { transcript: 'string' } } },
{ id: 'speech.asr', room: 'speech', entry: 'transcript', judge: 'automatic_grader', name: 'Speech recognition', metrics: ['word_error_rate', 'cost_usd'], schema: { entry: 'transcript', payload: { transcript: 'string' } } },
{ id: 'translate.ja', room: 'translate', entry: 'translation', judge: 'human_pairwise', name: 'Japanese translation', metrics: ['elo', 'cost_usd'], schema: { entry: 'translation', payload: { translation: 'string' }, prompt: { source_text: 'string', source_language: 'string', audience: 'string' }, rubric: ['Preserve the source meaning', 'Use natural Japanese for the audience', 'Preserve names, numbers and uncertainty'], seeds: [{ source_text: 'The meeting starts at 3 p.m. Please bring the draft.', source_language: 'en', audience: 'coworker' }] } },
{ id: 'translate.ja', room: 'translate', entry: 'translation', judge: 'human_pairwise', name: 'Japanese translation', metrics: ['elo', 'cost_usd'], schema: { entry: 'translation', payload: { translation: 'string' }, prompt: { source_text: 'string', source_language: 'string', audience: 'string' }, rubric: ['Preserve the source meaning', 'Use natural Japanese for the audience', 'Preserve names, numbers and uncertainty'], seeds: [{ source_text: 'The meeting starts at 3 p.m. Please bring the draft.', source_language: 'en', audience: 'coworker' }] } },
{ id: 'search.answer', room: 'search', entry: 'cited_answer', judge: 'human_pairwise', name: 'Cited answers', metrics: ['elo', 'cost_usd'], schema: { entry: 'cited_answer', payload: { answer: 'string', citations: 'array' }, prompt: { question: 'string', source_urls: 'array' }, rubric: ['Answer from the supplied sources', 'Cite claims with supporting URLs', 'State when the sources do not establish an answer'], seeds: [{ question: 'What year was the first release announced?', source_urls: ['https://example.org/releases'] }] } },
{ id: 'search.answer', room: 'search', entry: 'cited_answer', judge: 'human_pairwise', name: 'Cited answers', metrics: ['elo', 'cost_usd'], schema: { entry: 'cited_answer', payload: { answer: 'string', citations: 'array' }, prompt: { question: 'string', source_urls: 'array' }, rubric: ['Answer from the supplied sources', 'Cite claims with supporting URLs', 'State when the sources do not establish an answer'], seeds: [{ question: 'What year was the first release announced?', source_urls: ['https://example.org/releases'] }] } },
+ // B11: a plain text answer is the default kind for a person's own prompts. No renderer: the answer is text, and
+ // the expected answer (when the pasted CSV carries one) tells the automatic grader what correct looks like.
+ { id: 'text.answer', room: 'bench', entry: 'model_answer', judge: 'automatic_grader', name: 'Text answers', metrics: ['pass_rate', 'cost_usd', 'ms'], schema: { entry: 'model_answer', payload: { text: 'string' }, judges: ['automatic_grader'], rubric: ['Answer the prompt directly', 'Respect any format the prompt asks for', 'Do not add unrequested commentary'] } },
// The nine showcase kinds (spec 2, 2026-10-30). Script-bearing kinds render on the isolated render origin
// The nine showcase kinds (spec 2, 2026-10-30). Script-bearing kinds render on the isolated render origin
// (server/bench/render.js): the inert in-page sandbox allows no scripts, so a game, a canvas loop, three.js or
// (server/bench/render.js): the inert in-page sandbox allows no scripts, so a game, a canvas loop, three.js or
// a shader cannot run there.
// a shader cannot run there.
@@ -16,7 +16,7 @@ function home({ kinds = [], counts = { kinds: 0 }, benchmarks = [] } = {}) {
@@ -16,7 +16,7 @@ function home({ kinds = [], counts = { kinds: 0 }, benchmarks = [] } = {}) {
return `<div class="bm-wrap bm-home" id="bm-home">
return `<div class="bm-wrap bm-home" id="bm-home">
<header class="bm-hero"><p class="bm-kicker">OpenVibe.Bench</p><h1>See what AI really makes</h1>
<header class="bm-hero"><p class="bm-kicker">OpenVibe.Bench</p><h1>See what AI really makes</h1>
<p class="bm-lede">Every model on the same tasks: real pages, games, scenes and fixes, shown live side by side. Every run is published with what it cost and how long it took, and anyone can run it again.</p>
<p class="bm-lede">Every model on the same tasks: real pages, games, scenes and fixes, shown live side by side. Every run is published with what it cost and how long it took, and anyone can run it again.</p>
-<div class="bm-actions"><a class="bm-btn" href="/benchmarks">Browse benchmarks</a><a class="bm-btn bm-btn-quiet" href="/showcase">Vote in the showcase</a></div></header>
+<div class="bm-actions"><a class="bm-btn" href="/benchmarks">Browse benchmarks</a><a class="bm-btn bm-btn-quiet" href="/benchmarks/new">Make your own prompts</a><a class="bm-btn bm-btn-quiet" href="/showcase">Vote in the showcase</a></div></header>
<ul class="bm-why" aria-label="What Bench promises">
<ul class="bm-why" aria-label="What Bench promises">
<li><b>Outputs, not just scores</b><span>The pages, games and drawings themselves, live in sandboxed frames, next to the numbers.</span></li>
<li><b>Outputs, not just scores</b><span>The pages, games and drawings themselves, live in sandboxed frames, next to the numbers.</span></li>
<li><b>Every run published</b><span>Whatever it scored. No private variants, no best of ten: the service that ran it publishes it.</span></li>
<li><b>Every run published</b><span>Whatever it scored. No private variants, no best of ten: the service that ran it publishes it.</span></li>
@@ -44,6 +44,7 @@ ${ep('GET', '/api/v1/rounds/:id', 'One round with entries; entries stay blind un
@@ -44,6 +44,7 @@ ${ep('GET', '/api/v1/rounds/:id', 'One round with entries; entries stay blind un
${ep('GET', '/api/v1/boards/:kindId?type=elo|pareto|cost', 'A rendered board.')}
${ep('GET', '/api/v1/boards/:kindId?type=elo|pareto|cost', 'A rendered board.')}
${ep('GET', '/api/v1/models/:model', 'One model across every kind.')}
${ep('GET', '/api/v1/models/:model', 'One model across every kind.')}
${ep('GET', '/api/v1/datasets/:kindId.jsonl', 'The open dataset (prompts, entries, judgements without voter ids, ratings).')}
${ep('GET', '/api/v1/datasets/:kindId.jsonl', 'The open dataset (prompts, entries, judgements without voter ids, ratings).')}
+${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its visible tasks and every run, with no person id.')}
</dl></section>
</dl></section>
<section><h2>Write</h2><dl class="bm-eps">
<section><h2>Write</h2><dl class="bm-eps">
${ep('POST', '/api/v1/kinds/:kindId/tasks', '<code>{ prompt, task_class?, public? }</code>. A person or a service with bench.task.create.')}
${ep('POST', '/api/v1/kinds/:kindId/tasks', '<code>{ prompt, task_class?, public? }</code>. A person or a service with bench.task.create.')}
@@ -55,10 +56,11 @@ ${ep('POST', '/api/v1/evals', 'Create a private task suite (never in the public
@@ -55,10 +56,11 @@ ${ep('POST', '/api/v1/evals', 'Create a private task suite (never in the public
${ep('POST', '/api/v1/evals/:id/run', 'Run a private eval against chosen offers, under the free daily cap.')}
${ep('POST', '/api/v1/evals/:id/run', 'Run a private eval against chosen offers, under the free daily cap.')}
${ep('GET', '/api/v1/evals/:id/results', 'A private eval and its runs.')}
${ep('GET', '/api/v1/evals/:id/results', 'A private eval and its runs.')}
</dl></section>
</dl></section>
-<section id="br-publish"><h2>Benchmarks and runs</h2><p class="bm-fine">A benchmark is a versioned set of tasks, judges and limits; a frozen version never changes once a run names it. A run published by a service (openvibe.codes, Run, a Node worker) is <b>verified</b>; one published with a person's token is <b>submitted</b>, labelled, and never ranked by default.</p><dl class="bm-eps">
-${ep('GET', '/api/v1/benchmarks?kind=&tag=&q=&mine=1', 'The public benchmarks; <code>mine=1</code> adds your own unlisted ones.')}
-${ep('GET', '/api/v1/benchmarks/:id[@version]', 'A definition. Held-out tasks keep their prompt for the executor and the owner.')}
+<section id="br-publish"><h2>Benchmarks and runs</h2><p class="bm-fine">A benchmark is a versioned set of tasks, judges and limits; a frozen version never changes once a run names it. A run published by a service (openvibe.codes, Run, a Node worker) is <b>verified</b>; one published with a person's token is <b>submitted</b>, labelled, and never ranked by default. Your own prompts can be pasted at <a href="/benchmarks/new">/benchmarks/new</a>: 5 to 50 of them, private by default, with an unlisted share link or a public listing when you choose.</p><dl class="bm-eps">
+${ep('GET', '/api/v1/benchmarks?kind=&tag=&q=&mine=1', 'The public benchmarks; <code>mine=1</code> adds your own unlisted and private ones.')}
+${ep('GET', '/api/v1/benchmarks/:id[@version]', 'A definition. Held-out tasks keep their prompt for the executor and the owner; a private benchmark is a 404 to everyone else.')}
${ep('POST', '/api/v1/benchmarks', 'Create one (<code>bench.benchmark.write</code>); <code>PUT</code> saves a draft, <code>POST …/versions</code> freezes it, <code>POST …/fork</code> forks it.')}
${ep('POST', '/api/v1/benchmarks', 'Create one (<code>bench.benchmark.write</code>); <code>PUT</code> saves a draft, <code>POST …/versions</code> freezes it, <code>POST …/fork</code> forks it.')}
+${ep('POST', '/api/v1/benchmarks/:id/share', '<code>{ visibility: "private" | "unlisted" | "public" }</code>. The owner only. An unlisted call returns a new share token once; the old link stops working.')}
${ep('POST', '/api/v1/runs', 'Publish a run: <code>{ benchmark: "bmk_…@N", model, provider, harness: { name, version }, executor, tasks: [{ task_id, score, cost_usd, ms, tokens, artifacts }], score, cost_usd, ms, tokens }</code>.')}
${ep('POST', '/api/v1/runs', 'Publish a run: <code>{ benchmark: "bmk_…@N", model, provider, harness: { name, version }, executor, tasks: [{ task_id, score, cost_usd, ms, tokens, artifacts }], score, cost_usd, ms, tokens }</code>.')}
${ep('GET', '/api/v1/runs/:id', 'A run with its tasks; <code>/api/v1/compare?runs=a,b</code> puts up to four side by side.')}
${ep('GET', '/api/v1/runs/:id', 'A run with its tasks; <code>/api/v1/compare?runs=a,b</code> puts up to four side by side.')}
${ep('GET', '/api/v1/rankings?benchmark=&metric=score|value|speed|consistency&harness=&since=', 'Medians per model over verified runs; value is score per dollar, speed is score per minute.')}
${ep('GET', '/api/v1/rankings?benchmark=&metric=score|value|speed|consistency&harness=&since=', 'Medians per model over verified runs; value is score per dollar, speed is score per minute.')}
@@ -154,7 +154,44 @@ function metricTabs(base, current) {
@@ -154,7 +154,44 @@ function metricTabs(base, current) {
function rankingsTable(entries, metric) {
function rankingsTable(entries, metric) {
if (!entries.length) return '';
if (!entries.length) return '';
const col = { value: (e) => (e.value == null ? '—' : `${fmtScore(e.value)} / $`), speed: (e) => (e.speed == null ? '—' : `${fmtScore(e.speed)} / min`), consistency: (e) => (e.consistency == null ? '—' : `±${fmtScore(e.consistency)}`) }[metric];
const col = { value: (e) => (e.value == null ? '—' : `${fmtScore(e.value)} / $`), speed: (e) => (e.speed == null ? '—' : `${fmtScore(e.speed)} / min`), consistency: (e) => (e.consistency == null ? '—' : `±${fmtScore(e.consistency)}`) }[metric];
- return html`<div class="br-table-wrap"><table class="br-table"><thead><tr><th scope="col">#</th><th scope="col">Model</th><th scope="col">Score</th>${col ? html`<th scope="col">${METRICS.find((m) => m.id === metric).label}</th>` : ''}<th scope="col" class="br-num">Cost / run</th><th scope="col" class="br-num">Time</th><th scope="col" class="br-num">Runs</th></tr></thead><tbody>${entries.map((e, i) => html`<tr><td class="br-rank">${i + 1}</td><td><a class="br-model" href="${modelHref(e.model)}">${e.model}</a></td><td>${scoreBar(e.score)}</td>${col ? html`<td class="br-num">${col(e)}</td>` : ''}<td class="br-num">${fmtUsd(e.cost_usd)}</td><td class="br-num">${fmtMs(e.ms)}</td><td class="br-num">${e.runs}</td></tr>`)}</tbody></table></div>`;
+ const bands = metric === 'score' && entries.some((e) => e.band);
+ return html`<div class="br-table-wrap"><table class="br-table br-rank-table"><thead><tr><th scope="col">#</th>${bands ? html`<th scope="col">Band</th>` : ''}<th scope="col">Model</th><th scope="col">Score</th>${col ? html`<th scope="col">${METRICS.find((m) => m.id === metric).label}</th>` : ''}<th scope="col" class="br-num">Cost / run</th><th scope="col" class="br-num">Time</th><th scope="col" class="br-num">Runs</th></tr></thead><tbody>${entries.map((e, i) => html`<tr><td class="br-rank">${i + 1}</td>${bands ? html`<td class="br-c-band"><span class="rt-band">${e.band || '—'}</span></td>` : ''}<td class="br-c-model"><a class="br-model" href="${modelHref(e.model)}">${e.model}</a></td><td class="br-c-score">${scoreBar(e.score)}</td>${col ? html`<td class="br-num br-c-metric" data-label="${METRICS.find((m) => m.id === metric).label}">${col(e)}</td>` : ''}<td class="br-num br-c-cost" data-label="cost/run">${fmtUsd(e.cost_usd)}</td><td class="br-num br-c-time" data-label="time">${fmtMs(e.ms)}</td><td class="br-num br-c-runs" data-label="runs">${e.runs}</td></tr>`)}</tbody></table></div>`;
+}
+
+/**
+ * One leaderboard in the same `br-` table language for a person's own prompts and the public board beside it.
+ * Bands are the overlapping score intervals (2nd–4th, not a fake third place); the cost is per solved task, the
+ * rules' value metric (the median cost of an attempt divided by the success rate).
+ */
+function promptLeaderboard(entries, { emptyText = 'No runs yet.', caption = 'Models in rank bands, with their score, cost per solved task and run count' } = {}) {
+ if (!entries.length) return html`<div class="bm-empty"><p>${emptyText}</p></div>`;
+ return html`<div class="br-table-wrap"><table class="br-table rt-table br-prompt-table">
+<caption class="br-sr">${caption}</caption>
+<thead><tr><th scope="col">Band</th><th scope="col">Model</th><th scope="col">Score</th><th scope="col" class="br-num">Cost / solved task</th><th scope="col" class="br-num">Runs</th></tr></thead>
+<tbody>${entries.map((e) => html`<tr data-model="${e.model}"><td class="rt-col-band"><span class="rt-band">${e.band || '—'}</span></td><td class="rt-col-model"><a class="rt-model br-model" href="${modelHref(e.model)}">${e.model}</a></td><td class="rt-col-controlled">${scoreBar(e.score)}${e.spread ? html`<span class="br-fine br-spread">${fmtScore(e.min_score)}–${fmtScore(e.max_score)} over ${plural(e.runs, 'run')}</span>` : ''}</td><td class="rt-col-raw br-num rt-prompt-cost">${e.cost_per_solved == null ? '—' : fmtUsd(e.cost_per_solved)}</td><td class="rt-col-votes br-num">${e.runs}</td></tr>`)}</tbody></table></div>`;
+}
+
+/** The owner's leaderboard beside the public one for the same kind; the two are never mixed. */
+function promptBoards({ ownEntries = [], publicBoard = null, ownRuns = 0, visibility = 'private' }) {
+ const mineLabel = `Your prompts, ${ownRuns} ${ownRuns === 1 ? 'run' : 'runs'}`;
+ const publicEntries = (publicBoard && publicBoard.entries) || [];
+ return html`<section class="bm-section br-your-prompts" id="br-rank" aria-labelledby="br-yours-h">
+<div class="br-duo">
+<section class="br-duo-pane" aria-labelledby="br-yours-h">
+<div class="bm-head-row"><h2 id="br-yours-h">${mineLabel}</h2><span class="br-chip br-vis">${visibility}</span></div>
+<p class="br-fine">Your own tasks, your runs. This board is never mixed into the public ranks.</p>
+${ownEntries.length ? valueChart(ownEntries) : ''}
+${promptLeaderboard(ownEntries, { emptyText: 'No runs yet. Start it on 3 to 6 models; every run lands here, whatever it scores.' })}
+</section>
+<section class="br-duo-pane" aria-labelledby="br-public-h">
+<div class="bm-head-row"><h2 id="br-public-h">The public board for this kind</h2>${publicBoard && publicBoard.benchmark ? html`<a class="bm-more" href="${benchmarkHref(publicBoard.benchmark.id)}">${publicBoard.benchmark.title}</a>` : ''}</div>
+<p class="br-fine">Other people's runs of the public set, ranked the same way. Your prompts are not in it.</p>
+${publicEntries.length
+ ? promptLeaderboard(publicEntries, { emptyText: '' })
+ : empty('No public board yet', 'No public benchmark of this kind has verified runs yet, so there is nothing to stand beside. Your own board still counts your runs.')}
+</section>
+</div>
+</section>`;
}
}
// ---- runs list -----------------------------------------------------------------------------------------------
// ---- runs list -----------------------------------------------------------------------------------------------
@@ -181,28 +218,92 @@ function artifact(a, label) {
@@ -181,28 +218,92 @@ function artifact(a, label) {
// ---- pages ---------------------------------------------------------------------------------------------------
// ---- pages ---------------------------------------------------------------------------------------------------
-function benchmarksIndex({ benchmarks = [], kinds = [], kind = null, q = '' } = {}) {
+function benchmarksIndex({ benchmarks = [], kinds = [], kind = null, q = '', mine = [], signedIn = false } = {}) {
const families = [...new Set(kinds.map((k) => String(k).split('.')[0]))].sort();
const families = [...new Set(kinds.map((k) => String(k).split('.')[0]))].sort();
const chip = (label, value) => {
const chip = (label, value) => {
const on = (value || null) === (kind || null);
const on = (value || null) === (kind || null);
const href = value ? `/benchmarks?kind=${encodeURIComponent(value)}` : '/benchmarks';
const href = value ? `/benchmarks?kind=${encodeURIComponent(value)}` : '/benchmarks';
return html`<a class="br-filter" href="${href}"${on ? raw(' aria-current="page"') : ''}>${label}</a>`;
return html`<a class="br-filter" href="${href}"${on ? raw(' aria-current="page"') : ''}>${label}</a>`;
};
};
+ const own = mine.length
+ ? html`<section class="bm-section br-mine" aria-labelledby="br-mine-h"><div class="bm-head-row"><h2 id="br-mine-h">Your benchmarks</h2><span class="bm-count">${mine.length}</span></div>
+<p class="br-fine">Private ones are yours alone. An unlisted one opens with its share link; a public one is listed below.</p>
+<ul class="br-cards">${mine.map(benchmarkCard)}</ul></section>`
+ : '';
return html`<div class="bm-wrap br" id="br-benchmarks">
return html`<div class="bm-wrap br" id="br-benchmarks">
-<header class="bm-page-h"><p class="bm-kicker">Benchmarks</p><h1>Benchmarks anyone can run again</h1>
-<p class="bm-lede">Each one is a versioned set of tasks with its judges and limits. A version never changes once a run names it, so every score here can be repeated.</p></header>
+<header class="bm-page-h"><div class="bm-head-row"><div><p class="bm-kicker">Benchmarks</p><h1>Benchmarks anyone can run again</h1>
+<p class="bm-lede">Each one is a versioned set of tasks with its judges and limits. A version never changes once a run names it, so every score here can be repeated.</p></div><a class="bm-btn" href="/benchmarks/new">Make your own</a></div></header>
<form class="br-search" method="get" action="/benchmarks" role="search"><label class="br-sr" for="br-q">Search benchmarks</label><input id="br-q" name="q" type="search" value="${q}" placeholder="Search by title, tag or kind" autocomplete="off">${kind ? html`<input type="hidden" name="kind" value="${kind}">` : ''}<button class="bm-btn bm-btn-small" type="submit">Search</button></form>
<form class="br-search" method="get" action="/benchmarks" role="search"><label class="br-sr" for="br-q">Search benchmarks</label><input id="br-q" name="q" type="search" value="${q}" placeholder="Search by title, tag or kind" autocomplete="off">${kind ? html`<input type="hidden" name="kind" value="${kind}">` : ''}<button class="bm-btn bm-btn-small" type="submit">Search</button></form>
<nav class="br-filters" aria-label="Filter by family">${chip('All', null)}${families.map((f) => chip(f, f))}</nav>
<nav class="br-filters" aria-label="Filter by family">${chip('All', null)}${families.map((f) => chip(f, f))}</nav>
+${own}
+${signedIn ? '' : html`<p class="br-fine br-mine-hint">Signed in, you can paste 5 to 50 of your own prompts as a private benchmark: <a href="/benchmarks/new">make your own</a>.</p>`}
${benchmarks.length ? html`<ul class="br-cards">${benchmarks.map(benchmarkCard)}</ul>` : empty('No benchmark matches', q ? 'Try a shorter search, or clear the filter.' : 'None are published yet.', html`<a class="bm-btn bm-btn-quiet bm-btn-small" href="/benchmarks">Show all</a>`)}
${benchmarks.length ? html`<ul class="br-cards">${benchmarks.map(benchmarkCard)}</ul>` : empty('No benchmark matches', q ? 'Try a shorter search, or clear the filter.' : 'None are published yet.', html`<a class="bm-btn bm-btn-quiet bm-btn-small" href="/benchmarks">Show all</a>`)}
</div>`;
</div>`;
}
}
function benchmarkCard(b) {
function benchmarkCard(b) {
const tasks = ((b.draft && b.draft.tasks) || []).length;
const tasks = ((b.draft && b.draft.tasks) || []).length;
- return html`<li class="br-card"><a class="br-card-link" href="${benchmarkHref(b.id)}"><span class="br-card-top">${kindChip(b.kind)}<span class="br-card-v">v${b.latest_version || 0}</span></span><span class="br-card-title">${b.title}</span><span class="br-card-desc">${b.description}</span><span class="br-card-foot">${plural(tasks, 'task')}${b.runs != null ? html` · ${plural(b.runs, 'run')}` : ''}${b.parent ? ' · a fork' : ''}</span></a></li>`;
+ const vis = b.visibility && b.visibility !== 'public'
+ ? html`<span class="br-chip br-vis">${b.visibility === 'private' ? 'private' : 'unlisted'}</span>`
+ : '';
+ return html`<li class="br-card"><a class="br-card-link" href="${benchmarkHref(b.id)}"><span class="br-card-top">${kindChip(b.kind)}<span class="br-card-v">v${b.latest_version || 0}</span>${vis}</span><span class="br-card-title">${b.title}</span><span class="br-card-desc">${b.description}</span><span class="br-card-foot">${plural(tasks, 'task')}${b.runs != null ? html` · ${plural(b.runs, 'run')}` : ''}${b.parent ? ' · a fork' : ''}</span></a></li>`;
+}
+
+/** The owner's sharing controls: private, an unlisted link, or public. Plain form posts, no JavaScript. */
+function shareBox({ benchmark, shareUrl = null, error = null }) {
+ const visibility = benchmark.visibility || 'private';
+ const note = visibility === 'private'
+ ? 'Private: only you can see this benchmark and its prompts. No link works.'
+ : visibility === 'unlisted'
+ ? 'Unlisted: anyone with the link can see it. It is not in any list, search, sitemap or the open data.'
+ : 'Public: it is in the benchmark list, the sitemap and the open data, with every run published.';
+ return html`<section class="bm-section br-share" id="br-share" aria-labelledby="br-share-h">
+<div class="bm-head-row"><h2 id="br-share-h">Sharing</h2><span class="br-chip br-vis">${visibility}</span></div>
+<p class="br-fine">${note}</p>
+${error ? html`<div class="bm-note is-bad" role="alert">${error}</div>` : ''}
+${shareUrl ? html`<div class="bm-note br-share-now" role="status"><p>Your unlisted link is ready. It is shown once; copy it now. Share it again for a new link; the old one stops working.</p><label class="br-field"><span>The link</span><input class="br-share-link" type="text" readonly value="${shareUrl}" onfocus="this.select()"></label></div>` : ''}
+<form class="br-share-form" method="post" action="/benchmarks/${encodeURIComponent(benchmark.id)}/share">
+<button class="bm-btn" type="submit" name="visibility" value="unlisted">${visibility === 'unlisted' ? 'New unlisted link' : 'Share an unlisted link'}</button>
+<button class="bm-btn bm-btn-quiet" type="submit" name="visibility" value="public">Publish public</button>
+<button class="bm-btn bm-btn-quiet" type="submit" name="visibility" value="private">Make private</button>
+</form>
+<p class="br-fine">A person with the unlisted link can read the prompts and the runs. Only you can start a run on it.</p>
+</section>`;
+}
+
+/** The new-benchmark form: 5–50 pasted prompts, one per line or CSV with an optional expected answer. */
+function newBenchmarkPage({ kinds = [], values = {}, error = null, signedIn = false, loginUrl = '/auth/login' }) {
+ const head = html`<nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><span>Make your own</span></nav>
+<header class="bm-page-h"><p class="bm-kicker">Your prompts</p><h1>Make a benchmark from your own prompts</h1>
+<p class="bm-lede">Paste 5 to 50 prompts you really use. They become the tasks of a private benchmark: no model is called and nothing is charged until you run it on openvibe.codes.</p></header>`;
+ if (!signedIn) {
+ return html`<div class="bm-wrap br bm-narrow" id="br-new">${head}${empty('Sign in first', 'A benchmark of your own is attached to your account, so the prompts can be exported or deleted with you.', html`<a class="bm-btn bm-btn-small" href="${loginUrl}">Sign in with OpenVibe</a>`)}</div>`;
+ }
+ const kindList = [...kinds].sort((a, b) => (a.id === 'text.answer' ? -1 : b.id === 'text.answer' ? 1 : String(a.name || a.id).localeCompare(String(b.name || b.id))));
+ const kind = values.kind || 'text.answer';
+ const format = values.format || 'auto';
+ return html`<div class="bm-wrap br bm-narrow" id="br-new">${head}
+${error ? html`<div class="bm-note is-bad" role="alert">${error}</div>` : ''}
+<form class="br-form br-new-form" method="post" action="/benchmarks/new">
+<label class="br-field"><span>Name (optional)</span><input name="title" type="text" maxlength="160" value="${values.title || ''}" placeholder="Your prompts, e.g. the support replies I really send" autocomplete="off"></label>
+<label class="br-field"><span>Description (optional)</span><textarea name="description" rows="3" maxlength="4000" placeholder="What these prompts are for.">${values.description || ''}</textarea></label>
+<label class="br-field"><span>Kind</span><select name="kind">${kindList.map((k) => html`<option value="${k.id}"${k.id === kind ? raw(' selected') : ''}>${k.name || k.id}${k.id === 'text.answer' ? ' (a plain text answer)' : ''}</option>`)}</select></label>
+<fieldset class="br-paste"><legend>Your prompts</legend>
+<label class="br-field"><span>Format</span><select name="format">
+<option value="auto"${format === 'auto' ? raw(' selected') : ''}>Auto-detect</option>
+<option value="lines"${format === 'lines' ? raw(' selected') : ''}>One prompt per line</option>
+<option value="csv"${format === 'csv' ? raw(' selected') : ''}>CSV: prompt, expected answer (optional)</option>
+</select></label>
+<label class="br-field"><span>Paste 5 to 50 prompts</span><textarea class="br-prompts" name="prompts" rows="12" required placeholder="Write a haiku about rain.\nExplain a database index to a new developer.\nWhat is 17 × 24?">${values.prompts || ''}</textarea></label>
+<p class="br-fine">One per line, or CSV with an optional expected answer in the second column. A quoted field may hold commas. The benchmark is private by default; you can share a link or publish it after it exists.</p>
+</fieldset>
+<div class="bm-actions"><button class="bm-btn" type="submit">Make my benchmark</button><a class="bm-btn bm-btn-quiet" href="/benchmarks">Cancel</a></div>
+</form>
+<p class="br-fine">The prompts are never logged. They are exported and deleted with your account, like the rest of your data, and they are only in a public page if you publish the benchmark.</p>
+</div>`;
}
}
-function benchmarkPage({ benchmark, definition, version, rankings, metric = 'score', runs = [], codesRunUrl = null, viewer = null, baseUrl = '' }) {
+function benchmarkPage({ benchmark, definition, version, rankings, metric = 'score', runs = [], codesRunUrl = null, viewer = null, baseUrl = '', ownBoards = null, share = null, shared = false, owner = false }) {
const d = definition || {};
const d = definition || {};
const tasks = Array.isArray(d.tasks) ? d.tasks : [];
const tasks = Array.isArray(d.tasks) ? d.tasks : [];
const shown = tasks.filter((t) => !t.holdout && t.prompt);
const shown = tasks.filter((t) => !t.holdout && t.prompt);
@@ -211,21 +312,34 @@ function benchmarkPage({ benchmark, definition, version, rankings, metric = 'sco
@@ -211,21 +312,34 @@ function benchmarkPage({ benchmark, definition, version, rankings, metric = 'sco
const limits = d.limits || {};
const limits = d.limits || {};
const entries = (rankings && rankings.entries) || [];
const entries = (rankings && rankings.entries) || [];
const base = benchmarkHref(benchmark.id);
const base = benchmarkHref(benchmark.id);
- const runYourself = codesRunUrl
- ? html`<a class="bm-btn" href="${benchmarkHref(benchmark.id)}/run">Run it yourself</a>`
- : html`<a class="bm-btn bm-btn-quiet" href="/docs#br-publish">Publish a run</a>`;
+ const canRun = Boolean(codesRunUrl) && !shared;
+ const runYourself = shared
+ ? ''
+ : canRun
+ ? html`<a class="bm-btn" href="${benchmarkHref(benchmark.id)}/run">Run it yourself</a>`
+ : html`<a class="bm-btn bm-btn-quiet" href="/docs#br-publish">Publish a run</a>`;
+ const byline = owner ? 'by you' : shared ? 'by its owner' : `by ${benchmark.owner}`;
+ const privateNote = benchmark.visibility === 'private'
+ ? html`<div class="bm-note br-private" role="status"><b>Private.</b> Only you can see this benchmark and its prompts. It is not in any list, search or the open data.</div>`
+ : shared
+ ? html`<div class="bm-note br-shared" role="status"><b>Shared by its owner.</b> This page is not in any list or search engine. The link can stop working if the owner makes it private.</div>`
+ : '';
return html`<div class="bm-wrap br" id="br-benchmark">
return html`<div class="bm-wrap br" id="br-benchmark">
<nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><span>${benchmark.title}</span></nav>
<nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><span>${benchmark.title}</span></nav>
-<header class="br-head"><div class="br-head-main"><p class="br-head-meta">${kindChip(benchmark.kind)}<span>v${version || benchmark.latest_version || 0}</span><span>by ${benchmark.owner}</span>${benchmark.parent ? html`<a href="${benchmarkHref(benchmark.parent)}">forked</a>` : ''}</p>
+<header class="br-head"><div class="br-head-main"><p class="br-head-meta">${kindChip(benchmark.kind)}<span>v${version || benchmark.latest_version || 0}</span><span>${byline}</span>${benchmark.visibility !== 'public' ? html`<span class="br-chip br-vis">${benchmark.visibility}</span>` : ''}${benchmark.parent ? html`<a href="${benchmarkHref(benchmark.parent)}">forked</a>` : ''}</p>
<h1>${benchmark.title}</h1>${benchmark.description ? html`<p class="bm-lede">${benchmark.description}</p>` : ''}</div>
<h1>${benchmark.title}</h1>${benchmark.description ? html`<p class="bm-lede">${benchmark.description}</p>` : ''}</div>
-<div class="br-head-act">${runYourself}<p class="br-fine">${codesRunUrl ? 'On openvibe.codes, with your own keys. The result is published here.' : 'Running it on openvibe.codes with your own keys is coming. Today a run is published through the API.'}</p></div></header>
+<div class="br-head-act">${runYourself}<p class="br-fine">${canRun ? 'On openvibe.codes, with your own keys or the free pool through OpenVibe.AI. The result is published here.' : codesRunUrl ? 'Only the owner starts runs on this benchmark. Its results are readable here.' : 'Running it on openvibe.codes with your own keys is coming. Today a run is published through the API.'}</p></div></header>
+${privateNote}
<dl class="br-facts"><div><dt>Tasks</dt><dd>${tasks.length}${hidden ? html` <span class="br-fine">(${hidden} hidden)</span>` : ''}</dd></div><div><dt>Judged by</dt><dd>${judges.length ? judges.map((j) => String(j.type).replace(/_/g, ' ')).join(', ') : '—'}</dd></div><div><dt>Limits</dt><dd>${[limits.ms ? fmtMs(limits.ms) : null, limits.tokens ? `${fmtTokens(limits.tokens)} tokens` : null, limits.usd ? fmtUsd(limits.usd) : null].filter(Boolean).join(' · ') || '—'}</dd></div><div><dt>Verified runs</dt><dd>${entries.reduce((n, e) => n + e.runs, 0)}</dd></div></dl>
<dl class="br-facts"><div><dt>Tasks</dt><dd>${tasks.length}${hidden ? html` <span class="br-fine">(${hidden} hidden)</span>` : ''}</dd></div><div><dt>Judged by</dt><dd>${judges.length ? judges.map((j) => String(j.type).replace(/_/g, ' ')).join(', ') : '—'}</dd></div><div><dt>Limits</dt><dd>${[limits.ms ? fmtMs(limits.ms) : null, limits.tokens ? `${fmtTokens(limits.tokens)} tokens` : null, limits.usd ? fmtUsd(limits.usd) : null].filter(Boolean).join(' · ') || '—'}</dd></div><div><dt>Verified runs</dt><dd>${entries.reduce((n, e) => n + e.runs, 0)}</dd></div></dl>
-<section class="bm-section" id="br-rank" aria-labelledby="br-rank-h"><div class="bm-head-row"><h2 id="br-rank-h">Rankings</h2>${metricTabs(base, metric)}</div>
+${owner ? shareBox({ benchmark, shareUrl: share && share.shareUrl, error: share && share.error }) : ''}
+${ownBoards
+ ? promptBoards(ownBoards)
+ : html`<section class="bm-section" id="br-rank" aria-labelledby="br-rank-h"><div class="bm-head-row"><h2 id="br-rank-h">Rankings</h2>${metricTabs(base, metric)}</div>
${valueChart(entries)}
${valueChart(entries)}
${entries.length ? rankingsTable(entries, metric) : empty('No verified runs yet', 'A run counts here once openvibe.codes, Run or a Node worker ran it and published it. Submitted runs are listed below but never ranked.')}
${entries.length ? rankingsTable(entries, metric) : empty('No verified runs yet', 'A run counts here once openvibe.codes, Run or a Node worker ran it and published it. Submitted runs are listed below but never ranked.')}
-</section>
+</section>`}
<section class="bm-section" aria-labelledby="br-tasks-h"><div class="bm-head-row"><h2 id="br-tasks-h">Tasks</h2><span class="bm-count">${tasks.length}</span></div>
<section class="bm-section" aria-labelledby="br-tasks-h"><div class="bm-head-row"><h2 id="br-tasks-h">Tasks</h2><span class="bm-count">${tasks.length}</span></div>
-<ol class="br-tasks">${shown.map((t) => html`<li class="br-task"><span class="br-task-id br-mono">${t.id}</span><p>${t.prompt}</p></li>`)}${hidden ? html`<li class="br-task is-hidden"><span class="br-task-id br-mono">${plural(hidden, 'task')}</span><p>Held out: only the executor sees ${hidden === 1 ? 'it' : 'them'}, so nobody can tune for the answer.</p></li>` : ''}</ol></section>
+<ol class="br-tasks">${shown.map((t) => html`<li class="br-task"><span class="br-task-id br-mono">${t.id}</span><p>${t.prompt}</p>${t.expected && t.expected.answer ? html`<p class="br-fine">Expected: ${String(t.expected.answer).slice(0, 300)}</p>` : ''}</li>`)}${hidden ? html`<li class="br-task is-hidden"><span class="br-task-id br-mono">${plural(hidden, 'task')}</span><p>Held out: only the executor sees ${hidden === 1 ? 'it' : 'them'}, so nobody can tune for the answer.</p></li>` : ''}</ol></section>
<section class="bm-section" aria-labelledby="br-latest-h"><div class="bm-head-row"><h2 id="br-latest-h">Latest runs</h2>${runs.length > 1 ? html`<a class="bm-more" href="${compareHref(runs.slice(0, 2).map((r) => r.id))}">Compare the newest two</a>` : ''}</div>
<section class="bm-section" aria-labelledby="br-latest-h"><div class="bm-head-row"><h2 id="br-latest-h">Latest runs</h2>${runs.length > 1 ? html`<a class="bm-more" href="${compareHref(runs.slice(0, 2).map((r) => r.id))}">Compare the newest two</a>` : ''}</div>
${runs.length ? runsList(runs) : empty('No runs yet', 'The first run shows up here, verified or submitted.')}</section>
${runs.length ? runsList(runs) : empty('No runs yet', 'The first run shows up here, verified or submitted.')}</section>
${benchmarkUseThis({ baseUrl, benchmark, entries, viewer })}
${benchmarkUseThis({ baseUrl, benchmark, entries, viewer })}
@@ -250,17 +364,17 @@ function runForm({ benchmark, definition, version, signedIn, harnesses = [], sug
@@ -250,17 +364,17 @@ function runForm({ benchmark, definition, version, signedIn, harnesses = [], sug
for (const id of p.harnesses) if (!harnessList.some((h) => h.id === id)) harnessList.push({ id, label: id });
for (const id of p.harnesses) if (!harnessList.some((h) => h.id === id)) harnessList.push({ id, label: id });
const base = benchmarkHref(benchmark.id);
const base = benchmarkHref(benchmark.id);
const head = html`<nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><a href="${base}">${benchmark.title}</a><span aria-hidden="true">/</span><span>Run it</span></nav>
const head = html`<nav class="br-crumbs" aria-label="Breadcrumb"><a href="/benchmarks">Benchmarks</a><span aria-hidden="true">/</span><a href="${base}">${benchmark.title}</a><span aria-hidden="true">/</span><span>Run it</span></nav>
-<header class="bm-page-h"><p class="bm-kicker">Run it yourself</p><h1>${benchmark.title}</h1><p class="bm-lede">openvibe.codes runs every task on each model and harness you pick, with your own keys, and publishes each run here as verified, whatever it scores.</p></header>`;
+<header class="bm-page-h"><p class="bm-kicker">Run it yourself</p><h1>${benchmark.title}</h1><p class="bm-lede">Pick 3 to 6 models. openvibe.codes runs every task on each model and harness, with your own keys through OpenVibe.AI or the free pool, and publishes each run here as verified, whatever it scores.</p></header>`;
if (!signedIn) {
if (!signedIn) {
- return html`<div class="bm-wrap br bm-narrow" id="br-run-form">${head}${empty('Sign in first', 'A run uses your own provider keys on openvibe.codes, so it starts as you.', html`<a class="bm-btn bm-btn-small" href="${loginUrl}">Sign in with OpenVibe</a>`)}</div>`;
+ return html`<div class="bm-wrap br bm-narrow" id="br-run-form">${head}${empty('Sign in first', 'A run uses your own provider keys, or the free pool, through OpenVibe.AI on openvibe.codes, so it starts as you.', html`<a class="bm-btn bm-btn-small" href="${loginUrl}">Sign in with OpenVibe</a>`)}</div>`;
}
}
const check = (name, value, label, on) => html`<label class="br-check"><input type="checkbox" name="${name}" value="${value}"${on ? raw(' checked') : ''}><span>${label}</span></label>`;
const check = (name, value, label, on) => html`<label class="br-check"><input type="checkbox" name="${name}" value="${value}"${on ? raw(' checked') : ''}><span>${label}</span></label>`;
return html`<div class="bm-wrap br bm-narrow" id="br-run-form">${head}
return html`<div class="bm-wrap br bm-narrow" id="br-run-form">${head}
${error ? html`<div class="bm-note is-bad" role="alert">${error}</div>` : ''}
${error ? html`<div class="bm-note is-bad" role="alert">${error}</div>` : ''}
-${confirm === 'unknown' ? html`<div class="bm-note" role="status">This benchmark sets no dollar limit per task, so what it costs depends on the models you pick. It runs on your own keys. Check the choices below, then confirm.</div>` : confirm != null ? html`<div class="bm-note" role="status">This run may cost up to <b>${fmtUsd(confirm)}</b> on your own keys. Check the choices below, then confirm.</div>` : ''}
+${confirm === 'unknown' ? html`<div class="bm-note" role="status">This benchmark sets no dollar limit per task, so what it costs depends on the models you pick. It runs on your own keys or the free pool, through OpenVibe.AI. Check the choices below, then confirm.</div>` : confirm != null ? html`<div class="bm-note" role="status">This run may cost up to <b>${fmtUsd(confirm)}</b> on your own keys. Check the choices below, then confirm.</div>` : ''}
<form class="br-form" method="post" action="${base}/run" data-tasks="${tasks}" data-usd="${usd}">
<form class="br-form" method="post" action="${base}/run" data-tasks="${tasks}" data-usd="${usd}">
<fieldset><legend>Models</legend>${modelList.length ? html`<div class="br-checks">${modelList.map((m) => check('models', m, m, p.models.includes(m)))}</div>` : ''}
<fieldset><legend>Models</legend>${modelList.length ? html`<div class="br-checks">${modelList.map((m) => check('models', m, m, p.models.includes(m)))}</div>` : ''}
-<label class="br-field"><span>${modelList.length ? 'More models, separated by commas' : 'Models, separated by commas'}</span><input name="more_models" type="text" autocomplete="off" placeholder="deepseek-v4.1-flash, kimi-k3"></label></fieldset>
+<label class="br-field"><span>${modelList.length ? 'More models, separated by commas' : 'Models, separated by commas (3 to 6)'}</span><input name="more_models" type="text" autocomplete="off" placeholder="deepseek-v4.1-flash, kimi-k3"></label></fieldset>
<fieldset><legend>Harnesses</legend><div class="br-checks">${harnessList.map((h) => check('harnesses', h.id, h.label, p.harnesses.includes(h.id)))}</div></fieldset>
<fieldset><legend>Harnesses</legend><div class="br-checks">${harnessList.map((h) => check('harnesses', h.id, h.label, p.harnesses.includes(h.id)))}</div></fieldset>
<label class="br-field"><span>Runs of each</span><select name="repeat">${[1, 2, 3, 5].map((n) => html`<option value="${n}"${n === Number(p.repeat) ? raw(' selected') : ''}>${n === 1 ? 'Once' : `${n} times (shows how steady it is)`}</option>`)}</select></label>
<label class="br-field"><span>Runs of each</span><select name="repeat">${[1, 2, 3, 5].map((n) => html`<option value="${n}"${n === Number(p.repeat) ? raw(' selected') : ''}>${n === 1 ? 'Once' : `${n} times (shows how steady it is)`}</option>`)}</select></label>
<p class="br-fine" data-estimate>${usd > 0 ? `${tasks} ${tasks === 1 ? 'task' : 'tasks'} × your picks × runs, at up to ${fmtUsd(usd)} a task. Over $1 asks you once more.` : `${tasks} ${tasks === 1 ? 'task' : 'tasks'} × your picks × runs. This benchmark sets no dollar limit per task, so you confirm before it starts.`}</p>
<p class="br-fine" data-estimate>${usd > 0 ? `${tasks} ${tasks === 1 ? 'task' : 'tasks'} × your picks × runs, at up to ${fmtUsd(usd)} a task. Over $1 asks you once more.` : `${tasks} ${tasks === 1 ? 'task' : 'tasks'} × your picks × runs. This benchmark sets no dollar limit per task, so you confirm before it starts.`}</p>
@@ -337,6 +451,7 @@ ${modelUseThis({ baseUrl, model, rows, viewer })}
@@ -337,6 +451,7 @@ ${modelUseThis({ baseUrl, model, rows, viewer })}
}
}
module.exports = {
module.exports = {
- benchmarksIndex, benchmarkPage, runForm, runPage, comparePage, modelPage, valueChart, METRICS,
+ benchmarksIndex, benchmarkPage, newBenchmarkPage, promptLeaderboard, promptBoards, shareBox,
+ runForm, runPage, comparePage, modelPage, valueChart, METRICS,
fmtScore, fmtUsd, fmtMs, fmtTokens, ordinal, benchmarkCard, render,
fmtScore, fmtUsd, fmtMs, fmtTokens, ordinal, benchmarkCard, render,
};
};
@@ -28,6 +28,8 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
@@ -28,6 +28,8 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
let judgementId;
let judgementId;
let sharedBenchmark;
let sharedBenchmark;
let lonelyBenchmark;
let lonelyBenchmark;
+ let privateBenchmark;
+ let sharedPromptBenchmark;
const signedEvent = (event) => {
const signedEvent = (event) => {
const raw = JSON.stringify({ event });
const raw = JSON.stringify({ event });
@@ -73,6 +75,22 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
@@ -73,6 +75,22 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
await db.exec(`INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)
await db.exec(`INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)
VALUES ($1, 1, '{"kind":"design.page"}', $2, $3, now())`, [id, 'a'.repeat(64), owner]);
VALUES ($1, 1, '{"kind":"design.page"}', $2, $3, now())`, [id, 'a'.repeat(64), owner]);
}
}
+ // Her own prompts: a private benchmark with a service-run on it. Both go with her.
+ privateBenchmark = ulid('bmk', 'AAC');
+ await db.exec(`INSERT INTO bench_benchmarks (id, owner, title, kind, visibility, share_token_hash, created_by, created_at, updated_at)
+ VALUES ($1, $2, 'Private prompts', 'text.answer', 'private', $3, $2, now(), now())`, [privateBenchmark, alice.subject, 'b'.repeat(64)]);
+ await db.exec(`INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)
+ VALUES ($1, 1, $2::jsonb, $3, $4, now())`, [privateBenchmark, JSON.stringify({ kind: 'text.answer', tasks: [{ id: 't1', prompt: 'A very private prompt ZEBRA', holdout: false }] }), 'c'.repeat(64), alice.subject]);
+ await db.exec(`INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, provider, provenance, created_by, created_at)
+ VALUES ($1, $2, 1, 'model-private', 'acme', 'verified', 'svc:private-exec', now())`, [ulid('brn', 'AAC'), privateBenchmark]);
+ // A prompt benchmark she shared: it still goes with her, because the prompts are hers.
+ sharedPromptBenchmark = ulid('bmk', 'AAD');
+ await db.exec(`INSERT INTO bench_benchmarks (id, owner, title, kind, visibility, tags, share_token_hash, created_by, created_at, updated_at)
+ VALUES ($1, $2, 'Shared prompts', 'text.answer', 'unlisted', '["your-prompts"]'::jsonb, $3, $2, now(), now())`, [sharedPromptBenchmark, alice.subject, 'd'.repeat(64)]);
+ await db.exec(`INSERT INTO bench_benchmark_versions (benchmark_id, version, definition, hash, created_by, created_at)
+ VALUES ($1, 1, $2::jsonb, $3, $4, now())`, [sharedPromptBenchmark, JSON.stringify({ kind: 'text.answer', tasks: [{ id: 't1', prompt: 'A shared private prompt ZEBRA', holdout: false }] }), 'e'.repeat(64), alice.subject]);
+ await db.exec(`INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, provider, provenance, created_by, created_at)
+ VALUES ($1, $2, 1, 'model-shared', 'acme', 'verified', $3, now())`, [ulid('brn', 'AAD'), sharedPromptBenchmark, carol.subject]);
// Her run on the benchmark, and Carol's run on the same one.
// Her run on the benchmark, and Carol's run on the same one.
await db.exec(`INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, provider, provenance, created_by, created_at)
await db.exec(`INSERT INTO bench_runs (id, benchmark_id, benchmark_version, model, provider, provenance, created_by, created_at)
VALUES ($1, $2, 1, 'model-a', 'openai', 'submitted', $3, now())`, [ulid('brn', 'AAA'), sharedBenchmark, alice.subject]);
VALUES ($1, $2, 1, 'model-a', 'openai', 'submitted', $3, now())`, [ulid('brn', 'AAA'), sharedBenchmark, alice.subject]);
@@ -101,9 +119,15 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
@@ -101,9 +119,15 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
assert.ok(part.body.files.some((f) => f.name === 'bench_runs.json' && f.content.length === 1));
assert.ok(part.body.files.some((f) => f.name === 'bench_runs.json' && f.content.length === 1));
assert.ok(part.body.files.some((f) => f.name === 'bench_battle_seen.json' && f.content.length === 1));
assert.ok(part.body.files.some((f) => f.name === 'bench_battle_seen.json' && f.content.length === 1));
const benchmarks = part.body.files.find((f) => f.name === 'benchmarks.json');
const benchmarks = part.body.files.find((f) => f.name === 'benchmarks.json');
- assert.ok(benchmarks && benchmarks.content.length === 2, 'her benchmarks are in the part');
+ assert.ok(benchmarks && benchmarks.content.length === 4, 'her benchmarks are in the part');
const versions = part.body.files.find((f) => f.name === 'benchmark_versions.json');
const versions = part.body.files.find((f) => f.name === 'benchmark_versions.json');
- assert.ok(versions && versions.content.length === 2, 'the versions of her benchmarks are in the part');
+ assert.ok(versions && versions.content.length === 4, 'the versions of her benchmarks are in the part');
+ assert.ok(JSON.stringify(versions).includes('A very private prompt ZEBRA'), 'her private prompts are in her export');
+ assert.ok(JSON.stringify(versions).includes('A shared private prompt ZEBRA'), 'her shared prompts are in her export');
+ const privateRuns = part.body.files.find((f) => f.name === 'benchmark_runs.json');
+ assert.ok(privateRuns && privateRuns.content.length === 2, 'the runs on her prompt benchmarks are in her export');
+ assert.ok(privateRuns.content.some((r) => r.model === 'model-private'));
+ assert.ok(privateRuns.content.some((r) => r.model === 'model-shared'));
assert.ok(!JSON.stringify(part.body).includes(carol.subject), 'nobody else\'s rows');
assert.ok(!JSON.stringify(part.body).includes(carol.subject), 'nobody else\'s rows');
});
});
@@ -117,8 +141,8 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
@@ -117,8 +141,8 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
assert.strictEqual(confirm.body.subject, alice.subject);
assert.strictEqual(confirm.body.subject, alice.subject);
assert.strictEqual(confirm.body.erased.bench_tasks, 1);
assert.strictEqual(confirm.body.erased.bench_tasks, 1);
assert.strictEqual(confirm.body.erased.bench_rounds, 1);
assert.strictEqual(confirm.body.erased.bench_rounds, 1);
- assert.strictEqual(confirm.body.erased.bench_runs, 1, 'her run is gone');
- assert.strictEqual(confirm.body.erased.bench_benchmarks, 1, 'the benchmark nobody else ran is gone');
+ assert.strictEqual(confirm.body.erased.bench_runs, 3, 'her run and the runs on her prompt benchmarks are gone');
+ assert.strictEqual(confirm.body.erased.bench_benchmarks, 3, 'the benchmark nobody else ran and both prompt benchmarks are gone');
assert.strictEqual(confirm.body.erased.bench_battle_seen, 1, 'the pairs she had seen are gone');
assert.strictEqual(confirm.body.erased.bench_battle_seen, 1, 'the pairs she had seen are gone');
// Three rows stay without her: her judgement, the benchmark other people ran and its frozen version.
// Three rows stay without her: her judgement, the benchmark other people ran and its frozen version.
assert.strictEqual(confirm.body.retained.tombstones, 3, 'her judgement, the benchmark others ran and its version stay');
assert.strictEqual(confirm.body.retained.tombstones, 3, 'her judgement, the benchmark others ran and its version stay');
@@ -135,6 +159,12 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
@@ -135,6 +159,12 @@ const ulid = (prefix, tail = 'AAA') => `${prefix}_01JZ${'0'.repeat(19)}${tail}`;
assert.strictEqual(kept.created_by, 'deleted');
assert.strictEqual(kept.created_by, 'deleted');
assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [sharedBenchmark])), 1, 'Carol\'s run is still there');
assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [sharedBenchmark])), 1, 'Carol\'s run is still there');
assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmarks WHERE id = $1', [lonelyBenchmark])), 0, 'the benchmark nobody ran is gone');
assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmarks WHERE id = $1', [lonelyBenchmark])), 0, 'the benchmark nobody ran is gone');
+ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmarks WHERE id = $1', [privateBenchmark])), 0, 'the private benchmark is gone');
+ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmark_versions WHERE benchmark_id = $1', [privateBenchmark])), 0, 'its prompts go with it');
+ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [privateBenchmark])), 0, 'the run on it is gone too');
+ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmarks WHERE id = $1', [sharedPromptBenchmark])), 0, 'a shared prompt benchmark goes too');
+ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_benchmark_versions WHERE benchmark_id = $1', [sharedPromptBenchmark])), 0, 'its shared prompts go with it');
+ assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_runs WHERE benchmark_id = $1', [sharedPromptBenchmark])), 0, 'its run goes too');
if (await db.value("SELECT to_regclass('bench_watches') IS NOT NULL")) {
if (await db.value("SELECT to_regclass('bench_watches') IS NOT NULL")) {
assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_watches WHERE subject = $1', [alice.subject])), 0, 'her watch is gone');
assert.strictEqual(Number(await db.value('SELECT count(*)::int FROM bench_watches WHERE subject = $1', [alice.subject])), 0, 'her watch is gone');
}
}
@@ -17,7 +17,7 @@ const { boot, check, done } = require('./helpers/boot');
@@ -17,7 +17,7 @@ const { boot, check, done } = require('./helpers/boot');
await check('kinds are seeded from code (the 15 of §4.12 plus the nine showcase kinds)', async () => {
await check('kinds are seeded from code (the 15 of §4.12 plus the nine showcase kinds)', async () => {
const r = await t.get('/api/v1/kinds');
const r = await t.get('/api/v1/kinds');
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
- assert.strictEqual(r.json().kinds.length, 24);
+ assert.strictEqual(r.json().kinds.length, 25);
const ids = r.json().kinds.map((k) => k.id);
const ids = r.json().kinds.map((k) => k.id);
for (const want of ['design.page', 'games.match', 'meme.caption', 'vision.answer', 'codes.patch', 'speech.tts', 'speech.asr', 'translate.ja', 'search.answer']) {
for (const want of ['design.page', 'games.match', 'meme.caption', 'vision.answer', 'codes.patch', 'speech.tts', 'speech.asr', 'translate.ja', 'search.answer']) {
assert.ok(ids.includes(want), `missing kind ${want}`);
assert.ok(ids.includes(want), `missing kind ${want}`);
@@ -58,7 +58,7 @@ const rpc = (method, params, id = 1) => ({ jsonrpc: '2.0', id, method, ...(param
@@ -58,7 +58,7 @@ const rpc = (method, params, id = 1) => ({ jsonrpc: '2.0', id, method, ...(param
await check('bench.list_kinds lists every kind Bench runs', async () => {
await check('bench.list_kinds lists every kind Bench runs', async () => {
const r = await t.get('/mcp', { json: rpc('tools/call', { name: 'bench.list_kinds', arguments: {} }, 8) });
const r = await t.get('/mcp', { json: rpc('tools/call', { name: 'bench.list_kinds', arguments: {} }, 8) });
assert.strictEqual(r.json().result.isError, false);
assert.strictEqual(r.json().result.isError, false);
- assert.strictEqual(r.json().result.structuredContent.kinds.length, 24);
+ assert.strictEqual(r.json().result.structuredContent.kinds.length, 25);
});
});
await check('bench.get_board renders a board without a person', async () => {
await check('bench.get_board renders a board without a person', async () => {
@@ -28,7 +28,7 @@ const HOSTILE = [
@@ -28,7 +28,7 @@ const HOSTILE = [
const r = await t.get('/api/v1/kinds');
const r = await t.get('/api/v1/kinds');
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
const byId = new Map(r.json().kinds.map((k) => [k.id, k]));
const byId = new Map(r.json().kinds.map((k) => [k.id, k]));
- assert.strictEqual(r.json().kinds.length, 24);
+ assert.strictEqual(r.json().kinds.length, 25);
const want = ['games.play', 'games.three', 'design.svg', 'design.motion', 'design.ui', 'design.dataviz', 'design.model3d', 'design.shader', 'design.interface'];
const want = ['games.play', 'games.three', 'design.svg', 'design.motion', 'design.ui', 'design.dataviz', 'design.model3d', 'design.shader', 'design.interface'];
for (const id of want) {
for (const id of want) {
const k = byId.get(id);
const k = byId.get(id);
new file mode 100644
new file mode 100644
@@ -0,0 +1,298 @@
@@ -0,0 +1,298 @@
+'use strict';
+
+/**
+ * B11: a person's own prompts as a private benchmark.
+ *
+ * Under test: the paste parser (lines, CSV with an optional expected answer, 5–50 bounds); a private benchmark by
+ * default; another person gets a 404 and nothing private is listed, in the sitemap or in the open data; the owner's
+ * leaderboard sits beside the public board for the same kind (bands and cost per solved task, never mixed); the run
+ * goes through the existing launcher and still asks before it starts; an unlisted share link with a long random
+ * token, its "shared by its owner" page, noindex, and unlisting/unsharing/public/private; CSRF.
+ */
+const assert = require('assert');
+const http = require('http');
+const { boot, check, done } = require('./helpers/boot');
+const { parsePrompts, MAX_PROMPTS } = require('../server/bench/your-prompts');
+const { shareTokenMatches } = require('../server/bench/registry');
+
+function fakeCodes() {
+ const calls = [];
+ const server = http.createServer((req, res) => {
+ let body = '';
+ req.on('data', (c) => { body += c; });
+ req.on('end', () => {
+ calls.push({ method: req.method, url: req.url, headers: req.headers, body: body ? JSON.parse(body) : null });
+ res.setHeader('content-type', 'application/json');
+ if (req.url === '/api/v1/harnesses') return res.end(JSON.stringify({ harnesses: [{ id: 'codex', label: 'Codex' }, { id: 'claude-code', label: 'Claude Code' }] }));
+ if (/^\/api\/v1\/benchmarks\/[^/]+\/run$/.test(req.url) && req.method === 'POST') {
+ res.statusCode = 202;
+ return res.end(JSON.stringify({ session: 'ses_b11', tasks: [{ id: 'tsk_b11' }], task: { id: 'tsk_b11' }, total: 1 }));
+ }
+ res.statusCode = 404; res.end('{}');
+ });
+ });
+ return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ url: `http://127.0.0.1:${server.address().port}`, calls, close: () => server.close() })));
+}
+
+(async () => {
+ const codes = await fakeCodes();
+ const t = await boot({ env: { CODES_BENCH_RUN_URL: codes.url } });
+ const alice = t.network.addUser('alice');
+ const bob = t.network.addUser('bob');
+ const publisher = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.run.publish', 'bench.benchmark.write'] });
+ const base = String(t.config.baseUrl).replace(/\/+$/, '');
+ const MARK = 'ZEBRA-B11-PROMPT';
+
+ const form = (o) => new URLSearchParams(o).toString();
+ const post = (path, o, opt = {}) => t.get(path, { as: alice, raw: form(o), headers: { 'content-type': 'application/x-www-form-urlencoded' }, ...opt });
+ const share = (id, visibility) => post(`/benchmarks/${id}/share`, { visibility });
+ const tokenOf = (res) => {
+ const m = /\/b\/([A-Za-z0-9_-]{20,120})/.exec(res.text);
+ assert.ok(m, `a share token in the page: ${res.text.slice(0, 300)}`);
+ return m[1];
+ };
+ const publish = async (ref, model, scores, { cost = 0.03 } = {}) => {
+ const tasks = scores.map((score, i) => ({ task_id: `t${i + 1}`, score, cost_usd: cost / scores.length, ms: 1000 + i, tokens: 100, artifacts: [] }));
+ const body = {
+ benchmark: ref, model, route: { provider: 'acme' }, provider: 'acme',
+ harness: { name: 'codex', version: '1.0' }, settings: {}, executor: { kind: 'codes', id: 'tsk_b11' },
+ tasks, score: scores.reduce((a, b) => a + b, 0) / scores.length, cost_usd: cost, ms: 2000, tokens: 600,
+ };
+ const res = await t.get('/api/v1/runs', { bearer: publisher, json: body });
+ assert.strictEqual(res.status, 201, res.text);
+ return res.json().run.id;
+ };
+
+ let id;
+ let version;
+ let token;
+ const prompts = Array.from({ length: 6 }, (_, i) => `${MARK} number ${i + 1}`);
+
+ await check('the parser takes one prompt per line and enforces 5–50', async () => {
+ const good = parsePrompts(prompts.join('\n'), { format: 'lines' });
+ assert.strictEqual(good.ok, true, good.detail);
+ assert.strictEqual(good.prompts.length, 6);
+ assert.ok(good.prompts.every((p) => p.expected === null));
+ assert.strictEqual(parsePrompts(prompts.slice(0, 4).join('\n')).ok, false, 'four is too few');
+ assert.strictEqual(parsePrompts(Array.from({ length: MAX_PROMPTS + 1 }, (_, i) => `p${i}`).join('\n')).ok, false, 'fifty-one is too many');
+ });
+
+ await check('the parser reads CSV with an optional expected answer and quoted commas', async () => {
+ const csv = 'prompt,expected\n"What is 2 + 2, exactly?","4"\n"Say hello, in French.",bonjour\nName one primary colour.,red\nWhat is 3 × 3?,9\nFinish the word: sun—,flower\n';
+ const parsed = parsePrompts(csv);
+ assert.strictEqual(parsed.ok, true, parsed.detail);
+ assert.strictEqual(parsed.format, 'csv', 'a prompt/expected header is detected');
+ assert.strictEqual(parsed.prompts.length, 5);
+ assert.strictEqual(parsed.prompts[0].prompt, 'What is 2 + 2, exactly?');
+ assert.strictEqual(parsed.prompts[0].expected, '4');
+ assert.strictEqual(parsed.prompts[1].prompt, 'Say hello, in French.');
+ assert.strictEqual(parsed.prompts[1].expected, 'bonjour');
+ const noHeader = parsePrompts('one,two\nthree,four\nfive,six\nseven,eight\nnine,ten', { format: 'csv' });
+ assert.strictEqual(noHeader.ok, true, noHeader.detail);
+ assert.deepStrictEqual(noHeader.prompts[0], { prompt: 'one', expected: 'two' });
+ });
+
+ await check('a share token is stored as a hash and checked in constant time', async () => {
+ assert.strictEqual(shareTokenMatches('abc', 'a'.repeat(64)), false);
+ assert.strictEqual(shareTokenMatches('abc', null), false);
+ });
+
+ await check('the form makes a private benchmark from pasted prompts', async () => {
+ const res = await post('/benchmarks/new', {
+ title: 'My private prompts', description: 'Six real prompts.',
+ kind: 'text.answer', format: 'lines', prompts: prompts.join('\n'),
+ });
+ assert.strictEqual(res.status, 303, res.text.slice(0, 400));
+ id = res.headers.get('location').split('/').pop();
+ const r = await t.get(`/api/v1/benchmarks/${id}`, { as: alice });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.json().benchmark.visibility, 'private', 'private by default');
+ assert.strictEqual(r.json().benchmark.title, 'My private prompts');
+ assert.strictEqual(r.json().benchmark.latest_version, 1, 'frozen into a version');
+ version = r.json().version.version;
+ assert.strictEqual(r.json().definition.kind, 'text.answer');
+ assert.strictEqual(r.json().definition.tasks.length, 6);
+ assert.deepStrictEqual(r.json().definition.tasks.map((x) => x.id), ['t1', 't2', 't3', 't4', 't5', 't6']);
+ assert.ok(r.json().definition.tasks.every((x) => x.prompt.startsWith(MARK)));
+ assert.strictEqual(r.json().definition.tasks[0].expected, null);
+ assert.strictEqual(r.json().definition.limits.usd, 0, 'no dollar cap is added or raised');
+ assert.ok(!t.logs().includes(MARK), 'a prompt never reaches the log');
+ });
+
+ await check('a CSV paste keeps the expected answer of each row', async () => {
+ const csv = 'prompt,expected\nQ1,A1\nQ2,A2\nQ3,A3\nQ4,A4\nQ5,A5\n';
+ const res = await post('/benchmarks/new', { title: 'CSV prompts', kind: 'design.svg', format: 'csv', prompts: csv });
+ assert.strictEqual(res.status, 303, res.text.slice(0, 400));
+ const cid = res.headers.get('location').split('/').pop();
+ const r = await t.get(`/api/v1/benchmarks/${cid}`, { as: alice });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.json().definition.kind, 'design.svg', 'an existing kind can be chosen');
+ assert.deepStrictEqual(r.json().definition.tasks[0].expected, { answer: 'A1' });
+ assert.strictEqual(r.json().definition.tasks.length, 5);
+ });
+
+ await check('another person gets a 404 for the private benchmark everywhere', async () => {
+ for (const [path, what] of [
+ [`/benchmarks/${id}`, 'page'],
+ [`/benchmarks/${id}/run`, 'run form'],
+ [`/api/v1/benchmarks/${id}`, 'definition'],
+ [`/api/v1/rankings?benchmark=${id}`, 'rankings'],
+ [`/api/v1/percentile?benchmark=${id}&model=a&score=90`, 'percentile'],
+ ]) {
+ const res = await t.get(path, { as: bob });
+ assert.strictEqual(res.status, 404, `${what}: ${res.status} ${res.text.slice(0, 200)}`);
+ }
+ const run = await post(`/benchmarks/${id}/run`, { models: 'a', harnesses: 'codex', repeat: '1' }, { as: bob });
+ assert.strictEqual(run.status, 404, run.text.slice(0, 200));
+ });
+
+ await check('a private benchmark is never in a list, the sitemap or the open data', async () => {
+ const list = await t.get('/api/v1/benchmarks');
+ assert.ok(!list.json().benchmarks.some((b) => b.id === id), 'not in the public list');
+ const page = await t.get('/benchmarks');
+ assert.ok(!page.text.includes('My private prompts'), 'not on the benchmarks page');
+ const sitemap = await t.get('/sitemap.xml');
+ assert.ok(!sitemap.text.includes(id), 'not in the sitemap');
+ const dataset = await t.get('/api/v1/datasets/benchmarks.jsonl');
+ assert.ok(!dataset.text.includes(MARK), 'no prompt in the open data');
+ const mine = await t.get('/api/v1/benchmarks?mine=1', { as: alice });
+ assert.ok(mine.json().benchmarks.some((b) => b.id === id), 'the owner lists it');
+ assert.strictEqual(mine.headers.get('cache-control'), 'private, no-store', 'the owner list is not publicly cacheable');
+ const theirs = await t.get('/api/v1/benchmarks?mine=1', { as: bob });
+ assert.ok(!theirs.json().benchmarks.some((b) => b.id === id), 'nobody else does');
+ });
+
+ await check('the owner sees the private page and its prompts', async () => {
+ const res = await t.get(`/benchmarks/${id}`, { as: alice });
+ assert.strictEqual(res.status, 200, res.text.slice(0, 200));
+ assert.ok(res.text.includes('Private.'), 'the private notice');
+ assert.ok(res.text.includes(prompts[0]), 'the owner sees the prompt');
+ assert.ok(res.text.includes('noindex'), 'kept out of search');
+ assert.ok(res.text.includes('Share an unlisted link'), 'the owner can share');
+ assert.strictEqual(res.headers.get('cache-control'), 'private, no-store');
+ });
+
+ await check('runs published for the owner make the two separate leaderboards', async () => {
+ // A public benchmark of the same kind, with a run, is "the public one beside it".
+ const pub = await t.get('/api/v1/benchmarks', { bearer: publisher, json: {
+ title: 'Public text answers', description: 'The public board.', kind: 'text.answer', tags: [], visibility: 'public',
+ tasks: [{ id: 't1', prompt: 'A public question.', files: [], expected: null, holdout: false }],
+ judges: [{ type: 'automatic_grader', version: '1.0', config: {} }], scoring: { method: 'mean', scale: { min: 0, max: 100 } },
+ limits: { ms: 60000, tokens: 0, usd: 0 }, matrix: { models: [], harnesses: [] },
+ } });
+ assert.strictEqual(pub.status, 201, pub.text);
+ const pubId = pub.json().benchmark.id;
+ assert.strictEqual((await t.get(`/api/v1/benchmarks/${pubId}/versions`, { bearer: publisher, json: {} })).status, 201);
+ await publish(`${pubId}@1`, 'public-model', [90], { cost: 0.01 });
+
+ await publish(`${id}@${version}`, 'model-a', [80], { cost: 0.03 });
+ await publish(`${id}@${version}`, 'model-a', [95], { cost: 0.03 });
+ await publish(`${id}@${version}`, 'model-b', [85], { cost: 0.05 });
+ await publish(`${id}@${version}`, 'model-b', [96], { cost: 0.05 });
+ await publish(`${id}@${version}`, 'model-c', [50], { cost: 0.01 });
+ await publish(`${id}@${version}`, 'model-c', [55], { cost: 0.01 });
+
+ const res = await t.get(`/benchmarks/${id}`, { as: alice });
+ assert.strictEqual(res.status, 200, res.text.slice(0, 200));
+ assert.ok(res.text.includes('Your prompts, 6 runs'), 'the owner leaderboard label');
+ assert.ok(res.text.includes('The public board for this kind'), 'the public board is beside it');
+ assert.ok(res.text.includes('Cost / solved task'), 'cost per solved task');
+ assert.ok(/1st.{0,2}2nd/.test(res.text), 'overlapping intervals share a band');
+ assert.ok(res.text.includes('model-a') && res.text.includes('model-b') && res.text.includes('model-c'), 'the own models');
+ assert.ok(res.text.includes('public-model'), 'the public model');
+ const start = res.text.indexOf('The public board for this kind');
+ const end = res.text.indexOf('id="br-tasks-h"', start);
+ const publicPart = res.text.slice(start, end);
+ assert.ok(!publicPart.includes('model-a') && !publicPart.includes('model-b'), 'the private ranks are not mixed into the public board');
+ });
+
+ await check('the run goes through the launcher and still asks before it starts', async () => {
+ const before = codes.calls.filter((c) => c.method === 'POST').length;
+ const tooFew = await post(`/benchmarks/${id}/run`, { models: 'model-a', harnesses: 'codex', repeat: '1' });
+ assert.strictEqual(tooFew.status, 400, tooFew.text.slice(0, 200));
+ assert.ok(tooFew.text.includes('3 to 6 models'), '3 to 6 models');
+ assert.strictEqual(codes.calls.filter((c) => c.method === 'POST').length, before, 'nobody called Codes for one model');
+ const ask = await post(`/benchmarks/${id}/run`, { models: 'model-a', more_models: 'model-b, model-c', harnesses: 'codex', repeat: '1' });
+ assert.strictEqual(ask.status, 200, ask.text.slice(0, 200));
+ assert.ok(ask.text.includes('sets no dollar limit'), 'the no-limit confirmation');
+ assert.strictEqual(codes.calls.filter((c) => c.method === 'POST').length, before, 'Codes was not called yet');
+ const go = await post(`/benchmarks/${id}/run`, { models: 'model-a', more_models: 'model-b, model-c', harnesses: 'codex', repeat: '1', confirm: '1' });
+ assert.strictEqual(go.status, 303, go.text.slice(0, 200));
+ const calls = codes.calls.filter((c) => c.method === 'POST').slice(before);
+ assert.strictEqual(calls.length, 1);
+ assert.deepStrictEqual(calls[0].body.models, ['model-a', 'model-b', 'model-c']);
+ assert.ok(calls[0].headers['x-ov-subject'], 'as the person');
+ assert.match(String(calls[0].headers.authorization), /^Bearer /, 'with Bench’s service token');
+ const api = await t.get(`/api/v1/benchmarks/${id}/run`, { as: alice, json: { models: ['model-a', 'model-b', 'model-c'], harnesses: ['codex'], repeat: 1 } });
+ assert.strictEqual(api.status, 409, api.text);
+ assert.strictEqual(api.json().cost_estimate_usd, null, 'no estimate is invented');
+ });
+
+ await check('an unlisted share link opens the page once, with a shared note and noindex', async () => {
+ const res = await share(id, 'unlisted');
+ assert.strictEqual(res.status, 200, res.text.slice(0, 300));
+ token = tokenOf(res);
+ assert.ok(token.length >= 40, 'a long random token');
+ const page = await t.get(`/b/${token}`);
+ assert.strictEqual(page.status, 200, page.text.slice(0, 300));
+ assert.ok(page.text.includes('Shared by its owner'), 'the shared note');
+ assert.ok(page.text.includes('noindex'), 'noindex');
+ assert.ok(page.text.includes(MARK), 'the prompts are readable by the link');
+ assert.ok(!page.text.includes('Share an unlisted link'), 'a visitor cannot re-share');
+ assert.strictEqual((await t.get(`/benchmarks/${id}`)).status, 404, 'the id alone no longer opens it');
+ assert.strictEqual((await t.get(`/api/v1/benchmarks/${id}`)).status, 404, 'nor its API definition');
+ assert.strictEqual((await t.get('/b/not-a-real-token')).status, 404);
+ assert.strictEqual((await t.get(`/benchmarks/${id}`, { as: bob })).status, 404);
+ });
+
+ await check('making it private again stops the link', async () => {
+ const res = await share(id, 'private');
+ assert.strictEqual(res.status, 200, res.text.slice(0, 200));
+ assert.ok(res.text.includes('Private.'), 'private again');
+ assert.strictEqual((await t.get(`/b/${token}`)).status, 404, 'the old link is dead');
+ assert.strictEqual((await t.get(`/benchmarks/${id}`, { as: alice })).status, 200);
+ });
+
+ await check('sharing unlisted again makes a new link and invalidates the old one, then public lists it', async () => {
+ const res = await share(id, 'unlisted');
+ const second = tokenOf(res);
+ assert.notStrictEqual(second, token, 'a new share is a new token');
+ assert.strictEqual((await t.get(`/b/${token}`)).status, 404, 'the first link is dead');
+ assert.strictEqual((await t.get(`/b/${second}`)).status, 200);
+ const pub = await share(id, 'public');
+ assert.strictEqual(pub.status, 200, pub.text.slice(0, 200));
+ assert.strictEqual((await t.get(`/b/${second}`)).status, 404, 'the token is cleared when it becomes public');
+ assert.strictEqual((await t.get(`/benchmarks/${id}`)).status, 200, 'public by id');
+ const list = await t.get('/api/v1/benchmarks');
+ assert.ok(list.json().benchmarks.some((b) => b.id === id), 'in the public list');
+ const sitemap = await t.get('/sitemap.xml');
+ assert.ok(sitemap.text.includes(id), 'in the sitemap');
+ const dataset = await t.get('/api/v1/datasets/benchmarks.jsonl');
+ assert.ok(dataset.text.includes(MARK), 'its prompts are in the open data');
+ assert.ok(dataset.text.includes('benchmark_run'), 'with its runs');
+ assert.ok(!dataset.text.includes(alice.subject), 'no person id in the open data');
+ assert.ok(!dataset.text.includes('created_by'), 'no created_by field in the open data');
+ });
+
+ await check('making it private again takes it out of every public surface', async () => {
+ await share(id, 'private');
+ const list = await t.get('/api/v1/benchmarks');
+ assert.ok(!list.json().benchmarks.some((b) => b.id === id));
+ const dataset = await t.get('/api/v1/datasets/benchmarks.jsonl');
+ assert.ok(!dataset.text.includes(MARK), 'the prompts leave the open data');
+ const sitemap = await t.get('/sitemap.xml');
+ assert.ok(!sitemap.text.includes(id));
+ });
+
+ await check('every form post is same-origin', async () => {
+ const make = await post('/benchmarks/new', { kind: 'text.answer', format: 'lines', prompts: prompts.join('\n') }, { origin: 'https://evil.example' });
+ assert.strictEqual(make.status, 403, make.text.slice(0, 200));
+ const shared = await post(`/benchmarks/${id}/share`, { visibility: 'public' }, { origin: 'https://evil.example' });
+ assert.strictEqual(shared.status, 403, shared.text.slice(0, 200));
+ });
+
+ await check('stop', async () => { await t.close(); codes.close(); });
+
+ done();
+})();
+