openvibers/openvibe.design Commits
The Design Arena runs on OpenVibe.Bench: rounds, blind votes, Elo and cost boards, the dataset (#5)
Changes
@@@ -41,11 +41,17 @@ OV_AI_INTERNAL_URL=http://127.0.0.1:4700# Three explicit provider:model slots; unset pins AI's generate-route offers (fewer than three falls back to its pool).# DESIGN_ARENA_MODELS=shared:gpt-5-mini,local:qwen2.5--# Arena budgets (per UTC day) and generation jobs.++# OpenVibe.Bench owns new Arena rounds, entries, votes and boards.++OV_BENCH_INTERNAL_URL=http://127.0.0.1:5230++# OV_BENCH_URL=https://bench.openvibe.services++# DESIGN_BENCH_AUDIENCE=openvibe.bench++# DESIGN_BENCH_SCOPE=bench.task.create bench.round.create bench.round.read bench.judgement.cast bench.board.read++++# Legacy Arena settings retained for compatibility; Bench enforces active round limits and runs generation.# DESIGN_ARENA_PER_DAY=3# DESIGN_ARENA_DAILY=20# DESIGN_ARENA_JOB_INTERVAL_MS=1500--# The drill flag keeps the generation loop off; a test or a drill calls tick() itself.++# The drill flag controls other Design background work.# DESIGN_DRILL=0# OpenVibe.Events → this service for ADR-033 account export/deletion; unset turns the route off (503).@@@ -17,7 +17,7 @@ Network's theme API.- **Identity:** service id `design`, name OpenVibe.Design, port 5090, host 127.0.0.1. It stores no accounts andno tokens: the person's Network session lives in httpOnly cookies and their theme lives in OpenVibe.Network. The-- Arena's rounds, outputs, votes and Elo ratings live in this service's PostgreSQL database (PGlite in development).++ Arena rounds, entries, judgements and ratings live in OpenVibe.Bench. The legacy `arena_*` tables remain for historical account events and migration.- **Sign-in:** an OAuth 2 client of OpenVibe.Network (`design`), authorization code with PKCE S256, a state checkand a server-side code exchange.- **The theme API proxy:** every theme read and write goes to OpenVibe.Network's `/api/themes`, with the person's@@@ -38,13 +38,10 @@ Network's theme API.## The Design Arena--- **A round:** a signed-in person writes one prompt in one of six categories (landing, dashboard, mobile app, game-- UI, logo, slides). It is queued; the job loop reads the offers on OpenVibe.AI's `generate` routing and pins three-- distinct ones (`route: { provider, model }`), and each output stores the HTML (sanitized), the provider and model AI-- reports, its cost, its measured time and its tokens. When AI offers fewer than three, the remaining slots go to its-- pool and the round says so (`pool_fallback: true`).--- **Budgets:** `DESIGN_ARENA_PER_DAY` (default 3) rounds per person per UTC day and `DESIGN_ARENA_DAILY` (default 20)-- for the whole site. A refusal says which budget was reached and when to come back.++- **A round:** a signed-in person writes one prompt in one of six categories. Design creates a public Bench task of++ kind `design.page`, or `design.component` for logo, then creates a round. Bench queues and generates three entries++ through OpenVibe.AI, including their HTML, provider, model, cost, time and tokens.++- **Budgets:** Bench enforces its per-person and site daily round limits.- **Safety:** the model's page is text until it is rendered, and then only inside `<iframe sandbox srcdoc>` with no`allow-scripts` and no `allow-same-origin`. A CSP meta in the srcdoc blocks every network fetch(`default-src 'none'; style-src 'unsafe-inline'; img-src data:`). Scripts, `on*` handlers and `javascript:` URLs@@@ -52,13 +49,15 @@ Network's theme API.- **Voting:** `/arena/vote` serves two successful outputs in random order with the model names hidden. One vote perperson per round; the round's creator cannot vote. A vote reveals both models and updates the category's Eloratings (K = 32, a tie is 0.5, `both_bad` is a game for neither).--- **Leaderboard:** per category: model, Elo, games, wins, losses, ties, win rate, median cost and median time.--- **Open dataset:** `GET /api/v1/arena/dataset.jsonl` is one JSON object per round with its prompt, every output-- (metadata and HTML) and every vote without a voter id.--- **AI access:** Design is a first-party service and calls AI with a client-credentials token from Network:-- audience `openvibe.ai`, scope `ai.run.create` (and the `ai.generate` namespace in the grant). It pins one of that-- routing's listed offers per slot; AI refuses anything it does not offer in words, and a pin never bypasses its-- quotas or budgets. Generation is off under `DESIGN_DRILL=1`; the job loop starts with the process otherwise.++- **Leaderboard:** Bench Elo boards for both kinds, including confidence intervals and provisional status, and++ Bench Cost boards with entries, usable entries, total cost and cost per solved entry.++- **Open dataset:** `GET /api/v1/arena/dataset.jsonl` redirects to Bench's `design.page` JSONL dataset.++- **Bench access:** Design uses a client-credentials token (audience `openvibe.bench`) to create tasks and read++ boards. It sends the creator's Network token to create each round so Bench records ownership and enforces the++ personal budget and creator vote rule. Blind pairs and votes also use the person's token.++- **Legacy import:** `scripts/arena-to-bench.js` previews a migration by default. `--apply` copies old rounds and++ outputs and records votes as anonymous `automatic_grader` judgements. Bench's Elo board does not count these++ historical judgements. The importer needs `bench.entry.create` and is subject to Bench's round limits.## Pages@@@ -96,11 +95,11 @@ a Bearer write is not cookie-authenticated and is unaffected.| `POST /api/me/theme/history/:id/restore` | cookie + same-origin | Network's restore; `id` must be a positive integer, else 400 || `POST /api/themes/submit` | cookie + same-origin | Network `POST /api/themes` with `{ name, slug, description, mode, variables, tags }` || `POST /api/v1/arena/rounds` | cookie + same-origin | queue a round `{ category, prompt }` (subject to the two daily budgets) |--| `GET /api/v1/arena/rounds/:id` | none | one round with its outputs, cost and time |--| `GET /api/v1/arena/vote` | cookie | the next blind pair for this person (models hidden) |++| `GET /api/v1/arena/rounds/:id` | none | one round; entries are blind until Bench reveals them to this viewer |++| `GET /api/v1/arena/vote[?category=logo]` | cookie | the next blind pair for this person (models hidden) || `POST /api/v1/arena/vote` | cookie + same-origin | `{ round_id, left_output, right_output, winner }`; reveals both models and updates Elo |--| `GET /api/v1/arena/leaderboard?category=…` | none | Elo, games, win rate, median cost and median time |--| `GET /api/v1/arena/dataset.jsonl` | none | prompts, outputs (HTML included) and votes without voters |++| `GET /api/v1/arena/leaderboard?category=…&type=elo\|cost` | none | Bench Elo or Cost board for the category's kind |++| `GET /api/v1/arena/dataset.jsonl` | none | 302 to Bench's `design.page` JSONL dataset |- A Network **4xx** answer passes through with its status and body; a Network failure (no answer, timeout, 5xx)is a `502 network.unavailable` problem.@@@ -145,9 +144,8 @@ npm run dev # http://localhost:5090OV_ALLOW_SUITE=1 npm test```--`npm test` runs every `test/*.test.js` in its own process with an in-process OpenVibe.Network stand-in--(`test/helpers/network.js`): a real RS256 key pair, the JWKS, the OAuth token endpoint, the theme catalogue and--the person's theme routes. Nothing calls the real internet. `OV_ALLOW_SUITE=1` is the suite opt-in the shared++`npm test` runs every `test/*.test.js` in its own process with in-process OpenVibe.Network and OpenVibe.Bench++stand-ins (`test/helpers/network.js`, `test/helpers/bench.js`). Nothing calls the real internet. `OV_ALLOW_SUITE=1` is the suite opt-in the sharedtest runner expects.## Deploy (for the lead)nnew file mode 100644@@@ -0,0 +1,68 @@++#!/usr/bin/env node++'use strict';++++/** One-off legacy Arena import. Dry run is the default. No identity is sent with judgements. */++const { createBenchClient } = require('../server/arena/bench');++const { load } = require('../server/config');++const { createDb } = require('openvibe-sdk/db');++++async function readLegacy(db) {++ const rounds = await db.many('SELECT id, category, prompt, status, created_at FROM arena_rounds ORDER BY created_at, id');++ const outputs = await db.many('SELECT id, "round", model, provider, html, cost_usd, ms, tokens, error, ui_check, created_at FROM arena_outputs ORDER BY created_at, id');++ const votes = await db.many('SELECT "round", left_output, right_output, winner, at FROM arena_votes ORDER BY at, "round"');++ return { rounds, outputs, votes };++}++++function preflight({ rounds, outputs, votes }) {++ const problems = [];++ const ids = new Set(rounds.map((r) => r.id));++ const outputById = new Map(outputs.map((o) => [o.id, o]));++ for (const v of votes) {++ const left = outputById.get(v.left_output);++ const right = outputById.get(v.right_output);++ if (!ids.has(v.round) || !left || !right || left.round !== v.round || right.round !== v.round || left.error || right.error) {++ problems.push(`vote on ${v.round} has a missing, failed or foreign entry`);++ }++ }++ return problems;++}++++async function migrate({ db, bench, apply = false }) {++ const legacy = await readLegacy(db);++ const problems = preflight(legacy);++ const summary = { dry_run: !apply, rounds: legacy.rounds.length, entries: legacy.outputs.length, judgements: legacy.votes.length, problems };++ if (!apply) return summary;++ if (problems.length) throw new Error(`Import refused: ${problems.join('; ')}`);++ const outputMap = new Map();++ for (const r of legacy.rounds) {++ const { round } = await bench.createRound({ category: r.category, prompt: r.prompt });++ for (const o of legacy.outputs.filter((e) => e.round === r.id)) {++ const result = await bench.addEntry({ roundId: round.id, provider: o.provider, model: o.model,++ route: {}, payload: { html: o.html }, costUsd: o.cost_usd, ms: o.ms, tokens: o.tokens,++ error: o.error, uiCheck: o.ui_check });++ outputMap.set(o.id, result.entry.id);++ }++ for (const v of legacy.votes.filter((j) => j.round === r.id)) {++ await bench.castImportedVote({ roundId: round.id,++ pair: { left: outputMap.get(v.left_output), right: outputMap.get(v.right_output) }, winner: v.winner });++ }++ }++ return summary;++}++++async function main() {++ const apply = process.argv.includes('--apply');++ const config = load();++ if (!config.db.url) throw new Error('DATABASE_URL is required for the one-off import');++ config.bench.scope = `${config.bench.scope} bench.entry.create`;++ const db = createDb({ url: config.db.url, service: 'design-arena-import', max: 1 });++ try {++ const bench = apply ? createBenchClient({ config }) : null;++ const summary = await migrate({ db, bench, apply });++ process.stdout.write(`${JSON.stringify(summary)}\n`);++ if (summary.problems.length) process.exitCode = 2;++ } finally { await db.close(); }++}++++if (require.main === module) main().catch((err) => { console.error(err.message); process.exitCode = 1; });++module.exports = { readLegacy, preflight, migrate };@@@ -12,8 +12,7 @@* /api/health, /api/ready, /release.json liveness, truthful readiness, release metadata* /robots.txt, /sitemap.xml, /llms.txt crawl artifacts (http/discovery.js)*-- * The Arena's rounds are generated by a job loop (server/index.js starts it; DESIGN_DRILL=1 keeps it off). The-- * request that starts a round only queues it.++ * Bench generates Arena rounds asynchronously. The request that starts a round only queues it.*/const path = require('path');const express = require('express');@@@ -39,7 +38,7 @@ const { createDiscoveryRoutes } = require('./http/discovery');const { createServiceReadiness } = require('./observability');const { createArenaStore } = require('./arena/store');const { createArenaAi } = require('./arena/ai');--const { createArenaJobs } = require('./arena/jobs');++const { createBenchClient } = require('./arena/bench');const accountDataLib = require('./identity/account-data');const { PUBLIC_DIR, assetVersion } = require('./assets');const views = require('./views');@@@ -60,16 +59,18 @@ async function createApp(opts = {}) {const keys = createKeyStore({ config, fetchImpl, log });const sso = createSso({ config, keys, fetchImpl, log, now });const catalog = opts.catalog || createThemeCatalog({ config, fetchImpl, log, now });-- const arena = opts.arena || createArenaStore({ s, config, log });++ const bench = opts.bench || createBenchClient({ config, fetchImpl, log });++ const arena = opts.arena || createArenaStore({ s, config, bench, log });const ai = opts.arenaAi || opts.ai || createArenaAi({ config, fetchImpl, log });-- const jobs = createArenaJobs({ s, arena, ai, config, log });++ // The Arena generation job now lives on Bench; Design keeps a no-op loop for test compatibility.++ const jobs = opts.jobs || { tick: async () => null, start: () => null, stop: () => null, get running() { return false; } };const accountData = accountDataLib.create({ db: s.db, log });const accountSend = opts.accountSend || (config.oauth.clientSecret? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl }): async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Design cannot answer account events'); });const ctx = {-- config, s, store: s, keys, sso, catalog, arena, ai, jobs, accountData, accountSend, log,++ config, s, store: s, keys, sso, catalog, arena, bench, ai, jobs, accountData, accountSend, log,random: opts.random || Math.random,};nnew file mode 100644@@@ -0,0 +1,193 @@++'use strict';++++/**++ * OpenVibe.Bench client for the Design Arena. Design uses its service token for task creation, reads and boards,++ * and forwards each person's Network token for round creation, blind pairs, votes and reveals.++ *++ * service token (audience openvibe.bench, bench.task.create / bench.round.read / bench.board.read)++ * person token (Network user access token, audience openvibe.network)++ */++const { serviceAuth } = require('openvibe-contracts');++const { ArenaError } = require('./legacy-store');++++const BENCH_CAPABILITIES = Object.freeze([++ 'bench.task.create',++ 'bench.round.create',++ 'bench.round.read',++ 'bench.judgement.cast',++ 'bench.board.read',++].join(' '));++++class BenchClientError extends Error {++ constructor(status, code, detail) {++ super(detail || code);++ this.name = 'BenchClientError';++ this.status = status;++ this.code = code;++ this.detail = detail || code;++ }++}++++function mapBenchProblem(status, body) {++ const code = body && (body.code || body.error) ? String(body.code || body.error) : 'bench.error';++ const detail = body && (body.detail || body.message) ? String(body.detail || body.message) : 'OpenVibe.Bench returned an error.';++ switch (code) {++ case 'bench.limit_person': return new ArenaError(429, 'arena.limit_person', detail);++ case 'bench.limit_site': return new ArenaError(429, 'arena.limit_site', detail);++ case 'bench.kind_not_found': return new ArenaError(400, 'arena.invalid_category', detail);++ case 'bench.prompt_required':++ case 'bench.prompt_too_long': return new ArenaError(422, 'arena.prompt_required', detail);++ case 'bench.round_not_found': return new ArenaError(404, 'arena.round_not_found', detail);++ case 'bench.round_not_ready': return new ArenaError(409, 'arena.round_not_ready', detail);++ case 'bench.creator_cannot_vote': return new ArenaError(403, 'arena.creator_cannot_vote', detail);++ case 'bench.already_judged': return new ArenaError(409, 'arena.already_voted', detail);++ case 'bench.invalid_pair': return new ArenaError(422, 'arena.invalid_pair', detail);++ case 'bench.invalid_outcome': return new ArenaError(422, 'arena.invalid_winner', detail);++ default: return new BenchClientError(status, code, detail);++ }++}++++function kindForCategory(category) {++ return category === 'logo' ? 'design.component' : 'design.page';++}++++function createBenchClient({ config, fetchImpl = globalThis.fetch, log = console }) {++ const bench = config.bench;++ const enabled = Boolean(bench && bench.url);++ const tokenClient = enabled && config.oauth && config.oauth.clientSecret++ ? serviceAuth.createTokenClient({++ tokenUrl: `${config.networkInternalUrl || config.networkUrl}/oauth/token`,++ clientId: config.oauth.clientId,++ clientSecret: config.oauth.clientSecret,++ audience: bench.audience,++ scope: bench.scope,++ fetchImpl,++ })++ : null;++++ async function serviceHeaders() {++ if (!enabled) throw new BenchClientError(503, 'bench.not_configured', 'OpenVibe.Bench is not configured for the Arena.');++ if (!tokenClient) throw new BenchClientError(503, 'bench.not_configured', 'Design has no client secret to talk to Bench.');++ return { Accept: 'application/json', ...(await tokenClient.authHeaders()), 'Content-Type': 'application/json' };++ }++++ function personHeaders(token) {++ if (!enabled) throw new BenchClientError(503, 'bench.not_configured', 'OpenVibe.Bench is not configured for the Arena.');++ if (!token) throw new ArenaError(401, 'token.required', 'Sign in to use the Arena.');++ return { Accept: 'application/json', Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' };++ }++++ async function request(method, path, { body = null, headers = null, token = null } = {}) {++ const h = headers || (token ? personHeaders(token) : await serviceHeaders());++ const url = `${bench.url.replace(/\/+$/, '')}${path}`;++ const opts = { method, headers: h, signal: AbortSignal.timeout(config.timeoutMs || 8000) };++ if (body != null) opts.body = JSON.stringify(body);++ let res;++ try {++ res = await fetchImpl(url, opts);++ } catch (err) {++ throw new BenchClientError(502, 'bench.unreachable', err && err.name === 'TimeoutError' ? 'OpenVibe.Bench did not answer in time.' : 'OpenVibe.Bench could not be reached.');++ }++ const json = await res.json().catch(() => null);++ if (!res.ok) {++ const err = mapBenchProblem(res.status, json);++ throw err;++ }++ return json || {};++ }++++ async function createTask({ category, prompt }) {++ const kindId = kindForCategory(category);++ const out = await request('POST', `/api/v1/kinds/${encodeURIComponent(kindId)}/tasks`, {++ body: { prompt, public: true, task_class: category },++ });++ return out.task;++ }++++ async function createRound({ category, prompt, token, offers = null }) {++ const task = await createTask({ category, prompt });++ const body = {};++ if (offers) body.offers = offers;++ const out = await request('POST', `/api/v1/tasks/${encodeURIComponent(task.id)}/rounds`, {++ body,++ token,++ });++ return { task, round: { ...out.round, task_class: task.task_class, prompt: task.prompt } };++ }++++ async function getTask(id) {++ const out = await request('GET', `/api/v1/tasks/${encodeURIComponent(id)}`);++ return out.task;++ }++++ async function getRound(id, { token = null } = {}) {++ const out = await request('GET', `/api/v1/rounds/${encodeURIComponent(id)}`, { token });++ const round = out.round;++ if (round && round.task_id && !round.prompt) {++ try {++ const task = await getTask(round.task_id);++ round.task_class = task.task_class;++ round.prompt = task.prompt;++ } catch { /* keep blind */ }++ }++ return round;++ }++++ async function nextPair(kindId, token) {++ const out = await request('GET', `/api/v1/judgements/next?kind=${encodeURIComponent(kindId)}`, { token });++ return out && out.round ? out : null;++ }++++ async function castVote({ roundId, token, pair, winner }) {++ const out = await request('POST', `/api/v1/rounds/${encodeURIComponent(roundId)}/judgements`, {++ token,++ body: { pair, outcome: winner },++ });++ return out;++ }++++ async function castImportedVote({ roundId, pair, winner }) {++ return await request('POST', `/api/v1/rounds/${encodeURIComponent(roundId)}/judgements`, {++ body: { judge: 'automatic_grader', judge_name: 'design.arena.legacy_vote', pair, outcome: winner },++ });++ }++++ async function getBoard(kindId, { type = 'elo', token = null } = {}) {++ const out = await request('GET', `/api/v1/boards/${encodeURIComponent(kindId)}?type=${encodeURIComponent(type)}`, { token });++ return out.board;++ }++++ function datasetUrl(kindId) {++ return `${bench.publicUrl.replace(/\/+$/, '')}/api/v1/datasets/${encodeURIComponent(kindId)}.jsonl`;++ }++++ async function addEntry({ roundId, provider, model, route, payload, costUsd, ms, tokens, error, uiCheck, runId }) {++ return await request('POST', `/api/v1/rounds/${encodeURIComponent(roundId)}/entries`, {++ body: {++ provider, model, route, payload,++ cost_usd: costUsd,++ ms,++ tokens,++ error,++ ui_check: uiCheck,++ run_id: runId,++ },++ });++ }++++ return {++ enabled,++ kindForCategory,++ createTask,++ createRound,++ getRound,++ nextPair,++ castVote,++ castImportedVote,++ getBoard,++ datasetUrl,++ addEntry,++ };++}++++module.exports = { createBenchClient, BenchClientError, kindForCategory, BENCH_CAPABILITIES };nnew file mode 100644@@@ -0,0 +1,33 @@++'use strict';++++/**++ * Constants and helpers shared between the legacy Arena tables and the Bench-backed store.++ */++++const CATEGORIES = Object.freeze(['landing', 'dashboard', 'mobile_app', 'game_ui', 'logo', 'slides']);++const WINNERS = Object.freeze(['left', 'right', 'tie', 'both_bad']);++const K = 32;++const BASE_ELO = 1000;++++class ArenaError extends Error {++ constructor(status, code, detail, extra = {}) {++ super(detail);++ this.name = 'ArenaError';++ this.status = status;++ this.code = code;++ this.detail = detail;++ this.extra = extra;++ }++}++++/** Fisher–Yates, with an injectable random source so a test can pin the order. */++function shuffle(list, random = Math.random) {++ const a = list.slice();++ for (let i = a.length - 1; i > 0; i--) {++ const j = Math.floor(random() * (i + 1));++ [a[i], a[j]] = [a[j], a[i]];++ }++ return a;++}++++module.exports = { ArenaError, CATEGORIES, WINNERS, K, BASE_ELO, shuffle };@@@ -1,276 +1,166 @@'use strict';/**-- * The Arena's data access: rounds, outputs, votes, ratings, the open dataset and the day's budgets. Every-- * person-facing write is a transaction; a vote and the rating changes it causes commit together, so a crash can-- * never leave a vote without its Elo update. Votes are keyed by (round, voter_subject): one per person per round.++ * The Arena's data access, now backed by OpenVibe.Bench. Design keeps its own `arena_*` tables read-only++ * (legacy data and the migration script); the active engine lives on Bench.++ *++ * createRound() -> Bench task (service token) + round (creator token)++ * getRound() -> Bench round read++ * nextPair() -> Bench /judgements/next (person's Network token)++ * vote() -> Bench /rounds/:id/judgements (person's Network token)++ * leaderboard() -> Bench /boards/:kind?type=elo|cost++ * datasetJsonl() -> redirect to Bench /datasets/:kind.jsonl*/--const { ids } = require('openvibe-contracts');++const { ArenaError, CATEGORIES, WINNERS, K, BASE_ELO, shuffle } = require('./legacy-store');++const { sanitizeHtml } = require('./sanitize');--const CATEGORIES = Object.freeze(['landing', 'dashboard', 'mobile_app', 'game_ui', 'logo', 'slides']);--const WINNERS = Object.freeze(['left', 'right', 'tie', 'both_bad']);--const K = 32;--const BASE_ELO = 1000;----class ArenaError extends Error {-- constructor(status, code, detail, extra = {}) {-- super(detail);-- this.name = 'ArenaError';-- this.status = status;-- this.code = code;-- this.detail = detail;-- this.extra = extra;-- }--}----const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));--const roundOf = (r) => (r ? {-- id: r.id,-- category: r.category,-- prompt: r.prompt,-- status: r.status,-- // The offers the round asked AI for ({provider, model}); a pool slot is provider 'ai' with no model.-- offers: Array.isArray(r.offers) ? r.offers : [],-- // True only when the routing offered fewer than three distinct offers, so the rest came from the pool.-- pool_fallback: Boolean(r.pool_fallback),-- created_at: r.created_at,--} : null);--const outputOf = (o) => ({-- id: o.id,-- round: o.round,-- model: o.model,-- provider: o.provider,-- html: o.html,-- cost_usd: num(o.cost_usd),-- ms: num(o.ms),-- tokens: num(o.tokens),-- error: o.error || null,-- ui_check: o.ui_check == null ? null : o.ui_check,-- created_at: o.created_at,--});--const voteOf = (v) => ({ round: v.round, left_output: v.left_output, right_output: v.right_output, winner: v.winner, at: v.at });----/** Fisher–Yates, with an injectable random source so a test can pin the order. */--function shuffle(list, random = Math.random) {-- const a = list.slice();-- for (let i = a.length - 1; i > 0; i--) {-- const j = Math.floor(random() * (i + 1));-- [a[i], a[j]] = [a[j], a[i]];-- }-- return a;++function entryToOutput(e) {++ const payload = e.payload || {};++ return {++ id: e.id,++ round: e.round_id,++ model: e.model || 'unknown',++ provider: e.provider || null,++ html: sanitizeHtml(payload.html || ''),++ cost_usd: e.cost_usd == null ? 0 : Number(e.cost_usd),++ ms: e.ms == null ? 0 : Number(e.ms),++ tokens: e.tokens == null ? 0 : Number(e.tokens),++ error: e.error || null,++ ui_check: e.ui_check == null ? null : e.ui_check,++ created_at: e.created_at,++ };}--function createArenaStore({ s, config, log = console }) {-- const db = s.db;-- const perDay = config.arena.perDay;-- const daily = config.arena.daily;---- const dayStart = (at = s.now()) => {-- const d = new Date(at);-- d.setUTCHours(0, 0, 0, 0);-- return d.toISOString();++function roundToLocal(r) {++ const entries = (r.entries || []).map(entryToOutput);++ const category = r.task_class || (r.kind_id === 'design.component' ? 'logo' : 'landing');++ return {++ id: r.id,++ category,++ prompt: r.prompt || '',++ status: r.status === 'finished' ? 'ready' : (r.status === 'failed' ? 'failed' : (r.status || 'queued')),++ offers: r.offers || [],++ pool_fallback: Boolean(r.pool_fallback),++ created_at: r.created_at,++ outputs: entries,++ votes: Number(r.votes || 0),};++}-- async function counts(subject) {-- const start = dayStart();-- const used = num(await db.value('SELECT COUNT(*)::int AS n FROM arena_rounds WHERE created_by = $1 AND created_at >= $2::timestamptz', [subject, start]));-- const siteUsed = num(await db.value('SELECT COUNT(*)::int AS n FROM arena_rounds WHERE created_at >= $1::timestamptz', [start]));-- return {-- used,-- remaining: Math.max(0, perDay - used),-- per_day: perDay,-- site_used: siteUsed,-- site_daily: daily,-- site_remaining: Math.max(0, daily - siteUsed),-- };++function createArenaStore({ config, bench }) {++ async function counts(/* subject */) {++ // Bench enforces per-person and site budgets; Design no longer mirrors them.++ return null;}-- async function createRound({ category, prompt, subject }) {++ async function createRound({ category, prompt, token }) {if (!CATEGORIES.includes(category)) throw new ArenaError(422, 'arena.invalid_category', `Pick a category: ${CATEGORIES.join(', ')}.`);const text = String(prompt == null ? '' : prompt).trim();if (!text) throw new ArenaError(422, 'arena.prompt_required', 'Describe the page you want the models to design.');if (text.length > config.arena.maxPrompt) throw new ArenaError(422, 'arena.prompt_too_long', `The prompt must be at most ${config.arena.maxPrompt} characters.`);-- const at = s.now();-- const start = dayStart(at);-- const id = `arr_${ids.ulid(at)}`;-- const row = await db.maybe(-- `INSERT INTO arena_rounds (id, category, prompt, created_by, status, created_at)-- SELECT $1, $2, $3, $4, 'queued', $5::timestamptz-- WHERE (SELECT COUNT(*) FROM arena_rounds WHERE created_by = $4 AND created_at >= $6::timestamptz) < $7-- AND (SELECT COUNT(*) FROM arena_rounds WHERE created_at >= $6::timestamptz) < $8-- RETURNING *`,-- [id, category, text, subject, s.iso(), start, perDay, daily],-- );-- if (!row) {-- const c = await counts(subject);-- if (c.used >= perDay) {-- throw new ArenaError(429, 'arena.limit_person',-- `You can start ${perDay} Arena round${perDay === 1 ? '' : 's'} a day. You have started ${c.used} today; try again tomorrow.`);-- }-- throw new ArenaError(429, 'arena.limit_site',-- `The Arena has reached its ${daily} round${daily === 1 ? '' : 's'} for today. Try again tomorrow.`);-- }-- return await getRound(id);++ if (!token) throw new ArenaError(401, 'token.required', 'Sign in to use the Arena.');++ const { round } = await bench.createRound({ category, prompt: text, token });++ return await getRound(round.id, { token });}-- async function getRound(id) {-- const r = await db.maybe('SELECT * FROM arena_rounds WHERE id = $1', [id]);++ async function getRound(id, { token = null } = {}) {++ const r = await bench.getRound(id, { token });if (!r) return null;-- const outputs = (await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 ORDER BY created_at, id', [id])).map(outputOf);-- const votes = num(await db.value('SELECT COUNT(*)::int AS n FROM arena_votes WHERE "round" = $1', [id]));-- return { ...roundOf(r), outputs, votes };++ return roundToLocal(r);}-- /** A ready round this person has not voted on and did not start, with its two successful outputs shuffled. */-- async function nextPair(subject, { random = Math.random } = {}) {-- const r = await db.maybe(-- `SELECT r.* FROM arena_rounds r-- WHERE r.status = 'ready' AND r.created_by <> $1-- AND (SELECT COUNT(*) FROM arena_outputs o WHERE o."round" = r.id AND o.error IS NULL AND o.html <> '') >= 2-- AND NOT EXISTS (SELECT 1 FROM arena_votes v WHERE v."round" = r.id AND v.voter_subject = $1)-- ORDER BY random() LIMIT 1`,-- [subject],-- );-- if (!r) return null;-- const outputs = (await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 AND error IS NULL AND html <> \'\' ORDER BY id', [r.id])).map(outputOf);-- if (outputs.length < 2) return null;-- const [left, right] = shuffle(outputs, random).slice(0, 2);-- // Blind: the model, provider, cost and time are not part of this answer. The HTML is (in the iframe).++ async function nextPair(_subject, { token, category = 'landing', random: rng = Math.random } = {}) {++ if (!token) throw new ArenaError(401, 'token.required', 'Sign in at openvibe.design to use the Arena.');++ const kindId = categoryToKind(category);++ const pair = await bench.nextPair(kindId, token);++ if (!pair) return null;++ const [left, right] = shuffle([pair.left, pair.right], rng);return {-- round: { id: r.id, category: r.category, prompt: r.prompt },-- left: { id: left.id, html: left.html },-- right: { id: right.id, html: right.html },++ round: { id: pair.round.id, category: pair.round.task_class || (pair.round.kind_id === 'design.component' ? 'logo' : 'landing'), prompt: pair.round.prompt },++ left: { id: left.id, html: sanitizeHtml((left.payload || {}).html || '') },++ right: { id: right.id, html: sanitizeHtml((right.payload || {}).html || '') },};}-- async function applyElo(t, category, left, right, winner, atIso) {-- if (winner === 'both_bad') return; // a game for neither-- if (left.model === right.model) return; // two outputs of one model are not a match between designers-- const models = [left.model, right.model];-- for (const model of models) {-- await t.exec(`INSERT INTO arena_ratings (category, model, elo, games, wins, losses, ties, updated_at)-- VALUES ($1, $2, $3, 0, 0, 0, 0, $4::timestamptz)-- ON CONFLICT (category, model) DO NOTHING`, [category, model, BASE_ELO, atIso]);-- }-- const rows = await t.many('SELECT * FROM arena_ratings WHERE category = $1 AND model = ANY($2::text[]) ORDER BY model FOR UPDATE', [category, models]);-- const byModel = new Map(rows.map((r) => [r.model, r]));-- const a = byModel.get(left.model);-- const b = byModel.get(right.model);-- if (!a || !b) return;-- const ra = num(a.elo, BASE_ELO);-- const rb = num(b.elo, BASE_ELO);-- const ea = 1 / (1 + (10 ** ((rb - ra) / 400)));-- const eb = 1 - ea;-- const sa = winner === 'left' ? 1 : (winner === 'right' ? 0 : 0.5);-- const sb = 1 - sa;-- const nextA = Math.round((ra + K * (sa - ea)) * 100) / 100;-- const nextB = Math.round((rb + K * (sb - eb)) * 100) / 100;-- const update = async (row, elo, score, oppScore) => {-- await t.exec(`UPDATE arena_ratings SET elo = $1, games = games + 1,-- wins = wins + $2, losses = losses + $3, ties = ties + $4, updated_at = $5::timestamptz-- WHERE category = $6 AND model = $7`,-- [elo, score > oppScore ? 1 : 0, score < oppScore ? 1 : 0, score === oppScore ? 1 : 0, atIso, category, row.model]);++ async function vote({ roundId, leftOutput, rightOutput, winner, token }) {++ if (!WINNERS.includes(winner)) throw new ArenaError(422, 'arena.invalid_winner', 'The winner must be left, right, tie or both_bad.');++ if (!leftOutput || !rightOutput || leftOutput === rightOutput) throw new ArenaError(422, 'arena.invalid_pair', 'A vote needs two different outputs.');++ if (!token) throw new ArenaError(401, 'token.required', 'Sign in at openvibe.design to use the Arena.');++ const out = await bench.castVote({ roundId, token, pair: { left: leftOutput, right: rightOutput }, winner });++ const left = out.revealed && out.revealed.left ? entryToOutput(out.revealed.left) : null;++ const right = out.revealed && out.revealed.right ? entryToOutput(out.revealed.right) : null;++ return { round: { id: roundId }, winner, at: out.judgement && out.judgement.at, left, right };++ }++++ function boardRowToLocal(r) {++ return {++ category: r.category || 'all',++ model: r.subject || r.model || 'unknown',++ elo: r.elo == null ? BASE_ELO : Number(r.elo),++ games: Number(r.games || 0),++ wins: Number(r.wins || 0),++ losses: Number(r.losses || 0),++ ties: Number(r.ties || 0),++ win_rate: Number(r.win_rate || 0),++ median_cost_usd: r.median_cost_usd == null ? null : Number(r.median_cost_usd),++ median_cost: r.median_cost_usd == null ? null : Number(r.median_cost_usd),++ median_ms: r.median_ms == null ? null : Number(r.median_ms),++ ci_lower: r.ci_lower == null ? null : Number(r.ci_lower),++ ci_upper: r.ci_upper == null ? null : Number(r.ci_upper),++ provisional: Boolean(r.provisional),++ entries: Number(r.entries || 0),++ usable: Number(r.usable || 0),++ total_cost_usd: Number(r.total_cost_usd || 0),++ solved: Number(r.solved || 0),++ cost_per_solved: r.cost_per_solved == null ? null : Number(r.cost_per_solved),};-- await update(a, nextA, sa, sb);-- await update(b, nextB, sb, sa);}-- async function vote({ roundId, subject, leftOutput, rightOutput, winner }) {-- if (!WINNERS.includes(winner)) throw new ArenaError(422, 'arena.invalid_winner', 'The winner must be left, right, tie or both_bad.');-- if (!leftOutput || !rightOutput || leftOutput === rightOutput) throw new ArenaError(422, 'arena.invalid_pair', 'A vote needs two different outputs.');-- try {-- const result = await s.tx(async () => {-- const r = await db.maybe('SELECT * FROM arena_rounds WHERE id = $1 FOR UPDATE', [roundId]);-- if (!r) throw new ArenaError(404, 'arena.round_not_found', 'No such Arena round.');-- if (r.created_by === subject) throw new ArenaError(403, 'arena.creator_cannot_vote', 'You cannot vote on a round you started.');-- if (r.status !== 'ready') throw new ArenaError(409, 'arena.round_not_ready', 'This round is not ready to vote on yet.');-- const outputs = (await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 AND id = ANY($2::text[]) AND error IS NULL AND html <> \'\'', [roundId, [leftOutput, rightOutput]])).map(outputOf);-- if (outputs.length !== 2) throw new ArenaError(422, 'arena.invalid_pair', 'Those two outputs are not both part of this round.');-- const left = outputs.find((o) => o.id === leftOutput);-- const right = outputs.find((o) => o.id === rightOutput);-- const prior = await db.maybe('SELECT 1 AS n FROM arena_votes WHERE "round" = $1 AND voter_subject = $2', [roundId, subject]);-- if (prior) throw new ArenaError(409, 'arena.already_voted', 'You have already voted on this round.');-- const at = s.iso();-- await db.exec('INSERT INTO arena_votes ("round", voter_subject, left_output, right_output, winner, at) VALUES ($1, $2, $3, $4, $5, $6::timestamptz)',-- [roundId, subject, leftOutput, rightOutput, winner, at]);-- await applyElo(db, r.category, left, right, winner, at);-- return { round: roundOf(r), winner, at, left, right };-- });-- return result;-- } catch (err) {-- if (err && err.code === '23505') throw new ArenaError(409, 'arena.already_voted', 'You have already voted on this round.');-- throw err;-- }++ function categoryToKind(category) {++ return category === 'logo' || category === 'design.component' ? 'design.component' : 'design.page';}-- async function leaderboard(category = null) {-- const rows = await db.many(-- `WITH models AS (-- SELECT category, model FROM arena_ratings WHERE ($1::text IS NULL OR category = $1)-- UNION-- SELECT r.category, o.model FROM arena_outputs o JOIN arena_rounds r ON r.id = o."round"-- WHERE r.status = 'ready' AND o.error IS NULL AND ($1::text IS NULL OR r.category = $1)-- ), medians AS (-- SELECT r.category, o.model,-- percentile_cont(0.5) WITHIN GROUP (ORDER BY o.cost_usd) AS median_cost,-- percentile_cont(0.5) WITHIN GROUP (ORDER BY o.ms) AS median_ms-- FROM arena_outputs o JOIN arena_rounds r ON r.id = o."round"-- WHERE r.status = 'ready' AND o.error IS NULL AND ($1::text IS NULL OR r.category = $1)-- GROUP BY r.category, o.model-- )-- SELECT m.category, m.model, COALESCE(rt.elo, $2) AS elo,-- COALESCE(rt.games, 0) AS games, COALESCE(rt.wins, 0) AS wins,-- COALESCE(rt.losses, 0) AS losses, COALESCE(rt.ties, 0) AS ties,-- med.median_cost, med.median_ms-- FROM models m-- LEFT JOIN arena_ratings rt ON rt.category = m.category AND rt.model = m.model-- LEFT JOIN medians med ON med.category = m.category AND med.model = m.model-- ORDER BY m.category, elo DESC, games DESC, m.model`,-- [category, BASE_ELO],-- );-- const all = rows.map((r) => {-- const games = num(r.games);-- const wins = num(r.wins);-- const ties = num(r.ties);-- return {-- category: r.category,-- model: r.model,-- elo: Math.round(num(r.elo, BASE_ELO) * 100) / 100,-- games,-- wins,-- losses: num(r.losses),-- ties,-- // A tie is half a win, as in Elo's expected score.-- win_rate: games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0,-- median_cost_usd: r.median_cost == null ? null : Math.round(num(r.median_cost) * 1e6) / 1e6,-- median_cost: r.median_cost == null ? null : Math.round(num(r.median_cost) * 1e6) / 1e6,-- median_ms: r.median_ms == null ? null : Math.round(num(r.median_ms)),-- };-- });-- if (category) return all;-- return CATEGORIES.map((c) => ({ category: c, leaderboard: all.filter((r) => r.category === c) }));++ async function leaderboard(query = null) {++ const kindId = (!query || query === 'all') ? null : categoryToKind(query);++ if (kindId) {++ const elo = await bench.getBoard(kindId, { type: 'elo' });++ const rows = (elo.rows || []).map(boardRowToLocal);++ return [{ category: kindId === 'design.component' ? 'logo' : 'design.page', leaderboard: rows }];++ }++ const kinds = ['design.page', 'design.component'];++ const out = [];++ for (const kindId of kinds) {++ const elo = await bench.getBoard(kindId, { type: 'elo' });++ const category = kindId === 'design.component' ? 'logo' : 'design.page';++ out.push({ category, leaderboard: (elo.rows || []).map(boardRowToLocal) });++ }++ return out;}-- /** The open dataset: prompts, every output (HTML included) and every vote without its voter. */-- async function dataset() {-- const rounds = await db.many('SELECT id, category, prompt, status, created_at FROM arena_rounds ORDER BY created_at, id');-- const outputs = (await db.many('SELECT * FROM arena_outputs ORDER BY created_at, id')).map(outputOf);-- const votes = (await db.many('SELECT "round", left_output, right_output, winner, at FROM arena_votes ORDER BY at, "round"')).map(voteOf);-- const byRound = new Map(rounds.map((r) => [r.id, { ...r, outputs: [], votes: [] }]));-- for (const o of outputs) { const r = byRound.get(o.round); if (r) r.outputs.push(o); }-- for (const v of votes) { const r = byRound.get(v.round); if (r) r.votes.push(v); }-- return [...byRound.values()];++ async function costLeaderboard(query = null) {++ const kindId = (!query || query === 'all') ? null : categoryToKind(query);++ if (kindId) {++ const cost = await bench.getBoard(kindId, { type: 'cost' });++ return [{ category: kindId === 'design.component' ? 'logo' : 'design.page', leaderboard: (cost.rows || []).map(boardRowToLocal) }];++ }++ const kinds = ['design.page', 'design.component'];++ const out = [];++ for (const kindId of kinds) {++ const cost = await bench.getBoard(kindId, { type: 'cost' });++ const category = kindId === 'design.component' ? 'logo' : 'design.page';++ out.push({ category, leaderboard: (cost.rows || []).map(boardRowToLocal) });++ }++ return out;}async function datasetJsonl() {-- const lines = (await dataset()).map((r) => JSON.stringify(r));-- return lines.length ? `${lines.join('\n')}\n` : '';++ // The open dataset now lives on Bench; Design redirects to it.++ return { redirect: bench.datasetUrl('design.page') };}-- /** A current day's budget summary for the pages and the create answer. */return {-- CATEGORIES, WINNERS, K, BASE_ELO,-- counts, createRound, getRound, nextPair, vote, leaderboard, dataset, datasetJsonl, shuffle,++ CATEGORIES, WINNERS, K, BASE_ELO, shuffle,++ counts, createRound, getRound, nextPair, vote, leaderboard, costLeaderboard, datasetJsonl,};}@@@ -64,6 +64,15 @@ function load(env = process.env) {maxHtmlBytes: 200 * 1024,},++ // OpenVibe.Bench: Design creates tasks with its service token. A round uses its creator's Network token++ // so Bench can enforce personal budgets and prevent voting on one's own round.++ bench: {++ url: trim(env.OV_BENCH_INTERNAL_URL || 'http://127.0.0.1:5230'),++ publicUrl: trim(env.OV_BENCH_URL || 'https://bench.openvibe.services'),++ audience: env.DESIGN_BENCH_AUDIENCE || 'openvibe.bench',++ scope: env.DESIGN_BENCH_SCOPE || 'bench.task.create bench.round.create bench.round.read bench.judgement.cast bench.board.read',++ },++// Arena budgets and jobs. Both limits are per UTC day; the drill flag keeps the worker from starting.arena: {perDay: Math.max(1, int(env.DESIGN_ARENA_PER_DAY, 3)),@@@ -4,24 +4,29 @@* The Arena API, /api/v1/arena/*:** GET /categories the six categories-- * POST /rounds start a round (signed in, same-origin, daily budgets)++ * POST /rounds start a Bench task and round (signed in, same-origin)* GET /rounds/:id one round with its outputs* GET /vote the next blind pair for this person-- * POST /vote record a vote, reveal the two models and update Elo-- * GET /leaderboard[?category=…] Elo, games, win rate, median cost and ms-- * GET /dataset.jsonl every prompt, output and vote, without voter ids++ * POST /vote Bench records a vote and reveals the two models++ * GET /leaderboard[?type=cost] Bench Elo or Cost board++ * GET /dataset.jsonl redirect to Bench's design.page dataset*-- * A round is queued here; the job loop generates it. The request never waits for a model.++ * Bench queues and generates each round. The request never waits for a model.*/const { http } = require('openvibe-contracts');const { asyncRouter } = require('../http/router');const { ArenaError, CATEGORIES } = require('../arena/store');++const { BenchClientError } = require('../arena/bench');const { ownOrigin, requireSameOrigin } = require('./same-origin');function subjectOf(req) {return req.viewer && req.viewer.kind === 'user' && req.viewer.subject ? req.viewer.subject : null;}++function tokenOf(req) {++ return req.viewer && req.viewer.kind === 'user' ? req.viewer.token : null;++}++function createArenaApi(ctx) {const { arena, config, log } = ctx;const r = asyncRouter();@@@ -39,6 +44,7 @@ function createArenaApi(ctx) {const failed = (res, req, err) => {if (err instanceof ArenaError) return problem(res, req, err.status, err.code, err.detail);++ if (err instanceof BenchClientError) return problem(res, req, err.status >= 500 ? err.status : 502, err.code, err.detail);log.warn('[OpenVibe.Design] Arena API failed:', (err && err.message) || err);return problem(res, req, 500, 'internal.error', 'The Arena could not answer. Try again in a moment.');};@@@ -47,14 +53,20 @@ function createArenaApi(ctx) {async function leaderboard(req, res) {const category = req.query.category == null || req.query.category === '' ? null : String(req.query.category);++ const kind = req.query.kind == null || req.query.kind === '' ? null : String(req.query.kind);++ const type = req.query.type === 'cost' ? 'cost' : 'elo';if (category && !CATEGORIES.includes(category)) {return problem(res, req, 400, 'arena.invalid_category', `Pick a category: ${CATEGORIES.join(', ')}.`);}++ if (kind && kind !== 'design.page' && kind !== 'design.component') {++ return problem(res, req, 400, 'arena.invalid_category', 'Pick design.page or design.component.');++ }try {-- const list = await arena.leaderboard(category);++ const query = kind || category || null;++ const list = type === 'cost' ? await arena.costLeaderboard(query) : await arena.leaderboard(query);res.set('Cache-Control', 'public, max-age=30');-- if (category) return res.json({ category, leaderboard: list });-- return res.json({ categories: list, leaderboard: list.flatMap((c) => c.leaderboard) });++ if (query) return res.json({ category: query, type, leaderboard: list[0] ? list[0].leaderboard : [] });++ return res.json({ categories: list, type, leaderboard: list.flatMap((c) => c.leaderboard) });} catch (err) { return failed(res, req, err); }}r.get('/leaderboard', leaderboard);@@@ -62,7 +74,12 @@ function createArenaApi(ctx) {async function dataset(req, res) {try {-- const text = await arena.datasetJsonl();++ const out = await arena.datasetJsonl();++ if (out && out.redirect) {++ res.set('Cache-Control', 'public, max-age=60');++ return res.redirect(302, out.redirect);++ }++ const text = typeof out === 'string' ? out : '';res.set('Cache-Control', 'public, max-age=60');res.set('Content-Disposition', 'attachment; filename="arena-dataset.jsonl"');return res.type('application/x-ndjson').send(text);@@@ -74,7 +91,7 @@ function createArenaApi(ctx) {r.post('/rounds', requireOrigin, requireSubject, async (req, res) => {const b = req.body && typeof req.body === 'object' ? req.body : {};try {-- const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req) });++ const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req), token: tokenOf(req) });res.set('Cache-Control', 'private, no-store');res.set('Location', `/api/v1/arena/rounds/${round.id}`);return res.status(201).json({ round });@@@ -83,9 +100,9 @@ function createArenaApi(ctx) {async function readRound(req, res) {try {-- const round = await arena.getRound(String(req.params.id || ''));++ const round = await arena.getRound(String(req.params.id || ''), { token: tokenOf(req) });if (!round) return problem(res, req, 404, 'arena.round_not_found', 'No such Arena round.');-- res.set('Cache-Control', 'public, max-age=30');++ res.set('Cache-Control', tokenOf(req) ? 'private, no-store' : 'public, max-age=30');return res.json({ round });} catch (err) { return failed(res, req, err); }}@@@ -94,7 +111,9 @@ function createArenaApi(ctx) {async function nextPair(req, res) {try {-- const pair = await arena.nextPair(subjectOf(req), { random: ctx.random || Math.random });++ const category = String(req.query.category || 'landing');++ if (!CATEGORIES.includes(category)) return problem(res, req, 400, 'arena.invalid_category', 'Pick a valid category.');++ const pair = await arena.nextPair(subjectOf(req), { token: tokenOf(req), category, random: ctx.random || Math.random });res.set('Cache-Control', 'private, no-store');if (!pair) return res.json({ pair: null, detail: 'There is no round waiting for your vote. Start one, or come back when another is ready.' });return res.json(pair);@@@ -113,6 +132,7 @@ function createArenaApi(ctx) {leftOutput: String(b.left_output || b.left || ''),rightOutput: String(b.right_output || b.right || ''),winner: String(b.winner || ''),++ token: tokenOf(req),});res.set('Cache-Control', 'private, no-store');return res.status(201).json({@@@ -37,6 +37,7 @@ function meOf(viewer) {}const subjectOf = (req) => (req.viewer && req.viewer.kind === 'user' ? req.viewer.subject : null);++const tokenOf = (req) => (req.viewer && req.viewer.kind === 'user' ? req.viewer.token : null);function createPageRoutes(ctx) {const { config, catalog, arena, log } = ctx;@@@ -83,11 +84,16 @@ function createPageRoutes(ctx) {r.get('/arena', async (req, res) => {const me = meOf(req.viewer);let boards;-- try { boards = await arena.leaderboard(null); } catch (err) {++ let costBoards;++ try {++ boards = await arena.leaderboard(null);++ costBoards = await arena.costLeaderboard(null);++ } catch (err) {log.warn('[OpenVibe.Design] leaderboard failed:', (err && err.message) || err);boards = categories.map((category) => ({ category, leaderboard: [] }));++ costBoards = [];}-- return page(res, 200, views.arenaHome({ categories: boards, me, limits: await aria(req.viewer) }), req.viewer);++ return page(res, 200, views.arenaHome({ categories: boards, costCategories: costBoards, me, limits: await aria(req.viewer) }), req.viewer);});r.get('/arena/new', async (req, res) => {@@@ -100,7 +106,7 @@ function createPageRoutes(ctx) {if (!writeAllowed(req, config)) return res.status(403).type('text/plain').send('A round must be started from openvibe.design itself.');const b = req.body || {};try {-- const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req) });++ const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req), token: tokenOf(req) });return res.redirect(303, `/arena/r/${encodeURIComponent(round.id)}`);} catch (err) {const notice = err instanceof ArenaError ? err.detail : 'The round could not be started. Try again in a moment.';@@@ -115,7 +121,7 @@ function createPageRoutes(ctx) {let pair = null;let notice = null;if (subjectOf(req)) {-- try { pair = await arena.nextPair(subjectOf(req), { random: ctx.random || Math.random }); } catch (err) {++ try { pair = await arena.nextPair(subjectOf(req), { token: tokenOf(req), category: req.query.category === 'logo' ? 'logo' : 'landing', random: ctx.random || Math.random }); } catch (err) {log.warn('[OpenVibe.Design] picking a pair failed:', (err && err.message) || err);notice = 'The next pair could not be read. Try again in a moment.';}@@@ -135,6 +141,7 @@ function createPageRoutes(ctx) {leftOutput: String(b.left_output || ''),rightOutput: String(b.right_output || ''),winner: String(b.winner || ''),++ token: tokenOf(req),});return page(res, 201, views.arenaVote({ me, result: {vote: { round: out.round.id, winner: out.winner, at: out.at },@@@ -149,7 +156,7 @@ function createPageRoutes(ctx) {r.get('/arena/r/:id', async (req, res) => {let round = null;-- try { round = await arena.getRound(String(req.params.id || '')); } catch (err) {++ try { round = await arena.getRound(String(req.params.id || ''), { token: tokenOf(req) }); } catch (err) {log.warn('[OpenVibe.Design] round page failed:', (err && err.message) || err);}if (!round) return page(res, 404, views.notFound(), req.viewer);@@@ -161,6 +161,8 @@ const ARENA_CATEGORY = {game_ui: 'Game UI',logo: 'Logo',slides: 'Slides',++ 'design.page': 'Design page arena',++ 'design.component': 'Design component arena',};const arenaCost = (v) => `$${Number(v || 0).toFixed(4)}`;@@@ -189,24 +191,41 @@ function arenaViewSwitch() {<input type="radio" id="ar-view-phone" name="ar_view" value="phone" form="ar-view-none"><label for="ar-view-phone">Phone</label></div>`;}--function arenaLeaderboard(category, rows) {-- const body = rows.length ? rows.map((r, i) => `<tr class="ar-lb-row" data-model="${esc(r.model)}">++function arenaLeaderboard(category, rows, { type = 'elo' } = {}) {++ if (type === 'cost') {++ const body = rows.length ? rows.map((r, i) => `<tr class="ar-lb-row" data-model="${esc(r.model)}"><td class="ar-lb-rank">${i + 1}</td><td class="ar-lb-model">${esc(r.model)}</td>--<td class="ar-lb-elo">${Number(r.elo).toFixed(0)}</td>++<td class="ar-lb-entries">${r.entries}</td>++<td class="ar-lb-usable">${r.usable}</td>++<td class="ar-lb-total-cost">${arenaCost(r.total_cost_usd)}</td>++<td class="ar-lb-solved">${r.solved}</td>++<td class="ar-lb-cost-per-solved">${r.cost_per_solved == null ? '—' : arenaCost(r.cost_per_solved)}</td>++<td class="ar-lb-cost">${r.median_cost_usd == null ? '—' : arenaCost(r.median_cost_usd)}</td>++<td class="ar-lb-ms">${r.median_ms == null ? '—' : arenaMs(r.median_ms)}</td></tr>`).join('') : '<tr class="ar-lb-empty"><td colspan="9">No entries yet. Start a round to generate some.</td></tr>';++ return `<section class="ar-lb ar-lb--cost" id="ar-lb-cost-${esc(category)}" aria-labelledby="ar-lb-cost-h-${esc(category)}">++<div class="ar-lb-head"><h2 id="ar-lb-cost-h-${esc(category)}">${esc(ARENA_CATEGORY[category] || category)} · Cost</h2></div>++<table class="ar-table"><thead><tr><th scope="col">#</th><th scope="col">Model</th><th scope="col">Entries</th><th scope="col">Usable</th><th scope="col">Total cost</th><th scope="col">Solved</th><th scope="col">Cost per solved</th><th scope="col">Median cost</th><th scope="col">Median time</th></tr></thead>++<tbody>${body}</tbody></table></section>`;++ }++ const body = rows.length ? rows.map((r, i) => `<tr class="ar-lb-row" data-model="${esc(r.model)}">++<td class="ar-lb-rank">${i + 1}</td>++<td class="ar-lb-model">${esc(r.model)}${r.provisional ? ' <span class="ar-provisional" title="Fewer than 10 games">provisional</span>' : ''}</td>++<td class="ar-lb-elo">${Number(r.elo).toFixed(0)}${r.ci_lower != null && r.ci_upper != null ? ` <span class="ar-ci">(${Number(r.ci_lower).toFixed(0)}–${Number(r.ci_upper).toFixed(0)})</span>` : ''}</td><td class="ar-lb-games">${r.games}</td><td class="ar-lb-winrate">${arenaPct(r.win_rate)}</td><td class="ar-lb-cost">${r.median_cost_usd == null ? '—' : arenaCost(r.median_cost_usd)}</td><td class="ar-lb-ms">${r.median_ms == null ? '—' : arenaMs(r.median_ms)}</td></tr>`).join('') : '<tr class="ar-lb-empty"><td colspan="7">No votes yet. Be the first to judge this category.</td></tr>';return `<section class="ar-lb" id="ar-lb-${esc(category)}" aria-labelledby="ar-lb-h-${esc(category)}">--<div class="ar-lb-head"><h2 id="ar-lb-h-${esc(category)}">${esc(ARENA_CATEGORY[category] || category)}</h2><a class="ar-more" href="/arena/vote">Vote on this category</a></div>++<div class="ar-lb-head"><h2 id="ar-lb-h-${esc(category)}">${esc(ARENA_CATEGORY[category] || category)}</h2><a class="ar-more" href="/arena/vote${category === 'logo' ? '?category=logo' : ''}">Vote on this category</a></div><table class="ar-table"><thead><tr><th scope="col">#</th><th scope="col">Model</th><th scope="col">Elo</th><th scope="col">Games</th><th scope="col">Win rate</th><th scope="col">Median cost</th><th scope="col">Median time</th></tr></thead><tbody>${body}</tbody></table></section>`;}--function arenaHome({ categories = [], me = {}, limits = null } = {}) {++function arenaHome({ categories = [], costCategories = [], me = {}, limits = null } = {}) {const signedIn = Boolean(me.signedIn);const boards = categories.map((c) => arenaLeaderboard(c.category, c.leaderboard || [])).join('');++ const costBoards = costCategories.map((c) => arenaLeaderboard(c.category, c.leaderboard || [], { type: 'cost' })).join('');const rows = categories.flatMap((c) => c.leaderboard || []);const body = `<div class="ar-page" id="ar-home"><header class="ar-hero"><p class="ar-kicker">Design Arena</p><h1>Which model designs it best?</h1>@@@ -217,7 +236,9 @@ function arenaHome({ categories = [], me = {}, limits = null } = {}) {: 'Sign in to start a round or vote. The leaderboard and the dataset are open to everyone.'}</p></header><section class="ar-boards" id="ar-leaderboard" aria-labelledby="ar-boards-h"><div class="ar-lb-head"><h2 id="ar-boards-h">The leaderboard</h2><span class="ar-fine">${rows.length} model${rows.length === 1 ? '' : 's'} ranked · K = 32</span></div>${boards || '<p class="ar-empty">No rounds have been judged yet. Start the first one.</p>'}--<p class="ar-fine">A win rate counts a tie as half a win. <code>both_bad</code> is a game for neither model. The cost and time are the median over the outputs in this category.</p></section>++<p class="ar-fine">A win rate counts a tie as half a win. <code>both_bad</code> is a game for neither model. The cost and time are the median over the outputs. Ranges are 95% confidence intervals; <span class="ar-provisional">provisional</span> means fewer than 10 games.</p></section>++<section class="ar-boards ar-boards--cost" id="ar-cost" aria-labelledby="ar-cost-h"><div class="ar-lb-head"><h2 id="ar-cost-h">Cost view</h2><span class="ar-fine">Per model across all entries</span></div>++${costBoards || '<p class="ar-empty">No entries yet.</p>'}</section><section class="ar-open" id="ar-open"><h2>Everything is open</h2><p class="ar-lede">The prompts, every generated page and every vote are downloadable. No voter is named.</p><p><a class="ovk-btn ovk-btn--quiet" href="/api/v1/arena/dataset.jsonl">Download dataset.jsonl</a></p></section></div>`;@@@ -27,19 +27,15 @@ const bodyOf = (ev) => JSON.stringify({ event: ev, seq: 1 });const ana = t.network.addUser('ana');const bo = t.network.addUser('bo');try {-- // A starts a round (then it is given A's fixed account subject), and B votes on it.-- const aRound = (await t.get('/api/v1/arena/rounds', { as: ana, json: { category: 'landing', prompt: 'A round to export' }, origin: ORIGIN })).json().round;-- await t.tick();-- await t.ctx.s.db.exec('UPDATE arena_rounds SET created_by = $1 WHERE id = $2', [A, aRound.id]);-- const aOutputs = (await t.ctx.arena.getRound(aRound.id)).outputs;-- const pair = await t.ctx.arena.nextPair(bo.subject);-- const vote = await t.get('/api/v1/arena/vote', { as: bo, json: { round_id: pair.round.id, left_output: pair.left.id, right_output: pair.right.id, winner: 'left' }, origin: ORIGIN });-- assert.strictEqual(vote.status, 201, vote.text);---- // B starts a round and A's vote is stored on it directly (A is a fixture account, not a Network token here).-- const bRound = (await t.get('/api/v1/arena/rounds', { as: bo, json: { category: 'dashboard', prompt: 'B round stays' }, origin: ORIGIN })).json().round;-- await t.tick();-- const bOutputs = (await t.ctx.arena.getRound(bRound.id)).outputs;++ // Legacy Arena tables stay read-only in the active flow. Seed historical data to exercise ADR-033.++ const aRound = { id: 'arr_legacy_a' };++ const bRound = { id: 'arr_legacy_b' };++ const db = t.ctx.s.db;++ await db.exec("INSERT INTO arena_rounds (id, category, prompt, created_by, status) VALUES ($1, 'landing', 'A round to export', $2, 'ready'), ($3, 'dashboard', 'B round stays', $4, 'ready')", [aRound.id, A, bRound.id, B]);++ await db.exec("INSERT INTO arena_outputs (id, \"round\", model, provider, html) VALUES ('aro_legacy_a1', $1, 'model-a', 'ai', '<h1>A</h1>'), ('aro_legacy_a2', $1, 'model-b', 'ai', '<h1>B</h1>'), ('aro_legacy_b1', $2, 'model-a', 'ai', '<h1>C</h1>'), ('aro_legacy_b2', $2, 'model-b', 'ai', '<h1>D</h1>')", [aRound.id, bRound.id]);++ const aOutputs = await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 ORDER BY id', [aRound.id]);++ const bOutputs = await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 ORDER BY id', [bRound.id]);++ await db.exec("INSERT INTO arena_votes (\"round\", voter_subject, left_output, right_output, winner) VALUES ($1, $2, $3, $4, 'left')", [aRound.id, bo.subject, aOutputs[0].id, aOutputs[1].id]);await t.ctx.s.db.exec(`INSERT INTO arena_votes ("round", voter_subject, left_output, right_output, winner) VALUES ($1, $2, $3, $4, 'tie')`,[bRound.id, A, bOutputs[0].id, bOutputs[1].id]);nnew file mode 100644@@@ -0,0 +1,50 @@++'use strict';++const assert = require('assert');++const { createBenchClient } = require('../server/arena/bench');++const { check, done } = require('./helpers/boot');++++(async () => {++ const calls = [];++ const fetchImpl = async (url, opts) => {++ const path = new URL(url).pathname + new URL(url).search;++ calls.push({ path, opts });++ let body = {};++ if (path === '/oauth/token') body = { access_token: 'service-token', token_type: 'Bearer', expires_in: 300 };++ if (path === '/api/v1/kinds/design.page/tasks') body = { task: { id: 'btk_1', prompt: 'Page', task_class: 'landing' } };++ if (path === '/api/v1/tasks/btk_1/rounds') body = { round: { id: 'brd_1' } };++ if (path === '/api/v1/judgements/next?kind=design.page') body = { round: { id: 'brd_1' }, left: { id: 'bte_1', payload: { html: 'a' } }, right: { id: 'bte_2', payload: { html: 'b' } } };++ if (path === '/api/v1/rounds/brd_1/judgements') body = { judgement: { at: 'now' }, revealed: {} };++ return { ok: true, status: 200, json: async () => body };++ };++ const config = { bench: { url: 'http://bench.test', publicUrl: 'https://bench.openvibe.services', audience: 'openvibe.bench', scope: 'bench.task.create' },++ oauth: { clientId: 'design', clientSecret: 'fixture' }, networkInternalUrl: 'http://network.test' };++ const client = createBenchClient({ config, fetchImpl });++ await check('task uses service token and round uses the person token', async () => {++ const result = await client.createRound({ category: 'landing', prompt: 'Page', token: 'person-token' });++ assert.strictEqual(result.round.id, 'brd_1');++ const task = calls.find((c) => c.path === '/api/v1/kinds/design.page/tasks');++ const round = calls.find((c) => c.path === '/api/v1/tasks/btk_1/rounds');++ assert.strictEqual(task.opts.headers.Authorization, 'Bearer service-token');++ assert.strictEqual(round.opts.headers.Authorization, 'Bearer person-token');++ assert.deepStrictEqual(JSON.parse(round.opts.body), {});++ });++ await check('Bench pair and vote use the person token and exact wire shape', async () => {++ const pair = await client.nextPair('design.page', 'person-token');++ assert.strictEqual(pair.round.id, 'brd_1');++ await client.castVote({ roundId: 'brd_1', token: 'person-token', pair: { left: 'bte_1', right: 'bte_2' }, winner: 'left' });++ const vote = calls.find((c) => c.path === '/api/v1/rounds/brd_1/judgements');++ assert.strictEqual(vote.opts.headers.Authorization, 'Bearer person-token');++ assert.deepStrictEqual(JSON.parse(vote.opts.body), { pair: { left: 'bte_1', right: 'bte_2' }, outcome: 'left' });++ assert.strictEqual(client.datasetUrl('design.page'), 'https://bench.openvibe.services/api/v1/datasets/design.page.jsonl');++ });++ await check('imported votes use Bench service judgements without person identity', async () => {++ await client.castImportedVote({ roundId: 'brd_1', pair: { left: 'bte_1', right: 'bte_2' }, winner: 'both_bad' });++ const vote = calls.filter((c) => c.path === '/api/v1/rounds/brd_1/judgements').at(-1);++ assert.strictEqual(vote.opts.headers.Authorization, 'Bearer service-token');++ assert.deepStrictEqual(JSON.parse(vote.opts.body), {++ judge: 'automatic_grader', judge_name: 'design.arena.legacy_vote',++ pair: { left: 'bte_1', right: 'bte_2' }, outcome: 'both_bad',++ });++ });++ done();++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,51 @@++'use strict';++const assert = require('assert');++const { migrate } = require('../scripts/arena-to-bench');++const { check, done } = require('./helpers/boot');++++const rows = {++ rounds: [{ id: 'arr_a', category: 'landing', prompt: 'A page', status: 'ready' }],++ outputs: [++ { id: 'aro_a', round: 'arr_a', provider: 'ai', model: 'alpha', html: '<h1>A</h1>', cost_usd: 0.01, ms: 3, tokens: 4 },++ { id: 'aro_b', round: 'arr_a', provider: 'ai', model: 'beta', html: '<h1>B</h1>', cost_usd: 0.02, ms: 5, tokens: 7 },++ ],++ votes: [{ round: 'arr_a', left_output: 'aro_a', right_output: 'aro_b', winner: 'tie' }],++};++const db = (source) => ({ many: async (sql) => source[sql.includes('arena_rounds') ? 'rounds' : sql.includes('arena_outputs') ? 'outputs' : 'votes'] });++++(async () => {++ await check('migration dry run reads legacy tables and writes nothing', async () => {++ const result = await migrate({ db: db(rows), bench: null });++ assert.deepStrictEqual(result, { dry_run: true, rounds: 1, entries: 2, judgements: 1, problems: [] });++ });++ await check('migration maps entries and casts an anonymous service judgement', async () => {++ const calls = [];++ const bench = {++ createRound: async (input) => { calls.push(['round', input]); return { round: { id: 'brd_1' } }; },++ addEntry: async (input) => { calls.push(['entry', input]); return { entry: { id: `bte_${calls.length}` } }; },++ castImportedVote: async (input) => { calls.push(['vote', input]); },++ };++ await migrate({ db: db(rows), bench, apply: true });++ assert.strictEqual(calls[0][1].category, 'landing');++ assert.strictEqual(calls[1][1].payload.html, '<h1>A</h1>');++ assert.deepStrictEqual(calls[3][1].pair, { left: 'bte_2', right: 'bte_3' });++ assert.strictEqual(calls[3][1].winner, 'tie');++ assert.ok(!JSON.stringify(calls).includes('voter_subject'));++ });++ await check('migration retains multiple votes and both_bad without voter identities', async () => {++ const source = { ...rows, votes: [...rows.votes, { ...rows.votes[0], winner: 'both_bad' }] };++ const winners = [];++ const bench = {++ createRound: async () => ({ round: { id: 'brd_1' } }),++ addEntry: async (entry) => ({ entry: { id: entry.model } }),++ castImportedVote: async (vote) => winners.push(vote.winner),++ };++ await migrate({ db: db(source), bench, apply: true });++ assert.deepStrictEqual(winners, ['tie', 'both_bad']);++ });++ await check('invalid legacy references stop before any Bench write', async () => {++ const source = { ...rows, votes: [...rows.votes, { ...rows.votes[0], left_output: 'missing' }] };++ await assert.rejects(() => migrate({ db: db(source), bench: { createRound: () => { throw new Error('must not write'); } }, apply: true }), /Import refused/);++ });++ done();++})().catch((err) => { console.error(err); process.exit(1); });@@@ -1,342 +1,101 @@'use strict';--/**-- * The Arena end to end: a round generates three outputs through the job, the sanitizer is applied before storage,-- * the budgets refuse in words, the pair is blind, one vote per person, the creator cannot vote, Elo handles a win,-- * a tie and both_bad, the leaderboard reports games/win rate/medians, and the dataset names no voter.-- */const assert = require('assert');const { boot, check, done } = require('./helpers/boot');--const ORIGIN = 'https://openvibe.design';(async () => {-- const t = await boot({ random: () => 0, env: { DESIGN_ARENA_PER_DAY: '8', DESIGN_ARENA_DAILY: '40' } });++ const t = await boot();const ana = t.network.addUser('ana');const bo = t.network.addUser('bo');-- const cy = t.network.addUser('cy');---- const create = (user, category = 'landing', prompt = 'A landing page for a tiny coffee roastery in Seattle', extra = {}) => t.get('/api/v1/arena/rounds', {-- as: user, json: { category, prompt }, origin: ORIGIN, ...extra,-- });-- const getRound = async (id) => (await t.get(`/api/v1/arena/rounds/${id}`)).json().round;-- // Extra rounds would compete with A for the next blind pair; the checks that make them clean up after themselves.-- const dropRounds = async (...ids) => { for (const id of ids.filter(Boolean)) await t.ctx.s.db.exec('DELETE FROM arena_rounds WHERE id = $1', [id]); };---- let A;-- await check('a round is queued by the request, and no AI call happens in the request', async () => {-- const before = t.ai.requests.length;-- const r = await create(ana);-- assert.strictEqual(r.status, 201, r.text);-- A = r.json().round;-- assert.strictEqual(A.status, 'queued');-- assert.deepStrictEqual(A.outputs, []);-- assert.strictEqual(t.ai.requests.length, before, 'the job, not the request, calls AI');-- });---- await check('the job generates three outputs through AI with its reported model, provider, cost, time and tokens', async () => {-- const ready = await t.tick();-- assert.ok(ready, 'a queued round was claimed');-- assert.strictEqual(ready.id, A.id);-- assert.strictEqual(ready.status, 'ready');-- assert.strictEqual(ready.outputs.length, 3);-- const models = ready.outputs.map((o) => o.model);-- const providers = ready.outputs.map((o) => o.provider);-- assert.deepStrictEqual(models, ['alpha-1', 'beta-2', 'gamma-3']);-- assert.deepStrictEqual(providers, ['acme', 'beta', 'gamma']);-- for (const o of ready.outputs) {-- assert.ok(o.id.startsWith('aro_'));-- assert.strictEqual(o.error, null);-- assert.match(o.html, /^<!doctype html>/i);-- assert.strictEqual(o.cost_usd, 0.0012);-- assert.strictEqual(o.ms, 321);-- assert.strictEqual(o.tokens, 600);-- }-- assert.strictEqual(ready.pool_fallback, false, 'three offers were listed, so no pool fallback');-- assert.deepStrictEqual(ready.offers, [-- { provider: 'acme', model: 'alpha-1' },-- { provider: 'beta', model: 'beta-2' },-- { provider: 'gamma', model: 'gamma-3' },-- ]);-- const posts = t.ai.requests.filter((r) => r.method === 'POST' && r.path === '/api/v1/generate');-- assert.strictEqual(posts.length, 3);-- for (const [i, p] of posts.entries()) {-- assert.ok(p.body.options.cache === false);-- assert.strictEqual(p.body.options.model, undefined, 'the pin is a route, not an options.model hint');-- assert.deepStrictEqual(p.body.route, ready.offers[i], 'each slot asks for one distinct offer');-- assert.match(p.body.prompt, /coffee roastery/);-- assert.strictEqual(p.body.target.service, 'design');-- }-- });---- await check('Design asked Network for a client-credentials token for openvibe.ai with ai.run.create', async () => {-- const token = t.network.state.tokenRequests.find((f) => f.grant_type === 'client_credentials');-- assert.ok(token, 'a client-credentials grant was made');-- assert.strictEqual(token.client_id, 'design');-- assert.strictEqual(token.audience, 'openvibe.ai');-- assert.strictEqual(token.scope, 'ai.run.create');-- assert.ok(t.ai.state.tokenClaims);-- assert.ok(t.ai.state.tokenClaims.aud.includes('openvibe.ai'));-- assert.ok(t.ai.state.tokenClaims.cap.includes('ai.run.create'));-- assert.ok(t.ai.state.tokenClaims.ns.includes('ai.*'));-- });---- await check('the sanitizer runs before storage, including the 200 KB cap', async () => {-- t.ai.state.htmlFor = () => '<h1>safe</h1><script>alert(1)</script><img src="x" onerror="alert(2)"><a href="javascript:alert(3)">x</a>';-- const r = await create(ana, 'logo', 'A mark with a script');-- const scripted = r.json().round.id;-- await t.tick();-- const round = await getRound(scripted);-- for (const o of round.outputs) {-- assert.match(o.html, /safe/);-- assert.ok(!/<script/i.test(o.html));-- assert.ok(!/\sonerror\s*=/i.test(o.html));-- assert.ok(!/javascript\s*:/i.test(o.html));-- }-- t.ai.state.htmlFor = () => `<p>${'x'.repeat(300 * 1024)}</p>`;-- const big = await create(ana, 'logo', 'A big page');-- const bigId = big.json().round.id;-- await t.tick();-- const bigRound = await getRound(bigId);-- for (const o of bigRound.outputs) assert.ok(Buffer.byteLength(o.html) <= 200 * 1024, `${Buffer.byteLength(o.html)} bytes`);-- t.ai.state.htmlFor = null;-- await dropRounds(scripted, bigId);-- });---- await check('bad input and a cross-site Origin are refused in words', async () => {-- const signedOut = await t.get('/api/v1/arena/rounds', { json: { category: 'landing', prompt: 'x' } });-- assert.strictEqual(signedOut.status, 401);-- const badCat = await create(ana, 'poster');-- assert.strictEqual(badCat.status, 422);-- assert.match(badCat.json().detail, /category/i);-- const empty = await create(ana, 'landing', ' ');-- assert.strictEqual(empty.status, 422);-- assert.match(empty.json().detail, /describe/i);-- const cross = await create(ana, 'landing', 'x', { origin: 'https://evil.example' });-- assert.strictEqual(cross.status, 403);-- assert.strictEqual(cross.json().code, 'design.cross_site');-- });---- await check('a cookie write needs same-origin: missing headers are refused, Sec-Fetch-Site or Origin passes', async () => {-- const missing = await create(ana, 'landing', 'no header', { origin: null });-- assert.strictEqual(missing.status, 403);-- assert.strictEqual(missing.json().code, 'design.cross_site');-- assert.match(missing.json().detail, /openvibe\.design itself/);-- const byFetchSite = await create(ana, 'landing', 'fetch metadata', { origin: null, headers: { 'sec-fetch-site': 'same-origin' } });-- assert.strictEqual(byFetchSite.status, 201, byFetchSite.text);-- const crossSite = await create(ana, 'landing', 'cross site', { origin: null, headers: { 'sec-fetch-site': 'cross-site' } });-- assert.strictEqual(crossSite.status, 403);-- // A Bearer write is not cookie-authenticated: no Origin is needed.-- const bearer = await t.get('/api/v1/arena/rounds', {-- method: 'POST', json: { category: 'landing', prompt: 'a bearer write' },-- headers: { authorization: `Bearer ${t.network.userToken(ana)}` },++ const create = (as, category, prompt) => t.get('/api/v1/arena/rounds', { as, json: { category, prompt }, origin: ORIGIN });++ try {++ let round;++ await check('a signed-in start creates a Bench task and person-owned round', async () => {++ const r = await create(ana, 'landing', 'A coffee roastery landing page');++ assert.strictEqual(r.status, 201, r.text);++ round = r.json().round;++ assert.match(round.id, /^brd_/);++ assert.strictEqual(round.category, 'landing');++ assert.strictEqual(round.status, 'queued');++ const task = [...t.bench.state.tasks.values()][0];++ assert.strictEqual(task.kind_id, 'design.page');++ assert.strictEqual(task.task_class, 'landing');++ assert.strictEqual(t.bench.state.rounds.get(round.id).created_by, ana.subject);++ const taskCall = t.bench.state.requests.find((x) => x.path.endsWith('/tasks') && x.method === 'POST');++ const roundCall = t.bench.state.requests.find((x) => x.path.endsWith('/rounds') && x.method === 'POST');++ assert.ok(taskCall && roundCall);++ assert.notStrictEqual(taskCall.authorization, roundCall.authorization);++ const grant = t.network.state.tokenRequests.find((x) => x.audience === 'openvibe.bench');++ assert.ok(grant);++ assert.match(grant.scope, /bench.task.create/);});-- assert.strictEqual(bearer.status, 201, bearer.text);-- await dropRounds(byFetchSite.json().round.id, bearer.json().round.id);-- });---- await check('the no-JS forms enforce the same policy', async () => {-- const form = { 'content-type': 'application/x-www-form-urlencoded' };-- const body = 'category=landing&prompt=a+form+round';-- const missing = await t.get('/arena/new', { as: ana, method: 'POST', body, headers: form });-- assert.strictEqual(missing.status, 403);-- const same = await t.get('/arena/new', { as: ana, method: 'POST', body, headers: { ...form, 'sec-fetch-site': 'same-origin' } });-- assert.strictEqual(same.status, 303, same.text);-- assert.match(same.headers.get('location'), /^\/arena\/r\/arr_/);-- const cross = await t.get('/arena/new', { as: ana, method: 'POST', body, headers: { ...form, origin: 'https://evil.example' } });-- assert.strictEqual(cross.status, 403);-- await dropRounds(String(same.headers.get('location')).split('/').pop());-- });---- let winner;-- let loser;-- await check('the pair is blind: no model, provider, cost or time before the vote', async () => {-- const pair = await t.get('/api/v1/arena/vote', { as: bo });-- assert.strictEqual(pair.status, 200, pair.text);-- const p = pair.json();-- assert.strictEqual(p.round.id, A.id);-- assert.deepStrictEqual(Object.keys(p.left).sort(), ['html', 'id']);-- assert.deepStrictEqual(Object.keys(p.right).sort(), ['html', 'id']);-- assert.ok(!JSON.stringify(p).includes('alpha-1'));-- assert.ok(!JSON.stringify(p).includes('beta-2'));-- const creator = await t.get('/api/v1/arena/vote', { as: ana });-- assert.strictEqual(creator.json().pair, null, 'the creator cannot be offered their own round');-- t.state = p;-- });---- await check('the creator cannot vote on their own round', async () => {-- const p = t.state;-- const r = await t.get('/api/v1/arena/vote', { as: ana, json: { round_id: A.id, left_output: p.left.id, right_output: p.right.id, winner: 'left' }, origin: ORIGIN });-- assert.strictEqual(r.status, 403);-- assert.strictEqual(r.json().code, 'arena.creator_cannot_vote');-- });-- await check('bo votes once; the models are revealed and the vote is recorded', async () => {-- const p = t.state;-- const r = await t.get('/api/v1/arena/vote', { as: bo, json: { round_id: A.id, left_output: p.left.id, right_output: p.right.id, winner: 'left' }, origin: ORIGIN });-- assert.strictEqual(r.status, 201, r.text);-- const out = r.json();-- assert.strictEqual(out.vote.winner, 'left');-- assert.ok(out.revealed.left.model);-- assert.ok(out.revealed.right.model);-- const leftModel = out.revealed.left.model;-- const rightModel = out.revealed.right.model;-- assert.notStrictEqual(leftModel, rightModel);-- winner = leftModel;-- loser = rightModel;-- const again = await t.get('/api/v1/arena/vote', { as: bo, json: { round_id: A.id, left_output: p.left.id, right_output: p.right.id, winner: 'right' }, origin: ORIGIN });-- assert.strictEqual(again.status, 409);-- assert.strictEqual(again.json().code, 'arena.already_voted');-- });---- await check('the leaderboard updates Elo K=32, games and win rate, with median cost and time', async () => {-- const lb = (await t.get('/api/v1/arena/leaderboard?category=landing')).json().leaderboard;-- const w = lb.find((r) => r.model === winner);-- const l = lb.find((r) => r.model === loser);-- assert.ok(w && l);-- assert.strictEqual(w.elo, 1016);-- assert.strictEqual(l.elo, 984);-- assert.strictEqual(w.games, 1);-- assert.strictEqual(w.wins, 1);-- assert.strictEqual(w.losses, 0);-- assert.strictEqual(w.win_rate, 1);-- assert.strictEqual(l.games, 1);-- assert.strictEqual(l.wins, 0);-- assert.strictEqual(l.losses, 1);-- assert.strictEqual(l.win_rate, 0);-- assert.strictEqual(w.median_cost_usd, 0.0012);-- assert.strictEqual(w.median_ms, 321);-- });---- await check('a tie is half a win for each; both_bad is a game for neither', async () => {-- const b = (await create(bo, 'dashboard', 'An analytics dashboard')).json().round;-- // Earlier same-origin tests queued rounds too: drain until this one is generated.-- let bRound;-- for (let i = 0; i < 12; i++) {-- bRound = await getRound(b.id);-- if (bRound.status === 'ready' || bRound.status === 'failed') break;++ await check('Bench generates entries; the creator can read a sanitized round', async () => {await t.tick();-- }-- assert.strictEqual(bRound.status, 'ready');-- const tie = await t.ctx.arena.vote({ roundId: b.id, subject: cy.subject, leftOutput: bRound.outputs[0].id, rightOutput: bRound.outputs[1].id, winner: 'tie' });-- assert.strictEqual(tie.winner, 'tie');-- const lb = (await t.get('/api/v1/arena/leaderboard?category=dashboard')).json().leaderboard;-- const tied = lb.filter((r) => [bRound.outputs[0].model, bRound.outputs[1].model].includes(r.model));-- for (const row of tied) {-- assert.strictEqual(row.games, 1);-- assert.strictEqual(row.ties, 1);-- assert.strictEqual(row.wins, 0);-- assert.strictEqual(row.win_rate, 0.5);-- assert.strictEqual(row.elo, 1000, 'equal ratings tie to the same Elo');-- }---- const c = (await create(cy, 'game_ui', 'A game HUD')).json().round;-- await t.tick();-- const cRound = await getRound(c.id);-- const before = new Map(lb.map((r) => [r.model, r.games]));-- const both = await t.ctx.arena.vote({ roundId: c.id, subject: ana.subject, leftOutput: cRound.outputs[0].id, rightOutput: cRound.outputs[1].id, winner: 'both_bad' });-- assert.strictEqual(both.winner, 'both_bad');-- const after = (await t.get('/api/v1/arena/leaderboard?category=game_ui')).json().leaderboard;-- for (const row of after) {-- assert.strictEqual(row.games, 0, 'both_bad counted a game for neither');-- assert.strictEqual(row.elo, 1000);-- assert.strictEqual(row.ties, 0);-- }-- assert.ok(before.size >= 0);-- });---- await check('the dataset has prompts, output HTML and votes, and no voter ids', async () => {-- const ds = await t.get('/api/v1/arena/dataset.jsonl');-- assert.strictEqual(ds.status, 200);-- assert.match(ds.headers.get('content-type'), /ndjson|jsonl/);-- const lines = ds.text.trim().split('\n').map((l) => JSON.parse(l));-- assert.ok(lines.length >= 3);-- const a = lines.find((r) => r.id === A.id);-- assert.strictEqual(a.prompt, 'A landing page for a tiny coffee roastery in Seattle');-- assert.strictEqual(a.outputs.length, 3);-- assert.ok(a.outputs.every((o) => typeof o.html === 'string' && o.html.length));-- assert.strictEqual(a.votes.length, 1);-- assert.strictEqual(a.votes[0].winner, 'left');-- assert.ok(!ds.text.includes(ana.subject));-- assert.ok(!ds.text.includes(bo.subject));-- assert.ok(!ds.text.includes(cy.subject));-- assert.ok(!/voter_subject|"voter"/.test(ds.text));-- });++ const r = await t.get(`/api/v1/arena/rounds/${round.id}`, { as: ana });++ assert.strictEqual(r.status, 200, r.text);++ const ready = r.json().round;++ assert.strictEqual(ready.status, 'ready');++ assert.strictEqual(ready.outputs.length, 3);++ assert.ok(ready.outputs.every((o) => o.model && /<html>/i.test(o.html)));++ assert.strictEqual(Number(await t.ctx.s.db.value('SELECT COUNT(*) FROM arena_rounds')), 0);++ });-- await check('the pages render the leaderboard, the form and the frames with the CSP and both widths', async () => {-- const home = await t.get('/arena');-- assert.strictEqual(home.status, 200);-- assert.match(home.text, /id="ar-leaderboard"/);-- assert.match(home.text, /id="ar-lb-landing"/);-- assert.match(home.text, new RegExp(winner.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')));-- const round = await t.get(`/arena/r/${A.id}`);-- assert.strictEqual(round.status, 200);-- assert.match(round.text, /sandbox=""/);-- assert.match(round.text, /default-src 'none'/);-- assert.match(round.text, /width="390"/);-- assert.match(round.text, /width="1280"/);-- assert.match(round.text, /Phone · 390 px/);-- assert.match(round.text, /Desktop · 1280 px/);-- assert.doesNotMatch(round.text, /allow-scripts/);-- assert.doesNotMatch(round.text, /allow-same-origin/);-- const form = await t.get('/arena/new', { as: ana });-- assert.match(form.text, /id="ar-new-form"/);-- assert.match(form.text, /id="ar-category"/);-- assert.match(form.text, /id="ar-prompt"/);-- const anonVote = await t.get('/arena/vote');-- assert.match(anonVote.text, /Sign in to vote/);-- const gone = await t.get('/arena/r/nope');-- assert.strictEqual(gone.status, 404);-- });++ let pair;++ await check('Bench supplies a blind pair and excludes its creator', async () => {++ const r = await t.get('/api/v1/arena/vote', { as: bo });++ assert.strictEqual(r.status, 200, r.text);++ pair = r.json();++ assert.strictEqual(pair.round.id, round.id);++ assert.deepStrictEqual(Object.keys(pair.left).sort(), ['html', 'id']);++ assert.ok(!JSON.stringify(pair).includes('alpha-1'));++ const own = await t.get('/api/v1/arena/vote', { as: ana });++ assert.strictEqual(own.json().pair, null);++ });-- await check('only when fewer than three offers are listed does a round fall back to the pool, and say so', async () => {-- const two = await boot({ aiOptions: { models: [-- { provider: 'acme', model: 'alpha-1' },-- { provider: 'beta', model: 'beta-2' },-- ] } });-- try {-- const u = two.network.addUser('few');-- const r = await two.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'Two offers only' }, origin: ORIGIN });++ await check('Bench refuses creator vote, records one vote, and reveals models', async () => {++ const body = { round_id: round.id, left_output: pair.left.id, right_output: pair.right.id, winner: 'left' };++ const own = await t.get('/api/v1/arena/vote', { as: ana, json: body, origin: ORIGIN });++ assert.strictEqual(own.status, 403);++ const r = await t.get('/api/v1/arena/vote', { as: bo, json: body, origin: ORIGIN });assert.strictEqual(r.status, 201, r.text);-- const done = await two.tick();-- assert.strictEqual(done.status, 'ready');-- assert.strictEqual(done.pool_fallback, true);-- assert.deepStrictEqual(done.offers, [-- { provider: 'acme', model: 'alpha-1' },-- { provider: 'beta', model: 'beta-2' },-- { provider: 'ai', model: null },-- ]);-- const posts = two.ai.requests.filter((x) => x.method === 'POST' && x.path === '/api/v1/generate');-- assert.strictEqual(posts.length, 3);-- assert.deepStrictEqual(posts[0].body.route, { provider: 'acme', model: 'alpha-1' });-- assert.deepStrictEqual(posts[1].body.route, { provider: 'beta', model: 'beta-2' });-- assert.strictEqual(posts[2].body.route, undefined, 'the third slot is the routing pool');-- const page = await two.get(`/arena/r/${done.id}`);-- assert.match(page.text, /id="ar-round-pool"/);-- assert.match(page.text, /Fewer than three distinct offers/);-- } finally { await two.close(); }-- });++ assert.ok(r.json().revealed.left.model);++ assert.ok(r.json().revealed.right.model);++ const again = await t.get('/api/v1/arena/vote', { as: bo, json: body, origin: ORIGIN });++ assert.strictEqual(again.status, 409);++ });++++ await check('Elo and Cost views read Bench boards', async () => {++ const elo = await t.get('/api/v1/arena/leaderboard?category=landing');++ assert.strictEqual(elo.status, 200);++ assert.ok(elo.json().leaderboard.some((r) => r.games === 1 && r.provisional && r.ci_lower != null));++ const cost = await t.get('/api/v1/arena/leaderboard?type=cost&category=landing');++ assert.strictEqual(cost.status, 200);++ assert.ok(cost.json().leaderboard.some((r) => r.entries > 0 && r.cost_per_solved != null));++ assert.ok(t.bench.state.requests.some((x) => x.path === '/api/v1/boards/design.page' && x.query === '?type=cost'));++ const page = await t.get('/arena');++ assert.match(page.text, /Cost view/);++ assert.match(page.text, /provisional/);++ });-- await check('the create page refuses over the person budget in words and keeps a 429', async () => {-- const small = await boot({ env: { DESIGN_ARENA_PER_DAY: '2', DESIGN_ARENA_DAILY: '3' } });-- try {-- const u = small.network.addUser('limit');-- const v = small.network.addUser('other');-- assert.strictEqual((await small.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'one' }, origin: ORIGIN })).status, 201);-- assert.strictEqual((await small.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'two' }, origin: ORIGIN })).status, 201);-- const refused = await small.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'three' }, origin: ORIGIN });-- assert.strictEqual(refused.status, 429);-- assert.strictEqual(refused.json().code, 'arena.limit_person');-- assert.match(refused.json().detail, /You can start 2 Arena rounds a day/);-- const page = await small.get('/arena/new', { as: u });-- assert.match(page.text, /0 of 2 rounds left/);-- assert.strictEqual((await small.get('/api/v1/arena/rounds', { as: v, json: { category: 'landing', prompt: 'v one' }, origin: ORIGIN })).status, 201);-- const site = await small.get('/api/v1/arena/rounds', { as: v, json: { category: 'landing', prompt: 'v two' }, origin: ORIGIN });-- assert.strictEqual(site.status, 429);-- assert.strictEqual(site.json().code, 'arena.limit_site');-- assert.match(site.json().detail, /reached its 3 rounds for today/);-- } finally { await small.close(); }-- });++ await check('logo uses design.component and dataset redirects to public Bench', async () => {++ const logo = await create(bo, 'logo', 'A simple mark');++ assert.strictEqual(logo.status, 201, logo.text);++ assert.strictEqual([...t.bench.state.tasks.values()].at(-1).kind_id, 'design.component');++ await t.tick();++ const vote = await t.get('/api/v1/arena/vote?category=logo', { as: ana });++ assert.strictEqual(vote.status, 200);++ assert.strictEqual(vote.json().round.id, logo.json().round.id);++ const ds = await t.get('/api/v1/arena/dataset.jsonl');++ assert.strictEqual(ds.status, 302);++ assert.strictEqual(ds.headers.get('location'), `${t.bench.url}/api/v1/datasets/design.page.jsonl`);++ });-- await t.close();++ await check('bad input and cross-site cookie writes fail', async () => {++ assert.strictEqual((await create(ana, 'poster', 'x')).status, 422);++ assert.strictEqual((await create(ana, 'landing', ' ')).status, 422);++ assert.strictEqual((await t.get('/api/v1/arena/rounds', { as: ana, json: { category: 'landing', prompt: 'x' }, origin: 'https://evil.example' })).status, 403);++ });++ } finally { await t.close(); }done();})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,569 @@++'use strict';++++/**++ * A focused fake OpenVibe.Bench for Design's Arena tests. It implements just the routes Design needs,++ * verifies tokens against the shared fake Network public key, keeps state in memory, and can generate++ * AI entries when tick() is called.++ */++const http = require('http');++const crypto = require('crypto');++const { serviceAuth, ids } = require('openvibe-contracts');++++const K = 32;++const BASE_ELO = 1000;++const PROVISIONAL_GAMES = 10;++++function b64url(o) { return Buffer.from(JSON.stringify(o)).toString('base64url'); }++function signJwt(claims, privateKey) {++ const input = `${b64url({ alg: 'RS256', typ: 'JWT' })}.${b64url(claims)}`;++ return `${input}.${crypto.sign('RSA-SHA256', Buffer.from(input), privateKey).toString('base64url')}`;++}++++function shuffle(list, random = Math.random) {++ const a = list.slice();++ for (let i = a.length - 1; i > 0; i--) {++ const j = Math.floor(random() * (i + 1));++ [a[i], a[j]] = [a[j], a[i]];++ }++ return a;++}++++function num(v, d = 0) { return v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v); }++++function wilson(wins, games) {++ if (!games) return { lower: 0, upper: 1 };++ const z = 1.96;++ const p = wins / games;++ const denom = 1 + (z * z) / games;++ const centre = (p + (z * z) / (2 * games)) / denom;++ const width = (z * Math.sqrt((p * (1 - p) + (z * z) / (4 * games)) / games)) / denom;++ return { lower: Math.max(0, centre - width), upper: Math.min(1, centre + width) };++}++++function eloDelta(p) {++ if (!(p > 0)) return -400;++ if (p >= 1) return 400;++ return 400 * Math.log10(p / (1 - p));++}++++function ratingCi(elo, wins, losses, ties) {++ const games = wins + losses + ties;++ if (!games) return { ci_lower: null, ci_upper: null };++ const p = (wins + ties * 0.5) / games;++ const w = wilson(wins + ties * 0.5, games);++ const delta = eloDelta(p);++ return {++ ci_lower: Math.round((elo + (eloDelta(w.lower) - delta)) * 100) / 100,++ ci_upper: Math.round((elo + (eloDelta(w.upper) - delta)) * 100) / 100,++ };++}++++async function startBench({ network } = {}) {++ const publicKey = network.publicPem;++ const state = {++ tasks: new Map(),++ rounds: new Map(),++ entries: new Map(),++ judgements: [],++ ratings: new Map(),++ requests: [],++ };++ let nextId = 1;++ const id = (prefix) => `${prefix}_${String(nextId++).padStart(6, '0')}`;++++ function verify(token) {++ const r = serviceAuth.verifyServiceToken(token, { publicKey, issuer: network.url, audience: 'openvibe.bench' });++ if (r.ok) return { kind: 'service', sub: r.claims.sub, cap: r.claims.cap || [] };++ // try user token++ const parts = String(token || '').split('.');++ if (parts.length === 3) {++ const ok = crypto.verify('RSA-SHA256', Buffer.from(`${parts[0]}.${parts[1]}`), publicKey, Buffer.from(parts[2], 'base64url'));++ if (ok) {++ const c = JSON.parse(Buffer.from(parts[1], 'base64url').toString('utf8'));++ if (c.exp * 1000 > Date.now() && ids.isSubjectId('user', c.subject_id) &&++ (c.aud === 'openvibe.network' || (Array.isArray(c.aud) && c.aud.includes('openvibe.network')))) {++ return { kind: 'user', subject: c.subject_id, username: c.username };++ }++ }++ }++ return null;++ }++++ function requireService(req) {++ const v = verify(getAuth(req));++ if (!v || v.kind !== 'service') throw problem(401, 'token.required', 'Send a service token.');++ return v;++ }++++ function requireCapability(req, cap) {++ const v = requireService(req);++ if (!v.cap.includes(cap)) throw problem(403, 'bench.capability_required', `Need ${cap}.`);++ return v;++ }++++ function requirePerson(req) {++ const v = verify(getAuth(req));++ if (!v || v.kind !== 'user') throw problem(401, 'bench.sign_in_required', 'Sign in.');++ return v;++ }++++ function getAuth(req) {++ const h = String(req.headers.authorization || '');++ return h.startsWith('Bearer ') ? h.slice(7) : null;++ }++++ function problem(status, code, detail) {++ const err = new Error(detail);++ err.status = status;++ err.code = code;++ err.detail = detail;++ return err;++ }++++ const kinds = new Map([++ ['design.page', { id: 'design.page', room: 'design', entry: 'html_page', judge: 'human_pairwise', name: 'Design page arena', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'html_page', payload: { html: 'string' } } }],++ ['design.component', { id: 'design.component', room: 'design', entry: 'html_page', judge: 'human_pairwise', name: 'Design component arena', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'html_page', payload: { html: 'string' } } }],++ ]);++++ function defaultHtml(prompt) {++ const p = String(prompt || '').split('\n')[0].slice(0, 200);++ return `<!doctype html><html><head><meta charset="utf-8"><title>A generated design</title></head><body><h1>A page that works</h1><p>${p}</p><button>Start</button></body></html>`;++ }++++ async function generateEntries(round) {++ const pool = [++ { provider: 'acme', model: 'alpha-1' },++ { provider: 'beta', model: 'beta-2' },++ { provider: 'gamma', model: 'gamma-3' },++ ];++ const offers = round.offers && round.offers.length ? round.offers : pool;++ for (let i = 0; i < 3; i++) {++ const offer = offers[i] && offers[i].model ? offers[i] : pool[i];++ const model = offer.model;++ const provider = offer.provider;++ addEntry(round.id, {++ provider,++ model,++ route: offer,++ payload: { html: defaultHtml(round.prompt) },++ cost_usd: 0.0012,++ ms: 321,++ tokens: 600,++ });++ }++ }++++ function addEntry(roundId, entry) {++ const round = state.rounds.get(roundId);++ if (!round) throw problem(404, 'bench.round_not_found', 'No such round.');++ const e = {++ id: id('bte'),++ round_id: roundId,++ provider: entry.provider || 'ai',++ model: entry.model || 'unknown',++ route: entry.route || {},++ payload: entry.payload || {},++ cost_usd: num(entry.cost_usd),++ ms: num(entry.ms),++ tokens: num(entry.tokens),++ error: entry.error || null,++ ui_check: entry.ui_check || null,++ run_id: entry.run_id || null,++ created_at: new Date().toISOString(),++ };++ state.entries.set(e.id, e);++ const usable = [...state.entries.values()].filter((x) => x.round_id === roundId && !x.error).length;++ if (usable >= 2 && (round.status === 'queued' || round.status === 'running')) {++ round.status = 'finished';++ round.finished_at = new Date().toISOString();++ }++ return e;++ }++++ function ratingKey(kindId, subject) { return `${kindId}|${subject}`; }++++ function updateRating(kindId, left, right, outcome) {++ if (outcome === 'both_bad') return;++ if (!left || !right || left.model === right.model) return;++ const models = [left.model, right.model];++ for (const m of models) {++ const key = ratingKey(kindId, m);++ if (!state.ratings.has(key)) {++ state.ratings.set(key, { kind_id: kindId, category: 'all', subject: m, elo: BASE_ELO, games: 0, wins: 0, losses: 0, ties: 0, win_rate: 0, ci_lower: null, ci_upper: null });++ }++ }++ const a = state.ratings.get(ratingKey(kindId, left.model));++ const b = state.ratings.get(ratingKey(kindId, right.model));++ const sa = outcome === 'left' ? 1 : (outcome === 'right' ? 0 : 0.5);++ const sb = 1 - sa;++ const ea = 1 / (1 + 10 ** ((b.elo - a.elo) / 400));++ const eb = 1 - ea;++ const nextA = Math.round((a.elo + K * (sa - ea)) * 100) / 100;++ const nextB = Math.round((b.elo + K * (sb - eb)) * 100) / 100;++ for (const [row, elo, score, opp] of [[a, nextA, sa, sb], [b, nextB, sb, sa]]) {++ const games = row.games + 1;++ const wins = row.wins + (score > opp ? 1 : 0);++ const losses = row.losses + (score < opp ? 1 : 0);++ const ties = row.ties + (score === opp ? 1 : 0);++ const wr = games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0;++ const ci = ratingCi(elo, wins, losses, ties);++ row.elo = elo;++ row.games = games;++ row.wins = wins;++ row.losses = losses;++ row.ties = ties;++ row.win_rate = wr;++ row.ci_lower = ci.ci_lower;++ row.ci_upper = ci.ci_upper;++ }++ }++++ function blindEntry(e) {++ return {++ id: e.id,++ round_id: e.round_id,++ payload: e.payload,++ error: e.error,++ ui_check: e.ui_check,++ run_id: e.run_id,++ created_at: e.created_at,++ };++ }++++ function fullEntry(e) {++ return { ...e, route: e.route || {} };++ }++++ function revealed(round, subject) {++ if (!subject) return false;++ if (round.created_by === subject) return true;++ return state.judgements.some((j) => j.round_id === round.id && j.subject === subject);++ }++++ function wireRound(round, viewerSubject) {++ const entries = [...state.entries.values()].filter((e) => e.round_id === round.id).sort((a, b) => a.created_at.localeCompare(b.created_at));++ const isRevealed = revealed(round, viewerSubject);++ const out = {++ id: round.id,++ task_id: round.task_id,++ kind_id: round.kind_id,++ offers: round.offers,++ status: round.status,++ seed: round.seed,++ pool_fallback: round.pool_fallback,++ receipt: round.receipt,++ params: {},++ created_by: round.created_by,++ created_at: round.created_at,++ finished_at: round.finished_at || null,++ task_class: round.task_class,++ prompt: round.prompt,++ entries: entries.map((e) => isRevealed ? fullEntry(e) : blindEntry(e)),++ votes: state.judgements.filter((j) => j.round_id === round.id).length,++ };++ if (!isRevealed) out.blind = true;++ return out;++ }++++ function subjectsFor(kindId) {++ const set = new Set();++ for (const e of state.entries.values()) {++ const round = state.rounds.get(e.round_id);++ if (round && round.kind_id === kindId && !e.error) set.add(e.model);++ }++ return [...set].sort();++ }++++ function costRows(kindId) {++ const byModel = new Map();++ for (const e of state.entries.values()) {++ const round = state.rounds.get(e.round_id);++ if (!round || round.kind_id !== kindId || e.error) continue;++ if (!byModel.has(e.model)) byModel.set(e.model, { model: e.model, entries: 0, usable: 0, total_cost: 0, costs: [], mss: [] });++ const b = byModel.get(e.model);++ b.entries++;++ b.usable++;++ b.total_cost += e.cost_usd;++ b.costs.push(e.cost_usd);++ b.mss.push(e.ms);++ }++ return [...byModel.values()].map((r) => {++ const sortedCosts = r.costs.slice().sort((a, b) => a - b);++ const sortedMs = r.mss.slice().sort((a, b) => a - b);++ const median = (arr) => arr.length % 2 ? arr[Math.floor(arr.length / 2)] : (arr[arr.length / 2 - 1] + arr[arr.length / 2]) / 2;++ const solved = r.usable;++ return {++ subject: r.model,++ entries: r.entries,++ usable: r.usable,++ total_cost_usd: Math.round(r.total_cost * 1e6) / 1e6,++ solved,++ cost_per_solved: solved ? Math.round((r.total_cost / solved) * 1e6) / 1e6 : null,++ median_cost_usd: median(sortedCosts),++ median_ms: Math.round(median(sortedMs)),++ };++ }).sort((a, b) => (a.cost_per_solved == null ? Infinity : a.cost_per_solved) - (b.cost_per_solved == null ? Infinity : b.cost_per_solved) || a.subject.localeCompare(b.subject));++ }++++ function board(kindId, type) {++ if (!kinds.has(kindId)) return null;++ const subjects = subjectsFor(kindId);++ if (type === 'elo') {++ const rows = subjects.map((subject) => {++ const r = state.ratings.get(ratingKey(kindId, subject));++ const games = r ? r.games : 0;++ return {++ kind_id: kindId, category: 'all', subject,++ elo: r ? r.elo : BASE_ELO,++ games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,++ win_rate: r ? r.win_rate : 0,++ ci_lower: r ? r.ci_lower : null,++ ci_upper: r ? r.ci_upper : null,++ provisional: games < PROVISIONAL_GAMES,++ median_cost_usd: null,++ median_ms: null,++ };++ });++ rows.sort((a, b) => b.elo - a.elo || a.subject.localeCompare(b.subject));++ // Add median cost/time from cost rows++ const cost = costRows(kindId);++ for (const row of rows) {++ const c = cost.find((x) => x.subject === row.subject);++ if (c) { row.median_cost_usd = c.median_cost_usd; row.median_ms = c.median_ms; }++ }++ return { kind_id: kindId, category: 'all', period: 'all', type: 'elo', rows };++ }++ if (type === 'cost') {++ return { kind_id: kindId, category: 'all', period: 'all', type: 'cost', rows: costRows(kindId) };++ }++ return null;++ }++++ function datasetJsonl(kindId) {++ if (!kinds.has(kindId)) return null;++ const lines = [];++ for (const t of state.tasks.values()) {++ if (t.kind_id === kindId && t.public) lines.push(JSON.stringify({ type: 'task', id: t.id, kind_id: t.kind_id, prompt: t.prompt, task_class: t.task_class, public: t.public, created_at: t.created_at }));++ }++ for (const r of state.rounds.values()) {++ if (r.kind_id === kindId && r.status === 'finished') {++ lines.push(JSON.stringify({ type: 'round', id: r.id, task_id: r.task_id, kind_id: r.kind_id, offers: r.offers, status: r.status, seed: r.seed, pool_fallback: r.pool_fallback, receipt: r.receipt, created_at: r.created_at, finished_at: r.finished_at }));++ }++ }++ for (const e of state.entries.values()) {++ const r = state.rounds.get(e.round_id);++ if (r && r.kind_id === kindId && r.status === 'finished') lines.push(JSON.stringify({ type: 'entry', ...fullEntry(e) }));++ }++ for (const j of state.judgements) {++ const r = state.rounds.get(j.round_id);++ if (r && r.kind_id === kindId && r.status === 'finished') {++ lines.push(JSON.stringify({ type: 'judgement', id: j.id, round_id: j.round_id, kind_id: j.kind_id, judge: j.judge, method: j.method, pair: j.pair, outcome: j.outcome, score: j.score, at: j.at }));++ }++ }++ for (const r of state.ratings.values()) {++ if (r.kind_id === kindId) lines.push(JSON.stringify({ type: 'rating', ...r }));++ }++ return lines.length ? `${lines.join('\n')}\n` : '';++ }++++ const server = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', async () => {++ const raw = Buffer.concat(chunks);++ const text = raw.toString('utf8');++ let body = {};++ try { body = text ? JSON.parse(text) : {}; } catch { body = {}; }++ const url = new URL(req.url, 'http://x');++ state.requests.push({ method: req.method, path: url.pathname, query: url.search, authorization: req.headers.authorization || null, body: text });++++ const json = (status, obj, headers = {}) => {++ res.writeHead(status, { 'Content-Type': 'application/json', ...headers });++ res.end(JSON.stringify(obj));++ };++ const sendProblem = (status, code, detail) => {++ res.writeHead(status, { 'Content-Type': 'application/problem+json' });++ res.end(JSON.stringify({ type: `https://openvibe.network/problems/${code}`, title: 'Error', status, code, detail }));++ };++++ try {++ if (url.pathname === '/api/.well-known/jwks') return json(200, { public_key: publicKey, algorithm: 'RS256' });++++ if (req.method === 'GET' && url.pathname === '/api/v1/kinds') return json(200, { count: kinds.size, kinds: [...kinds.values()] });++ if (req.method === 'GET' && /^\/api\/v1\/kinds\/([^/]+)$/.test(url.pathname)) {++ const kindId = decodeURIComponent(url.pathname.split('/').pop());++ const k = kinds.get(kindId);++ if (!k) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');++ return json(200, { kind: k });++ }++++ if (req.method === 'POST' && /^\/api\/v1\/kinds\/([^/]+)\/tasks$/.test(url.pathname)) {++ requireCapability(req, 'bench.task.create');++ const kindId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);++ if (!kinds.has(kindId)) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');++ const prompt = String(body.prompt || '').trim();++ if (!prompt) return sendProblem(422, 'bench.prompt_required', 'A task needs a prompt.');++ const task = {++ id: id('btk'),++ kind_id: kindId,++ prompt,++ task_class: body.task_class || null,++ public: body.public !== false,++ created_by: 'service:design',++ created_at: new Date().toISOString(),++ };++ state.tasks.set(task.id, task);++ res.setHeader('Location', `/api/v1/tasks/${task.id}`);++ return json(201, { task });++ }++++ if (req.method === 'GET' && /^\/api\/v1\/tasks\/([^/]+)$/.test(url.pathname)) {++ const taskId = decodeURIComponent(url.pathname.split('/').pop());++ const task = state.tasks.get(taskId);++ if (!task) return sendProblem(404, 'bench.task_not_found', 'No such task.');++ return json(200, { task });++ }++++ if (req.method === 'POST' && /^\/api\/v1\/tasks\/([^/]+)\/rounds$/.test(url.pathname)) {++ const actor = verify(getAuth(req));++ if (!actor) return sendProblem(401, 'token.required', 'Send a token.');++ if (actor.kind === 'service' && !actor.cap.includes('bench.round.create')) return sendProblem(403, 'bench.capability_denied', 'Need bench.round.create.');++ const taskId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);++ const task = state.tasks.get(taskId);++ if (!task) return sendProblem(404, 'bench.task_not_found', 'No such task.');++ const offers = Array.isArray(body.offers) ? body.offers.slice(0, 3).map((o) => ({ provider: String(o.provider || ''), model: o.model == null ? null : String(o.model) })).filter((o) => o.provider) : [];++ const poolFallback = offers.length < 3;++ while (offers.length < 3) offers.push({ provider: 'ai', model: null });++ const round = {++ id: id('brd'),++ task_id: task.id,++ kind_id: task.kind_id,++ offers,++ status: 'queued',++ seed: 0,++ pool_fallback: poolFallback,++ receipt: null,++ params: {},++ created_by: actor.kind === 'user' ? actor.subject : 'service:design',++ created_at: new Date().toISOString(),++ finished_at: null,++ task_class: task.task_class,++ prompt: task.prompt,++ };++ state.rounds.set(round.id, round);++ res.setHeader('Location', `/api/v1/rounds/${round.id}`);++ return json(201, { round: wireRound(round, null) });++ }++++ if (req.method === 'GET' && /^\/api\/v1\/rounds\/([^/]+)$/.test(url.pathname)) {++ const roundId = decodeURIComponent(url.pathname.split('/').pop());++ const round = state.rounds.get(roundId);++ if (!round) return sendProblem(404, 'bench.round_not_found', 'No such round.');++ const viewer = verify(getAuth(req));++ return json(200, { round: wireRound(round, viewer && viewer.kind === 'user' ? viewer.subject : null) });++ }++++ if (req.method === 'POST' && /^\/api\/v1\/rounds\/([^/]+)\/entries$/.test(url.pathname)) {++ requireCapability(req, 'bench.entry.create');++ const roundId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);++ const entry = addEntry(roundId, body);++ res.setHeader('Location', `/api/v1/rounds/${roundId}`);++ return json(201, { entry: fullEntry(entry) });++ }++++ if (req.method === 'GET' && url.pathname === '/api/v1/judgements/next') {++ const person = requirePerson(req);++ const kindId = url.searchParams.get('kind') || 'design.page';++ const candidates = [...state.rounds.values()].filter((r) =>++ r.kind_id === kindId && r.status === 'finished' && r.created_by !== person.subject++ && !state.judgements.some((j) => j.round_id === r.id && j.subject === person.subject)++ );++ const usableRounds = candidates.filter((r) => [...state.entries.values()].filter((e) => e.round_id === r.id && !e.error).length >= 2);++ if (!usableRounds.length) return json(200, { pair: null, detail: 'There is no round waiting for your judgement right now.' });++ const round = usableRounds[0];++ const entries = shuffle([...state.entries.values()].filter((e) => e.round_id === round.id && !e.error));++ const [left, right] = entries.slice(0, 2);++ return json(200, {++ round: { id: round.id, kind_id: round.kind_id, prompt: round.prompt, task_class: round.task_class },++ left: blindEntry(left),++ right: blindEntry(right),++ });++ }++++ if (req.method === 'POST' && /^\/api\/v1\/rounds\/([^/]+)\/judgements$/.test(url.pathname)) {++ const person = requirePerson(req);++ const roundId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);++ const round = state.rounds.get(roundId);++ if (!round) return sendProblem(404, 'bench.round_not_found', 'No such round.');++ if (round.status !== 'finished') return sendProblem(409, 'bench.round_not_ready', 'Round not ready.');++ if (round.created_by === person.subject) return sendProblem(403, 'bench.creator_cannot_vote', 'You cannot judge a round you started.');++ const pair = body.pair || {};++ if (!pair.left || !pair.right || pair.left === pair.right) return sendProblem(422, 'bench.invalid_pair', 'Invalid pair.');++ if (state.judgements.some((j) => j.round_id === round.id && j.subject === person.subject)) return sendProblem(409, 'bench.already_judged', 'Already judged.');++ const entries = [...state.entries.values()].filter((e) => e.round_id === round.id && !e.error && (e.id === pair.left || e.id === pair.right));++ if (entries.length !== 2) return sendProblem(422, 'bench.invalid_pair', 'Entries not in round.');++ const outcome = body.outcome;++ if (!['left', 'right', 'tie', 'both_bad'].includes(outcome)) return sendProblem(422, 'bench.invalid_outcome', 'Invalid outcome.');++ const judgement = {++ id: id('bju'),++ round_id: round.id,++ kind_id: round.kind_id,++ judge: 'human_pairwise',++ method: 'human_pairwise',++ pair,++ outcome,++ score: null,++ subject: person.subject,++ at: new Date().toISOString(),++ };++ state.judgements.push(judgement);++ updateRating(round.kind_id, entries.find((e) => e.id === pair.left), entries.find((e) => e.id === pair.right), outcome);++ return json(201, { judgement, revealed: { left: fullEntry(entries.find((e) => e.id === pair.left)), right: fullEntry(entries.find((e) => e.id === pair.right)) } });++ }++++ if (req.method === 'GET' && /^\/api\/v1\/boards\/([^/]+)$/.test(url.pathname)) {++ const kindId = decodeURIComponent(url.pathname.split('/').pop());++ const type = url.searchParams.get('type') || 'elo';++ const b = board(kindId, type);++ if (!b) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');++ return json(200, { board: b });++ }++++ if (req.method === 'GET' && /^\/api\/v1\/datasets\/([^/]+)\.jsonl$/.test(url.pathname)) {++ const kindId = decodeURIComponent(url.pathname.split('/').pop().replace(/\.jsonl$/, ''));++ const text = datasetJsonl(kindId);++ if (text == null) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');++ res.setHeader('Cache-Control', 'public, max-age=60');++ res.setHeader('Content-Disposition', `attachment; filename="${kindId}-dataset.jsonl"`);++ res.writeHead(200, { 'Content-Type': 'application/x-ndjson' });++ return res.end(text);++ }++++ return sendProblem(404, 'route.not_found', 'No such route.');++ } catch (err) {++ if (err && err.status) return sendProblem(err.status, err.code, err.detail);++ sendProblem(500, 'internal.error', 'Bench fake failed.');++ }++ });++ });++++ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));++ const base = `http://127.0.0.1:${server.address().port}`;++++ async function tick() {++ for (const round of state.rounds.values()) {++ if (round.status === 'queued') {++ round.status = 'running';++ await generateEntries(round);++ }++ }++ }++++ async function close() {++ await new Promise((r) => server.close(r));++ }++++ return { base, url: base, state, tick, close, addEntry };++}++++module.exports = { startBench };@@@ -2,27 +2,28 @@// Many requests per test file: the per-address limits would answer 429 part-way.process.env.DESIGN_RATE_LIMIT_PER_MIN = process.env.DESIGN_RATE_LIMIT_PER_MIN || '100000';/**-- * Boots OpenVibe.Design on a temp database with a mock Network and AI, captures log lines, and returns a small-- * HTTP client. Every test file gets its own process, database, Network, AI and app.++ * Boots OpenVibe.Design on a temp database with mock Network and Bench, captures log lines, and returns a small++ * HTTP client. Every test file gets its own process, database, Network, Bench and app.** const t = await boot();* t.get(path, { as: user, json, headers, method, origin })* t.signIn(user) → cookie header value (design_at = a Network user token, as after /auth/callback)-- * await t.tick() → drive the Arena job loop once++ * await t.tick() → drive the fake Bench generation loop once* t.logs(), t.dbDump()*/const http = require('http');const { startNetwork } = require('./network');--const { startAi } = require('./ai');++const { startBench } = require('./bench');async function boot(opts = {}) {const network = await startNetwork(opts.network || {});-- const ai = opts.ai || await startAi(opts.aiOptions || {});++ const bench = opts.bench || await startBench({ network });const env = {NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.design', TRUST_PROXY: '1',OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url, OV_NETWORK_ISSUER: network.url,OV_OAUTH_CLIENT_ID: 'design', OV_OAUTH_CLIENT_SECRET: 'design-secret', COOKIE_SECURE: 'false',-- OV_AI_INTERNAL_URL: ai.url,++ OV_BENCH_INTERNAL_URL: bench.url,++ OV_BENCH_URL: bench.url,DESIGN_RATE_LIMIT_PER_MIN: '100000',DESIGN_DRILL: '1',DESIGN_AI_POLL_MS: '10',@@@ -46,7 +47,7 @@ async function boot(opts = {}) {const config = configLib.load(env);const testdb = opts.store ? null : await require('./db').testDb({ store: opts.testStore });const store = opts.store || createStore(testdb.db, { now: opts.now });-- const built = await createApp({ config, store, arenaAi: opts.arenaAi, log, now: opts.now, accountSend: opts.accountSend, random: opts.random });++ const built = await createApp({ config, store, log, now: opts.now, accountSend: opts.accountSend, random: opts.random });await built.ctx.keys.ensure();const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });const base = `http://127.0.0.1:${server.address().port}`;@@@ -80,8 +81,8 @@ async function boot(opts = {}) {}return {-- base, network, ai, config, ctx: built.ctx, store: built.ctx.s, get, signIn, dbDump,-- tick: () => built.ctx.jobs.tick(),++ base, network, bench, config, ctx: built.ctx, store: built.ctx.s, get, signIn, dbDump,++ tick: () => bench.tick(),logs: () => logLines.join('\n'),async close() {built.ctx.jobs.stop();@@@ -89,7 +90,7 @@ async function boot(opts = {}) {built.ctx.keys.client.stop();if (testdb) await testdb.close();await network.close();-- if (!opts.ai) await ai.close();++ if (!opts.bench) await bench.close();},};}@@@ -7,6 +7,7 @@*/const http = require('http');const crypto = require('crypto');++const { ids } = require('openvibe-contracts');const b64 = (o) => Buffer.from(JSON.stringify(o)).toString('base64url');function signJwt(claims, privateKey) {@@@ -58,7 +59,7 @@ async function startNetwork({ themes = DEFAULT_THEMES } = {}) {let nextUserId = 1;function addUser(username, { role = 'user' } = {}) {-- const u = { id: nextUserId++, subject: `usr_${crypto.randomBytes(12).toString('hex')}`, username, display_name: username, role };++ const u = { id: nextUserId++, subject: ids.newId('user'), username, display_name: username, role };st.users.set(u.id, u);return u;}@@@ -105,12 +106,13 @@ async function startNetwork({ themes = DEFAULT_THEMES } = {}) {return json(200, { access_token: userToken(u), refresh_token: rt, token_type: 'Bearer' });}if (form.grant_type === 'client_credentials') {-- // A first-party service token: audience openvibe.ai, capability ai.run.create, namespace ai.*.++ // A first-party service token for the requested audience and capabilities.const scope = String(form.scope || '');const now = Math.floor(Date.now() / 1000);const access = signJwt({iss: issuer, sub: 'svc:design', actor_type: 'service', aud: [form.audience || 'openvibe.ai'],-- cap: scope.split(/\s+/).filter(Boolean), ns: ['ai.*'], iat: now, exp: now + 300,++ cap: scope.split(/\s+/).filter(Boolean), ns: [form.audience === 'openvibe.bench' ? 'bench.*' : 'ai.*'],++ iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(12).toString('hex')}`,}, privatePem);return json(200, { access_token: access, token_type: 'Bearer', expires_in: 300, scope });}
@@ -41,11 +41,17 @@ OV_AI_INTERNAL_URL=http://127.0.0.1:4700
@@ -41,11 +41,17 @@ OV_AI_INTERNAL_URL=http://127.0.0.1:4700
# Three explicit provider:model slots; unset pins AI's generate-route offers (fewer than three falls back to its pool).
# Three explicit provider:model slots; unset pins AI's generate-route offers (fewer than three falls back to its pool).
# DESIGN_ARENA_MODELS=shared:gpt-5-mini,local:qwen2.5
# DESIGN_ARENA_MODELS=shared:gpt-5-mini,local:qwen2.5
-# Arena budgets (per UTC day) and generation jobs.
+# OpenVibe.Bench owns new Arena rounds, entries, votes and boards.
+OV_BENCH_INTERNAL_URL=http://127.0.0.1:5230
+# OV_BENCH_URL=https://bench.openvibe.services
+# DESIGN_BENCH_AUDIENCE=openvibe.bench
+# DESIGN_BENCH_SCOPE=bench.task.create bench.round.create bench.round.read bench.judgement.cast bench.board.read
+
+# Legacy Arena settings retained for compatibility; Bench enforces active round limits and runs generation.
# DESIGN_ARENA_PER_DAY=3
# DESIGN_ARENA_PER_DAY=3
# DESIGN_ARENA_DAILY=20
# DESIGN_ARENA_DAILY=20
# DESIGN_ARENA_JOB_INTERVAL_MS=1500
# DESIGN_ARENA_JOB_INTERVAL_MS=1500
-# The drill flag keeps the generation loop off; a test or a drill calls tick() itself.
+# The drill flag controls other Design background work.
# DESIGN_DRILL=0
# DESIGN_DRILL=0
# OpenVibe.Events → this service for ADR-033 account export/deletion; unset turns the route off (503).
# OpenVibe.Events → this service for ADR-033 account export/deletion; unset turns the route off (503).
@@ -17,7 +17,7 @@ Network's theme API.
@@ -17,7 +17,7 @@ Network's theme API.
- **Identity:** service id `design`, name OpenVibe.Design, port 5090, host 127.0.0.1. It stores no accounts and
- **Identity:** service id `design`, name OpenVibe.Design, port 5090, host 127.0.0.1. It stores no accounts and
no tokens: the person's Network session lives in httpOnly cookies and their theme lives in OpenVibe.Network. The
no tokens: the person's Network session lives in httpOnly cookies and their theme lives in OpenVibe.Network. The
- Arena's rounds, outputs, votes and Elo ratings live in this service's PostgreSQL database (PGlite in development).
+ Arena rounds, entries, judgements and ratings live in OpenVibe.Bench. The legacy `arena_*` tables remain for historical account events and migration.
- **Sign-in:** an OAuth 2 client of OpenVibe.Network (`design`), authorization code with PKCE S256, a state check
- **Sign-in:** an OAuth 2 client of OpenVibe.Network (`design`), authorization code with PKCE S256, a state check
and a server-side code exchange.
and a server-side code exchange.
- **The theme API proxy:** every theme read and write goes to OpenVibe.Network's `/api/themes`, with the person's
- **The theme API proxy:** every theme read and write goes to OpenVibe.Network's `/api/themes`, with the person's
@@ -38,13 +38,10 @@ Network's theme API.
@@ -38,13 +38,10 @@ Network's theme API.
## The Design Arena
## The Design Arena
-- **A round:** a signed-in person writes one prompt in one of six categories (landing, dashboard, mobile app, game
- UI, logo, slides). It is queued; the job loop reads the offers on OpenVibe.AI's `generate` routing and pins three
- distinct ones (`route: { provider, model }`), and each output stores the HTML (sanitized), the provider and model AI
- reports, its cost, its measured time and its tokens. When AI offers fewer than three, the remaining slots go to its
- pool and the round says so (`pool_fallback: true`).
-- **Budgets:** `DESIGN_ARENA_PER_DAY` (default 3) rounds per person per UTC day and `DESIGN_ARENA_DAILY` (default 20)
- for the whole site. A refusal says which budget was reached and when to come back.
+- **A round:** a signed-in person writes one prompt in one of six categories. Design creates a public Bench task of
+ kind `design.page`, or `design.component` for logo, then creates a round. Bench queues and generates three entries
+ through OpenVibe.AI, including their HTML, provider, model, cost, time and tokens.
+- **Budgets:** Bench enforces its per-person and site daily round limits.
- **Safety:** the model's page is text until it is rendered, and then only inside `<iframe sandbox srcdoc>` with no
- **Safety:** the model's page is text until it is rendered, and then only inside `<iframe sandbox srcdoc>` with no
`allow-scripts` and no `allow-same-origin`. A CSP meta in the srcdoc blocks every network fetch
`allow-scripts` and no `allow-same-origin`. A CSP meta in the srcdoc blocks every network fetch
(`default-src 'none'; style-src 'unsafe-inline'; img-src data:`). Scripts, `on*` handlers and `javascript:` URLs
(`default-src 'none'; style-src 'unsafe-inline'; img-src data:`). Scripts, `on*` handlers and `javascript:` URLs
@@ -52,13 +49,15 @@ Network's theme API.
@@ -52,13 +49,15 @@ Network's theme API.
- **Voting:** `/arena/vote` serves two successful outputs in random order with the model names hidden. One vote per
- **Voting:** `/arena/vote` serves two successful outputs in random order with the model names hidden. One vote per
person per round; the round's creator cannot vote. A vote reveals both models and updates the category's Elo
person per round; the round's creator cannot vote. A vote reveals both models and updates the category's Elo
ratings (K = 32, a tie is 0.5, `both_bad` is a game for neither).
ratings (K = 32, a tie is 0.5, `both_bad` is a game for neither).
-- **Leaderboard:** per category: model, Elo, games, wins, losses, ties, win rate, median cost and median time.
-- **Open dataset:** `GET /api/v1/arena/dataset.jsonl` is one JSON object per round with its prompt, every output
- (metadata and HTML) and every vote without a voter id.
-- **AI access:** Design is a first-party service and calls AI with a client-credentials token from Network:
- audience `openvibe.ai`, scope `ai.run.create` (and the `ai.generate` namespace in the grant). It pins one of that
- routing's listed offers per slot; AI refuses anything it does not offer in words, and a pin never bypasses its
- quotas or budgets. Generation is off under `DESIGN_DRILL=1`; the job loop starts with the process otherwise.
+- **Leaderboard:** Bench Elo boards for both kinds, including confidence intervals and provisional status, and
+ Bench Cost boards with entries, usable entries, total cost and cost per solved entry.
+- **Open dataset:** `GET /api/v1/arena/dataset.jsonl` redirects to Bench's `design.page` JSONL dataset.
+- **Bench access:** Design uses a client-credentials token (audience `openvibe.bench`) to create tasks and read
+ boards. It sends the creator's Network token to create each round so Bench records ownership and enforces the
+ personal budget and creator vote rule. Blind pairs and votes also use the person's token.
+- **Legacy import:** `scripts/arena-to-bench.js` previews a migration by default. `--apply` copies old rounds and
+ outputs and records votes as anonymous `automatic_grader` judgements. Bench's Elo board does not count these
+ historical judgements. The importer needs `bench.entry.create` and is subject to Bench's round limits.
## Pages
## Pages
@@ -96,11 +95,11 @@ a Bearer write is not cookie-authenticated and is unaffected.
@@ -96,11 +95,11 @@ a Bearer write is not cookie-authenticated and is unaffected.
| `POST /api/me/theme/history/:id/restore` | cookie + same-origin | Network's restore; `id` must be a positive integer, else 400 |
| `POST /api/me/theme/history/:id/restore` | cookie + same-origin | Network's restore; `id` must be a positive integer, else 400 |
| `POST /api/themes/submit` | cookie + same-origin | Network `POST /api/themes` with `{ name, slug, description, mode, variables, tags }` |
| `POST /api/themes/submit` | cookie + same-origin | Network `POST /api/themes` with `{ name, slug, description, mode, variables, tags }` |
| `POST /api/v1/arena/rounds` | cookie + same-origin | queue a round `{ category, prompt }` (subject to the two daily budgets) |
| `POST /api/v1/arena/rounds` | cookie + same-origin | queue a round `{ category, prompt }` (subject to the two daily budgets) |
-| `GET /api/v1/arena/rounds/:id` | none | one round with its outputs, cost and time |
-| `GET /api/v1/arena/vote` | cookie | the next blind pair for this person (models hidden) |
+| `GET /api/v1/arena/rounds/:id` | none | one round; entries are blind until Bench reveals them to this viewer |
+| `GET /api/v1/arena/vote[?category=logo]` | cookie | the next blind pair for this person (models hidden) |
| `POST /api/v1/arena/vote` | cookie + same-origin | `{ round_id, left_output, right_output, winner }`; reveals both models and updates Elo |
| `POST /api/v1/arena/vote` | cookie + same-origin | `{ round_id, left_output, right_output, winner }`; reveals both models and updates Elo |
-| `GET /api/v1/arena/leaderboard?category=…` | none | Elo, games, win rate, median cost and median time |
-| `GET /api/v1/arena/dataset.jsonl` | none | prompts, outputs (HTML included) and votes without voters |
+| `GET /api/v1/arena/leaderboard?category=…&type=elo\|cost` | none | Bench Elo or Cost board for the category's kind |
+| `GET /api/v1/arena/dataset.jsonl` | none | 302 to Bench's `design.page` JSONL dataset |
- A Network **4xx** answer passes through with its status and body; a Network failure (no answer, timeout, 5xx)
- A Network **4xx** answer passes through with its status and body; a Network failure (no answer, timeout, 5xx)
is a `502 network.unavailable` problem.
is a `502 network.unavailable` problem.
@@ -145,9 +144,8 @@ npm run dev # http://localhost:5090
@@ -145,9 +144,8 @@ npm run dev # http://localhost:5090
OV_ALLOW_SUITE=1 npm test
OV_ALLOW_SUITE=1 npm test
```
```
-`npm test` runs every `test/*.test.js` in its own process with an in-process OpenVibe.Network stand-in
-(`test/helpers/network.js`): a real RS256 key pair, the JWKS, the OAuth token endpoint, the theme catalogue and
-the person's theme routes. Nothing calls the real internet. `OV_ALLOW_SUITE=1` is the suite opt-in the shared
+`npm test` runs every `test/*.test.js` in its own process with in-process OpenVibe.Network and OpenVibe.Bench
+stand-ins (`test/helpers/network.js`, `test/helpers/bench.js`). Nothing calls the real internet. `OV_ALLOW_SUITE=1` is the suite opt-in the shared
test runner expects.
test runner expects.
## Deploy (for the lead)
## Deploy (for the lead)
new file mode 100644
new file mode 100644
@@ -0,0 +1,68 @@
@@ -0,0 +1,68 @@
+#!/usr/bin/env node
+'use strict';
+
+/** One-off legacy Arena import. Dry run is the default. No identity is sent with judgements. */
+const { createBenchClient } = require('../server/arena/bench');
+const { load } = require('../server/config');
+const { createDb } = require('openvibe-sdk/db');
+
+async function readLegacy(db) {
+ const rounds = await db.many('SELECT id, category, prompt, status, created_at FROM arena_rounds ORDER BY created_at, id');
+ const outputs = await db.many('SELECT id, "round", model, provider, html, cost_usd, ms, tokens, error, ui_check, created_at FROM arena_outputs ORDER BY created_at, id');
+ const votes = await db.many('SELECT "round", left_output, right_output, winner, at FROM arena_votes ORDER BY at, "round"');
+ return { rounds, outputs, votes };
+}
+
+function preflight({ rounds, outputs, votes }) {
+ const problems = [];
+ const ids = new Set(rounds.map((r) => r.id));
+ const outputById = new Map(outputs.map((o) => [o.id, o]));
+ for (const v of votes) {
+ const left = outputById.get(v.left_output);
+ const right = outputById.get(v.right_output);
+ if (!ids.has(v.round) || !left || !right || left.round !== v.round || right.round !== v.round || left.error || right.error) {
+ problems.push(`vote on ${v.round} has a missing, failed or foreign entry`);
+ }
+ }
+ return problems;
+}
+
+async function migrate({ db, bench, apply = false }) {
+ const legacy = await readLegacy(db);
+ const problems = preflight(legacy);
+ const summary = { dry_run: !apply, rounds: legacy.rounds.length, entries: legacy.outputs.length, judgements: legacy.votes.length, problems };
+ if (!apply) return summary;
+ if (problems.length) throw new Error(`Import refused: ${problems.join('; ')}`);
+ const outputMap = new Map();
+ for (const r of legacy.rounds) {
+ const { round } = await bench.createRound({ category: r.category, prompt: r.prompt });
+ for (const o of legacy.outputs.filter((e) => e.round === r.id)) {
+ const result = await bench.addEntry({ roundId: round.id, provider: o.provider, model: o.model,
+ route: {}, payload: { html: o.html }, costUsd: o.cost_usd, ms: o.ms, tokens: o.tokens,
+ error: o.error, uiCheck: o.ui_check });
+ outputMap.set(o.id, result.entry.id);
+ }
+ for (const v of legacy.votes.filter((j) => j.round === r.id)) {
+ await bench.castImportedVote({ roundId: round.id,
+ pair: { left: outputMap.get(v.left_output), right: outputMap.get(v.right_output) }, winner: v.winner });
+ }
+ }
+ return summary;
+}
+
+async function main() {
+ const apply = process.argv.includes('--apply');
+ const config = load();
+ if (!config.db.url) throw new Error('DATABASE_URL is required for the one-off import');
+ config.bench.scope = `${config.bench.scope} bench.entry.create`;
+ const db = createDb({ url: config.db.url, service: 'design-arena-import', max: 1 });
+ try {
+ const bench = apply ? createBenchClient({ config }) : null;
+ const summary = await migrate({ db, bench, apply });
+ process.stdout.write(`${JSON.stringify(summary)}\n`);
+ if (summary.problems.length) process.exitCode = 2;
+ } finally { await db.close(); }
+}
+
+if (require.main === module) main().catch((err) => { console.error(err.message); process.exitCode = 1; });
+module.exports = { readLegacy, preflight, migrate };
@@ -12,8 +12,7 @@
@@ -12,8 +12,7 @@
* /api/health, /api/ready, /release.json liveness, truthful readiness, release metadata
* /api/health, /api/ready, /release.json liveness, truthful readiness, release metadata
* /robots.txt, /sitemap.xml, /llms.txt crawl artifacts (http/discovery.js)
* /robots.txt, /sitemap.xml, /llms.txt crawl artifacts (http/discovery.js)
*
*
- * The Arena's rounds are generated by a job loop (server/index.js starts it; DESIGN_DRILL=1 keeps it off). The
- * request that starts a round only queues it.
+ * Bench generates Arena rounds asynchronously. The request that starts a round only queues it.
*/
*/
const path = require('path');
const path = require('path');
const express = require('express');
const express = require('express');
@@ -39,7 +38,7 @@ const { createDiscoveryRoutes } = require('./http/discovery');
@@ -39,7 +38,7 @@ const { createDiscoveryRoutes } = require('./http/discovery');
const { createServiceReadiness } = require('./observability');
const { createServiceReadiness } = require('./observability');
const { createArenaStore } = require('./arena/store');
const { createArenaStore } = require('./arena/store');
const { createArenaAi } = require('./arena/ai');
const { createArenaAi } = require('./arena/ai');
-const { createArenaJobs } = require('./arena/jobs');
+const { createBenchClient } = require('./arena/bench');
const accountDataLib = require('./identity/account-data');
const accountDataLib = require('./identity/account-data');
const { PUBLIC_DIR, assetVersion } = require('./assets');
const { PUBLIC_DIR, assetVersion } = require('./assets');
const views = require('./views');
const views = require('./views');
@@ -60,16 +59,18 @@ async function createApp(opts = {}) {
@@ -60,16 +59,18 @@ async function createApp(opts = {}) {
const keys = createKeyStore({ config, fetchImpl, log });
const keys = createKeyStore({ config, fetchImpl, log });
const sso = createSso({ config, keys, fetchImpl, log, now });
const sso = createSso({ config, keys, fetchImpl, log, now });
const catalog = opts.catalog || createThemeCatalog({ config, fetchImpl, log, now });
const catalog = opts.catalog || createThemeCatalog({ config, fetchImpl, log, now });
- const arena = opts.arena || createArenaStore({ s, config, log });
+ const bench = opts.bench || createBenchClient({ config, fetchImpl, log });
+ const arena = opts.arena || createArenaStore({ s, config, bench, log });
const ai = opts.arenaAi || opts.ai || createArenaAi({ config, fetchImpl, log });
const ai = opts.arenaAi || opts.ai || createArenaAi({ config, fetchImpl, log });
- const jobs = createArenaJobs({ s, arena, ai, config, log });
+ // The Arena generation job now lives on Bench; Design keeps a no-op loop for test compatibility.
+ const jobs = opts.jobs || { tick: async () => null, start: () => null, stop: () => null, get running() { return false; } };
const accountData = accountDataLib.create({ db: s.db, log });
const accountData = accountDataLib.create({ db: s.db, log });
const accountSend = opts.accountSend || (config.oauth.clientSecret
const accountSend = opts.accountSend || (config.oauth.clientSecret
? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })
? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Design cannot answer account events'); });
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Design cannot answer account events'); });
const ctx = {
const ctx = {
- config, s, store: s, keys, sso, catalog, arena, ai, jobs, accountData, accountSend, log,
+ config, s, store: s, keys, sso, catalog, arena, bench, ai, jobs, accountData, accountSend, log,
random: opts.random || Math.random,
random: opts.random || Math.random,
};
};
new file mode 100644
new file mode 100644
@@ -0,0 +1,193 @@
@@ -0,0 +1,193 @@
+'use strict';
+
+/**
+ * OpenVibe.Bench client for the Design Arena. Design uses its service token for task creation, reads and boards,
+ * and forwards each person's Network token for round creation, blind pairs, votes and reveals.
+ *
+ * service token (audience openvibe.bench, bench.task.create / bench.round.read / bench.board.read)
+ * person token (Network user access token, audience openvibe.network)
+ */
+const { serviceAuth } = require('openvibe-contracts');
+const { ArenaError } = require('./legacy-store');
+
+const BENCH_CAPABILITIES = Object.freeze([
+ 'bench.task.create',
+ 'bench.round.create',
+ 'bench.round.read',
+ 'bench.judgement.cast',
+ 'bench.board.read',
+].join(' '));
+
+class BenchClientError extends Error {
+ constructor(status, code, detail) {
+ super(detail || code);
+ this.name = 'BenchClientError';
+ this.status = status;
+ this.code = code;
+ this.detail = detail || code;
+ }
+}
+
+function mapBenchProblem(status, body) {
+ const code = body && (body.code || body.error) ? String(body.code || body.error) : 'bench.error';
+ const detail = body && (body.detail || body.message) ? String(body.detail || body.message) : 'OpenVibe.Bench returned an error.';
+ switch (code) {
+ case 'bench.limit_person': return new ArenaError(429, 'arena.limit_person', detail);
+ case 'bench.limit_site': return new ArenaError(429, 'arena.limit_site', detail);
+ case 'bench.kind_not_found': return new ArenaError(400, 'arena.invalid_category', detail);
+ case 'bench.prompt_required':
+ case 'bench.prompt_too_long': return new ArenaError(422, 'arena.prompt_required', detail);
+ case 'bench.round_not_found': return new ArenaError(404, 'arena.round_not_found', detail);
+ case 'bench.round_not_ready': return new ArenaError(409, 'arena.round_not_ready', detail);
+ case 'bench.creator_cannot_vote': return new ArenaError(403, 'arena.creator_cannot_vote', detail);
+ case 'bench.already_judged': return new ArenaError(409, 'arena.already_voted', detail);
+ case 'bench.invalid_pair': return new ArenaError(422, 'arena.invalid_pair', detail);
+ case 'bench.invalid_outcome': return new ArenaError(422, 'arena.invalid_winner', detail);
+ default: return new BenchClientError(status, code, detail);
+ }
+}
+
+function kindForCategory(category) {
+ return category === 'logo' ? 'design.component' : 'design.page';
+}
+
+function createBenchClient({ config, fetchImpl = globalThis.fetch, log = console }) {
+ const bench = config.bench;
+ const enabled = Boolean(bench && bench.url);
+ const tokenClient = enabled && config.oauth && config.oauth.clientSecret
+ ? serviceAuth.createTokenClient({
+ tokenUrl: `${config.networkInternalUrl || config.networkUrl}/oauth/token`,
+ clientId: config.oauth.clientId,
+ clientSecret: config.oauth.clientSecret,
+ audience: bench.audience,
+ scope: bench.scope,
+ fetchImpl,
+ })
+ : null;
+
+ async function serviceHeaders() {
+ if (!enabled) throw new BenchClientError(503, 'bench.not_configured', 'OpenVibe.Bench is not configured for the Arena.');
+ if (!tokenClient) throw new BenchClientError(503, 'bench.not_configured', 'Design has no client secret to talk to Bench.');
+ return { Accept: 'application/json', ...(await tokenClient.authHeaders()), 'Content-Type': 'application/json' };
+ }
+
+ function personHeaders(token) {
+ if (!enabled) throw new BenchClientError(503, 'bench.not_configured', 'OpenVibe.Bench is not configured for the Arena.');
+ if (!token) throw new ArenaError(401, 'token.required', 'Sign in to use the Arena.');
+ return { Accept: 'application/json', Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' };
+ }
+
+ async function request(method, path, { body = null, headers = null, token = null } = {}) {
+ const h = headers || (token ? personHeaders(token) : await serviceHeaders());
+ const url = `${bench.url.replace(/\/+$/, '')}${path}`;
+ const opts = { method, headers: h, signal: AbortSignal.timeout(config.timeoutMs || 8000) };
+ if (body != null) opts.body = JSON.stringify(body);
+ let res;
+ try {
+ res = await fetchImpl(url, opts);
+ } catch (err) {
+ throw new BenchClientError(502, 'bench.unreachable', err && err.name === 'TimeoutError' ? 'OpenVibe.Bench did not answer in time.' : 'OpenVibe.Bench could not be reached.');
+ }
+ const json = await res.json().catch(() => null);
+ if (!res.ok) {
+ const err = mapBenchProblem(res.status, json);
+ throw err;
+ }
+ return json || {};
+ }
+
+ async function createTask({ category, prompt }) {
+ const kindId = kindForCategory(category);
+ const out = await request('POST', `/api/v1/kinds/${encodeURIComponent(kindId)}/tasks`, {
+ body: { prompt, public: true, task_class: category },
+ });
+ return out.task;
+ }
+
+ async function createRound({ category, prompt, token, offers = null }) {
+ const task = await createTask({ category, prompt });
+ const body = {};
+ if (offers) body.offers = offers;
+ const out = await request('POST', `/api/v1/tasks/${encodeURIComponent(task.id)}/rounds`, {
+ body,
+ token,
+ });
+ return { task, round: { ...out.round, task_class: task.task_class, prompt: task.prompt } };
+ }
+
+ async function getTask(id) {
+ const out = await request('GET', `/api/v1/tasks/${encodeURIComponent(id)}`);
+ return out.task;
+ }
+
+ async function getRound(id, { token = null } = {}) {
+ const out = await request('GET', `/api/v1/rounds/${encodeURIComponent(id)}`, { token });
+ const round = out.round;
+ if (round && round.task_id && !round.prompt) {
+ try {
+ const task = await getTask(round.task_id);
+ round.task_class = task.task_class;
+ round.prompt = task.prompt;
+ } catch { /* keep blind */ }
+ }
+ return round;
+ }
+
+ async function nextPair(kindId, token) {
+ const out = await request('GET', `/api/v1/judgements/next?kind=${encodeURIComponent(kindId)}`, { token });
+ return out && out.round ? out : null;
+ }
+
+ async function castVote({ roundId, token, pair, winner }) {
+ const out = await request('POST', `/api/v1/rounds/${encodeURIComponent(roundId)}/judgements`, {
+ token,
+ body: { pair, outcome: winner },
+ });
+ return out;
+ }
+
+ async function castImportedVote({ roundId, pair, winner }) {
+ return await request('POST', `/api/v1/rounds/${encodeURIComponent(roundId)}/judgements`, {
+ body: { judge: 'automatic_grader', judge_name: 'design.arena.legacy_vote', pair, outcome: winner },
+ });
+ }
+
+ async function getBoard(kindId, { type = 'elo', token = null } = {}) {
+ const out = await request('GET', `/api/v1/boards/${encodeURIComponent(kindId)}?type=${encodeURIComponent(type)}`, { token });
+ return out.board;
+ }
+
+ function datasetUrl(kindId) {
+ return `${bench.publicUrl.replace(/\/+$/, '')}/api/v1/datasets/${encodeURIComponent(kindId)}.jsonl`;
+ }
+
+ async function addEntry({ roundId, provider, model, route, payload, costUsd, ms, tokens, error, uiCheck, runId }) {
+ return await request('POST', `/api/v1/rounds/${encodeURIComponent(roundId)}/entries`, {
+ body: {
+ provider, model, route, payload,
+ cost_usd: costUsd,
+ ms,
+ tokens,
+ error,
+ ui_check: uiCheck,
+ run_id: runId,
+ },
+ });
+ }
+
+ return {
+ enabled,
+ kindForCategory,
+ createTask,
+ createRound,
+ getRound,
+ nextPair,
+ castVote,
+ castImportedVote,
+ getBoard,
+ datasetUrl,
+ addEntry,
+ };
+}
+
+module.exports = { createBenchClient, BenchClientError, kindForCategory, BENCH_CAPABILITIES };
new file mode 100644
new file mode 100644
@@ -0,0 +1,33 @@
@@ -0,0 +1,33 @@
+'use strict';
+
+/**
+ * Constants and helpers shared between the legacy Arena tables and the Bench-backed store.
+ */
+
+const CATEGORIES = Object.freeze(['landing', 'dashboard', 'mobile_app', 'game_ui', 'logo', 'slides']);
+const WINNERS = Object.freeze(['left', 'right', 'tie', 'both_bad']);
+const K = 32;
+const BASE_ELO = 1000;
+
+class ArenaError extends Error {
+ constructor(status, code, detail, extra = {}) {
+ super(detail);
+ this.name = 'ArenaError';
+ this.status = status;
+ this.code = code;
+ this.detail = detail;
+ this.extra = extra;
+ }
+}
+
+/** Fisher–Yates, with an injectable random source so a test can pin the order. */
+function shuffle(list, random = Math.random) {
+ const a = list.slice();
+ for (let i = a.length - 1; i > 0; i--) {
+ const j = Math.floor(random() * (i + 1));
+ [a[i], a[j]] = [a[j], a[i]];
+ }
+ return a;
+}
+
+module.exports = { ArenaError, CATEGORIES, WINNERS, K, BASE_ELO, shuffle };
@@ -1,276 +1,166 @@
@@ -1,276 +1,166 @@
'use strict';
'use strict';
/**
/**
- * The Arena's data access: rounds, outputs, votes, ratings, the open dataset and the day's budgets. Every
- * person-facing write is a transaction; a vote and the rating changes it causes commit together, so a crash can
- * never leave a vote without its Elo update. Votes are keyed by (round, voter_subject): one per person per round.
+ * The Arena's data access, now backed by OpenVibe.Bench. Design keeps its own `arena_*` tables read-only
+ * (legacy data and the migration script); the active engine lives on Bench.
+ *
+ * createRound() -> Bench task (service token) + round (creator token)
+ * getRound() -> Bench round read
+ * nextPair() -> Bench /judgements/next (person's Network token)
+ * vote() -> Bench /rounds/:id/judgements (person's Network token)
+ * leaderboard() -> Bench /boards/:kind?type=elo|cost
+ * datasetJsonl() -> redirect to Bench /datasets/:kind.jsonl
*/
*/
-const { ids } = require('openvibe-contracts');
+const { ArenaError, CATEGORIES, WINNERS, K, BASE_ELO, shuffle } = require('./legacy-store');
+const { sanitizeHtml } = require('./sanitize');
-const CATEGORIES = Object.freeze(['landing', 'dashboard', 'mobile_app', 'game_ui', 'logo', 'slides']);
-const WINNERS = Object.freeze(['left', 'right', 'tie', 'both_bad']);
-const K = 32;
-const BASE_ELO = 1000;
-
-class ArenaError extends Error {
- constructor(status, code, detail, extra = {}) {
- super(detail);
- this.name = 'ArenaError';
- this.status = status;
- this.code = code;
- this.detail = detail;
- this.extra = extra;
- }
-}
-
-const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
-const roundOf = (r) => (r ? {
- id: r.id,
- category: r.category,
- prompt: r.prompt,
- status: r.status,
- // The offers the round asked AI for ({provider, model}); a pool slot is provider 'ai' with no model.
- offers: Array.isArray(r.offers) ? r.offers : [],
- // True only when the routing offered fewer than three distinct offers, so the rest came from the pool.
- pool_fallback: Boolean(r.pool_fallback),
- created_at: r.created_at,
-} : null);
-const outputOf = (o) => ({
- id: o.id,
- round: o.round,
- model: o.model,
- provider: o.provider,
- html: o.html,
- cost_usd: num(o.cost_usd),
- ms: num(o.ms),
- tokens: num(o.tokens),
- error: o.error || null,
- ui_check: o.ui_check == null ? null : o.ui_check,
- created_at: o.created_at,
-});
-const voteOf = (v) => ({ round: v.round, left_output: v.left_output, right_output: v.right_output, winner: v.winner, at: v.at });
-
-/** Fisher–Yates, with an injectable random source so a test can pin the order. */
-function shuffle(list, random = Math.random) {
- const a = list.slice();
- for (let i = a.length - 1; i > 0; i--) {
- const j = Math.floor(random() * (i + 1));
- [a[i], a[j]] = [a[j], a[i]];
- }
- return a;
+function entryToOutput(e) {
+ const payload = e.payload || {};
+ return {
+ id: e.id,
+ round: e.round_id,
+ model: e.model || 'unknown',
+ provider: e.provider || null,
+ html: sanitizeHtml(payload.html || ''),
+ cost_usd: e.cost_usd == null ? 0 : Number(e.cost_usd),
+ ms: e.ms == null ? 0 : Number(e.ms),
+ tokens: e.tokens == null ? 0 : Number(e.tokens),
+ error: e.error || null,
+ ui_check: e.ui_check == null ? null : e.ui_check,
+ created_at: e.created_at,
+ };
}
}
-function createArenaStore({ s, config, log = console }) {
- const db = s.db;
- const perDay = config.arena.perDay;
- const daily = config.arena.daily;
-
- const dayStart = (at = s.now()) => {
- const d = new Date(at);
- d.setUTCHours(0, 0, 0, 0);
- return d.toISOString();
+function roundToLocal(r) {
+ const entries = (r.entries || []).map(entryToOutput);
+ const category = r.task_class || (r.kind_id === 'design.component' ? 'logo' : 'landing');
+ return {
+ id: r.id,
+ category,
+ prompt: r.prompt || '',
+ status: r.status === 'finished' ? 'ready' : (r.status === 'failed' ? 'failed' : (r.status || 'queued')),
+ offers: r.offers || [],
+ pool_fallback: Boolean(r.pool_fallback),
+ created_at: r.created_at,
+ outputs: entries,
+ votes: Number(r.votes || 0),
};
};
+}
- async function counts(subject) {
- const start = dayStart();
- const used = num(await db.value('SELECT COUNT(*)::int AS n FROM arena_rounds WHERE created_by = $1 AND created_at >= $2::timestamptz', [subject, start]));
- const siteUsed = num(await db.value('SELECT COUNT(*)::int AS n FROM arena_rounds WHERE created_at >= $1::timestamptz', [start]));
- return {
- used,
- remaining: Math.max(0, perDay - used),
- per_day: perDay,
- site_used: siteUsed,
- site_daily: daily,
- site_remaining: Math.max(0, daily - siteUsed),
- };
+function createArenaStore({ config, bench }) {
+ async function counts(/* subject */) {
+ // Bench enforces per-person and site budgets; Design no longer mirrors them.
+ return null;
}
}
- async function createRound({ category, prompt, subject }) {
+ async function createRound({ category, prompt, token }) {
if (!CATEGORIES.includes(category)) throw new ArenaError(422, 'arena.invalid_category', `Pick a category: ${CATEGORIES.join(', ')}.`);
if (!CATEGORIES.includes(category)) throw new ArenaError(422, 'arena.invalid_category', `Pick a category: ${CATEGORIES.join(', ')}.`);
const text = String(prompt == null ? '' : prompt).trim();
const text = String(prompt == null ? '' : prompt).trim();
if (!text) throw new ArenaError(422, 'arena.prompt_required', 'Describe the page you want the models to design.');
if (!text) throw new ArenaError(422, 'arena.prompt_required', 'Describe the page you want the models to design.');
if (text.length > config.arena.maxPrompt) throw new ArenaError(422, 'arena.prompt_too_long', `The prompt must be at most ${config.arena.maxPrompt} characters.`);
if (text.length > config.arena.maxPrompt) throw new ArenaError(422, 'arena.prompt_too_long', `The prompt must be at most ${config.arena.maxPrompt} characters.`);
- const at = s.now();
- const start = dayStart(at);
- const id = `arr_${ids.ulid(at)}`;
- const row = await db.maybe(
- `INSERT INTO arena_rounds (id, category, prompt, created_by, status, created_at)
- SELECT $1, $2, $3, $4, 'queued', $5::timestamptz
- WHERE (SELECT COUNT(*) FROM arena_rounds WHERE created_by = $4 AND created_at >= $6::timestamptz) < $7
- AND (SELECT COUNT(*) FROM arena_rounds WHERE created_at >= $6::timestamptz) < $8
- RETURNING *`,
- [id, category, text, subject, s.iso(), start, perDay, daily],
- );
- if (!row) {
- const c = await counts(subject);
- if (c.used >= perDay) {
- throw new ArenaError(429, 'arena.limit_person',
- `You can start ${perDay} Arena round${perDay === 1 ? '' : 's'} a day. You have started ${c.used} today; try again tomorrow.`);
- }
- throw new ArenaError(429, 'arena.limit_site',
- `The Arena has reached its ${daily} round${daily === 1 ? '' : 's'} for today. Try again tomorrow.`);
- }
- return await getRound(id);
+ if (!token) throw new ArenaError(401, 'token.required', 'Sign in to use the Arena.');
+ const { round } = await bench.createRound({ category, prompt: text, token });
+ return await getRound(round.id, { token });
}
}
- async function getRound(id) {
- const r = await db.maybe('SELECT * FROM arena_rounds WHERE id = $1', [id]);
+ async function getRound(id, { token = null } = {}) {
+ const r = await bench.getRound(id, { token });
if (!r) return null;
if (!r) return null;
- const outputs = (await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 ORDER BY created_at, id', [id])).map(outputOf);
- const votes = num(await db.value('SELECT COUNT(*)::int AS n FROM arena_votes WHERE "round" = $1', [id]));
- return { ...roundOf(r), outputs, votes };
+ return roundToLocal(r);
}
}
- /** A ready round this person has not voted on and did not start, with its two successful outputs shuffled. */
- async function nextPair(subject, { random = Math.random } = {}) {
- const r = await db.maybe(
- `SELECT r.* FROM arena_rounds r
- WHERE r.status = 'ready' AND r.created_by <> $1
- AND (SELECT COUNT(*) FROM arena_outputs o WHERE o."round" = r.id AND o.error IS NULL AND o.html <> '') >= 2
- AND NOT EXISTS (SELECT 1 FROM arena_votes v WHERE v."round" = r.id AND v.voter_subject = $1)
- ORDER BY random() LIMIT 1`,
- [subject],
- );
- if (!r) return null;
- const outputs = (await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 AND error IS NULL AND html <> \'\' ORDER BY id', [r.id])).map(outputOf);
- if (outputs.length < 2) return null;
- const [left, right] = shuffle(outputs, random).slice(0, 2);
- // Blind: the model, provider, cost and time are not part of this answer. The HTML is (in the iframe).
+ async function nextPair(_subject, { token, category = 'landing', random: rng = Math.random } = {}) {
+ if (!token) throw new ArenaError(401, 'token.required', 'Sign in at openvibe.design to use the Arena.');
+ const kindId = categoryToKind(category);
+ const pair = await bench.nextPair(kindId, token);
+ if (!pair) return null;
+ const [left, right] = shuffle([pair.left, pair.right], rng);
return {
return {
- round: { id: r.id, category: r.category, prompt: r.prompt },
- left: { id: left.id, html: left.html },
- right: { id: right.id, html: right.html },
+ round: { id: pair.round.id, category: pair.round.task_class || (pair.round.kind_id === 'design.component' ? 'logo' : 'landing'), prompt: pair.round.prompt },
+ left: { id: left.id, html: sanitizeHtml((left.payload || {}).html || '') },
+ right: { id: right.id, html: sanitizeHtml((right.payload || {}).html || '') },
};
};
}
}
- async function applyElo(t, category, left, right, winner, atIso) {
- if (winner === 'both_bad') return; // a game for neither
- if (left.model === right.model) return; // two outputs of one model are not a match between designers
- const models = [left.model, right.model];
- for (const model of models) {
- await t.exec(`INSERT INTO arena_ratings (category, model, elo, games, wins, losses, ties, updated_at)
- VALUES ($1, $2, $3, 0, 0, 0, 0, $4::timestamptz)
- ON CONFLICT (category, model) DO NOTHING`, [category, model, BASE_ELO, atIso]);
- }
- const rows = await t.many('SELECT * FROM arena_ratings WHERE category = $1 AND model = ANY($2::text[]) ORDER BY model FOR UPDATE', [category, models]);
- const byModel = new Map(rows.map((r) => [r.model, r]));
- const a = byModel.get(left.model);
- const b = byModel.get(right.model);
- if (!a || !b) return;
- const ra = num(a.elo, BASE_ELO);
- const rb = num(b.elo, BASE_ELO);
- const ea = 1 / (1 + (10 ** ((rb - ra) / 400)));
- const eb = 1 - ea;
- const sa = winner === 'left' ? 1 : (winner === 'right' ? 0 : 0.5);
- const sb = 1 - sa;
- const nextA = Math.round((ra + K * (sa - ea)) * 100) / 100;
- const nextB = Math.round((rb + K * (sb - eb)) * 100) / 100;
- const update = async (row, elo, score, oppScore) => {
- await t.exec(`UPDATE arena_ratings SET elo = $1, games = games + 1,
- wins = wins + $2, losses = losses + $3, ties = ties + $4, updated_at = $5::timestamptz
- WHERE category = $6 AND model = $7`,
- [elo, score > oppScore ? 1 : 0, score < oppScore ? 1 : 0, score === oppScore ? 1 : 0, atIso, category, row.model]);
+ async function vote({ roundId, leftOutput, rightOutput, winner, token }) {
+ if (!WINNERS.includes(winner)) throw new ArenaError(422, 'arena.invalid_winner', 'The winner must be left, right, tie or both_bad.');
+ if (!leftOutput || !rightOutput || leftOutput === rightOutput) throw new ArenaError(422, 'arena.invalid_pair', 'A vote needs two different outputs.');
+ if (!token) throw new ArenaError(401, 'token.required', 'Sign in at openvibe.design to use the Arena.');
+ const out = await bench.castVote({ roundId, token, pair: { left: leftOutput, right: rightOutput }, winner });
+ const left = out.revealed && out.revealed.left ? entryToOutput(out.revealed.left) : null;
+ const right = out.revealed && out.revealed.right ? entryToOutput(out.revealed.right) : null;
+ return { round: { id: roundId }, winner, at: out.judgement && out.judgement.at, left, right };
+ }
+
+ function boardRowToLocal(r) {
+ return {
+ category: r.category || 'all',
+ model: r.subject || r.model || 'unknown',
+ elo: r.elo == null ? BASE_ELO : Number(r.elo),
+ games: Number(r.games || 0),
+ wins: Number(r.wins || 0),
+ losses: Number(r.losses || 0),
+ ties: Number(r.ties || 0),
+ win_rate: Number(r.win_rate || 0),
+ median_cost_usd: r.median_cost_usd == null ? null : Number(r.median_cost_usd),
+ median_cost: r.median_cost_usd == null ? null : Number(r.median_cost_usd),
+ median_ms: r.median_ms == null ? null : Number(r.median_ms),
+ ci_lower: r.ci_lower == null ? null : Number(r.ci_lower),
+ ci_upper: r.ci_upper == null ? null : Number(r.ci_upper),
+ provisional: Boolean(r.provisional),
+ entries: Number(r.entries || 0),
+ usable: Number(r.usable || 0),
+ total_cost_usd: Number(r.total_cost_usd || 0),
+ solved: Number(r.solved || 0),
+ cost_per_solved: r.cost_per_solved == null ? null : Number(r.cost_per_solved),
};
};
- await update(a, nextA, sa, sb);
- await update(b, nextB, sb, sa);
}
}
- async function vote({ roundId, subject, leftOutput, rightOutput, winner }) {
- if (!WINNERS.includes(winner)) throw new ArenaError(422, 'arena.invalid_winner', 'The winner must be left, right, tie or both_bad.');
- if (!leftOutput || !rightOutput || leftOutput === rightOutput) throw new ArenaError(422, 'arena.invalid_pair', 'A vote needs two different outputs.');
- try {
- const result = await s.tx(async () => {
- const r = await db.maybe('SELECT * FROM arena_rounds WHERE id = $1 FOR UPDATE', [roundId]);
- if (!r) throw new ArenaError(404, 'arena.round_not_found', 'No such Arena round.');
- if (r.created_by === subject) throw new ArenaError(403, 'arena.creator_cannot_vote', 'You cannot vote on a round you started.');
- if (r.status !== 'ready') throw new ArenaError(409, 'arena.round_not_ready', 'This round is not ready to vote on yet.');
- const outputs = (await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 AND id = ANY($2::text[]) AND error IS NULL AND html <> \'\'', [roundId, [leftOutput, rightOutput]])).map(outputOf);
- if (outputs.length !== 2) throw new ArenaError(422, 'arena.invalid_pair', 'Those two outputs are not both part of this round.');
- const left = outputs.find((o) => o.id === leftOutput);
- const right = outputs.find((o) => o.id === rightOutput);
- const prior = await db.maybe('SELECT 1 AS n FROM arena_votes WHERE "round" = $1 AND voter_subject = $2', [roundId, subject]);
- if (prior) throw new ArenaError(409, 'arena.already_voted', 'You have already voted on this round.');
- const at = s.iso();
- await db.exec('INSERT INTO arena_votes ("round", voter_subject, left_output, right_output, winner, at) VALUES ($1, $2, $3, $4, $5, $6::timestamptz)',
- [roundId, subject, leftOutput, rightOutput, winner, at]);
- await applyElo(db, r.category, left, right, winner, at);
- return { round: roundOf(r), winner, at, left, right };
- });
- return result;
- } catch (err) {
- if (err && err.code === '23505') throw new ArenaError(409, 'arena.already_voted', 'You have already voted on this round.');
- throw err;
- }
+ function categoryToKind(category) {
+ return category === 'logo' || category === 'design.component' ? 'design.component' : 'design.page';
}
}
- async function leaderboard(category = null) {
- const rows = await db.many(
- `WITH models AS (
- SELECT category, model FROM arena_ratings WHERE ($1::text IS NULL OR category = $1)
- UNION
- SELECT r.category, o.model FROM arena_outputs o JOIN arena_rounds r ON r.id = o."round"
- WHERE r.status = 'ready' AND o.error IS NULL AND ($1::text IS NULL OR r.category = $1)
- ), medians AS (
- SELECT r.category, o.model,
- percentile_cont(0.5) WITHIN GROUP (ORDER BY o.cost_usd) AS median_cost,
- percentile_cont(0.5) WITHIN GROUP (ORDER BY o.ms) AS median_ms
- FROM arena_outputs o JOIN arena_rounds r ON r.id = o."round"
- WHERE r.status = 'ready' AND o.error IS NULL AND ($1::text IS NULL OR r.category = $1)
- GROUP BY r.category, o.model
- )
- SELECT m.category, m.model, COALESCE(rt.elo, $2) AS elo,
- COALESCE(rt.games, 0) AS games, COALESCE(rt.wins, 0) AS wins,
- COALESCE(rt.losses, 0) AS losses, COALESCE(rt.ties, 0) AS ties,
- med.median_cost, med.median_ms
- FROM models m
- LEFT JOIN arena_ratings rt ON rt.category = m.category AND rt.model = m.model
- LEFT JOIN medians med ON med.category = m.category AND med.model = m.model
- ORDER BY m.category, elo DESC, games DESC, m.model`,
- [category, BASE_ELO],
- );
- const all = rows.map((r) => {
- const games = num(r.games);
- const wins = num(r.wins);
- const ties = num(r.ties);
- return {
- category: r.category,
- model: r.model,
- elo: Math.round(num(r.elo, BASE_ELO) * 100) / 100,
- games,
- wins,
- losses: num(r.losses),
- ties,
- // A tie is half a win, as in Elo's expected score.
- win_rate: games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0,
- median_cost_usd: r.median_cost == null ? null : Math.round(num(r.median_cost) * 1e6) / 1e6,
- median_cost: r.median_cost == null ? null : Math.round(num(r.median_cost) * 1e6) / 1e6,
- median_ms: r.median_ms == null ? null : Math.round(num(r.median_ms)),
- };
- });
- if (category) return all;
- return CATEGORIES.map((c) => ({ category: c, leaderboard: all.filter((r) => r.category === c) }));
+ async function leaderboard(query = null) {
+ const kindId = (!query || query === 'all') ? null : categoryToKind(query);
+ if (kindId) {
+ const elo = await bench.getBoard(kindId, { type: 'elo' });
+ const rows = (elo.rows || []).map(boardRowToLocal);
+ return [{ category: kindId === 'design.component' ? 'logo' : 'design.page', leaderboard: rows }];
+ }
+ const kinds = ['design.page', 'design.component'];
+ const out = [];
+ for (const kindId of kinds) {
+ const elo = await bench.getBoard(kindId, { type: 'elo' });
+ const category = kindId === 'design.component' ? 'logo' : 'design.page';
+ out.push({ category, leaderboard: (elo.rows || []).map(boardRowToLocal) });
+ }
+ return out;
}
}
- /** The open dataset: prompts, every output (HTML included) and every vote without its voter. */
- async function dataset() {
- const rounds = await db.many('SELECT id, category, prompt, status, created_at FROM arena_rounds ORDER BY created_at, id');
- const outputs = (await db.many('SELECT * FROM arena_outputs ORDER BY created_at, id')).map(outputOf);
- const votes = (await db.many('SELECT "round", left_output, right_output, winner, at FROM arena_votes ORDER BY at, "round"')).map(voteOf);
- const byRound = new Map(rounds.map((r) => [r.id, { ...r, outputs: [], votes: [] }]));
- for (const o of outputs) { const r = byRound.get(o.round); if (r) r.outputs.push(o); }
- for (const v of votes) { const r = byRound.get(v.round); if (r) r.votes.push(v); }
- return [...byRound.values()];
+ async function costLeaderboard(query = null) {
+ const kindId = (!query || query === 'all') ? null : categoryToKind(query);
+ if (kindId) {
+ const cost = await bench.getBoard(kindId, { type: 'cost' });
+ return [{ category: kindId === 'design.component' ? 'logo' : 'design.page', leaderboard: (cost.rows || []).map(boardRowToLocal) }];
+ }
+ const kinds = ['design.page', 'design.component'];
+ const out = [];
+ for (const kindId of kinds) {
+ const cost = await bench.getBoard(kindId, { type: 'cost' });
+ const category = kindId === 'design.component' ? 'logo' : 'design.page';
+ out.push({ category, leaderboard: (cost.rows || []).map(boardRowToLocal) });
+ }
+ return out;
}
}
async function datasetJsonl() {
async function datasetJsonl() {
- const lines = (await dataset()).map((r) => JSON.stringify(r));
- return lines.length ? `${lines.join('\n')}\n` : '';
+ // The open dataset now lives on Bench; Design redirects to it.
+ return { redirect: bench.datasetUrl('design.page') };
}
}
- /** A current day's budget summary for the pages and the create answer. */
return {
return {
- CATEGORIES, WINNERS, K, BASE_ELO,
- counts, createRound, getRound, nextPair, vote, leaderboard, dataset, datasetJsonl, shuffle,
+ CATEGORIES, WINNERS, K, BASE_ELO, shuffle,
+ counts, createRound, getRound, nextPair, vote, leaderboard, costLeaderboard, datasetJsonl,
};
};
}
}
@@ -64,6 +64,15 @@ function load(env = process.env) {
@@ -64,6 +64,15 @@ function load(env = process.env) {
maxHtmlBytes: 200 * 1024,
maxHtmlBytes: 200 * 1024,
},
},
+ // OpenVibe.Bench: Design creates tasks with its service token. A round uses its creator's Network token
+ // so Bench can enforce personal budgets and prevent voting on one's own round.
+ bench: {
+ url: trim(env.OV_BENCH_INTERNAL_URL || 'http://127.0.0.1:5230'),
+ publicUrl: trim(env.OV_BENCH_URL || 'https://bench.openvibe.services'),
+ audience: env.DESIGN_BENCH_AUDIENCE || 'openvibe.bench',
+ scope: env.DESIGN_BENCH_SCOPE || 'bench.task.create bench.round.create bench.round.read bench.judgement.cast bench.board.read',
+ },
+
// Arena budgets and jobs. Both limits are per UTC day; the drill flag keeps the worker from starting.
// Arena budgets and jobs. Both limits are per UTC day; the drill flag keeps the worker from starting.
arena: {
arena: {
perDay: Math.max(1, int(env.DESIGN_ARENA_PER_DAY, 3)),
perDay: Math.max(1, int(env.DESIGN_ARENA_PER_DAY, 3)),
@@ -4,24 +4,29 @@
@@ -4,24 +4,29 @@
* The Arena API, /api/v1/arena/*:
* The Arena API, /api/v1/arena/*:
*
*
* GET /categories the six categories
* GET /categories the six categories
- * POST /rounds start a round (signed in, same-origin, daily budgets)
+ * POST /rounds start a Bench task and round (signed in, same-origin)
* GET /rounds/:id one round with its outputs
* GET /rounds/:id one round with its outputs
* GET /vote the next blind pair for this person
* GET /vote the next blind pair for this person
- * POST /vote record a vote, reveal the two models and update Elo
- * GET /leaderboard[?category=…] Elo, games, win rate, median cost and ms
- * GET /dataset.jsonl every prompt, output and vote, without voter ids
+ * POST /vote Bench records a vote and reveals the two models
+ * GET /leaderboard[?type=cost] Bench Elo or Cost board
+ * GET /dataset.jsonl redirect to Bench's design.page dataset
*
*
- * A round is queued here; the job loop generates it. The request never waits for a model.
+ * Bench queues and generates each round. The request never waits for a model.
*/
*/
const { http } = require('openvibe-contracts');
const { http } = require('openvibe-contracts');
const { asyncRouter } = require('../http/router');
const { asyncRouter } = require('../http/router');
const { ArenaError, CATEGORIES } = require('../arena/store');
const { ArenaError, CATEGORIES } = require('../arena/store');
+const { BenchClientError } = require('../arena/bench');
const { ownOrigin, requireSameOrigin } = require('./same-origin');
const { ownOrigin, requireSameOrigin } = require('./same-origin');
function subjectOf(req) {
function subjectOf(req) {
return req.viewer && req.viewer.kind === 'user' && req.viewer.subject ? req.viewer.subject : null;
return req.viewer && req.viewer.kind === 'user' && req.viewer.subject ? req.viewer.subject : null;
}
}
+function tokenOf(req) {
+ return req.viewer && req.viewer.kind === 'user' ? req.viewer.token : null;
+}
+
function createArenaApi(ctx) {
function createArenaApi(ctx) {
const { arena, config, log } = ctx;
const { arena, config, log } = ctx;
const r = asyncRouter();
const r = asyncRouter();
@@ -39,6 +44,7 @@ function createArenaApi(ctx) {
@@ -39,6 +44,7 @@ function createArenaApi(ctx) {
const failed = (res, req, err) => {
const failed = (res, req, err) => {
if (err instanceof ArenaError) return problem(res, req, err.status, err.code, err.detail);
if (err instanceof ArenaError) return problem(res, req, err.status, err.code, err.detail);
+ if (err instanceof BenchClientError) return problem(res, req, err.status >= 500 ? err.status : 502, err.code, err.detail);
log.warn('[OpenVibe.Design] Arena API failed:', (err && err.message) || err);
log.warn('[OpenVibe.Design] Arena API failed:', (err && err.message) || err);
return problem(res, req, 500, 'internal.error', 'The Arena could not answer. Try again in a moment.');
return problem(res, req, 500, 'internal.error', 'The Arena could not answer. Try again in a moment.');
};
};
@@ -47,14 +53,20 @@ function createArenaApi(ctx) {
@@ -47,14 +53,20 @@ function createArenaApi(ctx) {
async function leaderboard(req, res) {
async function leaderboard(req, res) {
const category = req.query.category == null || req.query.category === '' ? null : String(req.query.category);
const category = req.query.category == null || req.query.category === '' ? null : String(req.query.category);
+ const kind = req.query.kind == null || req.query.kind === '' ? null : String(req.query.kind);
+ const type = req.query.type === 'cost' ? 'cost' : 'elo';
if (category && !CATEGORIES.includes(category)) {
if (category && !CATEGORIES.includes(category)) {
return problem(res, req, 400, 'arena.invalid_category', `Pick a category: ${CATEGORIES.join(', ')}.`);
return problem(res, req, 400, 'arena.invalid_category', `Pick a category: ${CATEGORIES.join(', ')}.`);
}
}
+ if (kind && kind !== 'design.page' && kind !== 'design.component') {
+ return problem(res, req, 400, 'arena.invalid_category', 'Pick design.page or design.component.');
+ }
try {
try {
- const list = await arena.leaderboard(category);
+ const query = kind || category || null;
+ const list = type === 'cost' ? await arena.costLeaderboard(query) : await arena.leaderboard(query);
res.set('Cache-Control', 'public, max-age=30');
res.set('Cache-Control', 'public, max-age=30');
- if (category) return res.json({ category, leaderboard: list });
- return res.json({ categories: list, leaderboard: list.flatMap((c) => c.leaderboard) });
+ if (query) return res.json({ category: query, type, leaderboard: list[0] ? list[0].leaderboard : [] });
+ return res.json({ categories: list, type, leaderboard: list.flatMap((c) => c.leaderboard) });
} catch (err) { return failed(res, req, err); }
} catch (err) { return failed(res, req, err); }
}
}
r.get('/leaderboard', leaderboard);
r.get('/leaderboard', leaderboard);
@@ -62,7 +74,12 @@ function createArenaApi(ctx) {
@@ -62,7 +74,12 @@ function createArenaApi(ctx) {
async function dataset(req, res) {
async function dataset(req, res) {
try {
try {
- const text = await arena.datasetJsonl();
+ const out = await arena.datasetJsonl();
+ if (out && out.redirect) {
+ res.set('Cache-Control', 'public, max-age=60');
+ return res.redirect(302, out.redirect);
+ }
+ const text = typeof out === 'string' ? out : '';
res.set('Cache-Control', 'public, max-age=60');
res.set('Cache-Control', 'public, max-age=60');
res.set('Content-Disposition', 'attachment; filename="arena-dataset.jsonl"');
res.set('Content-Disposition', 'attachment; filename="arena-dataset.jsonl"');
return res.type('application/x-ndjson').send(text);
return res.type('application/x-ndjson').send(text);
@@ -74,7 +91,7 @@ function createArenaApi(ctx) {
@@ -74,7 +91,7 @@ function createArenaApi(ctx) {
r.post('/rounds', requireOrigin, requireSubject, async (req, res) => {
r.post('/rounds', requireOrigin, requireSubject, async (req, res) => {
const b = req.body && typeof req.body === 'object' ? req.body : {};
const b = req.body && typeof req.body === 'object' ? req.body : {};
try {
try {
- const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req) });
+ const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req), token: tokenOf(req) });
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
res.set('Location', `/api/v1/arena/rounds/${round.id}`);
res.set('Location', `/api/v1/arena/rounds/${round.id}`);
return res.status(201).json({ round });
return res.status(201).json({ round });
@@ -83,9 +100,9 @@ function createArenaApi(ctx) {
@@ -83,9 +100,9 @@ function createArenaApi(ctx) {
async function readRound(req, res) {
async function readRound(req, res) {
try {
try {
- const round = await arena.getRound(String(req.params.id || ''));
+ const round = await arena.getRound(String(req.params.id || ''), { token: tokenOf(req) });
if (!round) return problem(res, req, 404, 'arena.round_not_found', 'No such Arena round.');
if (!round) return problem(res, req, 404, 'arena.round_not_found', 'No such Arena round.');
- res.set('Cache-Control', 'public, max-age=30');
+ res.set('Cache-Control', tokenOf(req) ? 'private, no-store' : 'public, max-age=30');
return res.json({ round });
return res.json({ round });
} catch (err) { return failed(res, req, err); }
} catch (err) { return failed(res, req, err); }
}
}
@@ -94,7 +111,9 @@ function createArenaApi(ctx) {
@@ -94,7 +111,9 @@ function createArenaApi(ctx) {
async function nextPair(req, res) {
async function nextPair(req, res) {
try {
try {
- const pair = await arena.nextPair(subjectOf(req), { random: ctx.random || Math.random });
+ const category = String(req.query.category || 'landing');
+ if (!CATEGORIES.includes(category)) return problem(res, req, 400, 'arena.invalid_category', 'Pick a valid category.');
+ const pair = await arena.nextPair(subjectOf(req), { token: tokenOf(req), category, random: ctx.random || Math.random });
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
if (!pair) return res.json({ pair: null, detail: 'There is no round waiting for your vote. Start one, or come back when another is ready.' });
if (!pair) return res.json({ pair: null, detail: 'There is no round waiting for your vote. Start one, or come back when another is ready.' });
return res.json(pair);
return res.json(pair);
@@ -113,6 +132,7 @@ function createArenaApi(ctx) {
@@ -113,6 +132,7 @@ function createArenaApi(ctx) {
leftOutput: String(b.left_output || b.left || ''),
leftOutput: String(b.left_output || b.left || ''),
rightOutput: String(b.right_output || b.right || ''),
rightOutput: String(b.right_output || b.right || ''),
winner: String(b.winner || ''),
winner: String(b.winner || ''),
+ token: tokenOf(req),
});
});
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
return res.status(201).json({
return res.status(201).json({
@@ -37,6 +37,7 @@ function meOf(viewer) {
@@ -37,6 +37,7 @@ function meOf(viewer) {
}
}
const subjectOf = (req) => (req.viewer && req.viewer.kind === 'user' ? req.viewer.subject : null);
const subjectOf = (req) => (req.viewer && req.viewer.kind === 'user' ? req.viewer.subject : null);
+const tokenOf = (req) => (req.viewer && req.viewer.kind === 'user' ? req.viewer.token : null);
function createPageRoutes(ctx) {
function createPageRoutes(ctx) {
const { config, catalog, arena, log } = ctx;
const { config, catalog, arena, log } = ctx;
@@ -83,11 +84,16 @@ function createPageRoutes(ctx) {
@@ -83,11 +84,16 @@ function createPageRoutes(ctx) {
r.get('/arena', async (req, res) => {
r.get('/arena', async (req, res) => {
const me = meOf(req.viewer);
const me = meOf(req.viewer);
let boards;
let boards;
- try { boards = await arena.leaderboard(null); } catch (err) {
+ let costBoards;
+ try {
+ boards = await arena.leaderboard(null);
+ costBoards = await arena.costLeaderboard(null);
+ } catch (err) {
log.warn('[OpenVibe.Design] leaderboard failed:', (err && err.message) || err);
log.warn('[OpenVibe.Design] leaderboard failed:', (err && err.message) || err);
boards = categories.map((category) => ({ category, leaderboard: [] }));
boards = categories.map((category) => ({ category, leaderboard: [] }));
+ costBoards = [];
}
}
- return page(res, 200, views.arenaHome({ categories: boards, me, limits: await aria(req.viewer) }), req.viewer);
+ return page(res, 200, views.arenaHome({ categories: boards, costCategories: costBoards, me, limits: await aria(req.viewer) }), req.viewer);
});
});
r.get('/arena/new', async (req, res) => {
r.get('/arena/new', async (req, res) => {
@@ -100,7 +106,7 @@ function createPageRoutes(ctx) {
@@ -100,7 +106,7 @@ function createPageRoutes(ctx) {
if (!writeAllowed(req, config)) return res.status(403).type('text/plain').send('A round must be started from openvibe.design itself.');
if (!writeAllowed(req, config)) return res.status(403).type('text/plain').send('A round must be started from openvibe.design itself.');
const b = req.body || {};
const b = req.body || {};
try {
try {
- const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req) });
+ const round = await arena.createRound({ category: String(b.category || ''), prompt: b.prompt, subject: subjectOf(req), token: tokenOf(req) });
return res.redirect(303, `/arena/r/${encodeURIComponent(round.id)}`);
return res.redirect(303, `/arena/r/${encodeURIComponent(round.id)}`);
} catch (err) {
} catch (err) {
const notice = err instanceof ArenaError ? err.detail : 'The round could not be started. Try again in a moment.';
const notice = err instanceof ArenaError ? err.detail : 'The round could not be started. Try again in a moment.';
@@ -115,7 +121,7 @@ function createPageRoutes(ctx) {
@@ -115,7 +121,7 @@ function createPageRoutes(ctx) {
let pair = null;
let pair = null;
let notice = null;
let notice = null;
if (subjectOf(req)) {
if (subjectOf(req)) {
- try { pair = await arena.nextPair(subjectOf(req), { random: ctx.random || Math.random }); } catch (err) {
+ try { pair = await arena.nextPair(subjectOf(req), { token: tokenOf(req), category: req.query.category === 'logo' ? 'logo' : 'landing', random: ctx.random || Math.random }); } catch (err) {
log.warn('[OpenVibe.Design] picking a pair failed:', (err && err.message) || err);
log.warn('[OpenVibe.Design] picking a pair failed:', (err && err.message) || err);
notice = 'The next pair could not be read. Try again in a moment.';
notice = 'The next pair could not be read. Try again in a moment.';
}
}
@@ -135,6 +141,7 @@ function createPageRoutes(ctx) {
@@ -135,6 +141,7 @@ function createPageRoutes(ctx) {
leftOutput: String(b.left_output || ''),
leftOutput: String(b.left_output || ''),
rightOutput: String(b.right_output || ''),
rightOutput: String(b.right_output || ''),
winner: String(b.winner || ''),
winner: String(b.winner || ''),
+ token: tokenOf(req),
});
});
return page(res, 201, views.arenaVote({ me, result: {
return page(res, 201, views.arenaVote({ me, result: {
vote: { round: out.round.id, winner: out.winner, at: out.at },
vote: { round: out.round.id, winner: out.winner, at: out.at },
@@ -149,7 +156,7 @@ function createPageRoutes(ctx) {
@@ -149,7 +156,7 @@ function createPageRoutes(ctx) {
r.get('/arena/r/:id', async (req, res) => {
r.get('/arena/r/:id', async (req, res) => {
let round = null;
let round = null;
- try { round = await arena.getRound(String(req.params.id || '')); } catch (err) {
+ try { round = await arena.getRound(String(req.params.id || ''), { token: tokenOf(req) }); } catch (err) {
log.warn('[OpenVibe.Design] round page failed:', (err && err.message) || err);
log.warn('[OpenVibe.Design] round page failed:', (err && err.message) || err);
}
}
if (!round) return page(res, 404, views.notFound(), req.viewer);
if (!round) return page(res, 404, views.notFound(), req.viewer);
@@ -161,6 +161,8 @@ const ARENA_CATEGORY = {
@@ -161,6 +161,8 @@ const ARENA_CATEGORY = {
game_ui: 'Game UI',
game_ui: 'Game UI',
logo: 'Logo',
logo: 'Logo',
slides: 'Slides',
slides: 'Slides',
+ 'design.page': 'Design page arena',
+ 'design.component': 'Design component arena',
};
};
const arenaCost = (v) => `$${Number(v || 0).toFixed(4)}`;
const arenaCost = (v) => `$${Number(v || 0).toFixed(4)}`;
@@ -189,24 +191,41 @@ function arenaViewSwitch() {
@@ -189,24 +191,41 @@ function arenaViewSwitch() {
<input type="radio" id="ar-view-phone" name="ar_view" value="phone" form="ar-view-none"><label for="ar-view-phone">Phone</label></div>`;
<input type="radio" id="ar-view-phone" name="ar_view" value="phone" form="ar-view-none"><label for="ar-view-phone">Phone</label></div>`;
}
}
-function arenaLeaderboard(category, rows) {
- const body = rows.length ? rows.map((r, i) => `<tr class="ar-lb-row" data-model="${esc(r.model)}">
+function arenaLeaderboard(category, rows, { type = 'elo' } = {}) {
+ if (type === 'cost') {
+ const body = rows.length ? rows.map((r, i) => `<tr class="ar-lb-row" data-model="${esc(r.model)}">
<td class="ar-lb-rank">${i + 1}</td>
<td class="ar-lb-rank">${i + 1}</td>
<td class="ar-lb-model">${esc(r.model)}</td>
<td class="ar-lb-model">${esc(r.model)}</td>
-<td class="ar-lb-elo">${Number(r.elo).toFixed(0)}</td>
+<td class="ar-lb-entries">${r.entries}</td>
+<td class="ar-lb-usable">${r.usable}</td>
+<td class="ar-lb-total-cost">${arenaCost(r.total_cost_usd)}</td>
+<td class="ar-lb-solved">${r.solved}</td>
+<td class="ar-lb-cost-per-solved">${r.cost_per_solved == null ? '—' : arenaCost(r.cost_per_solved)}</td>
+<td class="ar-lb-cost">${r.median_cost_usd == null ? '—' : arenaCost(r.median_cost_usd)}</td>
+<td class="ar-lb-ms">${r.median_ms == null ? '—' : arenaMs(r.median_ms)}</td></tr>`).join('') : '<tr class="ar-lb-empty"><td colspan="9">No entries yet. Start a round to generate some.</td></tr>';
+ return `<section class="ar-lb ar-lb--cost" id="ar-lb-cost-${esc(category)}" aria-labelledby="ar-lb-cost-h-${esc(category)}">
+<div class="ar-lb-head"><h2 id="ar-lb-cost-h-${esc(category)}">${esc(ARENA_CATEGORY[category] || category)} · Cost</h2></div>
+<table class="ar-table"><thead><tr><th scope="col">#</th><th scope="col">Model</th><th scope="col">Entries</th><th scope="col">Usable</th><th scope="col">Total cost</th><th scope="col">Solved</th><th scope="col">Cost per solved</th><th scope="col">Median cost</th><th scope="col">Median time</th></tr></thead>
+<tbody>${body}</tbody></table></section>`;
+ }
+ const body = rows.length ? rows.map((r, i) => `<tr class="ar-lb-row" data-model="${esc(r.model)}">
+<td class="ar-lb-rank">${i + 1}</td>
+<td class="ar-lb-model">${esc(r.model)}${r.provisional ? ' <span class="ar-provisional" title="Fewer than 10 games">provisional</span>' : ''}</td>
+<td class="ar-lb-elo">${Number(r.elo).toFixed(0)}${r.ci_lower != null && r.ci_upper != null ? ` <span class="ar-ci">(${Number(r.ci_lower).toFixed(0)}–${Number(r.ci_upper).toFixed(0)})</span>` : ''}</td>
<td class="ar-lb-games">${r.games}</td>
<td class="ar-lb-games">${r.games}</td>
<td class="ar-lb-winrate">${arenaPct(r.win_rate)}</td>
<td class="ar-lb-winrate">${arenaPct(r.win_rate)}</td>
<td class="ar-lb-cost">${r.median_cost_usd == null ? '—' : arenaCost(r.median_cost_usd)}</td>
<td class="ar-lb-cost">${r.median_cost_usd == null ? '—' : arenaCost(r.median_cost_usd)}</td>
<td class="ar-lb-ms">${r.median_ms == null ? '—' : arenaMs(r.median_ms)}</td></tr>`).join('') : '<tr class="ar-lb-empty"><td colspan="7">No votes yet. Be the first to judge this category.</td></tr>';
<td class="ar-lb-ms">${r.median_ms == null ? '—' : arenaMs(r.median_ms)}</td></tr>`).join('') : '<tr class="ar-lb-empty"><td colspan="7">No votes yet. Be the first to judge this category.</td></tr>';
return `<section class="ar-lb" id="ar-lb-${esc(category)}" aria-labelledby="ar-lb-h-${esc(category)}">
return `<section class="ar-lb" id="ar-lb-${esc(category)}" aria-labelledby="ar-lb-h-${esc(category)}">
-<div class="ar-lb-head"><h2 id="ar-lb-h-${esc(category)}">${esc(ARENA_CATEGORY[category] || category)}</h2><a class="ar-more" href="/arena/vote">Vote on this category</a></div>
+<div class="ar-lb-head"><h2 id="ar-lb-h-${esc(category)}">${esc(ARENA_CATEGORY[category] || category)}</h2><a class="ar-more" href="/arena/vote${category === 'logo' ? '?category=logo' : ''}">Vote on this category</a></div>
<table class="ar-table"><thead><tr><th scope="col">#</th><th scope="col">Model</th><th scope="col">Elo</th><th scope="col">Games</th><th scope="col">Win rate</th><th scope="col">Median cost</th><th scope="col">Median time</th></tr></thead>
<table class="ar-table"><thead><tr><th scope="col">#</th><th scope="col">Model</th><th scope="col">Elo</th><th scope="col">Games</th><th scope="col">Win rate</th><th scope="col">Median cost</th><th scope="col">Median time</th></tr></thead>
<tbody>${body}</tbody></table></section>`;
<tbody>${body}</tbody></table></section>`;
}
}
-function arenaHome({ categories = [], me = {}, limits = null } = {}) {
+function arenaHome({ categories = [], costCategories = [], me = {}, limits = null } = {}) {
const signedIn = Boolean(me.signedIn);
const signedIn = Boolean(me.signedIn);
const boards = categories.map((c) => arenaLeaderboard(c.category, c.leaderboard || [])).join('');
const boards = categories.map((c) => arenaLeaderboard(c.category, c.leaderboard || [])).join('');
+ const costBoards = costCategories.map((c) => arenaLeaderboard(c.category, c.leaderboard || [], { type: 'cost' })).join('');
const rows = categories.flatMap((c) => c.leaderboard || []);
const rows = categories.flatMap((c) => c.leaderboard || []);
const body = `<div class="ar-page" id="ar-home">
const body = `<div class="ar-page" id="ar-home">
<header class="ar-hero"><p class="ar-kicker">Design Arena</p><h1>Which model designs it best?</h1>
<header class="ar-hero"><p class="ar-kicker">Design Arena</p><h1>Which model designs it best?</h1>
@@ -217,7 +236,9 @@ function arenaHome({ categories = [], me = {}, limits = null } = {}) {
@@ -217,7 +236,9 @@ function arenaHome({ categories = [], me = {}, limits = null } = {}) {
: 'Sign in to start a round or vote. The leaderboard and the dataset are open to everyone.'}</p></header>
: 'Sign in to start a round or vote. The leaderboard and the dataset are open to everyone.'}</p></header>
<section class="ar-boards" id="ar-leaderboard" aria-labelledby="ar-boards-h"><div class="ar-lb-head"><h2 id="ar-boards-h">The leaderboard</h2><span class="ar-fine">${rows.length} model${rows.length === 1 ? '' : 's'} ranked · K = 32</span></div>
<section class="ar-boards" id="ar-leaderboard" aria-labelledby="ar-boards-h"><div class="ar-lb-head"><h2 id="ar-boards-h">The leaderboard</h2><span class="ar-fine">${rows.length} model${rows.length === 1 ? '' : 's'} ranked · K = 32</span></div>
${boards || '<p class="ar-empty">No rounds have been judged yet. Start the first one.</p>'}
${boards || '<p class="ar-empty">No rounds have been judged yet. Start the first one.</p>'}
-<p class="ar-fine">A win rate counts a tie as half a win. <code>both_bad</code> is a game for neither model. The cost and time are the median over the outputs in this category.</p></section>
+<p class="ar-fine">A win rate counts a tie as half a win. <code>both_bad</code> is a game for neither model. The cost and time are the median over the outputs. Ranges are 95% confidence intervals; <span class="ar-provisional">provisional</span> means fewer than 10 games.</p></section>
+<section class="ar-boards ar-boards--cost" id="ar-cost" aria-labelledby="ar-cost-h"><div class="ar-lb-head"><h2 id="ar-cost-h">Cost view</h2><span class="ar-fine">Per model across all entries</span></div>
+${costBoards || '<p class="ar-empty">No entries yet.</p>'}</section>
<section class="ar-open" id="ar-open"><h2>Everything is open</h2><p class="ar-lede">The prompts, every generated page and every vote are downloadable. No voter is named.</p>
<section class="ar-open" id="ar-open"><h2>Everything is open</h2><p class="ar-lede">The prompts, every generated page and every vote are downloadable. No voter is named.</p>
<p><a class="ovk-btn ovk-btn--quiet" href="/api/v1/arena/dataset.jsonl">Download dataset.jsonl</a></p></section>
<p><a class="ovk-btn ovk-btn--quiet" href="/api/v1/arena/dataset.jsonl">Download dataset.jsonl</a></p></section>
</div>`;
</div>`;
@@ -27,19 +27,15 @@ const bodyOf = (ev) => JSON.stringify({ event: ev, seq: 1 });
@@ -27,19 +27,15 @@ const bodyOf = (ev) => JSON.stringify({ event: ev, seq: 1 });
const ana = t.network.addUser('ana');
const ana = t.network.addUser('ana');
const bo = t.network.addUser('bo');
const bo = t.network.addUser('bo');
try {
try {
- // A starts a round (then it is given A's fixed account subject), and B votes on it.
- const aRound = (await t.get('/api/v1/arena/rounds', { as: ana, json: { category: 'landing', prompt: 'A round to export' }, origin: ORIGIN })).json().round;
- await t.tick();
- await t.ctx.s.db.exec('UPDATE arena_rounds SET created_by = $1 WHERE id = $2', [A, aRound.id]);
- const aOutputs = (await t.ctx.arena.getRound(aRound.id)).outputs;
- const pair = await t.ctx.arena.nextPair(bo.subject);
- const vote = await t.get('/api/v1/arena/vote', { as: bo, json: { round_id: pair.round.id, left_output: pair.left.id, right_output: pair.right.id, winner: 'left' }, origin: ORIGIN });
- assert.strictEqual(vote.status, 201, vote.text);
-
- // B starts a round and A's vote is stored on it directly (A is a fixture account, not a Network token here).
- const bRound = (await t.get('/api/v1/arena/rounds', { as: bo, json: { category: 'dashboard', prompt: 'B round stays' }, origin: ORIGIN })).json().round;
- await t.tick();
- const bOutputs = (await t.ctx.arena.getRound(bRound.id)).outputs;
+ // Legacy Arena tables stay read-only in the active flow. Seed historical data to exercise ADR-033.
+ const aRound = { id: 'arr_legacy_a' };
+ const bRound = { id: 'arr_legacy_b' };
+ const db = t.ctx.s.db;
+ await db.exec("INSERT INTO arena_rounds (id, category, prompt, created_by, status) VALUES ($1, 'landing', 'A round to export', $2, 'ready'), ($3, 'dashboard', 'B round stays', $4, 'ready')", [aRound.id, A, bRound.id, B]);
+ await db.exec("INSERT INTO arena_outputs (id, \"round\", model, provider, html) VALUES ('aro_legacy_a1', $1, 'model-a', 'ai', '<h1>A</h1>'), ('aro_legacy_a2', $1, 'model-b', 'ai', '<h1>B</h1>'), ('aro_legacy_b1', $2, 'model-a', 'ai', '<h1>C</h1>'), ('aro_legacy_b2', $2, 'model-b', 'ai', '<h1>D</h1>')", [aRound.id, bRound.id]);
+ const aOutputs = await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 ORDER BY id', [aRound.id]);
+ const bOutputs = await db.many('SELECT * FROM arena_outputs WHERE "round" = $1 ORDER BY id', [bRound.id]);
+ await db.exec("INSERT INTO arena_votes (\"round\", voter_subject, left_output, right_output, winner) VALUES ($1, $2, $3, $4, 'left')", [aRound.id, bo.subject, aOutputs[0].id, aOutputs[1].id]);
await t.ctx.s.db.exec(`INSERT INTO arena_votes ("round", voter_subject, left_output, right_output, winner) VALUES ($1, $2, $3, $4, 'tie')`,
await t.ctx.s.db.exec(`INSERT INTO arena_votes ("round", voter_subject, left_output, right_output, winner) VALUES ($1, $2, $3, $4, 'tie')`,
[bRound.id, A, bOutputs[0].id, bOutputs[1].id]);
[bRound.id, A, bOutputs[0].id, bOutputs[1].id]);
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+'use strict';
+const assert = require('assert');
+const { createBenchClient } = require('../server/arena/bench');
+const { check, done } = require('./helpers/boot');
+
+(async () => {
+ const calls = [];
+ const fetchImpl = async (url, opts) => {
+ const path = new URL(url).pathname + new URL(url).search;
+ calls.push({ path, opts });
+ let body = {};
+ if (path === '/oauth/token') body = { access_token: 'service-token', token_type: 'Bearer', expires_in: 300 };
+ if (path === '/api/v1/kinds/design.page/tasks') body = { task: { id: 'btk_1', prompt: 'Page', task_class: 'landing' } };
+ if (path === '/api/v1/tasks/btk_1/rounds') body = { round: { id: 'brd_1' } };
+ if (path === '/api/v1/judgements/next?kind=design.page') body = { round: { id: 'brd_1' }, left: { id: 'bte_1', payload: { html: 'a' } }, right: { id: 'bte_2', payload: { html: 'b' } } };
+ if (path === '/api/v1/rounds/brd_1/judgements') body = { judgement: { at: 'now' }, revealed: {} };
+ return { ok: true, status: 200, json: async () => body };
+ };
+ const config = { bench: { url: 'http://bench.test', publicUrl: 'https://bench.openvibe.services', audience: 'openvibe.bench', scope: 'bench.task.create' },
+ oauth: { clientId: 'design', clientSecret: 'fixture' }, networkInternalUrl: 'http://network.test' };
+ const client = createBenchClient({ config, fetchImpl });
+ await check('task uses service token and round uses the person token', async () => {
+ const result = await client.createRound({ category: 'landing', prompt: 'Page', token: 'person-token' });
+ assert.strictEqual(result.round.id, 'brd_1');
+ const task = calls.find((c) => c.path === '/api/v1/kinds/design.page/tasks');
+ const round = calls.find((c) => c.path === '/api/v1/tasks/btk_1/rounds');
+ assert.strictEqual(task.opts.headers.Authorization, 'Bearer service-token');
+ assert.strictEqual(round.opts.headers.Authorization, 'Bearer person-token');
+ assert.deepStrictEqual(JSON.parse(round.opts.body), {});
+ });
+ await check('Bench pair and vote use the person token and exact wire shape', async () => {
+ const pair = await client.nextPair('design.page', 'person-token');
+ assert.strictEqual(pair.round.id, 'brd_1');
+ await client.castVote({ roundId: 'brd_1', token: 'person-token', pair: { left: 'bte_1', right: 'bte_2' }, winner: 'left' });
+ const vote = calls.find((c) => c.path === '/api/v1/rounds/brd_1/judgements');
+ assert.strictEqual(vote.opts.headers.Authorization, 'Bearer person-token');
+ assert.deepStrictEqual(JSON.parse(vote.opts.body), { pair: { left: 'bte_1', right: 'bte_2' }, outcome: 'left' });
+ assert.strictEqual(client.datasetUrl('design.page'), 'https://bench.openvibe.services/api/v1/datasets/design.page.jsonl');
+ });
+ await check('imported votes use Bench service judgements without person identity', async () => {
+ await client.castImportedVote({ roundId: 'brd_1', pair: { left: 'bte_1', right: 'bte_2' }, winner: 'both_bad' });
+ const vote = calls.filter((c) => c.path === '/api/v1/rounds/brd_1/judgements').at(-1);
+ assert.strictEqual(vote.opts.headers.Authorization, 'Bearer service-token');
+ assert.deepStrictEqual(JSON.parse(vote.opts.body), {
+ judge: 'automatic_grader', judge_name: 'design.arena.legacy_vote',
+ pair: { left: 'bte_1', right: 'bte_2' }, outcome: 'both_bad',
+ });
+ });
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,51 @@
@@ -0,0 +1,51 @@
+'use strict';
+const assert = require('assert');
+const { migrate } = require('../scripts/arena-to-bench');
+const { check, done } = require('./helpers/boot');
+
+const rows = {
+ rounds: [{ id: 'arr_a', category: 'landing', prompt: 'A page', status: 'ready' }],
+ outputs: [
+ { id: 'aro_a', round: 'arr_a', provider: 'ai', model: 'alpha', html: '<h1>A</h1>', cost_usd: 0.01, ms: 3, tokens: 4 },
+ { id: 'aro_b', round: 'arr_a', provider: 'ai', model: 'beta', html: '<h1>B</h1>', cost_usd: 0.02, ms: 5, tokens: 7 },
+ ],
+ votes: [{ round: 'arr_a', left_output: 'aro_a', right_output: 'aro_b', winner: 'tie' }],
+};
+const db = (source) => ({ many: async (sql) => source[sql.includes('arena_rounds') ? 'rounds' : sql.includes('arena_outputs') ? 'outputs' : 'votes'] });
+
+(async () => {
+ await check('migration dry run reads legacy tables and writes nothing', async () => {
+ const result = await migrate({ db: db(rows), bench: null });
+ assert.deepStrictEqual(result, { dry_run: true, rounds: 1, entries: 2, judgements: 1, problems: [] });
+ });
+ await check('migration maps entries and casts an anonymous service judgement', async () => {
+ const calls = [];
+ const bench = {
+ createRound: async (input) => { calls.push(['round', input]); return { round: { id: 'brd_1' } }; },
+ addEntry: async (input) => { calls.push(['entry', input]); return { entry: { id: `bte_${calls.length}` } }; },
+ castImportedVote: async (input) => { calls.push(['vote', input]); },
+ };
+ await migrate({ db: db(rows), bench, apply: true });
+ assert.strictEqual(calls[0][1].category, 'landing');
+ assert.strictEqual(calls[1][1].payload.html, '<h1>A</h1>');
+ assert.deepStrictEqual(calls[3][1].pair, { left: 'bte_2', right: 'bte_3' });
+ assert.strictEqual(calls[3][1].winner, 'tie');
+ assert.ok(!JSON.stringify(calls).includes('voter_subject'));
+ });
+ await check('migration retains multiple votes and both_bad without voter identities', async () => {
+ const source = { ...rows, votes: [...rows.votes, { ...rows.votes[0], winner: 'both_bad' }] };
+ const winners = [];
+ const bench = {
+ createRound: async () => ({ round: { id: 'brd_1' } }),
+ addEntry: async (entry) => ({ entry: { id: entry.model } }),
+ castImportedVote: async (vote) => winners.push(vote.winner),
+ };
+ await migrate({ db: db(source), bench, apply: true });
+ assert.deepStrictEqual(winners, ['tie', 'both_bad']);
+ });
+ await check('invalid legacy references stop before any Bench write', async () => {
+ const source = { ...rows, votes: [...rows.votes, { ...rows.votes[0], left_output: 'missing' }] };
+ await assert.rejects(() => migrate({ db: db(source), bench: { createRound: () => { throw new Error('must not write'); } }, apply: true }), /Import refused/);
+ });
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,342 +1,101 @@
@@ -1,342 +1,101 @@
'use strict';
'use strict';
-/**
- * The Arena end to end: a round generates three outputs through the job, the sanitizer is applied before storage,
- * the budgets refuse in words, the pair is blind, one vote per person, the creator cannot vote, Elo handles a win,
- * a tie and both_bad, the leaderboard reports games/win rate/medians, and the dataset names no voter.
- */
const assert = require('assert');
const assert = require('assert');
const { boot, check, done } = require('./helpers/boot');
const { boot, check, done } = require('./helpers/boot');
-
const ORIGIN = 'https://openvibe.design';
const ORIGIN = 'https://openvibe.design';
(async () => {
(async () => {
- const t = await boot({ random: () => 0, env: { DESIGN_ARENA_PER_DAY: '8', DESIGN_ARENA_DAILY: '40' } });
+ const t = await boot();
const ana = t.network.addUser('ana');
const ana = t.network.addUser('ana');
const bo = t.network.addUser('bo');
const bo = t.network.addUser('bo');
- const cy = t.network.addUser('cy');
-
- const create = (user, category = 'landing', prompt = 'A landing page for a tiny coffee roastery in Seattle', extra = {}) => t.get('/api/v1/arena/rounds', {
- as: user, json: { category, prompt }, origin: ORIGIN, ...extra,
- });
- const getRound = async (id) => (await t.get(`/api/v1/arena/rounds/${id}`)).json().round;
- // Extra rounds would compete with A for the next blind pair; the checks that make them clean up after themselves.
- const dropRounds = async (...ids) => { for (const id of ids.filter(Boolean)) await t.ctx.s.db.exec('DELETE FROM arena_rounds WHERE id = $1', [id]); };
-
- let A;
- await check('a round is queued by the request, and no AI call happens in the request', async () => {
- const before = t.ai.requests.length;
- const r = await create(ana);
- assert.strictEqual(r.status, 201, r.text);
- A = r.json().round;
- assert.strictEqual(A.status, 'queued');
- assert.deepStrictEqual(A.outputs, []);
- assert.strictEqual(t.ai.requests.length, before, 'the job, not the request, calls AI');
- });
-
- await check('the job generates three outputs through AI with its reported model, provider, cost, time and tokens', async () => {
- const ready = await t.tick();
- assert.ok(ready, 'a queued round was claimed');
- assert.strictEqual(ready.id, A.id);
- assert.strictEqual(ready.status, 'ready');
- assert.strictEqual(ready.outputs.length, 3);
- const models = ready.outputs.map((o) => o.model);
- const providers = ready.outputs.map((o) => o.provider);
- assert.deepStrictEqual(models, ['alpha-1', 'beta-2', 'gamma-3']);
- assert.deepStrictEqual(providers, ['acme', 'beta', 'gamma']);
- for (const o of ready.outputs) {
- assert.ok(o.id.startsWith('aro_'));
- assert.strictEqual(o.error, null);
- assert.match(o.html, /^<!doctype html>/i);
- assert.strictEqual(o.cost_usd, 0.0012);
- assert.strictEqual(o.ms, 321);
- assert.strictEqual(o.tokens, 600);
- }
- assert.strictEqual(ready.pool_fallback, false, 'three offers were listed, so no pool fallback');
- assert.deepStrictEqual(ready.offers, [
- { provider: 'acme', model: 'alpha-1' },
- { provider: 'beta', model: 'beta-2' },
- { provider: 'gamma', model: 'gamma-3' },
- ]);
- const posts = t.ai.requests.filter((r) => r.method === 'POST' && r.path === '/api/v1/generate');
- assert.strictEqual(posts.length, 3);
- for (const [i, p] of posts.entries()) {
- assert.ok(p.body.options.cache === false);
- assert.strictEqual(p.body.options.model, undefined, 'the pin is a route, not an options.model hint');
- assert.deepStrictEqual(p.body.route, ready.offers[i], 'each slot asks for one distinct offer');
- assert.match(p.body.prompt, /coffee roastery/);
- assert.strictEqual(p.body.target.service, 'design');
- }
- });
-
- await check('Design asked Network for a client-credentials token for openvibe.ai with ai.run.create', async () => {
- const token = t.network.state.tokenRequests.find((f) => f.grant_type === 'client_credentials');
- assert.ok(token, 'a client-credentials grant was made');
- assert.strictEqual(token.client_id, 'design');
- assert.strictEqual(token.audience, 'openvibe.ai');
- assert.strictEqual(token.scope, 'ai.run.create');
- assert.ok(t.ai.state.tokenClaims);
- assert.ok(t.ai.state.tokenClaims.aud.includes('openvibe.ai'));
- assert.ok(t.ai.state.tokenClaims.cap.includes('ai.run.create'));
- assert.ok(t.ai.state.tokenClaims.ns.includes('ai.*'));
- });
-
- await check('the sanitizer runs before storage, including the 200 KB cap', async () => {
- t.ai.state.htmlFor = () => '<h1>safe</h1><script>alert(1)</script><img src="x" onerror="alert(2)"><a href="javascript:alert(3)">x</a>';
- const r = await create(ana, 'logo', 'A mark with a script');
- const scripted = r.json().round.id;
- await t.tick();
- const round = await getRound(scripted);
- for (const o of round.outputs) {
- assert.match(o.html, /safe/);
- assert.ok(!/<script/i.test(o.html));
- assert.ok(!/\sonerror\s*=/i.test(o.html));
- assert.ok(!/javascript\s*:/i.test(o.html));
- }
- t.ai.state.htmlFor = () => `<p>${'x'.repeat(300 * 1024)}</p>`;
- const big = await create(ana, 'logo', 'A big page');
- const bigId = big.json().round.id;
- await t.tick();
- const bigRound = await getRound(bigId);
- for (const o of bigRound.outputs) assert.ok(Buffer.byteLength(o.html) <= 200 * 1024, `${Buffer.byteLength(o.html)} bytes`);
- t.ai.state.htmlFor = null;
- await dropRounds(scripted, bigId);
- });
-
- await check('bad input and a cross-site Origin are refused in words', async () => {
- const signedOut = await t.get('/api/v1/arena/rounds', { json: { category: 'landing', prompt: 'x' } });
- assert.strictEqual(signedOut.status, 401);
- const badCat = await create(ana, 'poster');
- assert.strictEqual(badCat.status, 422);
- assert.match(badCat.json().detail, /category/i);
- const empty = await create(ana, 'landing', ' ');
- assert.strictEqual(empty.status, 422);
- assert.match(empty.json().detail, /describe/i);
- const cross = await create(ana, 'landing', 'x', { origin: 'https://evil.example' });
- assert.strictEqual(cross.status, 403);
- assert.strictEqual(cross.json().code, 'design.cross_site');
- });
-
- await check('a cookie write needs same-origin: missing headers are refused, Sec-Fetch-Site or Origin passes', async () => {
- const missing = await create(ana, 'landing', 'no header', { origin: null });
- assert.strictEqual(missing.status, 403);
- assert.strictEqual(missing.json().code, 'design.cross_site');
- assert.match(missing.json().detail, /openvibe\.design itself/);
- const byFetchSite = await create(ana, 'landing', 'fetch metadata', { origin: null, headers: { 'sec-fetch-site': 'same-origin' } });
- assert.strictEqual(byFetchSite.status, 201, byFetchSite.text);
- const crossSite = await create(ana, 'landing', 'cross site', { origin: null, headers: { 'sec-fetch-site': 'cross-site' } });
- assert.strictEqual(crossSite.status, 403);
- // A Bearer write is not cookie-authenticated: no Origin is needed.
- const bearer = await t.get('/api/v1/arena/rounds', {
- method: 'POST', json: { category: 'landing', prompt: 'a bearer write' },
- headers: { authorization: `Bearer ${t.network.userToken(ana)}` },
+ const create = (as, category, prompt) => t.get('/api/v1/arena/rounds', { as, json: { category, prompt }, origin: ORIGIN });
+ try {
+ let round;
+ await check('a signed-in start creates a Bench task and person-owned round', async () => {
+ const r = await create(ana, 'landing', 'A coffee roastery landing page');
+ assert.strictEqual(r.status, 201, r.text);
+ round = r.json().round;
+ assert.match(round.id, /^brd_/);
+ assert.strictEqual(round.category, 'landing');
+ assert.strictEqual(round.status, 'queued');
+ const task = [...t.bench.state.tasks.values()][0];
+ assert.strictEqual(task.kind_id, 'design.page');
+ assert.strictEqual(task.task_class, 'landing');
+ assert.strictEqual(t.bench.state.rounds.get(round.id).created_by, ana.subject);
+ const taskCall = t.bench.state.requests.find((x) => x.path.endsWith('/tasks') && x.method === 'POST');
+ const roundCall = t.bench.state.requests.find((x) => x.path.endsWith('/rounds') && x.method === 'POST');
+ assert.ok(taskCall && roundCall);
+ assert.notStrictEqual(taskCall.authorization, roundCall.authorization);
+ const grant = t.network.state.tokenRequests.find((x) => x.audience === 'openvibe.bench');
+ assert.ok(grant);
+ assert.match(grant.scope, /bench.task.create/);
});
});
- assert.strictEqual(bearer.status, 201, bearer.text);
- await dropRounds(byFetchSite.json().round.id, bearer.json().round.id);
- });
-
- await check('the no-JS forms enforce the same policy', async () => {
- const form = { 'content-type': 'application/x-www-form-urlencoded' };
- const body = 'category=landing&prompt=a+form+round';
- const missing = await t.get('/arena/new', { as: ana, method: 'POST', body, headers: form });
- assert.strictEqual(missing.status, 403);
- const same = await t.get('/arena/new', { as: ana, method: 'POST', body, headers: { ...form, 'sec-fetch-site': 'same-origin' } });
- assert.strictEqual(same.status, 303, same.text);
- assert.match(same.headers.get('location'), /^\/arena\/r\/arr_/);
- const cross = await t.get('/arena/new', { as: ana, method: 'POST', body, headers: { ...form, origin: 'https://evil.example' } });
- assert.strictEqual(cross.status, 403);
- await dropRounds(String(same.headers.get('location')).split('/').pop());
- });
-
- let winner;
- let loser;
- await check('the pair is blind: no model, provider, cost or time before the vote', async () => {
- const pair = await t.get('/api/v1/arena/vote', { as: bo });
- assert.strictEqual(pair.status, 200, pair.text);
- const p = pair.json();
- assert.strictEqual(p.round.id, A.id);
- assert.deepStrictEqual(Object.keys(p.left).sort(), ['html', 'id']);
- assert.deepStrictEqual(Object.keys(p.right).sort(), ['html', 'id']);
- assert.ok(!JSON.stringify(p).includes('alpha-1'));
- assert.ok(!JSON.stringify(p).includes('beta-2'));
- const creator = await t.get('/api/v1/arena/vote', { as: ana });
- assert.strictEqual(creator.json().pair, null, 'the creator cannot be offered their own round');
- t.state = p;
- });
-
- await check('the creator cannot vote on their own round', async () => {
- const p = t.state;
- const r = await t.get('/api/v1/arena/vote', { as: ana, json: { round_id: A.id, left_output: p.left.id, right_output: p.right.id, winner: 'left' }, origin: ORIGIN });
- assert.strictEqual(r.status, 403);
- assert.strictEqual(r.json().code, 'arena.creator_cannot_vote');
- });
- await check('bo votes once; the models are revealed and the vote is recorded', async () => {
- const p = t.state;
- const r = await t.get('/api/v1/arena/vote', { as: bo, json: { round_id: A.id, left_output: p.left.id, right_output: p.right.id, winner: 'left' }, origin: ORIGIN });
- assert.strictEqual(r.status, 201, r.text);
- const out = r.json();
- assert.strictEqual(out.vote.winner, 'left');
- assert.ok(out.revealed.left.model);
- assert.ok(out.revealed.right.model);
- const leftModel = out.revealed.left.model;
- const rightModel = out.revealed.right.model;
- assert.notStrictEqual(leftModel, rightModel);
- winner = leftModel;
- loser = rightModel;
- const again = await t.get('/api/v1/arena/vote', { as: bo, json: { round_id: A.id, left_output: p.left.id, right_output: p.right.id, winner: 'right' }, origin: ORIGIN });
- assert.strictEqual(again.status, 409);
- assert.strictEqual(again.json().code, 'arena.already_voted');
- });
-
- await check('the leaderboard updates Elo K=32, games and win rate, with median cost and time', async () => {
- const lb = (await t.get('/api/v1/arena/leaderboard?category=landing')).json().leaderboard;
- const w = lb.find((r) => r.model === winner);
- const l = lb.find((r) => r.model === loser);
- assert.ok(w && l);
- assert.strictEqual(w.elo, 1016);
- assert.strictEqual(l.elo, 984);
- assert.strictEqual(w.games, 1);
- assert.strictEqual(w.wins, 1);
- assert.strictEqual(w.losses, 0);
- assert.strictEqual(w.win_rate, 1);
- assert.strictEqual(l.games, 1);
- assert.strictEqual(l.wins, 0);
- assert.strictEqual(l.losses, 1);
- assert.strictEqual(l.win_rate, 0);
- assert.strictEqual(w.median_cost_usd, 0.0012);
- assert.strictEqual(w.median_ms, 321);
- });
-
- await check('a tie is half a win for each; both_bad is a game for neither', async () => {
- const b = (await create(bo, 'dashboard', 'An analytics dashboard')).json().round;
- // Earlier same-origin tests queued rounds too: drain until this one is generated.
- let bRound;
- for (let i = 0; i < 12; i++) {
- bRound = await getRound(b.id);
- if (bRound.status === 'ready' || bRound.status === 'failed') break;
+ await check('Bench generates entries; the creator can read a sanitized round', async () => {
await t.tick();
await t.tick();
- }
- assert.strictEqual(bRound.status, 'ready');
- const tie = await t.ctx.arena.vote({ roundId: b.id, subject: cy.subject, leftOutput: bRound.outputs[0].id, rightOutput: bRound.outputs[1].id, winner: 'tie' });
- assert.strictEqual(tie.winner, 'tie');
- const lb = (await t.get('/api/v1/arena/leaderboard?category=dashboard')).json().leaderboard;
- const tied = lb.filter((r) => [bRound.outputs[0].model, bRound.outputs[1].model].includes(r.model));
- for (const row of tied) {
- assert.strictEqual(row.games, 1);
- assert.strictEqual(row.ties, 1);
- assert.strictEqual(row.wins, 0);
- assert.strictEqual(row.win_rate, 0.5);
- assert.strictEqual(row.elo, 1000, 'equal ratings tie to the same Elo');
- }
-
- const c = (await create(cy, 'game_ui', 'A game HUD')).json().round;
- await t.tick();
- const cRound = await getRound(c.id);
- const before = new Map(lb.map((r) => [r.model, r.games]));
- const both = await t.ctx.arena.vote({ roundId: c.id, subject: ana.subject, leftOutput: cRound.outputs[0].id, rightOutput: cRound.outputs[1].id, winner: 'both_bad' });
- assert.strictEqual(both.winner, 'both_bad');
- const after = (await t.get('/api/v1/arena/leaderboard?category=game_ui')).json().leaderboard;
- for (const row of after) {
- assert.strictEqual(row.games, 0, 'both_bad counted a game for neither');
- assert.strictEqual(row.elo, 1000);
- assert.strictEqual(row.ties, 0);
- }
- assert.ok(before.size >= 0);
- });
-
- await check('the dataset has prompts, output HTML and votes, and no voter ids', async () => {
- const ds = await t.get('/api/v1/arena/dataset.jsonl');
- assert.strictEqual(ds.status, 200);
- assert.match(ds.headers.get('content-type'), /ndjson|jsonl/);
- const lines = ds.text.trim().split('\n').map((l) => JSON.parse(l));
- assert.ok(lines.length >= 3);
- const a = lines.find((r) => r.id === A.id);
- assert.strictEqual(a.prompt, 'A landing page for a tiny coffee roastery in Seattle');
- assert.strictEqual(a.outputs.length, 3);
- assert.ok(a.outputs.every((o) => typeof o.html === 'string' && o.html.length));
- assert.strictEqual(a.votes.length, 1);
- assert.strictEqual(a.votes[0].winner, 'left');
- assert.ok(!ds.text.includes(ana.subject));
- assert.ok(!ds.text.includes(bo.subject));
- assert.ok(!ds.text.includes(cy.subject));
- assert.ok(!/voter_subject|"voter"/.test(ds.text));
- });
+ const r = await t.get(`/api/v1/arena/rounds/${round.id}`, { as: ana });
+ assert.strictEqual(r.status, 200, r.text);
+ const ready = r.json().round;
+ assert.strictEqual(ready.status, 'ready');
+ assert.strictEqual(ready.outputs.length, 3);
+ assert.ok(ready.outputs.every((o) => o.model && /<html>/i.test(o.html)));
+ assert.strictEqual(Number(await t.ctx.s.db.value('SELECT COUNT(*) FROM arena_rounds')), 0);
+ });
- await check('the pages render the leaderboard, the form and the frames with the CSP and both widths', async () => {
- const home = await t.get('/arena');
- assert.strictEqual(home.status, 200);
- assert.match(home.text, /id="ar-leaderboard"/);
- assert.match(home.text, /id="ar-lb-landing"/);
- assert.match(home.text, new RegExp(winner.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')));
- const round = await t.get(`/arena/r/${A.id}`);
- assert.strictEqual(round.status, 200);
- assert.match(round.text, /sandbox=""/);
- assert.match(round.text, /default-src 'none'/);
- assert.match(round.text, /width="390"/);
- assert.match(round.text, /width="1280"/);
- assert.match(round.text, /Phone · 390 px/);
- assert.match(round.text, /Desktop · 1280 px/);
- assert.doesNotMatch(round.text, /allow-scripts/);
- assert.doesNotMatch(round.text, /allow-same-origin/);
- const form = await t.get('/arena/new', { as: ana });
- assert.match(form.text, /id="ar-new-form"/);
- assert.match(form.text, /id="ar-category"/);
- assert.match(form.text, /id="ar-prompt"/);
- const anonVote = await t.get('/arena/vote');
- assert.match(anonVote.text, /Sign in to vote/);
- const gone = await t.get('/arena/r/nope');
- assert.strictEqual(gone.status, 404);
- });
+ let pair;
+ await check('Bench supplies a blind pair and excludes its creator', async () => {
+ const r = await t.get('/api/v1/arena/vote', { as: bo });
+ assert.strictEqual(r.status, 200, r.text);
+ pair = r.json();
+ assert.strictEqual(pair.round.id, round.id);
+ assert.deepStrictEqual(Object.keys(pair.left).sort(), ['html', 'id']);
+ assert.ok(!JSON.stringify(pair).includes('alpha-1'));
+ const own = await t.get('/api/v1/arena/vote', { as: ana });
+ assert.strictEqual(own.json().pair, null);
+ });
- await check('only when fewer than three offers are listed does a round fall back to the pool, and say so', async () => {
- const two = await boot({ aiOptions: { models: [
- { provider: 'acme', model: 'alpha-1' },
- { provider: 'beta', model: 'beta-2' },
- ] } });
- try {
- const u = two.network.addUser('few');
- const r = await two.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'Two offers only' }, origin: ORIGIN });
+ await check('Bench refuses creator vote, records one vote, and reveals models', async () => {
+ const body = { round_id: round.id, left_output: pair.left.id, right_output: pair.right.id, winner: 'left' };
+ const own = await t.get('/api/v1/arena/vote', { as: ana, json: body, origin: ORIGIN });
+ assert.strictEqual(own.status, 403);
+ const r = await t.get('/api/v1/arena/vote', { as: bo, json: body, origin: ORIGIN });
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
- const done = await two.tick();
- assert.strictEqual(done.status, 'ready');
- assert.strictEqual(done.pool_fallback, true);
- assert.deepStrictEqual(done.offers, [
- { provider: 'acme', model: 'alpha-1' },
- { provider: 'beta', model: 'beta-2' },
- { provider: 'ai', model: null },
- ]);
- const posts = two.ai.requests.filter((x) => x.method === 'POST' && x.path === '/api/v1/generate');
- assert.strictEqual(posts.length, 3);
- assert.deepStrictEqual(posts[0].body.route, { provider: 'acme', model: 'alpha-1' });
- assert.deepStrictEqual(posts[1].body.route, { provider: 'beta', model: 'beta-2' });
- assert.strictEqual(posts[2].body.route, undefined, 'the third slot is the routing pool');
- const page = await two.get(`/arena/r/${done.id}`);
- assert.match(page.text, /id="ar-round-pool"/);
- assert.match(page.text, /Fewer than three distinct offers/);
- } finally { await two.close(); }
- });
+ assert.ok(r.json().revealed.left.model);
+ assert.ok(r.json().revealed.right.model);
+ const again = await t.get('/api/v1/arena/vote', { as: bo, json: body, origin: ORIGIN });
+ assert.strictEqual(again.status, 409);
+ });
+
+ await check('Elo and Cost views read Bench boards', async () => {
+ const elo = await t.get('/api/v1/arena/leaderboard?category=landing');
+ assert.strictEqual(elo.status, 200);
+ assert.ok(elo.json().leaderboard.some((r) => r.games === 1 && r.provisional && r.ci_lower != null));
+ const cost = await t.get('/api/v1/arena/leaderboard?type=cost&category=landing');
+ assert.strictEqual(cost.status, 200);
+ assert.ok(cost.json().leaderboard.some((r) => r.entries > 0 && r.cost_per_solved != null));
+ assert.ok(t.bench.state.requests.some((x) => x.path === '/api/v1/boards/design.page' && x.query === '?type=cost'));
+ const page = await t.get('/arena');
+ assert.match(page.text, /Cost view/);
+ assert.match(page.text, /provisional/);
+ });
- await check('the create page refuses over the person budget in words and keeps a 429', async () => {
- const small = await boot({ env: { DESIGN_ARENA_PER_DAY: '2', DESIGN_ARENA_DAILY: '3' } });
- try {
- const u = small.network.addUser('limit');
- const v = small.network.addUser('other');
- assert.strictEqual((await small.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'one' }, origin: ORIGIN })).status, 201);
- assert.strictEqual((await small.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'two' }, origin: ORIGIN })).status, 201);
- const refused = await small.get('/api/v1/arena/rounds', { as: u, json: { category: 'landing', prompt: 'three' }, origin: ORIGIN });
- assert.strictEqual(refused.status, 429);
- assert.strictEqual(refused.json().code, 'arena.limit_person');
- assert.match(refused.json().detail, /You can start 2 Arena rounds a day/);
- const page = await small.get('/arena/new', { as: u });
- assert.match(page.text, /0 of 2 rounds left/);
- assert.strictEqual((await small.get('/api/v1/arena/rounds', { as: v, json: { category: 'landing', prompt: 'v one' }, origin: ORIGIN })).status, 201);
- const site = await small.get('/api/v1/arena/rounds', { as: v, json: { category: 'landing', prompt: 'v two' }, origin: ORIGIN });
- assert.strictEqual(site.status, 429);
- assert.strictEqual(site.json().code, 'arena.limit_site');
- assert.match(site.json().detail, /reached its 3 rounds for today/);
- } finally { await small.close(); }
- });
+ await check('logo uses design.component and dataset redirects to public Bench', async () => {
+ const logo = await create(bo, 'logo', 'A simple mark');
+ assert.strictEqual(logo.status, 201, logo.text);
+ assert.strictEqual([...t.bench.state.tasks.values()].at(-1).kind_id, 'design.component');
+ await t.tick();
+ const vote = await t.get('/api/v1/arena/vote?category=logo', { as: ana });
+ assert.strictEqual(vote.status, 200);
+ assert.strictEqual(vote.json().round.id, logo.json().round.id);
+ const ds = await t.get('/api/v1/arena/dataset.jsonl');
+ assert.strictEqual(ds.status, 302);
+ assert.strictEqual(ds.headers.get('location'), `${t.bench.url}/api/v1/datasets/design.page.jsonl`);
+ });
- await t.close();
+ await check('bad input and cross-site cookie writes fail', async () => {
+ assert.strictEqual((await create(ana, 'poster', 'x')).status, 422);
+ assert.strictEqual((await create(ana, 'landing', ' ')).status, 422);
+ assert.strictEqual((await t.get('/api/v1/arena/rounds', { as: ana, json: { category: 'landing', prompt: 'x' }, origin: 'https://evil.example' })).status, 403);
+ });
+ } finally { await t.close(); }
done();
done();
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,569 @@
@@ -0,0 +1,569 @@
+'use strict';
+
+/**
+ * A focused fake OpenVibe.Bench for Design's Arena tests. It implements just the routes Design needs,
+ * verifies tokens against the shared fake Network public key, keeps state in memory, and can generate
+ * AI entries when tick() is called.
+ */
+const http = require('http');
+const crypto = require('crypto');
+const { serviceAuth, ids } = require('openvibe-contracts');
+
+const K = 32;
+const BASE_ELO = 1000;
+const PROVISIONAL_GAMES = 10;
+
+function b64url(o) { return Buffer.from(JSON.stringify(o)).toString('base64url'); }
+function signJwt(claims, privateKey) {
+ const input = `${b64url({ alg: 'RS256', typ: 'JWT' })}.${b64url(claims)}`;
+ return `${input}.${crypto.sign('RSA-SHA256', Buffer.from(input), privateKey).toString('base64url')}`;
+}
+
+function shuffle(list, random = Math.random) {
+ const a = list.slice();
+ for (let i = a.length - 1; i > 0; i--) {
+ const j = Math.floor(random() * (i + 1));
+ [a[i], a[j]] = [a[j], a[i]];
+ }
+ return a;
+}
+
+function num(v, d = 0) { return v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v); }
+
+function wilson(wins, games) {
+ if (!games) return { lower: 0, upper: 1 };
+ const z = 1.96;
+ const p = wins / games;
+ const denom = 1 + (z * z) / games;
+ const centre = (p + (z * z) / (2 * games)) / denom;
+ const width = (z * Math.sqrt((p * (1 - p) + (z * z) / (4 * games)) / games)) / denom;
+ return { lower: Math.max(0, centre - width), upper: Math.min(1, centre + width) };
+}
+
+function eloDelta(p) {
+ if (!(p > 0)) return -400;
+ if (p >= 1) return 400;
+ return 400 * Math.log10(p / (1 - p));
+}
+
+function ratingCi(elo, wins, losses, ties) {
+ const games = wins + losses + ties;
+ if (!games) return { ci_lower: null, ci_upper: null };
+ const p = (wins + ties * 0.5) / games;
+ const w = wilson(wins + ties * 0.5, games);
+ const delta = eloDelta(p);
+ return {
+ ci_lower: Math.round((elo + (eloDelta(w.lower) - delta)) * 100) / 100,
+ ci_upper: Math.round((elo + (eloDelta(w.upper) - delta)) * 100) / 100,
+ };
+}
+
+async function startBench({ network } = {}) {
+ const publicKey = network.publicPem;
+ const state = {
+ tasks: new Map(),
+ rounds: new Map(),
+ entries: new Map(),
+ judgements: [],
+ ratings: new Map(),
+ requests: [],
+ };
+ let nextId = 1;
+ const id = (prefix) => `${prefix}_${String(nextId++).padStart(6, '0')}`;
+
+ function verify(token) {
+ const r = serviceAuth.verifyServiceToken(token, { publicKey, issuer: network.url, audience: 'openvibe.bench' });
+ if (r.ok) return { kind: 'service', sub: r.claims.sub, cap: r.claims.cap || [] };
+ // try user token
+ const parts = String(token || '').split('.');
+ if (parts.length === 3) {
+ const ok = crypto.verify('RSA-SHA256', Buffer.from(`${parts[0]}.${parts[1]}`), publicKey, Buffer.from(parts[2], 'base64url'));
+ if (ok) {
+ const c = JSON.parse(Buffer.from(parts[1], 'base64url').toString('utf8'));
+ if (c.exp * 1000 > Date.now() && ids.isSubjectId('user', c.subject_id) &&
+ (c.aud === 'openvibe.network' || (Array.isArray(c.aud) && c.aud.includes('openvibe.network')))) {
+ return { kind: 'user', subject: c.subject_id, username: c.username };
+ }
+ }
+ }
+ return null;
+ }
+
+ function requireService(req) {
+ const v = verify(getAuth(req));
+ if (!v || v.kind !== 'service') throw problem(401, 'token.required', 'Send a service token.');
+ return v;
+ }
+
+ function requireCapability(req, cap) {
+ const v = requireService(req);
+ if (!v.cap.includes(cap)) throw problem(403, 'bench.capability_required', `Need ${cap}.`);
+ return v;
+ }
+
+ function requirePerson(req) {
+ const v = verify(getAuth(req));
+ if (!v || v.kind !== 'user') throw problem(401, 'bench.sign_in_required', 'Sign in.');
+ return v;
+ }
+
+ function getAuth(req) {
+ const h = String(req.headers.authorization || '');
+ return h.startsWith('Bearer ') ? h.slice(7) : null;
+ }
+
+ function problem(status, code, detail) {
+ const err = new Error(detail);
+ err.status = status;
+ err.code = code;
+ err.detail = detail;
+ return err;
+ }
+
+ const kinds = new Map([
+ ['design.page', { id: 'design.page', room: 'design', entry: 'html_page', judge: 'human_pairwise', name: 'Design page arena', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'html_page', payload: { html: 'string' } } }],
+ ['design.component', { id: 'design.component', room: 'design', entry: 'html_page', judge: 'human_pairwise', name: 'Design component arena', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'html_page', payload: { html: 'string' } } }],
+ ]);
+
+ function defaultHtml(prompt) {
+ const p = String(prompt || '').split('\n')[0].slice(0, 200);
+ return `<!doctype html><html><head><meta charset="utf-8"><title>A generated design</title></head><body><h1>A page that works</h1><p>${p}</p><button>Start</button></body></html>`;
+ }
+
+ async function generateEntries(round) {
+ const pool = [
+ { provider: 'acme', model: 'alpha-1' },
+ { provider: 'beta', model: 'beta-2' },
+ { provider: 'gamma', model: 'gamma-3' },
+ ];
+ const offers = round.offers && round.offers.length ? round.offers : pool;
+ for (let i = 0; i < 3; i++) {
+ const offer = offers[i] && offers[i].model ? offers[i] : pool[i];
+ const model = offer.model;
+ const provider = offer.provider;
+ addEntry(round.id, {
+ provider,
+ model,
+ route: offer,
+ payload: { html: defaultHtml(round.prompt) },
+ cost_usd: 0.0012,
+ ms: 321,
+ tokens: 600,
+ });
+ }
+ }
+
+ function addEntry(roundId, entry) {
+ const round = state.rounds.get(roundId);
+ if (!round) throw problem(404, 'bench.round_not_found', 'No such round.');
+ const e = {
+ id: id('bte'),
+ round_id: roundId,
+ provider: entry.provider || 'ai',
+ model: entry.model || 'unknown',
+ route: entry.route || {},
+ payload: entry.payload || {},
+ cost_usd: num(entry.cost_usd),
+ ms: num(entry.ms),
+ tokens: num(entry.tokens),
+ error: entry.error || null,
+ ui_check: entry.ui_check || null,
+ run_id: entry.run_id || null,
+ created_at: new Date().toISOString(),
+ };
+ state.entries.set(e.id, e);
+ const usable = [...state.entries.values()].filter((x) => x.round_id === roundId && !x.error).length;
+ if (usable >= 2 && (round.status === 'queued' || round.status === 'running')) {
+ round.status = 'finished';
+ round.finished_at = new Date().toISOString();
+ }
+ return e;
+ }
+
+ function ratingKey(kindId, subject) { return `${kindId}|${subject}`; }
+
+ function updateRating(kindId, left, right, outcome) {
+ if (outcome === 'both_bad') return;
+ if (!left || !right || left.model === right.model) return;
+ const models = [left.model, right.model];
+ for (const m of models) {
+ const key = ratingKey(kindId, m);
+ if (!state.ratings.has(key)) {
+ state.ratings.set(key, { kind_id: kindId, category: 'all', subject: m, elo: BASE_ELO, games: 0, wins: 0, losses: 0, ties: 0, win_rate: 0, ci_lower: null, ci_upper: null });
+ }
+ }
+ const a = state.ratings.get(ratingKey(kindId, left.model));
+ const b = state.ratings.get(ratingKey(kindId, right.model));
+ const sa = outcome === 'left' ? 1 : (outcome === 'right' ? 0 : 0.5);
+ const sb = 1 - sa;
+ const ea = 1 / (1 + 10 ** ((b.elo - a.elo) / 400));
+ const eb = 1 - ea;
+ const nextA = Math.round((a.elo + K * (sa - ea)) * 100) / 100;
+ const nextB = Math.round((b.elo + K * (sb - eb)) * 100) / 100;
+ for (const [row, elo, score, opp] of [[a, nextA, sa, sb], [b, nextB, sb, sa]]) {
+ const games = row.games + 1;
+ const wins = row.wins + (score > opp ? 1 : 0);
+ const losses = row.losses + (score < opp ? 1 : 0);
+ const ties = row.ties + (score === opp ? 1 : 0);
+ const wr = games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0;
+ const ci = ratingCi(elo, wins, losses, ties);
+ row.elo = elo;
+ row.games = games;
+ row.wins = wins;
+ row.losses = losses;
+ row.ties = ties;
+ row.win_rate = wr;
+ row.ci_lower = ci.ci_lower;
+ row.ci_upper = ci.ci_upper;
+ }
+ }
+
+ function blindEntry(e) {
+ return {
+ id: e.id,
+ round_id: e.round_id,
+ payload: e.payload,
+ error: e.error,
+ ui_check: e.ui_check,
+ run_id: e.run_id,
+ created_at: e.created_at,
+ };
+ }
+
+ function fullEntry(e) {
+ return { ...e, route: e.route || {} };
+ }
+
+ function revealed(round, subject) {
+ if (!subject) return false;
+ if (round.created_by === subject) return true;
+ return state.judgements.some((j) => j.round_id === round.id && j.subject === subject);
+ }
+
+ function wireRound(round, viewerSubject) {
+ const entries = [...state.entries.values()].filter((e) => e.round_id === round.id).sort((a, b) => a.created_at.localeCompare(b.created_at));
+ const isRevealed = revealed(round, viewerSubject);
+ const out = {
+ id: round.id,
+ task_id: round.task_id,
+ kind_id: round.kind_id,
+ offers: round.offers,
+ status: round.status,
+ seed: round.seed,
+ pool_fallback: round.pool_fallback,
+ receipt: round.receipt,
+ params: {},
+ created_by: round.created_by,
+ created_at: round.created_at,
+ finished_at: round.finished_at || null,
+ task_class: round.task_class,
+ prompt: round.prompt,
+ entries: entries.map((e) => isRevealed ? fullEntry(e) : blindEntry(e)),
+ votes: state.judgements.filter((j) => j.round_id === round.id).length,
+ };
+ if (!isRevealed) out.blind = true;
+ return out;
+ }
+
+ function subjectsFor(kindId) {
+ const set = new Set();
+ for (const e of state.entries.values()) {
+ const round = state.rounds.get(e.round_id);
+ if (round && round.kind_id === kindId && !e.error) set.add(e.model);
+ }
+ return [...set].sort();
+ }
+
+ function costRows(kindId) {
+ const byModel = new Map();
+ for (const e of state.entries.values()) {
+ const round = state.rounds.get(e.round_id);
+ if (!round || round.kind_id !== kindId || e.error) continue;
+ if (!byModel.has(e.model)) byModel.set(e.model, { model: e.model, entries: 0, usable: 0, total_cost: 0, costs: [], mss: [] });
+ const b = byModel.get(e.model);
+ b.entries++;
+ b.usable++;
+ b.total_cost += e.cost_usd;
+ b.costs.push(e.cost_usd);
+ b.mss.push(e.ms);
+ }
+ return [...byModel.values()].map((r) => {
+ const sortedCosts = r.costs.slice().sort((a, b) => a - b);
+ const sortedMs = r.mss.slice().sort((a, b) => a - b);
+ const median = (arr) => arr.length % 2 ? arr[Math.floor(arr.length / 2)] : (arr[arr.length / 2 - 1] + arr[arr.length / 2]) / 2;
+ const solved = r.usable;
+ return {
+ subject: r.model,
+ entries: r.entries,
+ usable: r.usable,
+ total_cost_usd: Math.round(r.total_cost * 1e6) / 1e6,
+ solved,
+ cost_per_solved: solved ? Math.round((r.total_cost / solved) * 1e6) / 1e6 : null,
+ median_cost_usd: median(sortedCosts),
+ median_ms: Math.round(median(sortedMs)),
+ };
+ }).sort((a, b) => (a.cost_per_solved == null ? Infinity : a.cost_per_solved) - (b.cost_per_solved == null ? Infinity : b.cost_per_solved) || a.subject.localeCompare(b.subject));
+ }
+
+ function board(kindId, type) {
+ if (!kinds.has(kindId)) return null;
+ const subjects = subjectsFor(kindId);
+ if (type === 'elo') {
+ const rows = subjects.map((subject) => {
+ const r = state.ratings.get(ratingKey(kindId, subject));
+ const games = r ? r.games : 0;
+ return {
+ kind_id: kindId, category: 'all', subject,
+ elo: r ? r.elo : BASE_ELO,
+ games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,
+ win_rate: r ? r.win_rate : 0,
+ ci_lower: r ? r.ci_lower : null,
+ ci_upper: r ? r.ci_upper : null,
+ provisional: games < PROVISIONAL_GAMES,
+ median_cost_usd: null,
+ median_ms: null,
+ };
+ });
+ rows.sort((a, b) => b.elo - a.elo || a.subject.localeCompare(b.subject));
+ // Add median cost/time from cost rows
+ const cost = costRows(kindId);
+ for (const row of rows) {
+ const c = cost.find((x) => x.subject === row.subject);
+ if (c) { row.median_cost_usd = c.median_cost_usd; row.median_ms = c.median_ms; }
+ }
+ return { kind_id: kindId, category: 'all', period: 'all', type: 'elo', rows };
+ }
+ if (type === 'cost') {
+ return { kind_id: kindId, category: 'all', period: 'all', type: 'cost', rows: costRows(kindId) };
+ }
+ return null;
+ }
+
+ function datasetJsonl(kindId) {
+ if (!kinds.has(kindId)) return null;
+ const lines = [];
+ for (const t of state.tasks.values()) {
+ if (t.kind_id === kindId && t.public) lines.push(JSON.stringify({ type: 'task', id: t.id, kind_id: t.kind_id, prompt: t.prompt, task_class: t.task_class, public: t.public, created_at: t.created_at }));
+ }
+ for (const r of state.rounds.values()) {
+ if (r.kind_id === kindId && r.status === 'finished') {
+ lines.push(JSON.stringify({ type: 'round', id: r.id, task_id: r.task_id, kind_id: r.kind_id, offers: r.offers, status: r.status, seed: r.seed, pool_fallback: r.pool_fallback, receipt: r.receipt, created_at: r.created_at, finished_at: r.finished_at }));
+ }
+ }
+ for (const e of state.entries.values()) {
+ const r = state.rounds.get(e.round_id);
+ if (r && r.kind_id === kindId && r.status === 'finished') lines.push(JSON.stringify({ type: 'entry', ...fullEntry(e) }));
+ }
+ for (const j of state.judgements) {
+ const r = state.rounds.get(j.round_id);
+ if (r && r.kind_id === kindId && r.status === 'finished') {
+ lines.push(JSON.stringify({ type: 'judgement', id: j.id, round_id: j.round_id, kind_id: j.kind_id, judge: j.judge, method: j.method, pair: j.pair, outcome: j.outcome, score: j.score, at: j.at }));
+ }
+ }
+ for (const r of state.ratings.values()) {
+ if (r.kind_id === kindId) lines.push(JSON.stringify({ type: 'rating', ...r }));
+ }
+ return lines.length ? `${lines.join('\n')}\n` : '';
+ }
+
+ const server = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', async () => {
+ const raw = Buffer.concat(chunks);
+ const text = raw.toString('utf8');
+ let body = {};
+ try { body = text ? JSON.parse(text) : {}; } catch { body = {}; }
+ const url = new URL(req.url, 'http://x');
+ state.requests.push({ method: req.method, path: url.pathname, query: url.search, authorization: req.headers.authorization || null, body: text });
+
+ const json = (status, obj, headers = {}) => {
+ res.writeHead(status, { 'Content-Type': 'application/json', ...headers });
+ res.end(JSON.stringify(obj));
+ };
+ const sendProblem = (status, code, detail) => {
+ res.writeHead(status, { 'Content-Type': 'application/problem+json' });
+ res.end(JSON.stringify({ type: `https://openvibe.network/problems/${code}`, title: 'Error', status, code, detail }));
+ };
+
+ try {
+ if (url.pathname === '/api/.well-known/jwks') return json(200, { public_key: publicKey, algorithm: 'RS256' });
+
+ if (req.method === 'GET' && url.pathname === '/api/v1/kinds') return json(200, { count: kinds.size, kinds: [...kinds.values()] });
+ if (req.method === 'GET' && /^\/api\/v1\/kinds\/([^/]+)$/.test(url.pathname)) {
+ const kindId = decodeURIComponent(url.pathname.split('/').pop());
+ const k = kinds.get(kindId);
+ if (!k) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');
+ return json(200, { kind: k });
+ }
+
+ if (req.method === 'POST' && /^\/api\/v1\/kinds\/([^/]+)\/tasks$/.test(url.pathname)) {
+ requireCapability(req, 'bench.task.create');
+ const kindId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);
+ if (!kinds.has(kindId)) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');
+ const prompt = String(body.prompt || '').trim();
+ if (!prompt) return sendProblem(422, 'bench.prompt_required', 'A task needs a prompt.');
+ const task = {
+ id: id('btk'),
+ kind_id: kindId,
+ prompt,
+ task_class: body.task_class || null,
+ public: body.public !== false,
+ created_by: 'service:design',
+ created_at: new Date().toISOString(),
+ };
+ state.tasks.set(task.id, task);
+ res.setHeader('Location', `/api/v1/tasks/${task.id}`);
+ return json(201, { task });
+ }
+
+ if (req.method === 'GET' && /^\/api\/v1\/tasks\/([^/]+)$/.test(url.pathname)) {
+ const taskId = decodeURIComponent(url.pathname.split('/').pop());
+ const task = state.tasks.get(taskId);
+ if (!task) return sendProblem(404, 'bench.task_not_found', 'No such task.');
+ return json(200, { task });
+ }
+
+ if (req.method === 'POST' && /^\/api\/v1\/tasks\/([^/]+)\/rounds$/.test(url.pathname)) {
+ const actor = verify(getAuth(req));
+ if (!actor) return sendProblem(401, 'token.required', 'Send a token.');
+ if (actor.kind === 'service' && !actor.cap.includes('bench.round.create')) return sendProblem(403, 'bench.capability_denied', 'Need bench.round.create.');
+ const taskId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);
+ const task = state.tasks.get(taskId);
+ if (!task) return sendProblem(404, 'bench.task_not_found', 'No such task.');
+ const offers = Array.isArray(body.offers) ? body.offers.slice(0, 3).map((o) => ({ provider: String(o.provider || ''), model: o.model == null ? null : String(o.model) })).filter((o) => o.provider) : [];
+ const poolFallback = offers.length < 3;
+ while (offers.length < 3) offers.push({ provider: 'ai', model: null });
+ const round = {
+ id: id('brd'),
+ task_id: task.id,
+ kind_id: task.kind_id,
+ offers,
+ status: 'queued',
+ seed: 0,
+ pool_fallback: poolFallback,
+ receipt: null,
+ params: {},
+ created_by: actor.kind === 'user' ? actor.subject : 'service:design',
+ created_at: new Date().toISOString(),
+ finished_at: null,
+ task_class: task.task_class,
+ prompt: task.prompt,
+ };
+ state.rounds.set(round.id, round);
+ res.setHeader('Location', `/api/v1/rounds/${round.id}`);
+ return json(201, { round: wireRound(round, null) });
+ }
+
+ if (req.method === 'GET' && /^\/api\/v1\/rounds\/([^/]+)$/.test(url.pathname)) {
+ const roundId = decodeURIComponent(url.pathname.split('/').pop());
+ const round = state.rounds.get(roundId);
+ if (!round) return sendProblem(404, 'bench.round_not_found', 'No such round.');
+ const viewer = verify(getAuth(req));
+ return json(200, { round: wireRound(round, viewer && viewer.kind === 'user' ? viewer.subject : null) });
+ }
+
+ if (req.method === 'POST' && /^\/api\/v1\/rounds\/([^/]+)\/entries$/.test(url.pathname)) {
+ requireCapability(req, 'bench.entry.create');
+ const roundId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);
+ const entry = addEntry(roundId, body);
+ res.setHeader('Location', `/api/v1/rounds/${roundId}`);
+ return json(201, { entry: fullEntry(entry) });
+ }
+
+ if (req.method === 'GET' && url.pathname === '/api/v1/judgements/next') {
+ const person = requirePerson(req);
+ const kindId = url.searchParams.get('kind') || 'design.page';
+ const candidates = [...state.rounds.values()].filter((r) =>
+ r.kind_id === kindId && r.status === 'finished' && r.created_by !== person.subject
+ && !state.judgements.some((j) => j.round_id === r.id && j.subject === person.subject)
+ );
+ const usableRounds = candidates.filter((r) => [...state.entries.values()].filter((e) => e.round_id === r.id && !e.error).length >= 2);
+ if (!usableRounds.length) return json(200, { pair: null, detail: 'There is no round waiting for your judgement right now.' });
+ const round = usableRounds[0];
+ const entries = shuffle([...state.entries.values()].filter((e) => e.round_id === round.id && !e.error));
+ const [left, right] = entries.slice(0, 2);
+ return json(200, {
+ round: { id: round.id, kind_id: round.kind_id, prompt: round.prompt, task_class: round.task_class },
+ left: blindEntry(left),
+ right: blindEntry(right),
+ });
+ }
+
+ if (req.method === 'POST' && /^\/api\/v1\/rounds\/([^/]+)\/judgements$/.test(url.pathname)) {
+ const person = requirePerson(req);
+ const roundId = decodeURIComponent(url.pathname.split('/').slice(-2)[0]);
+ const round = state.rounds.get(roundId);
+ if (!round) return sendProblem(404, 'bench.round_not_found', 'No such round.');
+ if (round.status !== 'finished') return sendProblem(409, 'bench.round_not_ready', 'Round not ready.');
+ if (round.created_by === person.subject) return sendProblem(403, 'bench.creator_cannot_vote', 'You cannot judge a round you started.');
+ const pair = body.pair || {};
+ if (!pair.left || !pair.right || pair.left === pair.right) return sendProblem(422, 'bench.invalid_pair', 'Invalid pair.');
+ if (state.judgements.some((j) => j.round_id === round.id && j.subject === person.subject)) return sendProblem(409, 'bench.already_judged', 'Already judged.');
+ const entries = [...state.entries.values()].filter((e) => e.round_id === round.id && !e.error && (e.id === pair.left || e.id === pair.right));
+ if (entries.length !== 2) return sendProblem(422, 'bench.invalid_pair', 'Entries not in round.');
+ const outcome = body.outcome;
+ if (!['left', 'right', 'tie', 'both_bad'].includes(outcome)) return sendProblem(422, 'bench.invalid_outcome', 'Invalid outcome.');
+ const judgement = {
+ id: id('bju'),
+ round_id: round.id,
+ kind_id: round.kind_id,
+ judge: 'human_pairwise',
+ method: 'human_pairwise',
+ pair,
+ outcome,
+ score: null,
+ subject: person.subject,
+ at: new Date().toISOString(),
+ };
+ state.judgements.push(judgement);
+ updateRating(round.kind_id, entries.find((e) => e.id === pair.left), entries.find((e) => e.id === pair.right), outcome);
+ return json(201, { judgement, revealed: { left: fullEntry(entries.find((e) => e.id === pair.left)), right: fullEntry(entries.find((e) => e.id === pair.right)) } });
+ }
+
+ if (req.method === 'GET' && /^\/api\/v1\/boards\/([^/]+)$/.test(url.pathname)) {
+ const kindId = decodeURIComponent(url.pathname.split('/').pop());
+ const type = url.searchParams.get('type') || 'elo';
+ const b = board(kindId, type);
+ if (!b) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');
+ return json(200, { board: b });
+ }
+
+ if (req.method === 'GET' && /^\/api\/v1\/datasets\/([^/]+)\.jsonl$/.test(url.pathname)) {
+ const kindId = decodeURIComponent(url.pathname.split('/').pop().replace(/\.jsonl$/, ''));
+ const text = datasetJsonl(kindId);
+ if (text == null) return sendProblem(404, 'bench.kind_not_found', 'No such kind.');
+ res.setHeader('Cache-Control', 'public, max-age=60');
+ res.setHeader('Content-Disposition', `attachment; filename="${kindId}-dataset.jsonl"`);
+ res.writeHead(200, { 'Content-Type': 'application/x-ndjson' });
+ return res.end(text);
+ }
+
+ return sendProblem(404, 'route.not_found', 'No such route.');
+ } catch (err) {
+ if (err && err.status) return sendProblem(err.status, err.code, err.detail);
+ sendProblem(500, 'internal.error', 'Bench fake failed.');
+ }
+ });
+ });
+
+ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
+ const base = `http://127.0.0.1:${server.address().port}`;
+
+ async function tick() {
+ for (const round of state.rounds.values()) {
+ if (round.status === 'queued') {
+ round.status = 'running';
+ await generateEntries(round);
+ }
+ }
+ }
+
+ async function close() {
+ await new Promise((r) => server.close(r));
+ }
+
+ return { base, url: base, state, tick, close, addEntry };
+}
+
+module.exports = { startBench };
@@ -2,27 +2,28 @@
@@ -2,27 +2,28 @@
// Many requests per test file: the per-address limits would answer 429 part-way.
// Many requests per test file: the per-address limits would answer 429 part-way.
process.env.DESIGN_RATE_LIMIT_PER_MIN = process.env.DESIGN_RATE_LIMIT_PER_MIN || '100000';
process.env.DESIGN_RATE_LIMIT_PER_MIN = process.env.DESIGN_RATE_LIMIT_PER_MIN || '100000';
/**
/**
- * Boots OpenVibe.Design on a temp database with a mock Network and AI, captures log lines, and returns a small
- * HTTP client. Every test file gets its own process, database, Network, AI and app.
+ * Boots OpenVibe.Design on a temp database with mock Network and Bench, captures log lines, and returns a small
+ * HTTP client. Every test file gets its own process, database, Network, Bench and app.
*
*
* const t = await boot();
* const t = await boot();
* t.get(path, { as: user, json, headers, method, origin })
* t.get(path, { as: user, json, headers, method, origin })
* t.signIn(user) → cookie header value (design_at = a Network user token, as after /auth/callback)
* t.signIn(user) → cookie header value (design_at = a Network user token, as after /auth/callback)
- * await t.tick() → drive the Arena job loop once
+ * await t.tick() → drive the fake Bench generation loop once
* t.logs(), t.dbDump()
* t.logs(), t.dbDump()
*/
*/
const http = require('http');
const http = require('http');
const { startNetwork } = require('./network');
const { startNetwork } = require('./network');
-const { startAi } = require('./ai');
+const { startBench } = require('./bench');
async function boot(opts = {}) {
async function boot(opts = {}) {
const network = await startNetwork(opts.network || {});
const network = await startNetwork(opts.network || {});
- const ai = opts.ai || await startAi(opts.aiOptions || {});
+ const bench = opts.bench || await startBench({ network });
const env = {
const env = {
NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.design', TRUST_PROXY: '1',
NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.design', TRUST_PROXY: '1',
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url, OV_NETWORK_ISSUER: network.url,
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url, OV_NETWORK_ISSUER: network.url,
OV_OAUTH_CLIENT_ID: 'design', OV_OAUTH_CLIENT_SECRET: 'design-secret', COOKIE_SECURE: 'false',
OV_OAUTH_CLIENT_ID: 'design', OV_OAUTH_CLIENT_SECRET: 'design-secret', COOKIE_SECURE: 'false',
- OV_AI_INTERNAL_URL: ai.url,
+ OV_BENCH_INTERNAL_URL: bench.url,
+ OV_BENCH_URL: bench.url,
DESIGN_RATE_LIMIT_PER_MIN: '100000',
DESIGN_RATE_LIMIT_PER_MIN: '100000',
DESIGN_DRILL: '1',
DESIGN_DRILL: '1',
DESIGN_AI_POLL_MS: '10',
DESIGN_AI_POLL_MS: '10',
@@ -46,7 +47,7 @@ async function boot(opts = {}) {
@@ -46,7 +47,7 @@ async function boot(opts = {}) {
const config = configLib.load(env);
const config = configLib.load(env);
const testdb = opts.store ? null : await require('./db').testDb({ store: opts.testStore });
const testdb = opts.store ? null : await require('./db').testDb({ store: opts.testStore });
const store = opts.store || createStore(testdb.db, { now: opts.now });
const store = opts.store || createStore(testdb.db, { now: opts.now });
- const built = await createApp({ config, store, arenaAi: opts.arenaAi, log, now: opts.now, accountSend: opts.accountSend, random: opts.random });
+ const built = await createApp({ config, store, log, now: opts.now, accountSend: opts.accountSend, random: opts.random });
await built.ctx.keys.ensure();
await built.ctx.keys.ensure();
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
@@ -80,8 +81,8 @@ async function boot(opts = {}) {
@@ -80,8 +81,8 @@ async function boot(opts = {}) {
}
}
return {
return {
- base, network, ai, config, ctx: built.ctx, store: built.ctx.s, get, signIn, dbDump,
- tick: () => built.ctx.jobs.tick(),
+ base, network, bench, config, ctx: built.ctx, store: built.ctx.s, get, signIn, dbDump,
+ tick: () => bench.tick(),
logs: () => logLines.join('\n'),
logs: () => logLines.join('\n'),
async close() {
async close() {
built.ctx.jobs.stop();
built.ctx.jobs.stop();
@@ -89,7 +90,7 @@ async function boot(opts = {}) {
@@ -89,7 +90,7 @@ async function boot(opts = {}) {
built.ctx.keys.client.stop();
built.ctx.keys.client.stop();
if (testdb) await testdb.close();
if (testdb) await testdb.close();
await network.close();
await network.close();
- if (!opts.ai) await ai.close();
+ if (!opts.bench) await bench.close();
},
},
};
};
}
}
@@ -7,6 +7,7 @@
@@ -7,6 +7,7 @@
*/
*/
const http = require('http');
const http = require('http');
const crypto = require('crypto');
const crypto = require('crypto');
+const { ids } = require('openvibe-contracts');
const b64 = (o) => Buffer.from(JSON.stringify(o)).toString('base64url');
const b64 = (o) => Buffer.from(JSON.stringify(o)).toString('base64url');
function signJwt(claims, privateKey) {
function signJwt(claims, privateKey) {
@@ -58,7 +59,7 @@ async function startNetwork({ themes = DEFAULT_THEMES } = {}) {
@@ -58,7 +59,7 @@ async function startNetwork({ themes = DEFAULT_THEMES } = {}) {
let nextUserId = 1;
let nextUserId = 1;
function addUser(username, { role = 'user' } = {}) {
function addUser(username, { role = 'user' } = {}) {
- const u = { id: nextUserId++, subject: `usr_${crypto.randomBytes(12).toString('hex')}`, username, display_name: username, role };
+ const u = { id: nextUserId++, subject: ids.newId('user'), username, display_name: username, role };
st.users.set(u.id, u);
st.users.set(u.id, u);
return u;
return u;
}
}
@@ -105,12 +106,13 @@ async function startNetwork({ themes = DEFAULT_THEMES } = {}) {
@@ -105,12 +106,13 @@ async function startNetwork({ themes = DEFAULT_THEMES } = {}) {
return json(200, { access_token: userToken(u), refresh_token: rt, token_type: 'Bearer' });
return json(200, { access_token: userToken(u), refresh_token: rt, token_type: 'Bearer' });
}
}
if (form.grant_type === 'client_credentials') {
if (form.grant_type === 'client_credentials') {
- // A first-party service token: audience openvibe.ai, capability ai.run.create, namespace ai.*.
+ // A first-party service token for the requested audience and capabilities.
const scope = String(form.scope || '');
const scope = String(form.scope || '');
const now = Math.floor(Date.now() / 1000);
const now = Math.floor(Date.now() / 1000);
const access = signJwt({
const access = signJwt({
iss: issuer, sub: 'svc:design', actor_type: 'service', aud: [form.audience || 'openvibe.ai'],
iss: issuer, sub: 'svc:design', actor_type: 'service', aud: [form.audience || 'openvibe.ai'],
- cap: scope.split(/\s+/).filter(Boolean), ns: ['ai.*'], iat: now, exp: now + 300,
+ cap: scope.split(/\s+/).filter(Boolean), ns: [form.audience === 'openvibe.bench' ? 'bench.*' : 'ai.*'],
+ iat: now, exp: now + 300, jti: `tok_${crypto.randomBytes(12).toString('hex')}`,
}, privatePem);
}, privatePem);
return json(200, { access_token: access, token_type: 'Bearer', expires_in: 300, scope });
return json(200, { access_token: access, token_type: 'Bearer', expires_in: 300, scope });
}
}