openvibers/openvibe.report Commits
AI: carry OpenVibe.Bench's pages as Report's AI section (#132)
Changes
@@@ -3,6 +3,9 @@What changed in OpenVibe.Report, newest first. Each site also publishes its patch notes at /updates onopenvibe.network.++## Unreleased++++- **The AI section (2026-11-08): OpenVibe.Bench's pages moved here.** `/ai` is the overview: every published benchmark with its leader, runs and last run, every task kind with quality, price and speed, and a fold for the jobs still waiting. `/ai/<benchmark>` has verified rankings, score against cost in SVG and the latest runs; `/ai/<kind>` keeps the Elo/Pareto/cost boards and adds the verified runs on the kind's benchmarks. `/ai/models/<model>` has runs over time, best and worst tasks, cost and recent runs; `/ai/compare` puts two or three models side by side; `/ai/method` says how a score is made and how runs are judged. `/ai/feed.xml` and `GET /api/v1/ai/results` publish new runs. Every number is Bench's, read through its public API with its 60-second cache and last-good copy; Bench's old HTML pages 301 here.## 0.1.0 — 2026-10-28The service skeleton: a stateless front for the network's open numbers, The Rundown and every stream's after-show@@@ -36,7 +36,7 @@ OpenVibe.Live in both modes.| **Report's tables** | `report_metric_days`, `report_metric_windows`, `report_streams` | the same routes while `REPORT_SOURCE=report` || **OpenVibe.Live** — The Rundown | `GET /api/rundown/`, `/api/rundown/topics`, `/api/rundown/t/:slug`, `/api/rundown/e/:streamId`, `/api/rundown/s/:username` | `/rundown`, `/rundown/t/:slug`, `/rundown/e/:streamId`, `/rundown/s/:username` || **OpenVibe.Live** — after-show reports | `GET /api/recap/:streamId`, `GET /api/recap/channel/:username` | `/after-show/:streamId`, `/c/:username` |--| **OpenVibe.Bench** | `GET /api/v1/kinds`, `/api/v1/boards/:kind?type=elo\|pareto\|cost`, `/api/v1/models/:model` | `/ai`, `/ai/:kind`, `/ai/models/:model` |++| **OpenVibe.Bench** | `GET /api/v1/kinds`, `/api/v1/boards/:kind?type=elo\|pareto\|cost`, `/api/v1/benchmarks`, `/api/v1/rankings`, `/api/v1/runs`, `/api/v1/overview`, `/api/v1/models/:model`, `/api/v1/models/:model/summary` | `/ai`, `/ai/:kind`, `/ai/<benchmark>`, `/ai/models/:model`, `/ai/compare`, `/ai/method`, `/ai/feed.xml`, `GET /api/v1/ai/results` || **OpenVibe.Billing** | `GET /api/v1/stats`, `GET /api/v1/transactions?subject=…` | the revenue read for a channel; network money counts || **OpenVibe.Billing** — settlements, through Events | `billing.credit.settled@1` on `POST /internal/events` (only with `REPORT_AI_EXCHANGE=1`) | the AI Exchange aggregates and the staff page/API |@@@ -64,6 +64,7 @@ never invents a zero and never throws a source error at the reader.- **Money** — OpenVibe.Billing. Report reads it with `billing.stats.read` and `billing.balance.read`; it holds noledger and computes no price.- **The Rundown and the after-show reports** — OpenVibe.Live. Report quotes them, including their counts and text.++- **The benchmark results** — OpenVibe.Bench. Report renders its rows and never derives a score, cost or time; Bench stayed the authority when its public pages moved here (2026-11-08) and its old HTML pages answer 301 to `/ai` counterparts.- **Identity, sessions and writes** — Report has no sign-in, no cookies and no public write route. `POST/internal/events` is loopback-only and HMAC-verified.- **The Shared Frame** — `openvibe-shared` (navbar, footer, theme-loader, static files), served from this site's own@@@ -83,9 +84,13 @@ All server-rendered in `server/views/index.js`; every dynamic value goes through| `/rundown/s/:username` | one streamer's episodes and topics; canonical is Live's `/rundown/:username` || `/after-show/:streamId` | one after-show report || `/c/:username` | one channel's latest after-show reports |--| `/ai` | benchmark kinds grouped by room, with leaders and game counts |--| `/ai/:kind` | Elo, Pareto quality against cost per solved, and cost per solved tables |--| `/ai/models/:model` | a model's ratings and entries across kinds |++| `/ai` | the AI overview: every published benchmark with its leader, runs and last run; every task kind with its leader, quality, cost and speed; a link to the method and compare pages |++| `/ai/:kind` | one task kind: Elo rank, quality against cost, cost per solved, the verified runs on its benchmarks, and the latest runs |++| `/ai/<benchmark>` | one public Bench benchmark: verified rankings with score, cost, speed and last run, a score-against-cost SVG, and the latest runs |++| `/ai/models/:model` | a model's runs over time, its best and worst tasks, cost, per-benchmark record and recent runs |++| `/ai/compare` | two or three models on the same benchmarks, side by side |++| `/ai/method` | how a score is made, how runs are judged, verified vs submitted, and what the counts mean |++| `/ai/feed.xml` | new public runs as RSS; the same rows are `GET /api/v1/ai/results` || `/ai/exchange` | **staff only**, with `REPORT_AI_EXCHANGE=1`: Billing's settlements by day, model and access, a daily chart and the exact amounts. Needs a signed-in person whose staff role holds `staff.site.view` |A bad path parameter (slug, stream id, username) or a source that answered nothing renders `views.notFound()` with@@@ -104,6 +109,7 @@ GET /api/v1/rundown/topicsGET /api/v1/rundown/t/:slugGET /api/v1/after-show/:streamIdGET /api/v1/c/:username++GET /api/v1/ai/results the new AI runs, newest first```A bad parameter or a missing source answer is a 404 `application/problem+json` body built with@@@ -19,7 +19,7 @@"pg v8.23.1","pglite v0.5.13 (dev)"],-- "summary": "OpenVibe.Report keeps its own PostgreSQL analytics store (migrations/), ingests Network's live.stream.ended and frame beacon plus its open-stats answer idempotently (report_inbox), computes the same site-wide, per-site and per-channel numbers by day, quotes Billing's revenue through Billing's read API, and runs a shadow job that records every difference from Network in report_shadow_diffs with GET /internal/shadow on loopback. REPORT_SOURCE=network (default) keeps the pages on today's Network numbers; REPORT_SOURCE=report answers them from Report's tables. The Rundown and after-show pages still read OpenVibe.Live.",++ "summary": "OpenVibe.Report keeps its own PostgreSQL analytics store (migrations/), ingests Network's live.stream.ended and frame beacon plus its open-stats answer idempotently (report_inbox), computes the same site-wide, per-site and per-channel numbers by day, quotes Billing's revenue through Billing's read API, and runs a shadow job that records every difference from Network in report_shadow_diffs with GET /internal/shadow on loopback. REPORT_SOURCE=network (default) keeps the pages on today's Network numbers; REPORT_SOURCE=report answers them from Report's tables. The Rundown and after-show pages still read OpenVibe.Live. The AI section reads OpenVibe.Bench's public API: /ai, /ai/<benchmark>, /ai/models/<model>, /ai/compare, /ai/method, /ai/feed.xml and /api/v1/ai/results; Bench's old HTML pages 301 here.","works": ["Database: server/db.js (openvibe-sdk/db, migrations at boot; PostgreSQL through PgBouncer, PGlite in development) and test/helpers/db.js (createTestDb)","Tables: report_inbox, report_events, report_streams, report_frame_days, report_metric_days, report_metric_windows, report_metric_json, report_dimensions, report_shadow_runs, report_shadow_diffs",@@@ -61,7 +61,9 @@"JSON API","crawl artifacts","readiness",-- "deploy files"++ "deploy files",++ "AI section from OpenVibe.Bench",++ "AI results RSS and JSON"],"deployment": "planned: openvibe-ovh, unit openvibe-report.service on 127.0.0.1:5080 (git checkout /opt/openvibe.report, env /etc/openvibe/report.env), public at https://openvibe.report; DATABASE_URL/DATABASE_DIRECT_URL from OpenVibe.Host roles/data add-service.sh report","updated": "2026-11-08"@@@ -372,3 +372,13 @@ html, body { margin: 0; background: var(--bg-primary, #0a0f1c); color: var(--tex.rp-step-st { color: var(--text-secondary, #9aa4b2); font-size: .82rem; font-weight: 500; }.rp-steps li.is-failed .rp-step-st { color: #f87171; }.rp-step-detail { color: var(--text-secondary, #9aa4b2); font-size: .9rem; }++++/* AI (2026-11-08): the compare form and the method page. The charts and tables are the shared kit's own. */++.rp-ai-compare-form { display: flex; flex-wrap: wrap; align-items: end; gap: 8px; margin: 18px 0 8px; padding: 14px 16px; border: 1px solid var(--border, #1f2d47); border-radius: 16px; background: var(--bg-card, #0f1729); }++.rp-ai-compare-form label { flex: 1 0 100%; font-size: .8rem; font-weight: 700; color: var(--text-secondary, #96a7c2); }++.rp-ai-compare-form input { flex: 1 1 240px; min-width: 0; min-height: 44px; padding: 0 12px; border-radius: 10px; border: 1px solid var(--border, #1f2d47); background: var(--bg-secondary, #101828); color: var(--text-primary, #e6edf7); font: inherit; }++.rp-ai-compare-form input:focus-visible { outline: 2px solid var(--room-accent); outline-offset: 1px; }++.rp-ai-compare-form .rp-btn { border: 0; cursor: pointer; font: inherit; }++.rp-ai-method h2 { margin: 28px 0 8px; font-size: 1.15rem; }++.rp-ai-method p { max-width: 820px; margin: 0 0 12px; line-height: 1.65; color: var(--text-secondary, #96a7c2); }++.rp-ai-method code { overflow-wrap: anywhere; }@@@ -18,7 +18,13 @@ function createAgentPages({ publicPages, bench }) {else if (path === '/design') page = publicPages.design();else if (path === '/stats') page = await publicPages.stats(normaliseDays(query && query.get('days')));else if (path === '/ai') page = await publicPages.aiIndex();++ else if (path === '/ai/method') page = await publicPages.aiMethod();++ else if (path === '/ai/compare') {++ const models = String((query && query.get('models')) || '').split(',').map((x) => x.trim()).filter(Boolean);++ page = await publicPages.aiCompare(models);++ }else if (model) page = await publicPages.aiModel(decodeURIComponent(model[1]));++ else if (kind && bench.isBenchmarkId(kind[1])) page = await publicPages.aiBenchmark(kind[1]);else if (kind) page = await publicPages.aiBoard(kind[1], bench.normaliseTab(query && query.get('tab')));return page ? { html: page.html } : null;}@@@ -287,7 +287,7 @@ async function createApp(opts = {}) {legacyHeaders: false,handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }),}));-- app.use('/api/v1', createApi({ network, live, analytics, aiUsage, aiExchange, sessionReports, config, now, log }));++ app.use('/api/v1', createApi({ network, live, bench, analytics, aiUsage, aiExchange, sessionReports, config, now, log }));app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));// ── Crawl artifacts, then the pages ─────────────────────@@@ -1,29 +1,52 @@'use strict';/**-- * OpenVibe.Bench's public reads (plan 2026-10-28 §4.12):++ * OpenVibe.Bench's public reads (plan 2026-10-28 §4.12; the 2026-11-08 move adds the registry reads):* GET /api/v1/kinds every benchmark kind* GET /api/v1/boards/:kind?type=elo|pareto|cost-- * GET /api/v1/models/:model one model across every kind++ * GET /api/v1/models/:model one model across every kind (the legacy boards)++ * GET /api/v1/benchmarks every public benchmark, its runs and last run++ * GET /api/v1/benchmarks/:id one benchmark and its definition++ * GET /api/v1/rankings?benchmark=&metric= verified per-model medians++ * GET /api/v1/runs?benchmark=&model=&kind= public runs newest first++ * GET /api/v1/overview public counts and the newest runs++ * GET /api/v1/models/:model/summary a model's runs, tasks, cost and last run** Reads are public and held for sixty seconds in this process. A read that has answered once keeps its copy* for as long as Bench is down: the client answers the last good one with the moment it was read (`at`) and* `stale: true`, so a board page stays a page — never an error page. A read that has never answered is null-- * after one short log line. A kind id, board type or model name that cannot exist is refused before any request-- * is made.++ * after one short log line. A kind id, benchmark id, board type or model name that cannot exist is refused++ * before any request is made.*/const { trim, requestJson, TIMEOUT_MS } = require('./source');const TTL_MS = 60_000;const KIND_RE = /^[a-z0-9][a-z0-9._-]{0,79}$/;++const BENCHMARK_RE = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;const MODEL_RE = /^[A-Za-z0-9][A-Za-z0-9._:+\/~\-]{0,119}$/;const BOARD_TYPES = Object.freeze(['elo', 'pareto', 'cost']);++const METRICS = Object.freeze(['score', 'value', 'speed', 'consistency', 'horizon']);const isKindId = (value) => typeof value === 'string' && KIND_RE.test(value);++/** A benchmark id as Bench mints it (bmk_ + a 26-character ULID), never a kind or a path. */++const isBenchmarkId = (value) => typeof value === 'string' && BENCHMARK_RE.test(value);++/** A model as Bench pins it: one path segment (its own /models/:model route cannot serve more), 120 characters at most. */const isModelName = (value) => typeof value === 'string' && MODEL_RE.test(value);++/** A ranking metric; anything else is the score, exactly as Bench's own rankings route defaults. */++function normaliseMetric(value) {++ const raw = Array.isArray(value) ? value[0] : value;++ return METRICS.includes(raw) ? raw : 'score';++}++++/** A public run limit: 1 to 100, the API's own cap; anything else is 20. */++function normaliseLimit(value) {++ const n = parseInt(Array.isArray(value) ? value[0] : value, 10);++ return Number.isFinite(n) ? Math.min(100, Math.max(1, n)) : 20;++}++/** ?tab=elo|pareto|cost; anything else is the Elo board, exactly as the page tabs read it. */function normaliseTab(value) {const raw = Array.isArray(value) ? value[0] : value;@@@ -41,6 +64,8 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = co// boards are a fixed small set; models are capped, the oldest inserted out first.const boards = new Map();const models = new Map();++ const catalog = new Map();++ const results = new Map();function keep(store, key, data, cap) {store.set(key, data);@@@ -86,7 +111,58 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = coreturn read(models, `model:${model}`, `/api/v1/models/${encodeURIComponent(model)}`, `bench model (${model})`, 200);}-- return { kinds, board, modelSummary, baseUrl, ttlMs: TTL_MS, isKindId, isModelName, normaliseTab };++ /** Every public benchmark the registry lists, with its runs and last run. */++ function benchmarks() {++ return read(catalog, 'benchmarks', '/api/v1/benchmarks?limit=200', 'bench benchmarks', 100);++ }++++ /** One public benchmark: { benchmark, version, definition, task_meta }. */++ async function benchmark(id) {++ if (!isBenchmarkId(id)) return null;++ return read(catalog, `benchmark:${id}`, `/api/v1/benchmarks/${encodeURIComponent(id)}`, `bench benchmark (${id})`, 500);++ }++++ /** Public counts and the newest runs: { kinds, benchmarks, tasks, runs, models, last_run, list, recent }. */++ function overview() {++ return read(catalog, 'overview', '/api/v1/overview', 'bench overview', 20);++ }++++ /** One benchmark's verified medians: { rankings: { benchmark, metric, entries } }. */++ async function rankings(benchmarkId, metric = 'score') {++ if (!isBenchmarkId(benchmarkId) || !METRICS.includes(metric)) return null;++ return read(results, `rankings:${benchmarkId}:${metric}`, `/api/v1/rankings?benchmark=${encodeURIComponent(benchmarkId)}&metric=${metric}`, `bench rankings (${benchmarkId}, ${metric})`, 500);++ }++++ /**++ * Public runs newest first. opts: { benchmark, model, kind, limit }. A kind is the task-kind filter the kind++ * page reads; a benchmark and a model are the registry ids. A value that cannot exist is refused before any++ * request, so a stray query never becomes a Bench call.++ */++ async function runs({ benchmark = null, model = null, kind = null, limit = 20 } = {}) {++ if (benchmark != null && !isBenchmarkId(benchmark)) return null;++ if (model != null && !isModelName(model)) return null;++ if (kind != null && !isKindId(kind)) return null;++ const params = new URLSearchParams();++ if (benchmark) params.set('benchmark', benchmark);++ if (model) params.set('model', model);++ if (kind) params.set('kind', kind);++ params.set('limit', String(normaliseLimit(limit)));++ const query = params.toString();++ return read(results, `runs:${query}`, `/api/v1/runs?${query}`, `bench runs (${query})`, 500);++ }++++ /** A model's public record: { model, runs, benchmarks, tasks, best_task, worst_task, recent_runs, total_cost_usd, last_run }. */++ async function modelDetail(model) {++ if (!isModelName(model)) return null;++ return read(models, `detail:${model}`, `/api/v1/models/${encodeURIComponent(model)}/summary`, `bench model detail (${model})`, 200);++ }++++ return {++ kinds, board, modelSummary, benchmarks, benchmark, overview, rankings, runs, modelDetail,++ baseUrl, ttlMs: TTL_MS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,++ };}--module.exports = { createBenchClient, TTL_MS, BOARD_TYPES, isKindId, isModelName, normaliseTab };++module.exports = {++ createBenchClient, TTL_MS, BOARD_TYPES, METRICS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,++};@@@ -30,7 +30,7 @@ const sharing = require('../sharing');const CACHE = 'public, max-age=120';--function createApi({ network, live, analytics = null, aiUsage = null, aiExchange = null, sessionReports = null, config = {}, now = () => Date.now(), log = console }) {++function createApi({ network, live, bench = null, analytics = null, aiUsage = null, aiExchange = null, sessionReports = null, config = {}, now = () => Date.now(), log = console }) {const r = express.Router();const serviceAuth = createServiceAuth({ config, now });@@@ -166,6 +166,22 @@ function createApi({ network, live, analytics = null, aiUsage = null, aiExchange});});++ // The new AI results, as JSON: the same public runs the /ai/feed.xml RSS carries. Only Bench's own rows; a++ // result with no run is not in it.++ r.get('/ai/results', async (req, res) => {++ if (!bench) return problem(req, res, 503, 'report.no_bench', 'Bench is not configured in this process.');++ const [runs, overview] = await Promise.all([bench.runs({ limit: 50 }), bench.overview()]);++ const list = runs && runs.data && Array.isArray(runs.data.runs) ? runs.data.runs : null;++ if (!list) return problem(req, res, 502, 'report.bench_unavailable', 'Bench did not answer with its runs.');++ const titles = new Map((overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : []).map((b) => [b.id, b.title]));++ const results = list.map((run) => ({++ id: run.id, benchmark_id: run.benchmark_id, benchmark: titles.get(run.benchmark_id) || null,++ model: run.model, score: run.score, cost_usd: run.cost_usd, ms: run.ms, provenance: run.provenance,++ created_at: run.created_at, url: `${String(config.baseUrl || 'https://openvibe.report').replace(/\/+$/, '')}/ai/${encodeURIComponent(run.benchmark_id)}`,++ }));++ return res.json({ count: results.length, generated_at: new Date(now()).toISOString(), results });++ });++// ── The AI meter (plan L40 and L41) ─────────────────────────────────────────────//// The public totals: the tokens, the cache's share, the own-key share and the spend by week. No person, no@@@ -4,7 +4,7 @@* Crawl artifacts for openvibe.report, built with openvibe-shared/seo:** GET /robots.txt the public pages are crawlable; the API is not-- * GET /sitemap.xml /, /stats, /design, /rundown and every /rundown/t/<slug> from the Rundown's topics++ * GET /sitemap.xml /, /stats, /design, /ai (and its benchmarks, models, method and compare), the Rundown's topics* GET /llms.txt a markdown map of the site and its machine-readable answers** The sitemap lists only the pages this site owns; The Rundown pages carry rel=canonical to openvibe.live and are left@@@ -46,8 +46,13 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {});r.get('/sitemap.xml', async (_req, res) => {-- const [ai, reports] = await Promise.all([bench.kinds(), sessionReports ? sessionReports.listPublic({ limit: 500 }) : []]);++ const [ai, overview, reports] = await Promise.all([++ bench.kinds(), bench.overview(), sessionReports ? sessionReports.listPublic({ limit: 500 }) : [],++ ]);const kinds = (ai && Array.isArray(ai.data.kinds)) ? ai.data.kinds : [];++ const o = overview && overview.data ? overview.data : {};++ const benches = Array.isArray(o.list) ? o.list : [];++ const models = Array.isArray(o.model_list) ? o.model_list.map((m) => m.model).filter(Boolean) : [...new Set((Array.isArray(o.recent) ? o.recent : []).map((r) => r && r.model).filter(Boolean))];const lastmod = siteUpdated();// Only the pages this site owns. /rundown and its topic, episode and speaker pages carry rel=canonical to// OpenVibe.Live's Rundown, so listing them here would advertise a duplicate.@@@ -56,19 +61,28 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {{ loc: abs('/stats'), changefreq: 'daily', priority: 0.9 },{ loc: abs('/design'), changefreq: 'weekly', priority: 0.6 },{ loc: abs('/ai'), changefreq: 'daily', priority: 0.9 },++ { loc: abs('/ai/method'), changefreq: 'weekly', priority: 0.6 },++ { loc: abs('/ai/compare'), changefreq: 'weekly', priority: 0.6 },...reports.map((rep) => ({ loc: abs('/s/' + encodeURIComponent(rep.id)), changefreq: 'daily', priority: 0.6 })),...kinds.filter((k) => k && bench.isKindId(k.id)).map((k) => ({loc: abs(`/ai/${encodeURIComponent(k.id)}`), changefreq: 'daily', priority: 0.7,})),-- ].map((p) => ({ ...p, ...(lastmod ? { lastmod } : {}) }));++ ...benches.filter((b) => b && bench.isBenchmarkId(b.id)).map((b) => ({++ loc: abs(`/ai/${encodeURIComponent(b.id)}`), changefreq: 'daily', priority: 0.8,++ ...(dayOf(b.last_run) ? { lastmod: dayOf(b.last_run) } : {}),++ })),++ ...models.filter((m) => bench.isModelName(m)).map((m) => ({++ loc: abs(`/ai/models/${encodeURIComponent(m)}`), changefreq: 'daily', priority: 0.6,++ })),++ ].map((p) => ({ ...p, ...(lastmod && !p.lastmod ? { lastmod } : {}) }));res.type('application/xml').set('Cache-Control', CACHE).send(seo.sitemapXml(pages));});// The site as llms.txt describes it, shared by /llms.txt and /llms-full.txt (SEO-33).const llmsSite = () => ({name: SITE_NAME,-- summary: 'Every number on OpenVibe, honest: the network\'s open stats, The Rundown of what streamers talked about, and every stream\'s after-show report.',-- details: 'The numbers and the write-ups are read from OpenVibe.Network and OpenVibe.Live, cached in this process for a few minutes, and shown as the sources answer them. Nothing here is about a person.',++ summary: 'Every number on OpenVibe, honest: the network\'s open stats, The Rundown of what streamers talked about, every stream\'s after-show report, and the AI results Bench publishes.',++ details: 'The numbers and the write-ups are read from OpenVibe.Network, OpenVibe.Live and OpenVibe.Bench, cached in this process for a few minutes, and shown as the sources answer them. Nothing here is about a person.',markdown: true,base: site,sections: [@@@ -77,7 +91,9 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {{ title: 'Open numbers', url: abs('/stats'), note: 'page views, accounts and traffic for every OpenVibe site' },{ title: 'Design health', url: abs('/design'), note: 'how closely each site follows the shared design kit, read from its code; per site at /design/<site>' },{ title: 'The Rundown', url: 'https://openvibe.live/rundown', note: 'what streamers talked about, topic by topic (OpenVibe.Live is its canonical home)' },-- { title: 'AI boards', url: abs('/ai'), note: 'Bench ratings, quality and cost across rooms' },++ { title: 'AI', url: abs('/ai'), note: 'OpenVibe.Bench results: quality, price and speed for every task kind and benchmark' },++ { title: 'How AI results are judged', url: abs('/ai/method'), note: 'scores, verified runs, cost per solved, speed and the frontier' },++ { title: 'Compare AI models', url: abs('/ai/compare'), note: 'two or three models on the same benchmarks' },] },// A JSON API is what an agent came for, not something to skip (AGT-5 review).{ title: 'API', links: [@@@ -85,6 +101,8 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {{ title: 'Design health (JSON)', url: abs('/api/v1/design-health'), note: 'every site; /api/v1/design-health/<site> for one' },{ title: 'The Rundown (JSON)', url: abs('/api/v1/rundown') },{ title: 'Topics (JSON)', url: abs('/api/v1/rundown/topics') },++ { title: 'New AI results (RSS)', url: abs('/ai/feed.xml') },++ { title: 'New AI results (JSON)', url: abs('/api/v1/ai/results') },] },],optional: [@@@ -11,6 +11,12 @@* GET /rundown/s/:username one streamer* GET /after-show/:streamId one after-show report* GET /c/:username one channel's after-show reports++ * GET /ai OpenVibe.Bench's public results: task kinds, benchmarks, runs and last runs++ * GET /ai/method how a score, a cost and a speed are made++ * GET /ai/compare two or three models side by side++ * GET /ai/feed.xml new runs as RSS++ * GET /ai/:ref one task kind (Elo/price/speed) or one public benchmark (bmk_…)++ * GET /ai/models/:model one model's runs over time, tasks, cost and recent runs** A bad path parameter is a 404 without a source request; so is a source that answers nothing. Pages are public* for two minutes.@@@ -49,25 +55,68 @@ function createPublicPages({ sessionReports, network, live, bench, analytics = n}async function aiIndex() {-- const kinds = await bench.kinds();++ const [overview, kinds] = await Promise.all([bench.overview(), bench.kinds()]);const list = kinds && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];-- const boards = await Promise.all(list.map((kind) => bench.board(kind.id, 'elo')));-- return { html: views.aiIndex({ kinds, boards }) };++ const benches = overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : [];++ const [boards, rankings] = await Promise.all([++ Promise.all(list.map((kind) => bench.board(kind.id, 'elo'))),++ Promise.all(benches.map((b) => bench.rankings(b.id, 'score'))),++ ]);++ return { html: views.aiIndex({ overview, kinds, boards, rankings }) };}async function aiBoard(kindId, tab) {if (!bench.isKindId(kindId)) return null;-- const [kinds, board, cost] = await Promise.all([-- bench.kinds(), bench.board(kindId, tab), tab === 'pareto' ? bench.board(kindId, 'cost') : Promise.resolve(null),++ const [kinds, board, cost, benchmarks, runs, overview] = await Promise.all([++ bench.kinds(),++ bench.board(kindId, tab),++ tab === 'pareto' ? bench.board(kindId, 'cost') : Promise.resolve(null),++ bench.benchmarks(),++ bench.runs({ kind: kindId, limit: 8 }),++ bench.overview(),]);const kind = kinds && Array.isArray(kinds.data.kinds) ? kinds.data.kinds.find((k) => k.id === kindId) : null;if (kinds && !kind) return null;-- return { html: views.aiBoard({ kindId, kind, tab, board, cost, kinds }) };++ const kindBenchmarks = benchmarks && benchmarks.data && Array.isArray(benchmarks.data.benchmarks)++ ? benchmarks.data.benchmarks.filter((b) => b.kind === kindId) : [];++ const rankings = await Promise.all(kindBenchmarks.map((b) => bench.rankings(b.id, 'score')));++ return { html: views.aiBoard({ kindId, kind, tab, board, cost, kinds, benchmarks, rankings, runs, overview }) };++ }++++ async function aiBenchmark(benchmarkId) {++ if (!bench.isBenchmarkId(benchmarkId)) return null;++ const [benchmark, rankings, runs, overview] = await Promise.all([++ bench.benchmark(benchmarkId), bench.rankings(benchmarkId, 'score'), bench.runs({ benchmark: benchmarkId, limit: 20 }), bench.overview(),++ ]);++ // A public id the overview does not list is a 404; a Bench that did not answer at all is the honest unavailable page.++ const list = overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : null;++ if (list && !list.some((b) => b.id === benchmarkId)) return null;++ return { html: views.aiBenchmark({ benchmark, rankings, runs }) };}async function aiModel(model) {if (!bench.isModelName(model)) return null;-- return { html: views.aiModel({ model, summary: await bench.modelSummary(model) }) };++ const [detail, legacy] = await Promise.all([bench.modelDetail(model), bench.modelSummary(model)]);++ return { html: views.aiModel({ model, detail, legacy }) };++ }++++ async function aiMethod() {++ const [overview, kinds] = await Promise.all([bench.overview(), bench.kinds()]);++ return { html: views.aiMethod({ overview, kinds }) };++ }++++ async function aiCompare(models = [], errors = []) {++ const clean = [...new Set((models || []).filter((m) => bench.isModelName(m)))].slice(0, 3);++ const [details, legacy] = await Promise.all([++ Promise.all(clean.map((m) => bench.modelDetail(m))),++ Promise.all(clean.map((m) => bench.modelSummary(m))),++ ]);++ return { models: clean, html: views.aiCompare({ models: clean, details, legacy, errors }) };++ }++++ async function aiFeed() {++ const [runs, overview] = await Promise.all([bench.runs({ limit: 20 }), bench.overview()]);++ return { runs, overview };}function design() { return { html: designViews.designHealth(designHealth.all()) }; }@@@ -80,7 +129,27 @@ function createPublicPages({ sessionReports, network, live, bench, analytics = nreturn { html: views.rundown({ rundown: r, topics, reports }) };}-- return { home, stats, developers, design, rundown, aiIndex, aiBoard, aiModel };++ return { home, stats, developers, design, rundown, aiIndex, aiBoard, aiBenchmark, aiModel, aiMethod, aiCompare, aiFeed };++}++++/** The new-results feed: one item per public run, with its own Bench date and no invented number. */++function rssFeed({ runs, overview, base }) {++ const xml = (s) => String(s == null ? '' : s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"');++ const list = runs && runs.data && Array.isArray(runs.data.runs) ? runs.data.runs : [];++ const titles = new Map((overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : []).map((b) => [b.id, b.title]));++ const site = String(base || 'https://openvibe.report').replace(/\/+$/, '');++ const items = list.map((r) => {++ const title = `${r.model} on ${titles.get(r.benchmark_id) || r.benchmark_id}`;++ const link = `${site}/ai/${encodeURIComponent(r.benchmark_id)}`;++ const score = r.score == null ? 'no score' : `${Math.round(Number(r.score) * 10) / 10}/100`;++ const cost = r.cost_usd == null ? 'cost not given' : `$${Number(r.cost_usd).toFixed(4).replace(/0+$/, '').replace(/\.$/, '')}`;++ const speed = r.ms == null ? 'time not given' : `${Math.round(Number(r.ms))} ms`;++ const at = Date.parse(r.created_at);++ return `<item><title>${xml(title)}</title><link>${xml(link)}</link><guid isPermaLink="false">${xml(r.id)}</guid>${Number.isFinite(at) ? `<pubDate>${new Date(at).toUTCString()}</pubDate>` : ''}<description>${xml(`${r.model} scored ${score} on ${titles.get(r.benchmark_id) || r.benchmark_id}, for ${cost} in ${speed} (${r.provenance || 'run'}).`)}</description></item>`;++ }).join('');++ const newest = list.length ? Date.parse(list[0].created_at) : NaN;++ return `<?xml version="1.0" encoding="UTF-8"?>++<rss version="2.0"><channel><title>OpenVibe AI: new results</title><link>${xml(`${site}/ai`)}</link><description>Verified and submitted AI runs as OpenVibe.Bench publishes them.</description><language>en</language>${Number.isFinite(newest) ? `<lastBuildDate>${new Date(newest).toUTCString()}</lastBuildDate>` : ''}${items}</channel></rss>`;}function createPageRoutes(ctx) {@@@ -137,6 +206,24 @@ function createPageRoutes(ctx) {return send(req, res, page.html, 200, 'no-store');});++ // The AI section's method, comparison and feed. They must sit before /ai/:ref, which would read them as a kind.++ r.get('/ai/method', async (req, res) => send(req, res, (await publicPages.aiMethod()).html, 200, 'no-store'));++++ r.get('/ai/compare', async (req, res) => {++ const raw = String(req.query.models || '').split(',').map((x) => x.trim()).filter(Boolean);++ const errors = [];++ const unique = [...new Set(raw)];++ if (unique.length < 2) errors.push('Pick at least two model ids, comma separated.');++ if (unique.length > 3) errors.push('At most three models can be compared at once.');++ const page = await publicPages.aiCompare(unique.slice(0, 3), errors);++ return send(req, res, page.html, 200, 'no-store');++ });++++ r.get('/ai/feed.xml', async (req, res) => {++ const feed = await publicPages.aiFeed();++ res.status(200).set('Cache-Control', 'public, max-age=120').type('application/rss+xml; charset=utf-8').send(rssFeed({ ...feed, base: ctx.config.baseUrl }));++ });++// The AI Exchange's daily aggregates (plan G-P9), staff-only, behind REPORT_AI_EXCHANGE. The gate is the signed-in// person's staff role (`staff.site.view`, the capability that covers analytics and money state), the same// openvibe-contracts staff map other staff pages use. The page is private and never cached.@@@ -155,8 +242,11 @@ function createPageRoutes(ctx) {}});-- r.get('/ai/:kind', async (req, res) => {-- const page = await publicPages.aiBoard(req.params.kind, bench.normaliseTab(req.query.tab));++ r.get('/ai/:ref', async (req, res) => {++ const ref = String(req.params.ref || '');++ const page = bench.isBenchmarkId(ref)++ ? await publicPages.aiBenchmark(ref)++ : await publicPages.aiBoard(ref, bench.normaliseTab(req.query.tab));if (!page) return notFound(req, res);return send(req, res, page.html, 200, 'no-store');});@@@ -80,7 +80,7 @@ const NAV = [{ label: 'You', href: '/you', icon: 'fa-user' },{ label: 'Numbers', href: '/stats', icon: 'fa-chart-line' },{ label: 'The Rundown', href: '/rundown', icon: 'fa-newspaper' },-- { label: 'AI boards', href: '/ai', icon: 'fa-robot' },++ { label: 'AI', href: '/ai', icon: 'fa-robot' },{ label: 'API', href: '/developers', icon: 'fa-code', more: true },];@@@ -90,7 +90,7 @@ function layout({ title, description, canonical = null, path = null, robots = 'iservice: 'report',apiBase: NETWORK,links: NAV.map((l) => ({ ...l, active: l.href === active || (l.href !== '/' && active.startsWith(l.href)) })),-- appBar: [{ label: 'Home', href: '/', icon: 'fa-house' }, { label: 'Numbers', href: '/stats', icon: 'fa-chart-line' }, { label: 'Rundown', href: '/rundown', icon: 'fa-newspaper' }, { label: 'AI boards', href: '/ai', icon: 'fa-robot' }],++ appBar: [{ label: 'Home', href: '/', icon: 'fa-house' }, { label: 'Numbers', href: '/stats', icon: 'fa-chart-line' }, { label: 'Rundown', href: '/rundown', icon: 'fa-newspaper' }, { label: 'AI', href: '/ai', icon: 'fa-robot' }],history: { type: 'page', title: title || NAME },// Sign-in is this site's own (openvibe-sdk/sso at /auth): the navbar's buttons and its silent check use it.silentLogin: `${BASE}/auth/login?silent=1&next={url}`,@@@ -121,6 +121,7 @@ function layout({ title, description, canonical = null, path = null, robots = 'i<link rel="stylesheet" href="${esc(ovServe.url('room.css'))}"><link rel="stylesheet" href="${esc(ovServe.url('showcase.css'))}"><link rel="stylesheet" href="${esc(asset('css/app.css'))}">++${path && String(path).startsWith('/ai') ? `<link rel="alternate" type="application/rss+xml" title="AI: new results" href="${esc(`${BASE}/ai/feed.xml`)}">` : ''}<script src="${esc(ovServe.url('kit-web.js'))}" defer></script><script src="${esc(ovServe.url('kit-data-web.js'))}" defer></script><script src="${esc(ovServe.url('motion.js'))}" defer></script>@@@ -253,7 +254,7 @@ function home({ stats, rundown, topics } = {}) {explore: { title: 'More on Report', items: [{ title: 'Every site\'s numbers', text: 'Page views, people, streams and how fast each service answers', href: '/stats', icon: 'ov:gauge' },{ title: 'The Rundown', text: 'What streamers talked about, topic by topic', href: '/rundown', icon: 'ov:news' },-- { title: 'AI boards', text: 'Every benchmark kind, its leader, quality against cost and cost per solved task', href: '/ai', icon: 'ov:ai' },++ { title: 'AI', text: 'Which AI does each job best, what it costs and how fast it is: Bench results from every room', href: '/ai', icon: 'ov:ai' },{ title: 'Status', text: 'Whether every OpenVibe service is up right now', href: 'https://openvibe.network/status', icon: 'ov:ping' },] },dev: { title: 'For developers', lede: 'Every number here is open JSON, readable from any site.', samples: [@@@ -644,6 +645,8 @@ function developers() {['GET /api/v1/rundown/t/:slug', 'One topic with its takes and related topics.'],['GET /api/v1/after-show/:streamId', 'A stream\'s after-show report: viewers minute by minute, chat, moments and the write-up.'],['GET /api/v1/c/:username', 'A channel\'s after-show reports, newest first.'],++ ['GET /api/v1/ai/results', 'New public AI runs from OpenVibe.Bench, newest first, with their score, cost and time.'],++ ['GET /ai/feed.xml', 'The same new AI results as an RSS feed.'],];const body = `<header class="rp-page-h"><p class="rp-kicker">For developers</p><h1>The Report API</h1><p class="rp-lede">Open JSON, no key, readable from any site (CORS). Counts only, never about a person.</p></header><section aria-labelledby="rp-ep-h"><h2 id="rp-ep-h">Endpoints</h2><dl class="rp-eps">${rows.map(([ep, what]) => `<div><dt><code>${esc(ep)}</code></dt><dd>${esc(what)}</dd></div>`).join('')}</dl></section>@@@ -652,140 +655,461 @@ function developers() {}function benchState(...reads) {-- const stale = reads.filter((r) => r && r.stale);++ const stale = reads.filter((r) => r && r.stale).sort((a, b) => String(a.at).localeCompare(String(b.at)));if (!stale.length) return '';-- const oldest = stale.map((r) => r.at).sort()[0];-- return `<p class="rp-fine" role="status">Bench is unavailable. Showing the last good copy, read ${esc(dayTime(oldest))}.</p>`;++ return `<p class="rp-fine" role="status">Bench is unavailable. Showing the last good copy, read ${esc(dayTime(stale[0].at))}.</p>`;}++/** The answer inside a Bench read ({ data, at, stale }), or null. */++const readData = (r) => (r && r.data != null ? r.data : null);const aiPath = (kind) => `/ai/${encodeURIComponent(kind)}`;const modelPath = (model) => `/ai/models/${encodeURIComponent(model)}`;const modelLink = (subject) => `<a href="${modelPath(subject)}">${esc(subject)}</a>`;const datasetLink = (kind) => `<p class="rp-fine">Download the <a href="${BENCH}/api/v1/datasets/${encodeURIComponent(kind)}.jsonl">dataset JSONL</a> from Bench.</p>`;++const scoreText = (v) => { const x = n(v); return x == null ? '—' : `${Math.round(x * 10) / 10}`; };++const scoreCell = (v) => { const x = n(v); return x == null ? '—' : `<span title="${esc(String(v))} / 100 from Bench">${esc(scoreText(v))}</span>`; };++const moneyCell = (v) => { const x = n(v); return x == null ? '—' : `<span title="${esc(String(v))} USD from Bench">${esc(usd(x))}</span>`; };++const speedCell = (v) => { const x = n(v); if (x == null) return '—'; return `<span title="${exact(x)} ms from Bench">${esc(msText(x))}</span>`; };++function msText(v) {++ const x = n(v);++ if (x == null) return '—';++ if (x >= 60_000) return `${(x / 60_000).toFixed(x >= 600_000 ? 0 : 1).replace(/\.0$/, '')} min`;++ if (x >= 1_000) return `${(x / 1_000).toFixed(x >= 10_000 ? 0 : 1).replace(/\.0$/, '')} s`;++ return `${Math.round(x)} ms`;++}++const lastRunText = (v) => (v ? `Last run ${dayTime(v)}` : 'No runs yet');++const provenanceCell = (p) => (p === 'verified' ? '<span class="rp-fine">verified</span>' : p === 'submitted' ? '<span class="rp-flag">submitted</span>' : '—');++/** A reading of quality against cost: a model is on the frontier when nothing beats it on both. */++function frontier(rows, xKey, yKey) {++ return rows.map((r) => {++ const x = n(r[xKey]), y = n(r[yKey]);++ const best = x != null && y != null && !rows.some((o) => {++ if (o === r) return false;++ const ox = n(o[xKey]), oy = n(o[yKey]);++ return ox != null && oy != null && ox <= x && oy >= y && (ox < x || oy > y);++ });++ return { ...r, frontier: best, cost: x, quality: y };++ });++}++/** A benchmark's leads and last run, from the overview list (no new number is derived). */++function byKindFromOverview(overview) {++ const map = new Map();++ for (const b of (overview && Array.isArray(overview.list) ? overview.list : [])) {++ if (!b || !b.kind) continue;++ const at = b.last_run;++ if (!at) continue;++ if (!map.has(b.kind) || String(at) > String(map.get(b.kind))) map.set(b.kind, at);++ }++ return map;++}--// ── AI boards (Kit 2): who does each job best, and what it costs ──--// The reader came to choose a model for a job. So the page leads with the leaders, one row per job, and keeps the--// jobs nobody has rated yet in one short list; each board is a ranking, a quality-against-cost chart and the exact--// numbers. Nothing here is drawn from a guess: every figure is Bench's.--const JUDGE_SHORT = { human_pairwise: 'blind votes', rubric: 'a rubric', automatic_grader: 'a grader', test_suite: 'a test suite', game_outcome: 'game results' };--// Where people rate a kind themselves (only blind-vote kinds are rated by people); a room without a site has no link.++// ── AI (OpenVibe.Bench, plan 2026-10-28 §4.12; the owner moved its pages here 2026-11-08) ──++// The reader came to choose a model for a job. Every figure is Bench's: a board's rating, a run's score, the++// cost and the milliseconds the run carried, and the moment it was published. A job nobody has rated yet is a++// short wait list, never a made-up leader.++const JUDGE_SHORT = { human_pairwise: 'blind votes', audience_poll: 'audience polls', rubric: 'a rubric', automatic_grader: 'a grader', test_suite: 'a test suite', game_outcome: 'game results' };++const JUDGE_LONG = {++ human_pairwise: 'People see two answers blind and pick the better one. A vote is one game between two models.',++ audience_poll: 'An audience votes on the outputs; the poll is recorded as one game, however many people voted.',++ rubric: 'A rubric scores the output against the task’s stated criteria.',++ automatic_grader: 'A grader checks the output against the expected answer or the task’s rules.',++ test_suite: 'A pinned test suite runs the output; it passes only when the tests pass.',++ game_outcome: 'The game itself decides: the result is the judgement, not a person’s opinion.',++};const ROOM_HOSTS = { chat: 'openvibe.chat', games: 'openvibe.games', search: 'search.openvibe.network', codes: 'openvibe.codes' };const roomHost = (room) => { const r = rooms.get(room); return (r && r.host) || ROOM_HOSTS[room] || null; };const rowsOf = (read) => (read && read.data && read.data.board && Array.isArray(read.data.board.rows) ? read.data.board.rows : []);--/**-- * Whether a board has results: on a ranking (rows carry games), a game has been played (a model entered but never-- * judged has 0); the quality and cost views carry no games, and any row counts.-- */const played = (rows) => {const list = rows || [];if (!list.some((r) => r && r.games != null)) return list.length > 0;return list.some((r) => (n(r && r.games) || 0) > 0);};--const howRated = (kind) => `Rated by ${JUDGE_SHORT[kind.judge] || 'Bench'}`;++const howRated = (kind) => `Rated by ${JUDGE_SHORT[kind && kind.judge] || 'Bench'}`;const METHOD = 'Ratings are Bradley–Terry Elo from Bench, with a 95% interval; a model with fewer than ten games is provisional.';++const CHART_NOTE = 'Up and to the left is better. The dashed line joins the models nothing beats on both.';++const BENCH_API = `${BENCH}/api/v1`;--/** What to do when a board is empty: rate it in its room (blind-vote kinds), or wait for runs (graded kinds). */function emptyFor(kind) {const host = kind && kind.judge === 'human_pairwise' ? roomHost(kind.room) : null;if (host) return kit.empty({ icon: 'trophy', title: 'No votes yet', text: `This board fills as people compare two answers blind on ${roomName(kind.room)} and pick the better one.`, action: { href: `https://${host}/`, label: `Vote on ${roomName(kind.room)}`, icon: 'forward' } });return kit.empty({ icon: 'trophy', title: 'No results yet', text: kind ? `This board fills as models are run and ${JUDGE_SHORT[kind.judge] || 'Bench'} scores them.` : 'This board fills as models are run on Bench.' });}--function aiIndex({ kinds, boards = [] }) {-- const list = kinds && kinds.data && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];-- const items = list.map((kind, i) => ({ kind, rows: rowsOf(boards[i]), available: Boolean(boards[i]) }));-- // A board is rated once a game is played. Bench lists a model as soon as it has an entry, at the starting 1000-- // with 0 games: that is no leader (11-03, every board on /ai named gpt-5-nano "leader" with 0 games).-- const rated = items.filter((x) => played(x.rows)).sort((a, b) => gamesOf(b.rows) - gamesOf(a.rows));-- const waiting = items.filter((x) => !played(x.rows));-- const leaders = rated.length ? kit.table({-- caption: 'Leaders', columns: [-- { key: 'job', label: 'Job', primary: true, html: true }, { key: 'leader', label: 'Leader', html: true },-- { key: 'rating', label: 'Rating', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' },-- ],-- rows: rated.map(({ kind, rows }) => ({ job: `<a href="${aiPath(kind.id)}">${esc(kind.name || kind.id)}</a>`, leader: `${modelLink(rows[0].subject)}${flag(rows[0])}`, rating: rating(rows[0]), games: gamesOf(rows) })),-- }) : '';-- const waitRows = waiting.map(({ kind, available }) => kit.row({ href: aiPath(kind.id), title: kind.name || kind.id, sub: `${roomName(kind.room)} · ${available ? howRated(kind) : 'board unavailable'}`, trail: kit.icon('chevron') }));-- const waitList = waitRows.length ? kit.list(waitRows, { flush: true }) : '';-- const none = !rated.length && list.length;-- const body = kit.head({ title: 'AI boards', sub: 'Which AI does each job best, and what it costs', sticky: false })-- + benchState(kinds, ...boards)-- + (!list.length ? kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'The boards come back as soon as Bench answers.' }) : '')-- + (none ? kit.notice({ kind: 'info', text: `No model is rated yet. ${list.length} jobs are set up; each board fills as people vote blind on two answers, or as graders score runs.` }) : '')-- + (rated.length ? kit.stats([[rated.length, rated.length === 1 ? 'job rated' : 'jobs rated'], [rated.reduce((s, x) => s + gamesOf(x.rows), 0), 'games played'], [list.length, 'jobs tracked']], { label: 'The boards in numbers' }) : '')-- + leaders-- + (waitList ? (none ? kit.section({ title: 'Jobs being rated', count: waiting.length, body: waitList }) : kit.fold({ title: 'Waiting for their first results', meta: String(waiting.length), body: waitList })) : '')-- + `<p class="rp-fine">${esc(METHOD)} Every board's data is open: <a href="${BENCH}/docs">Bench's API and datasets</a>.</p>`;-- return layout({ title: 'AI boards', description: 'Which AI does each job best, and what it costs: Bench ratings across OpenVibe.', body, active: '/ai', path: '/ai', agent: true });--}--++/** The cost-per-solved chart Bench's boards already use: quality (0–1) against cost per solved task. */function paretoRows(pareto, cost) {const costs = new Map(rowsOf(cost).map((r) => [r.subject, n(r.cost_per_solved)]));const rows = rowsOf(pareto).map((r) => ({ ...r, quality: n(r.quality), costPerSolved: costs.get(r.subject) }));-- return rows.map((r) => ({ ...r, frontier: r.quality != null && r.costPerSolved != null && !rows.some((other) => other !== r && other.quality != null && other.costPerSolved != null && other.quality >= r.quality && other.costPerSolved <= r.costPerSolved && (other.quality > r.quality || other.costPerSolved < r.costPerSolved)) }));++ return frontier(rows.map((r) => ({ ...r, x: r.costPerSolved, y: r.quality })), 'x', 'y');}----/** Quality against cost per solved task, the frontier dashed and labelled (kit.chart scatter). */function paretoChart(rows) {const points = rows.filter((r) => r.quality != null && r.costPerSolved != null && r.costPerSolved >= 0);if (!points.length) return '<p class="rp-fine">No quality and cost per solved pairs yet.</p>';return kit.chart({-- type: 'scatter', title: 'Quality against cost per solved task', note: 'Up and to the left is better. The dashed line joins the models nothing beats on both.',++ type: 'scatter', title: 'Quality against cost per solved task', note: CHART_NOTE,x: { label: 'Cost per solved task (USD, lower is better)', fmt: (v) => usd(v) }, y: { label: 'Quality', fmt: (v) => `${Math.round(v)}%`, zero: false },points: points.map((r) => ({ label: r.subject, x: r.costPerSolved, y: r.quality * 100, best: r.frontier, mark: r.frontier || points.length <= 8, href: modelPath(r.subject) })),height: 260,});}++/** A benchmark's verifier: score 0–100 against the median cost of a run. The frontier is the same reading. */++function scoreCostChart(entries) {++ const rows = frontier(entries.filter((r) => n(r.cost_usd) != null && n(r.score) != null).map((r) => ({ ...r, x: n(r.cost_usd), y: n(r.score) })), 'x', 'y');++ if (!rows.length) return '';++ return kit.chart({++ type: 'scatter', title: 'Score against cost per run', note: CHART_NOTE,++ x: { label: 'Median cost of a run (USD, lower is better)', fmt: (v) => usd(v) }, y: { label: 'Score (0–100)', fmt: (v) => `${Math.round(v)}`, zero: false },++ points: rows.map((r) => ({ label: r.model, x: r.cost, y: r.quality, best: r.frontier, mark: r.frontier || rows.length <= 8, href: modelPath(r.model) })),++ height: 260,++ });++}const TABS = [{ key: 'elo', label: 'Ranking' }, { key: 'pareto', label: 'Quality and cost' }, { key: 'cost', label: 'Cost per solved' }];--function aiBoard({ kindId, kind, tab, board, cost, kinds }) {++function aiIndex({ overview, kinds, boards = [], rankings = [] }) {++ const o = readData(overview) || {};++ const list = kinds && kinds.data && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];++ const benches = Array.isArray(o.list) ? o.list : [];++ const lastByKind = byKindFromOverview(o);++ const items = list.map((kind, i) => ({ kind, rows: rowsOf(boards[i]), available: Boolean(boards[i]) }));++ const rated = items.filter((x) => played(x.rows)).sort((a, b) => gamesOf(b.rows) - gamesOf(a.rows));++ const waiting = items.filter((x) => !played(x.rows));++ const leaders = rated.length ? kit.table({++ caption: 'Task kinds and their leaders', columns: [++ { key: 'job', label: 'Job', primary: true, html: true }, { key: 'leader', label: 'Leader', html: true },++ { key: 'quality', label: 'Quality', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' },++ { key: 'cost', label: 'Cost/run', num: true, html: true },++ { key: 'speed', label: 'Speed/run', num: true, html: true }, { key: 'last', label: 'Last run', html: true },++ ],++ rows: rated.map(({ kind, rows }) => {++ const top = rows[0];++ return {++ job: `<a href="${aiPath(kind.id)}">${esc(kind.name || kind.id)}</a>`,++ leader: `${modelLink(top.subject)}${flag(top)}`,++ quality: rating(top),++ games: gamesOf(rows),++ cost: moneyCell(top.median_cost_usd),++ speed: speedCell(top.median_ms),++ last: lastByKind.get(kind.id) ? `<span title="${esc(lastByKind.get(kind.id))}">${esc(dayTime(lastByKind.get(kind.id)))}</span>` : '—',++ };++ }),++ }) : '';++ const waitRows = waiting.map(({ kind, available }) => kit.row({ href: aiPath(kind.id), title: kind.name || kind.id, sub: `${roomName(kind.room)} · ${available ? howRated(kind) : 'board unavailable'}`, trail: kit.icon('chevron') }));++ const waitList = waitRows.length ? kit.list(waitRows, { flush: true }) : '';++ const benchRow = (b, i) => {++ const r = rankings[i] && readData(rankings[i]);++ const top = r && r.rankings && Array.isArray(r.rankings.entries) ? r.rankings.entries[0] : null;++ return {++ benchmark: `<a href="${aiPath(b.id)}">${esc(b.title)}</a>`,++ kind: esc(b.kind || '—'),++ leader: top ? `${modelLink(top.model)} ${scoreCell(top.score)}` : '—',++ runs: b.runs == null ? 0 : b.runs,++ last: b.last_run ? esc(dayTime(b.last_run)) : 'No runs yet',++ };++ };++ const activeBench = benches.filter((b) => Number(b.runs) > 0);++ const idleBench = benches.filter((b) => !(Number(b.runs) > 0));++ const benchTable = activeBench.length ? kit.table({++ caption: 'Benchmarks with published runs', columns: [++ { key: 'benchmark', label: 'Benchmark', primary: true, html: true }, { key: 'kind', label: 'Task kind' },++ { key: 'leader', label: 'Leader', html: true }, { key: 'runs', label: 'Runs', num: true, fmt: 'int' }, { key: 'last', label: 'Last run', html: true },++ ], rows: benches.map((b, i) => (Number(b.runs) > 0 ? benchRow(b, i) : null)).filter(Boolean),++ }) : '';++ const idleRows = benches.map((b, i) => (Number(b.runs) > 0 ? null : kit.row({ href: aiPath(b.id), title: b.title, sub: `${b.kind || 'benchmark'} · no run yet`, trail: kit.icon('chevron') }))).filter(Boolean);++ const idleList = idleRows.length ? kit.list(idleRows, { flush: true }) : '';++ const none = !rated.length && list.length;++ const body = kit.head({ title: 'AI', sub: 'Which AI does each job best, what it costs, and how fast it is', sticky: false })++ + benchState(overview, kinds, ...rankings, ...boards)++ + `<p class="rp-fine">Every number on this page is a published Bench result or vote. ${esc(METHOD)} <a href="/ai/method">How the runs are judged</a> · <a href="/ai/compare">Compare models</a> · <a href="/ai/feed.xml">New results (RSS)</a></p>`++ + (!list.length && !benches.length ? kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'The boards come back as soon as Bench answers.' }) : '')++ + (o.benchmarks != null || o.runs != null ? kit.stats([++ [o.benchmarks == null ? '—' : exact(o.benchmarks), o.benchmarks === 1 ? 'benchmark' : 'benchmarks'],++ [o.tasks == null ? '—' : exact(o.tasks), o.tasks === 1 ? 'task' : 'tasks'],++ [o.runs == null ? '—' : exact(o.runs), o.runs === 1 ? 'run' : 'runs'],++ [o.models == null ? '—' : exact(o.models), o.models === 1 ? 'model' : 'models'],++ [o.last_run ? dayTime(o.last_run) : 'No runs yet', 'last run'],++ ], { label: 'AI in numbers' }) : (rated.length ? kit.stats([++ [rated.length, rated.length === 1 ? 'job rated' : 'jobs rated'],++ [rated.reduce((sum, x) => sum + gamesOf(x.rows), 0), 'games played'],++ [list.length, 'jobs tracked'],++ ], { label: 'The boards in numbers' }) : ''))++ + (none ? kit.notice({ kind: 'info', text: `No model is rated on a task kind yet. ${list.length} jobs are set up; each board fills as people vote blind on two answers, or as graders score runs.` }) : '')++ + (leaders ? kit.section({ title: 'Task kinds', count: rated.length, body: leaders }) : '')++ + (benchTable ? kit.section({ title: 'Benchmarks with published runs', count: activeBench.length, body: benchTable }) : '')++ + (idleList ? kit.fold({ title: 'Waiting for their first run', meta: String(idleRows.length), body: idleList }) : '')++ + (waitList ? (none ? kit.fold({ title: 'Jobs being rated', meta: String(waiting.length), body: waitList }) : kit.fold({ title: 'Waiting for their first results', meta: String(waiting.length), body: waitList })) : '')++ + `<p class="rp-fine">The open data is Bench's: <a href="${BENCH}/docs">its API and datasets</a> stay on bench.openvibe.services.</p>`;++ return layout({ title: 'AI', description: 'Which AI does each job best, and what it costs: OpenVibe.Bench results, rendered by OpenVibe.Report.', body, active: '/ai', path: '/ai', agent: true });++}++++function aiBoard({ kindId, kind, tab, board, cost, kinds, benchmarks, rankings = [], runs, overview }) {const name = kind && kind.name || kindId;const rows = rowsOf(board);++ const o = readData(overview) || {};++ const lastByKind = byKindFromOverview(o);++ const kindLast = lastByKind.get(kindId) || null;++ const latest = readData(runs);++ const latestRows = latest && Array.isArray(latest.runs) ? latest.runs : [];++ // The registry's verified runs on this kind's public benchmarks: a second, task-level reading even while the++ // blind board has no votes. A model's numbers across the kind's benchmarks are medians, never a sum of scores.++ const kindBenchmarks = (() => {++ const b = readData(benchmarks);++ return b && Array.isArray(b.benchmarks) ? b.benchmarks.filter((x) => x.kind === kindId) : [];++ })();++ const byModel = new Map();++ kindBenchmarks.forEach((b, i) => {++ const r = rankings[i] && readData(rankings[i]);++ const entries = r && r.rankings && Array.isArray(r.rankings.entries) ? r.rankings.entries : [];++ for (const e of entries) {++ if (!byModel.has(e.model)) byModel.set(e.model, { model: e.model, scores: [], costs: [], times: [], runs: 0, last_run: null });++ const x = byModel.get(e.model);++ if (n(e.score) != null) x.scores.push(n(e.score));++ if (n(e.cost_usd) != null) x.costs.push(n(e.cost_usd));++ if (n(e.ms) != null) x.times.push(n(e.ms));++ x.runs += Number(e.runs) || 0;++ if (e.last_run && (!x.last_run || e.last_run > x.last_run)) x.last_run = e.last_run;++ }++ });++ const median = (list) => { const v = list.filter((x) => x != null).sort((a, b) => a - b); if (!v.length) return null; const m = Math.floor(v.length / 2); return v.length % 2 ? v[m] : (v[m - 1] + v[m]) / 2; };++ const registryRows = [...byModel.values()].map((x) => ({ model: x.model, score: median(x.scores), cost: median(x.costs), speed: median(x.times), runs: x.runs, last_run: x.last_run })).sort((a, b) => (b.score == null ? -1 : b.score) - (a.score == null ? -1 : a.score) || a.model.localeCompare(b.model));++ const registryTable = registryRows.length ? kit.table({ caption: 'Published runs on this kind', columns: [++ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Median score', num: true, html: true },++ { key: 'runs', label: 'Runs', num: true, fmt: 'int' }, { key: 'cost', label: 'Cost/run', num: true, html: true },++ { key: 'speed', label: 'Speed/run', num: true, html: true }, { key: 'last', label: 'Last run', html: true },++ ], rows: registryRows.map((r) => ({ model: modelLink(r.model), score: scoreText(r.score), runs: r.runs, cost: moneyCell(r.cost), speed: speedCell(r.speed), last: r.last_run ? esc(dayTime(r.last_run)) : '—' })) }) : '';let content;if (!board) content = kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'This board comes back as soon as Bench answers.' });++ else if (!played(rows) && registryRows.length) content = kit.notice({ kind: 'info', text: `No blind vote on this task kind yet. ${registryRows.length} ${registryRows.length === 1 ? 'model has' : 'models have'} verified runs below.` });else if (!played(rows)) content = emptyFor(kind);else if (tab === 'pareto') {const points = paretoRows(board, cost);-- content = paretoChart(points) + (points.length ? kit.table({ caption: 'The numbers', columns: [{ key: 'm', label: 'Model', primary: true, html: true }, { key: 'q', label: 'Quality', num: true, html: true }, { key: 'c', label: 'Cost per solved', num: true, html: true }],-- // The frontier is a word on the model's name, so a phone's card stays two lines.-- rows: points.map((r) => ({ m: `${modelLink(r.subject)}${r.frontier ? ` ${kit.badge('frontier', 'accent')}` : ''}`, q: share(r.quality), c: usd(r.costPerSolved) })) }) : '');++ content = paretoChart(points) + (points.length ? kit.table({ caption: 'The numbers', columns: [++ { key: 'm', label: 'Model', primary: true, html: true }, { key: 'q', label: 'Quality', num: true, html: true },++ { key: 'c', label: 'Cost per solved', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },++ ], rows: points.map((r) => ({ m: `${modelLink(r.subject)}${r.frontier ? ` ${kit.badge('frontier', 'accent')}` : ''}`, q: share(r.quality), c: usd(r.costPerSolved), speed: speedCell(r.median_ms) })) }) : '');} else if (tab === 'cost') {const crows = rowsOf(cost).length ? rowsOf(cost) : rows;-- content = kit.table({ caption: 'Cost per solved task', columns: [{ key: 'm', label: 'Model', primary: true, html: true }, { key: 'solved', label: 'Solved', num: true, fmt: 'int' }, { key: 'entries', label: 'Entries', num: true, fmt: 'int' }, { key: 'total', label: 'Total cost', num: true, html: true }, { key: 'per', label: 'Cost per solved', num: true, html: true }],-- rows: crows.map((r) => ({ m: modelLink(r.subject), solved: r.solved, entries: r.entries, total: usd(r.total_cost_usd), per: usd(r.cost_per_solved) })) });++ content = kit.table({ caption: 'Cost per solved task', columns: [++ { key: 'm', label: 'Model', primary: true, html: true }, { key: 'solved', label: 'Solved', num: true, fmt: 'int' },++ { key: 'entries', label: 'Entries', num: true, fmt: 'int' }, { key: 'total', label: 'Total cost', num: true, html: true },++ { key: 'per', label: 'Cost per solved', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },++ ], rows: crows.map((r) => ({ m: modelLink(r.subject), solved: r.solved, entries: r.entries, total: usd(r.total_cost_usd), per: usd(r.cost_per_solved), speed: speedCell(r.median_ms) })) });} else {-- // The bars start just under the lowest interval, so the differences that matter are the ones you see.const floor = Math.floor((Math.min(...rows.map((r) => n(r.ci_lower) ?? n(r.elo) ?? 0)) - 25) / 50) * 50;content = kit.rank({ label: `${name}: ranking`, base: floor, fmt: 'int', items: rows.map((r) => {const lo = n(r.ci_lower), hi = n(r.ci_upper), elo = n(r.elo);++ const costText = n(r.median_cost_usd) != null ? `${usd(r.median_cost_usd)}/run` : null;++ const speedText = n(r.median_ms) != null ? `${msText(r.median_ms)}/run` : null;return { name: r.subject, href: modelPath(r.subject), value: elo, interval: lo != null && hi != null && elo != null ? Math.round(Math.max(hi - elo, elo - lo)) : null,-- note: [n(r.games) != null ? `${exact(r.games)} games` : '', n(r.win_rate) != null ? `${share(r.win_rate)} won` : '', provisional(r) ? 'provisional' : ''].filter(Boolean).join(' · ') };-- }) }) + `<p class="rp-fine">${esc(METHOD)}</p>`;++ note: [n(r.games) != null ? `${exact(r.games)} games` : '', n(r.win_rate) != null ? `${share(r.win_rate)} won` : '', costText, speedText, provisional(r) ? 'provisional' : ''].filter(Boolean).join(' · ') };++ }) }) + `<p class="rp-fine">${esc(METHOD)}</p>`++ + kit.table({ caption: 'Quality, price and speed side by side', columns: [++ { key: 'm', label: 'Model', primary: true, html: true }, { key: 'q', label: 'Quality', num: true, html: true },++ { key: 'games', label: 'Games', num: true, fmt: 'int' }, { key: 'won', label: 'Won', num: true, html: true },++ { key: 'cost', label: 'Cost/run', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },++ ], rows: rows.map((r) => ({ m: modelLink(r.subject), q: rating(r), games: r.games, won: share(r.win_rate), cost: moneyCell(r.median_cost_usd), speed: speedCell(r.median_ms) })) });}-- const head = kit.head({ title: name, sub: kind ? `${roomName(kind.room)} · ${howRated(kind)}` : 'Bench results', back: { href: '/ai', label: 'AI boards' },++ const lastRunTable = latestRows.length ? kit.table({++ caption: 'Latest runs of this kind', columns: [++ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },++ { key: 'cost', label: 'Cost', num: true, html: true }, { key: 'speed', label: 'Time', num: true, html: true },++ { key: 'when', label: 'Published', html: true }, { key: 'how', label: 'How', html: true },++ ],++ rows: latestRows.map((r) => ({ model: modelLink(r.model), score: scoreCell(r.score), cost: usd(r.cost_usd), speed: speedCell(r.ms), when: esc(dayTime(r.created_at)), how: provenanceCell(r.provenance) })),++ }) : '<p class="rp-fine">No run of this task kind has been published yet.</p>';++ const head = kit.head({ title: name, sub: kind ? `${roomName(kind.room)} · ${howRated(kind)}${kindLast ? ` · ${lastRunText(kindLast)}` : ''}` : 'Bench results', back: { href: '/ai', label: 'AI' },tabs: played(rows) ? kit.tabs({ base: aiPath(kindId), items: TABS, active: tab, label: 'Board view' }) : '' });-- const body = head + benchState(kinds, board, cost) + `<section aria-label="${esc(TABS.find((t) => t.key === tab)?.label || 'Board')}">${content}</section>${datasetLink(kindId)}`;-- return layout({ title: `${name}: AI board`, description: `${name}: which AI does it best, and what it costs.`, body, active: '/ai', path: aiPath(kindId), agent: true });--}----function aiModel({ model, summary }) {-- const data = summary && summary.data || null;-- const ratings = data && Array.isArray(data.ratings) ? data.ratings : [];-- const entries = data && Array.isArray(data.entries) ? data.entries : [];-- const kinds = [...new Set([...ratings.map((r) => r.kind_id), ...entries.map((e) => e.kind_id)])].filter(Boolean);-- const rows = kinds.map((kind) => {-- const r = ratings.find((item) => item.kind_id === kind);-- return { job: `<a href="${aiPath(kind)}">${esc(kind)}</a>`, rating: r ? `${rating(r)}${flag(r)}` : '—', games: r ? r.games : null, won: r ? share(r.win_rate) : '—', recent: entries.filter((item) => item.kind_id === kind).length };++ const body = head + benchState(kinds, board, cost, benchmarks, runs, overview)++ + `<section aria-label="${esc(TABS.find((t) => t.key === tab)?.label || 'Board')}">${content}</section>`++ + (registryRows.length ? kit.section({ title: 'Verified runs on this task kind', count: registryRows.length, body: (registryRows.length > 1 ? scoreCostChart(registryRows.map((r) => ({ model: r.model, score: r.score, cost_usd: r.cost, ms: r.speed, runs: r.runs, last_run: r.last_run }))) : '') + registryTable }) : '')++ + kit.section({ title: 'Latest runs', count: latestRows.length, body: lastRunTable })++ + datasetLink(kindId);++ return layout({ title: `${name}: AI`, description: `${name}: which AI does it best, what it costs and how fast it is.`, body, active: '/ai', path: aiPath(kindId), agent: true });++}++++function aiBenchmark({ benchmark, rankings, runs }) {++ const b = readData(benchmark);++ const rank = readData(rankings);++ const latest = readData(runs);++ const entries = rank && rank.rankings && Array.isArray(rank.rankings.entries) ? rank.rankings.entries : [];++ const latestRows = latest && Array.isArray(latest.runs) ? latest.runs : [];++ const meta = b && b.benchmark ? b.benchmark : null;++ const definition = b && b.definition ? b.definition : null;++ const tasks = definition && Array.isArray(definition.tasks) ? definition.tasks : [];++ const version = b && b.version ? b.version.version : null;++ const last = entries.reduce((at, e) => (!at || (e.last_run && e.last_run > at) ? e.last_run : at), null)++ || (latestRows[0] && latestRows[0].created_at) || null;++ if (!meta) {++ const body = kit.head({ title: 'Benchmark', sub: 'Bench results', back: { href: '/ai', label: 'AI' }, sticky: false })++ + benchState(benchmark) + kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'This benchmark comes back as soon as Bench answers.' });++ return layout({ title: 'AI benchmark', description: 'A benchmark of AI runs from OpenVibe.Bench.', body, active: '/ai', path: '/ai', agent: true });++ }++ const chart = scoreCostChart(entries);++ const leaderboard = entries.length ? kit.table({++ caption: 'Verified results', columns: [++ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },++ { key: 'runs', label: 'Runs', num: true, fmt: 'int' }, { key: 'range', label: 'Best / worst', num: true, html: true },++ { key: 'cost', label: 'Cost/run', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },++ { key: 'solved', label: 'Cost per solved', num: true, html: true }, { key: 'last', label: 'Last run', html: true },++ ],++ rows: entries.map((e) => ({++ model: modelLink(e.model), score: scoreCell(e.score), runs: e.runs,++ range: `${scoreCell(e.max_score)} / ${scoreCell(e.min_score)}`, cost: moneyCell(e.cost_usd), speed: speedCell(e.ms),++ solved: moneyCell(e.cost_per_solved), last: e.last_run ? esc(dayTime(e.last_run)) : '—',++ })),++ }) : kit.empty({ icon: 'trophy', title: 'No verified results yet', text: 'This benchmark has no verified run yet. Submitted runs are labelled on the latest-runs list and never mixed into the ranking.' });++ const latestTable = latestRows.length ? kit.table({++ caption: 'Latest runs', columns: [++ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },++ { key: 'cost', label: 'Cost', num: true, html: true }, { key: 'speed', label: 'Time', num: true, html: true },++ { key: 'when', label: 'Published', html: true }, { key: 'how', label: 'How', html: true },++ ],++ rows: latestRows.map((r) => ({ model: modelLink(r.model), score: scoreCell(r.score), cost: usd(r.cost_usd), speed: speedCell(r.ms), when: esc(dayTime(r.created_at)), how: provenanceCell(r.provenance) })),++ }) : '<p class="rp-fine">No run has been published on this benchmark yet.</p>';++ const body = kit.head({ title: meta.title, sub: `${meta.kind || 'Benchmark'}${version ? ` · version ${version}` : ''} · ${tasks.length} ${tasks.length === 1 ? 'task' : 'tasks'} · ${lastRunText(last)}`, back: { href: '/ai', label: 'AI' }, sticky: false })++ + benchState(benchmark, rankings, runs)++ + (meta.description ? `<p class="rp-lede">${esc(meta.description)}</p>` : '')++ + kit.stats([[entries.length, entries.length === 1 ? 'model ranked' : 'models ranked'], [latestRows.length, 'latest runs'], [tasks.length, tasks.length === 1 ? 'task' : 'tasks'], [last ? dayTime(last) : 'No runs yet', 'last run']], { label: `${meta.title} in numbers` })++ + chart + leaderboard++ + kit.section({ title: 'Latest runs', count: latestRows.length, body: latestTable })++ + `<p class="rp-fine"><a href="/ai/method">How the runs are judged</a> · <a href="/ai/compare?models=${entries.slice(0, 3).map((e) => encodeURIComponent(e.model)).join(',')}">Compare the leaders</a> · <a href="${BENCH}/api/v1/datasets/benchmarks.jsonl">Bench's open data</a></p>`;++ return layout({ title: `${meta.title}: AI benchmark`, description: `${meta.title}: verified AI results, cost and speed from OpenVibe.Bench.`, body, active: '/ai', path: aiPath(meta.id), agent: true });++}++++function aiModel({ model, detail, legacy }) {++ const d = readData(detail) || null;++ const l = readData(legacy) || null;++ const runs = d && Array.isArray(d.recent_runs) ? d.recent_runs : [];++ const benchmarks = d && Array.isArray(d.benchmarks) ? d.benchmarks : [];++ const tasks = d && Array.isArray(d.tasks) ? d.tasks : [];++ const ratings = l && Array.isArray(l.ratings) ? l.ratings : [];++ const entries = l && Array.isArray(l.entries) ? l.entries : [];++ const hasRuns = Boolean(d && d.runs > 0);++ const history = runs.filter((r) => n(r.score) != null && ts(r.created_at) != null).sort((a, b) => String(a.created_at).localeCompare(String(b.created_at)));++ const historyChart = history.length >= 2 ? kit.chart({++ type: 'line', title: 'Score over time', note: 'Every public run, newest last.',++ x: { time: true, label: 'Published' }, y: { label: 'Score (0–100)', zero: false, fmt: (v) => `${Math.round(v)}` },++ series: [{ name: model, points: history.map((r) => [r.created_at, n(r.score)]) }], height: 220,++ }) : '';++ const best = tasks.slice(0, 5);++ const worst = tasks.length > 5 ? tasks.slice(-5).reverse() : [];++ const taskRow = (t) => ({++ task: `<span title="${esc(t.benchmark_title)}">${esc(t.task_id)}</span>`,++ benchmark: `<a href="${aiPath(t.benchmark_id)}">${esc(t.benchmark_title)}</a>`,++ score: scoreCell(t.score), runs: t.runs, last: t.last_run ? esc(dayTime(t.last_run)) : '—',});-- const games = ratings.reduce((s, r) => s + (n(r.games) || 0), 0);-- const head = kit.head({ title: model, sub: 'Every job this model has been rated on', back: { href: '/ai', label: 'AI boards' }, sticky: false });-- const body = head + benchState(summary) + (!data ? kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'This page comes back as soon as Bench answers.' })-- : kit.stats([[kinds.length, kinds.length === 1 ? 'job' : 'jobs'], [games, 'games'], [n(data.total_cost_usd) ? usd(data.total_cost_usd) : null, 'spent on runs']], { label: `${model} in numbers` })-- + (rows.length ? kit.table({ caption: 'By job', columns: [{ key: 'job', label: 'Job', primary: true, html: true }, { key: 'rating', label: 'Rating', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' }, { key: 'won', label: 'Won', num: true, html: true }, { key: 'recent', label: 'Recent entries', num: true, fmt: 'int' }], rows })-- : kit.empty({ icon: 'trophy', title: 'Not rated yet', text: 'This model appears here once it has played on a Bench board.' })))-- + `<p class="rp-fine">${esc(METHOD)}</p>`;-- return layout({ title: `${model}: AI model`, description: `${model} across OpenVibe benchmark jobs: ratings, games and cost.`, body, active: '/ai', path: modelPath(model), agent: true });++ const taskColumns = [++ { key: 'task', label: 'Task', primary: true, html: true }, { key: 'benchmark', label: 'Benchmark', html: true },++ { key: 'score', label: 'Median score', num: true, html: true }, { key: 'runs', label: 'Runs', num: true, fmt: 'int' },++ { key: 'last', label: 'Last run', html: true },++ ];++ const taskTable = best.length ? kit.table({ caption: 'Best tasks', columns: taskColumns, rows: best.map(taskRow) }) : '';++ const worstTable = worst.length ? kit.table({ caption: 'Worst tasks', columns: taskColumns, rows: worst.map(taskRow) }) : '';++ const benchmarkRows = benchmarks.map((b) => ({++ benchmark: `<a href="${aiPath(b.benchmark_id)}">${esc(b.title)}</a>`, kind: esc(b.kind || '—'), runs: b.runs,++ score: scoreCell(b.score), range: `${scoreCell(b.best)} / ${scoreCell(b.worst)}`, cost: moneyCell(b.cost_usd), speed: speedCell(b.ms), last: b.last_run ? esc(dayTime(b.last_run)) : '—',++ }));++ const runTitles = new Map(benchmarks.map((b) => [b.benchmark_id, b.title]));++ const runRows = runs.slice(0, 12).map((r) => ({++ benchmark: `<a href="${aiPath(r.benchmark_id)}">${esc(runTitles.get(r.benchmark_id) || r.benchmark_id)}</a>`, score: scoreCell(r.score), cost: usd(r.cost_usd), speed: speedCell(r.ms),++ when: esc(dayTime(r.created_at)), how: provenanceCell(r.provenance),++ }));++ const legacyRows = ratings.map((r) => ({ job: `<a href="${aiPath(r.kind_id)}">${esc(r.kind_id)}</a>`, rating: rating(r), games: r.games, won: share(r.win_rate) }));++ const body = kit.head({ title: model, sub: hasRuns ? `${exact(d.runs)} verified ${d.runs === 1 ? 'run' : 'runs'} · ${lastRunText(d.last_run)}` : 'No verified runs yet', back: { href: '/ai', label: 'AI' }, sticky: false })++ + benchState(detail, legacy)++ + (hasRuns ? kit.stats([++ [exact(d.runs), d.runs === 1 ? 'run' : 'runs'], [benchmarks.length, benchmarks.length === 1 ? 'benchmark' : 'benchmarks'],++ [d.best_task ? scoreText(d.best_task.score) : '—', 'best task'], [d.worst_task ? scoreText(d.worst_task.score) : '—', 'worst task'],++ [d.total_cost_usd == null ? '—' : usd(d.total_cost_usd), 'spent on runs'], [d.last_run ? dayTime(d.last_run) : 'No runs yet', 'last run'],++ ], { label: `${model} in numbers` }) : '')++ + (historyChart ? kit.section({ title: 'Runs over time', body: historyChart }) : '')++ + (benchmarkRows.length ? kit.section({ title: 'By benchmark', count: benchmarks.length, body: kit.table({ caption: 'By benchmark', columns: [++ { key: 'benchmark', label: 'Benchmark', primary: true, html: true }, { key: 'kind', label: 'Task kind' }, { key: 'runs', label: 'Runs', num: true, fmt: 'int' },++ { key: 'score', label: 'Score', num: true, html: true }, { key: 'range', label: 'Best / worst', num: true, html: true }, { key: 'cost', label: 'Cost/run', num: true, html: true },++ { key: 'speed', label: 'Speed/run', num: true, html: true }, { key: 'last', label: 'Last run', html: true },++ ], rows: benchmarkRows }) }) : '')++ + (taskTable ? kit.section({ title: 'Best and worst tasks', count: best.length + worst.length, body: taskTable + worstTable }) : '')++ + (runRows.length ? kit.section({ title: 'Recent runs', count: runRows.length, body: kit.table({ caption: 'Recent runs', columns: [++ { key: 'benchmark', label: 'Benchmark', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },++ { key: 'cost', label: 'Cost', num: true, html: true }, { key: 'speed', label: 'Time', num: true, html: true }, { key: 'when', label: 'Published', html: true }, { key: 'how', label: 'How', html: true },++ ], rows: runRows }) }) : '')++ + (!hasRuns ? kit.empty({ icon: 'trophy', title: 'No verified runs yet', text: 'This model appears on a benchmark once a verified run is published. Its blind-board ratings, when it has any, are below.' }) : '')++ + (legacyRows.length ? kit.section({ title: 'Blind-board ratings', count: legacyRows.length, body: kit.table({ caption: 'Ratings by job', columns: [++ { key: 'job', label: 'Job', primary: true, html: true }, { key: 'rating', label: 'Rating', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' }, { key: 'won', label: 'Won', num: true, html: true },++ ], rows: legacyRows }) }) : '')++ + (entries.length && !hasRuns ? `<p class="rp-fine">${exact(entries.length)} ${entries.length === 1 ? 'entry' : 'entries'} on Bench’s blind boards.</p>` : '')++ + `<p class="rp-fine"><a href="/ai/compare?models=${encodeURIComponent(model)}">Compare ${esc(model)} with another model</a> · <a href="/ai/method">How the runs are judged</a> · <a href="${BENCH}/api/v1/models/${encodeURIComponent(model)}">Bench’s own record</a></p>`;++ return layout({ title: `${model}: AI model`, description: `${model} across OpenVibe benchmark jobs: score, cost, speed, runs over time and its best and worst tasks.`, body, active: '/ai', path: modelPath(model), agent: true });++}++++function aiCompare({ models = [], details = [], legacy = [], errors = [] }) {++ const rowsByBenchmark = new Map();++ const names = [];++ const cards = [];++ models.forEach((model, i) => {++ const d = readData(details[i]) || null;++ const l = readData(legacy[i]) || null;++ names.push(model);++ const list = d && Array.isArray(d.benchmarks) ? d.benchmarks : [];++ for (const b of list) {++ if (!rowsByBenchmark.has(b.benchmark_id)) rowsByBenchmark.set(b.benchmark_id, { title: b.title, cells: new Map() });++ rowsByBenchmark.get(b.benchmark_id).cells.set(model, b);++ }++ cards.push({++ model, runs: d ? d.runs : 0, cost: d ? d.total_cost_usd : null, last: d ? d.last_run : null,++ ratingCount: l && Array.isArray(l.ratings) ? l.ratings.length : 0,++ });++ });++ const rows = [...rowsByBenchmark.entries()].map(([id, x]) => {++ const row = { benchmark: `<a href="${aiPath(id)}">${esc(x.title)}</a>` };++ for (const model of models) {++ const b = x.cells.get(model);++ row[model] = b ? `<b>${scoreCell(b.score)}</b> <small>${esc(usd(b.cost_usd))}/run · ${esc(msText(b.ms))} · ${exact(b.runs)} runs</small>` : '—';++ }++ return row;++ });++ const columns = [{ key: 'benchmark', label: 'Benchmark', primary: true, html: true }, ...models.map((m) => ({ key: m, label: m, html: true }))];++ const form = `<form class="rp-ai-compare-form" method="get" action="/ai/compare" role="search" aria-label="Compare models">++<label for="rp-compare-models">Two or three model ids, comma separated</label>++<input id="rp-compare-models" name="models" value="${esc(models.join(', '))}" placeholder="gpt-5, claude-opus-4" autocomplete="off">++<button class="rp-btn" type="submit">Compare</button></form>`;++ const cardHtml = cards.length ? kit.stats(cards.map((c) => [`${c.runs} · ${c.cost == null ? '—' : usd(c.cost)} · ${c.last ? day(c.last) : 'no runs'}`, c.model]), { label: 'Models in this comparison' }) : '';++ const body = kit.head({ title: 'Compare AI models', sub: 'The same benchmarks side by side: score, cost per run, speed and the last run', back: { href: '/ai', label: 'AI' }, sticky: false })++ + benchState(...details, ...legacy)++ + form++ + (errors.length ? kit.notice({ kind: 'warn', text: errors.join(' ') }) : '')++ + (models.length < 2 ? kit.empty({ icon: 'trophy', title: 'Pick two or three models', text: 'Put their Bench ids in the box above; the comparison is built from their published runs.' }) : '')++ + (rows.length ? kit.section({ title: 'Side by side', body: kit.table({ caption: 'Models side by side', columns, rows }) }) : (models.length >= 2 ? kit.empty({ icon: 'trophy', title: 'No shared benchmarks yet', text: 'These models have no verified run on a public benchmark in common.' }) : ''))++ + (cardHtml ? `<p class="rp-fine">${esc('Each card: verified runs · total run cost · last run, then the model.')}</p>${cardHtml}` : '')++ + `<p class="rp-fine">Cost and speed are the medians of each model's runs on that benchmark; a score is the median of its run scores. <a href="/ai/method">How the runs are judged</a>.</p>`;++ return layout({ title: 'Compare AI models', description: 'Compare two or three AI models on the same OpenVibe benchmarks: score, cost and speed.', body, active: '/ai', path: '/ai/compare', agent: true });++}++++function aiMethod({ overview, kinds }) {++ const o = readData(overview) || {};++ const list = kinds && kinds.data && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];++ const byJudge = new Map();++ for (const k of list) {++ if (!byJudge.has(k.judge)) byJudge.set(k.judge, []);++ byJudge.get(k.judge).push(k);++ }++ const judgeRows = [...byJudge.entries()].map(([judge, ks]) => ({++ how: `<b>${esc(JUDGE_SHORT[judge] || judge)}</b>`, judge: esc(judge), jobs: esc(ks.map((k) => k.name || k.id).join(', ')),++ }));++ const body = kit.head({ title: 'How the AI results are judged', sub: 'The method behind every score, price and speed on these pages', back: { href: '/ai', label: 'AI' }, sticky: false })++ + benchState(overview, kinds)++ + `<section class="rp-ai-method"><h2>What a score means</h2>++<p>A score is a run’s result on one benchmark, from 0 to 100. Bench’s frozen benchmark version names its tasks, its judges and how the task scores combine; the default is the mean of the task scores. A model’s score on a benchmark is the median of its <b>verified</b> runs, so one bad run does not decide the page. The best and worst columns are the runs themselves, not a second estimate.</p>++<h2>Quality, price and speed</h2>++<p><b>Quality</b> is that score. <b>Price</b> is the median cost of a run in US dollars, as the service that ran it published it; <b>cost per solved</b> is the run’s cost divided by its success rate (score ÷ 100), so it is the price of one solved task. <b>Cost per solved</b> is the same figure, named plainly. <b>Speed</b> is the median wall-clock time of a run. The quality-against-cost chart marks the frontier: the models nothing beats on both.</p>++<h2>How runs are judged</h2>++<p>Each task kind has one main judge. A benchmark may also name more judges in its definition; the exact list is in its public definition at <code>/api/v1/benchmarks/:id</code>.</p>++${judgeRows.length ? kit.table({ caption: 'Judges in use', columns: [{ key: 'how', label: 'How', html: true }, { key: 'judge', label: 'Bench judge' }, { key: 'jobs', label: 'Task kinds' }], rows: judgeRows }) : `<p class="rp-fine">${esc('Bench did not answer its kind list; the judge rules above are its published contract.')}</p>`}++<h2>Verified and submitted</h2>++<p>A <b>verified</b> run was executed by a service (openvibe.codes, Run, a Node worker); a <b>submitted</b> run was posted with a person’s token. The rankings read verified runs only, and a submitted run is labelled wherever the latest runs are listed.</p>++<h2>How often, and what is missing</h2>++<p>${o.runs == null ? 'Bench did not answer the public counts.' : `${exact(o.benchmarks)} public ${o.benchmarks === 1 ? 'benchmark' : 'benchmarks'}, ${exact(o.tasks)} ${o.tasks === 1 ? 'task' : 'tasks'}, ${exact(o.runs)} ${o.runs === 1 ? 'run' : 'runs'} from ${exact(o.models)} ${o.models === 1 ? 'model' : 'models'}; the newest run was ${o.last_run ? esc(dayTime(o.last_run)) : 'never published'}.`}</p>++<p class="rp-fine">A model with no verified run has no score here; a benchmark with no run says so. The public reads are Bench’s API at <a href="${BENCH}/docs">bench.openvibe.services/docs</a>; the open data is <a href="${BENCH}/api/v1/datasets/benchmarks.jsonl">benchmarks.jsonl</a> and the per-kind datasets. <a href="/ai/feed.xml">New results (RSS)</a> · <a href="/api/v1/ai/results">the same as JSON</a>.</p>`;++ return layout({ title: 'How the AI results are judged', description: 'The method behind OpenVibe.Bench’s scores, costs and speeds, and how to read them.', body, active: '/ai', path: '/ai/method', agent: true });}// ── The AI Exchange (plan G-P9): Billing's settlements, summed by day, model and access ────────────────────────@@@ -868,4 +1192,4 @@ function notFound() {return layout({ title: 'Not found', robots: 'noindex, follow', body: `<header class="rp-page-h"><h1>Nothing here</h1><p class="rp-lede">That page does not exist, or its source has nothing for it. <a href="/">Back to Report</a>.</p></header>` });}--module.exports = { home, stats, rundown, sessionReport, sharingChip, topic, episode, speaker, recap, channel, you, youSignedOut, developers, aiIndex, aiBoard, aiModel, aiExchange, staffOnly, usdNano, paretoRows, paretoChart, notFound, layout, setRelease, configure };++module.exports = { home, stats, rundown, sessionReport, sharingChip, topic, episode, speaker, recap, channel, you, youSignedOut, developers, aiIndex, aiBoard, aiBenchmark, aiModel, aiCompare, aiMethod, aiExchange, staffOnly, usdNano, paretoRows, paretoChart, notFound, layout, setRelease, configure };@@@ -40,8 +40,8 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')await check('/ai leads with the leaders: the job, its leader, the rating and the games', async () => {const r = await t.get('/ai');assert.strictEqual(r.status, 200);-- for (const s of ['<caption>Leaders</caption>', 'href="/ai/design.page">Page <design></a>', 'openai/gpt-4o', '1043 ± 38', 'data-label="Games">8</td>', 'job rated', 'provisional']) assert.ok(r.text.includes(s), s);-- assert.ok(!r.text.includes('No model is rated yet'), 'no empty notice while a board has a leader');++ for (const s of ['Task kinds and their leaders', 'href="/ai/design.page">Page <design></a>', 'openai/gpt-4o', '1043 ± 38', 'data-label="Games">8</td>', 'job rated', 'provisional']) assert.ok(r.text.includes(s), s);++ assert.ok(!r.text.includes('No model is rated on a task kind yet'), 'no empty notice while a board has a leader');assert.ok(!r.text.includes('Page <design>'));});@@@ -107,8 +107,8 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')});try {const r = await fresh.get('/ai');-- assert.ok(r.text.includes('No model is rated yet') && r.text.includes('Jobs being rated'), 'one notice, one list');-- assert.ok(!r.text.includes('<caption>Leaders</caption>') && !r.text.includes('jobs tracked'), 'no empty leaders table and no lone number');++ assert.ok(r.text.includes('No model is rated on a task kind yet') && r.text.includes('Jobs being rated'), 'one notice, one list');++ assert.ok(!r.text.includes('Task kinds and their leaders') && !r.text.includes('jobs tracked'), 'no empty leaders table and no lone number');const b = await fresh.get('/ai/design.page');assert.ok(b.text.includes('No votes yet') && b.text.includes('href="https://openvibe.design/"') && b.text.includes('Vote on Design'), 'a blind-vote job sends people to vote');assert.ok(!b.text.includes('class="ovk-tabs'), 'an empty board has no tabs');@@@ -130,7 +130,7 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')});try {const r = await unplayed.get('/ai');-- assert.ok(r.text.includes('No model is rated yet') && !r.text.includes('<caption>Leaders</caption>'), 'no leaders table');++ assert.ok(r.text.includes('No model is rated on a task kind yet') && !r.text.includes('Task kinds and their leaders'), 'no leaders table');assert.ok(!r.text.includes('gpt-5-nano-2025-08-07'), 'the unjudged model is named nowhere as a leader');const b = await unplayed.get('/ai/design.page');assert.ok(b.text.includes('No votes yet') && b.text.includes('Vote on Design') && !b.text.includes('class="ovk-tabs'), 'the board asks for votes');nnew file mode 100644@@@ -0,0 +1,161 @@++'use strict';++++/**++ * The AI section (owner, 2026-11-08): Bench's pages moved into Report. The overview, one task kind, one benchmark,++ * one model, the comparison, the method page and the new-results feeds read only Bench's public answers.++ */++const assert = require('assert');++const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot');++++const BMK = 'bmk_01M65KA3H4Y4ZENBYYZQA8SRNN';++const RUN_A = 'brn_01M65KA3H4Y4ZENBYYZQA8SRN1';++const RUN_B = 'brn_01M65KA3H4Y4ZENBYYZQA8SRN2';++++const overview = {++ kinds: 2, benchmarks: 1, tasks: 2, runs: 3, models: 2, last_run: '2026-11-07T10:00:00.000Z',++ list: [{ id: BMK, title: 'Playable games arena', kind: 'design.page', description: 'A test benchmark.', latest_version: 1, runs: 3, last_run: '2026-11-07T10:00:00.000Z' }],++ recent: [++ { id: RUN_B, benchmark_id: BMK, model: 'model-b', score: 70, cost_usd: 0.04, ms: 2000, provenance: 'verified', created_at: '2026-11-07T10:00:00.000Z' },++ { id: RUN_A, benchmark_id: BMK, model: 'model-a', score: 90, cost_usd: 0.02, ms: 1000, provenance: 'verified', created_at: '2026-11-06T10:00:00.000Z' },++ ],++};++++const runs = {++ count: 2, offset: 0, last_run: '2026-11-07T10:00:00.000Z',++ runs: [++ { id: RUN_B, benchmark_id: BMK, model: 'model-b', score: 70, cost_usd: 0.04, ms: 2000, tokens: 20, provenance: 'verified', created_at: '2026-11-07T10:00:00.000Z' },++ { id: RUN_A, benchmark_id: BMK, model: 'model-a', score: 90, cost_usd: 0.02, ms: 1000, tokens: 10, provenance: 'verified', created_at: '2026-11-06T10:00:00.000Z' },++ ],++};++++const rankings = { rankings: { benchmark: BMK, metric: 'score', entries: [++ { model: 'model-a', runs: 2, score: 85, cost_usd: 0.02, ms: 1000, cost_per_solved: 0.024, min_score: 80, max_score: 90, last_run: '2026-11-06T10:00:00.000Z' },++ { model: 'model-b', runs: 1, score: 70, cost_usd: 0.04, ms: 2000, cost_per_solved: 0.057, min_score: 70, max_score: 70, last_run: '2026-11-07T10:00:00.000Z' },++] } };++++const benchmark = {++ benchmark: { id: BMK, title: 'Playable games arena', kind: 'design.page', description: 'A test benchmark.' },++ version: { version: 1, created_at: '2026-10-30T05:07:52.000Z' },++ definition: { tasks: [{ id: 't1', prompt: 'Build one screen.' }, { id: 't2', prompt: 'Build another.' }] },++ task_meta: [],++};++++const modelDetail = {++ model: 'model-a', runs: 2,++ benchmarks: [{ benchmark_id: BMK, title: 'Playable games arena', kind: 'design.page', runs: 2, score: 85, best: 90, worst: 80, cost_usd: 0.02, ms: 1000, last_run: '2026-11-06T10:00:00.000Z' }],++ tasks: [++ { benchmark_id: BMK, benchmark_title: 'Playable games arena', task_id: 't1', runs: 2, score: 90, best: 90, worst: 90, last_run: '2026-11-06T10:00:00.000Z' },++ { benchmark_id: BMK, benchmark_title: 'Playable games arena', task_id: 't2', runs: 2, score: 80, best: 80, worst: 80, last_run: '2026-11-06T10:00:00.000Z' },++ ],++ best_task: { task_id: 't1', score: 90 }, worst_task: { task_id: 't2', score: 80 },++ total_cost_usd: 0.04, total_ms: 2000, total_tokens: 20, last_run: '2026-11-06T10:00:00.000Z',++ recent_runs: [++ ...runs.runs.filter((r) => r.model === 'model-a'),++ { id: 'brn_01M65KA3H4Y4ZENBYYZQA8SRN3', benchmark_id: BMK, model: 'model-a', score: 80, cost_usd: 0.02, ms: 1200, tokens: 10, provenance: 'verified', created_at: '2026-11-05T10:00:00.000Z' },++ ],++};++++const legacy = { model: 'model-a', ratings: [{ kind_id: 'design.page', elo: 1043, ci_lower: 1005, ci_upper: 1081, games: 8, win_rate: 0.75 }], entries: [], total_cost_usd: 0.04 };++++(async () => {++ const source = stubFetch();++ const fetchImpl = async (url, opts) => {++ const u = new URL(String(url));++ if (u.port !== '5230') return source(url, opts);++ const p = u.pathname;++ if (p === '/api/v1/kinds') return jsonResponse({ count: 1, kinds: [{ id: 'design.page', room: 'design', name: 'Design page arena', judge: 'human_pairwise' }] });++ if (p === '/api/v1/boards/design.page') {++ const type = u.searchParams.get('type');++ const rows = type === 'elo'++ ? [{ subject: 'model-a', elo: 1043, ci_lower: 1005, ci_upper: 1081, games: 8, win_rate: 0.75, median_cost_usd: 0.02, median_ms: 1000, provisional: true }]++ : type === 'pareto' ? [{ subject: 'model-a', quality: 0.8 }] : [{ subject: 'model-a', entries: 2, solved: 2, total_cost_usd: 0.04, cost_per_solved: 0.02, median_ms: 1000 }];++ return jsonResponse({ board: { kind_id: 'design.page', type, rows } });++ }++ if (p === '/api/v1/overview') return jsonResponse(overview);++ if (p === '/api/v1/benchmarks') return jsonResponse({ count: 1, benchmarks: overview.list });++ if (p === `/api/v1/benchmarks/${BMK}`) return jsonResponse(benchmark);++ if (p === '/api/v1/rankings') return jsonResponse(rankings);++ if (p === '/api/v1/runs') return jsonResponse(runs);++ if (p === '/api/v1/models/model-a') return jsonResponse(legacy);++ if (p === '/api/v1/models/model-a/summary') return jsonResponse(modelDetail);++ if (p === '/api/v1/models/model-b/summary') return jsonResponse({ ...modelDetail, model: 'model-b', runs: 1, total_cost_usd: 0.04, recent_runs: runs.runs.filter((r) => r.model === 'model-b') });++ return jsonResponse({ error: 'no stub for ' + p }, 404);++ };++ const t = await boot({ fetchImpl, env: { OV_BENCH_INTERNAL_URL: 'http://127.0.0.1:5230' } });++ try {++ await check('the AI overview leads with the real leader and states the last run', async () => {++ const r = await t.get('/ai');++ assert.strictEqual(r.status, 200);++ for (const s of ['Task kinds and their leaders', 'Design page arena', 'model-a', '1043 ± 38', '$0.02', '1 s', 'Playable games arena', 'Nov 7', 'AI in numbers']) assert.ok(r.text.includes(s), s);++ assert.ok(r.text.includes('href="/ai/method"') && r.text.includes('href="/ai/compare"') && r.text.includes('/ai/feed.xml'));++ });++++ await check('one task kind shows quality, price and speed side by side and its latest runs', async () => {++ const r = await t.get('/ai/design.page');++ assert.strictEqual(r.status, 200);++ for (const s of ['Quality, price and speed side by side', 'Cost/run', 'Speed/run', '$0.02', '1 s', 'Latest runs', 'model-a', 'Nov 7']) assert.ok(r.text.includes(s), s);++ });++++ await check('one benchmark page has a ranking, an SVG and the latest runs', async () => {++ const r = await t.get(`/ai/${BMK}`);++ assert.strictEqual(r.status, 200);++ for (const s of ['Playable games arena', 'Score against cost per run', 'ovk-chart--scatter', 'model-a', '85', '$0.02', 'Cost per solved', 'Latest runs', 'version 1']) assert.ok(r.text.includes(s), s);++ });++++ await check('one model page has its history, best and worst tasks and cost', async () => {++ const r = await t.get('/ai/models/model-a');++ assert.strictEqual(r.status, 200);++ for (const s of ['Score over time', 'Best and worst tasks', 't1', 't2', 'spent on runs', '$0.04', 'Published', 'Blind-board ratings', 'Playable games arena']) assert.ok(r.text.includes(s), s);++ assert.ok(!r.text.includes('<span title='), 'a stat is escaped HTML, not markup');++ });++++ await check('compare puts two models side by side, and refuses one', async () => {++ const two = await t.get('/ai/compare?models=model-a,model-b');++ assert.strictEqual(two.status, 200);++ assert.ok(two.text.includes('Playable games arena') && two.text.includes('model-a') && two.text.includes('model-b'), 'both models');++ const one = await t.get('/ai/compare?models=model-a');++ assert.strictEqual(one.status, 200);++ assert.ok(one.text.includes('Pick at least two model ids'), 'the refusal');++ });++++ await check('an unknown benchmark id is a 404 once Bench has answered its list', async () => {++ assert.strictEqual((await t.get('/ai/bmk_01M65KA3H4Y4ZENBYYZQA8SRN9')).status, 404);++ });++++ await check('the method page explains judging, verification and cost', async () => {++ const r = await t.get('/ai/method');++ assert.strictEqual(r.status, 200);++ for (const s of ['What a score means', 'How runs are judged', 'Verified and submitted', 'Cost per solved', 'blind votes', 'benchmarks.jsonl']) assert.ok(r.text.includes(s), s);++ });++++ await check('new results are RSS and JSON, from the same runs', async () => {++ const rss = await t.get('/ai/feed.xml');++ assert.strictEqual(rss.status, 200);++ assert.match(rss.headers.get('content-type') || '', /application\/rss\+xml/);++ assert.ok(rss.text.includes('<title>model-a on Playable games arena</title>') && rss.text.includes(RUN_A), 'the run is in the feed');++ const json = await t.get('/api/v1/ai/results');++ assert.strictEqual(json.status, 200);++ const body = json.json();++ assert.strictEqual(body.count, 2);++ assert.strictEqual(body.results[0].id, RUN_B);++ assert.strictEqual(body.results[0].benchmark, 'Playable games arena');++ assert.ok(body.results[0].url.endsWith('/ai/' + BMK));++ });++++ await check('the AI pages have Markdown twins, and the benchmark twin names its page', async () => {++ for (const [path, expected] of [[`/ai/${BMK}.md`, 'Playable games arena'], ['/ai/method.md', 'How the AI results are judged'], ['/ai/models/model-a.md', 'model-a']]) {++ const r = await t.get(path);++ assert.strictEqual(r.status, 200, path);++ assert.match(r.headers.get('content-type') || '', /^text\/markdown/, path);++ assert.ok(r.text.includes(expected), `${path}: ${expected}`);++ }++ });++++ await check('the sitemap lists the AI pages and the benchmark', async () => {++ const r = await t.get('/sitemap.xml');++ assert.strictEqual(r.status, 200);++ for (const s of ['/ai</loc>', '/ai/method', '/ai/compare', `/ai/${BMK}`, '/ai/design.page', '/ai/models/model-a']) assert.ok(r.text.includes(s), s);++ });++ } finally { await t.close(); }++ done();++})().catch((e) => { console.error(e); process.exit(1); });@@@ -43,8 +43,15 @@ const { check, done, jsonResponse } = require('./helpers/boot');assert.strictEqual(await bench.board('../bad'), null);assert.strictEqual(await bench.board('design.page', 'wrong'), null);assert.strictEqual(await bench.modelSummary('bad?name'), null);++ assert.strictEqual(await bench.benchmark('../bad'), null);++ assert.strictEqual(await bench.rankings('not-a-benchmark'), null);++ assert.strictEqual(await bench.runs({ benchmark: '../bad' }), null);++ assert.strictEqual(await bench.modelDetail('bad?name'), null);assert.strictEqual(calls, 2);assert.strictEqual(bench.isModelName('openai/gpt-4o'), true);++ assert.strictEqual(bench.isBenchmarkId('bmk_01M65KA3H4Y4ZENBYYZQA8SRNN'), true);++ assert.strictEqual(bench.normaliseLimit(0), 1);++ assert.strictEqual(bench.normaliseLimit(1000), 100);});done();
@@ -3,6 +3,9 @@
@@ -3,6 +3,9 @@
What changed in OpenVibe.Report, newest first. Each site also publishes its patch notes at /updates on
What changed in OpenVibe.Report, newest first. Each site also publishes its patch notes at /updates on
openvibe.network.
openvibe.network.
+## Unreleased
+
+- **The AI section (2026-11-08): OpenVibe.Bench's pages moved here.** `/ai` is the overview: every published benchmark with its leader, runs and last run, every task kind with quality, price and speed, and a fold for the jobs still waiting. `/ai/<benchmark>` has verified rankings, score against cost in SVG and the latest runs; `/ai/<kind>` keeps the Elo/Pareto/cost boards and adds the verified runs on the kind's benchmarks. `/ai/models/<model>` has runs over time, best and worst tasks, cost and recent runs; `/ai/compare` puts two or three models side by side; `/ai/method` says how a score is made and how runs are judged. `/ai/feed.xml` and `GET /api/v1/ai/results` publish new runs. Every number is Bench's, read through its public API with its 60-second cache and last-good copy; Bench's old HTML pages 301 here.
## 0.1.0 — 2026-10-28
## 0.1.0 — 2026-10-28
The service skeleton: a stateless front for the network's open numbers, The Rundown and every stream's after-show
The service skeleton: a stateless front for the network's open numbers, The Rundown and every stream's after-show
@@ -36,7 +36,7 @@ OpenVibe.Live in both modes.
@@ -36,7 +36,7 @@ OpenVibe.Live in both modes.
| **Report's tables** | `report_metric_days`, `report_metric_windows`, `report_streams` | the same routes while `REPORT_SOURCE=report` |
| **Report's tables** | `report_metric_days`, `report_metric_windows`, `report_streams` | the same routes while `REPORT_SOURCE=report` |
| **OpenVibe.Live** — The Rundown | `GET /api/rundown/`, `/api/rundown/topics`, `/api/rundown/t/:slug`, `/api/rundown/e/:streamId`, `/api/rundown/s/:username` | `/rundown`, `/rundown/t/:slug`, `/rundown/e/:streamId`, `/rundown/s/:username` |
| **OpenVibe.Live** — The Rundown | `GET /api/rundown/`, `/api/rundown/topics`, `/api/rundown/t/:slug`, `/api/rundown/e/:streamId`, `/api/rundown/s/:username` | `/rundown`, `/rundown/t/:slug`, `/rundown/e/:streamId`, `/rundown/s/:username` |
| **OpenVibe.Live** — after-show reports | `GET /api/recap/:streamId`, `GET /api/recap/channel/:username` | `/after-show/:streamId`, `/c/:username` |
| **OpenVibe.Live** — after-show reports | `GET /api/recap/:streamId`, `GET /api/recap/channel/:username` | `/after-show/:streamId`, `/c/:username` |
-| **OpenVibe.Bench** | `GET /api/v1/kinds`, `/api/v1/boards/:kind?type=elo\|pareto\|cost`, `/api/v1/models/:model` | `/ai`, `/ai/:kind`, `/ai/models/:model` |
+| **OpenVibe.Bench** | `GET /api/v1/kinds`, `/api/v1/boards/:kind?type=elo\|pareto\|cost`, `/api/v1/benchmarks`, `/api/v1/rankings`, `/api/v1/runs`, `/api/v1/overview`, `/api/v1/models/:model`, `/api/v1/models/:model/summary` | `/ai`, `/ai/:kind`, `/ai/<benchmark>`, `/ai/models/:model`, `/ai/compare`, `/ai/method`, `/ai/feed.xml`, `GET /api/v1/ai/results` |
| **OpenVibe.Billing** | `GET /api/v1/stats`, `GET /api/v1/transactions?subject=…` | the revenue read for a channel; network money counts |
| **OpenVibe.Billing** | `GET /api/v1/stats`, `GET /api/v1/transactions?subject=…` | the revenue read for a channel; network money counts |
| **OpenVibe.Billing** — settlements, through Events | `billing.credit.settled@1` on `POST /internal/events` (only with `REPORT_AI_EXCHANGE=1`) | the AI Exchange aggregates and the staff page/API |
| **OpenVibe.Billing** — settlements, through Events | `billing.credit.settled@1` on `POST /internal/events` (only with `REPORT_AI_EXCHANGE=1`) | the AI Exchange aggregates and the staff page/API |
@@ -64,6 +64,7 @@ never invents a zero and never throws a source error at the reader.
@@ -64,6 +64,7 @@ never invents a zero and never throws a source error at the reader.
- **Money** — OpenVibe.Billing. Report reads it with `billing.stats.read` and `billing.balance.read`; it holds no
- **Money** — OpenVibe.Billing. Report reads it with `billing.stats.read` and `billing.balance.read`; it holds no
ledger and computes no price.
ledger and computes no price.
- **The Rundown and the after-show reports** — OpenVibe.Live. Report quotes them, including their counts and text.
- **The Rundown and the after-show reports** — OpenVibe.Live. Report quotes them, including their counts and text.
+- **The benchmark results** — OpenVibe.Bench. Report renders its rows and never derives a score, cost or time; Bench stayed the authority when its public pages moved here (2026-11-08) and its old HTML pages answer 301 to `/ai` counterparts.
- **Identity, sessions and writes** — Report has no sign-in, no cookies and no public write route. `POST
- **Identity, sessions and writes** — Report has no sign-in, no cookies and no public write route. `POST
/internal/events` is loopback-only and HMAC-verified.
/internal/events` is loopback-only and HMAC-verified.
- **The Shared Frame** — `openvibe-shared` (navbar, footer, theme-loader, static files), served from this site's own
- **The Shared Frame** — `openvibe-shared` (navbar, footer, theme-loader, static files), served from this site's own
@@ -83,9 +84,13 @@ All server-rendered in `server/views/index.js`; every dynamic value goes through
@@ -83,9 +84,13 @@ All server-rendered in `server/views/index.js`; every dynamic value goes through
| `/rundown/s/:username` | one streamer's episodes and topics; canonical is Live's `/rundown/:username` |
| `/rundown/s/:username` | one streamer's episodes and topics; canonical is Live's `/rundown/:username` |
| `/after-show/:streamId` | one after-show report |
| `/after-show/:streamId` | one after-show report |
| `/c/:username` | one channel's latest after-show reports |
| `/c/:username` | one channel's latest after-show reports |
-| `/ai` | benchmark kinds grouped by room, with leaders and game counts |
-| `/ai/:kind` | Elo, Pareto quality against cost per solved, and cost per solved tables |
-| `/ai/models/:model` | a model's ratings and entries across kinds |
+| `/ai` | the AI overview: every published benchmark with its leader, runs and last run; every task kind with its leader, quality, cost and speed; a link to the method and compare pages |
+| `/ai/:kind` | one task kind: Elo rank, quality against cost, cost per solved, the verified runs on its benchmarks, and the latest runs |
+| `/ai/<benchmark>` | one public Bench benchmark: verified rankings with score, cost, speed and last run, a score-against-cost SVG, and the latest runs |
+| `/ai/models/:model` | a model's runs over time, its best and worst tasks, cost, per-benchmark record and recent runs |
+| `/ai/compare` | two or three models on the same benchmarks, side by side |
+| `/ai/method` | how a score is made, how runs are judged, verified vs submitted, and what the counts mean |
+| `/ai/feed.xml` | new public runs as RSS; the same rows are `GET /api/v1/ai/results` |
| `/ai/exchange` | **staff only**, with `REPORT_AI_EXCHANGE=1`: Billing's settlements by day, model and access, a daily chart and the exact amounts. Needs a signed-in person whose staff role holds `staff.site.view` |
| `/ai/exchange` | **staff only**, with `REPORT_AI_EXCHANGE=1`: Billing's settlements by day, model and access, a daily chart and the exact amounts. Needs a signed-in person whose staff role holds `staff.site.view` |
A bad path parameter (slug, stream id, username) or a source that answered nothing renders `views.notFound()` with
A bad path parameter (slug, stream id, username) or a source that answered nothing renders `views.notFound()` with
@@ -104,6 +109,7 @@ GET /api/v1/rundown/topics
@@ -104,6 +109,7 @@ GET /api/v1/rundown/topics
GET /api/v1/rundown/t/:slug
GET /api/v1/rundown/t/:slug
GET /api/v1/after-show/:streamId
GET /api/v1/after-show/:streamId
GET /api/v1/c/:username
GET /api/v1/c/:username
+GET /api/v1/ai/results the new AI runs, newest first
```
```
A bad parameter or a missing source answer is a 404 `application/problem+json` body built with
A bad parameter or a missing source answer is a 404 `application/problem+json` body built with
@@ -19,7 +19,7 @@
@@ -19,7 +19,7 @@
"pg v8.23.1",
"pg v8.23.1",
"pglite v0.5.13 (dev)"
"pglite v0.5.13 (dev)"
],
],
- "summary": "OpenVibe.Report keeps its own PostgreSQL analytics store (migrations/), ingests Network's live.stream.ended and frame beacon plus its open-stats answer idempotently (report_inbox), computes the same site-wide, per-site and per-channel numbers by day, quotes Billing's revenue through Billing's read API, and runs a shadow job that records every difference from Network in report_shadow_diffs with GET /internal/shadow on loopback. REPORT_SOURCE=network (default) keeps the pages on today's Network numbers; REPORT_SOURCE=report answers them from Report's tables. The Rundown and after-show pages still read OpenVibe.Live.",
+ "summary": "OpenVibe.Report keeps its own PostgreSQL analytics store (migrations/), ingests Network's live.stream.ended and frame beacon plus its open-stats answer idempotently (report_inbox), computes the same site-wide, per-site and per-channel numbers by day, quotes Billing's revenue through Billing's read API, and runs a shadow job that records every difference from Network in report_shadow_diffs with GET /internal/shadow on loopback. REPORT_SOURCE=network (default) keeps the pages on today's Network numbers; REPORT_SOURCE=report answers them from Report's tables. The Rundown and after-show pages still read OpenVibe.Live. The AI section reads OpenVibe.Bench's public API: /ai, /ai/<benchmark>, /ai/models/<model>, /ai/compare, /ai/method, /ai/feed.xml and /api/v1/ai/results; Bench's old HTML pages 301 here.",
"works": [
"works": [
"Database: server/db.js (openvibe-sdk/db, migrations at boot; PostgreSQL through PgBouncer, PGlite in development) and test/helpers/db.js (createTestDb)",
"Database: server/db.js (openvibe-sdk/db, migrations at boot; PostgreSQL through PgBouncer, PGlite in development) and test/helpers/db.js (createTestDb)",
"Tables: report_inbox, report_events, report_streams, report_frame_days, report_metric_days, report_metric_windows, report_metric_json, report_dimensions, report_shadow_runs, report_shadow_diffs",
"Tables: report_inbox, report_events, report_streams, report_frame_days, report_metric_days, report_metric_windows, report_metric_json, report_dimensions, report_shadow_runs, report_shadow_diffs",
@@ -61,7 +61,9 @@
@@ -61,7 +61,9 @@
"JSON API",
"JSON API",
"crawl artifacts",
"crawl artifacts",
"readiness",
"readiness",
- "deploy files"
+ "deploy files",
+ "AI section from OpenVibe.Bench",
+ "AI results RSS and JSON"
],
],
"deployment": "planned: openvibe-ovh, unit openvibe-report.service on 127.0.0.1:5080 (git checkout /opt/openvibe.report, env /etc/openvibe/report.env), public at https://openvibe.report; DATABASE_URL/DATABASE_DIRECT_URL from OpenVibe.Host roles/data add-service.sh report",
"deployment": "planned: openvibe-ovh, unit openvibe-report.service on 127.0.0.1:5080 (git checkout /opt/openvibe.report, env /etc/openvibe/report.env), public at https://openvibe.report; DATABASE_URL/DATABASE_DIRECT_URL from OpenVibe.Host roles/data add-service.sh report",
"updated": "2026-11-08"
"updated": "2026-11-08"
@@ -372,3 +372,13 @@ html, body { margin: 0; background: var(--bg-primary, #0a0f1c); color: var(--tex
@@ -372,3 +372,13 @@ html, body { margin: 0; background: var(--bg-primary, #0a0f1c); color: var(--tex
.rp-step-st { color: var(--text-secondary, #9aa4b2); font-size: .82rem; font-weight: 500; }
.rp-step-st { color: var(--text-secondary, #9aa4b2); font-size: .82rem; font-weight: 500; }
.rp-steps li.is-failed .rp-step-st { color: #f87171; }
.rp-steps li.is-failed .rp-step-st { color: #f87171; }
.rp-step-detail { color: var(--text-secondary, #9aa4b2); font-size: .9rem; }
.rp-step-detail { color: var(--text-secondary, #9aa4b2); font-size: .9rem; }
+
+/* AI (2026-11-08): the compare form and the method page. The charts and tables are the shared kit's own. */
+.rp-ai-compare-form { display: flex; flex-wrap: wrap; align-items: end; gap: 8px; margin: 18px 0 8px; padding: 14px 16px; border: 1px solid var(--border, #1f2d47); border-radius: 16px; background: var(--bg-card, #0f1729); }
+.rp-ai-compare-form label { flex: 1 0 100%; font-size: .8rem; font-weight: 700; color: var(--text-secondary, #96a7c2); }
+.rp-ai-compare-form input { flex: 1 1 240px; min-width: 0; min-height: 44px; padding: 0 12px; border-radius: 10px; border: 1px solid var(--border, #1f2d47); background: var(--bg-secondary, #101828); color: var(--text-primary, #e6edf7); font: inherit; }
+.rp-ai-compare-form input:focus-visible { outline: 2px solid var(--room-accent); outline-offset: 1px; }
+.rp-ai-compare-form .rp-btn { border: 0; cursor: pointer; font: inherit; }
+.rp-ai-method h2 { margin: 28px 0 8px; font-size: 1.15rem; }
+.rp-ai-method p { max-width: 820px; margin: 0 0 12px; line-height: 1.65; color: var(--text-secondary, #96a7c2); }
+.rp-ai-method code { overflow-wrap: anywhere; }
@@ -18,7 +18,13 @@ function createAgentPages({ publicPages, bench }) {
@@ -18,7 +18,13 @@ function createAgentPages({ publicPages, bench }) {
else if (path === '/design') page = publicPages.design();
else if (path === '/design') page = publicPages.design();
else if (path === '/stats') page = await publicPages.stats(normaliseDays(query && query.get('days')));
else if (path === '/stats') page = await publicPages.stats(normaliseDays(query && query.get('days')));
else if (path === '/ai') page = await publicPages.aiIndex();
else if (path === '/ai') page = await publicPages.aiIndex();
+ else if (path === '/ai/method') page = await publicPages.aiMethod();
+ else if (path === '/ai/compare') {
+ const models = String((query && query.get('models')) || '').split(',').map((x) => x.trim()).filter(Boolean);
+ page = await publicPages.aiCompare(models);
+ }
else if (model) page = await publicPages.aiModel(decodeURIComponent(model[1]));
else if (model) page = await publicPages.aiModel(decodeURIComponent(model[1]));
+ else if (kind && bench.isBenchmarkId(kind[1])) page = await publicPages.aiBenchmark(kind[1]);
else if (kind) page = await publicPages.aiBoard(kind[1], bench.normaliseTab(query && query.get('tab')));
else if (kind) page = await publicPages.aiBoard(kind[1], bench.normaliseTab(query && query.get('tab')));
return page ? { html: page.html } : null;
return page ? { html: page.html } : null;
}
}
@@ -287,7 +287,7 @@ async function createApp(opts = {}) {
@@ -287,7 +287,7 @@ async function createApp(opts = {}) {
legacyHeaders: false,
legacyHeaders: false,
handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }),
handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }),
}));
}));
- app.use('/api/v1', createApi({ network, live, analytics, aiUsage, aiExchange, sessionReports, config, now, log }));
+ app.use('/api/v1', createApi({ network, live, bench, analytics, aiUsage, aiExchange, sessionReports, config, now, log }));
app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));
app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));
// ── Crawl artifacts, then the pages ─────────────────────
// ── Crawl artifacts, then the pages ─────────────────────
@@ -1,29 +1,52 @@
@@ -1,29 +1,52 @@
'use strict';
'use strict';
/**
/**
- * OpenVibe.Bench's public reads (plan 2026-10-28 §4.12):
+ * OpenVibe.Bench's public reads (plan 2026-10-28 §4.12; the 2026-11-08 move adds the registry reads):
* GET /api/v1/kinds every benchmark kind
* GET /api/v1/kinds every benchmark kind
* GET /api/v1/boards/:kind?type=elo|pareto|cost
* GET /api/v1/boards/:kind?type=elo|pareto|cost
- * GET /api/v1/models/:model one model across every kind
+ * GET /api/v1/models/:model one model across every kind (the legacy boards)
+ * GET /api/v1/benchmarks every public benchmark, its runs and last run
+ * GET /api/v1/benchmarks/:id one benchmark and its definition
+ * GET /api/v1/rankings?benchmark=&metric= verified per-model medians
+ * GET /api/v1/runs?benchmark=&model=&kind= public runs newest first
+ * GET /api/v1/overview public counts and the newest runs
+ * GET /api/v1/models/:model/summary a model's runs, tasks, cost and last run
*
*
* Reads are public and held for sixty seconds in this process. A read that has answered once keeps its copy
* Reads are public and held for sixty seconds in this process. A read that has answered once keeps its copy
* for as long as Bench is down: the client answers the last good one with the moment it was read (`at`) and
* for as long as Bench is down: the client answers the last good one with the moment it was read (`at`) and
* `stale: true`, so a board page stays a page — never an error page. A read that has never answered is null
* `stale: true`, so a board page stays a page — never an error page. A read that has never answered is null
- * after one short log line. A kind id, board type or model name that cannot exist is refused before any request
- * is made.
+ * after one short log line. A kind id, benchmark id, board type or model name that cannot exist is refused
+ * before any request is made.
*/
*/
const { trim, requestJson, TIMEOUT_MS } = require('./source');
const { trim, requestJson, TIMEOUT_MS } = require('./source');
const TTL_MS = 60_000;
const TTL_MS = 60_000;
const KIND_RE = /^[a-z0-9][a-z0-9._-]{0,79}$/;
const KIND_RE = /^[a-z0-9][a-z0-9._-]{0,79}$/;
+const BENCHMARK_RE = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;
const MODEL_RE = /^[A-Za-z0-9][A-Za-z0-9._:+\/~\-]{0,119}$/;
const MODEL_RE = /^[A-Za-z0-9][A-Za-z0-9._:+\/~\-]{0,119}$/;
const BOARD_TYPES = Object.freeze(['elo', 'pareto', 'cost']);
const BOARD_TYPES = Object.freeze(['elo', 'pareto', 'cost']);
+const METRICS = Object.freeze(['score', 'value', 'speed', 'consistency', 'horizon']);
const isKindId = (value) => typeof value === 'string' && KIND_RE.test(value);
const isKindId = (value) => typeof value === 'string' && KIND_RE.test(value);
+/** A benchmark id as Bench mints it (bmk_ + a 26-character ULID), never a kind or a path. */
+const isBenchmarkId = (value) => typeof value === 'string' && BENCHMARK_RE.test(value);
+
/** A model as Bench pins it: one path segment (its own /models/:model route cannot serve more), 120 characters at most. */
/** A model as Bench pins it: one path segment (its own /models/:model route cannot serve more), 120 characters at most. */
const isModelName = (value) => typeof value === 'string' && MODEL_RE.test(value);
const isModelName = (value) => typeof value === 'string' && MODEL_RE.test(value);
+/** A ranking metric; anything else is the score, exactly as Bench's own rankings route defaults. */
+function normaliseMetric(value) {
+ const raw = Array.isArray(value) ? value[0] : value;
+ return METRICS.includes(raw) ? raw : 'score';
+}
+
+/** A public run limit: 1 to 100, the API's own cap; anything else is 20. */
+function normaliseLimit(value) {
+ const n = parseInt(Array.isArray(value) ? value[0] : value, 10);
+ return Number.isFinite(n) ? Math.min(100, Math.max(1, n)) : 20;
+}
+
/** ?tab=elo|pareto|cost; anything else is the Elo board, exactly as the page tabs read it. */
/** ?tab=elo|pareto|cost; anything else is the Elo board, exactly as the page tabs read it. */
function normaliseTab(value) {
function normaliseTab(value) {
const raw = Array.isArray(value) ? value[0] : value;
const raw = Array.isArray(value) ? value[0] : value;
@@ -41,6 +64,8 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = co
@@ -41,6 +64,8 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = co
// boards are a fixed small set; models are capped, the oldest inserted out first.
// boards are a fixed small set; models are capped, the oldest inserted out first.
const boards = new Map();
const boards = new Map();
const models = new Map();
const models = new Map();
+ const catalog = new Map();
+ const results = new Map();
function keep(store, key, data, cap) {
function keep(store, key, data, cap) {
store.set(key, data);
store.set(key, data);
@@ -86,7 +111,58 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = co
@@ -86,7 +111,58 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = co
return read(models, `model:${model}`, `/api/v1/models/${encodeURIComponent(model)}`, `bench model (${model})`, 200);
return read(models, `model:${model}`, `/api/v1/models/${encodeURIComponent(model)}`, `bench model (${model})`, 200);
}
}
- return { kinds, board, modelSummary, baseUrl, ttlMs: TTL_MS, isKindId, isModelName, normaliseTab };
+ /** Every public benchmark the registry lists, with its runs and last run. */
+ function benchmarks() {
+ return read(catalog, 'benchmarks', '/api/v1/benchmarks?limit=200', 'bench benchmarks', 100);
+ }
+
+ /** One public benchmark: { benchmark, version, definition, task_meta }. */
+ async function benchmark(id) {
+ if (!isBenchmarkId(id)) return null;
+ return read(catalog, `benchmark:${id}`, `/api/v1/benchmarks/${encodeURIComponent(id)}`, `bench benchmark (${id})`, 500);
+ }
+
+ /** Public counts and the newest runs: { kinds, benchmarks, tasks, runs, models, last_run, list, recent }. */
+ function overview() {
+ return read(catalog, 'overview', '/api/v1/overview', 'bench overview', 20);
+ }
+
+ /** One benchmark's verified medians: { rankings: { benchmark, metric, entries } }. */
+ async function rankings(benchmarkId, metric = 'score') {
+ if (!isBenchmarkId(benchmarkId) || !METRICS.includes(metric)) return null;
+ return read(results, `rankings:${benchmarkId}:${metric}`, `/api/v1/rankings?benchmark=${encodeURIComponent(benchmarkId)}&metric=${metric}`, `bench rankings (${benchmarkId}, ${metric})`, 500);
+ }
+
+ /**
+ * Public runs newest first. opts: { benchmark, model, kind, limit }. A kind is the task-kind filter the kind
+ * page reads; a benchmark and a model are the registry ids. A value that cannot exist is refused before any
+ * request, so a stray query never becomes a Bench call.
+ */
+ async function runs({ benchmark = null, model = null, kind = null, limit = 20 } = {}) {
+ if (benchmark != null && !isBenchmarkId(benchmark)) return null;
+ if (model != null && !isModelName(model)) return null;
+ if (kind != null && !isKindId(kind)) return null;
+ const params = new URLSearchParams();
+ if (benchmark) params.set('benchmark', benchmark);
+ if (model) params.set('model', model);
+ if (kind) params.set('kind', kind);
+ params.set('limit', String(normaliseLimit(limit)));
+ const query = params.toString();
+ return read(results, `runs:${query}`, `/api/v1/runs?${query}`, `bench runs (${query})`, 500);
+ }
+
+ /** A model's public record: { model, runs, benchmarks, tasks, best_task, worst_task, recent_runs, total_cost_usd, last_run }. */
+ async function modelDetail(model) {
+ if (!isModelName(model)) return null;
+ return read(models, `detail:${model}`, `/api/v1/models/${encodeURIComponent(model)}/summary`, `bench model detail (${model})`, 200);
+ }
+
+ return {
+ kinds, board, modelSummary, benchmarks, benchmark, overview, rankings, runs, modelDetail,
+ baseUrl, ttlMs: TTL_MS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,
+ };
}
}
-module.exports = { createBenchClient, TTL_MS, BOARD_TYPES, isKindId, isModelName, normaliseTab };
+module.exports = {
+ createBenchClient, TTL_MS, BOARD_TYPES, METRICS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,
+};
@@ -30,7 +30,7 @@ const sharing = require('../sharing');
@@ -30,7 +30,7 @@ const sharing = require('../sharing');
const CACHE = 'public, max-age=120';
const CACHE = 'public, max-age=120';
-function createApi({ network, live, analytics = null, aiUsage = null, aiExchange = null, sessionReports = null, config = {}, now = () => Date.now(), log = console }) {
+function createApi({ network, live, bench = null, analytics = null, aiUsage = null, aiExchange = null, sessionReports = null, config = {}, now = () => Date.now(), log = console }) {
const r = express.Router();
const r = express.Router();
const serviceAuth = createServiceAuth({ config, now });
const serviceAuth = createServiceAuth({ config, now });
@@ -166,6 +166,22 @@ function createApi({ network, live, analytics = null, aiUsage = null, aiExchange
@@ -166,6 +166,22 @@ function createApi({ network, live, analytics = null, aiUsage = null, aiExchange
});
});
});
});
+ // The new AI results, as JSON: the same public runs the /ai/feed.xml RSS carries. Only Bench's own rows; a
+ // result with no run is not in it.
+ r.get('/ai/results', async (req, res) => {
+ if (!bench) return problem(req, res, 503, 'report.no_bench', 'Bench is not configured in this process.');
+ const [runs, overview] = await Promise.all([bench.runs({ limit: 50 }), bench.overview()]);
+ const list = runs && runs.data && Array.isArray(runs.data.runs) ? runs.data.runs : null;
+ if (!list) return problem(req, res, 502, 'report.bench_unavailable', 'Bench did not answer with its runs.');
+ const titles = new Map((overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : []).map((b) => [b.id, b.title]));
+ const results = list.map((run) => ({
+ id: run.id, benchmark_id: run.benchmark_id, benchmark: titles.get(run.benchmark_id) || null,
+ model: run.model, score: run.score, cost_usd: run.cost_usd, ms: run.ms, provenance: run.provenance,
+ created_at: run.created_at, url: `${String(config.baseUrl || 'https://openvibe.report').replace(/\/+$/, '')}/ai/${encodeURIComponent(run.benchmark_id)}`,
+ }));
+ return res.json({ count: results.length, generated_at: new Date(now()).toISOString(), results });
+ });
+
// ── The AI meter (plan L40 and L41) ─────────────────────────────────────────────
// ── The AI meter (plan L40 and L41) ─────────────────────────────────────────────
//
//
// The public totals: the tokens, the cache's share, the own-key share and the spend by week. No person, no
// The public totals: the tokens, the cache's share, the own-key share and the spend by week. No person, no
@@ -4,7 +4,7 @@
@@ -4,7 +4,7 @@
* Crawl artifacts for openvibe.report, built with openvibe-shared/seo:
* Crawl artifacts for openvibe.report, built with openvibe-shared/seo:
*
*
* GET /robots.txt the public pages are crawlable; the API is not
* GET /robots.txt the public pages are crawlable; the API is not
- * GET /sitemap.xml /, /stats, /design, /rundown and every /rundown/t/<slug> from the Rundown's topics
+ * GET /sitemap.xml /, /stats, /design, /ai (and its benchmarks, models, method and compare), the Rundown's topics
* GET /llms.txt a markdown map of the site and its machine-readable answers
* GET /llms.txt a markdown map of the site and its machine-readable answers
*
*
* The sitemap lists only the pages this site owns; The Rundown pages carry rel=canonical to openvibe.live and are left
* The sitemap lists only the pages this site owns; The Rundown pages carry rel=canonical to openvibe.live and are left
@@ -46,8 +46,13 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
@@ -46,8 +46,13 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
});
});
r.get('/sitemap.xml', async (_req, res) => {
r.get('/sitemap.xml', async (_req, res) => {
- const [ai, reports] = await Promise.all([bench.kinds(), sessionReports ? sessionReports.listPublic({ limit: 500 }) : []]);
+ const [ai, overview, reports] = await Promise.all([
+ bench.kinds(), bench.overview(), sessionReports ? sessionReports.listPublic({ limit: 500 }) : [],
+ ]);
const kinds = (ai && Array.isArray(ai.data.kinds)) ? ai.data.kinds : [];
const kinds = (ai && Array.isArray(ai.data.kinds)) ? ai.data.kinds : [];
+ const o = overview && overview.data ? overview.data : {};
+ const benches = Array.isArray(o.list) ? o.list : [];
+ const models = Array.isArray(o.model_list) ? o.model_list.map((m) => m.model).filter(Boolean) : [...new Set((Array.isArray(o.recent) ? o.recent : []).map((r) => r && r.model).filter(Boolean))];
const lastmod = siteUpdated();
const lastmod = siteUpdated();
// Only the pages this site owns. /rundown and its topic, episode and speaker pages carry rel=canonical to
// Only the pages this site owns. /rundown and its topic, episode and speaker pages carry rel=canonical to
// OpenVibe.Live's Rundown, so listing them here would advertise a duplicate.
// OpenVibe.Live's Rundown, so listing them here would advertise a duplicate.
@@ -56,19 +61,28 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
@@ -56,19 +61,28 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
{ loc: abs('/stats'), changefreq: 'daily', priority: 0.9 },
{ loc: abs('/stats'), changefreq: 'daily', priority: 0.9 },
{ loc: abs('/design'), changefreq: 'weekly', priority: 0.6 },
{ loc: abs('/design'), changefreq: 'weekly', priority: 0.6 },
{ loc: abs('/ai'), changefreq: 'daily', priority: 0.9 },
{ loc: abs('/ai'), changefreq: 'daily', priority: 0.9 },
+ { loc: abs('/ai/method'), changefreq: 'weekly', priority: 0.6 },
+ { loc: abs('/ai/compare'), changefreq: 'weekly', priority: 0.6 },
...reports.map((rep) => ({ loc: abs('/s/' + encodeURIComponent(rep.id)), changefreq: 'daily', priority: 0.6 })),
...reports.map((rep) => ({ loc: abs('/s/' + encodeURIComponent(rep.id)), changefreq: 'daily', priority: 0.6 })),
...kinds.filter((k) => k && bench.isKindId(k.id)).map((k) => ({
...kinds.filter((k) => k && bench.isKindId(k.id)).map((k) => ({
loc: abs(`/ai/${encodeURIComponent(k.id)}`), changefreq: 'daily', priority: 0.7,
loc: abs(`/ai/${encodeURIComponent(k.id)}`), changefreq: 'daily', priority: 0.7,
})),
})),
- ].map((p) => ({ ...p, ...(lastmod ? { lastmod } : {}) }));
+ ...benches.filter((b) => b && bench.isBenchmarkId(b.id)).map((b) => ({
+ loc: abs(`/ai/${encodeURIComponent(b.id)}`), changefreq: 'daily', priority: 0.8,
+ ...(dayOf(b.last_run) ? { lastmod: dayOf(b.last_run) } : {}),
+ })),
+ ...models.filter((m) => bench.isModelName(m)).map((m) => ({
+ loc: abs(`/ai/models/${encodeURIComponent(m)}`), changefreq: 'daily', priority: 0.6,
+ })),
+ ].map((p) => ({ ...p, ...(lastmod && !p.lastmod ? { lastmod } : {}) }));
res.type('application/xml').set('Cache-Control', CACHE).send(seo.sitemapXml(pages));
res.type('application/xml').set('Cache-Control', CACHE).send(seo.sitemapXml(pages));
});
});
// The site as llms.txt describes it, shared by /llms.txt and /llms-full.txt (SEO-33).
// The site as llms.txt describes it, shared by /llms.txt and /llms-full.txt (SEO-33).
const llmsSite = () => ({
const llmsSite = () => ({
name: SITE_NAME,
name: SITE_NAME,
- summary: 'Every number on OpenVibe, honest: the network\'s open stats, The Rundown of what streamers talked about, and every stream\'s after-show report.',
- details: 'The numbers and the write-ups are read from OpenVibe.Network and OpenVibe.Live, cached in this process for a few minutes, and shown as the sources answer them. Nothing here is about a person.',
+ summary: 'Every number on OpenVibe, honest: the network\'s open stats, The Rundown of what streamers talked about, every stream\'s after-show report, and the AI results Bench publishes.',
+ details: 'The numbers and the write-ups are read from OpenVibe.Network, OpenVibe.Live and OpenVibe.Bench, cached in this process for a few minutes, and shown as the sources answer them. Nothing here is about a person.',
markdown: true,
markdown: true,
base: site,
base: site,
sections: [
sections: [
@@ -77,7 +91,9 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
@@ -77,7 +91,9 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
{ title: 'Open numbers', url: abs('/stats'), note: 'page views, accounts and traffic for every OpenVibe site' },
{ title: 'Open numbers', url: abs('/stats'), note: 'page views, accounts and traffic for every OpenVibe site' },
{ title: 'Design health', url: abs('/design'), note: 'how closely each site follows the shared design kit, read from its code; per site at /design/<site>' },
{ title: 'Design health', url: abs('/design'), note: 'how closely each site follows the shared design kit, read from its code; per site at /design/<site>' },
{ title: 'The Rundown', url: 'https://openvibe.live/rundown', note: 'what streamers talked about, topic by topic (OpenVibe.Live is its canonical home)' },
{ title: 'The Rundown', url: 'https://openvibe.live/rundown', note: 'what streamers talked about, topic by topic (OpenVibe.Live is its canonical home)' },
- { title: 'AI boards', url: abs('/ai'), note: 'Bench ratings, quality and cost across rooms' },
+ { title: 'AI', url: abs('/ai'), note: 'OpenVibe.Bench results: quality, price and speed for every task kind and benchmark' },
+ { title: 'How AI results are judged', url: abs('/ai/method'), note: 'scores, verified runs, cost per solved, speed and the frontier' },
+ { title: 'Compare AI models', url: abs('/ai/compare'), note: 'two or three models on the same benchmarks' },
] },
] },
// A JSON API is what an agent came for, not something to skip (AGT-5 review).
// A JSON API is what an agent came for, not something to skip (AGT-5 review).
{ title: 'API', links: [
{ title: 'API', links: [
@@ -85,6 +101,8 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
@@ -85,6 +101,8 @@ function createDiscoveryRoutes({ config, live, bench, sessionReports = null }) {
{ title: 'Design health (JSON)', url: abs('/api/v1/design-health'), note: 'every site; /api/v1/design-health/<site> for one' },
{ title: 'Design health (JSON)', url: abs('/api/v1/design-health'), note: 'every site; /api/v1/design-health/<site> for one' },
{ title: 'The Rundown (JSON)', url: abs('/api/v1/rundown') },
{ title: 'The Rundown (JSON)', url: abs('/api/v1/rundown') },
{ title: 'Topics (JSON)', url: abs('/api/v1/rundown/topics') },
{ title: 'Topics (JSON)', url: abs('/api/v1/rundown/topics') },
+ { title: 'New AI results (RSS)', url: abs('/ai/feed.xml') },
+ { title: 'New AI results (JSON)', url: abs('/api/v1/ai/results') },
] },
] },
],
],
optional: [
optional: [
@@ -11,6 +11,12 @@
@@ -11,6 +11,12 @@
* GET /rundown/s/:username one streamer
* GET /rundown/s/:username one streamer
* GET /after-show/:streamId one after-show report
* GET /after-show/:streamId one after-show report
* GET /c/:username one channel's after-show reports
* GET /c/:username one channel's after-show reports
+ * GET /ai OpenVibe.Bench's public results: task kinds, benchmarks, runs and last runs
+ * GET /ai/method how a score, a cost and a speed are made
+ * GET /ai/compare two or three models side by side
+ * GET /ai/feed.xml new runs as RSS
+ * GET /ai/:ref one task kind (Elo/price/speed) or one public benchmark (bmk_…)
+ * GET /ai/models/:model one model's runs over time, tasks, cost and recent runs
*
*
* A bad path parameter is a 404 without a source request; so is a source that answers nothing. Pages are public
* A bad path parameter is a 404 without a source request; so is a source that answers nothing. Pages are public
* for two minutes.
* for two minutes.
@@ -49,25 +55,68 @@ function createPublicPages({ sessionReports, network, live, bench, analytics = n
@@ -49,25 +55,68 @@ function createPublicPages({ sessionReports, network, live, bench, analytics = n
}
}
async function aiIndex() {
async function aiIndex() {
- const kinds = await bench.kinds();
+ const [overview, kinds] = await Promise.all([bench.overview(), bench.kinds()]);
const list = kinds && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];
const list = kinds && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];
- const boards = await Promise.all(list.map((kind) => bench.board(kind.id, 'elo')));
- return { html: views.aiIndex({ kinds, boards }) };
+ const benches = overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : [];
+ const [boards, rankings] = await Promise.all([
+ Promise.all(list.map((kind) => bench.board(kind.id, 'elo'))),
+ Promise.all(benches.map((b) => bench.rankings(b.id, 'score'))),
+ ]);
+ return { html: views.aiIndex({ overview, kinds, boards, rankings }) };
}
}
async function aiBoard(kindId, tab) {
async function aiBoard(kindId, tab) {
if (!bench.isKindId(kindId)) return null;
if (!bench.isKindId(kindId)) return null;
- const [kinds, board, cost] = await Promise.all([
- bench.kinds(), bench.board(kindId, tab), tab === 'pareto' ? bench.board(kindId, 'cost') : Promise.resolve(null),
+ const [kinds, board, cost, benchmarks, runs, overview] = await Promise.all([
+ bench.kinds(),
+ bench.board(kindId, tab),
+ tab === 'pareto' ? bench.board(kindId, 'cost') : Promise.resolve(null),
+ bench.benchmarks(),
+ bench.runs({ kind: kindId, limit: 8 }),
+ bench.overview(),
]);
]);
const kind = kinds && Array.isArray(kinds.data.kinds) ? kinds.data.kinds.find((k) => k.id === kindId) : null;
const kind = kinds && Array.isArray(kinds.data.kinds) ? kinds.data.kinds.find((k) => k.id === kindId) : null;
if (kinds && !kind) return null;
if (kinds && !kind) return null;
- return { html: views.aiBoard({ kindId, kind, tab, board, cost, kinds }) };
+ const kindBenchmarks = benchmarks && benchmarks.data && Array.isArray(benchmarks.data.benchmarks)
+ ? benchmarks.data.benchmarks.filter((b) => b.kind === kindId) : [];
+ const rankings = await Promise.all(kindBenchmarks.map((b) => bench.rankings(b.id, 'score')));
+ return { html: views.aiBoard({ kindId, kind, tab, board, cost, kinds, benchmarks, rankings, runs, overview }) };
+ }
+
+ async function aiBenchmark(benchmarkId) {
+ if (!bench.isBenchmarkId(benchmarkId)) return null;
+ const [benchmark, rankings, runs, overview] = await Promise.all([
+ bench.benchmark(benchmarkId), bench.rankings(benchmarkId, 'score'), bench.runs({ benchmark: benchmarkId, limit: 20 }), bench.overview(),
+ ]);
+ // A public id the overview does not list is a 404; a Bench that did not answer at all is the honest unavailable page.
+ const list = overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : null;
+ if (list && !list.some((b) => b.id === benchmarkId)) return null;
+ return { html: views.aiBenchmark({ benchmark, rankings, runs }) };
}
}
async function aiModel(model) {
async function aiModel(model) {
if (!bench.isModelName(model)) return null;
if (!bench.isModelName(model)) return null;
- return { html: views.aiModel({ model, summary: await bench.modelSummary(model) }) };
+ const [detail, legacy] = await Promise.all([bench.modelDetail(model), bench.modelSummary(model)]);
+ return { html: views.aiModel({ model, detail, legacy }) };
+ }
+
+ async function aiMethod() {
+ const [overview, kinds] = await Promise.all([bench.overview(), bench.kinds()]);
+ return { html: views.aiMethod({ overview, kinds }) };
+ }
+
+ async function aiCompare(models = [], errors = []) {
+ const clean = [...new Set((models || []).filter((m) => bench.isModelName(m)))].slice(0, 3);
+ const [details, legacy] = await Promise.all([
+ Promise.all(clean.map((m) => bench.modelDetail(m))),
+ Promise.all(clean.map((m) => bench.modelSummary(m))),
+ ]);
+ return { models: clean, html: views.aiCompare({ models: clean, details, legacy, errors }) };
+ }
+
+ async function aiFeed() {
+ const [runs, overview] = await Promise.all([bench.runs({ limit: 20 }), bench.overview()]);
+ return { runs, overview };
}
}
function design() { return { html: designViews.designHealth(designHealth.all()) }; }
function design() { return { html: designViews.designHealth(designHealth.all()) }; }
@@ -80,7 +129,27 @@ function createPublicPages({ sessionReports, network, live, bench, analytics = n
@@ -80,7 +129,27 @@ function createPublicPages({ sessionReports, network, live, bench, analytics = n
return { html: views.rundown({ rundown: r, topics, reports }) };
return { html: views.rundown({ rundown: r, topics, reports }) };
}
}
- return { home, stats, developers, design, rundown, aiIndex, aiBoard, aiModel };
+ return { home, stats, developers, design, rundown, aiIndex, aiBoard, aiBenchmark, aiModel, aiMethod, aiCompare, aiFeed };
+}
+
+/** The new-results feed: one item per public run, with its own Bench date and no invented number. */
+function rssFeed({ runs, overview, base }) {
+ const xml = (s) => String(s == null ? '' : s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"');
+ const list = runs && runs.data && Array.isArray(runs.data.runs) ? runs.data.runs : [];
+ const titles = new Map((overview && overview.data && Array.isArray(overview.data.list) ? overview.data.list : []).map((b) => [b.id, b.title]));
+ const site = String(base || 'https://openvibe.report').replace(/\/+$/, '');
+ const items = list.map((r) => {
+ const title = `${r.model} on ${titles.get(r.benchmark_id) || r.benchmark_id}`;
+ const link = `${site}/ai/${encodeURIComponent(r.benchmark_id)}`;
+ const score = r.score == null ? 'no score' : `${Math.round(Number(r.score) * 10) / 10}/100`;
+ const cost = r.cost_usd == null ? 'cost not given' : `$${Number(r.cost_usd).toFixed(4).replace(/0+$/, '').replace(/\.$/, '')}`;
+ const speed = r.ms == null ? 'time not given' : `${Math.round(Number(r.ms))} ms`;
+ const at = Date.parse(r.created_at);
+ return `<item><title>${xml(title)}</title><link>${xml(link)}</link><guid isPermaLink="false">${xml(r.id)}</guid>${Number.isFinite(at) ? `<pubDate>${new Date(at).toUTCString()}</pubDate>` : ''}<description>${xml(`${r.model} scored ${score} on ${titles.get(r.benchmark_id) || r.benchmark_id}, for ${cost} in ${speed} (${r.provenance || 'run'}).`)}</description></item>`;
+ }).join('');
+ const newest = list.length ? Date.parse(list[0].created_at) : NaN;
+ return `<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0"><channel><title>OpenVibe AI: new results</title><link>${xml(`${site}/ai`)}</link><description>Verified and submitted AI runs as OpenVibe.Bench publishes them.</description><language>en</language>${Number.isFinite(newest) ? `<lastBuildDate>${new Date(newest).toUTCString()}</lastBuildDate>` : ''}${items}</channel></rss>`;
}
}
function createPageRoutes(ctx) {
function createPageRoutes(ctx) {
@@ -137,6 +206,24 @@ function createPageRoutes(ctx) {
@@ -137,6 +206,24 @@ function createPageRoutes(ctx) {
return send(req, res, page.html, 200, 'no-store');
return send(req, res, page.html, 200, 'no-store');
});
});
+ // The AI section's method, comparison and feed. They must sit before /ai/:ref, which would read them as a kind.
+ r.get('/ai/method', async (req, res) => send(req, res, (await publicPages.aiMethod()).html, 200, 'no-store'));
+
+ r.get('/ai/compare', async (req, res) => {
+ const raw = String(req.query.models || '').split(',').map((x) => x.trim()).filter(Boolean);
+ const errors = [];
+ const unique = [...new Set(raw)];
+ if (unique.length < 2) errors.push('Pick at least two model ids, comma separated.');
+ if (unique.length > 3) errors.push('At most three models can be compared at once.');
+ const page = await publicPages.aiCompare(unique.slice(0, 3), errors);
+ return send(req, res, page.html, 200, 'no-store');
+ });
+
+ r.get('/ai/feed.xml', async (req, res) => {
+ const feed = await publicPages.aiFeed();
+ res.status(200).set('Cache-Control', 'public, max-age=120').type('application/rss+xml; charset=utf-8').send(rssFeed({ ...feed, base: ctx.config.baseUrl }));
+ });
+
// The AI Exchange's daily aggregates (plan G-P9), staff-only, behind REPORT_AI_EXCHANGE. The gate is the signed-in
// The AI Exchange's daily aggregates (plan G-P9), staff-only, behind REPORT_AI_EXCHANGE. The gate is the signed-in
// person's staff role (`staff.site.view`, the capability that covers analytics and money state), the same
// person's staff role (`staff.site.view`, the capability that covers analytics and money state), the same
// openvibe-contracts staff map other staff pages use. The page is private and never cached.
// openvibe-contracts staff map other staff pages use. The page is private and never cached.
@@ -155,8 +242,11 @@ function createPageRoutes(ctx) {
@@ -155,8 +242,11 @@ function createPageRoutes(ctx) {
}
}
});
});
- r.get('/ai/:kind', async (req, res) => {
- const page = await publicPages.aiBoard(req.params.kind, bench.normaliseTab(req.query.tab));
+ r.get('/ai/:ref', async (req, res) => {
+ const ref = String(req.params.ref || '');
+ const page = bench.isBenchmarkId(ref)
+ ? await publicPages.aiBenchmark(ref)
+ : await publicPages.aiBoard(ref, bench.normaliseTab(req.query.tab));
if (!page) return notFound(req, res);
if (!page) return notFound(req, res);
return send(req, res, page.html, 200, 'no-store');
return send(req, res, page.html, 200, 'no-store');
});
});
@@ -80,7 +80,7 @@ const NAV = [
@@ -80,7 +80,7 @@ const NAV = [
{ label: 'You', href: '/you', icon: 'fa-user' },
{ label: 'You', href: '/you', icon: 'fa-user' },
{ label: 'Numbers', href: '/stats', icon: 'fa-chart-line' },
{ label: 'Numbers', href: '/stats', icon: 'fa-chart-line' },
{ label: 'The Rundown', href: '/rundown', icon: 'fa-newspaper' },
{ label: 'The Rundown', href: '/rundown', icon: 'fa-newspaper' },
- { label: 'AI boards', href: '/ai', icon: 'fa-robot' },
+ { label: 'AI', href: '/ai', icon: 'fa-robot' },
{ label: 'API', href: '/developers', icon: 'fa-code', more: true },
{ label: 'API', href: '/developers', icon: 'fa-code', more: true },
];
];
@@ -90,7 +90,7 @@ function layout({ title, description, canonical = null, path = null, robots = 'i
@@ -90,7 +90,7 @@ function layout({ title, description, canonical = null, path = null, robots = 'i
service: 'report',
service: 'report',
apiBase: NETWORK,
apiBase: NETWORK,
links: NAV.map((l) => ({ ...l, active: l.href === active || (l.href !== '/' && active.startsWith(l.href)) })),
links: NAV.map((l) => ({ ...l, active: l.href === active || (l.href !== '/' && active.startsWith(l.href)) })),
- appBar: [{ label: 'Home', href: '/', icon: 'fa-house' }, { label: 'Numbers', href: '/stats', icon: 'fa-chart-line' }, { label: 'Rundown', href: '/rundown', icon: 'fa-newspaper' }, { label: 'AI boards', href: '/ai', icon: 'fa-robot' }],
+ appBar: [{ label: 'Home', href: '/', icon: 'fa-house' }, { label: 'Numbers', href: '/stats', icon: 'fa-chart-line' }, { label: 'Rundown', href: '/rundown', icon: 'fa-newspaper' }, { label: 'AI', href: '/ai', icon: 'fa-robot' }],
history: { type: 'page', title: title || NAME },
history: { type: 'page', title: title || NAME },
// Sign-in is this site's own (openvibe-sdk/sso at /auth): the navbar's buttons and its silent check use it.
// Sign-in is this site's own (openvibe-sdk/sso at /auth): the navbar's buttons and its silent check use it.
silentLogin: `${BASE}/auth/login?silent=1&next={url}`,
silentLogin: `${BASE}/auth/login?silent=1&next={url}`,
@@ -121,6 +121,7 @@ function layout({ title, description, canonical = null, path = null, robots = 'i
@@ -121,6 +121,7 @@ function layout({ title, description, canonical = null, path = null, robots = 'i
<link rel="stylesheet" href="${esc(ovServe.url('room.css'))}">
<link rel="stylesheet" href="${esc(ovServe.url('room.css'))}">
<link rel="stylesheet" href="${esc(ovServe.url('showcase.css'))}">
<link rel="stylesheet" href="${esc(ovServe.url('showcase.css'))}">
<link rel="stylesheet" href="${esc(asset('css/app.css'))}">
<link rel="stylesheet" href="${esc(asset('css/app.css'))}">
+${path && String(path).startsWith('/ai') ? `<link rel="alternate" type="application/rss+xml" title="AI: new results" href="${esc(`${BASE}/ai/feed.xml`)}">` : ''}
<script src="${esc(ovServe.url('kit-web.js'))}" defer></script>
<script src="${esc(ovServe.url('kit-web.js'))}" defer></script>
<script src="${esc(ovServe.url('kit-data-web.js'))}" defer></script>
<script src="${esc(ovServe.url('kit-data-web.js'))}" defer></script>
<script src="${esc(ovServe.url('motion.js'))}" defer></script>
<script src="${esc(ovServe.url('motion.js'))}" defer></script>
@@ -253,7 +254,7 @@ function home({ stats, rundown, topics } = {}) {
@@ -253,7 +254,7 @@ function home({ stats, rundown, topics } = {}) {
explore: { title: 'More on Report', items: [
explore: { title: 'More on Report', items: [
{ title: 'Every site\'s numbers', text: 'Page views, people, streams and how fast each service answers', href: '/stats', icon: 'ov:gauge' },
{ title: 'Every site\'s numbers', text: 'Page views, people, streams and how fast each service answers', href: '/stats', icon: 'ov:gauge' },
{ title: 'The Rundown', text: 'What streamers talked about, topic by topic', href: '/rundown', icon: 'ov:news' },
{ title: 'The Rundown', text: 'What streamers talked about, topic by topic', href: '/rundown', icon: 'ov:news' },
- { title: 'AI boards', text: 'Every benchmark kind, its leader, quality against cost and cost per solved task', href: '/ai', icon: 'ov:ai' },
+ { title: 'AI', text: 'Which AI does each job best, what it costs and how fast it is: Bench results from every room', href: '/ai', icon: 'ov:ai' },
{ title: 'Status', text: 'Whether every OpenVibe service is up right now', href: 'https://openvibe.network/status', icon: 'ov:ping' },
{ title: 'Status', text: 'Whether every OpenVibe service is up right now', href: 'https://openvibe.network/status', icon: 'ov:ping' },
] },
] },
dev: { title: 'For developers', lede: 'Every number here is open JSON, readable from any site.', samples: [
dev: { title: 'For developers', lede: 'Every number here is open JSON, readable from any site.', samples: [
@@ -644,6 +645,8 @@ function developers() {
@@ -644,6 +645,8 @@ function developers() {
['GET /api/v1/rundown/t/:slug', 'One topic with its takes and related topics.'],
['GET /api/v1/rundown/t/:slug', 'One topic with its takes and related topics.'],
['GET /api/v1/after-show/:streamId', 'A stream\'s after-show report: viewers minute by minute, chat, moments and the write-up.'],
['GET /api/v1/after-show/:streamId', 'A stream\'s after-show report: viewers minute by minute, chat, moments and the write-up.'],
['GET /api/v1/c/:username', 'A channel\'s after-show reports, newest first.'],
['GET /api/v1/c/:username', 'A channel\'s after-show reports, newest first.'],
+ ['GET /api/v1/ai/results', 'New public AI runs from OpenVibe.Bench, newest first, with their score, cost and time.'],
+ ['GET /ai/feed.xml', 'The same new AI results as an RSS feed.'],
];
];
const body = `<header class="rp-page-h"><p class="rp-kicker">For developers</p><h1>The Report API</h1><p class="rp-lede">Open JSON, no key, readable from any site (CORS). Counts only, never about a person.</p></header>
const body = `<header class="rp-page-h"><p class="rp-kicker">For developers</p><h1>The Report API</h1><p class="rp-lede">Open JSON, no key, readable from any site (CORS). Counts only, never about a person.</p></header>
<section aria-labelledby="rp-ep-h"><h2 id="rp-ep-h">Endpoints</h2><dl class="rp-eps">${rows.map(([ep, what]) => `<div><dt><code>${esc(ep)}</code></dt><dd>${esc(what)}</dd></div>`).join('')}</dl></section>
<section aria-labelledby="rp-ep-h"><h2 id="rp-ep-h">Endpoints</h2><dl class="rp-eps">${rows.map(([ep, what]) => `<div><dt><code>${esc(ep)}</code></dt><dd>${esc(what)}</dd></div>`).join('')}</dl></section>
@@ -652,140 +655,461 @@ function developers() {
@@ -652,140 +655,461 @@ function developers() {
}
}
function benchState(...reads) {
function benchState(...reads) {
- const stale = reads.filter((r) => r && r.stale);
+ const stale = reads.filter((r) => r && r.stale).sort((a, b) => String(a.at).localeCompare(String(b.at)));
if (!stale.length) return '';
if (!stale.length) return '';
- const oldest = stale.map((r) => r.at).sort()[0];
- return `<p class="rp-fine" role="status">Bench is unavailable. Showing the last good copy, read ${esc(dayTime(oldest))}.</p>`;
+ return `<p class="rp-fine" role="status">Bench is unavailable. Showing the last good copy, read ${esc(dayTime(stale[0].at))}.</p>`;
}
}
+/** The answer inside a Bench read ({ data, at, stale }), or null. */
+const readData = (r) => (r && r.data != null ? r.data : null);
const aiPath = (kind) => `/ai/${encodeURIComponent(kind)}`;
const aiPath = (kind) => `/ai/${encodeURIComponent(kind)}`;
const modelPath = (model) => `/ai/models/${encodeURIComponent(model)}`;
const modelPath = (model) => `/ai/models/${encodeURIComponent(model)}`;
const modelLink = (subject) => `<a href="${modelPath(subject)}">${esc(subject)}</a>`;
const modelLink = (subject) => `<a href="${modelPath(subject)}">${esc(subject)}</a>`;
const datasetLink = (kind) => `<p class="rp-fine">Download the <a href="${BENCH}/api/v1/datasets/${encodeURIComponent(kind)}.jsonl">dataset JSONL</a> from Bench.</p>`;
const datasetLink = (kind) => `<p class="rp-fine">Download the <a href="${BENCH}/api/v1/datasets/${encodeURIComponent(kind)}.jsonl">dataset JSONL</a> from Bench.</p>`;
+const scoreText = (v) => { const x = n(v); return x == null ? '—' : `${Math.round(x * 10) / 10}`; };
+const scoreCell = (v) => { const x = n(v); return x == null ? '—' : `<span title="${esc(String(v))} / 100 from Bench">${esc(scoreText(v))}</span>`; };
+const moneyCell = (v) => { const x = n(v); return x == null ? '—' : `<span title="${esc(String(v))} USD from Bench">${esc(usd(x))}</span>`; };
+const speedCell = (v) => { const x = n(v); if (x == null) return '—'; return `<span title="${exact(x)} ms from Bench">${esc(msText(x))}</span>`; };
+function msText(v) {
+ const x = n(v);
+ if (x == null) return '—';
+ if (x >= 60_000) return `${(x / 60_000).toFixed(x >= 600_000 ? 0 : 1).replace(/\.0$/, '')} min`;
+ if (x >= 1_000) return `${(x / 1_000).toFixed(x >= 10_000 ? 0 : 1).replace(/\.0$/, '')} s`;
+ return `${Math.round(x)} ms`;
+}
+const lastRunText = (v) => (v ? `Last run ${dayTime(v)}` : 'No runs yet');
+const provenanceCell = (p) => (p === 'verified' ? '<span class="rp-fine">verified</span>' : p === 'submitted' ? '<span class="rp-flag">submitted</span>' : '—');
+/** A reading of quality against cost: a model is on the frontier when nothing beats it on both. */
+function frontier(rows, xKey, yKey) {
+ return rows.map((r) => {
+ const x = n(r[xKey]), y = n(r[yKey]);
+ const best = x != null && y != null && !rows.some((o) => {
+ if (o === r) return false;
+ const ox = n(o[xKey]), oy = n(o[yKey]);
+ return ox != null && oy != null && ox <= x && oy >= y && (ox < x || oy > y);
+ });
+ return { ...r, frontier: best, cost: x, quality: y };
+ });
+}
+/** A benchmark's leads and last run, from the overview list (no new number is derived). */
+function byKindFromOverview(overview) {
+ const map = new Map();
+ for (const b of (overview && Array.isArray(overview.list) ? overview.list : [])) {
+ if (!b || !b.kind) continue;
+ const at = b.last_run;
+ if (!at) continue;
+ if (!map.has(b.kind) || String(at) > String(map.get(b.kind))) map.set(b.kind, at);
+ }
+ return map;
+}
-// ── AI boards (Kit 2): who does each job best, and what it costs ──
-// The reader came to choose a model for a job. So the page leads with the leaders, one row per job, and keeps the
-// jobs nobody has rated yet in one short list; each board is a ranking, a quality-against-cost chart and the exact
-// numbers. Nothing here is drawn from a guess: every figure is Bench's.
-const JUDGE_SHORT = { human_pairwise: 'blind votes', rubric: 'a rubric', automatic_grader: 'a grader', test_suite: 'a test suite', game_outcome: 'game results' };
-// Where people rate a kind themselves (only blind-vote kinds are rated by people); a room without a site has no link.
+// ── AI (OpenVibe.Bench, plan 2026-10-28 §4.12; the owner moved its pages here 2026-11-08) ──
+// The reader came to choose a model for a job. Every figure is Bench's: a board's rating, a run's score, the
+// cost and the milliseconds the run carried, and the moment it was published. A job nobody has rated yet is a
+// short wait list, never a made-up leader.
+const JUDGE_SHORT = { human_pairwise: 'blind votes', audience_poll: 'audience polls', rubric: 'a rubric', automatic_grader: 'a grader', test_suite: 'a test suite', game_outcome: 'game results' };
+const JUDGE_LONG = {
+ human_pairwise: 'People see two answers blind and pick the better one. A vote is one game between two models.',
+ audience_poll: 'An audience votes on the outputs; the poll is recorded as one game, however many people voted.',
+ rubric: 'A rubric scores the output against the task’s stated criteria.',
+ automatic_grader: 'A grader checks the output against the expected answer or the task’s rules.',
+ test_suite: 'A pinned test suite runs the output; it passes only when the tests pass.',
+ game_outcome: 'The game itself decides: the result is the judgement, not a person’s opinion.',
+};
const ROOM_HOSTS = { chat: 'openvibe.chat', games: 'openvibe.games', search: 'search.openvibe.network', codes: 'openvibe.codes' };
const ROOM_HOSTS = { chat: 'openvibe.chat', games: 'openvibe.games', search: 'search.openvibe.network', codes: 'openvibe.codes' };
const roomHost = (room) => { const r = rooms.get(room); return (r && r.host) || ROOM_HOSTS[room] || null; };
const roomHost = (room) => { const r = rooms.get(room); return (r && r.host) || ROOM_HOSTS[room] || null; };
const rowsOf = (read) => (read && read.data && read.data.board && Array.isArray(read.data.board.rows) ? read.data.board.rows : []);
const rowsOf = (read) => (read && read.data && read.data.board && Array.isArray(read.data.board.rows) ? read.data.board.rows : []);
-/**
- * Whether a board has results: on a ranking (rows carry games), a game has been played (a model entered but never
- * judged has 0); the quality and cost views carry no games, and any row counts.
- */
const played = (rows) => {
const played = (rows) => {
const list = rows || [];
const list = rows || [];
if (!list.some((r) => r && r.games != null)) return list.length > 0;
if (!list.some((r) => r && r.games != null)) return list.length > 0;
return list.some((r) => (n(r && r.games) || 0) > 0);
return list.some((r) => (n(r && r.games) || 0) > 0);
};
};
-const howRated = (kind) => `Rated by ${JUDGE_SHORT[kind.judge] || 'Bench'}`;
+const howRated = (kind) => `Rated by ${JUDGE_SHORT[kind && kind.judge] || 'Bench'}`;
const METHOD = 'Ratings are Bradley–Terry Elo from Bench, with a 95% interval; a model with fewer than ten games is provisional.';
const METHOD = 'Ratings are Bradley–Terry Elo from Bench, with a 95% interval; a model with fewer than ten games is provisional.';
+const CHART_NOTE = 'Up and to the left is better. The dashed line joins the models nothing beats on both.';
+const BENCH_API = `${BENCH}/api/v1`;
-/** What to do when a board is empty: rate it in its room (blind-vote kinds), or wait for runs (graded kinds). */
function emptyFor(kind) {
function emptyFor(kind) {
const host = kind && kind.judge === 'human_pairwise' ? roomHost(kind.room) : null;
const host = kind && kind.judge === 'human_pairwise' ? roomHost(kind.room) : null;
if (host) return kit.empty({ icon: 'trophy', title: 'No votes yet', text: `This board fills as people compare two answers blind on ${roomName(kind.room)} and pick the better one.`, action: { href: `https://${host}/`, label: `Vote on ${roomName(kind.room)}`, icon: 'forward' } });
if (host) return kit.empty({ icon: 'trophy', title: 'No votes yet', text: `This board fills as people compare two answers blind on ${roomName(kind.room)} and pick the better one.`, action: { href: `https://${host}/`, label: `Vote on ${roomName(kind.room)}`, icon: 'forward' } });
return kit.empty({ icon: 'trophy', title: 'No results yet', text: kind ? `This board fills as models are run and ${JUDGE_SHORT[kind.judge] || 'Bench'} scores them.` : 'This board fills as models are run on Bench.' });
return kit.empty({ icon: 'trophy', title: 'No results yet', text: kind ? `This board fills as models are run and ${JUDGE_SHORT[kind.judge] || 'Bench'} scores them.` : 'This board fills as models are run on Bench.' });
}
}
-function aiIndex({ kinds, boards = [] }) {
- const list = kinds && kinds.data && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];
- const items = list.map((kind, i) => ({ kind, rows: rowsOf(boards[i]), available: Boolean(boards[i]) }));
- // A board is rated once a game is played. Bench lists a model as soon as it has an entry, at the starting 1000
- // with 0 games: that is no leader (11-03, every board on /ai named gpt-5-nano "leader" with 0 games).
- const rated = items.filter((x) => played(x.rows)).sort((a, b) => gamesOf(b.rows) - gamesOf(a.rows));
- const waiting = items.filter((x) => !played(x.rows));
- const leaders = rated.length ? kit.table({
- caption: 'Leaders', columns: [
- { key: 'job', label: 'Job', primary: true, html: true }, { key: 'leader', label: 'Leader', html: true },
- { key: 'rating', label: 'Rating', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' },
- ],
- rows: rated.map(({ kind, rows }) => ({ job: `<a href="${aiPath(kind.id)}">${esc(kind.name || kind.id)}</a>`, leader: `${modelLink(rows[0].subject)}${flag(rows[0])}`, rating: rating(rows[0]), games: gamesOf(rows) })),
- }) : '';
- const waitRows = waiting.map(({ kind, available }) => kit.row({ href: aiPath(kind.id), title: kind.name || kind.id, sub: `${roomName(kind.room)} · ${available ? howRated(kind) : 'board unavailable'}`, trail: kit.icon('chevron') }));
- const waitList = waitRows.length ? kit.list(waitRows, { flush: true }) : '';
- const none = !rated.length && list.length;
- const body = kit.head({ title: 'AI boards', sub: 'Which AI does each job best, and what it costs', sticky: false })
- + benchState(kinds, ...boards)
- + (!list.length ? kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'The boards come back as soon as Bench answers.' }) : '')
- + (none ? kit.notice({ kind: 'info', text: `No model is rated yet. ${list.length} jobs are set up; each board fills as people vote blind on two answers, or as graders score runs.` }) : '')
- + (rated.length ? kit.stats([[rated.length, rated.length === 1 ? 'job rated' : 'jobs rated'], [rated.reduce((s, x) => s + gamesOf(x.rows), 0), 'games played'], [list.length, 'jobs tracked']], { label: 'The boards in numbers' }) : '')
- + leaders
- + (waitList ? (none ? kit.section({ title: 'Jobs being rated', count: waiting.length, body: waitList }) : kit.fold({ title: 'Waiting for their first results', meta: String(waiting.length), body: waitList })) : '')
- + `<p class="rp-fine">${esc(METHOD)} Every board's data is open: <a href="${BENCH}/docs">Bench's API and datasets</a>.</p>`;
- return layout({ title: 'AI boards', description: 'Which AI does each job best, and what it costs: Bench ratings across OpenVibe.', body, active: '/ai', path: '/ai', agent: true });
-}
-
+/** The cost-per-solved chart Bench's boards already use: quality (0–1) against cost per solved task. */
function paretoRows(pareto, cost) {
function paretoRows(pareto, cost) {
const costs = new Map(rowsOf(cost).map((r) => [r.subject, n(r.cost_per_solved)]));
const costs = new Map(rowsOf(cost).map((r) => [r.subject, n(r.cost_per_solved)]));
const rows = rowsOf(pareto).map((r) => ({ ...r, quality: n(r.quality), costPerSolved: costs.get(r.subject) }));
const rows = rowsOf(pareto).map((r) => ({ ...r, quality: n(r.quality), costPerSolved: costs.get(r.subject) }));
- return rows.map((r) => ({ ...r, frontier: r.quality != null && r.costPerSolved != null && !rows.some((other) => other !== r && other.quality != null && other.costPerSolved != null && other.quality >= r.quality && other.costPerSolved <= r.costPerSolved && (other.quality > r.quality || other.costPerSolved < r.costPerSolved)) }));
+ return frontier(rows.map((r) => ({ ...r, x: r.costPerSolved, y: r.quality })), 'x', 'y');
}
}
-
-/** Quality against cost per solved task, the frontier dashed and labelled (kit.chart scatter). */
function paretoChart(rows) {
function paretoChart(rows) {
const points = rows.filter((r) => r.quality != null && r.costPerSolved != null && r.costPerSolved >= 0);
const points = rows.filter((r) => r.quality != null && r.costPerSolved != null && r.costPerSolved >= 0);
if (!points.length) return '<p class="rp-fine">No quality and cost per solved pairs yet.</p>';
if (!points.length) return '<p class="rp-fine">No quality and cost per solved pairs yet.</p>';
return kit.chart({
return kit.chart({
- type: 'scatter', title: 'Quality against cost per solved task', note: 'Up and to the left is better. The dashed line joins the models nothing beats on both.',
+ type: 'scatter', title: 'Quality against cost per solved task', note: CHART_NOTE,
x: { label: 'Cost per solved task (USD, lower is better)', fmt: (v) => usd(v) }, y: { label: 'Quality', fmt: (v) => `${Math.round(v)}%`, zero: false },
x: { label: 'Cost per solved task (USD, lower is better)', fmt: (v) => usd(v) }, y: { label: 'Quality', fmt: (v) => `${Math.round(v)}%`, zero: false },
points: points.map((r) => ({ label: r.subject, x: r.costPerSolved, y: r.quality * 100, best: r.frontier, mark: r.frontier || points.length <= 8, href: modelPath(r.subject) })),
points: points.map((r) => ({ label: r.subject, x: r.costPerSolved, y: r.quality * 100, best: r.frontier, mark: r.frontier || points.length <= 8, href: modelPath(r.subject) })),
height: 260,
height: 260,
});
});
}
}
+/** A benchmark's verifier: score 0–100 against the median cost of a run. The frontier is the same reading. */
+function scoreCostChart(entries) {
+ const rows = frontier(entries.filter((r) => n(r.cost_usd) != null && n(r.score) != null).map((r) => ({ ...r, x: n(r.cost_usd), y: n(r.score) })), 'x', 'y');
+ if (!rows.length) return '';
+ return kit.chart({
+ type: 'scatter', title: 'Score against cost per run', note: CHART_NOTE,
+ x: { label: 'Median cost of a run (USD, lower is better)', fmt: (v) => usd(v) }, y: { label: 'Score (0–100)', fmt: (v) => `${Math.round(v)}`, zero: false },
+ points: rows.map((r) => ({ label: r.model, x: r.cost, y: r.quality, best: r.frontier, mark: r.frontier || rows.length <= 8, href: modelPath(r.model) })),
+ height: 260,
+ });
+}
const TABS = [{ key: 'elo', label: 'Ranking' }, { key: 'pareto', label: 'Quality and cost' }, { key: 'cost', label: 'Cost per solved' }];
const TABS = [{ key: 'elo', label: 'Ranking' }, { key: 'pareto', label: 'Quality and cost' }, { key: 'cost', label: 'Cost per solved' }];
-function aiBoard({ kindId, kind, tab, board, cost, kinds }) {
+function aiIndex({ overview, kinds, boards = [], rankings = [] }) {
+ const o = readData(overview) || {};
+ const list = kinds && kinds.data && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];
+ const benches = Array.isArray(o.list) ? o.list : [];
+ const lastByKind = byKindFromOverview(o);
+ const items = list.map((kind, i) => ({ kind, rows: rowsOf(boards[i]), available: Boolean(boards[i]) }));
+ const rated = items.filter((x) => played(x.rows)).sort((a, b) => gamesOf(b.rows) - gamesOf(a.rows));
+ const waiting = items.filter((x) => !played(x.rows));
+ const leaders = rated.length ? kit.table({
+ caption: 'Task kinds and their leaders', columns: [
+ { key: 'job', label: 'Job', primary: true, html: true }, { key: 'leader', label: 'Leader', html: true },
+ { key: 'quality', label: 'Quality', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' },
+ { key: 'cost', label: 'Cost/run', num: true, html: true },
+ { key: 'speed', label: 'Speed/run', num: true, html: true }, { key: 'last', label: 'Last run', html: true },
+ ],
+ rows: rated.map(({ kind, rows }) => {
+ const top = rows[0];
+ return {
+ job: `<a href="${aiPath(kind.id)}">${esc(kind.name || kind.id)}</a>`,
+ leader: `${modelLink(top.subject)}${flag(top)}`,
+ quality: rating(top),
+ games: gamesOf(rows),
+ cost: moneyCell(top.median_cost_usd),
+ speed: speedCell(top.median_ms),
+ last: lastByKind.get(kind.id) ? `<span title="${esc(lastByKind.get(kind.id))}">${esc(dayTime(lastByKind.get(kind.id)))}</span>` : '—',
+ };
+ }),
+ }) : '';
+ const waitRows = waiting.map(({ kind, available }) => kit.row({ href: aiPath(kind.id), title: kind.name || kind.id, sub: `${roomName(kind.room)} · ${available ? howRated(kind) : 'board unavailable'}`, trail: kit.icon('chevron') }));
+ const waitList = waitRows.length ? kit.list(waitRows, { flush: true }) : '';
+ const benchRow = (b, i) => {
+ const r = rankings[i] && readData(rankings[i]);
+ const top = r && r.rankings && Array.isArray(r.rankings.entries) ? r.rankings.entries[0] : null;
+ return {
+ benchmark: `<a href="${aiPath(b.id)}">${esc(b.title)}</a>`,
+ kind: esc(b.kind || '—'),
+ leader: top ? `${modelLink(top.model)} ${scoreCell(top.score)}` : '—',
+ runs: b.runs == null ? 0 : b.runs,
+ last: b.last_run ? esc(dayTime(b.last_run)) : 'No runs yet',
+ };
+ };
+ const activeBench = benches.filter((b) => Number(b.runs) > 0);
+ const idleBench = benches.filter((b) => !(Number(b.runs) > 0));
+ const benchTable = activeBench.length ? kit.table({
+ caption: 'Benchmarks with published runs', columns: [
+ { key: 'benchmark', label: 'Benchmark', primary: true, html: true }, { key: 'kind', label: 'Task kind' },
+ { key: 'leader', label: 'Leader', html: true }, { key: 'runs', label: 'Runs', num: true, fmt: 'int' }, { key: 'last', label: 'Last run', html: true },
+ ], rows: benches.map((b, i) => (Number(b.runs) > 0 ? benchRow(b, i) : null)).filter(Boolean),
+ }) : '';
+ const idleRows = benches.map((b, i) => (Number(b.runs) > 0 ? null : kit.row({ href: aiPath(b.id), title: b.title, sub: `${b.kind || 'benchmark'} · no run yet`, trail: kit.icon('chevron') }))).filter(Boolean);
+ const idleList = idleRows.length ? kit.list(idleRows, { flush: true }) : '';
+ const none = !rated.length && list.length;
+ const body = kit.head({ title: 'AI', sub: 'Which AI does each job best, what it costs, and how fast it is', sticky: false })
+ + benchState(overview, kinds, ...rankings, ...boards)
+ + `<p class="rp-fine">Every number on this page is a published Bench result or vote. ${esc(METHOD)} <a href="/ai/method">How the runs are judged</a> · <a href="/ai/compare">Compare models</a> · <a href="/ai/feed.xml">New results (RSS)</a></p>`
+ + (!list.length && !benches.length ? kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'The boards come back as soon as Bench answers.' }) : '')
+ + (o.benchmarks != null || o.runs != null ? kit.stats([
+ [o.benchmarks == null ? '—' : exact(o.benchmarks), o.benchmarks === 1 ? 'benchmark' : 'benchmarks'],
+ [o.tasks == null ? '—' : exact(o.tasks), o.tasks === 1 ? 'task' : 'tasks'],
+ [o.runs == null ? '—' : exact(o.runs), o.runs === 1 ? 'run' : 'runs'],
+ [o.models == null ? '—' : exact(o.models), o.models === 1 ? 'model' : 'models'],
+ [o.last_run ? dayTime(o.last_run) : 'No runs yet', 'last run'],
+ ], { label: 'AI in numbers' }) : (rated.length ? kit.stats([
+ [rated.length, rated.length === 1 ? 'job rated' : 'jobs rated'],
+ [rated.reduce((sum, x) => sum + gamesOf(x.rows), 0), 'games played'],
+ [list.length, 'jobs tracked'],
+ ], { label: 'The boards in numbers' }) : ''))
+ + (none ? kit.notice({ kind: 'info', text: `No model is rated on a task kind yet. ${list.length} jobs are set up; each board fills as people vote blind on two answers, or as graders score runs.` }) : '')
+ + (leaders ? kit.section({ title: 'Task kinds', count: rated.length, body: leaders }) : '')
+ + (benchTable ? kit.section({ title: 'Benchmarks with published runs', count: activeBench.length, body: benchTable }) : '')
+ + (idleList ? kit.fold({ title: 'Waiting for their first run', meta: String(idleRows.length), body: idleList }) : '')
+ + (waitList ? (none ? kit.fold({ title: 'Jobs being rated', meta: String(waiting.length), body: waitList }) : kit.fold({ title: 'Waiting for their first results', meta: String(waiting.length), body: waitList })) : '')
+ + `<p class="rp-fine">The open data is Bench's: <a href="${BENCH}/docs">its API and datasets</a> stay on bench.openvibe.services.</p>`;
+ return layout({ title: 'AI', description: 'Which AI does each job best, and what it costs: OpenVibe.Bench results, rendered by OpenVibe.Report.', body, active: '/ai', path: '/ai', agent: true });
+}
+
+function aiBoard({ kindId, kind, tab, board, cost, kinds, benchmarks, rankings = [], runs, overview }) {
const name = kind && kind.name || kindId;
const name = kind && kind.name || kindId;
const rows = rowsOf(board);
const rows = rowsOf(board);
+ const o = readData(overview) || {};
+ const lastByKind = byKindFromOverview(o);
+ const kindLast = lastByKind.get(kindId) || null;
+ const latest = readData(runs);
+ const latestRows = latest && Array.isArray(latest.runs) ? latest.runs : [];
+ // The registry's verified runs on this kind's public benchmarks: a second, task-level reading even while the
+ // blind board has no votes. A model's numbers across the kind's benchmarks are medians, never a sum of scores.
+ const kindBenchmarks = (() => {
+ const b = readData(benchmarks);
+ return b && Array.isArray(b.benchmarks) ? b.benchmarks.filter((x) => x.kind === kindId) : [];
+ })();
+ const byModel = new Map();
+ kindBenchmarks.forEach((b, i) => {
+ const r = rankings[i] && readData(rankings[i]);
+ const entries = r && r.rankings && Array.isArray(r.rankings.entries) ? r.rankings.entries : [];
+ for (const e of entries) {
+ if (!byModel.has(e.model)) byModel.set(e.model, { model: e.model, scores: [], costs: [], times: [], runs: 0, last_run: null });
+ const x = byModel.get(e.model);
+ if (n(e.score) != null) x.scores.push(n(e.score));
+ if (n(e.cost_usd) != null) x.costs.push(n(e.cost_usd));
+ if (n(e.ms) != null) x.times.push(n(e.ms));
+ x.runs += Number(e.runs) || 0;
+ if (e.last_run && (!x.last_run || e.last_run > x.last_run)) x.last_run = e.last_run;
+ }
+ });
+ const median = (list) => { const v = list.filter((x) => x != null).sort((a, b) => a - b); if (!v.length) return null; const m = Math.floor(v.length / 2); return v.length % 2 ? v[m] : (v[m - 1] + v[m]) / 2; };
+ const registryRows = [...byModel.values()].map((x) => ({ model: x.model, score: median(x.scores), cost: median(x.costs), speed: median(x.times), runs: x.runs, last_run: x.last_run })).sort((a, b) => (b.score == null ? -1 : b.score) - (a.score == null ? -1 : a.score) || a.model.localeCompare(b.model));
+ const registryTable = registryRows.length ? kit.table({ caption: 'Published runs on this kind', columns: [
+ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Median score', num: true, html: true },
+ { key: 'runs', label: 'Runs', num: true, fmt: 'int' }, { key: 'cost', label: 'Cost/run', num: true, html: true },
+ { key: 'speed', label: 'Speed/run', num: true, html: true }, { key: 'last', label: 'Last run', html: true },
+ ], rows: registryRows.map((r) => ({ model: modelLink(r.model), score: scoreText(r.score), runs: r.runs, cost: moneyCell(r.cost), speed: speedCell(r.speed), last: r.last_run ? esc(dayTime(r.last_run)) : '—' })) }) : '';
let content;
let content;
if (!board) content = kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'This board comes back as soon as Bench answers.' });
if (!board) content = kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'This board comes back as soon as Bench answers.' });
+ else if (!played(rows) && registryRows.length) content = kit.notice({ kind: 'info', text: `No blind vote on this task kind yet. ${registryRows.length} ${registryRows.length === 1 ? 'model has' : 'models have'} verified runs below.` });
else if (!played(rows)) content = emptyFor(kind);
else if (!played(rows)) content = emptyFor(kind);
else if (tab === 'pareto') {
else if (tab === 'pareto') {
const points = paretoRows(board, cost);
const points = paretoRows(board, cost);
- content = paretoChart(points) + (points.length ? kit.table({ caption: 'The numbers', columns: [{ key: 'm', label: 'Model', primary: true, html: true }, { key: 'q', label: 'Quality', num: true, html: true }, { key: 'c', label: 'Cost per solved', num: true, html: true }],
- // The frontier is a word on the model's name, so a phone's card stays two lines.
- rows: points.map((r) => ({ m: `${modelLink(r.subject)}${r.frontier ? ` ${kit.badge('frontier', 'accent')}` : ''}`, q: share(r.quality), c: usd(r.costPerSolved) })) }) : '');
+ content = paretoChart(points) + (points.length ? kit.table({ caption: 'The numbers', columns: [
+ { key: 'm', label: 'Model', primary: true, html: true }, { key: 'q', label: 'Quality', num: true, html: true },
+ { key: 'c', label: 'Cost per solved', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },
+ ], rows: points.map((r) => ({ m: `${modelLink(r.subject)}${r.frontier ? ` ${kit.badge('frontier', 'accent')}` : ''}`, q: share(r.quality), c: usd(r.costPerSolved), speed: speedCell(r.median_ms) })) }) : '');
} else if (tab === 'cost') {
} else if (tab === 'cost') {
const crows = rowsOf(cost).length ? rowsOf(cost) : rows;
const crows = rowsOf(cost).length ? rowsOf(cost) : rows;
- content = kit.table({ caption: 'Cost per solved task', columns: [{ key: 'm', label: 'Model', primary: true, html: true }, { key: 'solved', label: 'Solved', num: true, fmt: 'int' }, { key: 'entries', label: 'Entries', num: true, fmt: 'int' }, { key: 'total', label: 'Total cost', num: true, html: true }, { key: 'per', label: 'Cost per solved', num: true, html: true }],
- rows: crows.map((r) => ({ m: modelLink(r.subject), solved: r.solved, entries: r.entries, total: usd(r.total_cost_usd), per: usd(r.cost_per_solved) })) });
+ content = kit.table({ caption: 'Cost per solved task', columns: [
+ { key: 'm', label: 'Model', primary: true, html: true }, { key: 'solved', label: 'Solved', num: true, fmt: 'int' },
+ { key: 'entries', label: 'Entries', num: true, fmt: 'int' }, { key: 'total', label: 'Total cost', num: true, html: true },
+ { key: 'per', label: 'Cost per solved', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },
+ ], rows: crows.map((r) => ({ m: modelLink(r.subject), solved: r.solved, entries: r.entries, total: usd(r.total_cost_usd), per: usd(r.cost_per_solved), speed: speedCell(r.median_ms) })) });
} else {
} else {
- // The bars start just under the lowest interval, so the differences that matter are the ones you see.
const floor = Math.floor((Math.min(...rows.map((r) => n(r.ci_lower) ?? n(r.elo) ?? 0)) - 25) / 50) * 50;
const floor = Math.floor((Math.min(...rows.map((r) => n(r.ci_lower) ?? n(r.elo) ?? 0)) - 25) / 50) * 50;
content = kit.rank({ label: `${name}: ranking`, base: floor, fmt: 'int', items: rows.map((r) => {
content = kit.rank({ label: `${name}: ranking`, base: floor, fmt: 'int', items: rows.map((r) => {
const lo = n(r.ci_lower), hi = n(r.ci_upper), elo = n(r.elo);
const lo = n(r.ci_lower), hi = n(r.ci_upper), elo = n(r.elo);
+ const costText = n(r.median_cost_usd) != null ? `${usd(r.median_cost_usd)}/run` : null;
+ const speedText = n(r.median_ms) != null ? `${msText(r.median_ms)}/run` : null;
return { name: r.subject, href: modelPath(r.subject), value: elo, interval: lo != null && hi != null && elo != null ? Math.round(Math.max(hi - elo, elo - lo)) : null,
return { name: r.subject, href: modelPath(r.subject), value: elo, interval: lo != null && hi != null && elo != null ? Math.round(Math.max(hi - elo, elo - lo)) : null,
- note: [n(r.games) != null ? `${exact(r.games)} games` : '', n(r.win_rate) != null ? `${share(r.win_rate)} won` : '', provisional(r) ? 'provisional' : ''].filter(Boolean).join(' · ') };
- }) }) + `<p class="rp-fine">${esc(METHOD)}</p>`;
+ note: [n(r.games) != null ? `${exact(r.games)} games` : '', n(r.win_rate) != null ? `${share(r.win_rate)} won` : '', costText, speedText, provisional(r) ? 'provisional' : ''].filter(Boolean).join(' · ') };
+ }) }) + `<p class="rp-fine">${esc(METHOD)}</p>`
+ + kit.table({ caption: 'Quality, price and speed side by side', columns: [
+ { key: 'm', label: 'Model', primary: true, html: true }, { key: 'q', label: 'Quality', num: true, html: true },
+ { key: 'games', label: 'Games', num: true, fmt: 'int' }, { key: 'won', label: 'Won', num: true, html: true },
+ { key: 'cost', label: 'Cost/run', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },
+ ], rows: rows.map((r) => ({ m: modelLink(r.subject), q: rating(r), games: r.games, won: share(r.win_rate), cost: moneyCell(r.median_cost_usd), speed: speedCell(r.median_ms) })) });
}
}
- const head = kit.head({ title: name, sub: kind ? `${roomName(kind.room)} · ${howRated(kind)}` : 'Bench results', back: { href: '/ai', label: 'AI boards' },
+ const lastRunTable = latestRows.length ? kit.table({
+ caption: 'Latest runs of this kind', columns: [
+ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },
+ { key: 'cost', label: 'Cost', num: true, html: true }, { key: 'speed', label: 'Time', num: true, html: true },
+ { key: 'when', label: 'Published', html: true }, { key: 'how', label: 'How', html: true },
+ ],
+ rows: latestRows.map((r) => ({ model: modelLink(r.model), score: scoreCell(r.score), cost: usd(r.cost_usd), speed: speedCell(r.ms), when: esc(dayTime(r.created_at)), how: provenanceCell(r.provenance) })),
+ }) : '<p class="rp-fine">No run of this task kind has been published yet.</p>';
+ const head = kit.head({ title: name, sub: kind ? `${roomName(kind.room)} · ${howRated(kind)}${kindLast ? ` · ${lastRunText(kindLast)}` : ''}` : 'Bench results', back: { href: '/ai', label: 'AI' },
tabs: played(rows) ? kit.tabs({ base: aiPath(kindId), items: TABS, active: tab, label: 'Board view' }) : '' });
tabs: played(rows) ? kit.tabs({ base: aiPath(kindId), items: TABS, active: tab, label: 'Board view' }) : '' });
- const body = head + benchState(kinds, board, cost) + `<section aria-label="${esc(TABS.find((t) => t.key === tab)?.label || 'Board')}">${content}</section>${datasetLink(kindId)}`;
- return layout({ title: `${name}: AI board`, description: `${name}: which AI does it best, and what it costs.`, body, active: '/ai', path: aiPath(kindId), agent: true });
-}
-
-function aiModel({ model, summary }) {
- const data = summary && summary.data || null;
- const ratings = data && Array.isArray(data.ratings) ? data.ratings : [];
- const entries = data && Array.isArray(data.entries) ? data.entries : [];
- const kinds = [...new Set([...ratings.map((r) => r.kind_id), ...entries.map((e) => e.kind_id)])].filter(Boolean);
- const rows = kinds.map((kind) => {
- const r = ratings.find((item) => item.kind_id === kind);
- return { job: `<a href="${aiPath(kind)}">${esc(kind)}</a>`, rating: r ? `${rating(r)}${flag(r)}` : '—', games: r ? r.games : null, won: r ? share(r.win_rate) : '—', recent: entries.filter((item) => item.kind_id === kind).length };
+ const body = head + benchState(kinds, board, cost, benchmarks, runs, overview)
+ + `<section aria-label="${esc(TABS.find((t) => t.key === tab)?.label || 'Board')}">${content}</section>`
+ + (registryRows.length ? kit.section({ title: 'Verified runs on this task kind', count: registryRows.length, body: (registryRows.length > 1 ? scoreCostChart(registryRows.map((r) => ({ model: r.model, score: r.score, cost_usd: r.cost, ms: r.speed, runs: r.runs, last_run: r.last_run }))) : '') + registryTable }) : '')
+ + kit.section({ title: 'Latest runs', count: latestRows.length, body: lastRunTable })
+ + datasetLink(kindId);
+ return layout({ title: `${name}: AI`, description: `${name}: which AI does it best, what it costs and how fast it is.`, body, active: '/ai', path: aiPath(kindId), agent: true });
+}
+
+function aiBenchmark({ benchmark, rankings, runs }) {
+ const b = readData(benchmark);
+ const rank = readData(rankings);
+ const latest = readData(runs);
+ const entries = rank && rank.rankings && Array.isArray(rank.rankings.entries) ? rank.rankings.entries : [];
+ const latestRows = latest && Array.isArray(latest.runs) ? latest.runs : [];
+ const meta = b && b.benchmark ? b.benchmark : null;
+ const definition = b && b.definition ? b.definition : null;
+ const tasks = definition && Array.isArray(definition.tasks) ? definition.tasks : [];
+ const version = b && b.version ? b.version.version : null;
+ const last = entries.reduce((at, e) => (!at || (e.last_run && e.last_run > at) ? e.last_run : at), null)
+ || (latestRows[0] && latestRows[0].created_at) || null;
+ if (!meta) {
+ const body = kit.head({ title: 'Benchmark', sub: 'Bench results', back: { href: '/ai', label: 'AI' }, sticky: false })
+ + benchState(benchmark) + kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'This benchmark comes back as soon as Bench answers.' });
+ return layout({ title: 'AI benchmark', description: 'A benchmark of AI runs from OpenVibe.Bench.', body, active: '/ai', path: '/ai', agent: true });
+ }
+ const chart = scoreCostChart(entries);
+ const leaderboard = entries.length ? kit.table({
+ caption: 'Verified results', columns: [
+ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },
+ { key: 'runs', label: 'Runs', num: true, fmt: 'int' }, { key: 'range', label: 'Best / worst', num: true, html: true },
+ { key: 'cost', label: 'Cost/run', num: true, html: true }, { key: 'speed', label: 'Speed/run', num: true, html: true },
+ { key: 'solved', label: 'Cost per solved', num: true, html: true }, { key: 'last', label: 'Last run', html: true },
+ ],
+ rows: entries.map((e) => ({
+ model: modelLink(e.model), score: scoreCell(e.score), runs: e.runs,
+ range: `${scoreCell(e.max_score)} / ${scoreCell(e.min_score)}`, cost: moneyCell(e.cost_usd), speed: speedCell(e.ms),
+ solved: moneyCell(e.cost_per_solved), last: e.last_run ? esc(dayTime(e.last_run)) : '—',
+ })),
+ }) : kit.empty({ icon: 'trophy', title: 'No verified results yet', text: 'This benchmark has no verified run yet. Submitted runs are labelled on the latest-runs list and never mixed into the ranking.' });
+ const latestTable = latestRows.length ? kit.table({
+ caption: 'Latest runs', columns: [
+ { key: 'model', label: 'Model', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },
+ { key: 'cost', label: 'Cost', num: true, html: true }, { key: 'speed', label: 'Time', num: true, html: true },
+ { key: 'when', label: 'Published', html: true }, { key: 'how', label: 'How', html: true },
+ ],
+ rows: latestRows.map((r) => ({ model: modelLink(r.model), score: scoreCell(r.score), cost: usd(r.cost_usd), speed: speedCell(r.ms), when: esc(dayTime(r.created_at)), how: provenanceCell(r.provenance) })),
+ }) : '<p class="rp-fine">No run has been published on this benchmark yet.</p>';
+ const body = kit.head({ title: meta.title, sub: `${meta.kind || 'Benchmark'}${version ? ` · version ${version}` : ''} · ${tasks.length} ${tasks.length === 1 ? 'task' : 'tasks'} · ${lastRunText(last)}`, back: { href: '/ai', label: 'AI' }, sticky: false })
+ + benchState(benchmark, rankings, runs)
+ + (meta.description ? `<p class="rp-lede">${esc(meta.description)}</p>` : '')
+ + kit.stats([[entries.length, entries.length === 1 ? 'model ranked' : 'models ranked'], [latestRows.length, 'latest runs'], [tasks.length, tasks.length === 1 ? 'task' : 'tasks'], [last ? dayTime(last) : 'No runs yet', 'last run']], { label: `${meta.title} in numbers` })
+ + chart + leaderboard
+ + kit.section({ title: 'Latest runs', count: latestRows.length, body: latestTable })
+ + `<p class="rp-fine"><a href="/ai/method">How the runs are judged</a> · <a href="/ai/compare?models=${entries.slice(0, 3).map((e) => encodeURIComponent(e.model)).join(',')}">Compare the leaders</a> · <a href="${BENCH}/api/v1/datasets/benchmarks.jsonl">Bench's open data</a></p>`;
+ return layout({ title: `${meta.title}: AI benchmark`, description: `${meta.title}: verified AI results, cost and speed from OpenVibe.Bench.`, body, active: '/ai', path: aiPath(meta.id), agent: true });
+}
+
+function aiModel({ model, detail, legacy }) {
+ const d = readData(detail) || null;
+ const l = readData(legacy) || null;
+ const runs = d && Array.isArray(d.recent_runs) ? d.recent_runs : [];
+ const benchmarks = d && Array.isArray(d.benchmarks) ? d.benchmarks : [];
+ const tasks = d && Array.isArray(d.tasks) ? d.tasks : [];
+ const ratings = l && Array.isArray(l.ratings) ? l.ratings : [];
+ const entries = l && Array.isArray(l.entries) ? l.entries : [];
+ const hasRuns = Boolean(d && d.runs > 0);
+ const history = runs.filter((r) => n(r.score) != null && ts(r.created_at) != null).sort((a, b) => String(a.created_at).localeCompare(String(b.created_at)));
+ const historyChart = history.length >= 2 ? kit.chart({
+ type: 'line', title: 'Score over time', note: 'Every public run, newest last.',
+ x: { time: true, label: 'Published' }, y: { label: 'Score (0–100)', zero: false, fmt: (v) => `${Math.round(v)}` },
+ series: [{ name: model, points: history.map((r) => [r.created_at, n(r.score)]) }], height: 220,
+ }) : '';
+ const best = tasks.slice(0, 5);
+ const worst = tasks.length > 5 ? tasks.slice(-5).reverse() : [];
+ const taskRow = (t) => ({
+ task: `<span title="${esc(t.benchmark_title)}">${esc(t.task_id)}</span>`,
+ benchmark: `<a href="${aiPath(t.benchmark_id)}">${esc(t.benchmark_title)}</a>`,
+ score: scoreCell(t.score), runs: t.runs, last: t.last_run ? esc(dayTime(t.last_run)) : '—',
});
});
- const games = ratings.reduce((s, r) => s + (n(r.games) || 0), 0);
- const head = kit.head({ title: model, sub: 'Every job this model has been rated on', back: { href: '/ai', label: 'AI boards' }, sticky: false });
- const body = head + benchState(summary) + (!data ? kit.empty({ icon: 'trophy', title: 'Bench is unavailable', text: 'This page comes back as soon as Bench answers.' })
- : kit.stats([[kinds.length, kinds.length === 1 ? 'job' : 'jobs'], [games, 'games'], [n(data.total_cost_usd) ? usd(data.total_cost_usd) : null, 'spent on runs']], { label: `${model} in numbers` })
- + (rows.length ? kit.table({ caption: 'By job', columns: [{ key: 'job', label: 'Job', primary: true, html: true }, { key: 'rating', label: 'Rating', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' }, { key: 'won', label: 'Won', num: true, html: true }, { key: 'recent', label: 'Recent entries', num: true, fmt: 'int' }], rows })
- : kit.empty({ icon: 'trophy', title: 'Not rated yet', text: 'This model appears here once it has played on a Bench board.' })))
- + `<p class="rp-fine">${esc(METHOD)}</p>`;
- return layout({ title: `${model}: AI model`, description: `${model} across OpenVibe benchmark jobs: ratings, games and cost.`, body, active: '/ai', path: modelPath(model), agent: true });
+ const taskColumns = [
+ { key: 'task', label: 'Task', primary: true, html: true }, { key: 'benchmark', label: 'Benchmark', html: true },
+ { key: 'score', label: 'Median score', num: true, html: true }, { key: 'runs', label: 'Runs', num: true, fmt: 'int' },
+ { key: 'last', label: 'Last run', html: true },
+ ];
+ const taskTable = best.length ? kit.table({ caption: 'Best tasks', columns: taskColumns, rows: best.map(taskRow) }) : '';
+ const worstTable = worst.length ? kit.table({ caption: 'Worst tasks', columns: taskColumns, rows: worst.map(taskRow) }) : '';
+ const benchmarkRows = benchmarks.map((b) => ({
+ benchmark: `<a href="${aiPath(b.benchmark_id)}">${esc(b.title)}</a>`, kind: esc(b.kind || '—'), runs: b.runs,
+ score: scoreCell(b.score), range: `${scoreCell(b.best)} / ${scoreCell(b.worst)}`, cost: moneyCell(b.cost_usd), speed: speedCell(b.ms), last: b.last_run ? esc(dayTime(b.last_run)) : '—',
+ }));
+ const runTitles = new Map(benchmarks.map((b) => [b.benchmark_id, b.title]));
+ const runRows = runs.slice(0, 12).map((r) => ({
+ benchmark: `<a href="${aiPath(r.benchmark_id)}">${esc(runTitles.get(r.benchmark_id) || r.benchmark_id)}</a>`, score: scoreCell(r.score), cost: usd(r.cost_usd), speed: speedCell(r.ms),
+ when: esc(dayTime(r.created_at)), how: provenanceCell(r.provenance),
+ }));
+ const legacyRows = ratings.map((r) => ({ job: `<a href="${aiPath(r.kind_id)}">${esc(r.kind_id)}</a>`, rating: rating(r), games: r.games, won: share(r.win_rate) }));
+ const body = kit.head({ title: model, sub: hasRuns ? `${exact(d.runs)} verified ${d.runs === 1 ? 'run' : 'runs'} · ${lastRunText(d.last_run)}` : 'No verified runs yet', back: { href: '/ai', label: 'AI' }, sticky: false })
+ + benchState(detail, legacy)
+ + (hasRuns ? kit.stats([
+ [exact(d.runs), d.runs === 1 ? 'run' : 'runs'], [benchmarks.length, benchmarks.length === 1 ? 'benchmark' : 'benchmarks'],
+ [d.best_task ? scoreText(d.best_task.score) : '—', 'best task'], [d.worst_task ? scoreText(d.worst_task.score) : '—', 'worst task'],
+ [d.total_cost_usd == null ? '—' : usd(d.total_cost_usd), 'spent on runs'], [d.last_run ? dayTime(d.last_run) : 'No runs yet', 'last run'],
+ ], { label: `${model} in numbers` }) : '')
+ + (historyChart ? kit.section({ title: 'Runs over time', body: historyChart }) : '')
+ + (benchmarkRows.length ? kit.section({ title: 'By benchmark', count: benchmarks.length, body: kit.table({ caption: 'By benchmark', columns: [
+ { key: 'benchmark', label: 'Benchmark', primary: true, html: true }, { key: 'kind', label: 'Task kind' }, { key: 'runs', label: 'Runs', num: true, fmt: 'int' },
+ { key: 'score', label: 'Score', num: true, html: true }, { key: 'range', label: 'Best / worst', num: true, html: true }, { key: 'cost', label: 'Cost/run', num: true, html: true },
+ { key: 'speed', label: 'Speed/run', num: true, html: true }, { key: 'last', label: 'Last run', html: true },
+ ], rows: benchmarkRows }) }) : '')
+ + (taskTable ? kit.section({ title: 'Best and worst tasks', count: best.length + worst.length, body: taskTable + worstTable }) : '')
+ + (runRows.length ? kit.section({ title: 'Recent runs', count: runRows.length, body: kit.table({ caption: 'Recent runs', columns: [
+ { key: 'benchmark', label: 'Benchmark', primary: true, html: true }, { key: 'score', label: 'Score', num: true, html: true },
+ { key: 'cost', label: 'Cost', num: true, html: true }, { key: 'speed', label: 'Time', num: true, html: true }, { key: 'when', label: 'Published', html: true }, { key: 'how', label: 'How', html: true },
+ ], rows: runRows }) }) : '')
+ + (!hasRuns ? kit.empty({ icon: 'trophy', title: 'No verified runs yet', text: 'This model appears on a benchmark once a verified run is published. Its blind-board ratings, when it has any, are below.' }) : '')
+ + (legacyRows.length ? kit.section({ title: 'Blind-board ratings', count: legacyRows.length, body: kit.table({ caption: 'Ratings by job', columns: [
+ { key: 'job', label: 'Job', primary: true, html: true }, { key: 'rating', label: 'Rating', num: true, html: true }, { key: 'games', label: 'Games', num: true, fmt: 'int' }, { key: 'won', label: 'Won', num: true, html: true },
+ ], rows: legacyRows }) }) : '')
+ + (entries.length && !hasRuns ? `<p class="rp-fine">${exact(entries.length)} ${entries.length === 1 ? 'entry' : 'entries'} on Bench’s blind boards.</p>` : '')
+ + `<p class="rp-fine"><a href="/ai/compare?models=${encodeURIComponent(model)}">Compare ${esc(model)} with another model</a> · <a href="/ai/method">How the runs are judged</a> · <a href="${BENCH}/api/v1/models/${encodeURIComponent(model)}">Bench’s own record</a></p>`;
+ return layout({ title: `${model}: AI model`, description: `${model} across OpenVibe benchmark jobs: score, cost, speed, runs over time and its best and worst tasks.`, body, active: '/ai', path: modelPath(model), agent: true });
+}
+
+function aiCompare({ models = [], details = [], legacy = [], errors = [] }) {
+ const rowsByBenchmark = new Map();
+ const names = [];
+ const cards = [];
+ models.forEach((model, i) => {
+ const d = readData(details[i]) || null;
+ const l = readData(legacy[i]) || null;
+ names.push(model);
+ const list = d && Array.isArray(d.benchmarks) ? d.benchmarks : [];
+ for (const b of list) {
+ if (!rowsByBenchmark.has(b.benchmark_id)) rowsByBenchmark.set(b.benchmark_id, { title: b.title, cells: new Map() });
+ rowsByBenchmark.get(b.benchmark_id).cells.set(model, b);
+ }
+ cards.push({
+ model, runs: d ? d.runs : 0, cost: d ? d.total_cost_usd : null, last: d ? d.last_run : null,
+ ratingCount: l && Array.isArray(l.ratings) ? l.ratings.length : 0,
+ });
+ });
+ const rows = [...rowsByBenchmark.entries()].map(([id, x]) => {
+ const row = { benchmark: `<a href="${aiPath(id)}">${esc(x.title)}</a>` };
+ for (const model of models) {
+ const b = x.cells.get(model);
+ row[model] = b ? `<b>${scoreCell(b.score)}</b> <small>${esc(usd(b.cost_usd))}/run · ${esc(msText(b.ms))} · ${exact(b.runs)} runs</small>` : '—';
+ }
+ return row;
+ });
+ const columns = [{ key: 'benchmark', label: 'Benchmark', primary: true, html: true }, ...models.map((m) => ({ key: m, label: m, html: true }))];
+ const form = `<form class="rp-ai-compare-form" method="get" action="/ai/compare" role="search" aria-label="Compare models">
+<label for="rp-compare-models">Two or three model ids, comma separated</label>
+<input id="rp-compare-models" name="models" value="${esc(models.join(', '))}" placeholder="gpt-5, claude-opus-4" autocomplete="off">
+<button class="rp-btn" type="submit">Compare</button></form>`;
+ const cardHtml = cards.length ? kit.stats(cards.map((c) => [`${c.runs} · ${c.cost == null ? '—' : usd(c.cost)} · ${c.last ? day(c.last) : 'no runs'}`, c.model]), { label: 'Models in this comparison' }) : '';
+ const body = kit.head({ title: 'Compare AI models', sub: 'The same benchmarks side by side: score, cost per run, speed and the last run', back: { href: '/ai', label: 'AI' }, sticky: false })
+ + benchState(...details, ...legacy)
+ + form
+ + (errors.length ? kit.notice({ kind: 'warn', text: errors.join(' ') }) : '')
+ + (models.length < 2 ? kit.empty({ icon: 'trophy', title: 'Pick two or three models', text: 'Put their Bench ids in the box above; the comparison is built from their published runs.' }) : '')
+ + (rows.length ? kit.section({ title: 'Side by side', body: kit.table({ caption: 'Models side by side', columns, rows }) }) : (models.length >= 2 ? kit.empty({ icon: 'trophy', title: 'No shared benchmarks yet', text: 'These models have no verified run on a public benchmark in common.' }) : ''))
+ + (cardHtml ? `<p class="rp-fine">${esc('Each card: verified runs · total run cost · last run, then the model.')}</p>${cardHtml}` : '')
+ + `<p class="rp-fine">Cost and speed are the medians of each model's runs on that benchmark; a score is the median of its run scores. <a href="/ai/method">How the runs are judged</a>.</p>`;
+ return layout({ title: 'Compare AI models', description: 'Compare two or three AI models on the same OpenVibe benchmarks: score, cost and speed.', body, active: '/ai', path: '/ai/compare', agent: true });
+}
+
+function aiMethod({ overview, kinds }) {
+ const o = readData(overview) || {};
+ const list = kinds && kinds.data && Array.isArray(kinds.data.kinds) ? kinds.data.kinds : [];
+ const byJudge = new Map();
+ for (const k of list) {
+ if (!byJudge.has(k.judge)) byJudge.set(k.judge, []);
+ byJudge.get(k.judge).push(k);
+ }
+ const judgeRows = [...byJudge.entries()].map(([judge, ks]) => ({
+ how: `<b>${esc(JUDGE_SHORT[judge] || judge)}</b>`, judge: esc(judge), jobs: esc(ks.map((k) => k.name || k.id).join(', ')),
+ }));
+ const body = kit.head({ title: 'How the AI results are judged', sub: 'The method behind every score, price and speed on these pages', back: { href: '/ai', label: 'AI' }, sticky: false })
+ + benchState(overview, kinds)
+ + `<section class="rp-ai-method"><h2>What a score means</h2>
+<p>A score is a run’s result on one benchmark, from 0 to 100. Bench’s frozen benchmark version names its tasks, its judges and how the task scores combine; the default is the mean of the task scores. A model’s score on a benchmark is the median of its <b>verified</b> runs, so one bad run does not decide the page. The best and worst columns are the runs themselves, not a second estimate.</p>
+<h2>Quality, price and speed</h2>
+<p><b>Quality</b> is that score. <b>Price</b> is the median cost of a run in US dollars, as the service that ran it published it; <b>cost per solved</b> is the run’s cost divided by its success rate (score ÷ 100), so it is the price of one solved task. <b>Cost per solved</b> is the same figure, named plainly. <b>Speed</b> is the median wall-clock time of a run. The quality-against-cost chart marks the frontier: the models nothing beats on both.</p>
+<h2>How runs are judged</h2>
+<p>Each task kind has one main judge. A benchmark may also name more judges in its definition; the exact list is in its public definition at <code>/api/v1/benchmarks/:id</code>.</p>
+${judgeRows.length ? kit.table({ caption: 'Judges in use', columns: [{ key: 'how', label: 'How', html: true }, { key: 'judge', label: 'Bench judge' }, { key: 'jobs', label: 'Task kinds' }], rows: judgeRows }) : `<p class="rp-fine">${esc('Bench did not answer its kind list; the judge rules above are its published contract.')}</p>`}
+<h2>Verified and submitted</h2>
+<p>A <b>verified</b> run was executed by a service (openvibe.codes, Run, a Node worker); a <b>submitted</b> run was posted with a person’s token. The rankings read verified runs only, and a submitted run is labelled wherever the latest runs are listed.</p>
+<h2>How often, and what is missing</h2>
+<p>${o.runs == null ? 'Bench did not answer the public counts.' : `${exact(o.benchmarks)} public ${o.benchmarks === 1 ? 'benchmark' : 'benchmarks'}, ${exact(o.tasks)} ${o.tasks === 1 ? 'task' : 'tasks'}, ${exact(o.runs)} ${o.runs === 1 ? 'run' : 'runs'} from ${exact(o.models)} ${o.models === 1 ? 'model' : 'models'}; the newest run was ${o.last_run ? esc(dayTime(o.last_run)) : 'never published'}.`}</p>
+<p class="rp-fine">A model with no verified run has no score here; a benchmark with no run says so. The public reads are Bench’s API at <a href="${BENCH}/docs">bench.openvibe.services/docs</a>; the open data is <a href="${BENCH}/api/v1/datasets/benchmarks.jsonl">benchmarks.jsonl</a> and the per-kind datasets. <a href="/ai/feed.xml">New results (RSS)</a> · <a href="/api/v1/ai/results">the same as JSON</a>.</p>`;
+ return layout({ title: 'How the AI results are judged', description: 'The method behind OpenVibe.Bench’s scores, costs and speeds, and how to read them.', body, active: '/ai', path: '/ai/method', agent: true });
}
}
// ── The AI Exchange (plan G-P9): Billing's settlements, summed by day, model and access ────────────────────────
// ── The AI Exchange (plan G-P9): Billing's settlements, summed by day, model and access ────────────────────────
@@ -868,4 +1192,4 @@ function notFound() {
@@ -868,4 +1192,4 @@ function notFound() {
return layout({ title: 'Not found', robots: 'noindex, follow', body: `<header class="rp-page-h"><h1>Nothing here</h1><p class="rp-lede">That page does not exist, or its source has nothing for it. <a href="/">Back to Report</a>.</p></header>` });
return layout({ title: 'Not found', robots: 'noindex, follow', body: `<header class="rp-page-h"><h1>Nothing here</h1><p class="rp-lede">That page does not exist, or its source has nothing for it. <a href="/">Back to Report</a>.</p></header>` });
}
}
-module.exports = { home, stats, rundown, sessionReport, sharingChip, topic, episode, speaker, recap, channel, you, youSignedOut, developers, aiIndex, aiBoard, aiModel, aiExchange, staffOnly, usdNano, paretoRows, paretoChart, notFound, layout, setRelease, configure };
+module.exports = { home, stats, rundown, sessionReport, sharingChip, topic, episode, speaker, recap, channel, you, youSignedOut, developers, aiIndex, aiBoard, aiBenchmark, aiModel, aiCompare, aiMethod, aiExchange, staffOnly, usdNano, paretoRows, paretoChart, notFound, layout, setRelease, configure };
@@ -40,8 +40,8 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')
@@ -40,8 +40,8 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')
await check('/ai leads with the leaders: the job, its leader, the rating and the games', async () => {
await check('/ai leads with the leaders: the job, its leader, the rating and the games', async () => {
const r = await t.get('/ai');
const r = await t.get('/ai');
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
- for (const s of ['<caption>Leaders</caption>', 'href="/ai/design.page">Page <design></a>', 'openai/gpt-4o', '1043 ± 38', 'data-label="Games">8</td>', 'job rated', 'provisional']) assert.ok(r.text.includes(s), s);
- assert.ok(!r.text.includes('No model is rated yet'), 'no empty notice while a board has a leader');
+ for (const s of ['Task kinds and their leaders', 'href="/ai/design.page">Page <design></a>', 'openai/gpt-4o', '1043 ± 38', 'data-label="Games">8</td>', 'job rated', 'provisional']) assert.ok(r.text.includes(s), s);
+ assert.ok(!r.text.includes('No model is rated on a task kind yet'), 'no empty notice while a board has a leader');
assert.ok(!r.text.includes('Page <design>'));
assert.ok(!r.text.includes('Page <design>'));
});
});
@@ -107,8 +107,8 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')
@@ -107,8 +107,8 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')
});
});
try {
try {
const r = await fresh.get('/ai');
const r = await fresh.get('/ai');
- assert.ok(r.text.includes('No model is rated yet') && r.text.includes('Jobs being rated'), 'one notice, one list');
- assert.ok(!r.text.includes('<caption>Leaders</caption>') && !r.text.includes('jobs tracked'), 'no empty leaders table and no lone number');
+ assert.ok(r.text.includes('No model is rated on a task kind yet') && r.text.includes('Jobs being rated'), 'one notice, one list');
+ assert.ok(!r.text.includes('Task kinds and their leaders') && !r.text.includes('jobs tracked'), 'no empty leaders table and no lone number');
const b = await fresh.get('/ai/design.page');
const b = await fresh.get('/ai/design.page');
assert.ok(b.text.includes('No votes yet') && b.text.includes('href="https://openvibe.design/"') && b.text.includes('Vote on Design'), 'a blind-vote job sends people to vote');
assert.ok(b.text.includes('No votes yet') && b.text.includes('href="https://openvibe.design/"') && b.text.includes('Vote on Design'), 'a blind-vote job sends people to vote');
assert.ok(!b.text.includes('class="ovk-tabs'), 'an empty board has no tabs');
assert.ok(!b.text.includes('class="ovk-tabs'), 'an empty board has no tabs');
@@ -130,7 +130,7 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')
@@ -130,7 +130,7 @@ const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot')
});
});
try {
try {
const r = await unplayed.get('/ai');
const r = await unplayed.get('/ai');
- assert.ok(r.text.includes('No model is rated yet') && !r.text.includes('<caption>Leaders</caption>'), 'no leaders table');
+ assert.ok(r.text.includes('No model is rated on a task kind yet') && !r.text.includes('Task kinds and their leaders'), 'no leaders table');
assert.ok(!r.text.includes('gpt-5-nano-2025-08-07'), 'the unjudged model is named nowhere as a leader');
assert.ok(!r.text.includes('gpt-5-nano-2025-08-07'), 'the unjudged model is named nowhere as a leader');
const b = await unplayed.get('/ai/design.page');
const b = await unplayed.get('/ai/design.page');
assert.ok(b.text.includes('No votes yet') && b.text.includes('Vote on Design') && !b.text.includes('class="ovk-tabs'), 'the board asks for votes');
assert.ok(b.text.includes('No votes yet') && b.text.includes('Vote on Design') && !b.text.includes('class="ovk-tabs'), 'the board asks for votes');
new file mode 100644
new file mode 100644
@@ -0,0 +1,161 @@
@@ -0,0 +1,161 @@
+'use strict';
+
+/**
+ * The AI section (owner, 2026-11-08): Bench's pages moved into Report. The overview, one task kind, one benchmark,
+ * one model, the comparison, the method page and the new-results feeds read only Bench's public answers.
+ */
+const assert = require('assert');
+const { boot, check, done, stubFetch, jsonResponse } = require('./helpers/boot');
+
+const BMK = 'bmk_01M65KA3H4Y4ZENBYYZQA8SRNN';
+const RUN_A = 'brn_01M65KA3H4Y4ZENBYYZQA8SRN1';
+const RUN_B = 'brn_01M65KA3H4Y4ZENBYYZQA8SRN2';
+
+const overview = {
+ kinds: 2, benchmarks: 1, tasks: 2, runs: 3, models: 2, last_run: '2026-11-07T10:00:00.000Z',
+ list: [{ id: BMK, title: 'Playable games arena', kind: 'design.page', description: 'A test benchmark.', latest_version: 1, runs: 3, last_run: '2026-11-07T10:00:00.000Z' }],
+ recent: [
+ { id: RUN_B, benchmark_id: BMK, model: 'model-b', score: 70, cost_usd: 0.04, ms: 2000, provenance: 'verified', created_at: '2026-11-07T10:00:00.000Z' },
+ { id: RUN_A, benchmark_id: BMK, model: 'model-a', score: 90, cost_usd: 0.02, ms: 1000, provenance: 'verified', created_at: '2026-11-06T10:00:00.000Z' },
+ ],
+};
+
+const runs = {
+ count: 2, offset: 0, last_run: '2026-11-07T10:00:00.000Z',
+ runs: [
+ { id: RUN_B, benchmark_id: BMK, model: 'model-b', score: 70, cost_usd: 0.04, ms: 2000, tokens: 20, provenance: 'verified', created_at: '2026-11-07T10:00:00.000Z' },
+ { id: RUN_A, benchmark_id: BMK, model: 'model-a', score: 90, cost_usd: 0.02, ms: 1000, tokens: 10, provenance: 'verified', created_at: '2026-11-06T10:00:00.000Z' },
+ ],
+};
+
+const rankings = { rankings: { benchmark: BMK, metric: 'score', entries: [
+ { model: 'model-a', runs: 2, score: 85, cost_usd: 0.02, ms: 1000, cost_per_solved: 0.024, min_score: 80, max_score: 90, last_run: '2026-11-06T10:00:00.000Z' },
+ { model: 'model-b', runs: 1, score: 70, cost_usd: 0.04, ms: 2000, cost_per_solved: 0.057, min_score: 70, max_score: 70, last_run: '2026-11-07T10:00:00.000Z' },
+] } };
+
+const benchmark = {
+ benchmark: { id: BMK, title: 'Playable games arena', kind: 'design.page', description: 'A test benchmark.' },
+ version: { version: 1, created_at: '2026-10-30T05:07:52.000Z' },
+ definition: { tasks: [{ id: 't1', prompt: 'Build one screen.' }, { id: 't2', prompt: 'Build another.' }] },
+ task_meta: [],
+};
+
+const modelDetail = {
+ model: 'model-a', runs: 2,
+ benchmarks: [{ benchmark_id: BMK, title: 'Playable games arena', kind: 'design.page', runs: 2, score: 85, best: 90, worst: 80, cost_usd: 0.02, ms: 1000, last_run: '2026-11-06T10:00:00.000Z' }],
+ tasks: [
+ { benchmark_id: BMK, benchmark_title: 'Playable games arena', task_id: 't1', runs: 2, score: 90, best: 90, worst: 90, last_run: '2026-11-06T10:00:00.000Z' },
+ { benchmark_id: BMK, benchmark_title: 'Playable games arena', task_id: 't2', runs: 2, score: 80, best: 80, worst: 80, last_run: '2026-11-06T10:00:00.000Z' },
+ ],
+ best_task: { task_id: 't1', score: 90 }, worst_task: { task_id: 't2', score: 80 },
+ total_cost_usd: 0.04, total_ms: 2000, total_tokens: 20, last_run: '2026-11-06T10:00:00.000Z',
+ recent_runs: [
+ ...runs.runs.filter((r) => r.model === 'model-a'),
+ { id: 'brn_01M65KA3H4Y4ZENBYYZQA8SRN3', benchmark_id: BMK, model: 'model-a', score: 80, cost_usd: 0.02, ms: 1200, tokens: 10, provenance: 'verified', created_at: '2026-11-05T10:00:00.000Z' },
+ ],
+};
+
+const legacy = { model: 'model-a', ratings: [{ kind_id: 'design.page', elo: 1043, ci_lower: 1005, ci_upper: 1081, games: 8, win_rate: 0.75 }], entries: [], total_cost_usd: 0.04 };
+
+(async () => {
+ const source = stubFetch();
+ const fetchImpl = async (url, opts) => {
+ const u = new URL(String(url));
+ if (u.port !== '5230') return source(url, opts);
+ const p = u.pathname;
+ if (p === '/api/v1/kinds') return jsonResponse({ count: 1, kinds: [{ id: 'design.page', room: 'design', name: 'Design page arena', judge: 'human_pairwise' }] });
+ if (p === '/api/v1/boards/design.page') {
+ const type = u.searchParams.get('type');
+ const rows = type === 'elo'
+ ? [{ subject: 'model-a', elo: 1043, ci_lower: 1005, ci_upper: 1081, games: 8, win_rate: 0.75, median_cost_usd: 0.02, median_ms: 1000, provisional: true }]
+ : type === 'pareto' ? [{ subject: 'model-a', quality: 0.8 }] : [{ subject: 'model-a', entries: 2, solved: 2, total_cost_usd: 0.04, cost_per_solved: 0.02, median_ms: 1000 }];
+ return jsonResponse({ board: { kind_id: 'design.page', type, rows } });
+ }
+ if (p === '/api/v1/overview') return jsonResponse(overview);
+ if (p === '/api/v1/benchmarks') return jsonResponse({ count: 1, benchmarks: overview.list });
+ if (p === `/api/v1/benchmarks/${BMK}`) return jsonResponse(benchmark);
+ if (p === '/api/v1/rankings') return jsonResponse(rankings);
+ if (p === '/api/v1/runs') return jsonResponse(runs);
+ if (p === '/api/v1/models/model-a') return jsonResponse(legacy);
+ if (p === '/api/v1/models/model-a/summary') return jsonResponse(modelDetail);
+ if (p === '/api/v1/models/model-b/summary') return jsonResponse({ ...modelDetail, model: 'model-b', runs: 1, total_cost_usd: 0.04, recent_runs: runs.runs.filter((r) => r.model === 'model-b') });
+ return jsonResponse({ error: 'no stub for ' + p }, 404);
+ };
+ const t = await boot({ fetchImpl, env: { OV_BENCH_INTERNAL_URL: 'http://127.0.0.1:5230' } });
+ try {
+ await check('the AI overview leads with the real leader and states the last run', async () => {
+ const r = await t.get('/ai');
+ assert.strictEqual(r.status, 200);
+ for (const s of ['Task kinds and their leaders', 'Design page arena', 'model-a', '1043 ± 38', '$0.02', '1 s', 'Playable games arena', 'Nov 7', 'AI in numbers']) assert.ok(r.text.includes(s), s);
+ assert.ok(r.text.includes('href="/ai/method"') && r.text.includes('href="/ai/compare"') && r.text.includes('/ai/feed.xml'));
+ });
+
+ await check('one task kind shows quality, price and speed side by side and its latest runs', async () => {
+ const r = await t.get('/ai/design.page');
+ assert.strictEqual(r.status, 200);
+ for (const s of ['Quality, price and speed side by side', 'Cost/run', 'Speed/run', '$0.02', '1 s', 'Latest runs', 'model-a', 'Nov 7']) assert.ok(r.text.includes(s), s);
+ });
+
+ await check('one benchmark page has a ranking, an SVG and the latest runs', async () => {
+ const r = await t.get(`/ai/${BMK}`);
+ assert.strictEqual(r.status, 200);
+ for (const s of ['Playable games arena', 'Score against cost per run', 'ovk-chart--scatter', 'model-a', '85', '$0.02', 'Cost per solved', 'Latest runs', 'version 1']) assert.ok(r.text.includes(s), s);
+ });
+
+ await check('one model page has its history, best and worst tasks and cost', async () => {
+ const r = await t.get('/ai/models/model-a');
+ assert.strictEqual(r.status, 200);
+ for (const s of ['Score over time', 'Best and worst tasks', 't1', 't2', 'spent on runs', '$0.04', 'Published', 'Blind-board ratings', 'Playable games arena']) assert.ok(r.text.includes(s), s);
+ assert.ok(!r.text.includes('<span title='), 'a stat is escaped HTML, not markup');
+ });
+
+ await check('compare puts two models side by side, and refuses one', async () => {
+ const two = await t.get('/ai/compare?models=model-a,model-b');
+ assert.strictEqual(two.status, 200);
+ assert.ok(two.text.includes('Playable games arena') && two.text.includes('model-a') && two.text.includes('model-b'), 'both models');
+ const one = await t.get('/ai/compare?models=model-a');
+ assert.strictEqual(one.status, 200);
+ assert.ok(one.text.includes('Pick at least two model ids'), 'the refusal');
+ });
+
+ await check('an unknown benchmark id is a 404 once Bench has answered its list', async () => {
+ assert.strictEqual((await t.get('/ai/bmk_01M65KA3H4Y4ZENBYYZQA8SRN9')).status, 404);
+ });
+
+ await check('the method page explains judging, verification and cost', async () => {
+ const r = await t.get('/ai/method');
+ assert.strictEqual(r.status, 200);
+ for (const s of ['What a score means', 'How runs are judged', 'Verified and submitted', 'Cost per solved', 'blind votes', 'benchmarks.jsonl']) assert.ok(r.text.includes(s), s);
+ });
+
+ await check('new results are RSS and JSON, from the same runs', async () => {
+ const rss = await t.get('/ai/feed.xml');
+ assert.strictEqual(rss.status, 200);
+ assert.match(rss.headers.get('content-type') || '', /application\/rss\+xml/);
+ assert.ok(rss.text.includes('<title>model-a on Playable games arena</title>') && rss.text.includes(RUN_A), 'the run is in the feed');
+ const json = await t.get('/api/v1/ai/results');
+ assert.strictEqual(json.status, 200);
+ const body = json.json();
+ assert.strictEqual(body.count, 2);
+ assert.strictEqual(body.results[0].id, RUN_B);
+ assert.strictEqual(body.results[0].benchmark, 'Playable games arena');
+ assert.ok(body.results[0].url.endsWith('/ai/' + BMK));
+ });
+
+ await check('the AI pages have Markdown twins, and the benchmark twin names its page', async () => {
+ for (const [path, expected] of [[`/ai/${BMK}.md`, 'Playable games arena'], ['/ai/method.md', 'How the AI results are judged'], ['/ai/models/model-a.md', 'model-a']]) {
+ const r = await t.get(path);
+ assert.strictEqual(r.status, 200, path);
+ assert.match(r.headers.get('content-type') || '', /^text\/markdown/, path);
+ assert.ok(r.text.includes(expected), `${path}: ${expected}`);
+ }
+ });
+
+ await check('the sitemap lists the AI pages and the benchmark', async () => {
+ const r = await t.get('/sitemap.xml');
+ assert.strictEqual(r.status, 200);
+ for (const s of ['/ai</loc>', '/ai/method', '/ai/compare', `/ai/${BMK}`, '/ai/design.page', '/ai/models/model-a']) assert.ok(r.text.includes(s), s);
+ });
+ } finally { await t.close(); }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
@@ -43,8 +43,15 @@ const { check, done, jsonResponse } = require('./helpers/boot');
@@ -43,8 +43,15 @@ const { check, done, jsonResponse } = require('./helpers/boot');
assert.strictEqual(await bench.board('../bad'), null);
assert.strictEqual(await bench.board('../bad'), null);
assert.strictEqual(await bench.board('design.page', 'wrong'), null);
assert.strictEqual(await bench.board('design.page', 'wrong'), null);
assert.strictEqual(await bench.modelSummary('bad?name'), null);
assert.strictEqual(await bench.modelSummary('bad?name'), null);
+ assert.strictEqual(await bench.benchmark('../bad'), null);
+ assert.strictEqual(await bench.rankings('not-a-benchmark'), null);
+ assert.strictEqual(await bench.runs({ benchmark: '../bad' }), null);
+ assert.strictEqual(await bench.modelDetail('bad?name'), null);
assert.strictEqual(calls, 2);
assert.strictEqual(calls, 2);
assert.strictEqual(bench.isModelName('openai/gpt-4o'), true);
assert.strictEqual(bench.isModelName('openai/gpt-4o'), true);
+ assert.strictEqual(bench.isBenchmarkId('bmk_01M65KA3H4Y4ZENBYYZQA8SRNN'), true);
+ assert.strictEqual(bench.normaliseLimit(0), 1);
+ assert.strictEqual(bench.normaliseLimit(1000), 100);
});
});
done();
done();