openvibers/openvibe.compare Commits
Harness effects, the harness x model matrix, and an agent pair's split (#24)
Changes
@@@ -20,9 +20,12 @@ API**. Server-rendered pages; `openvibe-shared` for the shell.and `/models/:org/:model` 301 to `/ai/models/:model`.- `GET /compute` — compute prices from Exchange: filters, sorts and a two/three-offer side-by-side.- `GET /ai`, `/ai/leaderboard`, `/ai/providers`, `/ai/models/:model`, `/ai/compare[/:a-vs-:b]`,-- `/ai/method`, `/ai/feed.xml`, `/ai/:ref` (a Bench task kind or benchmark) and `/ai/arena` — Report's-- AI comparison pages, with Bench's boards through `openvibe-sdk/bench`, the AI catalogue and Exchange-- market joined, and Report's measured first token, tokens per second and success beside the prices.++ `/ai/method`, `/ai/feed.xml`, `/ai/harnesses`, `/ai/matrix`, `/ai/:ref` (a Bench task kind or++ benchmark) and `/ai/arena` — Report's AI comparison pages, with Bench's boards through++ `openvibe-sdk/bench`, the AI catalogue and Exchange market joined, and Report's measured first++ token, tokens per second and success beside the prices. `/ai/harnesses` draws each harness's effect++ against the neutral `ov-bare` with its interval; `/ai/matrix` crosses harness and model; an agent++ pair (`<harness>+<model>-vs-<harness>+<model>`) splits its gap into model, harness and interaction.`/comparisons/ai-models` and an AI model's `/things/:id` 301 to the AI pages.- `GET /api/v1/comparisons[/:slug]` and `GET /api/v1/compute` — the same data as JSON, CORS-allowedfor the origins in [server/origins.json](server/origins.json).@@@ -47,6 +50,7 @@ and the compute book in [server/compute.js](server/compute.js) `SOURCE`:| `openvibe.report/api/v1/ai/performance?group=model&days=30` | measured AI time to first token, tokens per second and success rate, public model rows with a 20-call floor (OpenVibe.Report) || `openvibe.exchange/api/compute/prices` | reviewed compute tariffs per provider and region (OpenVibe.Exchange) || `bench.openvibe.services/api/v1/{kinds,ratings,benchmarks,rankings,runs,overview}` | AI boards, benchmarks and new runs through `openvibe-sdk/bench` (OpenVibe.Bench) |++| `bench.openvibe.services/api/v1/factors?task_kind=&days=` | model and harness apart: harness effects, agents and interactions, with intervals (OpenVibe.Bench) || `api.openvibe.exchange/api/v1/exchange/{catalog,market}` | AI catalogue and dated provider prices (OpenVibe.AI / Exchange) |It does not call AI, Live, Network or any authenticated API. Nothing here writes anywhere.@@@ -59,8 +63,9 @@ It does not call AI, Live, Network or any authenticated API. Nothing here writes- **Data:** [server/data.js](server/data.js) — fetches the seven feeds (8 s timeout; 3 s for Report), builds the threecomparisons, coalesces concurrent reads and caches for ten minutes; a failed feed becomes `null`.- **AI data:** [server/ai-data.js](server/ai-data.js) joins the catalogue/market and Bench's boards with-- Report's measured rows; [server/ai-views.js](server/ai-views.js) renders the Report pages and the Bench boards.-- A failed source leaves a worded status and a dash, never a zero; pages cache for two minutes.++ Report's measured rows and reads Bench's factor fit for the harness board, the matrix and an agent++ pair; [server/ai-views.js](server/ai-views.js) renders the Report pages, the Bench boards and the factor++ pages. A failed source leaves a worded status and a dash, never a zero; pages cache for two minutes.- **Compute:** [server/compute.js](server/compute.js) normalizes Exchange's tariffs and derives themonth (730 h) and GPU GB-hour prices; [server/compute-views.js](server/compute-views.js) renders thepage, Markdown and Dataset JSON-LD. A failed feed is not cached.@@@ -68,6 +73,7 @@ It does not call AI, Live, Network or any authenticated API. Nothing here writes- **Ranking:** [public/rank.js](public/rank.js) — the one ranking implementation, required by both theserver views and the browser.- **Browser:** [public/comparison.js](public/comparison.js), [public/compare-account.js](public/compare-account.js),++ [public/ai-factors.js](public/ai-factors.js) (an agent pair's gap split, drawn with `OVMetrics.split`),[public/compare.css](public/compare.css).- **Deploy:** [deploy/systemd/openvibe-compare.service](deploy/systemd/openvibe-compare.service),[deploy/nginx/openvibe.compare.conf](deploy/nginx/openvibe.compare.conf).@@@ -5,7 +5,7 @@ Compare v1 serves public comparisons at https://openvibe.compare. Run `npm ci`,Data: Exchange price-book market and its reviewed compute tariffs, Report's public AI performance rows (30 days, model group), Bench public arena/verified runs, Codes harness catalog, Billing public plans. Reads are coalesced and cached for ten minutes. Price provenance retains the provider's checked date; a board without an update timestamp labels its date as the observation date. Failed feeds produce explicit unavailable states. The AI models comparison shows Report's measured time to first token p50, tokens per second p50 and success rate beside each provider's price; a model under 20 calls or with Report down is a dash, never zero. There is no private data or paid execution. Compute prices are read from OpenVibe.Exchange's `GET /api/compute/prices`; Compare adds only derived arithmetic (per month at 24/7, per GPU GB-hour), never a copied tariff.--Pages: `/`, `/comparisons/:slug`, `/things/:id` (URI-encoded identity), `/compute` (filters by kind, GPU model, region and interruptible; sorts by price per hour, per GPU GB-hour and per month at 24/7; a two- or three-offer side-by-side), and the AI pages `/ai` (hub), `/ai/leaderboard`, `/ai/providers`, `/ai/models/:model`, `/ai/compare` and `/ai/compare/:a-vs-:b` (canonical sorted pair), `/ai/method`, `/ai/feed.xml`, `/ai/:ref` (a Bench task kind or benchmark) and `/ai/arena`. The old `/comparisons/ai-models` and an AI model's `/things/:id` 301 to the AI pages. AI ratings and prices come from OpenVibe.AI's catalogue and Exchange market joined to OpenVibe.Bench's boards through `openvibe-sdk/bench`; Report's measured rows sit beside the prices. JSON at `/api/v1/comparisons[/slug]` and `/api/v1/compute`. Pages negotiate `Accept: text/markdown`. Dataset JSON-LD, sitemap and llms.txt are generated from the same data. Desktop tables and phone cards share ranking logic; compute cards run to 899 px and its table starts at 900 px. Weight URL parameters are `w.<criterion>`; all-zero weights leave things unranked. Builder and version-history fields are reserved, with no write API in v1.++Pages: `/`, `/comparisons/:slug`, `/things/:id` (URI-encoded identity), `/compute` (filters by kind, GPU model, region and interruptible; sorts by price per hour, per GPU GB-hour and per month at 24/7; a two- or three-offer side-by-side), and the AI pages `/ai` (hub), `/ai/leaderboard`, `/ai/providers`, `/ai/models/:model`, `/ai/compare` and `/ai/compare/:a-vs-:b` (canonical sorted pair; two agents get a model/harness/interaction split), `/ai/harnesses` (each harness's effect with its interval), `/ai/matrix` (harness × model), `/ai/method`, `/ai/feed.xml`, `/ai/:ref` (a Bench task kind or benchmark) and `/ai/arena`. The old `/comparisons/ai-models` and an AI model's `/things/:id` 301 to the AI pages. AI ratings and prices come from OpenVibe.AI's catalogue and Exchange market joined to OpenVibe.Bench's boards through `openvibe-sdk/bench`; Report's measured rows sit beside the prices. JSON at `/api/v1/comparisons[/slug]` and `/api/v1/compute`. Pages negotiate `Accept: text/markdown`. Dataset JSON-LD, sitemap and llms.txt are generated from the same data. Desktop tables and phone cards share ranking logic; compute cards run to 899 px and its table starts at 900 px. Weight URL parameters are `w.<criterion>`; all-zero weights leave things unranked. Builder and version-history fields are reserved, with no write API in v1.Launch: Host register-compare.py registers port 5280 with a lifecycle override until the separate Contracts manifest is released. Use merge-deploy.sh, then ov-vhost-install for deploy/nginx/openvibe.compare.conf. No database or service credentials are required.@@@ -129,3 +129,51 @@.ai-page .ai-head h1{overflow-wrap:anywhere}.ai-page .ov-parts-bar label{min-height:44px}}++++/* Harness effects: one scale for every harness. The range is its 95% interval, the dot its effect, the line zero. */++.ai-harness-board{list-style:none;margin:0;padding:0;display:grid;gap:12px}++.ai-harness{display:grid;gap:10px;padding:14px 16px;border:1px solid var(--border);border-radius:14px;background:var(--bg-card);min-width:0}++.ai-harness-head{display:flex;flex-wrap:wrap;align-items:baseline;gap:2px 12px;min-width:0}++.ai-harness-head>b{font-size:1.05rem;overflow-wrap:anywhere}++.ai-harness-effect{font-weight:700;font-variant-numeric:tabular-nums}++.ai-harness-meta{color:var(--text-muted);font-size:.82rem;font-variant-numeric:tabular-nums;overflow-wrap:anywhere;min-width:0}++.ai-hstrip{position:relative;height:26px;min-width:0;margin:2px 8px;border-radius:999px;background:var(--bg-tertiary,color-mix(in srgb,var(--text-primary) 8%,transparent))}++.ai-hstrip::before{content:"";position:absolute;top:0;bottom:0;left:var(--zero,50%);width:1px;background:var(--text-muted);opacity:.55}++.ai-hstrip-range{position:absolute;top:8px;height:10px;border-radius:999px;background:color-mix(in srgb,var(--accent) 32%,transparent)}++.ai-hstrip-dot{position:absolute;top:5px;width:16px;height:16px;margin-left:-8px;border-radius:50%;background:var(--accent);border:2px solid var(--bg-card);box-sizing:border-box}++.ai-harness-agents{margin:0;display:flex;flex-wrap:wrap;gap:4px 12px;align-items:baseline;font-size:.86rem;color:var(--text-secondary);min-width:0}++.ai-agent-item{display:inline-flex;flex-wrap:wrap;gap:2px 6px;align-items:baseline;min-width:0}++.ai-agent-item>a{color:var(--text-primary);text-decoration:none;min-width:0}++.ai-agent-item>a:hover{color:var(--accent);text-decoration:underline}++.ai-agent{display:inline-flex;flex-wrap:wrap;gap:1px;align-items:baseline;min-width:0;overflow-wrap:anywhere}++.ai-agent>b{font-weight:700}++.ai-agent>span[aria-hidden]{color:var(--text-muted);margin:0 1px}++.ai-harness-agents b{font-variant-numeric:tabular-nums}++.ai-harness-agents small{display:block;color:var(--text-muted);font-size:.78rem;font-variant-numeric:tabular-nums}++++/* The matrix: a cell is an agent's score, its interval in words and the tooltip; the best pair per model is marked. */++.ai-interactions{margin:12px 0 0;padding:10px 12px;border:1px solid var(--border);border-radius:12px;background:var(--bg-card);font-size:.88rem;line-height:1.5;min-width:0;overflow-wrap:anywhere}++.ai-interactions b{font-variant-numeric:tabular-nums}++.ai-page .ovk-table td .ovk-badge{margin-left:4px;font-size:.7rem;white-space:nowrap}++@media (max-width:600px){++ .ai-page .ovk-table--stack td .ovk-badge{margin-left:0;display:inline-block;margin-top:4px}++ .ai-harness{padding:12px}++ .ai-hstrip{margin:2px 0}++}++++/* An agent pair: the gap split into model, harness and interaction, drawn by OVMetrics.split. */++.ai-gap{display:grid;gap:10px;padding:16px;border:1px solid var(--border);border-radius:16px;background:var(--bg-card);min-width:0}++.ai-gap h2{margin:0;font-size:1.05rem}++.ai-gap-say{margin:0;font-size:1.02rem;line-height:1.55;overflow-wrap:anywhere}++.ai-gap-split{min-width:0}++.ai-vs-card h2{margin:0;font-size:1.08rem;line-height:1.3;overflow-wrap:anywhere}++@media (max-width:600px){.ai-gap{padding:14px}.ai-gap-say{font-size:.98rem}}++++/* The factor pages' own numbers: a grid of tiles, never a row that a phone has to scroll sideways to read. */++.ai-page .ovk-stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,140px),1fr));overflow:visible}++.ai-page .ovk-table th small{display:block;margin-top:2px;color:var(--text-muted);font-size:.78rem;font-weight:400;font-variant-numeric:tabular-nums}++.ai-vs-card dd small{display:block;margin-top:2px;color:var(--text-muted);font-size:.78rem;font-weight:400;font-variant-numeric:tabular-nums}++@media (max-width:600px){++ .ai-vs-form .ai-filter{grid-column:1/-1}++ .ai-page .ovk-stats{grid-template-columns:repeat(auto-fit,minmax(min(100%,120px),1fr))}++}nnew file mode 100644@@@ -0,0 +1,44 @@++/* The agent pair's gap split, drawn with openvibe-shared/ov-metrics.js (OVMetrics.split), never with a chart of++ our own. The page writes the three signed parts; this file hands their sizes to the split bar and keeps the sign++ in the sentence beside it. Without JavaScript the sentence is the whole answer. */++(function (root, factory) {++ const api = factory(typeof document !== 'undefined' ? document : null);++ if (typeof module === 'object' && module.exports) module.exports = api;++ else root.AiFactors = api;++})(typeof globalThis !== 'undefined' ? globalThis : this, function (doc) {++ 'use strict';++ const LABELS = [['model', 'the model'], ['harness', 'the harness'], ['interaction', 'the two together']];++ const num = (v) => (v == null || !Number.isFinite(Number(v)) ? null : Number(v));++++ /** [{ key, label, value, signed, tone }] — value is the size the bar draws, signed is the direction. */++ function gapParts(parts = {}) {++ return LABELS.map(([key, label], i) => {++ const signed = num(parts[key]);++ return { key, label, value: signed == null ? null : Math.abs(signed), signed, tone: i + 1 };++ }).filter((p) => p.value != null);++ }++++ function draw(d, data, mount) {++ const win = d && d.defaultView;++ const OVMetrics = win && win.OVMetrics;++ if (!OVMetrics || !mount || !data || !data.parts) return null;++ const parts = gapParts(data.parts);++ if (!parts.some((p) => p.value > 0)) return null;++ const split = OVMetrics.split(parts.map((p) => ({ label: p.label, value: p.value, tone: p.tone })), { label: data.label || 'Where the gap comes from' });++ mount.replaceChildren(split);++ return split;++ }++++ function mount() {++ const data = doc.getElementById('ai-gap-data');++ const target = doc.getElementById('ai-gap-split');++ if (!data || !target) return;++ try { draw(doc, JSON.parse(data.textContent), target); } catch { /* a page without a gap is fine */ }++ }++++ if (doc) {++ if (doc.readyState === 'loading') doc.addEventListener('DOMContentLoaded', mount, { once: true });++ else mount();++ }++ return { gapParts, draw };++});@@@ -23,11 +23,23 @@ const BENCHMARK_RE = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;const MODEL_RE = /^[A-Za-z0-9][A-Za-z0-9._:+\/~\-]{0,119}$/;const BOARD_TYPES = Object.freeze(['elo', 'pareto', 'cost']);const METRICS = Object.freeze(['score', 'value', 'speed', 'consistency', 'horizon']);++const FACTOR_KINDS = Object.freeze(['chat', 'code', 'design', 'image', 'speech', 'search', 'computer', 'game']);++const FACTOR_DAYS = Object.freeze([7, 30, 90, 365]);const isKindId = (value) => typeof value === 'string' && KIND_RE.test(value);const isBenchmarkId = (value) => typeof value === 'string' && BENCHMARK_RE.test(value);const isModelName = (value) => typeof value === 'string' && MODEL_RE.test(value);++function normaliseFactorKind(value) {++ const raw = Array.isArray(value) ? value[0] : value;++ return FACTOR_KINDS.includes(raw) ? raw : 'code';++}++++function normaliseFactorDays(value) {++ const n = Number(Array.isArray(value) ? value[0] : value);++ return FACTOR_DAYS.includes(n) ? n : 90;++}++function normaliseMetric(value) {const raw = Array.isArray(value) ? value[0] : value;return METRICS.includes(raw) ? raw : 'score';@@@ -137,12 +149,25 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = coreturn read(models, `detail:${model}`, `/api/v1/models/${encodeURIComponent(model)}/summary`, `bench model detail (${model})`, 200);}++ // Model and harness apart (bench.rating@1 1.1.0 factors): one fit per task kind, category and window.++ async function factors({ taskKind = 'code', category = null, days = 90 } = {}) {++ const kind = normaliseFactorKind(taskKind);++ const span = normaliseFactorDays(days);++ const cat = category ? String(category).slice(0, 64) : null;++ const params = new URLSearchParams({ task_kind: kind, days: String(span) });++ if (cat) params.set('category', cat);++ const query = params.toString();++ return read(results, `factors:${query}`, `/api/v1/factors?${query}`, `bench factors (${query})`, 500);++ }++return {-- kinds, board, modelSummary, benchmarks, benchmark, overview, rankings, runs, modelDetail,++ kinds, board, modelSummary, benchmarks, benchmark, overview, rankings, runs, modelDetail, factors,baseUrl, ttlMs: TTL_MS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,++ normaliseFactorKind, normaliseFactorDays,};}module.exports = {-- createBenchClient, TTL_MS, BOARD_TYPES, METRICS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,++ createBenchClient, TTL_MS, BOARD_TYPES, METRICS, FACTOR_KINDS, FACTOR_DAYS,++ isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit, normaliseFactorKind, normaliseFactorDays,};@@@ -6,7 +6,7 @@const { createAiCatalogue } = require('./ai-catalogue');const { comparisonCatalogue } = require('./ai-market');const { findCatalogModel, makerOf } = require('./ai-join');--const { createBenchClient, isKindId, isModelName, normaliseTab } = require('./ai-bench');++const { createBenchClient, isKindId, isModelName, normaliseTab, normaliseFactorKind, normaliseFactorDays, FACTOR_KINDS } = require('./ai-bench');const { requestJson } = require('./ai-source');const { measuredRows, measuredOf } = require('./data');@@@ -106,6 +106,59 @@ function buildComparison({ kindsRead = null, boards = [], catalogue = null, perfreturn { rows, providers: catOffers, tasks, catalogue, availability, stale, measured_at: perfAt, updated_at: dates.sort().at(-1) || iso(now) };}++// ── Model and harness, apart ──++// Bench's factor fit (bench.rating@1 1.1.0 `factors`) is the only source that can say which half of an agent's++// score is the model and which is the harness. Its numbers are kept as read; a missing fit is null, never zero.++const factorList = (x) => (Array.isArray(x) ? x : []);++const factorNumber = (x) => x == null || !Number.isFinite(Number(x)) ? null : Number(x);++const agentLabel = (id) => String(id || '').replace(/^agent:/, '').replace(/#.*$/, '');++const looksLikeAgent = (id) => String(id || '').includes('+');++const round1 = (x) => factorNumber(x) == null ? null : Math.round(factorNumber(x) * 10) / 10;++++function normaliseFactors(read) {++ const raw = read && read.data && typeof read.data === 'object' ? read.data : null;++ if (!raw) return null;++ const models = factorList(raw.models).map((m) => ({++ model: String(m.model || ''), score: factorNumber(m.score), ci_lower: factorNumber(m.ci_lower), ci_upper: factorNumber(m.ci_upper), games: factorNumber(m.games),++ })).filter((m) => m.model);++ const harnesses = factorList(raw.harnesses).map((h) => ({++ harness: String(h.harness || ''), effect: factorNumber(h.effect), ci_lower: factorNumber(h.ci_lower), ci_upper: factorNumber(h.ci_upper), games: factorNumber(h.games),++ })).filter((h) => h.harness);++ const agents = factorList(raw.agents).map((a) => {++ const f = a.factors && typeof a.factors === 'object' ? a.factors : {};++ const part = (p) => ({ score: factorNumber(p && p.score) });++ return {++ id: String(a.id || ''), model: String(a.model || ''), harness: String(a.harness || ''),++ score: factorNumber(a.score), ci_lower: factorNumber(a.ci_lower), ci_upper: factorNumber(a.ci_upper),++ factors: { model: part(f.model), harness: part(f.harness), interaction: part(f.interaction) },++ };++ }).filter((a) => a.id && a.model && a.harness);++ return {++ task_kind: raw.task_kind || null, category: raw.category || null, days: factorNumber(raw.days),++ matches: factorNumber(raw.matches) || 0, games: factorNumber(raw.games) || 0, style_control: raw.style_control !== false,++ models, harnesses, agents, method: typeof raw.method === 'string' ? raw.method : null,++ };++}++++function pairAgent(fit, side) {++ const wanted = String(side || '');++ if (!fit || !wanted) return null;++ return fit.agents.find((a) => a.id === wanted || a.id === `agent:${wanted}` || agentLabel(a.id) === wanted || `${a.harness}+${a.model}` === wanted) || null;++}++++/** The gap between two agents in one fit, split into its model, harness and interaction parts (Elo points). */++function splitOf(fit, a, b, taskKind) {++ const A = pairAgent(fit, a);++ const B = pairAgent(fit, b);++ if (!A || !B || A.score == null || B.score == null) return null;++ const d = (part) => round1((A.factors[part].score == null ? 0 : A.factors[part].score) - (B.factors[part].score == null ? 0 : B.factors[part].score));++ return {++ task_kind: taskKind || fit.task_kind, days: fit.days, games: fit.games, matches: fit.matches, method: fit.method,++ a: A, b: B, gap: round1(A.score - B.score),++ parts: { model: d('model'), harness: d('harness'), interaction: d('interaction') },++ };++}++function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new Date().toISOString(), ttl = TTL_MS, bench = null, catalogue = null } = {}) {const clock = typeof now === 'function' ? now : () => new Date().toISOString();const benchClient = bench || createBenchClient({ fetchImpl, log, now: () => Date.parse(clock()) || Date.now() });@@@ -182,8 +235,37 @@ function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new}async function compare(models = []) {-- const data = await comparison();-- return { models: models.map((id) => data.rows.find((x) => x.id === id) || { id, reason: 'No public model data.' }), options: data.rows.map((x) => x.id), availability: data.availability, stale: data.stale, measured_at: data.measured_at };++ const [data, agentPair] = await Promise.all([++ comparison(),++ models.length === 2 ? agentSplit(models[0], models[1]) : Promise.resolve(null),++ ]);++ return { models: models.map((id) => data.rows.find((x) => x.id === id) || { id, reason: 'No public model data.' }), options: data.rows.map((x) => x.id), availability: data.availability, stale: data.stale, measured_at: data.measured_at, agentPair };++ }++++ // The harness board and the matrix read the same fit; the pair page reads it only when both sides look like++ // agents (`harness+model`). One failed first read stops the search: Bench is down, not merely missing a pair.++ async function factorFit(taskKind, days) {++ const read = await benchClient.factors({ taskKind, days });++ return { fit: normaliseFactors(read), read };++ }++++ async function factorBoard(query = {}) {++ const taskKind = normaliseFactorKind(query.task_kind);++ const days = normaliseFactorDays(query.days);++ const { fit, read } = await factorFit(taskKind, days);++ return { taskKind, days, fit, unavailable: !read, stale: read && read.stale ? read.at : null };++ }++++ async function agentSplit(a, b) {++ if (!looksLikeAgent(a) || !looksLikeAgent(b)) return null;++ const first = await factorFit('code', 365);++ if (!first.read) return null;++ const direct = first.fit ? splitOf(first.fit, a, b, 'code') : null;++ if (direct) return direct;++ const rest = FACTOR_KINDS.filter((kind) => kind !== 'code');++ const fits = await Promise.all(rest.map(async (kind) => ({ kind, ...(await factorFit(kind, 365)) })));++ const found = fits.filter((x) => x.fit && pairAgent(x.fit, a) && pairAgent(x.fit, b)).sort((x, y) => (y.fit.games || 0) - (x.fit.games || 0))[0];++ return found ? splitOf(found.fit, a, b, found.kind) : null;}async function leaderboard(query = {}) {@@@ -214,10 +296,10 @@ function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new}return {-- comparison, kindBoard, benchmarkPage, method, feed, modelPage, compare, leaderboard,++ comparison, kindBoard, benchmarkPage, method, feed, modelPage, compare, leaderboard, factorBoard, agentSplit,bench: benchClient, catalogue: catClient, readRows, ttlMs: ttl,isKindId: benchClient.isKindId, isBenchmarkId: benchClient.isBenchmarkId, isModelName: benchClient.isModelName, normaliseTab: benchClient.normaliseTab,};}--module.exports = { createAi, buildComparison, measuredRow, TTL_MS, PERF_URL };++module.exports = { createAi, buildComparison, measuredRow, normaliseFactors, splitOf, agentLabel, TTL_MS, PERF_URL };@@@ -29,6 +29,12 @@ const word = { ms: metrics.format.ms, tps: metrics.format.tps, rate: metrics.forconst measuredWord = { ttft: metrics.format.ms, tps: metrics.format.tps, success: metrics.format.rate };const sourceHint = (s) => { const x = s && typeof s === 'object' ? s : { name: s }; return `Source: ${x.name || 'not listed'}${x.date ? ` · ${String(x.date).slice(0, 10)}` : ''}${x.sample_size != null ? ` · n = ${x.sample_size}` : ''}`; };const readData = (r) => (r && r.data != null ? r.data : null);++const jsonData = (x) => JSON.stringify(x).replace(/</g, '\\u003c');++++// Bench's factor board covers these task kinds and windows; the page's form offers exactly these.++const FACTOR_KINDS = [['code', 'Code'], ['chat', 'Chat'], ['design', 'Design'], ['image', 'Image'], ['speech', 'Speech'], ['search', 'Search'], ['computer', 'Computer use'], ['game', 'Games']];++const FACTOR_DAYS = [[7, '7 days'], [30, '30 days'], [90, '90 days'], [365, '365 days']];++const KIND_NAME = Object.fromEntries(FACTOR_KINDS);const HEAD = `<link rel="stylesheet" href="/ai-comparison.css"><link rel="stylesheet" href="/shared/kit-data.css"><link rel="stylesheet" href="/shared/showcase.css"><style>${marks.CSS}\n${PARTS_CSS}</style>`;function page(title, description, body, path, ld = []) {@@@ -209,6 +215,7 @@ function hub(data = {}) {+ kit.stats([[ratedRows.length, 'rated'], [rows.length, 'models listed'], [providerNames.size, 'providers'], [measuredOffers.length, 'measured endpoints']], { label: 'The leaderboard in numbers' })+ `</header>`+ (pickList || pending.length ? kit.section({ title: 'Top picks', body: `${pickList ? `<ul class="ai-picks">${pickList}</ul>` : ''}${pending.length ? `<p class="ai-fine">Not enough data yet for ${esc(pending.join(', '))}.</p>` : ''}` }) : '')++ + kit.section({ title: 'Harness and model, apart', body: `<p class="ai-fine">A coding agent is a harness driving a model, and the harness can move a score as much as the model. Bench fits the two apart, with an interval on each harness effect. Every agent number here names its harness.</p><ul class="ai-chips"><li><a href="/ai/harnesses">Harness effects</a></li><li><a href="/ai/matrix">Harness × model matrix</a></li><li><a href="/ai/method">How the fit works</a></li></ul>` })// A phone shows the parts below the top picks one at a time as tabs; every part stays in the HTML.+ parts({ name: 'board', label: 'The leaderboard\'s parts', items: [{ id: 'leaderboard', label: 'Leaderboard', count: chat.length, html: kit.section({ title: 'Leaderboard', count: chat.length, actions: [act('/ai/leaderboard', chat.length > 25 ? `All ${chat.length} models` : 'Filter and sort')], body: board(chat.slice(0, 25)) + (others ? `<p class="ai-fine">Chat models only. ${others} embedding and speech ${others === 1 ? 'model is' : 'models are'} in the <a href="/ai/leaderboard?type=all">full list</a>.</p>` : '') + dataLinks() + measuredNote(data) }) },@@@ -349,6 +356,7 @@ function model(data = {}) {// ── /ai/compare ──function compare(data = {}) {++ if (data.agentPair) return agentCompare(data, data.agentPair);const models = data.models || [];const options = data.options || [];const two = models.length === 2;@@@ -371,6 +379,188 @@ function compare(data = {}) {return page(two ? `${shortName(models[0].id)} vs ${shortName(models[1].id)}` : 'Compare AI models', 'Compare AI models side by side: ratings, prices and measured speed.', body, '/ai/compare');}++// ── Harness and model, apart (/ai/harnesses, /ai/matrix, an agent pair) ──++const agentName = (a) => `<span class="ai-agent"><b>${esc(a.harness)}</b><span aria-hidden="true">+</span>${esc(a.model)}</span>`;++const agentSlug = (a) => `${a.harness}+${a.model}`;++const pairPath = (a, b) => {++ const [x, y] = [agentSlug(a), agentSlug(b)].sort();++ return `/ai/compare/${encodeURIComponent(x)}-vs-${encodeURIComponent(y)}`;++};++const eloText = (x) => value(x) == null ? '—' : String(Math.round(Number(x)));++const pointsText = (x) => { const n = value(x); return n == null ? '—' : `${n > 0 ? '+' : ''}${Math.round(n * 10) / 10}`; };++const agentScoreCell = (a) => {++ const lo = value(a.ci_lower), hi = value(a.ci_upper);++ const spread = lo != null && hi != null ? `${Math.round(lo)}–${Math.round(hi)}` : null;++ return `<b>${eloText(a.score)}</b>${spread ? `<small>95% ${esc(spread)}</small>` : ''}`;++};++++function factorForm(path, data = {}) {++ const kinds = FACTOR_KINDS.map(([id, name]) => `<option value="${id}"${id === (data.taskKind || 'code') ? ' selected' : ''}>${esc(name)}</option>`).join('');++ const days = FACTOR_DAYS.map(([n, name]) => `<option value="${n}"${n === Number(data.days || 90) ? ' selected' : ''}>${esc(name)}</option>`).join('');++ return `<form class="ai-filters ai-factor-form" action="${esc(path)}" method="get"><label class="ai-filter"><span>Task kind</span><select name="task_kind">${kinds}</select></label><label class="ai-filter"><span>Window</span><select name="days">${days}</select></label><button class="ovk-btn ovk-btn--primary">Show</button></form>`;++}++++function factorState(data = {}) {++ if (data.unavailable) return `<p class="ai-note" role="status">OpenVibe.Bench did not answer its factor board. This page comes back when it answers; nothing is filled in.</p>`;++ return data.stale ? benchState(data.stale) : '';++}++++function factorMethod(fit, kind, days) {++ const matches = fit && fit.matches ? `${Number(fit.matches).toLocaleString('en-US')} matches` : 'no matches yet';++ return `<p class="ai-fine">${fit && fit.method ? `${esc(fit.method)}. ` : ''}Model and harness are fitted apart from every match of ${esc(KIND_NAME[kind] || kind)} in ${esc(String(days))} days (${esc(matches)}); the neutral harness <code>ov-bare</code> is the zero when it appears, otherwise the effects are centred on their average. <a href="${BENCH}/docs">Bench's method</a>.</p>`;++}++++// One scale for every harness, so the strips compare by their position: zero is ov-bare when it anchored the fit.++function stripScale(harnesses) {++ const values = harnesses.flatMap((h) => [value(h.ci_lower), value(h.ci_upper), value(h.effect)]).filter((x) => x != null).concat([0]);++ const lo = Math.min(...values), hi = Math.max(...values);++ const pad = Math.max(10, (hi - lo) * 0.06);++ return { lo: lo - pad, hi: hi + pad };++}++function harnessStrip(h, scale) {++ const lo = value(h.ci_lower), hi = value(h.ci_upper), effect = value(h.effect);++ if (lo == null || hi == null || effect == null) return `<p class="ai-fine">Effect interval not reported for ${esc(h.harness)}.</p>`;++ const span = scale.hi - scale.lo || 1;++ const at = (x) => ((x - scale.lo) / span) * 100;++ const label = `${h.harness}: ${pointsText(effect)} Elo effect, 95% interval ${Math.round(lo)} to ${Math.round(hi)}`;++ return `<div class="ai-hstrip" role="img" aria-label="${esc(label)}" title="${esc(label)}" style="--zero:${at(0).toFixed(2)}%">`++ + `<span class="ai-hstrip-range" style="left:${at(lo).toFixed(2)}%;width:${Math.max(0, at(hi) - at(lo)).toFixed(2)}%"></span>`++ + `<span class="ai-hstrip-dot" style="left:${at(effect).toFixed(2)}%"></span></div>`;++}++++// ── /ai/harnesses ──++function harnesses(data = {}) {++ const fit = data.fit || null;++ const list = fit ? fit.harnesses || [] : [];++ const agents = fit ? fit.agents || [] : [];++ const top = [...agents].sort((a, b) => (value(b.score) == null ? -Infinity : value(b.score)) - (value(a.score) == null ? -Infinity : value(a.score)));++ const byHarness = new Map();++ for (const a of agents) {++ if (!byHarness.has(a.harness)) byHarness.set(a.harness, []);++ byHarness.get(a.harness).push(a);++ }++ const scale = stripScale(list);++ const rows = list.map((h) => {++ const own = (byHarness.get(h.harness) || []).sort((a, b) => (value(b.score) == null ? -Infinity : value(b.score)) - (value(a.score) == null ? -Infinity : value(a.score)));++ const other = top.find((x) => !own.some((a) => a.id === x.id)) || null;++ const agentLine = own.length++ ? `<p class="ai-harness-agents">Agents: ${own.map((a) => {++ const body = other ? `<a href="${pairPath(a, other)}">${agentName(a)}</a> ${agentScoreCell(a)}` : `${agentName(a)} ${agentScoreCell(a)}`;++ return `<span class="ai-agent-item">${body}</span>`;++ }).join(' · ')}</p>`++ : `<p class="ai-harness-agents">No agent row names this harness yet.</p>`;++ return `<li class="ai-harness"><div class="ai-harness-head"><b>${esc(h.harness)}</b><span class="ai-harness-effect">${esc(pointsText(h.effect))} Elo</span>`++ + `<span class="ai-harness-meta">95% ${esc(value(h.ci_lower) == null ? '—' : `${Math.round(value(h.ci_lower))} to ${Math.round(value(h.ci_upper))}`)} · ${esc(value(h.games) == null ? 'games not reported' : `${Number(h.games).toLocaleString('en-US')} games`)}</span></div>`++ + harnessStrip(h, scale) + agentLine + `</li>`;++ }).join('');++ const empty = !list.length;++ const body = `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>Harness effects</h1>`++ + `<p class="ai-lede">What a harness adds, in Elo points, apart from the model it drives. Each strip is its 95% interval on one scale; zero is <code>ov-bare</code> when it anchored the fit, otherwise the average harness. Kind and window choose the fit.</p>`++ + factorForm('/ai/harnesses', data) + `</header>`++ + factorState(data)++ + (!fit ? '' : kit.stats([[list.length, list.length === 1 ? 'harness' : 'harnesses'], [fit.games || 0, 'games'], [fit.matches || 0, 'matches'], [agents.length, agents.length === 1 ? 'agent' : 'agents'], [fit.models ? fit.models.length : 0, 'models']], { label: 'The harness board in numbers' }))++ + (empty ? kit.empty({ icon: 'trophy', title: 'No harness board yet', text: 'No harness has enough matches yet: a harness shows after it has run against others on the same tasks.' })++ : kit.section({ title: 'The harness board', count: list.length, body: `<ol class="ai-harness-board">${rows}</ol>${agents.length ? `<p class="ai-fine">An agent is named harness+model. <a href="/ai/matrix?task_kind=${encodeURIComponent(data.taskKind || 'code')}&days=${encodeURIComponent(String(data.days || 90))}">See every harness against every model</a>.</p>` : ''}` }))++ + (fit ? factorMethod(fit, data.taskKind || 'code', data.days || 90) : '');++ return page('Harness effects — AI', 'Each harness\'s effect apart from its model, with its 95% interval and games, from OpenVibe.Bench.', body, '/ai/harnesses', [{ '@context': 'https://schema.org', '@type': 'Dataset', name: 'OpenVibe harness effects', url: 'https://openvibe.compare/ai/harnesses' }]);++}++++// ── /ai/matrix ──++function matrix(data = {}) {++ const fit = data.fit || null;++ const harnessList = fit ? (fit.harnesses || []) : [];++ const agents = fit ? fit.agents || [] : [];++ const models = fit ? [...(fit.models || [])].sort((a, b) => (value(b.score) == null ? -Infinity : value(b.score)) - (value(a.score) == null ? -Infinity : value(a.score))) : [];++ const shownModels = models.slice(0, 8);++ const shownHarnesses = harnessList.slice(0, 12);++ const cellOf = new Map(agents.map((a) => [`${a.harness}\u0000${a.model}`, a]));++ const bestOf = new Map();++ for (const h of shownHarnesses) for (const m of shownModels) {++ const a = cellOf.get(`${h.harness}\u0000${m.model}`);++ if (a && (bestOf.get(m.model) == null || (value(a.score) || -Infinity) > (value(bestOf.get(m.model).score) || -Infinity))) bestOf.set(m.model, a);++ }++ const tableRows = shownHarnesses.map((h) => {++ const row = { harness: `<b>${esc(h.harness)}</b><small>${esc(pointsText(h.effect))} Elo · ${esc(value(h.games) == null ? 'games not reported' : `${Number(h.games).toLocaleString('en-US')} games`)}</small>` };++ for (const m of shownModels) {++ const a = cellOf.get(`${h.harness}\u0000${m.model}`);++ if (!a) { row[`m:${m.model}`] = quiet('not run'); continue; }++ const lo = value(a.ci_lower), hi = value(a.ci_upper);++ const interval = lo != null && hi != null ? `95% interval ${Math.round(lo)}–${Math.round(hi)}` : 'interval not reported';++ const best = bestOf.get(m.model) === a;++ row[`m:${m.model}`] = `<span class="ai-cell" title="${esc(`${a.harness}+${a.model} · ${interval}`)}"><b>${esc(eloText(a.score))}</b><small>${lo != null && hi != null ? `± ${Math.round((hi - lo) / 2)}` : '—'}</small>${best ? ` ${kit.badge('best pair', 'accent')}` : ''}</span>`;++ }++ return row;++ });++ const columns = [{ key: 'harness', label: 'Harness', primary: true, html: true }].concat(shownModels.map((m) => ({ key: `m:${m.model}`, label: m.model, html: true })));++ const notable = agents.filter((a) => Math.abs(value(a.factors.interaction.score) || 0) > 30)++ .sort((a, b) => Math.abs(value(b.factors.interaction.score) || 0) - Math.abs(value(a.factors.interaction.score) || 0));++ const named = notable.slice(0, 8).map((a) => {++ const gain = value(a.factors.interaction.score) || 0;++ return `<b>${esc(a.harness)}</b> ${gain > 0 ? 'gains' : 'loses'} ${esc(Math.abs(Math.round(gain)))} points with ${esc(a.model)}`;++ }).join('; ');++ const interactionLine = notable.length++ ? `<p class="ai-interactions"><b>Interactions over 30 points:</b> ${named}${notable.length > 8 ? `; and ${notable.length - 8} more` : ''}.</p>`++ : `<p class="ai-interactions ai-fine">No interaction is over 30 points: in this fit every harness adds about the same with every model.</p>`;++ const hasCells = shownModels.length > 0 && shownHarnesses.length > 0 && agents.length > 0;++ const empty = !hasCells;++ const body = `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>Harness × model</h1>`++ + `<p class="ai-lede">Every pair is an agent: one harness driving one model. A cell is that agent's score with its 95% interval in the tooltip; “not run” means no match has joined the two yet. The best pair for each model is marked, and interactions over 30 points are named below the table.</p>`++ + factorForm('/ai/matrix', data) + `</header>`++ + factorState(data)++ + (!fit ? '' : kit.stats([[shownHarnesses.length, 'harnesses'], [shownModels.length, 'models'], [agents.length, agents.length === 1 ? 'agent' : 'agents'], [fit.games || 0, 'games']], { label: 'The matrix in numbers' }))++ + (empty ? kit.empty({ icon: 'trophy', title: 'No agent yet', text: 'No harness × model pair has run yet: a cell fills when an agent has run on that harness and model on the same tasks.' })++ : kit.section({ title: 'The matrix', count: agents.length, body: kit.table({ caption: `Score by harness and model · ${KIND_NAME[data.taskKind] || data.taskKind || 'code'} · ${data.days || 90} days`, columns, rows: tableRows, label: 'Harness by model' }) + interactionLine }))++ + (models.length > shownModels.length ? `<p class="ai-fine">The ${shownModels.length} highest-rated models are shown; ${models.length - shownModels.length} more are in the fit.</p>` : '')++ + (fit ? factorMethod(fit, data.taskKind || 'code', data.days || 90) : '');++ return page('Harness × model — AI', 'Every harness and model as an agent, with its score, interval and interaction, from OpenVibe.Bench.', body, '/ai/matrix', [{ '@context': 'https://schema.org', '@type': 'Dataset', name: 'OpenVibe harness by model matrix', url: 'https://openvibe.compare/ai/matrix' }]);++}++++// ── An agent pair: the gap, split ──++function agentCompare(data, pair) {++ const a = pair.a, b = pair.b;++ const gap = value(pair.gap) || 0;++ const lead = gap >= 0 ? a : b;++ const trail = gap >= 0 ? b : a;++ const sign = gap >= 0 ? 1 : -1;++ const part = (k) => { const n = value(pair.parts && pair.parts[k]); return n == null ? 0 : Math.round(sign * n * 10) / 10 || 0; };++ const parts = { model: part('model'), harness: part('harness'), interaction: part('interaction') };++ const gapText = Math.abs(Math.round(gap * 10) / 10);++ const sentence = gap === 0++ ? `${agentName(a)} and ${agentName(b)} are level. Of this 0-point gap, ${pointsText(parts.model)} is the model, ${pointsText(parts.harness)} the harness, ${pointsText(parts.interaction)} the two together.`++ : `${agentName(lead)} is ${gapText} Elo points ahead of ${agentName(trail)}. Of this ${gapText}-point gap, ${pointsText(parts.model)} is the model, ${pointsText(parts.harness)} the harness, ${pointsText(parts.interaction)} the two together.`;++ const negative = Object.values(parts).some((x) => x < 0);++ const sideFacts = (x) => [++ ['Harness', `<b>${esc(x.harness)}</b>`],++ ['Model', `<b>${esc(x.model)}</b>`],++ ['Agent score', agentScoreCell(x)],++ ['Model part', `${eloText(x.factors.model.score)} Elo`],++ ['Harness effect', `${pointsText(x.factors.harness.score)} Elo`],++ ['Interaction', `${pointsText(x.factors.interaction.score)} Elo`],++ ];++ const cards = [a, b].map((x) => `<article class="ai-vs-card"><h2>${agentName(x)}</h2><dl>${sideFacts(x).map(([k, val]) => `<div><dt>${esc(k)}</dt><dd>${val}</dd></div>`).join('')}</dl></article>`).join('');++ const options = data.options || [];++ const list = `<datalist id="ai-models">${options.slice(0, 500).map((o) => `<option value="${esc(o)}"></option>`).join('')}</datalist>`;++ const form = `<form class="ai-filters ai-vs-form" action="/ai/compare"><label class="ai-filter"><span>First entrant</span><input name="a" list="ai-models" value="${esc(agentSlug(a))}" required></label>`++ + `<label class="ai-filter"><span>Second entrant</span><input name="b" list="ai-models" value="${esc(agentSlug(b))}" required></label><button class="ovk-btn ovk-btn--primary">Compare</button>${list}</form>`;++ const split = { label: `Where the ${gapText}-point gap comes from`, parts };++ const canonical = pairPath(a, b);++ const body = `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>${agentName(a)} vs ${agentName(b)}</h1>`++ + `<p class="ai-lede">Two agents on the same fit: the same model and harness contributions are added for each. ${fitNote(pair)}</p></header>`++ + availabilityBlock(data) + `<div class="ai-vs">${cards}</div>`++ + `<section class="ai-gap" aria-labelledby="ai-gap-title"><h2 id="ai-gap-title">The gap, split</h2><p class="ai-gap-say">${sentence}</p>`++ + `<div class="ai-gap-split" id="ai-gap-split"></div>`++ + (negative ? '<p class="ai-fine">A part below zero is drawn by its size; the sign is in the sentence above.</p>' : '')++ + `</section>`++ + form++ + `<p class="ai-fine">The split comes from OpenVibe.Bench's ${esc(KIND_NAME[pair.task_kind] || pair.task_kind || 'code')} fit over ${esc(String(pair.days || 365))} days (${esc(Number(pair.matches || 0).toLocaleString('en-US'))} matches).${pair.method ? ` ${esc(pair.method)}.` : ''} <a href="/ai/harnesses?task_kind=${encodeURIComponent(pair.task_kind || 'code')}&days=${encodeURIComponent(String(pair.days || 365))}">Every harness effect</a> · <a href="/ai/matrix?task_kind=${encodeURIComponent(pair.task_kind || 'code')}&days=${encodeURIComponent(String(pair.days || 365))}">The matrix</a>.</p>`++ + `<script type="application/json" id="ai-gap-data">${jsonData(split)}</script><script src="/ai-factors.js" defer></script>`;++ return page(`${agentSlug(a)} vs ${agentSlug(b)}`, `${agentSlug(a)} vs ${agentSlug(b)}: the score gap split into model, harness and interaction.`, body, canonical, [{ '@context': 'https://schema.org', '@type': 'Dataset', name: `${agentSlug(a)} vs ${agentSlug(b)}`, url: `https://openvibe.compare${canonical}` }]);++}++++function fitNote(pair) {++ return `${esc(KIND_NAME[pair.task_kind] || pair.task_kind || 'Code')} · ${esc(String(pair.days || 365))} days · ${esc(Number(pair.games || 0).toLocaleString('en-US'))} games.`;++}++function arena() {return page('AI Arena', 'Compare anonymous AI answers and vote.', `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>AI Arena</h1><p class="ai-lede">Two anonymous answers to one prompt; you pick the better one, then see which models wrote them. Every vote moves the ratings here.</p></header><p><a class="ovk-btn ovk-btn--primary" href="https://openvibe.exchange/arena">Vote in the Arena</a></p>`, '/ai/arena');}@@@ -610,6 +800,8 @@ function methodPage(data = {}) {${judgeRows.length ? kit.table({ caption: 'Judges in use', columns: [{ key: 'how', label: 'How', html: true }, { key: 'judge', label: 'Bench judge' }, { key: 'jobs', label: 'Task kinds' }], rows: judgeRows }) : `<p class="ai-fine">${esc('Bench did not answer its kind list; the judge rules above are its published contract.')}</p>`}<h2>Verified and submitted</h2><p>A <b>verified</b> run was executed by a service (openvibe.codes, Run, a Node worker); a <b>submitted</b> run was posted with a person’s token. The rankings read verified runs only, and a submitted run is labelled wherever the latest runs are listed.</p>++<h2>Model and harness, apart</h2>++<p>An agent is a harness driving a model, so an agent score has two parts. Bench's factor board fits <code>agent = model + harness + model×harness</code> over the same matches, with partial pooling and a 95% interval on each part; the neutral harness <code>ov-bare</code> is the zero when it appears (otherwise the effects are centred), so a harness's effect is in Elo points against that zero. A negative interaction says the harness adds less with that model than it does on average. <a href="/ai/harnesses">The harness board</a> and <a href="/ai/matrix">the harness × model matrix</a> draw those numbers; an agent pair's page splits its gap into the three parts.</p><h2>How often, and what is missing</h2><p>${o.runs == null ? 'Bench did not answer the public counts.' : `${Number(o.benchmarks || 0).toLocaleString('en-US')} public ${o.benchmarks === 1 ? 'benchmark' : 'benchmarks'}, ${Number(o.tasks || 0).toLocaleString('en-US')} ${o.tasks === 1 ? 'task' : 'tasks'}, ${Number(o.runs || 0).toLocaleString('en-US')} ${o.runs === 1 ? 'run' : 'runs'} from ${Number(o.models || 0).toLocaleString('en-US')} ${o.models === 1 ? 'model' : 'models'}; the newest run was ${o.last_run ? esc(dayTime(o.last_run)) : 'never published'}.`}</p><p class="ai-fine">A model with no verified run has no score here; a benchmark with no run says so. The public reads are Bench’s API at <a href="${BENCH}/docs">bench.openvibe.services/docs</a>; the open data is <a href="${BENCH}/api/v1/datasets/benchmarks.jsonl">benchmarks.jsonl</a> and the per-kind datasets. <a href="/ai/feed.xml">New results (RSS)</a> · <a href="https://openvibe.report/api/v1/ai/results">the same as JSON</a>.</p>`;@@@ -638,6 +830,6 @@ function rss({ runs, overview, base = 'https://openvibe.compare' } = {}) {}module.exports = {-- hub, leaderboard, providers, model, compare, arena, boardPage, benchmarkPage, methodPage, rss,++ hub, leaderboard, providers, model, compare, arena, boardPage, benchmarkPage, methodPage, rss, harnesses, matrix,availabilityBlock, measuredNote, benchState, page, scriptLd, PARTS_CSS,};@@@ -27,9 +27,9 @@ function createApp({getData=createData(),getCompute=createCompute(),ai=createAi(app.get('/api/v1/comparisons',wrap(async(req,res)=>res.json({comparisons:await getData()})));app.get('/api/v1/comparisons/:slug',wrap(async(req,res)=>{const c=(await getData()).find(c=>c.slug===req.params.slug);return c?res.json(c):res.status(404).json({error:'comparison.not_found'});}));app.get('/robots.txt',(_q,r)=>r.type('text').send('User-agent: *\nAllow: /\nAllow: /ai\nSitemap: https://openvibe.compare/sitemap.xml\n'));-- app.get('/llms.txt',wrap(async(_q,r)=>{const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);return r.type('text').send('# OpenVibe.Compare\n\nDated comparisons. Missing evidence is not zero.\n\n'+`- [Compute prices](https://openvibe.compare/compute): GPUs and computers from OpenVibe.Exchange, per hour, per GPU GB-hour and per month at 24/7${compute.checked_at?`; checked ${compute.checked_at.slice(0,10)}`:''}.\n`+`- [AI model leaderboard](https://openvibe.compare/ai): ratings from blind votes, provider prices and measured speed.\n- [AI leaderboard](https://openvibe.compare/ai/leaderboard): every model with filters and sorts.\n- [AI providers](https://openvibe.compare/ai/providers): every provider offer with price, reply time, errors and uptime.\n- [AI method](https://openvibe.compare/ai/method): how a score, a cost and a speed are made.\n- [AI results feed](https://openvibe.compare/ai/feed.xml): new public Bench runs as RSS.\n`+aiData.rows.map(c=>`- [${c.id}](https://openvibe.compare/ai/models/${encodeURIComponent(c.id)}): rating, prices and measured speed.`).join('\n')+'\n'+cs.map(c=>`- [${c.title}](https://openvibe.compare${c.slug==='ai-models'?'/ai':`/comparisons/${c.slug}`}): ${c.summary}`).join('\n')+'\n\nJSON: https://openvibe.compare/api/v1/comparisons\nCompute JSON: https://openvibe.compare/api/v1/compute\nEach page also accepts Accept: text/markdown.\n')}));++ app.get('/llms.txt',wrap(async(_q,r)=>{const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);return r.type('text').send('# OpenVibe.Compare\n\nDated comparisons. Missing evidence is not zero.\n\n'+`- [Compute prices](https://openvibe.compare/compute): GPUs and computers from OpenVibe.Exchange, per hour, per GPU GB-hour and per month at 24/7${compute.checked_at?`; checked ${compute.checked_at.slice(0,10)}`:''}.\n`+`- [AI model leaderboard](https://openvibe.compare/ai): ratings from blind votes, provider prices and measured speed.\n- [AI leaderboard](https://openvibe.compare/ai/leaderboard): every model with filters and sorts.\n- [AI providers](https://openvibe.compare/ai/providers): every provider offer with price, reply time, errors and uptime.\n- [AI method](https://openvibe.compare/ai/method): how a score, a cost and a speed are made.\n- [Harness effects](https://openvibe.compare/ai/harnesses): each harness's Elo effect apart from its model, with an interval.\n- [Harness × model matrix](https://openvibe.compare/ai/matrix): every agent as its harness and model, with interactions.\n- [AI results feed](https://openvibe.compare/ai/feed.xml): new public Bench runs as RSS.\n`+aiData.rows.map(c=>`- [${c.id}](https://openvibe.compare/ai/models/${encodeURIComponent(c.id)}): rating, prices and measured speed.`).join('\n')+'\n'+cs.map(c=>`- [${c.title}](https://openvibe.compare${c.slug==='ai-models'?'/ai':`/comparisons/${c.slug}`}): ${c.summary}`).join('\n')+'\n\nJSON: https://openvibe.compare/api/v1/comparisons\nCompute JSON: https://openvibe.compare/api/v1/compute\nEach page also accepts Accept: text/markdown.\n')}));app.get('/sitemap.xml',wrap(async(_q,r)=>{-- const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);const newest=cs.map(c=>c.updated_at).sort().at(-1);const paths=new Map([['/',newest],['/compute',compute.checked_at||newest],['/ai',aiData.updated_at],['/ai/leaderboard',aiData.updated_at],['/ai/providers',aiData.updated_at],['/ai/method',aiData.updated_at],['/ai/arena',aiData.updated_at]]);++ const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);const newest=cs.map(c=>c.updated_at).sort().at(-1);const paths=new Map([['/',newest],['/compute',compute.checked_at||newest],['/ai',aiData.updated_at],['/ai/leaderboard',aiData.updated_at],['/ai/providers',aiData.updated_at],['/ai/method',aiData.updated_at],['/ai/harnesses',aiData.updated_at],['/ai/matrix',aiData.updated_at],['/ai/arena',aiData.updated_at]]);for(const c of cs){if(c.slug==='ai-models')continue;paths.set(`/comparisons/${c.slug}`,c.updated_at);for(const t of c.things)paths.set(v.thingPath(t.id),c.updated_at);}for(const row of aiData.rows)paths.set(`/ai/models/${encodeURIComponent(row.id)}`,row.source&&row.source.date||aiData.updated_at);for(const task of aiData.tasks)paths.set(`/ai/${encodeURIComponent(task.id)}`,aiData.updated_at);@@@ -44,6 +44,8 @@ function createApp({getData=createData(),getCompute=createCompute(),ai=createAi(app.get('/ai',wrap(async(_q,r)=>r.send(aiViews.hub(await ai.comparison()))));app.get('/ai/leaderboard',wrap(async(req,res)=>res.send(aiViews.leaderboard(await ai.leaderboard(req.query)))));app.get('/ai/providers',wrap(async(_q,r)=>{const data=await ai.comparison();return r.send(aiViews.providers(data));}));++ app.get('/ai/harnesses',wrap(async(req,res)=>res.send(aiViews.harnesses(await ai.factorBoard(req.query)))));++ app.get('/ai/matrix',wrap(async(req,res)=>res.send(aiViews.matrix(await ai.factorBoard(req.query)))));app.get('/ai/arena',(_q,r)=>r.send(aiViews.arena()));app.get('/ai/compare',wrap(async(req,res)=>{// The picker sends a and b; older links send models=a,b.nnew file mode 100644@@@ -0,0 +1,182 @@++'use strict';++++// The harness board, the harness × model matrix and an agent pair: each with a stubbed Bench factor fit, an empty++// fit, and Bench down. A page that cannot read the fit still answers 200 with words, never a blank or a zero.++const test = require('node:test');++const assert = require('node:assert/strict');++const { once } = require('node:events');++const { createApp } = require('../server/app');++const { createAi } = require('../server/ai-data');++const { gapParts } = require('../public/ai-factors');++++const at = '2026-11-13T00:00:00.000Z';++const KINDS = [++ { id: 'code', room: 'codes', judge: 'test_suite', name: 'Code' },++ { id: 'chat', room: 'exchange', judge: 'human_pairwise', name: 'Chat' },++];++const catalogue = {++ generated_at: at,++ models: [++ { slug: 'gpt-5-mini', canonical: 'gpt-5-mini', name: 'gpt-5-mini', kind: 'chat', maker: 'OpenAI', offers: [] },++ { slug: 'glm-5.3', canonical: 'glm-5.3', name: 'glm-5.3', kind: 'chat', maker: 'Z.ai', offers: [] },++ ],++};++const market = { generated_at: at, offers: [] };++const perf = { range: { from: '2026-10-15', to: '2026-11-13' }, rows: [] };++++const fit = {++ task_kind: 'code', category: null, days: 90, matches: 180, games: 180, style_control: true,++ models: [++ { model: 'gpt-5-mini', score: 1050, ci_lower: 1020, ci_upper: 1080, games: 100 },++ { model: 'glm-5.3', score: 1030, ci_lower: 1000, ci_upper: 1060, games: 80 },++ ],++ harnesses: [++ { harness: 'codex', effect: 45, ci_lower: 20, ci_upper: 70, games: 110 },++ { harness: 'opencode', effect: -15, ci_lower: -40, ci_upper: 10, games: 70 },++ { harness: 'aider', effect: 5, ci_lower: -20, ci_upper: 30, games: 20 },++ ],++ agents: [++ { id: 'agent:codex+gpt-5-mini#0123abcd', model: 'gpt-5-mini', harness: 'codex', score: 1095, ci_lower: 1070, ci_upper: 1120, factors: { model: { score: 1050 }, harness: { score: 45 }, interaction: { score: 0 } } },++ { id: 'agent:opencode+gpt-5-mini#0123abcd', model: 'gpt-5-mini', harness: 'opencode', score: 1035, ci_lower: 1010, ci_upper: 1060, factors: { model: { score: 1050 }, harness: { score: -15 }, interaction: { score: 0 } } },++ { id: 'agent:codex+glm-5.3#0123abcd', model: 'glm-5.3', harness: 'codex', score: 1040, ci_lower: 1015, ci_upper: 1065, factors: { model: { score: 1030 }, harness: { score: 45 }, interaction: { score: -35 } } },++ { id: 'agent:opencode+glm-5.3#0123abcd', model: 'glm-5.3', harness: 'opencode', score: 1070, ci_lower: 1045, ci_upper: 1095, factors: { model: { score: 1030 }, harness: { score: -15 }, interaction: { score: 55 } } },++ ],++ method: 'penalised conditional logit: agent = model + harness + model×harness, 150 bootstrap draws, seed 20261113',++};++const emptyFit = { task_kind: 'code', category: null, days: 90, matches: 0, games: 0, style_control: true, models: [], harnesses: [], agents: [], method: fit.method };++++const json = (body) => ({ ok: true, status: 200, headers: { get: () => 'application/json' }, text: async () => JSON.stringify(body), json: async () => body });++++function stub({ down = false, empty = false } = {}) {++ return async (url) => {++ const u = new URL(url);++ const p = u.pathname;++ if (p === '/api/v1/factors') {++ if (down) throw Error('offline');++ const days = Number(u.searchParams.get('days')) || 90;++ const task_kind = u.searchParams.get('task_kind') || 'code';++ return json(empty ? { ...emptyFit, task_kind, days } : { ...fit, task_kind, days });++ }++ if (down) throw Error('offline');++ if (p === '/api/v1/exchange/catalog') return json(catalogue);++ if (p === '/api/v1/exchange/market') return json(market);++ if (p === '/api/v1/ai/performance') return json(perf);++ if (p === '/api/v1/kinds') return json({ count: KINDS.length, kinds: KINDS });++ if (p.startsWith('/api/v1/ratings/')) return json({ board: { kind_id: p.split('/').at(-1), rows: [] } });++ if (p.startsWith('/api/v1/boards/')) return json({ board: { kind_id: p.split('/').at(-1), type: 'elo', rows: [] } });++ return json(null);++ };++}++++async function withServer(opts, fn) {++ const ai = createAi({ fetchImpl: stub(opts), log: { warn() {} } });++ const server = createApp({ ai }).listen(0, '127.0.0.1');++ await once(server, 'listening');++ try {++ await fn(`http://127.0.0.1:${server.address().port}`);++ } finally {++ await new Promise((resolve) => server.close(resolve));++ }++}++++test('the harness board draws each effect with its interval on one scale, games and the agents it appears in', async () => {++ await withServer({}, async (base) => {++ const r = await fetch(`${base}/ai/harnesses?task_kind=code&days=90`);++ assert.equal(r.status, 200);++ const html = await r.text();++ assert.match(html, /Harness effects/);++ assert.match(html, /\+45 Elo/);++ assert.match(html, /95% 20 to 70/);++ assert.match(html, /110 games/);++ assert.match(html, /class="ai-hstrip"/);++ assert.match(html, /--zero:/);++ // Every agent number names its harness.++ assert.match(html, /<b>codex<\/b><span aria-hidden="true">\+<\/span>gpt-5-mini/);++ assert.match(html, /<b>opencode<\/b><span aria-hidden="true">\+<\/span>glm-5\.3/);++ assert.match(html, /95% 1070–1120/);++ assert.match(html, /aider/);++ assert.match(html, /name="task_kind"/);++ assert.match(html, /name="days"/);++ assert.match(html, /neutral harness <code>ov-bare<\/code>/);++ assert.doesNotMatch(html, /No harness has enough matches yet/);++ });++});++++test('the matrix is a labelled table (cards under 600 px), not run cells, the best pair per model and named interactions', async () => {++ await withServer({}, async (base) => {++ const r = await fetch(`${base}/ai/matrix?task_kind=code&days=90`);++ assert.equal(r.status, 200);++ const html = await r.text();++ assert.match(html, /Harness × model/);++ assert.match(html, /ovk-table--stack/);++ assert.match(html, /data-label="gpt-5-mini"/);++ assert.match(html, /title="codex\+gpt-5-mini · 95% interval 1070–1120"/);++ assert.match(html, /best pair/);++ assert.match(html, /not run/);++ assert.match(html, /Interactions over 30 points/);++ assert.match(html, /<b>opencode<\/b> gains 55 points with glm-5\.3/);++ assert.match(html, /<b>codex<\/b> loses 35 points with glm-5\.3/);++ assert.match(html, /Harness × model matrix|Harness × model/);++ });++});++++test('an agent pair splits its gap into the model, the harness and the two together', async () => {++ await withServer({}, async (base) => {++ const r = await fetch(`${base}/ai/compare/codex%2Bgpt-5-mini-vs-opencode%2Bglm-5.3`);++ assert.equal(r.status, 200);++ const html = await r.text();++ assert.match(html, /Of this 25-point gap, \+20 is the model, \+60 the harness, -55 the two together/);++ assert.match(html, /id="ai-gap-data"/);++ assert.match(html, /id="ai-gap-split"/);++ assert.match(html, /\/ai-factors\.js/);++ assert.match(html, /codex<\/b><span aria-hidden="true">\+<\/span>gpt-5-mini/);++ assert.match(html, /OpenVibe\.Bench's Code fit over 365 days/);++ });++});++++test('an empty answer says what fills the board, and the hub links both pages', async () => {++ await withServer({ empty: true }, async (base) => {++ const harnesses = await (await fetch(`${base}/ai/harnesses`)).text();++ assert.match(harnesses, /No harness has enough matches yet: a harness shows after it has run against others on the same tasks\./);++ const matrix = await (await fetch(`${base}/ai/matrix`)).text();++ assert.match(matrix, /No harness × model pair has run yet: a cell fills when an agent has run on that harness and model on the same tasks\./);++ const pair = await fetch(`${base}/ai/compare/codex%2Bgpt-5-mini-vs-opencode%2Bglm-5.3`);++ assert.equal(pair.status, 200);++ assert.doesNotMatch(await pair.text(), /Of this .*point gap/);++ const hub = await (await fetch(`${base}/ai`)).text();++ assert.match(hub, /href="\/ai\/harnesses"/);++ assert.match(hub, /href="\/ai\/matrix"/);++ assert.match(hub, /Harness and model, apart/);++ });++});++++test('Bench down still answers 200 with words on each page', async () => {++ await withServer({ down: true }, async (base) => {++ for (const path of ['/ai/harnesses', '/ai/matrix']) {++ const r = await fetch(base + path);++ assert.equal(r.status, 200, path);++ const html = await r.text();++ assert.match(html, /OpenVibe\.Bench did not answer its factor board\. This page comes back when it answers; nothing is filled in\./, path);++ }++ const pair = await fetch(`${base}/ai/compare/codex%2Bgpt-5-mini-vs-opencode%2Bglm-5.3`);++ assert.equal(pair.status, 200);++ const html = await pair.text();++ assert.match(html, /codex\+gpt-5-mini vs opencode\+glm-5\.3/);++ assert.doesNotMatch(html, /Of this .*point gap/);++ assert.doesNotMatch(html, /id="ai-gap-data"/);++ });++});++++test('the sitemap carries the two pages', async () => {++ await withServer({}, async (base) => {++ const xml = await (await fetch(`${base}/sitemap.xml`)).text();++ assert.match(xml, /https:\/\/openvibe\.compare\/ai\/harnesses/);++ assert.match(xml, /https:\/\/openvibe\.compare\/ai\/matrix/);++ });++});++++test('gapParts hands OVMetrics.split a size per part and keeps the sign', () => {++ const parts = gapParts({ model: 20, harness: 60, interaction: -55 });++ assert.deepEqual(parts.map((p) => [p.key, p.value, p.signed, p.tone]), [['model', 20, 20, 1], ['harness', 60, 60, 2], ['interaction', 55, -55, 3]]);++ assert.equal(gapParts({ model: null, harness: null, interaction: null }).length, 0);++});
@@ -20,9 +20,12 @@ API**. Server-rendered pages; `openvibe-shared` for the shell.
@@ -20,9 +20,12 @@ API**. Server-rendered pages; `openvibe-shared` for the shell.
and `/models/:org/:model` 301 to `/ai/models/:model`.
and `/models/:org/:model` 301 to `/ai/models/:model`.
- `GET /compute` — compute prices from Exchange: filters, sorts and a two/three-offer side-by-side.
- `GET /compute` — compute prices from Exchange: filters, sorts and a two/three-offer side-by-side.
- `GET /ai`, `/ai/leaderboard`, `/ai/providers`, `/ai/models/:model`, `/ai/compare[/:a-vs-:b]`,
- `GET /ai`, `/ai/leaderboard`, `/ai/providers`, `/ai/models/:model`, `/ai/compare[/:a-vs-:b]`,
- `/ai/method`, `/ai/feed.xml`, `/ai/:ref` (a Bench task kind or benchmark) and `/ai/arena` — Report's
- AI comparison pages, with Bench's boards through `openvibe-sdk/bench`, the AI catalogue and Exchange
- market joined, and Report's measured first token, tokens per second and success beside the prices.
+ `/ai/method`, `/ai/feed.xml`, `/ai/harnesses`, `/ai/matrix`, `/ai/:ref` (a Bench task kind or
+ benchmark) and `/ai/arena` — Report's AI comparison pages, with Bench's boards through
+ `openvibe-sdk/bench`, the AI catalogue and Exchange market joined, and Report's measured first
+ token, tokens per second and success beside the prices. `/ai/harnesses` draws each harness's effect
+ against the neutral `ov-bare` with its interval; `/ai/matrix` crosses harness and model; an agent
+ pair (`<harness>+<model>-vs-<harness>+<model>`) splits its gap into model, harness and interaction.
`/comparisons/ai-models` and an AI model's `/things/:id` 301 to the AI pages.
`/comparisons/ai-models` and an AI model's `/things/:id` 301 to the AI pages.
- `GET /api/v1/comparisons[/:slug]` and `GET /api/v1/compute` — the same data as JSON, CORS-allowed
- `GET /api/v1/comparisons[/:slug]` and `GET /api/v1/compute` — the same data as JSON, CORS-allowed
for the origins in [server/origins.json](server/origins.json).
for the origins in [server/origins.json](server/origins.json).
@@ -47,6 +50,7 @@ and the compute book in [server/compute.js](server/compute.js) `SOURCE`:
@@ -47,6 +50,7 @@ and the compute book in [server/compute.js](server/compute.js) `SOURCE`:
| `openvibe.report/api/v1/ai/performance?group=model&days=30` | measured AI time to first token, tokens per second and success rate, public model rows with a 20-call floor (OpenVibe.Report) |
| `openvibe.report/api/v1/ai/performance?group=model&days=30` | measured AI time to first token, tokens per second and success rate, public model rows with a 20-call floor (OpenVibe.Report) |
| `openvibe.exchange/api/compute/prices` | reviewed compute tariffs per provider and region (OpenVibe.Exchange) |
| `openvibe.exchange/api/compute/prices` | reviewed compute tariffs per provider and region (OpenVibe.Exchange) |
| `bench.openvibe.services/api/v1/{kinds,ratings,benchmarks,rankings,runs,overview}` | AI boards, benchmarks and new runs through `openvibe-sdk/bench` (OpenVibe.Bench) |
| `bench.openvibe.services/api/v1/{kinds,ratings,benchmarks,rankings,runs,overview}` | AI boards, benchmarks and new runs through `openvibe-sdk/bench` (OpenVibe.Bench) |
+| `bench.openvibe.services/api/v1/factors?task_kind=&days=` | model and harness apart: harness effects, agents and interactions, with intervals (OpenVibe.Bench) |
| `api.openvibe.exchange/api/v1/exchange/{catalog,market}` | AI catalogue and dated provider prices (OpenVibe.AI / Exchange) |
| `api.openvibe.exchange/api/v1/exchange/{catalog,market}` | AI catalogue and dated provider prices (OpenVibe.AI / Exchange) |
It does not call AI, Live, Network or any authenticated API. Nothing here writes anywhere.
It does not call AI, Live, Network or any authenticated API. Nothing here writes anywhere.
@@ -59,8 +63,9 @@ It does not call AI, Live, Network or any authenticated API. Nothing here writes
@@ -59,8 +63,9 @@ It does not call AI, Live, Network or any authenticated API. Nothing here writes
- **Data:** [server/data.js](server/data.js) — fetches the seven feeds (8 s timeout; 3 s for Report), builds the three
- **Data:** [server/data.js](server/data.js) — fetches the seven feeds (8 s timeout; 3 s for Report), builds the three
comparisons, coalesces concurrent reads and caches for ten minutes; a failed feed becomes `null`.
comparisons, coalesces concurrent reads and caches for ten minutes; a failed feed becomes `null`.
- **AI data:** [server/ai-data.js](server/ai-data.js) joins the catalogue/market and Bench's boards with
- **AI data:** [server/ai-data.js](server/ai-data.js) joins the catalogue/market and Bench's boards with
- Report's measured rows; [server/ai-views.js](server/ai-views.js) renders the Report pages and the Bench boards.
- A failed source leaves a worded status and a dash, never a zero; pages cache for two minutes.
+ Report's measured rows and reads Bench's factor fit for the harness board, the matrix and an agent
+ pair; [server/ai-views.js](server/ai-views.js) renders the Report pages, the Bench boards and the factor
+ pages. A failed source leaves a worded status and a dash, never a zero; pages cache for two minutes.
- **Compute:** [server/compute.js](server/compute.js) normalizes Exchange's tariffs and derives the
- **Compute:** [server/compute.js](server/compute.js) normalizes Exchange's tariffs and derives the
month (730 h) and GPU GB-hour prices; [server/compute-views.js](server/compute-views.js) renders the
month (730 h) and GPU GB-hour prices; [server/compute-views.js](server/compute-views.js) renders the
page, Markdown and Dataset JSON-LD. A failed feed is not cached.
page, Markdown and Dataset JSON-LD. A failed feed is not cached.
@@ -68,6 +73,7 @@ It does not call AI, Live, Network or any authenticated API. Nothing here writes
@@ -68,6 +73,7 @@ It does not call AI, Live, Network or any authenticated API. Nothing here writes
- **Ranking:** [public/rank.js](public/rank.js) — the one ranking implementation, required by both the
- **Ranking:** [public/rank.js](public/rank.js) — the one ranking implementation, required by both the
server views and the browser.
server views and the browser.
- **Browser:** [public/comparison.js](public/comparison.js), [public/compare-account.js](public/compare-account.js),
- **Browser:** [public/comparison.js](public/comparison.js), [public/compare-account.js](public/compare-account.js),
+ [public/ai-factors.js](public/ai-factors.js) (an agent pair's gap split, drawn with `OVMetrics.split`),
[public/compare.css](public/compare.css).
[public/compare.css](public/compare.css).
- **Deploy:** [deploy/systemd/openvibe-compare.service](deploy/systemd/openvibe-compare.service),
- **Deploy:** [deploy/systemd/openvibe-compare.service](deploy/systemd/openvibe-compare.service),
[deploy/nginx/openvibe.compare.conf](deploy/nginx/openvibe.compare.conf).
[deploy/nginx/openvibe.compare.conf](deploy/nginx/openvibe.compare.conf).
@@ -5,7 +5,7 @@ Compare v1 serves public comparisons at https://openvibe.compare. Run `npm ci`,
@@ -5,7 +5,7 @@ Compare v1 serves public comparisons at https://openvibe.compare. Run `npm ci`,
Data: Exchange price-book market and its reviewed compute tariffs, Report's public AI performance rows (30 days, model group), Bench public arena/verified runs, Codes harness catalog, Billing public plans. Reads are coalesced and cached for ten minutes. Price provenance retains the provider's checked date; a board without an update timestamp labels its date as the observation date. Failed feeds produce explicit unavailable states. The AI models comparison shows Report's measured time to first token p50, tokens per second p50 and success rate beside each provider's price; a model under 20 calls or with Report down is a dash, never zero. There is no private data or paid execution. Compute prices are read from OpenVibe.Exchange's `GET /api/compute/prices`; Compare adds only derived arithmetic (per month at 24/7, per GPU GB-hour), never a copied tariff.
Data: Exchange price-book market and its reviewed compute tariffs, Report's public AI performance rows (30 days, model group), Bench public arena/verified runs, Codes harness catalog, Billing public plans. Reads are coalesced and cached for ten minutes. Price provenance retains the provider's checked date; a board without an update timestamp labels its date as the observation date. Failed feeds produce explicit unavailable states. The AI models comparison shows Report's measured time to first token p50, tokens per second p50 and success rate beside each provider's price; a model under 20 calls or with Report down is a dash, never zero. There is no private data or paid execution. Compute prices are read from OpenVibe.Exchange's `GET /api/compute/prices`; Compare adds only derived arithmetic (per month at 24/7, per GPU GB-hour), never a copied tariff.
-Pages: `/`, `/comparisons/:slug`, `/things/:id` (URI-encoded identity), `/compute` (filters by kind, GPU model, region and interruptible; sorts by price per hour, per GPU GB-hour and per month at 24/7; a two- or three-offer side-by-side), and the AI pages `/ai` (hub), `/ai/leaderboard`, `/ai/providers`, `/ai/models/:model`, `/ai/compare` and `/ai/compare/:a-vs-:b` (canonical sorted pair), `/ai/method`, `/ai/feed.xml`, `/ai/:ref` (a Bench task kind or benchmark) and `/ai/arena`. The old `/comparisons/ai-models` and an AI model's `/things/:id` 301 to the AI pages. AI ratings and prices come from OpenVibe.AI's catalogue and Exchange market joined to OpenVibe.Bench's boards through `openvibe-sdk/bench`; Report's measured rows sit beside the prices. JSON at `/api/v1/comparisons[/slug]` and `/api/v1/compute`. Pages negotiate `Accept: text/markdown`. Dataset JSON-LD, sitemap and llms.txt are generated from the same data. Desktop tables and phone cards share ranking logic; compute cards run to 899 px and its table starts at 900 px. Weight URL parameters are `w.<criterion>`; all-zero weights leave things unranked. Builder and version-history fields are reserved, with no write API in v1.
+Pages: `/`, `/comparisons/:slug`, `/things/:id` (URI-encoded identity), `/compute` (filters by kind, GPU model, region and interruptible; sorts by price per hour, per GPU GB-hour and per month at 24/7; a two- or three-offer side-by-side), and the AI pages `/ai` (hub), `/ai/leaderboard`, `/ai/providers`, `/ai/models/:model`, `/ai/compare` and `/ai/compare/:a-vs-:b` (canonical sorted pair; two agents get a model/harness/interaction split), `/ai/harnesses` (each harness's effect with its interval), `/ai/matrix` (harness × model), `/ai/method`, `/ai/feed.xml`, `/ai/:ref` (a Bench task kind or benchmark) and `/ai/arena`. The old `/comparisons/ai-models` and an AI model's `/things/:id` 301 to the AI pages. AI ratings and prices come from OpenVibe.AI's catalogue and Exchange market joined to OpenVibe.Bench's boards through `openvibe-sdk/bench`; Report's measured rows sit beside the prices. JSON at `/api/v1/comparisons[/slug]` and `/api/v1/compute`. Pages negotiate `Accept: text/markdown`. Dataset JSON-LD, sitemap and llms.txt are generated from the same data. Desktop tables and phone cards share ranking logic; compute cards run to 899 px and its table starts at 900 px. Weight URL parameters are `w.<criterion>`; all-zero weights leave things unranked. Builder and version-history fields are reserved, with no write API in v1.
Launch: Host register-compare.py registers port 5280 with a lifecycle override until the separate Contracts manifest is released. Use merge-deploy.sh, then ov-vhost-install for deploy/nginx/openvibe.compare.conf. No database or service credentials are required.
Launch: Host register-compare.py registers port 5280 with a lifecycle override until the separate Contracts manifest is released. Use merge-deploy.sh, then ov-vhost-install for deploy/nginx/openvibe.compare.conf. No database or service credentials are required.
@@ -129,3 +129,51 @@
@@ -129,3 +129,51 @@
.ai-page .ai-head h1{overflow-wrap:anywhere}
.ai-page .ai-head h1{overflow-wrap:anywhere}
.ai-page .ov-parts-bar label{min-height:44px}
.ai-page .ov-parts-bar label{min-height:44px}
}
}
+
+/* Harness effects: one scale for every harness. The range is its 95% interval, the dot its effect, the line zero. */
+.ai-harness-board{list-style:none;margin:0;padding:0;display:grid;gap:12px}
+.ai-harness{display:grid;gap:10px;padding:14px 16px;border:1px solid var(--border);border-radius:14px;background:var(--bg-card);min-width:0}
+.ai-harness-head{display:flex;flex-wrap:wrap;align-items:baseline;gap:2px 12px;min-width:0}
+.ai-harness-head>b{font-size:1.05rem;overflow-wrap:anywhere}
+.ai-harness-effect{font-weight:700;font-variant-numeric:tabular-nums}
+.ai-harness-meta{color:var(--text-muted);font-size:.82rem;font-variant-numeric:tabular-nums;overflow-wrap:anywhere;min-width:0}
+.ai-hstrip{position:relative;height:26px;min-width:0;margin:2px 8px;border-radius:999px;background:var(--bg-tertiary,color-mix(in srgb,var(--text-primary) 8%,transparent))}
+.ai-hstrip::before{content:"";position:absolute;top:0;bottom:0;left:var(--zero,50%);width:1px;background:var(--text-muted);opacity:.55}
+.ai-hstrip-range{position:absolute;top:8px;height:10px;border-radius:999px;background:color-mix(in srgb,var(--accent) 32%,transparent)}
+.ai-hstrip-dot{position:absolute;top:5px;width:16px;height:16px;margin-left:-8px;border-radius:50%;background:var(--accent);border:2px solid var(--bg-card);box-sizing:border-box}
+.ai-harness-agents{margin:0;display:flex;flex-wrap:wrap;gap:4px 12px;align-items:baseline;font-size:.86rem;color:var(--text-secondary);min-width:0}
+.ai-agent-item{display:inline-flex;flex-wrap:wrap;gap:2px 6px;align-items:baseline;min-width:0}
+.ai-agent-item>a{color:var(--text-primary);text-decoration:none;min-width:0}
+.ai-agent-item>a:hover{color:var(--accent);text-decoration:underline}
+.ai-agent{display:inline-flex;flex-wrap:wrap;gap:1px;align-items:baseline;min-width:0;overflow-wrap:anywhere}
+.ai-agent>b{font-weight:700}
+.ai-agent>span[aria-hidden]{color:var(--text-muted);margin:0 1px}
+.ai-harness-agents b{font-variant-numeric:tabular-nums}
+.ai-harness-agents small{display:block;color:var(--text-muted);font-size:.78rem;font-variant-numeric:tabular-nums}
+
+/* The matrix: a cell is an agent's score, its interval in words and the tooltip; the best pair per model is marked. */
+.ai-interactions{margin:12px 0 0;padding:10px 12px;border:1px solid var(--border);border-radius:12px;background:var(--bg-card);font-size:.88rem;line-height:1.5;min-width:0;overflow-wrap:anywhere}
+.ai-interactions b{font-variant-numeric:tabular-nums}
+.ai-page .ovk-table td .ovk-badge{margin-left:4px;font-size:.7rem;white-space:nowrap}
+@media (max-width:600px){
+ .ai-page .ovk-table--stack td .ovk-badge{margin-left:0;display:inline-block;margin-top:4px}
+ .ai-harness{padding:12px}
+ .ai-hstrip{margin:2px 0}
+}
+
+/* An agent pair: the gap split into model, harness and interaction, drawn by OVMetrics.split. */
+.ai-gap{display:grid;gap:10px;padding:16px;border:1px solid var(--border);border-radius:16px;background:var(--bg-card);min-width:0}
+.ai-gap h2{margin:0;font-size:1.05rem}
+.ai-gap-say{margin:0;font-size:1.02rem;line-height:1.55;overflow-wrap:anywhere}
+.ai-gap-split{min-width:0}
+.ai-vs-card h2{margin:0;font-size:1.08rem;line-height:1.3;overflow-wrap:anywhere}
+@media (max-width:600px){.ai-gap{padding:14px}.ai-gap-say{font-size:.98rem}}
+
+/* The factor pages' own numbers: a grid of tiles, never a row that a phone has to scroll sideways to read. */
+.ai-page .ovk-stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,140px),1fr));overflow:visible}
+.ai-page .ovk-table th small{display:block;margin-top:2px;color:var(--text-muted);font-size:.78rem;font-weight:400;font-variant-numeric:tabular-nums}
+.ai-vs-card dd small{display:block;margin-top:2px;color:var(--text-muted);font-size:.78rem;font-weight:400;font-variant-numeric:tabular-nums}
+@media (max-width:600px){
+ .ai-vs-form .ai-filter{grid-column:1/-1}
+ .ai-page .ovk-stats{grid-template-columns:repeat(auto-fit,minmax(min(100%,120px),1fr))}
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,44 @@
@@ -0,0 +1,44 @@
+/* The agent pair's gap split, drawn with openvibe-shared/ov-metrics.js (OVMetrics.split), never with a chart of
+ our own. The page writes the three signed parts; this file hands their sizes to the split bar and keeps the sign
+ in the sentence beside it. Without JavaScript the sentence is the whole answer. */
+(function (root, factory) {
+ const api = factory(typeof document !== 'undefined' ? document : null);
+ if (typeof module === 'object' && module.exports) module.exports = api;
+ else root.AiFactors = api;
+})(typeof globalThis !== 'undefined' ? globalThis : this, function (doc) {
+ 'use strict';
+ const LABELS = [['model', 'the model'], ['harness', 'the harness'], ['interaction', 'the two together']];
+ const num = (v) => (v == null || !Number.isFinite(Number(v)) ? null : Number(v));
+
+ /** [{ key, label, value, signed, tone }] — value is the size the bar draws, signed is the direction. */
+ function gapParts(parts = {}) {
+ return LABELS.map(([key, label], i) => {
+ const signed = num(parts[key]);
+ return { key, label, value: signed == null ? null : Math.abs(signed), signed, tone: i + 1 };
+ }).filter((p) => p.value != null);
+ }
+
+ function draw(d, data, mount) {
+ const win = d && d.defaultView;
+ const OVMetrics = win && win.OVMetrics;
+ if (!OVMetrics || !mount || !data || !data.parts) return null;
+ const parts = gapParts(data.parts);
+ if (!parts.some((p) => p.value > 0)) return null;
+ const split = OVMetrics.split(parts.map((p) => ({ label: p.label, value: p.value, tone: p.tone })), { label: data.label || 'Where the gap comes from' });
+ mount.replaceChildren(split);
+ return split;
+ }
+
+ function mount() {
+ const data = doc.getElementById('ai-gap-data');
+ const target = doc.getElementById('ai-gap-split');
+ if (!data || !target) return;
+ try { draw(doc, JSON.parse(data.textContent), target); } catch { /* a page without a gap is fine */ }
+ }
+
+ if (doc) {
+ if (doc.readyState === 'loading') doc.addEventListener('DOMContentLoaded', mount, { once: true });
+ else mount();
+ }
+ return { gapParts, draw };
+});
@@ -23,11 +23,23 @@ const BENCHMARK_RE = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;
@@ -23,11 +23,23 @@ const BENCHMARK_RE = /^bmk_[0-9A-HJKMNP-TV-Z]{26}$/;
const MODEL_RE = /^[A-Za-z0-9][A-Za-z0-9._:+\/~\-]{0,119}$/;
const MODEL_RE = /^[A-Za-z0-9][A-Za-z0-9._:+\/~\-]{0,119}$/;
const BOARD_TYPES = Object.freeze(['elo', 'pareto', 'cost']);
const BOARD_TYPES = Object.freeze(['elo', 'pareto', 'cost']);
const METRICS = Object.freeze(['score', 'value', 'speed', 'consistency', 'horizon']);
const METRICS = Object.freeze(['score', 'value', 'speed', 'consistency', 'horizon']);
+const FACTOR_KINDS = Object.freeze(['chat', 'code', 'design', 'image', 'speech', 'search', 'computer', 'game']);
+const FACTOR_DAYS = Object.freeze([7, 30, 90, 365]);
const isKindId = (value) => typeof value === 'string' && KIND_RE.test(value);
const isKindId = (value) => typeof value === 'string' && KIND_RE.test(value);
const isBenchmarkId = (value) => typeof value === 'string' && BENCHMARK_RE.test(value);
const isBenchmarkId = (value) => typeof value === 'string' && BENCHMARK_RE.test(value);
const isModelName = (value) => typeof value === 'string' && MODEL_RE.test(value);
const isModelName = (value) => typeof value === 'string' && MODEL_RE.test(value);
+function normaliseFactorKind(value) {
+ const raw = Array.isArray(value) ? value[0] : value;
+ return FACTOR_KINDS.includes(raw) ? raw : 'code';
+}
+
+function normaliseFactorDays(value) {
+ const n = Number(Array.isArray(value) ? value[0] : value);
+ return FACTOR_DAYS.includes(n) ? n : 90;
+}
+
function normaliseMetric(value) {
function normaliseMetric(value) {
const raw = Array.isArray(value) ? value[0] : value;
const raw = Array.isArray(value) ? value[0] : value;
return METRICS.includes(raw) ? raw : 'score';
return METRICS.includes(raw) ? raw : 'score';
@@ -137,12 +149,25 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = co
@@ -137,12 +149,25 @@ function createBenchClient({ config = {}, fetchImpl = globalThis.fetch, log = co
return read(models, `detail:${model}`, `/api/v1/models/${encodeURIComponent(model)}/summary`, `bench model detail (${model})`, 200);
return read(models, `detail:${model}`, `/api/v1/models/${encodeURIComponent(model)}/summary`, `bench model detail (${model})`, 200);
}
}
+ // Model and harness apart (bench.rating@1 1.1.0 factors): one fit per task kind, category and window.
+ async function factors({ taskKind = 'code', category = null, days = 90 } = {}) {
+ const kind = normaliseFactorKind(taskKind);
+ const span = normaliseFactorDays(days);
+ const cat = category ? String(category).slice(0, 64) : null;
+ const params = new URLSearchParams({ task_kind: kind, days: String(span) });
+ if (cat) params.set('category', cat);
+ const query = params.toString();
+ return read(results, `factors:${query}`, `/api/v1/factors?${query}`, `bench factors (${query})`, 500);
+ }
+
return {
return {
- kinds, board, modelSummary, benchmarks, benchmark, overview, rankings, runs, modelDetail,
+ kinds, board, modelSummary, benchmarks, benchmark, overview, rankings, runs, modelDetail, factors,
baseUrl, ttlMs: TTL_MS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,
baseUrl, ttlMs: TTL_MS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,
+ normaliseFactorKind, normaliseFactorDays,
};
};
}
}
module.exports = {
module.exports = {
- createBenchClient, TTL_MS, BOARD_TYPES, METRICS, isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit,
+ createBenchClient, TTL_MS, BOARD_TYPES, METRICS, FACTOR_KINDS, FACTOR_DAYS,
+ isKindId, isBenchmarkId, isModelName, normaliseTab, normaliseMetric, normaliseLimit, normaliseFactorKind, normaliseFactorDays,
};
};
@@ -6,7 +6,7 @@
@@ -6,7 +6,7 @@
const { createAiCatalogue } = require('./ai-catalogue');
const { createAiCatalogue } = require('./ai-catalogue');
const { comparisonCatalogue } = require('./ai-market');
const { comparisonCatalogue } = require('./ai-market');
const { findCatalogModel, makerOf } = require('./ai-join');
const { findCatalogModel, makerOf } = require('./ai-join');
-const { createBenchClient, isKindId, isModelName, normaliseTab } = require('./ai-bench');
+const { createBenchClient, isKindId, isModelName, normaliseTab, normaliseFactorKind, normaliseFactorDays, FACTOR_KINDS } = require('./ai-bench');
const { requestJson } = require('./ai-source');
const { requestJson } = require('./ai-source');
const { measuredRows, measuredOf } = require('./data');
const { measuredRows, measuredOf } = require('./data');
@@ -106,6 +106,59 @@ function buildComparison({ kindsRead = null, boards = [], catalogue = null, perf
@@ -106,6 +106,59 @@ function buildComparison({ kindsRead = null, boards = [], catalogue = null, perf
return { rows, providers: catOffers, tasks, catalogue, availability, stale, measured_at: perfAt, updated_at: dates.sort().at(-1) || iso(now) };
return { rows, providers: catOffers, tasks, catalogue, availability, stale, measured_at: perfAt, updated_at: dates.sort().at(-1) || iso(now) };
}
}
+// ── Model and harness, apart ──
+// Bench's factor fit (bench.rating@1 1.1.0 `factors`) is the only source that can say which half of an agent's
+// score is the model and which is the harness. Its numbers are kept as read; a missing fit is null, never zero.
+const factorList = (x) => (Array.isArray(x) ? x : []);
+const factorNumber = (x) => x == null || !Number.isFinite(Number(x)) ? null : Number(x);
+const agentLabel = (id) => String(id || '').replace(/^agent:/, '').replace(/#.*$/, '');
+const looksLikeAgent = (id) => String(id || '').includes('+');
+const round1 = (x) => factorNumber(x) == null ? null : Math.round(factorNumber(x) * 10) / 10;
+
+function normaliseFactors(read) {
+ const raw = read && read.data && typeof read.data === 'object' ? read.data : null;
+ if (!raw) return null;
+ const models = factorList(raw.models).map((m) => ({
+ model: String(m.model || ''), score: factorNumber(m.score), ci_lower: factorNumber(m.ci_lower), ci_upper: factorNumber(m.ci_upper), games: factorNumber(m.games),
+ })).filter((m) => m.model);
+ const harnesses = factorList(raw.harnesses).map((h) => ({
+ harness: String(h.harness || ''), effect: factorNumber(h.effect), ci_lower: factorNumber(h.ci_lower), ci_upper: factorNumber(h.ci_upper), games: factorNumber(h.games),
+ })).filter((h) => h.harness);
+ const agents = factorList(raw.agents).map((a) => {
+ const f = a.factors && typeof a.factors === 'object' ? a.factors : {};
+ const part = (p) => ({ score: factorNumber(p && p.score) });
+ return {
+ id: String(a.id || ''), model: String(a.model || ''), harness: String(a.harness || ''),
+ score: factorNumber(a.score), ci_lower: factorNumber(a.ci_lower), ci_upper: factorNumber(a.ci_upper),
+ factors: { model: part(f.model), harness: part(f.harness), interaction: part(f.interaction) },
+ };
+ }).filter((a) => a.id && a.model && a.harness);
+ return {
+ task_kind: raw.task_kind || null, category: raw.category || null, days: factorNumber(raw.days),
+ matches: factorNumber(raw.matches) || 0, games: factorNumber(raw.games) || 0, style_control: raw.style_control !== false,
+ models, harnesses, agents, method: typeof raw.method === 'string' ? raw.method : null,
+ };
+}
+
+function pairAgent(fit, side) {
+ const wanted = String(side || '');
+ if (!fit || !wanted) return null;
+ return fit.agents.find((a) => a.id === wanted || a.id === `agent:${wanted}` || agentLabel(a.id) === wanted || `${a.harness}+${a.model}` === wanted) || null;
+}
+
+/** The gap between two agents in one fit, split into its model, harness and interaction parts (Elo points). */
+function splitOf(fit, a, b, taskKind) {
+ const A = pairAgent(fit, a);
+ const B = pairAgent(fit, b);
+ if (!A || !B || A.score == null || B.score == null) return null;
+ const d = (part) => round1((A.factors[part].score == null ? 0 : A.factors[part].score) - (B.factors[part].score == null ? 0 : B.factors[part].score));
+ return {
+ task_kind: taskKind || fit.task_kind, days: fit.days, games: fit.games, matches: fit.matches, method: fit.method,
+ a: A, b: B, gap: round1(A.score - B.score),
+ parts: { model: d('model'), harness: d('harness'), interaction: d('interaction') },
+ };
+}
+
function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new Date().toISOString(), ttl = TTL_MS, bench = null, catalogue = null } = {}) {
function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new Date().toISOString(), ttl = TTL_MS, bench = null, catalogue = null } = {}) {
const clock = typeof now === 'function' ? now : () => new Date().toISOString();
const clock = typeof now === 'function' ? now : () => new Date().toISOString();
const benchClient = bench || createBenchClient({ fetchImpl, log, now: () => Date.parse(clock()) || Date.now() });
const benchClient = bench || createBenchClient({ fetchImpl, log, now: () => Date.parse(clock()) || Date.now() });
@@ -182,8 +235,37 @@ function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new
@@ -182,8 +235,37 @@ function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new
}
}
async function compare(models = []) {
async function compare(models = []) {
- const data = await comparison();
- return { models: models.map((id) => data.rows.find((x) => x.id === id) || { id, reason: 'No public model data.' }), options: data.rows.map((x) => x.id), availability: data.availability, stale: data.stale, measured_at: data.measured_at };
+ const [data, agentPair] = await Promise.all([
+ comparison(),
+ models.length === 2 ? agentSplit(models[0], models[1]) : Promise.resolve(null),
+ ]);
+ return { models: models.map((id) => data.rows.find((x) => x.id === id) || { id, reason: 'No public model data.' }), options: data.rows.map((x) => x.id), availability: data.availability, stale: data.stale, measured_at: data.measured_at, agentPair };
+ }
+
+ // The harness board and the matrix read the same fit; the pair page reads it only when both sides look like
+ // agents (`harness+model`). One failed first read stops the search: Bench is down, not merely missing a pair.
+ async function factorFit(taskKind, days) {
+ const read = await benchClient.factors({ taskKind, days });
+ return { fit: normaliseFactors(read), read };
+ }
+
+ async function factorBoard(query = {}) {
+ const taskKind = normaliseFactorKind(query.task_kind);
+ const days = normaliseFactorDays(query.days);
+ const { fit, read } = await factorFit(taskKind, days);
+ return { taskKind, days, fit, unavailable: !read, stale: read && read.stale ? read.at : null };
+ }
+
+ async function agentSplit(a, b) {
+ if (!looksLikeAgent(a) || !looksLikeAgent(b)) return null;
+ const first = await factorFit('code', 365);
+ if (!first.read) return null;
+ const direct = first.fit ? splitOf(first.fit, a, b, 'code') : null;
+ if (direct) return direct;
+ const rest = FACTOR_KINDS.filter((kind) => kind !== 'code');
+ const fits = await Promise.all(rest.map(async (kind) => ({ kind, ...(await factorFit(kind, 365)) })));
+ const found = fits.filter((x) => x.fit && pairAgent(x.fit, a) && pairAgent(x.fit, b)).sort((x, y) => (y.fit.games || 0) - (x.fit.games || 0))[0];
+ return found ? splitOf(found.fit, a, b, found.kind) : null;
}
}
async function leaderboard(query = {}) {
async function leaderboard(query = {}) {
@@ -214,10 +296,10 @@ function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new
@@ -214,10 +296,10 @@ function createAi({ fetchImpl = globalThis.fetch, log = console, now = () => new
}
}
return {
return {
- comparison, kindBoard, benchmarkPage, method, feed, modelPage, compare, leaderboard,
+ comparison, kindBoard, benchmarkPage, method, feed, modelPage, compare, leaderboard, factorBoard, agentSplit,
bench: benchClient, catalogue: catClient, readRows, ttlMs: ttl,
bench: benchClient, catalogue: catClient, readRows, ttlMs: ttl,
isKindId: benchClient.isKindId, isBenchmarkId: benchClient.isBenchmarkId, isModelName: benchClient.isModelName, normaliseTab: benchClient.normaliseTab,
isKindId: benchClient.isKindId, isBenchmarkId: benchClient.isBenchmarkId, isModelName: benchClient.isModelName, normaliseTab: benchClient.normaliseTab,
};
};
}
}
-module.exports = { createAi, buildComparison, measuredRow, TTL_MS, PERF_URL };
+module.exports = { createAi, buildComparison, measuredRow, normaliseFactors, splitOf, agentLabel, TTL_MS, PERF_URL };
@@ -29,6 +29,12 @@ const word = { ms: metrics.format.ms, tps: metrics.format.tps, rate: metrics.for
@@ -29,6 +29,12 @@ const word = { ms: metrics.format.ms, tps: metrics.format.tps, rate: metrics.for
const measuredWord = { ttft: metrics.format.ms, tps: metrics.format.tps, success: metrics.format.rate };
const measuredWord = { ttft: metrics.format.ms, tps: metrics.format.tps, success: metrics.format.rate };
const sourceHint = (s) => { const x = s && typeof s === 'object' ? s : { name: s }; return `Source: ${x.name || 'not listed'}${x.date ? ` · ${String(x.date).slice(0, 10)}` : ''}${x.sample_size != null ? ` · n = ${x.sample_size}` : ''}`; };
const sourceHint = (s) => { const x = s && typeof s === 'object' ? s : { name: s }; return `Source: ${x.name || 'not listed'}${x.date ? ` · ${String(x.date).slice(0, 10)}` : ''}${x.sample_size != null ? ` · n = ${x.sample_size}` : ''}`; };
const readData = (r) => (r && r.data != null ? r.data : null);
const readData = (r) => (r && r.data != null ? r.data : null);
+const jsonData = (x) => JSON.stringify(x).replace(/</g, '\\u003c');
+
+// Bench's factor board covers these task kinds and windows; the page's form offers exactly these.
+const FACTOR_KINDS = [['code', 'Code'], ['chat', 'Chat'], ['design', 'Design'], ['image', 'Image'], ['speech', 'Speech'], ['search', 'Search'], ['computer', 'Computer use'], ['game', 'Games']];
+const FACTOR_DAYS = [[7, '7 days'], [30, '30 days'], [90, '90 days'], [365, '365 days']];
+const KIND_NAME = Object.fromEntries(FACTOR_KINDS);
const HEAD = `<link rel="stylesheet" href="/ai-comparison.css"><link rel="stylesheet" href="/shared/kit-data.css"><link rel="stylesheet" href="/shared/showcase.css"><style>${marks.CSS}\n${PARTS_CSS}</style>`;
const HEAD = `<link rel="stylesheet" href="/ai-comparison.css"><link rel="stylesheet" href="/shared/kit-data.css"><link rel="stylesheet" href="/shared/showcase.css"><style>${marks.CSS}\n${PARTS_CSS}</style>`;
function page(title, description, body, path, ld = []) {
function page(title, description, body, path, ld = []) {
@@ -209,6 +215,7 @@ function hub(data = {}) {
@@ -209,6 +215,7 @@ function hub(data = {}) {
+ kit.stats([[ratedRows.length, 'rated'], [rows.length, 'models listed'], [providerNames.size, 'providers'], [measuredOffers.length, 'measured endpoints']], { label: 'The leaderboard in numbers' })
+ kit.stats([[ratedRows.length, 'rated'], [rows.length, 'models listed'], [providerNames.size, 'providers'], [measuredOffers.length, 'measured endpoints']], { label: 'The leaderboard in numbers' })
+ `</header>`
+ `</header>`
+ (pickList || pending.length ? kit.section({ title: 'Top picks', body: `${pickList ? `<ul class="ai-picks">${pickList}</ul>` : ''}${pending.length ? `<p class="ai-fine">Not enough data yet for ${esc(pending.join(', '))}.</p>` : ''}` }) : '')
+ (pickList || pending.length ? kit.section({ title: 'Top picks', body: `${pickList ? `<ul class="ai-picks">${pickList}</ul>` : ''}${pending.length ? `<p class="ai-fine">Not enough data yet for ${esc(pending.join(', '))}.</p>` : ''}` }) : '')
+ + kit.section({ title: 'Harness and model, apart', body: `<p class="ai-fine">A coding agent is a harness driving a model, and the harness can move a score as much as the model. Bench fits the two apart, with an interval on each harness effect. Every agent number here names its harness.</p><ul class="ai-chips"><li><a href="/ai/harnesses">Harness effects</a></li><li><a href="/ai/matrix">Harness × model matrix</a></li><li><a href="/ai/method">How the fit works</a></li></ul>` })
// A phone shows the parts below the top picks one at a time as tabs; every part stays in the HTML.
// A phone shows the parts below the top picks one at a time as tabs; every part stays in the HTML.
+ parts({ name: 'board', label: 'The leaderboard\'s parts', items: [
+ parts({ name: 'board', label: 'The leaderboard\'s parts', items: [
{ id: 'leaderboard', label: 'Leaderboard', count: chat.length, html: kit.section({ title: 'Leaderboard', count: chat.length, actions: [act('/ai/leaderboard', chat.length > 25 ? `All ${chat.length} models` : 'Filter and sort')], body: board(chat.slice(0, 25)) + (others ? `<p class="ai-fine">Chat models only. ${others} embedding and speech ${others === 1 ? 'model is' : 'models are'} in the <a href="/ai/leaderboard?type=all">full list</a>.</p>` : '') + dataLinks() + measuredNote(data) }) },
{ id: 'leaderboard', label: 'Leaderboard', count: chat.length, html: kit.section({ title: 'Leaderboard', count: chat.length, actions: [act('/ai/leaderboard', chat.length > 25 ? `All ${chat.length} models` : 'Filter and sort')], body: board(chat.slice(0, 25)) + (others ? `<p class="ai-fine">Chat models only. ${others} embedding and speech ${others === 1 ? 'model is' : 'models are'} in the <a href="/ai/leaderboard?type=all">full list</a>.</p>` : '') + dataLinks() + measuredNote(data) }) },
@@ -349,6 +356,7 @@ function model(data = {}) {
@@ -349,6 +356,7 @@ function model(data = {}) {
// ── /ai/compare ──
// ── /ai/compare ──
function compare(data = {}) {
function compare(data = {}) {
+ if (data.agentPair) return agentCompare(data, data.agentPair);
const models = data.models || [];
const models = data.models || [];
const options = data.options || [];
const options = data.options || [];
const two = models.length === 2;
const two = models.length === 2;
@@ -371,6 +379,188 @@ function compare(data = {}) {
@@ -371,6 +379,188 @@ function compare(data = {}) {
return page(two ? `${shortName(models[0].id)} vs ${shortName(models[1].id)}` : 'Compare AI models', 'Compare AI models side by side: ratings, prices and measured speed.', body, '/ai/compare');
return page(two ? `${shortName(models[0].id)} vs ${shortName(models[1].id)}` : 'Compare AI models', 'Compare AI models side by side: ratings, prices and measured speed.', body, '/ai/compare');
}
}
+// ── Harness and model, apart (/ai/harnesses, /ai/matrix, an agent pair) ──
+const agentName = (a) => `<span class="ai-agent"><b>${esc(a.harness)}</b><span aria-hidden="true">+</span>${esc(a.model)}</span>`;
+const agentSlug = (a) => `${a.harness}+${a.model}`;
+const pairPath = (a, b) => {
+ const [x, y] = [agentSlug(a), agentSlug(b)].sort();
+ return `/ai/compare/${encodeURIComponent(x)}-vs-${encodeURIComponent(y)}`;
+};
+const eloText = (x) => value(x) == null ? '—' : String(Math.round(Number(x)));
+const pointsText = (x) => { const n = value(x); return n == null ? '—' : `${n > 0 ? '+' : ''}${Math.round(n * 10) / 10}`; };
+const agentScoreCell = (a) => {
+ const lo = value(a.ci_lower), hi = value(a.ci_upper);
+ const spread = lo != null && hi != null ? `${Math.round(lo)}–${Math.round(hi)}` : null;
+ return `<b>${eloText(a.score)}</b>${spread ? `<small>95% ${esc(spread)}</small>` : ''}`;
+};
+
+function factorForm(path, data = {}) {
+ const kinds = FACTOR_KINDS.map(([id, name]) => `<option value="${id}"${id === (data.taskKind || 'code') ? ' selected' : ''}>${esc(name)}</option>`).join('');
+ const days = FACTOR_DAYS.map(([n, name]) => `<option value="${n}"${n === Number(data.days || 90) ? ' selected' : ''}>${esc(name)}</option>`).join('');
+ return `<form class="ai-filters ai-factor-form" action="${esc(path)}" method="get"><label class="ai-filter"><span>Task kind</span><select name="task_kind">${kinds}</select></label><label class="ai-filter"><span>Window</span><select name="days">${days}</select></label><button class="ovk-btn ovk-btn--primary">Show</button></form>`;
+}
+
+function factorState(data = {}) {
+ if (data.unavailable) return `<p class="ai-note" role="status">OpenVibe.Bench did not answer its factor board. This page comes back when it answers; nothing is filled in.</p>`;
+ return data.stale ? benchState(data.stale) : '';
+}
+
+function factorMethod(fit, kind, days) {
+ const matches = fit && fit.matches ? `${Number(fit.matches).toLocaleString('en-US')} matches` : 'no matches yet';
+ return `<p class="ai-fine">${fit && fit.method ? `${esc(fit.method)}. ` : ''}Model and harness are fitted apart from every match of ${esc(KIND_NAME[kind] || kind)} in ${esc(String(days))} days (${esc(matches)}); the neutral harness <code>ov-bare</code> is the zero when it appears, otherwise the effects are centred on their average. <a href="${BENCH}/docs">Bench's method</a>.</p>`;
+}
+
+// One scale for every harness, so the strips compare by their position: zero is ov-bare when it anchored the fit.
+function stripScale(harnesses) {
+ const values = harnesses.flatMap((h) => [value(h.ci_lower), value(h.ci_upper), value(h.effect)]).filter((x) => x != null).concat([0]);
+ const lo = Math.min(...values), hi = Math.max(...values);
+ const pad = Math.max(10, (hi - lo) * 0.06);
+ return { lo: lo - pad, hi: hi + pad };
+}
+function harnessStrip(h, scale) {
+ const lo = value(h.ci_lower), hi = value(h.ci_upper), effect = value(h.effect);
+ if (lo == null || hi == null || effect == null) return `<p class="ai-fine">Effect interval not reported for ${esc(h.harness)}.</p>`;
+ const span = scale.hi - scale.lo || 1;
+ const at = (x) => ((x - scale.lo) / span) * 100;
+ const label = `${h.harness}: ${pointsText(effect)} Elo effect, 95% interval ${Math.round(lo)} to ${Math.round(hi)}`;
+ return `<div class="ai-hstrip" role="img" aria-label="${esc(label)}" title="${esc(label)}" style="--zero:${at(0).toFixed(2)}%">`
+ + `<span class="ai-hstrip-range" style="left:${at(lo).toFixed(2)}%;width:${Math.max(0, at(hi) - at(lo)).toFixed(2)}%"></span>`
+ + `<span class="ai-hstrip-dot" style="left:${at(effect).toFixed(2)}%"></span></div>`;
+}
+
+// ── /ai/harnesses ──
+function harnesses(data = {}) {
+ const fit = data.fit || null;
+ const list = fit ? fit.harnesses || [] : [];
+ const agents = fit ? fit.agents || [] : [];
+ const top = [...agents].sort((a, b) => (value(b.score) == null ? -Infinity : value(b.score)) - (value(a.score) == null ? -Infinity : value(a.score)));
+ const byHarness = new Map();
+ for (const a of agents) {
+ if (!byHarness.has(a.harness)) byHarness.set(a.harness, []);
+ byHarness.get(a.harness).push(a);
+ }
+ const scale = stripScale(list);
+ const rows = list.map((h) => {
+ const own = (byHarness.get(h.harness) || []).sort((a, b) => (value(b.score) == null ? -Infinity : value(b.score)) - (value(a.score) == null ? -Infinity : value(a.score)));
+ const other = top.find((x) => !own.some((a) => a.id === x.id)) || null;
+ const agentLine = own.length
+ ? `<p class="ai-harness-agents">Agents: ${own.map((a) => {
+ const body = other ? `<a href="${pairPath(a, other)}">${agentName(a)}</a> ${agentScoreCell(a)}` : `${agentName(a)} ${agentScoreCell(a)}`;
+ return `<span class="ai-agent-item">${body}</span>`;
+ }).join(' · ')}</p>`
+ : `<p class="ai-harness-agents">No agent row names this harness yet.</p>`;
+ return `<li class="ai-harness"><div class="ai-harness-head"><b>${esc(h.harness)}</b><span class="ai-harness-effect">${esc(pointsText(h.effect))} Elo</span>`
+ + `<span class="ai-harness-meta">95% ${esc(value(h.ci_lower) == null ? '—' : `${Math.round(value(h.ci_lower))} to ${Math.round(value(h.ci_upper))}`)} · ${esc(value(h.games) == null ? 'games not reported' : `${Number(h.games).toLocaleString('en-US')} games`)}</span></div>`
+ + harnessStrip(h, scale) + agentLine + `</li>`;
+ }).join('');
+ const empty = !list.length;
+ const body = `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>Harness effects</h1>`
+ + `<p class="ai-lede">What a harness adds, in Elo points, apart from the model it drives. Each strip is its 95% interval on one scale; zero is <code>ov-bare</code> when it anchored the fit, otherwise the average harness. Kind and window choose the fit.</p>`
+ + factorForm('/ai/harnesses', data) + `</header>`
+ + factorState(data)
+ + (!fit ? '' : kit.stats([[list.length, list.length === 1 ? 'harness' : 'harnesses'], [fit.games || 0, 'games'], [fit.matches || 0, 'matches'], [agents.length, agents.length === 1 ? 'agent' : 'agents'], [fit.models ? fit.models.length : 0, 'models']], { label: 'The harness board in numbers' }))
+ + (empty ? kit.empty({ icon: 'trophy', title: 'No harness board yet', text: 'No harness has enough matches yet: a harness shows after it has run against others on the same tasks.' })
+ : kit.section({ title: 'The harness board', count: list.length, body: `<ol class="ai-harness-board">${rows}</ol>${agents.length ? `<p class="ai-fine">An agent is named harness+model. <a href="/ai/matrix?task_kind=${encodeURIComponent(data.taskKind || 'code')}&days=${encodeURIComponent(String(data.days || 90))}">See every harness against every model</a>.</p>` : ''}` }))
+ + (fit ? factorMethod(fit, data.taskKind || 'code', data.days || 90) : '');
+ return page('Harness effects — AI', 'Each harness\'s effect apart from its model, with its 95% interval and games, from OpenVibe.Bench.', body, '/ai/harnesses', [{ '@context': 'https://schema.org', '@type': 'Dataset', name: 'OpenVibe harness effects', url: 'https://openvibe.compare/ai/harnesses' }]);
+}
+
+// ── /ai/matrix ──
+function matrix(data = {}) {
+ const fit = data.fit || null;
+ const harnessList = fit ? (fit.harnesses || []) : [];
+ const agents = fit ? fit.agents || [] : [];
+ const models = fit ? [...(fit.models || [])].sort((a, b) => (value(b.score) == null ? -Infinity : value(b.score)) - (value(a.score) == null ? -Infinity : value(a.score))) : [];
+ const shownModels = models.slice(0, 8);
+ const shownHarnesses = harnessList.slice(0, 12);
+ const cellOf = new Map(agents.map((a) => [`${a.harness}\u0000${a.model}`, a]));
+ const bestOf = new Map();
+ for (const h of shownHarnesses) for (const m of shownModels) {
+ const a = cellOf.get(`${h.harness}\u0000${m.model}`);
+ if (a && (bestOf.get(m.model) == null || (value(a.score) || -Infinity) > (value(bestOf.get(m.model).score) || -Infinity))) bestOf.set(m.model, a);
+ }
+ const tableRows = shownHarnesses.map((h) => {
+ const row = { harness: `<b>${esc(h.harness)}</b><small>${esc(pointsText(h.effect))} Elo · ${esc(value(h.games) == null ? 'games not reported' : `${Number(h.games).toLocaleString('en-US')} games`)}</small>` };
+ for (const m of shownModels) {
+ const a = cellOf.get(`${h.harness}\u0000${m.model}`);
+ if (!a) { row[`m:${m.model}`] = quiet('not run'); continue; }
+ const lo = value(a.ci_lower), hi = value(a.ci_upper);
+ const interval = lo != null && hi != null ? `95% interval ${Math.round(lo)}–${Math.round(hi)}` : 'interval not reported';
+ const best = bestOf.get(m.model) === a;
+ row[`m:${m.model}`] = `<span class="ai-cell" title="${esc(`${a.harness}+${a.model} · ${interval}`)}"><b>${esc(eloText(a.score))}</b><small>${lo != null && hi != null ? `± ${Math.round((hi - lo) / 2)}` : '—'}</small>${best ? ` ${kit.badge('best pair', 'accent')}` : ''}</span>`;
+ }
+ return row;
+ });
+ const columns = [{ key: 'harness', label: 'Harness', primary: true, html: true }].concat(shownModels.map((m) => ({ key: `m:${m.model}`, label: m.model, html: true })));
+ const notable = agents.filter((a) => Math.abs(value(a.factors.interaction.score) || 0) > 30)
+ .sort((a, b) => Math.abs(value(b.factors.interaction.score) || 0) - Math.abs(value(a.factors.interaction.score) || 0));
+ const named = notable.slice(0, 8).map((a) => {
+ const gain = value(a.factors.interaction.score) || 0;
+ return `<b>${esc(a.harness)}</b> ${gain > 0 ? 'gains' : 'loses'} ${esc(Math.abs(Math.round(gain)))} points with ${esc(a.model)}`;
+ }).join('; ');
+ const interactionLine = notable.length
+ ? `<p class="ai-interactions"><b>Interactions over 30 points:</b> ${named}${notable.length > 8 ? `; and ${notable.length - 8} more` : ''}.</p>`
+ : `<p class="ai-interactions ai-fine">No interaction is over 30 points: in this fit every harness adds about the same with every model.</p>`;
+ const hasCells = shownModels.length > 0 && shownHarnesses.length > 0 && agents.length > 0;
+ const empty = !hasCells;
+ const body = `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>Harness × model</h1>`
+ + `<p class="ai-lede">Every pair is an agent: one harness driving one model. A cell is that agent's score with its 95% interval in the tooltip; “not run” means no match has joined the two yet. The best pair for each model is marked, and interactions over 30 points are named below the table.</p>`
+ + factorForm('/ai/matrix', data) + `</header>`
+ + factorState(data)
+ + (!fit ? '' : kit.stats([[shownHarnesses.length, 'harnesses'], [shownModels.length, 'models'], [agents.length, agents.length === 1 ? 'agent' : 'agents'], [fit.games || 0, 'games']], { label: 'The matrix in numbers' }))
+ + (empty ? kit.empty({ icon: 'trophy', title: 'No agent yet', text: 'No harness × model pair has run yet: a cell fills when an agent has run on that harness and model on the same tasks.' })
+ : kit.section({ title: 'The matrix', count: agents.length, body: kit.table({ caption: `Score by harness and model · ${KIND_NAME[data.taskKind] || data.taskKind || 'code'} · ${data.days || 90} days`, columns, rows: tableRows, label: 'Harness by model' }) + interactionLine }))
+ + (models.length > shownModels.length ? `<p class="ai-fine">The ${shownModels.length} highest-rated models are shown; ${models.length - shownModels.length} more are in the fit.</p>` : '')
+ + (fit ? factorMethod(fit, data.taskKind || 'code', data.days || 90) : '');
+ return page('Harness × model — AI', 'Every harness and model as an agent, with its score, interval and interaction, from OpenVibe.Bench.', body, '/ai/matrix', [{ '@context': 'https://schema.org', '@type': 'Dataset', name: 'OpenVibe harness by model matrix', url: 'https://openvibe.compare/ai/matrix' }]);
+}
+
+// ── An agent pair: the gap, split ──
+function agentCompare(data, pair) {
+ const a = pair.a, b = pair.b;
+ const gap = value(pair.gap) || 0;
+ const lead = gap >= 0 ? a : b;
+ const trail = gap >= 0 ? b : a;
+ const sign = gap >= 0 ? 1 : -1;
+ const part = (k) => { const n = value(pair.parts && pair.parts[k]); return n == null ? 0 : Math.round(sign * n * 10) / 10 || 0; };
+ const parts = { model: part('model'), harness: part('harness'), interaction: part('interaction') };
+ const gapText = Math.abs(Math.round(gap * 10) / 10);
+ const sentence = gap === 0
+ ? `${agentName(a)} and ${agentName(b)} are level. Of this 0-point gap, ${pointsText(parts.model)} is the model, ${pointsText(parts.harness)} the harness, ${pointsText(parts.interaction)} the two together.`
+ : `${agentName(lead)} is ${gapText} Elo points ahead of ${agentName(trail)}. Of this ${gapText}-point gap, ${pointsText(parts.model)} is the model, ${pointsText(parts.harness)} the harness, ${pointsText(parts.interaction)} the two together.`;
+ const negative = Object.values(parts).some((x) => x < 0);
+ const sideFacts = (x) => [
+ ['Harness', `<b>${esc(x.harness)}</b>`],
+ ['Model', `<b>${esc(x.model)}</b>`],
+ ['Agent score', agentScoreCell(x)],
+ ['Model part', `${eloText(x.factors.model.score)} Elo`],
+ ['Harness effect', `${pointsText(x.factors.harness.score)} Elo`],
+ ['Interaction', `${pointsText(x.factors.interaction.score)} Elo`],
+ ];
+ const cards = [a, b].map((x) => `<article class="ai-vs-card"><h2>${agentName(x)}</h2><dl>${sideFacts(x).map(([k, val]) => `<div><dt>${esc(k)}</dt><dd>${val}</dd></div>`).join('')}</dl></article>`).join('');
+ const options = data.options || [];
+ const list = `<datalist id="ai-models">${options.slice(0, 500).map((o) => `<option value="${esc(o)}"></option>`).join('')}</datalist>`;
+ const form = `<form class="ai-filters ai-vs-form" action="/ai/compare"><label class="ai-filter"><span>First entrant</span><input name="a" list="ai-models" value="${esc(agentSlug(a))}" required></label>`
+ + `<label class="ai-filter"><span>Second entrant</span><input name="b" list="ai-models" value="${esc(agentSlug(b))}" required></label><button class="ovk-btn ovk-btn--primary">Compare</button>${list}</form>`;
+ const split = { label: `Where the ${gapText}-point gap comes from`, parts };
+ const canonical = pairPath(a, b);
+ const body = `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>${agentName(a)} vs ${agentName(b)}</h1>`
+ + `<p class="ai-lede">Two agents on the same fit: the same model and harness contributions are added for each. ${fitNote(pair)}</p></header>`
+ + availabilityBlock(data) + `<div class="ai-vs">${cards}</div>`
+ + `<section class="ai-gap" aria-labelledby="ai-gap-title"><h2 id="ai-gap-title">The gap, split</h2><p class="ai-gap-say">${sentence}</p>`
+ + `<div class="ai-gap-split" id="ai-gap-split"></div>`
+ + (negative ? '<p class="ai-fine">A part below zero is drawn by its size; the sign is in the sentence above.</p>' : '')
+ + `</section>`
+ + form
+ + `<p class="ai-fine">The split comes from OpenVibe.Bench's ${esc(KIND_NAME[pair.task_kind] || pair.task_kind || 'code')} fit over ${esc(String(pair.days || 365))} days (${esc(Number(pair.matches || 0).toLocaleString('en-US'))} matches).${pair.method ? ` ${esc(pair.method)}.` : ''} <a href="/ai/harnesses?task_kind=${encodeURIComponent(pair.task_kind || 'code')}&days=${encodeURIComponent(String(pair.days || 365))}">Every harness effect</a> · <a href="/ai/matrix?task_kind=${encodeURIComponent(pair.task_kind || 'code')}&days=${encodeURIComponent(String(pair.days || 365))}">The matrix</a>.</p>`
+ + `<script type="application/json" id="ai-gap-data">${jsonData(split)}</script><script src="/ai-factors.js" defer></script>`;
+ return page(`${agentSlug(a)} vs ${agentSlug(b)}`, `${agentSlug(a)} vs ${agentSlug(b)}: the score gap split into model, harness and interaction.`, body, canonical, [{ '@context': 'https://schema.org', '@type': 'Dataset', name: `${agentSlug(a)} vs ${agentSlug(b)}`, url: `https://openvibe.compare${canonical}` }]);
+}
+
+function fitNote(pair) {
+ return `${esc(KIND_NAME[pair.task_kind] || pair.task_kind || 'Code')} · ${esc(String(pair.days || 365))} days · ${esc(Number(pair.games || 0).toLocaleString('en-US'))} games.`;
+}
+
function arena() {
function arena() {
return page('AI Arena', 'Compare anonymous AI answers and vote.', `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>AI Arena</h1><p class="ai-lede">Two anonymous answers to one prompt; you pick the better one, then see which models wrote them. Every vote moves the ratings here.</p></header><p><a class="ovk-btn ovk-btn--primary" href="https://openvibe.exchange/arena">Vote in the Arena</a></p>`, '/ai/arena');
return page('AI Arena', 'Compare anonymous AI answers and vote.', `<header class="ai-head ai-head--page"><p class="ai-kicker"><a href="/ai">AI comparison</a></p><h1>AI Arena</h1><p class="ai-lede">Two anonymous answers to one prompt; you pick the better one, then see which models wrote them. Every vote moves the ratings here.</p></header><p><a class="ovk-btn ovk-btn--primary" href="https://openvibe.exchange/arena">Vote in the Arena</a></p>`, '/ai/arena');
}
}
@@ -610,6 +800,8 @@ function methodPage(data = {}) {
@@ -610,6 +800,8 @@ function methodPage(data = {}) {
${judgeRows.length ? kit.table({ caption: 'Judges in use', columns: [{ key: 'how', label: 'How', html: true }, { key: 'judge', label: 'Bench judge' }, { key: 'jobs', label: 'Task kinds' }], rows: judgeRows }) : `<p class="ai-fine">${esc('Bench did not answer its kind list; the judge rules above are its published contract.')}</p>`}
${judgeRows.length ? kit.table({ caption: 'Judges in use', columns: [{ key: 'how', label: 'How', html: true }, { key: 'judge', label: 'Bench judge' }, { key: 'jobs', label: 'Task kinds' }], rows: judgeRows }) : `<p class="ai-fine">${esc('Bench did not answer its kind list; the judge rules above are its published contract.')}</p>`}
<h2>Verified and submitted</h2>
<h2>Verified and submitted</h2>
<p>A <b>verified</b> run was executed by a service (openvibe.codes, Run, a Node worker); a <b>submitted</b> run was posted with a person’s token. The rankings read verified runs only, and a submitted run is labelled wherever the latest runs are listed.</p>
<p>A <b>verified</b> run was executed by a service (openvibe.codes, Run, a Node worker); a <b>submitted</b> run was posted with a person’s token. The rankings read verified runs only, and a submitted run is labelled wherever the latest runs are listed.</p>
+<h2>Model and harness, apart</h2>
+<p>An agent is a harness driving a model, so an agent score has two parts. Bench's factor board fits <code>agent = model + harness + model×harness</code> over the same matches, with partial pooling and a 95% interval on each part; the neutral harness <code>ov-bare</code> is the zero when it appears (otherwise the effects are centred), so a harness's effect is in Elo points against that zero. A negative interaction says the harness adds less with that model than it does on average. <a href="/ai/harnesses">The harness board</a> and <a href="/ai/matrix">the harness × model matrix</a> draw those numbers; an agent pair's page splits its gap into the three parts.</p>
<h2>How often, and what is missing</h2>
<h2>How often, and what is missing</h2>
<p>${o.runs == null ? 'Bench did not answer the public counts.' : `${Number(o.benchmarks || 0).toLocaleString('en-US')} public ${o.benchmarks === 1 ? 'benchmark' : 'benchmarks'}, ${Number(o.tasks || 0).toLocaleString('en-US')} ${o.tasks === 1 ? 'task' : 'tasks'}, ${Number(o.runs || 0).toLocaleString('en-US')} ${o.runs === 1 ? 'run' : 'runs'} from ${Number(o.models || 0).toLocaleString('en-US')} ${o.models === 1 ? 'model' : 'models'}; the newest run was ${o.last_run ? esc(dayTime(o.last_run)) : 'never published'}.`}</p>
<p>${o.runs == null ? 'Bench did not answer the public counts.' : `${Number(o.benchmarks || 0).toLocaleString('en-US')} public ${o.benchmarks === 1 ? 'benchmark' : 'benchmarks'}, ${Number(o.tasks || 0).toLocaleString('en-US')} ${o.tasks === 1 ? 'task' : 'tasks'}, ${Number(o.runs || 0).toLocaleString('en-US')} ${o.runs === 1 ? 'run' : 'runs'} from ${Number(o.models || 0).toLocaleString('en-US')} ${o.models === 1 ? 'model' : 'models'}; the newest run was ${o.last_run ? esc(dayTime(o.last_run)) : 'never published'}.`}</p>
<p class="ai-fine">A model with no verified run has no score here; a benchmark with no run says so. The public reads are Bench’s API at <a href="${BENCH}/docs">bench.openvibe.services/docs</a>; the open data is <a href="${BENCH}/api/v1/datasets/benchmarks.jsonl">benchmarks.jsonl</a> and the per-kind datasets. <a href="/ai/feed.xml">New results (RSS)</a> · <a href="https://openvibe.report/api/v1/ai/results">the same as JSON</a>.</p>`;
<p class="ai-fine">A model with no verified run has no score here; a benchmark with no run says so. The public reads are Bench’s API at <a href="${BENCH}/docs">bench.openvibe.services/docs</a>; the open data is <a href="${BENCH}/api/v1/datasets/benchmarks.jsonl">benchmarks.jsonl</a> and the per-kind datasets. <a href="/ai/feed.xml">New results (RSS)</a> · <a href="https://openvibe.report/api/v1/ai/results">the same as JSON</a>.</p>`;
@@ -638,6 +830,6 @@ function rss({ runs, overview, base = 'https://openvibe.compare' } = {}) {
@@ -638,6 +830,6 @@ function rss({ runs, overview, base = 'https://openvibe.compare' } = {}) {
}
}
module.exports = {
module.exports = {
- hub, leaderboard, providers, model, compare, arena, boardPage, benchmarkPage, methodPage, rss,
+ hub, leaderboard, providers, model, compare, arena, boardPage, benchmarkPage, methodPage, rss, harnesses, matrix,
availabilityBlock, measuredNote, benchState, page, scriptLd, PARTS_CSS,
availabilityBlock, measuredNote, benchState, page, scriptLd, PARTS_CSS,
};
};
@@ -27,9 +27,9 @@ function createApp({getData=createData(),getCompute=createCompute(),ai=createAi(
@@ -27,9 +27,9 @@ function createApp({getData=createData(),getCompute=createCompute(),ai=createAi(
app.get('/api/v1/comparisons',wrap(async(req,res)=>res.json({comparisons:await getData()})));
app.get('/api/v1/comparisons',wrap(async(req,res)=>res.json({comparisons:await getData()})));
app.get('/api/v1/comparisons/:slug',wrap(async(req,res)=>{const c=(await getData()).find(c=>c.slug===req.params.slug);return c?res.json(c):res.status(404).json({error:'comparison.not_found'});}));
app.get('/api/v1/comparisons/:slug',wrap(async(req,res)=>{const c=(await getData()).find(c=>c.slug===req.params.slug);return c?res.json(c):res.status(404).json({error:'comparison.not_found'});}));
app.get('/robots.txt',(_q,r)=>r.type('text').send('User-agent: *\nAllow: /\nAllow: /ai\nSitemap: https://openvibe.compare/sitemap.xml\n'));
app.get('/robots.txt',(_q,r)=>r.type('text').send('User-agent: *\nAllow: /\nAllow: /ai\nSitemap: https://openvibe.compare/sitemap.xml\n'));
- app.get('/llms.txt',wrap(async(_q,r)=>{const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);return r.type('text').send('# OpenVibe.Compare\n\nDated comparisons. Missing evidence is not zero.\n\n'+`- [Compute prices](https://openvibe.compare/compute): GPUs and computers from OpenVibe.Exchange, per hour, per GPU GB-hour and per month at 24/7${compute.checked_at?`; checked ${compute.checked_at.slice(0,10)}`:''}.\n`+`- [AI model leaderboard](https://openvibe.compare/ai): ratings from blind votes, provider prices and measured speed.\n- [AI leaderboard](https://openvibe.compare/ai/leaderboard): every model with filters and sorts.\n- [AI providers](https://openvibe.compare/ai/providers): every provider offer with price, reply time, errors and uptime.\n- [AI method](https://openvibe.compare/ai/method): how a score, a cost and a speed are made.\n- [AI results feed](https://openvibe.compare/ai/feed.xml): new public Bench runs as RSS.\n`+aiData.rows.map(c=>`- [${c.id}](https://openvibe.compare/ai/models/${encodeURIComponent(c.id)}): rating, prices and measured speed.`).join('\n')+'\n'+cs.map(c=>`- [${c.title}](https://openvibe.compare${c.slug==='ai-models'?'/ai':`/comparisons/${c.slug}`}): ${c.summary}`).join('\n')+'\n\nJSON: https://openvibe.compare/api/v1/comparisons\nCompute JSON: https://openvibe.compare/api/v1/compute\nEach page also accepts Accept: text/markdown.\n')}));
+ app.get('/llms.txt',wrap(async(_q,r)=>{const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);return r.type('text').send('# OpenVibe.Compare\n\nDated comparisons. Missing evidence is not zero.\n\n'+`- [Compute prices](https://openvibe.compare/compute): GPUs and computers from OpenVibe.Exchange, per hour, per GPU GB-hour and per month at 24/7${compute.checked_at?`; checked ${compute.checked_at.slice(0,10)}`:''}.\n`+`- [AI model leaderboard](https://openvibe.compare/ai): ratings from blind votes, provider prices and measured speed.\n- [AI leaderboard](https://openvibe.compare/ai/leaderboard): every model with filters and sorts.\n- [AI providers](https://openvibe.compare/ai/providers): every provider offer with price, reply time, errors and uptime.\n- [AI method](https://openvibe.compare/ai/method): how a score, a cost and a speed are made.\n- [Harness effects](https://openvibe.compare/ai/harnesses): each harness's Elo effect apart from its model, with an interval.\n- [Harness × model matrix](https://openvibe.compare/ai/matrix): every agent as its harness and model, with interactions.\n- [AI results feed](https://openvibe.compare/ai/feed.xml): new public Bench runs as RSS.\n`+aiData.rows.map(c=>`- [${c.id}](https://openvibe.compare/ai/models/${encodeURIComponent(c.id)}): rating, prices and measured speed.`).join('\n')+'\n'+cs.map(c=>`- [${c.title}](https://openvibe.compare${c.slug==='ai-models'?'/ai':`/comparisons/${c.slug}`}): ${c.summary}`).join('\n')+'\n\nJSON: https://openvibe.compare/api/v1/comparisons\nCompute JSON: https://openvibe.compare/api/v1/compute\nEach page also accepts Accept: text/markdown.\n')}));
app.get('/sitemap.xml',wrap(async(_q,r)=>{
app.get('/sitemap.xml',wrap(async(_q,r)=>{
- const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);const newest=cs.map(c=>c.updated_at).sort().at(-1);const paths=new Map([['/',newest],['/compute',compute.checked_at||newest],['/ai',aiData.updated_at],['/ai/leaderboard',aiData.updated_at],['/ai/providers',aiData.updated_at],['/ai/method',aiData.updated_at],['/ai/arena',aiData.updated_at]]);
+ const [cs,compute,aiData]=await Promise.all([getData(),getCompute(),ai.comparison()]);const newest=cs.map(c=>c.updated_at).sort().at(-1);const paths=new Map([['/',newest],['/compute',compute.checked_at||newest],['/ai',aiData.updated_at],['/ai/leaderboard',aiData.updated_at],['/ai/providers',aiData.updated_at],['/ai/method',aiData.updated_at],['/ai/harnesses',aiData.updated_at],['/ai/matrix',aiData.updated_at],['/ai/arena',aiData.updated_at]]);
for(const c of cs){if(c.slug==='ai-models')continue;paths.set(`/comparisons/${c.slug}`,c.updated_at);for(const t of c.things)paths.set(v.thingPath(t.id),c.updated_at);}
for(const c of cs){if(c.slug==='ai-models')continue;paths.set(`/comparisons/${c.slug}`,c.updated_at);for(const t of c.things)paths.set(v.thingPath(t.id),c.updated_at);}
for(const row of aiData.rows)paths.set(`/ai/models/${encodeURIComponent(row.id)}`,row.source&&row.source.date||aiData.updated_at);
for(const row of aiData.rows)paths.set(`/ai/models/${encodeURIComponent(row.id)}`,row.source&&row.source.date||aiData.updated_at);
for(const task of aiData.tasks)paths.set(`/ai/${encodeURIComponent(task.id)}`,aiData.updated_at);
for(const task of aiData.tasks)paths.set(`/ai/${encodeURIComponent(task.id)}`,aiData.updated_at);
@@ -44,6 +44,8 @@ function createApp({getData=createData(),getCompute=createCompute(),ai=createAi(
@@ -44,6 +44,8 @@ function createApp({getData=createData(),getCompute=createCompute(),ai=createAi(
app.get('/ai',wrap(async(_q,r)=>r.send(aiViews.hub(await ai.comparison()))));
app.get('/ai',wrap(async(_q,r)=>r.send(aiViews.hub(await ai.comparison()))));
app.get('/ai/leaderboard',wrap(async(req,res)=>res.send(aiViews.leaderboard(await ai.leaderboard(req.query)))));
app.get('/ai/leaderboard',wrap(async(req,res)=>res.send(aiViews.leaderboard(await ai.leaderboard(req.query)))));
app.get('/ai/providers',wrap(async(_q,r)=>{const data=await ai.comparison();return r.send(aiViews.providers(data));}));
app.get('/ai/providers',wrap(async(_q,r)=>{const data=await ai.comparison();return r.send(aiViews.providers(data));}));
+ app.get('/ai/harnesses',wrap(async(req,res)=>res.send(aiViews.harnesses(await ai.factorBoard(req.query)))));
+ app.get('/ai/matrix',wrap(async(req,res)=>res.send(aiViews.matrix(await ai.factorBoard(req.query)))));
app.get('/ai/arena',(_q,r)=>r.send(aiViews.arena()));
app.get('/ai/arena',(_q,r)=>r.send(aiViews.arena()));
app.get('/ai/compare',wrap(async(req,res)=>{
app.get('/ai/compare',wrap(async(req,res)=>{
// The picker sends a and b; older links send models=a,b.
// The picker sends a and b; older links send models=a,b.
new file mode 100644
new file mode 100644
@@ -0,0 +1,182 @@
@@ -0,0 +1,182 @@
+'use strict';
+
+// The harness board, the harness × model matrix and an agent pair: each with a stubbed Bench factor fit, an empty
+// fit, and Bench down. A page that cannot read the fit still answers 200 with words, never a blank or a zero.
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const { once } = require('node:events');
+const { createApp } = require('../server/app');
+const { createAi } = require('../server/ai-data');
+const { gapParts } = require('../public/ai-factors');
+
+const at = '2026-11-13T00:00:00.000Z';
+const KINDS = [
+ { id: 'code', room: 'codes', judge: 'test_suite', name: 'Code' },
+ { id: 'chat', room: 'exchange', judge: 'human_pairwise', name: 'Chat' },
+];
+const catalogue = {
+ generated_at: at,
+ models: [
+ { slug: 'gpt-5-mini', canonical: 'gpt-5-mini', name: 'gpt-5-mini', kind: 'chat', maker: 'OpenAI', offers: [] },
+ { slug: 'glm-5.3', canonical: 'glm-5.3', name: 'glm-5.3', kind: 'chat', maker: 'Z.ai', offers: [] },
+ ],
+};
+const market = { generated_at: at, offers: [] };
+const perf = { range: { from: '2026-10-15', to: '2026-11-13' }, rows: [] };
+
+const fit = {
+ task_kind: 'code', category: null, days: 90, matches: 180, games: 180, style_control: true,
+ models: [
+ { model: 'gpt-5-mini', score: 1050, ci_lower: 1020, ci_upper: 1080, games: 100 },
+ { model: 'glm-5.3', score: 1030, ci_lower: 1000, ci_upper: 1060, games: 80 },
+ ],
+ harnesses: [
+ { harness: 'codex', effect: 45, ci_lower: 20, ci_upper: 70, games: 110 },
+ { harness: 'opencode', effect: -15, ci_lower: -40, ci_upper: 10, games: 70 },
+ { harness: 'aider', effect: 5, ci_lower: -20, ci_upper: 30, games: 20 },
+ ],
+ agents: [
+ { id: 'agent:codex+gpt-5-mini#0123abcd', model: 'gpt-5-mini', harness: 'codex', score: 1095, ci_lower: 1070, ci_upper: 1120, factors: { model: { score: 1050 }, harness: { score: 45 }, interaction: { score: 0 } } },
+ { id: 'agent:opencode+gpt-5-mini#0123abcd', model: 'gpt-5-mini', harness: 'opencode', score: 1035, ci_lower: 1010, ci_upper: 1060, factors: { model: { score: 1050 }, harness: { score: -15 }, interaction: { score: 0 } } },
+ { id: 'agent:codex+glm-5.3#0123abcd', model: 'glm-5.3', harness: 'codex', score: 1040, ci_lower: 1015, ci_upper: 1065, factors: { model: { score: 1030 }, harness: { score: 45 }, interaction: { score: -35 } } },
+ { id: 'agent:opencode+glm-5.3#0123abcd', model: 'glm-5.3', harness: 'opencode', score: 1070, ci_lower: 1045, ci_upper: 1095, factors: { model: { score: 1030 }, harness: { score: -15 }, interaction: { score: 55 } } },
+ ],
+ method: 'penalised conditional logit: agent = model + harness + model×harness, 150 bootstrap draws, seed 20261113',
+};
+const emptyFit = { task_kind: 'code', category: null, days: 90, matches: 0, games: 0, style_control: true, models: [], harnesses: [], agents: [], method: fit.method };
+
+const json = (body) => ({ ok: true, status: 200, headers: { get: () => 'application/json' }, text: async () => JSON.stringify(body), json: async () => body });
+
+function stub({ down = false, empty = false } = {}) {
+ return async (url) => {
+ const u = new URL(url);
+ const p = u.pathname;
+ if (p === '/api/v1/factors') {
+ if (down) throw Error('offline');
+ const days = Number(u.searchParams.get('days')) || 90;
+ const task_kind = u.searchParams.get('task_kind') || 'code';
+ return json(empty ? { ...emptyFit, task_kind, days } : { ...fit, task_kind, days });
+ }
+ if (down) throw Error('offline');
+ if (p === '/api/v1/exchange/catalog') return json(catalogue);
+ if (p === '/api/v1/exchange/market') return json(market);
+ if (p === '/api/v1/ai/performance') return json(perf);
+ if (p === '/api/v1/kinds') return json({ count: KINDS.length, kinds: KINDS });
+ if (p.startsWith('/api/v1/ratings/')) return json({ board: { kind_id: p.split('/').at(-1), rows: [] } });
+ if (p.startsWith('/api/v1/boards/')) return json({ board: { kind_id: p.split('/').at(-1), type: 'elo', rows: [] } });
+ return json(null);
+ };
+}
+
+async function withServer(opts, fn) {
+ const ai = createAi({ fetchImpl: stub(opts), log: { warn() {} } });
+ const server = createApp({ ai }).listen(0, '127.0.0.1');
+ await once(server, 'listening');
+ try {
+ await fn(`http://127.0.0.1:${server.address().port}`);
+ } finally {
+ await new Promise((resolve) => server.close(resolve));
+ }
+}
+
+test('the harness board draws each effect with its interval on one scale, games and the agents it appears in', async () => {
+ await withServer({}, async (base) => {
+ const r = await fetch(`${base}/ai/harnesses?task_kind=code&days=90`);
+ assert.equal(r.status, 200);
+ const html = await r.text();
+ assert.match(html, /Harness effects/);
+ assert.match(html, /\+45 Elo/);
+ assert.match(html, /95% 20 to 70/);
+ assert.match(html, /110 games/);
+ assert.match(html, /class="ai-hstrip"/);
+ assert.match(html, /--zero:/);
+ // Every agent number names its harness.
+ assert.match(html, /<b>codex<\/b><span aria-hidden="true">\+<\/span>gpt-5-mini/);
+ assert.match(html, /<b>opencode<\/b><span aria-hidden="true">\+<\/span>glm-5\.3/);
+ assert.match(html, /95% 1070–1120/);
+ assert.match(html, /aider/);
+ assert.match(html, /name="task_kind"/);
+ assert.match(html, /name="days"/);
+ assert.match(html, /neutral harness <code>ov-bare<\/code>/);
+ assert.doesNotMatch(html, /No harness has enough matches yet/);
+ });
+});
+
+test('the matrix is a labelled table (cards under 600 px), not run cells, the best pair per model and named interactions', async () => {
+ await withServer({}, async (base) => {
+ const r = await fetch(`${base}/ai/matrix?task_kind=code&days=90`);
+ assert.equal(r.status, 200);
+ const html = await r.text();
+ assert.match(html, /Harness × model/);
+ assert.match(html, /ovk-table--stack/);
+ assert.match(html, /data-label="gpt-5-mini"/);
+ assert.match(html, /title="codex\+gpt-5-mini · 95% interval 1070–1120"/);
+ assert.match(html, /best pair/);
+ assert.match(html, /not run/);
+ assert.match(html, /Interactions over 30 points/);
+ assert.match(html, /<b>opencode<\/b> gains 55 points with glm-5\.3/);
+ assert.match(html, /<b>codex<\/b> loses 35 points with glm-5\.3/);
+ assert.match(html, /Harness × model matrix|Harness × model/);
+ });
+});
+
+test('an agent pair splits its gap into the model, the harness and the two together', async () => {
+ await withServer({}, async (base) => {
+ const r = await fetch(`${base}/ai/compare/codex%2Bgpt-5-mini-vs-opencode%2Bglm-5.3`);
+ assert.equal(r.status, 200);
+ const html = await r.text();
+ assert.match(html, /Of this 25-point gap, \+20 is the model, \+60 the harness, -55 the two together/);
+ assert.match(html, /id="ai-gap-data"/);
+ assert.match(html, /id="ai-gap-split"/);
+ assert.match(html, /\/ai-factors\.js/);
+ assert.match(html, /codex<\/b><span aria-hidden="true">\+<\/span>gpt-5-mini/);
+ assert.match(html, /OpenVibe\.Bench's Code fit over 365 days/);
+ });
+});
+
+test('an empty answer says what fills the board, and the hub links both pages', async () => {
+ await withServer({ empty: true }, async (base) => {
+ const harnesses = await (await fetch(`${base}/ai/harnesses`)).text();
+ assert.match(harnesses, /No harness has enough matches yet: a harness shows after it has run against others on the same tasks\./);
+ const matrix = await (await fetch(`${base}/ai/matrix`)).text();
+ assert.match(matrix, /No harness × model pair has run yet: a cell fills when an agent has run on that harness and model on the same tasks\./);
+ const pair = await fetch(`${base}/ai/compare/codex%2Bgpt-5-mini-vs-opencode%2Bglm-5.3`);
+ assert.equal(pair.status, 200);
+ assert.doesNotMatch(await pair.text(), /Of this .*point gap/);
+ const hub = await (await fetch(`${base}/ai`)).text();
+ assert.match(hub, /href="\/ai\/harnesses"/);
+ assert.match(hub, /href="\/ai\/matrix"/);
+ assert.match(hub, /Harness and model, apart/);
+ });
+});
+
+test('Bench down still answers 200 with words on each page', async () => {
+ await withServer({ down: true }, async (base) => {
+ for (const path of ['/ai/harnesses', '/ai/matrix']) {
+ const r = await fetch(base + path);
+ assert.equal(r.status, 200, path);
+ const html = await r.text();
+ assert.match(html, /OpenVibe\.Bench did not answer its factor board\. This page comes back when it answers; nothing is filled in\./, path);
+ }
+ const pair = await fetch(`${base}/ai/compare/codex%2Bgpt-5-mini-vs-opencode%2Bglm-5.3`);
+ assert.equal(pair.status, 200);
+ const html = await pair.text();
+ assert.match(html, /codex\+gpt-5-mini vs opencode\+glm-5\.3/);
+ assert.doesNotMatch(html, /Of this .*point gap/);
+ assert.doesNotMatch(html, /id="ai-gap-data"/);
+ });
+});
+
+test('the sitemap carries the two pages', async () => {
+ await withServer({}, async (base) => {
+ const xml = await (await fetch(`${base}/sitemap.xml`)).text();
+ assert.match(xml, /https:\/\/openvibe\.compare\/ai\/harnesses/);
+ assert.match(xml, /https:\/\/openvibe\.compare\/ai\/matrix/);
+ });
+});
+
+test('gapParts hands OVMetrics.split a size per part and keeps the sign', () => {
+ const parts = gapParts({ model: 20, harness: 60, interaction: -55 });
+ assert.deepEqual(parts.map((p) => [p.key, p.value, p.signed, p.tone]), [['model', 20, 20, 1], ['harness', 60, 60, 2], ['interaction', 55, -55, 3]]);
+ assert.equal(gapParts({ model: null, harness: null, interaction: null }).length, 0);
+});