openvibers/openvibe.bench Commits
Bench B3: rank intervals, category boards, the OpenVibe Index and the holdout gap (#176)
Changes
@@@ -229,6 +229,22 @@ observations is returned only when every observation counts that outcome; a missignores any such fields, and the read routes never join a task, benchmark or entry. The only text in a row is anexact id, a family, and the id of the round or run that produced it.++## The boards: ranks, categories, the Index and the holdout++++`GET /api/v1/boards/:kindId?type=elo` returns the fit, not just the Elo updates: each row carries `elo`, its++`ci_lower`/`ci_upper` rating interval, and `rank`/`rank_lo`/`rank_hi` — the 95% rank interval from the same bootstrap.++The board also carries:++++| Field | What it holds |++|---|---|++| `categories` | One board per task category with at least 30 games: `{ category, games, rows }`, each row fitted the same way. A category below the floor is left out. `?category=<slug>` answers for one category alone. |++| `index` | The OpenVibe Index per model: `{ weights, rating_anchor, benchmark_count, benchmarks[], rows[] }`. Each row is `{ subject, index, n_components, weight_sum, components[] }`; `index` is null when fewer than two components exist. |++| `holdout` | `{ available, held_out_tasks, rows[] }`; each row is `{ subject, public_n, public_score, holdout_n, holdout_score, gap }`, with a null gap when either side has no scores. |++++The open dataset (`/api/v1/datasets/:kindId.jsonl`) also emits `type: "category"`, `type: "index"` and++`type: "holdout"` lines, and its rating lines carry the rank interval. The formulas, the 0.5/0.5 weights and the++30-game floor are in [index.md](index.md).++## The older APIBench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the privatennew file mode 100644@@@ -0,0 +1,60 @@++# The numbers on Bench++++Every board is a Bradley-Terry fit over the votes, centred on 1000, with a bootstrap that gives each model a++rating interval and a 95% rank interval. A kind's board answers at `GET /api/v1/boards/:kindId`; the open dataset++for the kind (`GET /api/v1/datasets/:kindId.jsonl`) carries the same numbers. The shapes are in++[api.md](api.md).++++## Rank intervals++++A rank is a band, not a place. Every fit resamples the games 200 times with a fixed seed, ranks the models in each++resample (equal ratings share a rank), and takes the 2.5th and 97.5th percentile of those places:++++```json++{ "subject": "model-a", "elo": 1042.5, "rank": 2, "rank_lo": 1, "rank_hi": 4 }++```++++`rank` is the place on the fitted board. `rank_lo` and `rank_hi` are the 95% rank interval: the places the votes++allow. A model whose interval contains 1 may be first; overlapping intervals are why the page shows "1st–3rd" and++never pretends the order is settled.++++## Per-category boards++++A task can carry a `task_class` (code, writing, chat, reasoning, arcade, …). Every vote is counted in the task's++category as well as the kind's whole board. The elo board publishes `categories`: one board per category with at++least **30 games**, fitted the same way (same Bradley-Terry maths, same bootstrap intervals, same rank intervals).++A category below 30 games is left out; it is not shown with a thin score. `GET /api/v1/boards/:kindId?category=arcade`++answers for one category alone.++++## The OpenVibe Index++++The Index is one number per model per kind, on the same 0-100 scale as every other Bench score. It is a weighted mean++of two kinds of evidence:++++| Component | Weight | Value |++|---|---|---|++| The model's human rating | 0.5 | Its Bradley-Terry rating put on 0-100 by the Elo expectation against a 1000-rated opponent: `100 / (1 + 10 ** ((1000 - rating) / 400))`. A 1000 rating is exactly 50. |++| The kind's public benchmarks | 0.5, split equally between them | The model's median verified score (0-100) on each public benchmark of that kind. |++++The weights are published in every answer (`index.weights` and the per-benchmark `index.benchmarks[].weight`) and in++each row's `components`. A model missing a benchmark gets that benchmark's weight renormalised away rather than++letting one benchmark stand in for half the Index; `weight_sum` says how much of the 1.0 the model actually has.++++**Fewer than two components is not an Index.** A model with only a rating, or only one benchmark, gets `index: null`.++Null is not zero and is not sorted as a score.++++## The holdout gap++++A benchmark task may be held out (`holdout: true`): only the executor and the benchmark's owner see its prompt. The++holdout gap compares the same model's verified scores on the public tasks and on the held-out tasks of the kind:++++```++gap = mean(public task scores) - mean(held-out task scores)++```++++The answer carries `public_n`, `public_score`, `holdout_n`, `holdout_score` and `gap`. A kind with no held-out tasks++says `available: false`. A model missing either side gets `gap: null`, never zero: a gap cannot be read from one side.++A large positive gap is a contamination signal, not a proof; it says the held-out set scored lower and is worth a look.++++Both statistics travel with every board (`index` and `holdout`), and the open dataset has `type: "index"` and++`type: "holdout"` lines beside the games, so a reader can check them without the page.@@@ -79,8 +79,13 @@ const lib = require('./lib');// each output as covariates, bootstrap intervals and rank bands.const style = require('./style');++// The OpenVibe Index and the holdout gap (stage B3): pure maths, so a test and a page agree on the numbers.++const stats = require('./stats');++module.exports = {eloUpdate, bradleyTerry: lib.bradleyTerry, voterWeights: lib.voterWeights, wilson: lib.wilson, percentile: lib.percentile, pareto, costPerSolved, BASE_ELO, K_DEFAULT,style,fitRatings: style.fitRatings, rankBands: style.rankBands, styleFeatures: style.styleFeatures, markupCount: style.markupCount,++ openvibeIndex: stats.openvibeIndex, holdoutGap: stats.holdoutGap, ratingScore: stats.ratingScore,++ INDEX_WEIGHTS: stats.INDEX_WEIGHTS, RATING_ANCHOR: stats.RATING_ANCHOR,};@@@ -92,6 +92,16 @@ function seededRandom(seed) {};}++/**++ * Competition ranks by strength: the best subject is 1st and equal ratings share the best rank. The bootstrap++ * ranks use the same rule on every resampled fit, so a rank interval is the spread of the places the votes allow,++ * not a second guess from the two ends of the rating interval.++ */++function ranksOf(ratings) {++ const list = Array.isArray(ratings) ? ratings : [];++ return list.map((rating) => 1 + list.filter((other) => other > rating + 1e-9).length);++}++function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterations = 200, weights } = {}) {const active = games.filter(game => game.outcome !== 'both_bad');const subjects = [...new Set(active.flatMap(game => [game.a, game.b]))].sort();@@@ -110,19 +120,25 @@ function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterati}const ratings = ratingsFor(active, subjects, weights, iterations, anchor);const samples = subjects.map(() => []);++ const rankSamples = subjects.map(() => []);if (bootstrap > 0 && active.length) {const random = seededRandom(seed);for (let i = 0; i < bootstrap; i++) {const resampled = Array.from({ length: active.length }, () => active[Math.floor(random() * active.length)]);const draw = ratingsFor(resampled, subjects, weights, iterations, anchor);draw.forEach((rating, j) => samples[j].push(rating));++ ranksOf(draw).forEach((rank, j) => rankSamples[j].push(rank));}}++ const pointRanks = ranksOf(ratings);return subjects.map((subject, i) => ({subject,rating: ratings[i],ci_lower: samples[i].length ? percentile(samples[i], 2.5) : ratings[i],ci_upper: samples[i].length ? percentile(samples[i], 97.5) : ratings[i],++ rank: pointRanks[i],++ rank_lo: rankSamples[i].length ? Math.round(percentile(rankSamples[i], 2.5)) : pointRanks[i],++ rank_hi: rankSamples[i].length ? Math.round(percentile(rankSamples[i], 97.5)) : pointRanks[i],...stats.get(subject),provisional: stats.get(subject).games < 10,})).sort((a, b) => b.rating - a.rating || String(a.subject).localeCompare(String(b.subject)));@@@ -171,4 +187,4 @@ function costPerSolved(entries) {return { cost_usd, solved, per_solved: solved ? cost_usd / solved : null };}--module.exports = { eloUpdate, bradleyTerry, voterWeights, wilson, pareto, costPerSolved, percentile };++module.exports = { eloUpdate, bradleyTerry, voterWeights, wilson, pareto, costPerSolved, percentile, ranksOf };nnew file mode 100644@@@ -0,0 +1,124 @@++'use strict';++++/**++ * The OpenVibe Index and the holdout gap (plan 2026-11-09, stage B3).++ *++ * The Index is one number per model per kind, on the 0-100 scale every other Bench score uses. It is a weighted++ * mean of two kinds of evidence:++ *++ * rating the model's Bradley-Terry rating from the human votes, put on 0-100 by the Elo expectation++ * against a 1000-rated opponent: 100 / (1 + 10 ** ((1000 - rating) / 400)). A 1000 rating is 50,++ * not a made-up score of its own.++ * benchmarks the median verified score of the model on each public benchmark of that kind (0-100).++ *++ * The weights are published with every answer: half the Index is the rating and half is split equally between the++ * kind's public benchmarks. A model that has no score on one of them gets that benchmark's weight renormalised++ * away, so the Index is always a weighted mean of the evidence the model actually has. Fewer than two components is++ * not an Index: the answer is null, never a number from one component.++ *++ * The holdout gap is the public score minus the held-out score, points on the same 0-100 scale, with the number of++ * scored tasks on each side. A kind without held-out tasks, or a model missing one side, gets a null gap.++ */++++/** The published weights: half the rating, half the objective benchmarks (split equally between them). */++const INDEX_WEIGHTS = Object.freeze({ rating: 0.5, benchmarks: 0.5 });++/** The rating the logistic is read against, so a 1000 rating is 50 points. */++const RATING_ANCHOR = 1000;++++const num = (v) => (v == null || v === '' || Number.isNaN(Number(v)) ? null : Number(v));++const round = (v, places = 4) => {++ if (v == null || !Number.isFinite(Number(v))) return null;++ const factor = 10 ** places;++ return Math.round(Number(v) * factor) / factor;++};++const clamp = (v, low, high) => Math.min(high, Math.max(low, v));++++function mean(values) {++ const list = (Array.isArray(values) ? values : []).map(num).filter((v) => v != null);++ return list.length ? list.reduce((sum, v) => sum + v, 0) / list.length : null;++}++++function median(values) {++ const list = (Array.isArray(values) ? values : []).map(num).filter((v) => v != null).sort((a, b) => a - b);++ if (!list.length) return null;++ const middle = Math.floor(list.length / 2);++ return list.length % 2 ? list[middle] : (list[middle - 1] + list[middle]) / 2;++}++++/** A Bradley-Terry rating on the 0-100 scale: the Elo expectation against a 1000-rated opponent, as points. */++function ratingScore(rating, anchor = RATING_ANCHOR) {++ const value = num(rating);++ if (value == null) return null;++ return 100 / (1 + 10 ** ((anchor - value) / 400));++}++++/**++ * The Index for one model: `{ index, n_components, weight_sum, components }`.++ *++ * `benchmarks` are the kind's public benchmarks the model has a verified median score on, as++ * `{ id, label?, score, runs? }`. `benchmarkCount` is how many public benchmarks the kind has; the benchmark half++ * of the weight is split over that whole list, so a model's missing benchmarks reduce its weight sum rather than++ * silently making its one benchmark worth half the Index. When the caller does not pass it, the list's length is++ * the count (the model has every benchmark of the kind).++ */++function openvibeIndex({ rating = null, benchmarks = [], benchmarkCount = null, weights = INDEX_WEIGHTS } = {}) {++ const ratingWeight = num(weights && weights.rating);++ const benchmarkWeight = num(weights && weights.benchmarks);++ const list = (Array.isArray(benchmarks) ? benchmarks : [])++ .filter((b) => b && num(b.score) != null)++ .map((b) => ({++ id: String(b.id),++ label: b.label == null ? String(b.id) : String(b.label),++ value: clamp(num(b.score), 0, 100),++ runs: b.runs == null ? null : num(b.runs),++ }));++ const count = Math.max(list.length, Math.max(0, num(benchmarkCount) || 0));++ const perBenchmark = count ? (benchmarkWeight || 0) / count : 0;++++ const components = [];++ const ratingValue = ratingScore(rating);++ if (ratingValue != null && ratingWeight != null) {++ components.push({ id: 'rating', kind: 'rating', label: 'Human rating', value: ratingValue, weight: ratingWeight, runs: null });++ }++ for (const benchmark of list) {++ components.push({ id: benchmark.id, kind: 'benchmark', label: benchmark.label, value: benchmark.value, weight: perBenchmark, runs: benchmark.runs });++ }++ const weightSum = components.reduce((sum, component) => sum + component.weight, 0);++ const value = components.length >= 2 && weightSum > 0++ ? components.reduce((sum, component) => sum + component.value * component.weight, 0) / weightSum++ : null;++ return {++ index: round(value, 2),++ n_components: components.length,++ weight_sum: round(weightSum, 6),++ benchmark_count: count,++ components: components.map((component) => ({++ ...component,++ value: round(component.value, 2),++ weight: round(component.weight, 6),++ })),++ };++}++++/**++ * The holdout gap for one model: the mean score on the public tasks minus the mean on the held-out tasks, with the++ * number of scored tasks on each side. A side with no scores makes the gap null, never zero.++ */++function holdoutGap(publicScores = [], holdoutScores = []) {++ const publicList = (Array.isArray(publicScores) ? publicScores : []).map(num).filter((v) => v != null);++ const holdoutList = (Array.isArray(holdoutScores) ? holdoutScores : []).map(num).filter((v) => v != null);++ const publicScore = mean(publicList);++ const holdoutScore = mean(holdoutList);++ return {++ public_n: publicList.length,++ public_score: round(publicScore, 2),++ holdout_n: holdoutList.length,++ holdout_score: round(holdoutScore, 2),++ gap: publicScore != null && holdoutScore != null ? round(publicScore - holdoutScore, 2) : null,++ };++}++++module.exports = {++ INDEX_WEIGHTS, RATING_ANCHOR,++ openvibeIndex, holdoutGap, ratingScore, mean, median,++};@@@ -34,7 +34,7 @@* Bradley-Terry so the numbers mean what a reader expects.*/--const { percentile } = require('./lib');++const { percentile, ranksOf } = require('./lib');/** Elo points per unit of log-odds: the usual 400 / ln(10). */const SCALE = 400 / Math.log(10);@@@ -381,6 +381,8 @@ function fitRatings(games, {const rawSamples = new Map();const ctrlSamples = new Map();++ const rawRankSamples = new Map();++ const ctrlRankSamples = new Map();const random = seededRandom(seed);const rounds = Math.max(0, Math.min(BOOTSTRAP, Number(bootstrap) || 0));const n = prepared.count;@@@ -391,16 +393,33 @@ function fitRatings(games, {for (let j = 0; j < n; j++) order[j] = Math.floor(random() * n);const r = fitPrepared(prepared, { order, use: [], ridge, maxIter: 60, warm: { theta: raw.theta, beta: [] } });const c = fitPrepared(prepared, { order, use: covariates, ridge, betaRidge, maxIter: 60, warm: { theta: controlled.theta, beta: controlled.beta } });++ const rawRanks = ranksOf(r.theta);++ const ctrlRanks = ranksOf(c.theta);for (const [subject, position] of rawIndex) {-- if (!rawSamples.has(subject)) { rawSamples.set(subject, []); ctrlSamples.set(subject, []); }++ if (!rawSamples.has(subject)) {++ rawSamples.set(subject, []); ctrlSamples.set(subject, []);++ rawRankSamples.set(subject, []); ctrlRankSamples.set(subject, []);++ }rawSamples.get(subject).push(r.theta[position]);-- ctrlSamples.get(subject).push(c.theta[ctrlIndex.has(subject) ? ctrlIndex.get(subject) : 0]);++ rawRankSamples.get(subject).push(rawRanks[position]);++ const ctrlPosition = ctrlIndex.has(subject) ? ctrlIndex.get(subject) : 0;++ ctrlSamples.get(subject).push(c.theta[ctrlPosition]);++ ctrlRankSamples.get(subject).push(ctrlRanks[ctrlPosition]);}}const toRating = (theta) => ANCHOR + SCALE * theta;const rawTheta = new Map(raw.subjects.map((s, i) => [s, raw.theta[i]]));const ctrlTheta = new Map(controlled.subjects.map((s, i) => [s, controlled.theta[i]]));++ const rawPointRankList = ranksOf(raw.theta);++ const ctrlPointRankList = ranksOf(controlled.theta);++ const rawPointRanks = new Map(raw.subjects.map((s, i) => [s, rawPointRankList[i]]));++ const ctrlPointRanks = new Map(controlled.subjects.map((s, i) => [s, ctrlPointRankList[i]]));++ const rankCell = (samples, point) => ({++ rank: point,++ rank_lo: samples.length ? Math.round(percentile(samples, 0.025)) : point,++ rank_hi: samples.length ? Math.round(percentile(samples, 0.975)) : point,++ });const rows = raw.subjects.map((subject) => {const thetaRaw = rawTheta.get(subject);const thetaCtrl = ctrlTheta.get(subject);@@@ -413,11 +432,13 @@ function fitRatings(games, {rating: Math.round(toRating(thetaRaw) * 100) / 100,ci_lower: rawP.length ? Math.round(toRating(percentile(rawP, 0.025)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,ci_upper: rawP.length ? Math.round(toRating(percentile(rawP, 0.975)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,++ ...rankCell(rawRankSamples.get(subject) || [], rawPointRanks.get(subject)),},controlled: {rating: Math.round(toRating(thetaCtrl) * 100) / 100,ci_lower: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.025)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,ci_upper: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.975)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,++ ...rankCell(ctrlRankSamples.get(subject) || [], ctrlPointRanks.get(subject)),},};}).sort((x, y) => y.raw.rating - x.raw.rating || x.model.localeCompare(y.model));@@@ -13,6 +13,7 @@ const clipLib = require('./bench/clip');const clipSource = require('./bench/clip-source');const ratingLib = require('./rating');const styleLib = require('./rating/style');++const statsLib = require('./rating/stats');const { checkSvg } = require('./bench/svg-check');const BASE_ELO = ratingLib.BASE_ELO;@@@ -203,6 +204,14 @@ function ratingCi(elo, wins, losses, ties) {function createStore(s, { log = console, random = Math.random, config = null } = {}) {const db = s.db;++ // The kind boards' fits are pure functions of the games, so a board asks for the same fit only once per++ // vote set. The key changes with the number of games, and a short TTL covers a vote that lands mid-request.++ const boardFits = new Map();++ const BOARD_BOOTSTRAP = 200;++ const BOARD_FIT_TTL_MS = 5 * 60_000;++ // A per-category board is published only when its category has this many games; below it there is not++ // enough play to separate models, so the board is left out of the category list rather than shown thin.++ const CATEGORY_MIN_GAMES = 30;// ── Outbox (in the same ambient transaction as the change that caused it) ──async function insertOutbox(topic, payload) {@@@ -460,24 +469,43 @@ function createStore(s, { log = console, random = Math.random, config = null } =}// ── Judgements ──────────────────────────────────────────++ /**++ * The pairwise games of a board, as the rating maths reads them: two model names and an outcome of++ * 'a', 'b' or 'tie'. The pair on a judgement names two entries, so the models come from the entries, never++ * from the entry ids.++ *++ * The category picks the games the board counts: 'all' every vote, 'people' the signed-in voters, 'anon' the++ * anonymous ones, and any other slug the task's own category (code, writing, chat, reasoning…).++ */async function pairwiseGames(kindId, category = CATEGORY_DEFAULT) {++ const cat = String(category == null ? CATEGORY_DEFAULT : category);const rows = await db.many(-- `SELECT * FROM bench_judgements-- WHERE kind_id = $1 AND method IN ('human_pairwise', 'game_outcome', 'audience_poll') AND pair IS NOT NULL-- AND ($2::text = 'all' OR outcome IS NOT NULL)-- ORDER BY at, id`,-- [kindId, category],++ `SELECT j.id, j.method, j.outcome, j.vote_counts, l.model AS left_model, e.model AS right_model++ FROM bench_judgements j++ JOIN bench_rounds rd ON rd.id = j.round_id++ JOIN bench_tasks t ON t.id = rd.task_id++ JOIN bench_entries l ON l.id = j.pair->>'left' AND l.round_id = j.round_id++ JOIN bench_entries e ON e.id = j.pair->>'right' AND e.round_id = j.round_id++ WHERE j.kind_id = $1++ AND j.method IN ('human_pairwise', 'game_outcome', 'audience_poll')++ AND j.pair IS NOT NULL AND j.outcome IS NOT NULL++ AND ($2::text <> 'people' OR COALESCE(j.voter_kind, 'person') = 'person')++ AND ($2::text <> 'anon' OR j.voter_kind = 'anon')++ AND ($2::text IN ('all', 'people', 'anon') OR t.task_class = $2)++ ORDER BY j.at, j.id`,++ [kindId, cat],);const out = [];for (const j of rows) {-- const pair = typeof j.pair === 'string' ? JSON.parse(j.pair) : j.pair;-- if (!pair || !pair.left || !pair.right) continue;if (j.outcome === 'both_bad') continue;if (j.method === 'audience_poll') {const counts = typeof j.vote_counts === 'string' ? JSON.parse(j.vote_counts) : j.vote_counts;if (!counts || !num(counts.voters)) continue;}-- out.push({ a: pair.left, b: pair.right, outcome: j.outcome === 'left' ? 1 : j.outcome === 'right' ? 0 : 0.5 });++ if (!j.left_model || !j.right_model || j.left_model === j.right_model) continue;++ const outcome = j.outcome === 'left' ? 'a' : j.outcome === 'right' ? 'b' : (j.outcome === 'tie' || j.outcome === 'draw') ? 'tie' : null;++ if (!outcome) continue;++ out.push({ id: j.id, a: j.left_model, b: j.right_model, outcome });}return out;}@@@ -640,23 +668,47 @@ function createStore(s, { log = console, random = Math.random, config = null } =return rows.map((r) => r.model);}++ /**++ * The Bradley-Terry fit of one board, cached on the vote set. The fit is the same maths the ratings page uses++ * (server/rating/lib.js): every game is one vote, the ratings are centred on 1000, and the bootstrap supplies++ * both the rating interval and the 95% rank interval. The seed is fixed, so the same games give the same board.++ */++ async function pairwiseFit(kindId, category) {++ const games = await pairwiseGames(kindId, category);++ const key = `${kindId}:${category}:${games.length}:${games.length ? games[games.length - 1].id : '-'}`;++ const hit = boardFits.get(key);++ if (hit && Date.now() - hit.at < BOARD_FIT_TTL_MS) return hit.fit;++ const fit = ratingLib.bradleyTerry(games, { bootstrap: BOARD_BOOTSTRAP, seed: 1 });++ boardFits.set(key, { at: Date.now(), fit });++ if (boardFits.size > 200) {++ const oldest = [...boardFits.entries()].sort((a, b) => a[1].at - b[1].at).slice(0, 100);++ for (const [stale] of oldest) boardFits.delete(stale);++ }++ return fit;++ }++async function boardElo(kindId, category = CATEGORY_DEFAULT) {const subjects = await subjectsFor(kindId);-- const ratings = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2', [kindId, category]);-- const by = new Map(ratings.map((r) => [r.subject, wireRating(r)]));++ const fit = await pairwiseFit(kindId, category);++ const by = new Map(fit.map((r) => [r.subject, r]));// A task category only lists the models that played in it: a model with no game in this kind of// work has no place on its board, rather than a baseline score it never earned.const taskCategory = ![CATEGORY_DEFAULT, CATEGORY_PEOPLE, CATEGORY_ANON].includes(category);-- const rows = (taskCategory ? subjects.filter((subject) => by.has(subject)) : subjects).map((subject) => {++ const listed = new Set(taskCategory ? subjects.filter((subject) => by.has(subject)) : subjects);++ for (const r of fit) listed.add(r.subject);++ const rows = [...listed].map((subject) => {const r = by.get(subject);const games = r ? r.games : 0;return {kind_id: kindId, category, subject,-- elo: r ? r.elo : BASE_ELO,++ elo: r ? Math.round(r.rating * 100) / 100 : BASE_ELO,games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,-- win_rate: r ? r.win_rate : 0,-- ci_lower: r ? r.ci_lower : null,-- ci_upper: r ? r.ci_upper : null,++ win_rate: games ? Math.round(((r.wins + r.ties * 0.5) / games) * 10000) / 10000 : 0,++ ci_lower: r ? Math.round(r.ci_lower * 100) / 100 : null,++ ci_upper: r ? Math.round(r.ci_upper * 100) / 100 : null,++ rank: r ? r.rank : null,++ rank_lo: r ? r.rank_lo : null,++ rank_hi: r ? r.rank_hi : null,provisional: games < 10,};});@@@ -664,6 +716,136 @@ function createStore(s, { log = console, random = Math.random, config = null } =return rows;}++ /**++ * The per-category boards: one for every task category of the kind with at least CATEGORY_MIN_GAMES games, in++ * the same shape as the kind's own board. A category below the floor is not listed at all, so a page can say++ * "not enough games yet" without drawing a board that cannot separate anything.++ */++ async function categoryBoards(kindId) {++ const rows = await db.many(++ `SELECT t.task_class AS category, COUNT(*)::int AS games++ FROM bench_judgements j++ JOIN bench_rounds rd ON rd.id = j.round_id++ JOIN bench_tasks t ON t.id = rd.task_id++ JOIN bench_entries l ON l.id = j.pair->>'left' AND l.round_id = j.round_id++ JOIN bench_entries e ON e.id = j.pair->>'right' AND e.round_id = j.round_id++ WHERE j.kind_id = $1++ AND j.method IN ('human_pairwise', 'game_outcome', 'audience_poll')++ AND j.pair IS NOT NULL AND j.outcome IS NOT NULL AND j.outcome <> 'both_bad'++ AND t.task_class IS NOT NULL++ AND t.task_class NOT IN ('all', 'people', 'anon')++ GROUP BY t.task_class++ HAVING COUNT(*) >= $2++ ORDER BY COUNT(*) DESC, t.task_class`,++ [kindId, CATEGORY_MIN_GAMES],++ );++ const out = [];++ for (const row of rows) {++ // Use the same eligible votes as the fit: self-pairs and empty polls do not earn a board.++ const games = (await pairwiseGames(kindId, row.category)).length;++ if (games < CATEGORY_MIN_GAMES) continue;++ out.push({ category: row.category, games, rows: await boardElo(kindId, row.category) });++ }++ return out.sort((a, b) => b.games - a.games || a.category.localeCompare(b.category));++ }++++ /**++ * The OpenVibe Index for a kind: the Bradley-Terry rating plus the model's median verified score on each public++ * benchmark of that kind, weighted 50/50 and split equally between the benchmarks. The weights are in the++ * answer; a model with fewer than two components gets null, never a number from one component.++ */++ async function kindIndex(kindId, eloRows) {++ const rows = await db.many(++ `SELECT b.id, b.title, r.model, r.score++ FROM bench_runs r++ JOIN bench_benchmarks b ON b.id = r.benchmark_id++ WHERE b.kind = $1 AND b.visibility = 'public' AND r.provenance = 'verified'++ AND r.score IS NOT NULL++ ORDER BY b.created_at, b.id, r.model, r.created_at, r.id`,++ [kindId],++ );++ const benchmarks = new Map();++ const scores = new Map();++ for (const row of rows) {++ if (!benchmarks.has(row.id)) benchmarks.set(row.id, { id: row.id, label: row.title });++ const key = `${row.id}\u0000${row.model}`;++ if (!scores.has(key)) scores.set(key, []);++ scores.get(key).push(num(row.score));++ }++ const benchmarkList = [...benchmarks.values()];++ const modelNames = new Set(eloRows.filter((row) => row.rank != null).map((row) => row.subject));++ for (const key of scores.keys()) modelNames.add(key.split('\u0000')[1]);++ const indexRows = [...modelNames].map((subject) => {++ const ratingRow = eloRows.find((row) => row.subject === subject);++ const present = benchmarkList++ .map((benchmark) => ({++ id: benchmark.id,++ label: benchmark.label,++ runs: (scores.get(`${benchmark.id}\u0000${subject}`) || []).length,++ score: statsLib.median(scores.get(`${benchmark.id}\u0000${subject}`) || []),++ }))++ .filter((benchmark) => benchmark.score != null);++ const value = statsLib.openvibeIndex({++ rating: ratingRow && ratingRow.rank != null ? ratingRow.elo : null,++ benchmarks: present,++ benchmarkCount: benchmarkList.length,++ });++ return { subject, ...value };++ });++ indexRows.sort((a, b) => {++ if (a.index == null && b.index == null) return a.subject.localeCompare(b.subject);++ if (a.index == null) return 1;++ if (b.index == null) return -1;++ return b.index - a.index || a.subject.localeCompare(b.subject);++ });++ return {++ kind_id: kindId,++ weights: { ...statsLib.INDEX_WEIGHTS },++ rating_anchor: statsLib.RATING_ANCHOR,++ benchmark_count: benchmarkList.length,++ benchmarks: benchmarkList.map((benchmark) => ({++ id: benchmark.id,++ label: benchmark.label,++ weight: benchmarkList.length ? Math.round((statsLib.INDEX_WEIGHTS.benchmarks / benchmarkList.length) * 1e6) / 1e6 : 0,++ })),++ rows: indexRows,++ };++ }++++ /**++ * The holdout gap for a kind: per model, the mean score on public tasks minus the mean score on held-out tasks,++ * with the number of scored tasks on each side. Verified runs of public benchmarks only. A kind without++ * held-out tasks says `available: false`; a model missing one side gets a null gap.++ */++ async function kindHoldout(kindId) {++ const rows = await db.many(++ `SELECT b.id AS benchmark_id, r.model, t.task_id, t.score, t.held_out++ FROM bench_run_tasks t++ JOIN bench_runs r ON r.id = t.run_id++ JOIN bench_benchmarks b ON b.id = r.benchmark_id++ WHERE b.kind = $1 AND b.visibility = 'public' AND r.provenance = 'verified'++ AND t.score IS NOT NULL++ ORDER BY r.model, t.created_at, t.id`,++ [kindId],++ );++ const by = new Map();++ for (const row of rows) {++ if (!by.has(row.model)) by.set(row.model, { public: [], holdout: [] });++ const bucket = by.get(row.model);++ if (row.held_out) bucket.holdout.push(num(row.score));++ else bucket.public.push(num(row.score));++ }++ const out = [...by.entries()].map(([subject, bucket]) => ({ subject, ...statsLib.holdoutGap(bucket.public, bucket.holdout) }));++ out.sort((a, b) => (b.gap == null ? -Infinity : b.gap) - (a.gap == null ? -Infinity : a.gap) || a.subject.localeCompare(b.subject));++ const available = rows.some((row) => row.held_out);++ return {++ kind_id: kindId,++ available,++ held_out_tasks: new Set(rows.filter((row) => row.held_out).map((row) => `${row.benchmark_id}:${row.task_id}`)).size,++ rows: out,++ };++ }++async function costStats(kindId) {const rows = await db.many(`SELECT e.model,@@@ -763,7 +945,19 @@ function createStore(s, { log = console, random = Math.random, config = null } =} else {return null;}-- return { kind_id: kindId, category: cat, period, type, rows, voters: await voterCounts(kindId) };++ // The kind's own statistics travel with every board: the rating board also carries its per-category++ // boards, and every type carries the OpenVibe Index and the holdout gap. The Index always reads the++ // whole kind's rating, even when the rows are filtered to one kind of voter.++ const allEloRows = cat === CATEGORY_DEFAULT ? eloRows : await boardElo(kindId);++ const [voterKinds, categories, index, holdout] = await Promise.all([++ voterCounts(kindId),++ type === 'elo' ? categoryBoards(kindId) : Promise.resolve(undefined),++ kindIndex(kindId, allEloRows),++ kindHoldout(kindId),++ ]);++ const out = { kind_id: kindId, category: cat, period, type, rows, voters: voterKinds, index, holdout };++ if (categories !== undefined) out.categories = categories;++ return out;}/**@@@ -830,8 +1024,25 @@ function createStore(s, { log = console, random = Math.random, config = null } =entries = (await db.many('SELECT * FROM bench_entries WHERE round_id = ANY($1::text[]) ORDER BY created_at, id', [roundIds])).map(wireEntry);judgements = (await db.many('SELECT * FROM bench_judgements WHERE round_id = ANY($1::text[]) ORDER BY at, id', [roundIds])).map((j) => wireJudgement(j));}-- const ratings = (await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 ORDER BY subject', [kindId])).map(wireRating);-- return { kind_id: kindId, tasks, rounds, entries, judgements, ratings };++ const allEloRows = await boardElo(kindId);++ const savedRatings = (await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 ORDER BY subject', [kindId])).map(wireRating);++ const fitted = new Map([[CATEGORY_DEFAULT, allEloRows]]);++ for (const category of new Set(savedRatings.map((rating) => rating.category))) {++ if (!fitted.has(category)) fitted.set(category, await boardElo(kindId, category));++ }++ // Export the same fit and intervals as the board, not the old incremental Elo beside a fitted rank.++ const ratings = [...fitted.values()].flat().filter((rating) => rating.games > 0).map((rating) => ({++ ...savedRatings.find((saved) => saved.subject === rating.subject && saved.category === rating.category),++ ...rating,++ }));++ const [categories, index, holdout] = await Promise.all([++ categoryBoards(kindId),++ kindIndex(kindId, allEloRows),++ kindHoldout(kindId),++ ]);++ // The open dataset carries the same four statistics the board API answers with: every rating's rank++ // interval, the per-category boards, the OpenVibe Index with its weights, and the holdout gap.++ return { kind_id: kindId, tasks, rounds, entries, judgements, ratings, categories, index, holdout };}async function datasetJsonl(kindId) {@@@ -843,6 +1054,9 @@ function createStore(s, { log = console, random = Math.random, config = null } =for (const e of d.entries) lines.push(JSON.stringify({ type: 'entry', ...e }));for (const j of d.judgements) lines.push(JSON.stringify({ type: 'judgement', ...j }));for (const r of d.ratings) lines.push(JSON.stringify({ type: 'rating', ...r }));++ for (const c of d.categories) lines.push(JSON.stringify({ type: 'category', ...c }));++ lines.push(JSON.stringify({ type: 'index', ...d.index }));++ lines.push(JSON.stringify({ type: 'holdout', ...d.holdout }));return lines.length ? `${lines.join('\n')}\n` : '';}@@@ -42,6 +42,128 @@ const { boot, check, done } = require('./helpers/boot');assert.ok(after.some((row) => row.games === 1 && row.provisional === true), 'one game is still provisional');});++ await check('every elo board carries 95% rank intervals from the bootstrap', async () => {++ const board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;++ assert.ok(board.rows.length >= 2);++ for (const row of board.rows) {++ assert.ok(Number.isInteger(row.rank) && row.rank >= 1 && row.rank <= board.rows.length, `${row.subject} rank`);++ assert.ok(Number.isInteger(row.rank_lo) && row.rank_lo >= 1, `${row.subject} rank_lo`);++ assert.ok(Number.isInteger(row.rank_hi) && row.rank_hi <= board.rows.length, `${row.subject} rank_hi`);++ assert.ok(row.rank_lo <= row.rank_hi, `${row.subject} interval is ordered`);++ assert.ok(row.ci_lower != null && row.ci_upper != null, `${row.subject} rating interval`);++ }++ const exported = (await t.get('/api/v1/datasets/design.page.jsonl')).text.trim().split('\n').map(JSON.parse);++ for (const row of board.rows) {++ const saved = exported.find((entry) => entry.type === 'rating' && entry.category === 'all' && entry.subject === row.subject);++ for (const field of ['elo', 'games', 'ci_lower', 'ci_upper', 'rank', 'rank_lo', 'rank_hi']) {++ assert.strictEqual(saved[field], row[field], `dataset ${row.subject} ${field} matches the board`);++ }++ }++ assert.ok(board.index.rows.every((row) => row.index === null), 'a rating alone is never an Index');++ assert.strictEqual(board.holdout.available, false, 'this kind has no held-out tasks');++ assert.deepStrictEqual(board.holdout.rows, []);++ });++++ // A category board: a task class with at least 30 games earns one; 29 does not.++ async function roundForCategory(category) {++ const task = await t.get('/api/v1/kinds/design.page/tasks', { as: alice, json: { prompt: 'A ' + category + ' page.', public: true, task_class: category } });++ const rr = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });++ const id = rr.json().round.id;++ const a = await t.get(`/api/v1/rounds/${id}/entries`, { bearer: entrySvc, json: { provider: 'acme', model: 'alpha-1', payload: { html: '<h1>a</h1>' } } });++ const b = await t.get(`/api/v1/rounds/${id}/entries`, { bearer: entrySvc, json: { provider: 'beta', model: 'beta-2', payload: { html: '<h1>b</h1>' } } });++ return { roundId: id, left: a.json().entry.id, right: b.json().entry.id };++ }++++ async function seedGames(round, count) {++ for (let i = 0; i < count; i++) {++ await t.ctx.s.db.exec(++ `INSERT INTO bench_judgements (id, round_id, kind_id, judge, method, pair, outcome, subject, voter_kind)++ VALUES ($1, $2, 'design.page', 'human_pairwise', 'human_pairwise', $3::jsonb, $4, $5, 'person')`,++ [t.store.newId('bju'), round.roundId, JSON.stringify({ left: round.left, right: round.right }), i % 2 ? 'left' : 'right', 'usr_seed_' + i],++ );++ }++ }++++ await check('a category board appears at 30 games and not at 29', async () => {++ const arcade = await roundForCategory('arcade');++ const writing = await roundForCategory('writing');++ await seedGames(writing, 29);++ const selfOnly = await roundForCategory('self-only');++ await seedGames(selfOnly, 30);++ await t.ctx.s.db.exec('UPDATE bench_entries SET model = $1 WHERE id = $2', ['alpha-1', selfOnly.right]);++ let board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;++ assert.strictEqual(board.categories.length, 0, '29 games is below the floor: ' + JSON.stringify(board.categories));++ await seedGames(arcade, 30);++ board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;++ const category = board.categories.find((c) => c.category === 'arcade');++ assert.ok(category, 'the 30-game category has a board');++ assert.strictEqual(category.games, 30);++ assert.ok(category.rows.every((row) => row.games > 0), 'only models that played are listed');++ assert.ok(category.rows.every((row) => Number.isInteger(row.rank_lo) && row.rank_lo <= row.rank_hi), 'the category board has rank intervals');++ assert.strictEqual(board.categories.find((c) => c.category === 'writing'), undefined, '29 games is still out');++ // The filtered board answers for one category alone.++ const one = (await t.get('/api/v1/boards/design.page?category=arcade')).json().board;++ assert.strictEqual(one.category, 'arcade');++ assert.strictEqual(one.rows.reduce((n, row) => n + row.games, 0), 60, '30 games are 60 model-games');++ });++++ await check('the OpenVibe Index needs two components and publishes its weights', async () => {++ const definition = {++ title: 'Design pages', description: 'The B3 index fixture.', kind: 'design.page', tags: [], visibility: 'public',++ tasks: [++ { id: 't1', prompt: 'A public page.', files: [], expected: null, holdout: false },++ { id: 't2', prompt: 'A held-out page.', files: [], expected: null, holdout: true },++ ],++ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },++ limits: { ms: 10000, tokens: 10000, usd: 1 },++ matrix: { models: ['alpha-1'], harnesses: ['fixture'] },++ };++ const writer = t.network.serviceToken({ sub: 'svc:fixture', audience: 'openvibe.bench', cap: ['bench.benchmark.write', 'bench.run.publish'] });++ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition });++ assert.strictEqual(created.status, 201, created.text);++ const benchmarkId = created.json().benchmark.id;++ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: writer, json: {} });++ assert.strictEqual(frozen.status, 201, frozen.text);++ const run = await t.get('/api/v1/runs', {++ bearer: writer,++ json: {++ benchmark: `${benchmarkId}@1`, model: 'alpha-1', route: { provider: 'acme' }, provider: 'acme',++ harness: { name: 'fixture', version: '1' }, settings: {}, executor: { kind: 'codes', id: 'fixture' },++ tasks: [++ { task_id: 't1', score: 90, cost_usd: 0.01, ms: 100, tokens: 10, artifacts: [] },++ { task_id: 't2', score: 60, cost_usd: 0.01, ms: 100, tokens: 10, artifacts: [] },++ ],++ score: 75, cost_usd: 0.02, ms: 200, tokens: 20,++ },++ });++ assert.strictEqual(run.status, 201, run.text);++++ const board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;++ assert.deepStrictEqual(board.index.weights, { rating: 0.5, benchmarks: 0.5 });++ assert.strictEqual(board.index.benchmark_count, 1);++ assert.strictEqual(board.index.benchmarks[0].weight, 0.5);++ const alpha = board.index.rows.find((row) => row.subject === 'alpha-1');++ const beta = board.index.rows.find((row) => row.subject === 'beta-2');++ assert.ok(alpha, 'the rated model is in the index');++ assert.strictEqual(alpha.n_components, 2, 'rating + one benchmark: ' + JSON.stringify(alpha.components));++ assert.ok(typeof alpha.index === 'number' && alpha.index >= 0 && alpha.index <= 100, 'the index is 0-100');++ assert.ok(alpha.components.some((c) => c.kind === 'rating') && alpha.components.some((c) => c.kind === 'benchmark'), 'both kinds of component are named');++ assert.strictEqual(beta.index, null, 'a single component is never an index');++ assert.strictEqual(beta.n_components, 1);++++ assert.strictEqual(board.holdout.available, true);++ const gap = board.holdout.rows.find((row) => row.subject === 'alpha-1');++ assert.deepStrictEqual(gap, { subject: 'alpha-1', public_n: 1, public_score: 90, holdout_n: 1, holdout_score: 60, gap: 30 });++++ const dataset = (await t.get('/api/v1/datasets/design.page.jsonl')).text.trim().split('\n').map((line) => JSON.parse(line));++ assert.ok(dataset.some((line) => line.type === 'index' && line.weights.rating === 0.5), 'the dataset carries the index');++ assert.ok(dataset.some((line) => line.type === 'holdout' && line.rows.some((row) => row.gap === 30)), 'the dataset carries the holdout gap');++ assert.ok(dataset.some((line) => line.type === 'category' && line.category === 'arcade'), 'the dataset carries the category board');++ assert.ok(dataset.some((line) => line.type === 'rating' && line.rank_lo !== null), 'the dataset carries the rank interval');++ });++++await check('a model page aggregates across kinds', async () => {const task = await t.get('/api/v1/kinds/games.match/tasks', { as: alice, json: { prompt: 'A match.', public: true } });const rr = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });nnew file mode 100644@@@ -0,0 +1,34 @@++{++ "cases": [++ {++ "name": "a public set and a holdout: the gap is public minus holdout",++ "public": [80, 90],++ "holdout": [60],++ "expect": { "public_n": 2, "public_score": 85, "holdout_n": 1, "holdout_score": 60, "gap": 25 }++ },++ {++ "name": "no held-out score: the gap is null, never zero",++ "public": [90, 70],++ "holdout": [],++ "expect": { "public_n": 2, "public_score": 80, "holdout_n": 0, "holdout_score": null, "gap": null }++ },++ {++ "name": "no public score: the gap is null too",++ "public": [],++ "holdout": [40],++ "expect": { "public_n": 0, "public_score": null, "holdout_n": 1, "holdout_score": 40, "gap": null }++ },++ {++ "name": "nothing scored: every number is absent, not zero",++ "public": [],++ "holdout": [],++ "expect": { "public_n": 0, "public_score": null, "holdout_n": 0, "holdout_score": null, "gap": null }++ },++ {++ "name": "a negative gap says the held-out set scored higher",++ "public": [55.5],++ "holdout": [70.25],++ "expect": { "public_n": 1, "public_score": 55.5, "holdout_n": 1, "holdout_score": 70.25, "gap": -14.75 }++ }++ ]++}nnew file mode 100644@@@ -0,0 +1,50 @@++{++ "weights": { "rating": 0.5, "benchmarks": 0.5 },++ "cases": [++ {++ "name": "a rating alone is not an Index",++ "rating": 1200,++ "benchmarks": [],++ "benchmarkCount": 0,++ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5 }++ },++ {++ "name": "one benchmark alone is not an Index either",++ "rating": null,++ "benchmarks": [{ "id": "b1", "label": "One", "score": 80, "runs": 2 }],++ "benchmarkCount": 1,++ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5 }++ },++ {++ "name": "a rating and one of two benchmarks: the missing benchmark's weight renormalises",++ "rating": 1000,++ "benchmarks": [{ "id": "b1", "label": "One", "score": 80, "runs": 3 }],++ "benchmarkCount": 2,++ "expect": { "index": 60, "n_components": 2, "weight_sum": 0.75 }++ },++ {++ "name": "two benchmarks and no rating still make an Index",++ "rating": null,++ "benchmarks": [++ { "id": "b1", "label": "One", "score": 40, "runs": 1 },++ { "id": "b2", "label": "Two", "score": 80, "runs": 1 }++ ],++ "benchmarkCount": 2,++ "expect": { "index": 60, "n_components": 2, "weight_sum": 0.5 }++ },++ {++ "name": "a score over 100 is clamped, not trusted",++ "rating": 1000,++ "benchmarks": [{ "id": "b1", "label": "One", "score": 140, "runs": 1 }],++ "benchmarkCount": 1,++ "expect": { "index": 75, "n_components": 2, "weight_sum": 1 }++ },++ {++ "name": "the rating is the Elo expectation against 1000, as 0-100 points",++ "rating": 1000,++ "benchmarks": [],++ "benchmarkCount": 0,++ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5, "rating_value": 50 }++ }++ ]++}@@@ -105,3 +105,35 @@ test('percentile interpolates and handles empty input', () => {assert.ok(Math.abs(percentile([1, 2, 3, 4], 95) - 3.85) < 1e-12);assert.equal(percentile([], 95), null);});++++test('rank intervals come from the same bootstrap as the rating intervals', () => {++ const games = [++ ...Array.from({ length: 24 }, () => game('A', 'B', 'a')),++ ...Array.from({ length: 6 }, () => game('A', 'B', 'b')),++ ...Array.from({ length: 20 }, () => game('B', 'C', 'a')),++ ...Array.from({ length: 10 }, () => game('B', 'C', 'b')),++ ...Array.from({ length: 25 }, () => game('A', 'C', 'a')),++ ...Array.from({ length: 5 }, () => game('A', 'C', 'b')),++ ];++ const fit = bradleyTerry(games, { bootstrap: 200, seed: 11 });++ const by = bySubject(fit);++ assert.equal(by.get('A').rank, 1, 'the planted winner is first');++ assert.equal(by.get('A').rank_lo, 1);++ assert.ok(by.get('A').rank_hi >= 1);++ assert.equal(by.get('C').rank, 3);++ for (const row of fit) {++ assert.ok(Number.isInteger(row.rank) && row.rank >= 1 && row.rank <= fit.length, `${row.subject} rank`);++ assert.ok(Number.isInteger(row.rank_lo) && row.rank_lo >= 1, `${row.subject} rank_lo`);++ assert.ok(Number.isInteger(row.rank_hi) && row.rank_hi <= fit.length, `${row.subject} rank_hi`);++ assert.ok(row.rank_lo <= row.rank_hi, `${row.subject} interval is ordered`);++ }++ assert.deepEqual(bradleyTerry(games, { bootstrap: 200, seed: 11 }), fit, 'the rank intervals are seeded');++});++++test('without a bootstrap the rank interval is the point rank, never a fraction', () => {++ const fit = bradleyTerry([game('A', 'B', 'a'), game('A', 'B', 'a'), game('B', 'C', 'a')], { bootstrap: 0 });++ for (const row of fit) {++ assert.equal(row.rank_lo, row.rank);++ assert.equal(row.rank_hi, row.rank);++ }++});nnew file mode 100644@@@ -0,0 +1,71 @@++'use strict';++++/**++ * The OpenVibe Index and the holdout gap (stage B3): the published weights, the null for one component, the++ * renormalisation over the evidence a model actually has, and the holdout gap's n on both sides.++ */++const test = require('node:test');++const assert = require('node:assert/strict');++const fs = require('node:fs');++const path = require('node:path');++const { openvibeIndex, holdoutGap, ratingScore, INDEX_WEIGHTS, RATING_ANCHOR } = require('../server/rating/stats');++++const fixture = (name) => JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures/rating', name), 'utf8'));++++test('the published weights are the ones the docs and the API name', () => {++ assert.deepEqual(INDEX_WEIGHTS, { rating: 0.5, benchmarks: 0.5 });++ assert.equal(RATING_ANCHOR, 1000);++ assert.equal(ratingScore(1000), 50);++ assert.ok(Math.abs(ratingScore(1200) - 75.97469266479578) < 1e-9);++ assert.equal(ratingScore(null), null);++});++++test('the Index fixture: two components or null, with the weights in the answer', () => {++ for (const testCase of fixture('index-cases.json').cases) {++ const got = openvibeIndex({++ rating: testCase.rating,++ benchmarks: testCase.benchmarks,++ benchmarkCount: testCase.benchmarkCount,++ });++ assert.equal(got.index, testCase.expect.index, testCase.name);++ assert.equal(got.n_components, testCase.expect.n_components, testCase.name);++ assert.equal(got.weight_sum, testCase.expect.weight_sum, testCase.name);++ if (testCase.expect.rating_value != null) {++ assert.equal(got.components[0].value, testCase.expect.rating_value, testCase.name);++ }++ for (const component of got.components) {++ assert.ok(component.weight > 0, testCase.name + ': every component has its weight');++ }++ }++});++++test('the Index never invents a score from one component, and null is not zero', () => {++ assert.equal(openvibeIndex({ rating: 1300 }).index, null);++ assert.equal(openvibeIndex({ benchmarks: [{ id: 'b', score: 99 }] }).index, null);++ assert.equal(openvibeIndex({}).index, null);++ assert.equal(openvibeIndex({ rating: 1100, benchmarks: [{ id: 'b', score: 90 }] }).index != null, true);++});++++test('a model missing one benchmark of the kind gets the weight renormalised, not a second benchmark', () => {++ const got = openvibeIndex({++ rating: 1000,++ benchmarks: [{ id: 'b1', score: 80 }],++ benchmarkCount: 3,++ });++ assert.equal(got.n_components, 2);++ assert.equal(got.weight_sum, 0.666667);++ assert.equal(got.components.find((c) => c.id === 'b1').weight, 0.166667);++ assert.ok(got.index > 50 && got.index < 80);++});++++test('the holdout gap fixture: public minus holdout, with n on each side', () => {++ for (const testCase of fixture('holdout-cases.json').cases) {++ assert.deepEqual(holdoutGap(testCase.public, testCase.holdout), testCase.expect, testCase.name);++ }++});++++test('the holdout gap ignores missing scores and keeps the side count honest', () => {++ assert.deepEqual(holdoutGap([80, null, undefined], [null]), {++ public_n: 1, public_score: 80, holdout_n: 0, holdout_score: null, gap: null,++ });++});@@@ -86,6 +86,25 @@ function rng(seed) {assert.strictEqual(style.bandLabel(1, 1), '1st');});++ await check('the fit publishes rank intervals for both boards', async () => {++ const games = [];++ for (let i = 0; i < 60; i++) {++ games.push({ id: 'ra' + i, a: 'alpha', b: 'beta', outcome: 'a', fa: { length: 100, markup: 1 }, fb: { length: 100, markup: 1 } });++ games.push({ id: 'rb' + i, a: 'beta', b: 'gamma', outcome: 'a', fa: { length: 100, markup: 1 }, fb: { length: 100, markup: 1 } });++ }++ const fit = style.fitRatings(games, { bootstrap: 80 });++ const by = new Map(fit.models.map((m) => [m.model, m]));++ assert.strictEqual(by.get('alpha').raw.rank, 1);++ for (const model of fit.models) {++ for (const cell of [model.raw, model.controlled]) {++ assert.ok(Number.isInteger(cell.rank) && cell.rank >= 1 && cell.rank <= fit.models.length, model.model + ' rank');++ assert.ok(Number.isInteger(cell.rank_lo) && cell.rank_lo >= 1, model.model + ' rank_lo');++ assert.ok(Number.isInteger(cell.rank_hi) && cell.rank_hi <= fit.models.length, model.model + ' rank_hi');++ assert.ok(cell.rank_lo <= cell.rank_hi, model.model + ' ordered interval');++ }++ }++ });++await check('the style features count length and markup', async () => {const f = style.styleFeatures({ html: '<h1>Title</h1><p><strong>bold</strong></p><ul><li>one</li></ul>' });assert.ok(f.length > 20, 'length: ' + f.length);
@@ -229,6 +229,22 @@ observations is returned only when every observation counts that outcome; a miss
@@ -229,6 +229,22 @@ observations is returned only when every observation counts that outcome; a miss
ignores any such fields, and the read routes never join a task, benchmark or entry. The only text in a row is an
ignores any such fields, and the read routes never join a task, benchmark or entry. The only text in a row is an
exact id, a family, and the id of the round or run that produced it.
exact id, a family, and the id of the round or run that produced it.
+## The boards: ranks, categories, the Index and the holdout
+
+`GET /api/v1/boards/:kindId?type=elo` returns the fit, not just the Elo updates: each row carries `elo`, its
+`ci_lower`/`ci_upper` rating interval, and `rank`/`rank_lo`/`rank_hi` — the 95% rank interval from the same bootstrap.
+The board also carries:
+
+| Field | What it holds |
+|---|---|
+| `categories` | One board per task category with at least 30 games: `{ category, games, rows }`, each row fitted the same way. A category below the floor is left out. `?category=<slug>` answers for one category alone. |
+| `index` | The OpenVibe Index per model: `{ weights, rating_anchor, benchmark_count, benchmarks[], rows[] }`. Each row is `{ subject, index, n_components, weight_sum, components[] }`; `index` is null when fewer than two components exist. |
+| `holdout` | `{ available, held_out_tasks, rows[] }`; each row is `{ subject, public_n, public_score, holdout_n, holdout_score, gap }`, with a null gap when either side has no scores. |
+
+The open dataset (`/api/v1/datasets/:kindId.jsonl`) also emits `type: "category"`, `type: "index"` and
+`type: "holdout"` lines, and its rating lines carry the rank interval. The formulas, the 0.5/0.5 weights and the
+30-game floor are in [index.md](index.md).
+
## The older API
## The older API
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
new file mode 100644
new file mode 100644
@@ -0,0 +1,60 @@
@@ -0,0 +1,60 @@
+# The numbers on Bench
+
+Every board is a Bradley-Terry fit over the votes, centred on 1000, with a bootstrap that gives each model a
+rating interval and a 95% rank interval. A kind's board answers at `GET /api/v1/boards/:kindId`; the open dataset
+for the kind (`GET /api/v1/datasets/:kindId.jsonl`) carries the same numbers. The shapes are in
+[api.md](api.md).
+
+## Rank intervals
+
+A rank is a band, not a place. Every fit resamples the games 200 times with a fixed seed, ranks the models in each
+resample (equal ratings share a rank), and takes the 2.5th and 97.5th percentile of those places:
+
+```json
+{ "subject": "model-a", "elo": 1042.5, "rank": 2, "rank_lo": 1, "rank_hi": 4 }
+```
+
+`rank` is the place on the fitted board. `rank_lo` and `rank_hi` are the 95% rank interval: the places the votes
+allow. A model whose interval contains 1 may be first; overlapping intervals are why the page shows "1st–3rd" and
+never pretends the order is settled.
+
+## Per-category boards
+
+A task can carry a `task_class` (code, writing, chat, reasoning, arcade, …). Every vote is counted in the task's
+category as well as the kind's whole board. The elo board publishes `categories`: one board per category with at
+least **30 games**, fitted the same way (same Bradley-Terry maths, same bootstrap intervals, same rank intervals).
+A category below 30 games is left out; it is not shown with a thin score. `GET /api/v1/boards/:kindId?category=arcade`
+answers for one category alone.
+
+## The OpenVibe Index
+
+The Index is one number per model per kind, on the same 0-100 scale as every other Bench score. It is a weighted mean
+of two kinds of evidence:
+
+| Component | Weight | Value |
+|---|---|---|
+| The model's human rating | 0.5 | Its Bradley-Terry rating put on 0-100 by the Elo expectation against a 1000-rated opponent: `100 / (1 + 10 ** ((1000 - rating) / 400))`. A 1000 rating is exactly 50. |
+| The kind's public benchmarks | 0.5, split equally between them | The model's median verified score (0-100) on each public benchmark of that kind. |
+
+The weights are published in every answer (`index.weights` and the per-benchmark `index.benchmarks[].weight`) and in
+each row's `components`. A model missing a benchmark gets that benchmark's weight renormalised away rather than
+letting one benchmark stand in for half the Index; `weight_sum` says how much of the 1.0 the model actually has.
+
+**Fewer than two components is not an Index.** A model with only a rating, or only one benchmark, gets `index: null`.
+Null is not zero and is not sorted as a score.
+
+## The holdout gap
+
+A benchmark task may be held out (`holdout: true`): only the executor and the benchmark's owner see its prompt. The
+holdout gap compares the same model's verified scores on the public tasks and on the held-out tasks of the kind:
+
+```
+gap = mean(public task scores) - mean(held-out task scores)
+```
+
+The answer carries `public_n`, `public_score`, `holdout_n`, `holdout_score` and `gap`. A kind with no held-out tasks
+says `available: false`. A model missing either side gets `gap: null`, never zero: a gap cannot be read from one side.
+A large positive gap is a contamination signal, not a proof; it says the held-out set scored lower and is worth a look.
+
+Both statistics travel with every board (`index` and `holdout`), and the open dataset has `type: "index"` and
+`type: "holdout"` lines beside the games, so a reader can check them without the page.
@@ -79,8 +79,13 @@ const lib = require('./lib');
@@ -79,8 +79,13 @@ const lib = require('./lib');
// each output as covariates, bootstrap intervals and rank bands.
// each output as covariates, bootstrap intervals and rank bands.
const style = require('./style');
const style = require('./style');
+// The OpenVibe Index and the holdout gap (stage B3): pure maths, so a test and a page agree on the numbers.
+const stats = require('./stats');
+
module.exports = {
module.exports = {
eloUpdate, bradleyTerry: lib.bradleyTerry, voterWeights: lib.voterWeights, wilson: lib.wilson, percentile: lib.percentile, pareto, costPerSolved, BASE_ELO, K_DEFAULT,
eloUpdate, bradleyTerry: lib.bradleyTerry, voterWeights: lib.voterWeights, wilson: lib.wilson, percentile: lib.percentile, pareto, costPerSolved, BASE_ELO, K_DEFAULT,
style,
style,
fitRatings: style.fitRatings, rankBands: style.rankBands, styleFeatures: style.styleFeatures, markupCount: style.markupCount,
fitRatings: style.fitRatings, rankBands: style.rankBands, styleFeatures: style.styleFeatures, markupCount: style.markupCount,
+ openvibeIndex: stats.openvibeIndex, holdoutGap: stats.holdoutGap, ratingScore: stats.ratingScore,
+ INDEX_WEIGHTS: stats.INDEX_WEIGHTS, RATING_ANCHOR: stats.RATING_ANCHOR,
};
};
@@ -92,6 +92,16 @@ function seededRandom(seed) {
@@ -92,6 +92,16 @@ function seededRandom(seed) {
};
};
}
}
+/**
+ * Competition ranks by strength: the best subject is 1st and equal ratings share the best rank. The bootstrap
+ * ranks use the same rule on every resampled fit, so a rank interval is the spread of the places the votes allow,
+ * not a second guess from the two ends of the rating interval.
+ */
+function ranksOf(ratings) {
+ const list = Array.isArray(ratings) ? ratings : [];
+ return list.map((rating) => 1 + list.filter((other) => other > rating + 1e-9).length);
+}
+
function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterations = 200, weights } = {}) {
function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterations = 200, weights } = {}) {
const active = games.filter(game => game.outcome !== 'both_bad');
const active = games.filter(game => game.outcome !== 'both_bad');
const subjects = [...new Set(active.flatMap(game => [game.a, game.b]))].sort();
const subjects = [...new Set(active.flatMap(game => [game.a, game.b]))].sort();
@@ -110,19 +120,25 @@ function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterati
@@ -110,19 +120,25 @@ function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterati
}
}
const ratings = ratingsFor(active, subjects, weights, iterations, anchor);
const ratings = ratingsFor(active, subjects, weights, iterations, anchor);
const samples = subjects.map(() => []);
const samples = subjects.map(() => []);
+ const rankSamples = subjects.map(() => []);
if (bootstrap > 0 && active.length) {
if (bootstrap > 0 && active.length) {
const random = seededRandom(seed);
const random = seededRandom(seed);
for (let i = 0; i < bootstrap; i++) {
for (let i = 0; i < bootstrap; i++) {
const resampled = Array.from({ length: active.length }, () => active[Math.floor(random() * active.length)]);
const resampled = Array.from({ length: active.length }, () => active[Math.floor(random() * active.length)]);
const draw = ratingsFor(resampled, subjects, weights, iterations, anchor);
const draw = ratingsFor(resampled, subjects, weights, iterations, anchor);
draw.forEach((rating, j) => samples[j].push(rating));
draw.forEach((rating, j) => samples[j].push(rating));
+ ranksOf(draw).forEach((rank, j) => rankSamples[j].push(rank));
}
}
}
}
+ const pointRanks = ranksOf(ratings);
return subjects.map((subject, i) => ({
return subjects.map((subject, i) => ({
subject,
subject,
rating: ratings[i],
rating: ratings[i],
ci_lower: samples[i].length ? percentile(samples[i], 2.5) : ratings[i],
ci_lower: samples[i].length ? percentile(samples[i], 2.5) : ratings[i],
ci_upper: samples[i].length ? percentile(samples[i], 97.5) : ratings[i],
ci_upper: samples[i].length ? percentile(samples[i], 97.5) : ratings[i],
+ rank: pointRanks[i],
+ rank_lo: rankSamples[i].length ? Math.round(percentile(rankSamples[i], 2.5)) : pointRanks[i],
+ rank_hi: rankSamples[i].length ? Math.round(percentile(rankSamples[i], 97.5)) : pointRanks[i],
...stats.get(subject),
...stats.get(subject),
provisional: stats.get(subject).games < 10,
provisional: stats.get(subject).games < 10,
})).sort((a, b) => b.rating - a.rating || String(a.subject).localeCompare(String(b.subject)));
})).sort((a, b) => b.rating - a.rating || String(a.subject).localeCompare(String(b.subject)));
@@ -171,4 +187,4 @@ function costPerSolved(entries) {
@@ -171,4 +187,4 @@ function costPerSolved(entries) {
return { cost_usd, solved, per_solved: solved ? cost_usd / solved : null };
return { cost_usd, solved, per_solved: solved ? cost_usd / solved : null };
}
}
-module.exports = { eloUpdate, bradleyTerry, voterWeights, wilson, pareto, costPerSolved, percentile };
+module.exports = { eloUpdate, bradleyTerry, voterWeights, wilson, pareto, costPerSolved, percentile, ranksOf };
new file mode 100644
new file mode 100644
@@ -0,0 +1,124 @@
@@ -0,0 +1,124 @@
+'use strict';
+
+/**
+ * The OpenVibe Index and the holdout gap (plan 2026-11-09, stage B3).
+ *
+ * The Index is one number per model per kind, on the 0-100 scale every other Bench score uses. It is a weighted
+ * mean of two kinds of evidence:
+ *
+ * rating the model's Bradley-Terry rating from the human votes, put on 0-100 by the Elo expectation
+ * against a 1000-rated opponent: 100 / (1 + 10 ** ((1000 - rating) / 400)). A 1000 rating is 50,
+ * not a made-up score of its own.
+ * benchmarks the median verified score of the model on each public benchmark of that kind (0-100).
+ *
+ * The weights are published with every answer: half the Index is the rating and half is split equally between the
+ * kind's public benchmarks. A model that has no score on one of them gets that benchmark's weight renormalised
+ * away, so the Index is always a weighted mean of the evidence the model actually has. Fewer than two components is
+ * not an Index: the answer is null, never a number from one component.
+ *
+ * The holdout gap is the public score minus the held-out score, points on the same 0-100 scale, with the number of
+ * scored tasks on each side. A kind without held-out tasks, or a model missing one side, gets a null gap.
+ */
+
+/** The published weights: half the rating, half the objective benchmarks (split equally between them). */
+const INDEX_WEIGHTS = Object.freeze({ rating: 0.5, benchmarks: 0.5 });
+/** The rating the logistic is read against, so a 1000 rating is 50 points. */
+const RATING_ANCHOR = 1000;
+
+const num = (v) => (v == null || v === '' || Number.isNaN(Number(v)) ? null : Number(v));
+const round = (v, places = 4) => {
+ if (v == null || !Number.isFinite(Number(v))) return null;
+ const factor = 10 ** places;
+ return Math.round(Number(v) * factor) / factor;
+};
+const clamp = (v, low, high) => Math.min(high, Math.max(low, v));
+
+function mean(values) {
+ const list = (Array.isArray(values) ? values : []).map(num).filter((v) => v != null);
+ return list.length ? list.reduce((sum, v) => sum + v, 0) / list.length : null;
+}
+
+function median(values) {
+ const list = (Array.isArray(values) ? values : []).map(num).filter((v) => v != null).sort((a, b) => a - b);
+ if (!list.length) return null;
+ const middle = Math.floor(list.length / 2);
+ return list.length % 2 ? list[middle] : (list[middle - 1] + list[middle]) / 2;
+}
+
+/** A Bradley-Terry rating on the 0-100 scale: the Elo expectation against a 1000-rated opponent, as points. */
+function ratingScore(rating, anchor = RATING_ANCHOR) {
+ const value = num(rating);
+ if (value == null) return null;
+ return 100 / (1 + 10 ** ((anchor - value) / 400));
+}
+
+/**
+ * The Index for one model: `{ index, n_components, weight_sum, components }`.
+ *
+ * `benchmarks` are the kind's public benchmarks the model has a verified median score on, as
+ * `{ id, label?, score, runs? }`. `benchmarkCount` is how many public benchmarks the kind has; the benchmark half
+ * of the weight is split over that whole list, so a model's missing benchmarks reduce its weight sum rather than
+ * silently making its one benchmark worth half the Index. When the caller does not pass it, the list's length is
+ * the count (the model has every benchmark of the kind).
+ */
+function openvibeIndex({ rating = null, benchmarks = [], benchmarkCount = null, weights = INDEX_WEIGHTS } = {}) {
+ const ratingWeight = num(weights && weights.rating);
+ const benchmarkWeight = num(weights && weights.benchmarks);
+ const list = (Array.isArray(benchmarks) ? benchmarks : [])
+ .filter((b) => b && num(b.score) != null)
+ .map((b) => ({
+ id: String(b.id),
+ label: b.label == null ? String(b.id) : String(b.label),
+ value: clamp(num(b.score), 0, 100),
+ runs: b.runs == null ? null : num(b.runs),
+ }));
+ const count = Math.max(list.length, Math.max(0, num(benchmarkCount) || 0));
+ const perBenchmark = count ? (benchmarkWeight || 0) / count : 0;
+
+ const components = [];
+ const ratingValue = ratingScore(rating);
+ if (ratingValue != null && ratingWeight != null) {
+ components.push({ id: 'rating', kind: 'rating', label: 'Human rating', value: ratingValue, weight: ratingWeight, runs: null });
+ }
+ for (const benchmark of list) {
+ components.push({ id: benchmark.id, kind: 'benchmark', label: benchmark.label, value: benchmark.value, weight: perBenchmark, runs: benchmark.runs });
+ }
+ const weightSum = components.reduce((sum, component) => sum + component.weight, 0);
+ const value = components.length >= 2 && weightSum > 0
+ ? components.reduce((sum, component) => sum + component.value * component.weight, 0) / weightSum
+ : null;
+ return {
+ index: round(value, 2),
+ n_components: components.length,
+ weight_sum: round(weightSum, 6),
+ benchmark_count: count,
+ components: components.map((component) => ({
+ ...component,
+ value: round(component.value, 2),
+ weight: round(component.weight, 6),
+ })),
+ };
+}
+
+/**
+ * The holdout gap for one model: the mean score on the public tasks minus the mean on the held-out tasks, with the
+ * number of scored tasks on each side. A side with no scores makes the gap null, never zero.
+ */
+function holdoutGap(publicScores = [], holdoutScores = []) {
+ const publicList = (Array.isArray(publicScores) ? publicScores : []).map(num).filter((v) => v != null);
+ const holdoutList = (Array.isArray(holdoutScores) ? holdoutScores : []).map(num).filter((v) => v != null);
+ const publicScore = mean(publicList);
+ const holdoutScore = mean(holdoutList);
+ return {
+ public_n: publicList.length,
+ public_score: round(publicScore, 2),
+ holdout_n: holdoutList.length,
+ holdout_score: round(holdoutScore, 2),
+ gap: publicScore != null && holdoutScore != null ? round(publicScore - holdoutScore, 2) : null,
+ };
+}
+
+module.exports = {
+ INDEX_WEIGHTS, RATING_ANCHOR,
+ openvibeIndex, holdoutGap, ratingScore, mean, median,
+};
@@ -34,7 +34,7 @@
@@ -34,7 +34,7 @@
* Bradley-Terry so the numbers mean what a reader expects.
* Bradley-Terry so the numbers mean what a reader expects.
*/
*/
-const { percentile } = require('./lib');
+const { percentile, ranksOf } = require('./lib');
/** Elo points per unit of log-odds: the usual 400 / ln(10). */
/** Elo points per unit of log-odds: the usual 400 / ln(10). */
const SCALE = 400 / Math.log(10);
const SCALE = 400 / Math.log(10);
@@ -381,6 +381,8 @@ function fitRatings(games, {
@@ -381,6 +381,8 @@ function fitRatings(games, {
const rawSamples = new Map();
const rawSamples = new Map();
const ctrlSamples = new Map();
const ctrlSamples = new Map();
+ const rawRankSamples = new Map();
+ const ctrlRankSamples = new Map();
const random = seededRandom(seed);
const random = seededRandom(seed);
const rounds = Math.max(0, Math.min(BOOTSTRAP, Number(bootstrap) || 0));
const rounds = Math.max(0, Math.min(BOOTSTRAP, Number(bootstrap) || 0));
const n = prepared.count;
const n = prepared.count;
@@ -391,16 +393,33 @@ function fitRatings(games, {
@@ -391,16 +393,33 @@ function fitRatings(games, {
for (let j = 0; j < n; j++) order[j] = Math.floor(random() * n);
for (let j = 0; j < n; j++) order[j] = Math.floor(random() * n);
const r = fitPrepared(prepared, { order, use: [], ridge, maxIter: 60, warm: { theta: raw.theta, beta: [] } });
const r = fitPrepared(prepared, { order, use: [], ridge, maxIter: 60, warm: { theta: raw.theta, beta: [] } });
const c = fitPrepared(prepared, { order, use: covariates, ridge, betaRidge, maxIter: 60, warm: { theta: controlled.theta, beta: controlled.beta } });
const c = fitPrepared(prepared, { order, use: covariates, ridge, betaRidge, maxIter: 60, warm: { theta: controlled.theta, beta: controlled.beta } });
+ const rawRanks = ranksOf(r.theta);
+ const ctrlRanks = ranksOf(c.theta);
for (const [subject, position] of rawIndex) {
for (const [subject, position] of rawIndex) {
- if (!rawSamples.has(subject)) { rawSamples.set(subject, []); ctrlSamples.set(subject, []); }
+ if (!rawSamples.has(subject)) {
+ rawSamples.set(subject, []); ctrlSamples.set(subject, []);
+ rawRankSamples.set(subject, []); ctrlRankSamples.set(subject, []);
+ }
rawSamples.get(subject).push(r.theta[position]);
rawSamples.get(subject).push(r.theta[position]);
- ctrlSamples.get(subject).push(c.theta[ctrlIndex.has(subject) ? ctrlIndex.get(subject) : 0]);
+ rawRankSamples.get(subject).push(rawRanks[position]);
+ const ctrlPosition = ctrlIndex.has(subject) ? ctrlIndex.get(subject) : 0;
+ ctrlSamples.get(subject).push(c.theta[ctrlPosition]);
+ ctrlRankSamples.get(subject).push(ctrlRanks[ctrlPosition]);
}
}
}
}
const toRating = (theta) => ANCHOR + SCALE * theta;
const toRating = (theta) => ANCHOR + SCALE * theta;
const rawTheta = new Map(raw.subjects.map((s, i) => [s, raw.theta[i]]));
const rawTheta = new Map(raw.subjects.map((s, i) => [s, raw.theta[i]]));
const ctrlTheta = new Map(controlled.subjects.map((s, i) => [s, controlled.theta[i]]));
const ctrlTheta = new Map(controlled.subjects.map((s, i) => [s, controlled.theta[i]]));
+ const rawPointRankList = ranksOf(raw.theta);
+ const ctrlPointRankList = ranksOf(controlled.theta);
+ const rawPointRanks = new Map(raw.subjects.map((s, i) => [s, rawPointRankList[i]]));
+ const ctrlPointRanks = new Map(controlled.subjects.map((s, i) => [s, ctrlPointRankList[i]]));
+ const rankCell = (samples, point) => ({
+ rank: point,
+ rank_lo: samples.length ? Math.round(percentile(samples, 0.025)) : point,
+ rank_hi: samples.length ? Math.round(percentile(samples, 0.975)) : point,
+ });
const rows = raw.subjects.map((subject) => {
const rows = raw.subjects.map((subject) => {
const thetaRaw = rawTheta.get(subject);
const thetaRaw = rawTheta.get(subject);
const thetaCtrl = ctrlTheta.get(subject);
const thetaCtrl = ctrlTheta.get(subject);
@@ -413,11 +432,13 @@ function fitRatings(games, {
@@ -413,11 +432,13 @@ function fitRatings(games, {
rating: Math.round(toRating(thetaRaw) * 100) / 100,
rating: Math.round(toRating(thetaRaw) * 100) / 100,
ci_lower: rawP.length ? Math.round(toRating(percentile(rawP, 0.025)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,
ci_lower: rawP.length ? Math.round(toRating(percentile(rawP, 0.025)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,
ci_upper: rawP.length ? Math.round(toRating(percentile(rawP, 0.975)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,
ci_upper: rawP.length ? Math.round(toRating(percentile(rawP, 0.975)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,
+ ...rankCell(rawRankSamples.get(subject) || [], rawPointRanks.get(subject)),
},
},
controlled: {
controlled: {
rating: Math.round(toRating(thetaCtrl) * 100) / 100,
rating: Math.round(toRating(thetaCtrl) * 100) / 100,
ci_lower: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.025)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,
ci_lower: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.025)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,
ci_upper: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.975)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,
ci_upper: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.975)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,
+ ...rankCell(ctrlRankSamples.get(subject) || [], ctrlPointRanks.get(subject)),
},
},
};
};
}).sort((x, y) => y.raw.rating - x.raw.rating || x.model.localeCompare(y.model));
}).sort((x, y) => y.raw.rating - x.raw.rating || x.model.localeCompare(y.model));
@@ -13,6 +13,7 @@ const clipLib = require('./bench/clip');
@@ -13,6 +13,7 @@ const clipLib = require('./bench/clip');
const clipSource = require('./bench/clip-source');
const clipSource = require('./bench/clip-source');
const ratingLib = require('./rating');
const ratingLib = require('./rating');
const styleLib = require('./rating/style');
const styleLib = require('./rating/style');
+const statsLib = require('./rating/stats');
const { checkSvg } = require('./bench/svg-check');
const { checkSvg } = require('./bench/svg-check');
const BASE_ELO = ratingLib.BASE_ELO;
const BASE_ELO = ratingLib.BASE_ELO;
@@ -203,6 +204,14 @@ function ratingCi(elo, wins, losses, ties) {
@@ -203,6 +204,14 @@ function ratingCi(elo, wins, losses, ties) {
function createStore(s, { log = console, random = Math.random, config = null } = {}) {
function createStore(s, { log = console, random = Math.random, config = null } = {}) {
const db = s.db;
const db = s.db;
+ // The kind boards' fits are pure functions of the games, so a board asks for the same fit only once per
+ // vote set. The key changes with the number of games, and a short TTL covers a vote that lands mid-request.
+ const boardFits = new Map();
+ const BOARD_BOOTSTRAP = 200;
+ const BOARD_FIT_TTL_MS = 5 * 60_000;
+ // A per-category board is published only when its category has this many games; below it there is not
+ // enough play to separate models, so the board is left out of the category list rather than shown thin.
+ const CATEGORY_MIN_GAMES = 30;
// ── Outbox (in the same ambient transaction as the change that caused it) ──
// ── Outbox (in the same ambient transaction as the change that caused it) ──
async function insertOutbox(topic, payload) {
async function insertOutbox(topic, payload) {
@@ -460,24 +469,43 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -460,24 +469,43 @@ function createStore(s, { log = console, random = Math.random, config = null } =
}
}
// ── Judgements ──────────────────────────────────────────
// ── Judgements ──────────────────────────────────────────
+ /**
+ * The pairwise games of a board, as the rating maths reads them: two model names and an outcome of
+ * 'a', 'b' or 'tie'. The pair on a judgement names two entries, so the models come from the entries, never
+ * from the entry ids.
+ *
+ * The category picks the games the board counts: 'all' every vote, 'people' the signed-in voters, 'anon' the
+ * anonymous ones, and any other slug the task's own category (code, writing, chat, reasoning…).
+ */
async function pairwiseGames(kindId, category = CATEGORY_DEFAULT) {
async function pairwiseGames(kindId, category = CATEGORY_DEFAULT) {
+ const cat = String(category == null ? CATEGORY_DEFAULT : category);
const rows = await db.many(
const rows = await db.many(
- `SELECT * FROM bench_judgements
- WHERE kind_id = $1 AND method IN ('human_pairwise', 'game_outcome', 'audience_poll') AND pair IS NOT NULL
- AND ($2::text = 'all' OR outcome IS NOT NULL)
- ORDER BY at, id`,
- [kindId, category],
+ `SELECT j.id, j.method, j.outcome, j.vote_counts, l.model AS left_model, e.model AS right_model
+ FROM bench_judgements j
+ JOIN bench_rounds rd ON rd.id = j.round_id
+ JOIN bench_tasks t ON t.id = rd.task_id
+ JOIN bench_entries l ON l.id = j.pair->>'left' AND l.round_id = j.round_id
+ JOIN bench_entries e ON e.id = j.pair->>'right' AND e.round_id = j.round_id
+ WHERE j.kind_id = $1
+ AND j.method IN ('human_pairwise', 'game_outcome', 'audience_poll')
+ AND j.pair IS NOT NULL AND j.outcome IS NOT NULL
+ AND ($2::text <> 'people' OR COALESCE(j.voter_kind, 'person') = 'person')
+ AND ($2::text <> 'anon' OR j.voter_kind = 'anon')
+ AND ($2::text IN ('all', 'people', 'anon') OR t.task_class = $2)
+ ORDER BY j.at, j.id`,
+ [kindId, cat],
);
);
const out = [];
const out = [];
for (const j of rows) {
for (const j of rows) {
- const pair = typeof j.pair === 'string' ? JSON.parse(j.pair) : j.pair;
- if (!pair || !pair.left || !pair.right) continue;
if (j.outcome === 'both_bad') continue;
if (j.outcome === 'both_bad') continue;
if (j.method === 'audience_poll') {
if (j.method === 'audience_poll') {
const counts = typeof j.vote_counts === 'string' ? JSON.parse(j.vote_counts) : j.vote_counts;
const counts = typeof j.vote_counts === 'string' ? JSON.parse(j.vote_counts) : j.vote_counts;
if (!counts || !num(counts.voters)) continue;
if (!counts || !num(counts.voters)) continue;
}
}
- out.push({ a: pair.left, b: pair.right, outcome: j.outcome === 'left' ? 1 : j.outcome === 'right' ? 0 : 0.5 });
+ if (!j.left_model || !j.right_model || j.left_model === j.right_model) continue;
+ const outcome = j.outcome === 'left' ? 'a' : j.outcome === 'right' ? 'b' : (j.outcome === 'tie' || j.outcome === 'draw') ? 'tie' : null;
+ if (!outcome) continue;
+ out.push({ id: j.id, a: j.left_model, b: j.right_model, outcome });
}
}
return out;
return out;
}
}
@@ -640,23 +668,47 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -640,23 +668,47 @@ function createStore(s, { log = console, random = Math.random, config = null } =
return rows.map((r) => r.model);
return rows.map((r) => r.model);
}
}
+ /**
+ * The Bradley-Terry fit of one board, cached on the vote set. The fit is the same maths the ratings page uses
+ * (server/rating/lib.js): every game is one vote, the ratings are centred on 1000, and the bootstrap supplies
+ * both the rating interval and the 95% rank interval. The seed is fixed, so the same games give the same board.
+ */
+ async function pairwiseFit(kindId, category) {
+ const games = await pairwiseGames(kindId, category);
+ const key = `${kindId}:${category}:${games.length}:${games.length ? games[games.length - 1].id : '-'}`;
+ const hit = boardFits.get(key);
+ if (hit && Date.now() - hit.at < BOARD_FIT_TTL_MS) return hit.fit;
+ const fit = ratingLib.bradleyTerry(games, { bootstrap: BOARD_BOOTSTRAP, seed: 1 });
+ boardFits.set(key, { at: Date.now(), fit });
+ if (boardFits.size > 200) {
+ const oldest = [...boardFits.entries()].sort((a, b) => a[1].at - b[1].at).slice(0, 100);
+ for (const [stale] of oldest) boardFits.delete(stale);
+ }
+ return fit;
+ }
+
async function boardElo(kindId, category = CATEGORY_DEFAULT) {
async function boardElo(kindId, category = CATEGORY_DEFAULT) {
const subjects = await subjectsFor(kindId);
const subjects = await subjectsFor(kindId);
- const ratings = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2', [kindId, category]);
- const by = new Map(ratings.map((r) => [r.subject, wireRating(r)]));
+ const fit = await pairwiseFit(kindId, category);
+ const by = new Map(fit.map((r) => [r.subject, r]));
// A task category only lists the models that played in it: a model with no game in this kind of
// A task category only lists the models that played in it: a model with no game in this kind of
// work has no place on its board, rather than a baseline score it never earned.
// work has no place on its board, rather than a baseline score it never earned.
const taskCategory = ![CATEGORY_DEFAULT, CATEGORY_PEOPLE, CATEGORY_ANON].includes(category);
const taskCategory = ![CATEGORY_DEFAULT, CATEGORY_PEOPLE, CATEGORY_ANON].includes(category);
- const rows = (taskCategory ? subjects.filter((subject) => by.has(subject)) : subjects).map((subject) => {
+ const listed = new Set(taskCategory ? subjects.filter((subject) => by.has(subject)) : subjects);
+ for (const r of fit) listed.add(r.subject);
+ const rows = [...listed].map((subject) => {
const r = by.get(subject);
const r = by.get(subject);
const games = r ? r.games : 0;
const games = r ? r.games : 0;
return {
return {
kind_id: kindId, category, subject,
kind_id: kindId, category, subject,
- elo: r ? r.elo : BASE_ELO,
+ elo: r ? Math.round(r.rating * 100) / 100 : BASE_ELO,
games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,
games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,
- win_rate: r ? r.win_rate : 0,
- ci_lower: r ? r.ci_lower : null,
- ci_upper: r ? r.ci_upper : null,
+ win_rate: games ? Math.round(((r.wins + r.ties * 0.5) / games) * 10000) / 10000 : 0,
+ ci_lower: r ? Math.round(r.ci_lower * 100) / 100 : null,
+ ci_upper: r ? Math.round(r.ci_upper * 100) / 100 : null,
+ rank: r ? r.rank : null,
+ rank_lo: r ? r.rank_lo : null,
+ rank_hi: r ? r.rank_hi : null,
provisional: games < 10,
provisional: games < 10,
};
};
});
});
@@ -664,6 +716,136 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -664,6 +716,136 @@ function createStore(s, { log = console, random = Math.random, config = null } =
return rows;
return rows;
}
}
+ /**
+ * The per-category boards: one for every task category of the kind with at least CATEGORY_MIN_GAMES games, in
+ * the same shape as the kind's own board. A category below the floor is not listed at all, so a page can say
+ * "not enough games yet" without drawing a board that cannot separate anything.
+ */
+ async function categoryBoards(kindId) {
+ const rows = await db.many(
+ `SELECT t.task_class AS category, COUNT(*)::int AS games
+ FROM bench_judgements j
+ JOIN bench_rounds rd ON rd.id = j.round_id
+ JOIN bench_tasks t ON t.id = rd.task_id
+ JOIN bench_entries l ON l.id = j.pair->>'left' AND l.round_id = j.round_id
+ JOIN bench_entries e ON e.id = j.pair->>'right' AND e.round_id = j.round_id
+ WHERE j.kind_id = $1
+ AND j.method IN ('human_pairwise', 'game_outcome', 'audience_poll')
+ AND j.pair IS NOT NULL AND j.outcome IS NOT NULL AND j.outcome <> 'both_bad'
+ AND t.task_class IS NOT NULL
+ AND t.task_class NOT IN ('all', 'people', 'anon')
+ GROUP BY t.task_class
+ HAVING COUNT(*) >= $2
+ ORDER BY COUNT(*) DESC, t.task_class`,
+ [kindId, CATEGORY_MIN_GAMES],
+ );
+ const out = [];
+ for (const row of rows) {
+ // Use the same eligible votes as the fit: self-pairs and empty polls do not earn a board.
+ const games = (await pairwiseGames(kindId, row.category)).length;
+ if (games < CATEGORY_MIN_GAMES) continue;
+ out.push({ category: row.category, games, rows: await boardElo(kindId, row.category) });
+ }
+ return out.sort((a, b) => b.games - a.games || a.category.localeCompare(b.category));
+ }
+
+ /**
+ * The OpenVibe Index for a kind: the Bradley-Terry rating plus the model's median verified score on each public
+ * benchmark of that kind, weighted 50/50 and split equally between the benchmarks. The weights are in the
+ * answer; a model with fewer than two components gets null, never a number from one component.
+ */
+ async function kindIndex(kindId, eloRows) {
+ const rows = await db.many(
+ `SELECT b.id, b.title, r.model, r.score
+ FROM bench_runs r
+ JOIN bench_benchmarks b ON b.id = r.benchmark_id
+ WHERE b.kind = $1 AND b.visibility = 'public' AND r.provenance = 'verified'
+ AND r.score IS NOT NULL
+ ORDER BY b.created_at, b.id, r.model, r.created_at, r.id`,
+ [kindId],
+ );
+ const benchmarks = new Map();
+ const scores = new Map();
+ for (const row of rows) {
+ if (!benchmarks.has(row.id)) benchmarks.set(row.id, { id: row.id, label: row.title });
+ const key = `${row.id}\u0000${row.model}`;
+ if (!scores.has(key)) scores.set(key, []);
+ scores.get(key).push(num(row.score));
+ }
+ const benchmarkList = [...benchmarks.values()];
+ const modelNames = new Set(eloRows.filter((row) => row.rank != null).map((row) => row.subject));
+ for (const key of scores.keys()) modelNames.add(key.split('\u0000')[1]);
+ const indexRows = [...modelNames].map((subject) => {
+ const ratingRow = eloRows.find((row) => row.subject === subject);
+ const present = benchmarkList
+ .map((benchmark) => ({
+ id: benchmark.id,
+ label: benchmark.label,
+ runs: (scores.get(`${benchmark.id}\u0000${subject}`) || []).length,
+ score: statsLib.median(scores.get(`${benchmark.id}\u0000${subject}`) || []),
+ }))
+ .filter((benchmark) => benchmark.score != null);
+ const value = statsLib.openvibeIndex({
+ rating: ratingRow && ratingRow.rank != null ? ratingRow.elo : null,
+ benchmarks: present,
+ benchmarkCount: benchmarkList.length,
+ });
+ return { subject, ...value };
+ });
+ indexRows.sort((a, b) => {
+ if (a.index == null && b.index == null) return a.subject.localeCompare(b.subject);
+ if (a.index == null) return 1;
+ if (b.index == null) return -1;
+ return b.index - a.index || a.subject.localeCompare(b.subject);
+ });
+ return {
+ kind_id: kindId,
+ weights: { ...statsLib.INDEX_WEIGHTS },
+ rating_anchor: statsLib.RATING_ANCHOR,
+ benchmark_count: benchmarkList.length,
+ benchmarks: benchmarkList.map((benchmark) => ({
+ id: benchmark.id,
+ label: benchmark.label,
+ weight: benchmarkList.length ? Math.round((statsLib.INDEX_WEIGHTS.benchmarks / benchmarkList.length) * 1e6) / 1e6 : 0,
+ })),
+ rows: indexRows,
+ };
+ }
+
+ /**
+ * The holdout gap for a kind: per model, the mean score on public tasks minus the mean score on held-out tasks,
+ * with the number of scored tasks on each side. Verified runs of public benchmarks only. A kind without
+ * held-out tasks says `available: false`; a model missing one side gets a null gap.
+ */
+ async function kindHoldout(kindId) {
+ const rows = await db.many(
+ `SELECT b.id AS benchmark_id, r.model, t.task_id, t.score, t.held_out
+ FROM bench_run_tasks t
+ JOIN bench_runs r ON r.id = t.run_id
+ JOIN bench_benchmarks b ON b.id = r.benchmark_id
+ WHERE b.kind = $1 AND b.visibility = 'public' AND r.provenance = 'verified'
+ AND t.score IS NOT NULL
+ ORDER BY r.model, t.created_at, t.id`,
+ [kindId],
+ );
+ const by = new Map();
+ for (const row of rows) {
+ if (!by.has(row.model)) by.set(row.model, { public: [], holdout: [] });
+ const bucket = by.get(row.model);
+ if (row.held_out) bucket.holdout.push(num(row.score));
+ else bucket.public.push(num(row.score));
+ }
+ const out = [...by.entries()].map(([subject, bucket]) => ({ subject, ...statsLib.holdoutGap(bucket.public, bucket.holdout) }));
+ out.sort((a, b) => (b.gap == null ? -Infinity : b.gap) - (a.gap == null ? -Infinity : a.gap) || a.subject.localeCompare(b.subject));
+ const available = rows.some((row) => row.held_out);
+ return {
+ kind_id: kindId,
+ available,
+ held_out_tasks: new Set(rows.filter((row) => row.held_out).map((row) => `${row.benchmark_id}:${row.task_id}`)).size,
+ rows: out,
+ };
+ }
+
async function costStats(kindId) {
async function costStats(kindId) {
const rows = await db.many(
const rows = await db.many(
`SELECT e.model,
`SELECT e.model,
@@ -763,7 +945,19 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -763,7 +945,19 @@ function createStore(s, { log = console, random = Math.random, config = null } =
} else {
} else {
return null;
return null;
}
}
- return { kind_id: kindId, category: cat, period, type, rows, voters: await voterCounts(kindId) };
+ // The kind's own statistics travel with every board: the rating board also carries its per-category
+ // boards, and every type carries the OpenVibe Index and the holdout gap. The Index always reads the
+ // whole kind's rating, even when the rows are filtered to one kind of voter.
+ const allEloRows = cat === CATEGORY_DEFAULT ? eloRows : await boardElo(kindId);
+ const [voterKinds, categories, index, holdout] = await Promise.all([
+ voterCounts(kindId),
+ type === 'elo' ? categoryBoards(kindId) : Promise.resolve(undefined),
+ kindIndex(kindId, allEloRows),
+ kindHoldout(kindId),
+ ]);
+ const out = { kind_id: kindId, category: cat, period, type, rows, voters: voterKinds, index, holdout };
+ if (categories !== undefined) out.categories = categories;
+ return out;
}
}
/**
/**
@@ -830,8 +1024,25 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -830,8 +1024,25 @@ function createStore(s, { log = console, random = Math.random, config = null } =
entries = (await db.many('SELECT * FROM bench_entries WHERE round_id = ANY($1::text[]) ORDER BY created_at, id', [roundIds])).map(wireEntry);
entries = (await db.many('SELECT * FROM bench_entries WHERE round_id = ANY($1::text[]) ORDER BY created_at, id', [roundIds])).map(wireEntry);
judgements = (await db.many('SELECT * FROM bench_judgements WHERE round_id = ANY($1::text[]) ORDER BY at, id', [roundIds])).map((j) => wireJudgement(j));
judgements = (await db.many('SELECT * FROM bench_judgements WHERE round_id = ANY($1::text[]) ORDER BY at, id', [roundIds])).map((j) => wireJudgement(j));
}
}
- const ratings = (await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 ORDER BY subject', [kindId])).map(wireRating);
- return { kind_id: kindId, tasks, rounds, entries, judgements, ratings };
+ const allEloRows = await boardElo(kindId);
+ const savedRatings = (await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 ORDER BY subject', [kindId])).map(wireRating);
+ const fitted = new Map([[CATEGORY_DEFAULT, allEloRows]]);
+ for (const category of new Set(savedRatings.map((rating) => rating.category))) {
+ if (!fitted.has(category)) fitted.set(category, await boardElo(kindId, category));
+ }
+ // Export the same fit and intervals as the board, not the old incremental Elo beside a fitted rank.
+ const ratings = [...fitted.values()].flat().filter((rating) => rating.games > 0).map((rating) => ({
+ ...savedRatings.find((saved) => saved.subject === rating.subject && saved.category === rating.category),
+ ...rating,
+ }));
+ const [categories, index, holdout] = await Promise.all([
+ categoryBoards(kindId),
+ kindIndex(kindId, allEloRows),
+ kindHoldout(kindId),
+ ]);
+ // The open dataset carries the same four statistics the board API answers with: every rating's rank
+ // interval, the per-category boards, the OpenVibe Index with its weights, and the holdout gap.
+ return { kind_id: kindId, tasks, rounds, entries, judgements, ratings, categories, index, holdout };
}
}
async function datasetJsonl(kindId) {
async function datasetJsonl(kindId) {
@@ -843,6 +1054,9 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -843,6 +1054,9 @@ function createStore(s, { log = console, random = Math.random, config = null } =
for (const e of d.entries) lines.push(JSON.stringify({ type: 'entry', ...e }));
for (const e of d.entries) lines.push(JSON.stringify({ type: 'entry', ...e }));
for (const j of d.judgements) lines.push(JSON.stringify({ type: 'judgement', ...j }));
for (const j of d.judgements) lines.push(JSON.stringify({ type: 'judgement', ...j }));
for (const r of d.ratings) lines.push(JSON.stringify({ type: 'rating', ...r }));
for (const r of d.ratings) lines.push(JSON.stringify({ type: 'rating', ...r }));
+ for (const c of d.categories) lines.push(JSON.stringify({ type: 'category', ...c }));
+ lines.push(JSON.stringify({ type: 'index', ...d.index }));
+ lines.push(JSON.stringify({ type: 'holdout', ...d.holdout }));
return lines.length ? `${lines.join('\n')}\n` : '';
return lines.length ? `${lines.join('\n')}\n` : '';
}
}
@@ -42,6 +42,128 @@ const { boot, check, done } = require('./helpers/boot');
@@ -42,6 +42,128 @@ const { boot, check, done } = require('./helpers/boot');
assert.ok(after.some((row) => row.games === 1 && row.provisional === true), 'one game is still provisional');
assert.ok(after.some((row) => row.games === 1 && row.provisional === true), 'one game is still provisional');
});
});
+ await check('every elo board carries 95% rank intervals from the bootstrap', async () => {
+ const board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
+ assert.ok(board.rows.length >= 2);
+ for (const row of board.rows) {
+ assert.ok(Number.isInteger(row.rank) && row.rank >= 1 && row.rank <= board.rows.length, `${row.subject} rank`);
+ assert.ok(Number.isInteger(row.rank_lo) && row.rank_lo >= 1, `${row.subject} rank_lo`);
+ assert.ok(Number.isInteger(row.rank_hi) && row.rank_hi <= board.rows.length, `${row.subject} rank_hi`);
+ assert.ok(row.rank_lo <= row.rank_hi, `${row.subject} interval is ordered`);
+ assert.ok(row.ci_lower != null && row.ci_upper != null, `${row.subject} rating interval`);
+ }
+ const exported = (await t.get('/api/v1/datasets/design.page.jsonl')).text.trim().split('\n').map(JSON.parse);
+ for (const row of board.rows) {
+ const saved = exported.find((entry) => entry.type === 'rating' && entry.category === 'all' && entry.subject === row.subject);
+ for (const field of ['elo', 'games', 'ci_lower', 'ci_upper', 'rank', 'rank_lo', 'rank_hi']) {
+ assert.strictEqual(saved[field], row[field], `dataset ${row.subject} ${field} matches the board`);
+ }
+ }
+ assert.ok(board.index.rows.every((row) => row.index === null), 'a rating alone is never an Index');
+ assert.strictEqual(board.holdout.available, false, 'this kind has no held-out tasks');
+ assert.deepStrictEqual(board.holdout.rows, []);
+ });
+
+ // A category board: a task class with at least 30 games earns one; 29 does not.
+ async function roundForCategory(category) {
+ const task = await t.get('/api/v1/kinds/design.page/tasks', { as: alice, json: { prompt: 'A ' + category + ' page.', public: true, task_class: category } });
+ const rr = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });
+ const id = rr.json().round.id;
+ const a = await t.get(`/api/v1/rounds/${id}/entries`, { bearer: entrySvc, json: { provider: 'acme', model: 'alpha-1', payload: { html: '<h1>a</h1>' } } });
+ const b = await t.get(`/api/v1/rounds/${id}/entries`, { bearer: entrySvc, json: { provider: 'beta', model: 'beta-2', payload: { html: '<h1>b</h1>' } } });
+ return { roundId: id, left: a.json().entry.id, right: b.json().entry.id };
+ }
+
+ async function seedGames(round, count) {
+ for (let i = 0; i < count; i++) {
+ await t.ctx.s.db.exec(
+ `INSERT INTO bench_judgements (id, round_id, kind_id, judge, method, pair, outcome, subject, voter_kind)
+ VALUES ($1, $2, 'design.page', 'human_pairwise', 'human_pairwise', $3::jsonb, $4, $5, 'person')`,
+ [t.store.newId('bju'), round.roundId, JSON.stringify({ left: round.left, right: round.right }), i % 2 ? 'left' : 'right', 'usr_seed_' + i],
+ );
+ }
+ }
+
+ await check('a category board appears at 30 games and not at 29', async () => {
+ const arcade = await roundForCategory('arcade');
+ const writing = await roundForCategory('writing');
+ await seedGames(writing, 29);
+ const selfOnly = await roundForCategory('self-only');
+ await seedGames(selfOnly, 30);
+ await t.ctx.s.db.exec('UPDATE bench_entries SET model = $1 WHERE id = $2', ['alpha-1', selfOnly.right]);
+ let board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
+ assert.strictEqual(board.categories.length, 0, '29 games is below the floor: ' + JSON.stringify(board.categories));
+ await seedGames(arcade, 30);
+ board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
+ const category = board.categories.find((c) => c.category === 'arcade');
+ assert.ok(category, 'the 30-game category has a board');
+ assert.strictEqual(category.games, 30);
+ assert.ok(category.rows.every((row) => row.games > 0), 'only models that played are listed');
+ assert.ok(category.rows.every((row) => Number.isInteger(row.rank_lo) && row.rank_lo <= row.rank_hi), 'the category board has rank intervals');
+ assert.strictEqual(board.categories.find((c) => c.category === 'writing'), undefined, '29 games is still out');
+ // The filtered board answers for one category alone.
+ const one = (await t.get('/api/v1/boards/design.page?category=arcade')).json().board;
+ assert.strictEqual(one.category, 'arcade');
+ assert.strictEqual(one.rows.reduce((n, row) => n + row.games, 0), 60, '30 games are 60 model-games');
+ });
+
+ await check('the OpenVibe Index needs two components and publishes its weights', async () => {
+ const definition = {
+ title: 'Design pages', description: 'The B3 index fixture.', kind: 'design.page', tags: [], visibility: 'public',
+ tasks: [
+ { id: 't1', prompt: 'A public page.', files: [], expected: null, holdout: false },
+ { id: 't2', prompt: 'A held-out page.', files: [], expected: null, holdout: true },
+ ],
+ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],
+ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
+ limits: { ms: 10000, tokens: 10000, usd: 1 },
+ matrix: { models: ['alpha-1'], harnesses: ['fixture'] },
+ };
+ const writer = t.network.serviceToken({ sub: 'svc:fixture', audience: 'openvibe.bench', cap: ['bench.benchmark.write', 'bench.run.publish'] });
+ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition });
+ assert.strictEqual(created.status, 201, created.text);
+ const benchmarkId = created.json().benchmark.id;
+ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: writer, json: {} });
+ assert.strictEqual(frozen.status, 201, frozen.text);
+ const run = await t.get('/api/v1/runs', {
+ bearer: writer,
+ json: {
+ benchmark: `${benchmarkId}@1`, model: 'alpha-1', route: { provider: 'acme' }, provider: 'acme',
+ harness: { name: 'fixture', version: '1' }, settings: {}, executor: { kind: 'codes', id: 'fixture' },
+ tasks: [
+ { task_id: 't1', score: 90, cost_usd: 0.01, ms: 100, tokens: 10, artifacts: [] },
+ { task_id: 't2', score: 60, cost_usd: 0.01, ms: 100, tokens: 10, artifacts: [] },
+ ],
+ score: 75, cost_usd: 0.02, ms: 200, tokens: 20,
+ },
+ });
+ assert.strictEqual(run.status, 201, run.text);
+
+ const board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
+ assert.deepStrictEqual(board.index.weights, { rating: 0.5, benchmarks: 0.5 });
+ assert.strictEqual(board.index.benchmark_count, 1);
+ assert.strictEqual(board.index.benchmarks[0].weight, 0.5);
+ const alpha = board.index.rows.find((row) => row.subject === 'alpha-1');
+ const beta = board.index.rows.find((row) => row.subject === 'beta-2');
+ assert.ok(alpha, 'the rated model is in the index');
+ assert.strictEqual(alpha.n_components, 2, 'rating + one benchmark: ' + JSON.stringify(alpha.components));
+ assert.ok(typeof alpha.index === 'number' && alpha.index >= 0 && alpha.index <= 100, 'the index is 0-100');
+ assert.ok(alpha.components.some((c) => c.kind === 'rating') && alpha.components.some((c) => c.kind === 'benchmark'), 'both kinds of component are named');
+ assert.strictEqual(beta.index, null, 'a single component is never an index');
+ assert.strictEqual(beta.n_components, 1);
+
+ assert.strictEqual(board.holdout.available, true);
+ const gap = board.holdout.rows.find((row) => row.subject === 'alpha-1');
+ assert.deepStrictEqual(gap, { subject: 'alpha-1', public_n: 1, public_score: 90, holdout_n: 1, holdout_score: 60, gap: 30 });
+
+ const dataset = (await t.get('/api/v1/datasets/design.page.jsonl')).text.trim().split('\n').map((line) => JSON.parse(line));
+ assert.ok(dataset.some((line) => line.type === 'index' && line.weights.rating === 0.5), 'the dataset carries the index');
+ assert.ok(dataset.some((line) => line.type === 'holdout' && line.rows.some((row) => row.gap === 30)), 'the dataset carries the holdout gap');
+ assert.ok(dataset.some((line) => line.type === 'category' && line.category === 'arcade'), 'the dataset carries the category board');
+ assert.ok(dataset.some((line) => line.type === 'rating' && line.rank_lo !== null), 'the dataset carries the rank interval');
+ });
+
+
await check('a model page aggregates across kinds', async () => {
await check('a model page aggregates across kinds', async () => {
const task = await t.get('/api/v1/kinds/games.match/tasks', { as: alice, json: { prompt: 'A match.', public: true } });
const task = await t.get('/api/v1/kinds/games.match/tasks', { as: alice, json: { prompt: 'A match.', public: true } });
const rr = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });
const rr = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });
new file mode 100644
new file mode 100644
@@ -0,0 +1,34 @@
@@ -0,0 +1,34 @@
+{
+ "cases": [
+ {
+ "name": "a public set and a holdout: the gap is public minus holdout",
+ "public": [80, 90],
+ "holdout": [60],
+ "expect": { "public_n": 2, "public_score": 85, "holdout_n": 1, "holdout_score": 60, "gap": 25 }
+ },
+ {
+ "name": "no held-out score: the gap is null, never zero",
+ "public": [90, 70],
+ "holdout": [],
+ "expect": { "public_n": 2, "public_score": 80, "holdout_n": 0, "holdout_score": null, "gap": null }
+ },
+ {
+ "name": "no public score: the gap is null too",
+ "public": [],
+ "holdout": [40],
+ "expect": { "public_n": 0, "public_score": null, "holdout_n": 1, "holdout_score": 40, "gap": null }
+ },
+ {
+ "name": "nothing scored: every number is absent, not zero",
+ "public": [],
+ "holdout": [],
+ "expect": { "public_n": 0, "public_score": null, "holdout_n": 0, "holdout_score": null, "gap": null }
+ },
+ {
+ "name": "a negative gap says the held-out set scored higher",
+ "public": [55.5],
+ "holdout": [70.25],
+ "expect": { "public_n": 1, "public_score": 55.5, "holdout_n": 1, "holdout_score": 70.25, "gap": -14.75 }
+ }
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+{
+ "weights": { "rating": 0.5, "benchmarks": 0.5 },
+ "cases": [
+ {
+ "name": "a rating alone is not an Index",
+ "rating": 1200,
+ "benchmarks": [],
+ "benchmarkCount": 0,
+ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5 }
+ },
+ {
+ "name": "one benchmark alone is not an Index either",
+ "rating": null,
+ "benchmarks": [{ "id": "b1", "label": "One", "score": 80, "runs": 2 }],
+ "benchmarkCount": 1,
+ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5 }
+ },
+ {
+ "name": "a rating and one of two benchmarks: the missing benchmark's weight renormalises",
+ "rating": 1000,
+ "benchmarks": [{ "id": "b1", "label": "One", "score": 80, "runs": 3 }],
+ "benchmarkCount": 2,
+ "expect": { "index": 60, "n_components": 2, "weight_sum": 0.75 }
+ },
+ {
+ "name": "two benchmarks and no rating still make an Index",
+ "rating": null,
+ "benchmarks": [
+ { "id": "b1", "label": "One", "score": 40, "runs": 1 },
+ { "id": "b2", "label": "Two", "score": 80, "runs": 1 }
+ ],
+ "benchmarkCount": 2,
+ "expect": { "index": 60, "n_components": 2, "weight_sum": 0.5 }
+ },
+ {
+ "name": "a score over 100 is clamped, not trusted",
+ "rating": 1000,
+ "benchmarks": [{ "id": "b1", "label": "One", "score": 140, "runs": 1 }],
+ "benchmarkCount": 1,
+ "expect": { "index": 75, "n_components": 2, "weight_sum": 1 }
+ },
+ {
+ "name": "the rating is the Elo expectation against 1000, as 0-100 points",
+ "rating": 1000,
+ "benchmarks": [],
+ "benchmarkCount": 0,
+ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5, "rating_value": 50 }
+ }
+ ]
+}
@@ -105,3 +105,35 @@ test('percentile interpolates and handles empty input', () => {
@@ -105,3 +105,35 @@ test('percentile interpolates and handles empty input', () => {
assert.ok(Math.abs(percentile([1, 2, 3, 4], 95) - 3.85) < 1e-12);
assert.ok(Math.abs(percentile([1, 2, 3, 4], 95) - 3.85) < 1e-12);
assert.equal(percentile([], 95), null);
assert.equal(percentile([], 95), null);
});
});
+
+test('rank intervals come from the same bootstrap as the rating intervals', () => {
+ const games = [
+ ...Array.from({ length: 24 }, () => game('A', 'B', 'a')),
+ ...Array.from({ length: 6 }, () => game('A', 'B', 'b')),
+ ...Array.from({ length: 20 }, () => game('B', 'C', 'a')),
+ ...Array.from({ length: 10 }, () => game('B', 'C', 'b')),
+ ...Array.from({ length: 25 }, () => game('A', 'C', 'a')),
+ ...Array.from({ length: 5 }, () => game('A', 'C', 'b')),
+ ];
+ const fit = bradleyTerry(games, { bootstrap: 200, seed: 11 });
+ const by = bySubject(fit);
+ assert.equal(by.get('A').rank, 1, 'the planted winner is first');
+ assert.equal(by.get('A').rank_lo, 1);
+ assert.ok(by.get('A').rank_hi >= 1);
+ assert.equal(by.get('C').rank, 3);
+ for (const row of fit) {
+ assert.ok(Number.isInteger(row.rank) && row.rank >= 1 && row.rank <= fit.length, `${row.subject} rank`);
+ assert.ok(Number.isInteger(row.rank_lo) && row.rank_lo >= 1, `${row.subject} rank_lo`);
+ assert.ok(Number.isInteger(row.rank_hi) && row.rank_hi <= fit.length, `${row.subject} rank_hi`);
+ assert.ok(row.rank_lo <= row.rank_hi, `${row.subject} interval is ordered`);
+ }
+ assert.deepEqual(bradleyTerry(games, { bootstrap: 200, seed: 11 }), fit, 'the rank intervals are seeded');
+});
+
+test('without a bootstrap the rank interval is the point rank, never a fraction', () => {
+ const fit = bradleyTerry([game('A', 'B', 'a'), game('A', 'B', 'a'), game('B', 'C', 'a')], { bootstrap: 0 });
+ for (const row of fit) {
+ assert.equal(row.rank_lo, row.rank);
+ assert.equal(row.rank_hi, row.rank);
+ }
+});
new file mode 100644
new file mode 100644
@@ -0,0 +1,71 @@
@@ -0,0 +1,71 @@
+'use strict';
+
+/**
+ * The OpenVibe Index and the holdout gap (stage B3): the published weights, the null for one component, the
+ * renormalisation over the evidence a model actually has, and the holdout gap's n on both sides.
+ */
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const { openvibeIndex, holdoutGap, ratingScore, INDEX_WEIGHTS, RATING_ANCHOR } = require('../server/rating/stats');
+
+const fixture = (name) => JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures/rating', name), 'utf8'));
+
+test('the published weights are the ones the docs and the API name', () => {
+ assert.deepEqual(INDEX_WEIGHTS, { rating: 0.5, benchmarks: 0.5 });
+ assert.equal(RATING_ANCHOR, 1000);
+ assert.equal(ratingScore(1000), 50);
+ assert.ok(Math.abs(ratingScore(1200) - 75.97469266479578) < 1e-9);
+ assert.equal(ratingScore(null), null);
+});
+
+test('the Index fixture: two components or null, with the weights in the answer', () => {
+ for (const testCase of fixture('index-cases.json').cases) {
+ const got = openvibeIndex({
+ rating: testCase.rating,
+ benchmarks: testCase.benchmarks,
+ benchmarkCount: testCase.benchmarkCount,
+ });
+ assert.equal(got.index, testCase.expect.index, testCase.name);
+ assert.equal(got.n_components, testCase.expect.n_components, testCase.name);
+ assert.equal(got.weight_sum, testCase.expect.weight_sum, testCase.name);
+ if (testCase.expect.rating_value != null) {
+ assert.equal(got.components[0].value, testCase.expect.rating_value, testCase.name);
+ }
+ for (const component of got.components) {
+ assert.ok(component.weight > 0, testCase.name + ': every component has its weight');
+ }
+ }
+});
+
+test('the Index never invents a score from one component, and null is not zero', () => {
+ assert.equal(openvibeIndex({ rating: 1300 }).index, null);
+ assert.equal(openvibeIndex({ benchmarks: [{ id: 'b', score: 99 }] }).index, null);
+ assert.equal(openvibeIndex({}).index, null);
+ assert.equal(openvibeIndex({ rating: 1100, benchmarks: [{ id: 'b', score: 90 }] }).index != null, true);
+});
+
+test('a model missing one benchmark of the kind gets the weight renormalised, not a second benchmark', () => {
+ const got = openvibeIndex({
+ rating: 1000,
+ benchmarks: [{ id: 'b1', score: 80 }],
+ benchmarkCount: 3,
+ });
+ assert.equal(got.n_components, 2);
+ assert.equal(got.weight_sum, 0.666667);
+ assert.equal(got.components.find((c) => c.id === 'b1').weight, 0.166667);
+ assert.ok(got.index > 50 && got.index < 80);
+});
+
+test('the holdout gap fixture: public minus holdout, with n on each side', () => {
+ for (const testCase of fixture('holdout-cases.json').cases) {
+ assert.deepEqual(holdoutGap(testCase.public, testCase.holdout), testCase.expect, testCase.name);
+ }
+});
+
+test('the holdout gap ignores missing scores and keeps the side count honest', () => {
+ assert.deepEqual(holdoutGap([80, null, undefined], [null]), {
+ public_n: 1, public_score: 80, holdout_n: 0, holdout_score: null, gap: null,
+ });
+});
@@ -86,6 +86,25 @@ function rng(seed) {
@@ -86,6 +86,25 @@ function rng(seed) {
assert.strictEqual(style.bandLabel(1, 1), '1st');
assert.strictEqual(style.bandLabel(1, 1), '1st');
});
});
+ await check('the fit publishes rank intervals for both boards', async () => {
+ const games = [];
+ for (let i = 0; i < 60; i++) {
+ games.push({ id: 'ra' + i, a: 'alpha', b: 'beta', outcome: 'a', fa: { length: 100, markup: 1 }, fb: { length: 100, markup: 1 } });
+ games.push({ id: 'rb' + i, a: 'beta', b: 'gamma', outcome: 'a', fa: { length: 100, markup: 1 }, fb: { length: 100, markup: 1 } });
+ }
+ const fit = style.fitRatings(games, { bootstrap: 80 });
+ const by = new Map(fit.models.map((m) => [m.model, m]));
+ assert.strictEqual(by.get('alpha').raw.rank, 1);
+ for (const model of fit.models) {
+ for (const cell of [model.raw, model.controlled]) {
+ assert.ok(Number.isInteger(cell.rank) && cell.rank >= 1 && cell.rank <= fit.models.length, model.model + ' rank');
+ assert.ok(Number.isInteger(cell.rank_lo) && cell.rank_lo >= 1, model.model + ' rank_lo');
+ assert.ok(Number.isInteger(cell.rank_hi) && cell.rank_hi <= fit.models.length, model.model + ' rank_hi');
+ assert.ok(cell.rank_lo <= cell.rank_hi, model.model + ' ordered interval');
+ }
+ }
+ });
+
await check('the style features count length and markup', async () => {
await check('the style features count length and markup', async () => {
const f = style.styleFeatures({ html: '<h1>Title</h1><p><strong>bold</strong></p><ul><li>one</li></ul>' });
const f = style.styleFeatures({ html: '<h1>Title</h1><p><strong>bold</strong></p><ul><li>one</li></ul>' });
assert.ok(f.length > 20, 'length: ' + f.length);
assert.ok(f.length > 20, 'length: ' + f.length);