openvibers/openvibe.bench Commits

Bench B3: rank intervals, category boards, the OpenVibe Index and the holdout gap (#176)

OP OpenVibers committed 3 days ago · bedd4d889e · 13 files, +806 −22

* Bench B3: rank intervals, category boards, the OpenVibe Index and the holdout gap

* Merge main and align board datasets with eligible fitted games

Changes

ddiff --git a/docs/api.md b/docs/api.md
iindex 2717563..3d914ee 100644
---- a/docs/api.md
++++ b/docs/api.md
@@@ -229,6 +229,22 @@ observations is returned only when every observation counts that outcome; a miss
ignores any such fields, and the read routes never join a task, benchmark or entry. The only text in a row is an
exact id, a family, and the id of the round or run that produced it.
++## The boards: ranks, categories, the Index and the holdout
++
++`GET /api/v1/boards/:kindId?type=elo` returns the fit, not just the Elo updates: each row carries `elo`, its
++`ci_lower`/`ci_upper` rating interval, and `rank`/`rank_lo`/`rank_hi` — the 95% rank interval from the same bootstrap.
++The board also carries:
++
++| Field | What it holds |
++|---|---|
++| `categories` | One board per task category with at least 30 games: `{ category, games, rows }`, each row fitted the same way. A category below the floor is left out. `?category=<slug>` answers for one category alone. |
++| `index` | The OpenVibe Index per model: `{ weights, rating_anchor, benchmark_count, benchmarks[], rows[] }`. Each row is `{ subject, index, n_components, weight_sum, components[] }`; `index` is null when fewer than two components exist. |
++| `holdout` | `{ available, held_out_tasks, rows[] }`; each row is `{ subject, public_n, public_score, holdout_n, holdout_score, gap }`, with a null gap when either side has no scores. |
++
++The open dataset (`/api/v1/datasets/:kindId.jsonl`) also emits `type: "category"`, `type: "index"` and
++`type: "holdout"` lines, and its rating lines carry the rank interval. The formulas, the 0.5/0.5 weights and the
++30-game floor are in [index.md](index.md).
++
## The older API
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
ddiff --git a/docs/index.md b/docs/index.md
nnew file mode 100644
iindex 0000000..4bf125f
---- /dev/null
++++ b/docs/index.md
@@@ -0,0 +1,60 @@
++# The numbers on Bench
++
++Every board is a Bradley-Terry fit over the votes, centred on 1000, with a bootstrap that gives each model a
++rating interval and a 95% rank interval. A kind's board answers at `GET /api/v1/boards/:kindId`; the open dataset
++for the kind (`GET /api/v1/datasets/:kindId.jsonl`) carries the same numbers. The shapes are in
++[api.md](api.md).
++
++## Rank intervals
++
++A rank is a band, not a place. Every fit resamples the games 200 times with a fixed seed, ranks the models in each
++resample (equal ratings share a rank), and takes the 2.5th and 97.5th percentile of those places:
++
++```json
++{ "subject": "model-a", "elo": 1042.5, "rank": 2, "rank_lo": 1, "rank_hi": 4 }
++```
++
++`rank` is the place on the fitted board. `rank_lo` and `rank_hi` are the 95% rank interval: the places the votes
++allow. A model whose interval contains 1 may be first; overlapping intervals are why the page shows "1st–3rd" and
++never pretends the order is settled.
++
++## Per-category boards
++
++A task can carry a `task_class` (code, writing, chat, reasoning, arcade, …). Every vote is counted in the task's
++category as well as the kind's whole board. The elo board publishes `categories`: one board per category with at
++least **30 games**, fitted the same way (same Bradley-Terry maths, same bootstrap intervals, same rank intervals).
++A category below 30 games is left out; it is not shown with a thin score. `GET /api/v1/boards/:kindId?category=arcade`
++answers for one category alone.
++
++## The OpenVibe Index
++
++The Index is one number per model per kind, on the same 0-100 scale as every other Bench score. It is a weighted mean
++of two kinds of evidence:
++
++| Component | Weight | Value |
++|---|---|---|
++| The model's human rating | 0.5 | Its Bradley-Terry rating put on 0-100 by the Elo expectation against a 1000-rated opponent: `100 / (1 + 10 ** ((1000 - rating) / 400))`. A 1000 rating is exactly 50. |
++| The kind's public benchmarks | 0.5, split equally between them | The model's median verified score (0-100) on each public benchmark of that kind. |
++
++The weights are published in every answer (`index.weights` and the per-benchmark `index.benchmarks[].weight`) and in
++each row's `components`. A model missing a benchmark gets that benchmark's weight renormalised away rather than
++letting one benchmark stand in for half the Index; `weight_sum` says how much of the 1.0 the model actually has.
++
++**Fewer than two components is not an Index.** A model with only a rating, or only one benchmark, gets `index: null`.
++Null is not zero and is not sorted as a score.
++
++## The holdout gap
++
++A benchmark task may be held out (`holdout: true`): only the executor and the benchmark's owner see its prompt. The
++holdout gap compares the same model's verified scores on the public tasks and on the held-out tasks of the kind:
++
++```
++gap = mean(public task scores) - mean(held-out task scores)
++```
++
++The answer carries `public_n`, `public_score`, `holdout_n`, `holdout_score` and `gap`. A kind with no held-out tasks
++says `available: false`. A model missing either side gets `gap: null`, never zero: a gap cannot be read from one side.
++A large positive gap is a contamination signal, not a proof; it says the held-out set scored lower and is worth a look.
++
++Both statistics travel with every board (`index` and `holdout`), and the open dataset has `type: "index"` and
++`type: "holdout"` lines beside the games, so a reader can check them without the page.
ddiff --git a/server/rating/index.js b/server/rating/index.js
iindex 4e5e939..65caef0 100644
---- a/server/rating/index.js
++++ b/server/rating/index.js
@@@ -79,8 +79,13 @@ const lib = require('./lib');
// each output as covariates, bootstrap intervals and rank bands.
const style = require('./style');
++// The OpenVibe Index and the holdout gap (stage B3): pure maths, so a test and a page agree on the numbers.
++const stats = require('./stats');
++
module.exports = {
eloUpdate, bradleyTerry: lib.bradleyTerry, voterWeights: lib.voterWeights, wilson: lib.wilson, percentile: lib.percentile, pareto, costPerSolved, BASE_ELO, K_DEFAULT,
style,
fitRatings: style.fitRatings, rankBands: style.rankBands, styleFeatures: style.styleFeatures, markupCount: style.markupCount,
++ openvibeIndex: stats.openvibeIndex, holdoutGap: stats.holdoutGap, ratingScore: stats.ratingScore,
++ INDEX_WEIGHTS: stats.INDEX_WEIGHTS, RATING_ANCHOR: stats.RATING_ANCHOR,
};
ddiff --git a/server/rating/lib.js b/server/rating/lib.js
iindex bd0668e..91c0847 100644
---- a/server/rating/lib.js
++++ b/server/rating/lib.js
@@@ -92,6 +92,16 @@ function seededRandom(seed) {
};
}
++/**
++ * Competition ranks by strength: the best subject is 1st and equal ratings share the best rank. The bootstrap
++ * ranks use the same rule on every resampled fit, so a rank interval is the spread of the places the votes allow,
++ * not a second guess from the two ends of the rating interval.
++ */
++function ranksOf(ratings) {
++ const list = Array.isArray(ratings) ? ratings : [];
++ return list.map((rating) => 1 + list.filter((other) => other > rating + 1e-9).length);
++}
++
function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterations = 200, weights } = {}) {
const active = games.filter(game => game.outcome !== 'both_bad');
const subjects = [...new Set(active.flatMap(game => [game.a, game.b]))].sort();
@@@ -110,19 +120,25 @@ function bradleyTerry(games, { bootstrap = 200, seed = 1, anchor = 1000, iterati
}
const ratings = ratingsFor(active, subjects, weights, iterations, anchor);
const samples = subjects.map(() => []);
++ const rankSamples = subjects.map(() => []);
if (bootstrap > 0 && active.length) {
const random = seededRandom(seed);
for (let i = 0; i < bootstrap; i++) {
const resampled = Array.from({ length: active.length }, () => active[Math.floor(random() * active.length)]);
const draw = ratingsFor(resampled, subjects, weights, iterations, anchor);
draw.forEach((rating, j) => samples[j].push(rating));
++ ranksOf(draw).forEach((rank, j) => rankSamples[j].push(rank));
}
}
++ const pointRanks = ranksOf(ratings);
return subjects.map((subject, i) => ({
subject,
rating: ratings[i],
ci_lower: samples[i].length ? percentile(samples[i], 2.5) : ratings[i],
ci_upper: samples[i].length ? percentile(samples[i], 97.5) : ratings[i],
++ rank: pointRanks[i],
++ rank_lo: rankSamples[i].length ? Math.round(percentile(rankSamples[i], 2.5)) : pointRanks[i],
++ rank_hi: rankSamples[i].length ? Math.round(percentile(rankSamples[i], 97.5)) : pointRanks[i],
...stats.get(subject),
provisional: stats.get(subject).games < 10,
})).sort((a, b) => b.rating - a.rating || String(a.subject).localeCompare(String(b.subject)));
@@@ -171,4 +187,4 @@ function costPerSolved(entries) {
return { cost_usd, solved, per_solved: solved ? cost_usd / solved : null };
}
--module.exports = { eloUpdate, bradleyTerry, voterWeights, wilson, pareto, costPerSolved, percentile };
++module.exports = { eloUpdate, bradleyTerry, voterWeights, wilson, pareto, costPerSolved, percentile, ranksOf };
ddiff --git a/server/rating/stats.js b/server/rating/stats.js
nnew file mode 100644
iindex 0000000..a107cf6
---- /dev/null
++++ b/server/rating/stats.js
@@@ -0,0 +1,124 @@
++'use strict';
++
++/**
++ * The OpenVibe Index and the holdout gap (plan 2026-11-09, stage B3).
++ *
++ * The Index is one number per model per kind, on the 0-100 scale every other Bench score uses. It is a weighted
++ * mean of two kinds of evidence:
++ *
++ * rating the model's Bradley-Terry rating from the human votes, put on 0-100 by the Elo expectation
++ * against a 1000-rated opponent: 100 / (1 + 10 ** ((1000 - rating) / 400)). A 1000 rating is 50,
++ * not a made-up score of its own.
++ * benchmarks the median verified score of the model on each public benchmark of that kind (0-100).
++ *
++ * The weights are published with every answer: half the Index is the rating and half is split equally between the
++ * kind's public benchmarks. A model that has no score on one of them gets that benchmark's weight renormalised
++ * away, so the Index is always a weighted mean of the evidence the model actually has. Fewer than two components is
++ * not an Index: the answer is null, never a number from one component.
++ *
++ * The holdout gap is the public score minus the held-out score, points on the same 0-100 scale, with the number of
++ * scored tasks on each side. A kind without held-out tasks, or a model missing one side, gets a null gap.
++ */
++
++/** The published weights: half the rating, half the objective benchmarks (split equally between them). */
++const INDEX_WEIGHTS = Object.freeze({ rating: 0.5, benchmarks: 0.5 });
++/** The rating the logistic is read against, so a 1000 rating is 50 points. */
++const RATING_ANCHOR = 1000;
++
++const num = (v) => (v == null || v === '' || Number.isNaN(Number(v)) ? null : Number(v));
++const round = (v, places = 4) => {
++ if (v == null || !Number.isFinite(Number(v))) return null;
++ const factor = 10 ** places;
++ return Math.round(Number(v) * factor) / factor;
++};
++const clamp = (v, low, high) => Math.min(high, Math.max(low, v));
++
++function mean(values) {
++ const list = (Array.isArray(values) ? values : []).map(num).filter((v) => v != null);
++ return list.length ? list.reduce((sum, v) => sum + v, 0) / list.length : null;
++}
++
++function median(values) {
++ const list = (Array.isArray(values) ? values : []).map(num).filter((v) => v != null).sort((a, b) => a - b);
++ if (!list.length) return null;
++ const middle = Math.floor(list.length / 2);
++ return list.length % 2 ? list[middle] : (list[middle - 1] + list[middle]) / 2;
++}
++
++/** A Bradley-Terry rating on the 0-100 scale: the Elo expectation against a 1000-rated opponent, as points. */
++function ratingScore(rating, anchor = RATING_ANCHOR) {
++ const value = num(rating);
++ if (value == null) return null;
++ return 100 / (1 + 10 ** ((anchor - value) / 400));
++}
++
++/**
++ * The Index for one model: `{ index, n_components, weight_sum, components }`.
++ *
++ * `benchmarks` are the kind's public benchmarks the model has a verified median score on, as
++ * `{ id, label?, score, runs? }`. `benchmarkCount` is how many public benchmarks the kind has; the benchmark half
++ * of the weight is split over that whole list, so a model's missing benchmarks reduce its weight sum rather than
++ * silently making its one benchmark worth half the Index. When the caller does not pass it, the list's length is
++ * the count (the model has every benchmark of the kind).
++ */
++function openvibeIndex({ rating = null, benchmarks = [], benchmarkCount = null, weights = INDEX_WEIGHTS } = {}) {
++ const ratingWeight = num(weights && weights.rating);
++ const benchmarkWeight = num(weights && weights.benchmarks);
++ const list = (Array.isArray(benchmarks) ? benchmarks : [])
++ .filter((b) => b && num(b.score) != null)
++ .map((b) => ({
++ id: String(b.id),
++ label: b.label == null ? String(b.id) : String(b.label),
++ value: clamp(num(b.score), 0, 100),
++ runs: b.runs == null ? null : num(b.runs),
++ }));
++ const count = Math.max(list.length, Math.max(0, num(benchmarkCount) || 0));
++ const perBenchmark = count ? (benchmarkWeight || 0) / count : 0;
++
++ const components = [];
++ const ratingValue = ratingScore(rating);
++ if (ratingValue != null && ratingWeight != null) {
++ components.push({ id: 'rating', kind: 'rating', label: 'Human rating', value: ratingValue, weight: ratingWeight, runs: null });
++ }
++ for (const benchmark of list) {
++ components.push({ id: benchmark.id, kind: 'benchmark', label: benchmark.label, value: benchmark.value, weight: perBenchmark, runs: benchmark.runs });
++ }
++ const weightSum = components.reduce((sum, component) => sum + component.weight, 0);
++ const value = components.length >= 2 && weightSum > 0
++ ? components.reduce((sum, component) => sum + component.value * component.weight, 0) / weightSum
++ : null;
++ return {
++ index: round(value, 2),
++ n_components: components.length,
++ weight_sum: round(weightSum, 6),
++ benchmark_count: count,
++ components: components.map((component) => ({
++ ...component,
++ value: round(component.value, 2),
++ weight: round(component.weight, 6),
++ })),
++ };
++}
++
++/**
++ * The holdout gap for one model: the mean score on the public tasks minus the mean on the held-out tasks, with the
++ * number of scored tasks on each side. A side with no scores makes the gap null, never zero.
++ */
++function holdoutGap(publicScores = [], holdoutScores = []) {
++ const publicList = (Array.isArray(publicScores) ? publicScores : []).map(num).filter((v) => v != null);
++ const holdoutList = (Array.isArray(holdoutScores) ? holdoutScores : []).map(num).filter((v) => v != null);
++ const publicScore = mean(publicList);
++ const holdoutScore = mean(holdoutList);
++ return {
++ public_n: publicList.length,
++ public_score: round(publicScore, 2),
++ holdout_n: holdoutList.length,
++ holdout_score: round(holdoutScore, 2),
++ gap: publicScore != null && holdoutScore != null ? round(publicScore - holdoutScore, 2) : null,
++ };
++}
++
++module.exports = {
++ INDEX_WEIGHTS, RATING_ANCHOR,
++ openvibeIndex, holdoutGap, ratingScore, mean, median,
++};
ddiff --git a/server/rating/style.js b/server/rating/style.js
iindex b69d84f..7f19d99 100644
---- a/server/rating/style.js
++++ b/server/rating/style.js
@@@ -34,7 +34,7 @@
* Bradley-Terry so the numbers mean what a reader expects.
*/
--const { percentile } = require('./lib');
++const { percentile, ranksOf } = require('./lib');
/** Elo points per unit of log-odds: the usual 400 / ln(10). */
const SCALE = 400 / Math.log(10);
@@@ -381,6 +381,8 @@ function fitRatings(games, {
const rawSamples = new Map();
const ctrlSamples = new Map();
++ const rawRankSamples = new Map();
++ const ctrlRankSamples = new Map();
const random = seededRandom(seed);
const rounds = Math.max(0, Math.min(BOOTSTRAP, Number(bootstrap) || 0));
const n = prepared.count;
@@@ -391,16 +393,33 @@ function fitRatings(games, {
for (let j = 0; j < n; j++) order[j] = Math.floor(random() * n);
const r = fitPrepared(prepared, { order, use: [], ridge, maxIter: 60, warm: { theta: raw.theta, beta: [] } });
const c = fitPrepared(prepared, { order, use: covariates, ridge, betaRidge, maxIter: 60, warm: { theta: controlled.theta, beta: controlled.beta } });
++ const rawRanks = ranksOf(r.theta);
++ const ctrlRanks = ranksOf(c.theta);
for (const [subject, position] of rawIndex) {
-- if (!rawSamples.has(subject)) { rawSamples.set(subject, []); ctrlSamples.set(subject, []); }
++ if (!rawSamples.has(subject)) {
++ rawSamples.set(subject, []); ctrlSamples.set(subject, []);
++ rawRankSamples.set(subject, []); ctrlRankSamples.set(subject, []);
++ }
rawSamples.get(subject).push(r.theta[position]);
-- ctrlSamples.get(subject).push(c.theta[ctrlIndex.has(subject) ? ctrlIndex.get(subject) : 0]);
++ rawRankSamples.get(subject).push(rawRanks[position]);
++ const ctrlPosition = ctrlIndex.has(subject) ? ctrlIndex.get(subject) : 0;
++ ctrlSamples.get(subject).push(c.theta[ctrlPosition]);
++ ctrlRankSamples.get(subject).push(ctrlRanks[ctrlPosition]);
}
}
const toRating = (theta) => ANCHOR + SCALE * theta;
const rawTheta = new Map(raw.subjects.map((s, i) => [s, raw.theta[i]]));
const ctrlTheta = new Map(controlled.subjects.map((s, i) => [s, controlled.theta[i]]));
++ const rawPointRankList = ranksOf(raw.theta);
++ const ctrlPointRankList = ranksOf(controlled.theta);
++ const rawPointRanks = new Map(raw.subjects.map((s, i) => [s, rawPointRankList[i]]));
++ const ctrlPointRanks = new Map(controlled.subjects.map((s, i) => [s, ctrlPointRankList[i]]));
++ const rankCell = (samples, point) => ({
++ rank: point,
++ rank_lo: samples.length ? Math.round(percentile(samples, 0.025)) : point,
++ rank_hi: samples.length ? Math.round(percentile(samples, 0.975)) : point,
++ });
const rows = raw.subjects.map((subject) => {
const thetaRaw = rawTheta.get(subject);
const thetaCtrl = ctrlTheta.get(subject);
@@@ -413,11 +432,13 @@ function fitRatings(games, {
rating: Math.round(toRating(thetaRaw) * 100) / 100,
ci_lower: rawP.length ? Math.round(toRating(percentile(rawP, 0.025)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,
ci_upper: rawP.length ? Math.round(toRating(percentile(rawP, 0.975)) * 100) / 100 : Math.round(toRating(thetaRaw) * 100) / 100,
++ ...rankCell(rawRankSamples.get(subject) || [], rawPointRanks.get(subject)),
},
controlled: {
rating: Math.round(toRating(thetaCtrl) * 100) / 100,
ci_lower: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.025)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,
ci_upper: ctrlP.length ? Math.round(toRating(percentile(ctrlP, 0.975)) * 100) / 100 : Math.round(toRating(thetaCtrl) * 100) / 100,
++ ...rankCell(ctrlRankSamples.get(subject) || [], ctrlPointRanks.get(subject)),
},
};
}).sort((x, y) => y.raw.rating - x.raw.rating || x.model.localeCompare(y.model));
ddiff --git a/server/store.js b/server/store.js
iindex d9c0f09..98458cf 100644
---- a/server/store.js
++++ b/server/store.js
@@@ -13,6 +13,7 @@ const clipLib = require('./bench/clip');
const clipSource = require('./bench/clip-source');
const ratingLib = require('./rating');
const styleLib = require('./rating/style');
++const statsLib = require('./rating/stats');
const { checkSvg } = require('./bench/svg-check');
const BASE_ELO = ratingLib.BASE_ELO;
@@@ -203,6 +204,14 @@ function ratingCi(elo, wins, losses, ties) {
function createStore(s, { log = console, random = Math.random, config = null } = {}) {
const db = s.db;
++ // The kind boards' fits are pure functions of the games, so a board asks for the same fit only once per
++ // vote set. The key changes with the number of games, and a short TTL covers a vote that lands mid-request.
++ const boardFits = new Map();
++ const BOARD_BOOTSTRAP = 200;
++ const BOARD_FIT_TTL_MS = 5 * 60_000;
++ // A per-category board is published only when its category has this many games; below it there is not
++ // enough play to separate models, so the board is left out of the category list rather than shown thin.
++ const CATEGORY_MIN_GAMES = 30;
// ── Outbox (in the same ambient transaction as the change that caused it) ──
async function insertOutbox(topic, payload) {
@@@ -460,24 +469,43 @@ function createStore(s, { log = console, random = Math.random, config = null } =
}
// ── Judgements ──────────────────────────────────────────
++ /**
++ * The pairwise games of a board, as the rating maths reads them: two model names and an outcome of
++ * 'a', 'b' or 'tie'. The pair on a judgement names two entries, so the models come from the entries, never
++ * from the entry ids.
++ *
++ * The category picks the games the board counts: 'all' every vote, 'people' the signed-in voters, 'anon' the
++ * anonymous ones, and any other slug the task's own category (code, writing, chat, reasoning…).
++ */
async function pairwiseGames(kindId, category = CATEGORY_DEFAULT) {
++ const cat = String(category == null ? CATEGORY_DEFAULT : category);
const rows = await db.many(
-- `SELECT * FROM bench_judgements
-- WHERE kind_id = $1 AND method IN ('human_pairwise', 'game_outcome', 'audience_poll') AND pair IS NOT NULL
-- AND ($2::text = 'all' OR outcome IS NOT NULL)
-- ORDER BY at, id`,
-- [kindId, category],
++ `SELECT j.id, j.method, j.outcome, j.vote_counts, l.model AS left_model, e.model AS right_model
++ FROM bench_judgements j
++ JOIN bench_rounds rd ON rd.id = j.round_id
++ JOIN bench_tasks t ON t.id = rd.task_id
++ JOIN bench_entries l ON l.id = j.pair->>'left' AND l.round_id = j.round_id
++ JOIN bench_entries e ON e.id = j.pair->>'right' AND e.round_id = j.round_id
++ WHERE j.kind_id = $1
++ AND j.method IN ('human_pairwise', 'game_outcome', 'audience_poll')
++ AND j.pair IS NOT NULL AND j.outcome IS NOT NULL
++ AND ($2::text <> 'people' OR COALESCE(j.voter_kind, 'person') = 'person')
++ AND ($2::text <> 'anon' OR j.voter_kind = 'anon')
++ AND ($2::text IN ('all', 'people', 'anon') OR t.task_class = $2)
++ ORDER BY j.at, j.id`,
++ [kindId, cat],
);
const out = [];
for (const j of rows) {
-- const pair = typeof j.pair === 'string' ? JSON.parse(j.pair) : j.pair;
-- if (!pair || !pair.left || !pair.right) continue;
if (j.outcome === 'both_bad') continue;
if (j.method === 'audience_poll') {
const counts = typeof j.vote_counts === 'string' ? JSON.parse(j.vote_counts) : j.vote_counts;
if (!counts || !num(counts.voters)) continue;
}
-- out.push({ a: pair.left, b: pair.right, outcome: j.outcome === 'left' ? 1 : j.outcome === 'right' ? 0 : 0.5 });
++ if (!j.left_model || !j.right_model || j.left_model === j.right_model) continue;
++ const outcome = j.outcome === 'left' ? 'a' : j.outcome === 'right' ? 'b' : (j.outcome === 'tie' || j.outcome === 'draw') ? 'tie' : null;
++ if (!outcome) continue;
++ out.push({ id: j.id, a: j.left_model, b: j.right_model, outcome });
}
return out;
}
@@@ -640,23 +668,47 @@ function createStore(s, { log = console, random = Math.random, config = null } =
return rows.map((r) => r.model);
}
++ /**
++ * The Bradley-Terry fit of one board, cached on the vote set. The fit is the same maths the ratings page uses
++ * (server/rating/lib.js): every game is one vote, the ratings are centred on 1000, and the bootstrap supplies
++ * both the rating interval and the 95% rank interval. The seed is fixed, so the same games give the same board.
++ */
++ async function pairwiseFit(kindId, category) {
++ const games = await pairwiseGames(kindId, category);
++ const key = `${kindId}:${category}:${games.length}:${games.length ? games[games.length - 1].id : '-'}`;
++ const hit = boardFits.get(key);
++ if (hit && Date.now() - hit.at < BOARD_FIT_TTL_MS) return hit.fit;
++ const fit = ratingLib.bradleyTerry(games, { bootstrap: BOARD_BOOTSTRAP, seed: 1 });
++ boardFits.set(key, { at: Date.now(), fit });
++ if (boardFits.size > 200) {
++ const oldest = [...boardFits.entries()].sort((a, b) => a[1].at - b[1].at).slice(0, 100);
++ for (const [stale] of oldest) boardFits.delete(stale);
++ }
++ return fit;
++ }
++
async function boardElo(kindId, category = CATEGORY_DEFAULT) {
const subjects = await subjectsFor(kindId);
-- const ratings = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2', [kindId, category]);
-- const by = new Map(ratings.map((r) => [r.subject, wireRating(r)]));
++ const fit = await pairwiseFit(kindId, category);
++ const by = new Map(fit.map((r) => [r.subject, r]));
// A task category only lists the models that played in it: a model with no game in this kind of
// work has no place on its board, rather than a baseline score it never earned.
const taskCategory = ![CATEGORY_DEFAULT, CATEGORY_PEOPLE, CATEGORY_ANON].includes(category);
-- const rows = (taskCategory ? subjects.filter((subject) => by.has(subject)) : subjects).map((subject) => {
++ const listed = new Set(taskCategory ? subjects.filter((subject) => by.has(subject)) : subjects);
++ for (const r of fit) listed.add(r.subject);
++ const rows = [...listed].map((subject) => {
const r = by.get(subject);
const games = r ? r.games : 0;
return {
kind_id: kindId, category, subject,
-- elo: r ? r.elo : BASE_ELO,
++ elo: r ? Math.round(r.rating * 100) / 100 : BASE_ELO,
games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,
-- win_rate: r ? r.win_rate : 0,
-- ci_lower: r ? r.ci_lower : null,
-- ci_upper: r ? r.ci_upper : null,
++ win_rate: games ? Math.round(((r.wins + r.ties * 0.5) / games) * 10000) / 10000 : 0,
++ ci_lower: r ? Math.round(r.ci_lower * 100) / 100 : null,
++ ci_upper: r ? Math.round(r.ci_upper * 100) / 100 : null,
++ rank: r ? r.rank : null,
++ rank_lo: r ? r.rank_lo : null,
++ rank_hi: r ? r.rank_hi : null,
provisional: games < 10,
};
});
@@@ -664,6 +716,136 @@ function createStore(s, { log = console, random = Math.random, config = null } =
return rows;
}
++ /**
++ * The per-category boards: one for every task category of the kind with at least CATEGORY_MIN_GAMES games, in
++ * the same shape as the kind's own board. A category below the floor is not listed at all, so a page can say
++ * "not enough games yet" without drawing a board that cannot separate anything.
++ */
++ async function categoryBoards(kindId) {
++ const rows = await db.many(
++ `SELECT t.task_class AS category, COUNT(*)::int AS games
++ FROM bench_judgements j
++ JOIN bench_rounds rd ON rd.id = j.round_id
++ JOIN bench_tasks t ON t.id = rd.task_id
++ JOIN bench_entries l ON l.id = j.pair->>'left' AND l.round_id = j.round_id
++ JOIN bench_entries e ON e.id = j.pair->>'right' AND e.round_id = j.round_id
++ WHERE j.kind_id = $1
++ AND j.method IN ('human_pairwise', 'game_outcome', 'audience_poll')
++ AND j.pair IS NOT NULL AND j.outcome IS NOT NULL AND j.outcome <> 'both_bad'
++ AND t.task_class IS NOT NULL
++ AND t.task_class NOT IN ('all', 'people', 'anon')
++ GROUP BY t.task_class
++ HAVING COUNT(*) >= $2
++ ORDER BY COUNT(*) DESC, t.task_class`,
++ [kindId, CATEGORY_MIN_GAMES],
++ );
++ const out = [];
++ for (const row of rows) {
++ // Use the same eligible votes as the fit: self-pairs and empty polls do not earn a board.
++ const games = (await pairwiseGames(kindId, row.category)).length;
++ if (games < CATEGORY_MIN_GAMES) continue;
++ out.push({ category: row.category, games, rows: await boardElo(kindId, row.category) });
++ }
++ return out.sort((a, b) => b.games - a.games || a.category.localeCompare(b.category));
++ }
++
++ /**
++ * The OpenVibe Index for a kind: the Bradley-Terry rating plus the model's median verified score on each public
++ * benchmark of that kind, weighted 50/50 and split equally between the benchmarks. The weights are in the
++ * answer; a model with fewer than two components gets null, never a number from one component.
++ */
++ async function kindIndex(kindId, eloRows) {
++ const rows = await db.many(
++ `SELECT b.id, b.title, r.model, r.score
++ FROM bench_runs r
++ JOIN bench_benchmarks b ON b.id = r.benchmark_id
++ WHERE b.kind = $1 AND b.visibility = 'public' AND r.provenance = 'verified'
++ AND r.score IS NOT NULL
++ ORDER BY b.created_at, b.id, r.model, r.created_at, r.id`,
++ [kindId],
++ );
++ const benchmarks = new Map();
++ const scores = new Map();
++ for (const row of rows) {
++ if (!benchmarks.has(row.id)) benchmarks.set(row.id, { id: row.id, label: row.title });
++ const key = `${row.id}\u0000${row.model}`;
++ if (!scores.has(key)) scores.set(key, []);
++ scores.get(key).push(num(row.score));
++ }
++ const benchmarkList = [...benchmarks.values()];
++ const modelNames = new Set(eloRows.filter((row) => row.rank != null).map((row) => row.subject));
++ for (const key of scores.keys()) modelNames.add(key.split('\u0000')[1]);
++ const indexRows = [...modelNames].map((subject) => {
++ const ratingRow = eloRows.find((row) => row.subject === subject);
++ const present = benchmarkList
++ .map((benchmark) => ({
++ id: benchmark.id,
++ label: benchmark.label,
++ runs: (scores.get(`${benchmark.id}\u0000${subject}`) || []).length,
++ score: statsLib.median(scores.get(`${benchmark.id}\u0000${subject}`) || []),
++ }))
++ .filter((benchmark) => benchmark.score != null);
++ const value = statsLib.openvibeIndex({
++ rating: ratingRow && ratingRow.rank != null ? ratingRow.elo : null,
++ benchmarks: present,
++ benchmarkCount: benchmarkList.length,
++ });
++ return { subject, ...value };
++ });
++ indexRows.sort((a, b) => {
++ if (a.index == null && b.index == null) return a.subject.localeCompare(b.subject);
++ if (a.index == null) return 1;
++ if (b.index == null) return -1;
++ return b.index - a.index || a.subject.localeCompare(b.subject);
++ });
++ return {
++ kind_id: kindId,
++ weights: { ...statsLib.INDEX_WEIGHTS },
++ rating_anchor: statsLib.RATING_ANCHOR,
++ benchmark_count: benchmarkList.length,
++ benchmarks: benchmarkList.map((benchmark) => ({
++ id: benchmark.id,
++ label: benchmark.label,
++ weight: benchmarkList.length ? Math.round((statsLib.INDEX_WEIGHTS.benchmarks / benchmarkList.length) * 1e6) / 1e6 : 0,
++ })),
++ rows: indexRows,
++ };
++ }
++
++ /**
++ * The holdout gap for a kind: per model, the mean score on public tasks minus the mean score on held-out tasks,
++ * with the number of scored tasks on each side. Verified runs of public benchmarks only. A kind without
++ * held-out tasks says `available: false`; a model missing one side gets a null gap.
++ */
++ async function kindHoldout(kindId) {
++ const rows = await db.many(
++ `SELECT b.id AS benchmark_id, r.model, t.task_id, t.score, t.held_out
++ FROM bench_run_tasks t
++ JOIN bench_runs r ON r.id = t.run_id
++ JOIN bench_benchmarks b ON b.id = r.benchmark_id
++ WHERE b.kind = $1 AND b.visibility = 'public' AND r.provenance = 'verified'
++ AND t.score IS NOT NULL
++ ORDER BY r.model, t.created_at, t.id`,
++ [kindId],
++ );
++ const by = new Map();
++ for (const row of rows) {
++ if (!by.has(row.model)) by.set(row.model, { public: [], holdout: [] });
++ const bucket = by.get(row.model);
++ if (row.held_out) bucket.holdout.push(num(row.score));
++ else bucket.public.push(num(row.score));
++ }
++ const out = [...by.entries()].map(([subject, bucket]) => ({ subject, ...statsLib.holdoutGap(bucket.public, bucket.holdout) }));
++ out.sort((a, b) => (b.gap == null ? -Infinity : b.gap) - (a.gap == null ? -Infinity : a.gap) || a.subject.localeCompare(b.subject));
++ const available = rows.some((row) => row.held_out);
++ return {
++ kind_id: kindId,
++ available,
++ held_out_tasks: new Set(rows.filter((row) => row.held_out).map((row) => `${row.benchmark_id}:${row.task_id}`)).size,
++ rows: out,
++ };
++ }
++
async function costStats(kindId) {
const rows = await db.many(
`SELECT e.model,
@@@ -763,7 +945,19 @@ function createStore(s, { log = console, random = Math.random, config = null } =
} else {
return null;
}
-- return { kind_id: kindId, category: cat, period, type, rows, voters: await voterCounts(kindId) };
++ // The kind's own statistics travel with every board: the rating board also carries its per-category
++ // boards, and every type carries the OpenVibe Index and the holdout gap. The Index always reads the
++ // whole kind's rating, even when the rows are filtered to one kind of voter.
++ const allEloRows = cat === CATEGORY_DEFAULT ? eloRows : await boardElo(kindId);
++ const [voterKinds, categories, index, holdout] = await Promise.all([
++ voterCounts(kindId),
++ type === 'elo' ? categoryBoards(kindId) : Promise.resolve(undefined),
++ kindIndex(kindId, allEloRows),
++ kindHoldout(kindId),
++ ]);
++ const out = { kind_id: kindId, category: cat, period, type, rows, voters: voterKinds, index, holdout };
++ if (categories !== undefined) out.categories = categories;
++ return out;
}
/**
@@@ -830,8 +1024,25 @@ function createStore(s, { log = console, random = Math.random, config = null } =
entries = (await db.many('SELECT * FROM bench_entries WHERE round_id = ANY($1::text[]) ORDER BY created_at, id', [roundIds])).map(wireEntry);
judgements = (await db.many('SELECT * FROM bench_judgements WHERE round_id = ANY($1::text[]) ORDER BY at, id', [roundIds])).map((j) => wireJudgement(j));
}
-- const ratings = (await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 ORDER BY subject', [kindId])).map(wireRating);
-- return { kind_id: kindId, tasks, rounds, entries, judgements, ratings };
++ const allEloRows = await boardElo(kindId);
++ const savedRatings = (await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 ORDER BY subject', [kindId])).map(wireRating);
++ const fitted = new Map([[CATEGORY_DEFAULT, allEloRows]]);
++ for (const category of new Set(savedRatings.map((rating) => rating.category))) {
++ if (!fitted.has(category)) fitted.set(category, await boardElo(kindId, category));
++ }
++ // Export the same fit and intervals as the board, not the old incremental Elo beside a fitted rank.
++ const ratings = [...fitted.values()].flat().filter((rating) => rating.games > 0).map((rating) => ({
++ ...savedRatings.find((saved) => saved.subject === rating.subject && saved.category === rating.category),
++ ...rating,
++ }));
++ const [categories, index, holdout] = await Promise.all([
++ categoryBoards(kindId),
++ kindIndex(kindId, allEloRows),
++ kindHoldout(kindId),
++ ]);
++ // The open dataset carries the same four statistics the board API answers with: every rating's rank
++ // interval, the per-category boards, the OpenVibe Index with its weights, and the holdout gap.
++ return { kind_id: kindId, tasks, rounds, entries, judgements, ratings, categories, index, holdout };
}
async function datasetJsonl(kindId) {
@@@ -843,6 +1054,9 @@ function createStore(s, { log = console, random = Math.random, config = null } =
for (const e of d.entries) lines.push(JSON.stringify({ type: 'entry', ...e }));
for (const j of d.judgements) lines.push(JSON.stringify({ type: 'judgement', ...j }));
for (const r of d.ratings) lines.push(JSON.stringify({ type: 'rating', ...r }));
++ for (const c of d.categories) lines.push(JSON.stringify({ type: 'category', ...c }));
++ lines.push(JSON.stringify({ type: 'index', ...d.index }));
++ lines.push(JSON.stringify({ type: 'holdout', ...d.holdout }));
return lines.length ? `${lines.join('\n')}\n` : '';
}
ddiff --git a/test/boards.test.js b/test/boards.test.js
iindex e64c06a..65ecd88 100644
---- a/test/boards.test.js
++++ b/test/boards.test.js
@@@ -42,6 +42,128 @@ const { boot, check, done } = require('./helpers/boot');
assert.ok(after.some((row) => row.games === 1 && row.provisional === true), 'one game is still provisional');
});
++ await check('every elo board carries 95% rank intervals from the bootstrap', async () => {
++ const board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
++ assert.ok(board.rows.length >= 2);
++ for (const row of board.rows) {
++ assert.ok(Number.isInteger(row.rank) && row.rank >= 1 && row.rank <= board.rows.length, `${row.subject} rank`);
++ assert.ok(Number.isInteger(row.rank_lo) && row.rank_lo >= 1, `${row.subject} rank_lo`);
++ assert.ok(Number.isInteger(row.rank_hi) && row.rank_hi <= board.rows.length, `${row.subject} rank_hi`);
++ assert.ok(row.rank_lo <= row.rank_hi, `${row.subject} interval is ordered`);
++ assert.ok(row.ci_lower != null && row.ci_upper != null, `${row.subject} rating interval`);
++ }
++ const exported = (await t.get('/api/v1/datasets/design.page.jsonl')).text.trim().split('\n').map(JSON.parse);
++ for (const row of board.rows) {
++ const saved = exported.find((entry) => entry.type === 'rating' && entry.category === 'all' && entry.subject === row.subject);
++ for (const field of ['elo', 'games', 'ci_lower', 'ci_upper', 'rank', 'rank_lo', 'rank_hi']) {
++ assert.strictEqual(saved[field], row[field], `dataset ${row.subject} ${field} matches the board`);
++ }
++ }
++ assert.ok(board.index.rows.every((row) => row.index === null), 'a rating alone is never an Index');
++ assert.strictEqual(board.holdout.available, false, 'this kind has no held-out tasks');
++ assert.deepStrictEqual(board.holdout.rows, []);
++ });
++
++ // A category board: a task class with at least 30 games earns one; 29 does not.
++ async function roundForCategory(category) {
++ const task = await t.get('/api/v1/kinds/design.page/tasks', { as: alice, json: { prompt: 'A ' + category + ' page.', public: true, task_class: category } });
++ const rr = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });
++ const id = rr.json().round.id;
++ const a = await t.get(`/api/v1/rounds/${id}/entries`, { bearer: entrySvc, json: { provider: 'acme', model: 'alpha-1', payload: { html: '<h1>a</h1>' } } });
++ const b = await t.get(`/api/v1/rounds/${id}/entries`, { bearer: entrySvc, json: { provider: 'beta', model: 'beta-2', payload: { html: '<h1>b</h1>' } } });
++ return { roundId: id, left: a.json().entry.id, right: b.json().entry.id };
++ }
++
++ async function seedGames(round, count) {
++ for (let i = 0; i < count; i++) {
++ await t.ctx.s.db.exec(
++ `INSERT INTO bench_judgements (id, round_id, kind_id, judge, method, pair, outcome, subject, voter_kind)
++ VALUES ($1, $2, 'design.page', 'human_pairwise', 'human_pairwise', $3::jsonb, $4, $5, 'person')`,
++ [t.store.newId('bju'), round.roundId, JSON.stringify({ left: round.left, right: round.right }), i % 2 ? 'left' : 'right', 'usr_seed_' + i],
++ );
++ }
++ }
++
++ await check('a category board appears at 30 games and not at 29', async () => {
++ const arcade = await roundForCategory('arcade');
++ const writing = await roundForCategory('writing');
++ await seedGames(writing, 29);
++ const selfOnly = await roundForCategory('self-only');
++ await seedGames(selfOnly, 30);
++ await t.ctx.s.db.exec('UPDATE bench_entries SET model = $1 WHERE id = $2', ['alpha-1', selfOnly.right]);
++ let board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
++ assert.strictEqual(board.categories.length, 0, '29 games is below the floor: ' + JSON.stringify(board.categories));
++ await seedGames(arcade, 30);
++ board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
++ const category = board.categories.find((c) => c.category === 'arcade');
++ assert.ok(category, 'the 30-game category has a board');
++ assert.strictEqual(category.games, 30);
++ assert.ok(category.rows.every((row) => row.games > 0), 'only models that played are listed');
++ assert.ok(category.rows.every((row) => Number.isInteger(row.rank_lo) && row.rank_lo <= row.rank_hi), 'the category board has rank intervals');
++ assert.strictEqual(board.categories.find((c) => c.category === 'writing'), undefined, '29 games is still out');
++ // The filtered board answers for one category alone.
++ const one = (await t.get('/api/v1/boards/design.page?category=arcade')).json().board;
++ assert.strictEqual(one.category, 'arcade');
++ assert.strictEqual(one.rows.reduce((n, row) => n + row.games, 0), 60, '30 games are 60 model-games');
++ });
++
++ await check('the OpenVibe Index needs two components and publishes its weights', async () => {
++ const definition = {
++ title: 'Design pages', description: 'The B3 index fixture.', kind: 'design.page', tags: [], visibility: 'public',
++ tasks: [
++ { id: 't1', prompt: 'A public page.', files: [], expected: null, holdout: false },
++ { id: 't2', prompt: 'A held-out page.', files: [], expected: null, holdout: true },
++ ],
++ judges: [{ type: 'human_pairwise', version: '1.0', config: {} }],
++ scoring: { method: 'mean', scale: { min: 0, max: 100 } },
++ limits: { ms: 10000, tokens: 10000, usd: 1 },
++ matrix: { models: ['alpha-1'], harnesses: ['fixture'] },
++ };
++ const writer = t.network.serviceToken({ sub: 'svc:fixture', audience: 'openvibe.bench', cap: ['bench.benchmark.write', 'bench.run.publish'] });
++ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition });
++ assert.strictEqual(created.status, 201, created.text);
++ const benchmarkId = created.json().benchmark.id;
++ const frozen = await t.get(`/api/v1/benchmarks/${benchmarkId}/versions`, { bearer: writer, json: {} });
++ assert.strictEqual(frozen.status, 201, frozen.text);
++ const run = await t.get('/api/v1/runs', {
++ bearer: writer,
++ json: {
++ benchmark: `${benchmarkId}@1`, model: 'alpha-1', route: { provider: 'acme' }, provider: 'acme',
++ harness: { name: 'fixture', version: '1' }, settings: {}, executor: { kind: 'codes', id: 'fixture' },
++ tasks: [
++ { task_id: 't1', score: 90, cost_usd: 0.01, ms: 100, tokens: 10, artifacts: [] },
++ { task_id: 't2', score: 60, cost_usd: 0.01, ms: 100, tokens: 10, artifacts: [] },
++ ],
++ score: 75, cost_usd: 0.02, ms: 200, tokens: 20,
++ },
++ });
++ assert.strictEqual(run.status, 201, run.text);
++
++ const board = (await t.get('/api/v1/boards/design.page?type=elo')).json().board;
++ assert.deepStrictEqual(board.index.weights, { rating: 0.5, benchmarks: 0.5 });
++ assert.strictEqual(board.index.benchmark_count, 1);
++ assert.strictEqual(board.index.benchmarks[0].weight, 0.5);
++ const alpha = board.index.rows.find((row) => row.subject === 'alpha-1');
++ const beta = board.index.rows.find((row) => row.subject === 'beta-2');
++ assert.ok(alpha, 'the rated model is in the index');
++ assert.strictEqual(alpha.n_components, 2, 'rating + one benchmark: ' + JSON.stringify(alpha.components));
++ assert.ok(typeof alpha.index === 'number' && alpha.index >= 0 && alpha.index <= 100, 'the index is 0-100');
++ assert.ok(alpha.components.some((c) => c.kind === 'rating') && alpha.components.some((c) => c.kind === 'benchmark'), 'both kinds of component are named');
++ assert.strictEqual(beta.index, null, 'a single component is never an index');
++ assert.strictEqual(beta.n_components, 1);
++
++ assert.strictEqual(board.holdout.available, true);
++ const gap = board.holdout.rows.find((row) => row.subject === 'alpha-1');
++ assert.deepStrictEqual(gap, { subject: 'alpha-1', public_n: 1, public_score: 90, holdout_n: 1, holdout_score: 60, gap: 30 });
++
++ const dataset = (await t.get('/api/v1/datasets/design.page.jsonl')).text.trim().split('\n').map((line) => JSON.parse(line));
++ assert.ok(dataset.some((line) => line.type === 'index' && line.weights.rating === 0.5), 'the dataset carries the index');
++ assert.ok(dataset.some((line) => line.type === 'holdout' && line.rows.some((row) => row.gap === 30)), 'the dataset carries the holdout gap');
++ assert.ok(dataset.some((line) => line.type === 'category' && line.category === 'arcade'), 'the dataset carries the category board');
++ assert.ok(dataset.some((line) => line.type === 'rating' && line.rank_lo !== null), 'the dataset carries the rank interval');
++ });
++
++
await check('a model page aggregates across kinds', async () => {
const task = await t.get('/api/v1/kinds/games.match/tasks', { as: alice, json: { prompt: 'A match.', public: true } });
const rr = await t.get(`/api/v1/tasks/${task.json().task.id}/rounds`, { as: alice, json: {} });
ddiff --git a/test/fixtures/rating/holdout-cases.json b/test/fixtures/rating/holdout-cases.json
nnew file mode 100644
iindex 0000000..0642b96
---- /dev/null
++++ b/test/fixtures/rating/holdout-cases.json
@@@ -0,0 +1,34 @@
++{
++ "cases": [
++ {
++ "name": "a public set and a holdout: the gap is public minus holdout",
++ "public": [80, 90],
++ "holdout": [60],
++ "expect": { "public_n": 2, "public_score": 85, "holdout_n": 1, "holdout_score": 60, "gap": 25 }
++ },
++ {
++ "name": "no held-out score: the gap is null, never zero",
++ "public": [90, 70],
++ "holdout": [],
++ "expect": { "public_n": 2, "public_score": 80, "holdout_n": 0, "holdout_score": null, "gap": null }
++ },
++ {
++ "name": "no public score: the gap is null too",
++ "public": [],
++ "holdout": [40],
++ "expect": { "public_n": 0, "public_score": null, "holdout_n": 1, "holdout_score": 40, "gap": null }
++ },
++ {
++ "name": "nothing scored: every number is absent, not zero",
++ "public": [],
++ "holdout": [],
++ "expect": { "public_n": 0, "public_score": null, "holdout_n": 0, "holdout_score": null, "gap": null }
++ },
++ {
++ "name": "a negative gap says the held-out set scored higher",
++ "public": [55.5],
++ "holdout": [70.25],
++ "expect": { "public_n": 1, "public_score": 55.5, "holdout_n": 1, "holdout_score": 70.25, "gap": -14.75 }
++ }
++ ]
++}
ddiff --git a/test/fixtures/rating/index-cases.json b/test/fixtures/rating/index-cases.json
nnew file mode 100644
iindex 0000000..56a5b80
---- /dev/null
++++ b/test/fixtures/rating/index-cases.json
@@@ -0,0 +1,50 @@
++{
++ "weights": { "rating": 0.5, "benchmarks": 0.5 },
++ "cases": [
++ {
++ "name": "a rating alone is not an Index",
++ "rating": 1200,
++ "benchmarks": [],
++ "benchmarkCount": 0,
++ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5 }
++ },
++ {
++ "name": "one benchmark alone is not an Index either",
++ "rating": null,
++ "benchmarks": [{ "id": "b1", "label": "One", "score": 80, "runs": 2 }],
++ "benchmarkCount": 1,
++ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5 }
++ },
++ {
++ "name": "a rating and one of two benchmarks: the missing benchmark's weight renormalises",
++ "rating": 1000,
++ "benchmarks": [{ "id": "b1", "label": "One", "score": 80, "runs": 3 }],
++ "benchmarkCount": 2,
++ "expect": { "index": 60, "n_components": 2, "weight_sum": 0.75 }
++ },
++ {
++ "name": "two benchmarks and no rating still make an Index",
++ "rating": null,
++ "benchmarks": [
++ { "id": "b1", "label": "One", "score": 40, "runs": 1 },
++ { "id": "b2", "label": "Two", "score": 80, "runs": 1 }
++ ],
++ "benchmarkCount": 2,
++ "expect": { "index": 60, "n_components": 2, "weight_sum": 0.5 }
++ },
++ {
++ "name": "a score over 100 is clamped, not trusted",
++ "rating": 1000,
++ "benchmarks": [{ "id": "b1", "label": "One", "score": 140, "runs": 1 }],
++ "benchmarkCount": 1,
++ "expect": { "index": 75, "n_components": 2, "weight_sum": 1 }
++ },
++ {
++ "name": "the rating is the Elo expectation against 1000, as 0-100 points",
++ "rating": 1000,
++ "benchmarks": [],
++ "benchmarkCount": 0,
++ "expect": { "index": null, "n_components": 1, "weight_sum": 0.5, "rating_value": 50 }
++ }
++ ]
++}
ddiff --git a/test/rating-lib.test.js b/test/rating-lib.test.js
iindex 2171fb4..067f4ff 100644
---- a/test/rating-lib.test.js
++++ b/test/rating-lib.test.js
@@@ -105,3 +105,35 @@ test('percentile interpolates and handles empty input', () => {
assert.ok(Math.abs(percentile([1, 2, 3, 4], 95) - 3.85) < 1e-12);
assert.equal(percentile([], 95), null);
});
++
++test('rank intervals come from the same bootstrap as the rating intervals', () => {
++ const games = [
++ ...Array.from({ length: 24 }, () => game('A', 'B', 'a')),
++ ...Array.from({ length: 6 }, () => game('A', 'B', 'b')),
++ ...Array.from({ length: 20 }, () => game('B', 'C', 'a')),
++ ...Array.from({ length: 10 }, () => game('B', 'C', 'b')),
++ ...Array.from({ length: 25 }, () => game('A', 'C', 'a')),
++ ...Array.from({ length: 5 }, () => game('A', 'C', 'b')),
++ ];
++ const fit = bradleyTerry(games, { bootstrap: 200, seed: 11 });
++ const by = bySubject(fit);
++ assert.equal(by.get('A').rank, 1, 'the planted winner is first');
++ assert.equal(by.get('A').rank_lo, 1);
++ assert.ok(by.get('A').rank_hi >= 1);
++ assert.equal(by.get('C').rank, 3);
++ for (const row of fit) {
++ assert.ok(Number.isInteger(row.rank) && row.rank >= 1 && row.rank <= fit.length, `${row.subject} rank`);
++ assert.ok(Number.isInteger(row.rank_lo) && row.rank_lo >= 1, `${row.subject} rank_lo`);
++ assert.ok(Number.isInteger(row.rank_hi) && row.rank_hi <= fit.length, `${row.subject} rank_hi`);
++ assert.ok(row.rank_lo <= row.rank_hi, `${row.subject} interval is ordered`);
++ }
++ assert.deepEqual(bradleyTerry(games, { bootstrap: 200, seed: 11 }), fit, 'the rank intervals are seeded');
++});
++
++test('without a bootstrap the rank interval is the point rank, never a fraction', () => {
++ const fit = bradleyTerry([game('A', 'B', 'a'), game('A', 'B', 'a'), game('B', 'C', 'a')], { bootstrap: 0 });
++ for (const row of fit) {
++ assert.equal(row.rank_lo, row.rank);
++ assert.equal(row.rank_hi, row.rank);
++ }
++});
ddiff --git a/test/rating-stats.test.js b/test/rating-stats.test.js
nnew file mode 100644
iindex 0000000..db93e8a
---- /dev/null
++++ b/test/rating-stats.test.js
@@@ -0,0 +1,71 @@
++'use strict';
++
++/**
++ * The OpenVibe Index and the holdout gap (stage B3): the published weights, the null for one component, the
++ * renormalisation over the evidence a model actually has, and the holdout gap's n on both sides.
++ */
++const test = require('node:test');
++const assert = require('node:assert/strict');
++const fs = require('node:fs');
++const path = require('node:path');
++const { openvibeIndex, holdoutGap, ratingScore, INDEX_WEIGHTS, RATING_ANCHOR } = require('../server/rating/stats');
++
++const fixture = (name) => JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures/rating', name), 'utf8'));
++
++test('the published weights are the ones the docs and the API name', () => {
++ assert.deepEqual(INDEX_WEIGHTS, { rating: 0.5, benchmarks: 0.5 });
++ assert.equal(RATING_ANCHOR, 1000);
++ assert.equal(ratingScore(1000), 50);
++ assert.ok(Math.abs(ratingScore(1200) - 75.97469266479578) < 1e-9);
++ assert.equal(ratingScore(null), null);
++});
++
++test('the Index fixture: two components or null, with the weights in the answer', () => {
++ for (const testCase of fixture('index-cases.json').cases) {
++ const got = openvibeIndex({
++ rating: testCase.rating,
++ benchmarks: testCase.benchmarks,
++ benchmarkCount: testCase.benchmarkCount,
++ });
++ assert.equal(got.index, testCase.expect.index, testCase.name);
++ assert.equal(got.n_components, testCase.expect.n_components, testCase.name);
++ assert.equal(got.weight_sum, testCase.expect.weight_sum, testCase.name);
++ if (testCase.expect.rating_value != null) {
++ assert.equal(got.components[0].value, testCase.expect.rating_value, testCase.name);
++ }
++ for (const component of got.components) {
++ assert.ok(component.weight > 0, testCase.name + ': every component has its weight');
++ }
++ }
++});
++
++test('the Index never invents a score from one component, and null is not zero', () => {
++ assert.equal(openvibeIndex({ rating: 1300 }).index, null);
++ assert.equal(openvibeIndex({ benchmarks: [{ id: 'b', score: 99 }] }).index, null);
++ assert.equal(openvibeIndex({}).index, null);
++ assert.equal(openvibeIndex({ rating: 1100, benchmarks: [{ id: 'b', score: 90 }] }).index != null, true);
++});
++
++test('a model missing one benchmark of the kind gets the weight renormalised, not a second benchmark', () => {
++ const got = openvibeIndex({
++ rating: 1000,
++ benchmarks: [{ id: 'b1', score: 80 }],
++ benchmarkCount: 3,
++ });
++ assert.equal(got.n_components, 2);
++ assert.equal(got.weight_sum, 0.666667);
++ assert.equal(got.components.find((c) => c.id === 'b1').weight, 0.166667);
++ assert.ok(got.index > 50 && got.index < 80);
++});
++
++test('the holdout gap fixture: public minus holdout, with n on each side', () => {
++ for (const testCase of fixture('holdout-cases.json').cases) {
++ assert.deepEqual(holdoutGap(testCase.public, testCase.holdout), testCase.expect, testCase.name);
++ }
++});
++
++test('the holdout gap ignores missing scores and keeps the side count honest', () => {
++ assert.deepEqual(holdoutGap([80, null, undefined], [null]), {
++ public_n: 1, public_score: 80, holdout_n: 0, holdout_score: null, gap: null,
++ });
++});
ddiff --git a/test/rating-style.test.js b/test/rating-style.test.js
iindex 5b12edc..0ddc2a5 100644
---- a/test/rating-style.test.js
++++ b/test/rating-style.test.js
@@@ -86,6 +86,25 @@ function rng(seed) {
assert.strictEqual(style.bandLabel(1, 1), '1st');
});
++ await check('the fit publishes rank intervals for both boards', async () => {
++ const games = [];
++ for (let i = 0; i < 60; i++) {
++ games.push({ id: 'ra' + i, a: 'alpha', b: 'beta', outcome: 'a', fa: { length: 100, markup: 1 }, fb: { length: 100, markup: 1 } });
++ games.push({ id: 'rb' + i, a: 'beta', b: 'gamma', outcome: 'a', fa: { length: 100, markup: 1 }, fb: { length: 100, markup: 1 } });
++ }
++ const fit = style.fitRatings(games, { bootstrap: 80 });
++ const by = new Map(fit.models.map((m) => [m.model, m]));
++ assert.strictEqual(by.get('alpha').raw.rank, 1);
++ for (const model of fit.models) {
++ for (const cell of [model.raw, model.controlled]) {
++ assert.ok(Number.isInteger(cell.rank) && cell.rank >= 1 && cell.rank <= fit.models.length, model.model + ' rank');
++ assert.ok(Number.isInteger(cell.rank_lo) && cell.rank_lo >= 1, model.model + ' rank_lo');
++ assert.ok(Number.isInteger(cell.rank_hi) && cell.rank_hi <= fit.models.length, model.model + ' rank_hi');
++ assert.ok(cell.rank_lo <= cell.rank_hi, model.model + ' ordered interval');
++ }
++ }
++ });
++
await check('the style features count length and markup', async () => {
const f = style.styleFeatures({ html: '<h1>Title</h1><p><strong>bold</strong></p><ul><li>one</li></ul>' });
assert.ok(f.length > 20, 'length: ' + f.length);