openvibers/openvibe.bench Commits
Signed-out voting and the daily rounds (plan L51, steps 1 and 2) (#64)
Changes
@@@ -41,6 +41,60 @@ run under it, with no person id. Nothing private or unlisted is ever read there.| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. || `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |++## Voting without signing in++++A person who has not signed in may still judge a pairwise round (`human_pairwise` kinds only). There are two ways in,++and both are capped.++++**On Bench's own pages.** `/showcase/round/:id` (and `POST /api/v1/showcase/rounds/:id/vote`) hands the visitor a++random opaque id in a signed, httpOnly cookie (`bench_anon`). The id says nothing about the visitor, and the cookie++is signed with a server secret, so a client cannot mint a new voter id at will. Bench never stores the address: the++per-address cap counts a hash of it, salted per day, so two days of addresses cannot be linked.++++**Relayed by a first-party service.** A service that may relay sends its own service token (with++`bench.judgement.cast`) and the header `X-OV-Voter: anon:<opaque id it derived from its own cookie>`. Only a service++on the allow-list may relay: `BENCH_ANON_RELAY_SERVICES` (default `design`). The relay's own daily total is capped++apart from the per-voter cap.++++| Route | Who | What |++|---|---|---|++| `GET /api/v1/judgements/next?kind=` | a person, an anonymous voter (cookie), or a relaying service (`X-OV-Voter`) | The next ready round of that kind the caller has not judged and did not start, blind. A signed-in person keeps the old answer (`200` with `pair: null` when there is nothing); an anonymous voter gets `204` when there is nothing to judge. |++| `POST /api/v1/rounds/:id/judgements` | a person, a service with `bench.judgement.cast`, or a relaying service with `X-OV-Voter` | The vote. A relayed vote is a `human_pairwise` vote of the anonymous voter the header names. |++| `POST /api/v1/showcase/rounds/:id/vote` | a person, or an anonymous voter (the cookie) | The vote from Bench's own page. |++| `GET /api/v1/boards/:kindId?voters=people\|anon` | anyone | The board of the signed-in votes alone, or of the anonymous votes alone. Every board says how many votes of each kind of voter it holds (`voters`). |++++What is stored is never the opaque id: the judgement carries `voter_kind` `anon` and the hash of the id, in the same++column a person's id would use. One vote per voter per round, and a creator never judges their own round, follow from++that. The open dataset never carries a voter id.++++Every cap answers the same way: `429` with the code `bench.limit_anon` and the detail "Sign in to keep voting today".++++| Cap | Environment variable | Default |++|---|---|---|++| One anonymous voter, per day | `BENCH_ANON_VOTER_PER_DAY` | 30 |++| One address (a salted hash of it, salted per day), per day | `BENCH_ANON_IP_PER_DAY` | 60 |++| One relaying service, per day | `BENCH_ANON_RELAY_PER_DAY` | 3000 |++| Every anonymous vote on the site, per day | `BENCH_ANON_GLOBAL_PER_DAY` | 20000 |++| The services that may relay | `BENCH_ANON_RELAY_SERVICES` | `design` |++| Signed-out voting on or off | `BENCH_ANON_ENABLED` | on |++| The secret the cookie is signed with | `BENCH_ANON_SECRET` | the OAuth client secret |++++## The daily rounds++++Bench makes one round per design kind per day, at one fixed UTC hour, on its own budget. It is off until the owner++turns it on (`BENCH_DAILY_ROUNDS`), and it never starts under the drill. The prompts come from a curated list of++about twenty per kind (`server/bench/daily-prompts.js`), and the two models of every round come from the pool the++ordinary rounds use. A kind whose share of the day's budget is gone is skipped, and what the day spent is written++down in `bench_daily_rounds`.++++| Setting | Environment variable | Default |++|---|---|---|++| The job on or off | `BENCH_DAILY_ROUNDS` | off |++| The job under the drill (the tests) | `BENCH_DAILY_ROUNDS_IN_DRILL` | off |++| The UTC hour of the day | `BENCH_DAILY_ROUNDS_HOUR_UTC` | 8 |++| What the day's rounds may cost | `BENCH_DAILY_BUDGET_USD` | 1.00 |++| What one round is assumed to cost | `BENCH_DAILY_ROUND_ESTIMATE_USD` | 0.05 |++## The older APIBench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the privatennew file mode 100644@@@ -0,0 +1,47 @@++-- phase: expand++-- Plan L51 step 1 (signed-out voting) and step 2 (the daily rounds job).++--++-- 1. A judgement now says who voted: a person, a first-party service, or an anonymous voter. The anonymous++-- voter is a hash, never an id: the raw opaque id never reaches the database.++ALTER TABLE bench_judgements ADD COLUMN voter_kind text COLLATE "C";++UPDATE bench_judgements++ SET voter_kind = CASE WHEN subject IS NULL THEN 'service'++ WHEN left(subject, 4) = 'usr_' THEN 'person'++ ELSE 'service' END;++ALTER TABLE bench_judgements ALTER COLUMN voter_kind SET DEFAULT 'person';++ALTER TABLE bench_judgements ADD CONSTRAINT bench_judgements_voter_kind++ CHECK (voter_kind IS NULL OR voter_kind IN ('person', 'anon', 'service'));++CREATE INDEX bench_judgements_by_voter_kind ON bench_judgements (kind_id, voter_kind, at);++++-- 2. The daily counters of the anonymous vote: one row per scope, key and UTC day.++-- scope is 'voter' (the hashed opaque id), 'ip' (the salted hash of the address, salted per day),++-- 'relay' (the relaying service) or 'global'. The address itself is never written anywhere.++CREATE TABLE bench_anon_counters (++ scope text COLLATE "C" NOT NULL, -- voter | ip | relay | global++ key text COLLATE "C" NOT NULL, -- the hashed voter id, the hashed address, the service, or 'all'++ day text COLLATE "C" NOT NULL, -- the UTC day, 'YYYY-MM-DD'++ count integer NOT NULL DEFAULT 0,++ PRIMARY KEY (scope, key, day),++ CONSTRAINT bench_anon_counters_count CHECK (count >= 0)++);++CREATE INDEX bench_anon_counters_day ON bench_anon_counters (day);++++-- 3. The daily rounds job: what it planned for every kind, what it skipped and what it spent.++CREATE TABLE bench_daily_rounds (++ id text COLLATE "C" PRIMARY KEY, -- bdr_ + ULID++ day text COLLATE "C" NOT NULL, -- the UTC day, 'YYYY-MM-DD'++ kind_id text COLLATE "C" NOT NULL,++ prompt text,++ models jsonb,++ task_id text,++ round_id text,++ estimated_usd numeric(12, 6) NOT NULL DEFAULT 0,++ spent_usd numeric(12, 6),++ status text COLLATE "C" NOT NULL, -- planned | created | skipped | failed++ skip_reason text,++ created_at timestamptz NOT NULL DEFAULT now(),++ CONSTRAINT bench_daily_rounds_id CHECK (id ~ '^bdr_[0-9A-HJKMNP-TV-Z]{26}$'),++ CONSTRAINT bench_daily_rounds_status CHECK (status IN ('planned', 'created', 'skipped', 'failed'))++);++CREATE UNIQUE INDEX bench_daily_rounds_day_kind ON bench_daily_rounds (day, kind_id);++CREATE INDEX bench_daily_rounds_day ON bench_daily_rounds (day);@@@ -35,6 +35,8 @@ const { createEmbedRoutes } = require('./http/embed');const { createWatchPages } = require('./http/watch');const { createWatches } = require('./watch/watch');const { createBattles } = require('./bench/battles');++const { createAnon } = require('./bench/anon');++const { createDailyRounds } = require('./bench/daily');const { createBattlePages } = require('./http/battle');const { createRatingsPages } = require('./http/ratings');const { createRenderRoutes } = require('./http/render');@@@ -80,6 +82,9 @@ async function createApp(opts = {}) {const mirror = require('./bench/mirror').createMirror({ s, registry, fetchImpl, log });const ai = opts.benchAi || createBenchAi({ config, fetchImpl, log });const jobs = createBenchJobs({ s, bench, ai, config, log });++ // The signed-out voter (the cookie, the relay header and the daily caps) and the daily rounds job.++ const anon = createAnon({ s, config, log });++ const daily = createDailyRounds({ s, bench, ai, config, anon, log });const publisher = createPublisher({ config, s, fetchImpl, log });await bench.seedKinds();@@@ -94,6 +99,7 @@ async function createApp(opts = {}) {const launcher = createLauncher({ config, registry, fetchImpl: fetchImpl || globalThis.fetch, log });const ctx = {config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,++ anon, daily,imports, mirror,random: opts.random || Math.random,};nnew file mode 100644@@@ -0,0 +1,228 @@++'use strict';++++/**++ * Anonymous voting (plan L51, step 1): a person who has not signed in may still judge a pairwise round.++ *++ * There are two ways in, and both end at the same place:++ *++ * 1. Bench's own vote page. The visitor gets a random opaque id in a signed, httpOnly cookie. The id says++ * nothing about the person, and the cookie is signed with a server secret so a client cannot mint a new++ * voter id at will. Bench never stores the address: the per-address cap counts a salted hash of it, and++ * the salt is derived from the day, so yesterday's hash says nothing about today's.++ * 2. A first-party service (OpenVibe.Design) that relays the vote. It sends its service token and the++ * header X-OV-Voter: anon:<opaque id it derived from its own cookie>. Only a service on the allow-list++ * (BENCH_ANON_RELAY_SERVICES, 'design' by default) may relay, and the relay's own daily total is capped.++ *++ * What is stored is never the opaque id: the judgement carries voter_kind 'anon' and the hash of the id, in++ * the same subject column a person's id would use. One vote per voter per round falls out of that for free.++ *++ * Every cap answers the same way: 429 bench.limit_anon, "Sign in to keep voting today".++ */++++const crypto = require('crypto');++const { BenchError } = require('../store');++++const HEADER = 'x-ov-voter';++const HEADER_PREFIX = 'anon:';++const RELAY_CAP = 'bench.judgement.cast';++const LIMIT_DETAIL = 'Sign in to keep voting today';++const SUBJECT_PREFIX = 'anon_';++const KEEP_DAYS = 7;++++const num = (v, d = 0) => (v == null || Number.isNaN(Number(v)) ? d : Number(v));++const b64u = (buf) => Buffer.from(buf).toString('base64url');++const hmac = (secret, msg) => crypto.createHmac('sha256', secret).update(String(msg), 'utf8').digest();++const sha256hex = (s) => crypto.createHash('sha256').update(String(s), 'utf8').digest('hex');++++function createAnon({ s, config, log = console }) {++ const db = s.db;++ const cfg = () => (config && config.bench && config.bench.anon) || {};++ let secretCache = null;++++ /** The secret the cookie is signed with and the voter id is hashed with. */++ function secret() {++ if (secretCache) return secretCache;++ const c = cfg();++ const own = String(c.secret || '').trim();++ const oauth = String((config && config.oauth && config.oauth.clientSecret) || '').trim();++ secretCache = own || oauth || crypto.randomBytes(32).toString('hex');++ if (!own && !oauth && log && typeof log.warn === 'function') {++ log.warn('[OpenVibe.Bench] neither BENCH_ANON_SECRET nor an OAuth client secret is set: the anonymous voter cookie is signed with a per-process secret, so a restart resets the anonymous caps.');++ }++ return secretCache;++ }++++ const cookieName = () => String(cfg().cookieName || 'bench_anon');++ const enabled = () => cfg().enabled !== false;++++ /** The UTC day, 'YYYY-MM-DD'. */++ function dayOf(now) {++ return new Date(now == null ? s.now() : now).toISOString().slice(0, 10);++ }++++ /** A new opaque voter id: random bytes, nothing about the person. */++ function newOpaqueId() {++ return b64u(crypto.randomBytes(18));++ }++++ /** The cookie value: the id and its signature. */++ function cookieValue(opaque) {++ return `${opaque}.${hmac(secret(), 'cookie|' + opaque).toString('base64url')}`;++ }++++ /** The id inside a signed cookie, or null when the cookie is not ours or is broken. */++ function opaqueFromCookie(raw) {++ const value = String(raw || '');++ const dot = value.indexOf('.');++ if (dot < 1) return null;++ const id = value.slice(0, dot);++ const sig = value.slice(dot + 1);++ if (!/^[A-Za-z0-9_-]{16,64}$/.test(id)) return null;++ const want = hmac(secret(), 'cookie|' + id).toString('base64url');++ const a = Buffer.from(String(sig));++ const b = Buffer.from(want);++ if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) return null;++ return id;++ }++++ /** The hash of the opaque id: what the database holds, never the id itself. */++ function voterHash(scope, opaque) {++ return SUBJECT_PREFIX + hmac(secret(), `${scope}|${opaque}`).toString('hex');++ }++++ /** The address hash: salted per day, so the rows of two days cannot be linked. */++ function ipHash(ip, day) {++ const salt = hmac(secret(), 'ip-salt|' + day).toString('hex');++ return sha256hex(`bench-anon-ip|${day}|${salt}|${ip}`);++ }++++ /** The address of the request, as the proxy chain reports it. */++ function ipOf(req) {++ const ip = String((req && req.ip) || (req && req.socket && req.socket.remoteAddress) || '').trim();++ return ip || null;++ }++++ /** The name of a service principal: 'service:design' is 'design'. */++ function serviceName(requester) {++ const r = String(requester || '').trim().toLowerCase();++ return r.startsWith('service:') ? r.slice('service:'.length) : r;++ }++++ /** Whether a service may relay an anonymous voter. */++ function relayAllowed(service) {++ const list = cfg().relayServices || [];++ const name = serviceName(service);++ return list.includes(name) || list.includes('service:' + name);++ }++++ /**++ * Who the anonymous voter is, if the request names one:++ * - a relaying service and its X-OV-Voter header;++ * - the signed cookie Bench set earlier;++ * - a fresh cookie, when the caller is nobody in particular (mint).++ * Returns null when the request carries no anonymous voter and none should be made.++ */++ function resolve(req, principal, { mint = true } = {}) {++ const p = principal || { kind: 'anonymous' };++ const header = String((req && req.headers && req.headers[HEADER]) || '').trim();++ if (header) {++ if (p.kind !== 'app') throw new BenchError(403, 'bench.relay_not_allowed', 'Only a first-party service may relay an anonymous voter.');++ const service = serviceName(p.requester);++ if (!relayAllowed(service)) throw new BenchError(403, 'bench.relay_not_allowed', 'This service may not relay anonymous votes.');++ if (typeof p.hasCapability === 'function' && !p.hasCapability(RELAY_CAP)) {++ throw new BenchError(403, 'bench.capability_denied', `A service token needs ${RELAY_CAP}.`);++ }++ if (!header.startsWith(HEADER_PREFIX)) throw new BenchError(400, 'bench.anon_voter_invalid', 'The X-OV-Voter header must be anon:<opaque id>.');++ const opaque = header.slice(HEADER_PREFIX.length).trim();++ if (!/^[A-Za-z0-9._~:-]{6,200}$/.test(opaque)) throw new BenchError(400, 'bench.anon_voter_invalid', 'The X-OV-Voter header must be anon:<opaque id>.');++ return {++ kind: 'anon', source: 'relay', opaque,++ subject: voterHash('relay:' + service, opaque),++ relay: service, creator: 'service:' + service, ip: null, minted: false,++ };++ }++ const cookies = (req && req.cookies) || {};++ const raw = cookies[cookieName()];++ const id = raw ? opaqueFromCookie(raw) : null;++ if (id) {++ return {++ kind: 'anon', source: 'cookie', opaque: id,++ subject: voterHash('cookie', id),++ relay: null, creator: null, ip: ipOf(req), minted: false,++ };++ }++ if (!mint) return null;++ if (p.kind && p.kind !== 'anonymous') return null;++ const fresh = newOpaqueId();++ return {++ kind: 'anon', source: 'cookie', opaque: fresh,++ subject: voterHash('cookie', fresh),++ relay: null, creator: null, ip: ipOf(req), minted: true,++ };++ }++++ /** Give the browser its voter cookie. */++ function setCookie(res, opaque) {++ if (!res || typeof res.cookie !== 'function') return;++ const days = Math.max(1, num(cfg().cookieDays, 180));++ res.cookie(cookieName(), cookieValue(opaque), {++ httpOnly: true,++ sameSite: 'lax',++ secure: Boolean(config && config.cookies && config.cookies.secure),++ maxAge: days * 86_400_000,++ path: '/',++ });++ }++++ /**++ * The daily counters this vote spends: the voter, the address (only for Bench's own cookie), the relaying++ * service and the site. The counters move inside the vote's own transaction, so a refused vote spends++ * nothing and a vote that is rolled back spends nothing either.++ */++ async function takeQuota(identity) {++ const c = cfg();++ const day = dayOf();++ const scopes = [{ scope: 'voter', key: identity.subject, cap: num(c.voterPerDay, 30) }];++ if (identity.source === 'cookie' && identity.ip) {++ scopes.push({ scope: 'ip', key: ipHash(identity.ip, day), cap: num(c.ipPerDay, 60) });++ }++ if (identity.relay) scopes.push({ scope: 'relay', key: identity.relay, cap: num(c.relayPerDay, 3000) });++ scopes.push({ scope: 'global', key: 'all', cap: num(c.globalPerDay, 20000) });++ for (const one of scopes) {++ const row = await db.maybe(++ `INSERT INTO bench_anon_counters (scope, key, day, count) VALUES ($1, $2, $3, 1)++ ON CONFLICT (scope, key, day) DO UPDATE SET count = bench_anon_counters.count + 1++ RETURNING count`,++ [one.scope, one.key, day],++ );++ if (num(row && row.count) > one.cap) throw new BenchError(429, 'bench.limit_anon', LIMIT_DETAIL);++ }++ return { day, scopes: scopes.map((one) => one.scope) };++ }++++ /** What the voter may still spend today, for the pages that want to say so. */++ async function remaining(identity) {++ const c = cfg();++ const day = dayOf();++ const used = num(await db.value(++ 'SELECT COUNT(*)::int AS n FROM bench_anon_counters WHERE scope = $1 AND key = $2 AND day = $3',++ ['voter', identity.subject, day],++ ));++ return { used, limit: num(c.voterPerDay, 30), left: Math.max(0, num(c.voterPerDay, 30) - used) };++ }++++ /** Old counters are dropped: the address hash of yesterday is worth nothing today. */++ async function prune(days = KEEP_DAYS) {++ const cutoff = new Date(s.now() - Math.max(1, days) * 86_400_000).toISOString().slice(0, 10);++ await db.exec('DELETE FROM bench_anon_counters WHERE day < $1', [cutoff]);++ }++++ return {++ HEADER, HEADER_PREFIX, RELAY_CAP, LIMIT_DETAIL,++ resolve, setCookie, takeQuota, remaining, prune,++ cookieName, enabled, dayOf, newOpaqueId, cookieValue, opaqueFromCookie, voterHash, ipHash, ipOf,++ serviceName, relayAllowed, secret,++ };++}++++module.exports = { createAnon, HEADER, HEADER_PREFIX, RELAY_CAP, LIMIT_DETAIL };nnew file mode 100644@@@ -0,0 +1,213 @@++'use strict';++++/**++ * The prompts of the daily rounds (plan L51, step 2): about twenty per design kind, so every day has something++ * worth judging and no two days ask the same thing. They are concrete briefs for one artefact of the kind, written++ * for a model that has never seen the site: no real people, no likenesses, no names of anyone who exists.++ */++++const DAILY_PROMPTS = Object.freeze({++ 'design.page': Object.freeze([++ 'A landing page for a fictional neighbourhood tool library: what it lends, a lending policy table, and opening hours.',++ 'A one-page guide to a fictional tide-pool reserve: three tide zones explained, a safety list and a printable checklist.',++ 'A landing page for a fictional language-learning app: what it teaches, a seven-day trial table and a pricing section.',++ 'A page for a fictional second-hand bicycle workshop: the services it offers, prices in a table and a booking form.',++ 'A single page for a fictional community seed library: how borrowing works and a month-by-month seed-saving calendar.',++ 'A product page for a fictional desk lamp: a specification table, an energy label and three rooms it fits in.',++ 'An announcement page for a fictional museum winter exhibition: dates, a curator note and a plan for the visit.',++ 'A one-page site for a fictional freelance bookkeeper: the services, the rates and a contact form.',++ 'A page for a fictional rooftop beekeeping co-op: where the hives stand, when honey is ready and how to join.',++ 'A page for a fictional ceramics studio: the class schedule as a table, the materials and the studio rules.',++ 'A landing page for a fictional city air-quality dashboard: what the scale means, where the data comes from and how to read it.',++ 'A page for a fictional mountain hut: a booking form, a hut-to-hut map and a safety checklist for the walk in.',++ 'A page for a fictional vinyl subscription: three tiers side by side and a preview of the listening notes.',++ 'A brochure page for a fictional makerspace: the tools, what each is for and how the induction works.',++ 'A page for a fictional river kayak club: the season calendar, the safety briefing and the membership form.',++ 'A landing page for a fictional tool that turns meeting notes into tasks: three steps, a sample and a sign-up.',++ 'A page for a fictional school science fair: the schedule, the judging criteria and the registration form.',++ 'A guide to growing herbs on a balcony: a table of light and water needs, and a shopping list.',++ 'A page for a fictional archive of letters between two invented towns: a timeline and a description of the exhibit room.',++ 'A page for a fictional repair cafe: the schedule, what can be fixed and what to bring.',++ ]),++ 'design.component': Object.freeze([++ 'A modal dialog that asks a person to confirm a destructive action: a clear question, cancel and confirm, and the focus trapped inside.',++ 'An accessible dropdown menu with a search field, grouped items and full keyboard support.',++ 'A price card that switches between monthly and yearly billing and shows what the year costs.',++ 'A file-upload drop zone with a progress bar, an error state and a retry button.',++ 'A four-step sign-up stepper showing which steps are done, which one is current and which are still to come.',++ 'An address field with suggestions, a loading state and a state for "nothing found".',++ 'A rating widget with half stars, a numeric readout and a hover preview.',++ 'A toast component in three variants: success, warning and error.',++ 'A date range picker with presets and a past range that cannot be picked.',++ 'An expandable question and answer item with a chevron, a smooth animation and the right ARIA state.',++ 'A tag input with suggestions, keyboard removal of a tag and a warning when there are too many.',++ 'A comparison table that turns into cards on a narrow screen.',++ 'A consent banner with accept, refuse and a "learn more" link.',++ 'A skeleton loading card with exactly the shape of the card it stands in for.',++ 'A breadcrumb trail with a middle item that is shortened and an overflow menu.',++ 'A segmented control that switches views without reloading the page.',++ 'An empty state with a slot for an illustration, a short explanation and one clear action.',++ 'A media uploader with thumbnail previews, a progress bar and a remove button on each item.',++ 'A form field with inline validation, help text under the label and an error message that says what to do.',++ 'A pagination control with first, previous, next and last, and a choice of page size.',++ ]),++ 'design.svg': Object.freeze([++ 'A flat illustration of a bicycle leaning against a lamppost at dusk.',++ 'A line-art icon set of five kitchen tools, drawn in one consistent weight.',++ 'A poster-style illustration of a whale swimming through clouds.',++ 'A map-style diagram of a fictional park with four paths and a pond in the middle.',++ 'A vintage travel poster for a fictional coastal railway line.',++ 'Six weather glyphs drawn as one geometric set.',++ 'A cutaway diagram of a beehive with its parts named.',++ 'A minimal geometric emblem for a fictional makerspace.',++ 'An illustrated scene of a rainy bus stop with a shelter, a bench and a puddle.',++ 'A botanical study of a fern frond in layered greens.',++ 'A chess diagram of a puzzle position, shown from White to move.',++ 'A technical drawing of a folding chair with dimension lines.',++ 'A diagram of the water cycle with labelled arrows.',++ 'An ornamental frame with a silhouette shape inside it, drawn so that it could be anyone and no one.',++ 'A comic-style burst shape with the word SALE hand-lettered across it.',++ 'Three progress rings showing three different values.',++ 'An isometric illustration of a small workshop desk with tools on it.',++ 'A constellation chart with the stars connected and the Greek letters beside them.',++ 'A food icon set: a loaf, a bowl, a whisk, a jar and a board.',++ 'A stylized mountain landscape in layered flat shapes with a moon above it.',++ ]),++ 'design.motion': Object.freeze([++ 'A hero whose headline rises twenty pixels and fades in, then its sub-line, then the button, each after the other.',++ 'A grid of cards that come in one after another as the page scrolls, sixty milliseconds apart.',++ 'A button with a ripple that grows from the point that was clicked.',++ 'A loading mark that changes shape every nine hundred milliseconds, from one form to another.',++ 'An accordion that opens and closes with a smooth change of height.',++ 'A hero video that pauses when it scrolls out of sight and plays again when it comes back.',++ 'A tab underline that slides from one tab to the other.',++ 'A number that counts up when its section scrolls into view.',++ 'A toast that slides in from the right and fades away after four seconds.',++ 'A parallax illustration whose background moves at half the speed of the page.',++ 'A burst of confetti after a successful sign-up, made small for visitors who ask for less movement.',++ 'A soft glow that follows the pointer, easing behind it.',++ 'A shimmer that sweeps across placeholder blocks while they load.',++ 'A chart that draws its line from left to right as it appears.',++ 'A slow strip of logos that stops when the pointer is over it.',++ 'A sheet that can be dragged away, follows the finger and snaps back when released early.',++ 'A headline revealed character by character, as if it were being typed.',++ 'A modal that grows from a slightly smaller size as it fades in, over a backdrop that goes soft.',++ 'A thin progress bar across the top of the page showing how far down the reader is.',++ 'A card that turns in three dimensions when the pointer is over it.',++ ]),++ 'design.ui': Object.freeze([++ 'A banking dashboard: the balance, a small spending chart and the pending transactions.',++ 'A train ticket booking flow: the date, the seat map and the fare summary.',++ 'A hospital appointment screen: a short list of triage questions, a summary and a consent box.',++ 'A warehouse picking app: a list of shelves, a scan field and a progress bar.',++ 'A support inbox: the threads on the left, the message in the middle and a reply box below it.',++ 'A recipe app in cooking mode: one step at a time with a timer, in a layout that reads from across a counter.',++ 'A fleet monitoring console: a map, the list of vehicles and an alert panel.',++ 'A course dashboard: progress so far, the deadlines ahead and the grades.',++ 'A music player: what is playing, the queue and the playback controls.',++ 'A calendar week view with a booking panel and a row of filters.',++ 'A stock control screen: the stock table and a panel for reordering.',++ 'A school portal: the timetable, an attendance card and the notice board.',++ 'A point of sale screen: the products in a grid, the basket and the payment panel.',++ 'A fitness summary: three rings, a week of bars and the list of goals.',++ 'A library catalogue: the search with its filters, the results and the detail pane.',++ 'A checkout: the address form, the shipping choices and the price broken down.',++ 'A control room for a water plant: the gauges, the alarm list and a trend chart.',++ 'A food delivery tracker: where the order is, which courier is bringing it and the messages.',++ 'A travel itinerary: the days one after another, with the bookings and a map.',++ 'A video call: the people in a grid, the chat beside it and the controls below.',++ ]),++ 'design.dataviz': Object.freeze([++ 'A chart of monthly rainfall for three towns, with a legend and axes that can be read.',++ 'A stacked bar chart of energy use by sector over four years.',++ 'A map of renewable energy capacity across countries, with a tooltip and a filter.',++ 'A line chart of daily temperature with a band showing the usual range.',++ 'A scatter plot of reading time against book length, coloured by genre.',++ 'A heat map of bus delays by hour and by weekday.',++ 'A treemap of a city budget split by department.',++ 'A dashboard of four charts that share the same filters.',++ 'A population pyramid comparing two towns.',++ 'A flow diagram of where a city water goes, the width of each band carrying the amount.',++ 'A slope chart of five countries scores in two years, twenty years apart.',++ 'A calendar heat map of how many issues were closed each day.',++ 'A box plot comparing the delivery times of five couriers.',++ 'A choropleth of fictional districts by median income, in a palette that reads without colour.',++ 'A gauge of a service level target with the threshold marked.',++ 'A sankey diagram of energy flowing from where it is made to where it is used.',++ 'A grouped bar chart of survey answers across three age groups.',++ 'A beeswarm plot of delivery ratings, each dot nudged aside so none hides another.',++ 'A grid of twelve small charts, one for each region.',++ 'A single headline number with the trend beside it and the target it is measured against.',++ ]),++ 'design.model3d': Object.freeze([++ 'A low-poly wooden stool: four legs and a round seat.',++ 'A simple tree with a trunk and layered foliage.',++ 'A coffee mug with a handle, standing on a saucer.',++ 'A stack of three books, each a little different in size.',++ 'A classic desk lamp with an arm that bends.',++ 'A bicycle with thin wheels, a frame, handlebars and a saddle.',++ 'A plant pot in clay, with a rim and a saucer under it.',++ 'A small wind turbine: a tower and three blades.',++ 'A cardboard box with its flaps open.',++ 'A park bench with wooden slats and metal legs.',++ 'A teapot with a round body, a spout and a lid.',++ 'A paper aeroplane with folded wings.',++ 'A chess rook on a turned base.',++ 'A watering can with a long spout and a handle.',++ 'A low-poly mountain with snow on the top.',++ 'A suitcase with a handle, wheels and a strap around it.',++ 'A lighthouse with a striped tower and a lamp room.',++ 'A small house with a pitched roof, a door and two windows.',++ 'A camera on a tripod with a long lens.',++ 'A shipping container with ribbed sides and corner fittings.',++ ]),++ 'design.shader': Object.freeze([++ 'A starfield that drifts, with the nearer stars moving faster than the far ones.',++ 'A soft glow that follows the mouse.',++ 'Lava flowing, built from layers of noise moving at different speeds.',++ 'A water surface that ripples, bending what is drawn behind it.',++ 'A checkerboard that fades into a diagonal gradient.',++ 'Heat haze above a field of moving waves.',++ 'A field of colour built from overlapping sine waves.',++ 'Concentric rings pulsing out from the centre.',++ 'A shadow that stretches and softens as the light moves.',++ 'The wall of a cave, drawn procedurally from layered noise.',++ 'An image repeated around the centre in a kaleidoscope.',++ 'Contour lines drawn over a noise field.',++ 'An aurora, hanging in vertical curtains.',++ 'A texture that breaks apart into blocks and disappears.',++ 'A picture reduced to pixel art in a palette of eight colours.',++ 'Flat coloured cells with thin borders between them, like a stained glass window.',++ 'A curved screen of an old monitor, with scan lines across it.',++ 'A soft-edged blob that follows the cursor and keeps a little of its own momentum.',++ 'A contour map of a landscape, drawn from layered noise.',++ 'A field of sparks that twinkle at random.',++ ]),++ 'design.interface': Object.freeze([++ 'A ticket app system: buttons, badges and a status chip, all from one set.',++ 'The form patterns of a checkout: fields, their validation states and a summary block.',++ 'A navigation system: a top bar, a side rail and a breadcrumb, sharing one spacing scale.',++ 'A catalogue card: picture, title, details, the actions and the empty version of it.',++ 'A data table kit: sorting, filtering, pagination and the skeleton it shows while loading.',++ 'A notification centre: a toast, a banner and an in-page alert, in one palette.',++ 'A modal and a drawer for a settings area, sharing the same header and footer.',++ 'A stepper and a set of progress marks for an onboarding flow.',++ 'A profile card, a set of avatars and an identity block for a community app.',++ 'A filter panel for a search page: chips, sliders and a way to reset everything.',++ 'An icon set for a productivity app, with a naming convention written down.',++ 'A pricing table with three tiers, the features side by side and one tier picked out.',++ 'A dashboard kit: a number tile, a small chart and a list of statuses.',++ 'A media kit: a picture, a video and a gallery, each with a caption.',++ 'An onboarding kit: welcome screens, a short tour and an empty state.',++ 'A chart set for an admin panel: axes, a legend and a tooltip.',++ 'A feedback kit: a thumbs up and down, a comment box and a prompt to rate.',++ 'A settings kit: a toggle, a choice, a slider and a place for the dangerous actions.',++ 'A chat kit: message bubbles, an input row and a typing indicator.',++ 'A landing page kit: the hero, a grid of features, the pricing, the questions and the footer.',++ ]),++});++++/** The kinds the daily job makes a round for: the nine design kinds. */++const DAILY_KIND_IDS = Object.freeze(Object.keys(DAILY_PROMPTS));++++module.exports = { DAILY_PROMPTS, DAILY_KIND_IDS };nnew file mode 100644@@@ -0,0 +1,229 @@++'use strict';++++/**++ * The daily rounds (plan L51, step 2): one round per design kind per day, so there is always something to judge.++ *++ * The job is off until the owner turns it on (BENCH_DAILY_ROUNDS), it runs at one fixed UTC hour++ * (BENCH_DAILY_ROUNDS_HOUR_UTC) and it spends Bench's own budget (BENCH_DAILY_BUDGET_USD, one US dollar a day by++ * default). A kind whose share of the budget is gone is skipped, and what it spent is written down, so the owner can++ * see what the day's rounds cost. The two models of every round come from the pool the ordinary rounds already use.++ *++ * Nothing here starts under BENCH_DRILL (the drill and the tests), unless the drill explicitly allows it++ * (BENCH_DAILY_ROUNDS_IN_DRILL). The tests call runDay() with a clock of their own instead.++ */++++const crypto = require('crypto');++const { DAILY_KIND_IDS, DAILY_PROMPTS } = require('./daily-prompts');++++/** The owner of the daily rounds. It is a service, not a person: no person can judge or create them. */++const SUBJECT = 'service:bench-daily';++++const num = (v, d = 0) => (v == null || Number.isNaN(Number(v)) ? d : Number(v));++++function createDailyRounds({ s, bench, ai, config, anon = null, log = console }) {++ const db = s.db;++ let timer = null;++ let stopped = false;++ let running = false;++ let prunedDay = null;++++ const cfg = () => (config && config.bench && config.bench.daily) || {};++ const dayOf = (now) => new Date(now == null ? s.now() : now).toISOString().slice(0, 10);++++ /** The prompt of the day for a kind: the same one all day, a different one tomorrow. */++ function promptFor(kindId, day) {++ const list = DAILY_PROMPTS[kindId] || [];++ if (!list.length) return null;++ const h = crypto.createHash('sha256').update(`${day}|${kindId}`).digest();++ return list[h.readUInt32BE(0) % list.length];++ }++++ /** The two models of the pool the ordinary rounds use. */++ async function modelsOfPool() {++ let list = [];++ try {++ if (typeof ai.slotPlan === 'function') {++ const plan = await ai.slotPlan();++ list = plan && Array.isArray(plan.slots) ? plan.slots : [];++ }++ } catch (err) {++ log.warn('[OpenVibe.Bench] daily rounds: the model pool is unavailable:', (err && err.message) || err);++ }++ if (list.length < 2) {++ try { list = await ai.offeredModels(); } catch { list = []; }++ }++ const out = [];++ const seen = new Set();++ for (const one of list) {++ const provider = String((one && one.provider) || '').trim();++ const model = one && one.model != null ? String(one.model) : null;++ if (!provider) continue;++ const key = provider + '|' + (model || '');++ if (seen.has(key)) continue;++ seen.add(key);++ out.push({ provider, model });++ if (out.length === 2) break;++ }++ return out;++ }++++ /** What one round is assumed to cost: the configured estimate, or what these models really cost. */++ async function estimateFor(models) {++ const base = num(cfg().estimateUsd, 0.05);++ const names = models.map((m) => m.model).filter(Boolean);++ if (!names.length) return base;++ const avg = num(await db.value(++ `SELECT COALESCE(AVG(cost_usd), 0)::numeric AS n FROM bench_entries++ WHERE model = ANY($1::text[]) AND created_at >= $2::timestamptz`,++ [names, new Date(s.now() - 7 * 86_400_000).toISOString()],++ ));++ return Math.max(base, Math.round(avg * models.length * 1e6) / 1e6);++ }++++ /** What the day has cost: what was really spent, plus what is still running. */++ async function budgetState(day) {++ const rows = await db.many('SELECT * FROM bench_daily_rounds WHERE day = $1', [day]);++ let spent = 0;++ let reserved = 0;++ for (const row of rows) {++ if (String(row.status) === 'skipped' || String(row.status) === 'failed') continue;++ if (row.spent_usd == null) reserved += num(row.estimated_usd);++ else spent += num(row.spent_usd);++ }++ return { spent, reserved, rows };++ }++++ /** What the day's rounds really cost, once their entries are in. */++ async function settle(day) {++ const rows = await db.many(++ "SELECT * FROM bench_daily_rounds WHERE day = $1 AND status = 'created' AND spent_usd IS NULL",++ [day],++ );++ for (const row of rows) {++ if (!row.round_id) continue;++ const usable = num(await db.value(++ 'SELECT COUNT(*)::int AS n FROM bench_entries WHERE round_id = $1 AND error IS NULL', [row.round_id]));++ if (usable < 2) continue;++ const spent = num(await db.value(++ 'SELECT COALESCE(SUM(cost_usd), 0)::numeric AS n FROM bench_entries WHERE round_id = $1', [row.round_id]));++ await db.exec('UPDATE bench_daily_rounds SET spent_usd = $1 WHERE id = $2', [spent, row.id]);++ }++ }++++ async function record({ day, kindId, prompt, models, estimated, status, reason = null, taskId = null, roundId = null }) {++ const id = s.newId('bdr');++ await db.exec(++ `INSERT INTO bench_daily_rounds (id, day, kind_id, prompt, models, task_id, round_id, estimated_usd, spent_usd, status, skip_reason, created_at)++ VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10, $11, $12::timestamptz)++ ON CONFLICT (day, kind_id) DO NOTHING`,++ [id, day, kindId, prompt || null, JSON.stringify(models || []), taskId, roundId, estimated,++ status === 'skipped' ? 0 : null, status, reason ? String(reason).slice(0, 200) : null, s.iso()],++ );++ return id;++ }++++ /**++ * One day of rounds: one per design kind, unless the day's budget is spent. A kind that already has a row for++ * today is left alone, so the job can be run again without making a second round of the same kind.++ */++ async function runDay({ now = s.now() } = {}) {++ const day = dayOf(now);++ const budget = num(cfg().budgetUsd, 1);++ await settle(day);++ const made = [];++ for (const kindId of DAILY_KIND_IDS) {++ const done = await db.maybe('SELECT id FROM bench_daily_rounds WHERE day = $1 AND kind_id = $2', [day, kindId]);++ if (done) continue;++ const prompt = promptFor(kindId, day);++ if (!prompt) continue;++ const models = await modelsOfPool();++ if (models.length < 2) {++ await record({ day, kindId, prompt, models, estimated: 0, status: 'skipped', reason: 'no_models' });++ made.push({ kind_id: kindId, status: 'skipped', reason: 'no_models' });++ continue;++ }++ const estimate = await estimateFor(models);++ const state = await budgetState(day);++ if (state.spent + state.reserved + estimate > budget) {++ await record({ day, kindId, prompt, models, estimated: estimate, status: 'skipped', reason: 'budget' });++ made.push({ kind_id: kindId, status: 'skipped', reason: 'budget' });++ continue;++ }++ try {++ const task = await bench.createTask({ kindId, prompt, taskClass: 'daily', public: true, subject: SUBJECT });++ if (!task.ok) throw new Error(task.message || 'the task was refused');++ const round = await bench.createRound({ taskId: task.task.id, offers: models, subject: SUBJECT, bypassQuota: true });++ if (!round.ok) throw new Error(round.message || 'the round was refused');++ await record({++ day, kindId, prompt, models, estimated: estimate, status: 'created',++ taskId: task.task.id, roundId: round.round.id,++ });++ made.push({ kind_id: kindId, status: 'created', round_id: round.round.id, estimated_usd: estimate });++ } catch (err) {++ log.warn('[OpenVibe.Bench] daily round failed:', (err && err.message) || err);++ await record({++ day, kindId, prompt, models, estimated: estimate, status: 'failed',++ reason: String((err && err.message) || err),++ });++ made.push({ kind_id: kindId, status: 'failed' });++ }++ }++ const state = await budgetState(day);++ const summary = {++ day, budget_usd: budget, spent_usd: state.spent, reserved_usd: state.reserved,++ created: made.filter((m) => m.status === 'created').length,++ skipped: made.filter((m) => m.status === 'skipped').length,++ rounds: made,++ };++ log.info(`[OpenVibe.Bench] daily rounds ${day}: ${summary.created} created, ${summary.skipped} skipped, $${state.spent.toFixed(4)} spent of $${budget.toFixed(2)}.`);++ return summary;++ }++++ /** One pass of the job: after the day's hour, and only if the day has no rounds yet. */++ async function tick() {++ if (running || stopped) return null;++ running = true;++ try {++ const now = s.now();++ const day = dayOf(now);++ const hour = new Date(now).getUTCHours();++ // The daily counters of the anonymous vote are dropped once a day: the hash of yesterday's address++ // is worth nothing today.++ if (anon && prunedDay !== day) {++ prunedDay = day;++ try { await anon.prune(); } catch (err) { log.warn('[OpenVibe.Bench] daily rounds: the old counters stayed:', (err && err.message) || err); }++ }++ if (hour < num(cfg().hourUtc, 8)) return null;++ const rows = num(await db.value('SELECT COUNT(*)::int AS n FROM bench_daily_rounds WHERE day = $1', [day]));++ if (rows >= DAILY_KIND_IDS.length) {++ await settle(day);++ return null;++ }++ return await runDay({ now });++ } finally {++ running = false;++ }++ }++++ /** The job starts only when the owner turns it on, and never under the drill unless the drill allows it. */++ function start() {++ if (timer || stopped) return null;++ if (!cfg().enabled) return null;++ if (config && config.drill && !cfg().enabledInDrill) return null;++ const run = () => tick().catch((err) => log.warn('[OpenVibe.Bench] daily rounds:', (err && err.message) || err));++ timer = setInterval(run, num(cfg().pollMs, 300_000));++ if (timer.unref) timer.unref();++ run();++ return timer;++ }++++ function stop() {++ stopped = true;++ if (timer) clearInterval(timer);++ timer = null;++ }++++ return { start, stop, tick, runDay, settle, promptFor, modelsOfPool, budgetState, dayOf, SUBJECT };++}++++module.exports = { createDailyRounds, SUBJECT };@@@ -9,6 +9,8 @@require('dotenv').config();const trim = (s) => String(s || '').replace(/\/+$/, '');const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);++const float = (v, def) => (Number.isFinite(parseFloat(v)) ? parseFloat(v) : def);++const bool = (v, def) => (v == null || v === '' ? def : (v === '1' || v === 'true' || v === 'yes' || v === 'on'));function load(env = process.env) {const nodeEnv = env.NODE_ENV || 'development';@@@ -89,6 +91,42 @@ function load(env = process.env) {battlePersonPerDay: Math.max(1, int(env.BENCH_BATTLE_PERSON_PER_DAY, 200)),battleNetworkPerHour: Math.max(1, int(env.BENCH_BATTLE_NETWORK_PER_HOUR, 120)),battleNetworkPerDay: Math.max(1, int(env.BENCH_BATTLE_NETWORK_PER_DAY, 1000)),++++ // Signed-out voting (plan L51 step 1). The caps are per UTC day and every one of them answers++ // 429 bench.limit_anon with the same plain detail: sign in to keep voting today.++ anon: {++ enabled: bool(env.BENCH_ANON_ENABLED, true),++ // Per anonymous voter per day.++ voterPerDay: Math.max(1, int(env.BENCH_ANON_VOTER_PER_DAY, 30)),++ // Per address per day. Bench keeps only a salted hash of the address, salted per day.++ ipPerDay: Math.max(1, int(env.BENCH_ANON_IP_PER_DAY, 60)),++ // Per relaying service per day.++ relayPerDay: Math.max(1, int(env.BENCH_ANON_RELAY_PER_DAY, 3000)),++ // Every anonymous vote on the site, per day.++ globalPerDay: Math.max(1, int(env.BENCH_ANON_GLOBAL_PER_DAY, 20000)),++ // The first-party services that may relay an anonymous voter (X-OV-Voter).++ relayServices: String(env.BENCH_ANON_RELAY_SERVICES || 'design').split(',').map((x) => x.trim().toLowerCase()).filter(Boolean),++ // The secret the voter cookie is signed with and the voter id is hashed with. Empty falls back to++ // the OAuth client secret, and then to a per-process secret (which resets the caps on a restart).++ secret: String(env.BENCH_ANON_SECRET || '').trim(),++ cookieName: String(env.BENCH_ANON_COOKIE || 'bench_anon').trim() || 'bench_anon',++ cookieDays: Math.max(1, int(env.BENCH_ANON_COOKIE_DAYS, 180)),++ },++ // The daily rounds (plan L51 step 2): one round per design kind per day, so there is always++ // something to judge. Off until the owner turns it on in production.++ daily: {++ enabled: bool(env.BENCH_DAILY_ROUNDS, false),++ // The job never starts under BENCH_DRILL unless this is set too (the tests use it).++ enabledInDrill: bool(env.BENCH_DAILY_ROUNDS_IN_DRILL, false),++ // The one UTC hour the day's rounds are made.++ hourUtc: Math.min(23, Math.max(0, int(env.BENCH_DAILY_ROUNDS_HOUR_UTC, 8))),++ // What the day's rounds may cost, in US dollars.++ budgetUsd: Math.max(0, float(env.BENCH_DAILY_BUDGET_USD, 1.00)),++ // What one round is assumed to cost, until the real cost of the models is known.++ estimateUsd: Math.max(0, float(env.BENCH_DAILY_ROUND_ESTIMATE_USD, 0.05)),++ // How often the job looks at the clock.++ pollMs: Math.max(30_000, int(env.BENCH_DAILY_POLL_MS, 300_000)),++ },},};}@@@ -12,7 +12,7 @@* POST /rounds/:id/entries (a room submits outside entries with a service token)* GET /judgements/next* POST /rounds/:id/judgements-- * GET /boards/:kindId?type=elo|pareto|cost++ * GET /boards/:kindId?type=elo|pareto|cost&voters=people|anon* GET /models/:model* GET /datasets/:kindId.jsonl* POST /evals@@@ -162,11 +162,39 @@ function createApi(ctx) {});// ── Judgements ──────────────────────────────────────────-- r.get('/judgements/next', requirePerson, async (req, res) => {++ /**++ * The next pair to judge. A signed-in person gets the next round they have not judged and did not start;++ * an anonymous voter gets one too, through Bench's own signed cookie or through the X-OV-Voter header a++ * relaying service sends. The anonymous answer is 204 when there is nothing left to judge.++ */++ r.get('/judgements/next', async (req, res) => {try {-- const pair = await bench.nextJudgement({ kind: String(req.query.kind || ''), subject: subjectOf(req), random: ctx.random || Math.random });++ const kind = String(req.query.kind || '');++ const person = subjectOf(req);++ if (person) {++ const pair = await bench.nextJudgement({ kind, subject: person, random: ctx.random || Math.random });++ res.set('Cache-Control', 'private, no-store');++ if (!pair) return res.json({ pair: null, detail: 'There is no round waiting for your judgement right now.' });++ return res.json(pair);++ }++ if (!ctx.anon.enabled()) {++ return problem(res, 403, 'bench.anon_disabled', 'Voting without signing in is off at the moment.');++ }++ const anon = ctx.anon.resolve(req, req.principal);++ if (!anon) {++ return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services, or send a service token that may relay anonymous votes.');++ }++ const kindRow = kindById(kind);++ if (!kindRow || kindRow.judge !== 'human_pairwise') {++ return problem(res, 403, 'bench.anon_kind_not_allowed', 'Only pairwise kinds can be judged without signing in.');++ }++ if (anon.minted) ctx.anon.setCookie(res, anon.opaque);++ const pair = await bench.nextJudgement({++ kind, subject: anon.subject, random: ctx.random || Math.random,++ extraCreators: anon.creator ? [anon.creator] : [],++ });res.set('Cache-Control', 'private, no-store');-- if (!pair) return res.json({ pair: null, detail: 'There is no round waiting for your judgement right now.' });++ if (!pair) return res.status(204).end();return res.json(pair);} catch (err) { return failed(res, err); }});@@@ -174,6 +202,36 @@ function createApi(ctx) {r.post('/rounds/:id/judgements', async (req, res) => {const b = req.body && typeof req.body === 'object' ? req.body : {};const roundId = String(req.params.id || '');++ // A first-party service may relay an anonymous voter: its own service token, and the voter id it derived++ // from its own cookie in the X-OV-Voter header. Only a service on the allow-list may do this.++ let relayed = null;++ try {++ const maybe = ctx.anon.resolve(req, req.principal, { mint: false });++ if (maybe && maybe.source === 'relay') relayed = maybe;++ } catch (err) { return failed(res, err); }++ if (relayed) {++ if (!ctx.anon.enabled()) {++ return problem(res, 403, 'bench.anon_disabled', 'Voting without signing in is off at the moment.');++ }++ try {++ const round = await bench.getRound(roundId, { viewerSubject: null });++ if (!round) return problem(res, 404, 'bench.round_not_found', 'No such round.');++ const row = kindById(round.kind_id);++ if (!row || row.judge !== 'human_pairwise') {++ return problem(res, 403, 'bench.anon_kind_not_allowed', 'Only pairwise kinds can be judged without signing in.');++ }++ const out = await bench.castJudgement({++ roundId, kindId: round.kind_id, judge: 'human_pairwise',++ judgeName: b.judge_name, judgeVersion: b.judge_version,++ pair: b.pair, outcome: b.outcome, score: b.score, voteCounts: b.vote_counts,++ subject: relayed.subject, voterKind: 'anon',++ extraCreators: relayed.creator ? [relayed.creator] : [],++ quota: () => ctx.anon.takeQuota(relayed),++ });++ res.set('Cache-Control', 'private, no-store');++ return res.status(201).json(out);++ } catch (err) { return failed(res, err); }++ }let judge = typeof b.judge === 'string' && b.judge ? b.judge : null;if (!judge) {const round = await bench.getRound(roundId, { viewerSubject: null });@@@ -210,6 +268,8 @@ function createApi(ctx) {type,category: typeof req.query.category === 'string' ? req.query.category : 'all',period: typeof req.query.period === 'string' ? req.query.period : 'all',++ // 'people' shows the signed-in votes alone, 'anon' the anonymous ones alone.++ voters: typeof req.query.voters === 'string' ? req.query.voters : null,});if (!board) return problem(res, 404, 'bench.kind_not_found', 'No such kind.');res.set('Cache-Control', 'public, max-age=30');@@@ -158,16 +158,23 @@ function pane(slot, revealed) {+ frame + facts + '</figure>';}--function voteForm(round, signedIn) {++/**++ * The vote form. A signed-in person votes as themselves. A visitor who has not signed in votes as the anonymous++ * voter their browser holds: one vote per browser, a limit a day, and the pair stays blind until they vote.++ */++function voteForm(round, { signedIn = false, canVote = true } = {}) {if (round.revealed) {return '<section class="sc-vote sc-vote-done" data-vote-state="revealed">'+ '<p>Thanks: the identities are open now.</p>'+ '<p><a href="/showcase/round/' + esc(round.id) + '?share=1">Share this pair</a></p></section>';}-- if (!signedIn) {++ if (!canVote) {return '<section class="sc-vote" data-vote-state="signed-out">'+ '<p><a href="/auth/login?next=' + encodeURIComponent('/showcase/round/' + round.id) + '">Sign in to vote</a></p></section>';}++ const note = signedIn++ ? ''++ : '<p class="sc-vote-note" data-vote-note="1">You are voting without signing in: one vote per browser, and a limit a day.</p>';const game = round.is_game? '<fieldset class="sc-vote-more"><legend>More fun</legend>'+ '<label><input type="radio" name="more_fun" value="A"> Pane A</label>'@@@ -175,7 +182,8 @@ function voteForm(round, signedIn) {+ '<label><input type="radio" name="more_fun" value="tie"> Same</label></fieldset>'+ '<label class="sc-vote-check"><input type="checkbox" name="crashed" value="1"> One of them crashed</label>': '';-- return '<form class="sc-vote" method="post" action="/showcase/round/' + esc(round.id) + '" data-vote-form="1">'++ return '<form class="sc-vote" method="post" action="/showcase/round/' + esc(round.id) + '" data-vote-form="1" data-vote-state="' + (signedIn ? 'signed-in' : 'anonymous') + '">'++ + note+ '<fieldset><legend>Which one wins?</legend>'+ '<label><input type="radio" name="outcome" value="A" required> Pane A</label>'+ '<label><input type="radio" name="outcome" value="B"> Pane B</label>'@@@ -201,14 +209,15 @@ function galleryView(pairs) {+ '</section>';}--function roundView(round, signedIn) {++function roundView(round, { signedIn = false, canVote = true, note = '' } = {}) {const panes = round.slots.map((s) => pane(s, round.revealed)).join('');return '<section class="sc-showcase sc-round" data-showcase="round" data-round-id="' + esc(round.id) + '" data-kind="' + esc(round.kind_id) + '">'+ '<h1 class="sc-round-prompt" data-prompt="1">' + esc(round.prompt) + '</h1>'+ '<p class="sc-round-meta"><span class="sc-kind">' + esc(round.kind_name) + '</span>'+ '<span class="sc-votes" data-votes="' + esc(round.votes) + '">' + esc(round.votes) + ' votes</span></p>'++ + (note ? '<p class="sc-vote-note" data-vote-note="1">' + esc(note) + '</p>' : '')+ '<div class="sc-round-panes" data-panes="2">' + panes + '</div>'-- + voteForm(round, signedIn)++ + voteForm(round, { signedIn, canVote })+ '</section>';}@@@ -220,6 +229,16 @@ function createShowcaseRoutes(ctx) {const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });const subjectOf = (req) => (req.principal && req.principal.kind === 'user' ? req.principal.subject : null);++ /** Who is looking: the person who is signed in, or the anonymous voter the signed cookie names. */++ async function viewerSubjectOf(req) {++ const person = subjectOf(req);++ if (person) return person;++ try {++ const anon = ctx.anon.resolve(req, req.principal, { mint: false });++ return anon ? anon.subject : null;++ } catch { return null; }++ }++r.get('/pairs', async (req, res) => {try {const pairs = await loadPairs(ctx, { kind: req.query.kind, limit: req.query.limit });@@@ -233,7 +252,7 @@ function createShowcaseRoutes(ctx) {r.get('/rounds/:id', async (req, res) => {try {-- const round = await loadRound(ctx, String(req.params.id || ''), subjectOf(req));++ const round = await loadRound(ctx, String(req.params.id || ''), await viewerSubjectOf(req));if (!round) return problem(res, 404, 'bench.round_not_found', 'No such round.');res.set('Cache-Control', round.revealed ? 'public, max-age=30' : 'private, no-store');return res.json({ round });@@@ -246,22 +265,43 @@ function createShowcaseRoutes(ctx) {r.post('/rounds/:id/vote', async (req, res) => {const b = req.body && typeof req.body === 'object' ? req.body : {};const subject = subjectOf(req);-- if (!subject) return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services to vote.');-- if (req.principal.viaSession && !sameOrigin(req, config)) {-- return problem(res, 403, 'bench.cross_site', 'A signed-in request that changes something must come from bench.openvibe.services itself.');-- }const map = { A: 'left', B: 'right', tie: 'tie', both_bad: 'both_bad' };const outcome = map[String(b.outcome || '')];if (!outcome) return problem(res, 422, 'bench.invalid_outcome', 'The outcome must be A, B, tie or both_bad.');const round = await loadRound(ctx, String(req.params.id || ''), null);if (!round || !round.ready) return problem(res, 404, 'bench.round_not_found', 'No such round.');++ // The voter: the person who is signed in, or the anonymous voter the browser cookie names.++ let voter = null;++ if (subject) {++ if (req.principal.viaSession && !sameOrigin(req, config)) {++ return problem(res, 403, 'bench.cross_site', 'A signed-in request that changes something must come from bench.openvibe.services itself.');++ }++ voter = { subject, voterKind: 'person', extraCreators: [] };++ } else {++ if (!ctx.anon.enabled()) {++ return problem(res, 403, 'bench.anon_disabled', 'Voting without signing in is off at the moment.');++ }++ if (!sameOrigin(req, config)) {++ return problem(res, 403, 'bench.cross_site', 'A vote that changes something must come from bench.openvibe.services itself.');++ }++ const anon = ctx.anon.resolve(req, req.principal);++ if (!anon) return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services to vote.');++ if (anon.minted) ctx.anon.setCookie(res, anon.opaque);++ voter = {++ subject: anon.subject, voterKind: 'anon', extraCreators: [],++ quota: () => ctx.anon.takeQuota(anon),++ };++ }try {const out = await bench.castJudgement({roundId: round.id,judge: 'human_pairwise',pair: { left: round.slots[0].entry_id, right: round.slots[1].entry_id },outcome,-- subject,++ subject: voter.subject,++ voterKind: voter.voterKind,++ extraCreators: voter.extraCreators,++ quota: voter.quota || null,});if (round.is_game && (b.more_fun || b.crashed)) {const moreFun = b.more_fun === 'A' ? 1 : b.more_fun === 'B' ? -1 : 0;@@@ -270,7 +310,8 @@ function createShowcaseRoutes(ctx) {judge: 'rubric',judgeName: b.crashed ? 'crashed' : 'more_fun',score: b.crashed ? 1 : moreFun,-- subject,++ subject: voter.subject,++ voterKind: voter.voterKind,});}res.set('Cache-Control', 'private, no-store');@@@ -299,6 +340,8 @@ function createShowcaseRoutes(ctx) {function createShowcasePages(ctx) {const r = express.Router();const { config, principal } = ctx;++ // The vote is a plain form post: the pair is judged without JavaScript.++ r.use(express.urlencoded({ extended: false, limit: '16kb' }));r.use(principal.middleware);const viewerOf = (req) => (req.principal && req.principal.kind === 'user'? { kind: 'user', subject: req.principal.subject, displayName: req.principal.displayName || req.principal.username }@@@ -318,17 +361,25 @@ function createShowcasePages(ctx) {r.get('/round/:id', async (req, res) => {const subject = req.principal && req.principal.kind === 'user' ? req.principal.subject : null;-- const round = await loadRound(ctx, String(req.params.id || ''), subject);++ // A visitor who has not signed in gets an anonymous voter id in a signed cookie, and may judge with it.++ let anon = null;++ if (!subject && ctx.anon.enabled()) {++ try { anon = ctx.anon.resolve(req, req.principal); } catch { anon = null; }++ if (anon && anon.minted) ctx.anon.setCookie(res, anon.opaque);++ }++ const viewerSubject = subject || (anon ? anon.subject : null);++ const round = await loadRound(ctx, String(req.params.id || ''), viewerSubject);if (!round) {return send(res, 404, { config, viewer: viewerOf(req), path: req.originalUrl, title: 'No such round', body: raw('<section><h1>No such round</h1></section>') });}++ const note = String(req.query.voted || '') === 'limit' ? 'Sign in to keep voting today.' : '';return send(res, 200, {config,viewer: viewerOf(req),path: req.originalUrl, // the navbar's current link and the canonical URLtitle: 'Compare two outputs',description: 'Two models, one prompt, side by side.',-- body: raw(roundView(round, Boolean(subject))),++ body: raw(roundView(round, { signedIn: Boolean(subject), canVote: Boolean(subject) || Boolean(anon), note })),});});@@@ -336,20 +387,37 @@ function createShowcasePages(ctx) {const subject = req.principal && req.principal.kind === 'user' ? req.principal.subject : null;const round = await loadRound(ctx, String(req.params.id || ''), null);if (!round) return res.redirect('/showcase');-- if (!subject) return res.redirect('/auth/login?next=' + encodeURIComponent('/showcase/round/' + round.id));const map = { A: 'left', B: 'right', tie: 'tie', both_bad: 'both_bad' };const outcome = map[String((req.body && req.body.outcome) || '')];if (!outcome) return res.redirect('/showcase/round/' + round.id);++ // The voter: the person who is signed in, or the anonymous voter the cookie names.++ let voter = null;++ if (subject) {++ voter = { subject, voterKind: 'person' };++ } else {++ const login = '/auth/login?next=' + encodeURIComponent('/showcase/round/' + round.id);++ if (!ctx.anon.enabled() || !sameOrigin(req, config)) return res.redirect(login);++ let anon = null;++ try { anon = ctx.anon.resolve(req, req.principal); } catch { anon = null; }++ if (!anon) return res.redirect(login);++ if (anon.minted) ctx.anon.setCookie(res, anon.opaque);++ voter = { subject: anon.subject, voterKind: 'anon', quota: () => ctx.anon.takeQuota(anon) };++ }try {await ctx.bench.castJudgement({roundId: round.id,judge: 'human_pairwise',pair: { left: round.slots[0].entry_id, right: round.slots[1].entry_id },outcome,-- subject,++ subject: voter.subject,++ voterKind: voter.voterKind,++ quota: voter.quota || null,});} catch (err) {ctx.log.warn('[OpenVibe.Bench] showcase page vote failed:', (err && err.message) || err);++ if (err && err.status === 429 && err.code === 'bench.limit_anon') {++ return res.redirect('/showcase/round/' + round.id + '?voted=limit');++ }}return res.redirect('/showcase/round/' + round.id);});@@@ -15,6 +15,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }() => { for (const t of timers) clearInterval(t); },() => ctx.keys.client.stop(),() => ctx.jobs.stop(),++ () => ctx.daily.stop(),() => ctx.publisher.stop(),async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },() => ctx.s.close(),@@@ -42,6 +43,8 @@ async function start() {// Jobs: rounds and private eval runs. Off under BENCH_DRILL; tests call tick().ctx.jobs.start();++ // The daily rounds (one per design kind per day). Off until BENCH_DAILY_ROUNDS is on; never under the drill.++ ctx.daily.start();// Outbox publisher → OpenVibe.Events.ctx.publisher.start();@@@ -15,6 +15,9 @@ const styleLib = require('./rating/style');const BASE_ELO = ratingLib.BASE_ELO;const K = 32;const CATEGORY_DEFAULT = 'all';++// The rating categories: everyone ('all'), the signed-in people ('people') and the anonymous voters ('anon').++const CATEGORY_PEOPLE = 'people';++const CATEGORY_ANON = 'anon';const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));@@@ -101,6 +104,27 @@ function wireJudgement(row, { withSubject = false } = {}) {return out;}++/**++ * Who voted: a person, a first-party service, or an anonymous voter. An anonymous voter is a hash (anon_…), never++ * an id, so the database never holds anything about the visitor.++ */++function kindOfVoter(voterKind, subject) {++ if (voterKind === 'person' || voterKind === 'anon' || voterKind === 'service') return voterKind;++ const s = String(subject || '');++ if (!s) return 'service';++ if (s.startsWith('anon_')) return 'anon';++ if (s.startsWith('usr_')) return 'person';++ return 'service';++}++++/** The rating categories a vote is counted in: everyone, and the kind of voter it came from. */++function categoriesOf(voterKind) {++ const out = [CATEGORY_DEFAULT];++ if (voterKind === 'person') out.push(CATEGORY_PEOPLE);++ else if (voterKind === 'anon') out.push(CATEGORY_ANON);++ return out;++}++function wireRating(row) {if (!row) return null;return {@@@ -232,13 +256,25 @@ function createStore(s, { log = console, random = Math.random, config = null } =};}-- async function createRound({ taskId, offers = null, seed = 0, subject }) {++ async function createRound({ taskId, offers = null, seed = 0, subject, bypassQuota = false }) {const task = await getTask(taskId);if (!task) return refusal(404, 'bench.task_not_found', 'No such task.');const kind = await getKind(task.kind_id);if (!kind) return refusal(422, 'bench.kind_not_found', 'The task\'s kind is gone.');const off = Array.isArray(offers) ? offers.slice(0, 3).map((o) => ({ provider: String(o.provider || ''), model: o.model == null ? null : String(o.model) })) : [];if (off.some((o) => !o.provider)) return refusal(422, 'bench.invalid_offers', 'Every offer needs a provider.');++ // The daily rounds job spends Bench's own budget, not a person's quota, so it may skip the counts.++ if (bypassQuota) {++ const ownId = s.newId('brd');++ const own = await db.maybe(++ `INSERT INTO bench_rounds (id, task_id, kind_id, offers, status, seed, params, pool_fallback, created_by, created_at)++ VALUES ($1, $2, $3, $4::jsonb, 'queued', $5, '{}'::jsonb, $6, $7, $8::timestamptz)++ RETURNING *`,++ [ownId, task.id, kind.id, JSON.stringify(off), Math.max(0, Number(seed) || 0), off.length < 3, subject, s.iso()],++ );++ if (!own) throw new BenchError(500, 'bench.round_not_created', 'Bench could not create the round.');++ return { ok: true, round: await getRound(ownId) };++ }const perDay = config ? config.bench.roundsPerDay : 3;const daily = config ? config.bench.roundsDaily : 20;const start = dayStart();@@@ -338,14 +374,16 @@ function createStore(s, { log = console, random = Math.random, config = null } =}/** A ready round this person has not judged and did not start, with two entries shuffled and identities hidden. */-- async function nextJudgement({ kind, subject, random: rng = random } = {}) {++ async function nextJudgement({ kind, subject, random: rng = random, extraCreators = [] } = {}) {++ const creators = (extraCreators || []).filter(Boolean);const row = await db.maybe(`SELECT r.* FROM bench_rounds rWHERE r.kind_id = $1 AND r.status = 'finished' AND r.created_by <> $2++ AND NOT (r.created_by = ANY($3::text[]))AND (SELECT COUNT(*) FROM bench_entries e WHERE e.round_id = r.id AND e.error IS NULL) >= 2AND NOT EXISTS (SELECT 1 FROM bench_judgements j WHERE j.round_id = r.id AND j.subject = $2)ORDER BY r.created_at, r.id LIMIT 1`,-- [String(kind || ''), subject],++ [String(kind || ''), subject, creators],);if (!row) return null;const entries = (await db.many('SELECT * FROM bench_entries WHERE round_id = $1 AND error IS NULL ORDER BY id', [row.id])).map(wireEntry);@@@ -358,7 +396,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =};}-- async function castJudgement({ roundId, kindId, judge, judgeName = null, judgeVersion = null, pair = null, outcome = null, score = null, voteCounts = null, question = null, answer = null, netHash = null, subject }) {++ async function castJudgement({ roundId, kindId, judge, judgeName = null, judgeVersion = null, pair = null, outcome = null, score = null, voteCounts = null, question = null, answer = null, netHash = null, subject, voterKind = null, extraCreators = [], quota = null }) {if (!JUDGES.includes(judge)) throw new BenchError(422, 'bench.invalid_judge', 'Unknown judge.');try {return await s.tx(async () => {@@@ -372,7 +410,13 @@ function createStore(s, { log = console, random = Math.random, config = null } =let left = null;let right = null;if (judge === 'human_pairwise' || judge === 'game_outcome' || judge === 'audience_poll') {-- if (judge === 'human_pairwise' && r.created_by === subject) throw new BenchError(403, 'bench.creator_cannot_vote', 'You cannot judge a round you started.');++ // A voter never judges a round they started: a person never judges their own, and the++ // service that relays an anonymous vote never judges a round it started itself.++ const extra = (extraCreators || []).filter(Boolean);++ const creators = [r.created_by, ...extra].filter(Boolean);++ if (judge === 'human_pairwise' && (creators.includes(subject) || extra.includes(r.created_by))) {++ throw new BenchError(403, 'bench.creator_cannot_vote', 'You cannot judge a round you started.');++ }const p = pair && typeof pair === 'object' ? pair : {};if (!p.left || !p.right || p.left === p.right) throw new BenchError(422, 'bench.invalid_pair', 'A pairwise judgement needs two different entries.');const prior = await db.maybe('SELECT 1 AS n FROM bench_judgements WHERE round_id = $1 AND subject = $2', [r.id, subject]);@@@ -394,18 +438,22 @@ function createStore(s, { log = console, random = Math.random, config = null } =} else if (voteCounts != null) throw new BenchError(422, 'bench.invalid_vote_counts', 'Only an audience poll carries vote counts.');}++ // The daily caps of the anonymous vote are spent here, inside the vote's own transaction: a++ // refused vote spends nothing, and a vote that is rolled back spends nothing either.++ if (quota) await quota();++const id = s.newId('bju');const row = await db.maybe(-- `INSERT INTO bench_judgements (id, round_id, kind_id, judge, judge_name, judge_version, method, pair, outcome, score, subject, at, vote_counts, question, answer, net_hash)-- VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10, $11, $12::timestamptz, $13::jsonb, $14, $15, $16) RETURNING *`,++ `INSERT INTO bench_judgements (id, round_id, kind_id, judge, judge_name, judge_version, method, pair, outcome, score, subject, at, vote_counts, question, answer, net_hash, voter_kind)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10, $11, $12::timestamptz, $13::jsonb, $14, $15, $16, $17) RETURNING *`,[id, r.id, r.kind_id, judge, judgeName, judgeVersion, judge, pair ? JSON.stringify(pair) : null,outcome || null, score == null ? null : num(score), subject || null, s.iso(), voteCounts ? JSON.stringify(voteCounts) : null,question == null ? null : String(question).slice(0, 200), answer == null ? null : String(answer).slice(0, 40),-- netHash == null ? null : String(netHash).slice(0, 64)],++ netHash == null ? null : String(netHash).slice(0, 64), kindOfVoter(voterKind, subject)],);// One stream poll is one game, regardless of audience size. The counts are evidence, not Elo weight.if (judge === 'human_pairwise' || judge === 'game_outcome' || (judge === 'audience_poll' && voteCounts.voters > 0)) {-- await applyPairwiseRating(r.id, r.kind_id, left, right, outcome);++ await applyPairwiseRating(r.id, r.kind_id, left, right, outcome, kindOfVoter(voterKind, subject));}await insertOutbox('bench.judgement.recorded', {judgement_id: id, round_id: r.id, kind_id: r.kind_id, judge, method: judge, outcome: outcome || null, score: score == null ? null : num(score),@@@ -418,42 +466,46 @@ function createStore(s, { log = console, random = Math.random, config = null } =}}-- async function applyPairwiseRating(roundId, kindId, left, right, outcome) {++ async function applyPairwiseRating(roundId, kindId, left, right, outcome, voterKind = 'person') {if (outcome === 'both_bad') return;if (!left || !right || left.model === right.model) return;const models = [left.model, right.model];const now = s.iso();-- for (const model of models) {-- await db.exec(-- `INSERT INTO bench_ratings (kind_id, category, subject, elo, games, wins, losses, ties, win_rate, ci_lower, ci_upper, updated_at)-- VALUES ($1, $2, $3, $4, 0, 0, 0, 0, 0, NULL, NULL, $5::timestamptz)-- ON CONFLICT (kind_id, category, subject) DO NOTHING`,-- [kindId, CATEGORY_DEFAULT, model, BASE_ELO, now],-- );-- }-- const rows = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2 AND subject = ANY($3::text[]) ORDER BY subject FOR UPDATE',-- [kindId, CATEGORY_DEFAULT, models]);-- const byModel = new Map(rows.map((r) => [r.subject, r]));-- const a = byModel.get(left.model);-- const b = byModel.get(right.model);-- if (!a || !b) return;-- const eloOutcome = outcome === 'draw' ? 'tie' : outcome;-- const update = ratingLib.eloUpdate(num(a.elo, BASE_ELO), num(b.elo, BASE_ELO), eloOutcome, K);-- const sa = outcome === 'left' ? 1 : outcome === 'right' ? 0 : 0.5;-- const sb = 1 - sa;-- for (const [row, elo, score, opp] of [[a, update.a, sa, sb], [b, update.b, sb, sa]]) {-- const games = num(row.games) + 1;-- const wins = num(row.wins) + (score > opp ? 1 : 0);-- const losses = num(row.losses) + (score < opp ? 1 : 0);-- const ties = num(row.ties) + (score === opp ? 1 : 0);-- const ci = ratingCi(elo, wins, losses, ties);-- await db.exec(-- `UPDATE bench_ratings SET elo = $1, games = $2, wins = $3, losses = $4, ties = $5,-- win_rate = $6, ci_lower = $7, ci_upper = $8, updated_at = $9::timestamptz-- WHERE kind_id = $10 AND category = $11 AND subject = $12`,-- [elo, games, wins, losses, ties, games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0,-- ci.ci_lower, ci.ci_upper, now, kindId, CATEGORY_DEFAULT, row.subject],-- );++ // The vote is counted twice: once for everyone ('all') and once for the kind of voter it came from++ // ('people' or 'anon'), so a board can show the signed-in votes alone and still count the rest.++ for (const category of categoriesOf(voterKind)) {++ for (const model of models) {++ await db.exec(++ `INSERT INTO bench_ratings (kind_id, category, subject, elo, games, wins, losses, ties, win_rate, ci_lower, ci_upper, updated_at)++ VALUES ($1, $2, $3, $4, 0, 0, 0, 0, 0, NULL, NULL, $5::timestamptz)++ ON CONFLICT (kind_id, category, subject) DO NOTHING`,++ [kindId, category, model, BASE_ELO, now],++ );++ }++ const rows = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2 AND subject = ANY($3::text[]) ORDER BY subject FOR UPDATE',++ [kindId, category, models]);++ const byModel = new Map(rows.map((r) => [r.subject, r]));++ const a = byModel.get(left.model);++ const b = byModel.get(right.model);++ if (!a || !b) continue;++ const eloOutcome = outcome === 'draw' ? 'tie' : outcome;++ const update = ratingLib.eloUpdate(num(a.elo, BASE_ELO), num(b.elo, BASE_ELO), eloOutcome, K);++ const sa = outcome === 'left' ? 1 : outcome === 'right' ? 0 : 0.5;++ const sb = 1 - sa;++ for (const [row, elo, score, opp] of [[a, update.a, sa, sb], [b, update.b, sb, sa]]) {++ const games = num(row.games) + 1;++ const wins = num(row.wins) + (score > opp ? 1 : 0);++ const losses = num(row.losses) + (score < opp ? 1 : 0);++ const ties = num(row.ties) + (score === opp ? 1 : 0);++ const ci = ratingCi(elo, wins, losses, ties);++ await db.exec(++ `UPDATE bench_ratings SET elo = $1, games = $2, wins = $3, losses = $4, ties = $5,++ win_rate = $6, ci_lower = $7, ci_upper = $8, updated_at = $9::timestamptz++ WHERE kind_id = $10 AND category = $11 AND subject = $12`,++ [elo, games, wins, losses, ties, games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0,++ ci.ci_lower, ci.ci_upper, now, kindId, category, row.subject],++ );++ }}await insertOutbox('bench.rating.changed', { kind_id: kindId, category: CATEGORY_DEFAULT, subjects: models });// The ratings have been refitted: this is where the watches are checked.@@@ -474,15 +526,15 @@ function createStore(s, { log = console, random = Math.random, config = null } =return rows.map((r) => r.model);}-- async function boardElo(kindId) {++ async function boardElo(kindId, category = CATEGORY_DEFAULT) {const subjects = await subjectsFor(kindId);-- const ratings = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2', [kindId, CATEGORY_DEFAULT]);++ const ratings = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2', [kindId, category]);const by = new Map(ratings.map((r) => [r.subject, wireRating(r)]));const rows = subjects.map((subject) => {const r = by.get(subject);const games = r ? r.games : 0;return {-- kind_id: kindId, category: CATEGORY_DEFAULT, subject,++ kind_id: kindId, category, subject,elo: r ? r.elo : BASE_ELO,games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,win_rate: r ? r.win_rate : 0,@@@ -524,12 +576,44 @@ function createStore(s, { log = console, random = Math.random, config = null } =return new Map(rows.map((r) => [r.model, r]));}-- async function board(kindId, { type = 'elo', category = CATEGORY_DEFAULT, period = 'all' } = {}) {++ /**++ * Which rating category a board reads: everyone ('all'), the signed-in people ('people') or the anonymous++ * voters ('anon'). The voters parameter wins over the older category parameter.++ */++ function categoryOf(voters, category) {++ const v = String(voters == null ? '' : voters).trim().toLowerCase();++ if (v === 'people' || v === 'person' || v === 'persons') return CATEGORY_PEOPLE;++ if (v === 'anon' || v === 'anonymous') return CATEGORY_ANON;++ const c = String(category == null ? '' : category).trim().toLowerCase();++ if (c === 'people' || c === 'person') return CATEGORY_PEOPLE;++ if (c === 'anon' || c === 'anonymous') return CATEGORY_ANON;++ return CATEGORY_DEFAULT;++ }++++ /** How many votes of each kind of voter a board holds. */++ async function voterCounts(kindId) {++ const rows = await db.many(++ `SELECT COALESCE(voter_kind, 'person') AS kind, COUNT(*)::int AS n++ FROM bench_judgements WHERE kind_id = $1 AND judge = 'human_pairwise'++ GROUP BY 1`,++ [kindId],++ );++ const out = { person: 0, anon: 0, service: 0, total: 0 };++ for (const row of rows) {++ const kind = String(row.kind || '');++ if (kind in out) out[kind] = num(row.n);++ out.total += num(row.n);++ }++ return out;++ }++++ async function board(kindId, { type = 'elo', category = CATEGORY_DEFAULT, period = 'all', voters = null } = {}) {const kind = await getKind(kindId);if (!kind) return null;++ const cat = categoryOf(voters, category);const cost = await costStats(kindId);const pass = await passRate(kindId);-- const eloRows = await boardElo(kindId);++ const eloRows = await boardElo(kindId, cat);let rows;if (type === 'elo') {@@@ -561,7 +645,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =} else {return null;}-- return { kind_id: kindId, category, period, type, rows };++ return { kind_id: kindId, category: cat, period, type, rows, voters: await voterCounts(kindId) };}/**@@@ -710,4 +794,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =};}--module.exports = { createStore, BenchError, KINDS, JUDGES, WINNERS, BASE_ELO, CATEGORY_DEFAULT, shuffle };++module.exports = {++ createStore, BenchError, KINDS, JUDGES, WINNERS, BASE_ELO, CATEGORY_DEFAULT, shuffle,++ CATEGORY_PEOPLE, CATEGORY_ANON, kindOfVoter, categoriesOf,++};@@@ -57,7 +57,7 @@ ${ep('GET', '/api/v1/kinds', 'The benchmark kinds.')}${ep('GET', '/api/v1/kinds/:kindId', 'One kind and its entry schema and metrics.')}${ep('GET', '/api/v1/tasks/:id', 'One task.')}${ep('GET', '/api/v1/rounds/:id', 'One round with entries; entries stay blind until the caller has judged or started the round.')}--${ep('GET', '/api/v1/boards/:kindId?type=elo|pareto|cost', 'A rendered board.')}++${ep('GET', '/api/v1/boards/:kindId?type=elo|pareto|cost&voters=people|anon', 'A rendered board. <code>voters=people</code> shows the signed-in votes alone, <code>voters=anon</code> the anonymous ones; every board says how many votes of each kind it holds.')}${ep('GET', '/api/v1/models/:model', 'One model across every kind.')}${ep('GET', '/api/v1/datasets/:kindId.jsonl', 'The open dataset (prompts, entries, judgements without voter ids, ratings).')}${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its visible tasks and every run, with no person id.')}@@@ -66,8 +66,8 @@ ${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its vi${ep('POST', '/api/v1/kinds/:kindId/tasks', '<code>{ prompt, task_class?, public? }</code>. A person or a service with bench.task.create.')}${ep('POST', '/api/v1/tasks/:id/rounds', '<code>{ offers?, seed? }</code>. Pins a reproducible round; the job generates entries.')}${ep('POST', '/api/v1/rounds/:id/entries', 'A room submits outside entries (a game match, a test-suite run) with a service token carrying bench.entry.create.')}--${ep('GET', '/api/v1/judgements/next?kind=&judge=', 'The next blind pair for this person.')}--${ep('POST', '/api/v1/rounds/:id/judgements', '<code>{ judge?, pair?, outcome?, score?, vote_counts? }</code>. Human judges are people; automatic/test/game judges and audience_poll are services with bench.judgement.cast. An audience_poll supplies { left, right, tie, voters }; it changes Elo as one game regardless of audience size. A zero-voter poll is recorded without changing ratings.')}++${ep('GET', '/api/v1/judgements/next?kind=&judge=', 'The next blind pair for this person, or for an anonymous voter (the signed cookie, or the X-OV-Voter header a relaying service sends).')}++${ep('POST', '/api/v1/rounds/:id/judgements', '<code>{ judge?, pair?, outcome?, score?, vote_counts? }</code>. Human judges are people; automatic/test/game judges and audience_poll are services with bench.judgement.cast. An audience_poll supplies { left, right, tie, voters }; it changes Elo as one game regardless of audience size. A zero-voter poll is recorded without changing ratings. A service on the relay allow-list may cast the vote of an anonymous voter it names in <code>X-OV-Voter: anon:<id></code>.')}${ep('POST', '/api/v1/evals', 'Create a private task suite (never in the public dataset).')}${ep('POST', '/api/v1/evals/:id/run', 'Run a private eval against chosen offers, under the free daily cap.')}${ep('GET', '/api/v1/evals/:id/results', 'A private eval and its runs.')}nnew file mode 100644@@@ -0,0 +1,405 @@++'use strict';++++/**++ * Signed-out voting (plan L51, step 1): the anonymous voter, the relay, and every cap.++ *++ * The caps are read from the environment, so each group of rules gets its own Bench with only that cap low.++ */++++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++++const FORM = { 'content-type': 'application/x-www-form-urlencoded' };++const SAME = { secFetchSite: 'same-origin' };++++/** A round with two entries: two things to judge. */++async function makeRound(t, alice, svc, n, owner) {++ const task = await t.get('/api/v1/kinds/design.ui/tasks', { as: alice, json: { prompt: 'A dashboard, prompt ' + n + '.', public: true } });++ assert.strictEqual(task.status, 201, task.text);++ const rr = await t.get('/api/v1/tasks/' + task.json().task.id + '/rounds', { as: alice, json: {} });++ assert.strictEqual(rr.status, 201, rr.text);++ const roundId = rr.json().round.id;++ const e1 = await t.get('/api/v1/rounds/' + roundId + '/entries', { bearer: svc, json: { provider: 'acme', model: 'alpha-' + n, payload: { html: '<p>a</p>' }, cost_usd: 0.01 } });++ const e2 = await t.get('/api/v1/rounds/' + roundId + '/entries', { bearer: svc, json: { provider: 'beta', model: 'beta-' + n, payload: { html: '<p>b</p>' }, cost_usd: 0.02 } });++ assert.strictEqual(e1.status, 201, e1.text);++ assert.strictEqual(e2.status, 201, e2.text);++ return { id: roundId, a: e1.json().entry.id, b: e2.json().entry.id };++}++++/** The cookie Bench hands a visitor who has not signed in. */++function cookieOf(res) {++ const raw = res.headers.get('set-cookie') || '';++ const first = String(raw).split(';')[0];++ return first.includes('=') ? first : '';++}++++(async () => {++ // ── The voter: one cookie, one vote per round, a cap a day ──────────────++ const t = await boot({++ env: {++ BENCH_ROUNDS_PER_DAY: '20',++ BENCH_ANON_VOTER_PER_DAY: '2',++ BENCH_ANON_IP_PER_DAY: '1000',++ BENCH_ANON_RELAY_PER_DAY: '1000',++ BENCH_ANON_GLOBAL_PER_DAY: '1000',++ BENCH_ANON_RELAY_SERVICES: 'design',++ },++ });++ const alice = t.network.addUser('alice');++ const bob = t.network.addUser('bob');++ const svc = t.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });++ const design = t.network.serviceToken({++ sub: 'svc:design', audience: 'openvibe.bench',++ cap: ['bench.task.create', 'bench.round.create', 'bench.judgement.cast'],++ });++ const rounds = [];++ for (const n of [1, 2, 3]) rounds.push(await makeRound(t, alice, svc, n));++++ let anonCookie = '';++ let anonSubject = '';++++ await check('the next-pair route answers an anonymous voter and hands out a signed cookie', async () => {++ const r = await t.get('/api/v1/judgements/next?kind=design.ui');++ assert.strictEqual(r.status, 200, r.text);++ anonCookie = cookieOf(r);++ assert.ok(anonCookie.startsWith('bench_anon='), 'the voter cookie: ' + anonCookie);++ assert.ok(/HttpOnly/i.test(r.headers.get('set-cookie') || ''), 'the cookie is httpOnly');++ assert.ok(r.json().left && r.json().right, 'a pair');++ assert.strictEqual(r.json().left.model, undefined, 'blind before the vote');++ assert.strictEqual(r.json().right.model, undefined, 'blind before the vote');++ });++++ await check('a signed-out visitor votes through the cookie, and the pair opens after the vote', async () => {++ const before = await t.get('/api/v1/showcase/rounds/' + rounds[0].id, { cookie: anonCookie });++ assert.strictEqual(before.status, 200, before.text);++ assert.strictEqual(before.json().round.revealed, false, 'blind before the vote');++++ const vote = await t.get('/api/v1/showcase/rounds/' + rounds[0].id + '/vote', {++ method: 'POST', cookie: anonCookie, json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(vote.status, 201, vote.text);++ assert.strictEqual(vote.json().judged, true);++ assert.strictEqual(vote.json().revealed, true, 'the pair is open after the vote');++++ const after = await t.get('/api/v1/showcase/rounds/' + rounds[0].id, { cookie: anonCookie });++ assert.strictEqual(after.json().round.revealed, true, 'the pair is open for the voter who judged it');++ assert.ok(after.json().round.slots[0].model, 'the names are there for the voter who judged it');++ });++++ await check('the judgement is stored as an anonymous vote, with a hash and never the id', async () => {++ const row = await t.ctx.s.db.maybe("SELECT voter_kind, subject FROM bench_judgements WHERE judge = 'human_pairwise' ORDER BY at DESC LIMIT 1");++ assert.strictEqual(row.voter_kind, 'anon');++ assert.ok(String(row.subject).startsWith('anon_'), 'the subject is the hash of the opaque id: ' + row.subject);++ assert.ok(!String(row.subject).includes('bench_anon'), 'the cookie value is never stored');++ anonSubject = row.subject;++ // The open dataset never carries a voter id.++ const dataset = await t.get('/api/v1/datasets/design.ui.jsonl');++ assert.strictEqual(dataset.status, 200, dataset.text);++ assert.ok(!dataset.text.includes(anonSubject), 'the dataset carries no voter hash');++ assert.ok(!dataset.text.includes('anon_'), 'the dataset carries no anonymous voter');++ });++++ await check('the same anonymous voter never judges the same round twice', async () => {++ const again = await t.get('/api/v1/showcase/rounds/' + rounds[0].id + '/vote', {++ method: 'POST', cookie: anonCookie, json: { outcome: 'B' }, ...SAME,++ });++ assert.strictEqual(again.status, 409, again.text);++ assert.strictEqual(again.json().code, 'bench.already_judged', again.text);++ });++++ await check('the next-pair route never hands back a round the voter judged, and 204 when none is left', async () => {++ const more = await t.get('/api/v1/judgements/next?kind=design.ui', { cookie: anonCookie });++ assert.strictEqual(more.status, 200, more.text);++ assert.notStrictEqual(more.json().round.id, rounds[0].id, 'a round they have not judged');++ const vote = await t.get('/api/v1/showcase/rounds/' + rounds[1].id + '/vote', {++ method: 'POST', cookie: anonCookie, json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(vote.status, 201, vote.text);++ const none = await t.get('/api/v1/judgements/next?kind=design.svg', { cookie: anonCookie });++ assert.strictEqual(none.status, 204, none.text);++ assert.strictEqual(none.text, '', 'no body with 204');++ });++++ await check('over the daily cap: 429 bench.limit_anon, and a plain reason', async () => {++ const over = await t.get('/api/v1/showcase/rounds/' + rounds[2].id + '/vote', {++ method: 'POST', cookie: anonCookie, json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(over.status, 429, over.text);++ assert.strictEqual(over.json().code, 'bench.limit_anon', over.text);++ assert.strictEqual(over.json().detail, 'Sign in to keep voting today', over.text);++ const counted = await t.ctx.s.db.value("SELECT COUNT(*)::int AS n FROM bench_judgements WHERE voter_kind = 'anon' AND subject = $1", [anonSubject]);++ assert.strictEqual(counted, 2, 'the refused vote was not counted');++ });++++ await check("Bench's own page votes without signing in", async () => {++ const page = await t.get('/showcase/round/' + rounds[2].id);++ assert.strictEqual(page.status, 200, page.text);++ const cookie = cookieOf(page);++ assert.ok(cookie.startsWith('bench_anon='), 'the page hands out the voter cookie');++ assert.ok(page.text.includes('data-vote-form="1"'), 'the signed-out visitor gets the form');++ assert.ok(!page.text.includes('Sign in to vote'), 'not the sign-in prompt');++ const vote = await t.get('/showcase/round/' + rounds[2].id, {++ method: 'POST', cookie, headers: FORM, body: 'outcome=A', ...SAME,++ });++ assert.strictEqual(vote.status, 302, vote.text);++ const row = await t.ctx.s.db.maybe("SELECT voter_kind FROM bench_judgements WHERE round_id = $1 ORDER BY at DESC LIMIT 1", [rounds[2].id]);++ assert.strictEqual(row.voter_kind, 'anon', 'the page vote is an anonymous vote');++ });++++ await check('a board says how many votes of each kind of voter it holds', async () => {++ const person = await t.get('/api/v1/showcase/rounds/' + rounds[0].id, { as: bob });++ assert.strictEqual(person.status, 200, person.text);++ const bobVote = await t.get('/api/v1/rounds/' + rounds[0].id + '/judgements', {++ as: bob, json: { judge: 'human_pairwise', pair: { left: rounds[0].a, right: rounds[0].b }, outcome: 'left' },++ });++ assert.strictEqual(bobVote.status, 201, bobVote.text);++++ const counted = await t.ctx.s.db.many("SELECT voter_kind, COUNT(*)::int AS n FROM bench_judgements WHERE judge = 'human_pairwise' GROUP BY 1");++ const byKind = new Map(counted.map((row) => [row.voter_kind, row.n]));++ const all = await t.get('/api/v1/boards/design.ui');++ assert.strictEqual(all.status, 200, all.text);++ assert.ok(all.json().board.voters, 'the board counts its voters');++ assert.strictEqual(all.json().board.voters.anon, byKind.get('anon'), 'the anonymous votes: ' + JSON.stringify(all.json().board.voters));++ assert.strictEqual(all.json().board.voters.person, byKind.get('person'), 'the signed-in votes: ' + JSON.stringify(all.json().board.voters));++ assert.strictEqual(all.json().board.voters.person, 1, 'one signed-in vote');++++ const people = await t.get('/api/v1/boards/design.ui?voters=people');++ assert.strictEqual(people.status, 200, people.text);++ assert.strictEqual(people.json().board.category, 'people');++ // Every vote is a game for both models, so the games of a board are twice its votes.++ const peopleGames = people.json().board.rows.reduce((n, row) => n + row.games, 0);++ assert.strictEqual(peopleGames, byKind.get('person') * 2, 'the signed-in vote alone: ' + JSON.stringify(people.json().board.rows));++++ const anon = await t.get('/api/v1/boards/design.ui?voters=anon');++ assert.strictEqual(anon.status, 200, anon.text);++ assert.strictEqual(anon.json().board.category, 'anon');++ const anonGames = anon.json().board.rows.reduce((n, row) => n + row.games, 0);++ assert.strictEqual(anonGames, byKind.get('anon') * 2, 'the anonymous votes alone: ' + JSON.stringify(anon.json().board.rows));++ assert.ok(anonGames > peopleGames, 'the anonymous votes are counted apart from the signed-in ones');++ });++++ await check('a signed-in person still gets the pair they have not judged', async () => {++ const r = await t.get('/api/v1/judgements/next?kind=design.ui', { as: bob });++ assert.strictEqual(r.status, 200, r.text);++ assert.ok(r.json().left && r.json().right);++ });++++ await check('stop', async () => { await t.close(); });++++ // ── The relay and the address ────────────────────────────────────────────++ const t2 = await boot({++ env: {++ BENCH_ROUNDS_PER_DAY: '20',++ BENCH_ANON_VOTER_PER_DAY: '1000',++ BENCH_ANON_IP_PER_DAY: '2',++ BENCH_ANON_RELAY_PER_DAY: '1',++ BENCH_ANON_GLOBAL_PER_DAY: '1000',++ BENCH_ANON_RELAY_SERVICES: 'design',++ },++ });++ const alice2 = t2.network.addUser('alice');++ const svc2 = t2.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });++ const design2 = t2.network.serviceToken({++ sub: 'svc:design', audience: 'openvibe.bench',++ cap: ['bench.task.create', 'bench.round.create', 'bench.judgement.cast'],++ });++ const other2 = t2.network.serviceToken({++ sub: 'svc:other', audience: 'openvibe.bench',++ cap: ['bench.task.create', 'bench.round.create', 'bench.judgement.cast'],++ });++ const rounds2 = [];++ for (const n of [1, 2, 3, 4]) rounds2.push(await makeRound(t2, alice2, svc2, n));++++ await check('a service on the allow-list relays an anonymous vote', async () => {++ const r = await t2.get('/api/v1/rounds/' + rounds2[0].id + '/judgements', {++ bearer: design2,++ headers: { 'x-ov-voter': 'anon:visitor-one' },++ json: { pair: { left: rounds2[0].a, right: rounds2[0].b }, outcome: 'left' },++ });++ assert.strictEqual(r.status, 201, r.text);++ assert.ok(r.json().revealed.left.model, 'the names open after the vote');++ const row = await t2.ctx.s.db.maybe("SELECT voter_kind, subject FROM bench_judgements WHERE round_id = $1 ORDER BY at DESC LIMIT 1", [rounds2[0].id]);++ assert.strictEqual(row.voter_kind, 'anon');++ assert.ok(!String(row.subject).includes('visitor-one'), 'the opaque id is never stored: ' + row.subject);++ });++++ await check('the relay gets a blind pair for the anonymous voter it names', async () => {++ const r = await t2.get('/api/v1/judgements/next?kind=design.ui', {++ bearer: design2, headers: { 'x-ov-voter': 'anon:visitor-two' },++ });++ assert.strictEqual(r.status, 200, r.text);++ assert.ok(r.json().left && r.json().right);++ assert.strictEqual(r.json().left.model, undefined, 'blind before the vote');++ assert.ok(r.json().round.id, 'the round it names');++ });++++ await check('a service that is not on the allow-list may not relay', async () => {++ const r = await t2.get('/api/v1/rounds/' + rounds2[1].id + '/judgements', {++ bearer: other2,++ headers: { 'x-ov-voter': 'anon:visitor-three' },++ json: { pair: { left: rounds2[1].a, right: rounds2[1].b }, outcome: 'left' },++ });++ assert.strictEqual(r.status, 403, r.text);++ assert.strictEqual(r.json().code, 'bench.relay_not_allowed', r.text);++ });++++ await check('a service token without bench.judgement.cast may not relay', async () => {++ const weak = t2.network.serviceToken({ sub: 'svc:design', audience: 'openvibe.bench', cap: ['bench.board.read'] });++ const r = await t2.get('/api/v1/rounds/' + rounds2[1].id + '/judgements', {++ bearer: weak,++ headers: { 'x-ov-voter': 'anon:visitor-four' },++ json: { pair: { left: rounds2[1].a, right: rounds2[1].b }, outcome: 'left' },++ });++ assert.strictEqual(r.status, 403, r.text);++ assert.strictEqual(r.json().code, 'bench.capability_denied', r.text);++ });++++ await check('the relay never sees a round it started itself', async () => {++ const task = await t2.get('/api/v1/kinds/design.ui/tasks', { bearer: design2, json: { prompt: 'The relay\'s own round.', public: true } });++ assert.strictEqual(task.status, 201, task.text);++ const rr = await t2.get('/api/v1/tasks/' + task.json().task.id + '/rounds', { bearer: design2, json: {} });++ const own = rr.json().round.id;++ const e1 = await t2.get('/api/v1/rounds/' + own + '/entries', { bearer: svc2, json: { provider: 'acme', model: 'alpha-own', payload: { html: '<p>a</p>' } } });++ const e2 = await t2.get('/api/v1/rounds/' + own + '/entries', { bearer: svc2, json: { provider: 'beta', model: 'beta-own', payload: { html: '<p>b</p>' } } });++ const vote = await t2.get('/api/v1/rounds/' + own + '/judgements', {++ bearer: design2,++ headers: { 'x-ov-voter': 'anon:visitor-five' },++ json: { pair: { left: e1.json().entry.id, right: e2.json().entry.id }, outcome: 'left' },++ });++ assert.strictEqual(vote.status, 403, vote.text);++ assert.strictEqual(vote.json().code, 'bench.creator_cannot_vote', vote.text);++ });++++ await check('the relay has a daily total of its own', async () => {++ const over = await t2.get('/api/v1/rounds/' + rounds2[1].id + '/judgements', {++ bearer: design2,++ headers: { 'x-ov-voter': 'anon:visitor-six' },++ json: { pair: { left: rounds2[1].a, right: rounds2[1].b }, outcome: 'left' },++ });++ assert.strictEqual(over.status, 429, over.text);++ assert.strictEqual(over.json().code, 'bench.limit_anon', over.text);++ assert.strictEqual(over.json().detail, 'Sign in to keep voting today', over.text);++ });++++ await check('the per-address cap: the same address has one budget, another address has its own', async () => {++ const first = await t2.get('/api/v1/judgements/next?kind=design.ui');++ const c1 = cookieOf(first);++ const v1 = await t2.get('/api/v1/showcase/rounds/' + rounds2[1].id + '/vote', {++ method: 'POST', cookie: c1, json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(v1.status, 201, v1.text);++++ const second = await t2.get('/api/v1/judgements/next?kind=design.ui');++ const c2 = cookieOf(second);++ const v2 = await t2.get('/api/v1/showcase/rounds/' + rounds2[2].id + '/vote', {++ method: 'POST', cookie: c2, json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(v2.status, 201, v2.text);++++ // The third visitor from the same address is over the address budget.++ const third = await t2.get('/api/v1/judgements/next?kind=design.ui');++ const c3 = cookieOf(third);++ const v3 = await t2.get('/api/v1/showcase/rounds/' + rounds2[3].id + '/vote', {++ method: 'POST', cookie: c3, json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(v3.status, 429, v3.text);++ assert.strictEqual(v3.json().code, 'bench.limit_anon', v3.text);++++ // Another address has its own budget.++ const other = await t2.get('/api/v1/judgements/next?kind=design.ui', { headers: { 'x-forwarded-for': '203.0.113.9' } });++ const c4 = cookieOf(other);++ const v4 = await t2.get('/api/v1/showcase/rounds/' + rounds2[3].id + '/vote', {++ method: 'POST', cookie: c4, json: { outcome: 'A' }, headers: { 'x-forwarded-for': '203.0.113.9' }, ...SAME,++ });++ assert.strictEqual(v4.status, 201, v4.text);++++ // The address itself is never written down: only a salted hash of it.++ const rows = await t2.ctx.s.db.many("SELECT key FROM bench_anon_counters WHERE scope = 'ip'");++ assert.ok(rows.length >= 2, 'the address counters');++ for (const row of rows) {++ assert.ok(!row.key.includes('203.0.113.9') && !row.key.includes('127.0.0.1'), 'no address in the counters: ' + row.key);++ assert.ok(/^[0-9a-f]{64}$/.test(row.key), 'a hash: ' + row.key);++ }++ });++++ await check('stop', async () => { await t2.close(); });++++ // ── The site-wide cap ────────────────────────────────────────────────────++ const t3 = await boot({++ env: {++ BENCH_ROUNDS_PER_DAY: '20',++ BENCH_ANON_VOTER_PER_DAY: '1000',++ BENCH_ANON_IP_PER_DAY: '1000',++ BENCH_ANON_RELAY_PER_DAY: '1000',++ BENCH_ANON_GLOBAL_PER_DAY: '2',++ },++ });++ const alice3 = t3.network.addUser('alice');++ const svc3 = t3.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });++ const rounds3 = [];++ for (const n of [1, 2, 3]) rounds3.push(await makeRound(t3, alice3, svc3, n));++++ await check('over the site-wide cap every anonymous vote is refused', async () => {++ for (const [i, ip] of ['198.51.100.1', '198.51.100.2'].entries()) {++ const next = await t3.get('/api/v1/judgements/next?kind=design.ui', { headers: { 'x-forwarded-for': ip } });++ const cookie = cookieOf(next);++ const vote = await t3.get('/api/v1/showcase/rounds/' + rounds3[i].id + '/vote', {++ method: 'POST', cookie, json: { outcome: 'A' }, headers: { 'x-forwarded-for': ip }, ...SAME,++ });++ assert.strictEqual(vote.status, 201, vote.text);++ }++ const next = await t3.get('/api/v1/judgements/next?kind=design.ui', { headers: { 'x-forwarded-for': '198.51.100.3' } });++ const cookie = cookieOf(next);++ const over = await t3.get('/api/v1/showcase/rounds/' + rounds3[2].id + '/vote', {++ method: 'POST', cookie, json: { outcome: 'A' }, headers: { 'x-forwarded-for': '198.51.100.3' }, ...SAME,++ });++ assert.strictEqual(over.status, 429, over.text);++ assert.strictEqual(over.json().code, 'bench.limit_anon', over.text);++ assert.strictEqual(over.json().detail, 'Sign in to keep voting today', over.text);++ });++++ await check('stop', async () => { await t3.close(); });++++ // The address hash is salted per day, and the old counters go.++ let now = Date.parse('2026-11-01T09:00:00.000Z');++ const t4 = await boot({++ now: () => now,++ env: { BENCH_ROUNDS_PER_DAY: '20', BENCH_ANON_IP_PER_DAY: '1000', BENCH_ANON_GLOBAL_PER_DAY: '1000' },++ });++ const alice4 = t4.network.addUser('alice');++ const svc4 = t4.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });++ const rounds4 = [];++ for (const n of [1, 2]) rounds4.push(await makeRound(t4, alice4, svc4, n));++++ await check('the hash of an address is salted per day, so two days cannot be linked', async () => {++ const first = await t4.get('/api/v1/judgements/next?kind=design.ui');++ const v1 = await t4.get('/api/v1/showcase/rounds/' + rounds4[0].id + '/vote', {++ method: 'POST', cookie: cookieOf(first), json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(v1.status, 201, v1.text);++++ now = Date.parse('2026-11-02T09:00:00.000Z');++ const second = await t4.get('/api/v1/judgements/next?kind=design.ui');++ const v2 = await t4.get('/api/v1/showcase/rounds/' + rounds4[1].id + '/vote', {++ method: 'POST', cookie: cookieOf(second), json: { outcome: 'A' }, ...SAME,++ });++ assert.strictEqual(v2.status, 201, v2.text);++++ const rows = await t4.ctx.s.db.many("SELECT key, day FROM bench_anon_counters WHERE scope = 'ip' ORDER BY day");++ assert.strictEqual(rows.length, 2, 'two days, two rows: ' + JSON.stringify(rows));++ assert.notStrictEqual(rows[0].key, rows[1].key, 'the salt moves with the day');++ assert.strictEqual(rows[0].day, '2026-11-01');++ assert.strictEqual(rows[1].day, '2026-11-02');++ });++++ await check('the old counters are dropped after their week', async () => {++ now = Date.parse('2026-11-20T09:00:00.000Z');++ await t4.ctx.anon.prune();++ const rows = await t4.ctx.s.db.many('SELECT COUNT(*)::int AS n FROM bench_anon_counters');++ assert.strictEqual(rows[0].n, 0, 'the old rows are gone');++ });++++ await check('stop', async () => { await t4.close(); });++++ done();++})();nnew file mode 100644@@@ -0,0 +1,135 @@++'use strict';++++/**++ * The daily rounds (plan L51, step 2): one round per design kind per day, on Bench's own budget.++ */++++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const { DAILY_KIND_IDS, DAILY_PROMPTS } = require('../server/bench/daily-prompts');++++const CLOCK = Date.parse('2026-11-01T09:00:00.000Z');++const clock = () => CLOCK;++++(async () => {++ await check('the job is off until the owner turns it on, and never starts under the drill', async () => {++ const off = await boot();++ assert.strictEqual(off.ctx.daily.start(), null, 'off without BENCH_DAILY_ROUNDS');++ await off.close();++++ const drill = await boot({ env: { BENCH_DAILY_ROUNDS: '1' } });++ assert.strictEqual(drill.ctx.daily.start(), null, 'off under BENCH_DRILL');++ await drill.close();++ });++++ const t = await boot({++ now: clock,++ env: {++ BENCH_DAILY_ROUNDS: '1',++ BENCH_DAILY_ROUNDS_IN_DRILL: '1',++ BENCH_DAILY_ROUNDS_HOUR_UTC: '8',++ BENCH_DAILY_BUDGET_USD: '1.00',++ BENCH_DAILY_ROUND_ESTIMATE_USD: '0.05',++ },++ });++++ await check('the curated list has about twenty prompts for each of the nine design kinds', async () => {++ assert.strictEqual(DAILY_KIND_IDS.length, 9, 'nine design kinds');++ for (const kind of DAILY_KIND_IDS) {++ assert.ok(DAILY_PROMPTS[kind].length >= 18, kind + ' has about twenty prompts: ' + DAILY_PROMPTS[kind].length);++ for (const prompt of DAILY_PROMPTS[kind]) {++ assert.ok(prompt.length >= 20, kind + ': a prompt is a brief');++ assert.ok(!/\b(he|she|his|her)\b/i.test(prompt), kind + ': no people in the brief');++ }++ }++ });++++ let summary = null;++++ await check('one round per design kind, from the curated prompts, with two models of the pool', async () => {++ summary = await t.ctx.daily.runDay({ now: CLOCK });++ assert.strictEqual(summary.created, 9, 'nine rounds: ' + JSON.stringify(summary.rounds));++ assert.strictEqual(summary.skipped, 0);++++ const rows = await t.ctx.s.db.many('SELECT * FROM bench_daily_rounds ORDER BY kind_id');++ assert.strictEqual(rows.length, 9, 'one row per kind');++ for (const row of rows) {++ assert.strictEqual(row.status, 'created');++ assert.strictEqual(row.day, '2026-11-01');++ assert.ok(DAILY_PROMPTS[row.kind_id].includes(row.prompt), 'the prompt is one of the curated ones: ' + row.prompt);++ const models = Array.isArray(row.models) ? row.models : JSON.parse(row.models);++ assert.strictEqual(models.length, 2, 'two models: ' + JSON.stringify(models));++ assert.notStrictEqual(models[0].provider, models[1].provider, 'two different models: ' + JSON.stringify(models));++ const round = await t.ctx.s.db.maybe('SELECT * FROM bench_rounds WHERE id = $1', [row.round_id]);++ assert.ok(round, 'the round exists');++ assert.strictEqual(round.kind_id, row.kind_id);++ assert.strictEqual(round.created_by, 'service:bench-daily', 'the daily rounds belong to Bench');++ }++ // The prompt of a kind is the same all day, and comes from the curated list.++ assert.strictEqual(t.ctx.daily.promptFor('design.ui', '2026-11-01'), t.ctx.daily.promptFor('design.ui', '2026-11-01'));++ });++++ await check('the same day never makes a second round of a kind', async () => {++ const again = await t.ctx.daily.runDay({ now: CLOCK });++ assert.strictEqual(again.created, 0, 'nothing new: ' + JSON.stringify(again.rounds));++ const rows = await t.ctx.s.db.many('SELECT COUNT(*)::int AS n FROM bench_daily_rounds');++ assert.strictEqual(rows[0].n, 9);++ });++++ await check('the rounds are real rounds: two entries each, and something to judge', async () => {++ // The worker takes one queued round per pass, and there are nine of them.++ for (let i = 0; i < 12; i++) await t.tick();++ const rounds = await t.ctx.s.db.many("SELECT id, status FROM bench_rounds WHERE created_by = 'service:bench-daily'");++ assert.strictEqual(rounds.length, 9);++ for (const round of rounds) {++ assert.strictEqual(round.status, 'finished', 'the round is ready to judge');++ }++ const next = await t.get('/api/v1/judgements/next?kind=design.ui');++ assert.strictEqual(next.status, 200, next.text);++ assert.ok(next.json().left && next.json().right, 'there is something to judge');++ assert.strictEqual(next.json().left.model, undefined, 'the pair is blind');++ });++++ await check('what the day spent is written down', async () => {++ await t.ctx.daily.settle('2026-11-01');++ const rows = await t.ctx.s.db.many('SELECT kind_id, estimated_usd, spent_usd FROM bench_daily_rounds ORDER BY kind_id');++ let spent = 0;++ for (const row of rows) {++ assert.ok(Number(row.spent_usd) > 0, row.kind_id + ': what it spent');++ spent += Number(row.spent_usd);++ }++ assert.ok(spent > 0, 'the day spent something');++ const state = await t.ctx.daily.budgetState('2026-11-01');++ assert.ok(Math.abs(state.spent - spent) < 1e-6, 'the budget state knows what was spent');++ });++++ await check('stop', async () => { await t.close(); });++++ // ── The budget ───────────────────────────────────────────────────────────++ const lean = await boot({++ now: clock,++ env: {++ BENCH_DAILY_ROUNDS: '1',++ BENCH_DAILY_BUDGET_USD: '0.001',++ BENCH_DAILY_ROUND_ESTIMATE_USD: '0.05',++ },++ });++++ await check('a kind is skipped when the day\'s budget is spent', async () => {++ const out = await lean.ctx.daily.runDay({ now: CLOCK });++ assert.strictEqual(out.created, 0, 'nothing was made: ' + JSON.stringify(out.rounds));++ assert.strictEqual(out.skipped, 9);++ const rows = await lean.ctx.s.db.many('SELECT kind_id, status, skip_reason FROM bench_daily_rounds');++ assert.strictEqual(rows.length, 9);++ for (const row of rows) {++ assert.strictEqual(row.status, 'skipped');++ assert.strictEqual(row.skip_reason, 'budget', 'why it was skipped: ' + row.kind_id);++ }++ const rounds = await lean.ctx.s.db.many('SELECT COUNT(*)::int AS n FROM bench_rounds');++ assert.strictEqual(rounds[0].n, 0, 'no round was made');++ });++++ await check('stop', async () => { await lean.close(); });++++ done();++})();
@@ -41,6 +41,60 @@ run under it, with no person id. Nothing private or unlisted is ever read there.
@@ -41,6 +41,60 @@ run under it, with no person id. Nothing private or unlisted is ever read there.
| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. |
| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. |
| `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |
| `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |
+## Voting without signing in
+
+A person who has not signed in may still judge a pairwise round (`human_pairwise` kinds only). There are two ways in,
+and both are capped.
+
+**On Bench's own pages.** `/showcase/round/:id` (and `POST /api/v1/showcase/rounds/:id/vote`) hands the visitor a
+random opaque id in a signed, httpOnly cookie (`bench_anon`). The id says nothing about the visitor, and the cookie
+is signed with a server secret, so a client cannot mint a new voter id at will. Bench never stores the address: the
+per-address cap counts a hash of it, salted per day, so two days of addresses cannot be linked.
+
+**Relayed by a first-party service.** A service that may relay sends its own service token (with
+`bench.judgement.cast`) and the header `X-OV-Voter: anon:<opaque id it derived from its own cookie>`. Only a service
+on the allow-list may relay: `BENCH_ANON_RELAY_SERVICES` (default `design`). The relay's own daily total is capped
+apart from the per-voter cap.
+
+| Route | Who | What |
+|---|---|---|
+| `GET /api/v1/judgements/next?kind=` | a person, an anonymous voter (cookie), or a relaying service (`X-OV-Voter`) | The next ready round of that kind the caller has not judged and did not start, blind. A signed-in person keeps the old answer (`200` with `pair: null` when there is nothing); an anonymous voter gets `204` when there is nothing to judge. |
+| `POST /api/v1/rounds/:id/judgements` | a person, a service with `bench.judgement.cast`, or a relaying service with `X-OV-Voter` | The vote. A relayed vote is a `human_pairwise` vote of the anonymous voter the header names. |
+| `POST /api/v1/showcase/rounds/:id/vote` | a person, or an anonymous voter (the cookie) | The vote from Bench's own page. |
+| `GET /api/v1/boards/:kindId?voters=people\|anon` | anyone | The board of the signed-in votes alone, or of the anonymous votes alone. Every board says how many votes of each kind of voter it holds (`voters`). |
+
+What is stored is never the opaque id: the judgement carries `voter_kind` `anon` and the hash of the id, in the same
+column a person's id would use. One vote per voter per round, and a creator never judges their own round, follow from
+that. The open dataset never carries a voter id.
+
+Every cap answers the same way: `429` with the code `bench.limit_anon` and the detail "Sign in to keep voting today".
+
+| Cap | Environment variable | Default |
+|---|---|---|
+| One anonymous voter, per day | `BENCH_ANON_VOTER_PER_DAY` | 30 |
+| One address (a salted hash of it, salted per day), per day | `BENCH_ANON_IP_PER_DAY` | 60 |
+| One relaying service, per day | `BENCH_ANON_RELAY_PER_DAY` | 3000 |
+| Every anonymous vote on the site, per day | `BENCH_ANON_GLOBAL_PER_DAY` | 20000 |
+| The services that may relay | `BENCH_ANON_RELAY_SERVICES` | `design` |
+| Signed-out voting on or off | `BENCH_ANON_ENABLED` | on |
+| The secret the cookie is signed with | `BENCH_ANON_SECRET` | the OAuth client secret |
+
+## The daily rounds
+
+Bench makes one round per design kind per day, at one fixed UTC hour, on its own budget. It is off until the owner
+turns it on (`BENCH_DAILY_ROUNDS`), and it never starts under the drill. The prompts come from a curated list of
+about twenty per kind (`server/bench/daily-prompts.js`), and the two models of every round come from the pool the
+ordinary rounds use. A kind whose share of the day's budget is gone is skipped, and what the day spent is written
+down in `bench_daily_rounds`.
+
+| Setting | Environment variable | Default |
+|---|---|---|
+| The job on or off | `BENCH_DAILY_ROUNDS` | off |
+| The job under the drill (the tests) | `BENCH_DAILY_ROUNDS_IN_DRILL` | off |
+| The UTC hour of the day | `BENCH_DAILY_ROUNDS_HOUR_UTC` | 8 |
+| What the day's rounds may cost | `BENCH_DAILY_BUDGET_USD` | 1.00 |
+| What one round is assumed to cost | `BENCH_DAILY_ROUND_ESTIMATE_USD` | 0.05 |
+
## The older API
## The older API
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
Bench also serves kinds, tasks, rounds, entries, judgements, the boards, the models, the open dataset and the private
new file mode 100644
new file mode 100644
@@ -0,0 +1,47 @@
@@ -0,0 +1,47 @@
+-- phase: expand
+-- Plan L51 step 1 (signed-out voting) and step 2 (the daily rounds job).
+--
+-- 1. A judgement now says who voted: a person, a first-party service, or an anonymous voter. The anonymous
+-- voter is a hash, never an id: the raw opaque id never reaches the database.
+ALTER TABLE bench_judgements ADD COLUMN voter_kind text COLLATE "C";
+UPDATE bench_judgements
+ SET voter_kind = CASE WHEN subject IS NULL THEN 'service'
+ WHEN left(subject, 4) = 'usr_' THEN 'person'
+ ELSE 'service' END;
+ALTER TABLE bench_judgements ALTER COLUMN voter_kind SET DEFAULT 'person';
+ALTER TABLE bench_judgements ADD CONSTRAINT bench_judgements_voter_kind
+ CHECK (voter_kind IS NULL OR voter_kind IN ('person', 'anon', 'service'));
+CREATE INDEX bench_judgements_by_voter_kind ON bench_judgements (kind_id, voter_kind, at);
+
+-- 2. The daily counters of the anonymous vote: one row per scope, key and UTC day.
+-- scope is 'voter' (the hashed opaque id), 'ip' (the salted hash of the address, salted per day),
+-- 'relay' (the relaying service) or 'global'. The address itself is never written anywhere.
+CREATE TABLE bench_anon_counters (
+ scope text COLLATE "C" NOT NULL, -- voter | ip | relay | global
+ key text COLLATE "C" NOT NULL, -- the hashed voter id, the hashed address, the service, or 'all'
+ day text COLLATE "C" NOT NULL, -- the UTC day, 'YYYY-MM-DD'
+ count integer NOT NULL DEFAULT 0,
+ PRIMARY KEY (scope, key, day),
+ CONSTRAINT bench_anon_counters_count CHECK (count >= 0)
+);
+CREATE INDEX bench_anon_counters_day ON bench_anon_counters (day);
+
+-- 3. The daily rounds job: what it planned for every kind, what it skipped and what it spent.
+CREATE TABLE bench_daily_rounds (
+ id text COLLATE "C" PRIMARY KEY, -- bdr_ + ULID
+ day text COLLATE "C" NOT NULL, -- the UTC day, 'YYYY-MM-DD'
+ kind_id text COLLATE "C" NOT NULL,
+ prompt text,
+ models jsonb,
+ task_id text,
+ round_id text,
+ estimated_usd numeric(12, 6) NOT NULL DEFAULT 0,
+ spent_usd numeric(12, 6),
+ status text COLLATE "C" NOT NULL, -- planned | created | skipped | failed
+ skip_reason text,
+ created_at timestamptz NOT NULL DEFAULT now(),
+ CONSTRAINT bench_daily_rounds_id CHECK (id ~ '^bdr_[0-9A-HJKMNP-TV-Z]{26}$'),
+ CONSTRAINT bench_daily_rounds_status CHECK (status IN ('planned', 'created', 'skipped', 'failed'))
+);
+CREATE UNIQUE INDEX bench_daily_rounds_day_kind ON bench_daily_rounds (day, kind_id);
+CREATE INDEX bench_daily_rounds_day ON bench_daily_rounds (day);
@@ -35,6 +35,8 @@ const { createEmbedRoutes } = require('./http/embed');
@@ -35,6 +35,8 @@ const { createEmbedRoutes } = require('./http/embed');
const { createWatchPages } = require('./http/watch');
const { createWatchPages } = require('./http/watch');
const { createWatches } = require('./watch/watch');
const { createWatches } = require('./watch/watch');
const { createBattles } = require('./bench/battles');
const { createBattles } = require('./bench/battles');
+const { createAnon } = require('./bench/anon');
+const { createDailyRounds } = require('./bench/daily');
const { createBattlePages } = require('./http/battle');
const { createBattlePages } = require('./http/battle');
const { createRatingsPages } = require('./http/ratings');
const { createRatingsPages } = require('./http/ratings');
const { createRenderRoutes } = require('./http/render');
const { createRenderRoutes } = require('./http/render');
@@ -80,6 +82,9 @@ async function createApp(opts = {}) {
@@ -80,6 +82,9 @@ async function createApp(opts = {}) {
const mirror = require('./bench/mirror').createMirror({ s, registry, fetchImpl, log });
const mirror = require('./bench/mirror').createMirror({ s, registry, fetchImpl, log });
const ai = opts.benchAi || createBenchAi({ config, fetchImpl, log });
const ai = opts.benchAi || createBenchAi({ config, fetchImpl, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
const jobs = createBenchJobs({ s, bench, ai, config, log });
+ // The signed-out voter (the cookie, the relay header and the daily caps) and the daily rounds job.
+ const anon = createAnon({ s, config, log });
+ const daily = createDailyRounds({ s, bench, ai, config, anon, log });
const publisher = createPublisher({ config, s, fetchImpl, log });
const publisher = createPublisher({ config, s, fetchImpl, log });
await bench.seedKinds();
await bench.seedKinds();
@@ -94,6 +99,7 @@ async function createApp(opts = {}) {
@@ -94,6 +99,7 @@ async function createApp(opts = {}) {
const launcher = createLauncher({ config, registry, fetchImpl: fetchImpl || globalThis.fetch, log });
const launcher = createLauncher({ config, registry, fetchImpl: fetchImpl || globalThis.fetch, log });
const ctx = {
const ctx = {
config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,
config, s, keys, sso, principal, bench, registry, launcher, ai, jobs, publisher, accountData, accountSend, log, fetchImpl,
+ anon, daily,
imports, mirror,
imports, mirror,
random: opts.random || Math.random,
random: opts.random || Math.random,
};
};
new file mode 100644
new file mode 100644
@@ -0,0 +1,228 @@
@@ -0,0 +1,228 @@
+'use strict';
+
+/**
+ * Anonymous voting (plan L51, step 1): a person who has not signed in may still judge a pairwise round.
+ *
+ * There are two ways in, and both end at the same place:
+ *
+ * 1. Bench's own vote page. The visitor gets a random opaque id in a signed, httpOnly cookie. The id says
+ * nothing about the person, and the cookie is signed with a server secret so a client cannot mint a new
+ * voter id at will. Bench never stores the address: the per-address cap counts a salted hash of it, and
+ * the salt is derived from the day, so yesterday's hash says nothing about today's.
+ * 2. A first-party service (OpenVibe.Design) that relays the vote. It sends its service token and the
+ * header X-OV-Voter: anon:<opaque id it derived from its own cookie>. Only a service on the allow-list
+ * (BENCH_ANON_RELAY_SERVICES, 'design' by default) may relay, and the relay's own daily total is capped.
+ *
+ * What is stored is never the opaque id: the judgement carries voter_kind 'anon' and the hash of the id, in
+ * the same subject column a person's id would use. One vote per voter per round falls out of that for free.
+ *
+ * Every cap answers the same way: 429 bench.limit_anon, "Sign in to keep voting today".
+ */
+
+const crypto = require('crypto');
+const { BenchError } = require('../store');
+
+const HEADER = 'x-ov-voter';
+const HEADER_PREFIX = 'anon:';
+const RELAY_CAP = 'bench.judgement.cast';
+const LIMIT_DETAIL = 'Sign in to keep voting today';
+const SUBJECT_PREFIX = 'anon_';
+const KEEP_DAYS = 7;
+
+const num = (v, d = 0) => (v == null || Number.isNaN(Number(v)) ? d : Number(v));
+const b64u = (buf) => Buffer.from(buf).toString('base64url');
+const hmac = (secret, msg) => crypto.createHmac('sha256', secret).update(String(msg), 'utf8').digest();
+const sha256hex = (s) => crypto.createHash('sha256').update(String(s), 'utf8').digest('hex');
+
+function createAnon({ s, config, log = console }) {
+ const db = s.db;
+ const cfg = () => (config && config.bench && config.bench.anon) || {};
+ let secretCache = null;
+
+ /** The secret the cookie is signed with and the voter id is hashed with. */
+ function secret() {
+ if (secretCache) return secretCache;
+ const c = cfg();
+ const own = String(c.secret || '').trim();
+ const oauth = String((config && config.oauth && config.oauth.clientSecret) || '').trim();
+ secretCache = own || oauth || crypto.randomBytes(32).toString('hex');
+ if (!own && !oauth && log && typeof log.warn === 'function') {
+ log.warn('[OpenVibe.Bench] neither BENCH_ANON_SECRET nor an OAuth client secret is set: the anonymous voter cookie is signed with a per-process secret, so a restart resets the anonymous caps.');
+ }
+ return secretCache;
+ }
+
+ const cookieName = () => String(cfg().cookieName || 'bench_anon');
+ const enabled = () => cfg().enabled !== false;
+
+ /** The UTC day, 'YYYY-MM-DD'. */
+ function dayOf(now) {
+ return new Date(now == null ? s.now() : now).toISOString().slice(0, 10);
+ }
+
+ /** A new opaque voter id: random bytes, nothing about the person. */
+ function newOpaqueId() {
+ return b64u(crypto.randomBytes(18));
+ }
+
+ /** The cookie value: the id and its signature. */
+ function cookieValue(opaque) {
+ return `${opaque}.${hmac(secret(), 'cookie|' + opaque).toString('base64url')}`;
+ }
+
+ /** The id inside a signed cookie, or null when the cookie is not ours or is broken. */
+ function opaqueFromCookie(raw) {
+ const value = String(raw || '');
+ const dot = value.indexOf('.');
+ if (dot < 1) return null;
+ const id = value.slice(0, dot);
+ const sig = value.slice(dot + 1);
+ if (!/^[A-Za-z0-9_-]{16,64}$/.test(id)) return null;
+ const want = hmac(secret(), 'cookie|' + id).toString('base64url');
+ const a = Buffer.from(String(sig));
+ const b = Buffer.from(want);
+ if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) return null;
+ return id;
+ }
+
+ /** The hash of the opaque id: what the database holds, never the id itself. */
+ function voterHash(scope, opaque) {
+ return SUBJECT_PREFIX + hmac(secret(), `${scope}|${opaque}`).toString('hex');
+ }
+
+ /** The address hash: salted per day, so the rows of two days cannot be linked. */
+ function ipHash(ip, day) {
+ const salt = hmac(secret(), 'ip-salt|' + day).toString('hex');
+ return sha256hex(`bench-anon-ip|${day}|${salt}|${ip}`);
+ }
+
+ /** The address of the request, as the proxy chain reports it. */
+ function ipOf(req) {
+ const ip = String((req && req.ip) || (req && req.socket && req.socket.remoteAddress) || '').trim();
+ return ip || null;
+ }
+
+ /** The name of a service principal: 'service:design' is 'design'. */
+ function serviceName(requester) {
+ const r = String(requester || '').trim().toLowerCase();
+ return r.startsWith('service:') ? r.slice('service:'.length) : r;
+ }
+
+ /** Whether a service may relay an anonymous voter. */
+ function relayAllowed(service) {
+ const list = cfg().relayServices || [];
+ const name = serviceName(service);
+ return list.includes(name) || list.includes('service:' + name);
+ }
+
+ /**
+ * Who the anonymous voter is, if the request names one:
+ * - a relaying service and its X-OV-Voter header;
+ * - the signed cookie Bench set earlier;
+ * - a fresh cookie, when the caller is nobody in particular (mint).
+ * Returns null when the request carries no anonymous voter and none should be made.
+ */
+ function resolve(req, principal, { mint = true } = {}) {
+ const p = principal || { kind: 'anonymous' };
+ const header = String((req && req.headers && req.headers[HEADER]) || '').trim();
+ if (header) {
+ if (p.kind !== 'app') throw new BenchError(403, 'bench.relay_not_allowed', 'Only a first-party service may relay an anonymous voter.');
+ const service = serviceName(p.requester);
+ if (!relayAllowed(service)) throw new BenchError(403, 'bench.relay_not_allowed', 'This service may not relay anonymous votes.');
+ if (typeof p.hasCapability === 'function' && !p.hasCapability(RELAY_CAP)) {
+ throw new BenchError(403, 'bench.capability_denied', `A service token needs ${RELAY_CAP}.`);
+ }
+ if (!header.startsWith(HEADER_PREFIX)) throw new BenchError(400, 'bench.anon_voter_invalid', 'The X-OV-Voter header must be anon:<opaque id>.');
+ const opaque = header.slice(HEADER_PREFIX.length).trim();
+ if (!/^[A-Za-z0-9._~:-]{6,200}$/.test(opaque)) throw new BenchError(400, 'bench.anon_voter_invalid', 'The X-OV-Voter header must be anon:<opaque id>.');
+ return {
+ kind: 'anon', source: 'relay', opaque,
+ subject: voterHash('relay:' + service, opaque),
+ relay: service, creator: 'service:' + service, ip: null, minted: false,
+ };
+ }
+ const cookies = (req && req.cookies) || {};
+ const raw = cookies[cookieName()];
+ const id = raw ? opaqueFromCookie(raw) : null;
+ if (id) {
+ return {
+ kind: 'anon', source: 'cookie', opaque: id,
+ subject: voterHash('cookie', id),
+ relay: null, creator: null, ip: ipOf(req), minted: false,
+ };
+ }
+ if (!mint) return null;
+ if (p.kind && p.kind !== 'anonymous') return null;
+ const fresh = newOpaqueId();
+ return {
+ kind: 'anon', source: 'cookie', opaque: fresh,
+ subject: voterHash('cookie', fresh),
+ relay: null, creator: null, ip: ipOf(req), minted: true,
+ };
+ }
+
+ /** Give the browser its voter cookie. */
+ function setCookie(res, opaque) {
+ if (!res || typeof res.cookie !== 'function') return;
+ const days = Math.max(1, num(cfg().cookieDays, 180));
+ res.cookie(cookieName(), cookieValue(opaque), {
+ httpOnly: true,
+ sameSite: 'lax',
+ secure: Boolean(config && config.cookies && config.cookies.secure),
+ maxAge: days * 86_400_000,
+ path: '/',
+ });
+ }
+
+ /**
+ * The daily counters this vote spends: the voter, the address (only for Bench's own cookie), the relaying
+ * service and the site. The counters move inside the vote's own transaction, so a refused vote spends
+ * nothing and a vote that is rolled back spends nothing either.
+ */
+ async function takeQuota(identity) {
+ const c = cfg();
+ const day = dayOf();
+ const scopes = [{ scope: 'voter', key: identity.subject, cap: num(c.voterPerDay, 30) }];
+ if (identity.source === 'cookie' && identity.ip) {
+ scopes.push({ scope: 'ip', key: ipHash(identity.ip, day), cap: num(c.ipPerDay, 60) });
+ }
+ if (identity.relay) scopes.push({ scope: 'relay', key: identity.relay, cap: num(c.relayPerDay, 3000) });
+ scopes.push({ scope: 'global', key: 'all', cap: num(c.globalPerDay, 20000) });
+ for (const one of scopes) {
+ const row = await db.maybe(
+ `INSERT INTO bench_anon_counters (scope, key, day, count) VALUES ($1, $2, $3, 1)
+ ON CONFLICT (scope, key, day) DO UPDATE SET count = bench_anon_counters.count + 1
+ RETURNING count`,
+ [one.scope, one.key, day],
+ );
+ if (num(row && row.count) > one.cap) throw new BenchError(429, 'bench.limit_anon', LIMIT_DETAIL);
+ }
+ return { day, scopes: scopes.map((one) => one.scope) };
+ }
+
+ /** What the voter may still spend today, for the pages that want to say so. */
+ async function remaining(identity) {
+ const c = cfg();
+ const day = dayOf();
+ const used = num(await db.value(
+ 'SELECT COUNT(*)::int AS n FROM bench_anon_counters WHERE scope = $1 AND key = $2 AND day = $3',
+ ['voter', identity.subject, day],
+ ));
+ return { used, limit: num(c.voterPerDay, 30), left: Math.max(0, num(c.voterPerDay, 30) - used) };
+ }
+
+ /** Old counters are dropped: the address hash of yesterday is worth nothing today. */
+ async function prune(days = KEEP_DAYS) {
+ const cutoff = new Date(s.now() - Math.max(1, days) * 86_400_000).toISOString().slice(0, 10);
+ await db.exec('DELETE FROM bench_anon_counters WHERE day < $1', [cutoff]);
+ }
+
+ return {
+ HEADER, HEADER_PREFIX, RELAY_CAP, LIMIT_DETAIL,
+ resolve, setCookie, takeQuota, remaining, prune,
+ cookieName, enabled, dayOf, newOpaqueId, cookieValue, opaqueFromCookie, voterHash, ipHash, ipOf,
+ serviceName, relayAllowed, secret,
+ };
+}
+
+module.exports = { createAnon, HEADER, HEADER_PREFIX, RELAY_CAP, LIMIT_DETAIL };
new file mode 100644
new file mode 100644
@@ -0,0 +1,213 @@
@@ -0,0 +1,213 @@
+'use strict';
+
+/**
+ * The prompts of the daily rounds (plan L51, step 2): about twenty per design kind, so every day has something
+ * worth judging and no two days ask the same thing. They are concrete briefs for one artefact of the kind, written
+ * for a model that has never seen the site: no real people, no likenesses, no names of anyone who exists.
+ */
+
+const DAILY_PROMPTS = Object.freeze({
+ 'design.page': Object.freeze([
+ 'A landing page for a fictional neighbourhood tool library: what it lends, a lending policy table, and opening hours.',
+ 'A one-page guide to a fictional tide-pool reserve: three tide zones explained, a safety list and a printable checklist.',
+ 'A landing page for a fictional language-learning app: what it teaches, a seven-day trial table and a pricing section.',
+ 'A page for a fictional second-hand bicycle workshop: the services it offers, prices in a table and a booking form.',
+ 'A single page for a fictional community seed library: how borrowing works and a month-by-month seed-saving calendar.',
+ 'A product page for a fictional desk lamp: a specification table, an energy label and three rooms it fits in.',
+ 'An announcement page for a fictional museum winter exhibition: dates, a curator note and a plan for the visit.',
+ 'A one-page site for a fictional freelance bookkeeper: the services, the rates and a contact form.',
+ 'A page for a fictional rooftop beekeeping co-op: where the hives stand, when honey is ready and how to join.',
+ 'A page for a fictional ceramics studio: the class schedule as a table, the materials and the studio rules.',
+ 'A landing page for a fictional city air-quality dashboard: what the scale means, where the data comes from and how to read it.',
+ 'A page for a fictional mountain hut: a booking form, a hut-to-hut map and a safety checklist for the walk in.',
+ 'A page for a fictional vinyl subscription: three tiers side by side and a preview of the listening notes.',
+ 'A brochure page for a fictional makerspace: the tools, what each is for and how the induction works.',
+ 'A page for a fictional river kayak club: the season calendar, the safety briefing and the membership form.',
+ 'A landing page for a fictional tool that turns meeting notes into tasks: three steps, a sample and a sign-up.',
+ 'A page for a fictional school science fair: the schedule, the judging criteria and the registration form.',
+ 'A guide to growing herbs on a balcony: a table of light and water needs, and a shopping list.',
+ 'A page for a fictional archive of letters between two invented towns: a timeline and a description of the exhibit room.',
+ 'A page for a fictional repair cafe: the schedule, what can be fixed and what to bring.',
+ ]),
+ 'design.component': Object.freeze([
+ 'A modal dialog that asks a person to confirm a destructive action: a clear question, cancel and confirm, and the focus trapped inside.',
+ 'An accessible dropdown menu with a search field, grouped items and full keyboard support.',
+ 'A price card that switches between monthly and yearly billing and shows what the year costs.',
+ 'A file-upload drop zone with a progress bar, an error state and a retry button.',
+ 'A four-step sign-up stepper showing which steps are done, which one is current and which are still to come.',
+ 'An address field with suggestions, a loading state and a state for "nothing found".',
+ 'A rating widget with half stars, a numeric readout and a hover preview.',
+ 'A toast component in three variants: success, warning and error.',
+ 'A date range picker with presets and a past range that cannot be picked.',
+ 'An expandable question and answer item with a chevron, a smooth animation and the right ARIA state.',
+ 'A tag input with suggestions, keyboard removal of a tag and a warning when there are too many.',
+ 'A comparison table that turns into cards on a narrow screen.',
+ 'A consent banner with accept, refuse and a "learn more" link.',
+ 'A skeleton loading card with exactly the shape of the card it stands in for.',
+ 'A breadcrumb trail with a middle item that is shortened and an overflow menu.',
+ 'A segmented control that switches views without reloading the page.',
+ 'An empty state with a slot for an illustration, a short explanation and one clear action.',
+ 'A media uploader with thumbnail previews, a progress bar and a remove button on each item.',
+ 'A form field with inline validation, help text under the label and an error message that says what to do.',
+ 'A pagination control with first, previous, next and last, and a choice of page size.',
+ ]),
+ 'design.svg': Object.freeze([
+ 'A flat illustration of a bicycle leaning against a lamppost at dusk.',
+ 'A line-art icon set of five kitchen tools, drawn in one consistent weight.',
+ 'A poster-style illustration of a whale swimming through clouds.',
+ 'A map-style diagram of a fictional park with four paths and a pond in the middle.',
+ 'A vintage travel poster for a fictional coastal railway line.',
+ 'Six weather glyphs drawn as one geometric set.',
+ 'A cutaway diagram of a beehive with its parts named.',
+ 'A minimal geometric emblem for a fictional makerspace.',
+ 'An illustrated scene of a rainy bus stop with a shelter, a bench and a puddle.',
+ 'A botanical study of a fern frond in layered greens.',
+ 'A chess diagram of a puzzle position, shown from White to move.',
+ 'A technical drawing of a folding chair with dimension lines.',
+ 'A diagram of the water cycle with labelled arrows.',
+ 'An ornamental frame with a silhouette shape inside it, drawn so that it could be anyone and no one.',
+ 'A comic-style burst shape with the word SALE hand-lettered across it.',
+ 'Three progress rings showing three different values.',
+ 'An isometric illustration of a small workshop desk with tools on it.',
+ 'A constellation chart with the stars connected and the Greek letters beside them.',
+ 'A food icon set: a loaf, a bowl, a whisk, a jar and a board.',
+ 'A stylized mountain landscape in layered flat shapes with a moon above it.',
+ ]),
+ 'design.motion': Object.freeze([
+ 'A hero whose headline rises twenty pixels and fades in, then its sub-line, then the button, each after the other.',
+ 'A grid of cards that come in one after another as the page scrolls, sixty milliseconds apart.',
+ 'A button with a ripple that grows from the point that was clicked.',
+ 'A loading mark that changes shape every nine hundred milliseconds, from one form to another.',
+ 'An accordion that opens and closes with a smooth change of height.',
+ 'A hero video that pauses when it scrolls out of sight and plays again when it comes back.',
+ 'A tab underline that slides from one tab to the other.',
+ 'A number that counts up when its section scrolls into view.',
+ 'A toast that slides in from the right and fades away after four seconds.',
+ 'A parallax illustration whose background moves at half the speed of the page.',
+ 'A burst of confetti after a successful sign-up, made small for visitors who ask for less movement.',
+ 'A soft glow that follows the pointer, easing behind it.',
+ 'A shimmer that sweeps across placeholder blocks while they load.',
+ 'A chart that draws its line from left to right as it appears.',
+ 'A slow strip of logos that stops when the pointer is over it.',
+ 'A sheet that can be dragged away, follows the finger and snaps back when released early.',
+ 'A headline revealed character by character, as if it were being typed.',
+ 'A modal that grows from a slightly smaller size as it fades in, over a backdrop that goes soft.',
+ 'A thin progress bar across the top of the page showing how far down the reader is.',
+ 'A card that turns in three dimensions when the pointer is over it.',
+ ]),
+ 'design.ui': Object.freeze([
+ 'A banking dashboard: the balance, a small spending chart and the pending transactions.',
+ 'A train ticket booking flow: the date, the seat map and the fare summary.',
+ 'A hospital appointment screen: a short list of triage questions, a summary and a consent box.',
+ 'A warehouse picking app: a list of shelves, a scan field and a progress bar.',
+ 'A support inbox: the threads on the left, the message in the middle and a reply box below it.',
+ 'A recipe app in cooking mode: one step at a time with a timer, in a layout that reads from across a counter.',
+ 'A fleet monitoring console: a map, the list of vehicles and an alert panel.',
+ 'A course dashboard: progress so far, the deadlines ahead and the grades.',
+ 'A music player: what is playing, the queue and the playback controls.',
+ 'A calendar week view with a booking panel and a row of filters.',
+ 'A stock control screen: the stock table and a panel for reordering.',
+ 'A school portal: the timetable, an attendance card and the notice board.',
+ 'A point of sale screen: the products in a grid, the basket and the payment panel.',
+ 'A fitness summary: three rings, a week of bars and the list of goals.',
+ 'A library catalogue: the search with its filters, the results and the detail pane.',
+ 'A checkout: the address form, the shipping choices and the price broken down.',
+ 'A control room for a water plant: the gauges, the alarm list and a trend chart.',
+ 'A food delivery tracker: where the order is, which courier is bringing it and the messages.',
+ 'A travel itinerary: the days one after another, with the bookings and a map.',
+ 'A video call: the people in a grid, the chat beside it and the controls below.',
+ ]),
+ 'design.dataviz': Object.freeze([
+ 'A chart of monthly rainfall for three towns, with a legend and axes that can be read.',
+ 'A stacked bar chart of energy use by sector over four years.',
+ 'A map of renewable energy capacity across countries, with a tooltip and a filter.',
+ 'A line chart of daily temperature with a band showing the usual range.',
+ 'A scatter plot of reading time against book length, coloured by genre.',
+ 'A heat map of bus delays by hour and by weekday.',
+ 'A treemap of a city budget split by department.',
+ 'A dashboard of four charts that share the same filters.',
+ 'A population pyramid comparing two towns.',
+ 'A flow diagram of where a city water goes, the width of each band carrying the amount.',
+ 'A slope chart of five countries scores in two years, twenty years apart.',
+ 'A calendar heat map of how many issues were closed each day.',
+ 'A box plot comparing the delivery times of five couriers.',
+ 'A choropleth of fictional districts by median income, in a palette that reads without colour.',
+ 'A gauge of a service level target with the threshold marked.',
+ 'A sankey diagram of energy flowing from where it is made to where it is used.',
+ 'A grouped bar chart of survey answers across three age groups.',
+ 'A beeswarm plot of delivery ratings, each dot nudged aside so none hides another.',
+ 'A grid of twelve small charts, one for each region.',
+ 'A single headline number with the trend beside it and the target it is measured against.',
+ ]),
+ 'design.model3d': Object.freeze([
+ 'A low-poly wooden stool: four legs and a round seat.',
+ 'A simple tree with a trunk and layered foliage.',
+ 'A coffee mug with a handle, standing on a saucer.',
+ 'A stack of three books, each a little different in size.',
+ 'A classic desk lamp with an arm that bends.',
+ 'A bicycle with thin wheels, a frame, handlebars and a saddle.',
+ 'A plant pot in clay, with a rim and a saucer under it.',
+ 'A small wind turbine: a tower and three blades.',
+ 'A cardboard box with its flaps open.',
+ 'A park bench with wooden slats and metal legs.',
+ 'A teapot with a round body, a spout and a lid.',
+ 'A paper aeroplane with folded wings.',
+ 'A chess rook on a turned base.',
+ 'A watering can with a long spout and a handle.',
+ 'A low-poly mountain with snow on the top.',
+ 'A suitcase with a handle, wheels and a strap around it.',
+ 'A lighthouse with a striped tower and a lamp room.',
+ 'A small house with a pitched roof, a door and two windows.',
+ 'A camera on a tripod with a long lens.',
+ 'A shipping container with ribbed sides and corner fittings.',
+ ]),
+ 'design.shader': Object.freeze([
+ 'A starfield that drifts, with the nearer stars moving faster than the far ones.',
+ 'A soft glow that follows the mouse.',
+ 'Lava flowing, built from layers of noise moving at different speeds.',
+ 'A water surface that ripples, bending what is drawn behind it.',
+ 'A checkerboard that fades into a diagonal gradient.',
+ 'Heat haze above a field of moving waves.',
+ 'A field of colour built from overlapping sine waves.',
+ 'Concentric rings pulsing out from the centre.',
+ 'A shadow that stretches and softens as the light moves.',
+ 'The wall of a cave, drawn procedurally from layered noise.',
+ 'An image repeated around the centre in a kaleidoscope.',
+ 'Contour lines drawn over a noise field.',
+ 'An aurora, hanging in vertical curtains.',
+ 'A texture that breaks apart into blocks and disappears.',
+ 'A picture reduced to pixel art in a palette of eight colours.',
+ 'Flat coloured cells with thin borders between them, like a stained glass window.',
+ 'A curved screen of an old monitor, with scan lines across it.',
+ 'A soft-edged blob that follows the cursor and keeps a little of its own momentum.',
+ 'A contour map of a landscape, drawn from layered noise.',
+ 'A field of sparks that twinkle at random.',
+ ]),
+ 'design.interface': Object.freeze([
+ 'A ticket app system: buttons, badges and a status chip, all from one set.',
+ 'The form patterns of a checkout: fields, their validation states and a summary block.',
+ 'A navigation system: a top bar, a side rail and a breadcrumb, sharing one spacing scale.',
+ 'A catalogue card: picture, title, details, the actions and the empty version of it.',
+ 'A data table kit: sorting, filtering, pagination and the skeleton it shows while loading.',
+ 'A notification centre: a toast, a banner and an in-page alert, in one palette.',
+ 'A modal and a drawer for a settings area, sharing the same header and footer.',
+ 'A stepper and a set of progress marks for an onboarding flow.',
+ 'A profile card, a set of avatars and an identity block for a community app.',
+ 'A filter panel for a search page: chips, sliders and a way to reset everything.',
+ 'An icon set for a productivity app, with a naming convention written down.',
+ 'A pricing table with three tiers, the features side by side and one tier picked out.',
+ 'A dashboard kit: a number tile, a small chart and a list of statuses.',
+ 'A media kit: a picture, a video and a gallery, each with a caption.',
+ 'An onboarding kit: welcome screens, a short tour and an empty state.',
+ 'A chart set for an admin panel: axes, a legend and a tooltip.',
+ 'A feedback kit: a thumbs up and down, a comment box and a prompt to rate.',
+ 'A settings kit: a toggle, a choice, a slider and a place for the dangerous actions.',
+ 'A chat kit: message bubbles, an input row and a typing indicator.',
+ 'A landing page kit: the hero, a grid of features, the pricing, the questions and the footer.',
+ ]),
+});
+
+/** The kinds the daily job makes a round for: the nine design kinds. */
+const DAILY_KIND_IDS = Object.freeze(Object.keys(DAILY_PROMPTS));
+
+module.exports = { DAILY_PROMPTS, DAILY_KIND_IDS };
new file mode 100644
new file mode 100644
@@ -0,0 +1,229 @@
@@ -0,0 +1,229 @@
+'use strict';
+
+/**
+ * The daily rounds (plan L51, step 2): one round per design kind per day, so there is always something to judge.
+ *
+ * The job is off until the owner turns it on (BENCH_DAILY_ROUNDS), it runs at one fixed UTC hour
+ * (BENCH_DAILY_ROUNDS_HOUR_UTC) and it spends Bench's own budget (BENCH_DAILY_BUDGET_USD, one US dollar a day by
+ * default). A kind whose share of the budget is gone is skipped, and what it spent is written down, so the owner can
+ * see what the day's rounds cost. The two models of every round come from the pool the ordinary rounds already use.
+ *
+ * Nothing here starts under BENCH_DRILL (the drill and the tests), unless the drill explicitly allows it
+ * (BENCH_DAILY_ROUNDS_IN_DRILL). The tests call runDay() with a clock of their own instead.
+ */
+
+const crypto = require('crypto');
+const { DAILY_KIND_IDS, DAILY_PROMPTS } = require('./daily-prompts');
+
+/** The owner of the daily rounds. It is a service, not a person: no person can judge or create them. */
+const SUBJECT = 'service:bench-daily';
+
+const num = (v, d = 0) => (v == null || Number.isNaN(Number(v)) ? d : Number(v));
+
+function createDailyRounds({ s, bench, ai, config, anon = null, log = console }) {
+ const db = s.db;
+ let timer = null;
+ let stopped = false;
+ let running = false;
+ let prunedDay = null;
+
+ const cfg = () => (config && config.bench && config.bench.daily) || {};
+ const dayOf = (now) => new Date(now == null ? s.now() : now).toISOString().slice(0, 10);
+
+ /** The prompt of the day for a kind: the same one all day, a different one tomorrow. */
+ function promptFor(kindId, day) {
+ const list = DAILY_PROMPTS[kindId] || [];
+ if (!list.length) return null;
+ const h = crypto.createHash('sha256').update(`${day}|${kindId}`).digest();
+ return list[h.readUInt32BE(0) % list.length];
+ }
+
+ /** The two models of the pool the ordinary rounds use. */
+ async function modelsOfPool() {
+ let list = [];
+ try {
+ if (typeof ai.slotPlan === 'function') {
+ const plan = await ai.slotPlan();
+ list = plan && Array.isArray(plan.slots) ? plan.slots : [];
+ }
+ } catch (err) {
+ log.warn('[OpenVibe.Bench] daily rounds: the model pool is unavailable:', (err && err.message) || err);
+ }
+ if (list.length < 2) {
+ try { list = await ai.offeredModels(); } catch { list = []; }
+ }
+ const out = [];
+ const seen = new Set();
+ for (const one of list) {
+ const provider = String((one && one.provider) || '').trim();
+ const model = one && one.model != null ? String(one.model) : null;
+ if (!provider) continue;
+ const key = provider + '|' + (model || '');
+ if (seen.has(key)) continue;
+ seen.add(key);
+ out.push({ provider, model });
+ if (out.length === 2) break;
+ }
+ return out;
+ }
+
+ /** What one round is assumed to cost: the configured estimate, or what these models really cost. */
+ async function estimateFor(models) {
+ const base = num(cfg().estimateUsd, 0.05);
+ const names = models.map((m) => m.model).filter(Boolean);
+ if (!names.length) return base;
+ const avg = num(await db.value(
+ `SELECT COALESCE(AVG(cost_usd), 0)::numeric AS n FROM bench_entries
+ WHERE model = ANY($1::text[]) AND created_at >= $2::timestamptz`,
+ [names, new Date(s.now() - 7 * 86_400_000).toISOString()],
+ ));
+ return Math.max(base, Math.round(avg * models.length * 1e6) / 1e6);
+ }
+
+ /** What the day has cost: what was really spent, plus what is still running. */
+ async function budgetState(day) {
+ const rows = await db.many('SELECT * FROM bench_daily_rounds WHERE day = $1', [day]);
+ let spent = 0;
+ let reserved = 0;
+ for (const row of rows) {
+ if (String(row.status) === 'skipped' || String(row.status) === 'failed') continue;
+ if (row.spent_usd == null) reserved += num(row.estimated_usd);
+ else spent += num(row.spent_usd);
+ }
+ return { spent, reserved, rows };
+ }
+
+ /** What the day's rounds really cost, once their entries are in. */
+ async function settle(day) {
+ const rows = await db.many(
+ "SELECT * FROM bench_daily_rounds WHERE day = $1 AND status = 'created' AND spent_usd IS NULL",
+ [day],
+ );
+ for (const row of rows) {
+ if (!row.round_id) continue;
+ const usable = num(await db.value(
+ 'SELECT COUNT(*)::int AS n FROM bench_entries WHERE round_id = $1 AND error IS NULL', [row.round_id]));
+ if (usable < 2) continue;
+ const spent = num(await db.value(
+ 'SELECT COALESCE(SUM(cost_usd), 0)::numeric AS n FROM bench_entries WHERE round_id = $1', [row.round_id]));
+ await db.exec('UPDATE bench_daily_rounds SET spent_usd = $1 WHERE id = $2', [spent, row.id]);
+ }
+ }
+
+ async function record({ day, kindId, prompt, models, estimated, status, reason = null, taskId = null, roundId = null }) {
+ const id = s.newId('bdr');
+ await db.exec(
+ `INSERT INTO bench_daily_rounds (id, day, kind_id, prompt, models, task_id, round_id, estimated_usd, spent_usd, status, skip_reason, created_at)
+ VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10, $11, $12::timestamptz)
+ ON CONFLICT (day, kind_id) DO NOTHING`,
+ [id, day, kindId, prompt || null, JSON.stringify(models || []), taskId, roundId, estimated,
+ status === 'skipped' ? 0 : null, status, reason ? String(reason).slice(0, 200) : null, s.iso()],
+ );
+ return id;
+ }
+
+ /**
+ * One day of rounds: one per design kind, unless the day's budget is spent. A kind that already has a row for
+ * today is left alone, so the job can be run again without making a second round of the same kind.
+ */
+ async function runDay({ now = s.now() } = {}) {
+ const day = dayOf(now);
+ const budget = num(cfg().budgetUsd, 1);
+ await settle(day);
+ const made = [];
+ for (const kindId of DAILY_KIND_IDS) {
+ const done = await db.maybe('SELECT id FROM bench_daily_rounds WHERE day = $1 AND kind_id = $2', [day, kindId]);
+ if (done) continue;
+ const prompt = promptFor(kindId, day);
+ if (!prompt) continue;
+ const models = await modelsOfPool();
+ if (models.length < 2) {
+ await record({ day, kindId, prompt, models, estimated: 0, status: 'skipped', reason: 'no_models' });
+ made.push({ kind_id: kindId, status: 'skipped', reason: 'no_models' });
+ continue;
+ }
+ const estimate = await estimateFor(models);
+ const state = await budgetState(day);
+ if (state.spent + state.reserved + estimate > budget) {
+ await record({ day, kindId, prompt, models, estimated: estimate, status: 'skipped', reason: 'budget' });
+ made.push({ kind_id: kindId, status: 'skipped', reason: 'budget' });
+ continue;
+ }
+ try {
+ const task = await bench.createTask({ kindId, prompt, taskClass: 'daily', public: true, subject: SUBJECT });
+ if (!task.ok) throw new Error(task.message || 'the task was refused');
+ const round = await bench.createRound({ taskId: task.task.id, offers: models, subject: SUBJECT, bypassQuota: true });
+ if (!round.ok) throw new Error(round.message || 'the round was refused');
+ await record({
+ day, kindId, prompt, models, estimated: estimate, status: 'created',
+ taskId: task.task.id, roundId: round.round.id,
+ });
+ made.push({ kind_id: kindId, status: 'created', round_id: round.round.id, estimated_usd: estimate });
+ } catch (err) {
+ log.warn('[OpenVibe.Bench] daily round failed:', (err && err.message) || err);
+ await record({
+ day, kindId, prompt, models, estimated: estimate, status: 'failed',
+ reason: String((err && err.message) || err),
+ });
+ made.push({ kind_id: kindId, status: 'failed' });
+ }
+ }
+ const state = await budgetState(day);
+ const summary = {
+ day, budget_usd: budget, spent_usd: state.spent, reserved_usd: state.reserved,
+ created: made.filter((m) => m.status === 'created').length,
+ skipped: made.filter((m) => m.status === 'skipped').length,
+ rounds: made,
+ };
+ log.info(`[OpenVibe.Bench] daily rounds ${day}: ${summary.created} created, ${summary.skipped} skipped, $${state.spent.toFixed(4)} spent of $${budget.toFixed(2)}.`);
+ return summary;
+ }
+
+ /** One pass of the job: after the day's hour, and only if the day has no rounds yet. */
+ async function tick() {
+ if (running || stopped) return null;
+ running = true;
+ try {
+ const now = s.now();
+ const day = dayOf(now);
+ const hour = new Date(now).getUTCHours();
+ // The daily counters of the anonymous vote are dropped once a day: the hash of yesterday's address
+ // is worth nothing today.
+ if (anon && prunedDay !== day) {
+ prunedDay = day;
+ try { await anon.prune(); } catch (err) { log.warn('[OpenVibe.Bench] daily rounds: the old counters stayed:', (err && err.message) || err); }
+ }
+ if (hour < num(cfg().hourUtc, 8)) return null;
+ const rows = num(await db.value('SELECT COUNT(*)::int AS n FROM bench_daily_rounds WHERE day = $1', [day]));
+ if (rows >= DAILY_KIND_IDS.length) {
+ await settle(day);
+ return null;
+ }
+ return await runDay({ now });
+ } finally {
+ running = false;
+ }
+ }
+
+ /** The job starts only when the owner turns it on, and never under the drill unless the drill allows it. */
+ function start() {
+ if (timer || stopped) return null;
+ if (!cfg().enabled) return null;
+ if (config && config.drill && !cfg().enabledInDrill) return null;
+ const run = () => tick().catch((err) => log.warn('[OpenVibe.Bench] daily rounds:', (err && err.message) || err));
+ timer = setInterval(run, num(cfg().pollMs, 300_000));
+ if (timer.unref) timer.unref();
+ run();
+ return timer;
+ }
+
+ function stop() {
+ stopped = true;
+ if (timer) clearInterval(timer);
+ timer = null;
+ }
+
+ return { start, stop, tick, runDay, settle, promptFor, modelsOfPool, budgetState, dayOf, SUBJECT };
+}
+
+module.exports = { createDailyRounds, SUBJECT };
@@ -9,6 +9,8 @@
@@ -9,6 +9,8 @@
require('dotenv').config();
require('dotenv').config();
const trim = (s) => String(s || '').replace(/\/+$/, '');
const trim = (s) => String(s || '').replace(/\/+$/, '');
const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);
const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);
+const float = (v, def) => (Number.isFinite(parseFloat(v)) ? parseFloat(v) : def);
+const bool = (v, def) => (v == null || v === '' ? def : (v === '1' || v === 'true' || v === 'yes' || v === 'on'));
function load(env = process.env) {
function load(env = process.env) {
const nodeEnv = env.NODE_ENV || 'development';
const nodeEnv = env.NODE_ENV || 'development';
@@ -89,6 +91,42 @@ function load(env = process.env) {
@@ -89,6 +91,42 @@ function load(env = process.env) {
battlePersonPerDay: Math.max(1, int(env.BENCH_BATTLE_PERSON_PER_DAY, 200)),
battlePersonPerDay: Math.max(1, int(env.BENCH_BATTLE_PERSON_PER_DAY, 200)),
battleNetworkPerHour: Math.max(1, int(env.BENCH_BATTLE_NETWORK_PER_HOUR, 120)),
battleNetworkPerHour: Math.max(1, int(env.BENCH_BATTLE_NETWORK_PER_HOUR, 120)),
battleNetworkPerDay: Math.max(1, int(env.BENCH_BATTLE_NETWORK_PER_DAY, 1000)),
battleNetworkPerDay: Math.max(1, int(env.BENCH_BATTLE_NETWORK_PER_DAY, 1000)),
+
+ // Signed-out voting (plan L51 step 1). The caps are per UTC day and every one of them answers
+ // 429 bench.limit_anon with the same plain detail: sign in to keep voting today.
+ anon: {
+ enabled: bool(env.BENCH_ANON_ENABLED, true),
+ // Per anonymous voter per day.
+ voterPerDay: Math.max(1, int(env.BENCH_ANON_VOTER_PER_DAY, 30)),
+ // Per address per day. Bench keeps only a salted hash of the address, salted per day.
+ ipPerDay: Math.max(1, int(env.BENCH_ANON_IP_PER_DAY, 60)),
+ // Per relaying service per day.
+ relayPerDay: Math.max(1, int(env.BENCH_ANON_RELAY_PER_DAY, 3000)),
+ // Every anonymous vote on the site, per day.
+ globalPerDay: Math.max(1, int(env.BENCH_ANON_GLOBAL_PER_DAY, 20000)),
+ // The first-party services that may relay an anonymous voter (X-OV-Voter).
+ relayServices: String(env.BENCH_ANON_RELAY_SERVICES || 'design').split(',').map((x) => x.trim().toLowerCase()).filter(Boolean),
+ // The secret the voter cookie is signed with and the voter id is hashed with. Empty falls back to
+ // the OAuth client secret, and then to a per-process secret (which resets the caps on a restart).
+ secret: String(env.BENCH_ANON_SECRET || '').trim(),
+ cookieName: String(env.BENCH_ANON_COOKIE || 'bench_anon').trim() || 'bench_anon',
+ cookieDays: Math.max(1, int(env.BENCH_ANON_COOKIE_DAYS, 180)),
+ },
+ // The daily rounds (plan L51 step 2): one round per design kind per day, so there is always
+ // something to judge. Off until the owner turns it on in production.
+ daily: {
+ enabled: bool(env.BENCH_DAILY_ROUNDS, false),
+ // The job never starts under BENCH_DRILL unless this is set too (the tests use it).
+ enabledInDrill: bool(env.BENCH_DAILY_ROUNDS_IN_DRILL, false),
+ // The one UTC hour the day's rounds are made.
+ hourUtc: Math.min(23, Math.max(0, int(env.BENCH_DAILY_ROUNDS_HOUR_UTC, 8))),
+ // What the day's rounds may cost, in US dollars.
+ budgetUsd: Math.max(0, float(env.BENCH_DAILY_BUDGET_USD, 1.00)),
+ // What one round is assumed to cost, until the real cost of the models is known.
+ estimateUsd: Math.max(0, float(env.BENCH_DAILY_ROUND_ESTIMATE_USD, 0.05)),
+ // How often the job looks at the clock.
+ pollMs: Math.max(30_000, int(env.BENCH_DAILY_POLL_MS, 300_000)),
+ },
},
},
};
};
}
}
@@ -12,7 +12,7 @@
@@ -12,7 +12,7 @@
* POST /rounds/:id/entries (a room submits outside entries with a service token)
* POST /rounds/:id/entries (a room submits outside entries with a service token)
* GET /judgements/next
* GET /judgements/next
* POST /rounds/:id/judgements
* POST /rounds/:id/judgements
- * GET /boards/:kindId?type=elo|pareto|cost
+ * GET /boards/:kindId?type=elo|pareto|cost&voters=people|anon
* GET /models/:model
* GET /models/:model
* GET /datasets/:kindId.jsonl
* GET /datasets/:kindId.jsonl
* POST /evals
* POST /evals
@@ -162,11 +162,39 @@ function createApi(ctx) {
@@ -162,11 +162,39 @@ function createApi(ctx) {
});
});
// ── Judgements ──────────────────────────────────────────
// ── Judgements ──────────────────────────────────────────
- r.get('/judgements/next', requirePerson, async (req, res) => {
+ /**
+ * The next pair to judge. A signed-in person gets the next round they have not judged and did not start;
+ * an anonymous voter gets one too, through Bench's own signed cookie or through the X-OV-Voter header a
+ * relaying service sends. The anonymous answer is 204 when there is nothing left to judge.
+ */
+ r.get('/judgements/next', async (req, res) => {
try {
try {
- const pair = await bench.nextJudgement({ kind: String(req.query.kind || ''), subject: subjectOf(req), random: ctx.random || Math.random });
+ const kind = String(req.query.kind || '');
+ const person = subjectOf(req);
+ if (person) {
+ const pair = await bench.nextJudgement({ kind, subject: person, random: ctx.random || Math.random });
+ res.set('Cache-Control', 'private, no-store');
+ if (!pair) return res.json({ pair: null, detail: 'There is no round waiting for your judgement right now.' });
+ return res.json(pair);
+ }
+ if (!ctx.anon.enabled()) {
+ return problem(res, 403, 'bench.anon_disabled', 'Voting without signing in is off at the moment.');
+ }
+ const anon = ctx.anon.resolve(req, req.principal);
+ if (!anon) {
+ return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services, or send a service token that may relay anonymous votes.');
+ }
+ const kindRow = kindById(kind);
+ if (!kindRow || kindRow.judge !== 'human_pairwise') {
+ return problem(res, 403, 'bench.anon_kind_not_allowed', 'Only pairwise kinds can be judged without signing in.');
+ }
+ if (anon.minted) ctx.anon.setCookie(res, anon.opaque);
+ const pair = await bench.nextJudgement({
+ kind, subject: anon.subject, random: ctx.random || Math.random,
+ extraCreators: anon.creator ? [anon.creator] : [],
+ });
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
- if (!pair) return res.json({ pair: null, detail: 'There is no round waiting for your judgement right now.' });
+ if (!pair) return res.status(204).end();
return res.json(pair);
return res.json(pair);
} catch (err) { return failed(res, err); }
} catch (err) { return failed(res, err); }
});
});
@@ -174,6 +202,36 @@ function createApi(ctx) {
@@ -174,6 +202,36 @@ function createApi(ctx) {
r.post('/rounds/:id/judgements', async (req, res) => {
r.post('/rounds/:id/judgements', async (req, res) => {
const b = req.body && typeof req.body === 'object' ? req.body : {};
const b = req.body && typeof req.body === 'object' ? req.body : {};
const roundId = String(req.params.id || '');
const roundId = String(req.params.id || '');
+ // A first-party service may relay an anonymous voter: its own service token, and the voter id it derived
+ // from its own cookie in the X-OV-Voter header. Only a service on the allow-list may do this.
+ let relayed = null;
+ try {
+ const maybe = ctx.anon.resolve(req, req.principal, { mint: false });
+ if (maybe && maybe.source === 'relay') relayed = maybe;
+ } catch (err) { return failed(res, err); }
+ if (relayed) {
+ if (!ctx.anon.enabled()) {
+ return problem(res, 403, 'bench.anon_disabled', 'Voting without signing in is off at the moment.');
+ }
+ try {
+ const round = await bench.getRound(roundId, { viewerSubject: null });
+ if (!round) return problem(res, 404, 'bench.round_not_found', 'No such round.');
+ const row = kindById(round.kind_id);
+ if (!row || row.judge !== 'human_pairwise') {
+ return problem(res, 403, 'bench.anon_kind_not_allowed', 'Only pairwise kinds can be judged without signing in.');
+ }
+ const out = await bench.castJudgement({
+ roundId, kindId: round.kind_id, judge: 'human_pairwise',
+ judgeName: b.judge_name, judgeVersion: b.judge_version,
+ pair: b.pair, outcome: b.outcome, score: b.score, voteCounts: b.vote_counts,
+ subject: relayed.subject, voterKind: 'anon',
+ extraCreators: relayed.creator ? [relayed.creator] : [],
+ quota: () => ctx.anon.takeQuota(relayed),
+ });
+ res.set('Cache-Control', 'private, no-store');
+ return res.status(201).json(out);
+ } catch (err) { return failed(res, err); }
+ }
let judge = typeof b.judge === 'string' && b.judge ? b.judge : null;
let judge = typeof b.judge === 'string' && b.judge ? b.judge : null;
if (!judge) {
if (!judge) {
const round = await bench.getRound(roundId, { viewerSubject: null });
const round = await bench.getRound(roundId, { viewerSubject: null });
@@ -210,6 +268,8 @@ function createApi(ctx) {
@@ -210,6 +268,8 @@ function createApi(ctx) {
type,
type,
category: typeof req.query.category === 'string' ? req.query.category : 'all',
category: typeof req.query.category === 'string' ? req.query.category : 'all',
period: typeof req.query.period === 'string' ? req.query.period : 'all',
period: typeof req.query.period === 'string' ? req.query.period : 'all',
+ // 'people' shows the signed-in votes alone, 'anon' the anonymous ones alone.
+ voters: typeof req.query.voters === 'string' ? req.query.voters : null,
});
});
if (!board) return problem(res, 404, 'bench.kind_not_found', 'No such kind.');
if (!board) return problem(res, 404, 'bench.kind_not_found', 'No such kind.');
res.set('Cache-Control', 'public, max-age=30');
res.set('Cache-Control', 'public, max-age=30');
@@ -158,16 +158,23 @@ function pane(slot, revealed) {
@@ -158,16 +158,23 @@ function pane(slot, revealed) {
+ frame + facts + '</figure>';
+ frame + facts + '</figure>';
}
}
-function voteForm(round, signedIn) {
+/**
+ * The vote form. A signed-in person votes as themselves. A visitor who has not signed in votes as the anonymous
+ * voter their browser holds: one vote per browser, a limit a day, and the pair stays blind until they vote.
+ */
+function voteForm(round, { signedIn = false, canVote = true } = {}) {
if (round.revealed) {
if (round.revealed) {
return '<section class="sc-vote sc-vote-done" data-vote-state="revealed">'
return '<section class="sc-vote sc-vote-done" data-vote-state="revealed">'
+ '<p>Thanks: the identities are open now.</p>'
+ '<p>Thanks: the identities are open now.</p>'
+ '<p><a href="/showcase/round/' + esc(round.id) + '?share=1">Share this pair</a></p></section>';
+ '<p><a href="/showcase/round/' + esc(round.id) + '?share=1">Share this pair</a></p></section>';
}
}
- if (!signedIn) {
+ if (!canVote) {
return '<section class="sc-vote" data-vote-state="signed-out">'
return '<section class="sc-vote" data-vote-state="signed-out">'
+ '<p><a href="/auth/login?next=' + encodeURIComponent('/showcase/round/' + round.id) + '">Sign in to vote</a></p></section>';
+ '<p><a href="/auth/login?next=' + encodeURIComponent('/showcase/round/' + round.id) + '">Sign in to vote</a></p></section>';
}
}
+ const note = signedIn
+ ? ''
+ : '<p class="sc-vote-note" data-vote-note="1">You are voting without signing in: one vote per browser, and a limit a day.</p>';
const game = round.is_game
const game = round.is_game
? '<fieldset class="sc-vote-more"><legend>More fun</legend>'
? '<fieldset class="sc-vote-more"><legend>More fun</legend>'
+ '<label><input type="radio" name="more_fun" value="A"> Pane A</label>'
+ '<label><input type="radio" name="more_fun" value="A"> Pane A</label>'
@@ -175,7 +182,8 @@ function voteForm(round, signedIn) {
@@ -175,7 +182,8 @@ function voteForm(round, signedIn) {
+ '<label><input type="radio" name="more_fun" value="tie"> Same</label></fieldset>'
+ '<label><input type="radio" name="more_fun" value="tie"> Same</label></fieldset>'
+ '<label class="sc-vote-check"><input type="checkbox" name="crashed" value="1"> One of them crashed</label>'
+ '<label class="sc-vote-check"><input type="checkbox" name="crashed" value="1"> One of them crashed</label>'
: '';
: '';
- return '<form class="sc-vote" method="post" action="/showcase/round/' + esc(round.id) + '" data-vote-form="1">'
+ return '<form class="sc-vote" method="post" action="/showcase/round/' + esc(round.id) + '" data-vote-form="1" data-vote-state="' + (signedIn ? 'signed-in' : 'anonymous') + '">'
+ + note
+ '<fieldset><legend>Which one wins?</legend>'
+ '<fieldset><legend>Which one wins?</legend>'
+ '<label><input type="radio" name="outcome" value="A" required> Pane A</label>'
+ '<label><input type="radio" name="outcome" value="A" required> Pane A</label>'
+ '<label><input type="radio" name="outcome" value="B"> Pane B</label>'
+ '<label><input type="radio" name="outcome" value="B"> Pane B</label>'
@@ -201,14 +209,15 @@ function galleryView(pairs) {
@@ -201,14 +209,15 @@ function galleryView(pairs) {
+ '</section>';
+ '</section>';
}
}
-function roundView(round, signedIn) {
+function roundView(round, { signedIn = false, canVote = true, note = '' } = {}) {
const panes = round.slots.map((s) => pane(s, round.revealed)).join('');
const panes = round.slots.map((s) => pane(s, round.revealed)).join('');
return '<section class="sc-showcase sc-round" data-showcase="round" data-round-id="' + esc(round.id) + '" data-kind="' + esc(round.kind_id) + '">'
return '<section class="sc-showcase sc-round" data-showcase="round" data-round-id="' + esc(round.id) + '" data-kind="' + esc(round.kind_id) + '">'
+ '<h1 class="sc-round-prompt" data-prompt="1">' + esc(round.prompt) + '</h1>'
+ '<h1 class="sc-round-prompt" data-prompt="1">' + esc(round.prompt) + '</h1>'
+ '<p class="sc-round-meta"><span class="sc-kind">' + esc(round.kind_name) + '</span>'
+ '<p class="sc-round-meta"><span class="sc-kind">' + esc(round.kind_name) + '</span>'
+ '<span class="sc-votes" data-votes="' + esc(round.votes) + '">' + esc(round.votes) + ' votes</span></p>'
+ '<span class="sc-votes" data-votes="' + esc(round.votes) + '">' + esc(round.votes) + ' votes</span></p>'
+ + (note ? '<p class="sc-vote-note" data-vote-note="1">' + esc(note) + '</p>' : '')
+ '<div class="sc-round-panes" data-panes="2">' + panes + '</div>'
+ '<div class="sc-round-panes" data-panes="2">' + panes + '</div>'
- + voteForm(round, signedIn)
+ + voteForm(round, { signedIn, canVote })
+ '</section>';
+ '</section>';
}
}
@@ -220,6 +229,16 @@ function createShowcaseRoutes(ctx) {
@@ -220,6 +229,16 @@ function createShowcaseRoutes(ctx) {
const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });
const problem = (res, status, code, detail) => http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });
const subjectOf = (req) => (req.principal && req.principal.kind === 'user' ? req.principal.subject : null);
const subjectOf = (req) => (req.principal && req.principal.kind === 'user' ? req.principal.subject : null);
+ /** Who is looking: the person who is signed in, or the anonymous voter the signed cookie names. */
+ async function viewerSubjectOf(req) {
+ const person = subjectOf(req);
+ if (person) return person;
+ try {
+ const anon = ctx.anon.resolve(req, req.principal, { mint: false });
+ return anon ? anon.subject : null;
+ } catch { return null; }
+ }
+
r.get('/pairs', async (req, res) => {
r.get('/pairs', async (req, res) => {
try {
try {
const pairs = await loadPairs(ctx, { kind: req.query.kind, limit: req.query.limit });
const pairs = await loadPairs(ctx, { kind: req.query.kind, limit: req.query.limit });
@@ -233,7 +252,7 @@ function createShowcaseRoutes(ctx) {
@@ -233,7 +252,7 @@ function createShowcaseRoutes(ctx) {
r.get('/rounds/:id', async (req, res) => {
r.get('/rounds/:id', async (req, res) => {
try {
try {
- const round = await loadRound(ctx, String(req.params.id || ''), subjectOf(req));
+ const round = await loadRound(ctx, String(req.params.id || ''), await viewerSubjectOf(req));
if (!round) return problem(res, 404, 'bench.round_not_found', 'No such round.');
if (!round) return problem(res, 404, 'bench.round_not_found', 'No such round.');
res.set('Cache-Control', round.revealed ? 'public, max-age=30' : 'private, no-store');
res.set('Cache-Control', round.revealed ? 'public, max-age=30' : 'private, no-store');
return res.json({ round });
return res.json({ round });
@@ -246,22 +265,43 @@ function createShowcaseRoutes(ctx) {
@@ -246,22 +265,43 @@ function createShowcaseRoutes(ctx) {
r.post('/rounds/:id/vote', async (req, res) => {
r.post('/rounds/:id/vote', async (req, res) => {
const b = req.body && typeof req.body === 'object' ? req.body : {};
const b = req.body && typeof req.body === 'object' ? req.body : {};
const subject = subjectOf(req);
const subject = subjectOf(req);
- if (!subject) return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services to vote.');
- if (req.principal.viaSession && !sameOrigin(req, config)) {
- return problem(res, 403, 'bench.cross_site', 'A signed-in request that changes something must come from bench.openvibe.services itself.');
- }
const map = { A: 'left', B: 'right', tie: 'tie', both_bad: 'both_bad' };
const map = { A: 'left', B: 'right', tie: 'tie', both_bad: 'both_bad' };
const outcome = map[String(b.outcome || '')];
const outcome = map[String(b.outcome || '')];
if (!outcome) return problem(res, 422, 'bench.invalid_outcome', 'The outcome must be A, B, tie or both_bad.');
if (!outcome) return problem(res, 422, 'bench.invalid_outcome', 'The outcome must be A, B, tie or both_bad.');
const round = await loadRound(ctx, String(req.params.id || ''), null);
const round = await loadRound(ctx, String(req.params.id || ''), null);
if (!round || !round.ready) return problem(res, 404, 'bench.round_not_found', 'No such round.');
if (!round || !round.ready) return problem(res, 404, 'bench.round_not_found', 'No such round.');
+ // The voter: the person who is signed in, or the anonymous voter the browser cookie names.
+ let voter = null;
+ if (subject) {
+ if (req.principal.viaSession && !sameOrigin(req, config)) {
+ return problem(res, 403, 'bench.cross_site', 'A signed-in request that changes something must come from bench.openvibe.services itself.');
+ }
+ voter = { subject, voterKind: 'person', extraCreators: [] };
+ } else {
+ if (!ctx.anon.enabled()) {
+ return problem(res, 403, 'bench.anon_disabled', 'Voting without signing in is off at the moment.');
+ }
+ if (!sameOrigin(req, config)) {
+ return problem(res, 403, 'bench.cross_site', 'A vote that changes something must come from bench.openvibe.services itself.');
+ }
+ const anon = ctx.anon.resolve(req, req.principal);
+ if (!anon) return problem(res, 401, 'bench.sign_in_required', 'Sign in at bench.openvibe.services to vote.');
+ if (anon.minted) ctx.anon.setCookie(res, anon.opaque);
+ voter = {
+ subject: anon.subject, voterKind: 'anon', extraCreators: [],
+ quota: () => ctx.anon.takeQuota(anon),
+ };
+ }
try {
try {
const out = await bench.castJudgement({
const out = await bench.castJudgement({
roundId: round.id,
roundId: round.id,
judge: 'human_pairwise',
judge: 'human_pairwise',
pair: { left: round.slots[0].entry_id, right: round.slots[1].entry_id },
pair: { left: round.slots[0].entry_id, right: round.slots[1].entry_id },
outcome,
outcome,
- subject,
+ subject: voter.subject,
+ voterKind: voter.voterKind,
+ extraCreators: voter.extraCreators,
+ quota: voter.quota || null,
});
});
if (round.is_game && (b.more_fun || b.crashed)) {
if (round.is_game && (b.more_fun || b.crashed)) {
const moreFun = b.more_fun === 'A' ? 1 : b.more_fun === 'B' ? -1 : 0;
const moreFun = b.more_fun === 'A' ? 1 : b.more_fun === 'B' ? -1 : 0;
@@ -270,7 +310,8 @@ function createShowcaseRoutes(ctx) {
@@ -270,7 +310,8 @@ function createShowcaseRoutes(ctx) {
judge: 'rubric',
judge: 'rubric',
judgeName: b.crashed ? 'crashed' : 'more_fun',
judgeName: b.crashed ? 'crashed' : 'more_fun',
score: b.crashed ? 1 : moreFun,
score: b.crashed ? 1 : moreFun,
- subject,
+ subject: voter.subject,
+ voterKind: voter.voterKind,
});
});
}
}
res.set('Cache-Control', 'private, no-store');
res.set('Cache-Control', 'private, no-store');
@@ -299,6 +340,8 @@ function createShowcaseRoutes(ctx) {
@@ -299,6 +340,8 @@ function createShowcaseRoutes(ctx) {
function createShowcasePages(ctx) {
function createShowcasePages(ctx) {
const r = express.Router();
const r = express.Router();
const { config, principal } = ctx;
const { config, principal } = ctx;
+ // The vote is a plain form post: the pair is judged without JavaScript.
+ r.use(express.urlencoded({ extended: false, limit: '16kb' }));
r.use(principal.middleware);
r.use(principal.middleware);
const viewerOf = (req) => (req.principal && req.principal.kind === 'user'
const viewerOf = (req) => (req.principal && req.principal.kind === 'user'
? { kind: 'user', subject: req.principal.subject, displayName: req.principal.displayName || req.principal.username }
? { kind: 'user', subject: req.principal.subject, displayName: req.principal.displayName || req.principal.username }
@@ -318,17 +361,25 @@ function createShowcasePages(ctx) {
@@ -318,17 +361,25 @@ function createShowcasePages(ctx) {
r.get('/round/:id', async (req, res) => {
r.get('/round/:id', async (req, res) => {
const subject = req.principal && req.principal.kind === 'user' ? req.principal.subject : null;
const subject = req.principal && req.principal.kind === 'user' ? req.principal.subject : null;
- const round = await loadRound(ctx, String(req.params.id || ''), subject);
+ // A visitor who has not signed in gets an anonymous voter id in a signed cookie, and may judge with it.
+ let anon = null;
+ if (!subject && ctx.anon.enabled()) {
+ try { anon = ctx.anon.resolve(req, req.principal); } catch { anon = null; }
+ if (anon && anon.minted) ctx.anon.setCookie(res, anon.opaque);
+ }
+ const viewerSubject = subject || (anon ? anon.subject : null);
+ const round = await loadRound(ctx, String(req.params.id || ''), viewerSubject);
if (!round) {
if (!round) {
return send(res, 404, { config, viewer: viewerOf(req), path: req.originalUrl, title: 'No such round', body: raw('<section><h1>No such round</h1></section>') });
return send(res, 404, { config, viewer: viewerOf(req), path: req.originalUrl, title: 'No such round', body: raw('<section><h1>No such round</h1></section>') });
}
}
+ const note = String(req.query.voted || '') === 'limit' ? 'Sign in to keep voting today.' : '';
return send(res, 200, {
return send(res, 200, {
config,
config,
viewer: viewerOf(req),
viewer: viewerOf(req),
path: req.originalUrl, // the navbar's current link and the canonical URL
path: req.originalUrl, // the navbar's current link and the canonical URL
title: 'Compare two outputs',
title: 'Compare two outputs',
description: 'Two models, one prompt, side by side.',
description: 'Two models, one prompt, side by side.',
- body: raw(roundView(round, Boolean(subject))),
+ body: raw(roundView(round, { signedIn: Boolean(subject), canVote: Boolean(subject) || Boolean(anon), note })),
});
});
});
});
@@ -336,20 +387,37 @@ function createShowcasePages(ctx) {
@@ -336,20 +387,37 @@ function createShowcasePages(ctx) {
const subject = req.principal && req.principal.kind === 'user' ? req.principal.subject : null;
const subject = req.principal && req.principal.kind === 'user' ? req.principal.subject : null;
const round = await loadRound(ctx, String(req.params.id || ''), null);
const round = await loadRound(ctx, String(req.params.id || ''), null);
if (!round) return res.redirect('/showcase');
if (!round) return res.redirect('/showcase');
- if (!subject) return res.redirect('/auth/login?next=' + encodeURIComponent('/showcase/round/' + round.id));
const map = { A: 'left', B: 'right', tie: 'tie', both_bad: 'both_bad' };
const map = { A: 'left', B: 'right', tie: 'tie', both_bad: 'both_bad' };
const outcome = map[String((req.body && req.body.outcome) || '')];
const outcome = map[String((req.body && req.body.outcome) || '')];
if (!outcome) return res.redirect('/showcase/round/' + round.id);
if (!outcome) return res.redirect('/showcase/round/' + round.id);
+ // The voter: the person who is signed in, or the anonymous voter the cookie names.
+ let voter = null;
+ if (subject) {
+ voter = { subject, voterKind: 'person' };
+ } else {
+ const login = '/auth/login?next=' + encodeURIComponent('/showcase/round/' + round.id);
+ if (!ctx.anon.enabled() || !sameOrigin(req, config)) return res.redirect(login);
+ let anon = null;
+ try { anon = ctx.anon.resolve(req, req.principal); } catch { anon = null; }
+ if (!anon) return res.redirect(login);
+ if (anon.minted) ctx.anon.setCookie(res, anon.opaque);
+ voter = { subject: anon.subject, voterKind: 'anon', quota: () => ctx.anon.takeQuota(anon) };
+ }
try {
try {
await ctx.bench.castJudgement({
await ctx.bench.castJudgement({
roundId: round.id,
roundId: round.id,
judge: 'human_pairwise',
judge: 'human_pairwise',
pair: { left: round.slots[0].entry_id, right: round.slots[1].entry_id },
pair: { left: round.slots[0].entry_id, right: round.slots[1].entry_id },
outcome,
outcome,
- subject,
+ subject: voter.subject,
+ voterKind: voter.voterKind,
+ quota: voter.quota || null,
});
});
} catch (err) {
} catch (err) {
ctx.log.warn('[OpenVibe.Bench] showcase page vote failed:', (err && err.message) || err);
ctx.log.warn('[OpenVibe.Bench] showcase page vote failed:', (err && err.message) || err);
+ if (err && err.status === 429 && err.code === 'bench.limit_anon') {
+ return res.redirect('/showcase/round/' + round.id + '?voted=limit');
+ }
}
}
return res.redirect('/showcase/round/' + round.id);
return res.redirect('/showcase/round/' + round.id);
});
});
@@ -15,6 +15,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
@@ -15,6 +15,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
() => { for (const t of timers) clearInterval(t); },
() => { for (const t of timers) clearInterval(t); },
() => ctx.keys.client.stop(),
() => ctx.keys.client.stop(),
() => ctx.jobs.stop(),
() => ctx.jobs.stop(),
+ () => ctx.daily.stop(),
() => ctx.publisher.stop(),
() => ctx.publisher.stop(),
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
() => ctx.s.close(),
() => ctx.s.close(),
@@ -42,6 +43,8 @@ async function start() {
@@ -42,6 +43,8 @@ async function start() {
// Jobs: rounds and private eval runs. Off under BENCH_DRILL; tests call tick().
// Jobs: rounds and private eval runs. Off under BENCH_DRILL; tests call tick().
ctx.jobs.start();
ctx.jobs.start();
+ // The daily rounds (one per design kind per day). Off until BENCH_DAILY_ROUNDS is on; never under the drill.
+ ctx.daily.start();
// Outbox publisher → OpenVibe.Events.
// Outbox publisher → OpenVibe.Events.
ctx.publisher.start();
ctx.publisher.start();
@@ -15,6 +15,9 @@ const styleLib = require('./rating/style');
@@ -15,6 +15,9 @@ const styleLib = require('./rating/style');
const BASE_ELO = ratingLib.BASE_ELO;
const BASE_ELO = ratingLib.BASE_ELO;
const K = 32;
const K = 32;
const CATEGORY_DEFAULT = 'all';
const CATEGORY_DEFAULT = 'all';
+// The rating categories: everyone ('all'), the signed-in people ('people') and the anonymous voters ('anon').
+const CATEGORY_PEOPLE = 'people';
+const CATEGORY_ANON = 'anon';
const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
@@ -101,6 +104,27 @@ function wireJudgement(row, { withSubject = false } = {}) {
@@ -101,6 +104,27 @@ function wireJudgement(row, { withSubject = false } = {}) {
return out;
return out;
}
}
+/**
+ * Who voted: a person, a first-party service, or an anonymous voter. An anonymous voter is a hash (anon_…), never
+ * an id, so the database never holds anything about the visitor.
+ */
+function kindOfVoter(voterKind, subject) {
+ if (voterKind === 'person' || voterKind === 'anon' || voterKind === 'service') return voterKind;
+ const s = String(subject || '');
+ if (!s) return 'service';
+ if (s.startsWith('anon_')) return 'anon';
+ if (s.startsWith('usr_')) return 'person';
+ return 'service';
+}
+
+/** The rating categories a vote is counted in: everyone, and the kind of voter it came from. */
+function categoriesOf(voterKind) {
+ const out = [CATEGORY_DEFAULT];
+ if (voterKind === 'person') out.push(CATEGORY_PEOPLE);
+ else if (voterKind === 'anon') out.push(CATEGORY_ANON);
+ return out;
+}
+
function wireRating(row) {
function wireRating(row) {
if (!row) return null;
if (!row) return null;
return {
return {
@@ -232,13 +256,25 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -232,13 +256,25 @@ function createStore(s, { log = console, random = Math.random, config = null } =
};
};
}
}
- async function createRound({ taskId, offers = null, seed = 0, subject }) {
+ async function createRound({ taskId, offers = null, seed = 0, subject, bypassQuota = false }) {
const task = await getTask(taskId);
const task = await getTask(taskId);
if (!task) return refusal(404, 'bench.task_not_found', 'No such task.');
if (!task) return refusal(404, 'bench.task_not_found', 'No such task.');
const kind = await getKind(task.kind_id);
const kind = await getKind(task.kind_id);
if (!kind) return refusal(422, 'bench.kind_not_found', 'The task\'s kind is gone.');
if (!kind) return refusal(422, 'bench.kind_not_found', 'The task\'s kind is gone.');
const off = Array.isArray(offers) ? offers.slice(0, 3).map((o) => ({ provider: String(o.provider || ''), model: o.model == null ? null : String(o.model) })) : [];
const off = Array.isArray(offers) ? offers.slice(0, 3).map((o) => ({ provider: String(o.provider || ''), model: o.model == null ? null : String(o.model) })) : [];
if (off.some((o) => !o.provider)) return refusal(422, 'bench.invalid_offers', 'Every offer needs a provider.');
if (off.some((o) => !o.provider)) return refusal(422, 'bench.invalid_offers', 'Every offer needs a provider.');
+ // The daily rounds job spends Bench's own budget, not a person's quota, so it may skip the counts.
+ if (bypassQuota) {
+ const ownId = s.newId('brd');
+ const own = await db.maybe(
+ `INSERT INTO bench_rounds (id, task_id, kind_id, offers, status, seed, params, pool_fallback, created_by, created_at)
+ VALUES ($1, $2, $3, $4::jsonb, 'queued', $5, '{}'::jsonb, $6, $7, $8::timestamptz)
+ RETURNING *`,
+ [ownId, task.id, kind.id, JSON.stringify(off), Math.max(0, Number(seed) || 0), off.length < 3, subject, s.iso()],
+ );
+ if (!own) throw new BenchError(500, 'bench.round_not_created', 'Bench could not create the round.');
+ return { ok: true, round: await getRound(ownId) };
+ }
const perDay = config ? config.bench.roundsPerDay : 3;
const perDay = config ? config.bench.roundsPerDay : 3;
const daily = config ? config.bench.roundsDaily : 20;
const daily = config ? config.bench.roundsDaily : 20;
const start = dayStart();
const start = dayStart();
@@ -338,14 +374,16 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -338,14 +374,16 @@ function createStore(s, { log = console, random = Math.random, config = null } =
}
}
/** A ready round this person has not judged and did not start, with two entries shuffled and identities hidden. */
/** A ready round this person has not judged and did not start, with two entries shuffled and identities hidden. */
- async function nextJudgement({ kind, subject, random: rng = random } = {}) {
+ async function nextJudgement({ kind, subject, random: rng = random, extraCreators = [] } = {}) {
+ const creators = (extraCreators || []).filter(Boolean);
const row = await db.maybe(
const row = await db.maybe(
`SELECT r.* FROM bench_rounds r
`SELECT r.* FROM bench_rounds r
WHERE r.kind_id = $1 AND r.status = 'finished' AND r.created_by <> $2
WHERE r.kind_id = $1 AND r.status = 'finished' AND r.created_by <> $2
+ AND NOT (r.created_by = ANY($3::text[]))
AND (SELECT COUNT(*) FROM bench_entries e WHERE e.round_id = r.id AND e.error IS NULL) >= 2
AND (SELECT COUNT(*) FROM bench_entries e WHERE e.round_id = r.id AND e.error IS NULL) >= 2
AND NOT EXISTS (SELECT 1 FROM bench_judgements j WHERE j.round_id = r.id AND j.subject = $2)
AND NOT EXISTS (SELECT 1 FROM bench_judgements j WHERE j.round_id = r.id AND j.subject = $2)
ORDER BY r.created_at, r.id LIMIT 1`,
ORDER BY r.created_at, r.id LIMIT 1`,
- [String(kind || ''), subject],
+ [String(kind || ''), subject, creators],
);
);
if (!row) return null;
if (!row) return null;
const entries = (await db.many('SELECT * FROM bench_entries WHERE round_id = $1 AND error IS NULL ORDER BY id', [row.id])).map(wireEntry);
const entries = (await db.many('SELECT * FROM bench_entries WHERE round_id = $1 AND error IS NULL ORDER BY id', [row.id])).map(wireEntry);
@@ -358,7 +396,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -358,7 +396,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =
};
};
}
}
- async function castJudgement({ roundId, kindId, judge, judgeName = null, judgeVersion = null, pair = null, outcome = null, score = null, voteCounts = null, question = null, answer = null, netHash = null, subject }) {
+ async function castJudgement({ roundId, kindId, judge, judgeName = null, judgeVersion = null, pair = null, outcome = null, score = null, voteCounts = null, question = null, answer = null, netHash = null, subject, voterKind = null, extraCreators = [], quota = null }) {
if (!JUDGES.includes(judge)) throw new BenchError(422, 'bench.invalid_judge', 'Unknown judge.');
if (!JUDGES.includes(judge)) throw new BenchError(422, 'bench.invalid_judge', 'Unknown judge.');
try {
try {
return await s.tx(async () => {
return await s.tx(async () => {
@@ -372,7 +410,13 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -372,7 +410,13 @@ function createStore(s, { log = console, random = Math.random, config = null } =
let left = null;
let left = null;
let right = null;
let right = null;
if (judge === 'human_pairwise' || judge === 'game_outcome' || judge === 'audience_poll') {
if (judge === 'human_pairwise' || judge === 'game_outcome' || judge === 'audience_poll') {
- if (judge === 'human_pairwise' && r.created_by === subject) throw new BenchError(403, 'bench.creator_cannot_vote', 'You cannot judge a round you started.');
+ // A voter never judges a round they started: a person never judges their own, and the
+ // service that relays an anonymous vote never judges a round it started itself.
+ const extra = (extraCreators || []).filter(Boolean);
+ const creators = [r.created_by, ...extra].filter(Boolean);
+ if (judge === 'human_pairwise' && (creators.includes(subject) || extra.includes(r.created_by))) {
+ throw new BenchError(403, 'bench.creator_cannot_vote', 'You cannot judge a round you started.');
+ }
const p = pair && typeof pair === 'object' ? pair : {};
const p = pair && typeof pair === 'object' ? pair : {};
if (!p.left || !p.right || p.left === p.right) throw new BenchError(422, 'bench.invalid_pair', 'A pairwise judgement needs two different entries.');
if (!p.left || !p.right || p.left === p.right) throw new BenchError(422, 'bench.invalid_pair', 'A pairwise judgement needs two different entries.');
const prior = await db.maybe('SELECT 1 AS n FROM bench_judgements WHERE round_id = $1 AND subject = $2', [r.id, subject]);
const prior = await db.maybe('SELECT 1 AS n FROM bench_judgements WHERE round_id = $1 AND subject = $2', [r.id, subject]);
@@ -394,18 +438,22 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -394,18 +438,22 @@ function createStore(s, { log = console, random = Math.random, config = null } =
} else if (voteCounts != null) throw new BenchError(422, 'bench.invalid_vote_counts', 'Only an audience poll carries vote counts.');
} else if (voteCounts != null) throw new BenchError(422, 'bench.invalid_vote_counts', 'Only an audience poll carries vote counts.');
}
}
+ // The daily caps of the anonymous vote are spent here, inside the vote's own transaction: a
+ // refused vote spends nothing, and a vote that is rolled back spends nothing either.
+ if (quota) await quota();
+
const id = s.newId('bju');
const id = s.newId('bju');
const row = await db.maybe(
const row = await db.maybe(
- `INSERT INTO bench_judgements (id, round_id, kind_id, judge, judge_name, judge_version, method, pair, outcome, score, subject, at, vote_counts, question, answer, net_hash)
- VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10, $11, $12::timestamptz, $13::jsonb, $14, $15, $16) RETURNING *`,
+ `INSERT INTO bench_judgements (id, round_id, kind_id, judge, judge_name, judge_version, method, pair, outcome, score, subject, at, vote_counts, question, answer, net_hash, voter_kind)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10, $11, $12::timestamptz, $13::jsonb, $14, $15, $16, $17) RETURNING *`,
[id, r.id, r.kind_id, judge, judgeName, judgeVersion, judge, pair ? JSON.stringify(pair) : null,
[id, r.id, r.kind_id, judge, judgeName, judgeVersion, judge, pair ? JSON.stringify(pair) : null,
outcome || null, score == null ? null : num(score), subject || null, s.iso(), voteCounts ? JSON.stringify(voteCounts) : null,
outcome || null, score == null ? null : num(score), subject || null, s.iso(), voteCounts ? JSON.stringify(voteCounts) : null,
question == null ? null : String(question).slice(0, 200), answer == null ? null : String(answer).slice(0, 40),
question == null ? null : String(question).slice(0, 200), answer == null ? null : String(answer).slice(0, 40),
- netHash == null ? null : String(netHash).slice(0, 64)],
+ netHash == null ? null : String(netHash).slice(0, 64), kindOfVoter(voterKind, subject)],
);
);
// One stream poll is one game, regardless of audience size. The counts are evidence, not Elo weight.
// One stream poll is one game, regardless of audience size. The counts are evidence, not Elo weight.
if (judge === 'human_pairwise' || judge === 'game_outcome' || (judge === 'audience_poll' && voteCounts.voters > 0)) {
if (judge === 'human_pairwise' || judge === 'game_outcome' || (judge === 'audience_poll' && voteCounts.voters > 0)) {
- await applyPairwiseRating(r.id, r.kind_id, left, right, outcome);
+ await applyPairwiseRating(r.id, r.kind_id, left, right, outcome, kindOfVoter(voterKind, subject));
}
}
await insertOutbox('bench.judgement.recorded', {
await insertOutbox('bench.judgement.recorded', {
judgement_id: id, round_id: r.id, kind_id: r.kind_id, judge, method: judge, outcome: outcome || null, score: score == null ? null : num(score),
judgement_id: id, round_id: r.id, kind_id: r.kind_id, judge, method: judge, outcome: outcome || null, score: score == null ? null : num(score),
@@ -418,42 +466,46 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -418,42 +466,46 @@ function createStore(s, { log = console, random = Math.random, config = null } =
}
}
}
}
- async function applyPairwiseRating(roundId, kindId, left, right, outcome) {
+ async function applyPairwiseRating(roundId, kindId, left, right, outcome, voterKind = 'person') {
if (outcome === 'both_bad') return;
if (outcome === 'both_bad') return;
if (!left || !right || left.model === right.model) return;
if (!left || !right || left.model === right.model) return;
const models = [left.model, right.model];
const models = [left.model, right.model];
const now = s.iso();
const now = s.iso();
- for (const model of models) {
- await db.exec(
- `INSERT INTO bench_ratings (kind_id, category, subject, elo, games, wins, losses, ties, win_rate, ci_lower, ci_upper, updated_at)
- VALUES ($1, $2, $3, $4, 0, 0, 0, 0, 0, NULL, NULL, $5::timestamptz)
- ON CONFLICT (kind_id, category, subject) DO NOTHING`,
- [kindId, CATEGORY_DEFAULT, model, BASE_ELO, now],
- );
- }
- const rows = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2 AND subject = ANY($3::text[]) ORDER BY subject FOR UPDATE',
- [kindId, CATEGORY_DEFAULT, models]);
- const byModel = new Map(rows.map((r) => [r.subject, r]));
- const a = byModel.get(left.model);
- const b = byModel.get(right.model);
- if (!a || !b) return;
- const eloOutcome = outcome === 'draw' ? 'tie' : outcome;
- const update = ratingLib.eloUpdate(num(a.elo, BASE_ELO), num(b.elo, BASE_ELO), eloOutcome, K);
- const sa = outcome === 'left' ? 1 : outcome === 'right' ? 0 : 0.5;
- const sb = 1 - sa;
- for (const [row, elo, score, opp] of [[a, update.a, sa, sb], [b, update.b, sb, sa]]) {
- const games = num(row.games) + 1;
- const wins = num(row.wins) + (score > opp ? 1 : 0);
- const losses = num(row.losses) + (score < opp ? 1 : 0);
- const ties = num(row.ties) + (score === opp ? 1 : 0);
- const ci = ratingCi(elo, wins, losses, ties);
- await db.exec(
- `UPDATE bench_ratings SET elo = $1, games = $2, wins = $3, losses = $4, ties = $5,
- win_rate = $6, ci_lower = $7, ci_upper = $8, updated_at = $9::timestamptz
- WHERE kind_id = $10 AND category = $11 AND subject = $12`,
- [elo, games, wins, losses, ties, games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0,
- ci.ci_lower, ci.ci_upper, now, kindId, CATEGORY_DEFAULT, row.subject],
- );
+ // The vote is counted twice: once for everyone ('all') and once for the kind of voter it came from
+ // ('people' or 'anon'), so a board can show the signed-in votes alone and still count the rest.
+ for (const category of categoriesOf(voterKind)) {
+ for (const model of models) {
+ await db.exec(
+ `INSERT INTO bench_ratings (kind_id, category, subject, elo, games, wins, losses, ties, win_rate, ci_lower, ci_upper, updated_at)
+ VALUES ($1, $2, $3, $4, 0, 0, 0, 0, 0, NULL, NULL, $5::timestamptz)
+ ON CONFLICT (kind_id, category, subject) DO NOTHING`,
+ [kindId, category, model, BASE_ELO, now],
+ );
+ }
+ const rows = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2 AND subject = ANY($3::text[]) ORDER BY subject FOR UPDATE',
+ [kindId, category, models]);
+ const byModel = new Map(rows.map((r) => [r.subject, r]));
+ const a = byModel.get(left.model);
+ const b = byModel.get(right.model);
+ if (!a || !b) continue;
+ const eloOutcome = outcome === 'draw' ? 'tie' : outcome;
+ const update = ratingLib.eloUpdate(num(a.elo, BASE_ELO), num(b.elo, BASE_ELO), eloOutcome, K);
+ const sa = outcome === 'left' ? 1 : outcome === 'right' ? 0 : 0.5;
+ const sb = 1 - sa;
+ for (const [row, elo, score, opp] of [[a, update.a, sa, sb], [b, update.b, sb, sa]]) {
+ const games = num(row.games) + 1;
+ const wins = num(row.wins) + (score > opp ? 1 : 0);
+ const losses = num(row.losses) + (score < opp ? 1 : 0);
+ const ties = num(row.ties) + (score === opp ? 1 : 0);
+ const ci = ratingCi(elo, wins, losses, ties);
+ await db.exec(
+ `UPDATE bench_ratings SET elo = $1, games = $2, wins = $3, losses = $4, ties = $5,
+ win_rate = $6, ci_lower = $7, ci_upper = $8, updated_at = $9::timestamptz
+ WHERE kind_id = $10 AND category = $11 AND subject = $12`,
+ [elo, games, wins, losses, ties, games ? Math.round(((wins + ties * 0.5) / games) * 10000) / 10000 : 0,
+ ci.ci_lower, ci.ci_upper, now, kindId, category, row.subject],
+ );
+ }
}
}
await insertOutbox('bench.rating.changed', { kind_id: kindId, category: CATEGORY_DEFAULT, subjects: models });
await insertOutbox('bench.rating.changed', { kind_id: kindId, category: CATEGORY_DEFAULT, subjects: models });
// The ratings have been refitted: this is where the watches are checked.
// The ratings have been refitted: this is where the watches are checked.
@@ -474,15 +526,15 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -474,15 +526,15 @@ function createStore(s, { log = console, random = Math.random, config = null } =
return rows.map((r) => r.model);
return rows.map((r) => r.model);
}
}
- async function boardElo(kindId) {
+ async function boardElo(kindId, category = CATEGORY_DEFAULT) {
const subjects = await subjectsFor(kindId);
const subjects = await subjectsFor(kindId);
- const ratings = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2', [kindId, CATEGORY_DEFAULT]);
+ const ratings = await db.many('SELECT * FROM bench_ratings WHERE kind_id = $1 AND category = $2', [kindId, category]);
const by = new Map(ratings.map((r) => [r.subject, wireRating(r)]));
const by = new Map(ratings.map((r) => [r.subject, wireRating(r)]));
const rows = subjects.map((subject) => {
const rows = subjects.map((subject) => {
const r = by.get(subject);
const r = by.get(subject);
const games = r ? r.games : 0;
const games = r ? r.games : 0;
return {
return {
- kind_id: kindId, category: CATEGORY_DEFAULT, subject,
+ kind_id: kindId, category, subject,
elo: r ? r.elo : BASE_ELO,
elo: r ? r.elo : BASE_ELO,
games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,
games, wins: r ? r.wins : 0, losses: r ? r.losses : 0, ties: r ? r.ties : 0,
win_rate: r ? r.win_rate : 0,
win_rate: r ? r.win_rate : 0,
@@ -524,12 +576,44 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -524,12 +576,44 @@ function createStore(s, { log = console, random = Math.random, config = null } =
return new Map(rows.map((r) => [r.model, r]));
return new Map(rows.map((r) => [r.model, r]));
}
}
- async function board(kindId, { type = 'elo', category = CATEGORY_DEFAULT, period = 'all' } = {}) {
+ /**
+ * Which rating category a board reads: everyone ('all'), the signed-in people ('people') or the anonymous
+ * voters ('anon'). The voters parameter wins over the older category parameter.
+ */
+ function categoryOf(voters, category) {
+ const v = String(voters == null ? '' : voters).trim().toLowerCase();
+ if (v === 'people' || v === 'person' || v === 'persons') return CATEGORY_PEOPLE;
+ if (v === 'anon' || v === 'anonymous') return CATEGORY_ANON;
+ const c = String(category == null ? '' : category).trim().toLowerCase();
+ if (c === 'people' || c === 'person') return CATEGORY_PEOPLE;
+ if (c === 'anon' || c === 'anonymous') return CATEGORY_ANON;
+ return CATEGORY_DEFAULT;
+ }
+
+ /** How many votes of each kind of voter a board holds. */
+ async function voterCounts(kindId) {
+ const rows = await db.many(
+ `SELECT COALESCE(voter_kind, 'person') AS kind, COUNT(*)::int AS n
+ FROM bench_judgements WHERE kind_id = $1 AND judge = 'human_pairwise'
+ GROUP BY 1`,
+ [kindId],
+ );
+ const out = { person: 0, anon: 0, service: 0, total: 0 };
+ for (const row of rows) {
+ const kind = String(row.kind || '');
+ if (kind in out) out[kind] = num(row.n);
+ out.total += num(row.n);
+ }
+ return out;
+ }
+
+ async function board(kindId, { type = 'elo', category = CATEGORY_DEFAULT, period = 'all', voters = null } = {}) {
const kind = await getKind(kindId);
const kind = await getKind(kindId);
if (!kind) return null;
if (!kind) return null;
+ const cat = categoryOf(voters, category);
const cost = await costStats(kindId);
const cost = await costStats(kindId);
const pass = await passRate(kindId);
const pass = await passRate(kindId);
- const eloRows = await boardElo(kindId);
+ const eloRows = await boardElo(kindId, cat);
let rows;
let rows;
if (type === 'elo') {
if (type === 'elo') {
@@ -561,7 +645,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -561,7 +645,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =
} else {
} else {
return null;
return null;
}
}
- return { kind_id: kindId, category, period, type, rows };
+ return { kind_id: kindId, category: cat, period, type, rows, voters: await voterCounts(kindId) };
}
}
/**
/**
@@ -710,4 +794,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =
@@ -710,4 +794,7 @@ function createStore(s, { log = console, random = Math.random, config = null } =
};
};
}
}
-module.exports = { createStore, BenchError, KINDS, JUDGES, WINNERS, BASE_ELO, CATEGORY_DEFAULT, shuffle };
+module.exports = {
+ createStore, BenchError, KINDS, JUDGES, WINNERS, BASE_ELO, CATEGORY_DEFAULT, shuffle,
+ CATEGORY_PEOPLE, CATEGORY_ANON, kindOfVoter, categoriesOf,
+};
@@ -57,7 +57,7 @@ ${ep('GET', '/api/v1/kinds', 'The benchmark kinds.')}
@@ -57,7 +57,7 @@ ${ep('GET', '/api/v1/kinds', 'The benchmark kinds.')}
${ep('GET', '/api/v1/kinds/:kindId', 'One kind and its entry schema and metrics.')}
${ep('GET', '/api/v1/kinds/:kindId', 'One kind and its entry schema and metrics.')}
${ep('GET', '/api/v1/tasks/:id', 'One task.')}
${ep('GET', '/api/v1/tasks/:id', 'One task.')}
${ep('GET', '/api/v1/rounds/:id', 'One round with entries; entries stay blind until the caller has judged or started the round.')}
${ep('GET', '/api/v1/rounds/:id', 'One round with entries; entries stay blind until the caller has judged or started the round.')}
-${ep('GET', '/api/v1/boards/:kindId?type=elo|pareto|cost', 'A rendered board.')}
+${ep('GET', '/api/v1/boards/:kindId?type=elo|pareto|cost&voters=people|anon', 'A rendered board. <code>voters=people</code> shows the signed-in votes alone, <code>voters=anon</code> the anonymous ones; every board says how many votes of each kind it holds.')}
${ep('GET', '/api/v1/models/:model', 'One model across every kind.')}
${ep('GET', '/api/v1/models/:model', 'One model across every kind.')}
${ep('GET', '/api/v1/datasets/:kindId.jsonl', 'The open dataset (prompts, entries, judgements without voter ids, ratings).')}
${ep('GET', '/api/v1/datasets/:kindId.jsonl', 'The open dataset (prompts, entries, judgements without voter ids, ratings).')}
${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its visible tasks and every run, with no person id.')}
${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its visible tasks and every run, with no person id.')}
@@ -66,8 +66,8 @@ ${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its vi
@@ -66,8 +66,8 @@ ${ep('GET', '/api/v1/datasets/benchmarks.jsonl', 'Every public benchmark, its vi
${ep('POST', '/api/v1/kinds/:kindId/tasks', '<code>{ prompt, task_class?, public? }</code>. A person or a service with bench.task.create.')}
${ep('POST', '/api/v1/kinds/:kindId/tasks', '<code>{ prompt, task_class?, public? }</code>. A person or a service with bench.task.create.')}
${ep('POST', '/api/v1/tasks/:id/rounds', '<code>{ offers?, seed? }</code>. Pins a reproducible round; the job generates entries.')}
${ep('POST', '/api/v1/tasks/:id/rounds', '<code>{ offers?, seed? }</code>. Pins a reproducible round; the job generates entries.')}
${ep('POST', '/api/v1/rounds/:id/entries', 'A room submits outside entries (a game match, a test-suite run) with a service token carrying bench.entry.create.')}
${ep('POST', '/api/v1/rounds/:id/entries', 'A room submits outside entries (a game match, a test-suite run) with a service token carrying bench.entry.create.')}
-${ep('GET', '/api/v1/judgements/next?kind=&judge=', 'The next blind pair for this person.')}
-${ep('POST', '/api/v1/rounds/:id/judgements', '<code>{ judge?, pair?, outcome?, score?, vote_counts? }</code>. Human judges are people; automatic/test/game judges and audience_poll are services with bench.judgement.cast. An audience_poll supplies { left, right, tie, voters }; it changes Elo as one game regardless of audience size. A zero-voter poll is recorded without changing ratings.')}
+${ep('GET', '/api/v1/judgements/next?kind=&judge=', 'The next blind pair for this person, or for an anonymous voter (the signed cookie, or the X-OV-Voter header a relaying service sends).')}
+${ep('POST', '/api/v1/rounds/:id/judgements', '<code>{ judge?, pair?, outcome?, score?, vote_counts? }</code>. Human judges are people; automatic/test/game judges and audience_poll are services with bench.judgement.cast. An audience_poll supplies { left, right, tie, voters }; it changes Elo as one game regardless of audience size. A zero-voter poll is recorded without changing ratings. A service on the relay allow-list may cast the vote of an anonymous voter it names in <code>X-OV-Voter: anon:<id></code>.')}
${ep('POST', '/api/v1/evals', 'Create a private task suite (never in the public dataset).')}
${ep('POST', '/api/v1/evals', 'Create a private task suite (never in the public dataset).')}
${ep('POST', '/api/v1/evals/:id/run', 'Run a private eval against chosen offers, under the free daily cap.')}
${ep('POST', '/api/v1/evals/:id/run', 'Run a private eval against chosen offers, under the free daily cap.')}
${ep('GET', '/api/v1/evals/:id/results', 'A private eval and its runs.')}
${ep('GET', '/api/v1/evals/:id/results', 'A private eval and its runs.')}
new file mode 100644
new file mode 100644
@@ -0,0 +1,405 @@
@@ -0,0 +1,405 @@
+'use strict';
+
+/**
+ * Signed-out voting (plan L51, step 1): the anonymous voter, the relay, and every cap.
+ *
+ * The caps are read from the environment, so each group of rules gets its own Bench with only that cap low.
+ */
+
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+
+const FORM = { 'content-type': 'application/x-www-form-urlencoded' };
+const SAME = { secFetchSite: 'same-origin' };
+
+/** A round with two entries: two things to judge. */
+async function makeRound(t, alice, svc, n, owner) {
+ const task = await t.get('/api/v1/kinds/design.ui/tasks', { as: alice, json: { prompt: 'A dashboard, prompt ' + n + '.', public: true } });
+ assert.strictEqual(task.status, 201, task.text);
+ const rr = await t.get('/api/v1/tasks/' + task.json().task.id + '/rounds', { as: alice, json: {} });
+ assert.strictEqual(rr.status, 201, rr.text);
+ const roundId = rr.json().round.id;
+ const e1 = await t.get('/api/v1/rounds/' + roundId + '/entries', { bearer: svc, json: { provider: 'acme', model: 'alpha-' + n, payload: { html: '<p>a</p>' }, cost_usd: 0.01 } });
+ const e2 = await t.get('/api/v1/rounds/' + roundId + '/entries', { bearer: svc, json: { provider: 'beta', model: 'beta-' + n, payload: { html: '<p>b</p>' }, cost_usd: 0.02 } });
+ assert.strictEqual(e1.status, 201, e1.text);
+ assert.strictEqual(e2.status, 201, e2.text);
+ return { id: roundId, a: e1.json().entry.id, b: e2.json().entry.id };
+}
+
+/** The cookie Bench hands a visitor who has not signed in. */
+function cookieOf(res) {
+ const raw = res.headers.get('set-cookie') || '';
+ const first = String(raw).split(';')[0];
+ return first.includes('=') ? first : '';
+}
+
+(async () => {
+ // ── The voter: one cookie, one vote per round, a cap a day ──────────────
+ const t = await boot({
+ env: {
+ BENCH_ROUNDS_PER_DAY: '20',
+ BENCH_ANON_VOTER_PER_DAY: '2',
+ BENCH_ANON_IP_PER_DAY: '1000',
+ BENCH_ANON_RELAY_PER_DAY: '1000',
+ BENCH_ANON_GLOBAL_PER_DAY: '1000',
+ BENCH_ANON_RELAY_SERVICES: 'design',
+ },
+ });
+ const alice = t.network.addUser('alice');
+ const bob = t.network.addUser('bob');
+ const svc = t.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });
+ const design = t.network.serviceToken({
+ sub: 'svc:design', audience: 'openvibe.bench',
+ cap: ['bench.task.create', 'bench.round.create', 'bench.judgement.cast'],
+ });
+ const rounds = [];
+ for (const n of [1, 2, 3]) rounds.push(await makeRound(t, alice, svc, n));
+
+ let anonCookie = '';
+ let anonSubject = '';
+
+ await check('the next-pair route answers an anonymous voter and hands out a signed cookie', async () => {
+ const r = await t.get('/api/v1/judgements/next?kind=design.ui');
+ assert.strictEqual(r.status, 200, r.text);
+ anonCookie = cookieOf(r);
+ assert.ok(anonCookie.startsWith('bench_anon='), 'the voter cookie: ' + anonCookie);
+ assert.ok(/HttpOnly/i.test(r.headers.get('set-cookie') || ''), 'the cookie is httpOnly');
+ assert.ok(r.json().left && r.json().right, 'a pair');
+ assert.strictEqual(r.json().left.model, undefined, 'blind before the vote');
+ assert.strictEqual(r.json().right.model, undefined, 'blind before the vote');
+ });
+
+ await check('a signed-out visitor votes through the cookie, and the pair opens after the vote', async () => {
+ const before = await t.get('/api/v1/showcase/rounds/' + rounds[0].id, { cookie: anonCookie });
+ assert.strictEqual(before.status, 200, before.text);
+ assert.strictEqual(before.json().round.revealed, false, 'blind before the vote');
+
+ const vote = await t.get('/api/v1/showcase/rounds/' + rounds[0].id + '/vote', {
+ method: 'POST', cookie: anonCookie, json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(vote.status, 201, vote.text);
+ assert.strictEqual(vote.json().judged, true);
+ assert.strictEqual(vote.json().revealed, true, 'the pair is open after the vote');
+
+ const after = await t.get('/api/v1/showcase/rounds/' + rounds[0].id, { cookie: anonCookie });
+ assert.strictEqual(after.json().round.revealed, true, 'the pair is open for the voter who judged it');
+ assert.ok(after.json().round.slots[0].model, 'the names are there for the voter who judged it');
+ });
+
+ await check('the judgement is stored as an anonymous vote, with a hash and never the id', async () => {
+ const row = await t.ctx.s.db.maybe("SELECT voter_kind, subject FROM bench_judgements WHERE judge = 'human_pairwise' ORDER BY at DESC LIMIT 1");
+ assert.strictEqual(row.voter_kind, 'anon');
+ assert.ok(String(row.subject).startsWith('anon_'), 'the subject is the hash of the opaque id: ' + row.subject);
+ assert.ok(!String(row.subject).includes('bench_anon'), 'the cookie value is never stored');
+ anonSubject = row.subject;
+ // The open dataset never carries a voter id.
+ const dataset = await t.get('/api/v1/datasets/design.ui.jsonl');
+ assert.strictEqual(dataset.status, 200, dataset.text);
+ assert.ok(!dataset.text.includes(anonSubject), 'the dataset carries no voter hash');
+ assert.ok(!dataset.text.includes('anon_'), 'the dataset carries no anonymous voter');
+ });
+
+ await check('the same anonymous voter never judges the same round twice', async () => {
+ const again = await t.get('/api/v1/showcase/rounds/' + rounds[0].id + '/vote', {
+ method: 'POST', cookie: anonCookie, json: { outcome: 'B' }, ...SAME,
+ });
+ assert.strictEqual(again.status, 409, again.text);
+ assert.strictEqual(again.json().code, 'bench.already_judged', again.text);
+ });
+
+ await check('the next-pair route never hands back a round the voter judged, and 204 when none is left', async () => {
+ const more = await t.get('/api/v1/judgements/next?kind=design.ui', { cookie: anonCookie });
+ assert.strictEqual(more.status, 200, more.text);
+ assert.notStrictEqual(more.json().round.id, rounds[0].id, 'a round they have not judged');
+ const vote = await t.get('/api/v1/showcase/rounds/' + rounds[1].id + '/vote', {
+ method: 'POST', cookie: anonCookie, json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(vote.status, 201, vote.text);
+ const none = await t.get('/api/v1/judgements/next?kind=design.svg', { cookie: anonCookie });
+ assert.strictEqual(none.status, 204, none.text);
+ assert.strictEqual(none.text, '', 'no body with 204');
+ });
+
+ await check('over the daily cap: 429 bench.limit_anon, and a plain reason', async () => {
+ const over = await t.get('/api/v1/showcase/rounds/' + rounds[2].id + '/vote', {
+ method: 'POST', cookie: anonCookie, json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(over.status, 429, over.text);
+ assert.strictEqual(over.json().code, 'bench.limit_anon', over.text);
+ assert.strictEqual(over.json().detail, 'Sign in to keep voting today', over.text);
+ const counted = await t.ctx.s.db.value("SELECT COUNT(*)::int AS n FROM bench_judgements WHERE voter_kind = 'anon' AND subject = $1", [anonSubject]);
+ assert.strictEqual(counted, 2, 'the refused vote was not counted');
+ });
+
+ await check("Bench's own page votes without signing in", async () => {
+ const page = await t.get('/showcase/round/' + rounds[2].id);
+ assert.strictEqual(page.status, 200, page.text);
+ const cookie = cookieOf(page);
+ assert.ok(cookie.startsWith('bench_anon='), 'the page hands out the voter cookie');
+ assert.ok(page.text.includes('data-vote-form="1"'), 'the signed-out visitor gets the form');
+ assert.ok(!page.text.includes('Sign in to vote'), 'not the sign-in prompt');
+ const vote = await t.get('/showcase/round/' + rounds[2].id, {
+ method: 'POST', cookie, headers: FORM, body: 'outcome=A', ...SAME,
+ });
+ assert.strictEqual(vote.status, 302, vote.text);
+ const row = await t.ctx.s.db.maybe("SELECT voter_kind FROM bench_judgements WHERE round_id = $1 ORDER BY at DESC LIMIT 1", [rounds[2].id]);
+ assert.strictEqual(row.voter_kind, 'anon', 'the page vote is an anonymous vote');
+ });
+
+ await check('a board says how many votes of each kind of voter it holds', async () => {
+ const person = await t.get('/api/v1/showcase/rounds/' + rounds[0].id, { as: bob });
+ assert.strictEqual(person.status, 200, person.text);
+ const bobVote = await t.get('/api/v1/rounds/' + rounds[0].id + '/judgements', {
+ as: bob, json: { judge: 'human_pairwise', pair: { left: rounds[0].a, right: rounds[0].b }, outcome: 'left' },
+ });
+ assert.strictEqual(bobVote.status, 201, bobVote.text);
+
+ const counted = await t.ctx.s.db.many("SELECT voter_kind, COUNT(*)::int AS n FROM bench_judgements WHERE judge = 'human_pairwise' GROUP BY 1");
+ const byKind = new Map(counted.map((row) => [row.voter_kind, row.n]));
+ const all = await t.get('/api/v1/boards/design.ui');
+ assert.strictEqual(all.status, 200, all.text);
+ assert.ok(all.json().board.voters, 'the board counts its voters');
+ assert.strictEqual(all.json().board.voters.anon, byKind.get('anon'), 'the anonymous votes: ' + JSON.stringify(all.json().board.voters));
+ assert.strictEqual(all.json().board.voters.person, byKind.get('person'), 'the signed-in votes: ' + JSON.stringify(all.json().board.voters));
+ assert.strictEqual(all.json().board.voters.person, 1, 'one signed-in vote');
+
+ const people = await t.get('/api/v1/boards/design.ui?voters=people');
+ assert.strictEqual(people.status, 200, people.text);
+ assert.strictEqual(people.json().board.category, 'people');
+ // Every vote is a game for both models, so the games of a board are twice its votes.
+ const peopleGames = people.json().board.rows.reduce((n, row) => n + row.games, 0);
+ assert.strictEqual(peopleGames, byKind.get('person') * 2, 'the signed-in vote alone: ' + JSON.stringify(people.json().board.rows));
+
+ const anon = await t.get('/api/v1/boards/design.ui?voters=anon');
+ assert.strictEqual(anon.status, 200, anon.text);
+ assert.strictEqual(anon.json().board.category, 'anon');
+ const anonGames = anon.json().board.rows.reduce((n, row) => n + row.games, 0);
+ assert.strictEqual(anonGames, byKind.get('anon') * 2, 'the anonymous votes alone: ' + JSON.stringify(anon.json().board.rows));
+ assert.ok(anonGames > peopleGames, 'the anonymous votes are counted apart from the signed-in ones');
+ });
+
+ await check('a signed-in person still gets the pair they have not judged', async () => {
+ const r = await t.get('/api/v1/judgements/next?kind=design.ui', { as: bob });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.ok(r.json().left && r.json().right);
+ });
+
+ await check('stop', async () => { await t.close(); });
+
+ // ── The relay and the address ────────────────────────────────────────────
+ const t2 = await boot({
+ env: {
+ BENCH_ROUNDS_PER_DAY: '20',
+ BENCH_ANON_VOTER_PER_DAY: '1000',
+ BENCH_ANON_IP_PER_DAY: '2',
+ BENCH_ANON_RELAY_PER_DAY: '1',
+ BENCH_ANON_GLOBAL_PER_DAY: '1000',
+ BENCH_ANON_RELAY_SERVICES: 'design',
+ },
+ });
+ const alice2 = t2.network.addUser('alice');
+ const svc2 = t2.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });
+ const design2 = t2.network.serviceToken({
+ sub: 'svc:design', audience: 'openvibe.bench',
+ cap: ['bench.task.create', 'bench.round.create', 'bench.judgement.cast'],
+ });
+ const other2 = t2.network.serviceToken({
+ sub: 'svc:other', audience: 'openvibe.bench',
+ cap: ['bench.task.create', 'bench.round.create', 'bench.judgement.cast'],
+ });
+ const rounds2 = [];
+ for (const n of [1, 2, 3, 4]) rounds2.push(await makeRound(t2, alice2, svc2, n));
+
+ await check('a service on the allow-list relays an anonymous vote', async () => {
+ const r = await t2.get('/api/v1/rounds/' + rounds2[0].id + '/judgements', {
+ bearer: design2,
+ headers: { 'x-ov-voter': 'anon:visitor-one' },
+ json: { pair: { left: rounds2[0].a, right: rounds2[0].b }, outcome: 'left' },
+ });
+ assert.strictEqual(r.status, 201, r.text);
+ assert.ok(r.json().revealed.left.model, 'the names open after the vote');
+ const row = await t2.ctx.s.db.maybe("SELECT voter_kind, subject FROM bench_judgements WHERE round_id = $1 ORDER BY at DESC LIMIT 1", [rounds2[0].id]);
+ assert.strictEqual(row.voter_kind, 'anon');
+ assert.ok(!String(row.subject).includes('visitor-one'), 'the opaque id is never stored: ' + row.subject);
+ });
+
+ await check('the relay gets a blind pair for the anonymous voter it names', async () => {
+ const r = await t2.get('/api/v1/judgements/next?kind=design.ui', {
+ bearer: design2, headers: { 'x-ov-voter': 'anon:visitor-two' },
+ });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.ok(r.json().left && r.json().right);
+ assert.strictEqual(r.json().left.model, undefined, 'blind before the vote');
+ assert.ok(r.json().round.id, 'the round it names');
+ });
+
+ await check('a service that is not on the allow-list may not relay', async () => {
+ const r = await t2.get('/api/v1/rounds/' + rounds2[1].id + '/judgements', {
+ bearer: other2,
+ headers: { 'x-ov-voter': 'anon:visitor-three' },
+ json: { pair: { left: rounds2[1].a, right: rounds2[1].b }, outcome: 'left' },
+ });
+ assert.strictEqual(r.status, 403, r.text);
+ assert.strictEqual(r.json().code, 'bench.relay_not_allowed', r.text);
+ });
+
+ await check('a service token without bench.judgement.cast may not relay', async () => {
+ const weak = t2.network.serviceToken({ sub: 'svc:design', audience: 'openvibe.bench', cap: ['bench.board.read'] });
+ const r = await t2.get('/api/v1/rounds/' + rounds2[1].id + '/judgements', {
+ bearer: weak,
+ headers: { 'x-ov-voter': 'anon:visitor-four' },
+ json: { pair: { left: rounds2[1].a, right: rounds2[1].b }, outcome: 'left' },
+ });
+ assert.strictEqual(r.status, 403, r.text);
+ assert.strictEqual(r.json().code, 'bench.capability_denied', r.text);
+ });
+
+ await check('the relay never sees a round it started itself', async () => {
+ const task = await t2.get('/api/v1/kinds/design.ui/tasks', { bearer: design2, json: { prompt: 'The relay\'s own round.', public: true } });
+ assert.strictEqual(task.status, 201, task.text);
+ const rr = await t2.get('/api/v1/tasks/' + task.json().task.id + '/rounds', { bearer: design2, json: {} });
+ const own = rr.json().round.id;
+ const e1 = await t2.get('/api/v1/rounds/' + own + '/entries', { bearer: svc2, json: { provider: 'acme', model: 'alpha-own', payload: { html: '<p>a</p>' } } });
+ const e2 = await t2.get('/api/v1/rounds/' + own + '/entries', { bearer: svc2, json: { provider: 'beta', model: 'beta-own', payload: { html: '<p>b</p>' } } });
+ const vote = await t2.get('/api/v1/rounds/' + own + '/judgements', {
+ bearer: design2,
+ headers: { 'x-ov-voter': 'anon:visitor-five' },
+ json: { pair: { left: e1.json().entry.id, right: e2.json().entry.id }, outcome: 'left' },
+ });
+ assert.strictEqual(vote.status, 403, vote.text);
+ assert.strictEqual(vote.json().code, 'bench.creator_cannot_vote', vote.text);
+ });
+
+ await check('the relay has a daily total of its own', async () => {
+ const over = await t2.get('/api/v1/rounds/' + rounds2[1].id + '/judgements', {
+ bearer: design2,
+ headers: { 'x-ov-voter': 'anon:visitor-six' },
+ json: { pair: { left: rounds2[1].a, right: rounds2[1].b }, outcome: 'left' },
+ });
+ assert.strictEqual(over.status, 429, over.text);
+ assert.strictEqual(over.json().code, 'bench.limit_anon', over.text);
+ assert.strictEqual(over.json().detail, 'Sign in to keep voting today', over.text);
+ });
+
+ await check('the per-address cap: the same address has one budget, another address has its own', async () => {
+ const first = await t2.get('/api/v1/judgements/next?kind=design.ui');
+ const c1 = cookieOf(first);
+ const v1 = await t2.get('/api/v1/showcase/rounds/' + rounds2[1].id + '/vote', {
+ method: 'POST', cookie: c1, json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(v1.status, 201, v1.text);
+
+ const second = await t2.get('/api/v1/judgements/next?kind=design.ui');
+ const c2 = cookieOf(second);
+ const v2 = await t2.get('/api/v1/showcase/rounds/' + rounds2[2].id + '/vote', {
+ method: 'POST', cookie: c2, json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(v2.status, 201, v2.text);
+
+ // The third visitor from the same address is over the address budget.
+ const third = await t2.get('/api/v1/judgements/next?kind=design.ui');
+ const c3 = cookieOf(third);
+ const v3 = await t2.get('/api/v1/showcase/rounds/' + rounds2[3].id + '/vote', {
+ method: 'POST', cookie: c3, json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(v3.status, 429, v3.text);
+ assert.strictEqual(v3.json().code, 'bench.limit_anon', v3.text);
+
+ // Another address has its own budget.
+ const other = await t2.get('/api/v1/judgements/next?kind=design.ui', { headers: { 'x-forwarded-for': '203.0.113.9' } });
+ const c4 = cookieOf(other);
+ const v4 = await t2.get('/api/v1/showcase/rounds/' + rounds2[3].id + '/vote', {
+ method: 'POST', cookie: c4, json: { outcome: 'A' }, headers: { 'x-forwarded-for': '203.0.113.9' }, ...SAME,
+ });
+ assert.strictEqual(v4.status, 201, v4.text);
+
+ // The address itself is never written down: only a salted hash of it.
+ const rows = await t2.ctx.s.db.many("SELECT key FROM bench_anon_counters WHERE scope = 'ip'");
+ assert.ok(rows.length >= 2, 'the address counters');
+ for (const row of rows) {
+ assert.ok(!row.key.includes('203.0.113.9') && !row.key.includes('127.0.0.1'), 'no address in the counters: ' + row.key);
+ assert.ok(/^[0-9a-f]{64}$/.test(row.key), 'a hash: ' + row.key);
+ }
+ });
+
+ await check('stop', async () => { await t2.close(); });
+
+ // ── The site-wide cap ────────────────────────────────────────────────────
+ const t3 = await boot({
+ env: {
+ BENCH_ROUNDS_PER_DAY: '20',
+ BENCH_ANON_VOTER_PER_DAY: '1000',
+ BENCH_ANON_IP_PER_DAY: '1000',
+ BENCH_ANON_RELAY_PER_DAY: '1000',
+ BENCH_ANON_GLOBAL_PER_DAY: '2',
+ },
+ });
+ const alice3 = t3.network.addUser('alice');
+ const svc3 = t3.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });
+ const rounds3 = [];
+ for (const n of [1, 2, 3]) rounds3.push(await makeRound(t3, alice3, svc3, n));
+
+ await check('over the site-wide cap every anonymous vote is refused', async () => {
+ for (const [i, ip] of ['198.51.100.1', '198.51.100.2'].entries()) {
+ const next = await t3.get('/api/v1/judgements/next?kind=design.ui', { headers: { 'x-forwarded-for': ip } });
+ const cookie = cookieOf(next);
+ const vote = await t3.get('/api/v1/showcase/rounds/' + rounds3[i].id + '/vote', {
+ method: 'POST', cookie, json: { outcome: 'A' }, headers: { 'x-forwarded-for': ip }, ...SAME,
+ });
+ assert.strictEqual(vote.status, 201, vote.text);
+ }
+ const next = await t3.get('/api/v1/judgements/next?kind=design.ui', { headers: { 'x-forwarded-for': '198.51.100.3' } });
+ const cookie = cookieOf(next);
+ const over = await t3.get('/api/v1/showcase/rounds/' + rounds3[2].id + '/vote', {
+ method: 'POST', cookie, json: { outcome: 'A' }, headers: { 'x-forwarded-for': '198.51.100.3' }, ...SAME,
+ });
+ assert.strictEqual(over.status, 429, over.text);
+ assert.strictEqual(over.json().code, 'bench.limit_anon', over.text);
+ assert.strictEqual(over.json().detail, 'Sign in to keep voting today', over.text);
+ });
+
+ await check('stop', async () => { await t3.close(); });
+
+ // The address hash is salted per day, and the old counters go.
+ let now = Date.parse('2026-11-01T09:00:00.000Z');
+ const t4 = await boot({
+ now: () => now,
+ env: { BENCH_ROUNDS_PER_DAY: '20', BENCH_ANON_IP_PER_DAY: '1000', BENCH_ANON_GLOBAL_PER_DAY: '1000' },
+ });
+ const alice4 = t4.network.addUser('alice');
+ const svc4 = t4.network.serviceToken({ sub: 'svc:games', audience: 'openvibe.bench', cap: ['bench.entry.create'] });
+ const rounds4 = [];
+ for (const n of [1, 2]) rounds4.push(await makeRound(t4, alice4, svc4, n));
+
+ await check('the hash of an address is salted per day, so two days cannot be linked', async () => {
+ const first = await t4.get('/api/v1/judgements/next?kind=design.ui');
+ const v1 = await t4.get('/api/v1/showcase/rounds/' + rounds4[0].id + '/vote', {
+ method: 'POST', cookie: cookieOf(first), json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(v1.status, 201, v1.text);
+
+ now = Date.parse('2026-11-02T09:00:00.000Z');
+ const second = await t4.get('/api/v1/judgements/next?kind=design.ui');
+ const v2 = await t4.get('/api/v1/showcase/rounds/' + rounds4[1].id + '/vote', {
+ method: 'POST', cookie: cookieOf(second), json: { outcome: 'A' }, ...SAME,
+ });
+ assert.strictEqual(v2.status, 201, v2.text);
+
+ const rows = await t4.ctx.s.db.many("SELECT key, day FROM bench_anon_counters WHERE scope = 'ip' ORDER BY day");
+ assert.strictEqual(rows.length, 2, 'two days, two rows: ' + JSON.stringify(rows));
+ assert.notStrictEqual(rows[0].key, rows[1].key, 'the salt moves with the day');
+ assert.strictEqual(rows[0].day, '2026-11-01');
+ assert.strictEqual(rows[1].day, '2026-11-02');
+ });
+
+ await check('the old counters are dropped after their week', async () => {
+ now = Date.parse('2026-11-20T09:00:00.000Z');
+ await t4.ctx.anon.prune();
+ const rows = await t4.ctx.s.db.many('SELECT COUNT(*)::int AS n FROM bench_anon_counters');
+ assert.strictEqual(rows[0].n, 0, 'the old rows are gone');
+ });
+
+ await check('stop', async () => { await t4.close(); });
+
+ done();
+})();
new file mode 100644
new file mode 100644
@@ -0,0 +1,135 @@
@@ -0,0 +1,135 @@
+'use strict';
+
+/**
+ * The daily rounds (plan L51, step 2): one round per design kind per day, on Bench's own budget.
+ */
+
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const { DAILY_KIND_IDS, DAILY_PROMPTS } = require('../server/bench/daily-prompts');
+
+const CLOCK = Date.parse('2026-11-01T09:00:00.000Z');
+const clock = () => CLOCK;
+
+(async () => {
+ await check('the job is off until the owner turns it on, and never starts under the drill', async () => {
+ const off = await boot();
+ assert.strictEqual(off.ctx.daily.start(), null, 'off without BENCH_DAILY_ROUNDS');
+ await off.close();
+
+ const drill = await boot({ env: { BENCH_DAILY_ROUNDS: '1' } });
+ assert.strictEqual(drill.ctx.daily.start(), null, 'off under BENCH_DRILL');
+ await drill.close();
+ });
+
+ const t = await boot({
+ now: clock,
+ env: {
+ BENCH_DAILY_ROUNDS: '1',
+ BENCH_DAILY_ROUNDS_IN_DRILL: '1',
+ BENCH_DAILY_ROUNDS_HOUR_UTC: '8',
+ BENCH_DAILY_BUDGET_USD: '1.00',
+ BENCH_DAILY_ROUND_ESTIMATE_USD: '0.05',
+ },
+ });
+
+ await check('the curated list has about twenty prompts for each of the nine design kinds', async () => {
+ assert.strictEqual(DAILY_KIND_IDS.length, 9, 'nine design kinds');
+ for (const kind of DAILY_KIND_IDS) {
+ assert.ok(DAILY_PROMPTS[kind].length >= 18, kind + ' has about twenty prompts: ' + DAILY_PROMPTS[kind].length);
+ for (const prompt of DAILY_PROMPTS[kind]) {
+ assert.ok(prompt.length >= 20, kind + ': a prompt is a brief');
+ assert.ok(!/\b(he|she|his|her)\b/i.test(prompt), kind + ': no people in the brief');
+ }
+ }
+ });
+
+ let summary = null;
+
+ await check('one round per design kind, from the curated prompts, with two models of the pool', async () => {
+ summary = await t.ctx.daily.runDay({ now: CLOCK });
+ assert.strictEqual(summary.created, 9, 'nine rounds: ' + JSON.stringify(summary.rounds));
+ assert.strictEqual(summary.skipped, 0);
+
+ const rows = await t.ctx.s.db.many('SELECT * FROM bench_daily_rounds ORDER BY kind_id');
+ assert.strictEqual(rows.length, 9, 'one row per kind');
+ for (const row of rows) {
+ assert.strictEqual(row.status, 'created');
+ assert.strictEqual(row.day, '2026-11-01');
+ assert.ok(DAILY_PROMPTS[row.kind_id].includes(row.prompt), 'the prompt is one of the curated ones: ' + row.prompt);
+ const models = Array.isArray(row.models) ? row.models : JSON.parse(row.models);
+ assert.strictEqual(models.length, 2, 'two models: ' + JSON.stringify(models));
+ assert.notStrictEqual(models[0].provider, models[1].provider, 'two different models: ' + JSON.stringify(models));
+ const round = await t.ctx.s.db.maybe('SELECT * FROM bench_rounds WHERE id = $1', [row.round_id]);
+ assert.ok(round, 'the round exists');
+ assert.strictEqual(round.kind_id, row.kind_id);
+ assert.strictEqual(round.created_by, 'service:bench-daily', 'the daily rounds belong to Bench');
+ }
+ // The prompt of a kind is the same all day, and comes from the curated list.
+ assert.strictEqual(t.ctx.daily.promptFor('design.ui', '2026-11-01'), t.ctx.daily.promptFor('design.ui', '2026-11-01'));
+ });
+
+ await check('the same day never makes a second round of a kind', async () => {
+ const again = await t.ctx.daily.runDay({ now: CLOCK });
+ assert.strictEqual(again.created, 0, 'nothing new: ' + JSON.stringify(again.rounds));
+ const rows = await t.ctx.s.db.many('SELECT COUNT(*)::int AS n FROM bench_daily_rounds');
+ assert.strictEqual(rows[0].n, 9);
+ });
+
+ await check('the rounds are real rounds: two entries each, and something to judge', async () => {
+ // The worker takes one queued round per pass, and there are nine of them.
+ for (let i = 0; i < 12; i++) await t.tick();
+ const rounds = await t.ctx.s.db.many("SELECT id, status FROM bench_rounds WHERE created_by = 'service:bench-daily'");
+ assert.strictEqual(rounds.length, 9);
+ for (const round of rounds) {
+ assert.strictEqual(round.status, 'finished', 'the round is ready to judge');
+ }
+ const next = await t.get('/api/v1/judgements/next?kind=design.ui');
+ assert.strictEqual(next.status, 200, next.text);
+ assert.ok(next.json().left && next.json().right, 'there is something to judge');
+ assert.strictEqual(next.json().left.model, undefined, 'the pair is blind');
+ });
+
+ await check('what the day spent is written down', async () => {
+ await t.ctx.daily.settle('2026-11-01');
+ const rows = await t.ctx.s.db.many('SELECT kind_id, estimated_usd, spent_usd FROM bench_daily_rounds ORDER BY kind_id');
+ let spent = 0;
+ for (const row of rows) {
+ assert.ok(Number(row.spent_usd) > 0, row.kind_id + ': what it spent');
+ spent += Number(row.spent_usd);
+ }
+ assert.ok(spent > 0, 'the day spent something');
+ const state = await t.ctx.daily.budgetState('2026-11-01');
+ assert.ok(Math.abs(state.spent - spent) < 1e-6, 'the budget state knows what was spent');
+ });
+
+ await check('stop', async () => { await t.close(); });
+
+ // ── The budget ───────────────────────────────────────────────────────────
+ const lean = await boot({
+ now: clock,
+ env: {
+ BENCH_DAILY_ROUNDS: '1',
+ BENCH_DAILY_BUDGET_USD: '0.001',
+ BENCH_DAILY_ROUND_ESTIMATE_USD: '0.05',
+ },
+ });
+
+ await check('a kind is skipped when the day\'s budget is spent', async () => {
+ const out = await lean.ctx.daily.runDay({ now: CLOCK });
+ assert.strictEqual(out.created, 0, 'nothing was made: ' + JSON.stringify(out.rounds));
+ assert.strictEqual(out.skipped, 9);
+ const rows = await lean.ctx.s.db.many('SELECT kind_id, status, skip_reason FROM bench_daily_rounds');
+ assert.strictEqual(rows.length, 9);
+ for (const row of rows) {
+ assert.strictEqual(row.status, 'skipped');
+ assert.strictEqual(row.skip_reason, 'budget', 'why it was skipped: ' + row.kind_id);
+ }
+ const rounds = await lean.ctx.s.db.many('SELECT COUNT(*)::int AS n FROM bench_rounds');
+ assert.strictEqual(rounds[0].n, 0, 'no round was made');
+ });
+
+ await check('stop', async () => { await lean.close(); });
+
+ done();
+})();