AGENTS.md

AGENTS.md — OpenVibe.Bench

Project overview

The shared AI benchmark and comparison engine for every OpenVibe room: kinds, tasks, pinned
benchmark versions, entries, judgements, the Bradley–Terry ratings, boards, the open dataset and
private evals. Node.js/Express, PostgreSQL (ov_bench), CommonJS, no build step.

• Service id bench, port 5230, origin https://bench.openvibe.services; the render origin
  https://bench-render.openvibe.services serves the same app; unit openvibe-bench.service.
• Division of authority: AI runs models, Bench judges and rates, Report owns the public AI
  analytics pages, Route learns from the results, and each room keeps its own pages.
• Public pages moved to OpenVibe.Report on 2026-11-08: https://openvibe.report/ai,
  /ai/<benchmark>, /ai/models/<model>. This origin's old public HTML answers 301 there; its API,
  MCP endpoint, render origin, badges, embeds and owner pages stay here.

What it owns

• Benchmark kinds, tasks, a benchmark's drafts and frozen versions (bench.benchmark@1), entries
  and judged rounds; private by default, an owner-only prompt set, a share link and an audience.
• Votes and judgements, including anonymous votes, daily rounds, battles and the pick flow.
• Ratings: GET /api/v1/boards/:kindId, GET /api/v1/datasets/:kindId.jsonl,
  GET /api/v1/ratings/:kindId (the shared public read other sites consume).
• The registry API (/api/v1/benchmarks…), showcase, gallery, badges, embeds, watches, imports and
  variant evidence.
• POST /mcp, /openapi.json, /llms.txt, the AI section data Report reads.
• Owner-only pages: /benchmarks/mine (default audience, "apply to all", Share), /benchmarks/new,
  /b/<token> unlisted links, and the noindex task/run pages.

Not owned: model execution (AI), the site's analytics presentation (Report), the learning loop
(Route), each room's product pages.

Who calls it, and whom it calls

• Callers: OpenVibe.Report (public AI pages and /ai/method), Live and other rooms (boards/ratings,
  badges, embeds), AI (bench.rating.changed, GET /api/v1/boards/:kind?type=elo|cost for routed
  evidence), and people/agents through the API and MCP.
• Depends on: Network (JWKS, user/service tokens, SSO with PKCE in server/auth/sso.js); AI
  (server/bench/ai.js runs benchmark tasks); Events (outbox publisher, bench.rating.changed);
  PostgreSQL through openvibe-sdk/db; Valkey optional through openvibe-sdk/valkey.

Repository map

• Entry: [server/index.js](server/index.js) — createApp(), graceful shutdown of keys, jobs,
  daily rounds, EEE sync, fidelity job, publisher, telemetry.
• App: [server/app.js](server/app.js) — helmet/CSP, cookies, rate limits, /auth, /internal/events,
  /shared, /api/v1 (api, registry, variant evidence, showcase, arena, gallery), /api, /render,
  /vendor, pages, /mcp, /badge, /embed.
• Config: [server/config.js](server/config.js), [server/variant-evidence-config.js](server/variant-evidence-config.js).
• Database: [server/db.js](server/db.js); migrations 0001…0017 in [migrations/](migrations/).
• Store/domain: [server/store.js](server/store.js), [server/bench/](server/bench/) (battles, anon,
  daily, registry, launch, seed), [server/rating/](server/rating/) (statistics, boards, style).
• HTTP: [server/http/](server/http/) — api.js, router.js, registry.js, ratings.js,
  pages.js, badge.js, embed.js, watch.js, pick.js, arena.js, gallery.js, render.js.
• Auth: [server/auth/keys.js](server/auth/keys.js), [server/auth/sso.js](server/auth/sso.js),
  [server/http/principal.js](server/http/principal.js).
• Frontend: [public/js/bench.js](public/js/bench.js), [public/css/bench.css](public/css/bench.css);
  server-rendered pages in [server/render/](server/render/).
• MCP: [server/mcp.js](server/mcp.js).

Commands

npm ci
npm run dev                # NODE_ENV=development node server/index.js
npm start                  # node server/index.js, http://127.0.0.1:5230
OV_ALLOW_SUITE=1 npm test  # the suite refuses to run without the guard in some environments
npm run test:pg            # same suite on PostgreSQL (BENCH_TEST_STORE=pg)
npm test -- api            # only files whose name contains the word
ov test                    # the harness's sharded run of what the change reaches
node --check <file.js>

Tests use a temporary PGlite database, an in-process mock Network and a fake AI; nothing needs the
network or a paid key.

Database and migrations

PostgreSQL database ov_bench (host data role, ADR-035); development and tests use PGlite. Every
read/write is async through openvibe-sdk/db. Schema is migrations/NNNN_*.sql, applied at boot;
never edit a migration that has run, and add a new number. A benchmark version is frozen: the
document and its SHA-256 hash are immutable and addressable as <benchmark>@<version>; drafts are
separate. Do not "fix" a frozen version in place.

Auth

A person is the bench_at session cookie (with bench_rt, bench_oauth) or a Network user token as
Bearer; a service sends a client-credentials token with the bench.* capability. A session write
must come from bench.openvibe.services itself (same-origin). Capability ids come from the pinned
openvibe-contracts; an unknown id is refused. Anonymous browser reads are allowed only on the
public rating/board/embed paths, which return no prompts, answers, voters or site identifiers.

Frontend loading model

Server-rendered HTML (no SPA framework, no bundler, no transpiler) plus one page script,
[public/js/bench.js](public/js/bench.js); /shared/* comes from the pinned openvibe-shared tarball
through its serve handler. CSP and cache policy are in [server/app.js](server/app.js) and
openvibe-shared/cache-policy. The bench-render. vhost exists for rendering/embeds; it is the same
process, not a second app.

Deploy

Unit [deploy/systemd/openvibe-bench.service](deploy/systemd/openvibe-bench.service) runs
/opt/openvibe.bench/server/index.js on 127.0.0.1:5230, env /etc/openvibe/bench.env; two vhosts,
[deploy/nginx/bench.openvibe.services.conf](deploy/nginx/bench.openvibe.services.conf) and
[deploy/nginx/bench-render.openvibe.services.conf](deploy/nginx/bench-render.openvibe.services.conf).
The repository does not record an ovhost deploy command; keep the unit and vhost in sync with
deploy/. Migrations run at boot; no secret in a file or a log.

Ratings rules (the part people rely on)

• Every board is a Bradley–Terry fit over votes, centred on 1000, with a bootstrap giving each model
  a rating interval and a 95% rank interval ([docs/index.md](docs/index.md)).
• GET /api/v1/ratings/:kindId?category=all omits models with fewer than 20 eligible games,
  including their counts; clients must not average ratings from different kinds or sources
  ([docs/shared-ratings.md](docs/shared-ratings.md)).
• The dataset (/api/v1/datasets/:kindId.jsonl) carries the same numbers as the board.
• No prompts, answers, voters or site identifiers leave through the public read.

Tests

npm test runs test/run.js, each test/*.test.js in its own process with a 90 s timeout through
openvibe-shared/test-runner; failure of any file fails the run. The suite pins the registry/frozen
versions and hashes, daily rounds, anonymous votes, boards, the dataset and JSON-LD, Report-move
redirects, MCP, OpenAPI, badges, embeds, privacy and the contracts pin. ov test runs the same.

Conventions

• CommonJS, 4-space indent, single quotes, semicolons; factories (createApp, createApi,
  createStore) and injectable clock/fetch/log so tests need no network.
• Errors are application/problem+json with a stable code; never leak SQL or upstream text.
• Contracts and enums come from openvibe-contracts; bump the pin before using a new id.
• Keep the API backward-compatible: Report, AI, Live and rooms read these endpoints.

Pitfalls

• Frozen means frozen. A benchmark version is immutable and hashed; editing it breaks every run
  and judgement that names it.
• The public pages are Report's. Only owner, unlisted and private pages belong on this origin;
  redirect the rest instead of re-growing HTML.
• Same-origin writes. A session write from another origin is refused; do not relax it for embeds.
• Minimum 20 games and exact identity: a small or fuzzy sample is omitted, never shown as a weak
  rating.
• Private benchmarks stay private. Prompts are owner-only; a share link is the only door.
• OV_ALLOW_SUITE=1 is required by the suite harness in this environment; a plain npm test may
  refuse, which is not a test failure to "fix" by disabling the guard.
• bench-render. is the same service. A change there is a change here.

Documentation

• [README.md](README.md) — what works and the API summary.
• [docs/index.md](docs/index.md) — the rating method and intervals.
• [docs/api.md](docs/api.md) — the API, auth and the frozen/unlisted distinction.
• [docs/benchmarks.md](docs/benchmarks.md) — writing and versioning a benchmark.
• [docs/shared-ratings.md](docs/shared-ratings.md) — the public rating read other sites consume.