openvibers/openvibe.bench Commits
AGENTS.md: the context an agent starts from (#215)
Changes
nnew file mode 100644@@@ -0,0 +1,155 @@++# AGENTS.md — OpenVibe.Bench++++## Project overview++++The shared AI benchmark and comparison engine for every OpenVibe room: kinds, tasks, pinned++benchmark versions, entries, judgements, the Bradley–Terry ratings, boards, the open dataset and++private evals. Node.js/Express, PostgreSQL (`ov_bench`), CommonJS, **no build step**.++++- Service id `bench`, port `5230`, origin `https://bench.openvibe.services`; the render origin++ `https://bench-render.openvibe.services` serves the same app; unit `openvibe-bench.service`.++- Division of authority: **AI** runs models, **Bench** judges and rates, **Report** owns the public AI++ analytics pages, **Route** learns from the results, and each room keeps its own pages.++- Public pages moved to OpenVibe.Report on 2026-11-08: `https://openvibe.report/ai`,++ `/ai/<benchmark>`, `/ai/models/<model>`. This origin's old public HTML answers `301` there; its API,++ MCP endpoint, render origin, badges, embeds and **owner** pages stay here.++++## What it owns++++- Benchmark kinds, tasks, a benchmark's drafts and **frozen versions** (`bench.benchmark@1`), entries++ and judged rounds; private by default, an owner-only prompt set, a share link and an audience.++- Votes and judgements, including anonymous votes, daily rounds, battles and the pick flow.++- Ratings: `GET /api/v1/boards/:kindId`, `GET /api/v1/datasets/:kindId.jsonl`,++ `GET /api/v1/ratings/:kindId` (the shared public read other sites consume).++- The registry API (`/api/v1/benchmarks…`), showcase, gallery, badges, embeds, watches, imports and++ variant evidence.++- `POST /mcp`, `/openapi.json`, `/llms.txt`, the AI section data Report reads.++- Owner-only pages: `/benchmarks/mine` (default audience, "apply to all", Share), `/benchmarks/new`,++ `/b/<token>` unlisted links, and the noindex task/run pages.++++Not owned: model execution (AI), the site's analytics presentation (Report), the learning loop++(Route), each room's product pages.++++## Who calls it, and whom it calls++++- Callers: OpenVibe.Report (public AI pages and `/ai/method`), Live and other rooms (boards/ratings,++ badges, embeds), AI (`bench.rating.changed`, `GET /api/v1/boards/:kind?type=elo|cost` for routed++ evidence), and people/agents through the API and MCP.++- Depends on: Network (JWKS, user/service tokens, SSO with PKCE in `server/auth/sso.js`); AI++ (`server/bench/ai.js` runs benchmark tasks); Events (outbox publisher, `bench.rating.changed`);++ PostgreSQL through `openvibe-sdk/db`; Valkey optional through `openvibe-sdk/valkey`.++++## Repository map++++- **Entry:** [server/index.js](server/index.js) — `createApp()`, graceful shutdown of keys, jobs,++ daily rounds, EEE sync, fidelity job, publisher, telemetry.++- **App:** [server/app.js](server/app.js) — helmet/CSP, cookies, rate limits, `/auth`, `/internal/events`,++ `/shared`, `/api/v1` (api, registry, variant evidence, showcase, arena, gallery), `/api`, `/render`,++ `/vendor`, pages, `/mcp`, `/badge`, `/embed`.++- **Config:** [server/config.js](server/config.js), [server/variant-evidence-config.js](server/variant-evidence-config.js).++- **Database:** [server/db.js](server/db.js); migrations `0001…0017` in [migrations/](migrations/).++- **Store/domain:** [server/store.js](server/store.js), [server/bench/](server/bench/) (battles, anon,++ daily, registry, launch, seed), [server/rating/](server/rating/) (statistics, boards, style).++- **HTTP:** [server/http/](server/http/) — `api.js`, `router.js`, `registry.js`, `ratings.js`,++ `pages.js`, `badge.js`, `embed.js`, `watch.js`, `pick.js`, `arena.js`, `gallery.js`, `render.js`.++- **Auth:** [server/auth/keys.js](server/auth/keys.js), [server/auth/sso.js](server/auth/sso.js),++ [server/http/principal.js](server/http/principal.js).++- **Frontend:** [public/js/bench.js](public/js/bench.js), [public/css/bench.css](public/css/bench.css);++ server-rendered pages in [server/render/](server/render/).++- **MCP:** [server/mcp.js](server/mcp.js).++++## Commands++++```bash++npm ci++npm run dev # NODE_ENV=development node server/index.js++npm start # node server/index.js, http://127.0.0.1:5230++OV_ALLOW_SUITE=1 npm test # the suite refuses to run without the guard in some environments++npm run test:pg # same suite on PostgreSQL (BENCH_TEST_STORE=pg)++npm test -- api # only files whose name contains the word++ov test # the harness's sharded run of what the change reaches++node --check <file.js>++```++++Tests use a temporary PGlite database, an in-process mock Network and a fake AI; nothing needs the++network or a paid key.++++## Database and migrations++++PostgreSQL database `ov_bench` (host data role, ADR-035); development and tests use PGlite. Every++read/write is async through `openvibe-sdk/db`. Schema is `migrations/NNNN_*.sql`, applied at boot;++never edit a migration that has run, and add a new number. A **benchmark version is frozen**: the++document and its SHA-256 hash are immutable and addressable as `<benchmark>@<version>`; drafts are++separate. Do not "fix" a frozen version in place.++++## Auth++++A person is the `bench_at` session cookie (with `bench_rt`, `bench_oauth`) or a Network user token as++`Bearer`; a service sends a client-credentials token with the `bench.*` capability. A **session write++must come from `bench.openvibe.services` itself** (same-origin). Capability ids come from the pinned++`openvibe-contracts`; an unknown id is refused. Anonymous browser reads are allowed only on the++public rating/board/embed paths, which return no prompts, answers, voters or site identifiers.++++## Frontend loading model++++Server-rendered HTML (no SPA framework, no bundler, no transpiler) plus one page script,++[public/js/bench.js](public/js/bench.js); `/shared/*` comes from the pinned `openvibe-shared` tarball++through its serve handler. CSP and cache policy are in [server/app.js](server/app.js) and++`openvibe-shared/cache-policy`. The `bench-render.` vhost exists for rendering/embeds; it is the same++process, not a second app.++++## Deploy++++Unit [deploy/systemd/openvibe-bench.service](deploy/systemd/openvibe-bench.service) runs++`/opt/openvibe.bench/server/index.js` on `127.0.0.1:5230`, env `/etc/openvibe/bench.env`; two vhosts,++[deploy/nginx/bench.openvibe.services.conf](deploy/nginx/bench.openvibe.services.conf) and++[deploy/nginx/bench-render.openvibe.services.conf](deploy/nginx/bench-render.openvibe.services.conf).++The repository does not record an `ovhost deploy` command; keep the unit and vhost in sync with++`deploy/`. Migrations run at boot; no secret in a file or a log.++++## Ratings rules (the part people rely on)++++- Every board is a Bradley–Terry fit over votes, centred on 1000, with a bootstrap giving each model++ a rating interval and a 95% rank interval ([docs/index.md](docs/index.md)).++- `GET /api/v1/ratings/:kindId?category=all` omits models with fewer than 20 eligible games,++ including their counts; clients must **not average** ratings from different kinds or sources++ ([docs/shared-ratings.md](docs/shared-ratings.md)).++- The dataset (`/api/v1/datasets/:kindId.jsonl`) carries the same numbers as the board.++- No prompts, answers, voters or site identifiers leave through the public read.++++## Tests++++`npm test` runs `test/run.js`, each `test/*.test.js` in its own process with a 90 s timeout through++`openvibe-shared/test-runner`; failure of any file fails the run. The suite pins the registry/frozen++versions and hashes, daily rounds, anonymous votes, boards, the dataset and JSON-LD, Report-move++redirects, MCP, OpenAPI, badges, embeds, privacy and the contracts pin. `ov test` runs the same.++++## Conventions++++- CommonJS, 4-space indent, single quotes, semicolons; factories (`createApp`, `createApi`,++ `createStore`) and injectable clock/fetch/log so tests need no network.++- Errors are `application/problem+json` with a stable `code`; never leak SQL or upstream text.++- Contracts and enums come from `openvibe-contracts`; bump the pin before using a new id.++- Keep the API backward-compatible: Report, AI, Live and rooms read these endpoints.++++## Pitfalls++++- **Frozen means frozen.** A benchmark version is immutable and hashed; editing it breaks every run++ and judgement that names it.++- **The public pages are Report's.** Only owner, unlisted and private pages belong on this origin;++ redirect the rest instead of re-growing HTML.++- **Same-origin writes.** A session write from another origin is refused; do not relax it for embeds.++- **Minimum 20 games** and exact identity: a small or fuzzy sample is omitted, never shown as a weak++ rating.++- **Private benchmarks stay private.** Prompts are owner-only; a share link is the only door.++- **`OV_ALLOW_SUITE=1`** is required by the suite harness in this environment; a plain `npm test` may++ refuse, which is not a test failure to "fix" by disabling the guard.++- **`bench-render.` is the same service.** A change there is a change here.++++## Documentation++++- [README.md](README.md) — what works and the API summary.++- [docs/index.md](docs/index.md) — the rating method and intervals.++- [docs/api.md](docs/api.md) — the API, auth and the frozen/unlisted distinction.++- [docs/benchmarks.md](docs/benchmarks.md) — writing and versioning a benchmark.++- [docs/shared-ratings.md](docs/shared-ratings.md) — the public rating read other sites consume.nnew file mode 100644@@@ -0,0 +1 @@
new file mode 100644
new file mode 100644
@@ -0,0 +1,155 @@
@@ -0,0 +1,155 @@
+# AGENTS.md — OpenVibe.Bench
+
+## Project overview
+
+The shared AI benchmark and comparison engine for every OpenVibe room: kinds, tasks, pinned
+benchmark versions, entries, judgements, the Bradley–Terry ratings, boards, the open dataset and
+private evals. Node.js/Express, PostgreSQL (`ov_bench`), CommonJS, **no build step**.
+
+- Service id `bench`, port `5230`, origin `https://bench.openvibe.services`; the render origin
+ `https://bench-render.openvibe.services` serves the same app; unit `openvibe-bench.service`.
+- Division of authority: **AI** runs models, **Bench** judges and rates, **Report** owns the public AI
+ analytics pages, **Route** learns from the results, and each room keeps its own pages.
+- Public pages moved to OpenVibe.Report on 2026-11-08: `https://openvibe.report/ai`,
+ `/ai/<benchmark>`, `/ai/models/<model>`. This origin's old public HTML answers `301` there; its API,
+ MCP endpoint, render origin, badges, embeds and **owner** pages stay here.
+
+## What it owns
+
+- Benchmark kinds, tasks, a benchmark's drafts and **frozen versions** (`bench.benchmark@1`), entries
+ and judged rounds; private by default, an owner-only prompt set, a share link and an audience.
+- Votes and judgements, including anonymous votes, daily rounds, battles and the pick flow.
+- Ratings: `GET /api/v1/boards/:kindId`, `GET /api/v1/datasets/:kindId.jsonl`,
+ `GET /api/v1/ratings/:kindId` (the shared public read other sites consume).
+- The registry API (`/api/v1/benchmarks…`), showcase, gallery, badges, embeds, watches, imports and
+ variant evidence.
+- `POST /mcp`, `/openapi.json`, `/llms.txt`, the AI section data Report reads.
+- Owner-only pages: `/benchmarks/mine` (default audience, "apply to all", Share), `/benchmarks/new`,
+ `/b/<token>` unlisted links, and the noindex task/run pages.
+
+Not owned: model execution (AI), the site's analytics presentation (Report), the learning loop
+(Route), each room's product pages.
+
+## Who calls it, and whom it calls
+
+- Callers: OpenVibe.Report (public AI pages and `/ai/method`), Live and other rooms (boards/ratings,
+ badges, embeds), AI (`bench.rating.changed`, `GET /api/v1/boards/:kind?type=elo|cost` for routed
+ evidence), and people/agents through the API and MCP.
+- Depends on: Network (JWKS, user/service tokens, SSO with PKCE in `server/auth/sso.js`); AI
+ (`server/bench/ai.js` runs benchmark tasks); Events (outbox publisher, `bench.rating.changed`);
+ PostgreSQL through `openvibe-sdk/db`; Valkey optional through `openvibe-sdk/valkey`.
+
+## Repository map
+
+- **Entry:** [server/index.js](server/index.js) — `createApp()`, graceful shutdown of keys, jobs,
+ daily rounds, EEE sync, fidelity job, publisher, telemetry.
+- **App:** [server/app.js](server/app.js) — helmet/CSP, cookies, rate limits, `/auth`, `/internal/events`,
+ `/shared`, `/api/v1` (api, registry, variant evidence, showcase, arena, gallery), `/api`, `/render`,
+ `/vendor`, pages, `/mcp`, `/badge`, `/embed`.
+- **Config:** [server/config.js](server/config.js), [server/variant-evidence-config.js](server/variant-evidence-config.js).
+- **Database:** [server/db.js](server/db.js); migrations `0001…0017` in [migrations/](migrations/).
+- **Store/domain:** [server/store.js](server/store.js), [server/bench/](server/bench/) (battles, anon,
+ daily, registry, launch, seed), [server/rating/](server/rating/) (statistics, boards, style).
+- **HTTP:** [server/http/](server/http/) — `api.js`, `router.js`, `registry.js`, `ratings.js`,
+ `pages.js`, `badge.js`, `embed.js`, `watch.js`, `pick.js`, `arena.js`, `gallery.js`, `render.js`.
+- **Auth:** [server/auth/keys.js](server/auth/keys.js), [server/auth/sso.js](server/auth/sso.js),
+ [server/http/principal.js](server/http/principal.js).
+- **Frontend:** [public/js/bench.js](public/js/bench.js), [public/css/bench.css](public/css/bench.css);
+ server-rendered pages in [server/render/](server/render/).
+- **MCP:** [server/mcp.js](server/mcp.js).
+
+## Commands
+
+```bash
+npm ci
+npm run dev # NODE_ENV=development node server/index.js
+npm start # node server/index.js, http://127.0.0.1:5230
+OV_ALLOW_SUITE=1 npm test # the suite refuses to run without the guard in some environments
+npm run test:pg # same suite on PostgreSQL (BENCH_TEST_STORE=pg)
+npm test -- api # only files whose name contains the word
+ov test # the harness's sharded run of what the change reaches
+node --check <file.js>
+```
+
+Tests use a temporary PGlite database, an in-process mock Network and a fake AI; nothing needs the
+network or a paid key.
+
+## Database and migrations
+
+PostgreSQL database `ov_bench` (host data role, ADR-035); development and tests use PGlite. Every
+read/write is async through `openvibe-sdk/db`. Schema is `migrations/NNNN_*.sql`, applied at boot;
+never edit a migration that has run, and add a new number. A **benchmark version is frozen**: the
+document and its SHA-256 hash are immutable and addressable as `<benchmark>@<version>`; drafts are
+separate. Do not "fix" a frozen version in place.
+
+## Auth
+
+A person is the `bench_at` session cookie (with `bench_rt`, `bench_oauth`) or a Network user token as
+`Bearer`; a service sends a client-credentials token with the `bench.*` capability. A **session write
+must come from `bench.openvibe.services` itself** (same-origin). Capability ids come from the pinned
+`openvibe-contracts`; an unknown id is refused. Anonymous browser reads are allowed only on the
+public rating/board/embed paths, which return no prompts, answers, voters or site identifiers.
+
+## Frontend loading model
+
+Server-rendered HTML (no SPA framework, no bundler, no transpiler) plus one page script,
+[public/js/bench.js](public/js/bench.js); `/shared/*` comes from the pinned `openvibe-shared` tarball
+through its serve handler. CSP and cache policy are in [server/app.js](server/app.js) and
+`openvibe-shared/cache-policy`. The `bench-render.` vhost exists for rendering/embeds; it is the same
+process, not a second app.
+
+## Deploy
+
+Unit [deploy/systemd/openvibe-bench.service](deploy/systemd/openvibe-bench.service) runs
+`/opt/openvibe.bench/server/index.js` on `127.0.0.1:5230`, env `/etc/openvibe/bench.env`; two vhosts,
+[deploy/nginx/bench.openvibe.services.conf](deploy/nginx/bench.openvibe.services.conf) and
+[deploy/nginx/bench-render.openvibe.services.conf](deploy/nginx/bench-render.openvibe.services.conf).
+The repository does not record an `ovhost deploy` command; keep the unit and vhost in sync with
+`deploy/`. Migrations run at boot; no secret in a file or a log.
+
+## Ratings rules (the part people rely on)
+
+- Every board is a Bradley–Terry fit over votes, centred on 1000, with a bootstrap giving each model
+ a rating interval and a 95% rank interval ([docs/index.md](docs/index.md)).
+- `GET /api/v1/ratings/:kindId?category=all` omits models with fewer than 20 eligible games,
+ including their counts; clients must **not average** ratings from different kinds or sources
+ ([docs/shared-ratings.md](docs/shared-ratings.md)).
+- The dataset (`/api/v1/datasets/:kindId.jsonl`) carries the same numbers as the board.
+- No prompts, answers, voters or site identifiers leave through the public read.
+
+## Tests
+
+`npm test` runs `test/run.js`, each `test/*.test.js` in its own process with a 90 s timeout through
+`openvibe-shared/test-runner`; failure of any file fails the run. The suite pins the registry/frozen
+versions and hashes, daily rounds, anonymous votes, boards, the dataset and JSON-LD, Report-move
+redirects, MCP, OpenAPI, badges, embeds, privacy and the contracts pin. `ov test` runs the same.
+
+## Conventions
+
+- CommonJS, 4-space indent, single quotes, semicolons; factories (`createApp`, `createApi`,
+ `createStore`) and injectable clock/fetch/log so tests need no network.
+- Errors are `application/problem+json` with a stable `code`; never leak SQL or upstream text.
+- Contracts and enums come from `openvibe-contracts`; bump the pin before using a new id.
+- Keep the API backward-compatible: Report, AI, Live and rooms read these endpoints.
+
+## Pitfalls
+
+- **Frozen means frozen.** A benchmark version is immutable and hashed; editing it breaks every run
+ and judgement that names it.
+- **The public pages are Report's.** Only owner, unlisted and private pages belong on this origin;
+ redirect the rest instead of re-growing HTML.
+- **Same-origin writes.** A session write from another origin is refused; do not relax it for embeds.
+- **Minimum 20 games** and exact identity: a small or fuzzy sample is omitted, never shown as a weak
+ rating.
+- **Private benchmarks stay private.** Prompts are owner-only; a share link is the only door.
+- **`OV_ALLOW_SUITE=1`** is required by the suite harness in this environment; a plain `npm test` may
+ refuse, which is not a test failure to "fix" by disabling the guard.
+- **`bench-render.` is the same service.** A change there is a change here.
+
+## Documentation
+
+- [README.md](README.md) — what works and the API summary.
+- [docs/index.md](docs/index.md) — the rating method and intervals.
+- [docs/api.md](docs/api.md) — the API, auth and the frozen/unlisted distinction.
+- [docs/benchmarks.md](docs/benchmarks.md) — writing and versioning a benchmark.
+- [docs/shared-ratings.md](docs/shared-ratings.md) — the public rating read other sites consume.
new file mode 100644
new file mode 100644
@@ -0,0 +1 @@
@@ -0,0 +1 @@