openvibers/openvibe.bench AGENTS.md

AGENTS.md

main 9.1 KB · 156 lines · markdown

Raw HistoryBlame Repository
1# AGENTS.md — OpenVibe.Bench
2
3## Project overview
4
5The shared AI benchmark and comparison engine for every OpenVibe room: kinds, tasks, pinned
6benchmark versions, entries, judgements, the Bradley–Terry ratings, boards, the open dataset and
7private evals. Node.js/Express, PostgreSQL (`ov_bench`), CommonJS, **no build step**.
8
9- Service id `bench`, port `5230`, origin `https://bench.openvibe.services`; the render origin
10 `https://bench-render.openvibe.services` serves the same app; unit `openvibe-bench.service`.
11- Division of authority: **AI** runs models, **Bench** judges and rates, **Report** owns the public AI
12 analytics pages, **Route** learns from the results, and each room keeps its own pages.
13- Public pages moved to OpenVibe.Report on 2026-11-08: `https://openvibe.report/ai`,
14 `/ai/<benchmark>`, `/ai/models/<model>`. This origin's old public HTML answers `301` there; its API,
15 MCP endpoint, render origin, badges, embeds and **owner** pages stay here.
16
17## What it owns
18
19- Benchmark kinds, tasks, a benchmark's drafts and **frozen versions** (`bench.benchmark@1`), entries
20 and judged rounds; private by default, an owner-only prompt set, a share link and an audience.
21- Votes and judgements, including anonymous votes, daily rounds, battles and the pick flow.
22- Ratings: `GET /api/v1/boards/:kindId`, `GET /api/v1/datasets/:kindId.jsonl`,
23 `GET /api/v1/ratings/:kindId` (the shared public read other sites consume).
24- The registry API (`/api/v1/benchmarks…`), showcase, gallery, badges, embeds, watches, imports and
25 variant evidence.
26- `POST /mcp`, `/openapi.json`, `/llms.txt`, the AI section data Report reads.
27- Owner-only pages: `/benchmarks/mine` (default audience, "apply to all", Share), `/benchmarks/new`,
28 `/b/<token>` unlisted links, and the noindex task/run pages.
29
30Not owned: model execution (AI), the site's analytics presentation (Report), the learning loop
31(Route), each room's product pages.
32
33## Who calls it, and whom it calls
34
35- Callers: OpenVibe.Report (public AI pages and `/ai/method`), Live and other rooms (boards/ratings,
36 badges, embeds), AI (`bench.rating.changed`, `GET /api/v1/boards/:kind?type=elo|cost` for routed
37 evidence), and people/agents through the API and MCP.
38- Depends on: Network (JWKS, user/service tokens, SSO with PKCE in `server/auth/sso.js`); AI
39 (`server/bench/ai.js` runs benchmark tasks); Events (outbox publisher, `bench.rating.changed`);
40 PostgreSQL through `openvibe-sdk/db`; Valkey optional through `openvibe-sdk/valkey`.
41
42## Repository map
43
44- **Entry:** [server/index.js](server/index.js) — `createApp()`, graceful shutdown of keys, jobs,
45 daily rounds, EEE sync, fidelity job, publisher, telemetry.
46- **App:** [server/app.js](server/app.js) — helmet/CSP, cookies, rate limits, `/auth`, `/internal/events`,
47 `/shared`, `/api/v1` (api, registry, variant evidence, showcase, arena, gallery), `/api`, `/render`,
48 `/vendor`, pages, `/mcp`, `/badge`, `/embed`.
49- **Config:** [server/config.js](server/config.js), [server/variant-evidence-config.js](server/variant-evidence-config.js).
50- **Database:** [server/db.js](server/db.js); migrations `0001…0017` in [migrations/](migrations/).
51- **Store/domain:** [server/store.js](server/store.js), [server/bench/](server/bench/) (battles, anon,
52 daily, registry, launch, seed), [server/rating/](server/rating/) (statistics, boards, style).
53- **HTTP:** [server/http/](server/http/) — `api.js`, `router.js`, `registry.js`, `ratings.js`,
54 `pages.js`, `badge.js`, `embed.js`, `watch.js`, `pick.js`, `arena.js`, `gallery.js`, `render.js`.
55- **Auth:** [server/auth/keys.js](server/auth/keys.js), [server/auth/sso.js](server/auth/sso.js),
56 [server/http/principal.js](server/http/principal.js).
57- **Frontend:** [public/js/bench.js](public/js/bench.js), [public/css/bench.css](public/css/bench.css);
58 server-rendered pages in [server/render/](server/render/).
59- **MCP:** [server/mcp.js](server/mcp.js).
60
61## Commands
62
63```bash
64npm ci
65npm run dev # NODE_ENV=development node server/index.js
66npm start # node server/index.js, http://127.0.0.1:5230
67OV_ALLOW_SUITE=1 npm test # the suite refuses to run without the guard in some environments
68npm run test:pg # same suite on PostgreSQL (BENCH_TEST_STORE=pg)
69npm test -- api # only files whose name contains the word
70ov test # the harness's sharded run of what the change reaches
71node --check <file.js>
72```
73
74Tests use a temporary PGlite database, an in-process mock Network and a fake AI; nothing needs the
75network or a paid key.
76
77## Database and migrations
78
79PostgreSQL database `ov_bench` (host data role, ADR-035); development and tests use PGlite. Every
80read/write is async through `openvibe-sdk/db`. Schema is `migrations/NNNN_*.sql`, applied at boot;
81never edit a migration that has run, and add a new number. A **benchmark version is frozen**: the
82document and its SHA-256 hash are immutable and addressable as `<benchmark>@<version>`; drafts are
83separate. Do not "fix" a frozen version in place.
84
85## Auth
86
87A person is the `bench_at` session cookie (with `bench_rt`, `bench_oauth`) or a Network user token as
88`Bearer`; a service sends a client-credentials token with the `bench.*` capability. A **session write
89must come from `bench.openvibe.services` itself** (same-origin). Capability ids come from the pinned
90`openvibe-contracts`; an unknown id is refused. Anonymous browser reads are allowed only on the
91public rating/board/embed paths, which return no prompts, answers, voters or site identifiers.
92
93## Frontend loading model
94
95Server-rendered HTML (no SPA framework, no bundler, no transpiler) plus one page script,
96[public/js/bench.js](public/js/bench.js); `/shared/*` comes from the pinned `openvibe-shared` tarball
97through its serve handler. CSP and cache policy are in [server/app.js](server/app.js) and
98`openvibe-shared/cache-policy`. The `bench-render.` vhost exists for rendering/embeds; it is the same
99process, not a second app.
100
101## Deploy
102
103Unit [deploy/systemd/openvibe-bench.service](deploy/systemd/openvibe-bench.service) runs
104`/opt/openvibe.bench/server/index.js` on `127.0.0.1:5230`, env `/etc/openvibe/bench.env`; two vhosts,
105[deploy/nginx/bench.openvibe.services.conf](deploy/nginx/bench.openvibe.services.conf) and
106[deploy/nginx/bench-render.openvibe.services.conf](deploy/nginx/bench-render.openvibe.services.conf).
107The repository does not record an `ovhost deploy` command; keep the unit and vhost in sync with
108`deploy/`. Migrations run at boot; no secret in a file or a log.
109
110## Ratings rules (the part people rely on)
111
112- Every board is a Bradley–Terry fit over votes, centred on 1000, with a bootstrap giving each model
113 a rating interval and a 95% rank interval ([docs/index.md](docs/index.md)).
114- `GET /api/v1/ratings/:kindId?category=all` omits models with fewer than 20 eligible games,
115 including their counts; clients must **not average** ratings from different kinds or sources
116 ([docs/shared-ratings.md](docs/shared-ratings.md)).
117- The dataset (`/api/v1/datasets/:kindId.jsonl`) carries the same numbers as the board.
118- No prompts, answers, voters or site identifiers leave through the public read.
119
120## Tests
121
122`npm test` runs `test/run.js`, each `test/*.test.js` in its own process with a 90 s timeout through
123`openvibe-shared/test-runner`; failure of any file fails the run. The suite pins the registry/frozen
124versions and hashes, daily rounds, anonymous votes, boards, the dataset and JSON-LD, Report-move
125redirects, MCP, OpenAPI, badges, embeds, privacy and the contracts pin. `ov test` runs the same.
126
127## Conventions
128
129- CommonJS, 4-space indent, single quotes, semicolons; factories (`createApp`, `createApi`,
130 `createStore`) and injectable clock/fetch/log so tests need no network.
131- Errors are `application/problem+json` with a stable `code`; never leak SQL or upstream text.
132- Contracts and enums come from `openvibe-contracts`; bump the pin before using a new id.
133- Keep the API backward-compatible: Report, AI, Live and rooms read these endpoints.
134
135## Pitfalls
136
137- **Frozen means frozen.** A benchmark version is immutable and hashed; editing it breaks every run
138 and judgement that names it.
139- **The public pages are Report's.** Only owner, unlisted and private pages belong on this origin;
140 redirect the rest instead of re-growing HTML.
141- **Same-origin writes.** A session write from another origin is refused; do not relax it for embeds.
142- **Minimum 20 games** and exact identity: a small or fuzzy sample is omitted, never shown as a weak
143 rating.
144- **Private benchmarks stay private.** Prompts are owner-only; a share link is the only door.
145- **`OV_ALLOW_SUITE=1`** is required by the suite harness in this environment; a plain `npm test` may
146 refuse, which is not a test failure to "fix" by disabling the guard.
147- **`bench-render.` is the same service.** A change there is a change here.
148
149## Documentation
150
151- [README.md](README.md) — what works and the API summary.
152- [docs/index.md](docs/index.md) — the rating method and intervals.
153- [docs/api.md](docs/api.md) — the API, auth and the frozen/unlisted distinction.
154- [docs/benchmarks.md](docs/benchmarks.md) — writing and versioning a benchmark.
155- [docs/shared-ratings.md](docs/shared-ratings.md) — the public rating read other sites consume.
156