openvibers/openvibe.bench Commits

AGENTS.md: the context an agent starts from (#215)

OP OpenVibers committed 1 days ago · 02a969f284 · 2 files, +156 −0

Changes

ddiff --git a/AGENTS.md b/AGENTS.md
nnew file mode 100644
iindex 0000000..d0807fa
---- /dev/null
++++ b/AGENTS.md
@@@ -0,0 +1,155 @@
++# AGENTS.md — OpenVibe.Bench
++
++## Project overview
++
++The shared AI benchmark and comparison engine for every OpenVibe room: kinds, tasks, pinned
++benchmark versions, entries, judgements, the Bradley–Terry ratings, boards, the open dataset and
++private evals. Node.js/Express, PostgreSQL (`ov_bench`), CommonJS, **no build step**.
++
++- Service id `bench`, port `5230`, origin `https://bench.openvibe.services`; the render origin
++ `https://bench-render.openvibe.services` serves the same app; unit `openvibe-bench.service`.
++- Division of authority: **AI** runs models, **Bench** judges and rates, **Report** owns the public AI
++ analytics pages, **Route** learns from the results, and each room keeps its own pages.
++- Public pages moved to OpenVibe.Report on 2026-11-08: `https://openvibe.report/ai`,
++ `/ai/<benchmark>`, `/ai/models/<model>`. This origin's old public HTML answers `301` there; its API,
++ MCP endpoint, render origin, badges, embeds and **owner** pages stay here.
++
++## What it owns
++
++- Benchmark kinds, tasks, a benchmark's drafts and **frozen versions** (`bench.benchmark@1`), entries
++ and judged rounds; private by default, an owner-only prompt set, a share link and an audience.
++- Votes and judgements, including anonymous votes, daily rounds, battles and the pick flow.
++- Ratings: `GET /api/v1/boards/:kindId`, `GET /api/v1/datasets/:kindId.jsonl`,
++ `GET /api/v1/ratings/:kindId` (the shared public read other sites consume).
++- The registry API (`/api/v1/benchmarks…`), showcase, gallery, badges, embeds, watches, imports and
++ variant evidence.
++- `POST /mcp`, `/openapi.json`, `/llms.txt`, the AI section data Report reads.
++- Owner-only pages: `/benchmarks/mine` (default audience, "apply to all", Share), `/benchmarks/new`,
++ `/b/<token>` unlisted links, and the noindex task/run pages.
++
++Not owned: model execution (AI), the site's analytics presentation (Report), the learning loop
++(Route), each room's product pages.
++
++## Who calls it, and whom it calls
++
++- Callers: OpenVibe.Report (public AI pages and `/ai/method`), Live and other rooms (boards/ratings,
++ badges, embeds), AI (`bench.rating.changed`, `GET /api/v1/boards/:kind?type=elo|cost` for routed
++ evidence), and people/agents through the API and MCP.
++- Depends on: Network (JWKS, user/service tokens, SSO with PKCE in `server/auth/sso.js`); AI
++ (`server/bench/ai.js` runs benchmark tasks); Events (outbox publisher, `bench.rating.changed`);
++ PostgreSQL through `openvibe-sdk/db`; Valkey optional through `openvibe-sdk/valkey`.
++
++## Repository map
++
++- **Entry:** [server/index.js](server/index.js) — `createApp()`, graceful shutdown of keys, jobs,
++ daily rounds, EEE sync, fidelity job, publisher, telemetry.
++- **App:** [server/app.js](server/app.js) — helmet/CSP, cookies, rate limits, `/auth`, `/internal/events`,
++ `/shared`, `/api/v1` (api, registry, variant evidence, showcase, arena, gallery), `/api`, `/render`,
++ `/vendor`, pages, `/mcp`, `/badge`, `/embed`.
++- **Config:** [server/config.js](server/config.js), [server/variant-evidence-config.js](server/variant-evidence-config.js).
++- **Database:** [server/db.js](server/db.js); migrations `0001…0017` in [migrations/](migrations/).
++- **Store/domain:** [server/store.js](server/store.js), [server/bench/](server/bench/) (battles, anon,
++ daily, registry, launch, seed), [server/rating/](server/rating/) (statistics, boards, style).
++- **HTTP:** [server/http/](server/http/) — `api.js`, `router.js`, `registry.js`, `ratings.js`,
++ `pages.js`, `badge.js`, `embed.js`, `watch.js`, `pick.js`, `arena.js`, `gallery.js`, `render.js`.
++- **Auth:** [server/auth/keys.js](server/auth/keys.js), [server/auth/sso.js](server/auth/sso.js),
++ [server/http/principal.js](server/http/principal.js).
++- **Frontend:** [public/js/bench.js](public/js/bench.js), [public/css/bench.css](public/css/bench.css);
++ server-rendered pages in [server/render/](server/render/).
++- **MCP:** [server/mcp.js](server/mcp.js).
++
++## Commands
++
++```bash
++npm ci
++npm run dev # NODE_ENV=development node server/index.js
++npm start # node server/index.js, http://127.0.0.1:5230
++OV_ALLOW_SUITE=1 npm test # the suite refuses to run without the guard in some environments
++npm run test:pg # same suite on PostgreSQL (BENCH_TEST_STORE=pg)
++npm test -- api # only files whose name contains the word
++ov test # the harness's sharded run of what the change reaches
++node --check <file.js>
++```
++
++Tests use a temporary PGlite database, an in-process mock Network and a fake AI; nothing needs the
++network or a paid key.
++
++## Database and migrations
++
++PostgreSQL database `ov_bench` (host data role, ADR-035); development and tests use PGlite. Every
++read/write is async through `openvibe-sdk/db`. Schema is `migrations/NNNN_*.sql`, applied at boot;
++never edit a migration that has run, and add a new number. A **benchmark version is frozen**: the
++document and its SHA-256 hash are immutable and addressable as `<benchmark>@<version>`; drafts are
++separate. Do not "fix" a frozen version in place.
++
++## Auth
++
++A person is the `bench_at` session cookie (with `bench_rt`, `bench_oauth`) or a Network user token as
++`Bearer`; a service sends a client-credentials token with the `bench.*` capability. A **session write
++must come from `bench.openvibe.services` itself** (same-origin). Capability ids come from the pinned
++`openvibe-contracts`; an unknown id is refused. Anonymous browser reads are allowed only on the
++public rating/board/embed paths, which return no prompts, answers, voters or site identifiers.
++
++## Frontend loading model
++
++Server-rendered HTML (no SPA framework, no bundler, no transpiler) plus one page script,
++[public/js/bench.js](public/js/bench.js); `/shared/*` comes from the pinned `openvibe-shared` tarball
++through its serve handler. CSP and cache policy are in [server/app.js](server/app.js) and
++`openvibe-shared/cache-policy`. The `bench-render.` vhost exists for rendering/embeds; it is the same
++process, not a second app.
++
++## Deploy
++
++Unit [deploy/systemd/openvibe-bench.service](deploy/systemd/openvibe-bench.service) runs
++`/opt/openvibe.bench/server/index.js` on `127.0.0.1:5230`, env `/etc/openvibe/bench.env`; two vhosts,
++[deploy/nginx/bench.openvibe.services.conf](deploy/nginx/bench.openvibe.services.conf) and
++[deploy/nginx/bench-render.openvibe.services.conf](deploy/nginx/bench-render.openvibe.services.conf).
++The repository does not record an `ovhost deploy` command; keep the unit and vhost in sync with
++`deploy/`. Migrations run at boot; no secret in a file or a log.
++
++## Ratings rules (the part people rely on)
++
++- Every board is a Bradley–Terry fit over votes, centred on 1000, with a bootstrap giving each model
++ a rating interval and a 95% rank interval ([docs/index.md](docs/index.md)).
++- `GET /api/v1/ratings/:kindId?category=all` omits models with fewer than 20 eligible games,
++ including their counts; clients must **not average** ratings from different kinds or sources
++ ([docs/shared-ratings.md](docs/shared-ratings.md)).
++- The dataset (`/api/v1/datasets/:kindId.jsonl`) carries the same numbers as the board.
++- No prompts, answers, voters or site identifiers leave through the public read.
++
++## Tests
++
++`npm test` runs `test/run.js`, each `test/*.test.js` in its own process with a 90 s timeout through
++`openvibe-shared/test-runner`; failure of any file fails the run. The suite pins the registry/frozen
++versions and hashes, daily rounds, anonymous votes, boards, the dataset and JSON-LD, Report-move
++redirects, MCP, OpenAPI, badges, embeds, privacy and the contracts pin. `ov test` runs the same.
++
++## Conventions
++
++- CommonJS, 4-space indent, single quotes, semicolons; factories (`createApp`, `createApi`,
++ `createStore`) and injectable clock/fetch/log so tests need no network.
++- Errors are `application/problem+json` with a stable `code`; never leak SQL or upstream text.
++- Contracts and enums come from `openvibe-contracts`; bump the pin before using a new id.
++- Keep the API backward-compatible: Report, AI, Live and rooms read these endpoints.
++
++## Pitfalls
++
++- **Frozen means frozen.** A benchmark version is immutable and hashed; editing it breaks every run
++ and judgement that names it.
++- **The public pages are Report's.** Only owner, unlisted and private pages belong on this origin;
++ redirect the rest instead of re-growing HTML.
++- **Same-origin writes.** A session write from another origin is refused; do not relax it for embeds.
++- **Minimum 20 games** and exact identity: a small or fuzzy sample is omitted, never shown as a weak
++ rating.
++- **Private benchmarks stay private.** Prompts are owner-only; a share link is the only door.
++- **`OV_ALLOW_SUITE=1`** is required by the suite harness in this environment; a plain `npm test` may
++ refuse, which is not a test failure to "fix" by disabling the guard.
++- **`bench-render.` is the same service.** A change there is a change here.
++
++## Documentation
++
++- [README.md](README.md) — what works and the API summary.
++- [docs/index.md](docs/index.md) — the rating method and intervals.
++- [docs/api.md](docs/api.md) — the API, auth and the frozen/unlisted distinction.
++- [docs/benchmarks.md](docs/benchmarks.md) — writing and versioning a benchmark.
++- [docs/shared-ratings.md](docs/shared-ratings.md) — the public rating read other sites consume.
ddiff --git a/CLAUDE.md b/CLAUDE.md
nnew file mode 100644
iindex 0000000..43c994c
---- /dev/null
++++ b/CLAUDE.md
@@@ -0,0 +1 @@