openvibers/openvibe.bench CHANGELOG.md
CHANGELOG.md
| 1 | # Changelog |
| 2 | |
| 3 | ## Unreleased |
| 4 | |
| 5 | - **Whole tickets, the gate (AL8).** A new kind, `codes.ticket`, scores a whole ticket — a goal and a typed check |
| 6 | in `agent.goal@1`'s shape, with a token/turn/minute budget and a no-progress limit — over 3 to 5 trials per model |
| 7 | and effort. A run is one trial; a task's score is 100 when the check exited 0 and 0 when it did not, and the |
| 8 | check's test patch is applied only after the worker stops, so the merged pull request's own test grades the work. |
| 9 | `GET /api/v1/rankings?benchmark=…&metric=tickets&effort=…` (also `pass_k`, `pass@k`) reports pass@1, pass^k (every |
| 10 | one of k trials passed) and cost per solved by model and effort, with the per-ticket rows. The first suite |
| 11 | (`fixtures/codes.ticket/openvibe-merged-2026-11.json`) holds twelve small tickets from merged OpenVibe pull |
| 12 | requests across Bench, Codes, AI, Actor, Exchange, Report and Network; each goal is the change in words and each |
| 13 | check is the pull request's own test. `scripts/ticket-suite.js` prints the plan and runs only with `--run`; the |
| 14 | default models are `deepseek-flash` and `local`, and nothing is scheduled. |
| 15 | - **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, the |
| 16 | score of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each, |
| 17 | no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the |
| 18 | dry run: every call, the own-page price estimate and whether it fits the owner's daily cap. `POST |
| 19 | /api/v1/fidelity/run` runs one bounded pass; every call goes through OpenVibe.AI's normal generate path, pinned to |
| 20 | the exact offer, with a durable idempotency key per prompt and week. Off by default (`BENCH_FIDELITY=1`), daily |
| 21 | capped, and a weekly due-based job when on. The objective set is public at `GET /api/v1/fidelity/objective-set`; |
| 22 | answers are never stored, only which item passed. |
| 23 | - **The 18+ board (`chat.willing`).** A new adult kind and `GET /api/v1/boards/chat.willing`, marked `adult: true` on the board and every row. It reads OpenVibe.AI's measured refusal and hedge rates (n and Wilson intervals) on the published 60-prompt lawful adult set, plus the abliteration tax for each declared base/abliterated pair: the same objective set on both, with the score difference and its Newcombe interval. A non-willingness type on the adult kind is refused; when AI cannot answer, the board says the measurements are unavailable and invents no rows. |
| 24 | - **Every Eval Ever (B2).** Bench imports the public EEE datastore (`github.com/evaleval/every_eval_ever`, |
| 25 | `evaleval/EEE_datastore`) as *external* results: the model (joined by Report's rule — maker prefix and a date or |
| 26 | channel suffix dropped), the harness and its version, the task and its version, few-shot, the metric, the score, |
| 27 | the source link and the licence. A record whose licence does not allow redistribution is dropped and counted. |
| 28 | The rows live in `bench_eee_records` and are served by `GET /api/v1/external/records` with `external: true`; they |
| 29 | never enter Bench's ratings, boards or medians. A weekly, resumable, bounded job (`BENCH_EEE_SYNC`) imports them. |
| 30 | Bench's own public runs are exported as EEE `evaluation_run` records at `GET /api/v1/datasets/eee` (JSONL, paged; |
| 31 | per-sample scores at `/api/v1/datasets/eee/runs/:id.samples.jsonl`). |
| 32 | - **Two more harnesses (B7).** HF lighteval and Stanford HELM results join Inspect AI, Harbor, the |
| 33 | lm-evaluation-harness and promptfoo. Every import keeps the harness name and version, each task's version, the |
| 34 | few-shot and the seed on the run. |
| 35 | - **The public pages moved to OpenVibe.Report's AI section (2026-11-08).** `/`, `/docs`, `/benchmarks`, `/models/*`, `/compare` and the other page routes answer `301` to `https://openvibe.report/ai` counterparts; the API, MCP, render origin, badges and embeds stay on bench.openvibe.services. New public reads feed Report: `GET /api/v1/runs`, `/api/v1/overview` (with every public model and its last run) and `/api/v1/models/:model/summary` (per-benchmark and per-task medians, cost, recent runs). `/api/v1/benchmarks` now carries each benchmark's runs and last run. |
| 36 | - **The Exchange arena (`chat.arena`).** A new kind, and a first-party API only OpenVibe.Exchange may call: |
| 37 | `POST /api/v1/arena/rounds` builds a round from two answers the Exchange already has, and |
| 38 | `POST /api/v1/arena/rounds/:id/votes` records the person's own `human_pairwise` judgement from |
| 39 | `X-OV-Subject`. The service owns the round, so the person can vote on it; a second vote is 409 and the round |
| 40 | is revealed. The arena's rounds skip a person's daily quota and have their own daily cap. |
| 41 | - **Task-category ratings.** A judgement now also updates the task's `task_class` category (code, writing, chat, |
| 42 | reasoning, translation, analysis) beside `all` and the voter kind, and `GET /api/v1/boards/:kindId?category=…` |
| 43 | reads it. A category board lists only the models that played in it. |
| 44 | |
| 45 | ## 0.1.0 — 2026-10-28 |
| 46 | |
| 47 | - OpenVibe.Bench v1: the shared AI benchmark and comparison engine (plan 2026-10-28 §4.12). |
| 48 | - Fifteen kinds seeded from code; pinned, reproducible rounds; blind pairwise judgements with one vote per person per round and no vote on your own round. |
| 49 | - Elo, pareto and cost boards; open dataset per kind without voter ids; private evals (free daily cap, no charging in v1). |
| 50 | - MCP endpoint (`POST /mcp`) and events outbox (`bench.round.finished`, `bench.judgement.recorded`, `bench.rating.changed`). |
| 51 |