# Changelog

## Unreleased

- **Whole tickets, the gate (AL8).** A new kind, `codes.ticket`, scores a whole ticket — a goal and a typed check
  in `agent.goal@1`'s shape, with a token/turn/minute budget and a no-progress limit — over 3 to 5 trials per model
  and effort. A run is one trial; a task's score is 100 when the check exited 0 and 0 when it did not, and the
  check's test patch is applied only after the worker stops, so the merged pull request's own test grades the work.
  `GET /api/v1/rankings?benchmark=…&metric=tickets&effort=…` (also `pass_k`, `pass@k`) reports pass@1, pass^k (every
  one of k trials passed) and cost per solved by model and effort, with the per-ticket rows. The first suite
  (`fixtures/codes.ticket/openvibe-merged-2026-11.json`) holds twelve small tickets from merged OpenVibe pull
  requests across Bench, Codes, AI, Actor, Exchange, Report and Network; each goal is the change in words and each
  check is the pull request's own test. `scripts/ticket-suite.js` prints the plan and runs only with `--run`; the
  default models are `deepseek-flash` and `local`, and nothing is scheduled.
- **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, the
  score of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each,
  no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the
  dry run: every call, the own-page price estimate and whether it fits the owner's daily cap. `POST
  /api/v1/fidelity/run` runs one bounded pass; every call goes through OpenVibe.AI's normal generate path, pinned to
  the exact offer, with a durable idempotency key per prompt and week. Off by default (`BENCH_FIDELITY=1`), daily
  capped, and a weekly due-based job when on. The objective set is public at `GET /api/v1/fidelity/objective-set`;
  answers are never stored, only which item passed.
- **The 18+ board (`chat.willing`).** A new adult kind and `GET /api/v1/boards/chat.willing`, marked `adult: true` on the board and every row. It reads OpenVibe.AI's measured refusal and hedge rates (n and Wilson intervals) on the published 60-prompt lawful adult set, plus the abliteration tax for each declared base/abliterated pair: the same objective set on both, with the score difference and its Newcombe interval. A non-willingness type on the adult kind is refused; when AI cannot answer, the board says the measurements are unavailable and invents no rows.
- **Every Eval Ever (B2).** Bench imports the public EEE datastore (`github.com/evaleval/every_eval_ever`,
  `evaleval/EEE_datastore`) as *external* results: the model (joined by Report's rule — maker prefix and a date or
  channel suffix dropped), the harness and its version, the task and its version, few-shot, the metric, the score,
  the source link and the licence. A record whose licence does not allow redistribution is dropped and counted.
  The rows live in `bench_eee_records` and are served by `GET /api/v1/external/records` with `external: true`; they
  never enter Bench's ratings, boards or medians. A weekly, resumable, bounded job (`BENCH_EEE_SYNC`) imports them.
  Bench's own public runs are exported as EEE `evaluation_run` records at `GET /api/v1/datasets/eee` (JSONL, paged;
  per-sample scores at `/api/v1/datasets/eee/runs/:id.samples.jsonl`).
- **Two more harnesses (B7).** HF lighteval and Stanford HELM results join Inspect AI, Harbor, the
  lm-evaluation-harness and promptfoo. Every import keeps the harness name and version, each task's version, the
  few-shot and the seed on the run.
- **The public pages moved to OpenVibe.Report's AI section (2026-11-08).** `/`, `/docs`, `/benchmarks`, `/models/*`, `/compare` and the other page routes answer `301` to `https://openvibe.report/ai` counterparts; the API, MCP, render origin, badges and embeds stay on bench.openvibe.services. New public reads feed Report: `GET /api/v1/runs`, `/api/v1/overview` (with every public model and its last run) and `/api/v1/models/:model/summary` (per-benchmark and per-task medians, cost, recent runs). `/api/v1/benchmarks` now carries each benchmark's runs and last run.
- **The Exchange arena (`chat.arena`).** A new kind, and a first-party API only OpenVibe.Exchange may call:
  `POST /api/v1/arena/rounds` builds a round from two answers the Exchange already has, and
  `POST /api/v1/arena/rounds/:id/votes` records the person's own `human_pairwise` judgement from
  `X-OV-Subject`. The service owns the round, so the person can vote on it; a second vote is 409 and the round
  is revealed. The arena's rounds skip a person's daily quota and have their own daily cap.
- **Task-category ratings.** A judgement now also updates the task's `task_class` category (code, writing, chat,
  reasoning, translation, analysis) beside `all` and the voter kind, and `GET /api/v1/boards/:kindId?category=…`
  reads it. A category board lists only the models that played in it.

## 0.1.0 — 2026-10-28

- OpenVibe.Bench v1: the shared AI benchmark and comparison engine (plan 2026-10-28 §4.12).
- Fifteen kinds seeded from code; pinned, reproducible rounds; blind pairwise judgements with one vote per person per round and no vote on your own round.
- Elo, pareto and cost boards; open dataset per kind without voter ids; private evals (free daily cap, no charging in v1).
- MCP endpoint (`POST /mcp`) and events outbox (`bench.round.finished`, `bench.judgement.recorded`, `bench.rating.changed`).