openvibers/openvibe.ai Commits
AI's routing learns from Bench: bench.rating.changed evidence, opt-in placement prefers the cheapest offer tied with the leader (#187)
Changes
@@@ -48,6 +48,18 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000#AI_PERSON_SESSION_TTL_MIN=1440# Audience a Network user token must carry#AI_SSO_AUDIENCE=openvibe.network++# ── Events (the ai.run.* outbox relay and the Bench ratings subscription) ──++# EVENTS_URL is where AI publishes its own events and ensures its subscription. AI_EVENTS_SECRET is the++# v2 signing secret Events hands back on that subscription (32+ characters; comma-separated during rotation);++# without it POST /internal/events is off. AI_EVENTS_SUBSCRIBE=0 skips creating the subscription (an++# operator/script may create topic bench.rating.changed → http://127.0.0.1:4700/internal/events).++#EVENTS_URL=http://127.0.0.1:4300++#AI_EVENTS_SECRET=++#AI_EVENTS_SUBSCRIBE=on++# Where a sparse bench.rating.changed is filled from (boards, cost per solved).++#AI_BENCH_URL=http://127.0.0.1:5230++#AI_BENCH_TIMEOUT_MS=15000++# ── Billing (plan T5 lane F) ──# One platform.usage-sample@1 reading per accounted run, posted to billing.usage.record with AI's# service token, so Network must grant client `ai` billing.usage.record on openvibe.billing. Readings@@@ -43,7 +43,8 @@ owns publication truth: Wiki, Blog, News, Live, … decide what to publish.- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`- OpenVibe.Network (JWKS, service tokens, SSO for the operator console, user modules)--- OpenVibe.Events (the `ai.run.*` outbox relay; off without `EVENTS_URL`)++- OpenVibe.Events (the `ai.run.*` outbox relay, and the signed `bench.rating.changed` subscription; off without `EVENTS_URL`)++- OpenVibe.Bench (`bench.rating.changed` and `GET /api/v1/boards/:kind?type=elo|cost`, for opted-in routes)- OpenVibe.Media (recordings `media.analyze` reads, over allow-listed https URLs or signed URLs)- provider APIs configured by the operator (OpenAI-compatible, OpenAI Responses, Anthropic, the HTTP seam), whisper.cppand the local model server `openvibe-llm.service` on this host@@@ -59,8 +60,9 @@ Implemented here (the service manifest's `capabilities`, audience `openvibe.ai`;registered since openvibe-contracts 0.109.0). A service token's `ns`claim limits which workflow namespaces it may run; an app token is fixed to its project's.--Called elsewhere, as the service principal `ai`: `events.event.publish` (Events, the outbox relay)--and `network.modules.read` / `network.modules.write` for `ai.preferences` and `ai.usage_summary`++Called elsewhere, as the service principal `ai`: `events.event.publish` (Events, the outbox relay) and++`events.subscription.manage` (Events, creating AI's own `bench.rating.changed` subscription), and++`network.modules.read` / `network.modules.write` for `ai.preferences` and `ai.usage_summary`(Network). Other services grant `ai` their own draft capabilities (for example`blog.post.create`, `news.story.revise`, `reviews.summary.propose`) to receive drafts.@@@ -362,6 +364,21 @@ failure; a fallback that answers is recorded on the run (`fallback_used`), in th(objective, reasons, selected, and every candidate with its estimated cost, latency and excludedreason).++### Bench evidence (BENCH-11, plan §4.12 and §4.7)++++AI consumes OpenVibe.Bench's `bench.rating.changed` (through the signed `/internal/events` webhook, subscribed++at boot with `events.subscription.manage`) and keeps a small `bench_evidence` table: per kind and category, the++model, provider, Elo rating, confidence interval, games and Bench's cost per solved. A sparse event (Bench's++current outbox names only `subjects`) is filled from `GET /api/v1/boards/:kind?type=elo|cost`.++++A route learns from that only when it opts in — `learn_from_bench: true` on the route, default off; `bench_kind`++names the board (when absent AI infers it from the route key or operation, e.g. `chat.reply`), and++`bench_category` defaults to `all`. Among the route's eligible offers, placement then prefers the **cheapest**++offer whose rating interval overlaps the leader's (the highest non-provisional rating): statistically tied with++the best, so price decides. A provisional rating (under 10 games) never decides. With no evidence the order is++exactly the one placement produced. `GET /api/v1/routes/:key` carries a `bench` object with the evidence, the++leader, the tied offers, the selected offer and the reason; the operator console shows the same read-only.++`GET /api/v1/routes?operation=generate` lists each route's eligible offers as `{ provider, model }`.A first-party service holding `ai.run.create` may pin one on `POST /api/v1/generate` (and on`POST /api/v1/runs` for `ai.generate`) with `"route": { "provider": …, "model": … }`. The run tries@@@ -36,7 +36,9 @@"ai.run.failed","ai.run.cached"],-- "eventsConsumed": [],++ "eventsConsumed": [++ "bench.rating.changed"++ ],"namespacesOwned": ["ai.usage_summary"],@@@ -87,5 +89,5 @@"none": "no claims: one process owns ai.db (better-sqlite3, single writer); a run is taken with a compare-and-set on its status (queued → running)"}},-- "notes": "Providers, models, routing profiles, versioned prompt templates and workflows, runs, request logs, citations, scoped cache, quotas and audit. Workflow output is a draft/evidence package attributed to workflow + model + run. ai.run.queued|succeeded|failed|cached have payload contracts (v0.30.0, status planned): AI publishes them once its Events client is wired in; it publishes none today. ai.run.create takes ai.run-request@1 and returns ai.run@1."++ "notes": "Providers, models, routing profiles, versioned prompt templates and workflows, runs, request logs, citations, scoped cache, quotas and audit. Workflow output is a draft/evidence package attributed to workflow + model + run. ai.run.queued|succeeded|failed|cached have payload contracts; AI publishes them through its outbox. AI consumes bench.rating.changed through a signed Events subscription and keeps per-kind Bench evidence; only routes with learn_from_bench: true consult it in placement. ai.run.create takes ai.run-request@1 and returns ai.run@1."}nnew file mode 100644@@@ -0,0 +1,33 @@++-- phase: expand++-- BENCH-11 (plan §4.12 and §4.7): a route can learn from OpenVibe.Bench's measured outcomes.++-- Bench publishes bench.rating.changed; AI keeps the latest rating and cost-per-solved it has seen per++-- kind/category/model, and only a route that opts in (routes.learn_from_bench) consults it during placement.++-- The inbox is openvibe-sdk/events.inboxSchema('bench_event_inbox'): a redelivered event is a no-op.++CREATE TABLE IF NOT EXISTS bench_evidence (++ kind_id text COLLATE "C" NOT NULL, -- bench.kind@1's id, e.g. chat.reply++ category text COLLATE "C" NOT NULL DEFAULT 'all', -- bench.rating@1's category (the board's task class)++ model text COLLATE "C" NOT NULL, -- bench.rating@1's subject: the model rated++ provider text COLLATE "C" NOT NULL DEFAULT '', -- AI's provider for that model, '' when unknown/ambiguous++ rating double precision NOT NULL, -- Elo++ ci_lower double precision,++ ci_upper double precision,++ games bigint NOT NULL DEFAULT 0,++ provisional bigint NOT NULL DEFAULT 0, -- Bench's flag; games < 10 is provisional too++ cost_per_solved double precision, -- USD from GET /api/v1/boards/:kind?type=cost++ updated_at text COLLATE "C" NOT NULL, -- the event's at (ISO-8601)++ PRIMARY KEY (kind_id, category, model, provider)++);++CREATE INDEX IF NOT EXISTS bench_evidence_kind ON bench_evidence (kind_id, category);++++CREATE TABLE IF NOT EXISTS bench_event_inbox (++ consumer text NOT NULL,++ event_id text NOT NULL,++ processed_at bigint NOT NULL,++ PRIMARY KEY (consumer, event_id)++);++++-- The opt-in is per route version, default off. bench_kind names the board to learn from; when it is++-- null, AI infers it from the route key or its operation (chat.reply for chat/generate/...).++ALTER TABLE routes ADD COLUMN IF NOT EXISTS learn_from_bench bigint NOT NULL DEFAULT 0;++ALTER TABLE routes ADD COLUMN IF NOT EXISTS bench_kind text COLLATE "C";++ALTER TABLE routes ADD COLUMN IF NOT EXISTS bench_category text COLLATE "C";@@@ -133,7 +133,20 @@ function adminRouter({ db, registry, pool, quotas, cache, runs, auth, ops = creaif (!latest) throw new AiError(404, 'ai.not_found', `no route ${req.params.key}`);const versions = (await registry.listRoutes({ history: true })).filter(x => x.key === req.params.key).map(x => ({ version: x.version, status: x.status, created_by: x.created_by, created_at: x.created_at }));const operation = req.query.operation ? String(req.query.operation) : null;-- return { route: { ...latest, offers: await routeOffers(latest, operation, req) }, versions };++ // BENCH-11: the evidence this route used, the leader it compared against and why the offer order++ // changed (or stayed). Null while there is no placement to learn from (an inactive route).++ let bench = null;++ if (latest.status === 'active' && pool.benchView) {++ try {++ const target = latest.alias_of ? await registry.resolveRoute(latest.key) : latest;++ if (target && target.status === 'active') {++ bench = await pool.benchView(target, operationFor(target, operation), { requesterType: req.principal.subject.type, requesterId: req.principal.subject.id });++ }++ } catch (err) {++ log.warn(`[ai] bench evidence for route ${latest.key}: ${err.message}`);++ }++ }++ return { route: { ...latest, offers: await routeOffers(latest, operation, req), bench }, versions };}));for (const [plural, k] of Object.entries(kinds)) {@@@ -20,7 +20,7 @@ const ovServe = require('openvibe-shared/serve');const home = require('./home');const pkg = require('../package.json');--function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, vault = null }) {++function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, vault = null, benchEvents = null }) {const app = express();app.disable('x-powered-by');app.set('trust proxy', 'loopback');@@@ -87,6 +87,16 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys// GET /release.json (ADR-016) and POST /release-metrics (open tabs' update reports into /metrics).release.mount(app, { registry: metrics.registry });++ // Events → AI (BENCH-11): loopback only, like every service's /internal. Events posts the signed v2++ // delivery; nginx answers 404 for /internal, and a request carrying forwarding headers is not loopback.++ app.use('/internal', (req, res, next) => {++ if (req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip']) {++ return http.sendProblem(res, 404, 'ai.not_found', { ctx: req.ov });++ }++ return next();++ });++ if (benchEvents) app.use(benchEvents.router);++// Operator actions shared by the admin API and the operator console (one implementation, one audit row).const ops = createOps({ db, registry, pool, quotas, cache, runs });app.use(runsRouter({ runs, registry, auth, config, log }));nnew file mode 100644@@@ -0,0 +1,122 @@++'use strict';++/**++ * OpenVibe.Events → AI for Bench's ratings. This is AI's consumer side (the ai.run.* outbox in++ * server/events.js is its producer side).++ *++ * At boot it ensures one subscription — topic `bench.rating.changed`, endpoint this service's loopback++ * `/internal/events` — with AI's own Events token (audience openvibe.events, scope++ * events.subscription.manage), so Network must grant client `ai` that scope on openvibe.events. Creation++ * is idempotent (Events answers 409 with the existing subscription) and retried in the background. The++ * endpoint verifies the v2 signature with AI_EVENTS_SECRET (comma-separated during rotation) and hands the++ * envelope to server/bench/evidence.js, whose inbox makes a redelivery a no-op.++ *++ * Off without EVENTS_URL/AI_EVENTS_URL and AI_EVENTS_SECRET; AI_EVENTS_SUBSCRIBE=0 creates no subscription++ * (an operator/script may do it), while the endpoint still verifies deliveries when the secret is set.++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { serviceAuth } = require('openvibe-contracts');++const { parseDelivery } = require('openvibe-sdk/events');++++const TOPIC = 'bench.rating.changed';++++function createBenchEvents({ bench, config, log = console, fetchImpl = globalThis.fetch, clientSecret = process.env.OV_OAUTH_CLIENT_SECRET }) {++ const router = express.Router();++ const secrets = (config.events && config.events.secrets) || [];++++ function refused(res, status, code, detail) {++ return http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });++ }++++ /** Apply one verified envelope (also used by tests and an operator replay). */++ async function apply(event) {++ return await bench.applyEvent(event);++ }++++ router.post('/internal/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {++ if (!secrets.length) return refused(res, 503, 'bench.consumer_disabled', 'AI_EVENTS_SECRET is not set');++ const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body || '');++ let delivery = null;++ for (const secret of secrets) {++ delivery = parseDelivery(raw, req.headers, secret, { requireV2: true });++ if (delivery) break;++ }++ if (!delivery) return refused(res, 401, 'bench.bad_signature', 'the Events delivery signature does not verify');++ const event = delivery.event;++ if (!event || typeof event.event_id !== 'string' || !event.event_id) {++ return refused(res, 400, 'bench.bad_delivery', 'body must be { event: <envelope> }');++ }++ try {++ const out = await apply(event);++ return res.status(200).json({ event_id: event.event_id, ...out });++ } catch (err) {++ log.error(`[bench] event ${event.event_id} (${event.event_type}) failed: ${err.message}`);++ return refused(res, 500, 'bench.event_failed', 'processing failed; it will be retried');++ }++ });++++ /**++ * Ensure the subscription exists, idempotently and in the background. Returns a stop handle, or null++ * while the consumer is off. Nothing is created when AI_EVENTS_SUBSCRIBE=0 or no signing secret is set.++ */++ function startSubscriptions({ port, eventsUrl = config.events && config.events.url, networkUrl = config.networkInternalUrl, clientId = config.console && config.console.clientId } = {}) {++ if (!config.events || config.events.subscribe === false || !eventsUrl || !secrets.length || !clientSecret) return null;++ const base = String(eventsUrl).replace(/\/+$/, '');++ const endpoint = (config.events.endpoint || `http://127.0.0.1:${port}/internal/events`);++ const tokens = serviceAuth.createTokenClient({++ tokenUrl: `${networkUrl}/oauth/token`, clientId: clientId || 'ai', clientSecret,++ audience: 'openvibe.events', scope: 'events.subscription.manage', fetchImpl,++ });++ const call = async (method, path, body) => {++ const res = await fetchImpl(`${base}${path}`, {++ method,++ headers: { Accept: 'application/json', ...(body ? { 'Content-Type': 'application/json' } : {}), ...(await tokens.authHeaders()) },++ body: body ? JSON.stringify(body) : undefined,++ signal: AbortSignal.timeout(15000),++ });++ const json = await res.json().catch(() => ({}));++ return { status: res.status, ok: res.ok, body: json };++ };++ const attempt = async () => {++ const listed = await call('GET', '/api/v1/subscriptions');++ if (!listed.ok) throw new Error(`listing subscriptions: ${listed.status}`);++ const mine = (listed.body.subscriptions || []).filter((s) => s.endpoint === endpoint);++ if (mine.some((s) => s.topic_pattern === TOPIC)) return;++ const made = await call('POST', '/api/v1/subscriptions', { topic_pattern: TOPIC, endpoint, secret: secrets[0] });++ if (!made.ok && made.status !== 409) throw new Error(`subscribing to ${TOPIC}: ${made.status} ${made.body.code || ''}`);++ if (made.ok) log.log(`[bench] Events subscription created: ${made.body.id} (${TOPIC} → ${endpoint})`);++ };++ const delays = [0, 10_000, 60_000, 5 * 60_000, 15 * 60_000];++ let i = 0;++ let timer = null;++ let stopped = false;++ const run = () => {++ timer = null;++ if (stopped) return;++ attempt().catch((err) => {++ if (stopped) return;++ if (++i < delays.length) { timer = setTimeout(run, delays[i]); if (timer.unref) timer.unref(); }++ else log.warn(`[bench] Events subscription not created: ${err.message}`);++ });++ };++ timer = setTimeout(run, delays[0]);++ if (timer.unref) timer.unref();++ return {++ topic: TOPIC,++ endpoint,++ stop() { stopped = true; if (timer) clearTimeout(timer); timer = null; },++ };++ }++++ async function status() {++ return {++ enabled: Boolean(secrets.length),++ topic: TOPIC,++ evidence: await bench.list(null).catch(() => []),++ };++ }++++ return { router, apply, startSubscriptions, status, TOPIC };++}++++module.exports = { createBenchEvents, TOPIC };nnew file mode 100644@@@ -0,0 +1,376 @@++'use strict';++/**++ * BENCH-11 (plan §4.12 and §4.7): OpenVibe.Bench's measured outcomes as routing evidence.++ *++ * Bench publishes bench.rating.changed when a board rating moves. AI keeps what it needs from each event++ * in bench_evidence — kind, category, model, provider, Elo, the confidence interval, games and the board's++ * cost per solved — and a route that opts in (learn_from_bench: true, default off) may prefer the cheapest++ * offer whose interval overlaps the leader's. A provisional rating (under 10 games) never decides a route.++ * With no evidence, select() returns the order it was given: placement is unchanged.++ *++ * The event and the board disagree about richness: bench.rating.changed@1 carries one full rating, while++ * Bench's current outbox announces `{ subjects: [...] }`. Both are accepted; when the event is sparse the++ * board endpoints (GET /api/v1/boards/:kind?type=elo|cost) fill the gap. A redelivery is a no-op through++ * the openvibe-sdk inbox (bench_event_inbox).++ */++const { createPgInbox } = require('openvibe-sdk/events');++++const CONSUMER = 'ai-bench.rating';++const MIN_GAMES = 10;++const DEFAULT_CATEGORY = 'all';++const OPERATION_KIND = {++ chat: 'chat.reply',++ complete: 'chat.reply',++ generate: 'chat.reply',++ summarize: 'chat.reply',++ classify: 'chat.reply',++ extract: 'chat.reply',++ enrich: 'chat.reply',++ vision: 'vision.pair',++ transcribe: 'speech.asr',++ image: 'vision.answer',++};++++const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));++const flag = (v) => v === true || v === 1 || /^(1|true|yes|on)$/i.test(String(v || ''));++const text = (v) => (v == null ? '' : String(v).trim());++const uniq = (xs) => [...new Set(xs.filter(Boolean))];++++/** The board category (Bench's task class) a route reads; `all` is Bench's default. */++function categoryFor(route) {++ const c = (route && route.constraints) || {};++ const o = (route && route.options) || {};++ return text((route && route.bench_category) || c.bench_category || o.bench_category) || DEFAULT_CATEGORY;++}++++/** The boards a route may learn from, most specific first. */++function kindCandidates(route, operation) {++ const c = (route && route.constraints) || {};++ const o = (route && route.options) || {};++ const explicit = [++ route && route.bench_kind,++ c.bench_kind, c.task_class, c.kind_id, c.kind,++ o.bench_kind, o.task_class, o.kind_id, o.kind,++ ].map(text);++ // An explicit board is authoritative: no evidence there means no evidence, never a fallback board.++ const first = explicit.find(Boolean);++ if (first) return [first];++ const out = [];++ if (route && route.key) out.push(text(route.key));++ const op = text((route && route.capability) || operation);++ if (OPERATION_KIND[op]) out.push(OPERATION_KIND[op]);++ return uniq(out);++}++++function intervalOf(row) {++ const lower = num(row && row.ci_lower);++ const upper = num(row && row.ci_upper);++ if (lower == null || upper == null) return null;++ return { lower: Math.min(lower, upper), upper: Math.max(lower, upper) };++}++++function provisionalOf(row) {++ return Boolean(row && row.provisional) || Number(row && row.games) < MIN_GAMES;++}++++function usable(row) {++ return Boolean(row) && num(row.rating) != null && !provisionalOf(row) && intervalOf(row) != null;++}++++function overlaps(a, b) {++ const x = intervalOf(a);++ const y = intervalOf(b);++ return Boolean(x && y && x.lower <= y.upper && y.lower <= x.upper);++}++++/** An evidence row applies to an offer when the model matches; a known provider must match too. */++function matchesOffer(row, offer) {++ if (!row || !offer || !text(offer.model) || text(row.model) !== text(offer.model)) return false;++ return !text(row.provider) || !text(offer.provider) || text(row.provider) === text(offer.provider);++}++++function viewRow(row) {++ const rating = num(row.rating);++ const interval = intervalOf(row);++ return {++ kind_id: row.kind_id,++ category: row.category,++ model: row.model,++ provider: text(row.provider) || null,++ rating,++ // `elo` is the same number under Bench's own name, so a reader of either shape finds it.++ elo: rating,++ interval,++ ci_lower: interval ? interval.lower : null,++ ci_upper: interval ? interval.upper : null,++ games: Number(row.games) || 0,++ provisional: provisionalOf(row),++ cost_per_solved: num(row.cost_per_solved),++ updated_at: row.updated_at || null,++ };++}++++function viewSelected(row, offer) {++ return {++ ...viewRow(row),++ provider: offer.provider || null,++ model: offer.model || null,++ };++}++++function leaderSort(a, b) {++ return (num(b.rating) - num(a.rating))++ || (Number(b.games) - Number(a.games))++ || String(a.model).localeCompare(String(b.model))++ || String(a.provider || '').localeCompare(String(b.provider || ''));++}++++/**++ * Pure tie rule: given every evidence row for one kind/category and placement's eligible order, prefer the++ * cheapest offer statistically tied with the best non-provisional rating. Returns the (possibly reordered)++ * order and everything a caller needs to explain the decision. No database or clock here.++ */++function chooseFromEvidence({ kind_id = null, category = DEFAULT_CATEGORY, evidence = [], order = [] }) {++ const rows = Array.isArray(evidence) ? evidence : [];++ const offers = Array.isArray(order) ? order : [];++ const usableRows = rows.filter(usable).sort(leaderSort);++ const base = {++ enabled: true, kind_id, category, evidence: usableRows.map(viewRow),++ leader: null, tied: [], selected: null, status: 'no_evidence',++ reason: `no bench evidence for ${kind_id || 'this route'}${category && category !== DEFAULT_CATEGORY ? ` (${category})` : ''}`,++ };++ if (!usableRows.length) {++ if (rows.length) {++ base.status = 'provisional_or_no_interval';++ base.reason = `bench evidence for ${kind_id || 'this route'} is provisional (under ${MIN_GAMES} games) or has no rating interval`;++ }++ return { order: offers, view: base };++ }++ const leader = usableRows[0];++ const leaderView = viewRow(leader);++ base.leader = leaderView;++ const tiedRows = usableRows.filter((row) => overlaps(row, leader));++++ // One candidate per eligible offer; a model may appear under several providers, so pick that offer's++ // best rating first and its cheapest cost as the tie-break.++ const tied = [];++ for (let index = 0; index < offers.length; index++) {++ const offer = offers[index];++ const matched = tiedRows.filter((row) => matchesOffer(row, offer))++ .sort((a, b) => leaderSort(a, b) || (num(a.cost_per_solved) ?? Infinity) - (num(b.cost_per_solved) ?? Infinity));++ if (!matched.length) continue;++ tied.push({ ...viewSelected(matched[0], offer), index });++ }++ base.tied = tied.map(({ index, ...rest }) => rest);++ if (!tied.length) {++ base.status = 'no_tied_offer';++ base.reason = `no eligible offer is statistically tied with ${leader.model} (rating ${leaderView.rating})`;++ return { order: offers, view: base };++ }++ const priced = tied.filter((row) => row.cost_per_solved != null && row.cost_per_solved >= 0)++ .sort((a, b) => a.cost_per_solved - b.cost_per_solved || a.index - b.index);++ if (!priced.length) {++ base.status = 'no_cost';++ base.reason = `the offers tied with ${leader.model} have no cost per solved yet`;++ return { order: offers, view: base };++ }++ const winner = priced[0];++ const next = [...offers];++ if (winner.index > 0) {++ const [chosen] = next.splice(winner.index, 1);++ next.unshift(chosen);++ }++ base.status = 'selected';++ base.selected = {++ provider: winner.provider, model: winner.model,++ kind_id: winner.kind_id, category: winner.category,++ rating: winner.rating, interval: winner.interval, games: winner.games,++ cost_per_solved: winner.cost_per_solved, updated_at: winner.updated_at,++ };++ base.reason = `${winner.provider ? `${winner.provider}/` : ''}${winner.model} is the cheapest offer statistically tied with ${leader.model} `++ + `(rating ${leaderView.rating}, ${leaderView.games} games) at $${winner.cost_per_solved}/solved`;++ return { order: next, view: base };++}++++function createBenchEvidence({ db, registry = null, config = {}, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console } = {}) {++ const table = 'bench_evidence';++ const inbox = createPgInbox(db, { table: 'bench_event_inbox', now: () => clock.now() });++ const benchUrl = text(config.bench && config.bench.url).replace(/\/+$/, '');++ const timeoutMs = Math.max(1000, Number(config.bench && config.bench.timeoutMs) || 15000);++++ const selectRows = db.prepare(`SELECT * FROM ${table} WHERE kind_id = ? AND category = ? ORDER BY rating DESC, games DESC, model ASC, provider ASC`);++ async function list(kindId, category = DEFAULT_CATEGORY) {++ if (!kindId) return [];++ return await selectRows.all(text(kindId), text(category) || DEFAULT_CATEGORY);++ }++++ async function fetchBoard(kindId, type) {++ if (!benchUrl) throw new Error('AI_BENCH_URL is not set');++ const url = `${benchUrl}/api/v1/boards/${encodeURIComponent(kindId)}?type=${encodeURIComponent(type)}`;++ const res = await fetchImpl(url, { headers: { Accept: 'application/json' }, signal: AbortSignal.timeout(timeoutMs) });++ if (!res.ok) throw new Error(`bench board ${kindId} ${type}: ${res.status}`);++ const body = await res.json();++ const board = body && body.board ? body.board : body;++ if (!board || !Array.isArray(board.rows)) throw new Error(`bench board ${kindId} ${type}: no rows`);++ return board.rows;++ }++++ /** A model's provider from AI's registry, when exactly one provider offers it; Bench rates the model, not the seller. */++ async function providerMap() {++ const out = new Map();++ if (!registry || typeof registry.listModels !== 'function') return out;++ try {++ for (const m of await registry.listModels()) {++ if (!out.has(m.model_key)) out.set(m.model_key, new Set());++ out.get(m.model_key).add(m.provider_key);++ }++ } catch (err) {++ log.warn(`[bench] provider lookup failed: ${err.message}`);++ }++ return out;++ }++++ function ratingsFromPayload(payload) {++ const source = Array.isArray(payload.ratings) ? payload.ratings++ : Array.isArray(payload.rows) ? payload.rows++ : Array.isArray(payload.subjects) ? payload.subjects++ : (payload.subject || payload.model) ? [payload] : [];++ return source.map((raw) => {++ const r = raw && typeof raw === 'object' ? raw : { subject: raw };++ const subject = r.subject && typeof r.subject === 'object' ? r.subject.id : r.subject;++ return {++ model: text(r.model || subject || r.id),++ provider: text(r.provider),++ rating: num(r.rating != null ? r.rating : r.elo),++ ci_lower: num(r.ci_lower),++ ci_upper: num(r.ci_upper),++ games: num(r.games),++ provisional: r.provisional === undefined ? null : flag(r.provisional),++ cost_per_solved: num(r.cost_per_solved),++ updated_at: r.updated_at || r.at || null,++ };++ }).filter((r) => r.model);++ }++++ /** Fill a sparse event from Bench's boards; a full rating still gets the board's cost when it is reachable. */++ async function hydrate(payload, rows) {++ const kindId = text(payload.kind_id);++ const missingRating = rows.some((r) => r.rating == null || r.games == null || r.ci_lower == null || r.ci_upper == null);++ let eloRows = null;++ if (missingRating) {++ eloRows = await fetchBoard(kindId, 'elo'); // a sparse event is useless without it: let Events retry++ for (const row of rows) {++ const b = eloRows.find((x) => text(x.subject || x.model) === row.model);++ if (!b) continue;++ if (row.rating == null) row.rating = num(b.elo != null ? b.elo : b.rating);++ if (row.games == null) row.games = num(b.games);++ if (row.ci_lower == null) row.ci_lower = num(b.ci_lower);++ if (row.ci_upper == null) row.ci_upper = num(b.ci_upper);++ if (row.provisional == null) row.provisional = b.provisional === undefined ? null : flag(b.provisional);++ }++ }++ if (rows.some((r) => r.cost_per_solved == null)) {++ try {++ const costRows = await fetchBoard(kindId, 'cost');++ for (const row of rows) {++ const b = costRows.find((x) => text(x.subject || x.model) === row.model);++ if (b && row.cost_per_solved == null) row.cost_per_solved = num(b.cost_per_solved);++ }++ } catch (err) {++ // A rating without a price is still evidence; the next rating change fills the cost in.++ if (rows.some((r) => r.rating == null)) throw err;++ log.warn(`[bench] cost per solved unavailable for ${kindId}: ${err.message}`);++ }++ }++ return rows;++ }++++ /**++ * Apply one bench.rating.changed envelope, exactly once. Returns { duplicate, stored }.++ * Only source `bench` and the rating topic are ours; anything else is acknowledged and ignored.++ */++ async function applyEvent(event) {++ if (!event || event.event_type !== 'bench.rating.changed') return { ignored: true, reason: 'not bench.rating.changed' };++ if (event.source && event.source !== 'bench') return { ignored: true, reason: `source ${event.source}` };++ if (!event.event_id) throw new Error('bench.rating.changed has no event_id');++ const payload = event.payload && typeof event.payload === 'object' ? event.payload : {};++ const kindId = text(payload.kind_id);++ if (!kindId) throw new Error('bench.rating.changed has no payload.kind_id');++ const category = text(payload.category) || DEFAULT_CATEGORY;++ const eventAt = text(payload.at || event.timestamp);++ const when = eventAt && !Number.isNaN(new Date(eventAt).getTime()) ? new Date(eventAt).toISOString() : new Date(clock.now()).toISOString();++ // A redelivery is common (Events retries); do not call Bench's boards for an event already applied.++ if (await inbox.seen(CONSUMER, String(event.event_id))) return { duplicate: true, stored: 0 };++ const rows = await hydrate(payload, ratingsFromPayload(payload));++ const providers = await providerMap();++++ const receipt = await inbox.once(CONSUMER, String(event.event_id), async (t) => {++ let stored = 0;++ for (const row of rows) {++ if (row.rating == null) continue; // a board with no rating yet is not evidence++ const known = providers.get(row.model);++ const provider = row.provider || (known && known.size === 1 ? [...known][0] : '');++ const provisional = row.provisional == null ? (Number(row.games) || 0) < MIN_GAMES : row.provisional || (Number(row.games) || 0) < MIN_GAMES;++ await t.exec(++ `INSERT INTO ${table} (kind_id, category, model, provider, rating, ci_lower, ci_upper, games, provisional, cost_per_solved, updated_at)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)++ ON CONFLICT (kind_id, category, model, provider) DO UPDATE SET++ rating = excluded.rating, ci_lower = excluded.ci_lower, ci_upper = excluded.ci_upper,++ games = excluded.games, provisional = excluded.provisional,++ cost_per_solved = COALESCE(excluded.cost_per_solved, ${table}.cost_per_solved),++ updated_at = excluded.updated_at++ WHERE excluded.updated_at >= ${table}.updated_at`,++ [kindId, category, row.model, provider, row.rating, row.ci_lower, row.ci_upper,++ Number(row.games) || 0, provisional ? 1 : 0, row.cost_per_solved, when],++ );++ stored++;++ }++ return { stored };++ });++ return receipt.duplicate ? { duplicate: true, stored: 0 } : { duplicate: false, ...(receipt.result || {}) };++ }++++ /** The order a route should try, with the evidence and reason that changed (or left) it. */++ async function select({ route, order = [], operation = null } = {}) {++ const enabled = Boolean(route && route.learn_from_bench);++ const kinds = kindCandidates(route, operation);++ const category = categoryFor(route);++ if (!enabled) {++ return {++ order,++ view: {++ enabled: false, kind_id: kinds[0] || null, category, evidence: [],++ leader: null, tied: [], selected: null, status: 'off',++ reason: 'route does not opt in (learn_from_bench is false)',++ },++ };++ }++ for (const kindId of kinds) {++ const evidence = await list(kindId, category);++ if (!evidence.length) continue;++ const picked = chooseFromEvidence({ kind_id: kindId, category, evidence, order });++ return { order: picked.order, view: picked.view };++ }++ return {++ order,++ view: {++ enabled: true, kind_id: kinds[0] || null, category, evidence: [],++ leader: null, tied: [], selected: null, status: 'no_evidence',++ reason: kinds[0]++ ? `no bench evidence for ${kinds[0]}${category !== DEFAULT_CATEGORY ? ` (${category})` : ''}`++ : 'no bench kind on this route (set bench_kind or use a route key that names one)',++ },++ };++ }++++ async function view({ route, order = [], operation = null } = {}) {++ return (await select({ route, order, operation })).view;++ }++++ return { applyEvent, select, view, list, kindCandidates, categoryFor, CONSUMER, MIN_GAMES };++}++++module.exports = {++ createBenchEvidence, chooseFromEvidence, kindCandidates, categoryFor,++ intervalOf, usable, overlaps, matchesOffer, viewRow, MIN_GAMES, DEFAULT_CATEGORY, CONSUMER,++};@@@ -61,6 +61,22 @@ function load(env = process.env) {networkPublicKey: env.OV_NETWORK_PUBLIC_KEY ? env.OV_NETWORK_PUBLIC_KEY.replace(/\\n/g, '\n') : null,audience: 'openvibe.ai',++ // ── OpenVibe.Events → AI (server/bench/events.js): AI subscribes to bench.rating.changed at boot.++ // The secret verifies the signed delivery (v2); unset turns POST /internal/events off. EVENTS_URL++ // is the shared producer/consumer address, AI_EVENTS_URL overrides it. AI_EVENTS_SUBSCRIBE=0 (or no++ // secret) leaves subscription creation to the operator/script; the endpoint is this service's loopback.++ events: {++ url: trimUrl(env.AI_EVENTS_URL || env.EVENTS_URL),++ secrets: list(env.AI_EVENTS_SECRET || env.AI_BENCH_EVENTS_SECRET || env.EVENTS_SECRET, []),++ endpoint: String(env.AI_EVENTS_ENDPOINT || '').trim(),++ subscribe: bool(env.AI_EVENTS_SUBSCRIBE, true),++ },++ // Where AI reads board detail a sparse bench.rating.changed leaves out (cost per solved, intervals).++ bench: {++ url: trimUrl(env.AI_BENCH_URL || env.OV_BENCH_INTERNAL_URL || env.OV_BENCH_URL || env.BENCH_URL || 'http://127.0.0.1:5230'),++ timeoutMs: int(env.AI_BENCH_TIMEOUT_MS, 15000),++ },++// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },@@@ -394,6 +394,8 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, eprimary: { provider: p.key, model: model || p.default_model || null }, fallbacks: previous, capability: null, constraints: {}, pinned: [], alias_of: null,options: (effective && effective.options) || {}, max_output_tokens: effective ? effective.max_output_tokens : null,response_format: (effective && effective.response_format) || 'text', timeout_ms: effective ? effective.timeout_ms : null, status: 'active',++ learn_from_bench: Boolean(effective && effective.learn_from_bench),++ bench_kind: (effective && effective.bench_kind) || null, bench_category: (effective && effective.bench_category) || null,}, who(req));res.redirect(303, `/console/routes/${encodeURIComponent(route.key)}?done=route_used`);}));@@@ -420,8 +422,20 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, eif (!versions.length) return await send(res, 404, pages.message({ title: 'Not found', text: `No ${k.one} ${req.params.key}.`, back: `/console/${kind}`, ...common(req) }));const want = Number(req.query.version);const selected = (Number.isInteger(want) && versions.find((x) => x.version === want)) || versions[0];++ const current = await registry[k.get](req.params.key, selected.version);++ // BENCH-11: read-only evidence on a route's page. It changes what runs pick, so it is shown with++ // the leader, the tied offers and the reason; it is not editable here (bench_kind is an API field).++ let bench = null;++ if (kind === 'routes' && current.status === 'active' && pool.benchView) {++ try {++ const target = current.alias_of ? await registry.resolveRoute(current.key) : current;++ if (target && target.status === 'active') bench = await pool.benchView(target, target.capability || 'chat');++ } catch (err) {++ log.warn(`[console] bench evidence for route ${current.key}: ${err.message}`);++ }++ }return await send(res, 200, pages.definition({-- ...common(req), kind, key: req.params.key, versions, selected: await registry[k.get](req.params.key, selected.version),++ ...common(req), kind, key: req.params.key, versions, selected: current, bench,inUse: inUseMap(kind, versions).get(req.params.key), canManage: req.staff.caps.includes(CAPS.workflowManage), notice: notice(req),...(kind === 'routes' ? { canChoose: req.staff.caps.includes(CAPS.providerManage), providers: (await registry.listProviders()).filter((p) => p.status === 'active' && p.kind !== 'whisper' && p.kind !== 'stub'), models: await registry.listModels() } : {}),}));@@@ -264,6 +264,24 @@ const KINDS = {};const target = (c) => (c ? html`<code>${c.provider}</code>${c.model ? html` / <code>${c.model}</code>` : ''}` : '—');++/** BENCH-11: the evidence an opted-in route used, read-only. Never a form: changing policy is an API edit. */++function benchBlock(bench) {++ if (!bench) return '';++ if (!bench.enabled) return html`<h2>Bench evidence</h2><p class="muted">Learn from Bench is off for this route (<code>learn_from_bench: false</code>); placement keeps its own order.</p>`;++ const rows = bench.evidence || [];++ const tied = new Set((bench.tied || []).map((x) => `${x.provider || ''}|${x.model}`));++ const body = rows.map((r) => html`<tr><td><code>${r.model}</code>${bench.selected && bench.selected.model === r.model ? html` <span class="ok">selected</span>` : tied.has(`${r.provider || ''}|${r.model}`) ? html` <span class="muted">tied</span>` : ''}</td>++<td>${r.provider ? html`<code>${r.provider}</code>` : html`<span class="muted">any</span>`}</td><td>${r.rating == null ? '—' : n(r.rating)}</td>++<td>${r.interval ? `${n(r.interval.lower)} – ${n(r.interval.upper)}` : '—'}</td><td>${n(r.games)}${r.provisional ? ' (provisional)' : ''}</td>++<td>${r.cost_per_solved == null ? '—' : `$${r.cost_per_solved}`}</td><td>${r.updated_at ? when(r.updated_at) : '—'}</td></tr>`);++ return html`<h2>Bench evidence</h2>++<p class="muted">Read-only. ${bench.kind_id ? html`Board <code>${bench.kind_id}</code>` : 'No board named'} · category <code>${bench.category}</code>.++Provisional ratings (under 10 games) never decide a route.</p>++<p><strong>Why:</strong> ${bench.reason}</p>++${bench.leader ? html`<p>Leader: <code>${bench.leader.model}</code> at ${n(bench.leader.rating)} (${n(bench.leader.games)} games, interval ${bench.leader.interval ? `${n(bench.leader.interval.lower)} – ${n(bench.leader.interval.upper)}` : '—'}).</p>` : ''}++${rows.length ? table('Evidence used', ['Model', 'Provider', 'Rating', 'Interval', 'Games', 'Cost / solved', 'Updated'], body, 'no evidence.') : html`<p class="muted">No evidence yet: routing keeps its current order.</p>`}`;++}++function definitions({ staff, csrf, kind, rows, inUse }) {const k = KINDS[kind];const link = (key) => html`<a href="/console/${kind}/${enc(key)}"><code>${key}</code></a>`;@@@ -294,7 +312,7 @@ ${table(`${k.title} (newest version of each)`, head, body, `no ${kind}.`)}`,});}--function definition({ staff, csrf, kind, key, versions, selected, inUse, canManage, notice, error, canChoose = false, providers = [], models = [] }) {++function definition({ staff, csrf, kind, key, versions, selected, inUse, canManage, notice, error, canChoose = false, providers = [], models = [], bench = null }) {const k = KINDS[kind];const v = selected;let content;@@@ -302,9 +320,11 @@ function definition({ staff, csrf, kind, key, versions, selected, inUse, canManacontent = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd>${v.alias_of ? html`<dt>Alias of</dt><dd><a href="/console/routes/${enc(v.alias_of)}"><code>${v.alias_of}</code></a></dd>` : html`<dt>Primary</dt><dd>${target(v.primary)}</dd><dt>Fallbacks</dt><dd>${v.fallbacks.length ? html`<ol>${v.fallbacks.map((f) => html`<li>${target(f)}</li>`)}</ol>` : html`<span class="warn">none: an outage of the primary fails every run on this route</span>`}</dd>`}++<dt>Learn from Bench</dt><dd>${v.learn_from_bench ? html`on${v.bench_kind ? html` · board <code>${v.bench_kind}</code>` : ''}${v.bench_category ? html` · category <code>${v.bench_category}</code>` : ''}` : 'off'}</dd><dt>Response format</dt><dd>${v.response_format}</dd><dt>Max output tokens</dt><dd>${v.max_output_tokens != null ? n(v.max_output_tokens) : '—'}</dd><dt>Timeout</dt><dd>${v.timeout_ms ? `${n(v.timeout_ms)} ms` : 'provider default'}</dd><dt>Created</dt><dd>${v.created_by || '—'}, ${when(v.created_at)}</dd></dl>--<h3>Options</h3><pre>${JSON.stringify(v.options || {}, null, 2)}</pre>`;++<h3>Options</h3><pre>${JSON.stringify(v.options || {}, null, 2)}</pre>++${benchBlock(bench)}`;} else if (kind === 'templates') {content = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd><dt>Name</dt><dd>${v.name}</dd><dt>Description</dt><dd>${v.description || '—'}</dd><dt>Default route</dt><dd>${v.default_route ? html`<a href="/console/routes/${enc(v.default_route)}"><code>${v.default_route}</code></a>` : '—'}</dd>@@@ -11,6 +11,8 @@ const { load } = require('./config');const { openDb } = require('./db');const { createRegistry } = require('./registry');const { createProviderPool } = require('./providers');++const { createBenchEvidence } = require('./bench/evidence');++const { createBenchEvents } = require('./bench/events');const { createFetcher } = require('./fetcher');const { createEngine } = require('./workflows/engine');const { createCache } = require('./cache');@@@ -35,7 +37,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now// Provider keys an owner stored from the console (server/vault.js): into this process before any provider is built.const vault = require('./vault').createVault({ db, config, clock, log });await vault.load().catch((err) => log.warn(`[Vault] ${err.message}`));-- const pool = createProviderPool({ db, registry, config, clock, fetchImpl, env, log, credentialFetch, machineModels });++ // BENCH-11: Bench's ratings as routing evidence, and the opted-in tie rule inside placement.++ const bench = createBenchEvidence({ db, registry, config, clock, fetchImpl, log });++ const pool = createProviderPool({ db, registry, config, clock, fetchImpl, env, log, credentialFetch, machineModels, bench });// T6 step 8: the SaaS tier budgets on openvibe-sdk/govern in Valkey (server/govern.js); off unless AI_GOVERN_TIERS=1.const tiers = createTiers({ config, valkey, clock, log });// §2.1.8: the provider free allowance per subject, priced off the pool's own rate cards (server/free-allowance.js);@@@ -65,7 +69,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now// NODE_ENV=test (tests pin a PEM or point the URL at a stub and let the client fetch on first use).const keys = createNetworkKeys({ config });const auth = createAuth({ config, log });-- const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, vault });++ // The Events consumer side: a signed bench.rating.changed webhook, subscribed at boot (server/bench/events.js).++ const benchEvents = createBenchEvents({ bench, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });++ const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, vault, benchEvents });let jwks = null;let keyLoaded = Promise.resolve(null);if (!keys.pinned && config.nodeEnv !== 'test') {@@@ -85,6 +91,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowhousekeeping.unref?.();let server = null;++ let benchSubscription = null;if (listen) userModules.start();if (listen) {server = await new Promise((resolve, reject) => {@@@ -95,12 +102,14 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowserver.headersTimeout = 66000;server.requestTimeout = config.runs.maxWaitMs + 30000;log.log(`[ai] listening on http://${config.host}:${server.address().port} (${config.nodeEnv})`);++ benchSubscription = benchEvents.startSubscriptions({ port: server.address().port });const shared = await registry.getProvider('shared');log.log(`[ai] shared provider: ${shared.kind} ${shared.status}, credentials ${registry.publicProvider(shared).credentials}; stub fallback ${config.stubFallback ? 'on' : 'off'}`);}async function close() {clearInterval(housekeeping);++ if (benchSubscription) benchSubscription.stop();userModules.stop();if (jwks) jwks.stop();if (server) {@@@ -116,7 +125,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowif (!givenDb) await db.close();}-- return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, close };++ return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, bench, benchEvents, close };}if (require.main === module) {@@@ -43,7 +43,7 @@ function rateLimited(err) {return (err && err.status === 429) || (err && err.code === 'provider.rate_limited');}--function createProviderPool({ db, registry, config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, credentialFetch = null, machineModels = null }) {++function createProviderPool({ db, registry, config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, credentialFetch = null, machineModels = null, bench = null }) {// A person's own machine as their model (credential provider node, server/machines.js).const machines = machineModels || require('../machines').createMachineModels({ config, env, fetchImpl });const cache = new Map(); // key -> { stamp, adapter }@@@ -339,9 +339,20 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no}async function candidates(route, features, operation, ctx = {}) {-- const { order, explain, placement: result = null, ...rest } = route.capability && operation++ let { order, explain, placement: result = null, ...rest } = route.capability && operation? await poolCandidates(route, features, operation, ctx): await pinnedCandidates(route, features, ctx);++ // BENCH-11: an opted-in route may put the cheapest offer statistically tied with Bench's leader first.++ // A route without the opt-in (or without evidence) keeps the order placement just produced.++ if (bench) {++ const picked = await bench.select({ route, order, operation });++ order = picked.order;++ explain.bench = picked.view;++ if (picked.view.status === 'selected' && picked.view.selected) {++ explain.selected = offerId(picked.view.selected.provider, picked.view.selected.model);++ explain.reasons = [picked.view.reason, ...(explain.reasons || [])];++ }++ }// A scoped route never ends on synthetic text (media.paid falls back to its extractive overview); an image// has no such fallback, so the dev/test stub PNG (AI_STUB_FALLBACK, off in production) still closes it.if (config.stubFallback && (!(route.constraints && route.constraints.provider_scope) || operation === 'image')@@@ -455,10 +466,12 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no* a pinned run is accepted only when it names one of them.*/async function offers(route, operation, ctx = {}) {-- const { explain } = await candidates(route, [operation], operation, ctx);++ const { order, explain } = await candidates(route, [operation], operation, ctx);++ const rank = new Map((order || []).map((c, i) => [offerId(c.provider, c.model || null), i]));const out = [];const seen = new Set();-- for (const c of explain.candidates) {++ // The order placement would try, including a Bench-tie winner when an opted-in route has evidence.++ for (const c of [...explain.candidates].sort((a, b) => (rank.get(offerId(a.provider, a.model || null)) ?? Infinity) - (rank.get(offerId(b.provider, b.model || null)) ?? Infinity))) {if (!c || !c.eligible || !c.provider) continue;const id = offerId(c.provider, c.model || null);if (seen.has(id)) continue;@@@ -468,6 +481,13 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.noreturn out;}++ /** The Bench evidence a route used and why, for GET /api/v1/routes/:key and the operator console. */++ async function benchView(route, operation, ctx = {}) {++ if (!bench) return null;++ const { explain } = await candidates(route, [operation], operation, ctx);++ return explain.bench || null;++ }++async function execute(route, operation, req, ctx = {}) {if (ctx.credential) return await executeWithCredential(ctx.credential, operation, req, ctx);const features = [operation];@@@ -579,7 +599,7 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.nothrow new AiError(503, 'provider.unavailable', 'no provider on this route could answer', { tried, explain });}-- return { adapter, health, resetHealth, execute, executeWithCredential, offers, priceFor, costOf, rateCardsFor, providerStates, stats, statFor, candidates };++ return { adapter, health, resetHealth, execute, executeWithCredential, offers, benchView, priceFor, costOf, rateCardsFor, providerStates, stats, statFor, candidates };}module.exports = { createProviderPool };@@@ -18,6 +18,7 @@ const BILLING_PROFILES = ['metered', 'subscription', 'payg', 'free', 'byok'];const FEATURES = ['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'enrich', 'embed', 'vision', 'json', 'tools', 'transcribe', 'image'];const LIFECYCLE = ['draft', 'active', 'deprecated', 'archived'];const KEY_RE = /^[a-z][a-z0-9_-]*(\.[a-z0-9_-]+)*$/;++const flag = (v) => v === true || v === 1 || /^(1|true|yes|on)$/i.test(String(v == null ? '' : v));function createRegistry(db, { clock = { now: () => Date.now() }, env = process.env } = {}) {const now = () => iso(clock.now());@@@ -189,10 +190,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.efallbacks: parseJson(r.fallbacks, []), options: parseJson(r.options, {}), max_output_tokens: r.max_output_tokens,response_format: r.response_format, timeout_ms: r.timeout_ms, alias_of: r.alias_of,capability: r.capability || null, constraints: parseJson(r.constraints, {}), pinned: parseJson(r.pinned, []),++ // BENCH-11: a route's opt-in to Bench's measured outcomes, and the board it reads (null = infer).++ learn_from_bench: Boolean(r.learn_from_bench), bench_kind: r.bench_kind || null, bench_category: r.bench_category || null,created_by: r.created_by, created_at: r.created_at,};}-- const routeContent = (r) => sha256({ p: r.primary, f: r.fallbacks, o: r.options, m: r.max_output_tokens, rf: r.response_format, t: r.timeout_ms, a: r.alias_of, s: r.status, c: r.capability || null, x: r.constraints || {}, pi: r.pinned || [] });++ const routeContent = (r) => sha256({ p: r.primary, f: r.fallbacks, o: r.options, m: r.max_output_tokens, rf: r.response_format, t: r.timeout_ms, a: r.alias_of, s: r.status, c: r.capability || null, x: r.constraints || {}, pi: r.pinned || [], b: Boolean(r.learn_from_bench), bk: r.bench_kind || null, bc: r.bench_category || null });async function createRouteVersion(key, input, { actor = 'system', trace = null } = {}) {if (!KEY_RE.test(key || '')) throw new AiError(422, 'ai.invalid', 'route key must be lowercase dotted');const prev = decodeRoute(await latestAny('routes', key));@@@ -209,9 +212,15 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.eresponse_format: input.response_format || (prev && prev.response_format) || 'text',timeout_ms: input.timeout_ms !== undefined ? input.timeout_ms : (prev ? prev.timeout_ms : null),alias_of: input.alias_of !== undefined ? input.alias_of : (prev ? prev.alias_of : null),++ // BENCH-11: off unless the route opts in; an edit that omits these keeps the previous version's choice.++ learn_from_bench: input.learn_from_bench !== undefined ? flag(input.learn_from_bench) : Boolean(prev && prev.learn_from_bench),++ bench_kind: input.bench_kind !== undefined ? (String(input.bench_kind || '').trim() || null) : (prev ? prev.bench_kind : null),++ bench_category: input.bench_category !== undefined ? (String(input.bench_category || '').trim() || null) : (prev ? prev.bench_category : null),status: input.status || 'active',};if (r.capability && !FEATURES.includes(r.capability)) throw new AiError(422, 'ai.invalid', `route capability must be one of ${FEATURES.join(', ')}`);++ if (r.bench_kind && !KEY_RE.test(r.bench_kind)) throw new AiError(422, 'ai.invalid', 'bench_kind must be a lowercase dotted Bench kind id (e.g. chat.reply)');++ if (r.bench_category && r.bench_category.length > 100) throw new AiError(422, 'ai.invalid', 'bench_category must be at most 100 characters');if (!r.alias_of && !r.capability) {if (!r.primary || !r.primary.provider) throw new AiError(422, 'ai.invalid', 'route needs primary.provider or a capability');for (const c of [r.primary, ...(r.fallbacks || [])]) {@@@ -230,11 +239,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.econst version = await nextVersion('routes', key);const primaryProvider = r.alias_of ? ((r.primary && r.primary.provider) || 'stub'): (r.primary && r.primary.provider) || ((r.pinned && r.pinned[0] && r.pinned[0].provider) || 'stub');-- await db.prepare(`INSERT INTO routes (key, version, status, primary_provider, primary_model, fallbacks, options, max_output_tokens, response_format, timeout_ms, alias_of, capability, constraints, pinned, created_by, created_at)-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)++ await db.prepare(`INSERT INTO routes (key, version, status, primary_provider, primary_model, fallbacks, options, max_output_tokens, response_format, timeout_ms, alias_of, capability, constraints, pinned, learn_from_bench, bench_kind, bench_category, created_by, created_at)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(key, version, r.status, primaryProvider, (r.primary && r.primary.model) || null,JSON.stringify(r.fallbacks || []), JSON.stringify(r.options || {}), r.max_output_tokens, r.response_format, r.timeout_ms, r.alias_of,-- r.capability || null, JSON.stringify(r.constraints || {}), JSON.stringify(r.pinned || []), actor, now());++ r.capability || null, JSON.stringify(r.constraints || {}), JSON.stringify(r.pinned || []),++ r.learn_from_bench ? 1 : 0, r.bench_kind, r.bench_category, actor, now());await audit(actor, 'route.version', 'route', key, { trace, metadata: { version, previous: prev ? prev.version : null, status: r.status, alias_of: r.alias_of, capability: r.capability || null } });return await getRoute(key, version);}nnew file mode 100644@@@ -0,0 +1,184 @@++'use strict';++// BENCH-11 (plan §4.12 and §4.7): Bench's bench.rating.changed reaches AI through its Events consumer,++// becomes per-kind evidence (rating, interval, games, cost per solved), and only an opted-in route uses it:++// the cheapest offer statistically tied with the best non-provisional rating. Provisional evidence never++// decides; no evidence leaves placement's order exactly as it was.++const assert = require('assert');++const { signDeliveryHeaders } = require('openvibe-sdk/events');++const { boot, request, token, suite, seamServer, ALL } = require('./helpers');++++const t = suite('bench-learning');++const admin = token('live', ALL);++const SECRET = 'bench-events-secret-0123456789abcdef';++let h; let boards;++++const boardRows = [++ { subject: 'leader-model', elo: 1200, games: 40, ci_lower: 1180, ci_upper: 1220, provisional: false, median_cost_usd: 0.5 },++ { subject: 'cheap-model', elo: 1180, games: 40, ci_lower: 1160, ci_upper: 1200, provisional: false, median_cost_usd: 0.1 },++];++const costRows = boardRows.map((r) => ({ subject: r.subject, cost_per_solved: r.subject === 'leader-model' ? 0.5 : 0.1, median_cost_usd: r.median_cost_usd }));++++async function provider(key, model, price) {++ const r = await h.registry.upsertProvider({++ key, display_name: key, kind: 'http', status: 'active', base_url: 'http://127.0.0.1:9',++ auth_mode: 'none', default_model: model, capabilities: ['chat'],++ }, { actor: 'test' });++ assert.strictEqual(r.key, key);++ await h.registry.upsertModel({++ provider_key: key, model_key: model, type: 'chat',++ cost: { in_per_mtok: price, out_per_mtok: price },++ }, { actor: 'test' });++}++++async function rating({ eventId, model, provider, rating: elo, lower, upper, games, cost, provisional = false, kind = 'chat.reply' }) {++ return await h.bench.applyEvent({++ event_id: eventId, event_type: 'bench.rating.changed', version: 1, source: 'bench',++ timestamp: '2026-10-20T10:00:00.000Z',++ payload: {++ kind_id: kind, category: 'all', subject: model, elo, games, win_rate: 0.6,++ ci_lower: lower, ci_upper: upper, provisional, median_cost_usd: null,++ cost_per_solved: cost, at: '2026-10-20T10:00:00.000Z',++ ...(provider ? { provider } : {}),++ },++ });++}++++async function deliver(event) {++ const raw = JSON.stringify({ event });++ const headers = signDeliveryHeaders(raw, SECRET);++ const res = await fetch(`${h.base}/internal/events`, {++ method: 'POST', headers: { 'Content-Type': 'application/json', ...headers }, body: raw,++ });++ return { status: res.status, body: await res.json().catch(() => ({})) };++}++++t.test('boot with a fake Bench board endpoint and the Events consumer on', async () => {++ boards = await seamServer((_body, req) => {++ if (req.url.includes('/api/v1/boards/chat.reply') && req.url.includes('type=elo')) return { board: { kind_id: 'chat.reply', type: 'elo', rows: boardRows } };++ if (req.url.includes('/api/v1/boards/chat.reply') && req.url.includes('type=cost')) return { board: { kind_id: 'chat.reply', type: 'cost', rows: costRows } };++ return { status: 404, body: { code: 'bench.kind_not_found' } };++ });++ h = await boot({ env: {++ AI_EVENTS_SECRET: SECRET, AI_EVENTS_SUBSCRIBE: '0', AI_BENCH_URL: boards.url, AI_BENCH_TIMEOUT_MS: '2000',++ } });++ await provider('bench-leader', 'leader-model', 50);++ await provider('bench-cheap', 'cheap-model', 10);++ await provider('bench-far', 'far-model', 0.01);++ await provider('bench-prov', 'prov-model', 0.001);++});++++t.test('a signed bench.rating.changed stores evidence, hydrates a sparse event from the board, and is idempotent', async () => {++ const event = {++ event_id: 'evt_bench_full_0001', event_type: 'bench.rating.changed', version: 1, source: 'bench',++ timestamp: '2026-10-20T10:00:00.000Z',++ payload: {++ kind_id: 'chat.reply', category: 'all', subject: 'leader-model', elo: 1200, games: 40, win_rate: 0.6,++ ci_lower: 1180, ci_upper: 1220, median_cost_usd: null, provisional: false, at: '2026-10-20T10:00:00.000Z',++ },++ };++ const first = await deliver(event);++ assert.strictEqual(first.status, 200, JSON.stringify(first.body));++ assert.strictEqual(first.body.stored, 1);++ let row = await h.db.prepare("SELECT * FROM bench_evidence WHERE kind_id = 'chat.reply' AND model = 'leader-model'").get();++ assert.ok(row, 'the rating is stored');++ assert.strictEqual(row.provider, 'bench-leader', 'the provider comes from AI registry when Bench rates only the model');++ assert.strictEqual(Number(row.rating), 1200);++ assert.strictEqual(Number(row.ci_lower), 1180);++ assert.strictEqual(Number(row.ci_upper), 1220);++ assert.strictEqual(Number(row.games), 40);++ assert.strictEqual(Number(row.cost_per_solved), 0.5, 'the board cost endpoint fills cost per solved');++++ // Bench's current outbox announces only subjects; the consumer fills rating and cost from the boards.++ const sparse = await deliver({++ event_id: 'evt_bench_sparse_0002', event_type: 'bench.rating.changed', version: 1, source: 'bench',++ timestamp: '2026-10-20T10:05:00.000Z',++ payload: { kind_id: 'chat.reply', category: 'all', subjects: ['cheap-model'] },++ });++ assert.strictEqual(sparse.status, 200, JSON.stringify(sparse.body));++ row = await h.db.prepare("SELECT * FROM bench_evidence WHERE kind_id = 'chat.reply' AND model = 'cheap-model'").get();++ assert.strictEqual(Number(row.rating), 1180);++ assert.strictEqual(Number(row.cost_per_solved), 0.1);++++ const again = await deliver({++ event_id: 'evt_bench_sparse_0002', event_type: 'bench.rating.changed', version: 1, source: 'bench',++ timestamp: '2026-10-20T10:05:00.000Z',++ payload: { kind_id: 'chat.reply', category: 'all', subjects: ['cheap-model'] },++ });++ assert.strictEqual(again.status, 200);++ assert.strictEqual(again.body.duplicate, true, 'the inbox makes a redelivery a no-op');++ assert.strictEqual((await h.db.prepare("SELECT COUNT(*) AS n FROM bench_evidence WHERE model = 'cheap-model'").get()).n, 1);++});++++t.test('the tie rule picks the cheapest offer whose interval overlaps the leader; provisional evidence never wins', async () => {++ // A provisional model is the cheapest and highest-rated, but under 10 games: it is not the leader and++ // is not a candidate. The leader is non-provisional; cheap-model overlaps it; far-model does not.++ await rating({ eventId: 'evt_bench_leader_0003', model: 'leader-model', rating: 1200, lower: 1180, upper: 1220, games: 40, cost: 0.5 });++ await rating({ eventId: 'evt_bench_cheap_0004', model: 'cheap-model', rating: 1180, lower: 1160, upper: 1200, games: 40, cost: 0.1 });++ await rating({ eventId: 'evt_bench_far_0005', model: 'far-model', rating: 900, lower: 850, upper: 950, games: 40, cost: 0.01 });++ await rating({ eventId: 'evt_bench_prov_0006', model: 'prov-model', rating: 2000, lower: 1900, upper: 2100, games: 9, cost: 0.001, provisional: true });++++ const route = {++ key: 'bench.learning', capability: 'chat', constraints: { objective: 'cheapest' },++ learn_from_bench: true, bench_kind: 'chat.reply', pinned: [], fallbacks: [],++ };++ const out = await h.pool.candidates(route, ['chat'], 'chat');++ assert.strictEqual(out.order[0].provider, 'bench-cheap', `expected the tied cheap offer first, got ${JSON.stringify(out.order)}`);++ assert.strictEqual(out.order[0].model, 'cheap-model');++ assert.strictEqual(out.explain.bench.status, 'selected');++ assert.strictEqual(out.explain.bench.leader.model, 'leader-model');++ assert.strictEqual(out.explain.bench.selected.model, 'cheap-model');++ assert.match(out.explain.bench.reason, /cheapest offer statistically tied/i);++ assert.ok(!out.explain.bench.evidence.some((x) => x.model === 'prov-model'), 'provisional evidence is not used');++ assert.ok(!out.explain.bench.tied.some((x) => x.model === 'prov-model'), 'a provisional model is not tied');++ assert.ok(!out.explain.bench.tied.some((x) => x.model === 'far-model'), 'an interval that does not overlap is not tied');++});++++t.test('only a route with learn_from_bench opts in; no evidence keeps placement order', async () => {++ const base = { key: 'bench.learning.off', capability: 'chat', constraints: { objective: 'cheapest' }, bench_kind: 'chat.reply', pinned: [], fallbacks: [] };++ const off = await h.pool.candidates({ ...base, learn_from_bench: false }, ['chat'], 'chat');++ assert.strictEqual(off.explain.bench.enabled, false);++ assert.strictEqual(off.explain.bench.status, 'off');++ assert.strictEqual(off.order[0].provider, 'bench-prov', 'placement stays cheapest: the evidence did not change the order');++ assert.notStrictEqual(off.order[0].provider, 'bench-cheap');++++ const noEvidence = await h.pool.candidates({ ...base, key: 'bench.learning.none', learn_from_bench: true, bench_kind: 'codes.patch' }, ['chat'], 'chat');++ assert.strictEqual(noEvidence.explain.bench.status, 'no_evidence');++ assert.deepStrictEqual(noEvidence.order.map((x) => `${x.provider}:${x.model}`), off.order.map((x) => `${x.provider}:${x.model}`), 'no evidence leaves the order unchanged');++++ const provisionalOnly = await h.pool.candidates({ ...base, key: 'bench.learning.prov', learn_from_bench: true, bench_kind: 'vision.answer' }, ['chat'], 'chat');++ await rating({ eventId: 'evt_bench_onlyprov_0007', kind: 'vision.answer', model: 'prov-model', rating: 2000, lower: 1900, upper: 2100, games: 9, cost: 0.001, provisional: true });++ const afterProvisional = await h.pool.candidates({ ...base, key: 'bench.learning.prov', learn_from_bench: true, bench_kind: 'vision.answer' }, ['chat'], 'chat');++ assert.strictEqual(afterProvisional.explain.bench.status, 'provisional_or_no_interval');++ assert.deepStrictEqual(afterProvisional.order.map((x) => `${x.provider}:${x.model}`), provisionalOnly.order.map((x) => `${x.provider}:${x.model}`), 'a provisional-only board changes nothing');++});++++t.test('GET /api/v1/routes/:key shows the evidence and why, and lists the winning offer first', async () => {++ const created = await request(h.base, 'POST', '/api/v1/routes/bench.api/versions', { tok: admin, body: {++ capability: 'chat', constraints: { objective: 'cheapest' }, learn_from_bench: true, bench_kind: 'chat.reply',++ } });++ assert.strictEqual(created.status, 201, created.text);++ const r = await request(h.base, 'GET', '/api/v1/routes/bench.api', { tok: admin });++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.body.route.learn_from_bench, true);++ assert.strictEqual(r.body.route.bench_kind, 'chat.reply');++ assert.strictEqual(r.body.route.bench.status, 'selected');++ assert.strictEqual(r.body.route.bench.selected.model, 'cheap-model');++ assert.ok(r.body.route.bench.evidence.length >= 2, 'the evidence it used is visible');++ assert.strictEqual(r.body.route.offers[0].provider, 'bench-cheap', 'the exposed offer order is the one runs try');++ assert.match(r.body.route.bench.reason, /statistically tied/i);++++ const plain = await request(h.base, 'POST', '/api/v1/routes/bench.plain/versions', { tok: admin, body: {++ capability: 'chat', constraints: { objective: 'cheapest' },++ } });++ assert.strictEqual(plain.status, 201, plain.text);++ const p = await request(h.base, 'GET', '/api/v1/routes/bench.plain', { tok: admin });++ assert.strictEqual(p.body.route.learn_from_bench, false, 'off is the default');++ assert.strictEqual(p.body.route.bench.enabled, false);++});++++t.test('shutdown', async () => {++ await h.stop();++ await boards.close();++});++++t.run();
@@ -48,6 +48,18 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
@@ -48,6 +48,18 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
#AI_PERSON_SESSION_TTL_MIN=1440
#AI_PERSON_SESSION_TTL_MIN=1440
# Audience a Network user token must carry
# Audience a Network user token must carry
#AI_SSO_AUDIENCE=openvibe.network
#AI_SSO_AUDIENCE=openvibe.network
+# ── Events (the ai.run.* outbox relay and the Bench ratings subscription) ──
+# EVENTS_URL is where AI publishes its own events and ensures its subscription. AI_EVENTS_SECRET is the
+# v2 signing secret Events hands back on that subscription (32+ characters; comma-separated during rotation);
+# without it POST /internal/events is off. AI_EVENTS_SUBSCRIBE=0 skips creating the subscription (an
+# operator/script may create topic bench.rating.changed → http://127.0.0.1:4700/internal/events).
+#EVENTS_URL=http://127.0.0.1:4300
+#AI_EVENTS_SECRET=
+#AI_EVENTS_SUBSCRIBE=on
+# Where a sparse bench.rating.changed is filled from (boards, cost per solved).
+#AI_BENCH_URL=http://127.0.0.1:5230
+#AI_BENCH_TIMEOUT_MS=15000
+
# ── Billing (plan T5 lane F) ──
# ── Billing (plan T5 lane F) ──
# One platform.usage-sample@1 reading per accounted run, posted to billing.usage.record with AI's
# One platform.usage-sample@1 reading per accounted run, posted to billing.usage.record with AI's
# service token, so Network must grant client `ai` billing.usage.record on openvibe.billing. Readings
# service token, so Network must grant client `ai` billing.usage.record on openvibe.billing. Readings
@@ -43,7 +43,8 @@ owns publication truth: Wiki, Blog, News, Live, … decide what to publish.
@@ -43,7 +43,8 @@ owns publication truth: Wiki, Blog, News, Live, … decide what to publish.
- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`
- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`
- OpenVibe.Network (JWKS, service tokens, SSO for the operator console, user modules)
- OpenVibe.Network (JWKS, service tokens, SSO for the operator console, user modules)
-- OpenVibe.Events (the `ai.run.*` outbox relay; off without `EVENTS_URL`)
+- OpenVibe.Events (the `ai.run.*` outbox relay, and the signed `bench.rating.changed` subscription; off without `EVENTS_URL`)
+- OpenVibe.Bench (`bench.rating.changed` and `GET /api/v1/boards/:kind?type=elo|cost`, for opted-in routes)
- OpenVibe.Media (recordings `media.analyze` reads, over allow-listed https URLs or signed URLs)
- OpenVibe.Media (recordings `media.analyze` reads, over allow-listed https URLs or signed URLs)
- provider APIs configured by the operator (OpenAI-compatible, OpenAI Responses, Anthropic, the HTTP seam), whisper.cpp
- provider APIs configured by the operator (OpenAI-compatible, OpenAI Responses, Anthropic, the HTTP seam), whisper.cpp
and the local model server `openvibe-llm.service` on this host
and the local model server `openvibe-llm.service` on this host
@@ -59,8 +60,9 @@ Implemented here (the service manifest's `capabilities`, audience `openvibe.ai`;
@@ -59,8 +60,9 @@ Implemented here (the service manifest's `capabilities`, audience `openvibe.ai`;
registered since openvibe-contracts 0.109.0). A service token's `ns`
registered since openvibe-contracts 0.109.0). A service token's `ns`
claim limits which workflow namespaces it may run; an app token is fixed to its project's.
claim limits which workflow namespaces it may run; an app token is fixed to its project's.
-Called elsewhere, as the service principal `ai`: `events.event.publish` (Events, the outbox relay)
-and `network.modules.read` / `network.modules.write` for `ai.preferences` and `ai.usage_summary`
+Called elsewhere, as the service principal `ai`: `events.event.publish` (Events, the outbox relay) and
+`events.subscription.manage` (Events, creating AI's own `bench.rating.changed` subscription), and
+`network.modules.read` / `network.modules.write` for `ai.preferences` and `ai.usage_summary`
(Network). Other services grant `ai` their own draft capabilities (for example
(Network). Other services grant `ai` their own draft capabilities (for example
`blog.post.create`, `news.story.revise`, `reviews.summary.propose`) to receive drafts.
`blog.post.create`, `news.story.revise`, `reviews.summary.propose`) to receive drafts.
@@ -362,6 +364,21 @@ failure; a fallback that answers is recorded on the run (`fallback_used`), in th
@@ -362,6 +364,21 @@ failure; a fallback that answers is recorded on the run (`fallback_used`), in th
(objective, reasons, selected, and every candidate with its estimated cost, latency and excluded
(objective, reasons, selected, and every candidate with its estimated cost, latency and excluded
reason).
reason).
+### Bench evidence (BENCH-11, plan §4.12 and §4.7)
+
+AI consumes OpenVibe.Bench's `bench.rating.changed` (through the signed `/internal/events` webhook, subscribed
+at boot with `events.subscription.manage`) and keeps a small `bench_evidence` table: per kind and category, the
+model, provider, Elo rating, confidence interval, games and Bench's cost per solved. A sparse event (Bench's
+current outbox names only `subjects`) is filled from `GET /api/v1/boards/:kind?type=elo|cost`.
+
+A route learns from that only when it opts in — `learn_from_bench: true` on the route, default off; `bench_kind`
+names the board (when absent AI infers it from the route key or operation, e.g. `chat.reply`), and
+`bench_category` defaults to `all`. Among the route's eligible offers, placement then prefers the **cheapest**
+offer whose rating interval overlaps the leader's (the highest non-provisional rating): statistically tied with
+the best, so price decides. A provisional rating (under 10 games) never decides. With no evidence the order is
+exactly the one placement produced. `GET /api/v1/routes/:key` carries a `bench` object with the evidence, the
+leader, the tied offers, the selected offer and the reason; the operator console shows the same read-only.
+
`GET /api/v1/routes?operation=generate` lists each route's eligible offers as `{ provider, model }`.
`GET /api/v1/routes?operation=generate` lists each route's eligible offers as `{ provider, model }`.
A first-party service holding `ai.run.create` may pin one on `POST /api/v1/generate` (and on
A first-party service holding `ai.run.create` may pin one on `POST /api/v1/generate` (and on
`POST /api/v1/runs` for `ai.generate`) with `"route": { "provider": …, "model": … }`. The run tries
`POST /api/v1/runs` for `ai.generate`) with `"route": { "provider": …, "model": … }`. The run tries
@@ -36,7 +36,9 @@
@@ -36,7 +36,9 @@
"ai.run.failed",
"ai.run.failed",
"ai.run.cached"
"ai.run.cached"
],
],
- "eventsConsumed": [],
+ "eventsConsumed": [
+ "bench.rating.changed"
+ ],
"namespacesOwned": [
"namespacesOwned": [
"ai.usage_summary"
"ai.usage_summary"
],
],
@@ -87,5 +89,5 @@
@@ -87,5 +89,5 @@
"none": "no claims: one process owns ai.db (better-sqlite3, single writer); a run is taken with a compare-and-set on its status (queued → running)"
"none": "no claims: one process owns ai.db (better-sqlite3, single writer); a run is taken with a compare-and-set on its status (queued → running)"
}
}
},
},
- "notes": "Providers, models, routing profiles, versioned prompt templates and workflows, runs, request logs, citations, scoped cache, quotas and audit. Workflow output is a draft/evidence package attributed to workflow + model + run. ai.run.queued|succeeded|failed|cached have payload contracts (v0.30.0, status planned): AI publishes them once its Events client is wired in; it publishes none today. ai.run.create takes ai.run-request@1 and returns ai.run@1."
+ "notes": "Providers, models, routing profiles, versioned prompt templates and workflows, runs, request logs, citations, scoped cache, quotas and audit. Workflow output is a draft/evidence package attributed to workflow + model + run. ai.run.queued|succeeded|failed|cached have payload contracts; AI publishes them through its outbox. AI consumes bench.rating.changed through a signed Events subscription and keeps per-kind Bench evidence; only routes with learn_from_bench: true consult it in placement. ai.run.create takes ai.run-request@1 and returns ai.run@1."
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,33 @@
@@ -0,0 +1,33 @@
+-- phase: expand
+-- BENCH-11 (plan §4.12 and §4.7): a route can learn from OpenVibe.Bench's measured outcomes.
+-- Bench publishes bench.rating.changed; AI keeps the latest rating and cost-per-solved it has seen per
+-- kind/category/model, and only a route that opts in (routes.learn_from_bench) consults it during placement.
+-- The inbox is openvibe-sdk/events.inboxSchema('bench_event_inbox'): a redelivered event is a no-op.
+CREATE TABLE IF NOT EXISTS bench_evidence (
+ kind_id text COLLATE "C" NOT NULL, -- bench.kind@1's id, e.g. chat.reply
+ category text COLLATE "C" NOT NULL DEFAULT 'all', -- bench.rating@1's category (the board's task class)
+ model text COLLATE "C" NOT NULL, -- bench.rating@1's subject: the model rated
+ provider text COLLATE "C" NOT NULL DEFAULT '', -- AI's provider for that model, '' when unknown/ambiguous
+ rating double precision NOT NULL, -- Elo
+ ci_lower double precision,
+ ci_upper double precision,
+ games bigint NOT NULL DEFAULT 0,
+ provisional bigint NOT NULL DEFAULT 0, -- Bench's flag; games < 10 is provisional too
+ cost_per_solved double precision, -- USD from GET /api/v1/boards/:kind?type=cost
+ updated_at text COLLATE "C" NOT NULL, -- the event's at (ISO-8601)
+ PRIMARY KEY (kind_id, category, model, provider)
+);
+CREATE INDEX IF NOT EXISTS bench_evidence_kind ON bench_evidence (kind_id, category);
+
+CREATE TABLE IF NOT EXISTS bench_event_inbox (
+ consumer text NOT NULL,
+ event_id text NOT NULL,
+ processed_at bigint NOT NULL,
+ PRIMARY KEY (consumer, event_id)
+);
+
+-- The opt-in is per route version, default off. bench_kind names the board to learn from; when it is
+-- null, AI infers it from the route key or its operation (chat.reply for chat/generate/...).
+ALTER TABLE routes ADD COLUMN IF NOT EXISTS learn_from_bench bigint NOT NULL DEFAULT 0;
+ALTER TABLE routes ADD COLUMN IF NOT EXISTS bench_kind text COLLATE "C";
+ALTER TABLE routes ADD COLUMN IF NOT EXISTS bench_category text COLLATE "C";
@@ -133,7 +133,20 @@ function adminRouter({ db, registry, pool, quotas, cache, runs, auth, ops = crea
@@ -133,7 +133,20 @@ function adminRouter({ db, registry, pool, quotas, cache, runs, auth, ops = crea
if (!latest) throw new AiError(404, 'ai.not_found', `no route ${req.params.key}`);
if (!latest) throw new AiError(404, 'ai.not_found', `no route ${req.params.key}`);
const versions = (await registry.listRoutes({ history: true })).filter(x => x.key === req.params.key).map(x => ({ version: x.version, status: x.status, created_by: x.created_by, created_at: x.created_at }));
const versions = (await registry.listRoutes({ history: true })).filter(x => x.key === req.params.key).map(x => ({ version: x.version, status: x.status, created_by: x.created_by, created_at: x.created_at }));
const operation = req.query.operation ? String(req.query.operation) : null;
const operation = req.query.operation ? String(req.query.operation) : null;
- return { route: { ...latest, offers: await routeOffers(latest, operation, req) }, versions };
+ // BENCH-11: the evidence this route used, the leader it compared against and why the offer order
+ // changed (or stayed). Null while there is no placement to learn from (an inactive route).
+ let bench = null;
+ if (latest.status === 'active' && pool.benchView) {
+ try {
+ const target = latest.alias_of ? await registry.resolveRoute(latest.key) : latest;
+ if (target && target.status === 'active') {
+ bench = await pool.benchView(target, operationFor(target, operation), { requesterType: req.principal.subject.type, requesterId: req.principal.subject.id });
+ }
+ } catch (err) {
+ log.warn(`[ai] bench evidence for route ${latest.key}: ${err.message}`);
+ }
+ }
+ return { route: { ...latest, offers: await routeOffers(latest, operation, req), bench }, versions };
}));
}));
for (const [plural, k] of Object.entries(kinds)) {
for (const [plural, k] of Object.entries(kinds)) {
@@ -20,7 +20,7 @@ const ovServe = require('openvibe-shared/serve');
@@ -20,7 +20,7 @@ const ovServe = require('openvibe-shared/serve');
const home = require('./home');
const home = require('./home');
const pkg = require('../package.json');
const pkg = require('../package.json');
-function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, vault = null }) {
+function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, vault = null, benchEvents = null }) {
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
app.set('trust proxy', 'loopback');
@@ -87,6 +87,16 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
@@ -87,6 +87,16 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
// GET /release.json (ADR-016) and POST /release-metrics (open tabs' update reports into /metrics).
// GET /release.json (ADR-016) and POST /release-metrics (open tabs' update reports into /metrics).
release.mount(app, { registry: metrics.registry });
release.mount(app, { registry: metrics.registry });
+ // Events → AI (BENCH-11): loopback only, like every service's /internal. Events posts the signed v2
+ // delivery; nginx answers 404 for /internal, and a request carrying forwarding headers is not loopback.
+ app.use('/internal', (req, res, next) => {
+ if (req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip']) {
+ return http.sendProblem(res, 404, 'ai.not_found', { ctx: req.ov });
+ }
+ return next();
+ });
+ if (benchEvents) app.use(benchEvents.router);
+
// Operator actions shared by the admin API and the operator console (one implementation, one audit row).
// Operator actions shared by the admin API and the operator console (one implementation, one audit row).
const ops = createOps({ db, registry, pool, quotas, cache, runs });
const ops = createOps({ db, registry, pool, quotas, cache, runs });
app.use(runsRouter({ runs, registry, auth, config, log }));
app.use(runsRouter({ runs, registry, auth, config, log }));
new file mode 100644
new file mode 100644
@@ -0,0 +1,122 @@
@@ -0,0 +1,122 @@
+'use strict';
+/**
+ * OpenVibe.Events → AI for Bench's ratings. This is AI's consumer side (the ai.run.* outbox in
+ * server/events.js is its producer side).
+ *
+ * At boot it ensures one subscription — topic `bench.rating.changed`, endpoint this service's loopback
+ * `/internal/events` — with AI's own Events token (audience openvibe.events, scope
+ * events.subscription.manage), so Network must grant client `ai` that scope on openvibe.events. Creation
+ * is idempotent (Events answers 409 with the existing subscription) and retried in the background. The
+ * endpoint verifies the v2 signature with AI_EVENTS_SECRET (comma-separated during rotation) and hands the
+ * envelope to server/bench/evidence.js, whose inbox makes a redelivery a no-op.
+ *
+ * Off without EVENTS_URL/AI_EVENTS_URL and AI_EVENTS_SECRET; AI_EVENTS_SUBSCRIBE=0 creates no subscription
+ * (an operator/script may do it), while the endpoint still verifies deliveries when the secret is set.
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { serviceAuth } = require('openvibe-contracts');
+const { parseDelivery } = require('openvibe-sdk/events');
+
+const TOPIC = 'bench.rating.changed';
+
+function createBenchEvents({ bench, config, log = console, fetchImpl = globalThis.fetch, clientSecret = process.env.OV_OAUTH_CLIENT_SECRET }) {
+ const router = express.Router();
+ const secrets = (config.events && config.events.secrets) || [];
+
+ function refused(res, status, code, detail) {
+ return http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });
+ }
+
+ /** Apply one verified envelope (also used by tests and an operator replay). */
+ async function apply(event) {
+ return await bench.applyEvent(event);
+ }
+
+ router.post('/internal/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {
+ if (!secrets.length) return refused(res, 503, 'bench.consumer_disabled', 'AI_EVENTS_SECRET is not set');
+ const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body || '');
+ let delivery = null;
+ for (const secret of secrets) {
+ delivery = parseDelivery(raw, req.headers, secret, { requireV2: true });
+ if (delivery) break;
+ }
+ if (!delivery) return refused(res, 401, 'bench.bad_signature', 'the Events delivery signature does not verify');
+ const event = delivery.event;
+ if (!event || typeof event.event_id !== 'string' || !event.event_id) {
+ return refused(res, 400, 'bench.bad_delivery', 'body must be { event: <envelope> }');
+ }
+ try {
+ const out = await apply(event);
+ return res.status(200).json({ event_id: event.event_id, ...out });
+ } catch (err) {
+ log.error(`[bench] event ${event.event_id} (${event.event_type}) failed: ${err.message}`);
+ return refused(res, 500, 'bench.event_failed', 'processing failed; it will be retried');
+ }
+ });
+
+ /**
+ * Ensure the subscription exists, idempotently and in the background. Returns a stop handle, or null
+ * while the consumer is off. Nothing is created when AI_EVENTS_SUBSCRIBE=0 or no signing secret is set.
+ */
+ function startSubscriptions({ port, eventsUrl = config.events && config.events.url, networkUrl = config.networkInternalUrl, clientId = config.console && config.console.clientId } = {}) {
+ if (!config.events || config.events.subscribe === false || !eventsUrl || !secrets.length || !clientSecret) return null;
+ const base = String(eventsUrl).replace(/\/+$/, '');
+ const endpoint = (config.events.endpoint || `http://127.0.0.1:${port}/internal/events`);
+ const tokens = serviceAuth.createTokenClient({
+ tokenUrl: `${networkUrl}/oauth/token`, clientId: clientId || 'ai', clientSecret,
+ audience: 'openvibe.events', scope: 'events.subscription.manage', fetchImpl,
+ });
+ const call = async (method, path, body) => {
+ const res = await fetchImpl(`${base}${path}`, {
+ method,
+ headers: { Accept: 'application/json', ...(body ? { 'Content-Type': 'application/json' } : {}), ...(await tokens.authHeaders()) },
+ body: body ? JSON.stringify(body) : undefined,
+ signal: AbortSignal.timeout(15000),
+ });
+ const json = await res.json().catch(() => ({}));
+ return { status: res.status, ok: res.ok, body: json };
+ };
+ const attempt = async () => {
+ const listed = await call('GET', '/api/v1/subscriptions');
+ if (!listed.ok) throw new Error(`listing subscriptions: ${listed.status}`);
+ const mine = (listed.body.subscriptions || []).filter((s) => s.endpoint === endpoint);
+ if (mine.some((s) => s.topic_pattern === TOPIC)) return;
+ const made = await call('POST', '/api/v1/subscriptions', { topic_pattern: TOPIC, endpoint, secret: secrets[0] });
+ if (!made.ok && made.status !== 409) throw new Error(`subscribing to ${TOPIC}: ${made.status} ${made.body.code || ''}`);
+ if (made.ok) log.log(`[bench] Events subscription created: ${made.body.id} (${TOPIC} → ${endpoint})`);
+ };
+ const delays = [0, 10_000, 60_000, 5 * 60_000, 15 * 60_000];
+ let i = 0;
+ let timer = null;
+ let stopped = false;
+ const run = () => {
+ timer = null;
+ if (stopped) return;
+ attempt().catch((err) => {
+ if (stopped) return;
+ if (++i < delays.length) { timer = setTimeout(run, delays[i]); if (timer.unref) timer.unref(); }
+ else log.warn(`[bench] Events subscription not created: ${err.message}`);
+ });
+ };
+ timer = setTimeout(run, delays[0]);
+ if (timer.unref) timer.unref();
+ return {
+ topic: TOPIC,
+ endpoint,
+ stop() { stopped = true; if (timer) clearTimeout(timer); timer = null; },
+ };
+ }
+
+ async function status() {
+ return {
+ enabled: Boolean(secrets.length),
+ topic: TOPIC,
+ evidence: await bench.list(null).catch(() => []),
+ };
+ }
+
+ return { router, apply, startSubscriptions, status, TOPIC };
+}
+
+module.exports = { createBenchEvents, TOPIC };
new file mode 100644
new file mode 100644
@@ -0,0 +1,376 @@
@@ -0,0 +1,376 @@
+'use strict';
+/**
+ * BENCH-11 (plan §4.12 and §4.7): OpenVibe.Bench's measured outcomes as routing evidence.
+ *
+ * Bench publishes bench.rating.changed when a board rating moves. AI keeps what it needs from each event
+ * in bench_evidence — kind, category, model, provider, Elo, the confidence interval, games and the board's
+ * cost per solved — and a route that opts in (learn_from_bench: true, default off) may prefer the cheapest
+ * offer whose interval overlaps the leader's. A provisional rating (under 10 games) never decides a route.
+ * With no evidence, select() returns the order it was given: placement is unchanged.
+ *
+ * The event and the board disagree about richness: bench.rating.changed@1 carries one full rating, while
+ * Bench's current outbox announces `{ subjects: [...] }`. Both are accepted; when the event is sparse the
+ * board endpoints (GET /api/v1/boards/:kind?type=elo|cost) fill the gap. A redelivery is a no-op through
+ * the openvibe-sdk inbox (bench_event_inbox).
+ */
+const { createPgInbox } = require('openvibe-sdk/events');
+
+const CONSUMER = 'ai-bench.rating';
+const MIN_GAMES = 10;
+const DEFAULT_CATEGORY = 'all';
+const OPERATION_KIND = {
+ chat: 'chat.reply',
+ complete: 'chat.reply',
+ generate: 'chat.reply',
+ summarize: 'chat.reply',
+ classify: 'chat.reply',
+ extract: 'chat.reply',
+ enrich: 'chat.reply',
+ vision: 'vision.pair',
+ transcribe: 'speech.asr',
+ image: 'vision.answer',
+};
+
+const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
+const flag = (v) => v === true || v === 1 || /^(1|true|yes|on)$/i.test(String(v || ''));
+const text = (v) => (v == null ? '' : String(v).trim());
+const uniq = (xs) => [...new Set(xs.filter(Boolean))];
+
+/** The board category (Bench's task class) a route reads; `all` is Bench's default. */
+function categoryFor(route) {
+ const c = (route && route.constraints) || {};
+ const o = (route && route.options) || {};
+ return text((route && route.bench_category) || c.bench_category || o.bench_category) || DEFAULT_CATEGORY;
+}
+
+/** The boards a route may learn from, most specific first. */
+function kindCandidates(route, operation) {
+ const c = (route && route.constraints) || {};
+ const o = (route && route.options) || {};
+ const explicit = [
+ route && route.bench_kind,
+ c.bench_kind, c.task_class, c.kind_id, c.kind,
+ o.bench_kind, o.task_class, o.kind_id, o.kind,
+ ].map(text);
+ // An explicit board is authoritative: no evidence there means no evidence, never a fallback board.
+ const first = explicit.find(Boolean);
+ if (first) return [first];
+ const out = [];
+ if (route && route.key) out.push(text(route.key));
+ const op = text((route && route.capability) || operation);
+ if (OPERATION_KIND[op]) out.push(OPERATION_KIND[op]);
+ return uniq(out);
+}
+
+function intervalOf(row) {
+ const lower = num(row && row.ci_lower);
+ const upper = num(row && row.ci_upper);
+ if (lower == null || upper == null) return null;
+ return { lower: Math.min(lower, upper), upper: Math.max(lower, upper) };
+}
+
+function provisionalOf(row) {
+ return Boolean(row && row.provisional) || Number(row && row.games) < MIN_GAMES;
+}
+
+function usable(row) {
+ return Boolean(row) && num(row.rating) != null && !provisionalOf(row) && intervalOf(row) != null;
+}
+
+function overlaps(a, b) {
+ const x = intervalOf(a);
+ const y = intervalOf(b);
+ return Boolean(x && y && x.lower <= y.upper && y.lower <= x.upper);
+}
+
+/** An evidence row applies to an offer when the model matches; a known provider must match too. */
+function matchesOffer(row, offer) {
+ if (!row || !offer || !text(offer.model) || text(row.model) !== text(offer.model)) return false;
+ return !text(row.provider) || !text(offer.provider) || text(row.provider) === text(offer.provider);
+}
+
+function viewRow(row) {
+ const rating = num(row.rating);
+ const interval = intervalOf(row);
+ return {
+ kind_id: row.kind_id,
+ category: row.category,
+ model: row.model,
+ provider: text(row.provider) || null,
+ rating,
+ // `elo` is the same number under Bench's own name, so a reader of either shape finds it.
+ elo: rating,
+ interval,
+ ci_lower: interval ? interval.lower : null,
+ ci_upper: interval ? interval.upper : null,
+ games: Number(row.games) || 0,
+ provisional: provisionalOf(row),
+ cost_per_solved: num(row.cost_per_solved),
+ updated_at: row.updated_at || null,
+ };
+}
+
+function viewSelected(row, offer) {
+ return {
+ ...viewRow(row),
+ provider: offer.provider || null,
+ model: offer.model || null,
+ };
+}
+
+function leaderSort(a, b) {
+ return (num(b.rating) - num(a.rating))
+ || (Number(b.games) - Number(a.games))
+ || String(a.model).localeCompare(String(b.model))
+ || String(a.provider || '').localeCompare(String(b.provider || ''));
+}
+
+/**
+ * Pure tie rule: given every evidence row for one kind/category and placement's eligible order, prefer the
+ * cheapest offer statistically tied with the best non-provisional rating. Returns the (possibly reordered)
+ * order and everything a caller needs to explain the decision. No database or clock here.
+ */
+function chooseFromEvidence({ kind_id = null, category = DEFAULT_CATEGORY, evidence = [], order = [] }) {
+ const rows = Array.isArray(evidence) ? evidence : [];
+ const offers = Array.isArray(order) ? order : [];
+ const usableRows = rows.filter(usable).sort(leaderSort);
+ const base = {
+ enabled: true, kind_id, category, evidence: usableRows.map(viewRow),
+ leader: null, tied: [], selected: null, status: 'no_evidence',
+ reason: `no bench evidence for ${kind_id || 'this route'}${category && category !== DEFAULT_CATEGORY ? ` (${category})` : ''}`,
+ };
+ if (!usableRows.length) {
+ if (rows.length) {
+ base.status = 'provisional_or_no_interval';
+ base.reason = `bench evidence for ${kind_id || 'this route'} is provisional (under ${MIN_GAMES} games) or has no rating interval`;
+ }
+ return { order: offers, view: base };
+ }
+ const leader = usableRows[0];
+ const leaderView = viewRow(leader);
+ base.leader = leaderView;
+ const tiedRows = usableRows.filter((row) => overlaps(row, leader));
+
+ // One candidate per eligible offer; a model may appear under several providers, so pick that offer's
+ // best rating first and its cheapest cost as the tie-break.
+ const tied = [];
+ for (let index = 0; index < offers.length; index++) {
+ const offer = offers[index];
+ const matched = tiedRows.filter((row) => matchesOffer(row, offer))
+ .sort((a, b) => leaderSort(a, b) || (num(a.cost_per_solved) ?? Infinity) - (num(b.cost_per_solved) ?? Infinity));
+ if (!matched.length) continue;
+ tied.push({ ...viewSelected(matched[0], offer), index });
+ }
+ base.tied = tied.map(({ index, ...rest }) => rest);
+ if (!tied.length) {
+ base.status = 'no_tied_offer';
+ base.reason = `no eligible offer is statistically tied with ${leader.model} (rating ${leaderView.rating})`;
+ return { order: offers, view: base };
+ }
+ const priced = tied.filter((row) => row.cost_per_solved != null && row.cost_per_solved >= 0)
+ .sort((a, b) => a.cost_per_solved - b.cost_per_solved || a.index - b.index);
+ if (!priced.length) {
+ base.status = 'no_cost';
+ base.reason = `the offers tied with ${leader.model} have no cost per solved yet`;
+ return { order: offers, view: base };
+ }
+ const winner = priced[0];
+ const next = [...offers];
+ if (winner.index > 0) {
+ const [chosen] = next.splice(winner.index, 1);
+ next.unshift(chosen);
+ }
+ base.status = 'selected';
+ base.selected = {
+ provider: winner.provider, model: winner.model,
+ kind_id: winner.kind_id, category: winner.category,
+ rating: winner.rating, interval: winner.interval, games: winner.games,
+ cost_per_solved: winner.cost_per_solved, updated_at: winner.updated_at,
+ };
+ base.reason = `${winner.provider ? `${winner.provider}/` : ''}${winner.model} is the cheapest offer statistically tied with ${leader.model} `
+ + `(rating ${leaderView.rating}, ${leaderView.games} games) at $${winner.cost_per_solved}/solved`;
+ return { order: next, view: base };
+}
+
+function createBenchEvidence({ db, registry = null, config = {}, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console } = {}) {
+ const table = 'bench_evidence';
+ const inbox = createPgInbox(db, { table: 'bench_event_inbox', now: () => clock.now() });
+ const benchUrl = text(config.bench && config.bench.url).replace(/\/+$/, '');
+ const timeoutMs = Math.max(1000, Number(config.bench && config.bench.timeoutMs) || 15000);
+
+ const selectRows = db.prepare(`SELECT * FROM ${table} WHERE kind_id = ? AND category = ? ORDER BY rating DESC, games DESC, model ASC, provider ASC`);
+ async function list(kindId, category = DEFAULT_CATEGORY) {
+ if (!kindId) return [];
+ return await selectRows.all(text(kindId), text(category) || DEFAULT_CATEGORY);
+ }
+
+ async function fetchBoard(kindId, type) {
+ if (!benchUrl) throw new Error('AI_BENCH_URL is not set');
+ const url = `${benchUrl}/api/v1/boards/${encodeURIComponent(kindId)}?type=${encodeURIComponent(type)}`;
+ const res = await fetchImpl(url, { headers: { Accept: 'application/json' }, signal: AbortSignal.timeout(timeoutMs) });
+ if (!res.ok) throw new Error(`bench board ${kindId} ${type}: ${res.status}`);
+ const body = await res.json();
+ const board = body && body.board ? body.board : body;
+ if (!board || !Array.isArray(board.rows)) throw new Error(`bench board ${kindId} ${type}: no rows`);
+ return board.rows;
+ }
+
+ /** A model's provider from AI's registry, when exactly one provider offers it; Bench rates the model, not the seller. */
+ async function providerMap() {
+ const out = new Map();
+ if (!registry || typeof registry.listModels !== 'function') return out;
+ try {
+ for (const m of await registry.listModels()) {
+ if (!out.has(m.model_key)) out.set(m.model_key, new Set());
+ out.get(m.model_key).add(m.provider_key);
+ }
+ } catch (err) {
+ log.warn(`[bench] provider lookup failed: ${err.message}`);
+ }
+ return out;
+ }
+
+ function ratingsFromPayload(payload) {
+ const source = Array.isArray(payload.ratings) ? payload.ratings
+ : Array.isArray(payload.rows) ? payload.rows
+ : Array.isArray(payload.subjects) ? payload.subjects
+ : (payload.subject || payload.model) ? [payload] : [];
+ return source.map((raw) => {
+ const r = raw && typeof raw === 'object' ? raw : { subject: raw };
+ const subject = r.subject && typeof r.subject === 'object' ? r.subject.id : r.subject;
+ return {
+ model: text(r.model || subject || r.id),
+ provider: text(r.provider),
+ rating: num(r.rating != null ? r.rating : r.elo),
+ ci_lower: num(r.ci_lower),
+ ci_upper: num(r.ci_upper),
+ games: num(r.games),
+ provisional: r.provisional === undefined ? null : flag(r.provisional),
+ cost_per_solved: num(r.cost_per_solved),
+ updated_at: r.updated_at || r.at || null,
+ };
+ }).filter((r) => r.model);
+ }
+
+ /** Fill a sparse event from Bench's boards; a full rating still gets the board's cost when it is reachable. */
+ async function hydrate(payload, rows) {
+ const kindId = text(payload.kind_id);
+ const missingRating = rows.some((r) => r.rating == null || r.games == null || r.ci_lower == null || r.ci_upper == null);
+ let eloRows = null;
+ if (missingRating) {
+ eloRows = await fetchBoard(kindId, 'elo'); // a sparse event is useless without it: let Events retry
+ for (const row of rows) {
+ const b = eloRows.find((x) => text(x.subject || x.model) === row.model);
+ if (!b) continue;
+ if (row.rating == null) row.rating = num(b.elo != null ? b.elo : b.rating);
+ if (row.games == null) row.games = num(b.games);
+ if (row.ci_lower == null) row.ci_lower = num(b.ci_lower);
+ if (row.ci_upper == null) row.ci_upper = num(b.ci_upper);
+ if (row.provisional == null) row.provisional = b.provisional === undefined ? null : flag(b.provisional);
+ }
+ }
+ if (rows.some((r) => r.cost_per_solved == null)) {
+ try {
+ const costRows = await fetchBoard(kindId, 'cost');
+ for (const row of rows) {
+ const b = costRows.find((x) => text(x.subject || x.model) === row.model);
+ if (b && row.cost_per_solved == null) row.cost_per_solved = num(b.cost_per_solved);
+ }
+ } catch (err) {
+ // A rating without a price is still evidence; the next rating change fills the cost in.
+ if (rows.some((r) => r.rating == null)) throw err;
+ log.warn(`[bench] cost per solved unavailable for ${kindId}: ${err.message}`);
+ }
+ }
+ return rows;
+ }
+
+ /**
+ * Apply one bench.rating.changed envelope, exactly once. Returns { duplicate, stored }.
+ * Only source `bench` and the rating topic are ours; anything else is acknowledged and ignored.
+ */
+ async function applyEvent(event) {
+ if (!event || event.event_type !== 'bench.rating.changed') return { ignored: true, reason: 'not bench.rating.changed' };
+ if (event.source && event.source !== 'bench') return { ignored: true, reason: `source ${event.source}` };
+ if (!event.event_id) throw new Error('bench.rating.changed has no event_id');
+ const payload = event.payload && typeof event.payload === 'object' ? event.payload : {};
+ const kindId = text(payload.kind_id);
+ if (!kindId) throw new Error('bench.rating.changed has no payload.kind_id');
+ const category = text(payload.category) || DEFAULT_CATEGORY;
+ const eventAt = text(payload.at || event.timestamp);
+ const when = eventAt && !Number.isNaN(new Date(eventAt).getTime()) ? new Date(eventAt).toISOString() : new Date(clock.now()).toISOString();
+ // A redelivery is common (Events retries); do not call Bench's boards for an event already applied.
+ if (await inbox.seen(CONSUMER, String(event.event_id))) return { duplicate: true, stored: 0 };
+ const rows = await hydrate(payload, ratingsFromPayload(payload));
+ const providers = await providerMap();
+
+ const receipt = await inbox.once(CONSUMER, String(event.event_id), async (t) => {
+ let stored = 0;
+ for (const row of rows) {
+ if (row.rating == null) continue; // a board with no rating yet is not evidence
+ const known = providers.get(row.model);
+ const provider = row.provider || (known && known.size === 1 ? [...known][0] : '');
+ const provisional = row.provisional == null ? (Number(row.games) || 0) < MIN_GAMES : row.provisional || (Number(row.games) || 0) < MIN_GAMES;
+ await t.exec(
+ `INSERT INTO ${table} (kind_id, category, model, provider, rating, ci_lower, ci_upper, games, provisional, cost_per_solved, updated_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
+ ON CONFLICT (kind_id, category, model, provider) DO UPDATE SET
+ rating = excluded.rating, ci_lower = excluded.ci_lower, ci_upper = excluded.ci_upper,
+ games = excluded.games, provisional = excluded.provisional,
+ cost_per_solved = COALESCE(excluded.cost_per_solved, ${table}.cost_per_solved),
+ updated_at = excluded.updated_at
+ WHERE excluded.updated_at >= ${table}.updated_at`,
+ [kindId, category, row.model, provider, row.rating, row.ci_lower, row.ci_upper,
+ Number(row.games) || 0, provisional ? 1 : 0, row.cost_per_solved, when],
+ );
+ stored++;
+ }
+ return { stored };
+ });
+ return receipt.duplicate ? { duplicate: true, stored: 0 } : { duplicate: false, ...(receipt.result || {}) };
+ }
+
+ /** The order a route should try, with the evidence and reason that changed (or left) it. */
+ async function select({ route, order = [], operation = null } = {}) {
+ const enabled = Boolean(route && route.learn_from_bench);
+ const kinds = kindCandidates(route, operation);
+ const category = categoryFor(route);
+ if (!enabled) {
+ return {
+ order,
+ view: {
+ enabled: false, kind_id: kinds[0] || null, category, evidence: [],
+ leader: null, tied: [], selected: null, status: 'off',
+ reason: 'route does not opt in (learn_from_bench is false)',
+ },
+ };
+ }
+ for (const kindId of kinds) {
+ const evidence = await list(kindId, category);
+ if (!evidence.length) continue;
+ const picked = chooseFromEvidence({ kind_id: kindId, category, evidence, order });
+ return { order: picked.order, view: picked.view };
+ }
+ return {
+ order,
+ view: {
+ enabled: true, kind_id: kinds[0] || null, category, evidence: [],
+ leader: null, tied: [], selected: null, status: 'no_evidence',
+ reason: kinds[0]
+ ? `no bench evidence for ${kinds[0]}${category !== DEFAULT_CATEGORY ? ` (${category})` : ''}`
+ : 'no bench kind on this route (set bench_kind or use a route key that names one)',
+ },
+ };
+ }
+
+ async function view({ route, order = [], operation = null } = {}) {
+ return (await select({ route, order, operation })).view;
+ }
+
+ return { applyEvent, select, view, list, kindCandidates, categoryFor, CONSUMER, MIN_GAMES };
+}
+
+module.exports = {
+ createBenchEvidence, chooseFromEvidence, kindCandidates, categoryFor,
+ intervalOf, usable, overlaps, matchesOffer, viewRow, MIN_GAMES, DEFAULT_CATEGORY, CONSUMER,
+};
@@ -61,6 +61,22 @@ function load(env = process.env) {
@@ -61,6 +61,22 @@ function load(env = process.env) {
networkPublicKey: env.OV_NETWORK_PUBLIC_KEY ? env.OV_NETWORK_PUBLIC_KEY.replace(/\\n/g, '\n') : null,
networkPublicKey: env.OV_NETWORK_PUBLIC_KEY ? env.OV_NETWORK_PUBLIC_KEY.replace(/\\n/g, '\n') : null,
audience: 'openvibe.ai',
audience: 'openvibe.ai',
+ // ── OpenVibe.Events → AI (server/bench/events.js): AI subscribes to bench.rating.changed at boot.
+ // The secret verifies the signed delivery (v2); unset turns POST /internal/events off. EVENTS_URL
+ // is the shared producer/consumer address, AI_EVENTS_URL overrides it. AI_EVENTS_SUBSCRIBE=0 (or no
+ // secret) leaves subscription creation to the operator/script; the endpoint is this service's loopback.
+ events: {
+ url: trimUrl(env.AI_EVENTS_URL || env.EVENTS_URL),
+ secrets: list(env.AI_EVENTS_SECRET || env.AI_BENCH_EVENTS_SECRET || env.EVENTS_SECRET, []),
+ endpoint: String(env.AI_EVENTS_ENDPOINT || '').trim(),
+ subscribe: bool(env.AI_EVENTS_SUBSCRIBE, true),
+ },
+ // Where AI reads board detail a sparse bench.rating.changed leaves out (cost per solved, intervals).
+ bench: {
+ url: trimUrl(env.AI_BENCH_URL || env.OV_BENCH_INTERNAL_URL || env.OV_BENCH_URL || env.BENCH_URL || 'http://127.0.0.1:5230'),
+ timeoutMs: int(env.AI_BENCH_TIMEOUT_MS, 15000),
+ },
+
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
@@ -394,6 +394,8 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
@@ -394,6 +394,8 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
primary: { provider: p.key, model: model || p.default_model || null }, fallbacks: previous, capability: null, constraints: {}, pinned: [], alias_of: null,
primary: { provider: p.key, model: model || p.default_model || null }, fallbacks: previous, capability: null, constraints: {}, pinned: [], alias_of: null,
options: (effective && effective.options) || {}, max_output_tokens: effective ? effective.max_output_tokens : null,
options: (effective && effective.options) || {}, max_output_tokens: effective ? effective.max_output_tokens : null,
response_format: (effective && effective.response_format) || 'text', timeout_ms: effective ? effective.timeout_ms : null, status: 'active',
response_format: (effective && effective.response_format) || 'text', timeout_ms: effective ? effective.timeout_ms : null, status: 'active',
+ learn_from_bench: Boolean(effective && effective.learn_from_bench),
+ bench_kind: (effective && effective.bench_kind) || null, bench_category: (effective && effective.bench_category) || null,
}, who(req));
}, who(req));
res.redirect(303, `/console/routes/${encodeURIComponent(route.key)}?done=route_used`);
res.redirect(303, `/console/routes/${encodeURIComponent(route.key)}?done=route_used`);
}));
}));
@@ -420,8 +422,20 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
@@ -420,8 +422,20 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
if (!versions.length) return await send(res, 404, pages.message({ title: 'Not found', text: `No ${k.one} ${req.params.key}.`, back: `/console/${kind}`, ...common(req) }));
if (!versions.length) return await send(res, 404, pages.message({ title: 'Not found', text: `No ${k.one} ${req.params.key}.`, back: `/console/${kind}`, ...common(req) }));
const want = Number(req.query.version);
const want = Number(req.query.version);
const selected = (Number.isInteger(want) && versions.find((x) => x.version === want)) || versions[0];
const selected = (Number.isInteger(want) && versions.find((x) => x.version === want)) || versions[0];
+ const current = await registry[k.get](req.params.key, selected.version);
+ // BENCH-11: read-only evidence on a route's page. It changes what runs pick, so it is shown with
+ // the leader, the tied offers and the reason; it is not editable here (bench_kind is an API field).
+ let bench = null;
+ if (kind === 'routes' && current.status === 'active' && pool.benchView) {
+ try {
+ const target = current.alias_of ? await registry.resolveRoute(current.key) : current;
+ if (target && target.status === 'active') bench = await pool.benchView(target, target.capability || 'chat');
+ } catch (err) {
+ log.warn(`[console] bench evidence for route ${current.key}: ${err.message}`);
+ }
+ }
return await send(res, 200, pages.definition({
return await send(res, 200, pages.definition({
- ...common(req), kind, key: req.params.key, versions, selected: await registry[k.get](req.params.key, selected.version),
+ ...common(req), kind, key: req.params.key, versions, selected: current, bench,
inUse: inUseMap(kind, versions).get(req.params.key), canManage: req.staff.caps.includes(CAPS.workflowManage), notice: notice(req),
inUse: inUseMap(kind, versions).get(req.params.key), canManage: req.staff.caps.includes(CAPS.workflowManage), notice: notice(req),
...(kind === 'routes' ? { canChoose: req.staff.caps.includes(CAPS.providerManage), providers: (await registry.listProviders()).filter((p) => p.status === 'active' && p.kind !== 'whisper' && p.kind !== 'stub'), models: await registry.listModels() } : {}),
...(kind === 'routes' ? { canChoose: req.staff.caps.includes(CAPS.providerManage), providers: (await registry.listProviders()).filter((p) => p.status === 'active' && p.kind !== 'whisper' && p.kind !== 'stub'), models: await registry.listModels() } : {}),
}));
}));
@@ -264,6 +264,24 @@ const KINDS = {
@@ -264,6 +264,24 @@ const KINDS = {
};
};
const target = (c) => (c ? html`<code>${c.provider}</code>${c.model ? html` / <code>${c.model}</code>` : ''}` : '—');
const target = (c) => (c ? html`<code>${c.provider}</code>${c.model ? html` / <code>${c.model}</code>` : ''}` : '—');
+/** BENCH-11: the evidence an opted-in route used, read-only. Never a form: changing policy is an API edit. */
+function benchBlock(bench) {
+ if (!bench) return '';
+ if (!bench.enabled) return html`<h2>Bench evidence</h2><p class="muted">Learn from Bench is off for this route (<code>learn_from_bench: false</code>); placement keeps its own order.</p>`;
+ const rows = bench.evidence || [];
+ const tied = new Set((bench.tied || []).map((x) => `${x.provider || ''}|${x.model}`));
+ const body = rows.map((r) => html`<tr><td><code>${r.model}</code>${bench.selected && bench.selected.model === r.model ? html` <span class="ok">selected</span>` : tied.has(`${r.provider || ''}|${r.model}`) ? html` <span class="muted">tied</span>` : ''}</td>
+<td>${r.provider ? html`<code>${r.provider}</code>` : html`<span class="muted">any</span>`}</td><td>${r.rating == null ? '—' : n(r.rating)}</td>
+<td>${r.interval ? `${n(r.interval.lower)} – ${n(r.interval.upper)}` : '—'}</td><td>${n(r.games)}${r.provisional ? ' (provisional)' : ''}</td>
+<td>${r.cost_per_solved == null ? '—' : `$${r.cost_per_solved}`}</td><td>${r.updated_at ? when(r.updated_at) : '—'}</td></tr>`);
+ return html`<h2>Bench evidence</h2>
+<p class="muted">Read-only. ${bench.kind_id ? html`Board <code>${bench.kind_id}</code>` : 'No board named'} · category <code>${bench.category}</code>.
+Provisional ratings (under 10 games) never decide a route.</p>
+<p><strong>Why:</strong> ${bench.reason}</p>
+${bench.leader ? html`<p>Leader: <code>${bench.leader.model}</code> at ${n(bench.leader.rating)} (${n(bench.leader.games)} games, interval ${bench.leader.interval ? `${n(bench.leader.interval.lower)} – ${n(bench.leader.interval.upper)}` : '—'}).</p>` : ''}
+${rows.length ? table('Evidence used', ['Model', 'Provider', 'Rating', 'Interval', 'Games', 'Cost / solved', 'Updated'], body, 'no evidence.') : html`<p class="muted">No evidence yet: routing keeps its current order.</p>`}`;
+}
+
function definitions({ staff, csrf, kind, rows, inUse }) {
function definitions({ staff, csrf, kind, rows, inUse }) {
const k = KINDS[kind];
const k = KINDS[kind];
const link = (key) => html`<a href="/console/${kind}/${enc(key)}"><code>${key}</code></a>`;
const link = (key) => html`<a href="/console/${kind}/${enc(key)}"><code>${key}</code></a>`;
@@ -294,7 +312,7 @@ ${table(`${k.title} (newest version of each)`, head, body, `no ${kind}.`)}`,
@@ -294,7 +312,7 @@ ${table(`${k.title} (newest version of each)`, head, body, `no ${kind}.`)}`,
});
});
}
}
-function definition({ staff, csrf, kind, key, versions, selected, inUse, canManage, notice, error, canChoose = false, providers = [], models = [] }) {
+function definition({ staff, csrf, kind, key, versions, selected, inUse, canManage, notice, error, canChoose = false, providers = [], models = [], bench = null }) {
const k = KINDS[kind];
const k = KINDS[kind];
const v = selected;
const v = selected;
let content;
let content;
@@ -302,9 +320,11 @@ function definition({ staff, csrf, kind, key, versions, selected, inUse, canMana
@@ -302,9 +320,11 @@ function definition({ staff, csrf, kind, key, versions, selected, inUse, canMana
content = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd>
content = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd>
${v.alias_of ? html`<dt>Alias of</dt><dd><a href="/console/routes/${enc(v.alias_of)}"><code>${v.alias_of}</code></a></dd>` : html`<dt>Primary</dt><dd>${target(v.primary)}</dd>
${v.alias_of ? html`<dt>Alias of</dt><dd><a href="/console/routes/${enc(v.alias_of)}"><code>${v.alias_of}</code></a></dd>` : html`<dt>Primary</dt><dd>${target(v.primary)}</dd>
<dt>Fallbacks</dt><dd>${v.fallbacks.length ? html`<ol>${v.fallbacks.map((f) => html`<li>${target(f)}</li>`)}</ol>` : html`<span class="warn">none: an outage of the primary fails every run on this route</span>`}</dd>`}
<dt>Fallbacks</dt><dd>${v.fallbacks.length ? html`<ol>${v.fallbacks.map((f) => html`<li>${target(f)}</li>`)}</ol>` : html`<span class="warn">none: an outage of the primary fails every run on this route</span>`}</dd>`}
+<dt>Learn from Bench</dt><dd>${v.learn_from_bench ? html`on${v.bench_kind ? html` · board <code>${v.bench_kind}</code>` : ''}${v.bench_category ? html` · category <code>${v.bench_category}</code>` : ''}` : 'off'}</dd>
<dt>Response format</dt><dd>${v.response_format}</dd><dt>Max output tokens</dt><dd>${v.max_output_tokens != null ? n(v.max_output_tokens) : '—'}</dd>
<dt>Response format</dt><dd>${v.response_format}</dd><dt>Max output tokens</dt><dd>${v.max_output_tokens != null ? n(v.max_output_tokens) : '—'}</dd>
<dt>Timeout</dt><dd>${v.timeout_ms ? `${n(v.timeout_ms)} ms` : 'provider default'}</dd><dt>Created</dt><dd>${v.created_by || '—'}, ${when(v.created_at)}</dd></dl>
<dt>Timeout</dt><dd>${v.timeout_ms ? `${n(v.timeout_ms)} ms` : 'provider default'}</dd><dt>Created</dt><dd>${v.created_by || '—'}, ${when(v.created_at)}</dd></dl>
-<h3>Options</h3><pre>${JSON.stringify(v.options || {}, null, 2)}</pre>`;
+<h3>Options</h3><pre>${JSON.stringify(v.options || {}, null, 2)}</pre>
+${benchBlock(bench)}`;
} else if (kind === 'templates') {
} else if (kind === 'templates') {
content = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd><dt>Name</dt><dd>${v.name}</dd><dt>Description</dt><dd>${v.description || '—'}</dd>
content = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd><dt>Name</dt><dd>${v.name}</dd><dt>Description</dt><dd>${v.description || '—'}</dd>
<dt>Default route</dt><dd>${v.default_route ? html`<a href="/console/routes/${enc(v.default_route)}"><code>${v.default_route}</code></a>` : '—'}</dd>
<dt>Default route</dt><dd>${v.default_route ? html`<a href="/console/routes/${enc(v.default_route)}"><code>${v.default_route}</code></a>` : '—'}</dd>
@@ -11,6 +11,8 @@ const { load } = require('./config');
@@ -11,6 +11,8 @@ const { load } = require('./config');
const { openDb } = require('./db');
const { openDb } = require('./db');
const { createRegistry } = require('./registry');
const { createRegistry } = require('./registry');
const { createProviderPool } = require('./providers');
const { createProviderPool } = require('./providers');
+const { createBenchEvidence } = require('./bench/evidence');
+const { createBenchEvents } = require('./bench/events');
const { createFetcher } = require('./fetcher');
const { createFetcher } = require('./fetcher');
const { createEngine } = require('./workflows/engine');
const { createEngine } = require('./workflows/engine');
const { createCache } = require('./cache');
const { createCache } = require('./cache');
@@ -35,7 +37,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -35,7 +37,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
// Provider keys an owner stored from the console (server/vault.js): into this process before any provider is built.
// Provider keys an owner stored from the console (server/vault.js): into this process before any provider is built.
const vault = require('./vault').createVault({ db, config, clock, log });
const vault = require('./vault').createVault({ db, config, clock, log });
await vault.load().catch((err) => log.warn(`[Vault] ${err.message}`));
await vault.load().catch((err) => log.warn(`[Vault] ${err.message}`));
- const pool = createProviderPool({ db, registry, config, clock, fetchImpl, env, log, credentialFetch, machineModels });
+ // BENCH-11: Bench's ratings as routing evidence, and the opted-in tie rule inside placement.
+ const bench = createBenchEvidence({ db, registry, config, clock, fetchImpl, log });
+ const pool = createProviderPool({ db, registry, config, clock, fetchImpl, env, log, credentialFetch, machineModels, bench });
// T6 step 8: the SaaS tier budgets on openvibe-sdk/govern in Valkey (server/govern.js); off unless AI_GOVERN_TIERS=1.
// T6 step 8: the SaaS tier budgets on openvibe-sdk/govern in Valkey (server/govern.js); off unless AI_GOVERN_TIERS=1.
const tiers = createTiers({ config, valkey, clock, log });
const tiers = createTiers({ config, valkey, clock, log });
// §2.1.8: the provider free allowance per subject, priced off the pool's own rate cards (server/free-allowance.js);
// §2.1.8: the provider free allowance per subject, priced off the pool's own rate cards (server/free-allowance.js);
@@ -65,7 +69,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -65,7 +69,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
// NODE_ENV=test (tests pin a PEM or point the URL at a stub and let the client fetch on first use).
// NODE_ENV=test (tests pin a PEM or point the URL at a stub and let the client fetch on first use).
const keys = createNetworkKeys({ config });
const keys = createNetworkKeys({ config });
const auth = createAuth({ config, log });
const auth = createAuth({ config, log });
- const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, vault });
+ // The Events consumer side: a signed bench.rating.changed webhook, subscribed at boot (server/bench/events.js).
+ const benchEvents = createBenchEvents({ bench, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });
+ const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, vault, benchEvents });
let jwks = null;
let jwks = null;
let keyLoaded = Promise.resolve(null);
let keyLoaded = Promise.resolve(null);
if (!keys.pinned && config.nodeEnv !== 'test') {
if (!keys.pinned && config.nodeEnv !== 'test') {
@@ -85,6 +91,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -85,6 +91,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
housekeeping.unref?.();
housekeeping.unref?.();
let server = null;
let server = null;
+ let benchSubscription = null;
if (listen) userModules.start();
if (listen) userModules.start();
if (listen) {
if (listen) {
server = await new Promise((resolve, reject) => {
server = await new Promise((resolve, reject) => {
@@ -95,12 +102,14 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -95,12 +102,14 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
server.headersTimeout = 66000;
server.headersTimeout = 66000;
server.requestTimeout = config.runs.maxWaitMs + 30000;
server.requestTimeout = config.runs.maxWaitMs + 30000;
log.log(`[ai] listening on http://${config.host}:${server.address().port} (${config.nodeEnv})`);
log.log(`[ai] listening on http://${config.host}:${server.address().port} (${config.nodeEnv})`);
+ benchSubscription = benchEvents.startSubscriptions({ port: server.address().port });
const shared = await registry.getProvider('shared');
const shared = await registry.getProvider('shared');
log.log(`[ai] shared provider: ${shared.kind} ${shared.status}, credentials ${registry.publicProvider(shared).credentials}; stub fallback ${config.stubFallback ? 'on' : 'off'}`);
log.log(`[ai] shared provider: ${shared.kind} ${shared.status}, credentials ${registry.publicProvider(shared).credentials}; stub fallback ${config.stubFallback ? 'on' : 'off'}`);
}
}
async function close() {
async function close() {
clearInterval(housekeeping);
clearInterval(housekeeping);
+ if (benchSubscription) benchSubscription.stop();
userModules.stop();
userModules.stop();
if (jwks) jwks.stop();
if (jwks) jwks.stop();
if (server) {
if (server) {
@@ -116,7 +125,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -116,7 +125,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
if (!givenDb) await db.close();
if (!givenDb) await db.close();
}
}
- return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, close };
+ return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, bench, benchEvents, close };
}
}
if (require.main === module) {
if (require.main === module) {
@@ -43,7 +43,7 @@ function rateLimited(err) {
@@ -43,7 +43,7 @@ function rateLimited(err) {
return (err && err.status === 429) || (err && err.code === 'provider.rate_limited');
return (err && err.status === 429) || (err && err.code === 'provider.rate_limited');
}
}
-function createProviderPool({ db, registry, config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, credentialFetch = null, machineModels = null }) {
+function createProviderPool({ db, registry, config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, credentialFetch = null, machineModels = null, bench = null }) {
// A person's own machine as their model (credential provider node, server/machines.js).
// A person's own machine as their model (credential provider node, server/machines.js).
const machines = machineModels || require('../machines').createMachineModels({ config, env, fetchImpl });
const machines = machineModels || require('../machines').createMachineModels({ config, env, fetchImpl });
const cache = new Map(); // key -> { stamp, adapter }
const cache = new Map(); // key -> { stamp, adapter }
@@ -339,9 +339,20 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
@@ -339,9 +339,20 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
}
}
async function candidates(route, features, operation, ctx = {}) {
async function candidates(route, features, operation, ctx = {}) {
- const { order, explain, placement: result = null, ...rest } = route.capability && operation
+ let { order, explain, placement: result = null, ...rest } = route.capability && operation
? await poolCandidates(route, features, operation, ctx)
? await poolCandidates(route, features, operation, ctx)
: await pinnedCandidates(route, features, ctx);
: await pinnedCandidates(route, features, ctx);
+ // BENCH-11: an opted-in route may put the cheapest offer statistically tied with Bench's leader first.
+ // A route without the opt-in (or without evidence) keeps the order placement just produced.
+ if (bench) {
+ const picked = await bench.select({ route, order, operation });
+ order = picked.order;
+ explain.bench = picked.view;
+ if (picked.view.status === 'selected' && picked.view.selected) {
+ explain.selected = offerId(picked.view.selected.provider, picked.view.selected.model);
+ explain.reasons = [picked.view.reason, ...(explain.reasons || [])];
+ }
+ }
// A scoped route never ends on synthetic text (media.paid falls back to its extractive overview); an image
// A scoped route never ends on synthetic text (media.paid falls back to its extractive overview); an image
// has no such fallback, so the dev/test stub PNG (AI_STUB_FALLBACK, off in production) still closes it.
// has no such fallback, so the dev/test stub PNG (AI_STUB_FALLBACK, off in production) still closes it.
if (config.stubFallback && (!(route.constraints && route.constraints.provider_scope) || operation === 'image')
if (config.stubFallback && (!(route.constraints && route.constraints.provider_scope) || operation === 'image')
@@ -455,10 +466,12 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
@@ -455,10 +466,12 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
* a pinned run is accepted only when it names one of them.
* a pinned run is accepted only when it names one of them.
*/
*/
async function offers(route, operation, ctx = {}) {
async function offers(route, operation, ctx = {}) {
- const { explain } = await candidates(route, [operation], operation, ctx);
+ const { order, explain } = await candidates(route, [operation], operation, ctx);
+ const rank = new Map((order || []).map((c, i) => [offerId(c.provider, c.model || null), i]));
const out = [];
const out = [];
const seen = new Set();
const seen = new Set();
- for (const c of explain.candidates) {
+ // The order placement would try, including a Bench-tie winner when an opted-in route has evidence.
+ for (const c of [...explain.candidates].sort((a, b) => (rank.get(offerId(a.provider, a.model || null)) ?? Infinity) - (rank.get(offerId(b.provider, b.model || null)) ?? Infinity))) {
if (!c || !c.eligible || !c.provider) continue;
if (!c || !c.eligible || !c.provider) continue;
const id = offerId(c.provider, c.model || null);
const id = offerId(c.provider, c.model || null);
if (seen.has(id)) continue;
if (seen.has(id)) continue;
@@ -468,6 +481,13 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
@@ -468,6 +481,13 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
return out;
return out;
}
}
+ /** The Bench evidence a route used and why, for GET /api/v1/routes/:key and the operator console. */
+ async function benchView(route, operation, ctx = {}) {
+ if (!bench) return null;
+ const { explain } = await candidates(route, [operation], operation, ctx);
+ return explain.bench || null;
+ }
+
async function execute(route, operation, req, ctx = {}) {
async function execute(route, operation, req, ctx = {}) {
if (ctx.credential) return await executeWithCredential(ctx.credential, operation, req, ctx);
if (ctx.credential) return await executeWithCredential(ctx.credential, operation, req, ctx);
const features = [operation];
const features = [operation];
@@ -579,7 +599,7 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
@@ -579,7 +599,7 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
throw new AiError(503, 'provider.unavailable', 'no provider on this route could answer', { tried, explain });
throw new AiError(503, 'provider.unavailable', 'no provider on this route could answer', { tried, explain });
}
}
- return { adapter, health, resetHealth, execute, executeWithCredential, offers, priceFor, costOf, rateCardsFor, providerStates, stats, statFor, candidates };
+ return { adapter, health, resetHealth, execute, executeWithCredential, offers, benchView, priceFor, costOf, rateCardsFor, providerStates, stats, statFor, candidates };
}
}
module.exports = { createProviderPool };
module.exports = { createProviderPool };
@@ -18,6 +18,7 @@ const BILLING_PROFILES = ['metered', 'subscription', 'payg', 'free', 'byok'];
@@ -18,6 +18,7 @@ const BILLING_PROFILES = ['metered', 'subscription', 'payg', 'free', 'byok'];
const FEATURES = ['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'enrich', 'embed', 'vision', 'json', 'tools', 'transcribe', 'image'];
const FEATURES = ['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'enrich', 'embed', 'vision', 'json', 'tools', 'transcribe', 'image'];
const LIFECYCLE = ['draft', 'active', 'deprecated', 'archived'];
const LIFECYCLE = ['draft', 'active', 'deprecated', 'archived'];
const KEY_RE = /^[a-z][a-z0-9_-]*(\.[a-z0-9_-]+)*$/;
const KEY_RE = /^[a-z][a-z0-9_-]*(\.[a-z0-9_-]+)*$/;
+const flag = (v) => v === true || v === 1 || /^(1|true|yes|on)$/i.test(String(v == null ? '' : v));
function createRegistry(db, { clock = { now: () => Date.now() }, env = process.env } = {}) {
function createRegistry(db, { clock = { now: () => Date.now() }, env = process.env } = {}) {
const now = () => iso(clock.now());
const now = () => iso(clock.now());
@@ -189,10 +190,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
@@ -189,10 +190,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
fallbacks: parseJson(r.fallbacks, []), options: parseJson(r.options, {}), max_output_tokens: r.max_output_tokens,
fallbacks: parseJson(r.fallbacks, []), options: parseJson(r.options, {}), max_output_tokens: r.max_output_tokens,
response_format: r.response_format, timeout_ms: r.timeout_ms, alias_of: r.alias_of,
response_format: r.response_format, timeout_ms: r.timeout_ms, alias_of: r.alias_of,
capability: r.capability || null, constraints: parseJson(r.constraints, {}), pinned: parseJson(r.pinned, []),
capability: r.capability || null, constraints: parseJson(r.constraints, {}), pinned: parseJson(r.pinned, []),
+ // BENCH-11: a route's opt-in to Bench's measured outcomes, and the board it reads (null = infer).
+ learn_from_bench: Boolean(r.learn_from_bench), bench_kind: r.bench_kind || null, bench_category: r.bench_category || null,
created_by: r.created_by, created_at: r.created_at,
created_by: r.created_by, created_at: r.created_at,
};
};
}
}
- const routeContent = (r) => sha256({ p: r.primary, f: r.fallbacks, o: r.options, m: r.max_output_tokens, rf: r.response_format, t: r.timeout_ms, a: r.alias_of, s: r.status, c: r.capability || null, x: r.constraints || {}, pi: r.pinned || [] });
+ const routeContent = (r) => sha256({ p: r.primary, f: r.fallbacks, o: r.options, m: r.max_output_tokens, rf: r.response_format, t: r.timeout_ms, a: r.alias_of, s: r.status, c: r.capability || null, x: r.constraints || {}, pi: r.pinned || [], b: Boolean(r.learn_from_bench), bk: r.bench_kind || null, bc: r.bench_category || null });
async function createRouteVersion(key, input, { actor = 'system', trace = null } = {}) {
async function createRouteVersion(key, input, { actor = 'system', trace = null } = {}) {
if (!KEY_RE.test(key || '')) throw new AiError(422, 'ai.invalid', 'route key must be lowercase dotted');
if (!KEY_RE.test(key || '')) throw new AiError(422, 'ai.invalid', 'route key must be lowercase dotted');
const prev = decodeRoute(await latestAny('routes', key));
const prev = decodeRoute(await latestAny('routes', key));
@@ -209,9 +212,15 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
@@ -209,9 +212,15 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
response_format: input.response_format || (prev && prev.response_format) || 'text',
response_format: input.response_format || (prev && prev.response_format) || 'text',
timeout_ms: input.timeout_ms !== undefined ? input.timeout_ms : (prev ? prev.timeout_ms : null),
timeout_ms: input.timeout_ms !== undefined ? input.timeout_ms : (prev ? prev.timeout_ms : null),
alias_of: input.alias_of !== undefined ? input.alias_of : (prev ? prev.alias_of : null),
alias_of: input.alias_of !== undefined ? input.alias_of : (prev ? prev.alias_of : null),
+ // BENCH-11: off unless the route opts in; an edit that omits these keeps the previous version's choice.
+ learn_from_bench: input.learn_from_bench !== undefined ? flag(input.learn_from_bench) : Boolean(prev && prev.learn_from_bench),
+ bench_kind: input.bench_kind !== undefined ? (String(input.bench_kind || '').trim() || null) : (prev ? prev.bench_kind : null),
+ bench_category: input.bench_category !== undefined ? (String(input.bench_category || '').trim() || null) : (prev ? prev.bench_category : null),
status: input.status || 'active',
status: input.status || 'active',
};
};
if (r.capability && !FEATURES.includes(r.capability)) throw new AiError(422, 'ai.invalid', `route capability must be one of ${FEATURES.join(', ')}`);
if (r.capability && !FEATURES.includes(r.capability)) throw new AiError(422, 'ai.invalid', `route capability must be one of ${FEATURES.join(', ')}`);
+ if (r.bench_kind && !KEY_RE.test(r.bench_kind)) throw new AiError(422, 'ai.invalid', 'bench_kind must be a lowercase dotted Bench kind id (e.g. chat.reply)');
+ if (r.bench_category && r.bench_category.length > 100) throw new AiError(422, 'ai.invalid', 'bench_category must be at most 100 characters');
if (!r.alias_of && !r.capability) {
if (!r.alias_of && !r.capability) {
if (!r.primary || !r.primary.provider) throw new AiError(422, 'ai.invalid', 'route needs primary.provider or a capability');
if (!r.primary || !r.primary.provider) throw new AiError(422, 'ai.invalid', 'route needs primary.provider or a capability');
for (const c of [r.primary, ...(r.fallbacks || [])]) {
for (const c of [r.primary, ...(r.fallbacks || [])]) {
@@ -230,11 +239,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
@@ -230,11 +239,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
const version = await nextVersion('routes', key);
const version = await nextVersion('routes', key);
const primaryProvider = r.alias_of ? ((r.primary && r.primary.provider) || 'stub')
const primaryProvider = r.alias_of ? ((r.primary && r.primary.provider) || 'stub')
: (r.primary && r.primary.provider) || ((r.pinned && r.pinned[0] && r.pinned[0].provider) || 'stub');
: (r.primary && r.primary.provider) || ((r.pinned && r.pinned[0] && r.pinned[0].provider) || 'stub');
- await db.prepare(`INSERT INTO routes (key, version, status, primary_provider, primary_model, fallbacks, options, max_output_tokens, response_format, timeout_ms, alias_of, capability, constraints, pinned, created_by, created_at)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
+ await db.prepare(`INSERT INTO routes (key, version, status, primary_provider, primary_model, fallbacks, options, max_output_tokens, response_format, timeout_ms, alias_of, capability, constraints, pinned, learn_from_bench, bench_kind, bench_category, created_by, created_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
.run(key, version, r.status, primaryProvider, (r.primary && r.primary.model) || null,
.run(key, version, r.status, primaryProvider, (r.primary && r.primary.model) || null,
JSON.stringify(r.fallbacks || []), JSON.stringify(r.options || {}), r.max_output_tokens, r.response_format, r.timeout_ms, r.alias_of,
JSON.stringify(r.fallbacks || []), JSON.stringify(r.options || {}), r.max_output_tokens, r.response_format, r.timeout_ms, r.alias_of,
- r.capability || null, JSON.stringify(r.constraints || {}), JSON.stringify(r.pinned || []), actor, now());
+ r.capability || null, JSON.stringify(r.constraints || {}), JSON.stringify(r.pinned || []),
+ r.learn_from_bench ? 1 : 0, r.bench_kind, r.bench_category, actor, now());
await audit(actor, 'route.version', 'route', key, { trace, metadata: { version, previous: prev ? prev.version : null, status: r.status, alias_of: r.alias_of, capability: r.capability || null } });
await audit(actor, 'route.version', 'route', key, { trace, metadata: { version, previous: prev ? prev.version : null, status: r.status, alias_of: r.alias_of, capability: r.capability || null } });
return await getRoute(key, version);
return await getRoute(key, version);
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,184 @@
@@ -0,0 +1,184 @@
+'use strict';
+// BENCH-11 (plan §4.12 and §4.7): Bench's bench.rating.changed reaches AI through its Events consumer,
+// becomes per-kind evidence (rating, interval, games, cost per solved), and only an opted-in route uses it:
+// the cheapest offer statistically tied with the best non-provisional rating. Provisional evidence never
+// decides; no evidence leaves placement's order exactly as it was.
+const assert = require('assert');
+const { signDeliveryHeaders } = require('openvibe-sdk/events');
+const { boot, request, token, suite, seamServer, ALL } = require('./helpers');
+
+const t = suite('bench-learning');
+const admin = token('live', ALL);
+const SECRET = 'bench-events-secret-0123456789abcdef';
+let h; let boards;
+
+const boardRows = [
+ { subject: 'leader-model', elo: 1200, games: 40, ci_lower: 1180, ci_upper: 1220, provisional: false, median_cost_usd: 0.5 },
+ { subject: 'cheap-model', elo: 1180, games: 40, ci_lower: 1160, ci_upper: 1200, provisional: false, median_cost_usd: 0.1 },
+];
+const costRows = boardRows.map((r) => ({ subject: r.subject, cost_per_solved: r.subject === 'leader-model' ? 0.5 : 0.1, median_cost_usd: r.median_cost_usd }));
+
+async function provider(key, model, price) {
+ const r = await h.registry.upsertProvider({
+ key, display_name: key, kind: 'http', status: 'active', base_url: 'http://127.0.0.1:9',
+ auth_mode: 'none', default_model: model, capabilities: ['chat'],
+ }, { actor: 'test' });
+ assert.strictEqual(r.key, key);
+ await h.registry.upsertModel({
+ provider_key: key, model_key: model, type: 'chat',
+ cost: { in_per_mtok: price, out_per_mtok: price },
+ }, { actor: 'test' });
+}
+
+async function rating({ eventId, model, provider, rating: elo, lower, upper, games, cost, provisional = false, kind = 'chat.reply' }) {
+ return await h.bench.applyEvent({
+ event_id: eventId, event_type: 'bench.rating.changed', version: 1, source: 'bench',
+ timestamp: '2026-10-20T10:00:00.000Z',
+ payload: {
+ kind_id: kind, category: 'all', subject: model, elo, games, win_rate: 0.6,
+ ci_lower: lower, ci_upper: upper, provisional, median_cost_usd: null,
+ cost_per_solved: cost, at: '2026-10-20T10:00:00.000Z',
+ ...(provider ? { provider } : {}),
+ },
+ });
+}
+
+async function deliver(event) {
+ const raw = JSON.stringify({ event });
+ const headers = signDeliveryHeaders(raw, SECRET);
+ const res = await fetch(`${h.base}/internal/events`, {
+ method: 'POST', headers: { 'Content-Type': 'application/json', ...headers }, body: raw,
+ });
+ return { status: res.status, body: await res.json().catch(() => ({})) };
+}
+
+t.test('boot with a fake Bench board endpoint and the Events consumer on', async () => {
+ boards = await seamServer((_body, req) => {
+ if (req.url.includes('/api/v1/boards/chat.reply') && req.url.includes('type=elo')) return { board: { kind_id: 'chat.reply', type: 'elo', rows: boardRows } };
+ if (req.url.includes('/api/v1/boards/chat.reply') && req.url.includes('type=cost')) return { board: { kind_id: 'chat.reply', type: 'cost', rows: costRows } };
+ return { status: 404, body: { code: 'bench.kind_not_found' } };
+ });
+ h = await boot({ env: {
+ AI_EVENTS_SECRET: SECRET, AI_EVENTS_SUBSCRIBE: '0', AI_BENCH_URL: boards.url, AI_BENCH_TIMEOUT_MS: '2000',
+ } });
+ await provider('bench-leader', 'leader-model', 50);
+ await provider('bench-cheap', 'cheap-model', 10);
+ await provider('bench-far', 'far-model', 0.01);
+ await provider('bench-prov', 'prov-model', 0.001);
+});
+
+t.test('a signed bench.rating.changed stores evidence, hydrates a sparse event from the board, and is idempotent', async () => {
+ const event = {
+ event_id: 'evt_bench_full_0001', event_type: 'bench.rating.changed', version: 1, source: 'bench',
+ timestamp: '2026-10-20T10:00:00.000Z',
+ payload: {
+ kind_id: 'chat.reply', category: 'all', subject: 'leader-model', elo: 1200, games: 40, win_rate: 0.6,
+ ci_lower: 1180, ci_upper: 1220, median_cost_usd: null, provisional: false, at: '2026-10-20T10:00:00.000Z',
+ },
+ };
+ const first = await deliver(event);
+ assert.strictEqual(first.status, 200, JSON.stringify(first.body));
+ assert.strictEqual(first.body.stored, 1);
+ let row = await h.db.prepare("SELECT * FROM bench_evidence WHERE kind_id = 'chat.reply' AND model = 'leader-model'").get();
+ assert.ok(row, 'the rating is stored');
+ assert.strictEqual(row.provider, 'bench-leader', 'the provider comes from AI registry when Bench rates only the model');
+ assert.strictEqual(Number(row.rating), 1200);
+ assert.strictEqual(Number(row.ci_lower), 1180);
+ assert.strictEqual(Number(row.ci_upper), 1220);
+ assert.strictEqual(Number(row.games), 40);
+ assert.strictEqual(Number(row.cost_per_solved), 0.5, 'the board cost endpoint fills cost per solved');
+
+ // Bench's current outbox announces only subjects; the consumer fills rating and cost from the boards.
+ const sparse = await deliver({
+ event_id: 'evt_bench_sparse_0002', event_type: 'bench.rating.changed', version: 1, source: 'bench',
+ timestamp: '2026-10-20T10:05:00.000Z',
+ payload: { kind_id: 'chat.reply', category: 'all', subjects: ['cheap-model'] },
+ });
+ assert.strictEqual(sparse.status, 200, JSON.stringify(sparse.body));
+ row = await h.db.prepare("SELECT * FROM bench_evidence WHERE kind_id = 'chat.reply' AND model = 'cheap-model'").get();
+ assert.strictEqual(Number(row.rating), 1180);
+ assert.strictEqual(Number(row.cost_per_solved), 0.1);
+
+ const again = await deliver({
+ event_id: 'evt_bench_sparse_0002', event_type: 'bench.rating.changed', version: 1, source: 'bench',
+ timestamp: '2026-10-20T10:05:00.000Z',
+ payload: { kind_id: 'chat.reply', category: 'all', subjects: ['cheap-model'] },
+ });
+ assert.strictEqual(again.status, 200);
+ assert.strictEqual(again.body.duplicate, true, 'the inbox makes a redelivery a no-op');
+ assert.strictEqual((await h.db.prepare("SELECT COUNT(*) AS n FROM bench_evidence WHERE model = 'cheap-model'").get()).n, 1);
+});
+
+t.test('the tie rule picks the cheapest offer whose interval overlaps the leader; provisional evidence never wins', async () => {
+ // A provisional model is the cheapest and highest-rated, but under 10 games: it is not the leader and
+ // is not a candidate. The leader is non-provisional; cheap-model overlaps it; far-model does not.
+ await rating({ eventId: 'evt_bench_leader_0003', model: 'leader-model', rating: 1200, lower: 1180, upper: 1220, games: 40, cost: 0.5 });
+ await rating({ eventId: 'evt_bench_cheap_0004', model: 'cheap-model', rating: 1180, lower: 1160, upper: 1200, games: 40, cost: 0.1 });
+ await rating({ eventId: 'evt_bench_far_0005', model: 'far-model', rating: 900, lower: 850, upper: 950, games: 40, cost: 0.01 });
+ await rating({ eventId: 'evt_bench_prov_0006', model: 'prov-model', rating: 2000, lower: 1900, upper: 2100, games: 9, cost: 0.001, provisional: true });
+
+ const route = {
+ key: 'bench.learning', capability: 'chat', constraints: { objective: 'cheapest' },
+ learn_from_bench: true, bench_kind: 'chat.reply', pinned: [], fallbacks: [],
+ };
+ const out = await h.pool.candidates(route, ['chat'], 'chat');
+ assert.strictEqual(out.order[0].provider, 'bench-cheap', `expected the tied cheap offer first, got ${JSON.stringify(out.order)}`);
+ assert.strictEqual(out.order[0].model, 'cheap-model');
+ assert.strictEqual(out.explain.bench.status, 'selected');
+ assert.strictEqual(out.explain.bench.leader.model, 'leader-model');
+ assert.strictEqual(out.explain.bench.selected.model, 'cheap-model');
+ assert.match(out.explain.bench.reason, /cheapest offer statistically tied/i);
+ assert.ok(!out.explain.bench.evidence.some((x) => x.model === 'prov-model'), 'provisional evidence is not used');
+ assert.ok(!out.explain.bench.tied.some((x) => x.model === 'prov-model'), 'a provisional model is not tied');
+ assert.ok(!out.explain.bench.tied.some((x) => x.model === 'far-model'), 'an interval that does not overlap is not tied');
+});
+
+t.test('only a route with learn_from_bench opts in; no evidence keeps placement order', async () => {
+ const base = { key: 'bench.learning.off', capability: 'chat', constraints: { objective: 'cheapest' }, bench_kind: 'chat.reply', pinned: [], fallbacks: [] };
+ const off = await h.pool.candidates({ ...base, learn_from_bench: false }, ['chat'], 'chat');
+ assert.strictEqual(off.explain.bench.enabled, false);
+ assert.strictEqual(off.explain.bench.status, 'off');
+ assert.strictEqual(off.order[0].provider, 'bench-prov', 'placement stays cheapest: the evidence did not change the order');
+ assert.notStrictEqual(off.order[0].provider, 'bench-cheap');
+
+ const noEvidence = await h.pool.candidates({ ...base, key: 'bench.learning.none', learn_from_bench: true, bench_kind: 'codes.patch' }, ['chat'], 'chat');
+ assert.strictEqual(noEvidence.explain.bench.status, 'no_evidence');
+ assert.deepStrictEqual(noEvidence.order.map((x) => `${x.provider}:${x.model}`), off.order.map((x) => `${x.provider}:${x.model}`), 'no evidence leaves the order unchanged');
+
+ const provisionalOnly = await h.pool.candidates({ ...base, key: 'bench.learning.prov', learn_from_bench: true, bench_kind: 'vision.answer' }, ['chat'], 'chat');
+ await rating({ eventId: 'evt_bench_onlyprov_0007', kind: 'vision.answer', model: 'prov-model', rating: 2000, lower: 1900, upper: 2100, games: 9, cost: 0.001, provisional: true });
+ const afterProvisional = await h.pool.candidates({ ...base, key: 'bench.learning.prov', learn_from_bench: true, bench_kind: 'vision.answer' }, ['chat'], 'chat');
+ assert.strictEqual(afterProvisional.explain.bench.status, 'provisional_or_no_interval');
+ assert.deepStrictEqual(afterProvisional.order.map((x) => `${x.provider}:${x.model}`), provisionalOnly.order.map((x) => `${x.provider}:${x.model}`), 'a provisional-only board changes nothing');
+});
+
+t.test('GET /api/v1/routes/:key shows the evidence and why, and lists the winning offer first', async () => {
+ const created = await request(h.base, 'POST', '/api/v1/routes/bench.api/versions', { tok: admin, body: {
+ capability: 'chat', constraints: { objective: 'cheapest' }, learn_from_bench: true, bench_kind: 'chat.reply',
+ } });
+ assert.strictEqual(created.status, 201, created.text);
+ const r = await request(h.base, 'GET', '/api/v1/routes/bench.api', { tok: admin });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.body.route.learn_from_bench, true);
+ assert.strictEqual(r.body.route.bench_kind, 'chat.reply');
+ assert.strictEqual(r.body.route.bench.status, 'selected');
+ assert.strictEqual(r.body.route.bench.selected.model, 'cheap-model');
+ assert.ok(r.body.route.bench.evidence.length >= 2, 'the evidence it used is visible');
+ assert.strictEqual(r.body.route.offers[0].provider, 'bench-cheap', 'the exposed offer order is the one runs try');
+ assert.match(r.body.route.bench.reason, /statistically tied/i);
+
+ const plain = await request(h.base, 'POST', '/api/v1/routes/bench.plain/versions', { tok: admin, body: {
+ capability: 'chat', constraints: { objective: 'cheapest' },
+ } });
+ assert.strictEqual(plain.status, 201, plain.text);
+ const p = await request(h.base, 'GET', '/api/v1/routes/bench.plain', { tok: admin });
+ assert.strictEqual(p.body.route.learn_from_bench, false, 'off is the default');
+ assert.strictEqual(p.body.route.bench.enabled, false);
+});
+
+t.test('shutdown', async () => {
+ await h.stop();
+ await boards.close();
+});
+
+t.run();