openvibers/openvibe.ai Commits

AI's routing learns from Bench: bench.rating.changed evidence, opt-in placement prefers the cheapest offer tied with the leader (#187)

OP OpenVibers committed 16 days ago · eaa6e42474 · 15 files, +880 −22

Changes

ddiff --git a/.env.example b/.env.example
iindex 4dfe8f7..afc6d44 100644
---- a/.env.example
++++ b/.env.example
@@@ -48,6 +48,18 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
#AI_PERSON_SESSION_TTL_MIN=1440
# Audience a Network user token must carry
#AI_SSO_AUDIENCE=openvibe.network
++# ── Events (the ai.run.* outbox relay and the Bench ratings subscription) ──
++# EVENTS_URL is where AI publishes its own events and ensures its subscription. AI_EVENTS_SECRET is the
++# v2 signing secret Events hands back on that subscription (32+ characters; comma-separated during rotation);
++# without it POST /internal/events is off. AI_EVENTS_SUBSCRIBE=0 skips creating the subscription (an
++# operator/script may create topic bench.rating.changed → http://127.0.0.1:4700/internal/events).
++#EVENTS_URL=http://127.0.0.1:4300
++#AI_EVENTS_SECRET=
++#AI_EVENTS_SUBSCRIBE=on
++# Where a sparse bench.rating.changed is filled from (boards, cost per solved).
++#AI_BENCH_URL=http://127.0.0.1:5230
++#AI_BENCH_TIMEOUT_MS=15000
++
# ── Billing (plan T5 lane F) ──
# One platform.usage-sample@1 reading per accounted run, posted to billing.usage.record with AI's
# service token, so Network must grant client `ai` billing.usage.record on openvibe.billing. Readings
ddiff --git a/README.md b/README.md
iindex b1f410e..e529064 100644
---- a/README.md
++++ b/README.md
@@@ -43,7 +43,8 @@ owns publication truth: Wiki, Blog, News, Live, … decide what to publish.
- PostgreSQL 18 (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through `openvibe-sdk/db`
- OpenVibe.Network (JWKS, service tokens, SSO for the operator console, user modules)
--- OpenVibe.Events (the `ai.run.*` outbox relay; off without `EVENTS_URL`)
++- OpenVibe.Events (the `ai.run.*` outbox relay, and the signed `bench.rating.changed` subscription; off without `EVENTS_URL`)
++- OpenVibe.Bench (`bench.rating.changed` and `GET /api/v1/boards/:kind?type=elo|cost`, for opted-in routes)
- OpenVibe.Media (recordings `media.analyze` reads, over allow-listed https URLs or signed URLs)
- provider APIs configured by the operator (OpenAI-compatible, OpenAI Responses, Anthropic, the HTTP seam), whisper.cpp
and the local model server `openvibe-llm.service` on this host
@@@ -59,8 +60,9 @@ Implemented here (the service manifest's `capabilities`, audience `openvibe.ai`;
registered since openvibe-contracts 0.109.0). A service token's `ns`
claim limits which workflow namespaces it may run; an app token is fixed to its project's.
--Called elsewhere, as the service principal `ai`: `events.event.publish` (Events, the outbox relay)
--and `network.modules.read` / `network.modules.write` for `ai.preferences` and `ai.usage_summary`
++Called elsewhere, as the service principal `ai`: `events.event.publish` (Events, the outbox relay) and
++`events.subscription.manage` (Events, creating AI's own `bench.rating.changed` subscription), and
++`network.modules.read` / `network.modules.write` for `ai.preferences` and `ai.usage_summary`
(Network). Other services grant `ai` their own draft capabilities (for example
`blog.post.create`, `news.story.revise`, `reviews.summary.propose`) to receive drafts.
@@@ -362,6 +364,21 @@ failure; a fallback that answers is recorded on the run (`fallback_used`), in th
(objective, reasons, selected, and every candidate with its estimated cost, latency and excluded
reason).
++### Bench evidence (BENCH-11, plan §4.12 and §4.7)
++
++AI consumes OpenVibe.Bench's `bench.rating.changed` (through the signed `/internal/events` webhook, subscribed
++at boot with `events.subscription.manage`) and keeps a small `bench_evidence` table: per kind and category, the
++model, provider, Elo rating, confidence interval, games and Bench's cost per solved. A sparse event (Bench's
++current outbox names only `subjects`) is filled from `GET /api/v1/boards/:kind?type=elo|cost`.
++
++A route learns from that only when it opts in — `learn_from_bench: true` on the route, default off; `bench_kind`
++names the board (when absent AI infers it from the route key or operation, e.g. `chat.reply`), and
++`bench_category` defaults to `all`. Among the route's eligible offers, placement then prefers the **cheapest**
++offer whose rating interval overlaps the leader's (the highest non-provisional rating): statistically tied with
++the best, so price decides. A provisional rating (under 10 games) never decides. With no evidence the order is
++exactly the one placement produced. `GET /api/v1/routes/:key` carries a `bench` object with the evidence, the
++leader, the tied offers, the selected offer and the reason; the operator console shows the same read-only.
++
`GET /api/v1/routes?operation=generate` lists each route's eligible offers as `{ provider, model }`.
A first-party service holding `ai.run.create` may pin one on `POST /api/v1/generate` (and on
`POST /api/v1/runs` for `ai.generate`) with `"route": { "provider": …, "model": … }`. The run tries
ddiff --git a/docs/capabilities-proposal/service-manifest.ai.json b/docs/capabilities-proposal/service-manifest.ai.json
iindex 8b33614..0264487 100644
---- a/docs/capabilities-proposal/service-manifest.ai.json
++++ b/docs/capabilities-proposal/service-manifest.ai.json
@@@ -36,7 +36,9 @@
"ai.run.failed",
"ai.run.cached"
],
-- "eventsConsumed": [],
++ "eventsConsumed": [
++ "bench.rating.changed"
++ ],
"namespacesOwned": [
"ai.usage_summary"
],
@@@ -87,5 +89,5 @@
"none": "no claims: one process owns ai.db (better-sqlite3, single writer); a run is taken with a compare-and-set on its status (queued → running)"
}
},
-- "notes": "Providers, models, routing profiles, versioned prompt templates and workflows, runs, request logs, citations, scoped cache, quotas and audit. Workflow output is a draft/evidence package attributed to workflow + model + run. ai.run.queued|succeeded|failed|cached have payload contracts (v0.30.0, status planned): AI publishes them once its Events client is wired in; it publishes none today. ai.run.create takes ai.run-request@1 and returns ai.run@1."
++ "notes": "Providers, models, routing profiles, versioned prompt templates and workflows, runs, request logs, citations, scoped cache, quotas and audit. Workflow output is a draft/evidence package attributed to workflow + model + run. ai.run.queued|succeeded|failed|cached have payload contracts; AI publishes them through its outbox. AI consumes bench.rating.changed through a signed Events subscription and keeps per-kind Bench evidence; only routes with learn_from_bench: true consult it in placement. ai.run.create takes ai.run-request@1 and returns ai.run@1."
}
ddiff --git a/migrations/0013_bench_evidence.sql b/migrations/0013_bench_evidence.sql
nnew file mode 100644
iindex 0000000..f3279e7
---- /dev/null
++++ b/migrations/0013_bench_evidence.sql
@@@ -0,0 +1,33 @@
++-- phase: expand
++-- BENCH-11 (plan §4.12 and §4.7): a route can learn from OpenVibe.Bench's measured outcomes.
++-- Bench publishes bench.rating.changed; AI keeps the latest rating and cost-per-solved it has seen per
++-- kind/category/model, and only a route that opts in (routes.learn_from_bench) consults it during placement.
++-- The inbox is openvibe-sdk/events.inboxSchema('bench_event_inbox'): a redelivered event is a no-op.
++CREATE TABLE IF NOT EXISTS bench_evidence (
++ kind_id text COLLATE "C" NOT NULL, -- bench.kind@1's id, e.g. chat.reply
++ category text COLLATE "C" NOT NULL DEFAULT 'all', -- bench.rating@1's category (the board's task class)
++ model text COLLATE "C" NOT NULL, -- bench.rating@1's subject: the model rated
++ provider text COLLATE "C" NOT NULL DEFAULT '', -- AI's provider for that model, '' when unknown/ambiguous
++ rating double precision NOT NULL, -- Elo
++ ci_lower double precision,
++ ci_upper double precision,
++ games bigint NOT NULL DEFAULT 0,
++ provisional bigint NOT NULL DEFAULT 0, -- Bench's flag; games < 10 is provisional too
++ cost_per_solved double precision, -- USD from GET /api/v1/boards/:kind?type=cost
++ updated_at text COLLATE "C" NOT NULL, -- the event's at (ISO-8601)
++ PRIMARY KEY (kind_id, category, model, provider)
++);
++CREATE INDEX IF NOT EXISTS bench_evidence_kind ON bench_evidence (kind_id, category);
++
++CREATE TABLE IF NOT EXISTS bench_event_inbox (
++ consumer text NOT NULL,
++ event_id text NOT NULL,
++ processed_at bigint NOT NULL,
++ PRIMARY KEY (consumer, event_id)
++);
++
++-- The opt-in is per route version, default off. bench_kind names the board to learn from; when it is
++-- null, AI infers it from the route key or its operation (chat.reply for chat/generate/...).
++ALTER TABLE routes ADD COLUMN IF NOT EXISTS learn_from_bench bigint NOT NULL DEFAULT 0;
++ALTER TABLE routes ADD COLUMN IF NOT EXISTS bench_kind text COLLATE "C";
++ALTER TABLE routes ADD COLUMN IF NOT EXISTS bench_category text COLLATE "C";
ddiff --git a/server/api/admin.js b/server/api/admin.js
iindex bb004c3..fcf60dd 100644
---- a/server/api/admin.js
++++ b/server/api/admin.js
@@@ -133,7 +133,20 @@ function adminRouter({ db, registry, pool, quotas, cache, runs, auth, ops = crea
if (!latest) throw new AiError(404, 'ai.not_found', `no route ${req.params.key}`);
const versions = (await registry.listRoutes({ history: true })).filter(x => x.key === req.params.key).map(x => ({ version: x.version, status: x.status, created_by: x.created_by, created_at: x.created_at }));
const operation = req.query.operation ? String(req.query.operation) : null;
-- return { route: { ...latest, offers: await routeOffers(latest, operation, req) }, versions };
++ // BENCH-11: the evidence this route used, the leader it compared against and why the offer order
++ // changed (or stayed). Null while there is no placement to learn from (an inactive route).
++ let bench = null;
++ if (latest.status === 'active' && pool.benchView) {
++ try {
++ const target = latest.alias_of ? await registry.resolveRoute(latest.key) : latest;
++ if (target && target.status === 'active') {
++ bench = await pool.benchView(target, operationFor(target, operation), { requesterType: req.principal.subject.type, requesterId: req.principal.subject.id });
++ }
++ } catch (err) {
++ log.warn(`[ai] bench evidence for route ${latest.key}: ${err.message}`);
++ }
++ }
++ return { route: { ...latest, offers: await routeOffers(latest, operation, req), bench }, versions };
}));
for (const [plural, k] of Object.entries(kinds)) {
ddiff --git a/server/app.js b/server/app.js
iindex ac14fe0..b534247 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -20,7 +20,7 @@ const ovServe = require('openvibe-shared/serve');
const home = require('./home');
const pkg = require('../package.json');
--function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, vault = null }) {
++function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, vault = null, benchEvents = null }) {
const app = express();
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
@@@ -87,6 +87,16 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
// GET /release.json (ADR-016) and POST /release-metrics (open tabs' update reports into /metrics).
release.mount(app, { registry: metrics.registry });
++ // Events → AI (BENCH-11): loopback only, like every service's /internal. Events posts the signed v2
++ // delivery; nginx answers 404 for /internal, and a request carrying forwarding headers is not loopback.
++ app.use('/internal', (req, res, next) => {
++ if (req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.headers['cf-connecting-ip']) {
++ return http.sendProblem(res, 404, 'ai.not_found', { ctx: req.ov });
++ }
++ return next();
++ });
++ if (benchEvents) app.use(benchEvents.router);
++
// Operator actions shared by the admin API and the operator console (one implementation, one audit row).
const ops = createOps({ db, registry, pool, quotas, cache, runs });
app.use(runsRouter({ runs, registry, auth, config, log }));
ddiff --git a/server/bench/events.js b/server/bench/events.js
nnew file mode 100644
iindex 0000000..29be8d8
---- /dev/null
++++ b/server/bench/events.js
@@@ -0,0 +1,122 @@
++'use strict';
++/**
++ * OpenVibe.Events → AI for Bench's ratings. This is AI's consumer side (the ai.run.* outbox in
++ * server/events.js is its producer side).
++ *
++ * At boot it ensures one subscription — topic `bench.rating.changed`, endpoint this service's loopback
++ * `/internal/events` — with AI's own Events token (audience openvibe.events, scope
++ * events.subscription.manage), so Network must grant client `ai` that scope on openvibe.events. Creation
++ * is idempotent (Events answers 409 with the existing subscription) and retried in the background. The
++ * endpoint verifies the v2 signature with AI_EVENTS_SECRET (comma-separated during rotation) and hands the
++ * envelope to server/bench/evidence.js, whose inbox makes a redelivery a no-op.
++ *
++ * Off without EVENTS_URL/AI_EVENTS_URL and AI_EVENTS_SECRET; AI_EVENTS_SUBSCRIBE=0 creates no subscription
++ * (an operator/script may do it), while the endpoint still verifies deliveries when the secret is set.
++ */
++const express = require('express');
++const { http } = require('openvibe-contracts');
++const { serviceAuth } = require('openvibe-contracts');
++const { parseDelivery } = require('openvibe-sdk/events');
++
++const TOPIC = 'bench.rating.changed';
++
++function createBenchEvents({ bench, config, log = console, fetchImpl = globalThis.fetch, clientSecret = process.env.OV_OAUTH_CLIENT_SECRET }) {
++ const router = express.Router();
++ const secrets = (config.events && config.events.secrets) || [];
++
++ function refused(res, status, code, detail) {
++ return http.sendProblem(res, status, code, { detail, ctx: res.req && res.req.ov });
++ }
++
++ /** Apply one verified envelope (also used by tests and an operator replay). */
++ async function apply(event) {
++ return await bench.applyEvent(event);
++ }
++
++ router.post('/internal/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {
++ if (!secrets.length) return refused(res, 503, 'bench.consumer_disabled', 'AI_EVENTS_SECRET is not set');
++ const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body || '');
++ let delivery = null;
++ for (const secret of secrets) {
++ delivery = parseDelivery(raw, req.headers, secret, { requireV2: true });
++ if (delivery) break;
++ }
++ if (!delivery) return refused(res, 401, 'bench.bad_signature', 'the Events delivery signature does not verify');
++ const event = delivery.event;
++ if (!event || typeof event.event_id !== 'string' || !event.event_id) {
++ return refused(res, 400, 'bench.bad_delivery', 'body must be { event: <envelope> }');
++ }
++ try {
++ const out = await apply(event);
++ return res.status(200).json({ event_id: event.event_id, ...out });
++ } catch (err) {
++ log.error(`[bench] event ${event.event_id} (${event.event_type}) failed: ${err.message}`);
++ return refused(res, 500, 'bench.event_failed', 'processing failed; it will be retried');
++ }
++ });
++
++ /**
++ * Ensure the subscription exists, idempotently and in the background. Returns a stop handle, or null
++ * while the consumer is off. Nothing is created when AI_EVENTS_SUBSCRIBE=0 or no signing secret is set.
++ */
++ function startSubscriptions({ port, eventsUrl = config.events && config.events.url, networkUrl = config.networkInternalUrl, clientId = config.console && config.console.clientId } = {}) {
++ if (!config.events || config.events.subscribe === false || !eventsUrl || !secrets.length || !clientSecret) return null;
++ const base = String(eventsUrl).replace(/\/+$/, '');
++ const endpoint = (config.events.endpoint || `http://127.0.0.1:${port}/internal/events`);
++ const tokens = serviceAuth.createTokenClient({
++ tokenUrl: `${networkUrl}/oauth/token`, clientId: clientId || 'ai', clientSecret,
++ audience: 'openvibe.events', scope: 'events.subscription.manage', fetchImpl,
++ });
++ const call = async (method, path, body) => {
++ const res = await fetchImpl(`${base}${path}`, {
++ method,
++ headers: { Accept: 'application/json', ...(body ? { 'Content-Type': 'application/json' } : {}), ...(await tokens.authHeaders()) },
++ body: body ? JSON.stringify(body) : undefined,
++ signal: AbortSignal.timeout(15000),
++ });
++ const json = await res.json().catch(() => ({}));
++ return { status: res.status, ok: res.ok, body: json };
++ };
++ const attempt = async () => {
++ const listed = await call('GET', '/api/v1/subscriptions');
++ if (!listed.ok) throw new Error(`listing subscriptions: ${listed.status}`);
++ const mine = (listed.body.subscriptions || []).filter((s) => s.endpoint === endpoint);
++ if (mine.some((s) => s.topic_pattern === TOPIC)) return;
++ const made = await call('POST', '/api/v1/subscriptions', { topic_pattern: TOPIC, endpoint, secret: secrets[0] });
++ if (!made.ok && made.status !== 409) throw new Error(`subscribing to ${TOPIC}: ${made.status} ${made.body.code || ''}`);
++ if (made.ok) log.log(`[bench] Events subscription created: ${made.body.id} (${TOPIC} → ${endpoint})`);
++ };
++ const delays = [0, 10_000, 60_000, 5 * 60_000, 15 * 60_000];
++ let i = 0;
++ let timer = null;
++ let stopped = false;
++ const run = () => {
++ timer = null;
++ if (stopped) return;
++ attempt().catch((err) => {
++ if (stopped) return;
++ if (++i < delays.length) { timer = setTimeout(run, delays[i]); if (timer.unref) timer.unref(); }
++ else log.warn(`[bench] Events subscription not created: ${err.message}`);
++ });
++ };
++ timer = setTimeout(run, delays[0]);
++ if (timer.unref) timer.unref();
++ return {
++ topic: TOPIC,
++ endpoint,
++ stop() { stopped = true; if (timer) clearTimeout(timer); timer = null; },
++ };
++ }
++
++ async function status() {
++ return {
++ enabled: Boolean(secrets.length),
++ topic: TOPIC,
++ evidence: await bench.list(null).catch(() => []),
++ };
++ }
++
++ return { router, apply, startSubscriptions, status, TOPIC };
++}
++
++module.exports = { createBenchEvents, TOPIC };
ddiff --git a/server/bench/evidence.js b/server/bench/evidence.js
nnew file mode 100644
iindex 0000000..2f6ac66
---- /dev/null
++++ b/server/bench/evidence.js
@@@ -0,0 +1,376 @@
++'use strict';
++/**
++ * BENCH-11 (plan §4.12 and §4.7): OpenVibe.Bench's measured outcomes as routing evidence.
++ *
++ * Bench publishes bench.rating.changed when a board rating moves. AI keeps what it needs from each event
++ * in bench_evidence — kind, category, model, provider, Elo, the confidence interval, games and the board's
++ * cost per solved — and a route that opts in (learn_from_bench: true, default off) may prefer the cheapest
++ * offer whose interval overlaps the leader's. A provisional rating (under 10 games) never decides a route.
++ * With no evidence, select() returns the order it was given: placement is unchanged.
++ *
++ * The event and the board disagree about richness: bench.rating.changed@1 carries one full rating, while
++ * Bench's current outbox announces `{ subjects: [...] }`. Both are accepted; when the event is sparse the
++ * board endpoints (GET /api/v1/boards/:kind?type=elo|cost) fill the gap. A redelivery is a no-op through
++ * the openvibe-sdk inbox (bench_event_inbox).
++ */
++const { createPgInbox } = require('openvibe-sdk/events');
++
++const CONSUMER = 'ai-bench.rating';
++const MIN_GAMES = 10;
++const DEFAULT_CATEGORY = 'all';
++const OPERATION_KIND = {
++ chat: 'chat.reply',
++ complete: 'chat.reply',
++ generate: 'chat.reply',
++ summarize: 'chat.reply',
++ classify: 'chat.reply',
++ extract: 'chat.reply',
++ enrich: 'chat.reply',
++ vision: 'vision.pair',
++ transcribe: 'speech.asr',
++ image: 'vision.answer',
++};
++
++const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
++const flag = (v) => v === true || v === 1 || /^(1|true|yes|on)$/i.test(String(v || ''));
++const text = (v) => (v == null ? '' : String(v).trim());
++const uniq = (xs) => [...new Set(xs.filter(Boolean))];
++
++/** The board category (Bench's task class) a route reads; `all` is Bench's default. */
++function categoryFor(route) {
++ const c = (route && route.constraints) || {};
++ const o = (route && route.options) || {};
++ return text((route && route.bench_category) || c.bench_category || o.bench_category) || DEFAULT_CATEGORY;
++}
++
++/** The boards a route may learn from, most specific first. */
++function kindCandidates(route, operation) {
++ const c = (route && route.constraints) || {};
++ const o = (route && route.options) || {};
++ const explicit = [
++ route && route.bench_kind,
++ c.bench_kind, c.task_class, c.kind_id, c.kind,
++ o.bench_kind, o.task_class, o.kind_id, o.kind,
++ ].map(text);
++ // An explicit board is authoritative: no evidence there means no evidence, never a fallback board.
++ const first = explicit.find(Boolean);
++ if (first) return [first];
++ const out = [];
++ if (route && route.key) out.push(text(route.key));
++ const op = text((route && route.capability) || operation);
++ if (OPERATION_KIND[op]) out.push(OPERATION_KIND[op]);
++ return uniq(out);
++}
++
++function intervalOf(row) {
++ const lower = num(row && row.ci_lower);
++ const upper = num(row && row.ci_upper);
++ if (lower == null || upper == null) return null;
++ return { lower: Math.min(lower, upper), upper: Math.max(lower, upper) };
++}
++
++function provisionalOf(row) {
++ return Boolean(row && row.provisional) || Number(row && row.games) < MIN_GAMES;
++}
++
++function usable(row) {
++ return Boolean(row) && num(row.rating) != null && !provisionalOf(row) && intervalOf(row) != null;
++}
++
++function overlaps(a, b) {
++ const x = intervalOf(a);
++ const y = intervalOf(b);
++ return Boolean(x && y && x.lower <= y.upper && y.lower <= x.upper);
++}
++
++/** An evidence row applies to an offer when the model matches; a known provider must match too. */
++function matchesOffer(row, offer) {
++ if (!row || !offer || !text(offer.model) || text(row.model) !== text(offer.model)) return false;
++ return !text(row.provider) || !text(offer.provider) || text(row.provider) === text(offer.provider);
++}
++
++function viewRow(row) {
++ const rating = num(row.rating);
++ const interval = intervalOf(row);
++ return {
++ kind_id: row.kind_id,
++ category: row.category,
++ model: row.model,
++ provider: text(row.provider) || null,
++ rating,
++ // `elo` is the same number under Bench's own name, so a reader of either shape finds it.
++ elo: rating,
++ interval,
++ ci_lower: interval ? interval.lower : null,
++ ci_upper: interval ? interval.upper : null,
++ games: Number(row.games) || 0,
++ provisional: provisionalOf(row),
++ cost_per_solved: num(row.cost_per_solved),
++ updated_at: row.updated_at || null,
++ };
++}
++
++function viewSelected(row, offer) {
++ return {
++ ...viewRow(row),
++ provider: offer.provider || null,
++ model: offer.model || null,
++ };
++}
++
++function leaderSort(a, b) {
++ return (num(b.rating) - num(a.rating))
++ || (Number(b.games) - Number(a.games))
++ || String(a.model).localeCompare(String(b.model))
++ || String(a.provider || '').localeCompare(String(b.provider || ''));
++}
++
++/**
++ * Pure tie rule: given every evidence row for one kind/category and placement's eligible order, prefer the
++ * cheapest offer statistically tied with the best non-provisional rating. Returns the (possibly reordered)
++ * order and everything a caller needs to explain the decision. No database or clock here.
++ */
++function chooseFromEvidence({ kind_id = null, category = DEFAULT_CATEGORY, evidence = [], order = [] }) {
++ const rows = Array.isArray(evidence) ? evidence : [];
++ const offers = Array.isArray(order) ? order : [];
++ const usableRows = rows.filter(usable).sort(leaderSort);
++ const base = {
++ enabled: true, kind_id, category, evidence: usableRows.map(viewRow),
++ leader: null, tied: [], selected: null, status: 'no_evidence',
++ reason: `no bench evidence for ${kind_id || 'this route'}${category && category !== DEFAULT_CATEGORY ? ` (${category})` : ''}`,
++ };
++ if (!usableRows.length) {
++ if (rows.length) {
++ base.status = 'provisional_or_no_interval';
++ base.reason = `bench evidence for ${kind_id || 'this route'} is provisional (under ${MIN_GAMES} games) or has no rating interval`;
++ }
++ return { order: offers, view: base };
++ }
++ const leader = usableRows[0];
++ const leaderView = viewRow(leader);
++ base.leader = leaderView;
++ const tiedRows = usableRows.filter((row) => overlaps(row, leader));
++
++ // One candidate per eligible offer; a model may appear under several providers, so pick that offer's
++ // best rating first and its cheapest cost as the tie-break.
++ const tied = [];
++ for (let index = 0; index < offers.length; index++) {
++ const offer = offers[index];
++ const matched = tiedRows.filter((row) => matchesOffer(row, offer))
++ .sort((a, b) => leaderSort(a, b) || (num(a.cost_per_solved) ?? Infinity) - (num(b.cost_per_solved) ?? Infinity));
++ if (!matched.length) continue;
++ tied.push({ ...viewSelected(matched[0], offer), index });
++ }
++ base.tied = tied.map(({ index, ...rest }) => rest);
++ if (!tied.length) {
++ base.status = 'no_tied_offer';
++ base.reason = `no eligible offer is statistically tied with ${leader.model} (rating ${leaderView.rating})`;
++ return { order: offers, view: base };
++ }
++ const priced = tied.filter((row) => row.cost_per_solved != null && row.cost_per_solved >= 0)
++ .sort((a, b) => a.cost_per_solved - b.cost_per_solved || a.index - b.index);
++ if (!priced.length) {
++ base.status = 'no_cost';
++ base.reason = `the offers tied with ${leader.model} have no cost per solved yet`;
++ return { order: offers, view: base };
++ }
++ const winner = priced[0];
++ const next = [...offers];
++ if (winner.index > 0) {
++ const [chosen] = next.splice(winner.index, 1);
++ next.unshift(chosen);
++ }
++ base.status = 'selected';
++ base.selected = {
++ provider: winner.provider, model: winner.model,
++ kind_id: winner.kind_id, category: winner.category,
++ rating: winner.rating, interval: winner.interval, games: winner.games,
++ cost_per_solved: winner.cost_per_solved, updated_at: winner.updated_at,
++ };
++ base.reason = `${winner.provider ? `${winner.provider}/` : ''}${winner.model} is the cheapest offer statistically tied with ${leader.model} `
++ + `(rating ${leaderView.rating}, ${leaderView.games} games) at $${winner.cost_per_solved}/solved`;
++ return { order: next, view: base };
++}
++
++function createBenchEvidence({ db, registry = null, config = {}, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console } = {}) {
++ const table = 'bench_evidence';
++ const inbox = createPgInbox(db, { table: 'bench_event_inbox', now: () => clock.now() });
++ const benchUrl = text(config.bench && config.bench.url).replace(/\/+$/, '');
++ const timeoutMs = Math.max(1000, Number(config.bench && config.bench.timeoutMs) || 15000);
++
++ const selectRows = db.prepare(`SELECT * FROM ${table} WHERE kind_id = ? AND category = ? ORDER BY rating DESC, games DESC, model ASC, provider ASC`);
++ async function list(kindId, category = DEFAULT_CATEGORY) {
++ if (!kindId) return [];
++ return await selectRows.all(text(kindId), text(category) || DEFAULT_CATEGORY);
++ }
++
++ async function fetchBoard(kindId, type) {
++ if (!benchUrl) throw new Error('AI_BENCH_URL is not set');
++ const url = `${benchUrl}/api/v1/boards/${encodeURIComponent(kindId)}?type=${encodeURIComponent(type)}`;
++ const res = await fetchImpl(url, { headers: { Accept: 'application/json' }, signal: AbortSignal.timeout(timeoutMs) });
++ if (!res.ok) throw new Error(`bench board ${kindId} ${type}: ${res.status}`);
++ const body = await res.json();
++ const board = body && body.board ? body.board : body;
++ if (!board || !Array.isArray(board.rows)) throw new Error(`bench board ${kindId} ${type}: no rows`);
++ return board.rows;
++ }
++
++ /** A model's provider from AI's registry, when exactly one provider offers it; Bench rates the model, not the seller. */
++ async function providerMap() {
++ const out = new Map();
++ if (!registry || typeof registry.listModels !== 'function') return out;
++ try {
++ for (const m of await registry.listModels()) {
++ if (!out.has(m.model_key)) out.set(m.model_key, new Set());
++ out.get(m.model_key).add(m.provider_key);
++ }
++ } catch (err) {
++ log.warn(`[bench] provider lookup failed: ${err.message}`);
++ }
++ return out;
++ }
++
++ function ratingsFromPayload(payload) {
++ const source = Array.isArray(payload.ratings) ? payload.ratings
++ : Array.isArray(payload.rows) ? payload.rows
++ : Array.isArray(payload.subjects) ? payload.subjects
++ : (payload.subject || payload.model) ? [payload] : [];
++ return source.map((raw) => {
++ const r = raw && typeof raw === 'object' ? raw : { subject: raw };
++ const subject = r.subject && typeof r.subject === 'object' ? r.subject.id : r.subject;
++ return {
++ model: text(r.model || subject || r.id),
++ provider: text(r.provider),
++ rating: num(r.rating != null ? r.rating : r.elo),
++ ci_lower: num(r.ci_lower),
++ ci_upper: num(r.ci_upper),
++ games: num(r.games),
++ provisional: r.provisional === undefined ? null : flag(r.provisional),
++ cost_per_solved: num(r.cost_per_solved),
++ updated_at: r.updated_at || r.at || null,
++ };
++ }).filter((r) => r.model);
++ }
++
++ /** Fill a sparse event from Bench's boards; a full rating still gets the board's cost when it is reachable. */
++ async function hydrate(payload, rows) {
++ const kindId = text(payload.kind_id);
++ const missingRating = rows.some((r) => r.rating == null || r.games == null || r.ci_lower == null || r.ci_upper == null);
++ let eloRows = null;
++ if (missingRating) {
++ eloRows = await fetchBoard(kindId, 'elo'); // a sparse event is useless without it: let Events retry
++ for (const row of rows) {
++ const b = eloRows.find((x) => text(x.subject || x.model) === row.model);
++ if (!b) continue;
++ if (row.rating == null) row.rating = num(b.elo != null ? b.elo : b.rating);
++ if (row.games == null) row.games = num(b.games);
++ if (row.ci_lower == null) row.ci_lower = num(b.ci_lower);
++ if (row.ci_upper == null) row.ci_upper = num(b.ci_upper);
++ if (row.provisional == null) row.provisional = b.provisional === undefined ? null : flag(b.provisional);
++ }
++ }
++ if (rows.some((r) => r.cost_per_solved == null)) {
++ try {
++ const costRows = await fetchBoard(kindId, 'cost');
++ for (const row of rows) {
++ const b = costRows.find((x) => text(x.subject || x.model) === row.model);
++ if (b && row.cost_per_solved == null) row.cost_per_solved = num(b.cost_per_solved);
++ }
++ } catch (err) {
++ // A rating without a price is still evidence; the next rating change fills the cost in.
++ if (rows.some((r) => r.rating == null)) throw err;
++ log.warn(`[bench] cost per solved unavailable for ${kindId}: ${err.message}`);
++ }
++ }
++ return rows;
++ }
++
++ /**
++ * Apply one bench.rating.changed envelope, exactly once. Returns { duplicate, stored }.
++ * Only source `bench` and the rating topic are ours; anything else is acknowledged and ignored.
++ */
++ async function applyEvent(event) {
++ if (!event || event.event_type !== 'bench.rating.changed') return { ignored: true, reason: 'not bench.rating.changed' };
++ if (event.source && event.source !== 'bench') return { ignored: true, reason: `source ${event.source}` };
++ if (!event.event_id) throw new Error('bench.rating.changed has no event_id');
++ const payload = event.payload && typeof event.payload === 'object' ? event.payload : {};
++ const kindId = text(payload.kind_id);
++ if (!kindId) throw new Error('bench.rating.changed has no payload.kind_id');
++ const category = text(payload.category) || DEFAULT_CATEGORY;
++ const eventAt = text(payload.at || event.timestamp);
++ const when = eventAt && !Number.isNaN(new Date(eventAt).getTime()) ? new Date(eventAt).toISOString() : new Date(clock.now()).toISOString();
++ // A redelivery is common (Events retries); do not call Bench's boards for an event already applied.
++ if (await inbox.seen(CONSUMER, String(event.event_id))) return { duplicate: true, stored: 0 };
++ const rows = await hydrate(payload, ratingsFromPayload(payload));
++ const providers = await providerMap();
++
++ const receipt = await inbox.once(CONSUMER, String(event.event_id), async (t) => {
++ let stored = 0;
++ for (const row of rows) {
++ if (row.rating == null) continue; // a board with no rating yet is not evidence
++ const known = providers.get(row.model);
++ const provider = row.provider || (known && known.size === 1 ? [...known][0] : '');
++ const provisional = row.provisional == null ? (Number(row.games) || 0) < MIN_GAMES : row.provisional || (Number(row.games) || 0) < MIN_GAMES;
++ await t.exec(
++ `INSERT INTO ${table} (kind_id, category, model, provider, rating, ci_lower, ci_upper, games, provisional, cost_per_solved, updated_at)
++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
++ ON CONFLICT (kind_id, category, model, provider) DO UPDATE SET
++ rating = excluded.rating, ci_lower = excluded.ci_lower, ci_upper = excluded.ci_upper,
++ games = excluded.games, provisional = excluded.provisional,
++ cost_per_solved = COALESCE(excluded.cost_per_solved, ${table}.cost_per_solved),
++ updated_at = excluded.updated_at
++ WHERE excluded.updated_at >= ${table}.updated_at`,
++ [kindId, category, row.model, provider, row.rating, row.ci_lower, row.ci_upper,
++ Number(row.games) || 0, provisional ? 1 : 0, row.cost_per_solved, when],
++ );
++ stored++;
++ }
++ return { stored };
++ });
++ return receipt.duplicate ? { duplicate: true, stored: 0 } : { duplicate: false, ...(receipt.result || {}) };
++ }
++
++ /** The order a route should try, with the evidence and reason that changed (or left) it. */
++ async function select({ route, order = [], operation = null } = {}) {
++ const enabled = Boolean(route && route.learn_from_bench);
++ const kinds = kindCandidates(route, operation);
++ const category = categoryFor(route);
++ if (!enabled) {
++ return {
++ order,
++ view: {
++ enabled: false, kind_id: kinds[0] || null, category, evidence: [],
++ leader: null, tied: [], selected: null, status: 'off',
++ reason: 'route does not opt in (learn_from_bench is false)',
++ },
++ };
++ }
++ for (const kindId of kinds) {
++ const evidence = await list(kindId, category);
++ if (!evidence.length) continue;
++ const picked = chooseFromEvidence({ kind_id: kindId, category, evidence, order });
++ return { order: picked.order, view: picked.view };
++ }
++ return {
++ order,
++ view: {
++ enabled: true, kind_id: kinds[0] || null, category, evidence: [],
++ leader: null, tied: [], selected: null, status: 'no_evidence',
++ reason: kinds[0]
++ ? `no bench evidence for ${kinds[0]}${category !== DEFAULT_CATEGORY ? ` (${category})` : ''}`
++ : 'no bench kind on this route (set bench_kind or use a route key that names one)',
++ },
++ };
++ }
++
++ async function view({ route, order = [], operation = null } = {}) {
++ return (await select({ route, order, operation })).view;
++ }
++
++ return { applyEvent, select, view, list, kindCandidates, categoryFor, CONSUMER, MIN_GAMES };
++}
++
++module.exports = {
++ createBenchEvidence, chooseFromEvidence, kindCandidates, categoryFor,
++ intervalOf, usable, overlaps, matchesOffer, viewRow, MIN_GAMES, DEFAULT_CATEGORY, CONSUMER,
++};
ddiff --git a/server/config.js b/server/config.js
iindex d4e1591..fa80ebe 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -61,6 +61,22 @@ function load(env = process.env) {
networkPublicKey: env.OV_NETWORK_PUBLIC_KEY ? env.OV_NETWORK_PUBLIC_KEY.replace(/\\n/g, '\n') : null,
audience: 'openvibe.ai',
++ // ── OpenVibe.Events → AI (server/bench/events.js): AI subscribes to bench.rating.changed at boot.
++ // The secret verifies the signed delivery (v2); unset turns POST /internal/events off. EVENTS_URL
++ // is the shared producer/consumer address, AI_EVENTS_URL overrides it. AI_EVENTS_SUBSCRIBE=0 (or no
++ // secret) leaves subscription creation to the operator/script; the endpoint is this service's loopback.
++ events: {
++ url: trimUrl(env.AI_EVENTS_URL || env.EVENTS_URL),
++ secrets: list(env.AI_EVENTS_SECRET || env.AI_BENCH_EVENTS_SECRET || env.EVENTS_SECRET, []),
++ endpoint: String(env.AI_EVENTS_ENDPOINT || '').trim(),
++ subscribe: bool(env.AI_EVENTS_SUBSCRIBE, true),
++ },
++ // Where AI reads board detail a sparse bench.rating.changed leaves out (cost per solved, intervals).
++ bench: {
++ url: trimUrl(env.AI_BENCH_URL || env.OV_BENCH_INTERNAL_URL || env.OV_BENCH_URL || env.BENCH_URL || 'http://127.0.0.1:5230'),
++ timeoutMs: int(env.AI_BENCH_TIMEOUT_MS, 15000),
++ },
++
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
ddiff --git a/server/console/index.js b/server/console/index.js
iindex 14caee1..05442e3 100644
---- a/server/console/index.js
++++ b/server/console/index.js
@@@ -394,6 +394,8 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
primary: { provider: p.key, model: model || p.default_model || null }, fallbacks: previous, capability: null, constraints: {}, pinned: [], alias_of: null,
options: (effective && effective.options) || {}, max_output_tokens: effective ? effective.max_output_tokens : null,
response_format: (effective && effective.response_format) || 'text', timeout_ms: effective ? effective.timeout_ms : null, status: 'active',
++ learn_from_bench: Boolean(effective && effective.learn_from_bench),
++ bench_kind: (effective && effective.bench_kind) || null, bench_category: (effective && effective.bench_category) || null,
}, who(req));
res.redirect(303, `/console/routes/${encodeURIComponent(route.key)}?done=route_used`);
}));
@@@ -420,8 +422,20 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
if (!versions.length) return await send(res, 404, pages.message({ title: 'Not found', text: `No ${k.one} ${req.params.key}.`, back: `/console/${kind}`, ...common(req) }));
const want = Number(req.query.version);
const selected = (Number.isInteger(want) && versions.find((x) => x.version === want)) || versions[0];
++ const current = await registry[k.get](req.params.key, selected.version);
++ // BENCH-11: read-only evidence on a route's page. It changes what runs pick, so it is shown with
++ // the leader, the tied offers and the reason; it is not editable here (bench_kind is an API field).
++ let bench = null;
++ if (kind === 'routes' && current.status === 'active' && pool.benchView) {
++ try {
++ const target = current.alias_of ? await registry.resolveRoute(current.key) : current;
++ if (target && target.status === 'active') bench = await pool.benchView(target, target.capability || 'chat');
++ } catch (err) {
++ log.warn(`[console] bench evidence for route ${current.key}: ${err.message}`);
++ }
++ }
return await send(res, 200, pages.definition({
-- ...common(req), kind, key: req.params.key, versions, selected: await registry[k.get](req.params.key, selected.version),
++ ...common(req), kind, key: req.params.key, versions, selected: current, bench,
inUse: inUseMap(kind, versions).get(req.params.key), canManage: req.staff.caps.includes(CAPS.workflowManage), notice: notice(req),
...(kind === 'routes' ? { canChoose: req.staff.caps.includes(CAPS.providerManage), providers: (await registry.listProviders()).filter((p) => p.status === 'active' && p.kind !== 'whisper' && p.kind !== 'stub'), models: await registry.listModels() } : {}),
}));
ddiff --git a/server/console/views.js b/server/console/views.js
iindex ba33cde..ba437b3 100644
---- a/server/console/views.js
++++ b/server/console/views.js
@@@ -264,6 +264,24 @@ const KINDS = {
};
const target = (c) => (c ? html`<code>${c.provider}</code>${c.model ? html` / <code>${c.model}</code>` : ''}` : '—');
++/** BENCH-11: the evidence an opted-in route used, read-only. Never a form: changing policy is an API edit. */
++function benchBlock(bench) {
++ if (!bench) return '';
++ if (!bench.enabled) return html`<h2>Bench evidence</h2><p class="muted">Learn from Bench is off for this route (<code>learn_from_bench: false</code>); placement keeps its own order.</p>`;
++ const rows = bench.evidence || [];
++ const tied = new Set((bench.tied || []).map((x) => `${x.provider || ''}|${x.model}`));
++ const body = rows.map((r) => html`<tr><td><code>${r.model}</code>${bench.selected && bench.selected.model === r.model ? html` <span class="ok">selected</span>` : tied.has(`${r.provider || ''}|${r.model}`) ? html` <span class="muted">tied</span>` : ''}</td>
++<td>${r.provider ? html`<code>${r.provider}</code>` : html`<span class="muted">any</span>`}</td><td>${r.rating == null ? '—' : n(r.rating)}</td>
++<td>${r.interval ? `${n(r.interval.lower)} – ${n(r.interval.upper)}` : '—'}</td><td>${n(r.games)}${r.provisional ? ' (provisional)' : ''}</td>
++<td>${r.cost_per_solved == null ? '—' : `$${r.cost_per_solved}`}</td><td>${r.updated_at ? when(r.updated_at) : '—'}</td></tr>`);
++ return html`<h2>Bench evidence</h2>
++<p class="muted">Read-only. ${bench.kind_id ? html`Board <code>${bench.kind_id}</code>` : 'No board named'} · category <code>${bench.category}</code>.
++Provisional ratings (under 10 games) never decide a route.</p>
++<p><strong>Why:</strong> ${bench.reason}</p>
++${bench.leader ? html`<p>Leader: <code>${bench.leader.model}</code> at ${n(bench.leader.rating)} (${n(bench.leader.games)} games, interval ${bench.leader.interval ? `${n(bench.leader.interval.lower)} – ${n(bench.leader.interval.upper)}` : '—'}).</p>` : ''}
++${rows.length ? table('Evidence used', ['Model', 'Provider', 'Rating', 'Interval', 'Games', 'Cost / solved', 'Updated'], body, 'no evidence.') : html`<p class="muted">No evidence yet: routing keeps its current order.</p>`}`;
++}
++
function definitions({ staff, csrf, kind, rows, inUse }) {
const k = KINDS[kind];
const link = (key) => html`<a href="/console/${kind}/${enc(key)}"><code>${key}</code></a>`;
@@@ -294,7 +312,7 @@ ${table(`${k.title} (newest version of each)`, head, body, `no ${kind}.`)}`,
});
}
--function definition({ staff, csrf, kind, key, versions, selected, inUse, canManage, notice, error, canChoose = false, providers = [], models = [] }) {
++function definition({ staff, csrf, kind, key, versions, selected, inUse, canManage, notice, error, canChoose = false, providers = [], models = [], bench = null }) {
const k = KINDS[kind];
const v = selected;
let content;
@@@ -302,9 +320,11 @@ function definition({ staff, csrf, kind, key, versions, selected, inUse, canMana
content = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd>
${v.alias_of ? html`<dt>Alias of</dt><dd><a href="/console/routes/${enc(v.alias_of)}"><code>${v.alias_of}</code></a></dd>` : html`<dt>Primary</dt><dd>${target(v.primary)}</dd>
<dt>Fallbacks</dt><dd>${v.fallbacks.length ? html`<ol>${v.fallbacks.map((f) => html`<li>${target(f)}</li>`)}</ol>` : html`<span class="warn">none: an outage of the primary fails every run on this route</span>`}</dd>`}
++<dt>Learn from Bench</dt><dd>${v.learn_from_bench ? html`on${v.bench_kind ? html` · board <code>${v.bench_kind}</code>` : ''}${v.bench_category ? html` · category <code>${v.bench_category}</code>` : ''}` : 'off'}</dd>
<dt>Response format</dt><dd>${v.response_format}</dd><dt>Max output tokens</dt><dd>${v.max_output_tokens != null ? n(v.max_output_tokens) : '—'}</dd>
<dt>Timeout</dt><dd>${v.timeout_ms ? `${n(v.timeout_ms)} ms` : 'provider default'}</dd><dt>Created</dt><dd>${v.created_by || '—'}, ${when(v.created_at)}</dd></dl>
--<h3>Options</h3><pre>${JSON.stringify(v.options || {}, null, 2)}</pre>`;
++<h3>Options</h3><pre>${JSON.stringify(v.options || {}, null, 2)}</pre>
++${benchBlock(bench)}`;
} else if (kind === 'templates') {
content = html`<dl><dt>Status</dt><dd>${badge(v.status)}</dd><dt>Name</dt><dd>${v.name}</dd><dt>Description</dt><dd>${v.description || '—'}</dd>
<dt>Default route</dt><dd>${v.default_route ? html`<a href="/console/routes/${enc(v.default_route)}"><code>${v.default_route}</code></a>` : '—'}</dd>
ddiff --git a/server/index.js b/server/index.js
iindex 0598936..86c9066 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -11,6 +11,8 @@ const { load } = require('./config');
const { openDb } = require('./db');
const { createRegistry } = require('./registry');
const { createProviderPool } = require('./providers');
++const { createBenchEvidence } = require('./bench/evidence');
++const { createBenchEvents } = require('./bench/events');
const { createFetcher } = require('./fetcher');
const { createEngine } = require('./workflows/engine');
const { createCache } = require('./cache');
@@@ -35,7 +37,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
// Provider keys an owner stored from the console (server/vault.js): into this process before any provider is built.
const vault = require('./vault').createVault({ db, config, clock, log });
await vault.load().catch((err) => log.warn(`[Vault] ${err.message}`));
-- const pool = createProviderPool({ db, registry, config, clock, fetchImpl, env, log, credentialFetch, machineModels });
++ // BENCH-11: Bench's ratings as routing evidence, and the opted-in tie rule inside placement.
++ const bench = createBenchEvidence({ db, registry, config, clock, fetchImpl, log });
++ const pool = createProviderPool({ db, registry, config, clock, fetchImpl, env, log, credentialFetch, machineModels, bench });
// T6 step 8: the SaaS tier budgets on openvibe-sdk/govern in Valkey (server/govern.js); off unless AI_GOVERN_TIERS=1.
const tiers = createTiers({ config, valkey, clock, log });
// §2.1.8: the provider free allowance per subject, priced off the pool's own rate cards (server/free-allowance.js);
@@@ -65,7 +69,9 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
// NODE_ENV=test (tests pin a PEM or point the URL at a stub and let the client fetch on first use).
const keys = createNetworkKeys({ config });
const auth = createAuth({ config, log });
-- const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, vault });
++ // The Events consumer side: a signed bench.rating.changed webhook, subscribed at boot (server/bench/events.js).
++ const benchEvents = createBenchEvents({ bench, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });
++ const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, vault, benchEvents });
let jwks = null;
let keyLoaded = Promise.resolve(null);
if (!keys.pinned && config.nodeEnv !== 'test') {
@@@ -85,6 +91,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
housekeeping.unref?.();
let server = null;
++ let benchSubscription = null;
if (listen) userModules.start();
if (listen) {
server = await new Promise((resolve, reject) => {
@@@ -95,12 +102,14 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
server.headersTimeout = 66000;
server.requestTimeout = config.runs.maxWaitMs + 30000;
log.log(`[ai] listening on http://${config.host}:${server.address().port} (${config.nodeEnv})`);
++ benchSubscription = benchEvents.startSubscriptions({ port: server.address().port });
const shared = await registry.getProvider('shared');
log.log(`[ai] shared provider: ${shared.kind} ${shared.status}, credentials ${registry.publicProvider(shared).credentials}; stub fallback ${config.stubFallback ? 'on' : 'off'}`);
}
async function close() {
clearInterval(housekeeping);
++ if (benchSubscription) benchSubscription.stop();
userModules.stop();
if (jwks) jwks.stop();
if (server) {
@@@ -116,7 +125,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
if (!givenDb) await db.close();
}
-- return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, close };
++ return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, bench, benchEvents, close };
}
if (require.main === module) {
ddiff --git a/server/providers/index.js b/server/providers/index.js
iindex 765ad12..0fc3e5c 100644
---- a/server/providers/index.js
++++ b/server/providers/index.js
@@@ -43,7 +43,7 @@ function rateLimited(err) {
return (err && err.status === 429) || (err && err.code === 'provider.rate_limited');
}
--function createProviderPool({ db, registry, config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, credentialFetch = null, machineModels = null }) {
++function createProviderPool({ db, registry, config, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, credentialFetch = null, machineModels = null, bench = null }) {
// A person's own machine as their model (credential provider node, server/machines.js).
const machines = machineModels || require('../machines').createMachineModels({ config, env, fetchImpl });
const cache = new Map(); // key -> { stamp, adapter }
@@@ -339,9 +339,20 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
}
async function candidates(route, features, operation, ctx = {}) {
-- const { order, explain, placement: result = null, ...rest } = route.capability && operation
++ let { order, explain, placement: result = null, ...rest } = route.capability && operation
? await poolCandidates(route, features, operation, ctx)
: await pinnedCandidates(route, features, ctx);
++ // BENCH-11: an opted-in route may put the cheapest offer statistically tied with Bench's leader first.
++ // A route without the opt-in (or without evidence) keeps the order placement just produced.
++ if (bench) {
++ const picked = await bench.select({ route, order, operation });
++ order = picked.order;
++ explain.bench = picked.view;
++ if (picked.view.status === 'selected' && picked.view.selected) {
++ explain.selected = offerId(picked.view.selected.provider, picked.view.selected.model);
++ explain.reasons = [picked.view.reason, ...(explain.reasons || [])];
++ }
++ }
// A scoped route never ends on synthetic text (media.paid falls back to its extractive overview); an image
// has no such fallback, so the dev/test stub PNG (AI_STUB_FALLBACK, off in production) still closes it.
if (config.stubFallback && (!(route.constraints && route.constraints.provider_scope) || operation === 'image')
@@@ -455,10 +466,12 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
* a pinned run is accepted only when it names one of them.
*/
async function offers(route, operation, ctx = {}) {
-- const { explain } = await candidates(route, [operation], operation, ctx);
++ const { order, explain } = await candidates(route, [operation], operation, ctx);
++ const rank = new Map((order || []).map((c, i) => [offerId(c.provider, c.model || null), i]));
const out = [];
const seen = new Set();
-- for (const c of explain.candidates) {
++ // The order placement would try, including a Bench-tie winner when an opted-in route has evidence.
++ for (const c of [...explain.candidates].sort((a, b) => (rank.get(offerId(a.provider, a.model || null)) ?? Infinity) - (rank.get(offerId(b.provider, b.model || null)) ?? Infinity))) {
if (!c || !c.eligible || !c.provider) continue;
const id = offerId(c.provider, c.model || null);
if (seen.has(id)) continue;
@@@ -468,6 +481,13 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
return out;
}
++ /** The Bench evidence a route used and why, for GET /api/v1/routes/:key and the operator console. */
++ async function benchView(route, operation, ctx = {}) {
++ if (!bench) return null;
++ const { explain } = await candidates(route, [operation], operation, ctx);
++ return explain.bench || null;
++ }
++
async function execute(route, operation, req, ctx = {}) {
if (ctx.credential) return await executeWithCredential(ctx.credential, operation, req, ctx);
const features = [operation];
@@@ -579,7 +599,7 @@ function createProviderPool({ db, registry, config, clock = { now: () => Date.no
throw new AiError(503, 'provider.unavailable', 'no provider on this route could answer', { tried, explain });
}
-- return { adapter, health, resetHealth, execute, executeWithCredential, offers, priceFor, costOf, rateCardsFor, providerStates, stats, statFor, candidates };
++ return { adapter, health, resetHealth, execute, executeWithCredential, offers, benchView, priceFor, costOf, rateCardsFor, providerStates, stats, statFor, candidates };
}
module.exports = { createProviderPool };
ddiff --git a/server/registry.js b/server/registry.js
iindex f59d7ec..6c942ec 100644
---- a/server/registry.js
++++ b/server/registry.js
@@@ -18,6 +18,7 @@ const BILLING_PROFILES = ['metered', 'subscription', 'payg', 'free', 'byok'];
const FEATURES = ['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'enrich', 'embed', 'vision', 'json', 'tools', 'transcribe', 'image'];
const LIFECYCLE = ['draft', 'active', 'deprecated', 'archived'];
const KEY_RE = /^[a-z][a-z0-9_-]*(\.[a-z0-9_-]+)*$/;
++const flag = (v) => v === true || v === 1 || /^(1|true|yes|on)$/i.test(String(v == null ? '' : v));
function createRegistry(db, { clock = { now: () => Date.now() }, env = process.env } = {}) {
const now = () => iso(clock.now());
@@@ -189,10 +190,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
fallbacks: parseJson(r.fallbacks, []), options: parseJson(r.options, {}), max_output_tokens: r.max_output_tokens,
response_format: r.response_format, timeout_ms: r.timeout_ms, alias_of: r.alias_of,
capability: r.capability || null, constraints: parseJson(r.constraints, {}), pinned: parseJson(r.pinned, []),
++ // BENCH-11: a route's opt-in to Bench's measured outcomes, and the board it reads (null = infer).
++ learn_from_bench: Boolean(r.learn_from_bench), bench_kind: r.bench_kind || null, bench_category: r.bench_category || null,
created_by: r.created_by, created_at: r.created_at,
};
}
-- const routeContent = (r) => sha256({ p: r.primary, f: r.fallbacks, o: r.options, m: r.max_output_tokens, rf: r.response_format, t: r.timeout_ms, a: r.alias_of, s: r.status, c: r.capability || null, x: r.constraints || {}, pi: r.pinned || [] });
++ const routeContent = (r) => sha256({ p: r.primary, f: r.fallbacks, o: r.options, m: r.max_output_tokens, rf: r.response_format, t: r.timeout_ms, a: r.alias_of, s: r.status, c: r.capability || null, x: r.constraints || {}, pi: r.pinned || [], b: Boolean(r.learn_from_bench), bk: r.bench_kind || null, bc: r.bench_category || null });
async function createRouteVersion(key, input, { actor = 'system', trace = null } = {}) {
if (!KEY_RE.test(key || '')) throw new AiError(422, 'ai.invalid', 'route key must be lowercase dotted');
const prev = decodeRoute(await latestAny('routes', key));
@@@ -209,9 +212,15 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
response_format: input.response_format || (prev && prev.response_format) || 'text',
timeout_ms: input.timeout_ms !== undefined ? input.timeout_ms : (prev ? prev.timeout_ms : null),
alias_of: input.alias_of !== undefined ? input.alias_of : (prev ? prev.alias_of : null),
++ // BENCH-11: off unless the route opts in; an edit that omits these keeps the previous version's choice.
++ learn_from_bench: input.learn_from_bench !== undefined ? flag(input.learn_from_bench) : Boolean(prev && prev.learn_from_bench),
++ bench_kind: input.bench_kind !== undefined ? (String(input.bench_kind || '').trim() || null) : (prev ? prev.bench_kind : null),
++ bench_category: input.bench_category !== undefined ? (String(input.bench_category || '').trim() || null) : (prev ? prev.bench_category : null),
status: input.status || 'active',
};
if (r.capability && !FEATURES.includes(r.capability)) throw new AiError(422, 'ai.invalid', `route capability must be one of ${FEATURES.join(', ')}`);
++ if (r.bench_kind && !KEY_RE.test(r.bench_kind)) throw new AiError(422, 'ai.invalid', 'bench_kind must be a lowercase dotted Bench kind id (e.g. chat.reply)');
++ if (r.bench_category && r.bench_category.length > 100) throw new AiError(422, 'ai.invalid', 'bench_category must be at most 100 characters');
if (!r.alias_of && !r.capability) {
if (!r.primary || !r.primary.provider) throw new AiError(422, 'ai.invalid', 'route needs primary.provider or a capability');
for (const c of [r.primary, ...(r.fallbacks || [])]) {
@@@ -230,11 +239,12 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
const version = await nextVersion('routes', key);
const primaryProvider = r.alias_of ? ((r.primary && r.primary.provider) || 'stub')
: (r.primary && r.primary.provider) || ((r.pinned && r.pinned[0] && r.pinned[0].provider) || 'stub');
-- await db.prepare(`INSERT INTO routes (key, version, status, primary_provider, primary_model, fallbacks, options, max_output_tokens, response_format, timeout_ms, alias_of, capability, constraints, pinned, created_by, created_at)
-- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
++ await db.prepare(`INSERT INTO routes (key, version, status, primary_provider, primary_model, fallbacks, options, max_output_tokens, response_format, timeout_ms, alias_of, capability, constraints, pinned, learn_from_bench, bench_kind, bench_category, created_by, created_at)
++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
.run(key, version, r.status, primaryProvider, (r.primary && r.primary.model) || null,
JSON.stringify(r.fallbacks || []), JSON.stringify(r.options || {}), r.max_output_tokens, r.response_format, r.timeout_ms, r.alias_of,
-- r.capability || null, JSON.stringify(r.constraints || {}), JSON.stringify(r.pinned || []), actor, now());
++ r.capability || null, JSON.stringify(r.constraints || {}), JSON.stringify(r.pinned || []),
++ r.learn_from_bench ? 1 : 0, r.bench_kind, r.bench_category, actor, now());
await audit(actor, 'route.version', 'route', key, { trace, metadata: { version, previous: prev ? prev.version : null, status: r.status, alias_of: r.alias_of, capability: r.capability || null } });
return await getRoute(key, version);
}
ddiff --git a/test/bench-learning.test.js b/test/bench-learning.test.js
nnew file mode 100644
iindex 0000000..ce4be79
---- /dev/null
++++ b/test/bench-learning.test.js
@@@ -0,0 +1,184 @@
++'use strict';
++// BENCH-11 (plan §4.12 and §4.7): Bench's bench.rating.changed reaches AI through its Events consumer,
++// becomes per-kind evidence (rating, interval, games, cost per solved), and only an opted-in route uses it:
++// the cheapest offer statistically tied with the best non-provisional rating. Provisional evidence never
++// decides; no evidence leaves placement's order exactly as it was.
++const assert = require('assert');
++const { signDeliveryHeaders } = require('openvibe-sdk/events');
++const { boot, request, token, suite, seamServer, ALL } = require('./helpers');
++
++const t = suite('bench-learning');
++const admin = token('live', ALL);
++const SECRET = 'bench-events-secret-0123456789abcdef';
++let h; let boards;
++
++const boardRows = [
++ { subject: 'leader-model', elo: 1200, games: 40, ci_lower: 1180, ci_upper: 1220, provisional: false, median_cost_usd: 0.5 },
++ { subject: 'cheap-model', elo: 1180, games: 40, ci_lower: 1160, ci_upper: 1200, provisional: false, median_cost_usd: 0.1 },
++];
++const costRows = boardRows.map((r) => ({ subject: r.subject, cost_per_solved: r.subject === 'leader-model' ? 0.5 : 0.1, median_cost_usd: r.median_cost_usd }));
++
++async function provider(key, model, price) {
++ const r = await h.registry.upsertProvider({
++ key, display_name: key, kind: 'http', status: 'active', base_url: 'http://127.0.0.1:9',
++ auth_mode: 'none', default_model: model, capabilities: ['chat'],
++ }, { actor: 'test' });
++ assert.strictEqual(r.key, key);
++ await h.registry.upsertModel({
++ provider_key: key, model_key: model, type: 'chat',
++ cost: { in_per_mtok: price, out_per_mtok: price },
++ }, { actor: 'test' });
++}
++
++async function rating({ eventId, model, provider, rating: elo, lower, upper, games, cost, provisional = false, kind = 'chat.reply' }) {
++ return await h.bench.applyEvent({
++ event_id: eventId, event_type: 'bench.rating.changed', version: 1, source: 'bench',
++ timestamp: '2026-10-20T10:00:00.000Z',
++ payload: {
++ kind_id: kind, category: 'all', subject: model, elo, games, win_rate: 0.6,
++ ci_lower: lower, ci_upper: upper, provisional, median_cost_usd: null,
++ cost_per_solved: cost, at: '2026-10-20T10:00:00.000Z',
++ ...(provider ? { provider } : {}),
++ },
++ });
++}
++
++async function deliver(event) {
++ const raw = JSON.stringify({ event });
++ const headers = signDeliveryHeaders(raw, SECRET);
++ const res = await fetch(`${h.base}/internal/events`, {
++ method: 'POST', headers: { 'Content-Type': 'application/json', ...headers }, body: raw,
++ });
++ return { status: res.status, body: await res.json().catch(() => ({})) };
++}
++
++t.test('boot with a fake Bench board endpoint and the Events consumer on', async () => {
++ boards = await seamServer((_body, req) => {
++ if (req.url.includes('/api/v1/boards/chat.reply') && req.url.includes('type=elo')) return { board: { kind_id: 'chat.reply', type: 'elo', rows: boardRows } };
++ if (req.url.includes('/api/v1/boards/chat.reply') && req.url.includes('type=cost')) return { board: { kind_id: 'chat.reply', type: 'cost', rows: costRows } };
++ return { status: 404, body: { code: 'bench.kind_not_found' } };
++ });
++ h = await boot({ env: {
++ AI_EVENTS_SECRET: SECRET, AI_EVENTS_SUBSCRIBE: '0', AI_BENCH_URL: boards.url, AI_BENCH_TIMEOUT_MS: '2000',
++ } });
++ await provider('bench-leader', 'leader-model', 50);
++ await provider('bench-cheap', 'cheap-model', 10);
++ await provider('bench-far', 'far-model', 0.01);
++ await provider('bench-prov', 'prov-model', 0.001);
++});
++
++t.test('a signed bench.rating.changed stores evidence, hydrates a sparse event from the board, and is idempotent', async () => {
++ const event = {
++ event_id: 'evt_bench_full_0001', event_type: 'bench.rating.changed', version: 1, source: 'bench',
++ timestamp: '2026-10-20T10:00:00.000Z',
++ payload: {
++ kind_id: 'chat.reply', category: 'all', subject: 'leader-model', elo: 1200, games: 40, win_rate: 0.6,
++ ci_lower: 1180, ci_upper: 1220, median_cost_usd: null, provisional: false, at: '2026-10-20T10:00:00.000Z',
++ },
++ };
++ const first = await deliver(event);
++ assert.strictEqual(first.status, 200, JSON.stringify(first.body));
++ assert.strictEqual(first.body.stored, 1);
++ let row = await h.db.prepare("SELECT * FROM bench_evidence WHERE kind_id = 'chat.reply' AND model = 'leader-model'").get();
++ assert.ok(row, 'the rating is stored');
++ assert.strictEqual(row.provider, 'bench-leader', 'the provider comes from AI registry when Bench rates only the model');
++ assert.strictEqual(Number(row.rating), 1200);
++ assert.strictEqual(Number(row.ci_lower), 1180);
++ assert.strictEqual(Number(row.ci_upper), 1220);
++ assert.strictEqual(Number(row.games), 40);
++ assert.strictEqual(Number(row.cost_per_solved), 0.5, 'the board cost endpoint fills cost per solved');
++
++ // Bench's current outbox announces only subjects; the consumer fills rating and cost from the boards.
++ const sparse = await deliver({
++ event_id: 'evt_bench_sparse_0002', event_type: 'bench.rating.changed', version: 1, source: 'bench',
++ timestamp: '2026-10-20T10:05:00.000Z',
++ payload: { kind_id: 'chat.reply', category: 'all', subjects: ['cheap-model'] },
++ });
++ assert.strictEqual(sparse.status, 200, JSON.stringify(sparse.body));
++ row = await h.db.prepare("SELECT * FROM bench_evidence WHERE kind_id = 'chat.reply' AND model = 'cheap-model'").get();
++ assert.strictEqual(Number(row.rating), 1180);
++ assert.strictEqual(Number(row.cost_per_solved), 0.1);
++
++ const again = await deliver({
++ event_id: 'evt_bench_sparse_0002', event_type: 'bench.rating.changed', version: 1, source: 'bench',
++ timestamp: '2026-10-20T10:05:00.000Z',
++ payload: { kind_id: 'chat.reply', category: 'all', subjects: ['cheap-model'] },
++ });
++ assert.strictEqual(again.status, 200);
++ assert.strictEqual(again.body.duplicate, true, 'the inbox makes a redelivery a no-op');
++ assert.strictEqual((await h.db.prepare("SELECT COUNT(*) AS n FROM bench_evidence WHERE model = 'cheap-model'").get()).n, 1);
++});
++
++t.test('the tie rule picks the cheapest offer whose interval overlaps the leader; provisional evidence never wins', async () => {
++ // A provisional model is the cheapest and highest-rated, but under 10 games: it is not the leader and
++ // is not a candidate. The leader is non-provisional; cheap-model overlaps it; far-model does not.
++ await rating({ eventId: 'evt_bench_leader_0003', model: 'leader-model', rating: 1200, lower: 1180, upper: 1220, games: 40, cost: 0.5 });
++ await rating({ eventId: 'evt_bench_cheap_0004', model: 'cheap-model', rating: 1180, lower: 1160, upper: 1200, games: 40, cost: 0.1 });
++ await rating({ eventId: 'evt_bench_far_0005', model: 'far-model', rating: 900, lower: 850, upper: 950, games: 40, cost: 0.01 });
++ await rating({ eventId: 'evt_bench_prov_0006', model: 'prov-model', rating: 2000, lower: 1900, upper: 2100, games: 9, cost: 0.001, provisional: true });
++
++ const route = {
++ key: 'bench.learning', capability: 'chat', constraints: { objective: 'cheapest' },
++ learn_from_bench: true, bench_kind: 'chat.reply', pinned: [], fallbacks: [],
++ };
++ const out = await h.pool.candidates(route, ['chat'], 'chat');
++ assert.strictEqual(out.order[0].provider, 'bench-cheap', `expected the tied cheap offer first, got ${JSON.stringify(out.order)}`);
++ assert.strictEqual(out.order[0].model, 'cheap-model');
++ assert.strictEqual(out.explain.bench.status, 'selected');
++ assert.strictEqual(out.explain.bench.leader.model, 'leader-model');
++ assert.strictEqual(out.explain.bench.selected.model, 'cheap-model');
++ assert.match(out.explain.bench.reason, /cheapest offer statistically tied/i);
++ assert.ok(!out.explain.bench.evidence.some((x) => x.model === 'prov-model'), 'provisional evidence is not used');
++ assert.ok(!out.explain.bench.tied.some((x) => x.model === 'prov-model'), 'a provisional model is not tied');
++ assert.ok(!out.explain.bench.tied.some((x) => x.model === 'far-model'), 'an interval that does not overlap is not tied');
++});
++
++t.test('only a route with learn_from_bench opts in; no evidence keeps placement order', async () => {
++ const base = { key: 'bench.learning.off', capability: 'chat', constraints: { objective: 'cheapest' }, bench_kind: 'chat.reply', pinned: [], fallbacks: [] };
++ const off = await h.pool.candidates({ ...base, learn_from_bench: false }, ['chat'], 'chat');
++ assert.strictEqual(off.explain.bench.enabled, false);
++ assert.strictEqual(off.explain.bench.status, 'off');
++ assert.strictEqual(off.order[0].provider, 'bench-prov', 'placement stays cheapest: the evidence did not change the order');
++ assert.notStrictEqual(off.order[0].provider, 'bench-cheap');
++
++ const noEvidence = await h.pool.candidates({ ...base, key: 'bench.learning.none', learn_from_bench: true, bench_kind: 'codes.patch' }, ['chat'], 'chat');
++ assert.strictEqual(noEvidence.explain.bench.status, 'no_evidence');
++ assert.deepStrictEqual(noEvidence.order.map((x) => `${x.provider}:${x.model}`), off.order.map((x) => `${x.provider}:${x.model}`), 'no evidence leaves the order unchanged');
++
++ const provisionalOnly = await h.pool.candidates({ ...base, key: 'bench.learning.prov', learn_from_bench: true, bench_kind: 'vision.answer' }, ['chat'], 'chat');
++ await rating({ eventId: 'evt_bench_onlyprov_0007', kind: 'vision.answer', model: 'prov-model', rating: 2000, lower: 1900, upper: 2100, games: 9, cost: 0.001, provisional: true });
++ const afterProvisional = await h.pool.candidates({ ...base, key: 'bench.learning.prov', learn_from_bench: true, bench_kind: 'vision.answer' }, ['chat'], 'chat');
++ assert.strictEqual(afterProvisional.explain.bench.status, 'provisional_or_no_interval');
++ assert.deepStrictEqual(afterProvisional.order.map((x) => `${x.provider}:${x.model}`), provisionalOnly.order.map((x) => `${x.provider}:${x.model}`), 'a provisional-only board changes nothing');
++});
++
++t.test('GET /api/v1/routes/:key shows the evidence and why, and lists the winning offer first', async () => {
++ const created = await request(h.base, 'POST', '/api/v1/routes/bench.api/versions', { tok: admin, body: {
++ capability: 'chat', constraints: { objective: 'cheapest' }, learn_from_bench: true, bench_kind: 'chat.reply',
++ } });
++ assert.strictEqual(created.status, 201, created.text);
++ const r = await request(h.base, 'GET', '/api/v1/routes/bench.api', { tok: admin });
++ assert.strictEqual(r.status, 200, r.text);
++ assert.strictEqual(r.body.route.learn_from_bench, true);
++ assert.strictEqual(r.body.route.bench_kind, 'chat.reply');
++ assert.strictEqual(r.body.route.bench.status, 'selected');
++ assert.strictEqual(r.body.route.bench.selected.model, 'cheap-model');
++ assert.ok(r.body.route.bench.evidence.length >= 2, 'the evidence it used is visible');
++ assert.strictEqual(r.body.route.offers[0].provider, 'bench-cheap', 'the exposed offer order is the one runs try');
++ assert.match(r.body.route.bench.reason, /statistically tied/i);
++
++ const plain = await request(h.base, 'POST', '/api/v1/routes/bench.plain/versions', { tok: admin, body: {
++ capability: 'chat', constraints: { objective: 'cheapest' },
++ } });
++ assert.strictEqual(plain.status, 201, plain.text);
++ const p = await request(h.base, 'GET', '/api/v1/routes/bench.plain', { tok: admin });
++ assert.strictEqual(p.body.route.learn_from_bench, false, 'off is the default');
++ assert.strictEqual(p.body.route.bench.enabled, false);
++});
++
++t.test('shutdown', async () => {
++ await h.stop();
++ await boards.close();
++});
++
++t.run();