openvibers/openvibe.bench Commits
AL8: score whole tickets over repeated trials (codes.ticket) (#184)
Changes
@@@ -2,6 +2,16 @@## Unreleased++- **Whole tickets, the gate (AL8).** A new kind, `codes.ticket`, scores a whole ticket — a goal and a typed check++ in `agent.goal@1`'s shape, with a token/turn/minute budget and a no-progress limit — over 3 to 5 trials per model++ and effort. A run is one trial; a task's score is 100 when the check exited 0 and 0 when it did not, and the++ check's test patch is applied only after the worker stops, so the merged pull request's own test grades the work.++ `GET /api/v1/rankings?benchmark=…&metric=tickets&effort=…` (also `pass_k`, `pass@k`) reports pass@1, pass^k (every++ one of k trials passed) and cost per solved by model and effort, with the per-ticket rows. The first suite++ (`fixtures/codes.ticket/openvibe-merged-2026-11.json`) holds twelve small tickets from merged OpenVibe pull++ requests across Bench, Codes, AI, Actor, Exchange, Report and Network; each goal is the change in words and each++ check is the pull request's own test. `scripts/ticket-suite.js` prints the plan and runs only with `--run`; the++ default models are `deepseek-flash` and `local`, and nothing is scheduled.- **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, thescore of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each,no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the@@@ -38,9 +38,38 @@ run under it, with no person id. Nothing private or unlisted is ever read there.| `POST /api/v1/runs` | a person, or `bench.run.publish` | Publish a run. A service token with `bench.run.publish` makes a **verified** run; a person's token makes a **submitted** one, badged and out of the default rankings. || `GET /api/v1/runs/:id` | anyone | One run with its per-task results. `run.document` is the published `bench.run@1` document. || `GET /api/v1/compare?runs=a,b,c` | anyone | Up to four runs side by side, per task. |--| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. |++| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency\|tickets&harness=&since=&effort=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. `metric=tickets` (also `pass_k`, `pass@k`) is the AL8 gate: pass@1, pass^k and cost per solved by model and effort (`effort` filters one). || `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |++## Whole tickets (AL8, the gate)++++`codes.ticket` scores a whole ticket: a goal and its check, run over 3 to 5 trials on each model and effort. A run++is one trial; each task is one ticket, and its score is 100 when the check passed and 0 when it did not. The++results API answers the gate's table:++++```++GET /api/v1/rankings?benchmark=bmk_…@1&metric=tickets&effort=high++{++ "rankings": {++ "metric": "tickets",++ "entries": [{++ "model": "deepseek-flash", "effort": "high", "k": 3, "trials": 3, "task_trials": 9, "tickets": 3,++ "pass_at_1": 0.888889, // share of all (ticket, trial) pairs that passed++ "pass_at_k": 0.666667, // share of tickets whose every one of k trials passed++ "cost_usd": 0.31, "solved": 3,++ "cost_per_solved": 0.103333, // every trial's cost ÷ the tickets solved at least once++ "insufficient_trials": false,++ "per_ticket": [{ "task_id": "…", "trials": 3, "passes": 3, "pass_at_1": 1, "pass_at_k": true, "cost_usd": 0.1 }]++ }]++ }++}++```++++A group with fewer than three trials sets `insufficient_trials: true`; it is not a gate result. The suite file++(`bench.codes-ticket-suite@1`) and the runner are in `fixtures/codes.ticket/` and `scripts/ticket-suite.js`. A suite++runs only when a person passes `--run`; the default models are `deepseek-flash` and `local`, and nothing is++scheduled.++## Voting without signing inA person who has not signed in may still judge a pairwise round (`human_pairwise` kinds only). There are two ways in,nnew file mode 100644@@@ -0,0 +1,326 @@++{++ "format": "bench.codes-ticket-suite@1",++ "suite": "openvibe-merged-2026-11",++ "title": "Twelve merged OpenVibe tickets",++ "description": "The first AL8 gate suite: twelve small tickets from merged OpenVibe pull requests across Bench, Codes, AI, Actor, Exchange, Report and Network. Each goal is the merged change in words; each check is the pull request's own test, applied only after the worker stops. Base is the merge commit's parent, so the test fails before the change and passes after it.",++ "kind": "codes.ticket",++ "models": [++ "deepseek-flash",++ "local"++ ],++ "trials": 3,++ "tickets": [++ {++ "id": "bench-158-navbar-token",++ "goal": "The shared navbar can open Messages in Dock Chats in place: the signed-in person's verified Network access token is available through a same-origin, no-store GET /auth/navbar-token route, while /auth/me stays token-free.",++ "check": {++ "kind": "command",++ "value": "node test/sso.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 30,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Bench",++ "ref": "b7ca03eea0f20dc933da1b9c8c8035c2d8f2f52c",++ "check_patch": "diff --git a/test/sso.test.js b/test/sso.test.js\nindex 44dd1bf..d9ece5a 100644\n--- a/test/sso.test.js\n+++ b/test/sso.test.js\n@@ -53,6 +53,11 @@ const cookieJar = (res) => res.headers.getSetCookie().map((c) => c.split(';')[0]\n const me = await t.get('/auth/me', { cookie: `ov_token=${at}` });\n assert.strictEqual(me.json().user.username, 'kim');\n assert.ok(!me.text.includes(at), '/auth/me never returns the token');\n+ const nav = await t.get('/auth/navbar-token', { cookie: `ov_token=${at}` });\n+ assert.strictEqual(nav.status, 200);\n+ assert.strictEqual(nav.json().token, at, 'the Dock receives the verified Network token');\n+ assert.strictEqual(nav.headers.get('cache-control'), 'private, no-store');\n+ assert.strictEqual((await t.get('/auth/navbar-token')).status, 401, 'a guest receives no token');\n });\n \n await check('the old cookie is still the session, so nobody is signed out by the move', async () => {\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Bench",++ "pull_request": 158,++ "merge_commit": "3b2a91896b5d384e3f685e29e427b324ae93db5f",++ "merged_at": "2026-11-09T06:55:42Z"++ }++ },++ {++ "id": "codes-351-prompt-prefix",++ "goal": "Two task runs through the coding agent's tool loop send the same model, system message and tool schemas, in a fixed order, so a provider caches one prompt prefix; only the user message differs between tasks.",++ "check": {++ "kind": "command",++ "value": "node test/harness-prompt-prefix.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Codes",++ "ref": "19721715ed45fd669ca6e7eb5d9737c2fa9dc8f8",++ "check_patch": "diff --git a/test/harness-prompt-prefix.test.js b/test/harness-prompt-prefix.test.js\nnew file mode 100644\nindex 0000000..bb0ba2d\n--- /dev/null\n+++ b/test/harness-prompt-prefix.test.js\n@@ -0,0 +1,73 @@\n+'use strict';\n+/**\n+ * The prompt prefix is byte-identical between two different tasks (stage S3, cache-stable prompts): the system message\n+ * and the tool schemas never carry the task, tools come in a fixed name order, and only the user message differs.\n+ */\n+const assert = require('assert');\n+const fs = require('fs');\n+const os = require('os');\n+const path = require('path');\n+const http = require('http');\n+const { createHarness } = require('../harness');\n+const { createStore } = require('../harness/sessions');\n+const { SYSTEM } = require('../harness/adapters/api');\n+const { check, done } = require('./helpers/boot');\n+\n+/** A chat-completions server that records each request and always answers with a plain reply. */\n+async function mockApi() {\n+ const requests = [];\n+ const server = http.createServer((req, res) => {\n+ let body = '';\n+ req.on('data', (d) => { body += d; });\n+ req.on('end', () => {\n+ requests.push(JSON.parse(body || '{}'));\n+ res.writeHead(200, { 'content-type': 'application/json' });\n+ res.end(JSON.stringify({ choices: [{ message: { role: 'assistant', content: 'done' } }], usage: { prompt_tokens: 1, completion_tokens: 1 } }));\n+ });\n+ });\n+ await new Promise((r) => server.listen(0, '127.0.0.1', r));\n+ return { url: `http://127.0.0.1:${server.address().port}`, requests, close: () => new Promise((r) => server.close(r)) };\n+}\n+\n+const collect = async (run) => { const out = []; for await (const e of run) out.push(e); return out; };\n+/** The stable prefix of one request as it is serialized: the model, the system message and the tools, in that order. */\n+const prefixOf = (body) => JSON.stringify({ model: body.model, messages: [body.messages[0]], tools: body.tools, tool_choice: body.tool_choice });\n+\n+(async () => {\n+ const api = await mockApi();\n+ try {\n+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codes-prefix-'));\n+ const work = path.join(dir, 'work');\n+ fs.mkdirSync(work, { recursive: true });\n+ // An empty environment: the harness reads its base url and model from here, never from the machine's own.\n+ const env = { PATH: '/nonexistent', OPENVIBE_CODES_HOME: path.join(dir, 'home'), OPENVIBE_CODES_BASE_URL: api.url, OPENVIBE_CODES_MODEL: 'tiny', OPENVIBE_CODES_API_KEY: 'k-123' };\n+ const h = createHarness({ env, store: createStore({ env }) });\n+\n+ await check('the system message never carries the task or a date', async () => {\n+ assert.ok(SYSTEM.trim().length > 100);\n+ assert.ok(!/\\d{4}-\\d{2}-\\d{2}/.test(SYSTEM), 'no date in the stable system message');\n+ });\n+\n+ await check('two tasks send the same model, system message and tools; only the prompt differs', async () => {\n+ await collect(h.run({ prompt: 'rename name to count', agent: 'openai-compatible', cwd: work, permission: 'edit', handoff: false }));\n+ await collect(h.run({ prompt: 'a much longer second task about the same repository', agent: 'openai-compatible', cwd: work, permission: 'edit', handoff: false }));\n+ assert.strictEqual(api.requests.length, 2);\n+ const [a, b] = api.requests;\n+ assert.strictEqual(prefixOf(a), prefixOf(b), 'the serialized prefix is byte-identical');\n+ assert.strictEqual(a.messages[0].role, 'system');\n+ assert.strictEqual(a.messages[1].role, 'user');\n+ assert.strictEqual(a.messages[1].content, 'rename name to count');\n+ assert.strictEqual(b.messages[1].content, 'a much longer second task about the same repository');\n+ });\n+\n+ await check('tool schemas serialize in a fixed name order', async () => {\n+ const names = api.requests[0].tools.map((t) => t.function.name);\n+ assert.deepStrictEqual(names, [...names].sort(), 'tools are offered in name order');\n+ assert.deepStrictEqual(names, ['edit_file', 'grep', 'list_dir', 'read_file', 'write_file']);\n+ assert.strictEqual(JSON.stringify(api.requests[0].tools), JSON.stringify(api.requests[1].tools));\n+ });\n+ } finally {\n+ await api.close();\n+ done();\n+ }\n+})();\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Codes",++ "pull_request": 351,++ "merge_commit": "52381fb2bf6267b844536413446a00d6f4a4aa43",++ "merged_at": "2026-11-09T21:59:35Z"++ }++ },++ {++ "id": "codes-314-who-leads",++ "goal": "The composer's \"Who leads\" choice names the person's own saved setting and is the option selected; a person who never chose is not pinned to Opus when the Settings page is saved untouched.",++ "check": {++ "kind": "command",++ "value": "node test/conductor.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Codes",++ "ref": "c7320335b35d2998134ea7bc131aa921331cd5d9",++ "check_patch": "diff --git a/test/conductor.test.js b/test/conductor.test.js\nindex d032e17..79ee775 100644\n--- a/test/conductor.test.js\n+++ b/test/conductor.test.js\n@@ -196,6 +196,12 @@ const DEVICE = 'dev_01J8Z3Q4R5S6T7V8W9X0Y1Z2N1';\n const res = await get('/conductor');\n assert.strictEqual(res.status, 200);\n assert.match(res.text, /<select name=\"who\" data-c4-model>/, 'who leads is a list');\n+ // The first option is the person's own setting, said by name, and it is the one selected: the composer never\n+ // claims a default the person changed, and no other leader is preselected.\n+ assert.match(res.text, /<select name=\"who\" data-c4-model><option value=\"\">Your setting: [^<]+<\\/option>/, 'the first choice names the setting');\n+ const whoList = res.text.split('<select name=\"who\" data-c4-model>')[1].split('</select>')[0];\n+ assert.ok(!/ selected/.test(whoList), 'nothing else is preselected');\n+ assert.ok(!res.text.includes(\"Today's choice\"), 'no \"Today\\'s choice\" wording');\n for (const label of ['Claude Code · Opus', 'Claude Code · Sonnet', 'Codex', 'OpenCode', 'DeepSeek']) assert.ok(res.text.includes(label), `${label} is offered`);\n assert.match(res.text, /data-c4-folder-pick/, 'the recent folders are a choice beside the typed path');\n assert.ok(res.text.includes(`data-machine=\"${NODE}\"`), 'this conductor’s folder is one, on its machine');\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Codes",++ "pull_request": 314,++ "merge_commit": "092d1952b27a5dfe3cb0edc22624f5cf6097e23e",++ "merged_at": "2026-11-07T13:06:20Z"++ }++ },++ {++ "id": "ai-387-space-components",++ "goal": "The space.app.build template accepts an optional components input (up to 20,000 characters) and, when present, puts it in the system prompt as the design.component@1 interface vocabulary with the rule that every major part carries data-ov-component; without it the prompt is exactly as before.",++ "check": {++ "kind": "command",++ "value": "node test/space-workflows.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.AI",++ "ref": "13ffb0f203b182c72960a3af8f5c37fefa53af0a",++ "check_patch": "diff --git a/test/space-workflows.test.js b/test/space-workflows.test.js\nindex eea1494..a4509da 100644\n--- a/test/space-workflows.test.js\n+++ b/test/space-workflows.test.js\n@@ -23,6 +23,13 @@ t.test('the builder\\'s prompt: the SDK reference, and the current files when cha\n assert.match(sys, /OV\\.fn\\(name, body\\)/);\n assert.match(sys, /never request\\.user or request\\.ov/, 'the visitor is on ov, not on the request');\n assert.ok(!/There is no server code/.test(sys));\n+ assert.match(sys, /design\\.component@1/, 'the registry is named as the interface vocabulary');\n+ const withComponents = render(sys, { components: 'sheet — A modal task surface. Web: kit.sheet()' });\n+ assert.match(withComponents, /mark each major part with data-ov-component/);\n+ assert.match(withComponents, /sheet — A modal task surface/);\n+ assert.ok(!/\\{\\{#components\\}\\}/.test(withComponents), 'the section is rendered away');\n+ const prepWithRegistry = PREPARE['space.app.build']({ prompt: 'a pizza poll', components: 'sheet — A modal task surface.' });\n+ assert.strictEqual(prepWithRegistry.vars.components, 'sheet — A modal task surface.');\n const prep = PREPARE['space.app.build']({ prompt: 'a poll about pizza' });\n assert.ok(prep.jsonSchema && prep.jsonSchema.properties.files && !/maxLength|maxItems/.test(JSON.stringify(prep.jsonSchema)), 'the model answers in the uncapped, non-strict schema');\n const fresh = render(tpl('space.app.build').user_prompt, prep.vars);\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.AI",++ "pull_request": 387,++ "merge_commit": "5fd137763be08137d2c7e928dcce50ad1608c2f0",++ "merged_at": "2026-11-09T21:48:50Z"++ }++ },++ {++ "id": "actor-390-prompt-prefix",++ "goal": "Two assemblies of the agent's system prompt with different dates and plans share the whole rules prefix byte for byte, and the tool definitions serialize identically, because today's date, the plan, the ledger and the demoted-tools note come after the rules.",++ "check": {++ "kind": "command",++ "value": "node test/runtime-prompt-prefix.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Actor",++ "ref": "beb69b131c2dd20e30a5c04b1bfaf245408f14d7",++ "check_patch": "diff --git a/test/runtime-prompt-prefix.test.js b/test/runtime-prompt-prefix.test.js\nnew file mode 100644\nindex 0000000..2694c55\n--- /dev/null\n+++ b/test/runtime-prompt-prefix.test.js\n@@ -0,0 +1,55 @@\n+'use strict';\n+/**\n+ * The prompt prefix stays byte-identical between tasks (stage S3, cache-stable prompts).\n+ *\n+ * - the system prompt keeps its rules first and today's date, the plan, the ledger and a demoted-tools note last,\n+ * so two tasks share the whole stable prefix and a provider's cache breakpoint never covers a date;\n+ * - the tool definitions serialize the same way on every assembly, so a cache is not lost to tool order.\n+ */\n+const fs = require('fs');\n+const os = require('os');\n+const path = require('path');\n+const assert = require('assert');\n+const { SYSTEM } = require('../server/runtime/loop');\n+const { createRegistry } = require('../server/runtime/tools');\n+\n+const results = [];\n+const ok = async (name, fn) => { try { await fn(); console.log(` ✓ ${name}`); } catch (e) { results.push(e); console.log(` ✗ ${name}\\n ${(e.stack || String(e)).split('\\n').slice(0, 8).join('\\n ')}`); } };\n+\n+function tempTools(files) {\n+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'actor-prefix-'));\n+ for (const [name, body] of Object.entries(files)) fs.writeFileSync(path.join(dir, name), body);\n+ return dir;\n+}\n+const tool = (name) => `module.exports = { name: ${JSON.stringify(name)}, description: ${JSON.stringify(`The ${name} tool.`)}, input: { type: 'object', properties: { q: { type: 'string' } } }, scope: 'public', run: async () => ({ text: 'ok' }) };\\n`;\n+\n+(async () => {\n+ await ok('two tasks share the whole rules prefix; the date and the plan come after it', async () => {\n+ const a = SYSTEM({ date: '2026-11-09', plan: '1. Look up the price' });\n+ const b = SYSTEM({ date: '2027-01-01', plan: '1. Compare two offers', ledger: 'Progress: 1 of 2', note: 'Offered last: slow_tool.' });\n+ const at = a.indexOf('Today is ');\n+ const stableAt = a.indexOf('Your plan:');\n+ assert.ok(at > 0, 'the date is in the prompt');\n+ assert.ok(a.indexOf('How the answer is grounded:') < stableAt, 'the rules come before the plan');\n+ assert.ok(stableAt > 1500, `the stable prefix is the whole rules block (${stableAt} bytes)`);\n+ assert.strictEqual(a.slice(0, stableAt), b.slice(0, stableAt), 'the rules prefix is byte-identical between two tasks');\n+ assert.ok(at > stableAt && b.indexOf('Today is ') > stableAt, 'the date comes after the rules');\n+ assert.strictEqual(a.split('Today is ').length, 2, 'the date appears once');\n+ assert.ok(b.indexOf('1. Compare two offers') < b.indexOf('Today is '), 'the plan stays before the date');\n+ assert.ok(b.indexOf('Progress: 1 of 2') < b.indexOf('Today is '), 'the ledger stays before the date');\n+ assert.ok(b.indexOf('Offered last: slow_tool.') < b.indexOf('Today is '), 'the demoted-tools note stays before the date');\n+ });\n+\n+ await ok('tool definitions serialize identically on every assembly', async () => {\n+ const dir = tempTools({ 'zebra.js': tool('zebra'), 'alpha.js': tool('alpha'), 'middle.js': tool('middle') });\n+ const first = createRegistry({ dir, strict: true, log: { warn() {} } }).definitions({ requester: 'user:usr_1' });\n+ const second = createRegistry({ dir, strict: true, log: { warn() {} } }).definitions({ requester: 'user:usr_1' });\n+ assert.strictEqual(JSON.stringify(first), JSON.stringify(second));\n+ assert.deepStrictEqual(first.map((d) => d.function.name), ['alpha', 'middle', 'zebra'], 'files load in a fixed order');\n+ });\n+\n+ if (results.length) {\n+ for (const e of results) console.error(e.stack || String(e));\n+ process.exitCode = 1;\n+ }\n+})();\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Actor",++ "pull_request": 390,++ "merge_commit": "cfb27b4a3441d3382b9c242799a270725400fb2d",++ "merged_at": "2026-11-09T21:59:07Z"++ }++ },++ {++ "id": "actor-340-skill-negation",++ "goal": "The skills scan flags a URL to an undeclared host when a negation appears elsewhere in the sentence; a negation standing immediately before the URL (or one word before it) still passes it.",++ "check": {++ "kind": "command",++ "value": "node test/skills.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 30,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Actor",++ "ref": "e6dce59b91a45db5786074e4883d2d7094119ce8",++ "check_patch": "diff --git a/test/skills.test.js b/test/skills.test.js\nindex 5f5eea2..84d25a5 100644\n--- a/test/skills.test.js\n+++ b/test/skills.test.js\n@@ -63,6 +63,13 @@ const RULES = [\n good: { instructions: 'Read https://other.example/page for the answer.', requires: { hosts: ['other.example'] } },\n state: 'flagged',\n },\n+ {\n+ // A negation elsewhere in the sentence does not hide the host it reaches.\n+ rule: 'hosts.undeclared',\n+ bad: { instructions: 'Do not wait: read https://other.example/page for the answer.' },\n+ good: { instructions: 'Never open https://other.example/page, and answer from the notes.' },\n+ state: 'flagged',\n+ },\n ];\n \n (async () => {\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Actor",++ "pull_request": 340,++ "merge_commit": "cbdac596b1c45f7da826fe82a2dfd5e4ea6f4f38",++ "merged_at": "2026-11-08T02:39:40Z"++ }++ },++ {++ "id": "exchange-161-status",++ "goal": "The status board and a model's offer table show each offer's measured status, Up, Degraded when more than 5% of at least 10 requests in the last hour failed, or Down when the offer's circuit is open, with the plain status_reason and status_since that OpenVibe.AI sends in its catalog; Exchange infers nothing.",++ "check": {++ "kind": "command",++ "value": "node test/pages.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Exchange",++ "ref": "45c8ab948dc1a7e3232ae03d6993c4f51dab0e3f",++ "check_patch": "diff --git a/test/pages.test.js b/test/pages.test.js\nindex a4c6e18..c767900 100644\n--- a/test/pages.test.js\n+++ b/test/pages.test.js\n@@ -112,6 +112,27 @@ const { createFakeAi, downAi } = require('./helpers/ai');\n assert.ok(r.text.includes('99.0%') && r.text.includes('no traffic'));\n });\n \n+ await check('status is measured: down and degraded carry a plain reason and since when', async () => {\n+ const offer = ai.catalogData.models[0].offers[0];\n+ try {\n+ offer.status = 'down';\n+ offer.status_reason = 'the provider says the account is out of credit';\n+ offer.status_since = '2026-11-09T09:00:00.000Z';\n+ const down = await t.get('/status');\n+ assert.ok(down.text.includes('Down') && down.text.includes('the provider says the account is out of credit'), 'the status page says why');\n+ assert.ok(down.text.includes('since 2026-11-09 09:00 UTC'), 'and since when');\n+ offer.status = 'degraded';\n+ offer.status_reason = 'too many errors in the last hour';\n+ offer.status_since = '2026-11-09T10:00:00.000Z';\n+ const degraded = await t.get('/m/deepseek/deepseek-chat');\n+ assert.ok(degraded.text.includes('Degraded') && degraded.text.includes('too many errors in the last hour'), 'the model page shows the measured status');\n+ } finally {\n+ offer.status = 'up';\n+ offer.status_reason = null;\n+ offer.status_since = null;\n+ }\n+ });\n+\n await check('an unknown page is 404, and a personal page is never cached', async () => {\n const r = await t.get('/nope');\n assert.strictEqual(r.status, 404);\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Exchange",++ "pull_request": 161,++ "merge_commit": "277caa6ded619bf105fd4b5b36da83c2a2d70040",++ "merged_at": "2026-11-09T22:49:57Z"++ }++ },++ {++ "id": "exchange-160-pricing-console",++ "goal": "The owner console has owner-only JSON read endpoints for Providers, Models and the Pricebook, plus provider-settings and model-override form routes that forward audited changes to OpenVibe.AI; provider key values never appear in console data, and the existing owner and same-origin checks stay.",++ "check": {++ "kind": "command",++ "value": "node test/console-merge.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Exchange",++ "ref": "a5c516f7a7328972bcd17aa98a730d6b1ba7fa8f",++ "check_patch": "diff --git a/test/console-merge.test.js b/test/console-merge.test.js\nindex fd3a083..575860f 100644\n--- a/test/console-merge.test.js\n+++ b/test/console-merge.test.js\n@@ -24,7 +24,7 @@ app.use(createConsoleRoutes({\n await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));\n const base = `http://127.0.0.1:${server.address().port}`;\n try {\n- const consolePaths = ['/console', ...Object.keys(READS).map((section) => `/console/${section}`), '/console/providers/new', '/console/providers/openai', '/console/gpus/model-a/quote'];\n+ const consolePaths = ['/console', ...Object.keys(READS).map((section) => `/console/${section}`), '/console/providers/new', '/console/providers/openai', '/console/gpus/model-a/quote', '/console/data/providers', '/console/data/models', '/console/data/pricebook'];\n for (const path of consolePaths) for (const headers of [{}, { 'x-test-role': 'user' }, { 'x-test-role': 'admin' }]) {\n const response = await fetch(`${base}${path}`, { headers });\n assert.strictEqual(response.status, 404, `${path}: signed-out people and non-owners see no console route`);\n@@ -39,6 +39,12 @@ app.use(createConsoleRoutes({\n const page = await fetch(`${base}${path}`, { headers: { 'x-test-role': 'owner' } });\n assert.strictEqual(page.status, 200, `${path}: ${await page.text()}`);\n }\n+ for (const section of ['providers', 'models', 'pricebook']) {\n+ const data = await fetch(`${base}/console/data/${section}`, { headers: { 'x-test-role': 'owner' } });\n+ assert.strictEqual(data.status, 200);\n+ assert.ok((data.headers.get('content-type') || '').includes('application/json'));\n+ assert.ok(!JSON.stringify(await data.json()).includes(fakeSecret), `${section} data omits key values`);\n+ }\n const keyWrite = await fetch(`${base}/console/providers/openai/key`, {\n method: 'POST', headers: { 'x-test-role': 'owner', Origin: 'https://openvibe.exchange', 'content-type': 'application/x-www-form-urlencoded' },\n body: new URLSearchParams({ api_key: fakeSecret }), redirect: 'manual',\n@@ -55,6 +61,19 @@ app.use(createConsoleRoutes({\n assert.strictEqual(createProvider.status, 303);\n assert.ok(writes.some((write) => write.path === '/api/v1/admin/console/providers' && write.body.api_key === fakeSecret), 'provider creation uses AI JSON admin API');\n \n+ const providerSettings = await fetch(`${base}/console/providers/groq/settings`, {\n+ method: 'POST', headers: { 'x-test-role': 'owner', Origin: 'https://openvibe.exchange', 'content-type': 'application/x-www-form-urlencoded' },\n+ body: new URLSearchParams({ status: 'active', provider_access: 'AUTHORIZED_WHOLESALE', preference_order: '7', markup_percent: '12.5', uncensored: '1' }), redirect: 'manual',\n+ });\n+ assert.strictEqual(providerSettings.status, 303);\n+ assert.ok(writes.some((write) => write.path === '/api/v1/providers/groq' && write.body.markup_percent === 12.5 && write.body.uncensored === true));\n+ const modelOverride = await fetch(`${base}/console/models/groq/openai%2Fgpt-oss-120b/override`, {\n+ method: 'POST', headers: { 'x-test-role': 'owner', Origin: 'https://openvibe.exchange', 'content-type': 'application/x-www-form-urlencoded' },\n+ body: new URLSearchParams({ input_per_million: '0.2', cached_per_million: '', output_per_million: '0.7', reason: 'Owner approved adjustment' }), redirect: 'manual',\n+ });\n+ assert.strictEqual(modelOverride.status, 303);\n+ assert.ok(writes.some((write) => write.path === '/api/v1/models/groq/openai%2Fgpt-oss-120b' && write.body.owner_override.reason === 'Owner approved adjustment'));\n+\n } finally { await new Promise((resolve) => server.close(resolve)); }\n process.stdout.write('console-merge: owner gate and secret redaction passed\\n');\n })().catch((err) => { process.stderr.write(`${err.stack || err}\\n`); process.exitCode = 1; });\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Exchange",++ "pull_request": 160,++ "merge_commit": "418c289149ce2ed28e6a041d3ac9cde41f025fbf",++ "merged_at": "2026-11-09T20:19:25Z"++ }++ },++ {++ "id": "report-147-dataset-jsonld",++ "goal": "Report announces its tables as an open dataset through Dataset JSON-LD on / and /stats, built with the shared seo.jsonLd.dataset builder: name, description, creator OpenVibe, keywords and a distribution list pointing at /api/v1/stats, /api/v1/ai/results and /api/v1/rundown/topics.",++ "check": {++ "kind": "command",++ "value": "node test/pages.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 30,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Report",++ "ref": "b5b64d0ac69f555e12f72be0acfab01b311d38c7",++ "check_patch": "diff --git a/test/pages.test.js b/test/pages.test.js\nindex 8bd6c54..38e71ee 100644\n--- a/test/pages.test.js\n+++ b/test/pages.test.js\n@@ -70,6 +70,19 @@ const { boot, check, done, defaults } = require('./helpers/boot');\n assert.ok(org && Array.isArray(org.sameAs) && org.sameAs.includes('https://github.com/OpenVibers'), 'Organization');\n });\n \n+ await check('the home and the stats page carry Dataset JSON-LD with the JSON endpoints', async () => {\n+ for (const path of ['/', '/stats']) {\n+ const r = await t.get(path);\n+ const ld = [...r.text.matchAll(/<script type=\"application\\/ld\\+json\">([\\s\\S]*?)<\\/script>/g)].map((m) => JSON.parse(m[1])).flatMap((b) => b['@graph'] || [b]);\n+ const dataset = ld.find((x) => x['@type'] === 'Dataset');\n+ assert.ok(dataset, `${path}: Dataset`);\n+ assert.strictEqual(dataset.url, 'https://openvibe.report/stats', `${path}: Dataset url`);\n+ const urls = (dataset.distribution || []).map((d) => d.contentUrl);\n+ assert.ok(urls.includes('https://openvibe.report/api/v1/stats'), `${path}: stats JSON`);\n+ assert.ok(urls.includes('https://openvibe.report/api/v1/ai/results'), `${path}: AI JSON`);\n+ }\n+ });\n+\n await check('The Rundown pages name OpenVibe.Live as canonical and mount the Frame for service report', async () => {\n const r = await t.get('/rundown/t/seattle');\n assert.ok(r.text.includes('<link rel=\"canonical\" href=\"https://openvibe.live/rundown/t/seattle\">'), 'the topic is Live\\'s page');\n",++ "check_timeout_ms": 180000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Report",++ "pull_request": 147,++ "merge_commit": "87971dc92196d069a6c967b76304283f52d91595",++ "merged_at": "2026-11-09T09:55:25Z"++ }++ },++ {++ "id": "network-747-motion",++ "goal": "A display preference save keeps motion: full and motion: off as well as auto and reduced, reads each of the four back from the active theme, and still drops an unknown value.",++ "check": {++ "kind": "command",++ "value": "node --import ./test/helpers/pg-preload.mjs test/themes-review.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 30,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Network",++ "ref": "40712202526e8ee21de31d636a649db99dd6cb58",++ "check_patch": "diff --git a/test/themes-review.test.js b/test/themes-review.test.js\nindex b2dd68a..7abc6b9 100644\n--- a/test/themes-review.test.js\n+++ b/test/themes-review.test.js\n@@ -119,6 +119,13 @@ const token = async (id) => signToken(await db.prepare('SELECT * FROM users WHER\n assert.strictEqual(r.body.display.icons, 'solid', 'a person who prefers solid icons keeps that on every site');\n assert.strictEqual((await call('GET', '/api/themes/me/active', { as: 2 })).body.display.icons, 'solid');\n assert.strictEqual((await call('PUT', '/api/themes/me/display', { as: 2, body: { icons: 'outline' } })).body.display.icons, undefined, 'only auto, line or solid');\n+ // Motion (openvibe-shared 4.183.0): auto, full, reduced or off follows the account, and each one reads back.\n+ for (const motion of ['auto', 'full', 'reduced', 'off']) {\n+ r = await call('PUT', '/api/themes/me/display', { as: 2, body: { motion } });\n+ assert.strictEqual(r.body.display.motion, motion, `motion ${motion} is kept`);\n+ assert.strictEqual((await call('GET', '/api/themes/me/active', { as: 2 })).body.display.motion, motion, `motion ${motion} reads back`);\n+ }\n+ assert.strictEqual((await call('PUT', '/api/themes/me/display', { as: 2, body: { motion: 'calm' } })).body.display.motion, undefined, 'only auto, full, reduced or off');\n \n // ── The site switcher's pinned and recent sites follow the account (migration 0029) ──\n r = await call('GET', '/api/themes/me/sites', { as: 2 });\n",++ "check_timeout_ms": 240000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Network",++ "pull_request": 747,++ "merge_commit": "b9c60b27ab51d18d5dbbe6542c816345bfe3adca",++ "merged_at": "2026-11-10T02:44:47Z"++ }++ },++ {++ "id": "network-736-link-send",++ "goal": "openvibe.link's Send widget reads /auth/me before it asks for send keys, so a signed-out visitor sees its existing sign-in line with no 401 from /api/v1/me/nodes/send-keys.",++ "check": {++ "kind": "command",++ "value": "node --import ./test/helpers/pg-preload.mjs test/security-ssrf.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Network",++ "ref": "87944111b4aece1a8469a31ead30535636aa3553",++ "check_patch": "diff --git a/test/security-ssrf.test.js b/test/security-ssrf.test.js\nindex bb800f3..71da23e 100644\n--- a/test/security-ssrf.test.js\n+++ b/test/security-ssrf.test.js\n@@ -133,7 +133,7 @@ const REVIEWED = {\n 'server/admin/review-queue.js fetch': [2, 'the reviewer queue: Wiki/Blog/News/Trade internal URLs from operator config (loopback defaults), fixed /api/v1/review-queue and review paths, ids percent-encoded, the signed-in staff member\\'s own token, staff only'],\n 'server/admin/ai-config-proxy.js fetch': [1, 'the owner settings proxy and the Setup tab (server/admin/setup.js calls through it): the reachable services\\' internal URLs from config.services (operator-set, loopback defaults), fixed /internal/config and AI /api/v1/providers paths, service, namespace and key checked against the known names, owner only'],\n 'server/integrations/github.js fetch': [1, 'fixed host api.github.com (asserted below)'],\n- 'server/link/send.js fetch': [3, 'browser-only Link Send requests to fixed same-origin machine-key and transfer routes; page input supplies file bytes, never a URL or destination host'],\n+ 'server/link/send.js fetch': [4, 'browser-only Link Send requests to fixed same-origin routes (the session read, machine keys and transfers); page input supplies file bytes, never a URL or destination host'],\n 'server/notifications/email-service.js http.request': [1, 'fixed host api.resend.com'],\n 'server/actions/index.js fetch': [7, 'The Action Center: OpenVibe.Actor, OpenVibe.Codes and OpenVibe.AI only (OV_ACTOR_INTERNAL_URL, OV_CODES_INTERNAL_URL, OV_AI_INTERNAL_URL, loopback); the paths are fixed (the decide route, the cancel route, the retry route of Actor or AI, Actor\\'s public GET /api/v1/skills catalogue, and Actor\\'s POST /api/v1/runs for a catalogue skill whose id must match the skill id pattern), the subject is the signed-in person\\'s own usr_ id, and an item id must match [A-Za-z0-9_-]{1,80}'],\n 'server/profile/public.js fetch': [6, 'OpenVibe.Inventory, Quest and Play use operator-configured internal URLs and fixed paths with a checked usr_ id. Public place reads use only the configured Live, Community and Space service URLs, fixed public paths, encoded canonical usernames, 2 s timeouts and no user-supplied host or token; Space results are restricted to the exact owner and Published cards'],\n",++ "check_timeout_ms": 240000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Network",++ "pull_request": 736,++ "merge_commit": "ec99318366af37fce577f657242cf8c9c8554619",++ "merged_at": "2026-11-09T19:31:04Z"++ }++ },++ {++ "id": "network-724-wallet-cookie",++ "goal": "The frame wallet relay takes the person's ov_token cookie when no Bearer token is sent, forwards it upstream as the Bearer and never as a cookie, and only an origin already on the CORS allow-list (or no Origin) may use that path.",++ "check": {++ "kind": "command",++ "value": "node --import ./test/helpers/pg-preload.mjs test/frame-wallet.test.js"++ },++ "budget": {++ "tokens": 1000000,++ "turns": 40,++ "active_minutes": 6++ },++ "no_progress_limit": 4,++ "fixture": {++ "repo": "https://github.com/OpenVibers/OpenVibe.Network",++ "ref": "c69abee12d0e3f4cd7ffb7a5a049d38d71c8469b",++ "check_patch": "diff --git a/test/frame-wallet.test.js b/test/frame-wallet.test.js\nindex f574d5e..1a3c667 100644\n--- a/test/frame-wallet.test.js\n+++ b/test/frame-wallet.test.js\n@@ -75,6 +75,24 @@ const ITD = 'itd_01JZ00000000000000000000B1';\n assert.strictEqual(r.status, 401, 'no token, no wallet');\n assert.strictEqual(asked.filter((x) => x.url.startsWith('/api/v1/me/wallet')).length, 1, 'and nothing is asked upstream');\n \n+ // A site with its own cookie session has no token for the navbar: the person's Network cookie (ov_token) is the\n+ // token, sent upstream as the Bearer and never as a cookie; the answer keeps the allow-list's echoed Origin.\n+ const session = 'abcdefghijklmnop.cookie';\n+ ({ r, d } = await get('/wallet/api/v1/me/wallet', { Cookie: `ov_sso=x; ov_token=${session}`, Origin: 'https://openvibe.events' }));\n+ assert.strictEqual(r.status, 200);\n+ const viaCookie = asked.filter((x) => x.url.startsWith('/api/v1/me/wallet')).at(-1);\n+ assert.strictEqual(viaCookie.auth, `Bearer ${session}`, \"the person's session token goes along as the Bearer\");\n+ assert.strictEqual(viaCookie.cookie, '', 'the cookie itself never does');\n+ assert.notStrictEqual(r.headers.get('access-control-allow-origin'), '*', \"a credentialed answer is never '*'\");\n+ const askedBefore = asked.length;\n+ ({ r } = await send('/wallet/api/v1/me/vibes/buy', 'POST', { pack: 'vibes_500' }, { Authorization: '', Cookie: `ov_token=${session}` }));\n+ assert.strictEqual(r.status, 401, 'a write by cookie needs an Origin');\n+ ({ r } = await get('/wallet/api/v1/me/wallet', { Cookie: `ov_token=${session}`, Origin: 'chrome-extension://abcdefghijklmnopabcdefghijklmnop' }));\n+ assert.strictEqual(r.status, 401, 'never from an extension page');\n+ ({ r } = await get('/wallet/api/v1/me/wallet', { Cookie: 'ov_token=short' }));\n+ assert.strictEqual(r.status, 401, 'a malformed cookie is no token');\n+ assert.strictEqual(asked.length, askedBefore, 'none of these asked upstream');\n+\n // A page limit and cursor pass; every other query is dropped.\n ({ r } = await get('/wallet/api/v1/me/items?limit=500&cursor=c1&subject=usr_other', { Authorization: token }));\n assert.strictEqual(r.status, 200);\n",++ "check_timeout_ms": 240000++ },++ "source": {++ "repository": "OpenVibers/OpenVibe.Network",++ "pull_request": 724,++ "merge_commit": "783e1e08fc9583011a332ac000fa2c053d78e98d",++ "merged_at": "2026-11-09T14:18:43Z"++ }++ }++ ]++}nnew file mode 100644@@@ -0,0 +1,208 @@++'use strict';++++/**++ * The local executor for one ticket trial (AL8). It is called by scripts/ticket-suite.js (or by the real-run++ * harness) with one JSON document on stdin:++ *++ * { ticket, model, effort, trial, session, workspace, log_path? }++ *++ * It prepares a fresh checkout at the ticket's base commit, runs an OpenVibe.Codes agent on the goal, restores++ * everything under test/ (so the agent cannot grade itself), applies the pull request's test patch and runs the++ * check. It prints one JSON line on stdout:++ *++ * { passed, cost_usd, ms, tokens, turns, check_output, model, effort, workspace }++ *++ * Environment: `DEEPSEEK_API_KEY` for deepseek-flash; `OPENVIBE_CODES_DIR` for the harness (default++ * ~/openvibe/work/OpenVibe.Codes); `OPENVIBE_CODES_BASE_URL` (and `OPENVIBE_CODES_MODEL`) for the local model;++ * `OPENVIBE_REPO_CACHE` to clone from a local mirror instead of GitHub. Nothing here prints a secret: the agent's++ * events are consumed, its cost and tokens only are reported, and its log goes to log_path when one is named.++ */++const fs = require('fs');++const os = require('os');++const path = require('path');++const { spawn, execFileSync } = require('child_process');++++const MAX_OUTPUT = 4000;++++function sh(cmd, args, cwd, { timeout = 300000, env = process.env } = {}) {++ return new Promise((resolve) => {++ const child = spawn(cmd, args, { cwd, env, stdio: ['ignore', 'pipe', 'pipe'] });++ let out = '';++ const add = (d) => { out = (out + d.toString()).slice(-MAX_OUTPUT); };++ child.stdout.on('data', add); child.stderr.on('data', add);++ const timer = setTimeout(() => child.kill('SIGKILL'), timeout);++ child.on('close', (code) => { clearTimeout(timer); resolve({ ok: code === 0, code, out }); });++ child.on('error', (err) => { clearTimeout(timer); resolve({ ok: false, code: -1, out: err.message }); });++ });++}++++function git(args, cwd) {++ try { return { ok: true, out: execFileSync('git', args, { cwd, encoding: 'utf8', timeout: 120000, stdio: ['ignore', 'pipe', 'ignore'] }) }; }++ catch (err) { return { ok: false, out: String((err.stderr || err.stdout || err.message)).slice(-800) }; }++}++++/** Where a repo's local clone cache lives, when one is configured. */++function cachedRepo(url, cache) {++ if (!cache) return null;++ const name = String(url).replace(/\.git$/, '').split('/').slice(-2).join('/');++ for (const candidate of [path.join(cache, name), path.join(cache, `${name}.git`)]) {++ if (fs.existsSync(candidate)) return candidate;++ }++ return null;++}++++/** The checkout at the ticket's base commit: cloned once, reset before every trial, dependencies installed once. */++async function prepare(input) {++ const workspace = input.workspace || fs.mkdtempSync(path.join(os.tmpdir(), 'bench-ticket-'));++ const ref = input.ticket.fixture.ref;++ const fresh = !fs.existsSync(path.join(workspace, '.git'));++ if (fresh) {++ fs.mkdirSync(workspace, { recursive: true });++ const cache = cachedRepo(input.ticket.fixture.repo, process.env.OPENVIBE_REPO_CACHE);++ const clone = cache++ ? await sh('git', ['clone', '--shared', '-q', cache, workspace], path.dirname(workspace))++ : await sh('git', ['clone', '--filter=blob:none', '-q', input.ticket.fixture.repo, workspace], path.dirname(workspace));++ if (!clone.ok) throw new Error(`clone failed: ${clone.out}`);++ }++ const checkout = git(['checkout', '-q', ref], workspace);++ if (!checkout.ok) {++ const fetched = git(['fetch', '-q', 'origin', ref], workspace);++ if (!fetched.ok) throw new Error(`checkout ${ref} failed: ${checkout.out}`);++ const again = git(['checkout', '-q', ref], workspace);++ if (!again.ok) throw new Error(`checkout ${ref} failed: ${again.out}`);++ }++ git(['reset', '--hard', '-q', ref], workspace);++ git(['clean', '-fdq', '-e', 'node_modules'], workspace);++ if (!fs.existsSync(path.join(workspace, 'node_modules'))) {++ const installed = await sh('npm', ['ci', '--no-audit', '--no-fund', '--prefer-offline'], workspace, { timeout: 600000 });++ if (!installed.ok) throw new Error(`npm ci failed: ${installed.out.slice(-600)}`);++ }++ return { workspace, fresh };++}++++/** The prompt one agent gets: the goal, its check and the budget. The test patch is never in it. */++function promptFor(input) {++ const ticket = input.ticket;++ const budget = ticket.budget || {};++ return [++ `Work in the repository in ${input.workspace}. Its base commit is ${ticket.fixture.ref}.`,++ '',++ 'The goal:',++ ticket.goal,++ '',++ `The check: ${ticket.check.kind} — ${ticket.check.value}`,++ budget.turns ? `Turn budget: ${budget.turns}.` : '',++ budget.active_minutes ? `Time budget: about ${budget.active_minutes} minutes.` : '',++ '',++ 'Do not add or edit anything under test/: the graded tests are applied after you stop, and your copy of them is discarded.',++ 'Do not run the full suite (npm test or node test/run.js): run at most one focused test file, or node --check on the files you changed.',++ 'Make the smallest change in the code the goal names, and end with a short summary.',++ ].filter((line) => line !== '').join('\n');++}++++/** Run one OpenVibe.Codes agent on the ticket. Returns cost, tokens, turns and the final text. */++async function runAgent(input, workspace) {++ const codesDir = process.env.OPENVIBE_CODES_DIR || path.join(os.homedir(), 'openvibe', 'work', 'OpenVibe.Codes');++ let createHarness;++ try { ({ createHarness } = require(path.join(codesDir, 'harness'))); }++ catch (err) { throw new Error(`the OpenVibe.Codes harness is not readable at ${codesDir}: ${err.message}`); }++ const model = String(input.model);++ const agent = model === 'local' ? 'openai-compatible' : 'deepseek';++ const modelName = model === 'local' ? (process.env.OPENVIBE_CODES_MODEL || null) : model;++ const budget = input.ticket.budget || {};++ const harness = createHarness({ env: process.env });++ const run = harness.run({++ prompt: promptFor(input),++ agent,++ model: modelName,++ cwd: workspace,++ permission: 'full',++ handoff: false,++ maxTurns: Math.max(1, Number(budget.turns) || 30),++ maxAttempts: 1,++ stallMs: 4 * 60 * 1000,++ });++ // A hard wall clock: the harness's quiet timer cancels a model call but a command's grandchild can hold its++ // pipe open forever, so one trial is never allowed to outlive its budget. The agent's own spend still counts.++ const deadlineMs = Math.max(60_000, Math.min(8 * 60_000, Number(budget.active_minutes || 8) * 60_000));++ const consumer = (async () => {++ let result = null;++ for await (const event of run) {++ if (event.type === 'result') result = event;++ if (input.log_path) fs.appendFileSync(input.log_path, JSON.stringify({ type: event.type, subtype: event.subtype || null, text: event.result || (event.message && event.message.content && event.message.content[0] && event.message.content[0].text) || null }) + '\n');++ }++ const usage = (result && result.usage) || {};++ return {++ cost_usd: result ? Number(result.total_cost_usd) || 0 : 0,++ tokens: (Number(usage.input_tokens) || 0) + (Number(usage.output_tokens) || 0) + (Number(usage.cache_read_input_tokens) || 0),++ turns: result ? Number(result.num_turns) || 0 : 0,++ ok: Boolean(result && !result.is_error),++ text: result ? String(result.result || '') : '',++ };++ })();++ let timer;++ const timeout = new Promise((resolve) => { timer = setTimeout(() => resolve({ timed_out: true }), deadlineMs); timer.unref?.(); });++ const agentResult = await Promise.race([consumer, timeout]);++ clearTimeout(timer);++ if (agentResult.timed_out) {++ try { run.cancel(); } catch { /* the run is already gone */ }++ return { cost_usd: 0, tokens: 0, turns: 0, ok: false, text: `the agent did not finish within ${Math.round(deadlineMs / 60000)} minutes` };++ }++ return agentResult;++}++++/** Restore test/, apply the pull request's test patch, run the check. The only place a trial can pass. */++async function grade(input, workspace) {++ git(['checkout', '--', 'test'], workspace);++ git(['clean', '-fdq', 'test'], workspace);++ const patchFile = path.join(workspace, '.bench-check.patch');++ fs.writeFileSync(patchFile, input.ticket.fixture.check_patch);++ const applied = await sh('git', ['apply', '--whitespace=nowarn', '.bench-check.patch'], workspace);++ fs.rmSync(patchFile, { force: true });++ if (!applied.ok) return { passed: false, check_output: `the check patch did not apply: ${applied.out}` };++ const timeout = Number(input.ticket.fixture.check_timeout_ms) || 180000;++ const check = await sh('bash', ['-lc', input.ticket.check.value], workspace, { timeout });++ return { passed: check.ok, check_output: check.out };++}++++async function main(raw) {++ const input = JSON.parse(raw);++ const started = Date.now();++ const { workspace } = await prepare(input);++ let agent;++ try { agent = await runAgent(input, workspace); }++ catch (err) { agent = { cost_usd: 0, tokens: 0, turns: 0, ok: false, text: err.message }; }++ // A test command (a URL check and a rubric are graded by a person, not this executor) still needs a workspace.++ const graded = input.ticket.check.kind === 'command' ? await grade(input, workspace) : { passed: false, check_output: `a ${input.ticket.check.kind} check is not graded by the local executor` };++ return {++ passed: graded.passed,++ cost_usd: Math.round(agent.cost_usd * 1e6) / 1e6,++ ms: Date.now() - started,++ tokens: agent.tokens,++ turns: agent.turns,++ agent_ok: agent.ok,++ agent_text: agent.text.slice(0, 2000),++ check_output: String(graded.check_output || '').slice(-MAX_OUTPUT),++ model: input.model,++ effort: input.effort,++ workspace,++ };++}++++if (require.main === module) {++ let raw = '';++ process.stdin.setEncoding('utf8');++ process.stdin.on('data', (d) => { raw += d; });++ process.stdin.on('end', () => {++ main(raw).then((out) => {++ process.stdout.write(JSON.stringify(out) + '\n');++ // A timed-out agent may leave a child holding a pipe: the result is out, so the process may stop.++ setTimeout(() => process.exit(0), 250).unref();++ }, (err) => {++ process.stdout.write(JSON.stringify({ passed: false, cost_usd: 0, ms: 0, tokens: 0, check_output: String(err && err.message || err).slice(0, 2000) }) + '\n');++ setTimeout(() => process.exit(1), 250).unref();++ });++ });++}++++module.exports = { main, prepare, promptFor, grade, cachedRepo };nnew file mode 100644@@@ -0,0 +1,173 @@++#!/usr/bin/env node++'use strict';++++/**++ * The AL8 gate, from the command line. It reads a suite file (bench.codes-ticket-suite@1), checks it, and — only++ * when the caller passes `--run` — runs every ticket over 3 to 5 trials on deepseek-flash and the local model,++ * publishes one run per trial and reads the table back from the results API.++ *++ * node scripts/ticket-suite.js # the plan: no call, no cost++ * node scripts/ticket-suite.js --run --trials 3 \++ * --models deepseek-flash --effort high \++ * --publish https://bench.openvibe.services --token "$BENCH_SERVICE_TOKEN"++ *++ * There is no timer and no schedule: a suite runs when a person runs this command.++ *++ * The executor is a command that takes one JSON trial on stdin and answers one JSON result line (the default is++ * scripts/ticket-executor.js). `--benchmark bmk_…` reuses a frozen suite instead of creating a new one; a new run++ * of an old benchmark is what makes pass^k comparable over time.++ */++const fs = require('fs');++const path = require('path');++const os = require('os');++const { spawn } = require('child_process');++const tickets = require('../server/bench/tickets');++const runner = require('../server/bench/ticket-runner');++++const ROOT = path.join(__dirname, '..');++const DEFAULT_SUITE = path.join(ROOT, 'fixtures', 'codes.ticket', 'openvibe-merged-2026-11.json');++++function parseArgs(argv) {++ const out = { _: [] };++ for (let i = 0; i < argv.length; i++) {++ const arg = argv[i];++ if (!arg.startsWith('--')) { out._.push(arg); continue; }++ const key = arg.slice(2);++ const next = argv[i + 1];++ if (next == null || next.startsWith('--')) out[key] = true;++ else { out[key] = next; i++; }++ }++ return out;++}++++function list(value, fallback) {++ if (value == null || value === true) return fallback;++ return String(value).split(',').map((x) => x.trim()).filter(Boolean);++}++++/** A stable line the report quotes: pass@1, pass^k and cost per solved for each model and effort. */++function tableLines(entries) {++ return entries.map((e) => `${e.model} · effort ${e.effort || 'default'} · trials ${e.k} · pass@1 ${e.pass_at_1 == null ? '—' : e.pass_at_1.toFixed(3)} · pass^${e.k} ${e.pass_at_k == null ? '—' : e.pass_at_k.toFixed(3)} · cost per solved ${e.cost_per_solved == null ? '—' : '$' + e.cost_per_solved.toFixed(4)}${e.insufficient_trials ? ' (fewer than 3 trials: not a gate result)' : ''}`);++}++++async function callJson(base, pathname, { method = 'GET', body, token } = {}) {++ const response = await fetch(new URL(pathname, base), {++ method,++ headers: { accept: 'application/json', ...(body ? { 'content-type': 'application/json' } : {}), ...(token ? { authorization: `Bearer ${token}` } : {}) },++ body: body ? JSON.stringify(body) : undefined,++ signal: AbortSignal.timeout(30000),++ });++ const text = await response.text();++ let json = null;++ try { json = text ? JSON.parse(text) : null; } catch { /* a problem document may not be JSON */ }++ if (!response.ok) throw new Error(`${method} ${pathname} answered HTTP ${response.status}: ${(json && (json.detail || json.code)) || text.slice(0, 300)}`);++ return json;++}++++/** Create and freeze a benchmark for the suite, or reuse one the caller names. */++async function ensureBenchmark(base, token, suite, reuse, log) {++ if (reuse) {++ const found = await callJson(base, `/api/v1/benchmarks/${encodeURIComponent(reuse)}`, { token });++ const benchmark = found.benchmark || found;++ const version = benchmark.latest_version || (found.version && found.version.version);++ if (!version) throw new Error(`${reuse} has no frozen version`);++ log(`reusing ${benchmark.id}@${version}`);++ return { id: benchmark.id, version };++ }++ const definition = tickets.benchmarkDefinition(suite);++ const made = await callJson(base, '/api/v1/benchmarks', {++ method: 'POST', token,++ body: { title: definition.title, description: definition.description, kind: definition.kind, tags: definition.tags, visibility: 'unlisted', definition },++ });++ const id = made.benchmark.id;++ const frozen = await callJson(base, `/api/v1/benchmarks/${encodeURIComponent(id)}/versions`, { method: 'POST', token, body: {} });++ log(`created ${id}@${frozen.version.version}`);++ return { id, version: frozen.version.version };++}++++/** One trial through the executor command; its stdout's last JSON line is the result. */++function executeWith(command, input, logPath) {++ return new Promise((resolve) => {++ const child = spawn('bash', ['-lc', command], { stdio: ['pipe', 'pipe', 'pipe'] });++ let out = '';++ let err = '';++ child.stdout.on('data', (d) => { out += d.toString(); });++ child.stderr.on('data', (d) => { err += d.toString(); });++ child.stdin.end(JSON.stringify(input));++ const timer = setTimeout(() => child.kill('SIGKILL'), 30 * 60 * 1000);++ child.on('close', () => {++ clearTimeout(timer);++ if (logPath) fs.writeFileSync(logPath, err.slice(-20000));++ const lines = out.trim().split('\n').filter(Boolean);++ for (let i = lines.length - 1; i >= 0; i--) {++ try { return resolve(JSON.parse(lines[i])); } catch { /* earlier lines are not the result */ }++ }++ resolve({ passed: false, cost_usd: 0, ms: 0, tokens: 0, check_output: `the executor printed no result: ${out.slice(-500)} ${err.slice(-500)}` });++ });++ child.on('error', (error) => { clearTimeout(timer); resolve({ passed: false, cost_usd: 0, ms: 0, tokens: 0, check_output: error.message }); });++ });++}++++async function main() {++ const args = parseArgs(process.argv.slice(2));++ const suitePath = path.resolve(String(args.suite || DEFAULT_SUITE));++ if (!fs.existsSync(suitePath)) throw new Error(`no suite file at ${suitePath}`);++ const suite = JSON.parse(fs.readFileSync(suitePath, 'utf8'));++ const checked = tickets.validateSuite(suite);++ if (!checked.valid) throw new Error(`the suite is not valid: ${checked.detail}`);++ const models = list(args.models, suite.models || tickets.DEFAULT_MODELS);++ const trials = args.trials == null ? suite.trials : Number(args.trials);++ const effort = args.effort == null ? 'high' : String(args.effort);++ const plan = runner.plan(suite, { models, trials, effort });++++ process.stdout.write(`suite ${suite.suite}: ${plan.tickets.length} tickets × ${plan.models.join(', ')} × ${plan.trials} trials = ${plan.calls} calls, effort ${plan.effort}\n`);++ for (const ticket of plan.tickets) process.stdout.write(` ${ticket.id}: ${ticket.check}\n`);++++ if (!args.run) {++ process.stdout.write('not run: pass --run to spend money on these trials\n');++ return;++ }++ if (args.publish && !args.token) throw new Error('--publish needs --token (or BENCH_SERVICE_TOKEN)');++ const base = args.publish ? String(args.publish).replace(/\/+$/, '') : null;++ const token = args.token && args.token !== true ? String(args.token) : process.env.BENCH_SERVICE_TOKEN;++ const log = (message) => process.stderr.write(`[ticket-suite] ${message}\n`);++ const benchmark = base ? await ensureBenchmark(base, token, suite, args.benchmark, log) : null;++ const workdir = path.resolve(String(args.workdir || path.join(os.tmpdir(), `bench-tickets-${suite.suite}`)));++ const logDir = args['log-dir'] ? path.resolve(String(args['log-dir'])) : null;++ if (logDir) fs.mkdirSync(logDir, { recursive: true });++ const executor = String(args.executor || `node ${JSON.stringify(path.join(__dirname, 'ticket-executor.js'))}`);++++ const out = await runner.runSuite({++ suite, models, trials, effort, run: true,++ execute: async ({ ticket, model, trial, session }) => {++ const workspace = path.join(workdir, ticket.id);++ const logPath = logDir ? path.join(logDir, `${ticket.id}-${model.replace(/[^a-z0-9.-]/gi, '_')}-${trial}.log`) : null;++ log(`${ticket.id} · ${model} · trial ${trial}/${plan.trials}`);++ return executeWith(executor, { ticket, model, effort, trial, session, workspace, log_path: logPath }, logPath);++ },++ });++ if (!out.ok) throw new Error(`${out.code}: ${out.detail}`);++++ let published = [];++ if (base) {++ for (const run of out.runs) {++ const made = await callJson(base, '/api/v1/runs', { method: 'POST', token, body: { ...run, benchmark: `${benchmark.id}@${benchmark.version}`, benchmark_id: benchmark.id, benchmark_version: benchmark.version } });++ published.push(made.run.id);++ }++ log(`published ${published.length} runs on ${benchmark.id}@${benchmark.version}`);++ }++ let table = out.table;++ if (base) {++ const read = await callJson(base, `/api/v1/rankings?benchmark=${encodeURIComponent(benchmark.id + '@' + benchmark.version)}&metric=tickets&effort=${encodeURIComponent(effort)}`, { token });++ table = read.rankings.entries;++ }++ const report = { suite: suite.suite, session: out.session, benchmark, effort, models, trials: plan.trials, runs: published, table };++ if (args.out) fs.writeFileSync(path.resolve(String(args.out)), JSON.stringify(report, null, 2));++ for (const line of tableLines(table)) process.stdout.write(line + '\n');++}++++if (require.main === module) {++ main().catch((err) => { process.stderr.write(`[ticket-suite] ${err.message}\n`); process.exitCode = 1; });++}++++module.exports = { parseArgs, tableLines, ensureBenchmark, executeWith };@@@ -21,6 +21,10 @@ const { isHeldOut, addedOn, humanMinutes, humanMinutesSource } = require('./taskconst { cutoffFor } = require('./model-cutoffs');const { contaminationFor } = require('./contamination');const { estimateHorizon } = require('./horizon');++const { scoreRuns } = require('./tickets');++++/** The metric names the gate answers to. `tickets` is the API's name; pass@k aliases are what callers type. */++const TICKET_METRICS = Object.freeze(['tickets', 'pass_k', 'pass@k', 'pass^k']);const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));@@@ -831,13 +835,14 @@ function createRegistry(s, { log = console, sharing = null } = {}) {* speed the median score per minute* consistency the spread of a model's scores inside one session (the lower, the steadier)* horizon the 50% human-time horizon (minutes; higher is better)++ * tickets the AL8 gate: pass@1, pass^k and cost per solved by model and effort, for a whole-ticket suite** `since` is a monthly fresh set: with dated tasks, only the tasks added on or after that month are scored.* An older benchmark whose tasks carry no dates keeps the B2 meaning (runs created since the date), so nothing* that worked before stops working. Each entry also carries the contamination flag and the time horizon when* there is enough data for either.*/-- async function rankings({ benchmark, metric = 'score', harness = null, since = null, includeSubmitted = false }) {++ async function rankings({ benchmark, metric = 'score', harness = null, since = null, includeSubmitted = false, effort = null }) {const ref = parseRef(benchmark);if (!ref) return null;const versions = await db.many(@@@ -883,6 +888,21 @@ function createRegistry(s, { log = console, sharing = null } = {}) {if (!tasksByRun.has(row.run_id)) tasksByRun.set(row.run_id, []);tasksByRun.get(row.run_id).push(row);}++ // The AL8 gate reads the same stored runs, one row per model and effort. A ticket is a task; a run is one++ // trial of the whole suite, so pass^k needs every trial's task row.++ if (TICKET_METRICS.includes(metric)) {++ const scored = scoreRuns(rows.map((row) => wireRun(row)), tasksByRun, { effort });++ const latest = definitions.get(Math.max(...wanted));++ return {++ benchmark: ref.id,++ metric: 'tickets',++ kind: latest ? latest.kind : null,++ effort: effort == null ? null : String(effort),++ trials_required: { min: 3, max: 5 },++ entries: scored.entries,++ };++ }++const estimates = await estimateMedians(ref.id, wanted);/** The score and totals of one run, over the selected tasks when a monthly set is asked for. */nnew file mode 100644@@@ -0,0 +1,108 @@++'use strict';++++/**++ * The AL8 gate's runner: a whole ticket over 3 to 5 trials per model and effort, then one row per model and effort.++ *++ * Two rules are code here, not advice:++ *++ * 1. It never runs by itself. Without `run: true` (the CLI's `--run`) it returns the plan and makes no call, so++ * no timer, scheduler or page start can spend money.++ * 2. It runs only on the cheap lanes unless the caller widens them on purpose: `deepseek-flash` and the local++ * model (`local`). Anything else is refused before the first trial.++ *++ * The executor is injected. It receives one ticket trial and answers with what the check said:++ *++ * execute({ ticket, model, effort, trial, session }) → { passed, cost_usd, ms, tokens, check_output }++ *++ * `scripts/ticket-executor.js` is the local implementation (an OpenVibe.Codes agent in a fresh checkout); a Codes++ * benchmark runner can implement the same shape. Because the check patch is applied by the executor after the++ * agent stops, the agent never grades its own copy of the test.++ */++const tickets = require('./tickets');++++const PROVIDERS = Object.freeze({ 'deepseek-flash': 'deepseek', local: 'local' });++++/** The plan a run would follow: which models, how many trials, which tickets. No money moves. */++function plan(suite, { models, trials, effort = 'high' } = {}) {++ const list = Array.isArray(models) && models.length ? models.map((m) => String(m)) : (suite.models || tickets.DEFAULT_MODELS).map((m) => String(m));++ const k = trials == null ? suite.trials : Number(trials);++ return {++ suite: suite.suite,++ models: list,++ trials: k,++ effort: effort == null ? 'high' : String(effort),++ tickets: (suite.tickets || []).map((t) => ({ id: t.id, check: t.check && t.check.value })),++ calls: list.length * (suite.tickets || []).length * k,++ };++}++++/**++ * Run the suite, or say what it would do. `execute` is required only when `run` is true. Returns the trial runs++ * (ready for POST /api/v1/runs) and the gate table (the same rows the results API computes).++ */++async function runSuite({ suite, execute = null, run = false, models = null, trials = null, effort = 'high', session = null, now = () => new Date() } = {}) {++ const checked = tickets.validateSuite(suite);++ if (!checked.valid) return { ok: false, code: 'bench.ticket_suite_invalid', detail: checked.detail };++ const thePlan = plan(suite, { models, trials, effort });++ if (thePlan.models.some((m) => !tickets.DEFAULT_MODELS.includes(m))) {++ return { ok: false, code: 'bench.ticket_model_not_allowed', detail: `the gate runs only on ${tickets.DEFAULT_MODELS.join(' and ')} unless a person widens it` };++ }++ if (!Number.isInteger(thePlan.trials) || thePlan.trials < tickets.TRIAL_MIN || thePlan.trials > tickets.TRIAL_MAX) {++ return { ok: false, code: 'bench.ticket_trials', detail: `trials must be ${tickets.TRIAL_MIN}-${tickets.TRIAL_MAX}` };++ }++ if (!run) return { ok: true, ran: false, plan: thePlan };++ if (typeof execute !== 'function') return { ok: false, code: 'bench.ticket_executor_missing', detail: 'a run needs an executor' };++++ const sessionId = session || `ticket-${thePlan.suite}-${now().toISOString().replace(/[^0-9]/g, '').slice(0, 14)}`;++ const runs = [];++ const tasksByRun = new Map();++ for (const model of thePlan.models) {++ for (let trial = 1; trial <= thePlan.trials; trial++) {++ const results = [];++ for (const ticket of suite.tickets) {++ let out;++ try {++ out = await execute({ ticket, model, effort: thePlan.effort, trial, session: sessionId });++ } catch (err) {++ out = { passed: false, error: (err && err.message) || String(err), cost_usd: 0, ms: 0, tokens: 0, check_output: '' };++ }++ out = out && typeof out === 'object' ? out : { passed: false };++ results.push({++ task_id: ticket.id,++ score: out.passed === true ? 100 : 0,++ cost_usd: Number.isFinite(Number(out.cost_usd)) ? Number(out.cost_usd) : 0,++ ms: Number.isFinite(Number(out.ms)) ? Math.max(0, Math.round(Number(out.ms))) : 0,++ tokens: Number.isFinite(Number(out.tokens)) ? Math.max(0, Math.round(Number(out.tokens))) : 0,++ artifacts: [{ kind: 'check', passed: out.passed === true, output: String(out.check_output == null ? '' : out.check_output).slice(0, 2000) }],++ ...(out.transcript_url ? { transcript_url: String(out.transcript_url).slice(0, 500) } : {}),++ });++ }++ const cost = results.reduce((n, r) => n + r.cost_usd, 0);++ const ms = results.reduce((n, r) => n + r.ms, 0);++ const tokens = results.reduce((n, r) => n + r.tokens, 0);++ const score = results.length ? results.reduce((n, r) => n + r.score, 0) / results.length : 0;++ const runDoc = {++ model,++ provider: PROVIDERS[model] || 'unknown',++ route: { requested: model, ran: model, effort: thePlan.effort },++ harness: { name: 'openvibe-codes', version: '0.4.0' },++ settings: { effort: thePlan.effort },++ executor: { kind: 'codes', id: 'bench-ticket-suite' },++ session: sessionId,++ tasks: results,++ score: Math.round(score * 1000) / 1000,++ cost_usd: Math.round(cost * 1e6) / 1e6,++ ms,++ tokens,++ created_at: now().toISOString(),++ };++ tasksByRun.set(`trial-${runs.length + 1}`, results.map((r) => ({ task_id: r.task_id, score: r.score, cost_usd: r.cost_usd, ms: r.ms, tokens: r.tokens })));++ runs.push(runDoc);++ }++ }++ // The table is computed from the same shape the database will store, so a published run reads back identically.++ const scored = require('./tickets').scoreRuns(runs.map((r, i) => ({ ...r, id: `trial-${i + 1}`, created_at: now().toISOString() })), tasksByRun, { effort: thePlan.effort });++ return { ok: true, ran: true, session: sessionId, plan: thePlan, runs, table: scored.entries };++}++++module.exports = { PROVIDERS, plan, runSuite };nnew file mode 100644@@@ -0,0 +1,285 @@++'use strict';++++/**++ * Whole-ticket benchmarks (AL8, plan 2026-11-10): a ticket is a goal, its check and a budget, drawn from a real++ * merged pull request. The worker gets the goal; the check runs only after it stops, so a test added by the pull++ * request measures the work instead of the worker's copy of it.++ *++ * The goal and check are AL1's `agent.goal@1` shape (Contracts 0.389.0, still in review when this landed): the++ * goal is the condition in words, the check is a command, a URL assertion or a rubric, and the budget bounds the++ * tokens, turns and active minutes a trial may spend (the local executor in scripts/ticket-executor.js enforces++ * turns and active minutes; a token cap is carried for executors that can stop on usage). This module defines the++ * suite file Bench reads and the scoring every run is judged by; the runner that spends money lives in++ * server/bench/ticket-runner.js.++ *++ * A suite is `bench.codes-ticket-suite@1`:++ *++ * {++ * "format": "bench.codes-ticket-suite@1",++ * "suite": "openvibe-merged-2026-11",++ * "title": "Twelve merged OpenVibe tickets",++ * "kind": "codes.ticket",++ * "models": ["deepseek-flash", "local"],++ * "trials": 3,++ * "tickets": [{++ * "id": "actor-390-prompt-prefix",++ * "goal": "The coding agent's request prefix stays byte-identical …",++ * "check": { "kind": "command", "value": "node test/runtime-prompt-prefix.test.js" },++ * "budget": { "tokens": 80000, "turns": 40, "active_minutes": 8 },++ * "no_progress_limit": 4,++ * "fixture": {++ * "repo": "https://github.com/OpenVibers/OpenVibe.Actor",++ * "ref": "<the merge commit's parent>",++ * "check_patch": "<unified diff of the pull request's test files>",++ * "check_timeout_ms": 180000++ * },++ * "source": { "repository": "OpenVibers/OpenVibe.Actor", "pull_request": 390,++ * "merge_commit": "…", "merged_at": "2026-11-10T…" }++ * }]++ * }++ *++ * Only `deepseek-flash` and `local` are allowed by default: the gate is meant to run often and must not spend on++ * the dear models. A suite runs only when a caller says `--run`; nothing here is scheduled.++ */++++const TRIAL_MIN = 3;++const TRIAL_MAX = 5;++const FORMAT = 'bench.codes-ticket-suite@1';++const KIND = 'codes.ticket';++const DEFAULT_MODELS = Object.freeze(['deepseek-flash', 'local']);++const CHECK_KINDS = Object.freeze(['command', 'url', 'rubric']);++const REPO = /^https:\/\/github\.com\/[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/;++const SHA = /^[0-9a-f]{40}$/;++++const isStr = (v) => typeof v === 'string';++const isNum = (v) => typeof v === 'number' && Number.isFinite(v);++const isObj = (v) => v && typeof v === 'object' && !Array.isArray(v);++const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));++++class Problems {++ constructor() { this.list = []; }++ add(path, message) { this.list.push({ path, message }); return this; }++ get valid() { return this.list.length === 0; }++ get detail() { return this.list.map((e) => `${e.path} ${e.message}`).join('; '); }++}++++/** AL1's goal and check: the goal in words, a typed check, optional budgets and a no-progress limit. */++function validateGoal(goal, path = '') {++ const e = new Problems();++ if (!isObj(goal)) return e.add(path || '/', 'must be an object');++ if (!isStr(goal.goal) || !goal.goal.trim() || goal.goal.length > 4000) e.add(`${path}/goal`, 'must be 1-4000 characters');++ if (!isObj(goal.check)) e.add(`${path}/check`, 'must be an object');++ else {++ if (!CHECK_KINDS.includes(goal.check.kind)) e.add(`${path}/check/kind`, `must be one of ${CHECK_KINDS.join(', ')}`);++ if (!isStr(goal.check.value) || !goal.check.value.trim() || goal.check.value.length > 4000) e.add(`${path}/check/value`, 'must be 1-4000 characters');++ }++ if (goal.budget != null) {++ if (!isObj(goal.budget)) e.add(`${path}/budget`, 'must be an object');++ else {++ for (const key of ['tokens', 'turns', 'active_minutes']) {++ if (goal.budget[key] != null && (!Number.isInteger(goal.budget[key]) || goal.budget[key] < 1)) e.add(`${path}/budget/${key}`, 'must be a positive integer');++ }++ if (!Object.keys(goal.budget).some((k) => ['tokens', 'turns', 'active_minutes'].includes(k))) e.add(`${path}/budget`, 'needs tokens, turns or active_minutes');++ }++ }++ if (goal.no_progress_limit != null && (!Number.isInteger(goal.no_progress_limit) || goal.no_progress_limit < 1 || goal.no_progress_limit > 20)) {++ e.add(`${path}/no_progress_limit`, 'must be an integer 1-20');++ }++ return e;++}++++/** One ticket in a suite: the AL1 goal and check, plus where the base checkout comes from and how it is graded. */++function validateTicket(ticket, path = '') {++ const e = new Problems();++ if (!isObj(ticket)) return e.add(path || '/', 'must be an object');++ if (!isStr(ticket.id) || !/^[a-z0-9][a-z0-9-]{1,79}$/.test(ticket.id)) e.add(`${path}/id`, 'must be 2-80 lowercase letters, digits and dashes');++ const goal = validateGoal(ticket, path);++ for (const p of goal.list) e.add(p.path, p.message);++ if (!isObj(ticket.fixture)) e.add(`${path}/fixture`, 'must be an object');++ else {++ if (!isStr(ticket.fixture.repo) || !REPO.test(ticket.fixture.repo)) e.add(`${path}/fixture/repo`, 'must be an https GitHub repository URL');++ if (!isStr(ticket.fixture.ref) || !SHA.test(ticket.fixture.ref)) e.add(`${path}/fixture/ref`, 'must be a 40-character commit sha');++ if (!isStr(ticket.fixture.check_patch) || !ticket.fixture.check_patch.trim()) e.add(`${path}/fixture/check_patch`, 'must be the unified diff the check is applied from');++ if (ticket.fixture.check_timeout_ms != null && (!Number.isInteger(ticket.fixture.check_timeout_ms) || ticket.fixture.check_timeout_ms < 1000 || ticket.fixture.check_timeout_ms > 900000)) {++ e.add(`${path}/fixture/check_timeout_ms`, 'must be 1000-900000 milliseconds');++ }++ if (ticket.check && ticket.check.kind === 'command' && !ticket.fixture.check_patch.trim().startsWith('diff --git ') && !ticket.fixture.check_patch.trim().startsWith('--- ')) {++ e.add(`${path}/fixture/check_patch`, 'must be a unified diff (diff --git …)');++ }++ }++ if (!isObj(ticket.source)) e.add(`${path}/source`, 'must be an object');++ else {++ if (!isStr(ticket.source.repository) || !/^OpenVibers\/[A-Za-z0-9_.-]+$/.test(ticket.source.repository)) e.add(`${path}/source/repository`, 'must be an OpenVibers/Repo name');++ if (!Number.isInteger(ticket.source.pull_request) || ticket.source.pull_request < 1) e.add(`${path}/source/pull_request`, 'must be a positive pull request number');++ if (!isStr(ticket.source.merge_commit) || !SHA.test(ticket.source.merge_commit)) e.add(`${path}/source/merge_commit`, 'must be a 40-character commit sha');++ }++ return e;++}++++/** The whole suite file, checked before a single model call is made. */++function validateSuite(doc) {++ const e = new Problems();++ if (!isObj(doc)) return e.add('/', 'must be an object');++ if (doc.format !== FORMAT) e.add('/format', `must be ${FORMAT}`);++ if (!isStr(doc.suite) || !/^[a-z0-9][a-z0-9-]{1,79}$/.test(doc.suite)) e.add('/suite', 'must be 2-80 lowercase letters, digits and dashes');++ if (!isStr(doc.title) || !doc.title.trim() || doc.title.length > 160) e.add('/title', 'must be 1-160 characters');++ if (doc.kind != null && doc.kind !== KIND) e.add('/kind', `must be ${KIND}`);++ if (doc.models != null) {++ if (!Array.isArray(doc.models) || !doc.models.length) e.add('/models', 'must be a non-empty array');++ else doc.models.forEach((m, i) => {++ if (!isStr(m) || !m.trim()) e.add(`/models/${i}`, 'must be a model id');++ else if (!DEFAULT_MODELS.includes(m)) e.add(`/models/${i}`, `only ${DEFAULT_MODELS.join(' and ')} may run by default`);++ });++ }++ if (!Number.isInteger(doc.trials) || doc.trials < TRIAL_MIN || doc.trials > TRIAL_MAX) e.add('/trials', `must be an integer ${TRIAL_MIN}-${TRIAL_MAX}`);++ if (!Array.isArray(doc.tickets) || !doc.tickets.length) e.add('/tickets', 'must be a non-empty array');++ else {++ const ids = new Set();++ doc.tickets.forEach((ticket, i) => {++ const one = validateTicket(ticket, `/tickets/${i}`);++ for (const p of one.list) e.add(p.path, p.message);++ if (isObj(ticket) && isStr(ticket.id)) {++ if (ids.has(ticket.id)) e.add(`/tickets/${i}/id`, 'is used twice');++ ids.add(ticket.id);++ }++ });++ }++ return { valid: e.valid, errors: e.list };++}++++/** The benchmark definition a suite freezes: one task per ticket, the check kept as the task's expected result. */++function benchmarkDefinition(suite, { owner = 'openvibe', visibility = 'unlisted' } = {}) {++ const tickets = Array.isArray(suite.tickets) ? suite.tickets : [];++ return {++ owner,++ title: String(suite.title || suite.suite || 'Ticket suite').slice(0, 160),++ description: String(suite.description || 'A whole ticket: a goal and its check, scored over repeated trials.').slice(0, 4000),++ kind: KIND,++ tags: ['tickets', 'gate', `suite:${suite.suite}`],++ visibility,++ tasks: tickets.map((t) => ({++ id: t.id,++ prompt: String(t.goal).slice(0, 8000),++ // The check patch is withheld from readers: only the executor (bench.run.publish) and the owner see it.++ holdout: true,++ files: [{++ kind: 'codes.ticket',++ repo: t.fixture.repo,++ ref: t.fixture.ref,++ goal: t.goal,++ check: t.check,++ budget: t.budget || {},++ no_progress_limit: t.no_progress_limit == null ? 5 : t.no_progress_limit,++ }],++ expected: {++ passed: true,++ check: t.check,++ check_patch: t.fixture.check_patch,++ check_timeout_ms: t.fixture.check_timeout_ms == null ? 180000 : t.fixture.check_timeout_ms,++ source: t.source,++ },++ })),++ judges: [{ type: 'test_suite', version: 'codes.ticket@1', config: { pass_score: 100, trials: suite.trials, check: 'the task check exits 0 after its test patch is applied' } }],++ scoring: { method: 'pass_rate', scale: { min: 0, max: 100 }, aggregate: 'mean' },++ limits: { ms: tickets.reduce((n, t) => n + num(t.fixture && t.fixture.check_timeout_ms, 180000), 0) },++ matrix: { models: Array.isArray(suite.models) && suite.models.length ? [...suite.models] : [...DEFAULT_MODELS], harnesses: ['openvibe-codes'] },++ };++}++++/** One ticket's trials: how many ran, how many passed, and whether every one passed. */++function scoreTicket(trials) {++ const list = Array.isArray(trials) ? trials.filter((t) => t && t.score != null) : [];++ const n = list.length;++ const passes = list.filter((t) => num(t.score) > 0).length;++ return {++ trials: n,++ passes,++ pass_at_1: n ? passes / n : null,++ pass_at_k: n ? passes === n : false,++ cost_usd: Math.round(list.reduce((s, t) => s + num(t.cost_usd), 0) * 1e6) / 1e6,++ ms: list.reduce((s, t) => s + num(t.ms), 0),++ tokens: list.reduce((s, t) => s + num(t.tokens), 0),++ };++}++++/**++ * The gate's table: one row per model and effort, from runs whose tasks are ticket results (100 = the check++ * passed, 0 = it did not).++ *++ * pass@1 the share of all (ticket, trial) pairs that passed++ * pass^k the share of tickets whose every one of k trials passed (the customer-facing number: a 75%++ * per-trial agent passes three in a row about 42% of the time)++ * cost per solved every trial's cost, divided by the tickets solved at least once, so a flaky ticket is++ * charged for its failures and can be told from a model that never solves it++ *++ * `k` is the smallest number of trials any ticket got; a ticket with fewer trials than k cannot pass^k. A group++ * with fewer than three trials says so (`insufficient_trials`) instead of pretending to be a gate result.++ */++function scoreRuns(runs, tasksByRun, { effort = null } = {}) {++ const groups = new Map();++ for (const run of runs) {++ if (!run) continue;++ const runEffort = run.settings && run.settings.effort != null ? String(run.settings.effort) : null;++ if (effort != null && String(effort) !== (runEffort == null ? 'default' : runEffort)) continue;++ const key = `${String(run.model)}\u0000${runEffort == null ? 'default' : runEffort}`;++ if (!groups.has(key)) groups.set(key, { model: String(run.model), effort: runEffort, trialsByTicket: new Map(), runs: 0, cost_usd: 0, ms: 0, tokens: 0, session: run.session || null });++ const group = groups.get(key);++ group.runs++;++ group.cost_usd += num(run.cost_usd);++ group.ms += num(run.ms);++ group.tokens += num(run.tokens);++ if (!group.last_run || String(run.created_at || '') > group.last_run) group.last_run = String(run.created_at || '');++ for (const task of tasksByRun.get(run.id) || []) {++ if (task.score == null) continue;++ const id = String(task.task_id);++ if (!group.trialsByTicket.has(id)) group.trialsByTicket.set(id, []);++ group.trialsByTicket.get(id).push({ score: num(task.score), cost_usd: num(task.cost_usd), ms: num(task.ms), tokens: num(task.tokens), run: run.id });++ }++ }++ const entries = [];++ for (const group of groups.values()) {++ const perTicket = [...group.trialsByTicket.entries()].map(([id, trials]) => ({ task_id: id, ...scoreTicket(trials) }));++ const k = perTicket.reduce((n, t) => Math.min(n, t.trials), Infinity);++ const kk = Number.isFinite(k) ? k : 0;++ const totalPairs = perTicket.reduce((n, t) => n + t.trials, 0);++ const passedPairs = perTicket.reduce((n, t) => n + t.passes, 0);++ const fullyPassed = perTicket.filter((t) => t.trials >= kk && t.pass_at_k).length;++ const solved = perTicket.filter((t) => t.passes > 0).length;++ entries.push({++ model: group.model,++ effort: group.effort,++ runs: group.runs,++ trials: group.runs,++ task_trials: totalPairs,++ tickets: perTicket.length,++ k: kk,++ pass_at_1: totalPairs ? Math.round((passedPairs / totalPairs) * 1e6) / 1e6 : null,++ pass_at_k: perTicket.length ? Math.round((fullyPassed / perTicket.length) * 1e6) / 1e6 : null,++ solved,++ cost_usd: Math.round(group.cost_usd * 1e6) / 1e6,++ cost_per_solved: solved ? Math.round((group.cost_usd / solved) * 1e6) / 1e6 : null,++ ms: group.ms,++ tokens: group.tokens,++ insufficient_trials: kk < TRIAL_MIN,++ per_ticket: perTicket.map((t) => ({++ task_id: t.task_id,++ trials: t.trials,++ passes: t.passes,++ pass_at_1: t.pass_at_1 == null ? null : Math.round(t.pass_at_1 * 1e6) / 1e6,++ pass_at_k: t.pass_at_k,++ cost_usd: t.cost_usd,++ })).sort((a, b) => a.task_id.localeCompare(b.task_id)),++ last_run: group.last_run || null,++ });++ }++ entries.sort((a, b) => (b.pass_at_1 == null ? -1 : b.pass_at_1) - (a.pass_at_1 == null ? -1 : a.pass_at_1)++ || (a.cost_per_solved == null ? Infinity : a.cost_per_solved) - (b.cost_per_solved == null ? Infinity : b.cost_per_solved)++ || a.model.localeCompare(b.model)++ || String(a.effort || '').localeCompare(String(b.effort || '')));++ return { metric: 'tickets', entries };++}++++module.exports = {++ FORMAT, KIND, TRIAL_MIN, TRIAL_MAX, DEFAULT_MODELS, CHECK_KINDS,++ validateGoal, validateTicket, validateSuite, benchmarkDefinition, scoreTicket, scoreRuns,++};@@@ -366,8 +366,8 @@ function createRegistryApi(ctx) {const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');const metric = req.query.metric ? String(req.query.metric) : 'score';-- if (!['score', 'value', 'speed', 'consistency', 'horizon'].includes(metric)) {-- return invalid(res, 'metric is score, value, speed, consistency or horizon');++ if (!['score', 'value', 'speed', 'consistency', 'horizon', 'tickets', 'pass_k', 'pass@k', 'pass^k'].includes(metric)) {++ return invalid(res, 'metric is score, value, speed, consistency, horizon or tickets');}try {const bm = await registry.getBenchmark(benchmark.split('@')[0]);@@@ -378,6 +378,7 @@ function createRegistryApi(ctx) {harness: req.query.harness ? String(req.query.harness) : null,since: req.query.since ? String(req.query.since) : null,includeSubmitted: String(req.query.include_submitted || '') === 'true',++ effort: req.query.effort ? String(req.query.effort).slice(0, 40) : null,});if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');res.set('Cache-Control', bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');@@@ -21,6 +21,19 @@ const KINDS = Object.freeze([{ id: 'vision.pair', room: 'vision', entry: 'model_answer', judge: 'human_pairwise', name: 'Vision head-to-head', metrics: ['elo', 'cost_usd'], schema: { entry: 'model_answer', payload: { answer: 'string' } } },{ id: 'codes.patch', room: 'codes', entry: 'code_patch', judge: 'test_suite', name: 'Code patches', metrics: ['pass_rate', 'cost_usd'], schema: { entry: 'code_patch', payload: { patch: 'string' }, prompt: { repository: 'string', failing_test: 'string', constraints: 'string' }, rubric: ['Apply the patch in an isolated checkout', 'Run the pinned failing and regression tests', 'Pass only when all tests pass'], seeds: [{ repository: 'sample-js', failing_test: 'sum([]) should return 0', constraints: 'Change only sum.js' }] } },{ id: 'codes.review', room: 'codes', entry: 'review', judge: 'human_pairwise', name: 'Code diff review', metrics: ['elo', 'cost_usd'], schema: { entry: 'review', payload: { review: 'string' }, prompt: { diff: 'string', intent: 'string' }, rubric: ['Find specific correctness or security defects', 'Cite the affected lines and explain impact', 'Avoid speculative or duplicate findings'], seeds: [{ diff: 'A cache key changes from userId to a constant', intent: 'Review for cross-user data exposure' }] } },++ // AL8, the gate (plan 2026-11-10): one whole ticket run over 3 to 5 trials. The task carries AL1's shape —++ // goal and check, a budget and a no-progress limit — and the check is applied only after the worker stops, so a++ // test added by the merged pull request grades the work. A run is one trial; pass@1, pass^k and cost per++ // solved are read from GET /api/v1/rankings?benchmark=…&metric=tickets[&effort=…].++ { id: 'codes.ticket', room: 'codes', entry: 'ticket_result', judge: 'test_suite', name: 'Whole tickets', metrics: ['pass_at_1', 'pass_at_k', 'cost_per_solved'],++ schema: {++ trials: { min: 3, max: 5 },++ goal: { goal: 'string', check: { kind: 'command|url|rubric', value: 'string' }, budget: { tokens: 'integer', turns: 'integer', active_minutes: 'integer' }, no_progress_limit: 'integer' },++ entry: 'ticket_result', payload: { passed: 'boolean', check_output: 'string' },++ fixture: { repo: 'https://github.com/OpenVibers/<Repo>', ref: '40-character base commit', check_patch: 'unified diff of the pull request tests' },++ scoring: 'pass@1 is the share of trials that passed; pass^k is the share of tickets whose every one of k trials passed; cost per solved is every trial cost divided by the tickets solved at least once',++ seeds: [{ goal: 'The failing test passes.', check: { kind: 'command', value: 'node test/example.test.js' }, budget: { turns: 20, active_minutes: 5 } }],++ } },{ id: 'surf.desk', room: 'surf', entry: 'agent_on_desk', judge: 'human_pairwise', name: 'Agents on desks', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'agent_on_desk', payload: { desk: 'object' } } },{ id: 'chat.reply', room: 'chat', entry: 'conversation_turn', judge: 'human_pairwise', name: 'Chat replies', metrics: ['elo', 'cost_usd'], schema: { entry: 'conversation_turn', payload: { reply: 'string' }, prompt: { conversation: 'array', latest_user_message: 'string' }, rubric: ['Answer the latest request accurately', 'Use prior turns when relevant', 'Be clear and avoid invented facts'], seeds: [{ conversation: [{ role: 'user', content: 'I have 20 minutes for dinner.' }], latest_user_message: 'Suggest a quick vegetarian meal.' }] } },{ id: 'chat.arena', room: 'exchange', entry: 'model_answer', judge: 'human_pairwise', name: 'Exchange arena', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'model_answer', payload: { answer: 'string' }, prompt: { text: 'string' } } },nnew file mode 100644@@@ -0,0 +1,207 @@++'use strict';++++/**++ * The AL8 gate (plan 2026-11-10): the codes.ticket kind, the suite file, the trial maths and the results API that++ * reads pass@1, pass^k and cost per solved by model and effort. The runner's money rule is tested here too: it++ * makes no call without `run: true` and refuses every model but deepseek-flash and the local one.++ */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const { boot, check, done } = require('./helpers/boot');++const tickets = require('../server/bench/tickets');++const runner = require('../server/bench/ticket-runner');++const cli = require('../scripts/ticket-suite');++const executor = require('../scripts/ticket-executor');++const { createRegistryApi } = require('../server/http/registry');++const { kindById } = require('../server/kinds');++++/** One ticket in the committed shape, with a one-line check patch. */++function ticketDoc(id, prompt = 'The example test passes.') {++ return {++ id,++ goal: prompt,++ check: { kind: 'command', value: `node test/${id}.test.js` },++ budget: { tokens: 20000, turns: 10, active_minutes: 3 },++ no_progress_limit: 3,++ fixture: { repo: 'https://github.com/OpenVibers/OpenVibe.Bench', ref: '0'.repeat(40), check_patch: 'diff --git a/test/x b/test/x\n', check_timeout_ms: 60000 },++ source: { repository: 'OpenVibers/OpenVibe.Bench', pull_request: 1, merge_commit: '0'.repeat(40) },++ };++}++++/** A two-ticket suite in the committed shape. */++function kitchen(over = {}) {++ return { format: tickets.FORMAT, suite: 'kitchen-suite', title: 'Kitchen suite', kind: 'codes.ticket', models: ['deepseek-flash'], trials: 3, tickets: [ticketDoc('alpha'), ticketDoc('beta')], ...over };++}++++(async () => {++ const t = await boot();++ const writer = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.benchmark.write', 'bench.run.publish'] });++++ await check('the results API is the registry route and codes.ticket is a kind', async () => {++ assert.strictEqual(typeof createRegistryApi, 'function', 'the registry route is mounted');++ const kind = kindById('codes.ticket');++ assert.ok(kind, 'the kind is registered');++ assert.strictEqual(kind.judge, 'test_suite');++ assert.deepStrictEqual(kind.metrics, ['pass_at_1', 'pass_at_k', 'cost_per_solved']);++ assert.strictEqual(kind.schema.trials.min, 3);++ assert.strictEqual(kind.schema.trials.max, 5);++ assert.ok(kind.schema.goal.check, 'the AL1 goal and check shape');++ });++++ await check('a suite is checked before it runs', async () => {++ assert.ok(tickets.validateSuite(kitchen()).valid, 'the kitchen suite is valid');++ assert.ok(!tickets.validateSuite(kitchen({ trials: 2 })).valid, 'two trials is not a gate');++ assert.ok(!tickets.validateSuite(kitchen({ trials: 6 })).valid, 'six trials is not a gate');++ assert.ok(!tickets.validateSuite(kitchen({ models: ['claude-opus-4'] })).valid, 'a dear model is not the default gate');++ const badCheck = kitchen();++ badCheck.tickets[0].check = { kind: 'vibes', value: 'it feels right' };++ assert.ok(!tickets.validateSuite(badCheck).valid, 'a check has a known kind');++ const twice = kitchen();++ twice.tickets[1].id = twice.tickets[0].id;++ assert.ok(!tickets.validateSuite(twice).valid, 'two tickets cannot share an id');++ });++++ await check('a suite in the committed fixture is valid and holds twelve real tickets', async () => {++ const file = path.join(__dirname, '..', 'fixtures', 'codes.ticket', 'openvibe-merged-2026-11.json');++ const suite = JSON.parse(fs.readFileSync(file, 'utf8'));++ const checked = tickets.validateSuite(suite);++ assert.ok(checked.valid, checked.detail);++ assert.strictEqual(suite.tickets.length, 12);++ for (const ticket of suite.tickets) {++ assert.strictEqual(ticket.check.kind, 'command');++ assert.match(ticket.fixture.check_patch, /^diff --git /);++ assert.match(ticket.source.repository, /^OpenVibers\//);++ }++ });++++ await check('the benchmark task carries AL1 goal, check, budget and the grading patch', async () => {++ const definition = tickets.benchmarkDefinition(kitchen());++ assert.strictEqual(definition.kind, 'codes.ticket');++ assert.strictEqual(definition.tasks.length, 2);++ const task = definition.tasks[0];++ assert.strictEqual(task.prompt, 'The example test passes.');++ assert.strictEqual(task.holdout, true);++ assert.strictEqual(task.files[0].goal, 'The example test passes.');++ assert.strictEqual(task.files[0].check.kind, 'command');++ assert.strictEqual(task.files[0].budget.turns, 10);++ assert.ok(task.expected.check_patch.startsWith('diff --git '));++ });++++ await check('the runner makes no call without --run, and refuses a dear model before one', async () => {++ let calls = 0;++ const plan = await runner.runSuite({ suite: kitchen(), execute: () => { calls++; } });++ assert.strictEqual(plan.ok, true);++ assert.strictEqual(plan.ran, false);++ assert.strictEqual(calls, 0);++ const refused = await runner.runSuite({ suite: kitchen(), run: true, models: ['claude-opus-4'], execute: () => { calls++; } });++ assert.strictEqual(refused.ok, false);++ assert.strictEqual(refused.code, 'bench.ticket_model_not_allowed');++ assert.strictEqual(calls, 0);++ const short = await runner.runSuite({ suite: kitchen(), run: true, trials: 2, execute: () => { calls++; } });++ assert.strictEqual(short.ok, false);++ assert.strictEqual(short.code, 'bench.ticket_trials');++ assert.strictEqual(calls, 0);++ });++++ await check('the executor prompt carries the goal and check but never the test patch', async () => {++ const ticket = kitchen().tickets[0];++ const prompt = executor.promptFor({ ticket, workspace: '/tmp/example' });++ assert.ok(prompt.includes(ticket.goal), 'the goal is in the prompt');++ assert.ok(prompt.includes(ticket.check.value), 'the check is in the prompt');++ assert.ok(!prompt.includes(ticket.fixture.check_patch.split('\n')[0]), 'the test patch is never shown to the agent');++ assert.match(prompt, /Do not add or edit anything under test\//);++ });++++ await check('the command line prints the plan and parses the safe flags', async () => {++ const args = cli.parseArgs(['--run', '--trials', '3', '--models', 'deepseek-flash,local', '--effort', 'high']);++ assert.strictEqual(args.run, true);++ assert.strictEqual(args.trials, '3');++ assert.strictEqual(args.models, 'deepseek-flash,local');++ const lines = cli.tableLines([{ model: 'deepseek-flash', effort: 'high', k: 3, pass_at_1: 0.888889, pass_at_k: 0.666667, cost_per_solved: 0.298567 }]);++ assert.match(lines[0], /pass@1 0\.889/);++ assert.match(lines[0], /pass\^3 0\.667/);++ assert.match(lines[0], /cost per solved \$0\.2986/);++ });++++ await check('three trials make a table of pass@1, pass^3 and cost per solved', async () => {++ const out = await runner.runSuite({++ suite: kitchen(), run: true, trials: 3, models: ['deepseek-flash'],++ execute: ({ ticket, trial, model }) => ({ passed: ticket.id === 'alpha' || trial === 1, cost_usd: model === 'deepseek-flash' ? 0.001 : 0, ms: 5, tokens: 7, check_output: 'ok' }),++ });++ assert.strictEqual(out.ok, true);++ assert.strictEqual(out.runs.length, 3, 'one run per trial');++ assert.strictEqual(out.runs[0].settings.effort, 'high');++ assert.strictEqual(out.runs[0].provider, 'deepseek');++ assert.strictEqual(out.runs[0].tasks.length, 2);++ const row = out.table.find((e) => e.model === 'deepseek-flash' && e.effort === 'high');++ assert.ok(row, 'the table has the model and effort');++ assert.strictEqual(row.k, 3);++ assert.strictEqual(row.pass_at_1, 0.666667);++ assert.strictEqual(row.pass_at_k, 0.5);++ assert.strictEqual(row.cost_usd, 0.006);++ assert.strictEqual(row.cost_per_solved, 0.003);++ assert.strictEqual(row.insufficient_trials, false);++ });++++ await check('pass@1 and pass^k are read from the results API, by model and effort', async () => {++ const definition = tickets.benchmarkDefinition(kitchen({ tickets: ['alpha', 'beta', 'gamma', 'delta'].map((id) => ticketDoc(id)) }));++ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition });++ assert.strictEqual(created.status, 201, created.text);++ const id = created.json().benchmark.id;++ const frozen = await t.get(`/api/v1/benchmarks/${id}/versions`, { bearer: writer, json: {} });++ assert.strictEqual(frozen.status, 201, frozen.text);++ const benchmark = `${id}@${frozen.json().version.version}`;++ const trial = (scores, { effort = 'high' } = {}) => {++ const tasks = scores.map(([task_id, score]) => ({ task_id, score, cost_usd: 0.001, ms: 100, tokens: 10, artifacts: [] }));++ return {++ benchmark, model: 'deepseek-flash', provider: 'deepseek',++ route: { requested: 'deepseek-flash', ran: 'deepseek-flash', effort },++ harness: { name: 'openvibe-codes', version: '0.4.0' }, settings: { effort },++ executor: { kind: 'codes', id: 'test' }, session: 'gate-test',++ tasks, score: tasks.reduce((n, x) => n + x.score, 0) / tasks.length,++ cost_usd: tasks.reduce((n, x) => n + x.cost_usd, 0), ms: tasks.length * 100, tokens: tasks.length * 10,++ };++ };++ // alpha passes every trial, beta two of three, gamma one, delta none.++ const trials = [++ trial([['alpha', 100], ['beta', 100], ['gamma', 100], ['delta', 0]]),++ trial([['alpha', 100], ['beta', 100], ['gamma', 0], ['delta', 0]]),++ trial([['alpha', 100], ['beta', 0], ['gamma', 0], ['delta', 0]]),++ ];++ for (const body of trials) {++ const made = await t.get('/api/v1/runs', { bearer: writer, json: body });++ assert.strictEqual(made.status, 201, made.text);++ }++ // A one-trial low-effort run is read, but it is not a gate result.++ const low = await t.get('/api/v1/runs', { bearer: writer, json: trial([['alpha', 100]], { effort: 'low' }) });++ assert.strictEqual(low.status, 201, low.text);++ const read = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=tickets&effort=high`, { bearer: writer });++ assert.strictEqual(read.status, 200, read.text);++ const row = read.json().rankings.entries.find((e) => e.model === 'deepseek-flash' && e.effort === 'high');++ assert.ok(row, 'the entry names the model and effort');++ assert.strictEqual(row.k, 3);++ assert.strictEqual(row.trials, 3, 'three trial runs');++ assert.strictEqual(row.task_trials, 12, 'four tickets x three trials');++ assert.strictEqual(row.tickets, 4);++ assert.strictEqual(row.pass_at_1, 0.5);++ assert.strictEqual(row.pass_at_k, 0.25);++ assert.ok(Math.abs(row.cost_per_solved - 0.004) < 1e-9, 'every trial cost, divided by the three solved tickets');++ assert.strictEqual(row.per_ticket.find((x) => x.task_id === 'alpha').pass_at_k, true);++ assert.strictEqual(row.per_ticket.find((x) => x.task_id === 'delta').pass_at_k, false);++ const alias = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=pass_k`, { bearer: writer });++ assert.strictEqual(alias.status, 200, alias.text);++ assert.strictEqual(alias.json().rankings.metric, 'tickets');++ const other = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=tickets&effort=low`, { bearer: writer });++ assert.strictEqual(other.status, 200, other.text);++ assert.strictEqual(other.json().rankings.entries.length, 1, 'the low-effort group is its own row');++ assert.strictEqual(other.json().rankings.entries[0].k, 1);++ assert.strictEqual(other.json().rankings.entries[0].insufficient_trials, true, 'one trial is not a gate result');++ const wrong = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=sideways`, { bearer: writer });++ assert.strictEqual(wrong.status, 400, wrong.text);++ });++++ await check('stop', async () => { await t.close(); });++++ done();++})();
@@ -2,6 +2,16 @@
@@ -2,6 +2,16 @@
## Unreleased
## Unreleased
+- **Whole tickets, the gate (AL8).** A new kind, `codes.ticket`, scores a whole ticket — a goal and a typed check
+ in `agent.goal@1`'s shape, with a token/turn/minute budget and a no-progress limit — over 3 to 5 trials per model
+ and effort. A run is one trial; a task's score is 100 when the check exited 0 and 0 when it did not, and the
+ check's test patch is applied only after the worker stops, so the merged pull request's own test grades the work.
+ `GET /api/v1/rankings?benchmark=…&metric=tickets&effort=…` (also `pass_k`, `pass@k`) reports pass@1, pass^k (every
+ one of k trials passed) and cost per solved by model and effort, with the per-ticket rows. The first suite
+ (`fixtures/codes.ticket/openvibe-merged-2026-11.json`) holds twelve small tickets from merged OpenVibe pull
+ requests across Bench, Codes, AI, Actor, Exchange, Report and Network; each goal is the change in words and each
+ check is the pull request's own test. `scripts/ticket-suite.js` prints the plan and runs only with `--run`; the
+ default models are `deepseek-flash` and `local`, and nothing is scheduled.
- **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, the
- **Provider fidelity (B5).** `GET /api/v1/fidelity` publishes, for every measured open-weight model offer, the
score of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each,
score of the fixed public objective set (`fidelity-2026-11-09.v1`: ten objective items, a deterministic check each,
no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the
no judge), with n, its 95% Wilson interval and the model it was compared on. `POST /api/v1/fidelity/plan` is the
@@ -38,9 +38,38 @@ run under it, with no person id. Nothing private or unlisted is ever read there.
@@ -38,9 +38,38 @@ run under it, with no person id. Nothing private or unlisted is ever read there.
| `POST /api/v1/runs` | a person, or `bench.run.publish` | Publish a run. A service token with `bench.run.publish` makes a **verified** run; a person's token makes a **submitted** one, badged and out of the default rankings. |
| `POST /api/v1/runs` | a person, or `bench.run.publish` | Publish a run. A service token with `bench.run.publish` makes a **verified** run; a person's token makes a **submitted** one, badged and out of the default rankings. |
| `GET /api/v1/runs/:id` | anyone | One run with its per-task results. `run.document` is the published `bench.run@1` document. |
| `GET /api/v1/runs/:id` | anyone | One run with its per-task results. `run.document` is the published `bench.run@1` document. |
| `GET /api/v1/compare?runs=a,b,c` | anyone | Up to four runs side by side, per task. |
| `GET /api/v1/compare?runs=a,b,c` | anyone | Up to four runs side by side, per task. |
-| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency&harness=&since=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. |
+| `GET /api/v1/rankings?benchmark=&metric=score\|value\|speed\|consistency\|tickets&harness=&since=&effort=` | anyone | The rankings for a benchmark. Verified runs only, unless `include_submitted=true`. `metric=tickets` (also `pass_k`, `pass@k`) is the AL8 gate: pass@1, pass^k and cost per solved by model and effort (`effort` filters one). |
| `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |
| `GET /api/v1/percentile?benchmark=&model=&score=` | anyone | Where a score sits among that model's runs. |
+## Whole tickets (AL8, the gate)
+
+`codes.ticket` scores a whole ticket: a goal and its check, run over 3 to 5 trials on each model and effort. A run
+is one trial; each task is one ticket, and its score is 100 when the check passed and 0 when it did not. The
+results API answers the gate's table:
+
+```
+GET /api/v1/rankings?benchmark=bmk_…@1&metric=tickets&effort=high
+{
+ "rankings": {
+ "metric": "tickets",
+ "entries": [{
+ "model": "deepseek-flash", "effort": "high", "k": 3, "trials": 3, "task_trials": 9, "tickets": 3,
+ "pass_at_1": 0.888889, // share of all (ticket, trial) pairs that passed
+ "pass_at_k": 0.666667, // share of tickets whose every one of k trials passed
+ "cost_usd": 0.31, "solved": 3,
+ "cost_per_solved": 0.103333, // every trial's cost ÷ the tickets solved at least once
+ "insufficient_trials": false,
+ "per_ticket": [{ "task_id": "…", "trials": 3, "passes": 3, "pass_at_1": 1, "pass_at_k": true, "cost_usd": 0.1 }]
+ }]
+ }
+}
+```
+
+A group with fewer than three trials sets `insufficient_trials: true`; it is not a gate result. The suite file
+(`bench.codes-ticket-suite@1`) and the runner are in `fixtures/codes.ticket/` and `scripts/ticket-suite.js`. A suite
+runs only when a person passes `--run`; the default models are `deepseek-flash` and `local`, and nothing is
+scheduled.
+
## Voting without signing in
## Voting without signing in
A person who has not signed in may still judge a pairwise round (`human_pairwise` kinds only). There are two ways in,
A person who has not signed in may still judge a pairwise round (`human_pairwise` kinds only). There are two ways in,
new file mode 100644
new file mode 100644
@@ -0,0 +1,326 @@
@@ -0,0 +1,326 @@
+{
+ "format": "bench.codes-ticket-suite@1",
+ "suite": "openvibe-merged-2026-11",
+ "title": "Twelve merged OpenVibe tickets",
+ "description": "The first AL8 gate suite: twelve small tickets from merged OpenVibe pull requests across Bench, Codes, AI, Actor, Exchange, Report and Network. Each goal is the merged change in words; each check is the pull request's own test, applied only after the worker stops. Base is the merge commit's parent, so the test fails before the change and passes after it.",
+ "kind": "codes.ticket",
+ "models": [
+ "deepseek-flash",
+ "local"
+ ],
+ "trials": 3,
+ "tickets": [
+ {
+ "id": "bench-158-navbar-token",
+ "goal": "The shared navbar can open Messages in Dock Chats in place: the signed-in person's verified Network access token is available through a same-origin, no-store GET /auth/navbar-token route, while /auth/me stays token-free.",
+ "check": {
+ "kind": "command",
+ "value": "node test/sso.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 30,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Bench",
+ "ref": "b7ca03eea0f20dc933da1b9c8c8035c2d8f2f52c",
+ "check_patch": "diff --git a/test/sso.test.js b/test/sso.test.js\nindex 44dd1bf..d9ece5a 100644\n--- a/test/sso.test.js\n+++ b/test/sso.test.js\n@@ -53,6 +53,11 @@ const cookieJar = (res) => res.headers.getSetCookie().map((c) => c.split(';')[0]\n const me = await t.get('/auth/me', { cookie: `ov_token=${at}` });\n assert.strictEqual(me.json().user.username, 'kim');\n assert.ok(!me.text.includes(at), '/auth/me never returns the token');\n+ const nav = await t.get('/auth/navbar-token', { cookie: `ov_token=${at}` });\n+ assert.strictEqual(nav.status, 200);\n+ assert.strictEqual(nav.json().token, at, 'the Dock receives the verified Network token');\n+ assert.strictEqual(nav.headers.get('cache-control'), 'private, no-store');\n+ assert.strictEqual((await t.get('/auth/navbar-token')).status, 401, 'a guest receives no token');\n });\n \n await check('the old cookie is still the session, so nobody is signed out by the move', async () => {\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Bench",
+ "pull_request": 158,
+ "merge_commit": "3b2a91896b5d384e3f685e29e427b324ae93db5f",
+ "merged_at": "2026-11-09T06:55:42Z"
+ }
+ },
+ {
+ "id": "codes-351-prompt-prefix",
+ "goal": "Two task runs through the coding agent's tool loop send the same model, system message and tool schemas, in a fixed order, so a provider caches one prompt prefix; only the user message differs between tasks.",
+ "check": {
+ "kind": "command",
+ "value": "node test/harness-prompt-prefix.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Codes",
+ "ref": "19721715ed45fd669ca6e7eb5d9737c2fa9dc8f8",
+ "check_patch": "diff --git a/test/harness-prompt-prefix.test.js b/test/harness-prompt-prefix.test.js\nnew file mode 100644\nindex 0000000..bb0ba2d\n--- /dev/null\n+++ b/test/harness-prompt-prefix.test.js\n@@ -0,0 +1,73 @@\n+'use strict';\n+/**\n+ * The prompt prefix is byte-identical between two different tasks (stage S3, cache-stable prompts): the system message\n+ * and the tool schemas never carry the task, tools come in a fixed name order, and only the user message differs.\n+ */\n+const assert = require('assert');\n+const fs = require('fs');\n+const os = require('os');\n+const path = require('path');\n+const http = require('http');\n+const { createHarness } = require('../harness');\n+const { createStore } = require('../harness/sessions');\n+const { SYSTEM } = require('../harness/adapters/api');\n+const { check, done } = require('./helpers/boot');\n+\n+/** A chat-completions server that records each request and always answers with a plain reply. */\n+async function mockApi() {\n+ const requests = [];\n+ const server = http.createServer((req, res) => {\n+ let body = '';\n+ req.on('data', (d) => { body += d; });\n+ req.on('end', () => {\n+ requests.push(JSON.parse(body || '{}'));\n+ res.writeHead(200, { 'content-type': 'application/json' });\n+ res.end(JSON.stringify({ choices: [{ message: { role: 'assistant', content: 'done' } }], usage: { prompt_tokens: 1, completion_tokens: 1 } }));\n+ });\n+ });\n+ await new Promise((r) => server.listen(0, '127.0.0.1', r));\n+ return { url: `http://127.0.0.1:${server.address().port}`, requests, close: () => new Promise((r) => server.close(r)) };\n+}\n+\n+const collect = async (run) => { const out = []; for await (const e of run) out.push(e); return out; };\n+/** The stable prefix of one request as it is serialized: the model, the system message and the tools, in that order. */\n+const prefixOf = (body) => JSON.stringify({ model: body.model, messages: [body.messages[0]], tools: body.tools, tool_choice: body.tool_choice });\n+\n+(async () => {\n+ const api = await mockApi();\n+ try {\n+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codes-prefix-'));\n+ const work = path.join(dir, 'work');\n+ fs.mkdirSync(work, { recursive: true });\n+ // An empty environment: the harness reads its base url and model from here, never from the machine's own.\n+ const env = { PATH: '/nonexistent', OPENVIBE_CODES_HOME: path.join(dir, 'home'), OPENVIBE_CODES_BASE_URL: api.url, OPENVIBE_CODES_MODEL: 'tiny', OPENVIBE_CODES_API_KEY: 'k-123' };\n+ const h = createHarness({ env, store: createStore({ env }) });\n+\n+ await check('the system message never carries the task or a date', async () => {\n+ assert.ok(SYSTEM.trim().length > 100);\n+ assert.ok(!/\\d{4}-\\d{2}-\\d{2}/.test(SYSTEM), 'no date in the stable system message');\n+ });\n+\n+ await check('two tasks send the same model, system message and tools; only the prompt differs', async () => {\n+ await collect(h.run({ prompt: 'rename name to count', agent: 'openai-compatible', cwd: work, permission: 'edit', handoff: false }));\n+ await collect(h.run({ prompt: 'a much longer second task about the same repository', agent: 'openai-compatible', cwd: work, permission: 'edit', handoff: false }));\n+ assert.strictEqual(api.requests.length, 2);\n+ const [a, b] = api.requests;\n+ assert.strictEqual(prefixOf(a), prefixOf(b), 'the serialized prefix is byte-identical');\n+ assert.strictEqual(a.messages[0].role, 'system');\n+ assert.strictEqual(a.messages[1].role, 'user');\n+ assert.strictEqual(a.messages[1].content, 'rename name to count');\n+ assert.strictEqual(b.messages[1].content, 'a much longer second task about the same repository');\n+ });\n+\n+ await check('tool schemas serialize in a fixed name order', async () => {\n+ const names = api.requests[0].tools.map((t) => t.function.name);\n+ assert.deepStrictEqual(names, [...names].sort(), 'tools are offered in name order');\n+ assert.deepStrictEqual(names, ['edit_file', 'grep', 'list_dir', 'read_file', 'write_file']);\n+ assert.strictEqual(JSON.stringify(api.requests[0].tools), JSON.stringify(api.requests[1].tools));\n+ });\n+ } finally {\n+ await api.close();\n+ done();\n+ }\n+})();\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Codes",
+ "pull_request": 351,
+ "merge_commit": "52381fb2bf6267b844536413446a00d6f4a4aa43",
+ "merged_at": "2026-11-09T21:59:35Z"
+ }
+ },
+ {
+ "id": "codes-314-who-leads",
+ "goal": "The composer's \"Who leads\" choice names the person's own saved setting and is the option selected; a person who never chose is not pinned to Opus when the Settings page is saved untouched.",
+ "check": {
+ "kind": "command",
+ "value": "node test/conductor.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Codes",
+ "ref": "c7320335b35d2998134ea7bc131aa921331cd5d9",
+ "check_patch": "diff --git a/test/conductor.test.js b/test/conductor.test.js\nindex d032e17..79ee775 100644\n--- a/test/conductor.test.js\n+++ b/test/conductor.test.js\n@@ -196,6 +196,12 @@ const DEVICE = 'dev_01J8Z3Q4R5S6T7V8W9X0Y1Z2N1';\n const res = await get('/conductor');\n assert.strictEqual(res.status, 200);\n assert.match(res.text, /<select name=\"who\" data-c4-model>/, 'who leads is a list');\n+ // The first option is the person's own setting, said by name, and it is the one selected: the composer never\n+ // claims a default the person changed, and no other leader is preselected.\n+ assert.match(res.text, /<select name=\"who\" data-c4-model><option value=\"\">Your setting: [^<]+<\\/option>/, 'the first choice names the setting');\n+ const whoList = res.text.split('<select name=\"who\" data-c4-model>')[1].split('</select>')[0];\n+ assert.ok(!/ selected/.test(whoList), 'nothing else is preselected');\n+ assert.ok(!res.text.includes(\"Today's choice\"), 'no \"Today\\'s choice\" wording');\n for (const label of ['Claude Code · Opus', 'Claude Code · Sonnet', 'Codex', 'OpenCode', 'DeepSeek']) assert.ok(res.text.includes(label), `${label} is offered`);\n assert.match(res.text, /data-c4-folder-pick/, 'the recent folders are a choice beside the typed path');\n assert.ok(res.text.includes(`data-machine=\"${NODE}\"`), 'this conductor’s folder is one, on its machine');\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Codes",
+ "pull_request": 314,
+ "merge_commit": "092d1952b27a5dfe3cb0edc22624f5cf6097e23e",
+ "merged_at": "2026-11-07T13:06:20Z"
+ }
+ },
+ {
+ "id": "ai-387-space-components",
+ "goal": "The space.app.build template accepts an optional components input (up to 20,000 characters) and, when present, puts it in the system prompt as the design.component@1 interface vocabulary with the rule that every major part carries data-ov-component; without it the prompt is exactly as before.",
+ "check": {
+ "kind": "command",
+ "value": "node test/space-workflows.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.AI",
+ "ref": "13ffb0f203b182c72960a3af8f5c37fefa53af0a",
+ "check_patch": "diff --git a/test/space-workflows.test.js b/test/space-workflows.test.js\nindex eea1494..a4509da 100644\n--- a/test/space-workflows.test.js\n+++ b/test/space-workflows.test.js\n@@ -23,6 +23,13 @@ t.test('the builder\\'s prompt: the SDK reference, and the current files when cha\n assert.match(sys, /OV\\.fn\\(name, body\\)/);\n assert.match(sys, /never request\\.user or request\\.ov/, 'the visitor is on ov, not on the request');\n assert.ok(!/There is no server code/.test(sys));\n+ assert.match(sys, /design\\.component@1/, 'the registry is named as the interface vocabulary');\n+ const withComponents = render(sys, { components: 'sheet — A modal task surface. Web: kit.sheet()' });\n+ assert.match(withComponents, /mark each major part with data-ov-component/);\n+ assert.match(withComponents, /sheet — A modal task surface/);\n+ assert.ok(!/\\{\\{#components\\}\\}/.test(withComponents), 'the section is rendered away');\n+ const prepWithRegistry = PREPARE['space.app.build']({ prompt: 'a pizza poll', components: 'sheet — A modal task surface.' });\n+ assert.strictEqual(prepWithRegistry.vars.components, 'sheet — A modal task surface.');\n const prep = PREPARE['space.app.build']({ prompt: 'a poll about pizza' });\n assert.ok(prep.jsonSchema && prep.jsonSchema.properties.files && !/maxLength|maxItems/.test(JSON.stringify(prep.jsonSchema)), 'the model answers in the uncapped, non-strict schema');\n const fresh = render(tpl('space.app.build').user_prompt, prep.vars);\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.AI",
+ "pull_request": 387,
+ "merge_commit": "5fd137763be08137d2c7e928dcce50ad1608c2f0",
+ "merged_at": "2026-11-09T21:48:50Z"
+ }
+ },
+ {
+ "id": "actor-390-prompt-prefix",
+ "goal": "Two assemblies of the agent's system prompt with different dates and plans share the whole rules prefix byte for byte, and the tool definitions serialize identically, because today's date, the plan, the ledger and the demoted-tools note come after the rules.",
+ "check": {
+ "kind": "command",
+ "value": "node test/runtime-prompt-prefix.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Actor",
+ "ref": "beb69b131c2dd20e30a5c04b1bfaf245408f14d7",
+ "check_patch": "diff --git a/test/runtime-prompt-prefix.test.js b/test/runtime-prompt-prefix.test.js\nnew file mode 100644\nindex 0000000..2694c55\n--- /dev/null\n+++ b/test/runtime-prompt-prefix.test.js\n@@ -0,0 +1,55 @@\n+'use strict';\n+/**\n+ * The prompt prefix stays byte-identical between tasks (stage S3, cache-stable prompts).\n+ *\n+ * - the system prompt keeps its rules first and today's date, the plan, the ledger and a demoted-tools note last,\n+ * so two tasks share the whole stable prefix and a provider's cache breakpoint never covers a date;\n+ * - the tool definitions serialize the same way on every assembly, so a cache is not lost to tool order.\n+ */\n+const fs = require('fs');\n+const os = require('os');\n+const path = require('path');\n+const assert = require('assert');\n+const { SYSTEM } = require('../server/runtime/loop');\n+const { createRegistry } = require('../server/runtime/tools');\n+\n+const results = [];\n+const ok = async (name, fn) => { try { await fn(); console.log(` ✓ ${name}`); } catch (e) { results.push(e); console.log(` ✗ ${name}\\n ${(e.stack || String(e)).split('\\n').slice(0, 8).join('\\n ')}`); } };\n+\n+function tempTools(files) {\n+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'actor-prefix-'));\n+ for (const [name, body] of Object.entries(files)) fs.writeFileSync(path.join(dir, name), body);\n+ return dir;\n+}\n+const tool = (name) => `module.exports = { name: ${JSON.stringify(name)}, description: ${JSON.stringify(`The ${name} tool.`)}, input: { type: 'object', properties: { q: { type: 'string' } } }, scope: 'public', run: async () => ({ text: 'ok' }) };\\n`;\n+\n+(async () => {\n+ await ok('two tasks share the whole rules prefix; the date and the plan come after it', async () => {\n+ const a = SYSTEM({ date: '2026-11-09', plan: '1. Look up the price' });\n+ const b = SYSTEM({ date: '2027-01-01', plan: '1. Compare two offers', ledger: 'Progress: 1 of 2', note: 'Offered last: slow_tool.' });\n+ const at = a.indexOf('Today is ');\n+ const stableAt = a.indexOf('Your plan:');\n+ assert.ok(at > 0, 'the date is in the prompt');\n+ assert.ok(a.indexOf('How the answer is grounded:') < stableAt, 'the rules come before the plan');\n+ assert.ok(stableAt > 1500, `the stable prefix is the whole rules block (${stableAt} bytes)`);\n+ assert.strictEqual(a.slice(0, stableAt), b.slice(0, stableAt), 'the rules prefix is byte-identical between two tasks');\n+ assert.ok(at > stableAt && b.indexOf('Today is ') > stableAt, 'the date comes after the rules');\n+ assert.strictEqual(a.split('Today is ').length, 2, 'the date appears once');\n+ assert.ok(b.indexOf('1. Compare two offers') < b.indexOf('Today is '), 'the plan stays before the date');\n+ assert.ok(b.indexOf('Progress: 1 of 2') < b.indexOf('Today is '), 'the ledger stays before the date');\n+ assert.ok(b.indexOf('Offered last: slow_tool.') < b.indexOf('Today is '), 'the demoted-tools note stays before the date');\n+ });\n+\n+ await ok('tool definitions serialize identically on every assembly', async () => {\n+ const dir = tempTools({ 'zebra.js': tool('zebra'), 'alpha.js': tool('alpha'), 'middle.js': tool('middle') });\n+ const first = createRegistry({ dir, strict: true, log: { warn() {} } }).definitions({ requester: 'user:usr_1' });\n+ const second = createRegistry({ dir, strict: true, log: { warn() {} } }).definitions({ requester: 'user:usr_1' });\n+ assert.strictEqual(JSON.stringify(first), JSON.stringify(second));\n+ assert.deepStrictEqual(first.map((d) => d.function.name), ['alpha', 'middle', 'zebra'], 'files load in a fixed order');\n+ });\n+\n+ if (results.length) {\n+ for (const e of results) console.error(e.stack || String(e));\n+ process.exitCode = 1;\n+ }\n+})();\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Actor",
+ "pull_request": 390,
+ "merge_commit": "cfb27b4a3441d3382b9c242799a270725400fb2d",
+ "merged_at": "2026-11-09T21:59:07Z"
+ }
+ },
+ {
+ "id": "actor-340-skill-negation",
+ "goal": "The skills scan flags a URL to an undeclared host when a negation appears elsewhere in the sentence; a negation standing immediately before the URL (or one word before it) still passes it.",
+ "check": {
+ "kind": "command",
+ "value": "node test/skills.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 30,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Actor",
+ "ref": "e6dce59b91a45db5786074e4883d2d7094119ce8",
+ "check_patch": "diff --git a/test/skills.test.js b/test/skills.test.js\nindex 5f5eea2..84d25a5 100644\n--- a/test/skills.test.js\n+++ b/test/skills.test.js\n@@ -63,6 +63,13 @@ const RULES = [\n good: { instructions: 'Read https://other.example/page for the answer.', requires: { hosts: ['other.example'] } },\n state: 'flagged',\n },\n+ {\n+ // A negation elsewhere in the sentence does not hide the host it reaches.\n+ rule: 'hosts.undeclared',\n+ bad: { instructions: 'Do not wait: read https://other.example/page for the answer.' },\n+ good: { instructions: 'Never open https://other.example/page, and answer from the notes.' },\n+ state: 'flagged',\n+ },\n ];\n \n (async () => {\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Actor",
+ "pull_request": 340,
+ "merge_commit": "cbdac596b1c45f7da826fe82a2dfd5e4ea6f4f38",
+ "merged_at": "2026-11-08T02:39:40Z"
+ }
+ },
+ {
+ "id": "exchange-161-status",
+ "goal": "The status board and a model's offer table show each offer's measured status, Up, Degraded when more than 5% of at least 10 requests in the last hour failed, or Down when the offer's circuit is open, with the plain status_reason and status_since that OpenVibe.AI sends in its catalog; Exchange infers nothing.",
+ "check": {
+ "kind": "command",
+ "value": "node test/pages.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Exchange",
+ "ref": "45c8ab948dc1a7e3232ae03d6993c4f51dab0e3f",
+ "check_patch": "diff --git a/test/pages.test.js b/test/pages.test.js\nindex a4c6e18..c767900 100644\n--- a/test/pages.test.js\n+++ b/test/pages.test.js\n@@ -112,6 +112,27 @@ const { createFakeAi, downAi } = require('./helpers/ai');\n assert.ok(r.text.includes('99.0%') && r.text.includes('no traffic'));\n });\n \n+ await check('status is measured: down and degraded carry a plain reason and since when', async () => {\n+ const offer = ai.catalogData.models[0].offers[0];\n+ try {\n+ offer.status = 'down';\n+ offer.status_reason = 'the provider says the account is out of credit';\n+ offer.status_since = '2026-11-09T09:00:00.000Z';\n+ const down = await t.get('/status');\n+ assert.ok(down.text.includes('Down') && down.text.includes('the provider says the account is out of credit'), 'the status page says why');\n+ assert.ok(down.text.includes('since 2026-11-09 09:00 UTC'), 'and since when');\n+ offer.status = 'degraded';\n+ offer.status_reason = 'too many errors in the last hour';\n+ offer.status_since = '2026-11-09T10:00:00.000Z';\n+ const degraded = await t.get('/m/deepseek/deepseek-chat');\n+ assert.ok(degraded.text.includes('Degraded') && degraded.text.includes('too many errors in the last hour'), 'the model page shows the measured status');\n+ } finally {\n+ offer.status = 'up';\n+ offer.status_reason = null;\n+ offer.status_since = null;\n+ }\n+ });\n+\n await check('an unknown page is 404, and a personal page is never cached', async () => {\n const r = await t.get('/nope');\n assert.strictEqual(r.status, 404);\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Exchange",
+ "pull_request": 161,
+ "merge_commit": "277caa6ded619bf105fd4b5b36da83c2a2d70040",
+ "merged_at": "2026-11-09T22:49:57Z"
+ }
+ },
+ {
+ "id": "exchange-160-pricing-console",
+ "goal": "The owner console has owner-only JSON read endpoints for Providers, Models and the Pricebook, plus provider-settings and model-override form routes that forward audited changes to OpenVibe.AI; provider key values never appear in console data, and the existing owner and same-origin checks stay.",
+ "check": {
+ "kind": "command",
+ "value": "node test/console-merge.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Exchange",
+ "ref": "a5c516f7a7328972bcd17aa98a730d6b1ba7fa8f",
+ "check_patch": "diff --git a/test/console-merge.test.js b/test/console-merge.test.js\nindex fd3a083..575860f 100644\n--- a/test/console-merge.test.js\n+++ b/test/console-merge.test.js\n@@ -24,7 +24,7 @@ app.use(createConsoleRoutes({\n await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));\n const base = `http://127.0.0.1:${server.address().port}`;\n try {\n- const consolePaths = ['/console', ...Object.keys(READS).map((section) => `/console/${section}`), '/console/providers/new', '/console/providers/openai', '/console/gpus/model-a/quote'];\n+ const consolePaths = ['/console', ...Object.keys(READS).map((section) => `/console/${section}`), '/console/providers/new', '/console/providers/openai', '/console/gpus/model-a/quote', '/console/data/providers', '/console/data/models', '/console/data/pricebook'];\n for (const path of consolePaths) for (const headers of [{}, { 'x-test-role': 'user' }, { 'x-test-role': 'admin' }]) {\n const response = await fetch(`${base}${path}`, { headers });\n assert.strictEqual(response.status, 404, `${path}: signed-out people and non-owners see no console route`);\n@@ -39,6 +39,12 @@ app.use(createConsoleRoutes({\n const page = await fetch(`${base}${path}`, { headers: { 'x-test-role': 'owner' } });\n assert.strictEqual(page.status, 200, `${path}: ${await page.text()}`);\n }\n+ for (const section of ['providers', 'models', 'pricebook']) {\n+ const data = await fetch(`${base}/console/data/${section}`, { headers: { 'x-test-role': 'owner' } });\n+ assert.strictEqual(data.status, 200);\n+ assert.ok((data.headers.get('content-type') || '').includes('application/json'));\n+ assert.ok(!JSON.stringify(await data.json()).includes(fakeSecret), `${section} data omits key values`);\n+ }\n const keyWrite = await fetch(`${base}/console/providers/openai/key`, {\n method: 'POST', headers: { 'x-test-role': 'owner', Origin: 'https://openvibe.exchange', 'content-type': 'application/x-www-form-urlencoded' },\n body: new URLSearchParams({ api_key: fakeSecret }), redirect: 'manual',\n@@ -55,6 +61,19 @@ app.use(createConsoleRoutes({\n assert.strictEqual(createProvider.status, 303);\n assert.ok(writes.some((write) => write.path === '/api/v1/admin/console/providers' && write.body.api_key === fakeSecret), 'provider creation uses AI JSON admin API');\n \n+ const providerSettings = await fetch(`${base}/console/providers/groq/settings`, {\n+ method: 'POST', headers: { 'x-test-role': 'owner', Origin: 'https://openvibe.exchange', 'content-type': 'application/x-www-form-urlencoded' },\n+ body: new URLSearchParams({ status: 'active', provider_access: 'AUTHORIZED_WHOLESALE', preference_order: '7', markup_percent: '12.5', uncensored: '1' }), redirect: 'manual',\n+ });\n+ assert.strictEqual(providerSettings.status, 303);\n+ assert.ok(writes.some((write) => write.path === '/api/v1/providers/groq' && write.body.markup_percent === 12.5 && write.body.uncensored === true));\n+ const modelOverride = await fetch(`${base}/console/models/groq/openai%2Fgpt-oss-120b/override`, {\n+ method: 'POST', headers: { 'x-test-role': 'owner', Origin: 'https://openvibe.exchange', 'content-type': 'application/x-www-form-urlencoded' },\n+ body: new URLSearchParams({ input_per_million: '0.2', cached_per_million: '', output_per_million: '0.7', reason: 'Owner approved adjustment' }), redirect: 'manual',\n+ });\n+ assert.strictEqual(modelOverride.status, 303);\n+ assert.ok(writes.some((write) => write.path === '/api/v1/models/groq/openai%2Fgpt-oss-120b' && write.body.owner_override.reason === 'Owner approved adjustment'));\n+\n } finally { await new Promise((resolve) => server.close(resolve)); }\n process.stdout.write('console-merge: owner gate and secret redaction passed\\n');\n })().catch((err) => { process.stderr.write(`${err.stack || err}\\n`); process.exitCode = 1; });\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Exchange",
+ "pull_request": 160,
+ "merge_commit": "418c289149ce2ed28e6a041d3ac9cde41f025fbf",
+ "merged_at": "2026-11-09T20:19:25Z"
+ }
+ },
+ {
+ "id": "report-147-dataset-jsonld",
+ "goal": "Report announces its tables as an open dataset through Dataset JSON-LD on / and /stats, built with the shared seo.jsonLd.dataset builder: name, description, creator OpenVibe, keywords and a distribution list pointing at /api/v1/stats, /api/v1/ai/results and /api/v1/rundown/topics.",
+ "check": {
+ "kind": "command",
+ "value": "node test/pages.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 30,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Report",
+ "ref": "b5b64d0ac69f555e12f72be0acfab01b311d38c7",
+ "check_patch": "diff --git a/test/pages.test.js b/test/pages.test.js\nindex 8bd6c54..38e71ee 100644\n--- a/test/pages.test.js\n+++ b/test/pages.test.js\n@@ -70,6 +70,19 @@ const { boot, check, done, defaults } = require('./helpers/boot');\n assert.ok(org && Array.isArray(org.sameAs) && org.sameAs.includes('https://github.com/OpenVibers'), 'Organization');\n });\n \n+ await check('the home and the stats page carry Dataset JSON-LD with the JSON endpoints', async () => {\n+ for (const path of ['/', '/stats']) {\n+ const r = await t.get(path);\n+ const ld = [...r.text.matchAll(/<script type=\"application\\/ld\\+json\">([\\s\\S]*?)<\\/script>/g)].map((m) => JSON.parse(m[1])).flatMap((b) => b['@graph'] || [b]);\n+ const dataset = ld.find((x) => x['@type'] === 'Dataset');\n+ assert.ok(dataset, `${path}: Dataset`);\n+ assert.strictEqual(dataset.url, 'https://openvibe.report/stats', `${path}: Dataset url`);\n+ const urls = (dataset.distribution || []).map((d) => d.contentUrl);\n+ assert.ok(urls.includes('https://openvibe.report/api/v1/stats'), `${path}: stats JSON`);\n+ assert.ok(urls.includes('https://openvibe.report/api/v1/ai/results'), `${path}: AI JSON`);\n+ }\n+ });\n+\n await check('The Rundown pages name OpenVibe.Live as canonical and mount the Frame for service report', async () => {\n const r = await t.get('/rundown/t/seattle');\n assert.ok(r.text.includes('<link rel=\"canonical\" href=\"https://openvibe.live/rundown/t/seattle\">'), 'the topic is Live\\'s page');\n",
+ "check_timeout_ms": 180000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Report",
+ "pull_request": 147,
+ "merge_commit": "87971dc92196d069a6c967b76304283f52d91595",
+ "merged_at": "2026-11-09T09:55:25Z"
+ }
+ },
+ {
+ "id": "network-747-motion",
+ "goal": "A display preference save keeps motion: full and motion: off as well as auto and reduced, reads each of the four back from the active theme, and still drops an unknown value.",
+ "check": {
+ "kind": "command",
+ "value": "node --import ./test/helpers/pg-preload.mjs test/themes-review.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 30,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Network",
+ "ref": "40712202526e8ee21de31d636a649db99dd6cb58",
+ "check_patch": "diff --git a/test/themes-review.test.js b/test/themes-review.test.js\nindex b2dd68a..7abc6b9 100644\n--- a/test/themes-review.test.js\n+++ b/test/themes-review.test.js\n@@ -119,6 +119,13 @@ const token = async (id) => signToken(await db.prepare('SELECT * FROM users WHER\n assert.strictEqual(r.body.display.icons, 'solid', 'a person who prefers solid icons keeps that on every site');\n assert.strictEqual((await call('GET', '/api/themes/me/active', { as: 2 })).body.display.icons, 'solid');\n assert.strictEqual((await call('PUT', '/api/themes/me/display', { as: 2, body: { icons: 'outline' } })).body.display.icons, undefined, 'only auto, line or solid');\n+ // Motion (openvibe-shared 4.183.0): auto, full, reduced or off follows the account, and each one reads back.\n+ for (const motion of ['auto', 'full', 'reduced', 'off']) {\n+ r = await call('PUT', '/api/themes/me/display', { as: 2, body: { motion } });\n+ assert.strictEqual(r.body.display.motion, motion, `motion ${motion} is kept`);\n+ assert.strictEqual((await call('GET', '/api/themes/me/active', { as: 2 })).body.display.motion, motion, `motion ${motion} reads back`);\n+ }\n+ assert.strictEqual((await call('PUT', '/api/themes/me/display', { as: 2, body: { motion: 'calm' } })).body.display.motion, undefined, 'only auto, full, reduced or off');\n \n // ── The site switcher's pinned and recent sites follow the account (migration 0029) ──\n r = await call('GET', '/api/themes/me/sites', { as: 2 });\n",
+ "check_timeout_ms": 240000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Network",
+ "pull_request": 747,
+ "merge_commit": "b9c60b27ab51d18d5dbbe6542c816345bfe3adca",
+ "merged_at": "2026-11-10T02:44:47Z"
+ }
+ },
+ {
+ "id": "network-736-link-send",
+ "goal": "openvibe.link's Send widget reads /auth/me before it asks for send keys, so a signed-out visitor sees its existing sign-in line with no 401 from /api/v1/me/nodes/send-keys.",
+ "check": {
+ "kind": "command",
+ "value": "node --import ./test/helpers/pg-preload.mjs test/security-ssrf.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Network",
+ "ref": "87944111b4aece1a8469a31ead30535636aa3553",
+ "check_patch": "diff --git a/test/security-ssrf.test.js b/test/security-ssrf.test.js\nindex bb800f3..71da23e 100644\n--- a/test/security-ssrf.test.js\n+++ b/test/security-ssrf.test.js\n@@ -133,7 +133,7 @@ const REVIEWED = {\n 'server/admin/review-queue.js fetch': [2, 'the reviewer queue: Wiki/Blog/News/Trade internal URLs from operator config (loopback defaults), fixed /api/v1/review-queue and review paths, ids percent-encoded, the signed-in staff member\\'s own token, staff only'],\n 'server/admin/ai-config-proxy.js fetch': [1, 'the owner settings proxy and the Setup tab (server/admin/setup.js calls through it): the reachable services\\' internal URLs from config.services (operator-set, loopback defaults), fixed /internal/config and AI /api/v1/providers paths, service, namespace and key checked against the known names, owner only'],\n 'server/integrations/github.js fetch': [1, 'fixed host api.github.com (asserted below)'],\n- 'server/link/send.js fetch': [3, 'browser-only Link Send requests to fixed same-origin machine-key and transfer routes; page input supplies file bytes, never a URL or destination host'],\n+ 'server/link/send.js fetch': [4, 'browser-only Link Send requests to fixed same-origin routes (the session read, machine keys and transfers); page input supplies file bytes, never a URL or destination host'],\n 'server/notifications/email-service.js http.request': [1, 'fixed host api.resend.com'],\n 'server/actions/index.js fetch': [7, 'The Action Center: OpenVibe.Actor, OpenVibe.Codes and OpenVibe.AI only (OV_ACTOR_INTERNAL_URL, OV_CODES_INTERNAL_URL, OV_AI_INTERNAL_URL, loopback); the paths are fixed (the decide route, the cancel route, the retry route of Actor or AI, Actor\\'s public GET /api/v1/skills catalogue, and Actor\\'s POST /api/v1/runs for a catalogue skill whose id must match the skill id pattern), the subject is the signed-in person\\'s own usr_ id, and an item id must match [A-Za-z0-9_-]{1,80}'],\n 'server/profile/public.js fetch': [6, 'OpenVibe.Inventory, Quest and Play use operator-configured internal URLs and fixed paths with a checked usr_ id. Public place reads use only the configured Live, Community and Space service URLs, fixed public paths, encoded canonical usernames, 2 s timeouts and no user-supplied host or token; Space results are restricted to the exact owner and Published cards'],\n",
+ "check_timeout_ms": 240000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Network",
+ "pull_request": 736,
+ "merge_commit": "ec99318366af37fce577f657242cf8c9c8554619",
+ "merged_at": "2026-11-09T19:31:04Z"
+ }
+ },
+ {
+ "id": "network-724-wallet-cookie",
+ "goal": "The frame wallet relay takes the person's ov_token cookie when no Bearer token is sent, forwards it upstream as the Bearer and never as a cookie, and only an origin already on the CORS allow-list (or no Origin) may use that path.",
+ "check": {
+ "kind": "command",
+ "value": "node --import ./test/helpers/pg-preload.mjs test/frame-wallet.test.js"
+ },
+ "budget": {
+ "tokens": 1000000,
+ "turns": 40,
+ "active_minutes": 6
+ },
+ "no_progress_limit": 4,
+ "fixture": {
+ "repo": "https://github.com/OpenVibers/OpenVibe.Network",
+ "ref": "c69abee12d0e3f4cd7ffb7a5a049d38d71c8469b",
+ "check_patch": "diff --git a/test/frame-wallet.test.js b/test/frame-wallet.test.js\nindex f574d5e..1a3c667 100644\n--- a/test/frame-wallet.test.js\n+++ b/test/frame-wallet.test.js\n@@ -75,6 +75,24 @@ const ITD = 'itd_01JZ00000000000000000000B1';\n assert.strictEqual(r.status, 401, 'no token, no wallet');\n assert.strictEqual(asked.filter((x) => x.url.startsWith('/api/v1/me/wallet')).length, 1, 'and nothing is asked upstream');\n \n+ // A site with its own cookie session has no token for the navbar: the person's Network cookie (ov_token) is the\n+ // token, sent upstream as the Bearer and never as a cookie; the answer keeps the allow-list's echoed Origin.\n+ const session = 'abcdefghijklmnop.cookie';\n+ ({ r, d } = await get('/wallet/api/v1/me/wallet', { Cookie: `ov_sso=x; ov_token=${session}`, Origin: 'https://openvibe.events' }));\n+ assert.strictEqual(r.status, 200);\n+ const viaCookie = asked.filter((x) => x.url.startsWith('/api/v1/me/wallet')).at(-1);\n+ assert.strictEqual(viaCookie.auth, `Bearer ${session}`, \"the person's session token goes along as the Bearer\");\n+ assert.strictEqual(viaCookie.cookie, '', 'the cookie itself never does');\n+ assert.notStrictEqual(r.headers.get('access-control-allow-origin'), '*', \"a credentialed answer is never '*'\");\n+ const askedBefore = asked.length;\n+ ({ r } = await send('/wallet/api/v1/me/vibes/buy', 'POST', { pack: 'vibes_500' }, { Authorization: '', Cookie: `ov_token=${session}` }));\n+ assert.strictEqual(r.status, 401, 'a write by cookie needs an Origin');\n+ ({ r } = await get('/wallet/api/v1/me/wallet', { Cookie: `ov_token=${session}`, Origin: 'chrome-extension://abcdefghijklmnopabcdefghijklmnop' }));\n+ assert.strictEqual(r.status, 401, 'never from an extension page');\n+ ({ r } = await get('/wallet/api/v1/me/wallet', { Cookie: 'ov_token=short' }));\n+ assert.strictEqual(r.status, 401, 'a malformed cookie is no token');\n+ assert.strictEqual(asked.length, askedBefore, 'none of these asked upstream');\n+\n // A page limit and cursor pass; every other query is dropped.\n ({ r } = await get('/wallet/api/v1/me/items?limit=500&cursor=c1&subject=usr_other', { Authorization: token }));\n assert.strictEqual(r.status, 200);\n",
+ "check_timeout_ms": 240000
+ },
+ "source": {
+ "repository": "OpenVibers/OpenVibe.Network",
+ "pull_request": 724,
+ "merge_commit": "783e1e08fc9583011a332ac000fa2c053d78e98d",
+ "merged_at": "2026-11-09T14:18:43Z"
+ }
+ }
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,208 @@
@@ -0,0 +1,208 @@
+'use strict';
+
+/**
+ * The local executor for one ticket trial (AL8). It is called by scripts/ticket-suite.js (or by the real-run
+ * harness) with one JSON document on stdin:
+ *
+ * { ticket, model, effort, trial, session, workspace, log_path? }
+ *
+ * It prepares a fresh checkout at the ticket's base commit, runs an OpenVibe.Codes agent on the goal, restores
+ * everything under test/ (so the agent cannot grade itself), applies the pull request's test patch and runs the
+ * check. It prints one JSON line on stdout:
+ *
+ * { passed, cost_usd, ms, tokens, turns, check_output, model, effort, workspace }
+ *
+ * Environment: `DEEPSEEK_API_KEY` for deepseek-flash; `OPENVIBE_CODES_DIR` for the harness (default
+ * ~/openvibe/work/OpenVibe.Codes); `OPENVIBE_CODES_BASE_URL` (and `OPENVIBE_CODES_MODEL`) for the local model;
+ * `OPENVIBE_REPO_CACHE` to clone from a local mirror instead of GitHub. Nothing here prints a secret: the agent's
+ * events are consumed, its cost and tokens only are reported, and its log goes to log_path when one is named.
+ */
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const { spawn, execFileSync } = require('child_process');
+
+const MAX_OUTPUT = 4000;
+
+function sh(cmd, args, cwd, { timeout = 300000, env = process.env } = {}) {
+ return new Promise((resolve) => {
+ const child = spawn(cmd, args, { cwd, env, stdio: ['ignore', 'pipe', 'pipe'] });
+ let out = '';
+ const add = (d) => { out = (out + d.toString()).slice(-MAX_OUTPUT); };
+ child.stdout.on('data', add); child.stderr.on('data', add);
+ const timer = setTimeout(() => child.kill('SIGKILL'), timeout);
+ child.on('close', (code) => { clearTimeout(timer); resolve({ ok: code === 0, code, out }); });
+ child.on('error', (err) => { clearTimeout(timer); resolve({ ok: false, code: -1, out: err.message }); });
+ });
+}
+
+function git(args, cwd) {
+ try { return { ok: true, out: execFileSync('git', args, { cwd, encoding: 'utf8', timeout: 120000, stdio: ['ignore', 'pipe', 'ignore'] }) }; }
+ catch (err) { return { ok: false, out: String((err.stderr || err.stdout || err.message)).slice(-800) }; }
+}
+
+/** Where a repo's local clone cache lives, when one is configured. */
+function cachedRepo(url, cache) {
+ if (!cache) return null;
+ const name = String(url).replace(/\.git$/, '').split('/').slice(-2).join('/');
+ for (const candidate of [path.join(cache, name), path.join(cache, `${name}.git`)]) {
+ if (fs.existsSync(candidate)) return candidate;
+ }
+ return null;
+}
+
+/** The checkout at the ticket's base commit: cloned once, reset before every trial, dependencies installed once. */
+async function prepare(input) {
+ const workspace = input.workspace || fs.mkdtempSync(path.join(os.tmpdir(), 'bench-ticket-'));
+ const ref = input.ticket.fixture.ref;
+ const fresh = !fs.existsSync(path.join(workspace, '.git'));
+ if (fresh) {
+ fs.mkdirSync(workspace, { recursive: true });
+ const cache = cachedRepo(input.ticket.fixture.repo, process.env.OPENVIBE_REPO_CACHE);
+ const clone = cache
+ ? await sh('git', ['clone', '--shared', '-q', cache, workspace], path.dirname(workspace))
+ : await sh('git', ['clone', '--filter=blob:none', '-q', input.ticket.fixture.repo, workspace], path.dirname(workspace));
+ if (!clone.ok) throw new Error(`clone failed: ${clone.out}`);
+ }
+ const checkout = git(['checkout', '-q', ref], workspace);
+ if (!checkout.ok) {
+ const fetched = git(['fetch', '-q', 'origin', ref], workspace);
+ if (!fetched.ok) throw new Error(`checkout ${ref} failed: ${checkout.out}`);
+ const again = git(['checkout', '-q', ref], workspace);
+ if (!again.ok) throw new Error(`checkout ${ref} failed: ${again.out}`);
+ }
+ git(['reset', '--hard', '-q', ref], workspace);
+ git(['clean', '-fdq', '-e', 'node_modules'], workspace);
+ if (!fs.existsSync(path.join(workspace, 'node_modules'))) {
+ const installed = await sh('npm', ['ci', '--no-audit', '--no-fund', '--prefer-offline'], workspace, { timeout: 600000 });
+ if (!installed.ok) throw new Error(`npm ci failed: ${installed.out.slice(-600)}`);
+ }
+ return { workspace, fresh };
+}
+
+/** The prompt one agent gets: the goal, its check and the budget. The test patch is never in it. */
+function promptFor(input) {
+ const ticket = input.ticket;
+ const budget = ticket.budget || {};
+ return [
+ `Work in the repository in ${input.workspace}. Its base commit is ${ticket.fixture.ref}.`,
+ '',
+ 'The goal:',
+ ticket.goal,
+ '',
+ `The check: ${ticket.check.kind} — ${ticket.check.value}`,
+ budget.turns ? `Turn budget: ${budget.turns}.` : '',
+ budget.active_minutes ? `Time budget: about ${budget.active_minutes} minutes.` : '',
+ '',
+ 'Do not add or edit anything under test/: the graded tests are applied after you stop, and your copy of them is discarded.',
+ 'Do not run the full suite (npm test or node test/run.js): run at most one focused test file, or node --check on the files you changed.',
+ 'Make the smallest change in the code the goal names, and end with a short summary.',
+ ].filter((line) => line !== '').join('\n');
+}
+
+/** Run one OpenVibe.Codes agent on the ticket. Returns cost, tokens, turns and the final text. */
+async function runAgent(input, workspace) {
+ const codesDir = process.env.OPENVIBE_CODES_DIR || path.join(os.homedir(), 'openvibe', 'work', 'OpenVibe.Codes');
+ let createHarness;
+ try { ({ createHarness } = require(path.join(codesDir, 'harness'))); }
+ catch (err) { throw new Error(`the OpenVibe.Codes harness is not readable at ${codesDir}: ${err.message}`); }
+ const model = String(input.model);
+ const agent = model === 'local' ? 'openai-compatible' : 'deepseek';
+ const modelName = model === 'local' ? (process.env.OPENVIBE_CODES_MODEL || null) : model;
+ const budget = input.ticket.budget || {};
+ const harness = createHarness({ env: process.env });
+ const run = harness.run({
+ prompt: promptFor(input),
+ agent,
+ model: modelName,
+ cwd: workspace,
+ permission: 'full',
+ handoff: false,
+ maxTurns: Math.max(1, Number(budget.turns) || 30),
+ maxAttempts: 1,
+ stallMs: 4 * 60 * 1000,
+ });
+ // A hard wall clock: the harness's quiet timer cancels a model call but a command's grandchild can hold its
+ // pipe open forever, so one trial is never allowed to outlive its budget. The agent's own spend still counts.
+ const deadlineMs = Math.max(60_000, Math.min(8 * 60_000, Number(budget.active_minutes || 8) * 60_000));
+ const consumer = (async () => {
+ let result = null;
+ for await (const event of run) {
+ if (event.type === 'result') result = event;
+ if (input.log_path) fs.appendFileSync(input.log_path, JSON.stringify({ type: event.type, subtype: event.subtype || null, text: event.result || (event.message && event.message.content && event.message.content[0] && event.message.content[0].text) || null }) + '\n');
+ }
+ const usage = (result && result.usage) || {};
+ return {
+ cost_usd: result ? Number(result.total_cost_usd) || 0 : 0,
+ tokens: (Number(usage.input_tokens) || 0) + (Number(usage.output_tokens) || 0) + (Number(usage.cache_read_input_tokens) || 0),
+ turns: result ? Number(result.num_turns) || 0 : 0,
+ ok: Boolean(result && !result.is_error),
+ text: result ? String(result.result || '') : '',
+ };
+ })();
+ let timer;
+ const timeout = new Promise((resolve) => { timer = setTimeout(() => resolve({ timed_out: true }), deadlineMs); timer.unref?.(); });
+ const agentResult = await Promise.race([consumer, timeout]);
+ clearTimeout(timer);
+ if (agentResult.timed_out) {
+ try { run.cancel(); } catch { /* the run is already gone */ }
+ return { cost_usd: 0, tokens: 0, turns: 0, ok: false, text: `the agent did not finish within ${Math.round(deadlineMs / 60000)} minutes` };
+ }
+ return agentResult;
+}
+
+/** Restore test/, apply the pull request's test patch, run the check. The only place a trial can pass. */
+async function grade(input, workspace) {
+ git(['checkout', '--', 'test'], workspace);
+ git(['clean', '-fdq', 'test'], workspace);
+ const patchFile = path.join(workspace, '.bench-check.patch');
+ fs.writeFileSync(patchFile, input.ticket.fixture.check_patch);
+ const applied = await sh('git', ['apply', '--whitespace=nowarn', '.bench-check.patch'], workspace);
+ fs.rmSync(patchFile, { force: true });
+ if (!applied.ok) return { passed: false, check_output: `the check patch did not apply: ${applied.out}` };
+ const timeout = Number(input.ticket.fixture.check_timeout_ms) || 180000;
+ const check = await sh('bash', ['-lc', input.ticket.check.value], workspace, { timeout });
+ return { passed: check.ok, check_output: check.out };
+}
+
+async function main(raw) {
+ const input = JSON.parse(raw);
+ const started = Date.now();
+ const { workspace } = await prepare(input);
+ let agent;
+ try { agent = await runAgent(input, workspace); }
+ catch (err) { agent = { cost_usd: 0, tokens: 0, turns: 0, ok: false, text: err.message }; }
+ // A test command (a URL check and a rubric are graded by a person, not this executor) still needs a workspace.
+ const graded = input.ticket.check.kind === 'command' ? await grade(input, workspace) : { passed: false, check_output: `a ${input.ticket.check.kind} check is not graded by the local executor` };
+ return {
+ passed: graded.passed,
+ cost_usd: Math.round(agent.cost_usd * 1e6) / 1e6,
+ ms: Date.now() - started,
+ tokens: agent.tokens,
+ turns: agent.turns,
+ agent_ok: agent.ok,
+ agent_text: agent.text.slice(0, 2000),
+ check_output: String(graded.check_output || '').slice(-MAX_OUTPUT),
+ model: input.model,
+ effort: input.effort,
+ workspace,
+ };
+}
+
+if (require.main === module) {
+ let raw = '';
+ process.stdin.setEncoding('utf8');
+ process.stdin.on('data', (d) => { raw += d; });
+ process.stdin.on('end', () => {
+ main(raw).then((out) => {
+ process.stdout.write(JSON.stringify(out) + '\n');
+ // A timed-out agent may leave a child holding a pipe: the result is out, so the process may stop.
+ setTimeout(() => process.exit(0), 250).unref();
+ }, (err) => {
+ process.stdout.write(JSON.stringify({ passed: false, cost_usd: 0, ms: 0, tokens: 0, check_output: String(err && err.message || err).slice(0, 2000) }) + '\n');
+ setTimeout(() => process.exit(1), 250).unref();
+ });
+ });
+}
+
+module.exports = { main, prepare, promptFor, grade, cachedRepo };
new file mode 100644
new file mode 100644
@@ -0,0 +1,173 @@
@@ -0,0 +1,173 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * The AL8 gate, from the command line. It reads a suite file (bench.codes-ticket-suite@1), checks it, and — only
+ * when the caller passes `--run` — runs every ticket over 3 to 5 trials on deepseek-flash and the local model,
+ * publishes one run per trial and reads the table back from the results API.
+ *
+ * node scripts/ticket-suite.js # the plan: no call, no cost
+ * node scripts/ticket-suite.js --run --trials 3 \
+ * --models deepseek-flash --effort high \
+ * --publish https://bench.openvibe.services --token "$BENCH_SERVICE_TOKEN"
+ *
+ * There is no timer and no schedule: a suite runs when a person runs this command.
+ *
+ * The executor is a command that takes one JSON trial on stdin and answers one JSON result line (the default is
+ * scripts/ticket-executor.js). `--benchmark bmk_…` reuses a frozen suite instead of creating a new one; a new run
+ * of an old benchmark is what makes pass^k comparable over time.
+ */
+const fs = require('fs');
+const path = require('path');
+const os = require('os');
+const { spawn } = require('child_process');
+const tickets = require('../server/bench/tickets');
+const runner = require('../server/bench/ticket-runner');
+
+const ROOT = path.join(__dirname, '..');
+const DEFAULT_SUITE = path.join(ROOT, 'fixtures', 'codes.ticket', 'openvibe-merged-2026-11.json');
+
+function parseArgs(argv) {
+ const out = { _: [] };
+ for (let i = 0; i < argv.length; i++) {
+ const arg = argv[i];
+ if (!arg.startsWith('--')) { out._.push(arg); continue; }
+ const key = arg.slice(2);
+ const next = argv[i + 1];
+ if (next == null || next.startsWith('--')) out[key] = true;
+ else { out[key] = next; i++; }
+ }
+ return out;
+}
+
+function list(value, fallback) {
+ if (value == null || value === true) return fallback;
+ return String(value).split(',').map((x) => x.trim()).filter(Boolean);
+}
+
+/** A stable line the report quotes: pass@1, pass^k and cost per solved for each model and effort. */
+function tableLines(entries) {
+ return entries.map((e) => `${e.model} · effort ${e.effort || 'default'} · trials ${e.k} · pass@1 ${e.pass_at_1 == null ? '—' : e.pass_at_1.toFixed(3)} · pass^${e.k} ${e.pass_at_k == null ? '—' : e.pass_at_k.toFixed(3)} · cost per solved ${e.cost_per_solved == null ? '—' : '$' + e.cost_per_solved.toFixed(4)}${e.insufficient_trials ? ' (fewer than 3 trials: not a gate result)' : ''}`);
+}
+
+async function callJson(base, pathname, { method = 'GET', body, token } = {}) {
+ const response = await fetch(new URL(pathname, base), {
+ method,
+ headers: { accept: 'application/json', ...(body ? { 'content-type': 'application/json' } : {}), ...(token ? { authorization: `Bearer ${token}` } : {}) },
+ body: body ? JSON.stringify(body) : undefined,
+ signal: AbortSignal.timeout(30000),
+ });
+ const text = await response.text();
+ let json = null;
+ try { json = text ? JSON.parse(text) : null; } catch { /* a problem document may not be JSON */ }
+ if (!response.ok) throw new Error(`${method} ${pathname} answered HTTP ${response.status}: ${(json && (json.detail || json.code)) || text.slice(0, 300)}`);
+ return json;
+}
+
+/** Create and freeze a benchmark for the suite, or reuse one the caller names. */
+async function ensureBenchmark(base, token, suite, reuse, log) {
+ if (reuse) {
+ const found = await callJson(base, `/api/v1/benchmarks/${encodeURIComponent(reuse)}`, { token });
+ const benchmark = found.benchmark || found;
+ const version = benchmark.latest_version || (found.version && found.version.version);
+ if (!version) throw new Error(`${reuse} has no frozen version`);
+ log(`reusing ${benchmark.id}@${version}`);
+ return { id: benchmark.id, version };
+ }
+ const definition = tickets.benchmarkDefinition(suite);
+ const made = await callJson(base, '/api/v1/benchmarks', {
+ method: 'POST', token,
+ body: { title: definition.title, description: definition.description, kind: definition.kind, tags: definition.tags, visibility: 'unlisted', definition },
+ });
+ const id = made.benchmark.id;
+ const frozen = await callJson(base, `/api/v1/benchmarks/${encodeURIComponent(id)}/versions`, { method: 'POST', token, body: {} });
+ log(`created ${id}@${frozen.version.version}`);
+ return { id, version: frozen.version.version };
+}
+
+/** One trial through the executor command; its stdout's last JSON line is the result. */
+function executeWith(command, input, logPath) {
+ return new Promise((resolve) => {
+ const child = spawn('bash', ['-lc', command], { stdio: ['pipe', 'pipe', 'pipe'] });
+ let out = '';
+ let err = '';
+ child.stdout.on('data', (d) => { out += d.toString(); });
+ child.stderr.on('data', (d) => { err += d.toString(); });
+ child.stdin.end(JSON.stringify(input));
+ const timer = setTimeout(() => child.kill('SIGKILL'), 30 * 60 * 1000);
+ child.on('close', () => {
+ clearTimeout(timer);
+ if (logPath) fs.writeFileSync(logPath, err.slice(-20000));
+ const lines = out.trim().split('\n').filter(Boolean);
+ for (let i = lines.length - 1; i >= 0; i--) {
+ try { return resolve(JSON.parse(lines[i])); } catch { /* earlier lines are not the result */ }
+ }
+ resolve({ passed: false, cost_usd: 0, ms: 0, tokens: 0, check_output: `the executor printed no result: ${out.slice(-500)} ${err.slice(-500)}` });
+ });
+ child.on('error', (error) => { clearTimeout(timer); resolve({ passed: false, cost_usd: 0, ms: 0, tokens: 0, check_output: error.message }); });
+ });
+}
+
+async function main() {
+ const args = parseArgs(process.argv.slice(2));
+ const suitePath = path.resolve(String(args.suite || DEFAULT_SUITE));
+ if (!fs.existsSync(suitePath)) throw new Error(`no suite file at ${suitePath}`);
+ const suite = JSON.parse(fs.readFileSync(suitePath, 'utf8'));
+ const checked = tickets.validateSuite(suite);
+ if (!checked.valid) throw new Error(`the suite is not valid: ${checked.detail}`);
+ const models = list(args.models, suite.models || tickets.DEFAULT_MODELS);
+ const trials = args.trials == null ? suite.trials : Number(args.trials);
+ const effort = args.effort == null ? 'high' : String(args.effort);
+ const plan = runner.plan(suite, { models, trials, effort });
+
+ process.stdout.write(`suite ${suite.suite}: ${plan.tickets.length} tickets × ${plan.models.join(', ')} × ${plan.trials} trials = ${plan.calls} calls, effort ${plan.effort}\n`);
+ for (const ticket of plan.tickets) process.stdout.write(` ${ticket.id}: ${ticket.check}\n`);
+
+ if (!args.run) {
+ process.stdout.write('not run: pass --run to spend money on these trials\n');
+ return;
+ }
+ if (args.publish && !args.token) throw new Error('--publish needs --token (or BENCH_SERVICE_TOKEN)');
+ const base = args.publish ? String(args.publish).replace(/\/+$/, '') : null;
+ const token = args.token && args.token !== true ? String(args.token) : process.env.BENCH_SERVICE_TOKEN;
+ const log = (message) => process.stderr.write(`[ticket-suite] ${message}\n`);
+ const benchmark = base ? await ensureBenchmark(base, token, suite, args.benchmark, log) : null;
+ const workdir = path.resolve(String(args.workdir || path.join(os.tmpdir(), `bench-tickets-${suite.suite}`)));
+ const logDir = args['log-dir'] ? path.resolve(String(args['log-dir'])) : null;
+ if (logDir) fs.mkdirSync(logDir, { recursive: true });
+ const executor = String(args.executor || `node ${JSON.stringify(path.join(__dirname, 'ticket-executor.js'))}`);
+
+ const out = await runner.runSuite({
+ suite, models, trials, effort, run: true,
+ execute: async ({ ticket, model, trial, session }) => {
+ const workspace = path.join(workdir, ticket.id);
+ const logPath = logDir ? path.join(logDir, `${ticket.id}-${model.replace(/[^a-z0-9.-]/gi, '_')}-${trial}.log`) : null;
+ log(`${ticket.id} · ${model} · trial ${trial}/${plan.trials}`);
+ return executeWith(executor, { ticket, model, effort, trial, session, workspace, log_path: logPath }, logPath);
+ },
+ });
+ if (!out.ok) throw new Error(`${out.code}: ${out.detail}`);
+
+ let published = [];
+ if (base) {
+ for (const run of out.runs) {
+ const made = await callJson(base, '/api/v1/runs', { method: 'POST', token, body: { ...run, benchmark: `${benchmark.id}@${benchmark.version}`, benchmark_id: benchmark.id, benchmark_version: benchmark.version } });
+ published.push(made.run.id);
+ }
+ log(`published ${published.length} runs on ${benchmark.id}@${benchmark.version}`);
+ }
+ let table = out.table;
+ if (base) {
+ const read = await callJson(base, `/api/v1/rankings?benchmark=${encodeURIComponent(benchmark.id + '@' + benchmark.version)}&metric=tickets&effort=${encodeURIComponent(effort)}`, { token });
+ table = read.rankings.entries;
+ }
+ const report = { suite: suite.suite, session: out.session, benchmark, effort, models, trials: plan.trials, runs: published, table };
+ if (args.out) fs.writeFileSync(path.resolve(String(args.out)), JSON.stringify(report, null, 2));
+ for (const line of tableLines(table)) process.stdout.write(line + '\n');
+}
+
+if (require.main === module) {
+ main().catch((err) => { process.stderr.write(`[ticket-suite] ${err.message}\n`); process.exitCode = 1; });
+}
+
+module.exports = { parseArgs, tableLines, ensureBenchmark, executeWith };
@@ -21,6 +21,10 @@ const { isHeldOut, addedOn, humanMinutes, humanMinutesSource } = require('./task
@@ -21,6 +21,10 @@ const { isHeldOut, addedOn, humanMinutes, humanMinutesSource } = require('./task
const { cutoffFor } = require('./model-cutoffs');
const { cutoffFor } = require('./model-cutoffs');
const { contaminationFor } = require('./contamination');
const { contaminationFor } = require('./contamination');
const { estimateHorizon } = require('./horizon');
const { estimateHorizon } = require('./horizon');
+const { scoreRuns } = require('./tickets');
+
+/** The metric names the gate answers to. `tickets` is the API's name; pass@k aliases are what callers type. */
+const TICKET_METRICS = Object.freeze(['tickets', 'pass_k', 'pass@k', 'pass^k']);
const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
const iso = (v) => (v == null ? null : (v instanceof Date ? v.toISOString() : String(v)));
@@ -831,13 +835,14 @@ function createRegistry(s, { log = console, sharing = null } = {}) {
@@ -831,13 +835,14 @@ function createRegistry(s, { log = console, sharing = null } = {}) {
* speed the median score per minute
* speed the median score per minute
* consistency the spread of a model's scores inside one session (the lower, the steadier)
* consistency the spread of a model's scores inside one session (the lower, the steadier)
* horizon the 50% human-time horizon (minutes; higher is better)
* horizon the 50% human-time horizon (minutes; higher is better)
+ * tickets the AL8 gate: pass@1, pass^k and cost per solved by model and effort, for a whole-ticket suite
*
*
* `since` is a monthly fresh set: with dated tasks, only the tasks added on or after that month are scored.
* `since` is a monthly fresh set: with dated tasks, only the tasks added on or after that month are scored.
* An older benchmark whose tasks carry no dates keeps the B2 meaning (runs created since the date), so nothing
* An older benchmark whose tasks carry no dates keeps the B2 meaning (runs created since the date), so nothing
* that worked before stops working. Each entry also carries the contamination flag and the time horizon when
* that worked before stops working. Each entry also carries the contamination flag and the time horizon when
* there is enough data for either.
* there is enough data for either.
*/
*/
- async function rankings({ benchmark, metric = 'score', harness = null, since = null, includeSubmitted = false }) {
+ async function rankings({ benchmark, metric = 'score', harness = null, since = null, includeSubmitted = false, effort = null }) {
const ref = parseRef(benchmark);
const ref = parseRef(benchmark);
if (!ref) return null;
if (!ref) return null;
const versions = await db.many(
const versions = await db.many(
@@ -883,6 +888,21 @@ function createRegistry(s, { log = console, sharing = null } = {}) {
@@ -883,6 +888,21 @@ function createRegistry(s, { log = console, sharing = null } = {}) {
if (!tasksByRun.has(row.run_id)) tasksByRun.set(row.run_id, []);
if (!tasksByRun.has(row.run_id)) tasksByRun.set(row.run_id, []);
tasksByRun.get(row.run_id).push(row);
tasksByRun.get(row.run_id).push(row);
}
}
+ // The AL8 gate reads the same stored runs, one row per model and effort. A ticket is a task; a run is one
+ // trial of the whole suite, so pass^k needs every trial's task row.
+ if (TICKET_METRICS.includes(metric)) {
+ const scored = scoreRuns(rows.map((row) => wireRun(row)), tasksByRun, { effort });
+ const latest = definitions.get(Math.max(...wanted));
+ return {
+ benchmark: ref.id,
+ metric: 'tickets',
+ kind: latest ? latest.kind : null,
+ effort: effort == null ? null : String(effort),
+ trials_required: { min: 3, max: 5 },
+ entries: scored.entries,
+ };
+ }
+
const estimates = await estimateMedians(ref.id, wanted);
const estimates = await estimateMedians(ref.id, wanted);
/** The score and totals of one run, over the selected tasks when a monthly set is asked for. */
/** The score and totals of one run, over the selected tasks when a monthly set is asked for. */
new file mode 100644
new file mode 100644
@@ -0,0 +1,108 @@
@@ -0,0 +1,108 @@
+'use strict';
+
+/**
+ * The AL8 gate's runner: a whole ticket over 3 to 5 trials per model and effort, then one row per model and effort.
+ *
+ * Two rules are code here, not advice:
+ *
+ * 1. It never runs by itself. Without `run: true` (the CLI's `--run`) it returns the plan and makes no call, so
+ * no timer, scheduler or page start can spend money.
+ * 2. It runs only on the cheap lanes unless the caller widens them on purpose: `deepseek-flash` and the local
+ * model (`local`). Anything else is refused before the first trial.
+ *
+ * The executor is injected. It receives one ticket trial and answers with what the check said:
+ *
+ * execute({ ticket, model, effort, trial, session }) → { passed, cost_usd, ms, tokens, check_output }
+ *
+ * `scripts/ticket-executor.js` is the local implementation (an OpenVibe.Codes agent in a fresh checkout); a Codes
+ * benchmark runner can implement the same shape. Because the check patch is applied by the executor after the
+ * agent stops, the agent never grades its own copy of the test.
+ */
+const tickets = require('./tickets');
+
+const PROVIDERS = Object.freeze({ 'deepseek-flash': 'deepseek', local: 'local' });
+
+/** The plan a run would follow: which models, how many trials, which tickets. No money moves. */
+function plan(suite, { models, trials, effort = 'high' } = {}) {
+ const list = Array.isArray(models) && models.length ? models.map((m) => String(m)) : (suite.models || tickets.DEFAULT_MODELS).map((m) => String(m));
+ const k = trials == null ? suite.trials : Number(trials);
+ return {
+ suite: suite.suite,
+ models: list,
+ trials: k,
+ effort: effort == null ? 'high' : String(effort),
+ tickets: (suite.tickets || []).map((t) => ({ id: t.id, check: t.check && t.check.value })),
+ calls: list.length * (suite.tickets || []).length * k,
+ };
+}
+
+/**
+ * Run the suite, or say what it would do. `execute` is required only when `run` is true. Returns the trial runs
+ * (ready for POST /api/v1/runs) and the gate table (the same rows the results API computes).
+ */
+async function runSuite({ suite, execute = null, run = false, models = null, trials = null, effort = 'high', session = null, now = () => new Date() } = {}) {
+ const checked = tickets.validateSuite(suite);
+ if (!checked.valid) return { ok: false, code: 'bench.ticket_suite_invalid', detail: checked.detail };
+ const thePlan = plan(suite, { models, trials, effort });
+ if (thePlan.models.some((m) => !tickets.DEFAULT_MODELS.includes(m))) {
+ return { ok: false, code: 'bench.ticket_model_not_allowed', detail: `the gate runs only on ${tickets.DEFAULT_MODELS.join(' and ')} unless a person widens it` };
+ }
+ if (!Number.isInteger(thePlan.trials) || thePlan.trials < tickets.TRIAL_MIN || thePlan.trials > tickets.TRIAL_MAX) {
+ return { ok: false, code: 'bench.ticket_trials', detail: `trials must be ${tickets.TRIAL_MIN}-${tickets.TRIAL_MAX}` };
+ }
+ if (!run) return { ok: true, ran: false, plan: thePlan };
+ if (typeof execute !== 'function') return { ok: false, code: 'bench.ticket_executor_missing', detail: 'a run needs an executor' };
+
+ const sessionId = session || `ticket-${thePlan.suite}-${now().toISOString().replace(/[^0-9]/g, '').slice(0, 14)}`;
+ const runs = [];
+ const tasksByRun = new Map();
+ for (const model of thePlan.models) {
+ for (let trial = 1; trial <= thePlan.trials; trial++) {
+ const results = [];
+ for (const ticket of suite.tickets) {
+ let out;
+ try {
+ out = await execute({ ticket, model, effort: thePlan.effort, trial, session: sessionId });
+ } catch (err) {
+ out = { passed: false, error: (err && err.message) || String(err), cost_usd: 0, ms: 0, tokens: 0, check_output: '' };
+ }
+ out = out && typeof out === 'object' ? out : { passed: false };
+ results.push({
+ task_id: ticket.id,
+ score: out.passed === true ? 100 : 0,
+ cost_usd: Number.isFinite(Number(out.cost_usd)) ? Number(out.cost_usd) : 0,
+ ms: Number.isFinite(Number(out.ms)) ? Math.max(0, Math.round(Number(out.ms))) : 0,
+ tokens: Number.isFinite(Number(out.tokens)) ? Math.max(0, Math.round(Number(out.tokens))) : 0,
+ artifacts: [{ kind: 'check', passed: out.passed === true, output: String(out.check_output == null ? '' : out.check_output).slice(0, 2000) }],
+ ...(out.transcript_url ? { transcript_url: String(out.transcript_url).slice(0, 500) } : {}),
+ });
+ }
+ const cost = results.reduce((n, r) => n + r.cost_usd, 0);
+ const ms = results.reduce((n, r) => n + r.ms, 0);
+ const tokens = results.reduce((n, r) => n + r.tokens, 0);
+ const score = results.length ? results.reduce((n, r) => n + r.score, 0) / results.length : 0;
+ const runDoc = {
+ model,
+ provider: PROVIDERS[model] || 'unknown',
+ route: { requested: model, ran: model, effort: thePlan.effort },
+ harness: { name: 'openvibe-codes', version: '0.4.0' },
+ settings: { effort: thePlan.effort },
+ executor: { kind: 'codes', id: 'bench-ticket-suite' },
+ session: sessionId,
+ tasks: results,
+ score: Math.round(score * 1000) / 1000,
+ cost_usd: Math.round(cost * 1e6) / 1e6,
+ ms,
+ tokens,
+ created_at: now().toISOString(),
+ };
+ tasksByRun.set(`trial-${runs.length + 1}`, results.map((r) => ({ task_id: r.task_id, score: r.score, cost_usd: r.cost_usd, ms: r.ms, tokens: r.tokens })));
+ runs.push(runDoc);
+ }
+ }
+ // The table is computed from the same shape the database will store, so a published run reads back identically.
+ const scored = require('./tickets').scoreRuns(runs.map((r, i) => ({ ...r, id: `trial-${i + 1}`, created_at: now().toISOString() })), tasksByRun, { effort: thePlan.effort });
+ return { ok: true, ran: true, session: sessionId, plan: thePlan, runs, table: scored.entries };
+}
+
+module.exports = { PROVIDERS, plan, runSuite };
new file mode 100644
new file mode 100644
@@ -0,0 +1,285 @@
@@ -0,0 +1,285 @@
+'use strict';
+
+/**
+ * Whole-ticket benchmarks (AL8, plan 2026-11-10): a ticket is a goal, its check and a budget, drawn from a real
+ * merged pull request. The worker gets the goal; the check runs only after it stops, so a test added by the pull
+ * request measures the work instead of the worker's copy of it.
+ *
+ * The goal and check are AL1's `agent.goal@1` shape (Contracts 0.389.0, still in review when this landed): the
+ * goal is the condition in words, the check is a command, a URL assertion or a rubric, and the budget bounds the
+ * tokens, turns and active minutes a trial may spend (the local executor in scripts/ticket-executor.js enforces
+ * turns and active minutes; a token cap is carried for executors that can stop on usage). This module defines the
+ * suite file Bench reads and the scoring every run is judged by; the runner that spends money lives in
+ * server/bench/ticket-runner.js.
+ *
+ * A suite is `bench.codes-ticket-suite@1`:
+ *
+ * {
+ * "format": "bench.codes-ticket-suite@1",
+ * "suite": "openvibe-merged-2026-11",
+ * "title": "Twelve merged OpenVibe tickets",
+ * "kind": "codes.ticket",
+ * "models": ["deepseek-flash", "local"],
+ * "trials": 3,
+ * "tickets": [{
+ * "id": "actor-390-prompt-prefix",
+ * "goal": "The coding agent's request prefix stays byte-identical …",
+ * "check": { "kind": "command", "value": "node test/runtime-prompt-prefix.test.js" },
+ * "budget": { "tokens": 80000, "turns": 40, "active_minutes": 8 },
+ * "no_progress_limit": 4,
+ * "fixture": {
+ * "repo": "https://github.com/OpenVibers/OpenVibe.Actor",
+ * "ref": "<the merge commit's parent>",
+ * "check_patch": "<unified diff of the pull request's test files>",
+ * "check_timeout_ms": 180000
+ * },
+ * "source": { "repository": "OpenVibers/OpenVibe.Actor", "pull_request": 390,
+ * "merge_commit": "…", "merged_at": "2026-11-10T…" }
+ * }]
+ * }
+ *
+ * Only `deepseek-flash` and `local` are allowed by default: the gate is meant to run often and must not spend on
+ * the dear models. A suite runs only when a caller says `--run`; nothing here is scheduled.
+ */
+
+const TRIAL_MIN = 3;
+const TRIAL_MAX = 5;
+const FORMAT = 'bench.codes-ticket-suite@1';
+const KIND = 'codes.ticket';
+const DEFAULT_MODELS = Object.freeze(['deepseek-flash', 'local']);
+const CHECK_KINDS = Object.freeze(['command', 'url', 'rubric']);
+const REPO = /^https:\/\/github\.com\/[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/;
+const SHA = /^[0-9a-f]{40}$/;
+
+const isStr = (v) => typeof v === 'string';
+const isNum = (v) => typeof v === 'number' && Number.isFinite(v);
+const isObj = (v) => v && typeof v === 'object' && !Array.isArray(v);
+const num = (v, d = 0) => (v == null || v === '' || Number.isNaN(Number(v)) ? d : Number(v));
+
+class Problems {
+ constructor() { this.list = []; }
+ add(path, message) { this.list.push({ path, message }); return this; }
+ get valid() { return this.list.length === 0; }
+ get detail() { return this.list.map((e) => `${e.path} ${e.message}`).join('; '); }
+}
+
+/** AL1's goal and check: the goal in words, a typed check, optional budgets and a no-progress limit. */
+function validateGoal(goal, path = '') {
+ const e = new Problems();
+ if (!isObj(goal)) return e.add(path || '/', 'must be an object');
+ if (!isStr(goal.goal) || !goal.goal.trim() || goal.goal.length > 4000) e.add(`${path}/goal`, 'must be 1-4000 characters');
+ if (!isObj(goal.check)) e.add(`${path}/check`, 'must be an object');
+ else {
+ if (!CHECK_KINDS.includes(goal.check.kind)) e.add(`${path}/check/kind`, `must be one of ${CHECK_KINDS.join(', ')}`);
+ if (!isStr(goal.check.value) || !goal.check.value.trim() || goal.check.value.length > 4000) e.add(`${path}/check/value`, 'must be 1-4000 characters');
+ }
+ if (goal.budget != null) {
+ if (!isObj(goal.budget)) e.add(`${path}/budget`, 'must be an object');
+ else {
+ for (const key of ['tokens', 'turns', 'active_minutes']) {
+ if (goal.budget[key] != null && (!Number.isInteger(goal.budget[key]) || goal.budget[key] < 1)) e.add(`${path}/budget/${key}`, 'must be a positive integer');
+ }
+ if (!Object.keys(goal.budget).some((k) => ['tokens', 'turns', 'active_minutes'].includes(k))) e.add(`${path}/budget`, 'needs tokens, turns or active_minutes');
+ }
+ }
+ if (goal.no_progress_limit != null && (!Number.isInteger(goal.no_progress_limit) || goal.no_progress_limit < 1 || goal.no_progress_limit > 20)) {
+ e.add(`${path}/no_progress_limit`, 'must be an integer 1-20');
+ }
+ return e;
+}
+
+/** One ticket in a suite: the AL1 goal and check, plus where the base checkout comes from and how it is graded. */
+function validateTicket(ticket, path = '') {
+ const e = new Problems();
+ if (!isObj(ticket)) return e.add(path || '/', 'must be an object');
+ if (!isStr(ticket.id) || !/^[a-z0-9][a-z0-9-]{1,79}$/.test(ticket.id)) e.add(`${path}/id`, 'must be 2-80 lowercase letters, digits and dashes');
+ const goal = validateGoal(ticket, path);
+ for (const p of goal.list) e.add(p.path, p.message);
+ if (!isObj(ticket.fixture)) e.add(`${path}/fixture`, 'must be an object');
+ else {
+ if (!isStr(ticket.fixture.repo) || !REPO.test(ticket.fixture.repo)) e.add(`${path}/fixture/repo`, 'must be an https GitHub repository URL');
+ if (!isStr(ticket.fixture.ref) || !SHA.test(ticket.fixture.ref)) e.add(`${path}/fixture/ref`, 'must be a 40-character commit sha');
+ if (!isStr(ticket.fixture.check_patch) || !ticket.fixture.check_patch.trim()) e.add(`${path}/fixture/check_patch`, 'must be the unified diff the check is applied from');
+ if (ticket.fixture.check_timeout_ms != null && (!Number.isInteger(ticket.fixture.check_timeout_ms) || ticket.fixture.check_timeout_ms < 1000 || ticket.fixture.check_timeout_ms > 900000)) {
+ e.add(`${path}/fixture/check_timeout_ms`, 'must be 1000-900000 milliseconds');
+ }
+ if (ticket.check && ticket.check.kind === 'command' && !ticket.fixture.check_patch.trim().startsWith('diff --git ') && !ticket.fixture.check_patch.trim().startsWith('--- ')) {
+ e.add(`${path}/fixture/check_patch`, 'must be a unified diff (diff --git …)');
+ }
+ }
+ if (!isObj(ticket.source)) e.add(`${path}/source`, 'must be an object');
+ else {
+ if (!isStr(ticket.source.repository) || !/^OpenVibers\/[A-Za-z0-9_.-]+$/.test(ticket.source.repository)) e.add(`${path}/source/repository`, 'must be an OpenVibers/Repo name');
+ if (!Number.isInteger(ticket.source.pull_request) || ticket.source.pull_request < 1) e.add(`${path}/source/pull_request`, 'must be a positive pull request number');
+ if (!isStr(ticket.source.merge_commit) || !SHA.test(ticket.source.merge_commit)) e.add(`${path}/source/merge_commit`, 'must be a 40-character commit sha');
+ }
+ return e;
+}
+
+/** The whole suite file, checked before a single model call is made. */
+function validateSuite(doc) {
+ const e = new Problems();
+ if (!isObj(doc)) return e.add('/', 'must be an object');
+ if (doc.format !== FORMAT) e.add('/format', `must be ${FORMAT}`);
+ if (!isStr(doc.suite) || !/^[a-z0-9][a-z0-9-]{1,79}$/.test(doc.suite)) e.add('/suite', 'must be 2-80 lowercase letters, digits and dashes');
+ if (!isStr(doc.title) || !doc.title.trim() || doc.title.length > 160) e.add('/title', 'must be 1-160 characters');
+ if (doc.kind != null && doc.kind !== KIND) e.add('/kind', `must be ${KIND}`);
+ if (doc.models != null) {
+ if (!Array.isArray(doc.models) || !doc.models.length) e.add('/models', 'must be a non-empty array');
+ else doc.models.forEach((m, i) => {
+ if (!isStr(m) || !m.trim()) e.add(`/models/${i}`, 'must be a model id');
+ else if (!DEFAULT_MODELS.includes(m)) e.add(`/models/${i}`, `only ${DEFAULT_MODELS.join(' and ')} may run by default`);
+ });
+ }
+ if (!Number.isInteger(doc.trials) || doc.trials < TRIAL_MIN || doc.trials > TRIAL_MAX) e.add('/trials', `must be an integer ${TRIAL_MIN}-${TRIAL_MAX}`);
+ if (!Array.isArray(doc.tickets) || !doc.tickets.length) e.add('/tickets', 'must be a non-empty array');
+ else {
+ const ids = new Set();
+ doc.tickets.forEach((ticket, i) => {
+ const one = validateTicket(ticket, `/tickets/${i}`);
+ for (const p of one.list) e.add(p.path, p.message);
+ if (isObj(ticket) && isStr(ticket.id)) {
+ if (ids.has(ticket.id)) e.add(`/tickets/${i}/id`, 'is used twice');
+ ids.add(ticket.id);
+ }
+ });
+ }
+ return { valid: e.valid, errors: e.list };
+}
+
+/** The benchmark definition a suite freezes: one task per ticket, the check kept as the task's expected result. */
+function benchmarkDefinition(suite, { owner = 'openvibe', visibility = 'unlisted' } = {}) {
+ const tickets = Array.isArray(suite.tickets) ? suite.tickets : [];
+ return {
+ owner,
+ title: String(suite.title || suite.suite || 'Ticket suite').slice(0, 160),
+ description: String(suite.description || 'A whole ticket: a goal and its check, scored over repeated trials.').slice(0, 4000),
+ kind: KIND,
+ tags: ['tickets', 'gate', `suite:${suite.suite}`],
+ visibility,
+ tasks: tickets.map((t) => ({
+ id: t.id,
+ prompt: String(t.goal).slice(0, 8000),
+ // The check patch is withheld from readers: only the executor (bench.run.publish) and the owner see it.
+ holdout: true,
+ files: [{
+ kind: 'codes.ticket',
+ repo: t.fixture.repo,
+ ref: t.fixture.ref,
+ goal: t.goal,
+ check: t.check,
+ budget: t.budget || {},
+ no_progress_limit: t.no_progress_limit == null ? 5 : t.no_progress_limit,
+ }],
+ expected: {
+ passed: true,
+ check: t.check,
+ check_patch: t.fixture.check_patch,
+ check_timeout_ms: t.fixture.check_timeout_ms == null ? 180000 : t.fixture.check_timeout_ms,
+ source: t.source,
+ },
+ })),
+ judges: [{ type: 'test_suite', version: 'codes.ticket@1', config: { pass_score: 100, trials: suite.trials, check: 'the task check exits 0 after its test patch is applied' } }],
+ scoring: { method: 'pass_rate', scale: { min: 0, max: 100 }, aggregate: 'mean' },
+ limits: { ms: tickets.reduce((n, t) => n + num(t.fixture && t.fixture.check_timeout_ms, 180000), 0) },
+ matrix: { models: Array.isArray(suite.models) && suite.models.length ? [...suite.models] : [...DEFAULT_MODELS], harnesses: ['openvibe-codes'] },
+ };
+}
+
+/** One ticket's trials: how many ran, how many passed, and whether every one passed. */
+function scoreTicket(trials) {
+ const list = Array.isArray(trials) ? trials.filter((t) => t && t.score != null) : [];
+ const n = list.length;
+ const passes = list.filter((t) => num(t.score) > 0).length;
+ return {
+ trials: n,
+ passes,
+ pass_at_1: n ? passes / n : null,
+ pass_at_k: n ? passes === n : false,
+ cost_usd: Math.round(list.reduce((s, t) => s + num(t.cost_usd), 0) * 1e6) / 1e6,
+ ms: list.reduce((s, t) => s + num(t.ms), 0),
+ tokens: list.reduce((s, t) => s + num(t.tokens), 0),
+ };
+}
+
+/**
+ * The gate's table: one row per model and effort, from runs whose tasks are ticket results (100 = the check
+ * passed, 0 = it did not).
+ *
+ * pass@1 the share of all (ticket, trial) pairs that passed
+ * pass^k the share of tickets whose every one of k trials passed (the customer-facing number: a 75%
+ * per-trial agent passes three in a row about 42% of the time)
+ * cost per solved every trial's cost, divided by the tickets solved at least once, so a flaky ticket is
+ * charged for its failures and can be told from a model that never solves it
+ *
+ * `k` is the smallest number of trials any ticket got; a ticket with fewer trials than k cannot pass^k. A group
+ * with fewer than three trials says so (`insufficient_trials`) instead of pretending to be a gate result.
+ */
+function scoreRuns(runs, tasksByRun, { effort = null } = {}) {
+ const groups = new Map();
+ for (const run of runs) {
+ if (!run) continue;
+ const runEffort = run.settings && run.settings.effort != null ? String(run.settings.effort) : null;
+ if (effort != null && String(effort) !== (runEffort == null ? 'default' : runEffort)) continue;
+ const key = `${String(run.model)}\u0000${runEffort == null ? 'default' : runEffort}`;
+ if (!groups.has(key)) groups.set(key, { model: String(run.model), effort: runEffort, trialsByTicket: new Map(), runs: 0, cost_usd: 0, ms: 0, tokens: 0, session: run.session || null });
+ const group = groups.get(key);
+ group.runs++;
+ group.cost_usd += num(run.cost_usd);
+ group.ms += num(run.ms);
+ group.tokens += num(run.tokens);
+ if (!group.last_run || String(run.created_at || '') > group.last_run) group.last_run = String(run.created_at || '');
+ for (const task of tasksByRun.get(run.id) || []) {
+ if (task.score == null) continue;
+ const id = String(task.task_id);
+ if (!group.trialsByTicket.has(id)) group.trialsByTicket.set(id, []);
+ group.trialsByTicket.get(id).push({ score: num(task.score), cost_usd: num(task.cost_usd), ms: num(task.ms), tokens: num(task.tokens), run: run.id });
+ }
+ }
+ const entries = [];
+ for (const group of groups.values()) {
+ const perTicket = [...group.trialsByTicket.entries()].map(([id, trials]) => ({ task_id: id, ...scoreTicket(trials) }));
+ const k = perTicket.reduce((n, t) => Math.min(n, t.trials), Infinity);
+ const kk = Number.isFinite(k) ? k : 0;
+ const totalPairs = perTicket.reduce((n, t) => n + t.trials, 0);
+ const passedPairs = perTicket.reduce((n, t) => n + t.passes, 0);
+ const fullyPassed = perTicket.filter((t) => t.trials >= kk && t.pass_at_k).length;
+ const solved = perTicket.filter((t) => t.passes > 0).length;
+ entries.push({
+ model: group.model,
+ effort: group.effort,
+ runs: group.runs,
+ trials: group.runs,
+ task_trials: totalPairs,
+ tickets: perTicket.length,
+ k: kk,
+ pass_at_1: totalPairs ? Math.round((passedPairs / totalPairs) * 1e6) / 1e6 : null,
+ pass_at_k: perTicket.length ? Math.round((fullyPassed / perTicket.length) * 1e6) / 1e6 : null,
+ solved,
+ cost_usd: Math.round(group.cost_usd * 1e6) / 1e6,
+ cost_per_solved: solved ? Math.round((group.cost_usd / solved) * 1e6) / 1e6 : null,
+ ms: group.ms,
+ tokens: group.tokens,
+ insufficient_trials: kk < TRIAL_MIN,
+ per_ticket: perTicket.map((t) => ({
+ task_id: t.task_id,
+ trials: t.trials,
+ passes: t.passes,
+ pass_at_1: t.pass_at_1 == null ? null : Math.round(t.pass_at_1 * 1e6) / 1e6,
+ pass_at_k: t.pass_at_k,
+ cost_usd: t.cost_usd,
+ })).sort((a, b) => a.task_id.localeCompare(b.task_id)),
+ last_run: group.last_run || null,
+ });
+ }
+ entries.sort((a, b) => (b.pass_at_1 == null ? -1 : b.pass_at_1) - (a.pass_at_1 == null ? -1 : a.pass_at_1)
+ || (a.cost_per_solved == null ? Infinity : a.cost_per_solved) - (b.cost_per_solved == null ? Infinity : b.cost_per_solved)
+ || a.model.localeCompare(b.model)
+ || String(a.effort || '').localeCompare(String(b.effort || '')));
+ return { metric: 'tickets', entries };
+}
+
+module.exports = {
+ FORMAT, KIND, TRIAL_MIN, TRIAL_MAX, DEFAULT_MODELS, CHECK_KINDS,
+ validateGoal, validateTicket, validateSuite, benchmarkDefinition, scoreTicket, scoreRuns,
+};
@@ -366,8 +366,8 @@ function createRegistryApi(ctx) {
@@ -366,8 +366,8 @@ function createRegistryApi(ctx) {
const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';
const benchmark = req.query.benchmark ? String(req.query.benchmark) : '';
if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');
if (!benchmark) return invalid(res, 'benchmark is required: ?benchmark=bmk_…');
const metric = req.query.metric ? String(req.query.metric) : 'score';
const metric = req.query.metric ? String(req.query.metric) : 'score';
- if (!['score', 'value', 'speed', 'consistency', 'horizon'].includes(metric)) {
- return invalid(res, 'metric is score, value, speed, consistency or horizon');
+ if (!['score', 'value', 'speed', 'consistency', 'horizon', 'tickets', 'pass_k', 'pass@k', 'pass^k'].includes(metric)) {
+ return invalid(res, 'metric is score, value, speed, consistency, horizon or tickets');
}
}
try {
try {
const bm = await registry.getBenchmark(benchmark.split('@')[0]);
const bm = await registry.getBenchmark(benchmark.split('@')[0]);
@@ -378,6 +378,7 @@ function createRegistryApi(ctx) {
@@ -378,6 +378,7 @@ function createRegistryApi(ctx) {
harness: req.query.harness ? String(req.query.harness) : null,
harness: req.query.harness ? String(req.query.harness) : null,
since: req.query.since ? String(req.query.since) : null,
since: req.query.since ? String(req.query.since) : null,
includeSubmitted: String(req.query.include_submitted || '') === 'true',
includeSubmitted: String(req.query.include_submitted || '') === 'true',
+ effort: req.query.effort ? String(req.query.effort).slice(0, 40) : null,
});
});
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
if (!out) return problem(res, 404, 'bench.benchmark_not_found', 'No such benchmark.');
res.set('Cache-Control', bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
res.set('Cache-Control', bm.visibility === 'public' ? 'public, max-age=60' : 'private, no-store');
@@ -21,6 +21,19 @@ const KINDS = Object.freeze([
@@ -21,6 +21,19 @@ const KINDS = Object.freeze([
{ id: 'vision.pair', room: 'vision', entry: 'model_answer', judge: 'human_pairwise', name: 'Vision head-to-head', metrics: ['elo', 'cost_usd'], schema: { entry: 'model_answer', payload: { answer: 'string' } } },
{ id: 'vision.pair', room: 'vision', entry: 'model_answer', judge: 'human_pairwise', name: 'Vision head-to-head', metrics: ['elo', 'cost_usd'], schema: { entry: 'model_answer', payload: { answer: 'string' } } },
{ id: 'codes.patch', room: 'codes', entry: 'code_patch', judge: 'test_suite', name: 'Code patches', metrics: ['pass_rate', 'cost_usd'], schema: { entry: 'code_patch', payload: { patch: 'string' }, prompt: { repository: 'string', failing_test: 'string', constraints: 'string' }, rubric: ['Apply the patch in an isolated checkout', 'Run the pinned failing and regression tests', 'Pass only when all tests pass'], seeds: [{ repository: 'sample-js', failing_test: 'sum([]) should return 0', constraints: 'Change only sum.js' }] } },
{ id: 'codes.patch', room: 'codes', entry: 'code_patch', judge: 'test_suite', name: 'Code patches', metrics: ['pass_rate', 'cost_usd'], schema: { entry: 'code_patch', payload: { patch: 'string' }, prompt: { repository: 'string', failing_test: 'string', constraints: 'string' }, rubric: ['Apply the patch in an isolated checkout', 'Run the pinned failing and regression tests', 'Pass only when all tests pass'], seeds: [{ repository: 'sample-js', failing_test: 'sum([]) should return 0', constraints: 'Change only sum.js' }] } },
{ id: 'codes.review', room: 'codes', entry: 'review', judge: 'human_pairwise', name: 'Code diff review', metrics: ['elo', 'cost_usd'], schema: { entry: 'review', payload: { review: 'string' }, prompt: { diff: 'string', intent: 'string' }, rubric: ['Find specific correctness or security defects', 'Cite the affected lines and explain impact', 'Avoid speculative or duplicate findings'], seeds: [{ diff: 'A cache key changes from userId to a constant', intent: 'Review for cross-user data exposure' }] } },
{ id: 'codes.review', room: 'codes', entry: 'review', judge: 'human_pairwise', name: 'Code diff review', metrics: ['elo', 'cost_usd'], schema: { entry: 'review', payload: { review: 'string' }, prompt: { diff: 'string', intent: 'string' }, rubric: ['Find specific correctness or security defects', 'Cite the affected lines and explain impact', 'Avoid speculative or duplicate findings'], seeds: [{ diff: 'A cache key changes from userId to a constant', intent: 'Review for cross-user data exposure' }] } },
+ // AL8, the gate (plan 2026-11-10): one whole ticket run over 3 to 5 trials. The task carries AL1's shape —
+ // goal and check, a budget and a no-progress limit — and the check is applied only after the worker stops, so a
+ // test added by the merged pull request grades the work. A run is one trial; pass@1, pass^k and cost per
+ // solved are read from GET /api/v1/rankings?benchmark=…&metric=tickets[&effort=…].
+ { id: 'codes.ticket', room: 'codes', entry: 'ticket_result', judge: 'test_suite', name: 'Whole tickets', metrics: ['pass_at_1', 'pass_at_k', 'cost_per_solved'],
+ schema: {
+ trials: { min: 3, max: 5 },
+ goal: { goal: 'string', check: { kind: 'command|url|rubric', value: 'string' }, budget: { tokens: 'integer', turns: 'integer', active_minutes: 'integer' }, no_progress_limit: 'integer' },
+ entry: 'ticket_result', payload: { passed: 'boolean', check_output: 'string' },
+ fixture: { repo: 'https://github.com/OpenVibers/<Repo>', ref: '40-character base commit', check_patch: 'unified diff of the pull request tests' },
+ scoring: 'pass@1 is the share of trials that passed; pass^k is the share of tickets whose every one of k trials passed; cost per solved is every trial cost divided by the tickets solved at least once',
+ seeds: [{ goal: 'The failing test passes.', check: { kind: 'command', value: 'node test/example.test.js' }, budget: { turns: 20, active_minutes: 5 } }],
+ } },
{ id: 'surf.desk', room: 'surf', entry: 'agent_on_desk', judge: 'human_pairwise', name: 'Agents on desks', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'agent_on_desk', payload: { desk: 'object' } } },
{ id: 'surf.desk', room: 'surf', entry: 'agent_on_desk', judge: 'human_pairwise', name: 'Agents on desks', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'agent_on_desk', payload: { desk: 'object' } } },
{ id: 'chat.reply', room: 'chat', entry: 'conversation_turn', judge: 'human_pairwise', name: 'Chat replies', metrics: ['elo', 'cost_usd'], schema: { entry: 'conversation_turn', payload: { reply: 'string' }, prompt: { conversation: 'array', latest_user_message: 'string' }, rubric: ['Answer the latest request accurately', 'Use prior turns when relevant', 'Be clear and avoid invented facts'], seeds: [{ conversation: [{ role: 'user', content: 'I have 20 minutes for dinner.' }], latest_user_message: 'Suggest a quick vegetarian meal.' }] } },
{ id: 'chat.reply', room: 'chat', entry: 'conversation_turn', judge: 'human_pairwise', name: 'Chat replies', metrics: ['elo', 'cost_usd'], schema: { entry: 'conversation_turn', payload: { reply: 'string' }, prompt: { conversation: 'array', latest_user_message: 'string' }, rubric: ['Answer the latest request accurately', 'Use prior turns when relevant', 'Be clear and avoid invented facts'], seeds: [{ conversation: [{ role: 'user', content: 'I have 20 minutes for dinner.' }], latest_user_message: 'Suggest a quick vegetarian meal.' }] } },
{ id: 'chat.arena', room: 'exchange', entry: 'model_answer', judge: 'human_pairwise', name: 'Exchange arena', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'model_answer', payload: { answer: 'string' }, prompt: { text: 'string' } } },
{ id: 'chat.arena', room: 'exchange', entry: 'model_answer', judge: 'human_pairwise', name: 'Exchange arena', metrics: ['elo', 'cost_usd', 'ms'], schema: { entry: 'model_answer', payload: { answer: 'string' }, prompt: { text: 'string' } } },
new file mode 100644
new file mode 100644
@@ -0,0 +1,207 @@
@@ -0,0 +1,207 @@
+'use strict';
+
+/**
+ * The AL8 gate (plan 2026-11-10): the codes.ticket kind, the suite file, the trial maths and the results API that
+ * reads pass@1, pass^k and cost per solved by model and effort. The runner's money rule is tested here too: it
+ * makes no call without `run: true` and refuses every model but deepseek-flash and the local one.
+ */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const { boot, check, done } = require('./helpers/boot');
+const tickets = require('../server/bench/tickets');
+const runner = require('../server/bench/ticket-runner');
+const cli = require('../scripts/ticket-suite');
+const executor = require('../scripts/ticket-executor');
+const { createRegistryApi } = require('../server/http/registry');
+const { kindById } = require('../server/kinds');
+
+/** One ticket in the committed shape, with a one-line check patch. */
+function ticketDoc(id, prompt = 'The example test passes.') {
+ return {
+ id,
+ goal: prompt,
+ check: { kind: 'command', value: `node test/${id}.test.js` },
+ budget: { tokens: 20000, turns: 10, active_minutes: 3 },
+ no_progress_limit: 3,
+ fixture: { repo: 'https://github.com/OpenVibers/OpenVibe.Bench', ref: '0'.repeat(40), check_patch: 'diff --git a/test/x b/test/x\n', check_timeout_ms: 60000 },
+ source: { repository: 'OpenVibers/OpenVibe.Bench', pull_request: 1, merge_commit: '0'.repeat(40) },
+ };
+}
+
+/** A two-ticket suite in the committed shape. */
+function kitchen(over = {}) {
+ return { format: tickets.FORMAT, suite: 'kitchen-suite', title: 'Kitchen suite', kind: 'codes.ticket', models: ['deepseek-flash'], trials: 3, tickets: [ticketDoc('alpha'), ticketDoc('beta')], ...over };
+}
+
+(async () => {
+ const t = await boot();
+ const writer = t.network.serviceToken({ sub: 'svc:codes', audience: 'openvibe.bench', cap: ['bench.benchmark.write', 'bench.run.publish'] });
+
+ await check('the results API is the registry route and codes.ticket is a kind', async () => {
+ assert.strictEqual(typeof createRegistryApi, 'function', 'the registry route is mounted');
+ const kind = kindById('codes.ticket');
+ assert.ok(kind, 'the kind is registered');
+ assert.strictEqual(kind.judge, 'test_suite');
+ assert.deepStrictEqual(kind.metrics, ['pass_at_1', 'pass_at_k', 'cost_per_solved']);
+ assert.strictEqual(kind.schema.trials.min, 3);
+ assert.strictEqual(kind.schema.trials.max, 5);
+ assert.ok(kind.schema.goal.check, 'the AL1 goal and check shape');
+ });
+
+ await check('a suite is checked before it runs', async () => {
+ assert.ok(tickets.validateSuite(kitchen()).valid, 'the kitchen suite is valid');
+ assert.ok(!tickets.validateSuite(kitchen({ trials: 2 })).valid, 'two trials is not a gate');
+ assert.ok(!tickets.validateSuite(kitchen({ trials: 6 })).valid, 'six trials is not a gate');
+ assert.ok(!tickets.validateSuite(kitchen({ models: ['claude-opus-4'] })).valid, 'a dear model is not the default gate');
+ const badCheck = kitchen();
+ badCheck.tickets[0].check = { kind: 'vibes', value: 'it feels right' };
+ assert.ok(!tickets.validateSuite(badCheck).valid, 'a check has a known kind');
+ const twice = kitchen();
+ twice.tickets[1].id = twice.tickets[0].id;
+ assert.ok(!tickets.validateSuite(twice).valid, 'two tickets cannot share an id');
+ });
+
+ await check('a suite in the committed fixture is valid and holds twelve real tickets', async () => {
+ const file = path.join(__dirname, '..', 'fixtures', 'codes.ticket', 'openvibe-merged-2026-11.json');
+ const suite = JSON.parse(fs.readFileSync(file, 'utf8'));
+ const checked = tickets.validateSuite(suite);
+ assert.ok(checked.valid, checked.detail);
+ assert.strictEqual(suite.tickets.length, 12);
+ for (const ticket of suite.tickets) {
+ assert.strictEqual(ticket.check.kind, 'command');
+ assert.match(ticket.fixture.check_patch, /^diff --git /);
+ assert.match(ticket.source.repository, /^OpenVibers\//);
+ }
+ });
+
+ await check('the benchmark task carries AL1 goal, check, budget and the grading patch', async () => {
+ const definition = tickets.benchmarkDefinition(kitchen());
+ assert.strictEqual(definition.kind, 'codes.ticket');
+ assert.strictEqual(definition.tasks.length, 2);
+ const task = definition.tasks[0];
+ assert.strictEqual(task.prompt, 'The example test passes.');
+ assert.strictEqual(task.holdout, true);
+ assert.strictEqual(task.files[0].goal, 'The example test passes.');
+ assert.strictEqual(task.files[0].check.kind, 'command');
+ assert.strictEqual(task.files[0].budget.turns, 10);
+ assert.ok(task.expected.check_patch.startsWith('diff --git '));
+ });
+
+ await check('the runner makes no call without --run, and refuses a dear model before one', async () => {
+ let calls = 0;
+ const plan = await runner.runSuite({ suite: kitchen(), execute: () => { calls++; } });
+ assert.strictEqual(plan.ok, true);
+ assert.strictEqual(plan.ran, false);
+ assert.strictEqual(calls, 0);
+ const refused = await runner.runSuite({ suite: kitchen(), run: true, models: ['claude-opus-4'], execute: () => { calls++; } });
+ assert.strictEqual(refused.ok, false);
+ assert.strictEqual(refused.code, 'bench.ticket_model_not_allowed');
+ assert.strictEqual(calls, 0);
+ const short = await runner.runSuite({ suite: kitchen(), run: true, trials: 2, execute: () => { calls++; } });
+ assert.strictEqual(short.ok, false);
+ assert.strictEqual(short.code, 'bench.ticket_trials');
+ assert.strictEqual(calls, 0);
+ });
+
+ await check('the executor prompt carries the goal and check but never the test patch', async () => {
+ const ticket = kitchen().tickets[0];
+ const prompt = executor.promptFor({ ticket, workspace: '/tmp/example' });
+ assert.ok(prompt.includes(ticket.goal), 'the goal is in the prompt');
+ assert.ok(prompt.includes(ticket.check.value), 'the check is in the prompt');
+ assert.ok(!prompt.includes(ticket.fixture.check_patch.split('\n')[0]), 'the test patch is never shown to the agent');
+ assert.match(prompt, /Do not add or edit anything under test\//);
+ });
+
+ await check('the command line prints the plan and parses the safe flags', async () => {
+ const args = cli.parseArgs(['--run', '--trials', '3', '--models', 'deepseek-flash,local', '--effort', 'high']);
+ assert.strictEqual(args.run, true);
+ assert.strictEqual(args.trials, '3');
+ assert.strictEqual(args.models, 'deepseek-flash,local');
+ const lines = cli.tableLines([{ model: 'deepseek-flash', effort: 'high', k: 3, pass_at_1: 0.888889, pass_at_k: 0.666667, cost_per_solved: 0.298567 }]);
+ assert.match(lines[0], /pass@1 0\.889/);
+ assert.match(lines[0], /pass\^3 0\.667/);
+ assert.match(lines[0], /cost per solved \$0\.2986/);
+ });
+
+ await check('three trials make a table of pass@1, pass^3 and cost per solved', async () => {
+ const out = await runner.runSuite({
+ suite: kitchen(), run: true, trials: 3, models: ['deepseek-flash'],
+ execute: ({ ticket, trial, model }) => ({ passed: ticket.id === 'alpha' || trial === 1, cost_usd: model === 'deepseek-flash' ? 0.001 : 0, ms: 5, tokens: 7, check_output: 'ok' }),
+ });
+ assert.strictEqual(out.ok, true);
+ assert.strictEqual(out.runs.length, 3, 'one run per trial');
+ assert.strictEqual(out.runs[0].settings.effort, 'high');
+ assert.strictEqual(out.runs[0].provider, 'deepseek');
+ assert.strictEqual(out.runs[0].tasks.length, 2);
+ const row = out.table.find((e) => e.model === 'deepseek-flash' && e.effort === 'high');
+ assert.ok(row, 'the table has the model and effort');
+ assert.strictEqual(row.k, 3);
+ assert.strictEqual(row.pass_at_1, 0.666667);
+ assert.strictEqual(row.pass_at_k, 0.5);
+ assert.strictEqual(row.cost_usd, 0.006);
+ assert.strictEqual(row.cost_per_solved, 0.003);
+ assert.strictEqual(row.insufficient_trials, false);
+ });
+
+ await check('pass@1 and pass^k are read from the results API, by model and effort', async () => {
+ const definition = tickets.benchmarkDefinition(kitchen({ tickets: ['alpha', 'beta', 'gamma', 'delta'].map((id) => ticketDoc(id)) }));
+ const created = await t.get('/api/v1/benchmarks', { bearer: writer, json: definition });
+ assert.strictEqual(created.status, 201, created.text);
+ const id = created.json().benchmark.id;
+ const frozen = await t.get(`/api/v1/benchmarks/${id}/versions`, { bearer: writer, json: {} });
+ assert.strictEqual(frozen.status, 201, frozen.text);
+ const benchmark = `${id}@${frozen.json().version.version}`;
+ const trial = (scores, { effort = 'high' } = {}) => {
+ const tasks = scores.map(([task_id, score]) => ({ task_id, score, cost_usd: 0.001, ms: 100, tokens: 10, artifacts: [] }));
+ return {
+ benchmark, model: 'deepseek-flash', provider: 'deepseek',
+ route: { requested: 'deepseek-flash', ran: 'deepseek-flash', effort },
+ harness: { name: 'openvibe-codes', version: '0.4.0' }, settings: { effort },
+ executor: { kind: 'codes', id: 'test' }, session: 'gate-test',
+ tasks, score: tasks.reduce((n, x) => n + x.score, 0) / tasks.length,
+ cost_usd: tasks.reduce((n, x) => n + x.cost_usd, 0), ms: tasks.length * 100, tokens: tasks.length * 10,
+ };
+ };
+ // alpha passes every trial, beta two of three, gamma one, delta none.
+ const trials = [
+ trial([['alpha', 100], ['beta', 100], ['gamma', 100], ['delta', 0]]),
+ trial([['alpha', 100], ['beta', 100], ['gamma', 0], ['delta', 0]]),
+ trial([['alpha', 100], ['beta', 0], ['gamma', 0], ['delta', 0]]),
+ ];
+ for (const body of trials) {
+ const made = await t.get('/api/v1/runs', { bearer: writer, json: body });
+ assert.strictEqual(made.status, 201, made.text);
+ }
+ // A one-trial low-effort run is read, but it is not a gate result.
+ const low = await t.get('/api/v1/runs', { bearer: writer, json: trial([['alpha', 100]], { effort: 'low' }) });
+ assert.strictEqual(low.status, 201, low.text);
+ const read = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=tickets&effort=high`, { bearer: writer });
+ assert.strictEqual(read.status, 200, read.text);
+ const row = read.json().rankings.entries.find((e) => e.model === 'deepseek-flash' && e.effort === 'high');
+ assert.ok(row, 'the entry names the model and effort');
+ assert.strictEqual(row.k, 3);
+ assert.strictEqual(row.trials, 3, 'three trial runs');
+ assert.strictEqual(row.task_trials, 12, 'four tickets x three trials');
+ assert.strictEqual(row.tickets, 4);
+ assert.strictEqual(row.pass_at_1, 0.5);
+ assert.strictEqual(row.pass_at_k, 0.25);
+ assert.ok(Math.abs(row.cost_per_solved - 0.004) < 1e-9, 'every trial cost, divided by the three solved tickets');
+ assert.strictEqual(row.per_ticket.find((x) => x.task_id === 'alpha').pass_at_k, true);
+ assert.strictEqual(row.per_ticket.find((x) => x.task_id === 'delta').pass_at_k, false);
+ const alias = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=pass_k`, { bearer: writer });
+ assert.strictEqual(alias.status, 200, alias.text);
+ assert.strictEqual(alias.json().rankings.metric, 'tickets');
+ const other = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=tickets&effort=low`, { bearer: writer });
+ assert.strictEqual(other.status, 200, other.text);
+ assert.strictEqual(other.json().rankings.entries.length, 1, 'the low-effort group is its own row');
+ assert.strictEqual(other.json().rankings.entries[0].k, 1);
+ assert.strictEqual(other.json().rankings.entries[0].insufficient_trials, true, 'one trial is not a gate result');
+ const wrong = await t.get(`/api/v1/rankings?benchmark=${encodeURIComponent(benchmark)}&metric=sideways`, { bearer: writer });
+ assert.strictEqual(wrong.status, 400, wrong.text);
+ });
+
+ await check('stop', async () => { await t.close(); });
+
+ done();
+})();