openvibers/openvibe.contracts Commits
0.244.0: bench.benchmark@1 and bench.run@1 — the benchmark registry and the results API (#251)
Changes
@@@ -4,6 +4,18 @@ All notable changes to `openvibe-contracts`. Releases are git tags (`vX.Y.Z`) thfrom `https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/<tag>`. Before v0.30.0,the notes were in the tag and commit messages (`git tag -n1`).++## 0.244.0 — 2026-10-30++++- **Bench as an open benchmark platform, stage B1** (spec 2026-10-30): `bench.benchmark@1` and `bench.run@1`.++ A benchmark is an owned, versioned and forkable set: its tasks, its judges (pinned by version), how they combine++ into one score, the limits a run may not pass, the default model and harness matrix, and whether it is public or++ unlisted. A run is one result: a benchmark version × a model × a harness × settings × who ran it, with the++ per-task scores, cost, time, tokens, the artifacts, and its provenance — `verified` (an executor ran it) or++ `submitted` (posted with no executor, badged and out of the default rankings).++- **Three new capabilities** for Bench: `bench.benchmark.write` (create, edit, freeze and fork a benchmark),++ `bench.run.publish` (publish a run) and `bench.run.read` (the run, the comparison, the rankings and the percentile).++ OpenVibe.Bench's service manifest lists them.++## 0.243.0 — 2026-10-30- The release cut from `main` for codes.task@2 (#250): a coding task's attempts and outcome, one id with two majors.@@@ -40,7 +40,8 @@"OpenVibe.Inventory goes live (inventory.openvibe.network, site position 30) with six capabilities active (ADR-054, plan T21 step 2); a definition may name the services or apps that may grant it","the Workshop v1 (ADR-054 section 6): Workshop kinds, network.badge, staff review (inventory.review-request@1, inventory.definition.review active) and media_id on equipped slots","News, Reviews, Trade, Coupons and OpenRestream go live on their own domains (plan T11); the product at openre.stream is OpenRestream, with OpenVibe Live on by default (live.openre.slot.bind)",-- "ADR-055 (proposed): money and the inventory — primary sales first, gifting next; trading and a market stay closed, and chance is never paid for"++ "ADR-055 (proposed): money and the inventory — primary sales first, gifting next; trading and a market stay closed, and chance is never paid for",++ "bench.benchmark@1 and bench.run@1 with the capabilities bench.benchmark.write, bench.run.publish and bench.run.read: the benchmark registry and the results API (spec 2026-10-30, stage B1)"],-- "updated": "2026-10-22"++ "updated": "2026-10-30"}nnew file mode 100644@@@ -0,0 +1,266 @@++{++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/bench/benchmark.v1.json",++ "title": "BenchBenchmark",++ "description": "bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.",++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "owner",++ "title",++ "description",++ "kind",++ "tags",++ "visibility",++ "version",++ "parent",++ "tasks",++ "judges",++ "scoring",++ "limits",++ "matrix",++ "created_at"++ ],++ "properties": {++ "id": {++ "type": "string",++ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}$",++ "description": "The benchmark id; the same id carries every version."++ },++ "owner": {++ "type": "string",++ "minLength": 1,++ "maxLength": 120,++ "description": "Who owns the set: a person (usr_…), a project (prj_…) or a service."++ },++ "title": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ },++ "description": {++ "type": "string",++ "maxLength": 4000++ },++ "kind": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80,++ "description": "One of Bench's kinds; the kind sets the renderer."++ },++ "tags": {++ "type": "array",++ "maxItems": 32,++ "items": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ }++ },++ "visibility": {++ "enum": [++ "public",++ "unlisted"++ ],++ "description": "public appears in the registry; unlisted is reachable by id only."++ },++ "version": {++ "type": "integer",++ "minimum": 1,++ "description": "The version number this document is."++ },++ "parent": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 120,++ "description": "The benchmark id@version this one was forked from, or null."++ },++ "tasks": {++ "type": "array",++ "maxItems": 200,++ "items": {++ "$ref": "#/$defs/task"++ }++ },++ "judges": {++ "type": "array",++ "maxItems": 20,++ "items": {++ "$ref": "#/$defs/judge"++ }++ },++ "scoring": {++ "$ref": "#/$defs/scoring"++ },++ "limits": {++ "$ref": "#/$defs/limits"++ },++ "matrix": {++ "$ref": "#/$defs/matrix"++ },++ "created_at": {++ "type": "string",++ "format": "date-time"++ }++ },++ "$defs": {++ "task": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "prompt",++ "holdout"++ ],++ "properties": {++ "id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "prompt": {++ "type": "string",++ "minLength": 1,++ "maxLength": 8000++ },++ "files": {++ "type": "array",++ "maxItems": 20,++ "items": {++ "type": "object",++ "additionalProperties": true++ }++ },++ "expected": {++ "type": [++ "object",++ "null"++ ],++ "additionalProperties": true,++ "description": "What a correct answer looks like; the judge reads it."++ },++ "holdout": {++ "type": "boolean",++ "description": "A hidden task: it is withheld from everyone but the executor, against tuning to the set."++ }++ }++ },++ "judge": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "type",++ "version",++ "config"++ ],++ "properties": {++ "type": {++ "type": "string",++ "minLength": 1,++ "maxLength": 60,++ "description": "human_pairwise | rubric | automatic_grader | test_suite | game_outcome"++ },++ "version": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40,++ "description": "The pinned judge version; a run names the judge it used."++ },++ "config": {++ "type": "object",++ "additionalProperties": true++ }++ }++ },++ "scoring": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "method",++ "scale"++ ],++ "properties": {++ "method": {++ "enum": [++ "mean",++ "weighted",++ "pass_rate",++ "sum"++ ]++ },++ "scale": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "min",++ "max"++ ],++ "properties": {++ "min": {++ "type": "number"++ },++ "max": {++ "type": "number"++ }++ },++ "description": "The score scale; 0-100 for Bench."++ },++ "weights": {++ "type": "object",++ "additionalProperties": {++ "type": "number"++ }++ },++ "aggregate": {++ "type": "string",++ "maxLength": 120++ }++ }++ },++ "limits": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "ms": {++ "type": "number",++ "minimum": 0,++ "description": "The time one task may take."++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "usd": {++ "type": "number",++ "minimum": 0,++ "description": "The money one run may spend."++ }++ }++ },++ "matrix": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "models": {++ "type": "array",++ "maxItems": 50,++ "items": {++ "type": "string",++ "maxLength": 160++ }++ },++ "harnesses": {++ "type": "array",++ "maxItems": 50,++ "items": {++ "type": "string",++ "maxLength": 160++ }++ }++ }++ }++ }++}nnew file mode 100644@@@ -0,0 +1,241 @@++{++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/bench/run.v1.json",++ "title": "BenchRun",++ "description": "bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.",++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "benchmark",++ "model",++ "route",++ "provider",++ "harness",++ "settings",++ "executor",++ "tasks",++ "score",++ "cost_usd",++ "ms",++ "tokens",++ "provenance",++ "created_at"++ ],++ "properties": {++ "id": {++ "type": "string",++ "pattern": "^brn_[0-9A-HJKMNP-TV-Z]{26}$"++ },++ "benchmark": {++ "type": "string",++ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$",++ "description": "The benchmark id@version the run names. The version is immutable once a run names it."++ },++ "model": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ },++ "route": {++ "type": "object",++ "additionalProperties": true,++ "description": "The route that produced the run."++ },++ "provider": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "harness": {++ "$ref": "#/$defs/harness"++ },++ "settings": {++ "$ref": "#/$defs/settings"++ },++ "executor": {++ "$ref": "#/$defs/executor"++ },++ "hardware": {++ "type": [++ "object",++ "null"++ ],++ "additionalProperties": true,++ "description": "The machine for a local model: GPU, CPU, RAM, quantization."++ },++ "tasks": {++ "type": "array",++ "maxItems": 200,++ "items": {++ "$ref": "#/$defs/task"++ }++ },++ "score": {++ "type": "number",++ "minimum": 0,++ "maximum": 100,++ "description": "The benchmark's one score for this run, 0-100."++ },++ "cost_usd": {++ "type": "number",++ "minimum": 0++ },++ "ms": {++ "type": "integer",++ "minimum": 0++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "provenance": {++ "enum": [++ "verified",++ "submitted"++ ],++ "description": "verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings."++ },++ "created_at": {++ "type": "string",++ "format": "date-time"++ }++ },++ "$defs": {++ "harness": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "name",++ "version"++ ],++ "properties": {++ "name": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "version": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ }++ }++ },++ "settings": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "temperature": {++ "type": [++ "number",++ "null"++ ],++ "minimum": 0,++ "maximum": 2++ },++ "effort": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 40++ }++ }++ },++ "executor": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "kind",++ "id"++ ],++ "properties": {++ "kind": {++ "enum": [++ "codes",++ "run",++ "node",++ "submitted"++ ],++ "description": "codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it."++ },++ "id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ }++ }++ },++ "task": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "task_id",++ "score",++ "cost_usd",++ "ms",++ "tokens",++ "artifacts"++ ],++ "properties": {++ "task_id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "score": {++ "type": "number",++ "minimum": 0,++ "maximum": 100++ },++ "cost_usd": {++ "type": "number",++ "minimum": 0++ },++ "ms": {++ "type": "integer",++ "minimum": 0++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "artifacts": {++ "type": "array",++ "maxItems": 40,++ "items": {++ "$ref": "#/$defs/artifact"++ }++ },++ "transcript_url": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 500++ }++ }++ },++ "artifact": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "kind",++ "url"++ ],++ "properties": {++ "kind": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ },++ "url": {++ "type": "string",++ "minLength": 1,++ "maxLength": 500++ }++ }++ }++ }++}@@@ -8171,6 +8171,24 @@"visibility": "public","compatibility": "backward","status": "active"++ },++ {++ "id": "bench.benchmark",++ "version": "1.0.0",++ "owner": "bench",++ "schema": "bench/benchmark.v1.json",++ "visibility": "first-party",++ "compatibility": "backward",++ "status": "active"++ },++ {++ "id": "bench.run",++ "version": "1.0.0",++ "owner": "bench",++ "schema": "bench/run.v1.json",++ "visibility": "first-party",++ "compatibility": "backward",++ "status": "active"}]}nnew file mode 100644@@@ -0,0 +1,60 @@++{++ "id": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS",++ "owner": "openvibe",++ "title": "Playable games arena",++ "description": "One-screen arcade games, judged by play and by the frame.",++ "kind": "games.play",++ "tags": [++ "games",++ "arcade"++ ],++ "visibility": "public",++ "version": 1,++ "parent": null,++ "tasks": [++ {++ "id": "t1",++ "prompt": "x"++ }++ ],++ "judges": [++ {++ "type": "human_pairwise",++ "version": "1.0",++ "config": {++ "blind": true++ }++ },++ {++ "type": "game_outcome",++ "version": "1.0",++ "config": {}++ }++ ],++ "scoring": {++ "method": "mean",++ "scale": {++ "min": 0,++ "max": 100++ },++ "weights": {++ "human_pairwise": 1++ }++ },++ "limits": {++ "ms": 120000,++ "tokens": 200000,++ "usd": 0.5++ },++ "matrix": {++ "models": [++ "claude-opus-4",++ "gpt-5"++ ],++ "harnesses": [++ "claude-code",++ "opencode"++ ]++ },++ "created_at": "2026-10-30T12:00:00.000Z"++}nnew file mode 100644@@@ -0,0 +1,69 @@++{++ "id": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS",++ "owner": "openvibe",++ "title": "Playable games arena",++ "description": "One-screen arcade games, judged by play and by the frame.",++ "kind": "games.play",++ "tags": [++ "games",++ "arcade"++ ],++ "visibility": "secret",++ "version": 1,++ "parent": null,++ "tasks": [++ {++ "id": "t1",++ "prompt": "Build a one-screen arcade game with a paddle, bricks, a ball and a score.",++ "holdout": false++ },++ {++ "id": "t2",++ "prompt": "Build a snake game on a 20x20 grid.",++ "holdout": true,++ "expected": {++ "score": "food eaten"++ }++ }++ ],++ "judges": [++ {++ "type": "human_pairwise",++ "version": "1.0",++ "config": {++ "blind": true++ }++ },++ {++ "type": "game_outcome",++ "version": "1.0",++ "config": {}++ }++ ],++ "scoring": {++ "method": "mean",++ "scale": {++ "min": 0,++ "max": 100++ },++ "weights": {++ "human_pairwise": 1++ }++ },++ "limits": {++ "ms": 120000,++ "tokens": 200000,++ "usd": 0.5++ },++ "matrix": {++ "models": [++ "claude-opus-4",++ "gpt-5"++ ],++ "harnesses": [++ "claude-code",++ "opencode"++ ]++ },++ "created_at": "2026-10-30T12:00:00.000Z"++}nnew file mode 100644@@@ -0,0 +1,69 @@++{++ "id": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS",++ "owner": "openvibe",++ "title": "Playable games arena",++ "description": "One-screen arcade games, judged by play and by the frame.",++ "kind": "games.play",++ "tags": [++ "games",++ "arcade"++ ],++ "visibility": "public",++ "version": 1,++ "parent": null,++ "tasks": [++ {++ "id": "t1",++ "prompt": "Build a one-screen arcade game with a paddle, bricks, a ball and a score.",++ "holdout": false++ },++ {++ "id": "t2",++ "prompt": "Build a snake game on a 20x20 grid.",++ "holdout": true,++ "expected": {++ "score": "food eaten"++ }++ }++ ],++ "judges": [++ {++ "type": "human_pairwise",++ "version": "1.0",++ "config": {++ "blind": true++ }++ },++ {++ "type": "game_outcome",++ "version": "1.0",++ "config": {}++ }++ ],++ "scoring": {++ "method": "mean",++ "scale": {++ "min": 0,++ "max": 100++ },++ "weights": {++ "human_pairwise": 1++ }++ },++ "limits": {++ "ms": 120000,++ "tokens": 200000,++ "usd": 0.5++ },++ "matrix": {++ "models": [++ "claude-opus-4",++ "gpt-5"++ ],++ "harnesses": [++ "claude-code",++ "opencode"++ ]++ },++ "created_at": "2026-10-30T12:00:00.000Z"++}nnew file mode 100644@@@ -0,0 +1,45 @@++{++ "id": "brn_01JAB2C3D4E5F6G7H8J9K0MNPS",++ "benchmark": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS@1",++ "model": "claude-opus-4",++ "route": {++ "provider": "anthropic",++ "via": "router"++ },++ "provider": "anthropic",++ "harness": {++ "name": "claude-code",++ "version": "1.2.0"++ },++ "settings": {++ "temperature": 1,++ "effort": "high"++ },++ "executor": {++ "kind": "codes",++ "id": "task_01JAB2C3D4E5F6G7H8J9K0MNPS"++ },++ "hardware": null,++ "tasks": [++ {++ "task_id": "t1",++ "score": 92,++ "cost_usd": 0.031,++ "ms": 41200,++ "tokens": 8400,++ "artifacts": [++ {++ "kind": "html",++ "url": "https://bench.openvibe.services/render/x.html"++ }++ ],++ "transcript_url": "https://bench.openvibe.services/runs/x/t1"++ }++ ],++ "score": 92,++ "cost_usd": 0.031,++ "ms": 41200,++ "tokens": 8400,++ "provenance": "trusted",++ "created_at": "2026-10-30T12:05:00.000Z"++}nnew file mode 100644@@@ -0,0 +1,45 @@++{++ "id": "brn_01JAB2C3D4E5F6G7H8J9K0MNPS",++ "benchmark": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS@1",++ "model": "claude-opus-4",++ "route": {++ "provider": "anthropic",++ "via": "router"++ },++ "provider": "anthropic",++ "harness": {++ "name": "claude-code",++ "version": "1.2.0"++ },++ "settings": {++ "temperature": 1,++ "effort": "high"++ },++ "executor": {++ "kind": "codes",++ "id": "task_01JAB2C3D4E5F6G7H8J9K0MNPS"++ },++ "hardware": null,++ "tasks": [++ {++ "task_id": "t1",++ "score": 92,++ "cost_usd": 0.031,++ "ms": 41200,++ "tokens": 8400,++ "artifacts": [++ {++ "kind": "html",++ "url": "https://bench.openvibe.services/render/x.html"++ }++ ],++ "transcript_url": "https://bench.openvibe.services/runs/x/t1"++ }++ ],++ "score": 140,++ "cost_usd": 0.031,++ "ms": 41200,++ "tokens": 8400,++ "provenance": "verified",++ "created_at": "2026-10-30T12:05:00.000Z"++}nnew file mode 100644@@@ -0,0 +1,45 @@++{++ "id": "brn_01JAB2C3D4E5F6G7H8J9K0MNPS",++ "benchmark": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS@1",++ "model": "claude-opus-4",++ "route": {++ "provider": "anthropic",++ "via": "router"++ },++ "provider": "anthropic",++ "harness": {++ "name": "claude-code",++ "version": "1.2.0"++ },++ "settings": {++ "temperature": 1,++ "effort": "high"++ },++ "executor": {++ "kind": "codes",++ "id": "task_01JAB2C3D4E5F6G7H8J9K0MNPS"++ },++ "hardware": null,++ "tasks": [++ {++ "task_id": "t1",++ "score": 92,++ "cost_usd": 0.031,++ "ms": 41200,++ "tokens": 8400,++ "artifacts": [++ {++ "kind": "html",++ "url": "https://bench.openvibe.services/render/x.html"++ }++ ],++ "transcript_url": "https://bench.openvibe.services/runs/x/t1"++ }++ ],++ "score": 92,++ "cost_usd": 0.031,++ "ms": 41200,++ "tokens": 8400,++ "provenance": "verified",++ "created_at": "2026-10-30T12:05:00.000Z"++}@@@ -90866,6 +90866,523 @@}]}++ },++ "bench.benchmark": {++ "version": "1.0.0",++ "owner": "bench",++ "status": "active",++ "schema": {++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/bench/benchmark.v1.json",++ "title": "BenchBenchmark",++ "description": "bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.",++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "owner",++ "title",++ "description",++ "kind",++ "tags",++ "visibility",++ "version",++ "parent",++ "tasks",++ "judges",++ "scoring",++ "limits",++ "matrix",++ "created_at"++ ],++ "properties": {++ "id": {++ "type": "string",++ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}$",++ "description": "The benchmark id; the same id carries every version."++ },++ "owner": {++ "type": "string",++ "minLength": 1,++ "maxLength": 120,++ "description": "Who owns the set: a person (usr_…), a project (prj_…) or a service."++ },++ "title": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ },++ "description": {++ "type": "string",++ "maxLength": 4000++ },++ "kind": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80,++ "description": "One of Bench's kinds; the kind sets the renderer."++ },++ "tags": {++ "type": "array",++ "maxItems": 32,++ "items": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ }++ },++ "visibility": {++ "enum": [++ "public",++ "unlisted"++ ],++ "description": "public appears in the registry; unlisted is reachable by id only."++ },++ "version": {++ "type": "integer",++ "minimum": 1,++ "description": "The version number this document is."++ },++ "parent": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 120,++ "description": "The benchmark id@version this one was forked from, or null."++ },++ "tasks": {++ "type": "array",++ "maxItems": 200,++ "items": {++ "$ref": "#/$defs/task"++ }++ },++ "judges": {++ "type": "array",++ "maxItems": 20,++ "items": {++ "$ref": "#/$defs/judge"++ }++ },++ "scoring": {++ "$ref": "#/$defs/scoring"++ },++ "limits": {++ "$ref": "#/$defs/limits"++ },++ "matrix": {++ "$ref": "#/$defs/matrix"++ },++ "created_at": {++ "type": "string",++ "format": "date-time"++ }++ },++ "$defs": {++ "task": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "prompt",++ "holdout"++ ],++ "properties": {++ "id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "prompt": {++ "type": "string",++ "minLength": 1,++ "maxLength": 8000++ },++ "files": {++ "type": "array",++ "maxItems": 20,++ "items": {++ "type": "object",++ "additionalProperties": true++ }++ },++ "expected": {++ "type": [++ "object",++ "null"++ ],++ "additionalProperties": true,++ "description": "What a correct answer looks like; the judge reads it."++ },++ "holdout": {++ "type": "boolean",++ "description": "A hidden task: it is withheld from everyone but the executor, against tuning to the set."++ }++ }++ },++ "judge": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "type",++ "version",++ "config"++ ],++ "properties": {++ "type": {++ "type": "string",++ "minLength": 1,++ "maxLength": 60,++ "description": "human_pairwise | rubric | automatic_grader | test_suite | game_outcome"++ },++ "version": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40,++ "description": "The pinned judge version; a run names the judge it used."++ },++ "config": {++ "type": "object",++ "additionalProperties": true++ }++ }++ },++ "scoring": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "method",++ "scale"++ ],++ "properties": {++ "method": {++ "enum": [++ "mean",++ "weighted",++ "pass_rate",++ "sum"++ ]++ },++ "scale": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "min",++ "max"++ ],++ "properties": {++ "min": {++ "type": "number"++ },++ "max": {++ "type": "number"++ }++ },++ "description": "The score scale; 0-100 for Bench."++ },++ "weights": {++ "type": "object",++ "additionalProperties": {++ "type": "number"++ }++ },++ "aggregate": {++ "type": "string",++ "maxLength": 120++ }++ }++ },++ "limits": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "ms": {++ "type": "number",++ "minimum": 0,++ "description": "The time one task may take."++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "usd": {++ "type": "number",++ "minimum": 0,++ "description": "The money one run may spend."++ }++ }++ },++ "matrix": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "models": {++ "type": "array",++ "maxItems": 50,++ "items": {++ "type": "string",++ "maxLength": 160++ }++ },++ "harnesses": {++ "type": "array",++ "maxItems": 50,++ "items": {++ "type": "string",++ "maxLength": 160++ }++ }++ }++ }++ }++ }++ },++ "bench.run": {++ "version": "1.0.0",++ "owner": "bench",++ "status": "active",++ "schema": {++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/bench/run.v1.json",++ "title": "BenchRun",++ "description": "bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.",++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "benchmark",++ "model",++ "route",++ "provider",++ "harness",++ "settings",++ "executor",++ "tasks",++ "score",++ "cost_usd",++ "ms",++ "tokens",++ "provenance",++ "created_at"++ ],++ "properties": {++ "id": {++ "type": "string",++ "pattern": "^brn_[0-9A-HJKMNP-TV-Z]{26}$"++ },++ "benchmark": {++ "type": "string",++ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$",++ "description": "The benchmark id@version the run names. The version is immutable once a run names it."++ },++ "model": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ },++ "route": {++ "type": "object",++ "additionalProperties": true,++ "description": "The route that produced the run."++ },++ "provider": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "harness": {++ "$ref": "#/$defs/harness"++ },++ "settings": {++ "$ref": "#/$defs/settings"++ },++ "executor": {++ "$ref": "#/$defs/executor"++ },++ "hardware": {++ "type": [++ "object",++ "null"++ ],++ "additionalProperties": true,++ "description": "The machine for a local model: GPU, CPU, RAM, quantization."++ },++ "tasks": {++ "type": "array",++ "maxItems": 200,++ "items": {++ "$ref": "#/$defs/task"++ }++ },++ "score": {++ "type": "number",++ "minimum": 0,++ "maximum": 100,++ "description": "The benchmark's one score for this run, 0-100."++ },++ "cost_usd": {++ "type": "number",++ "minimum": 0++ },++ "ms": {++ "type": "integer",++ "minimum": 0++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "provenance": {++ "enum": [++ "verified",++ "submitted"++ ],++ "description": "verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings."++ },++ "created_at": {++ "type": "string",++ "format": "date-time"++ }++ },++ "$defs": {++ "harness": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "name",++ "version"++ ],++ "properties": {++ "name": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "version": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ }++ }++ },++ "settings": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "temperature": {++ "type": [++ "number",++ "null"++ ],++ "minimum": 0,++ "maximum": 2++ },++ "effort": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 40++ }++ }++ },++ "executor": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "kind",++ "id"++ ],++ "properties": {++ "kind": {++ "enum": [++ "codes",++ "run",++ "node",++ "submitted"++ ],++ "description": "codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it."++ },++ "id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ }++ }++ },++ "task": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "task_id",++ "score",++ "cost_usd",++ "ms",++ "tokens",++ "artifacts"++ ],++ "properties": {++ "task_id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "score": {++ "type": "number",++ "minimum": 0,++ "maximum": 100++ },++ "cost_usd": {++ "type": "number",++ "minimum": 0++ },++ "ms": {++ "type": "integer",++ "minimum": 0++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "artifacts": {++ "type": "array",++ "maxItems": 40,++ "items": {++ "$ref": "#/$defs/artifact"++ }++ },++ "transcript_url": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 500++ }++ }++ },++ "artifact": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "kind",++ "url"++ ],++ "properties": {++ "kind": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ },++ "url": {++ "type": "string",++ "minLength": 1,++ "maxLength": 500++ }++ }++ }++ }++ }}}}@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Actor API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Actor as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Actor as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.AI API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.AI as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.AI as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Bench API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Bench as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Bench as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -16,6 +16,260 @@}],"paths": {++ "/api/v1/benchmarks": {++ "post": {++ "operationId": "post_api_v1_benchmarks",++ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",++ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",++ "tags": [++ "bench.benchmark"++ ],++ "x-openvibe-capabilities": [++ "bench.benchmark.write"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.benchmark.write"++ ]++ }++ ],++ "requestBody": {++ "required": false,++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },++ "/api/v1/benchmarks/{id}": {++ "put": {++ "operationId": "put_api_v1_benchmarks_id",++ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",++ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",++ "tags": [++ "bench.benchmark"++ ],++ "x-openvibe-capabilities": [++ "bench.benchmark.write"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.benchmark.write"++ ]++ }++ ],++ "requestBody": {++ "required": false,++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "parameters": [++ {++ "name": "id",++ "in": "path",++ "required": true,++ "schema": {++ "type": "string"++ }++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },++ "/api/v1/benchmarks/{id}/fork": {++ "post": {++ "operationId": "post_api_v1_benchmarks_id_fork",++ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",++ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",++ "tags": [++ "bench.benchmark"++ ],++ "x-openvibe-capabilities": [++ "bench.benchmark.write"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.benchmark.write"++ ]++ }++ ],++ "requestBody": {++ "required": false,++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "parameters": [++ {++ "name": "id",++ "in": "path",++ "required": true,++ "schema": {++ "type": "string"++ }++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },++ "/api/v1/benchmarks/{id}/versions": {++ "post": {++ "operationId": "post_api_v1_benchmarks_id_versions",++ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",++ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",++ "tags": [++ "bench.benchmark"++ ],++ "x-openvibe-capabilities": [++ "bench.benchmark.write"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.benchmark.write"++ ]++ }++ ],++ "requestBody": {++ "required": false,++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "parameters": [++ {++ "name": "id",++ "in": "path",++ "required": true,++ "schema": {++ "type": "string"++ }++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.benchmark.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },"/api/v1/boards/{kindId}": {"get": {"operationId": "get_api_v1_boards_kindId",@@@ -72,6 +326,52 @@}}},++ "/api/v1/compare": {++ "get": {++ "operationId": "get_api_v1_compare",++ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",++ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.",++ "tags": [++ "bench.run"++ ],++ "x-openvibe-capabilities": [++ "bench.run.read"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.run.read"++ ]++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.run.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },"/api/v1/datasets/{kindId}.jsonl": {"get": {"operationId": "get_api_v1_datasets_kindId_jsonl",@@@ -193,13 +493,227 @@"security": [{"openvibeToken": [-- "bench.eval.manage"++ "bench.eval.manage"++ ]++ }++ ],++ "parameters": [++ {++ "name": "id",++ "in": "path",++ "required": true,++ "schema": {++ "type": "string"++ }++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.task.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },++ "/api/v1/evals/{id}/run": {++ "post": {++ "operationId": "post_api_v1_evals_id_run",++ "summary": "Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eva",++ "description": "**bench.eval.manage** (first-party): Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eval and its runs. Never in the public dataset.",++ "tags": [++ "bench.eval"++ ],++ "x-openvibe-capabilities": [++ "bench.eval.manage"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.eval.manage"++ ]++ }++ ],++ "parameters": [++ {++ "name": "id",++ "in": "path",++ "required": true,++ "schema": {++ "type": "string"++ }++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.task.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },++ "/api/v1/kinds": {++ "get": {++ "operationId": "get_api_v1_kinds",++ "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",++ "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",++ "tags": [++ "bench.kind"++ ],++ "x-openvibe-capabilities": [++ "bench.kind.read"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.kind.read"++ ]++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.kind.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },++ "/api/v1/kinds/{kindId}": {++ "get": {++ "operationId": "get_api_v1_kinds_kindId",++ "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",++ "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",++ "tags": [++ "bench.kind"++ ],++ "x-openvibe-capabilities": [++ "bench.kind.read"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.kind.read"++ ]++ }++ ],++ "parameters": [++ {++ "name": "kindId",++ "in": "path",++ "required": true,++ "schema": {++ "type": "string"++ }++ }++ ],++ "responses": {++ "2XX": {++ "description": "Success",++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.kind.v1"++ }++ }++ }++ },++ "default": {++ "description": "An error (RFC 9457 problem details)",++ "content": {++ "application/problem+json": {++ "schema": {++ "$ref": "#/components/schemas/errors.problem.v1"++ }++ }++ }++ }++ }++ }++ },++ "/api/v1/kinds/{kindId}/tasks": {++ "post": {++ "operationId": "post_api_v1_kinds_kindId_tasks",++ "summary": "Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public?",++ "description": "**bench.task.create** (first-party): Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public? }. A room or a person's own token may create a task.",++ "tags": [++ "bench.task"++ ],++ "x-openvibe-capabilities": [++ "bench.task.create"++ ],++ "x-openvibe-visibility": [++ "first-party"++ ],++ "x-openvibe-quota-class": "standard",++ "security": [++ {++ "openvibeToken": [++ "bench.task.create"]}],"parameters": [{-- "name": "id",++ "name": "kindId","in": "path","required": true,"schema": {@@@ -231,16 +745,16 @@}}},-- "/api/v1/evals/{id}/run": {-- "post": {-- "operationId": "post_api_v1_evals_id_run",-- "summary": "Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eva",-- "description": "**bench.eval.manage** (first-party): Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eval and its runs. Never in the public dataset.",++ "/api/v1/percentile": {++ "get": {++ "operationId": "get_api_v1_percentile",++ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",++ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.","tags": [-- "bench.eval"++ "bench.run"],"x-openvibe-capabilities": [-- "bench.eval.manage"++ "bench.run.read"],"x-openvibe-visibility": ["first-party"@@@ -249,27 +763,17 @@"security": [{"openvibeToken": [-- "bench.eval.manage"++ "bench.run.read"]}],-- "parameters": [-- {-- "name": "id",-- "in": "path",-- "required": true,-- "schema": {-- "type": "string"-- }-- }-- ],"responses": {"2XX": {"description": "Success","content": {"application/json": {"schema": {-- "$ref": "#/components/schemas/bench.task.v1"++ "$ref": "#/components/schemas/bench.run.v1"}}}@@@ -287,16 +791,16 @@}}},-- "/api/v1/kinds": {++ "/api/v1/rankings": {"get": {-- "operationId": "get_api_v1_kinds",-- "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",-- "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",++ "operationId": "get_api_v1_rankings",++ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",++ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.","tags": [-- "bench.kind"++ "bench.run"],"x-openvibe-capabilities": [-- "bench.kind.read"++ "bench.run.read"],"x-openvibe-visibility": ["first-party"@@@ -305,7 +809,7 @@"security": [{"openvibeToken": [-- "bench.kind.read"++ "bench.run.read"]}],@@@ -315,7 +819,7 @@"content": {"application/json": {"schema": {-- "$ref": "#/components/schemas/bench.kind.v1"++ "$ref": "#/components/schemas/bench.run.v1"}}}@@@ -333,16 +837,16 @@}}},-- "/api/v1/kinds/{kindId}": {++ "/api/v1/rounds/{id}": {"get": {-- "operationId": "get_api_v1_kinds_kindId",-- "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",-- "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",++ "operationId": "get_api_v1_rounds_id",++ "summary": "Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.",++ "description": "**bench.round.read** (first-party): Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.","tags": [-- "bench.kind"++ "bench.round"],"x-openvibe-capabilities": [-- "bench.kind.read"++ "bench.round.read"],"x-openvibe-visibility": ["first-party"@@@ -351,13 +855,13 @@"security": [{"openvibeToken": [-- "bench.kind.read"++ "bench.round.read"]}],"parameters": [{-- "name": "kindId",++ "name": "id","in": "path","required": true,"schema": {@@@ -371,7 +875,7 @@"content": {"application/json": {"schema": {-- "$ref": "#/components/schemas/bench.kind.v1"++ "$ref": "#/components/schemas/bench.round.v1"}}}@@@ -389,16 +893,16 @@}}},-- "/api/v1/kinds/{kindId}/tasks": {++ "/api/v1/rounds/{id}/entries": {"post": {-- "operationId": "post_api_v1_kinds_kindId_tasks",-- "summary": "Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public?",-- "description": "**bench.task.create** (first-party): Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public? }. A room or a person's own token may create a task.",++ "operationId": "post_api_v1_rounds_id_entries",++ "summary": "Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token.",++ "description": "**bench.entry.create** (first-party): Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token. A room submits a game match, a test-suite run or another room's result.","tags": [-- "bench.task"++ "bench.entry"],"x-openvibe-capabilities": [-- "bench.task.create"++ "bench.entry.create"],"x-openvibe-visibility": ["first-party"@@@ -407,13 +911,13 @@"security": [{"openvibeToken": [-- "bench.task.create"++ "bench.entry.create"]}],"parameters": [{-- "name": "kindId",++ "name": "id","in": "path","required": true,"schema": {@@@ -427,7 +931,7 @@"content": {"application/json": {"schema": {-- "$ref": "#/components/schemas/bench.task.v1"++ "$ref": "#/components/schemas/bench.entry.v1"}}}@@@ -445,16 +949,16 @@}}},-- "/api/v1/rounds/{id}": {-- "get": {-- "operationId": "get_api_v1_rounds_id",-- "summary": "Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.",-- "description": "**bench.round.read** (first-party): Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.",++ "/api/v1/rounds/{id}/judgements": {++ "post": {++ "operationId": "post_api_v1_rounds_id_judgements",++ "summary": "Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score?",++ "description": "**bench.judgement.cast** (first-party): Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score? }. Human judges are people; automatic, test-suite and game-outcome judges are services.","tags": [-- "bench.round"++ "bench.judgement"],"x-openvibe-capabilities": [-- "bench.round.read"++ "bench.judgement.cast"],"x-openvibe-visibility": ["first-party"@@@ -463,7 +967,7 @@"security": [{"openvibeToken": [-- "bench.round.read"++ "bench.judgement.cast"]}],@@@ -483,7 +987,7 @@"content": {"application/json": {"schema": {-- "$ref": "#/components/schemas/bench.round.v1"++ "$ref": "#/components/schemas/bench.judgement.v1"}}}@@@ -501,16 +1005,16 @@}}},-- "/api/v1/rounds/{id}/entries": {++ "/api/v1/runs": {"post": {-- "operationId": "post_api_v1_rounds_id_entries",-- "summary": "Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token.",-- "description": "**bench.entry.create** (first-party): Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token. A room submits a game match, a test-suite run or another room's result.",++ "operationId": "post_api_v1_runs",++ "summary": "Publish a run of a benchmark on OpenVibe.Bench: POST /api/v1/runs with the per-task scores, cost, time and tokens.",++ "description": "**bench.run.publish** (first-party): Publish a run of a benchmark on OpenVibe.Bench: POST /api/v1/runs with the per-task scores, cost, time and tokens. A service token (openvibe.codes, Run) makes a verified run; a person's token makes a submitted run, which is badged and kept out of the default rankings. A caller with this capability sees a benchmark's holdout tasks.","tags": [-- "bench.entry"++ "bench.run"],"x-openvibe-capabilities": [-- "bench.entry.create"++ "bench.run.publish"],"x-openvibe-visibility": ["first-party"@@@ -519,27 +1023,27 @@"security": [{"openvibeToken": [-- "bench.entry.create"++ "bench.run.publish"]}],-- "parameters": [-- {-- "name": "id",-- "in": "path",-- "required": true,-- "schema": {-- "type": "string"++ "requestBody": {++ "required": false,++ "content": {++ "application/json": {++ "schema": {++ "$ref": "#/components/schemas/bench.run.v1"++ }}}-- ],++ },"responses": {"2XX": {"description": "Success","content": {"application/json": {"schema": {-- "$ref": "#/components/schemas/bench.entry.v1"++ "$ref": "#/components/schemas/bench.run.v1"}}}@@@ -557,16 +1061,16 @@}}},-- "/api/v1/rounds/{id}/judgements": {-- "post": {-- "operationId": "post_api_v1_rounds_id_judgements",-- "summary": "Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score?",-- "description": "**bench.judgement.cast** (first-party): Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score? }. Human judges are people; automatic, test-suite and game-outcome judges are services.",++ "/api/v1/runs/{id}": {++ "get": {++ "operationId": "get_api_v1_runs_id",++ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",++ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.","tags": [-- "bench.judgement"++ "bench.run"],"x-openvibe-capabilities": [-- "bench.judgement.cast"++ "bench.run.read"],"x-openvibe-visibility": ["first-party"@@@ -575,7 +1079,7 @@"security": [{"openvibeToken": [-- "bench.judgement.cast"++ "bench.run.read"]}],@@@ -595,7 +1099,7 @@"content": {"application/json": {"schema": {-- "$ref": "#/components/schemas/bench.judgement.v1"++ "$ref": "#/components/schemas/bench.run.v1"}}}@@@ -672,6 +1176,270 @@},"components": {"schemas": {++ "bench.benchmark.v1": {++ "title": "BenchBenchmark",++ "description": "bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.",++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "owner",++ "title",++ "description",++ "kind",++ "tags",++ "visibility",++ "version",++ "parent",++ "tasks",++ "judges",++ "scoring",++ "limits",++ "matrix",++ "created_at"++ ],++ "properties": {++ "id": {++ "type": "string",++ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}$",++ "description": "The benchmark id; the same id carries every version."++ },++ "owner": {++ "type": "string",++ "minLength": 1,++ "maxLength": 120,++ "description": "Who owns the set: a person (usr_…), a project (prj_…) or a service."++ },++ "title": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ },++ "description": {++ "type": "string",++ "maxLength": 4000++ },++ "kind": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80,++ "description": "One of Bench's kinds; the kind sets the renderer."++ },++ "tags": {++ "type": "array",++ "maxItems": 32,++ "items": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ }++ },++ "visibility": {++ "enum": [++ "public",++ "unlisted"++ ],++ "description": "public appears in the registry; unlisted is reachable by id only."++ },++ "version": {++ "type": "integer",++ "minimum": 1,++ "description": "The version number this document is."++ },++ "parent": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 120,++ "description": "The benchmark id@version this one was forked from, or null."++ },++ "tasks": {++ "type": "array",++ "maxItems": 200,++ "items": {++ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/task"++ }++ },++ "judges": {++ "type": "array",++ "maxItems": 20,++ "items": {++ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/judge"++ }++ },++ "scoring": {++ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/scoring"++ },++ "limits": {++ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/limits"++ },++ "matrix": {++ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/matrix"++ },++ "created_at": {++ "type": "string",++ "format": "date-time"++ }++ },++ "$defs": {++ "task": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "prompt",++ "holdout"++ ],++ "properties": {++ "id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "prompt": {++ "type": "string",++ "minLength": 1,++ "maxLength": 8000++ },++ "files": {++ "type": "array",++ "maxItems": 20,++ "items": {++ "type": "object",++ "additionalProperties": true++ }++ },++ "expected": {++ "type": [++ "object",++ "null"++ ],++ "additionalProperties": true,++ "description": "What a correct answer looks like; the judge reads it."++ },++ "holdout": {++ "type": "boolean",++ "description": "A hidden task: it is withheld from everyone but the executor, against tuning to the set."++ }++ }++ },++ "judge": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "type",++ "version",++ "config"++ ],++ "properties": {++ "type": {++ "type": "string",++ "minLength": 1,++ "maxLength": 60,++ "description": "human_pairwise | rubric | automatic_grader | test_suite | game_outcome"++ },++ "version": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40,++ "description": "The pinned judge version; a run names the judge it used."++ },++ "config": {++ "type": "object",++ "additionalProperties": true++ }++ }++ },++ "scoring": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "method",++ "scale"++ ],++ "properties": {++ "method": {++ "enum": [++ "mean",++ "weighted",++ "pass_rate",++ "sum"++ ]++ },++ "scale": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "min",++ "max"++ ],++ "properties": {++ "min": {++ "type": "number"++ },++ "max": {++ "type": "number"++ }++ },++ "description": "The score scale; 0-100 for Bench."++ },++ "weights": {++ "type": "object",++ "additionalProperties": {++ "type": "number"++ }++ },++ "aggregate": {++ "type": "string",++ "maxLength": 120++ }++ }++ },++ "limits": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "ms": {++ "type": "number",++ "minimum": 0,++ "description": "The time one task may take."++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "usd": {++ "type": "number",++ "minimum": 0,++ "description": "The money one run may spend."++ }++ }++ },++ "matrix": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "models": {++ "type": "array",++ "maxItems": 50,++ "items": {++ "type": "string",++ "maxLength": 160++ }++ },++ "harnesses": {++ "type": "array",++ "maxItems": 50,++ "items": {++ "type": "string",++ "maxLength": 160++ }++ }++ }++ }++ }++ },"bench.entry.v1": {"title": "BenchEntry","description": "bench.entry@1: one submitted entry in a benchmark round on OpenVibe.Bench. PROVIDER and MODEL name the offer; ROUTE is the exact route taken; PAYLOAD is the entry's output; COST_USD, MS and TOKENS are the measured use.",@@@ -981,6 +1749,245 @@}}},++ "bench.run.v1": {++ "title": "BenchRun",++ "description": "bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.",++ "type": "object",++ "additionalProperties": false,++ "required": [++ "id",++ "benchmark",++ "model",++ "route",++ "provider",++ "harness",++ "settings",++ "executor",++ "tasks",++ "score",++ "cost_usd",++ "ms",++ "tokens",++ "provenance",++ "created_at"++ ],++ "properties": {++ "id": {++ "type": "string",++ "pattern": "^brn_[0-9A-HJKMNP-TV-Z]{26}$"++ },++ "benchmark": {++ "type": "string",++ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$",++ "description": "The benchmark id@version the run names. The version is immutable once a run names it."++ },++ "model": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ },++ "route": {++ "type": "object",++ "additionalProperties": true,++ "description": "The route that produced the run."++ },++ "provider": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "harness": {++ "$ref": "#/components/schemas/bench.run.v1/$defs/harness"++ },++ "settings": {++ "$ref": "#/components/schemas/bench.run.v1/$defs/settings"++ },++ "executor": {++ "$ref": "#/components/schemas/bench.run.v1/$defs/executor"++ },++ "hardware": {++ "type": [++ "object",++ "null"++ ],++ "additionalProperties": true,++ "description": "The machine for a local model: GPU, CPU, RAM, quantization."++ },++ "tasks": {++ "type": "array",++ "maxItems": 200,++ "items": {++ "$ref": "#/components/schemas/bench.run.v1/$defs/task"++ }++ },++ "score": {++ "type": "number",++ "minimum": 0,++ "maximum": 100,++ "description": "The benchmark's one score for this run, 0-100."++ },++ "cost_usd": {++ "type": "number",++ "minimum": 0++ },++ "ms": {++ "type": "integer",++ "minimum": 0++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "provenance": {++ "enum": [++ "verified",++ "submitted"++ ],++ "description": "verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings."++ },++ "created_at": {++ "type": "string",++ "format": "date-time"++ }++ },++ "$defs": {++ "harness": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "name",++ "version"++ ],++ "properties": {++ "name": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "version": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ }++ }++ },++ "settings": {++ "type": "object",++ "additionalProperties": false,++ "properties": {++ "temperature": {++ "type": [++ "number",++ "null"++ ],++ "minimum": 0,++ "maximum": 2++ },++ "effort": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 40++ }++ }++ },++ "executor": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "kind",++ "id"++ ],++ "properties": {++ "kind": {++ "enum": [++ "codes",++ "run",++ "node",++ "submitted"++ ],++ "description": "codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it."++ },++ "id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 160++ }++ }++ },++ "task": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "task_id",++ "score",++ "cost_usd",++ "ms",++ "tokens",++ "artifacts"++ ],++ "properties": {++ "task_id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 80++ },++ "score": {++ "type": "number",++ "minimum": 0,++ "maximum": 100++ },++ "cost_usd": {++ "type": "number",++ "minimum": 0++ },++ "ms": {++ "type": "integer",++ "minimum": 0++ },++ "tokens": {++ "type": "integer",++ "minimum": 0++ },++ "artifacts": {++ "type": "array",++ "maxItems": 40,++ "items": {++ "$ref": "#/components/schemas/bench.run.v1/$defs/artifact"++ }++ },++ "transcript_url": {++ "type": [++ "string",++ "null"++ ],++ "maxLength": 500++ }++ }++ },++ "artifact": {++ "type": "object",++ "additionalProperties": false,++ "required": [++ "kind",++ "url"++ ],++ "properties": {++ "kind": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40++ },++ "url": {++ "type": "string",++ "minLength": 1,++ "maxLength": 500++ }++ }++ }++ }++ },"bench.task.v1": {"title": "BenchTask","description": "bench.task@1: a benchmark task on OpenVibe.Bench. KIND_ID names the kind; PROMPT is the task; TASK_CLASS is an optional classifier for grouping; PUBLIC says whether the task may appear in the open dataset.",@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Billing API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Billing as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Billing as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Blog API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Blog as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Blog as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Bot API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Bot as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Bot as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Chat API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Chat as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Chat as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Codes API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Codes as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Codes as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Community API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Community as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Community as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Coupons API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Coupons as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Coupons as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Deals API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Deals as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Deals as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Domains API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Domains as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Domains as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Events API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Events as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Events as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Food API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Food as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Food as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Games API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Games as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Games as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Host API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Host as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Host as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -19,8 +19,8 @@"service": "bench","name": "OpenVibe.Bench","origin": "https://bench.openvibe.services",-- "operations": 12,-- "capabilities": 9++ "operations": 21,++ "capabilities": 12},{"service": "billing",@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Inventory API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Inventory as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Inventory as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Live API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Live as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Live as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.MediaHub API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.MediaHub as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.MediaHub as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Media API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Media as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Media as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Network API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Network as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Network as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.News API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.News as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.News as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenRestream API",-- "version": "0.243.0",-- "description": "Routes of OpenRestream as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenRestream as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Play API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Play as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Play as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Quest API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Quest as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Quest as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Reviews API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Reviews as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Reviews as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Run API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Run as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Run as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Search API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Search as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Search as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Services API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Services as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Services as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Sources API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Sources as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Sources as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Space API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Space as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Space as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Tips API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Tips as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Tips as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Tools API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Tools as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Tools as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Trade API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Trade as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Trade as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.VIP API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.VIP as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.VIP as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Vision API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Vision as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Vision as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Watch API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Watch as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Watch as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Wiki API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Wiki as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Wiki as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Work API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Work as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Work as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -3,8 +3,8 @@"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema","info": {"title": "OpenVibe.Zone API",-- "version": "0.243.0",-- "description": "Routes of OpenVibe.Zone as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",++ "version": "0.244.0",++ "description": "Routes of OpenVibe.Zone as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.","license": {"name": "MIT","identifier": "MIT"@@@ -215095,3 +215095,3614 @@ export type ReportMetric = {} & {[k: string]: unknown | undefined;};++++/** [email protected] (owner: bench) */++/**++ * bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.++ */++export interface BenchBenchmark {++ /**++ * The benchmark id; the same id carries every version.++ */++ id: string;++ /**++ * Who owns the set: a person (usr_…), a project (prj_…) or a service.++ */++ owner: string;++ title: string;++ description: string;++ /**++ * One of Bench's kinds; the kind sets the renderer.++ */++ kind: string;++ /**++ * @maxItems 32++ */++ tags: string[];++ /**++ * public appears in the registry; unlisted is reachable by id only.++ */++ visibility: "public" | "unlisted";++ /**++ * The version number this document is.++ */++ version: number;++ /**++ * The benchmark id@version this one was forked from, or null.++ */++ parent: string | null;++ /**++ * @maxItems 200++ */++ tasks: {++ id: string;++ prompt: string;++ /**++ * @maxItems 20++ */++ files?:++ | []++ | [++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ]++ | [++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ },++ {++ [k: string]: unknown | undefined;++ }++ ];++ /**++ * What a correct answer looks like; the judge reads it.++ */++ expected?: {++ [k: string]: unknown | undefined;++ } | null;++ /**++ * A hidden task: it is withheld from everyone but the executor, against tuning to the set.++ */++ holdout: boolean;++ }[];++ /**++ * @maxItems 20++ */++ judges:++ | []++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ]++ | [++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ },++ {++ /**++ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome++ */++ type: string;++ /**++ * The pinned judge version; a run names the judge it used.++ */++ version: string;++ config: {++ [k: string]: unknown | undefined;++ };++ }++ ];++ scoring: {++ method: "mean" | "weighted" | "pass_rate" | "sum";++ /**++ * The score scale; 0-100 for Bench.++ */++ scale: {++ min: number;++ max: number;++ };++ weights?: {++ [k: string]: number | undefined;++ };++ aggregate?: string;++ };++ limits: {++ /**++ * The time one task may take.++ */++ ms?: number;++ tokens?: number;++ /**++ * The money one run may spend.++ */++ usd?: number;++ };++ matrix: {++ /**++ * @maxItems 50++ */++ models?: string[];++ /**++ * @maxItems 50++ */++ harnesses?: string[];++ };++ created_at: string;++}++++/** [email protected] (owner: bench) */++/**++ * bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.++ */++export interface BenchRun {++ id: string;++ /**++ * The benchmark id@version the run names. The version is immutable once a run names it.++ */++ benchmark: string;++ model: string;++ /**++ * The route that produced the run.++ */++ route: {++ [k: string]: unknown | undefined;++ };++ provider: string;++ harness: {++ name: string;++ version: string;++ };++ settings: {++ temperature?: number | null;++ effort?: string | null;++ };++ executor: {++ /**++ * codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it.++ */++ kind: "codes" | "run" | "node" | "submitted";++ id: string;++ };++ /**++ * The machine for a local model: GPU, CPU, RAM, quantization.++ */++ hardware?: {++ [k: string]: unknown | undefined;++ } | null;++ /**++ * @maxItems 200++ */++ tasks: {++ task_id: string;++ score: number;++ cost_usd: number;++ ms: number;++ tokens: number;++ /**++ * @maxItems 40++ */++ artifacts: {++ kind: string;++ url: string;++ }[];++ transcript_url?: string | null;++ }[];++ /**++ * The benchmark's one score for this run, 0-100.++ */++ score: number;++ cost_usd: number;++ ms: number;++ tokens: number;++ /**++ * verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings.++ */++ provenance: "verified" | "submitted";++ created_at: string;++}nnew file mode 100644@@@ -0,0 +1,24 @@++{++ "id": "bench.benchmark.write",++ "version": "1.0",++ "owner": "bench",++ "status": "active",++ "visibility": "first-party",++ "description": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",++ "permissions": [++ "bench:benchmarks:write"++ ],++ "resourceConstraints": [++ "owner"++ ],++ "quotaClass": "standard",++ "events": [],++ "implementedBy": [++ "POST /api/v1/benchmarks",++ "PUT /api/v1/benchmarks/:id",++ "POST /api/v1/benchmarks/:id/versions",++ "POST /api/v1/benchmarks/:id/fork"++ ],++ "inputSchema": "bench.benchmark@1",++ "outputSchema": "bench.benchmark@1"++}nnew file mode 100644@@@ -0,0 +1,21 @@++{++ "id": "bench.run.publish",++ "version": "1.0",++ "owner": "bench",++ "status": "active",++ "visibility": "first-party",++ "description": "Publish a run of a benchmark on OpenVibe.Bench: POST /api/v1/runs with the per-task scores, cost, time and tokens. A service token (openvibe.codes, Run) makes a verified run; a person's token makes a submitted run, which is badged and kept out of the default rankings. A caller with this capability sees a benchmark's holdout tasks.",++ "permissions": [++ "bench:runs:publish"++ ],++ "resourceConstraints": [++ "none"++ ],++ "quotaClass": "standard",++ "events": [],++ "implementedBy": [++ "POST /api/v1/runs"++ ],++ "inputSchema": "bench.run@1",++ "outputSchema": "bench.run@1"++}nnew file mode 100644@@@ -0,0 +1,24 @@++{++ "id": "bench.run.read",++ "version": "1.0",++ "owner": "bench",++ "status": "active",++ "visibility": "first-party",++ "description": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.",++ "permissions": [++ "bench:runs:read"++ ],++ "resourceConstraints": [++ "none"++ ],++ "quotaClass": "standard",++ "events": [],++ "implementedBy": [++ "GET /api/v1/runs/:id",++ "GET /api/v1/compare",++ "GET /api/v1/rankings",++ "GET /api/v1/percentile"++ ],++ "inputSchema": "common.no-body@1",++ "outputSchema": "bench.run@1"++}@@@ -28,7 +28,10 @@"bench.judgement.cast","bench.board.read","bench.dataset.read",-- "bench.eval.manage"++ "bench.eval.manage",++ "bench.benchmark.write",++ "bench.run.publish",++ "bench.run.read"],"eventsProduced": ["bench.round.finished",@@@ -93,7 +96,7 @@"none": "no claims: one process owns its PostgreSQL database (ov_bench); a round is taken with a compare-and-set on its status (queued → running)"}},-- "notes": "Port 5230, unit openvibe-bench.service, database ov_bench. The shared AI benchmark and comparison engine (plan 2026-10-28 §4.12): kinds, pinned rounds, entries, blind judgements, ratings, boards, the open dataset and private evals. AI stays the run authority, Report the analytics authority. Bench answers account export and deletion (ADR-033) and subscribes to the two account Events topics. It calls AI's run API (ai.run.create, namespace ai.generate) for generation and publishes bench.* events through its outbox.",++ "notes": "Port 5230, unit openvibe-bench.service, database ov_bench. The shared AI benchmark and comparison engine (plan 2026-10-28 §4.12): kinds, pinned rounds, entries, blind judgements, ratings, boards, the open dataset and private evals. AI stays the run authority, Report the analytics authority. Bench answers account export and deletion (ADR-033) and subscribes to the two account Events topics. It calls AI's run API (ai.run.create, namespace ai.generate) for generation and publishes bench.* events through its outbox. Stage B1 (spec 2026-10-30): the registry — versioned, forkable benchmarks (bench.benchmark@1) and the results API (bench.run@1), with compare, rankings and percentiles.","site": {"name": "Bench","icon": "bench",@@@ -1,6 +1,6 @@{"name": "openvibe-contracts",-- "version": "0.243.0",++ "version": "0.244.0","description": "Machine-readable contracts for the OpenVibe network: schemas, manifests, ids, error envelope, capability checks.","main": "index.js","types": "generated/typescript/index.d.ts",
@@ -4,6 +4,18 @@ All notable changes to `openvibe-contracts`. Releases are git tags (`vX.Y.Z`) th
@@ -4,6 +4,18 @@ All notable changes to `openvibe-contracts`. Releases are git tags (`vX.Y.Z`) th
from `https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/<tag>`. Before v0.30.0,
from `https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/<tag>`. Before v0.30.0,
the notes were in the tag and commit messages (`git tag -n1`).
the notes were in the tag and commit messages (`git tag -n1`).
+## 0.244.0 — 2026-10-30
+
+- **Bench as an open benchmark platform, stage B1** (spec 2026-10-30): `bench.benchmark@1` and `bench.run@1`.
+ A benchmark is an owned, versioned and forkable set: its tasks, its judges (pinned by version), how they combine
+ into one score, the limits a run may not pass, the default model and harness matrix, and whether it is public or
+ unlisted. A run is one result: a benchmark version × a model × a harness × settings × who ran it, with the
+ per-task scores, cost, time, tokens, the artifacts, and its provenance — `verified` (an executor ran it) or
+ `submitted` (posted with no executor, badged and out of the default rankings).
+- **Three new capabilities** for Bench: `bench.benchmark.write` (create, edit, freeze and fork a benchmark),
+ `bench.run.publish` (publish a run) and `bench.run.read` (the run, the comparison, the rankings and the percentile).
+ OpenVibe.Bench's service manifest lists them.
+
## 0.243.0 — 2026-10-30
## 0.243.0 — 2026-10-30
- The release cut from `main` for codes.task@2 (#250): a coding task's attempts and outcome, one id with two majors.
- The release cut from `main` for codes.task@2 (#250): a coding task's attempts and outcome, one id with two majors.
@@ -40,7 +40,8 @@
@@ -40,7 +40,8 @@
"OpenVibe.Inventory goes live (inventory.openvibe.network, site position 30) with six capabilities active (ADR-054, plan T21 step 2); a definition may name the services or apps that may grant it",
"OpenVibe.Inventory goes live (inventory.openvibe.network, site position 30) with six capabilities active (ADR-054, plan T21 step 2); a definition may name the services or apps that may grant it",
"the Workshop v1 (ADR-054 section 6): Workshop kinds, network.badge, staff review (inventory.review-request@1, inventory.definition.review active) and media_id on equipped slots",
"the Workshop v1 (ADR-054 section 6): Workshop kinds, network.badge, staff review (inventory.review-request@1, inventory.definition.review active) and media_id on equipped slots",
"News, Reviews, Trade, Coupons and OpenRestream go live on their own domains (plan T11); the product at openre.stream is OpenRestream, with OpenVibe Live on by default (live.openre.slot.bind)",
"News, Reviews, Trade, Coupons and OpenRestream go live on their own domains (plan T11); the product at openre.stream is OpenRestream, with OpenVibe Live on by default (live.openre.slot.bind)",
- "ADR-055 (proposed): money and the inventory — primary sales first, gifting next; trading and a market stay closed, and chance is never paid for"
+ "ADR-055 (proposed): money and the inventory — primary sales first, gifting next; trading and a market stay closed, and chance is never paid for",
+ "bench.benchmark@1 and bench.run@1 with the capabilities bench.benchmark.write, bench.run.publish and bench.run.read: the benchmark registry and the results API (spec 2026-10-30, stage B1)"
],
],
- "updated": "2026-10-22"
+ "updated": "2026-10-30"
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,266 @@
@@ -0,0 +1,266 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/bench/benchmark.v1.json",
+ "title": "BenchBenchmark",
+ "description": "bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "owner",
+ "title",
+ "description",
+ "kind",
+ "tags",
+ "visibility",
+ "version",
+ "parent",
+ "tasks",
+ "judges",
+ "scoring",
+ "limits",
+ "matrix",
+ "created_at"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}$",
+ "description": "The benchmark id; the same id carries every version."
+ },
+ "owner": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 120,
+ "description": "Who owns the set: a person (usr_…), a project (prj_…) or a service."
+ },
+ "title": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ },
+ "description": {
+ "type": "string",
+ "maxLength": 4000
+ },
+ "kind": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80,
+ "description": "One of Bench's kinds; the kind sets the renderer."
+ },
+ "tags": {
+ "type": "array",
+ "maxItems": 32,
+ "items": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ }
+ },
+ "visibility": {
+ "enum": [
+ "public",
+ "unlisted"
+ ],
+ "description": "public appears in the registry; unlisted is reachable by id only."
+ },
+ "version": {
+ "type": "integer",
+ "minimum": 1,
+ "description": "The version number this document is."
+ },
+ "parent": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 120,
+ "description": "The benchmark id@version this one was forked from, or null."
+ },
+ "tasks": {
+ "type": "array",
+ "maxItems": 200,
+ "items": {
+ "$ref": "#/$defs/task"
+ }
+ },
+ "judges": {
+ "type": "array",
+ "maxItems": 20,
+ "items": {
+ "$ref": "#/$defs/judge"
+ }
+ },
+ "scoring": {
+ "$ref": "#/$defs/scoring"
+ },
+ "limits": {
+ "$ref": "#/$defs/limits"
+ },
+ "matrix": {
+ "$ref": "#/$defs/matrix"
+ },
+ "created_at": {
+ "type": "string",
+ "format": "date-time"
+ }
+ },
+ "$defs": {
+ "task": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "prompt",
+ "holdout"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "prompt": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 8000
+ },
+ "files": {
+ "type": "array",
+ "maxItems": 20,
+ "items": {
+ "type": "object",
+ "additionalProperties": true
+ }
+ },
+ "expected": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "additionalProperties": true,
+ "description": "What a correct answer looks like; the judge reads it."
+ },
+ "holdout": {
+ "type": "boolean",
+ "description": "A hidden task: it is withheld from everyone but the executor, against tuning to the set."
+ }
+ }
+ },
+ "judge": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "type",
+ "version",
+ "config"
+ ],
+ "properties": {
+ "type": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 60,
+ "description": "human_pairwise | rubric | automatic_grader | test_suite | game_outcome"
+ },
+ "version": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40,
+ "description": "The pinned judge version; a run names the judge it used."
+ },
+ "config": {
+ "type": "object",
+ "additionalProperties": true
+ }
+ }
+ },
+ "scoring": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "method",
+ "scale"
+ ],
+ "properties": {
+ "method": {
+ "enum": [
+ "mean",
+ "weighted",
+ "pass_rate",
+ "sum"
+ ]
+ },
+ "scale": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "min",
+ "max"
+ ],
+ "properties": {
+ "min": {
+ "type": "number"
+ },
+ "max": {
+ "type": "number"
+ }
+ },
+ "description": "The score scale; 0-100 for Bench."
+ },
+ "weights": {
+ "type": "object",
+ "additionalProperties": {
+ "type": "number"
+ }
+ },
+ "aggregate": {
+ "type": "string",
+ "maxLength": 120
+ }
+ }
+ },
+ "limits": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "ms": {
+ "type": "number",
+ "minimum": 0,
+ "description": "The time one task may take."
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "usd": {
+ "type": "number",
+ "minimum": 0,
+ "description": "The money one run may spend."
+ }
+ }
+ },
+ "matrix": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "models": {
+ "type": "array",
+ "maxItems": 50,
+ "items": {
+ "type": "string",
+ "maxLength": 160
+ }
+ },
+ "harnesses": {
+ "type": "array",
+ "maxItems": 50,
+ "items": {
+ "type": "string",
+ "maxLength": 160
+ }
+ }
+ }
+ }
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,241 @@
@@ -0,0 +1,241 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/bench/run.v1.json",
+ "title": "BenchRun",
+ "description": "bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "benchmark",
+ "model",
+ "route",
+ "provider",
+ "harness",
+ "settings",
+ "executor",
+ "tasks",
+ "score",
+ "cost_usd",
+ "ms",
+ "tokens",
+ "provenance",
+ "created_at"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "pattern": "^brn_[0-9A-HJKMNP-TV-Z]{26}$"
+ },
+ "benchmark": {
+ "type": "string",
+ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$",
+ "description": "The benchmark id@version the run names. The version is immutable once a run names it."
+ },
+ "model": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ },
+ "route": {
+ "type": "object",
+ "additionalProperties": true,
+ "description": "The route that produced the run."
+ },
+ "provider": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "harness": {
+ "$ref": "#/$defs/harness"
+ },
+ "settings": {
+ "$ref": "#/$defs/settings"
+ },
+ "executor": {
+ "$ref": "#/$defs/executor"
+ },
+ "hardware": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "additionalProperties": true,
+ "description": "The machine for a local model: GPU, CPU, RAM, quantization."
+ },
+ "tasks": {
+ "type": "array",
+ "maxItems": 200,
+ "items": {
+ "$ref": "#/$defs/task"
+ }
+ },
+ "score": {
+ "type": "number",
+ "minimum": 0,
+ "maximum": 100,
+ "description": "The benchmark's one score for this run, 0-100."
+ },
+ "cost_usd": {
+ "type": "number",
+ "minimum": 0
+ },
+ "ms": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "provenance": {
+ "enum": [
+ "verified",
+ "submitted"
+ ],
+ "description": "verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings."
+ },
+ "created_at": {
+ "type": "string",
+ "format": "date-time"
+ }
+ },
+ "$defs": {
+ "harness": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "name",
+ "version"
+ ],
+ "properties": {
+ "name": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "version": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ }
+ }
+ },
+ "settings": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "temperature": {
+ "type": [
+ "number",
+ "null"
+ ],
+ "minimum": 0,
+ "maximum": 2
+ },
+ "effort": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 40
+ }
+ }
+ },
+ "executor": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "kind",
+ "id"
+ ],
+ "properties": {
+ "kind": {
+ "enum": [
+ "codes",
+ "run",
+ "node",
+ "submitted"
+ ],
+ "description": "codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it."
+ },
+ "id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ }
+ }
+ },
+ "task": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "task_id",
+ "score",
+ "cost_usd",
+ "ms",
+ "tokens",
+ "artifacts"
+ ],
+ "properties": {
+ "task_id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "score": {
+ "type": "number",
+ "minimum": 0,
+ "maximum": 100
+ },
+ "cost_usd": {
+ "type": "number",
+ "minimum": 0
+ },
+ "ms": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "artifacts": {
+ "type": "array",
+ "maxItems": 40,
+ "items": {
+ "$ref": "#/$defs/artifact"
+ }
+ },
+ "transcript_url": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 500
+ }
+ }
+ },
+ "artifact": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "kind",
+ "url"
+ ],
+ "properties": {
+ "kind": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ },
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 500
+ }
+ }
+ }
+ }
+}
@@ -8171,6 +8171,24 @@
@@ -8171,6 +8171,24 @@
"visibility": "public",
"visibility": "public",
"compatibility": "backward",
"compatibility": "backward",
"status": "active"
"status": "active"
+ },
+ {
+ "id": "bench.benchmark",
+ "version": "1.0.0",
+ "owner": "bench",
+ "schema": "bench/benchmark.v1.json",
+ "visibility": "first-party",
+ "compatibility": "backward",
+ "status": "active"
+ },
+ {
+ "id": "bench.run",
+ "version": "1.0.0",
+ "owner": "bench",
+ "schema": "bench/run.v1.json",
+ "visibility": "first-party",
+ "compatibility": "backward",
+ "status": "active"
}
}
]
]
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,60 @@
@@ -0,0 +1,60 @@
+{
+ "id": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS",
+ "owner": "openvibe",
+ "title": "Playable games arena",
+ "description": "One-screen arcade games, judged by play and by the frame.",
+ "kind": "games.play",
+ "tags": [
+ "games",
+ "arcade"
+ ],
+ "visibility": "public",
+ "version": 1,
+ "parent": null,
+ "tasks": [
+ {
+ "id": "t1",
+ "prompt": "x"
+ }
+ ],
+ "judges": [
+ {
+ "type": "human_pairwise",
+ "version": "1.0",
+ "config": {
+ "blind": true
+ }
+ },
+ {
+ "type": "game_outcome",
+ "version": "1.0",
+ "config": {}
+ }
+ ],
+ "scoring": {
+ "method": "mean",
+ "scale": {
+ "min": 0,
+ "max": 100
+ },
+ "weights": {
+ "human_pairwise": 1
+ }
+ },
+ "limits": {
+ "ms": 120000,
+ "tokens": 200000,
+ "usd": 0.5
+ },
+ "matrix": {
+ "models": [
+ "claude-opus-4",
+ "gpt-5"
+ ],
+ "harnesses": [
+ "claude-code",
+ "opencode"
+ ]
+ },
+ "created_at": "2026-10-30T12:00:00.000Z"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,69 @@
@@ -0,0 +1,69 @@
+{
+ "id": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS",
+ "owner": "openvibe",
+ "title": "Playable games arena",
+ "description": "One-screen arcade games, judged by play and by the frame.",
+ "kind": "games.play",
+ "tags": [
+ "games",
+ "arcade"
+ ],
+ "visibility": "secret",
+ "version": 1,
+ "parent": null,
+ "tasks": [
+ {
+ "id": "t1",
+ "prompt": "Build a one-screen arcade game with a paddle, bricks, a ball and a score.",
+ "holdout": false
+ },
+ {
+ "id": "t2",
+ "prompt": "Build a snake game on a 20x20 grid.",
+ "holdout": true,
+ "expected": {
+ "score": "food eaten"
+ }
+ }
+ ],
+ "judges": [
+ {
+ "type": "human_pairwise",
+ "version": "1.0",
+ "config": {
+ "blind": true
+ }
+ },
+ {
+ "type": "game_outcome",
+ "version": "1.0",
+ "config": {}
+ }
+ ],
+ "scoring": {
+ "method": "mean",
+ "scale": {
+ "min": 0,
+ "max": 100
+ },
+ "weights": {
+ "human_pairwise": 1
+ }
+ },
+ "limits": {
+ "ms": 120000,
+ "tokens": 200000,
+ "usd": 0.5
+ },
+ "matrix": {
+ "models": [
+ "claude-opus-4",
+ "gpt-5"
+ ],
+ "harnesses": [
+ "claude-code",
+ "opencode"
+ ]
+ },
+ "created_at": "2026-10-30T12:00:00.000Z"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,69 @@
@@ -0,0 +1,69 @@
+{
+ "id": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS",
+ "owner": "openvibe",
+ "title": "Playable games arena",
+ "description": "One-screen arcade games, judged by play and by the frame.",
+ "kind": "games.play",
+ "tags": [
+ "games",
+ "arcade"
+ ],
+ "visibility": "public",
+ "version": 1,
+ "parent": null,
+ "tasks": [
+ {
+ "id": "t1",
+ "prompt": "Build a one-screen arcade game with a paddle, bricks, a ball and a score.",
+ "holdout": false
+ },
+ {
+ "id": "t2",
+ "prompt": "Build a snake game on a 20x20 grid.",
+ "holdout": true,
+ "expected": {
+ "score": "food eaten"
+ }
+ }
+ ],
+ "judges": [
+ {
+ "type": "human_pairwise",
+ "version": "1.0",
+ "config": {
+ "blind": true
+ }
+ },
+ {
+ "type": "game_outcome",
+ "version": "1.0",
+ "config": {}
+ }
+ ],
+ "scoring": {
+ "method": "mean",
+ "scale": {
+ "min": 0,
+ "max": 100
+ },
+ "weights": {
+ "human_pairwise": 1
+ }
+ },
+ "limits": {
+ "ms": 120000,
+ "tokens": 200000,
+ "usd": 0.5
+ },
+ "matrix": {
+ "models": [
+ "claude-opus-4",
+ "gpt-5"
+ ],
+ "harnesses": [
+ "claude-code",
+ "opencode"
+ ]
+ },
+ "created_at": "2026-10-30T12:00:00.000Z"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
+{
+ "id": "brn_01JAB2C3D4E5F6G7H8J9K0MNPS",
+ "benchmark": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS@1",
+ "model": "claude-opus-4",
+ "route": {
+ "provider": "anthropic",
+ "via": "router"
+ },
+ "provider": "anthropic",
+ "harness": {
+ "name": "claude-code",
+ "version": "1.2.0"
+ },
+ "settings": {
+ "temperature": 1,
+ "effort": "high"
+ },
+ "executor": {
+ "kind": "codes",
+ "id": "task_01JAB2C3D4E5F6G7H8J9K0MNPS"
+ },
+ "hardware": null,
+ "tasks": [
+ {
+ "task_id": "t1",
+ "score": 92,
+ "cost_usd": 0.031,
+ "ms": 41200,
+ "tokens": 8400,
+ "artifacts": [
+ {
+ "kind": "html",
+ "url": "https://bench.openvibe.services/render/x.html"
+ }
+ ],
+ "transcript_url": "https://bench.openvibe.services/runs/x/t1"
+ }
+ ],
+ "score": 92,
+ "cost_usd": 0.031,
+ "ms": 41200,
+ "tokens": 8400,
+ "provenance": "trusted",
+ "created_at": "2026-10-30T12:05:00.000Z"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
+{
+ "id": "brn_01JAB2C3D4E5F6G7H8J9K0MNPS",
+ "benchmark": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS@1",
+ "model": "claude-opus-4",
+ "route": {
+ "provider": "anthropic",
+ "via": "router"
+ },
+ "provider": "anthropic",
+ "harness": {
+ "name": "claude-code",
+ "version": "1.2.0"
+ },
+ "settings": {
+ "temperature": 1,
+ "effort": "high"
+ },
+ "executor": {
+ "kind": "codes",
+ "id": "task_01JAB2C3D4E5F6G7H8J9K0MNPS"
+ },
+ "hardware": null,
+ "tasks": [
+ {
+ "task_id": "t1",
+ "score": 92,
+ "cost_usd": 0.031,
+ "ms": 41200,
+ "tokens": 8400,
+ "artifacts": [
+ {
+ "kind": "html",
+ "url": "https://bench.openvibe.services/render/x.html"
+ }
+ ],
+ "transcript_url": "https://bench.openvibe.services/runs/x/t1"
+ }
+ ],
+ "score": 140,
+ "cost_usd": 0.031,
+ "ms": 41200,
+ "tokens": 8400,
+ "provenance": "verified",
+ "created_at": "2026-10-30T12:05:00.000Z"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
+{
+ "id": "brn_01JAB2C3D4E5F6G7H8J9K0MNPS",
+ "benchmark": "bmk_01JAB2C3D4E5F6G7H8J9K0MNPS@1",
+ "model": "claude-opus-4",
+ "route": {
+ "provider": "anthropic",
+ "via": "router"
+ },
+ "provider": "anthropic",
+ "harness": {
+ "name": "claude-code",
+ "version": "1.2.0"
+ },
+ "settings": {
+ "temperature": 1,
+ "effort": "high"
+ },
+ "executor": {
+ "kind": "codes",
+ "id": "task_01JAB2C3D4E5F6G7H8J9K0MNPS"
+ },
+ "hardware": null,
+ "tasks": [
+ {
+ "task_id": "t1",
+ "score": 92,
+ "cost_usd": 0.031,
+ "ms": 41200,
+ "tokens": 8400,
+ "artifacts": [
+ {
+ "kind": "html",
+ "url": "https://bench.openvibe.services/render/x.html"
+ }
+ ],
+ "transcript_url": "https://bench.openvibe.services/runs/x/t1"
+ }
+ ],
+ "score": 92,
+ "cost_usd": 0.031,
+ "ms": 41200,
+ "tokens": 8400,
+ "provenance": "verified",
+ "created_at": "2026-10-30T12:05:00.000Z"
+}
@@ -90866,6 +90866,523 @@
@@ -90866,6 +90866,523 @@
}
}
]
]
}
}
+ },
+ "bench.benchmark": {
+ "version": "1.0.0",
+ "owner": "bench",
+ "status": "active",
+ "schema": {
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/bench/benchmark.v1.json",
+ "title": "BenchBenchmark",
+ "description": "bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "owner",
+ "title",
+ "description",
+ "kind",
+ "tags",
+ "visibility",
+ "version",
+ "parent",
+ "tasks",
+ "judges",
+ "scoring",
+ "limits",
+ "matrix",
+ "created_at"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}$",
+ "description": "The benchmark id; the same id carries every version."
+ },
+ "owner": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 120,
+ "description": "Who owns the set: a person (usr_…), a project (prj_…) or a service."
+ },
+ "title": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ },
+ "description": {
+ "type": "string",
+ "maxLength": 4000
+ },
+ "kind": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80,
+ "description": "One of Bench's kinds; the kind sets the renderer."
+ },
+ "tags": {
+ "type": "array",
+ "maxItems": 32,
+ "items": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ }
+ },
+ "visibility": {
+ "enum": [
+ "public",
+ "unlisted"
+ ],
+ "description": "public appears in the registry; unlisted is reachable by id only."
+ },
+ "version": {
+ "type": "integer",
+ "minimum": 1,
+ "description": "The version number this document is."
+ },
+ "parent": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 120,
+ "description": "The benchmark id@version this one was forked from, or null."
+ },
+ "tasks": {
+ "type": "array",
+ "maxItems": 200,
+ "items": {
+ "$ref": "#/$defs/task"
+ }
+ },
+ "judges": {
+ "type": "array",
+ "maxItems": 20,
+ "items": {
+ "$ref": "#/$defs/judge"
+ }
+ },
+ "scoring": {
+ "$ref": "#/$defs/scoring"
+ },
+ "limits": {
+ "$ref": "#/$defs/limits"
+ },
+ "matrix": {
+ "$ref": "#/$defs/matrix"
+ },
+ "created_at": {
+ "type": "string",
+ "format": "date-time"
+ }
+ },
+ "$defs": {
+ "task": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "prompt",
+ "holdout"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "prompt": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 8000
+ },
+ "files": {
+ "type": "array",
+ "maxItems": 20,
+ "items": {
+ "type": "object",
+ "additionalProperties": true
+ }
+ },
+ "expected": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "additionalProperties": true,
+ "description": "What a correct answer looks like; the judge reads it."
+ },
+ "holdout": {
+ "type": "boolean",
+ "description": "A hidden task: it is withheld from everyone but the executor, against tuning to the set."
+ }
+ }
+ },
+ "judge": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "type",
+ "version",
+ "config"
+ ],
+ "properties": {
+ "type": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 60,
+ "description": "human_pairwise | rubric | automatic_grader | test_suite | game_outcome"
+ },
+ "version": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40,
+ "description": "The pinned judge version; a run names the judge it used."
+ },
+ "config": {
+ "type": "object",
+ "additionalProperties": true
+ }
+ }
+ },
+ "scoring": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "method",
+ "scale"
+ ],
+ "properties": {
+ "method": {
+ "enum": [
+ "mean",
+ "weighted",
+ "pass_rate",
+ "sum"
+ ]
+ },
+ "scale": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "min",
+ "max"
+ ],
+ "properties": {
+ "min": {
+ "type": "number"
+ },
+ "max": {
+ "type": "number"
+ }
+ },
+ "description": "The score scale; 0-100 for Bench."
+ },
+ "weights": {
+ "type": "object",
+ "additionalProperties": {
+ "type": "number"
+ }
+ },
+ "aggregate": {
+ "type": "string",
+ "maxLength": 120
+ }
+ }
+ },
+ "limits": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "ms": {
+ "type": "number",
+ "minimum": 0,
+ "description": "The time one task may take."
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "usd": {
+ "type": "number",
+ "minimum": 0,
+ "description": "The money one run may spend."
+ }
+ }
+ },
+ "matrix": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "models": {
+ "type": "array",
+ "maxItems": 50,
+ "items": {
+ "type": "string",
+ "maxLength": 160
+ }
+ },
+ "harnesses": {
+ "type": "array",
+ "maxItems": 50,
+ "items": {
+ "type": "string",
+ "maxLength": 160
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "bench.run": {
+ "version": "1.0.0",
+ "owner": "bench",
+ "status": "active",
+ "schema": {
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/bench/run.v1.json",
+ "title": "BenchRun",
+ "description": "bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "benchmark",
+ "model",
+ "route",
+ "provider",
+ "harness",
+ "settings",
+ "executor",
+ "tasks",
+ "score",
+ "cost_usd",
+ "ms",
+ "tokens",
+ "provenance",
+ "created_at"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "pattern": "^brn_[0-9A-HJKMNP-TV-Z]{26}$"
+ },
+ "benchmark": {
+ "type": "string",
+ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$",
+ "description": "The benchmark id@version the run names. The version is immutable once a run names it."
+ },
+ "model": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ },
+ "route": {
+ "type": "object",
+ "additionalProperties": true,
+ "description": "The route that produced the run."
+ },
+ "provider": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "harness": {
+ "$ref": "#/$defs/harness"
+ },
+ "settings": {
+ "$ref": "#/$defs/settings"
+ },
+ "executor": {
+ "$ref": "#/$defs/executor"
+ },
+ "hardware": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "additionalProperties": true,
+ "description": "The machine for a local model: GPU, CPU, RAM, quantization."
+ },
+ "tasks": {
+ "type": "array",
+ "maxItems": 200,
+ "items": {
+ "$ref": "#/$defs/task"
+ }
+ },
+ "score": {
+ "type": "number",
+ "minimum": 0,
+ "maximum": 100,
+ "description": "The benchmark's one score for this run, 0-100."
+ },
+ "cost_usd": {
+ "type": "number",
+ "minimum": 0
+ },
+ "ms": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "provenance": {
+ "enum": [
+ "verified",
+ "submitted"
+ ],
+ "description": "verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings."
+ },
+ "created_at": {
+ "type": "string",
+ "format": "date-time"
+ }
+ },
+ "$defs": {
+ "harness": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "name",
+ "version"
+ ],
+ "properties": {
+ "name": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "version": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ }
+ }
+ },
+ "settings": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "temperature": {
+ "type": [
+ "number",
+ "null"
+ ],
+ "minimum": 0,
+ "maximum": 2
+ },
+ "effort": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 40
+ }
+ }
+ },
+ "executor": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "kind",
+ "id"
+ ],
+ "properties": {
+ "kind": {
+ "enum": [
+ "codes",
+ "run",
+ "node",
+ "submitted"
+ ],
+ "description": "codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it."
+ },
+ "id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ }
+ }
+ },
+ "task": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "task_id",
+ "score",
+ "cost_usd",
+ "ms",
+ "tokens",
+ "artifacts"
+ ],
+ "properties": {
+ "task_id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "score": {
+ "type": "number",
+ "minimum": 0,
+ "maximum": 100
+ },
+ "cost_usd": {
+ "type": "number",
+ "minimum": 0
+ },
+ "ms": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "artifacts": {
+ "type": "array",
+ "maxItems": 40,
+ "items": {
+ "$ref": "#/$defs/artifact"
+ }
+ },
+ "transcript_url": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 500
+ }
+ }
+ },
+ "artifact": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "kind",
+ "url"
+ ],
+ "properties": {
+ "kind": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ },
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 500
+ }
+ }
+ }
+ }
+ }
}
}
}
}
}
}
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Actor API",
"title": "OpenVibe.Actor API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Actor as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Actor as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.AI API",
"title": "OpenVibe.AI API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.AI as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.AI as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Bench API",
"title": "OpenVibe.Bench API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Bench as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Bench as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -16,6 +16,260 @@
@@ -16,6 +16,260 @@
}
}
],
],
"paths": {
"paths": {
+ "/api/v1/benchmarks": {
+ "post": {
+ "operationId": "post_api_v1_benchmarks",
+ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",
+ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",
+ "tags": [
+ "bench.benchmark"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.benchmark.write"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.benchmark.write"
+ ]
+ }
+ ],
+ "requestBody": {
+ "required": false,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/benchmarks/{id}": {
+ "put": {
+ "operationId": "put_api_v1_benchmarks_id",
+ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",
+ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",
+ "tags": [
+ "bench.benchmark"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.benchmark.write"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.benchmark.write"
+ ]
+ }
+ ],
+ "requestBody": {
+ "required": false,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/benchmarks/{id}/fork": {
+ "post": {
+ "operationId": "post_api_v1_benchmarks_id_fork",
+ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",
+ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",
+ "tags": [
+ "bench.benchmark"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.benchmark.write"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.benchmark.write"
+ ]
+ }
+ ],
+ "requestBody": {
+ "required": false,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/benchmarks/{id}/versions": {
+ "post": {
+ "operationId": "post_api_v1_benchmarks_id_versions",
+ "summary": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork.",
+ "description": "**bench.benchmark.write** (first-party): Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",
+ "tags": [
+ "bench.benchmark"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.benchmark.write"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.benchmark.write"
+ ]
+ }
+ ],
+ "requestBody": {
+ "required": false,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.benchmark.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
"/api/v1/boards/{kindId}": {
"/api/v1/boards/{kindId}": {
"get": {
"get": {
"operationId": "get_api_v1_boards_kindId",
"operationId": "get_api_v1_boards_kindId",
@@ -72,6 +326,52 @@
@@ -72,6 +326,52 @@
}
}
}
}
},
},
+ "/api/v1/compare": {
+ "get": {
+ "operationId": "get_api_v1_compare",
+ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",
+ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.",
+ "tags": [
+ "bench.run"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.run.read"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.run.read"
+ ]
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.run.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
"/api/v1/datasets/{kindId}.jsonl": {
"/api/v1/datasets/{kindId}.jsonl": {
"get": {
"get": {
"operationId": "get_api_v1_datasets_kindId_jsonl",
"operationId": "get_api_v1_datasets_kindId_jsonl",
@@ -193,13 +493,227 @@
@@ -193,13 +493,227 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.eval.manage"
+ "bench.eval.manage"
+ ]
+ }
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.task.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/evals/{id}/run": {
+ "post": {
+ "operationId": "post_api_v1_evals_id_run",
+ "summary": "Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eva",
+ "description": "**bench.eval.manage** (first-party): Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eval and its runs. Never in the public dataset.",
+ "tags": [
+ "bench.eval"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.eval.manage"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.eval.manage"
+ ]
+ }
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.task.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/kinds": {
+ "get": {
+ "operationId": "get_api_v1_kinds",
+ "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",
+ "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",
+ "tags": [
+ "bench.kind"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.kind.read"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.kind.read"
+ ]
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.kind.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/kinds/{kindId}": {
+ "get": {
+ "operationId": "get_api_v1_kinds_kindId",
+ "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",
+ "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",
+ "tags": [
+ "bench.kind"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.kind.read"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.kind.read"
+ ]
+ }
+ ],
+ "parameters": [
+ {
+ "name": "kindId",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "2XX": {
+ "description": "Success",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.kind.v1"
+ }
+ }
+ }
+ },
+ "default": {
+ "description": "An error (RFC 9457 problem details)",
+ "content": {
+ "application/problem+json": {
+ "schema": {
+ "$ref": "#/components/schemas/errors.problem.v1"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v1/kinds/{kindId}/tasks": {
+ "post": {
+ "operationId": "post_api_v1_kinds_kindId_tasks",
+ "summary": "Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public?",
+ "description": "**bench.task.create** (first-party): Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public? }. A room or a person's own token may create a task.",
+ "tags": [
+ "bench.task"
+ ],
+ "x-openvibe-capabilities": [
+ "bench.task.create"
+ ],
+ "x-openvibe-visibility": [
+ "first-party"
+ ],
+ "x-openvibe-quota-class": "standard",
+ "security": [
+ {
+ "openvibeToken": [
+ "bench.task.create"
]
]
}
}
],
],
"parameters": [
"parameters": [
{
{
- "name": "id",
+ "name": "kindId",
"in": "path",
"in": "path",
"required": true,
"required": true,
"schema": {
"schema": {
@@ -231,16 +745,16 @@
@@ -231,16 +745,16 @@
}
}
}
}
},
},
- "/api/v1/evals/{id}/run": {
- "post": {
- "operationId": "post_api_v1_evals_id_run",
- "summary": "Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eva",
- "description": "**bench.eval.manage** (first-party): Manage private evals on OpenVibe.Bench: POST /api/v1/evals creates a private task suite, POST /api/v1/evals/:id/run runs it against chosen offers, and GET /api/v1/evals/:id/results reads a private eval and its runs. Never in the public dataset.",
+ "/api/v1/percentile": {
+ "get": {
+ "operationId": "get_api_v1_percentile",
+ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",
+ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.",
"tags": [
"tags": [
- "bench.eval"
+ "bench.run"
],
],
"x-openvibe-capabilities": [
"x-openvibe-capabilities": [
- "bench.eval.manage"
+ "bench.run.read"
],
],
"x-openvibe-visibility": [
"x-openvibe-visibility": [
"first-party"
"first-party"
@@ -249,27 +763,17 @@
@@ -249,27 +763,17 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.eval.manage"
+ "bench.run.read"
]
]
}
}
],
],
- "parameters": [
- {
- "name": "id",
- "in": "path",
- "required": true,
- "schema": {
- "type": "string"
- }
- }
- ],
"responses": {
"responses": {
"2XX": {
"2XX": {
"description": "Success",
"description": "Success",
"content": {
"content": {
"application/json": {
"application/json": {
"schema": {
"schema": {
- "$ref": "#/components/schemas/bench.task.v1"
+ "$ref": "#/components/schemas/bench.run.v1"
}
}
}
}
}
}
@@ -287,16 +791,16 @@
@@ -287,16 +791,16 @@
}
}
}
}
},
},
- "/api/v1/kinds": {
+ "/api/v1/rankings": {
"get": {
"get": {
- "operationId": "get_api_v1_kinds",
- "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",
- "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",
+ "operationId": "get_api_v1_rankings",
+ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",
+ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.",
"tags": [
"tags": [
- "bench.kind"
+ "bench.run"
],
],
"x-openvibe-capabilities": [
"x-openvibe-capabilities": [
- "bench.kind.read"
+ "bench.run.read"
],
],
"x-openvibe-visibility": [
"x-openvibe-visibility": [
"first-party"
"first-party"
@@ -305,7 +809,7 @@
@@ -305,7 +809,7 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.kind.read"
+ "bench.run.read"
]
]
}
}
],
],
@@ -315,7 +819,7 @@
@@ -315,7 +819,7 @@
"content": {
"content": {
"application/json": {
"application/json": {
"schema": {
"schema": {
- "$ref": "#/components/schemas/bench.kind.v1"
+ "$ref": "#/components/schemas/bench.run.v1"
}
}
}
}
}
}
@@ -333,16 +837,16 @@
@@ -333,16 +837,16 @@
}
}
}
}
},
},
- "/api/v1/kinds/{kindId}": {
+ "/api/v1/rounds/{id}": {
"get": {
"get": {
- "operationId": "get_api_v1_kinds_kindId",
- "summary": "Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics.",
- "description": "**bench.kind.read** (first-party): Read a benchmark kind on OpenVibe.Bench: GET /api/v1/kinds lists the kinds, GET /api/v1/kinds/:kindId reads one kind and its entry schema and metrics. Granted to rooms and developer services that render a kind or its board.",
+ "operationId": "get_api_v1_rounds_id",
+ "summary": "Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.",
+ "description": "**bench.round.read** (first-party): Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.",
"tags": [
"tags": [
- "bench.kind"
+ "bench.round"
],
],
"x-openvibe-capabilities": [
"x-openvibe-capabilities": [
- "bench.kind.read"
+ "bench.round.read"
],
],
"x-openvibe-visibility": [
"x-openvibe-visibility": [
"first-party"
"first-party"
@@ -351,13 +855,13 @@
@@ -351,13 +855,13 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.kind.read"
+ "bench.round.read"
]
]
}
}
],
],
"parameters": [
"parameters": [
{
{
- "name": "kindId",
+ "name": "id",
"in": "path",
"in": "path",
"required": true,
"required": true,
"schema": {
"schema": {
@@ -371,7 +875,7 @@
@@ -371,7 +875,7 @@
"content": {
"content": {
"application/json": {
"application/json": {
"schema": {
"schema": {
- "$ref": "#/components/schemas/bench.kind.v1"
+ "$ref": "#/components/schemas/bench.round.v1"
}
}
}
}
}
}
@@ -389,16 +893,16 @@
@@ -389,16 +893,16 @@
}
}
}
}
},
},
- "/api/v1/kinds/{kindId}/tasks": {
+ "/api/v1/rounds/{id}/entries": {
"post": {
"post": {
- "operationId": "post_api_v1_kinds_kindId_tasks",
- "summary": "Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public?",
- "description": "**bench.task.create** (first-party): Create a benchmark task in a kind on OpenVibe.Bench: POST /api/v1/kinds/:kindId/tasks with { prompt, task_class?, public? }. A room or a person's own token may create a task.",
+ "operationId": "post_api_v1_rounds_id_entries",
+ "summary": "Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token.",
+ "description": "**bench.entry.create** (first-party): Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token. A room submits a game match, a test-suite run or another room's result.",
"tags": [
"tags": [
- "bench.task"
+ "bench.entry"
],
],
"x-openvibe-capabilities": [
"x-openvibe-capabilities": [
- "bench.task.create"
+ "bench.entry.create"
],
],
"x-openvibe-visibility": [
"x-openvibe-visibility": [
"first-party"
"first-party"
@@ -407,13 +911,13 @@
@@ -407,13 +911,13 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.task.create"
+ "bench.entry.create"
]
]
}
}
],
],
"parameters": [
"parameters": [
{
{
- "name": "kindId",
+ "name": "id",
"in": "path",
"in": "path",
"required": true,
"required": true,
"schema": {
"schema": {
@@ -427,7 +931,7 @@
@@ -427,7 +931,7 @@
"content": {
"content": {
"application/json": {
"application/json": {
"schema": {
"schema": {
- "$ref": "#/components/schemas/bench.task.v1"
+ "$ref": "#/components/schemas/bench.entry.v1"
}
}
}
}
}
}
@@ -445,16 +949,16 @@
@@ -445,16 +949,16 @@
}
}
}
}
},
},
- "/api/v1/rounds/{id}": {
- "get": {
- "operationId": "get_api_v1_rounds_id",
- "summary": "Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.",
- "description": "**bench.round.read** (first-party): Read a benchmark round on OpenVibe.Bench: GET /api/v1/rounds/:id returns the round with entries; entries stay blind until the caller has judged or started the round.",
+ "/api/v1/rounds/{id}/judgements": {
+ "post": {
+ "operationId": "post_api_v1_rounds_id_judgements",
+ "summary": "Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score?",
+ "description": "**bench.judgement.cast** (first-party): Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score? }. Human judges are people; automatic, test-suite and game-outcome judges are services.",
"tags": [
"tags": [
- "bench.round"
+ "bench.judgement"
],
],
"x-openvibe-capabilities": [
"x-openvibe-capabilities": [
- "bench.round.read"
+ "bench.judgement.cast"
],
],
"x-openvibe-visibility": [
"x-openvibe-visibility": [
"first-party"
"first-party"
@@ -463,7 +967,7 @@
@@ -463,7 +967,7 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.round.read"
+ "bench.judgement.cast"
]
]
}
}
],
],
@@ -483,7 +987,7 @@
@@ -483,7 +987,7 @@
"content": {
"content": {
"application/json": {
"application/json": {
"schema": {
"schema": {
- "$ref": "#/components/schemas/bench.round.v1"
+ "$ref": "#/components/schemas/bench.judgement.v1"
}
}
}
}
}
}
@@ -501,16 +1005,16 @@
@@ -501,16 +1005,16 @@
}
}
}
}
},
},
- "/api/v1/rounds/{id}/entries": {
+ "/api/v1/runs": {
"post": {
"post": {
- "operationId": "post_api_v1_rounds_id_entries",
- "summary": "Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token.",
- "description": "**bench.entry.create** (first-party): Submit an outside entry to a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/entries with a service token. A room submits a game match, a test-suite run or another room's result.",
+ "operationId": "post_api_v1_runs",
+ "summary": "Publish a run of a benchmark on OpenVibe.Bench: POST /api/v1/runs with the per-task scores, cost, time and tokens.",
+ "description": "**bench.run.publish** (first-party): Publish a run of a benchmark on OpenVibe.Bench: POST /api/v1/runs with the per-task scores, cost, time and tokens. A service token (openvibe.codes, Run) makes a verified run; a person's token makes a submitted run, which is badged and kept out of the default rankings. A caller with this capability sees a benchmark's holdout tasks.",
"tags": [
"tags": [
- "bench.entry"
+ "bench.run"
],
],
"x-openvibe-capabilities": [
"x-openvibe-capabilities": [
- "bench.entry.create"
+ "bench.run.publish"
],
],
"x-openvibe-visibility": [
"x-openvibe-visibility": [
"first-party"
"first-party"
@@ -519,27 +1023,27 @@
@@ -519,27 +1023,27 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.entry.create"
+ "bench.run.publish"
]
]
}
}
],
],
- "parameters": [
- {
- "name": "id",
- "in": "path",
- "required": true,
- "schema": {
- "type": "string"
+ "requestBody": {
+ "required": false,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/bench.run.v1"
+ }
}
}
}
}
- ],
+ },
"responses": {
"responses": {
"2XX": {
"2XX": {
"description": "Success",
"description": "Success",
"content": {
"content": {
"application/json": {
"application/json": {
"schema": {
"schema": {
- "$ref": "#/components/schemas/bench.entry.v1"
+ "$ref": "#/components/schemas/bench.run.v1"
}
}
}
}
}
}
@@ -557,16 +1061,16 @@
@@ -557,16 +1061,16 @@
}
}
}
}
},
},
- "/api/v1/rounds/{id}/judgements": {
- "post": {
- "operationId": "post_api_v1_rounds_id_judgements",
- "summary": "Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score?",
- "description": "**bench.judgement.cast** (first-party): Cast a blind judgement on a benchmark round on OpenVibe.Bench: POST /api/v1/rounds/:id/judgements with { judge?, pair?, outcome?, score? }. Human judges are people; automatic, test-suite and game-outcome judges are services.",
+ "/api/v1/runs/{id}": {
+ "get": {
+ "operationId": "get_api_v1_runs_id",
+ "summary": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings an",
+ "description": "**bench.run.read** (first-party): Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.",
"tags": [
"tags": [
- "bench.judgement"
+ "bench.run"
],
],
"x-openvibe-capabilities": [
"x-openvibe-capabilities": [
- "bench.judgement.cast"
+ "bench.run.read"
],
],
"x-openvibe-visibility": [
"x-openvibe-visibility": [
"first-party"
"first-party"
@@ -575,7 +1079,7 @@
@@ -575,7 +1079,7 @@
"security": [
"security": [
{
{
"openvibeToken": [
"openvibeToken": [
- "bench.judgement.cast"
+ "bench.run.read"
]
]
}
}
],
],
@@ -595,7 +1099,7 @@
@@ -595,7 +1099,7 @@
"content": {
"content": {
"application/json": {
"application/json": {
"schema": {
"schema": {
- "$ref": "#/components/schemas/bench.judgement.v1"
+ "$ref": "#/components/schemas/bench.run.v1"
}
}
}
}
}
}
@@ -672,6 +1176,270 @@
@@ -672,6 +1176,270 @@
},
},
"components": {
"components": {
"schemas": {
"schemas": {
+ "bench.benchmark.v1": {
+ "title": "BenchBenchmark",
+ "description": "bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "owner",
+ "title",
+ "description",
+ "kind",
+ "tags",
+ "visibility",
+ "version",
+ "parent",
+ "tasks",
+ "judges",
+ "scoring",
+ "limits",
+ "matrix",
+ "created_at"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}$",
+ "description": "The benchmark id; the same id carries every version."
+ },
+ "owner": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 120,
+ "description": "Who owns the set: a person (usr_…), a project (prj_…) or a service."
+ },
+ "title": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ },
+ "description": {
+ "type": "string",
+ "maxLength": 4000
+ },
+ "kind": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80,
+ "description": "One of Bench's kinds; the kind sets the renderer."
+ },
+ "tags": {
+ "type": "array",
+ "maxItems": 32,
+ "items": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ }
+ },
+ "visibility": {
+ "enum": [
+ "public",
+ "unlisted"
+ ],
+ "description": "public appears in the registry; unlisted is reachable by id only."
+ },
+ "version": {
+ "type": "integer",
+ "minimum": 1,
+ "description": "The version number this document is."
+ },
+ "parent": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 120,
+ "description": "The benchmark id@version this one was forked from, or null."
+ },
+ "tasks": {
+ "type": "array",
+ "maxItems": 200,
+ "items": {
+ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/task"
+ }
+ },
+ "judges": {
+ "type": "array",
+ "maxItems": 20,
+ "items": {
+ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/judge"
+ }
+ },
+ "scoring": {
+ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/scoring"
+ },
+ "limits": {
+ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/limits"
+ },
+ "matrix": {
+ "$ref": "#/components/schemas/bench.benchmark.v1/$defs/matrix"
+ },
+ "created_at": {
+ "type": "string",
+ "format": "date-time"
+ }
+ },
+ "$defs": {
+ "task": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "prompt",
+ "holdout"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "prompt": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 8000
+ },
+ "files": {
+ "type": "array",
+ "maxItems": 20,
+ "items": {
+ "type": "object",
+ "additionalProperties": true
+ }
+ },
+ "expected": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "additionalProperties": true,
+ "description": "What a correct answer looks like; the judge reads it."
+ },
+ "holdout": {
+ "type": "boolean",
+ "description": "A hidden task: it is withheld from everyone but the executor, against tuning to the set."
+ }
+ }
+ },
+ "judge": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "type",
+ "version",
+ "config"
+ ],
+ "properties": {
+ "type": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 60,
+ "description": "human_pairwise | rubric | automatic_grader | test_suite | game_outcome"
+ },
+ "version": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40,
+ "description": "The pinned judge version; a run names the judge it used."
+ },
+ "config": {
+ "type": "object",
+ "additionalProperties": true
+ }
+ }
+ },
+ "scoring": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "method",
+ "scale"
+ ],
+ "properties": {
+ "method": {
+ "enum": [
+ "mean",
+ "weighted",
+ "pass_rate",
+ "sum"
+ ]
+ },
+ "scale": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "min",
+ "max"
+ ],
+ "properties": {
+ "min": {
+ "type": "number"
+ },
+ "max": {
+ "type": "number"
+ }
+ },
+ "description": "The score scale; 0-100 for Bench."
+ },
+ "weights": {
+ "type": "object",
+ "additionalProperties": {
+ "type": "number"
+ }
+ },
+ "aggregate": {
+ "type": "string",
+ "maxLength": 120
+ }
+ }
+ },
+ "limits": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "ms": {
+ "type": "number",
+ "minimum": 0,
+ "description": "The time one task may take."
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "usd": {
+ "type": "number",
+ "minimum": 0,
+ "description": "The money one run may spend."
+ }
+ }
+ },
+ "matrix": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "models": {
+ "type": "array",
+ "maxItems": 50,
+ "items": {
+ "type": "string",
+ "maxLength": 160
+ }
+ },
+ "harnesses": {
+ "type": "array",
+ "maxItems": 50,
+ "items": {
+ "type": "string",
+ "maxLength": 160
+ }
+ }
+ }
+ }
+ }
+ },
"bench.entry.v1": {
"bench.entry.v1": {
"title": "BenchEntry",
"title": "BenchEntry",
"description": "bench.entry@1: one submitted entry in a benchmark round on OpenVibe.Bench. PROVIDER and MODEL name the offer; ROUTE is the exact route taken; PAYLOAD is the entry's output; COST_USD, MS and TOKENS are the measured use.",
"description": "bench.entry@1: one submitted entry in a benchmark round on OpenVibe.Bench. PROVIDER and MODEL name the offer; ROUTE is the exact route taken; PAYLOAD is the entry's output; COST_USD, MS and TOKENS are the measured use.",
@@ -981,6 +1749,245 @@
@@ -981,6 +1749,245 @@
}
}
}
}
},
},
+ "bench.run.v1": {
+ "title": "BenchRun",
+ "description": "bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "id",
+ "benchmark",
+ "model",
+ "route",
+ "provider",
+ "harness",
+ "settings",
+ "executor",
+ "tasks",
+ "score",
+ "cost_usd",
+ "ms",
+ "tokens",
+ "provenance",
+ "created_at"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "pattern": "^brn_[0-9A-HJKMNP-TV-Z]{26}$"
+ },
+ "benchmark": {
+ "type": "string",
+ "pattern": "^bmk_[0-9A-HJKMNP-TV-Z]{26}@[0-9]+$",
+ "description": "The benchmark id@version the run names. The version is immutable once a run names it."
+ },
+ "model": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ },
+ "route": {
+ "type": "object",
+ "additionalProperties": true,
+ "description": "The route that produced the run."
+ },
+ "provider": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "harness": {
+ "$ref": "#/components/schemas/bench.run.v1/$defs/harness"
+ },
+ "settings": {
+ "$ref": "#/components/schemas/bench.run.v1/$defs/settings"
+ },
+ "executor": {
+ "$ref": "#/components/schemas/bench.run.v1/$defs/executor"
+ },
+ "hardware": {
+ "type": [
+ "object",
+ "null"
+ ],
+ "additionalProperties": true,
+ "description": "The machine for a local model: GPU, CPU, RAM, quantization."
+ },
+ "tasks": {
+ "type": "array",
+ "maxItems": 200,
+ "items": {
+ "$ref": "#/components/schemas/bench.run.v1/$defs/task"
+ }
+ },
+ "score": {
+ "type": "number",
+ "minimum": 0,
+ "maximum": 100,
+ "description": "The benchmark's one score for this run, 0-100."
+ },
+ "cost_usd": {
+ "type": "number",
+ "minimum": 0
+ },
+ "ms": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "provenance": {
+ "enum": [
+ "verified",
+ "submitted"
+ ],
+ "description": "verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings."
+ },
+ "created_at": {
+ "type": "string",
+ "format": "date-time"
+ }
+ },
+ "$defs": {
+ "harness": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "name",
+ "version"
+ ],
+ "properties": {
+ "name": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "version": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ }
+ }
+ },
+ "settings": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "temperature": {
+ "type": [
+ "number",
+ "null"
+ ],
+ "minimum": 0,
+ "maximum": 2
+ },
+ "effort": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 40
+ }
+ }
+ },
+ "executor": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "kind",
+ "id"
+ ],
+ "properties": {
+ "kind": {
+ "enum": [
+ "codes",
+ "run",
+ "node",
+ "submitted"
+ ],
+ "description": "codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it."
+ },
+ "id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 160
+ }
+ }
+ },
+ "task": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "task_id",
+ "score",
+ "cost_usd",
+ "ms",
+ "tokens",
+ "artifacts"
+ ],
+ "properties": {
+ "task_id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 80
+ },
+ "score": {
+ "type": "number",
+ "minimum": 0,
+ "maximum": 100
+ },
+ "cost_usd": {
+ "type": "number",
+ "minimum": 0
+ },
+ "ms": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "tokens": {
+ "type": "integer",
+ "minimum": 0
+ },
+ "artifacts": {
+ "type": "array",
+ "maxItems": 40,
+ "items": {
+ "$ref": "#/components/schemas/bench.run.v1/$defs/artifact"
+ }
+ },
+ "transcript_url": {
+ "type": [
+ "string",
+ "null"
+ ],
+ "maxLength": 500
+ }
+ }
+ },
+ "artifact": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "kind",
+ "url"
+ ],
+ "properties": {
+ "kind": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40
+ },
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 500
+ }
+ }
+ }
+ }
+ },
"bench.task.v1": {
"bench.task.v1": {
"title": "BenchTask",
"title": "BenchTask",
"description": "bench.task@1: a benchmark task on OpenVibe.Bench. KIND_ID names the kind; PROMPT is the task; TASK_CLASS is an optional classifier for grouping; PUBLIC says whether the task may appear in the open dataset.",
"description": "bench.task@1: a benchmark task on OpenVibe.Bench. KIND_ID names the kind; PROMPT is the task; TASK_CLASS is an optional classifier for grouping; PUBLIC says whether the task may appear in the open dataset.",
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Billing API",
"title": "OpenVibe.Billing API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Billing as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Billing as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Blog API",
"title": "OpenVibe.Blog API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Blog as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Blog as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Bot API",
"title": "OpenVibe.Bot API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Bot as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Bot as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Chat API",
"title": "OpenVibe.Chat API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Chat as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Chat as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Codes API",
"title": "OpenVibe.Codes API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Codes as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Codes as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Community API",
"title": "OpenVibe.Community API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Community as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Community as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Coupons API",
"title": "OpenVibe.Coupons API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Coupons as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Coupons as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Deals API",
"title": "OpenVibe.Deals API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Deals as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Deals as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Domains API",
"title": "OpenVibe.Domains API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Domains as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Domains as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Events API",
"title": "OpenVibe.Events API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Events as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Events as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Food API",
"title": "OpenVibe.Food API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Food as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Food as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Games API",
"title": "OpenVibe.Games API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Games as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Games as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Host API",
"title": "OpenVibe.Host API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Host as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Host as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -19,8 +19,8 @@
@@ -19,8 +19,8 @@
"service": "bench",
"service": "bench",
"name": "OpenVibe.Bench",
"name": "OpenVibe.Bench",
"origin": "https://bench.openvibe.services",
"origin": "https://bench.openvibe.services",
- "operations": 12,
- "capabilities": 9
+ "operations": 21,
+ "capabilities": 12
},
},
{
{
"service": "billing",
"service": "billing",
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Inventory API",
"title": "OpenVibe.Inventory API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Inventory as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Inventory as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Live API",
"title": "OpenVibe.Live API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Live as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Live as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.MediaHub API",
"title": "OpenVibe.MediaHub API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.MediaHub as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.MediaHub as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Media API",
"title": "OpenVibe.Media API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Media as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Media as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Network API",
"title": "OpenVibe.Network API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Network as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Network as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.News API",
"title": "OpenVibe.News API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.News as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.News as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenRestream API",
"title": "OpenRestream API",
- "version": "0.243.0",
- "description": "Routes of OpenRestream as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenRestream as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Play API",
"title": "OpenVibe.Play API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Play as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Play as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Quest API",
"title": "OpenVibe.Quest API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Quest as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Quest as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Reviews API",
"title": "OpenVibe.Reviews API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Reviews as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Reviews as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Run API",
"title": "OpenVibe.Run API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Run as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Run as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Search API",
"title": "OpenVibe.Search API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Search as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Search as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Services API",
"title": "OpenVibe.Services API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Services as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Services as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Sources API",
"title": "OpenVibe.Sources API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Sources as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Sources as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Space API",
"title": "OpenVibe.Space API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Space as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Space as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Tips API",
"title": "OpenVibe.Tips API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Tips as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Tips as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Tools API",
"title": "OpenVibe.Tools API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Tools as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Tools as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Trade API",
"title": "OpenVibe.Trade API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Trade as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Trade as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.VIP API",
"title": "OpenVibe.VIP API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.VIP as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.VIP as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Vision API",
"title": "OpenVibe.Vision API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Vision as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Vision as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Watch API",
"title": "OpenVibe.Watch API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Watch as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Watch as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Wiki API",
"title": "OpenVibe.Wiki API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Wiki as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Wiki as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Work API",
"title": "OpenVibe.Work API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Work as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Work as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -3,8 +3,8 @@
@@ -3,8 +3,8 @@
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"jsonSchemaDialect": "https://json-schema.org/draft/2020-12/schema",
"info": {
"info": {
"title": "OpenVibe.Zone API",
"title": "OpenVibe.Zone API",
- "version": "0.243.0",
- "description": "Routes of OpenVibe.Zone as the OpenVibe contracts describe them (openvibe-contracts 0.243.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
+ "version": "0.244.0",
+ "description": "Routes of OpenVibe.Zone as the OpenVibe contracts describe them (openvibe-contracts 0.244.0): each operation lists the capabilities it performs; a token for the service's audience must carry one of them unless the route is open. Generated by openvibe-contracts lib/openapi.js; do not edit.",
"license": {
"license": {
"name": "MIT",
"name": "MIT",
"identifier": "MIT"
"identifier": "MIT"
@@ -215095,3 +215095,3614 @@ export type ReportMetric = {
@@ -215095,3 +215095,3614 @@ export type ReportMetric = {
} & {
} & {
[k: string]: unknown | undefined;
[k: string]: unknown | undefined;
};
};
+
+/** [email protected] (owner: bench) */
+/**
+ * bench.benchmark@1: a versioned, forkable benchmark on OpenVibe.Bench (spec 2026-10-30, stage B1). It holds the tasks, the judges, how they combine into one score, the limits a run may not pass, the default model and harness matrix, and the visibility of the set. A version is immutable once a run names it; a fork records its parent. Holdout tasks are hidden from everyone but the executor.
+ */
+export interface BenchBenchmark {
+ /**
+ * The benchmark id; the same id carries every version.
+ */
+ id: string;
+ /**
+ * Who owns the set: a person (usr_…), a project (prj_…) or a service.
+ */
+ owner: string;
+ title: string;
+ description: string;
+ /**
+ * One of Bench's kinds; the kind sets the renderer.
+ */
+ kind: string;
+ /**
+ * @maxItems 32
+ */
+ tags: string[];
+ /**
+ * public appears in the registry; unlisted is reachable by id only.
+ */
+ visibility: "public" | "unlisted";
+ /**
+ * The version number this document is.
+ */
+ version: number;
+ /**
+ * The benchmark id@version this one was forked from, or null.
+ */
+ parent: string | null;
+ /**
+ * @maxItems 200
+ */
+ tasks: {
+ id: string;
+ prompt: string;
+ /**
+ * @maxItems 20
+ */
+ files?:
+ | []
+ | [
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ]
+ | [
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ },
+ {
+ [k: string]: unknown | undefined;
+ }
+ ];
+ /**
+ * What a correct answer looks like; the judge reads it.
+ */
+ expected?: {
+ [k: string]: unknown | undefined;
+ } | null;
+ /**
+ * A hidden task: it is withheld from everyone but the executor, against tuning to the set.
+ */
+ holdout: boolean;
+ }[];
+ /**
+ * @maxItems 20
+ */
+ judges:
+ | []
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ]
+ | [
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ },
+ {
+ /**
+ * human_pairwise | rubric | automatic_grader | test_suite | game_outcome
+ */
+ type: string;
+ /**
+ * The pinned judge version; a run names the judge it used.
+ */
+ version: string;
+ config: {
+ [k: string]: unknown | undefined;
+ };
+ }
+ ];
+ scoring: {
+ method: "mean" | "weighted" | "pass_rate" | "sum";
+ /**
+ * The score scale; 0-100 for Bench.
+ */
+ scale: {
+ min: number;
+ max: number;
+ };
+ weights?: {
+ [k: string]: number | undefined;
+ };
+ aggregate?: string;
+ };
+ limits: {
+ /**
+ * The time one task may take.
+ */
+ ms?: number;
+ tokens?: number;
+ /**
+ * The money one run may spend.
+ */
+ usd?: number;
+ };
+ matrix: {
+ /**
+ * @maxItems 50
+ */
+ models?: string[];
+ /**
+ * @maxItems 50
+ */
+ harnesses?: string[];
+ };
+ created_at: string;
+}
+
+/** [email protected] (owner: bench) */
+/**
+ * bench.run@1: one result on OpenVibe.Bench (spec 2026-10-30, stage B1): a benchmark version × a model × a harness × settings × who ran it. It records the environment, the per-task scores, cost, time and tokens, the rendered artifacts, and the provenance: verified (executed by openvibe.codes or a Run worker) or submitted (posted through the API with no executor). Submitted runs are badged and kept out of the default rankings.
+ */
+export interface BenchRun {
+ id: string;
+ /**
+ * The benchmark id@version the run names. The version is immutable once a run names it.
+ */
+ benchmark: string;
+ model: string;
+ /**
+ * The route that produced the run.
+ */
+ route: {
+ [k: string]: unknown | undefined;
+ };
+ provider: string;
+ harness: {
+ name: string;
+ version: string;
+ };
+ settings: {
+ temperature?: number | null;
+ effort?: string | null;
+ };
+ executor: {
+ /**
+ * codes: an openvibe.codes task. run: a Run worker. node: the person's own machine. submitted: nobody ran it.
+ */
+ kind: "codes" | "run" | "node" | "submitted";
+ id: string;
+ };
+ /**
+ * The machine for a local model: GPU, CPU, RAM, quantization.
+ */
+ hardware?: {
+ [k: string]: unknown | undefined;
+ } | null;
+ /**
+ * @maxItems 200
+ */
+ tasks: {
+ task_id: string;
+ score: number;
+ cost_usd: number;
+ ms: number;
+ tokens: number;
+ /**
+ * @maxItems 40
+ */
+ artifacts: {
+ kind: string;
+ url: string;
+ }[];
+ transcript_url?: string | null;
+ }[];
+ /**
+ * The benchmark's one score for this run, 0-100.
+ */
+ score: number;
+ cost_usd: number;
+ ms: number;
+ tokens: number;
+ /**
+ * verified: an executor ran it. submitted: posted with no executor, badged and out of the default rankings.
+ */
+ provenance: "verified" | "submitted";
+ created_at: string;
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,24 @@
@@ -0,0 +1,24 @@
+{
+ "id": "bench.benchmark.write",
+ "version": "1.0",
+ "owner": "bench",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "Create, edit, freeze and fork a benchmark on OpenVibe.Bench: POST /api/v1/benchmarks, PUT /api/v1/benchmarks/:id, POST /api/v1/benchmarks/:id/versions (freeze) and POST /api/v1/benchmarks/:id/fork. A version is immutable once a run names it; a fork records its parent.",
+ "permissions": [
+ "bench:benchmarks:write"
+ ],
+ "resourceConstraints": [
+ "owner"
+ ],
+ "quotaClass": "standard",
+ "events": [],
+ "implementedBy": [
+ "POST /api/v1/benchmarks",
+ "PUT /api/v1/benchmarks/:id",
+ "POST /api/v1/benchmarks/:id/versions",
+ "POST /api/v1/benchmarks/:id/fork"
+ ],
+ "inputSchema": "bench.benchmark@1",
+ "outputSchema": "bench.benchmark@1"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+{
+ "id": "bench.run.publish",
+ "version": "1.0",
+ "owner": "bench",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "Publish a run of a benchmark on OpenVibe.Bench: POST /api/v1/runs with the per-task scores, cost, time and tokens. A service token (openvibe.codes, Run) makes a verified run; a person's token makes a submitted run, which is badged and kept out of the default rankings. A caller with this capability sees a benchmark's holdout tasks.",
+ "permissions": [
+ "bench:runs:publish"
+ ],
+ "resourceConstraints": [
+ "none"
+ ],
+ "quotaClass": "standard",
+ "events": [],
+ "implementedBy": [
+ "POST /api/v1/runs"
+ ],
+ "inputSchema": "bench.run@1",
+ "outputSchema": "bench.run@1"
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,24 @@
@@ -0,0 +1,24 @@
+{
+ "id": "bench.run.read",
+ "version": "1.0",
+ "owner": "bench",
+ "status": "active",
+ "visibility": "first-party",
+ "description": "Read the runs of a benchmark on OpenVibe.Bench: GET /api/v1/runs/:id returns the result, GET /api/v1/compare?runs=a,b,c puts runs side by side, GET /api/v1/rankings gives the per-benchmark rankings and GET /api/v1/percentile says where a score sits among a model's runs.",
+ "permissions": [
+ "bench:runs:read"
+ ],
+ "resourceConstraints": [
+ "none"
+ ],
+ "quotaClass": "standard",
+ "events": [],
+ "implementedBy": [
+ "GET /api/v1/runs/:id",
+ "GET /api/v1/compare",
+ "GET /api/v1/rankings",
+ "GET /api/v1/percentile"
+ ],
+ "inputSchema": "common.no-body@1",
+ "outputSchema": "bench.run@1"
+}
@@ -28,7 +28,10 @@
@@ -28,7 +28,10 @@
"bench.judgement.cast",
"bench.judgement.cast",
"bench.board.read",
"bench.board.read",
"bench.dataset.read",
"bench.dataset.read",
- "bench.eval.manage"
+ "bench.eval.manage",
+ "bench.benchmark.write",
+ "bench.run.publish",
+ "bench.run.read"
],
],
"eventsProduced": [
"eventsProduced": [
"bench.round.finished",
"bench.round.finished",
@@ -93,7 +96,7 @@
@@ -93,7 +96,7 @@
"none": "no claims: one process owns its PostgreSQL database (ov_bench); a round is taken with a compare-and-set on its status (queued → running)"
"none": "no claims: one process owns its PostgreSQL database (ov_bench); a round is taken with a compare-and-set on its status (queued → running)"
}
}
},
},
- "notes": "Port 5230, unit openvibe-bench.service, database ov_bench. The shared AI benchmark and comparison engine (plan 2026-10-28 §4.12): kinds, pinned rounds, entries, blind judgements, ratings, boards, the open dataset and private evals. AI stays the run authority, Report the analytics authority. Bench answers account export and deletion (ADR-033) and subscribes to the two account Events topics. It calls AI's run API (ai.run.create, namespace ai.generate) for generation and publishes bench.* events through its outbox.",
+ "notes": "Port 5230, unit openvibe-bench.service, database ov_bench. The shared AI benchmark and comparison engine (plan 2026-10-28 §4.12): kinds, pinned rounds, entries, blind judgements, ratings, boards, the open dataset and private evals. AI stays the run authority, Report the analytics authority. Bench answers account export and deletion (ADR-033) and subscribes to the two account Events topics. It calls AI's run API (ai.run.create, namespace ai.generate) for generation and publishes bench.* events through its outbox. Stage B1 (spec 2026-10-30): the registry — versioned, forkable benchmarks (bench.benchmark@1) and the results API (bench.run@1), with compare, rankings and percentiles.",
"site": {
"site": {
"name": "Bench",
"name": "Bench",
"icon": "bench",
"icon": "bench",
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
{
{
"name": "openvibe-contracts",
"name": "openvibe-contracts",
- "version": "0.243.0",
+ "version": "0.244.0",
"description": "Machine-readable contracts for the OpenVibe network: schemas, manifests, ids, error envelope, capability checks.",
"description": "Machine-readable contracts for the OpenVibe network: schemas, manifests, ids, error envelope, capability checks.",
"main": "index.js",
"main": "index.js",
"types": "generated/typescript/index.d.ts",
"types": "generated/typescript/index.d.ts",