openvibers/openvibe.sdk Commits

0.145.0: shared Bench ratings for every site (#198)

OP OpenVibers committed 3 days ago · 87f08a3c0f · 11 files, +87 −13

Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex 923d96b..fc13537 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -1,5 +1,9 @@
  # Changelog
  
++## 0.145.0 — 2026-11-10
++
++- `bench.ratings.get`: shared fitted ratings and confidence intervals with a twenty-game floor. Browser-safe, anonymous and typed; pairs with `ai.metrics` across sites.
++
  ## 0.144.0 — 2026-11-10
  
  - `ai.metrics`: public daily model series, up-to-six-model comparisons, task leaderboards, listed price history and hourly provider health. Typed sample counts and nullable measurements; available in browser and server clients without credentials.
ddiff --git a/README.md b/README.md
iindex 7e52990..7fb03bf 100644
---- a/README.md
++++ b/README.md
@@@ -13,7 +13,7 @@ npm install https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags
  
  > Supported browser and server clients for the OpenVibe platform.
  
--**Status:** alpha, v0.144.0 (the five 1.0 core entries — `core`, `auth`, `events`, `db` and `media` — are frozen: their exported names change only with a major release; roadmap Wave 2; developer apps from Wave 20; Media objects v2 from Wave 4; the Tools platform API; Search, Chat, AI, Vision and consent clients; sign-in with recovery and a client health check). Every release is tagged (see [CHANGELOG.md](CHANGELOG.md)). Tested against local stub servers and the built-in mock platform only, never against the live platform. Services pin release tags; each repository's `STATUS.json` names the one it uses.
++**Status:** alpha, v0.145.0 (the five 1.0 core entries — `core`, `auth`, `events`, `db` and `media` — are frozen: their exported names change only with a major release; roadmap Wave 2; developer apps from Wave 20; Media objects v2 from Wave 4; the Tools platform API; Search, Chat, AI, Vision and consent clients; sign-in with recovery and a client health check). Every release is tagged (see [CHANGELOG.md](CHANGELOG.md)). Tested against local stub servers and the built-in mock platform only, never against the live platform. Services pin release tags; each repository's `STATUS.json` names the one it uses.
  **Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30.  
  **License:** MIT ([LICENSE](LICENSE)). This package is a library that apps outside the network embed, so it uses MIT. The OpenVibe services themselves stay AGPL-3.0.
  
@@@ -22,7 +22,7 @@ npm install https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags
  After v0.97.1 is tagged:
  
  ```bash
--npm install https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.144.0
++npm install https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.145.0
  ```
  
  It has no runtime dependencies. It needs Node ≥ 20, or any browser with `fetch`, Web Crypto and `TextDecoder`. There is no build step. The package is CommonJS with ESM entry points (`import` works). Each subpath has its own `.d.ts`. For a page with no bundler, `browser/openvibe-sdk.mjs` is one self-contained ES module of the browser-safe subpaths (see [Browser without a bundler](#browser-without-a-bundler)). It does not depend on `openvibe-contracts` at runtime: it copies the contract types it uses (from the temporary Contracts sharing branch, a devDependency the tests check them against).
@@@ -206,7 +206,7 @@ Until OpenVibe.Tools serves the gateway facade (Tools S6), point the jobs client
  | `openvibe-sdk/home-feed` | both | `createHomeFeedClient(client)` gives `ov.home.get()/set(layout)`, `ov.network.today()`, `ov.feed.list({ for: 'me', cursor })`, `ov.feed.post(text)`, and `ov.feed.contribute({ id, text, url })`. Personal calls need a user Bearer token; contribution needs an app token granted `community.pulse.contribute`. Today is public and cached by Network for five minutes. |
  | `openvibe-sdk/bot` | both | `createBotClient(client, {actingSubject?})`: OpenVibe.Bot robots. `robots.list/get/create/update/delete/pairingCode`, `robots.operators.list/add/remove`, `robots.devices`, `robots.streaming.get/set`, `robots.audit`/`iterateAudit`, `robots.estop`/`clearEstop`, `robots.command` (`bot.command@1`), `devices.rotate/revoke`, public `profiles.list/get` and `kits.list/get`, `pair` (the agent's one-time code, never retried), `as(subject)`; a service acts for a person with `X-OV-Subject` (capabilities `bot.robot.read`/`manage`/`control`, `bot.device.connect`) |
  | `openvibe-sdk/themes` | both | `createThemesClient(client)`: `catalog({mode, search, sort, limit})` (seasonal themes carry `season`), `get(idOrSlug)` (null when missing), `export(idOrSlug)`, `active()` (`{theme_id, custom_variables, display}`), `set(themeId, {custom})`, `setDisplay({motion, text, seasonal})`, `seasons({date})`, `submissions()`, `submit(theme)`, `import(file)`. In the browser: `followTheme(fn)` calls back now and on every theme, display or season change with `readTheme()` (`{id, mode, tokens, display: {motion, text, textScale, seasonal, calm}, season, seasonMark}`) and returns `stop()`, for a canvas or game to keep up; `applyTheme(vars, {root})`, `seasonOn(date, seasons)`, `onColor(hex)`, `DISPLAY_OPTIONS`, `TOKENS` |
--| `openvibe-sdk/bench` | both | `createBenchClient(client, {actingSubject?})`: `kinds.list/get`, `tasks.create/get`, `rounds.create/get`, `entries.submit`, `judgements.next/cast`, `boards.get`, `models.get`, `datasets.get/iterate`, `evals.create/get/run/results`, `mcpTools()`. Audience `openvibe.bench`; service id `bench`. |
++| `openvibe-sdk/bench` | both | `createBenchClient(client, {actingSubject?})`: `kinds.list/get`, `tasks.create/get`, `rounds.create/get`, `entries.submit`, `judgements.next/cast`, `boards.get`, `ratings.get` (public, at least twenty games), `models.get`, `datasets.get/iterate`, `evals.create/get/run/results`, `mcpTools()`. Audience `openvibe.bench`; service id `bench`. |
  | `openvibe-sdk/vision` | both | `createVisionClient(client)`: image questions, OCR, private search, network tools, UI checks, consent and face privacy, and `/mcp`. Audience `openvibe.vision`; service id `vision`. |
  | `openvibe-sdk/consent` | server | `createConsent({networkUrl, token, ttlMs?})`: the L24 check before any use of a person's content or likeness for AI or data. `allowed(subject, purpose, {kind, client, feature})` → boolean, `filter(subjects, purpose, opts)` → the allowed subset (batches of 500, input order), `onChange(event)` drops that subject's answers for the purpose on `network.consent.changed`, `mock(map)` for tests. **Fails closed** (network error, 2 s timeout, non-200 or malformed answer → false); caches 60 s; logs at most once a minute. [Example](docs/consent.md) |
  | `openvibe-sdk/search` | both | `createSearchClient(client, {actingSubject?})`: `query(text, {owner, type, lang, filter, facets, limit, cursor})`, `iterate(text, {…, max})`, `suggest(text, {owner, type, limit})`, `document(owner, type, id)` (null when missing or not visible) |
ddiff --git a/STATUS.json b/STATUS.json
iindex 087c9e2..0079520 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -3,7 +3,7 @@
    "stage": "alpha",
    "code": true,
    "wave": "W2",
--  "version": "0.144.0",
++  "version": "0.145.0",
    "plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), \u00a73.2; roadmap \u00a730",
    "deployed": false,
    "deployment": "none: a package released as git tags; services and apps pin a tag",
ddiff --git a/browser/openvibe-sdk.mjs b/browser/openvibe-sdk.mjs
iindex 7aa2356..8ef2866 100644
---- a/browser/openvibe-sdk.mjs
++++ b/browser/openvibe-sdk.mjs
@@@ -1,4 +1,4 @@
--// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.144.0, MIT). Do not edit.
++// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.145.0, MIT). Do not edit.
  // One self-contained ES module: core at the top level plus the browser-safe subpaths as namespaces
  // (auth, locker, registry, trust, levels, modules, realtime, media, cam, mediaUpload, ui, community, homeFeed, bot, search, bench, meme, vision, themes, design, redact, chat, ai, credentials, jobs, tools, projects, geo). No server-only code: no service tokens, JWT verification, events, identity.
  /* eslint-disable */
@@@ -57,7 +57,7 @@ const { paginate, offsetPager } = __load(7);
  const semver = __load(6);
  
  /** The package version (test/version.test.js keeps it equal to package.json). */
--const SDK_VERSION = '0.144.0';
++const SDK_VERSION = '0.145.0';
  
  module.exports = {
      createClient,
@@@ -3982,7 +3982,7 @@ const path = (part, id) => `/api/v1/${part}/${enc(id)}`;
  
  function createBenchClient(client, defaults = {}) {
      function options(opts = {}, perCall = {}) {
--        const acting = perCall.actingSubject !== undefined ? perCall.actingSubject : defaults.actingSubject;
++        const acting = opts.auth === false ? null : perCall.actingSubject !== undefined ? perCall.actingSubject : defaults.actingSubject;
          const headers = { ...opts.headers };
          if (acting) {
              if (!isActingSubjectId(acting)) throw new TypeError('actingSubject must be a usr_… or gst_… subject id');
@@@ -4017,6 +4017,13 @@ function createBenchClient(client, defaults = {}) {
          boards: {
              get: (kind, { type, category, period, ...o } = {}) => json({ path: path('boards', kind), query: { type, category, period } }, o),
          },
++        ratings: {
++            get(kind, { category, signal } = {}) {
++                if (typeof kind !== 'string' || !/^[a-zA-Z0-9_.:-]{1,100}$/.test(kind)) throw new TypeError('Use a Bench kind id');
++                if (category !== undefined && (typeof category !== 'string' || !/^[a-zA-Z0-9_.:-]{1,80}$/.test(category))) throw new TypeError('Use a task category of at most 80 characters');
++                return json({ method: 'GET', auth: false, path: path('ratings', kind), query: { category } }, { signal });
++            },
++        },
          models: {
              get: (model, o = {}) => json({ path: path('models', model) }, o),
          },
ddiff --git a/docs/shared-ratings.md b/docs/shared-ratings.md
nnew file mode 100644
iindex 0000000..875fd97
---- /dev/null
++++ b/docs/shared-ratings.md
@@@ -0,0 +1,18 @@
++# Shared Bench ratings
++
++```js
++const { createClient } = require('openvibe-sdk/core');
++const { createBenchClient } = require('openvibe-sdk/bench');
++const bench = createBenchClient(createClient(), { baseUrl: 'https://bench.openvibe.services' });
++const { board } = await bench.ratings.get('design.page', { category: 'all' });
++```
++
++This browser-safe GET sends no token or acting subject. It uses Bench's fitted
++rating, confidence and rank intervals. Models with fewer than twenty eligible
++games are omitted, including their counts. An empty board is insufficient
++evidence, not a score. `signal` is accepted for cancellation.
++
++Use `subject` as the model identity and keep kind/category with the rating.
++Do not average ratings across different kinds, model revisions or categories.
++Combine these quality observations with `ai.metrics` for receipt-based speed and
++cost; those measures have their own twenty-observation floor.
ddiff --git a/package-lock.json b/package-lock.json
iindex 6a785f2..8e4d808 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -1,12 +1,12 @@
  {
    "name": "openvibe-sdk",
--  "version": "0.144.0",
++  "version": "0.145.0",
    "lockfileVersion": 3,
    "requires": true,
    "packages": {
      "": {
        "name": "openvibe-sdk",
--      "version": "0.144.0",
++      "version": "0.145.0",
        "license": "MIT",
        "bin": {
          "openvibe-download": "bin/openvibe-download.js"
ddiff --git a/package.json b/package.json
iindex 4561067..ef9a4d7 100644
---- a/package.json
++++ b/package.json
@@@ -1,6 +1,6 @@
  {
    "name": "openvibe-sdk",
--  "version": "0.144.0",
++  "version": "0.145.0",
    "description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, Bot robots, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.",
    "main": "index.js",
    "types": "types/index.d.ts",
ddiff --git a/src/bench.js b/src/bench.js
iindex b8f4418..4c9e452 100644
---- a/src/bench.js
++++ b/src/bench.js
@@@ -8,7 +8,7 @@ const path = (part, id) => `/api/v1/${part}/${enc(id)}`;
  
  function createBenchClient(client, defaults = {}) {
      function options(opts = {}, perCall = {}) {
--        const acting = perCall.actingSubject !== undefined ? perCall.actingSubject : defaults.actingSubject;
++        const acting = opts.auth === false ? null : perCall.actingSubject !== undefined ? perCall.actingSubject : defaults.actingSubject;
          const headers = { ...opts.headers };
          if (acting) {
              if (!isActingSubjectId(acting)) throw new TypeError('actingSubject must be a usr_… or gst_… subject id');
@@@ -43,6 +43,13 @@ function createBenchClient(client, defaults = {}) {
          boards: {
              get: (kind, { type, category, period, ...o } = {}) => json({ path: path('boards', kind), query: { type, category, period } }, o),
          },
++        ratings: {
++            get(kind, { category, signal } = {}) {
++                if (typeof kind !== 'string' || !/^[a-zA-Z0-9_.:-]{1,100}$/.test(kind)) throw new TypeError('Use a Bench kind id');
++                if (category !== undefined && (typeof category !== 'string' || !/^[a-zA-Z0-9_.:-]{1,80}$/.test(category))) throw new TypeError('Use a task category of at most 80 characters');
++                return json({ method: 'GET', auth: false, path: path('ratings', kind), query: { category } }, { signal });
++            },
++        },
          models: {
              get: (model, o = {}) => json({ path: path('models', model) }, o),
          },
ddiff --git a/src/core/index.js b/src/core/index.js
iindex 8bef2d2..043b8b6 100644
---- a/src/core/index.js
++++ b/src/core/index.js
@@@ -8,7 +8,7 @@ const { paginate, offsetPager } = require('./paginate');
  const semver = require('./semver');
  
  /** The package version (test/version.test.js keeps it equal to package.json). */
--const SDK_VERSION = '0.144.0';
++const SDK_VERSION = '0.145.0';
  
  module.exports = {
      createClient,
ddiff --git a/test/bench-ratings.test.js b/test/bench-ratings.test.js
nnew file mode 100644
iindex 0000000..ea80965
---- /dev/null
++++ b/test/bench-ratings.test.js
@@@ -0,0 +1,26 @@
++'use strict';
++const assert = require('node:assert/strict');
++const { createBenchClient } = require('../src/bench');
++const { createClient } = require('../src/core');
++(async () => {
++    const calls = [];
++    const board = { schema: 'bench.ratings@1', kind_id: 'design.page', minimum_samples: 20, rows: [] };
++    const bench = createBenchClient({ json: async (request) => { calls.push(request); return { board }; } }, { actingSubject: 'bad-on-purpose' });
++    const signal = new AbortController().signal;
++    assert.deepEqual(await bench.ratings.get('design.page', { category: 'writing', signal }), { board });
++    assert.equal(calls[0].path, '/api/v1/ratings/design.page');
++    assert.equal(calls[0].auth, false);
++    assert.equal(calls[0].method, 'GET');
++    assert.equal(calls[0].query.category, 'writing');
++    assert.equal(calls[0].signal, signal);
++    assert.deepEqual(calls[0].headers, {});
++    assert.throws(() => bench.ratings.get('design/page'), /kind id/);
++    assert.throws(() => bench.ratings.get('design.page', { category: ['a', 'b'] }), /category/);
++    const publicClient = createBenchClient(createClient({ fetch: async (_url, init) => {
++        const headers = new Headers(init.headers);
++        assert(!headers.has('authorization') && !headers.has('x-ov-subject'));
++        return new Response(JSON.stringify({ board }), { headers: { 'Content-Type': 'application/json' } });
++    }, auth: { getToken: async () => { throw new Error('Public ratings must not request a token'); } } }), { baseUrl: 'https://bench.example.test', actingSubject: 'usr_01JAB2C3D4E5F6G7H8J9K0MNPQ' });
++    assert.equal((await publicClient.ratings.get('design.page')).board.minimum_samples, 20);
++    console.log('bench.ratings: public fitted reads preserve empty evidence, forward signals and send no identity');
++})().catch((err) => { console.error(err); process.exitCode = 1; });
ddiff --git a/types/bench.d.ts b/types/bench.d.ts
iindex 48add62..a8fd6a9 100644
---- a/types/bench.d.ts
++++ b/types/bench.d.ts
@@@ -23,6 +23,16 @@ export interface BenchJudgementInput {
      pair?: { left: string; right: string }; outcome?: 'left' | 'right' | 'tie' | 'both_bad' | 'draw'; score?: number;
  }
  export interface BenchBoard { kind_id: string; type: 'elo' | 'pareto' | 'cost'; category: string; period: string; rows: Array<Record<string, unknown>>; }
++export interface BenchRating {
++    kind_id: string; category: string; subject: string; elo: number; games: number;
++    wins: number; losses: number; ties: number; win_rate: number; provisional: boolean;
++    ci_lower: number | null; ci_upper: number | null;
++    rank: number | null; rank_lo: number | null; rank_hi: number | null;
++}
++export interface BenchSharedRatings {
++    schema: 'bench.ratings@1'; kind_id: string; category: string; type: 'elo'; adult: false;
++    minimum_samples: 20; source: string; methodology: string; rows: BenchRating[];
++}
  export interface BenchEval { id: string; owner: string; kind_id: string; name: string; test_suite: unknown[]; created_at: string; }
  export interface BenchEvalRun { id: string; eval_id: string; offers: BenchOffer[]; status: string; created_at: string; [field: string]: unknown; }
  export type BenchDatasetLine = ({ type: 'task' } & BenchTask) | ({ type: 'round' } & BenchRound) | ({ type: 'entry' } & BenchEntry) | ({ type: 'judgement' | 'rating' } & Record<string, unknown>);
@@@ -49,7 +59,9 @@ export interface BenchClient {
          cast(roundId: string, judgement: BenchJudgementInput, opts?: BenchOptions): Promise<{ judgement: Record<string, unknown>; revealed: { left: BenchEntry; right: BenchEntry } | null }>;
      };
      boards: { get(kind: string, query?: { type?: 'elo' | 'pareto' | 'cost'; category?: string; period?: string } & BenchOptions): Promise<{ board: BenchBoard }> };
--    models: { get(model: string, opts?: BenchOptions): Promise<{ model: string; ratings: Array<Record<string, unknown>>; entries: BenchEntry[]; total_cost_usd: number }> };
++    /** Public fitted ratings. Models below twenty eligible games are omitted. No identity is sent. */
++    ratings: { get(kind: string, query?: { category?: string; signal?: AbortSignal }): Promise<{ board: BenchSharedRatings }> };
++    models: { get(model: string, opts?: BenchOptions): Promise<{ model: string; ratings: BenchRating[]; ratings_minimum_samples?: number; entries: BenchEntry[]; total_cost_usd: number }> };
      datasets: {
          /** Raw application/x-ndjson text. */
          get(kind: string, opts?: BenchOptions): Promise<string>;