openvibers/openvibe.vision Commits
VIS-5: Vision's capabilities, MCP tools and events (contracts 0.240.0, sdk 0.72.0) (#11)
Changes
@@@ -2,6 +2,15 @@What changed in OpenVibe.Vision, newest first.++## Unreleased — VIS-5++++- Added a Streamable HTTP MCP endpoint with eleven tools, protected-resource metadata, token challenges and the++ same permission boundaries as the JSON API. Face identity and consent mutation remain person-only API flows.++- Aligned service-token checks with the twenty capability IDs in Contracts 0.240.0. Anonymous local and privacy++ tools remain open. Verification gives the same refusal before and after consent until enrollment is complete.++- Publishes `vision.result.ready` and `vision.alert` through OpenVibe.Events when Events is configured. Event++ payloads contain provenance and class names, never image bytes, answer text or face templates.++## 0.1.0 — 2026-10-28The first version: see with the best model for the job.@@@ -92,26 +92,43 @@ General answer tables store no image bytes, token or provider key. The opt-in fa## Capabilities and grants--The service offers eleven capabilities. The main session registers their manifests and grants them; v1 checks the--grant list directly, so a token that already carries one works.++Contracts 0.240.0 registers twenty Vision capabilities. The service checks each machine token's grant on its route;++private media and face identity operations also require the verified owner or the person's purpose-scoped consent.| Capability | Routes | Typical caller ||---|---|---|--| `vision.ask` | `POST /api/v1/ask`, `GET /api/v1/answers/:id` | a first-party service or a person |--| `vision.ocr` | `POST /api/v1/ocr` (the AI fallback also needs a person or this grant) | a first-party service or a person |--| `vision.compare` | `POST /api/v1/compare` | a first-party service or a person |++| `vision.image.ask` | `POST /api/v1/ask`, `GET /api/v1/answers/:id` | a first-party service or a person |++| `vision.text.read` | `POST /api/v1/ocr` (the AI fallback also needs a person or this grant) | a first-party service or a person |++| `vision.model.compare` | `POST /api/v1/compare` | a first-party service or a person || `vision.ui.check` | `POST /api/v1/ui-check`, `GET /api/v1/ui-checks/:id` | **OpenVibe.Design's Arena**, or a person || `vision.face.cast.match` | `POST /api/v1/faces/cast/match` (a signed `channel_id` claim must match the request) | **OpenVibe.Live**, or a person inspecting their own tags |--| `vision.alt` | `POST /api/v1/alt` | Pics, Media and Chat |--| `vision.find` | `POST /api/v1/index`, `GET`/`POST /api/v1/find` (owner user token) | Media and Search |--| `vision.receipt` | `POST /api/v1/receipt` | Food |--| `vision.product` | `POST /api/v1/product` | Deals |--| `vision.watch` | `POST /api/v1/watch` (owner user token) | Cam |--| `vision.moments` | `POST /api/v1/moments` (owner user token) | Live and Media |++| `vision.alt.write` | `POST /api/v1/alt` | Pics, Media and Chat |++| `vision.media.find` | `POST /api/v1/index`, `GET`/`POST /api/v1/find` (owner user token) | Media and Search |++| `vision.receipt.read` | `POST /api/v1/receipt` | Food |++| `vision.product.match` | `POST /api/v1/product` | Deals |++| `vision.camera.watch` | `POST /api/v1/watch` (owner user token) | Cam |++| `vision.video.moments` | `POST /api/v1/moments` (owner user token) | Live and Media |++| `vision.face.detect`, `vision.face.blur` | `POST /api/v1/faces/detect`, `/blur` | Public privacy tools; a service token needs the matching grant |++| `vision.face.consent.manage`, `vision.face.verify.self`, `vision.face.find.self`, `vision.face.library`, `vision.face.cast.read` | `/api/v1/faces/*` | The person themselves, with separate signed consent for each matching purpose |++| `vision.models.read`, `vision.benchmarks.read` | `GET /api/v1/models`, `/benchmarks` | Public reads; a service token needs the matching grant |Outgoing: **`ai.run.create`** to OpenVibe.AI (audience `openvibe.ai`, namespace `ai.*`), minted by Network for theclient `vision`. No new AI capability, no provider key, no direct paid provider.++## MCP and events++++`POST /mcp` serves Streamable HTTP JSON-RPC 2.0 (`initialize`, `ping`, `tools/list`, `tools/call`).++`/.well-known/oauth-protected-resource` advertises its Network authorization server; a private tool without a++token responds with a `WWW-Authenticate` challenge pointing to that metadata. The tools are `vision.ask_image`,++`vision.read_text`, `vision.translate_sign`, `vision.find_similar`, `vision.search_my_media`,++`vision.describe_clip`, `vision.detect_objects`, `vision.blur_faces`, `vision.check_ui`,++`vision.compare_models` and `vision.get_job`. Face enrolment, verification, cast matching and consent mutation++are available only through their person or Live-specific API routes, never through MCP.++++When configured with Events and a Network client secret, successful owner results publish `vision.result.ready`++with provenance, cost and retention, without image or answer text. A watch result with `source_ref` and detections++publishes `vision.alert` with classes and confidence. Publishing failures are logged without exposing payloads.++## APIJSON under `/api/v1`; errors are `application/problem+json` with a `code`. An image is a multipart `image` field, a@@@ -125,7 +142,7 @@ must hold the route's capability (403 `capability.denied`).| `POST /api/v1/ocr` | multipart/data URL/URL; `{ fallback?, lang?, model? }`; returns the OCR row with `text`, `answer.confidence`, `answer.words`, cost 0. || `POST /api/v1/compare` | `{ task?, question?, models?: string[] }` plus an image; returns `{ answer, compare }`, one row per engine. || `POST /api/v1/ui-check` | `{ html }`, `{ url }`, or a screenshot plus `{ viewport: { w, h } }`; returns `{ check }` with `findings`, `counts`, `severity`, `warnings`. |--| `GET /api/v1/answers/:id` | one saved answer: its owner, or a service holding `vision.ask`. |++| `GET /api/v1/answers/:id` | one saved answer: its owner, or a service holding `vision.image.ask`. || `GET /api/v1/ui-checks/:id` | one saved UI check: its owner, or a service holding `vision.ui.check`. || `GET /api/v1/models` | AI's offered `provider:model` hints and the local engine. No subject data. || `GET /api/v1/benchmarks?task=&limit=` | aggregate compare samples: task, engine, provider, model, samples, mean latency, mean cost. No subject or answer text. |@@@ -156,7 +173,7 @@ Answer (abridged):Validation refusals: `vision.invalid`, `vision.invalid_task`, `vision.invalid_mode`, `vision.question_required`,`vision.image_required`, `vision.image_too_large`, `vision.image_private_url`, `vision.image_unsupported`,--`vision.ocr_unavailable`, `vision.ai_unavailable`, `vision.not_found`, `vision.rate_limited`, and the face codes++`vision.text.read_unavailable`, `vision.ai_unavailable`, `vision.not_found`, `vision.rate_limited`, and the face codes`faces.consent_required`, `faces.signature_required`, `faces.adult_required`, `faces.region_disabled`,`faces.enrolment_required`, `faces.one_face_required`, `faces.model_unavailable`, `faces.key_unavailable`,`faces.live_grant_required`.@@@ -229,12 +246,12 @@ All six tools are JSON APIs. Successful `answer` objects carry a `receipt` with| Room call | Capability | Notes ||---|---|---|--| Pics, Media, Chat → `POST /api/v1/alt` | `vision.alt` | Routed VLM through OpenVibe.AI; short editable `alt_text`; noLooks in the prompt and output. |--| Media, Search → `POST /api/v1/index`, `GET /api/v1/find?q=`, `POST /api/v1/find` | `vision.find` | Owner's verified user token; index `{item_id,image,result_url?}`; search by text or image. Owner is in every index read predicate. |--| Food → `POST /api/v1/receipt` | `vision.receipt` | Tesseract first; `fallback=true` permits routed VLM if local extraction is incomplete. |--| Deals → `POST /api/v1/product` | `vision.product` | CLIP vector and routed product description; `allow_rerank=true` permits another routed VLM call to rerank candidate offers. |--| Cam → `POST /api/v1/watch` | `vision.watch` | Owner's verified user token; upload still and `classes` array; optional `previous_image` data URL for scene change. |--| Live, Media → `POST /api/v1/moments` | `vision.moments` | Owner's verified user token; `{frames:[{timestamp_ms,image|image_url}],prompts?}`. Scene histogram and CLIP prompt matching. |++| Pics, Media, Chat → `POST /api/v1/alt` | `vision.alt.write` | Routed VLM through OpenVibe.AI; short editable `alt_text`; noLooks in the prompt and output. |++| Media, Search → `POST /api/v1/index`, `GET /api/v1/find?q=`, `POST /api/v1/find` | `vision.media.find` | Owner's verified user token; index `{item_id,image,result_url?}`; search by text or image. Owner is in every index read predicate. |++| Food → `POST /api/v1/receipt` | `vision.receipt.read` | Tesseract first; `fallback=true` permits routed VLM if local extraction is incomplete. |++| Deals → `POST /api/v1/product` | `vision.product.match` | CLIP vector and routed product description; `allow_rerank=true` permits another routed VLM call to rerank candidate offers. |++| Cam → `POST /api/v1/watch` | `vision.camera.watch` | Owner's verified user token; upload still and `classes` array; optional `previous_image` data URL for scene change. |++| Live, Media → `POST /api/v1/moments` | `vision.video.moments` | Owner's verified user token; `{frames:[{timestamp_ms,image|image_url}],prompts?}`. Scene histogram and CLIP prompt matching. |For private index, watch and moments, a service capability alone does not assert a source owner's grant. The room should call with the owner's delegated Network user token. Private index and watch reject `image_url`; the owner must supply bytes or a data URL. Moments accepts public frame URLs under Vision's existing guarded image fetch, or frame data URLs. The room that opens a returned `result_url` must check its own source grant again.@@@ -8,26 +8,35 @@"runtime": "node 22, express 4, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, Tesseract 5, sharp, onnxruntime-node (YuNet/SFace), Transformers.js (CLIP/DETR)","port": "5130","domain": "openvibe.vision",-- "contracts": "openvibe-contracts v0.235.0",-- "sdk": "openvibe-sdk v0.70.0",++ "contracts": "openvibe-contracts v0.240.0",++ "sdk": "openvibe-sdk v0.72.0","shared": "openvibe-shared v4.41.0","packages": [-- "openvibe-contracts v0.235.0",-- "openvibe-sdk v0.70.0",++ "openvibe-contracts v0.240.0",++ "openvibe-sdk v0.72.0","openvibe-shared v4.41.0"],"capabilities": [-- "vision.ask",-- "vision.ocr",-- "vision.compare",++ "vision.image.ask",++ "vision.text.read",++ "vision.model.compare","vision.ui.check","vision.face.cast.match",-- "vision.alt",-- "vision.find",-- "vision.receipt",-- "vision.product",-- "vision.watch",-- "vision.moments"++ "vision.alt.write",++ "vision.media.find",++ "vision.receipt.read",++ "vision.product.match",++ "vision.camera.watch",++ "vision.video.moments",++ "vision.face.detect",++ "vision.face.blur",++ "vision.face.consent.manage",++ "vision.face.verify.self",++ "vision.face.find.self",++ "vision.face.library",++ "vision.face.cast.read",++ "vision.models.read",++ "vision.benchmarks.read"],"outboundCapabilities": ["ai.run.create"@@@ -46,6 +55,8 @@"Consent-scoped SFace matching: verify (1:1), self search in own Vision uploads, adult cast rosters and region-gated private library grouping","Encrypted face templates, opt-in uploads and vectors (AES-256-GCM); immediate template/tag erasure on opt-out and a daily three-year retention job","Network tools: routed alt text, owner-scoped CLIP index/find, Tesseract-first receipts, product vectors, DETR watch and moments, each with a receipt",++ "MCP on /mcp with RFC 9728 protected-resource metadata, token challenges and route-equivalent capability checks",++ "vision.result.ready and vision.alert published through OpenVibe.Events when configured","Answers, compare samples, private index and UI checks stored without image bytes; account export/deletion over every table including faces and the private index","Network sign-in as OAuth client vision (PKCE S256, vision_at/vision_rt/vision_oauth), truthful readiness, loopback metrics","Server-rendered pages with stable vi- ids/classes, discovery artifacts, deploy unit and nginx vhost, CI"@@@ -18,8 +18,8 @@"iovalkey": "^0.4.0","multer": "^2.5.0","onnxruntime-node": "^1.31.1",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.240.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.72.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.41.0","pg": "^8.23.0","sharp": "^0.33.5"@@@ -2625,9 +2625,9 @@"license": "MIT"},"node_modules/openvibe-contracts": {-- "version": "0.235.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",-- "integrity": "sha512-Rfmy2sDZOO+fegEPBweQvCKlamkYwOXGyr1M80VkYHXHl8gg7CnyDL7HrbGwuXG3LUcPqzhAMYeeB//jKMtzaw==",++ "version": "0.240.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.240.0",++ "integrity": "sha512-GiwXO9Y3YHhIuthBNzUrkZByIdOPK+ZjIklCLt1pYQRkPdyG59yGhi5ha9GMZ5d6z1AAkQ0QTwexGfJ23lgB0g==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -2641,9 +2641,9 @@}},"node_modules/openvibe-sdk": {-- "version": "0.70.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",-- "integrity": "sha512-iMNxlnRm0fmyj60nHiC/lKAaG3JAqiJgu42lShnkby4yF4dQJRG3G0trrqOoULZKBalrYHb/gjauRwNQYN/tog==",++ "version": "0.72.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.72.0",++ "integrity": "sha512-RaMMhd1n280MXeGd4c+EVTOALyoQcnhEO2UnbgMu+7lalxdXDkvivao+C6TraojrhPFdNGskbShcywvt2KVd7w==","license": "MIT","bin": {"openvibe-download": "bin/openvibe-download.js"@@@ -28,8 +28,8 @@"iovalkey": "^0.4.0","multer": "^2.5.0","onnxruntime-node": "^1.31.1",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.240.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.72.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.41.0","pg": "^8.23.0","sharp": "^0.33.5"@@@ -29,6 +29,7 @@ const { createSso } = require('./auth/sso');const { createPrincipal } = require('./http/principal');const { createApi } = require('./http/api');const { createFaceApi } = require('./http/faces-api');++const { mcpRouter } = require('./http/mcp');const { createPageRoutes } = require('./http/pages');const { createServiceReadiness } = require('./observability');const { assetVersion, send } = require('./render/layout');@@@ -42,6 +43,7 @@ const { createVision } = require('./vision/service');const { createFacesModel } = require('./vision/faces-model');const { createFaces } = require('./vision/faces');const { createNetworkSender } = require('openvibe-sdk/account-data');++const { createPublisher } = require('./events/publisher');const PUBLIC_DIR = path.join(__dirname, '..', 'public');const VERSION = require('../package.json').version;@@@ -77,7 +79,8 @@ async function createApp(opts = {}) {? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl }): async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Vision cannot answer account events'); });-- const ctx = { config, s, keys, sso, principal, store, ocr, ai, vision, faces, models, networkTools, accountData, accountSend, log, fetchImpl };++ const eventPublisher = opts.eventPublisher || createPublisher({ config, fetchImpl, log });++ const ctx = { config, s, keys, sso, principal, store, ocr, ai, vision, faces, models, networkTools, accountData, accountSend, eventPublisher, log, fetchImpl };const app = express();app.disable('x-powered-by');@@@ -153,6 +156,7 @@ async function createApp(opts = {}) {}));// ── API ─────────────────────────────────────────────────++ app.use(mcpRouter(ctx));app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.VISION_API_RATE_LIMIT_PER_MIN) || 120, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));app.use('/api/v1/faces', createFaceApi(ctx));app.use('/api/v1', createApi(ctx));nnew file mode 100644@@@ -0,0 +1,68 @@++'use strict';++++const { createClient } = require('openvibe-sdk/core');++const { createServiceTokenClient } = require('openvibe-sdk/auth');++const { createEventsClient } = require('openvibe-sdk/events');++++const TASKS = Object.freeze({++ '/ask': 'ask', '/describe': 'describe', '/ocr': 'ocr', '/compare': 'compare', '/alt': 'alt',++ '/find': 'find', '/receipt': 'receipt', '/product': 'product', '/watch': 'watch',++ '/moments': 'moments', '/ui-check': 'ui_check',++});++++function createPublisher({ config, fetchImpl = globalThis.fetch, log = console } = {}) {++ if (config.drill || !config.events.url || !config.oauth.clientSecret) return { result: async () => {}, alert: async () => {} };++ const tokenProvider = createServiceTokenClient({++ tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId,++ clientSecret: config.oauth.clientSecret, audience: 'openvibe.events', scope: config.events.scope, fetch: fetchImpl,++ });++ const client = createClient({ baseUrls: { events: config.events.url }, tokenProvider, fetch: fetchImpl, retries: 1 });++ const events = createEventsClient(client, { source: 'vision' });++++ function subject(req) {++ const p = req.principal;++ if (p && p.kind === 'user') return { type: 'user', id: p.subject };++ if (p && p.kind === 'service') return { type: 'service', id: p.sub };++ return null;++ }++ async function publish(req, type, payload) {++ const ref = subject(req);++ if (!ref) return;++ try {++ await events.publish({ event_type: type, actor: ref, subject: ref, visibility: ref.type === 'user' ? 'subject' : 'internal', priority: 'important', payload }, { traceparent: req.ov && req.ov.traceparent });++ } catch (err) { log.warn('[OpenVibe.Vision] event publish failed:', String(err && err.message || err).slice(0, 180)); }++ }++ async function result(req, out) {++ const task = req.visionTask || TASKS[req.path];++ if (!task || !out) return;++ const value = out.answer || out.check;++ if (!value) return;++ const receipt = value.receipt || {};++ await publish(req, 'vision.result.ready', {++ result_id: value.id || null, task,++ source: req.file ? 'upload' : req.body && req.body.image ? 'data' : req.body && (req.body.image_url || req.body.url) ? 'url' : req.body && req.body.html ? 'html' : 'none',++ model: value.model || receipt.model || null,++ provider: value.provider || null,++ route: value.route || receipt.engine || null,++ cost_usd: Number(value.cost_usd ?? receipt.cost_usd ?? 0),++ latency_ms: Math.max(0, Math.round(Number(value.latency_ms ?? receipt.ms ?? 0))),++ confidence: Number.isFinite(Number(receipt.confidence)) && receipt.confidence != null ? Number(receipt.confidence) : null,++ retention: String(receipt.retention || (value.id ? 'result stored for owner' : 'not stored')).slice(0, 200),++ consent_id: null, created_at: new Date().toISOString(),++ });++ }++ async function alert(req, out) {++ if (req.path !== '/watch' || !out || !out.answer || !Array.isArray(out.answer.detections) || !out.answer.detections.length) return;++ const ref = req.body && req.body.source_ref;++ if (typeof ref !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9:/_-]{1,159}$/.test(ref)) return;++ const detections = out.answer.detections.slice(0, 20);++ await publish(req, 'vision.alert', {++ kind: 'object', source_ref: ref, classes: detections.map((d) => String(d.class).slice(0, 60)),++ confidence: Number.isFinite(Number(detections[0].confidence)) ? Number(detections[0].confidence) : null,++ detected_at: new Date().toISOString(),++ });++ }++ return { result, alert };++}++++module.exports = { createPublisher };@@@ -8,7 +8,7 @@* POST /ocr the local Tesseract reader; `fallback=1` asks AI only when it finds no text* POST /compare local OCR vs one or more AI model hints, side by side, with cost and latency* POST /ui-check overflow, cut-off, contrast, overlap and layering faults from html, url or image-- * GET /answers/:id one saved answer (its owner, or a service holding vision.ask)++ * GET /answers/:id one saved answer (its owner, or a service holding vision.image.ask)* GET /ui-checks/:id one saved UI check (its owner, or a service holding vision.ui.check)* GET /models the AI route's offered provider:model hints (no subject data)* GET /benchmarks aggregate compare samples: model, task, samples, mean latency and cost@@@ -25,7 +25,7 @@ const { loadImage } = require('../vision/images');const { VisionError } = require('../vision/service');function createApi(ctx) {-- const { config, s, principal, store, vision, ai, ocr, fetchImpl, networkTools, models: localModels } = ctx;++ const { config, s, principal, store, vision, ai, ocr, fetchImpl, networkTools, models: localModels, eventPublisher } = ctx;const r = asyncRouter();const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: config.images.maxMb * 1024 * 1024, files: 1 } });@@@ -94,6 +94,8 @@ function createApi(ctx) {async function run(res, fn, status = 200) {try {const out = await fn();++ await eventPublisher.result(res.req, out);++ await eventPublisher.alert(res.req, out);return res.status(status).json(out);} catch (err) {if (err instanceof VisionError) {@@@ -116,27 +118,27 @@ function createApi(ctx) {const owner = (req) => req.principal.subject;const noRemoteSource = (req, res, next) => bodyOf(req).image_url ? invalid(res, 'private tools accept uploaded bytes or a data URL, not an unverified source URL') : next();-- r.post('/alt', principal.requireCapability('vision.alt'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.alt(await imageOf(req)) }), 201));-- r.post('/index', principal.requireCapability('vision.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {++ r.post('/alt', principal.requireCapability('vision.alt.write'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.alt(await imageOf(req)) }), 201));++ r.post('/index', principal.requireCapability('vision.media.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {const b = bodyOf(req);return { answer: await networkTools.index(owner(req), b.item_id, await imageOf(req), b.result_url) };}, 201));-- r.get('/find', principal.requireCapability('vision.find'), ownerOnly, (req, res) => run(res, async () => {++ r.get('/find', principal.requireCapability('vision.media.find'), ownerOnly, (req, res) => run(res, async () => {const q = String(req.query.q || '').trim();if (!q || q.length > 500) throw new VisionError(400, 'vision.invalid_query', 'q must contain 1–500 characters');return { answer: await networkTools.find(owner(req), q) };}));-- r.post('/find', principal.requireCapability('vision.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => ({ answer: await networkTools.find(owner(req), await imageOf(req)) })));-- r.post('/receipt', principal.requireCapability('vision.receipt'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.readReceipt(await imageOf(req), bodyOf(req).fallback === true || bodyOf(req).fallback === 'true') }), 201));-- r.post('/product', principal.requireCapability('vision.product'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.product(await imageOf(req), bodyOf(req).allow_rerank === true || bodyOf(req).allow_rerank === 'true') }), 201));-- r.post('/watch', principal.requireCapability('vision.watch'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {++ r.post('/find', principal.requireCapability('vision.media.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => ({ answer: await networkTools.find(owner(req), await imageOf(req)) })));++ r.post('/receipt', principal.requireCapability('vision.receipt.read'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.readReceipt(await imageOf(req), bodyOf(req).fallback === true || bodyOf(req).fallback === 'true') }), 201));++ r.post('/product', principal.requireCapability('vision.product.match'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.product(await imageOf(req), bodyOf(req).allow_rerank === true || bodyOf(req).allow_rerank === 'true') }), 201));++ r.post('/watch', principal.requireCapability('vision.camera.watch'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {const b = bodyOf(req);const classes = Array.isArray(b.classes) ? b.classes : String(b.classes || '').split(',').map((x) => x.trim()).filter(Boolean);if (!classes.length || classes.length > 20) throw new VisionError(400, 'vision.invalid_classes', 'send 1–20 watched classes');const previous = b.previous_image ? await loadImage({ image: b.previous_image, config, fetchImpl }) : null;return { answer: await networkTools.watch(await imageOf(req), classes, previous) };}, 201));-- r.post('/moments', principal.requireCapability('vision.moments'), ownerOnly, (req, res) => run(res, async () => {++ r.post('/moments', principal.requireCapability('vision.video.moments'), ownerOnly, (req, res) => run(res, async () => {const b = bodyOf(req);if (!Array.isArray(b.frames) || b.frames.length < 1 || b.frames.length > 30) throw new VisionError(400, 'vision.invalid_frames', 'send 1–30 timestamped frames');const frames = [];@@@ -150,10 +152,10 @@ function createApi(ctx) {}, 201));// ── OCR: local first, anonymous local calls allowed ─────-- r.post('/ocr', attachUpload, principal.optional('vision.ocr', { allowAnonymous: true }), async (req, res) => {++ r.post('/ocr', attachUpload, principal.optional('vision.text.read', { allowAnonymous: true }), async (req, res) => {const b = bodyOf(req);const fallback = b.fallback === true || b.fallback === '1' || b.fallback === 'true';-- if (req.principal.kind === 'anonymous' && fallback) return problem(res, 401, 'token.required', 'AI fallback needs a signed-in person or a service token holding vision.ocr');++ if (req.principal.kind === 'anonymous' && fallback) return problem(res, 401, 'token.required', 'AI fallback needs a signed-in person or a service token holding vision.text.read');return run(res, async () => {const image = await imageOf(req);return await vision.ocr({@@@ -172,7 +174,7 @@ function createApi(ctx) {const question = String(b.question || b.q || '').trim();const readish = vision.isReadTask(task, question);if (req.principal.kind === 'anonymous' && mode !== 'local' && !readish) {-- return problem(res, 401, 'token.required', 'Ask needs a signed-in person or a service token holding vision.ask; anonymous callers can use mode=local or a read question that the local OCR engine answers.');++ return problem(res, 401, 'token.required', 'Ask needs a signed-in person or a service token holding vision.image.ask; anonymous callers can use mode=local or a read question that the local OCR engine answers.');}return run(res, async () => {const image = await imageOf(req);@@@ -183,11 +185,11 @@ function createApi(ctx) {return { answer: out };}, 201);};-- r.post('/ask', attachUpload, principal.optional('vision.ask', { allowAnonymous: true }), askHandler('ask'));-- r.post('/describe', attachUpload, principal.optional('vision.ask', { allowAnonymous: true }), askHandler('describe'));++ r.post('/ask', attachUpload, principal.optional('vision.image.ask', { allowAnonymous: true }), askHandler('ask'));++ r.post('/describe', attachUpload, principal.optional('vision.image.ask', { allowAnonymous: true }), askHandler('describe'));// ── Compare: local + AI hints, side by side ────────────-- r.post('/compare', principal.optional('vision.compare'), attachUpload, async (req, res) => {++ r.post('/compare', principal.optional('vision.model.compare'), attachUpload, async (req, res) => {const b = bodyOf(req);const models = Array.isArray(b.models) ? b.models : (typeof b.models === 'string' && b.models.trim() ? b.models.split(',').map((x) => x.trim()) : []);return run(res, async () => {@@@ -229,7 +231,7 @@ function createApi(ctx) {});// ── Reads ──────────────────────────────────────────────-- r.get('/answers/:id', principal.optional('vision.ask', { allowAnonymous: true }), async (req, res) => {++ r.get('/answers/:id', principal.optional('vision.image.ask', { allowAnonymous: true }), async (req, res) => {const row = await store.answerById(String(req.params.id || ''));if (!row) return problem(res, 404, 'vision.not_found', 'No such answer.');const owner = await store.answerSubject(row.id);@@@ -253,7 +255,7 @@ function createApi(ctx) {// ── Model hints and benchmarks (no subject data) ───────let modelCache = null;-- r.get('/models', async (_req, res) => {++ r.get('/models', principal.optional('vision.models.read', { allowAnonymous: true }), async (_req, res) => {if (!modelCache || Date.now() - modelCache.at > 5 * 60_000) {const offered = ai.enabled ? await ai.offeredModels().catch(() => []) : [];modelCache = { at: Date.now(), offered };@@@ -268,7 +270,7 @@ function createApi(ctx) {});});-- r.get('/benchmarks', async (req, res) => {++ r.get('/benchmarks', principal.optional('vision.benchmarks.read', { allowAnonymous: true }), async (req, res) => {const task = typeof req.query.task === 'string' && req.query.task ? req.query.task : null;const limit = req.query.limit == null ? 50 : Number.parseInt(req.query.limit, 10);if (!Number.isFinite(limit) || limit < 1 || limit > 100) return invalid(res, 'limit must be from 1 to 100');@@@ -38,8 +38,8 @@ function createFaceApi({ config, principal, faces, fetchImpl }) {standardHeaders: true, legacyHeaders: false,handler: (req, res) => problem(res, 429, 'vision.rate_limited', 'too many face privacy requests from this address; retry shortly'),});-- r.post('/detect', privacyLimit, imageField, run(async (req) => faces.detect((await image(req)).buffer)));-- r.post('/blur', privacyLimit, imageField, async (req, res) => {++ r.post('/detect', privacyLimit, principal.optional('vision.face.detect', { allowAnonymous: true }), imageField, run(async (req) => faces.detect((await image(req)).buffer)));++ r.post('/blur', privacyLimit, principal.optional('vision.face.blur', { allowAnonymous: true }), imageField, async (req, res) => {try {const out = await faces.blur((await image(req)).buffer);res.set('X-Faces-Blurred', String(out.count));nnew file mode 100644@@@ -0,0 +1,186 @@++'use strict';++++const express = require('express');++const rateLimit = require('express-rate-limit');++const { loadImage } = require('../vision/images');++const { check } = require('../vision/ui-check');++const { cleanAnswer } = require('../text/no-looks');++const { sameOrigin } = require('./principal');++++const VERSION = '2025-06-18';++const VERSIONS = new Set([VERSION, '2025-03-26']);++const input = (properties, required = []) => ({ type: 'object', properties, required, additionalProperties: false });++const image = { image: { type: 'string', description: 'Image data URL' }, image_url: { type: 'string', description: 'Public HTTPS image URL' } };++const TOOLS = Object.freeze([++ { name: 'vision.ask_image', description: 'Ask about an image. Anonymous requests use local OCR only.', inputSchema: input({ ...image, question: { type: 'string' }, mode: { type: 'string', enum: ['auto', 'local', 'ai'] } }) },++ { name: 'vision.read_text', description: 'Read visible text with local OCR.', inputSchema: input({ ...image, lang: { type: 'string' } }) },++ { name: 'vision.translate_sign', description: 'Translate visible sign text through a routed model.', inputSchema: input({ ...image, lang: { type: 'string' } }) },++ { name: 'vision.find_similar', description: 'Search your private visual index by image.', inputSchema: input({ image: image.image }, ['image']) },++ { name: 'vision.search_my_media', description: 'Search your private visual index by words.', inputSchema: input({ query: { type: 'string' } }, ['query']) },++ { name: 'vision.describe_clip', description: 'Find scene changes and moments in your own frames.', inputSchema: input({ frames: { type: 'array', items: { type: 'object' } }, prompts: { type: 'array', items: { type: 'string' } } }, ['frames']) },++ { name: 'vision.detect_objects', description: 'Detect requested object classes in a source you own.', inputSchema: input({ image: image.image, classes: { type: 'array', items: { type: 'string' } }, source_ref: { type: 'string' } }, ['image', 'classes']) },++ { name: 'vision.blur_faces', description: 'Blur faces without identifying anyone.', inputSchema: input(image) },++ { name: 'vision.check_ui', description: 'Check a page or screenshot for layout and accessibility faults.', inputSchema: input({ ...image, html: { type: 'string' }, url: { type: 'string' }, viewport: { type: 'object' } }) },++ { name: 'vision.compare_models', description: 'Compare OCR with up to three routed model hints.', inputSchema: input({ ...image, question: { type: 'string' }, models: { type: 'array', items: { type: 'string' } } }) },++ { name: 'vision.get_job', description: 'Get a saved answer or UI check that you may read.', inputSchema: input({ id: { type: 'string' } }, ['id']) },++]);++++const ACCESS = Object.freeze({++ 'vision.ask_image': ['vision.image.ask', 'local'], 'vision.read_text': ['vision.text.read', 'anonymous'],++ 'vision.translate_sign': ['vision.image.ask'], 'vision.find_similar': ['vision.media.find', 'person'],++ 'vision.search_my_media': ['vision.media.find', 'person'], 'vision.describe_clip': ['vision.video.moments', 'person'],++ 'vision.detect_objects': ['vision.camera.watch', 'person'], 'vision.blur_faces': ['vision.face.blur', 'anonymous'],++ 'vision.check_ui': ['vision.ui.check', 'anonymous'], 'vision.compare_models': ['vision.model.compare'],++ 'vision.get_job': ['vision.image.ask'],++});++const EVENT_PATH = Object.freeze({++ 'vision.ask_image': '/ask', 'vision.read_text': '/ocr', 'vision.translate_sign': '/ask',++ 'vision.find_similar': '/find', 'vision.search_my_media': '/find', 'vision.describe_clip': '/moments',++ 'vision.detect_objects': '/watch', 'vision.check_ui': '/ui-check', 'vision.compare_models': '/compare',++});++++function rpcResult(id, value) { return { jsonrpc: '2.0', id, result: value }; }++function rpcError(id, code, message) { return { jsonrpc: '2.0', id: id === undefined ? null : id, error: { code, message } }; }++function toolResult(id, value) {++ if (value && typeof value.image === 'string' && value.image.startsWith('data:image/png;base64,')) {++ return rpcResult(id, { content: [++ { type: 'text', text: `Blurred ${value.count} face(s).` },++ { type: 'image', mimeType: 'image/png', data: value.image.slice('data:image/png;base64,'.length) },++ ], structuredContent: { count: value.count }, isError: false });++ }++ return rpcResult(id, { content: [{ type: 'text', text: cleanAnswer(JSON.stringify(value)) }], structuredContent: value, isError: false });++}++function toolError(id, message) { return rpcResult(id, { content: [{ type: 'text', text: message }], isError: true }); }++function owner(req) { return req.principal && req.principal.kind === 'user' ? req.principal.subject : null; }++function access(req, name, ctx, args = {}) {++ let [cap, policy] = ACCESS[name];++ if (name === 'vision.get_job' && String(args.id || '').startsWith('vuc_')) cap = 'vision.ui.check';++ const p = req.principal;++ if (policy === 'person' && !owner(req)) return 'A verified owner user token is required.';++ if (p.kind === 'user') return p.viaSession && !sameOrigin(req, ctx.config.baseUrl) ? 'A signed-in change must come from Vision.' : null;++ if (p.kind === 'service') return cap && !ctx.principal.granted(p, cap) ? `${cap} not granted.` : null;++ if (policy === 'anonymous') return null;++ if (policy === 'local' && name === 'vision.ask_image') return null;++ return 'A Network token is required.';++}++++async function invoke(name, args, req, ctx) {++ const { config, fetchImpl, vision, networkTools, faces, store, ocr } = ctx;++ const load = () => loadImage({ image: args.image, imageUrl: args.image_url, config, fetchImpl });++ const loadPrivate = () => {++ if (args.image_url || typeof args.image !== 'string' || !args.image.startsWith('data:image/')) throw new Error('Private tools accept an image data URL only.');++ return load();++ };++ const requester = req.principal.requester || 'anonymous';++ const subject = owner(req);++ const request = { requester, subject };++ switch (name) {++ case 'vision.ask_image': {++ const question = String(args.question || '').slice(0, 2000);++ const mode = req.principal.kind === 'anonymous' ? 'local' : (args.mode || 'auto');++ return { answer: await vision.ask({ ...request, image: await load(), task: 'ask', question, mode }) };++ }++ case 'vision.read_text': return await vision.ocr({ ...request, image: await load(), lang: args.lang, fallback: false });++ case 'vision.translate_sign': return { answer: await vision.ask({ ...request, image: await load(), task: 'translate', question: 'Translate the visible sign text.', mode: 'ai', lang: args.lang }) };++ case 'vision.find_similar': return { answer: await networkTools.find(subject, await loadPrivate()) };++ case 'vision.search_my_media': {++ if (typeof args.query !== 'string' || !args.query.trim() || args.query.length > 500) throw new Error('query must contain 1–500 characters');++ return { answer: await networkTools.find(subject, args.query) };++ }++ case 'vision.describe_clip': {++ if (!Array.isArray(args.frames) || !args.frames.length || args.frames.length > 30) throw new Error('send 1–30 frames');++ const frames = [];++ for (const f of args.frames) {++ if (!Number.isFinite(f.timestamp_ms) || f.timestamp_ms < 0 || !f.image) throw new Error('each frame needs timestamp_ms and an image data URL');++ frames.push({ timestamp_ms: f.timestamp_ms, image: await loadImage({ image: f.image, config, fetchImpl }) });++ }++ return { answer: await networkTools.moments(frames, args.prompts) };++ }++ case 'vision.detect_objects': {++ if (!Array.isArray(args.classes) || !args.classes.length || args.classes.length > 20) throw new Error('send 1–20 classes');++ return { answer: await networkTools.watch(await loadPrivate(), args.classes) };++ }++ case 'vision.blur_faces': {++ const out = await faces.blur((await load()).buffer);++ return { count: out.count, image: `data:image/png;base64,${out.buffer.toString('base64')}` };++ }++ case 'vision.check_ui': {++ const viewport = args.viewport && typeof args.viewport === 'object' ? args.viewport : { w: 1280, h: 800 };++ const source = args.image || args.image_url ? { imageBuffer: (await load()).buffer } : args.html ? { html: args.html } : { url: args.url };++ if (!source.imageBuffer && !source.html && !source.url) throw new Error('send an image, html or url');++ return { check: await check({ ...source, viewport, config, fetchImpl, ocr, maxFindings: config.ui.maxFindings }) };++ }++ case 'vision.compare_models': return await vision.compare({ ...request, image: await load(), question: args.question || '', task: 'ask', models: args.models || [] });++ case 'vision.get_job': {++ const id = String(args.id || '');++ const ui = id.startsWith('vuc_');++ const row = ui ? await store.uiCheckById(id) : await store.answerById(id);++ const rowOwner = row && (ui ? await store.uiCheckSubject(id) : await store.answerSubject(id));++ if (!row || (req.principal.kind !== 'service' && (!subject || rowOwner !== subject))) throw new Error('No such result.');++ return ui ? { check: row } : { answer: row };++ }++ default: throw new Error('Unknown tool');++ }++}++++function mcpRouter(ctx) {++ const r = express.Router();++ const metadata = `${ctx.config.baseUrl}/.well-known/oauth-protected-resource`;++ const challenge = (res, cap) => res.set('WWW-Authenticate', `Bearer resource_metadata="${metadata}"${cap ? `, scope="${cap}"` : ''}`);++ const document = () => ({ resource: `${ctx.config.baseUrl}/mcp`, authorization_servers: [ctx.config.networkUrl], scopes_supported: [...new Set(Object.values(ACCESS).map(([cap]) => cap).filter(Boolean))], bearer_methods_supported: ['header'], resource_documentation: `${ctx.config.baseUrl}/developers` });++ r.get('/.well-known/oauth-protected-resource', (_req, res) => res.json(document()));++ r.get('/.well-known/oauth-protected-resource/mcp', (_req, res) => res.json(document()));++ r.options('/mcp', (_req, res) => res.set({ 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'POST, OPTIONS', 'Access-Control-Allow-Headers': 'Content-Type, Authorization, MCP-Protocol-Version' }).status(204).end());++ r.post('/mcp', rateLimit({ windowMs: 60_000, limit: Number(process.env.VISION_MCP_RATE_LIMIT_PER_MIN) || 60, standardHeaders: true, legacyHeaders: false }), express.json({ limit: `${Math.ceil(ctx.config.images.maxMb * 1.5) + 2}mb`, strict: false }), async (req, res, next) => {++ res.set('Cache-Control', 'no-store');++ try {++ const resolved = await ctx.principal.resolve(req);++ if (resolved.error) { challenge(res); return res.status(resolved.error[0]).json(rpcError(null, -32001, resolved.error[2])); }++ req.principal = resolved.principal;++ if (req.get('mcp-protocol-version') && !VERSIONS.has(req.get('mcp-protocol-version'))) return res.status(400).json(rpcError(null, -32600, 'Unsupported protocol version'));++ const batch = Array.isArray(req.body);++ const messages = batch ? req.body : [req.body];++ if (!messages.length) return res.json(rpcError(null, -32600, 'Invalid Request'));++ const responses = [];++ for (const msg of messages) {++ if (!msg || typeof msg !== 'object' || Array.isArray(msg) || msg.jsonrpc !== '2.0' || typeof msg.method !== 'string' || (Object.hasOwn(msg, 'id') && !(msg.id === null || typeof msg.id === 'string' || Number.isFinite(msg.id)))) { responses.push(rpcError(null, -32600, 'Invalid Request')); continue; }++ if (!Object.hasOwn(msg, 'id')) continue;++ const id = msg.id, params = msg.params && typeof msg.params === 'object' && !Array.isArray(msg.params) ? msg.params : {};++ if (msg.method === 'initialize') responses.push(rpcResult(id, { protocolVersion: VERSIONS.has(params.protocolVersion) ? params.protocolVersion : VERSION, capabilities: { tools: { listChanged: false } }, serverInfo: { name: 'openvibe-vision', version: require('../../package.json').version } }));++ else if (msg.method === 'ping') responses.push(rpcResult(id, {}));++ else if (msg.method === 'tools/list') responses.push(rpcResult(id, { tools: TOOLS }));++ else if (msg.method === 'tools/call') {++ const name = params.name;++ if (!Object.hasOwn(ACCESS, name)) { responses.push(rpcError(id, -32602, 'Unknown tool')); continue; }++ const args = params.arguments && typeof params.arguments === 'object' && !Array.isArray(params.arguments) ? params.arguments : {};++ const denied = access(req, name, ctx, args);++ if (denied) {++ if (req.principal.kind === 'anonymous') challenge(res, name === 'vision.get_job' && String(args.id || '').startsWith('vuc_') ? 'vision.ui.check' : ACCESS[name][0]);++ responses.push(toolError(id, denied)); continue;++ }++ try {++ const value = await invoke(name, args, req, ctx);++ if (EVENT_PATH[name]) {++ const eventReq = { principal: req.principal, ov: req.ov, path: EVENT_PATH[name], body: args, visionTask: name === 'vision.translate_sign' ? 'translate' : undefined };++ await ctx.eventPublisher.result(eventReq, value);++ await ctx.eventPublisher.alert(eventReq, value);++ }++ responses.push(toolResult(id, value));++ }++ catch (err) { responses.push(toolError(id, err.detail || err.message || 'Tool failed')); }++ } else responses.push(rpcError(id, -32601, 'Method not found'));++ }++ if (!responses.length) return res.status(202).end();++ return res.status(res.getHeader('WWW-Authenticate') ? 401 : 200).json(batch ? responses : responses[0]);++ } catch (err) { next(err); }++ });++ r.all('/mcp', (_req, res) => res.set('Allow', 'POST').status(405).json(rpcError(null, -32600, 'Method Not Allowed')));++ r.use((err, _req, res, next) => {++ if (err && err.type === 'entity.parse.failed') return res.status(400).json(rpcError(null, -32700, 'Parse error'));++ if (err && err.type === 'entity.too.large') return res.status(413).json(rpcError(null, -32600, 'Request body too large'));++ next(err);++ });++ return r;++}++++module.exports = { mcpRouter, TOOLS };@@@ -7,10 +7,9 @@* a person: their Network user token as a Bearer, or this site's session cookie (vision_at).* { kind: 'service', sub: 'svc:design', cap: [...], project, claims }* a Network service, app or agent token for audience openvibe.vision. Each machine route names exactly one-- * capability (vision.ui.check for Design's Arena, vision.face.cast.match for Live's channel roster, vision.ask,-- * vision.ocr, vision.compare, and the six network tools) and checks the token's grant list with-- * openvibe-contracts capabilities.grants. Unknown capability ids are accepted here until the main session-- * registers the manifests; the main session adds the grants.++ * capability (vision.ui.check for Design's Arena, vision.face.cast.match for Live's channel roster,++ * vision.image.ask, vision.text.read, vision.model.compare, and the network tools) and checks the token's++ * grant list with openvibe-contracts capabilities.grants.* { kind: 'anonymous' }** A request that presents a token is judged on that token alone: a bad one is refused, never downgraded to the@@@ -22,10 +21,14 @@ const { serviceAuth, capabilities, http, ids } = require('openvibe-contracts');const PRINCIPAL_SUB = /^(svc|app|mod|agent):/;const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;--/** Vision's machine capabilities (the main session registers them; Vision accepts the grant directly). */++/** Vision's registered capabilities (OpenVibe.Contracts manifests/services/vision.json). */const CAPABILITIES = Object.freeze([-- 'vision.ask', 'vision.ocr', 'vision.compare', 'vision.ui.check', 'vision.face.cast.match',-- 'vision.alt', 'vision.find', 'vision.receipt', 'vision.product', 'vision.watch', 'vision.moments',++ 'vision.ui.check', 'vision.image.ask', 'vision.text.read', 'vision.model.compare',++ 'vision.alt.write', 'vision.media.find', 'vision.receipt.read', 'vision.product.match',++ 'vision.camera.watch', 'vision.video.moments', 'vision.face.detect', 'vision.face.blur',++ 'vision.face.consent.manage', 'vision.face.verify.self', 'vision.face.find.self',++ 'vision.face.library', 'vision.face.cast.match', 'vision.face.cast.read',++ 'vision.models.read', 'vision.benchmarks.read',]);function decodePayload(token) {@@@ -241,16 +241,16 @@ function developers({ ai = false, models = [] } = {}) {<header class="vi-page-h"><p class="vi-kicker">Developers</p><h1>The Vision API</h1><p class="vi-lede">JSON under <code>/api/v1</code>, with <code>application/problem+json</code> errors. A person is the <code>vision_at</code> session cookie or a Network user token as a Bearer. A service token needs the capability on the route; Design's Arena calls the UI check with <code>vision.ui.check</code>.</p></header><section><h2>Ask, read, compare</h2><dl class="vi-eps">--${ep('POST', '/api/v1/ask (alias /api/v1/describe)', 'One image and a question: <code>{ task, question, mode, image_url }</code>, a multipart <code>image</code>, or an <code>image</code> data URL. Local OCR answers a read question first, otherwise OpenVibe.AI. Capability: <code>vision.ask</code>.')}--${ep('POST', '/api/v1/ocr', 'The local Tesseract reader. <code>fallback=1</code> asks AI only when no text is found and the caller is signed in or holds <code>vision.ocr</code>.')}--${ep('POST', '/api/v1/compare', 'The same image through local OCR and one to three AI hints (<code>models</code>). Returns each engine\'s answer, provider, model, cost and latency. Capability: <code>vision.compare</code>.')}++${ep('POST', '/api/v1/ask (alias /api/v1/describe)', 'One image and a question: <code>{ task, question, mode, image_url }</code>, a multipart <code>image</code>, or an <code>image</code> data URL. Local OCR answers a read question first, otherwise OpenVibe.AI. Capability: <code>vision.image.ask</code>.')}++${ep('POST', '/api/v1/ocr', 'The local Tesseract reader. <code>fallback=1</code> asks AI only when no text is found and the caller is signed in or holds <code>vision.text.read</code>.')}++${ep('POST', '/api/v1/compare', 'The same image through local OCR and one to three AI hints (<code>models</code>). Returns each engine\'s answer, provider, model, cost and latency. Capability: <code>vision.model.compare</code>.')}${ep('POST', '/api/v1/ui-check', 'HTML, URL or screenshot with <code>viewport</code>. Returns <code>findings</code>: overflow, cut_off, contrast, overlap and layering. Capability: <code>vision.ui.check</code>.')}--${ep('POST', '/api/v1/alt', 'Pics, Media and Chat send an image for short editable alt text. Capability: <code>vision.alt</code>.')}--${ep('POST', '/api/v1/index; GET or POST /api/v1/find', 'Media and Search index an owner image and find it by words or image. Requires the owner user token. Capability: <code>vision.find</code>.')}--${ep('POST', '/api/v1/receipt', 'Food sends a receipt image; Tesseract reads store, items and total first. Capability: <code>vision.receipt</code>.')}--${ep('POST', '/api/v1/product', 'Deals sends a product image for description, CLIP vector and configured offer candidates. Capability: <code>vision.product</code>.')}--${ep('POST', '/api/v1/watch', 'Cam sends an owner-provided still and watched classes. Requires the owner user token. Capability: <code>vision.watch</code>.')}--${ep('POST', '/api/v1/moments', 'Live or Media sends timestamped frames for scenes and candidate moments. Requires the owner user token. Capability: <code>vision.moments</code>.')}++${ep('POST', '/api/v1/alt', 'Pics, Media and Chat send an image for short editable alt text. Capability: <code>vision.alt.write</code>.')}++${ep('POST', '/api/v1/index; GET or POST /api/v1/find', 'Media and Search index an owner image and find it by words or image. Requires the owner user token. Capability: <code>vision.media.find</code>.')}++${ep('POST', '/api/v1/receipt', 'Food sends a receipt image; Tesseract reads store, items and total first. Capability: <code>vision.receipt.read</code>.')}++${ep('POST', '/api/v1/product', 'Deals sends a product image for description, CLIP vector and configured offer candidates. Capability: <code>vision.product.match</code>.')}++${ep('POST', '/api/v1/watch', 'Cam sends an owner-provided still and watched classes. Requires the owner user token. Capability: <code>vision.camera.watch</code>.')}++${ep('POST', '/api/v1/moments', 'Live or Media sends timestamped frames for scenes and candidate moments. Requires the owner user token. Capability: <code>vision.video.moments</code>.')}${ep('GET', '/api/v1/answers/:id', 'One saved answer, for its owner or a service holding the route capability.')}${ep('GET', '/api/v1/ui-checks/:id', 'One saved UI check, for its owner or a service holding <code>vision.ui.check</code>.')}${ep('GET', '/api/v1/models', 'The AI route\'s offered provider:model hints and the local engine. No subject data.')}@@@ -272,7 +272,7 @@ ${ep('POST', '/api/v1/faces/library/correct', 'A person\'s own group correction:</dl><p class="vi-fine"><a href="/people">Manage choices</a> · <a href="/people/privacy">Retention and privacy</a></p></section><section id="vi-capabilities"><h2>Capabilities and grants</h2>--<p class="vi-fine">Vision needs no new AI capability: outgoing calls are <code>ai.run.create</code> to OpenVibe.AI, audience <code>openvibe.ai</code>, namespace <code>ai.*</code>. The capabilities Vision offers are <code>vision.ask</code>, <code>vision.ocr</code>, <code>vision.compare</code>, <code>vision.ui.check</code> and <code>vision.face.cast.match</code>; the main session registers the manifests and grants them to Design (the Arena), Live (the cast roster) and any other caller.</p>++<p class="vi-fine">Vision calls OpenVibe.AI with <code>ai.run.create</code>, audience <code>openvibe.ai</code> and namespace <code>ai.*</code>. The twenty Vision capability manifests cover image questions, text, comparison, UI checks, network tools and consent-scoped face operations. A service token needs the capability named on its route; a person manages only their own face consent. Vision also offers eleven tools on <code>/mcp</code>, with Network token discovery at <code>/.well-known/oauth-protected-resource</code>.</p><pre class="vi-code">POST /api/v1/ui-checkAuthorization: Bearer <Network service token for openvibe.vision>Content-Type: application/json@@@ -103,7 +103,7 @@ function createFaces({ s, config, model }) {async function template(subject, feature, scope = '') {requireKey(); await requireConsent(subject, feature, scope);const row = await db.maybe('SELECT ciphertext FROM face_templates WHERE subject=$1 AND feature=$2 AND scope=$3', [subject, feature, scope]);-- if (!row) deny('enrolment_required', `Enrol in ${feature} first.`);++ if (!row) deny('consent_required', `Enable ${feature} consent first.`);return open(key, row.ciphertext);}async function touch(subject, feature, scope = '') {@@@ -101,14 +101,14 @@ const { boot, check, done } = require('./helpers/boot');const good = t.network.serviceToken({ sub: 'svc:design', cap: ['vision.ui.check'] });const ok = await t.get('/api/v1/ui-check', { bearer: good, json: { html: '<div style="width:900px">x</div>', viewport: { w: 390, h: 844 } } });assert.strictEqual(ok.status, 201, ok.text);-- const bad = t.network.serviceToken({ sub: 'svc:design', cap: ['vision.ask'] });++ const bad = t.network.serviceToken({ sub: 'svc:design', cap: ['vision.image.ask'] });const no = await t.get('/api/v1/ui-check', { bearer: bad, json: { html: '<p>x</p>' } });assert.strictEqual(no.status, 403);assert.strictEqual(no.json().code, 'capability.denied');});await check('a service token with vision.ask can ask; the answer has no subject', async () => {-- const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.ask'] });++ const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.image.ask'] });const r = await t.get('/api/v1/ask', { bearer: tok, multipart: { fields: { task: 'describe', question: 'What is this?' }, image: png } });assert.strictEqual(r.status, 201, r.text);assert.strictEqual(r.json().answer.answer.engine, 'ai');@@@ -122,7 +122,7 @@ const { boot, check, done } = require('./helpers/boot');const id = r.json().answer.id;assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { as: user })).status, 200);assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { as: other })).status, 404);-- const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.ask'] });++ const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.image.ask'] });assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { bearer: tok })).status, 200);});nnew file mode 100644@@@ -0,0 +1,76 @@++'use strict';++++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const { createPublisher } = require('../server/events/publisher');++++(async () => {++ const sent = [];++ const config = {++ drill: false, networkInternalUrl: 'https://network.example',++ oauth: { clientId: 'vision', clientSecret: 'test-secret' },++ events: { url: 'https://events.example', scope: 'events.event.publish' },++ };++ const fetchImpl = async (url, options) => {++ if (String(url).endsWith('/oauth/token')) return Response.json({ access_token: 'test-token', expires_in: 300 });++ if (String(url).endsWith('/api/v1/events')) {++ sent.push(JSON.parse(options.body));++ return Response.json({ event_id: sent.at(-1).event_id, cursor: 'cursor' });++ }++ throw new Error('unexpected URL');++ };++ const publisher = createPublisher({ config, fetchImpl, log: { warn() { throw new Error('event publish failed'); } } });++ const principal = { kind: 'user', subject: 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A4' };++ const req = (path, body) => ({ path, body, principal, ov: {} });++++ await check('ready event carries provenance but no answer text or image', async () => {++ await publisher.result(req('/ask', { image: 'data:image/png;base64,AAAA' }), {++ answer: { id: 'vis_01J8Z3Q4R5S6T7V8W9X0Y1Z2A4', text: 'A red bicycle.', model: 'model', provider: 'ai', route: 'routed', cost_usd: 0.01, latency_ms: 120 },++ });++ assert.strictEqual(sent.length, 1);++ assert.strictEqual(sent[0].event_type, 'vision.result.ready');++ assert.deepStrictEqual(sent[0].subject, { type: 'user', id: principal.subject });++ assert.strictEqual(sent[0].payload.task, 'ask');++ assert.strictEqual(sent[0].payload.source, 'data');++ assert.ok(!JSON.stringify(sent[0]).includes('red bicycle'));++ assert.ok(!JSON.stringify(sent[0]).includes('AAAA'));++ });++++ await check('watch event names only the owner-granted source and detected classes', async () => {++ await publisher.alert(req('/watch', { source_ref: 'cam:front-door' }), { answer: { detections: [{ class: 'animal', confidence: 0.92 }] } });++ assert.strictEqual(sent[1].event_type, 'vision.alert');++ assert.strictEqual(sent[1].payload.source_ref, 'cam:front-door');++ assert.deepStrictEqual(sent[1].payload.classes, ['animal']);++ await publisher.alert(req('/watch', {}), { answer: { detections: [{ class: 'animal', confidence: 0.92 }] } });++ assert.strictEqual(sent.length, 2);++ });++++ await check('anonymous results create no subject event', async () => {++ await publisher.result({ ...req('/ocr', {}), principal: { kind: 'anonymous' } }, { answer: { text: 'x' } });++ assert.strictEqual(sent.length, 2);++ });++++ await check('successful API results call the publisher with the authenticated owner', async () => {++ const calls = [];++ const t = await boot({ eventPublisher: {++ result: async (request, out) => calls.push({ path: request.path, subject: request.principal.subject, id: out.check && out.check.id }),++ alert: async () => {},++ } });++ try {++ const user = t.network.addUser('event-owner');++ const r = await t.get('/api/v1/ui-check', { as: user, json: { html: '<p>hello</p>' } });++ assert.strictEqual(r.status, 201, r.text);++ assert.deepStrictEqual(calls, [{ path: '/ui-check', subject: user.subject, id: r.json().check.id }]);++ const mcp = await t.get('/mcp', { bearer: t.network.userToken(user), json: {++ jsonrpc: '2.0', id: 1, method: 'tools/call',++ params: { name: 'vision.check_ui', arguments: { html: '<p>hello</p>' } },++ } });++ assert.strictEqual(mcp.status, 200, mcp.text);++ assert.strictEqual(mcp.json().result.isError, false, mcp.text);++ assert.strictEqual(calls.length, 2);++ assert.strictEqual(calls[1].path, '/ui-check');++ assert.strictEqual(calls[1].subject, user.subject);++ } finally { await t.close(); }++ });++ done();++})();@@@ -135,6 +135,10 @@ function liveToken(t, channel) {assert.strictEqual(row.signature, 'Test Person');assert.strictEqual(row.revoked_at, null);assert.ok(Date.parse(row.granted_at));++ const waitingForEnrolment = await t.get('/api/v1/faces/verify', { as: ana, multipart: { image: png } });++ assert.strictEqual(waitingForEnrolment.status, 403);++ assert.strictEqual(waitingForEnrolment.json().code, 'faces.consent_required');++ assert.strictEqual(waitingForEnrolment.json().detail, 'Enable verify consent first.');assert.strictEqual((await consent(ana, 'find')).status, 200);assert.strictEqual((await consent(ana, 'cast', { channel: 'live:one', adult: true })).status, 200);@@@ -46,7 +46,7 @@ async function boot(opts = {}) {const { createStore } = require('../../server/db');const testdb = await require('./db').testDb();const store = createStore(testdb.db, { now: opts.now });-- const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, accountSend: opts.accountSend, ocr: opts.ocr, ai: opts.aiClient, facesModel: opts.facesModel, clip: opts.clip, detector: opts.detector });++ const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, accountSend: opts.accountSend, eventPublisher: opts.eventPublisher, ocr: opts.ocr, ai: opts.aiClient, facesModel: opts.facesModel, clip: opts.clip, detector: opts.detector });await built.ctx.keys.ensure();const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });const base = `http://127.0.0.1:${server.address().port}`;nnew file mode 100644@@@ -0,0 +1,72 @@++'use strict';++++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++++const rpc = (method, params = {}, id = 1) => ({ jsonrpc: '2.0', id, method, params });++++(async () => {++ const t = await boot();++ const owner = t.network.addUser('mcp-owner');++ try {++ await check('MCP advertises RFC 9728 metadata and its supported tools', async () => {++ const meta = await t.get('/.well-known/oauth-protected-resource');++ assert.strictEqual(meta.status, 200);++ assert.strictEqual(meta.json().resource, 'https://openvibe.vision/mcp');++ assert.deepStrictEqual(meta.json().authorization_servers, ['http://' + new URL(t.network.url).host]);++ const init = await t.get('/mcp', { json: rpc('initialize', { protocolVersion: '2025-06-18' }) });++ assert.strictEqual(init.status, 200, init.text);++ assert.strictEqual(init.json().result.serverInfo.name, 'openvibe-vision');++ const list = await t.get('/mcp', { json: rpc('tools/list') });++ assert.strictEqual(list.json().result.tools.length, 11);++ assert.ok(list.json().result.tools.some((x) => x.name === 'vision.blur_faces'));++ assert.ok(!list.json().result.tools.some((x) => /enrol|verify|cast/.test(x.name)));++ });++++ await check('private tools challenge anonymous callers with resource metadata', async () => {++ const r = await t.get('/mcp', { json: rpc('tools/call', { name: 'vision.search_my_media', arguments: { query: 'desk' } }) });++ assert.strictEqual(r.status, 401, r.text);++ assert.match(r.headers.get('www-authenticate'), /resource_metadata="https:\/\/openvibe.vision\/\.well-known\/oauth-protected-resource"/);++ assert.strictEqual(r.json().result.isError, true);++ });++++ await check('a service token needs the exact tool capability', async () => {++ const denied = await t.get('/mcp', { bearer: t.network.serviceToken({ cap: ['vision.ui.check'] }), json: rpc('tools/call', { name: 'vision.compare_models', arguments: {} }) });++ assert.strictEqual(denied.status, 200);++ assert.strictEqual(denied.json().result.isError, true);++ assert.match(denied.json().result.content[0].text, /vision.model.compare/);++ });++++ await check('private image tools refuse a remote source before fetching it', async () => {++ const r = await t.get('/mcp', { bearer: t.network.userToken(owner), json: rpc('tools/call', { name: 'vision.find_similar', arguments: { image_url: t.images.url + '/image.png' } }) });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.json().result.isError, true);++ assert.match(r.json().result.content[0].text, /image data URL only/);++ });++++ await check('an owner may read their own UI check, while another owner cannot', async () => {++ const saved = await t.get('/api/v1/ui-check', { as: owner, json: { html: '<div>hello</div>' } });++ assert.strictEqual(saved.status, 201, saved.text);++ const id = saved.json().check.id;++ const own = await t.get('/mcp', { bearer: t.network.userToken(owner), json: rpc('tools/call', { name: 'vision.get_job', arguments: { id } }) });++ assert.strictEqual(own.json().result.isError, false, own.text);++ assert.strictEqual(own.json().result.structuredContent.check.id, id);++ const other = t.network.addUser('mcp-other');++ const refused = await t.get('/mcp', { bearer: t.network.userToken(other), json: rpc('tools/call', { name: 'vision.get_job', arguments: { id } }) });++ assert.strictEqual(refused.json().result.isError, true);++ assert.strictEqual(refused.json().result.content[0].text, 'No such result.');++ });++++ await check('notifications, batches and parse errors follow JSON-RPC', async () => {++ const batch = await t.get('/mcp', { json: [rpc('ping', {}, 1), rpc('tools/list', {}, 2)] });++ assert.strictEqual(batch.status, 200);++ assert.strictEqual(batch.json().length, 2);++ const notification = await t.get('/mcp', { json: { jsonrpc: '2.0', method: 'notifications/initialized' } });++ assert.strictEqual(notification.status, 202);++ const malformed = await t.get('/mcp', { body: '{', headers: { 'content-type': 'application/json' } });++ assert.strictEqual(malformed.status, 400);++ assert.strictEqual(malformed.json().error.code, -32700);++ });++ } finally { await t.close(); }++ done();++})();@@@ -23,7 +23,7 @@ const { parseReceipt } = require('../server/vision/network-tools');const theirs = await t.get('/api/v1/find?q=bike', { as: other });assert.strictEqual(theirs.status, 200);assert.deepStrictEqual(theirs.json().answer.results, []);-- const svc = t.network.serviceToken({ sub: 'svc:media', cap: ['vision.find'] });++ const svc = t.network.serviceToken({ sub: 'svc:media', cap: ['vision.media.find'] });assert.strictEqual((await t.get('/api/v1/find?q=bike', { bearer: svc })).status, 403);const byImage = await t.get('/api/v1/find', { as: owner, json: { image } });assert.strictEqual(byImage.json().answer.results[0].item_id, 'photo-1');@@@ -54,7 +54,7 @@ const { parseReceipt } = require('../server/vision/network-tools');assert.strictEqual(m.status, 201, m.text);assert.ok(m.json().answer.receipt);assert.strictEqual(m.json().answer.candidate_moments[0].timestamp_ms, 1000);-- const svc = t.network.serviceToken({ sub: 'svc:cam', cap: ['vision.watch'] });++ const svc = t.network.serviceToken({ sub: 'svc:cam', cap: ['vision.camera.watch'] });assert.strictEqual((await t.get('/api/v1/watch', { bearer: svc, json: { image, classes: ['cat'] } })).status, 403);});await check('product has embedding and receipt even if offer APIs are not configured', async () => {
@@ -2,6 +2,15 @@
@@ -2,6 +2,15 @@
What changed in OpenVibe.Vision, newest first.
What changed in OpenVibe.Vision, newest first.
+## Unreleased — VIS-5
+
+- Added a Streamable HTTP MCP endpoint with eleven tools, protected-resource metadata, token challenges and the
+ same permission boundaries as the JSON API. Face identity and consent mutation remain person-only API flows.
+- Aligned service-token checks with the twenty capability IDs in Contracts 0.240.0. Anonymous local and privacy
+ tools remain open. Verification gives the same refusal before and after consent until enrollment is complete.
+- Publishes `vision.result.ready` and `vision.alert` through OpenVibe.Events when Events is configured. Event
+ payloads contain provenance and class names, never image bytes, answer text or face templates.
+
## 0.1.0 — 2026-10-28
## 0.1.0 — 2026-10-28
The first version: see with the best model for the job.
The first version: see with the best model for the job.
@@ -92,26 +92,43 @@ General answer tables store no image bytes, token or provider key. The opt-in fa
@@ -92,26 +92,43 @@ General answer tables store no image bytes, token or provider key. The opt-in fa
## Capabilities and grants
## Capabilities and grants
-The service offers eleven capabilities. The main session registers their manifests and grants them; v1 checks the
-grant list directly, so a token that already carries one works.
+Contracts 0.240.0 registers twenty Vision capabilities. The service checks each machine token's grant on its route;
+private media and face identity operations also require the verified owner or the person's purpose-scoped consent.
| Capability | Routes | Typical caller |
| Capability | Routes | Typical caller |
|---|---|---|
|---|---|---|
-| `vision.ask` | `POST /api/v1/ask`, `GET /api/v1/answers/:id` | a first-party service or a person |
-| `vision.ocr` | `POST /api/v1/ocr` (the AI fallback also needs a person or this grant) | a first-party service or a person |
-| `vision.compare` | `POST /api/v1/compare` | a first-party service or a person |
+| `vision.image.ask` | `POST /api/v1/ask`, `GET /api/v1/answers/:id` | a first-party service or a person |
+| `vision.text.read` | `POST /api/v1/ocr` (the AI fallback also needs a person or this grant) | a first-party service or a person |
+| `vision.model.compare` | `POST /api/v1/compare` | a first-party service or a person |
| `vision.ui.check` | `POST /api/v1/ui-check`, `GET /api/v1/ui-checks/:id` | **OpenVibe.Design's Arena**, or a person |
| `vision.ui.check` | `POST /api/v1/ui-check`, `GET /api/v1/ui-checks/:id` | **OpenVibe.Design's Arena**, or a person |
| `vision.face.cast.match` | `POST /api/v1/faces/cast/match` (a signed `channel_id` claim must match the request) | **OpenVibe.Live**, or a person inspecting their own tags |
| `vision.face.cast.match` | `POST /api/v1/faces/cast/match` (a signed `channel_id` claim must match the request) | **OpenVibe.Live**, or a person inspecting their own tags |
-| `vision.alt` | `POST /api/v1/alt` | Pics, Media and Chat |
-| `vision.find` | `POST /api/v1/index`, `GET`/`POST /api/v1/find` (owner user token) | Media and Search |
-| `vision.receipt` | `POST /api/v1/receipt` | Food |
-| `vision.product` | `POST /api/v1/product` | Deals |
-| `vision.watch` | `POST /api/v1/watch` (owner user token) | Cam |
-| `vision.moments` | `POST /api/v1/moments` (owner user token) | Live and Media |
+| `vision.alt.write` | `POST /api/v1/alt` | Pics, Media and Chat |
+| `vision.media.find` | `POST /api/v1/index`, `GET`/`POST /api/v1/find` (owner user token) | Media and Search |
+| `vision.receipt.read` | `POST /api/v1/receipt` | Food |
+| `vision.product.match` | `POST /api/v1/product` | Deals |
+| `vision.camera.watch` | `POST /api/v1/watch` (owner user token) | Cam |
+| `vision.video.moments` | `POST /api/v1/moments` (owner user token) | Live and Media |
+| `vision.face.detect`, `vision.face.blur` | `POST /api/v1/faces/detect`, `/blur` | Public privacy tools; a service token needs the matching grant |
+| `vision.face.consent.manage`, `vision.face.verify.self`, `vision.face.find.self`, `vision.face.library`, `vision.face.cast.read` | `/api/v1/faces/*` | The person themselves, with separate signed consent for each matching purpose |
+| `vision.models.read`, `vision.benchmarks.read` | `GET /api/v1/models`, `/benchmarks` | Public reads; a service token needs the matching grant |
Outgoing: **`ai.run.create`** to OpenVibe.AI (audience `openvibe.ai`, namespace `ai.*`), minted by Network for the
Outgoing: **`ai.run.create`** to OpenVibe.AI (audience `openvibe.ai`, namespace `ai.*`), minted by Network for the
client `vision`. No new AI capability, no provider key, no direct paid provider.
client `vision`. No new AI capability, no provider key, no direct paid provider.
+## MCP and events
+
+`POST /mcp` serves Streamable HTTP JSON-RPC 2.0 (`initialize`, `ping`, `tools/list`, `tools/call`).
+`/.well-known/oauth-protected-resource` advertises its Network authorization server; a private tool without a
+token responds with a `WWW-Authenticate` challenge pointing to that metadata. The tools are `vision.ask_image`,
+`vision.read_text`, `vision.translate_sign`, `vision.find_similar`, `vision.search_my_media`,
+`vision.describe_clip`, `vision.detect_objects`, `vision.blur_faces`, `vision.check_ui`,
+`vision.compare_models` and `vision.get_job`. Face enrolment, verification, cast matching and consent mutation
+are available only through their person or Live-specific API routes, never through MCP.
+
+When configured with Events and a Network client secret, successful owner results publish `vision.result.ready`
+with provenance, cost and retention, without image or answer text. A watch result with `source_ref` and detections
+publishes `vision.alert` with classes and confidence. Publishing failures are logged without exposing payloads.
+
## API
## API
JSON under `/api/v1`; errors are `application/problem+json` with a `code`. An image is a multipart `image` field, a
JSON under `/api/v1`; errors are `application/problem+json` with a `code`. An image is a multipart `image` field, a
@@ -125,7 +142,7 @@ must hold the route's capability (403 `capability.denied`).
@@ -125,7 +142,7 @@ must hold the route's capability (403 `capability.denied`).
| `POST /api/v1/ocr` | multipart/data URL/URL; `{ fallback?, lang?, model? }`; returns the OCR row with `text`, `answer.confidence`, `answer.words`, cost 0. |
| `POST /api/v1/ocr` | multipart/data URL/URL; `{ fallback?, lang?, model? }`; returns the OCR row with `text`, `answer.confidence`, `answer.words`, cost 0. |
| `POST /api/v1/compare` | `{ task?, question?, models?: string[] }` plus an image; returns `{ answer, compare }`, one row per engine. |
| `POST /api/v1/compare` | `{ task?, question?, models?: string[] }` plus an image; returns `{ answer, compare }`, one row per engine. |
| `POST /api/v1/ui-check` | `{ html }`, `{ url }`, or a screenshot plus `{ viewport: { w, h } }`; returns `{ check }` with `findings`, `counts`, `severity`, `warnings`. |
| `POST /api/v1/ui-check` | `{ html }`, `{ url }`, or a screenshot plus `{ viewport: { w, h } }`; returns `{ check }` with `findings`, `counts`, `severity`, `warnings`. |
-| `GET /api/v1/answers/:id` | one saved answer: its owner, or a service holding `vision.ask`. |
+| `GET /api/v1/answers/:id` | one saved answer: its owner, or a service holding `vision.image.ask`. |
| `GET /api/v1/ui-checks/:id` | one saved UI check: its owner, or a service holding `vision.ui.check`. |
| `GET /api/v1/ui-checks/:id` | one saved UI check: its owner, or a service holding `vision.ui.check`. |
| `GET /api/v1/models` | AI's offered `provider:model` hints and the local engine. No subject data. |
| `GET /api/v1/models` | AI's offered `provider:model` hints and the local engine. No subject data. |
| `GET /api/v1/benchmarks?task=&limit=` | aggregate compare samples: task, engine, provider, model, samples, mean latency, mean cost. No subject or answer text. |
| `GET /api/v1/benchmarks?task=&limit=` | aggregate compare samples: task, engine, provider, model, samples, mean latency, mean cost. No subject or answer text. |
@@ -156,7 +173,7 @@ Answer (abridged):
@@ -156,7 +173,7 @@ Answer (abridged):
Validation refusals: `vision.invalid`, `vision.invalid_task`, `vision.invalid_mode`, `vision.question_required`,
Validation refusals: `vision.invalid`, `vision.invalid_task`, `vision.invalid_mode`, `vision.question_required`,
`vision.image_required`, `vision.image_too_large`, `vision.image_private_url`, `vision.image_unsupported`,
`vision.image_required`, `vision.image_too_large`, `vision.image_private_url`, `vision.image_unsupported`,
-`vision.ocr_unavailable`, `vision.ai_unavailable`, `vision.not_found`, `vision.rate_limited`, and the face codes
+`vision.text.read_unavailable`, `vision.ai_unavailable`, `vision.not_found`, `vision.rate_limited`, and the face codes
`faces.consent_required`, `faces.signature_required`, `faces.adult_required`, `faces.region_disabled`,
`faces.consent_required`, `faces.signature_required`, `faces.adult_required`, `faces.region_disabled`,
`faces.enrolment_required`, `faces.one_face_required`, `faces.model_unavailable`, `faces.key_unavailable`,
`faces.enrolment_required`, `faces.one_face_required`, `faces.model_unavailable`, `faces.key_unavailable`,
`faces.live_grant_required`.
`faces.live_grant_required`.
@@ -229,12 +246,12 @@ All six tools are JSON APIs. Successful `answer` objects carry a `receipt` with
@@ -229,12 +246,12 @@ All six tools are JSON APIs. Successful `answer` objects carry a `receipt` with
| Room call | Capability | Notes |
| Room call | Capability | Notes |
|---|---|---|
|---|---|---|
-| Pics, Media, Chat → `POST /api/v1/alt` | `vision.alt` | Routed VLM through OpenVibe.AI; short editable `alt_text`; noLooks in the prompt and output. |
-| Media, Search → `POST /api/v1/index`, `GET /api/v1/find?q=`, `POST /api/v1/find` | `vision.find` | Owner's verified user token; index `{item_id,image,result_url?}`; search by text or image. Owner is in every index read predicate. |
-| Food → `POST /api/v1/receipt` | `vision.receipt` | Tesseract first; `fallback=true` permits routed VLM if local extraction is incomplete. |
-| Deals → `POST /api/v1/product` | `vision.product` | CLIP vector and routed product description; `allow_rerank=true` permits another routed VLM call to rerank candidate offers. |
-| Cam → `POST /api/v1/watch` | `vision.watch` | Owner's verified user token; upload still and `classes` array; optional `previous_image` data URL for scene change. |
-| Live, Media → `POST /api/v1/moments` | `vision.moments` | Owner's verified user token; `{frames:[{timestamp_ms,image|image_url}],prompts?}`. Scene histogram and CLIP prompt matching. |
+| Pics, Media, Chat → `POST /api/v1/alt` | `vision.alt.write` | Routed VLM through OpenVibe.AI; short editable `alt_text`; noLooks in the prompt and output. |
+| Media, Search → `POST /api/v1/index`, `GET /api/v1/find?q=`, `POST /api/v1/find` | `vision.media.find` | Owner's verified user token; index `{item_id,image,result_url?}`; search by text or image. Owner is in every index read predicate. |
+| Food → `POST /api/v1/receipt` | `vision.receipt.read` | Tesseract first; `fallback=true` permits routed VLM if local extraction is incomplete. |
+| Deals → `POST /api/v1/product` | `vision.product.match` | CLIP vector and routed product description; `allow_rerank=true` permits another routed VLM call to rerank candidate offers. |
+| Cam → `POST /api/v1/watch` | `vision.camera.watch` | Owner's verified user token; upload still and `classes` array; optional `previous_image` data URL for scene change. |
+| Live, Media → `POST /api/v1/moments` | `vision.video.moments` | Owner's verified user token; `{frames:[{timestamp_ms,image|image_url}],prompts?}`. Scene histogram and CLIP prompt matching. |
For private index, watch and moments, a service capability alone does not assert a source owner's grant. The room should call with the owner's delegated Network user token. Private index and watch reject `image_url`; the owner must supply bytes or a data URL. Moments accepts public frame URLs under Vision's existing guarded image fetch, or frame data URLs. The room that opens a returned `result_url` must check its own source grant again.
For private index, watch and moments, a service capability alone does not assert a source owner's grant. The room should call with the owner's delegated Network user token. Private index and watch reject `image_url`; the owner must supply bytes or a data URL. Moments accepts public frame URLs under Vision's existing guarded image fetch, or frame data URLs. The room that opens a returned `result_url` must check its own source grant again.
@@ -8,26 +8,35 @@
@@ -8,26 +8,35 @@
"runtime": "node 22, express 4, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, Tesseract 5, sharp, onnxruntime-node (YuNet/SFace), Transformers.js (CLIP/DETR)",
"runtime": "node 22, express 4, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, Tesseract 5, sharp, onnxruntime-node (YuNet/SFace), Transformers.js (CLIP/DETR)",
"port": "5130",
"port": "5130",
"domain": "openvibe.vision",
"domain": "openvibe.vision",
- "contracts": "openvibe-contracts v0.235.0",
- "sdk": "openvibe-sdk v0.70.0",
+ "contracts": "openvibe-contracts v0.240.0",
+ "sdk": "openvibe-sdk v0.72.0",
"shared": "openvibe-shared v4.41.0",
"shared": "openvibe-shared v4.41.0",
"packages": [
"packages": [
- "openvibe-contracts v0.235.0",
- "openvibe-sdk v0.70.0",
+ "openvibe-contracts v0.240.0",
+ "openvibe-sdk v0.72.0",
"openvibe-shared v4.41.0"
"openvibe-shared v4.41.0"
],
],
"capabilities": [
"capabilities": [
- "vision.ask",
- "vision.ocr",
- "vision.compare",
+ "vision.image.ask",
+ "vision.text.read",
+ "vision.model.compare",
"vision.ui.check",
"vision.ui.check",
"vision.face.cast.match",
"vision.face.cast.match",
- "vision.alt",
- "vision.find",
- "vision.receipt",
- "vision.product",
- "vision.watch",
- "vision.moments"
+ "vision.alt.write",
+ "vision.media.find",
+ "vision.receipt.read",
+ "vision.product.match",
+ "vision.camera.watch",
+ "vision.video.moments",
+ "vision.face.detect",
+ "vision.face.blur",
+ "vision.face.consent.manage",
+ "vision.face.verify.self",
+ "vision.face.find.self",
+ "vision.face.library",
+ "vision.face.cast.read",
+ "vision.models.read",
+ "vision.benchmarks.read"
],
],
"outboundCapabilities": [
"outboundCapabilities": [
"ai.run.create"
"ai.run.create"
@@ -46,6 +55,8 @@
@@ -46,6 +55,8 @@
"Consent-scoped SFace matching: verify (1:1), self search in own Vision uploads, adult cast rosters and region-gated private library grouping",
"Consent-scoped SFace matching: verify (1:1), self search in own Vision uploads, adult cast rosters and region-gated private library grouping",
"Encrypted face templates, opt-in uploads and vectors (AES-256-GCM); immediate template/tag erasure on opt-out and a daily three-year retention job",
"Encrypted face templates, opt-in uploads and vectors (AES-256-GCM); immediate template/tag erasure on opt-out and a daily three-year retention job",
"Network tools: routed alt text, owner-scoped CLIP index/find, Tesseract-first receipts, product vectors, DETR watch and moments, each with a receipt",
"Network tools: routed alt text, owner-scoped CLIP index/find, Tesseract-first receipts, product vectors, DETR watch and moments, each with a receipt",
+ "MCP on /mcp with RFC 9728 protected-resource metadata, token challenges and route-equivalent capability checks",
+ "vision.result.ready and vision.alert published through OpenVibe.Events when configured",
"Answers, compare samples, private index and UI checks stored without image bytes; account export/deletion over every table including faces and the private index",
"Answers, compare samples, private index and UI checks stored without image bytes; account export/deletion over every table including faces and the private index",
"Network sign-in as OAuth client vision (PKCE S256, vision_at/vision_rt/vision_oauth), truthful readiness, loopback metrics",
"Network sign-in as OAuth client vision (PKCE S256, vision_at/vision_rt/vision_oauth), truthful readiness, loopback metrics",
"Server-rendered pages with stable vi- ids/classes, discovery artifacts, deploy unit and nginx vhost, CI"
"Server-rendered pages with stable vi- ids/classes, discovery artifacts, deploy unit and nginx vhost, CI"
@@ -18,8 +18,8 @@
@@ -18,8 +18,8 @@
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"multer": "^2.5.0",
"multer": "^2.5.0",
"onnxruntime-node": "^1.31.1",
"onnxruntime-node": "^1.31.1",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.240.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.72.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.41.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.41.0",
"pg": "^8.23.0",
"pg": "^8.23.0",
"sharp": "^0.33.5"
"sharp": "^0.33.5"
@@ -2625,9 +2625,9 @@
@@ -2625,9 +2625,9 @@
"license": "MIT"
"license": "MIT"
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.235.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",
- "integrity": "sha512-Rfmy2sDZOO+fegEPBweQvCKlamkYwOXGyr1M80VkYHXHl8gg7CnyDL7HrbGwuXG3LUcPqzhAMYeeB//jKMtzaw==",
+ "version": "0.240.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.240.0",
+ "integrity": "sha512-GiwXO9Y3YHhIuthBNzUrkZByIdOPK+ZjIklCLt1pYQRkPdyG59yGhi5ha9GMZ5d6z1AAkQ0QTwexGfJ23lgB0g==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -2641,9 +2641,9 @@
@@ -2641,9 +2641,9 @@
}
}
},
},
"node_modules/openvibe-sdk": {
"node_modules/openvibe-sdk": {
- "version": "0.70.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",
- "integrity": "sha512-iMNxlnRm0fmyj60nHiC/lKAaG3JAqiJgu42lShnkby4yF4dQJRG3G0trrqOoULZKBalrYHb/gjauRwNQYN/tog==",
+ "version": "0.72.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.72.0",
+ "integrity": "sha512-RaMMhd1n280MXeGd4c+EVTOALyoQcnhEO2UnbgMu+7lalxdXDkvivao+C6TraojrhPFdNGskbShcywvt2KVd7w==",
"license": "MIT",
"license": "MIT",
"bin": {
"bin": {
"openvibe-download": "bin/openvibe-download.js"
"openvibe-download": "bin/openvibe-download.js"
@@ -28,8 +28,8 @@
@@ -28,8 +28,8 @@
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"multer": "^2.5.0",
"multer": "^2.5.0",
"onnxruntime-node": "^1.31.1",
"onnxruntime-node": "^1.31.1",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.240.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.72.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.41.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.41.0",
"pg": "^8.23.0",
"pg": "^8.23.0",
"sharp": "^0.33.5"
"sharp": "^0.33.5"
@@ -29,6 +29,7 @@ const { createSso } = require('./auth/sso');
@@ -29,6 +29,7 @@ const { createSso } = require('./auth/sso');
const { createPrincipal } = require('./http/principal');
const { createPrincipal } = require('./http/principal');
const { createApi } = require('./http/api');
const { createApi } = require('./http/api');
const { createFaceApi } = require('./http/faces-api');
const { createFaceApi } = require('./http/faces-api');
+const { mcpRouter } = require('./http/mcp');
const { createPageRoutes } = require('./http/pages');
const { createPageRoutes } = require('./http/pages');
const { createServiceReadiness } = require('./observability');
const { createServiceReadiness } = require('./observability');
const { assetVersion, send } = require('./render/layout');
const { assetVersion, send } = require('./render/layout');
@@ -42,6 +43,7 @@ const { createVision } = require('./vision/service');
@@ -42,6 +43,7 @@ const { createVision } = require('./vision/service');
const { createFacesModel } = require('./vision/faces-model');
const { createFacesModel } = require('./vision/faces-model');
const { createFaces } = require('./vision/faces');
const { createFaces } = require('./vision/faces');
const { createNetworkSender } = require('openvibe-sdk/account-data');
const { createNetworkSender } = require('openvibe-sdk/account-data');
+const { createPublisher } = require('./events/publisher');
const PUBLIC_DIR = path.join(__dirname, '..', 'public');
const PUBLIC_DIR = path.join(__dirname, '..', 'public');
const VERSION = require('../package.json').version;
const VERSION = require('../package.json').version;
@@ -77,7 +79,8 @@ async function createApp(opts = {}) {
@@ -77,7 +79,8 @@ async function createApp(opts = {}) {
? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })
? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Vision cannot answer account events'); });
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Vision cannot answer account events'); });
- const ctx = { config, s, keys, sso, principal, store, ocr, ai, vision, faces, models, networkTools, accountData, accountSend, log, fetchImpl };
+ const eventPublisher = opts.eventPublisher || createPublisher({ config, fetchImpl, log });
+ const ctx = { config, s, keys, sso, principal, store, ocr, ai, vision, faces, models, networkTools, accountData, accountSend, eventPublisher, log, fetchImpl };
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
@@ -153,6 +156,7 @@ async function createApp(opts = {}) {
@@ -153,6 +156,7 @@ async function createApp(opts = {}) {
}));
}));
// ── API ─────────────────────────────────────────────────
// ── API ─────────────────────────────────────────────────
+ app.use(mcpRouter(ctx));
app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.VISION_API_RATE_LIMIT_PER_MIN) || 120, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));
app.use('/api/v1', rateLimit({ windowMs: 60_000, limit: Number(process.env.VISION_API_RATE_LIMIT_PER_MIN) || 120, standardHeaders: true, legacyHeaders: false, handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }) }));
app.use('/api/v1/faces', createFaceApi(ctx));
app.use('/api/v1/faces', createFaceApi(ctx));
app.use('/api/v1', createApi(ctx));
app.use('/api/v1', createApi(ctx));
new file mode 100644
new file mode 100644
@@ -0,0 +1,68 @@
@@ -0,0 +1,68 @@
+'use strict';
+
+const { createClient } = require('openvibe-sdk/core');
+const { createServiceTokenClient } = require('openvibe-sdk/auth');
+const { createEventsClient } = require('openvibe-sdk/events');
+
+const TASKS = Object.freeze({
+ '/ask': 'ask', '/describe': 'describe', '/ocr': 'ocr', '/compare': 'compare', '/alt': 'alt',
+ '/find': 'find', '/receipt': 'receipt', '/product': 'product', '/watch': 'watch',
+ '/moments': 'moments', '/ui-check': 'ui_check',
+});
+
+function createPublisher({ config, fetchImpl = globalThis.fetch, log = console } = {}) {
+ if (config.drill || !config.events.url || !config.oauth.clientSecret) return { result: async () => {}, alert: async () => {} };
+ const tokenProvider = createServiceTokenClient({
+ tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId,
+ clientSecret: config.oauth.clientSecret, audience: 'openvibe.events', scope: config.events.scope, fetch: fetchImpl,
+ });
+ const client = createClient({ baseUrls: { events: config.events.url }, tokenProvider, fetch: fetchImpl, retries: 1 });
+ const events = createEventsClient(client, { source: 'vision' });
+
+ function subject(req) {
+ const p = req.principal;
+ if (p && p.kind === 'user') return { type: 'user', id: p.subject };
+ if (p && p.kind === 'service') return { type: 'service', id: p.sub };
+ return null;
+ }
+ async function publish(req, type, payload) {
+ const ref = subject(req);
+ if (!ref) return;
+ try {
+ await events.publish({ event_type: type, actor: ref, subject: ref, visibility: ref.type === 'user' ? 'subject' : 'internal', priority: 'important', payload }, { traceparent: req.ov && req.ov.traceparent });
+ } catch (err) { log.warn('[OpenVibe.Vision] event publish failed:', String(err && err.message || err).slice(0, 180)); }
+ }
+ async function result(req, out) {
+ const task = req.visionTask || TASKS[req.path];
+ if (!task || !out) return;
+ const value = out.answer || out.check;
+ if (!value) return;
+ const receipt = value.receipt || {};
+ await publish(req, 'vision.result.ready', {
+ result_id: value.id || null, task,
+ source: req.file ? 'upload' : req.body && req.body.image ? 'data' : req.body && (req.body.image_url || req.body.url) ? 'url' : req.body && req.body.html ? 'html' : 'none',
+ model: value.model || receipt.model || null,
+ provider: value.provider || null,
+ route: value.route || receipt.engine || null,
+ cost_usd: Number(value.cost_usd ?? receipt.cost_usd ?? 0),
+ latency_ms: Math.max(0, Math.round(Number(value.latency_ms ?? receipt.ms ?? 0))),
+ confidence: Number.isFinite(Number(receipt.confidence)) && receipt.confidence != null ? Number(receipt.confidence) : null,
+ retention: String(receipt.retention || (value.id ? 'result stored for owner' : 'not stored')).slice(0, 200),
+ consent_id: null, created_at: new Date().toISOString(),
+ });
+ }
+ async function alert(req, out) {
+ if (req.path !== '/watch' || !out || !out.answer || !Array.isArray(out.answer.detections) || !out.answer.detections.length) return;
+ const ref = req.body && req.body.source_ref;
+ if (typeof ref !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9:/_-]{1,159}$/.test(ref)) return;
+ const detections = out.answer.detections.slice(0, 20);
+ await publish(req, 'vision.alert', {
+ kind: 'object', source_ref: ref, classes: detections.map((d) => String(d.class).slice(0, 60)),
+ confidence: Number.isFinite(Number(detections[0].confidence)) ? Number(detections[0].confidence) : null,
+ detected_at: new Date().toISOString(),
+ });
+ }
+ return { result, alert };
+}
+
+module.exports = { createPublisher };
@@ -8,7 +8,7 @@
@@ -8,7 +8,7 @@
* POST /ocr the local Tesseract reader; `fallback=1` asks AI only when it finds no text
* POST /ocr the local Tesseract reader; `fallback=1` asks AI only when it finds no text
* POST /compare local OCR vs one or more AI model hints, side by side, with cost and latency
* POST /compare local OCR vs one or more AI model hints, side by side, with cost and latency
* POST /ui-check overflow, cut-off, contrast, overlap and layering faults from html, url or image
* POST /ui-check overflow, cut-off, contrast, overlap and layering faults from html, url or image
- * GET /answers/:id one saved answer (its owner, or a service holding vision.ask)
+ * GET /answers/:id one saved answer (its owner, or a service holding vision.image.ask)
* GET /ui-checks/:id one saved UI check (its owner, or a service holding vision.ui.check)
* GET /ui-checks/:id one saved UI check (its owner, or a service holding vision.ui.check)
* GET /models the AI route's offered provider:model hints (no subject data)
* GET /models the AI route's offered provider:model hints (no subject data)
* GET /benchmarks aggregate compare samples: model, task, samples, mean latency and cost
* GET /benchmarks aggregate compare samples: model, task, samples, mean latency and cost
@@ -25,7 +25,7 @@ const { loadImage } = require('../vision/images');
@@ -25,7 +25,7 @@ const { loadImage } = require('../vision/images');
const { VisionError } = require('../vision/service');
const { VisionError } = require('../vision/service');
function createApi(ctx) {
function createApi(ctx) {
- const { config, s, principal, store, vision, ai, ocr, fetchImpl, networkTools, models: localModels } = ctx;
+ const { config, s, principal, store, vision, ai, ocr, fetchImpl, networkTools, models: localModels, eventPublisher } = ctx;
const r = asyncRouter();
const r = asyncRouter();
const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: config.images.maxMb * 1024 * 1024, files: 1 } });
const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: config.images.maxMb * 1024 * 1024, files: 1 } });
@@ -94,6 +94,8 @@ function createApi(ctx) {
@@ -94,6 +94,8 @@ function createApi(ctx) {
async function run(res, fn, status = 200) {
async function run(res, fn, status = 200) {
try {
try {
const out = await fn();
const out = await fn();
+ await eventPublisher.result(res.req, out);
+ await eventPublisher.alert(res.req, out);
return res.status(status).json(out);
return res.status(status).json(out);
} catch (err) {
} catch (err) {
if (err instanceof VisionError) {
if (err instanceof VisionError) {
@@ -116,27 +118,27 @@ function createApi(ctx) {
@@ -116,27 +118,27 @@ function createApi(ctx) {
const owner = (req) => req.principal.subject;
const owner = (req) => req.principal.subject;
const noRemoteSource = (req, res, next) => bodyOf(req).image_url ? invalid(res, 'private tools accept uploaded bytes or a data URL, not an unverified source URL') : next();
const noRemoteSource = (req, res, next) => bodyOf(req).image_url ? invalid(res, 'private tools accept uploaded bytes or a data URL, not an unverified source URL') : next();
- r.post('/alt', principal.requireCapability('vision.alt'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.alt(await imageOf(req)) }), 201));
- r.post('/index', principal.requireCapability('vision.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {
+ r.post('/alt', principal.requireCapability('vision.alt.write'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.alt(await imageOf(req)) }), 201));
+ r.post('/index', principal.requireCapability('vision.media.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {
const b = bodyOf(req);
const b = bodyOf(req);
return { answer: await networkTools.index(owner(req), b.item_id, await imageOf(req), b.result_url) };
return { answer: await networkTools.index(owner(req), b.item_id, await imageOf(req), b.result_url) };
}, 201));
}, 201));
- r.get('/find', principal.requireCapability('vision.find'), ownerOnly, (req, res) => run(res, async () => {
+ r.get('/find', principal.requireCapability('vision.media.find'), ownerOnly, (req, res) => run(res, async () => {
const q = String(req.query.q || '').trim();
const q = String(req.query.q || '').trim();
if (!q || q.length > 500) throw new VisionError(400, 'vision.invalid_query', 'q must contain 1–500 characters');
if (!q || q.length > 500) throw new VisionError(400, 'vision.invalid_query', 'q must contain 1–500 characters');
return { answer: await networkTools.find(owner(req), q) };
return { answer: await networkTools.find(owner(req), q) };
}));
}));
- r.post('/find', principal.requireCapability('vision.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => ({ answer: await networkTools.find(owner(req), await imageOf(req)) })));
- r.post('/receipt', principal.requireCapability('vision.receipt'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.readReceipt(await imageOf(req), bodyOf(req).fallback === true || bodyOf(req).fallback === 'true') }), 201));
- r.post('/product', principal.requireCapability('vision.product'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.product(await imageOf(req), bodyOf(req).allow_rerank === true || bodyOf(req).allow_rerank === 'true') }), 201));
- r.post('/watch', principal.requireCapability('vision.watch'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {
+ r.post('/find', principal.requireCapability('vision.media.find'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => ({ answer: await networkTools.find(owner(req), await imageOf(req)) })));
+ r.post('/receipt', principal.requireCapability('vision.receipt.read'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.readReceipt(await imageOf(req), bodyOf(req).fallback === true || bodyOf(req).fallback === 'true') }), 201));
+ r.post('/product', principal.requireCapability('vision.product.match'), attachUpload, (req, res) => run(res, async () => ({ answer: await networkTools.product(await imageOf(req), bodyOf(req).allow_rerank === true || bodyOf(req).allow_rerank === 'true') }), 201));
+ r.post('/watch', principal.requireCapability('vision.camera.watch'), ownerOnly, attachUpload, noRemoteSource, (req, res) => run(res, async () => {
const b = bodyOf(req);
const b = bodyOf(req);
const classes = Array.isArray(b.classes) ? b.classes : String(b.classes || '').split(',').map((x) => x.trim()).filter(Boolean);
const classes = Array.isArray(b.classes) ? b.classes : String(b.classes || '').split(',').map((x) => x.trim()).filter(Boolean);
if (!classes.length || classes.length > 20) throw new VisionError(400, 'vision.invalid_classes', 'send 1–20 watched classes');
if (!classes.length || classes.length > 20) throw new VisionError(400, 'vision.invalid_classes', 'send 1–20 watched classes');
const previous = b.previous_image ? await loadImage({ image: b.previous_image, config, fetchImpl }) : null;
const previous = b.previous_image ? await loadImage({ image: b.previous_image, config, fetchImpl }) : null;
return { answer: await networkTools.watch(await imageOf(req), classes, previous) };
return { answer: await networkTools.watch(await imageOf(req), classes, previous) };
}, 201));
}, 201));
- r.post('/moments', principal.requireCapability('vision.moments'), ownerOnly, (req, res) => run(res, async () => {
+ r.post('/moments', principal.requireCapability('vision.video.moments'), ownerOnly, (req, res) => run(res, async () => {
const b = bodyOf(req);
const b = bodyOf(req);
if (!Array.isArray(b.frames) || b.frames.length < 1 || b.frames.length > 30) throw new VisionError(400, 'vision.invalid_frames', 'send 1–30 timestamped frames');
if (!Array.isArray(b.frames) || b.frames.length < 1 || b.frames.length > 30) throw new VisionError(400, 'vision.invalid_frames', 'send 1–30 timestamped frames');
const frames = [];
const frames = [];
@@ -150,10 +152,10 @@ function createApi(ctx) {
@@ -150,10 +152,10 @@ function createApi(ctx) {
}, 201));
}, 201));
// ── OCR: local first, anonymous local calls allowed ─────
// ── OCR: local first, anonymous local calls allowed ─────
- r.post('/ocr', attachUpload, principal.optional('vision.ocr', { allowAnonymous: true }), async (req, res) => {
+ r.post('/ocr', attachUpload, principal.optional('vision.text.read', { allowAnonymous: true }), async (req, res) => {
const b = bodyOf(req);
const b = bodyOf(req);
const fallback = b.fallback === true || b.fallback === '1' || b.fallback === 'true';
const fallback = b.fallback === true || b.fallback === '1' || b.fallback === 'true';
- if (req.principal.kind === 'anonymous' && fallback) return problem(res, 401, 'token.required', 'AI fallback needs a signed-in person or a service token holding vision.ocr');
+ if (req.principal.kind === 'anonymous' && fallback) return problem(res, 401, 'token.required', 'AI fallback needs a signed-in person or a service token holding vision.text.read');
return run(res, async () => {
return run(res, async () => {
const image = await imageOf(req);
const image = await imageOf(req);
return await vision.ocr({
return await vision.ocr({
@@ -172,7 +174,7 @@ function createApi(ctx) {
@@ -172,7 +174,7 @@ function createApi(ctx) {
const question = String(b.question || b.q || '').trim();
const question = String(b.question || b.q || '').trim();
const readish = vision.isReadTask(task, question);
const readish = vision.isReadTask(task, question);
if (req.principal.kind === 'anonymous' && mode !== 'local' && !readish) {
if (req.principal.kind === 'anonymous' && mode !== 'local' && !readish) {
- return problem(res, 401, 'token.required', 'Ask needs a signed-in person or a service token holding vision.ask; anonymous callers can use mode=local or a read question that the local OCR engine answers.');
+ return problem(res, 401, 'token.required', 'Ask needs a signed-in person or a service token holding vision.image.ask; anonymous callers can use mode=local or a read question that the local OCR engine answers.');
}
}
return run(res, async () => {
return run(res, async () => {
const image = await imageOf(req);
const image = await imageOf(req);
@@ -183,11 +185,11 @@ function createApi(ctx) {
@@ -183,11 +185,11 @@ function createApi(ctx) {
return { answer: out };
return { answer: out };
}, 201);
}, 201);
};
};
- r.post('/ask', attachUpload, principal.optional('vision.ask', { allowAnonymous: true }), askHandler('ask'));
- r.post('/describe', attachUpload, principal.optional('vision.ask', { allowAnonymous: true }), askHandler('describe'));
+ r.post('/ask', attachUpload, principal.optional('vision.image.ask', { allowAnonymous: true }), askHandler('ask'));
+ r.post('/describe', attachUpload, principal.optional('vision.image.ask', { allowAnonymous: true }), askHandler('describe'));
// ── Compare: local + AI hints, side by side ────────────
// ── Compare: local + AI hints, side by side ────────────
- r.post('/compare', principal.optional('vision.compare'), attachUpload, async (req, res) => {
+ r.post('/compare', principal.optional('vision.model.compare'), attachUpload, async (req, res) => {
const b = bodyOf(req);
const b = bodyOf(req);
const models = Array.isArray(b.models) ? b.models : (typeof b.models === 'string' && b.models.trim() ? b.models.split(',').map((x) => x.trim()) : []);
const models = Array.isArray(b.models) ? b.models : (typeof b.models === 'string' && b.models.trim() ? b.models.split(',').map((x) => x.trim()) : []);
return run(res, async () => {
return run(res, async () => {
@@ -229,7 +231,7 @@ function createApi(ctx) {
@@ -229,7 +231,7 @@ function createApi(ctx) {
});
});
// ── Reads ──────────────────────────────────────────────
// ── Reads ──────────────────────────────────────────────
- r.get('/answers/:id', principal.optional('vision.ask', { allowAnonymous: true }), async (req, res) => {
+ r.get('/answers/:id', principal.optional('vision.image.ask', { allowAnonymous: true }), async (req, res) => {
const row = await store.answerById(String(req.params.id || ''));
const row = await store.answerById(String(req.params.id || ''));
if (!row) return problem(res, 404, 'vision.not_found', 'No such answer.');
if (!row) return problem(res, 404, 'vision.not_found', 'No such answer.');
const owner = await store.answerSubject(row.id);
const owner = await store.answerSubject(row.id);
@@ -253,7 +255,7 @@ function createApi(ctx) {
@@ -253,7 +255,7 @@ function createApi(ctx) {
// ── Model hints and benchmarks (no subject data) ───────
// ── Model hints and benchmarks (no subject data) ───────
let modelCache = null;
let modelCache = null;
- r.get('/models', async (_req, res) => {
+ r.get('/models', principal.optional('vision.models.read', { allowAnonymous: true }), async (_req, res) => {
if (!modelCache || Date.now() - modelCache.at > 5 * 60_000) {
if (!modelCache || Date.now() - modelCache.at > 5 * 60_000) {
const offered = ai.enabled ? await ai.offeredModels().catch(() => []) : [];
const offered = ai.enabled ? await ai.offeredModels().catch(() => []) : [];
modelCache = { at: Date.now(), offered };
modelCache = { at: Date.now(), offered };
@@ -268,7 +270,7 @@ function createApi(ctx) {
@@ -268,7 +270,7 @@ function createApi(ctx) {
});
});
});
});
- r.get('/benchmarks', async (req, res) => {
+ r.get('/benchmarks', principal.optional('vision.benchmarks.read', { allowAnonymous: true }), async (req, res) => {
const task = typeof req.query.task === 'string' && req.query.task ? req.query.task : null;
const task = typeof req.query.task === 'string' && req.query.task ? req.query.task : null;
const limit = req.query.limit == null ? 50 : Number.parseInt(req.query.limit, 10);
const limit = req.query.limit == null ? 50 : Number.parseInt(req.query.limit, 10);
if (!Number.isFinite(limit) || limit < 1 || limit > 100) return invalid(res, 'limit must be from 1 to 100');
if (!Number.isFinite(limit) || limit < 1 || limit > 100) return invalid(res, 'limit must be from 1 to 100');
@@ -38,8 +38,8 @@ function createFaceApi({ config, principal, faces, fetchImpl }) {
@@ -38,8 +38,8 @@ function createFaceApi({ config, principal, faces, fetchImpl }) {
standardHeaders: true, legacyHeaders: false,
standardHeaders: true, legacyHeaders: false,
handler: (req, res) => problem(res, 429, 'vision.rate_limited', 'too many face privacy requests from this address; retry shortly'),
handler: (req, res) => problem(res, 429, 'vision.rate_limited', 'too many face privacy requests from this address; retry shortly'),
});
});
- r.post('/detect', privacyLimit, imageField, run(async (req) => faces.detect((await image(req)).buffer)));
- r.post('/blur', privacyLimit, imageField, async (req, res) => {
+ r.post('/detect', privacyLimit, principal.optional('vision.face.detect', { allowAnonymous: true }), imageField, run(async (req) => faces.detect((await image(req)).buffer)));
+ r.post('/blur', privacyLimit, principal.optional('vision.face.blur', { allowAnonymous: true }), imageField, async (req, res) => {
try {
try {
const out = await faces.blur((await image(req)).buffer);
const out = await faces.blur((await image(req)).buffer);
res.set('X-Faces-Blurred', String(out.count));
res.set('X-Faces-Blurred', String(out.count));
new file mode 100644
new file mode 100644
@@ -0,0 +1,186 @@
@@ -0,0 +1,186 @@
+'use strict';
+
+const express = require('express');
+const rateLimit = require('express-rate-limit');
+const { loadImage } = require('../vision/images');
+const { check } = require('../vision/ui-check');
+const { cleanAnswer } = require('../text/no-looks');
+const { sameOrigin } = require('./principal');
+
+const VERSION = '2025-06-18';
+const VERSIONS = new Set([VERSION, '2025-03-26']);
+const input = (properties, required = []) => ({ type: 'object', properties, required, additionalProperties: false });
+const image = { image: { type: 'string', description: 'Image data URL' }, image_url: { type: 'string', description: 'Public HTTPS image URL' } };
+const TOOLS = Object.freeze([
+ { name: 'vision.ask_image', description: 'Ask about an image. Anonymous requests use local OCR only.', inputSchema: input({ ...image, question: { type: 'string' }, mode: { type: 'string', enum: ['auto', 'local', 'ai'] } }) },
+ { name: 'vision.read_text', description: 'Read visible text with local OCR.', inputSchema: input({ ...image, lang: { type: 'string' } }) },
+ { name: 'vision.translate_sign', description: 'Translate visible sign text through a routed model.', inputSchema: input({ ...image, lang: { type: 'string' } }) },
+ { name: 'vision.find_similar', description: 'Search your private visual index by image.', inputSchema: input({ image: image.image }, ['image']) },
+ { name: 'vision.search_my_media', description: 'Search your private visual index by words.', inputSchema: input({ query: { type: 'string' } }, ['query']) },
+ { name: 'vision.describe_clip', description: 'Find scene changes and moments in your own frames.', inputSchema: input({ frames: { type: 'array', items: { type: 'object' } }, prompts: { type: 'array', items: { type: 'string' } } }, ['frames']) },
+ { name: 'vision.detect_objects', description: 'Detect requested object classes in a source you own.', inputSchema: input({ image: image.image, classes: { type: 'array', items: { type: 'string' } }, source_ref: { type: 'string' } }, ['image', 'classes']) },
+ { name: 'vision.blur_faces', description: 'Blur faces without identifying anyone.', inputSchema: input(image) },
+ { name: 'vision.check_ui', description: 'Check a page or screenshot for layout and accessibility faults.', inputSchema: input({ ...image, html: { type: 'string' }, url: { type: 'string' }, viewport: { type: 'object' } }) },
+ { name: 'vision.compare_models', description: 'Compare OCR with up to three routed model hints.', inputSchema: input({ ...image, question: { type: 'string' }, models: { type: 'array', items: { type: 'string' } } }) },
+ { name: 'vision.get_job', description: 'Get a saved answer or UI check that you may read.', inputSchema: input({ id: { type: 'string' } }, ['id']) },
+]);
+
+const ACCESS = Object.freeze({
+ 'vision.ask_image': ['vision.image.ask', 'local'], 'vision.read_text': ['vision.text.read', 'anonymous'],
+ 'vision.translate_sign': ['vision.image.ask'], 'vision.find_similar': ['vision.media.find', 'person'],
+ 'vision.search_my_media': ['vision.media.find', 'person'], 'vision.describe_clip': ['vision.video.moments', 'person'],
+ 'vision.detect_objects': ['vision.camera.watch', 'person'], 'vision.blur_faces': ['vision.face.blur', 'anonymous'],
+ 'vision.check_ui': ['vision.ui.check', 'anonymous'], 'vision.compare_models': ['vision.model.compare'],
+ 'vision.get_job': ['vision.image.ask'],
+});
+const EVENT_PATH = Object.freeze({
+ 'vision.ask_image': '/ask', 'vision.read_text': '/ocr', 'vision.translate_sign': '/ask',
+ 'vision.find_similar': '/find', 'vision.search_my_media': '/find', 'vision.describe_clip': '/moments',
+ 'vision.detect_objects': '/watch', 'vision.check_ui': '/ui-check', 'vision.compare_models': '/compare',
+});
+
+function rpcResult(id, value) { return { jsonrpc: '2.0', id, result: value }; }
+function rpcError(id, code, message) { return { jsonrpc: '2.0', id: id === undefined ? null : id, error: { code, message } }; }
+function toolResult(id, value) {
+ if (value && typeof value.image === 'string' && value.image.startsWith('data:image/png;base64,')) {
+ return rpcResult(id, { content: [
+ { type: 'text', text: `Blurred ${value.count} face(s).` },
+ { type: 'image', mimeType: 'image/png', data: value.image.slice('data:image/png;base64,'.length) },
+ ], structuredContent: { count: value.count }, isError: false });
+ }
+ return rpcResult(id, { content: [{ type: 'text', text: cleanAnswer(JSON.stringify(value)) }], structuredContent: value, isError: false });
+}
+function toolError(id, message) { return rpcResult(id, { content: [{ type: 'text', text: message }], isError: true }); }
+function owner(req) { return req.principal && req.principal.kind === 'user' ? req.principal.subject : null; }
+function access(req, name, ctx, args = {}) {
+ let [cap, policy] = ACCESS[name];
+ if (name === 'vision.get_job' && String(args.id || '').startsWith('vuc_')) cap = 'vision.ui.check';
+ const p = req.principal;
+ if (policy === 'person' && !owner(req)) return 'A verified owner user token is required.';
+ if (p.kind === 'user') return p.viaSession && !sameOrigin(req, ctx.config.baseUrl) ? 'A signed-in change must come from Vision.' : null;
+ if (p.kind === 'service') return cap && !ctx.principal.granted(p, cap) ? `${cap} not granted.` : null;
+ if (policy === 'anonymous') return null;
+ if (policy === 'local' && name === 'vision.ask_image') return null;
+ return 'A Network token is required.';
+}
+
+async function invoke(name, args, req, ctx) {
+ const { config, fetchImpl, vision, networkTools, faces, store, ocr } = ctx;
+ const load = () => loadImage({ image: args.image, imageUrl: args.image_url, config, fetchImpl });
+ const loadPrivate = () => {
+ if (args.image_url || typeof args.image !== 'string' || !args.image.startsWith('data:image/')) throw new Error('Private tools accept an image data URL only.');
+ return load();
+ };
+ const requester = req.principal.requester || 'anonymous';
+ const subject = owner(req);
+ const request = { requester, subject };
+ switch (name) {
+ case 'vision.ask_image': {
+ const question = String(args.question || '').slice(0, 2000);
+ const mode = req.principal.kind === 'anonymous' ? 'local' : (args.mode || 'auto');
+ return { answer: await vision.ask({ ...request, image: await load(), task: 'ask', question, mode }) };
+ }
+ case 'vision.read_text': return await vision.ocr({ ...request, image: await load(), lang: args.lang, fallback: false });
+ case 'vision.translate_sign': return { answer: await vision.ask({ ...request, image: await load(), task: 'translate', question: 'Translate the visible sign text.', mode: 'ai', lang: args.lang }) };
+ case 'vision.find_similar': return { answer: await networkTools.find(subject, await loadPrivate()) };
+ case 'vision.search_my_media': {
+ if (typeof args.query !== 'string' || !args.query.trim() || args.query.length > 500) throw new Error('query must contain 1–500 characters');
+ return { answer: await networkTools.find(subject, args.query) };
+ }
+ case 'vision.describe_clip': {
+ if (!Array.isArray(args.frames) || !args.frames.length || args.frames.length > 30) throw new Error('send 1–30 frames');
+ const frames = [];
+ for (const f of args.frames) {
+ if (!Number.isFinite(f.timestamp_ms) || f.timestamp_ms < 0 || !f.image) throw new Error('each frame needs timestamp_ms and an image data URL');
+ frames.push({ timestamp_ms: f.timestamp_ms, image: await loadImage({ image: f.image, config, fetchImpl }) });
+ }
+ return { answer: await networkTools.moments(frames, args.prompts) };
+ }
+ case 'vision.detect_objects': {
+ if (!Array.isArray(args.classes) || !args.classes.length || args.classes.length > 20) throw new Error('send 1–20 classes');
+ return { answer: await networkTools.watch(await loadPrivate(), args.classes) };
+ }
+ case 'vision.blur_faces': {
+ const out = await faces.blur((await load()).buffer);
+ return { count: out.count, image: `data:image/png;base64,${out.buffer.toString('base64')}` };
+ }
+ case 'vision.check_ui': {
+ const viewport = args.viewport && typeof args.viewport === 'object' ? args.viewport : { w: 1280, h: 800 };
+ const source = args.image || args.image_url ? { imageBuffer: (await load()).buffer } : args.html ? { html: args.html } : { url: args.url };
+ if (!source.imageBuffer && !source.html && !source.url) throw new Error('send an image, html or url');
+ return { check: await check({ ...source, viewport, config, fetchImpl, ocr, maxFindings: config.ui.maxFindings }) };
+ }
+ case 'vision.compare_models': return await vision.compare({ ...request, image: await load(), question: args.question || '', task: 'ask', models: args.models || [] });
+ case 'vision.get_job': {
+ const id = String(args.id || '');
+ const ui = id.startsWith('vuc_');
+ const row = ui ? await store.uiCheckById(id) : await store.answerById(id);
+ const rowOwner = row && (ui ? await store.uiCheckSubject(id) : await store.answerSubject(id));
+ if (!row || (req.principal.kind !== 'service' && (!subject || rowOwner !== subject))) throw new Error('No such result.');
+ return ui ? { check: row } : { answer: row };
+ }
+ default: throw new Error('Unknown tool');
+ }
+}
+
+function mcpRouter(ctx) {
+ const r = express.Router();
+ const metadata = `${ctx.config.baseUrl}/.well-known/oauth-protected-resource`;
+ const challenge = (res, cap) => res.set('WWW-Authenticate', `Bearer resource_metadata="${metadata}"${cap ? `, scope="${cap}"` : ''}`);
+ const document = () => ({ resource: `${ctx.config.baseUrl}/mcp`, authorization_servers: [ctx.config.networkUrl], scopes_supported: [...new Set(Object.values(ACCESS).map(([cap]) => cap).filter(Boolean))], bearer_methods_supported: ['header'], resource_documentation: `${ctx.config.baseUrl}/developers` });
+ r.get('/.well-known/oauth-protected-resource', (_req, res) => res.json(document()));
+ r.get('/.well-known/oauth-protected-resource/mcp', (_req, res) => res.json(document()));
+ r.options('/mcp', (_req, res) => res.set({ 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'POST, OPTIONS', 'Access-Control-Allow-Headers': 'Content-Type, Authorization, MCP-Protocol-Version' }).status(204).end());
+ r.post('/mcp', rateLimit({ windowMs: 60_000, limit: Number(process.env.VISION_MCP_RATE_LIMIT_PER_MIN) || 60, standardHeaders: true, legacyHeaders: false }), express.json({ limit: `${Math.ceil(ctx.config.images.maxMb * 1.5) + 2}mb`, strict: false }), async (req, res, next) => {
+ res.set('Cache-Control', 'no-store');
+ try {
+ const resolved = await ctx.principal.resolve(req);
+ if (resolved.error) { challenge(res); return res.status(resolved.error[0]).json(rpcError(null, -32001, resolved.error[2])); }
+ req.principal = resolved.principal;
+ if (req.get('mcp-protocol-version') && !VERSIONS.has(req.get('mcp-protocol-version'))) return res.status(400).json(rpcError(null, -32600, 'Unsupported protocol version'));
+ const batch = Array.isArray(req.body);
+ const messages = batch ? req.body : [req.body];
+ if (!messages.length) return res.json(rpcError(null, -32600, 'Invalid Request'));
+ const responses = [];
+ for (const msg of messages) {
+ if (!msg || typeof msg !== 'object' || Array.isArray(msg) || msg.jsonrpc !== '2.0' || typeof msg.method !== 'string' || (Object.hasOwn(msg, 'id') && !(msg.id === null || typeof msg.id === 'string' || Number.isFinite(msg.id)))) { responses.push(rpcError(null, -32600, 'Invalid Request')); continue; }
+ if (!Object.hasOwn(msg, 'id')) continue;
+ const id = msg.id, params = msg.params && typeof msg.params === 'object' && !Array.isArray(msg.params) ? msg.params : {};
+ if (msg.method === 'initialize') responses.push(rpcResult(id, { protocolVersion: VERSIONS.has(params.protocolVersion) ? params.protocolVersion : VERSION, capabilities: { tools: { listChanged: false } }, serverInfo: { name: 'openvibe-vision', version: require('../../package.json').version } }));
+ else if (msg.method === 'ping') responses.push(rpcResult(id, {}));
+ else if (msg.method === 'tools/list') responses.push(rpcResult(id, { tools: TOOLS }));
+ else if (msg.method === 'tools/call') {
+ const name = params.name;
+ if (!Object.hasOwn(ACCESS, name)) { responses.push(rpcError(id, -32602, 'Unknown tool')); continue; }
+ const args = params.arguments && typeof params.arguments === 'object' && !Array.isArray(params.arguments) ? params.arguments : {};
+ const denied = access(req, name, ctx, args);
+ if (denied) {
+ if (req.principal.kind === 'anonymous') challenge(res, name === 'vision.get_job' && String(args.id || '').startsWith('vuc_') ? 'vision.ui.check' : ACCESS[name][0]);
+ responses.push(toolError(id, denied)); continue;
+ }
+ try {
+ const value = await invoke(name, args, req, ctx);
+ if (EVENT_PATH[name]) {
+ const eventReq = { principal: req.principal, ov: req.ov, path: EVENT_PATH[name], body: args, visionTask: name === 'vision.translate_sign' ? 'translate' : undefined };
+ await ctx.eventPublisher.result(eventReq, value);
+ await ctx.eventPublisher.alert(eventReq, value);
+ }
+ responses.push(toolResult(id, value));
+ }
+ catch (err) { responses.push(toolError(id, err.detail || err.message || 'Tool failed')); }
+ } else responses.push(rpcError(id, -32601, 'Method not found'));
+ }
+ if (!responses.length) return res.status(202).end();
+ return res.status(res.getHeader('WWW-Authenticate') ? 401 : 200).json(batch ? responses : responses[0]);
+ } catch (err) { next(err); }
+ });
+ r.all('/mcp', (_req, res) => res.set('Allow', 'POST').status(405).json(rpcError(null, -32600, 'Method Not Allowed')));
+ r.use((err, _req, res, next) => {
+ if (err && err.type === 'entity.parse.failed') return res.status(400).json(rpcError(null, -32700, 'Parse error'));
+ if (err && err.type === 'entity.too.large') return res.status(413).json(rpcError(null, -32600, 'Request body too large'));
+ next(err);
+ });
+ return r;
+}
+
+module.exports = { mcpRouter, TOOLS };
@@ -7,10 +7,9 @@
@@ -7,10 +7,9 @@
* a person: their Network user token as a Bearer, or this site's session cookie (vision_at).
* a person: their Network user token as a Bearer, or this site's session cookie (vision_at).
* { kind: 'service', sub: 'svc:design', cap: [...], project, claims }
* { kind: 'service', sub: 'svc:design', cap: [...], project, claims }
* a Network service, app or agent token for audience openvibe.vision. Each machine route names exactly one
* a Network service, app or agent token for audience openvibe.vision. Each machine route names exactly one
- * capability (vision.ui.check for Design's Arena, vision.face.cast.match for Live's channel roster, vision.ask,
- * vision.ocr, vision.compare, and the six network tools) and checks the token's grant list with
- * openvibe-contracts capabilities.grants. Unknown capability ids are accepted here until the main session
- * registers the manifests; the main session adds the grants.
+ * capability (vision.ui.check for Design's Arena, vision.face.cast.match for Live's channel roster,
+ * vision.image.ask, vision.text.read, vision.model.compare, and the network tools) and checks the token's
+ * grant list with openvibe-contracts capabilities.grants.
* { kind: 'anonymous' }
* { kind: 'anonymous' }
*
*
* A request that presents a token is judged on that token alone: a bad one is refused, never downgraded to the
* A request that presents a token is judged on that token alone: a bad one is refused, never downgraded to the
@@ -22,10 +21,14 @@ const { serviceAuth, capabilities, http, ids } = require('openvibe-contracts');
@@ -22,10 +21,14 @@ const { serviceAuth, capabilities, http, ids } = require('openvibe-contracts');
const PRINCIPAL_SUB = /^(svc|app|mod|agent):/;
const PRINCIPAL_SUB = /^(svc|app|mod|agent):/;
const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;
const PROJECT_RE = /^prj_[0-9A-HJKMNP-TV-Z]{26}$/;
-/** Vision's machine capabilities (the main session registers them; Vision accepts the grant directly). */
+/** Vision's registered capabilities (OpenVibe.Contracts manifests/services/vision.json). */
const CAPABILITIES = Object.freeze([
const CAPABILITIES = Object.freeze([
- 'vision.ask', 'vision.ocr', 'vision.compare', 'vision.ui.check', 'vision.face.cast.match',
- 'vision.alt', 'vision.find', 'vision.receipt', 'vision.product', 'vision.watch', 'vision.moments',
+ 'vision.ui.check', 'vision.image.ask', 'vision.text.read', 'vision.model.compare',
+ 'vision.alt.write', 'vision.media.find', 'vision.receipt.read', 'vision.product.match',
+ 'vision.camera.watch', 'vision.video.moments', 'vision.face.detect', 'vision.face.blur',
+ 'vision.face.consent.manage', 'vision.face.verify.self', 'vision.face.find.self',
+ 'vision.face.library', 'vision.face.cast.match', 'vision.face.cast.read',
+ 'vision.models.read', 'vision.benchmarks.read',
]);
]);
function decodePayload(token) {
function decodePayload(token) {
@@ -241,16 +241,16 @@ function developers({ ai = false, models = [] } = {}) {
@@ -241,16 +241,16 @@ function developers({ ai = false, models = [] } = {}) {
<header class="vi-page-h"><p class="vi-kicker">Developers</p><h1>The Vision API</h1>
<header class="vi-page-h"><p class="vi-kicker">Developers</p><h1>The Vision API</h1>
<p class="vi-lede">JSON under <code>/api/v1</code>, with <code>application/problem+json</code> errors. A person is the <code>vision_at</code> session cookie or a Network user token as a Bearer. A service token needs the capability on the route; Design's Arena calls the UI check with <code>vision.ui.check</code>.</p></header>
<p class="vi-lede">JSON under <code>/api/v1</code>, with <code>application/problem+json</code> errors. A person is the <code>vision_at</code> session cookie or a Network user token as a Bearer. A service token needs the capability on the route; Design's Arena calls the UI check with <code>vision.ui.check</code>.</p></header>
<section><h2>Ask, read, compare</h2><dl class="vi-eps">
<section><h2>Ask, read, compare</h2><dl class="vi-eps">
-${ep('POST', '/api/v1/ask (alias /api/v1/describe)', 'One image and a question: <code>{ task, question, mode, image_url }</code>, a multipart <code>image</code>, or an <code>image</code> data URL. Local OCR answers a read question first, otherwise OpenVibe.AI. Capability: <code>vision.ask</code>.')}
-${ep('POST', '/api/v1/ocr', 'The local Tesseract reader. <code>fallback=1</code> asks AI only when no text is found and the caller is signed in or holds <code>vision.ocr</code>.')}
-${ep('POST', '/api/v1/compare', 'The same image through local OCR and one to three AI hints (<code>models</code>). Returns each engine\'s answer, provider, model, cost and latency. Capability: <code>vision.compare</code>.')}
+${ep('POST', '/api/v1/ask (alias /api/v1/describe)', 'One image and a question: <code>{ task, question, mode, image_url }</code>, a multipart <code>image</code>, or an <code>image</code> data URL. Local OCR answers a read question first, otherwise OpenVibe.AI. Capability: <code>vision.image.ask</code>.')}
+${ep('POST', '/api/v1/ocr', 'The local Tesseract reader. <code>fallback=1</code> asks AI only when no text is found and the caller is signed in or holds <code>vision.text.read</code>.')}
+${ep('POST', '/api/v1/compare', 'The same image through local OCR and one to three AI hints (<code>models</code>). Returns each engine\'s answer, provider, model, cost and latency. Capability: <code>vision.model.compare</code>.')}
${ep('POST', '/api/v1/ui-check', 'HTML, URL or screenshot with <code>viewport</code>. Returns <code>findings</code>: overflow, cut_off, contrast, overlap and layering. Capability: <code>vision.ui.check</code>.')}
${ep('POST', '/api/v1/ui-check', 'HTML, URL or screenshot with <code>viewport</code>. Returns <code>findings</code>: overflow, cut_off, contrast, overlap and layering. Capability: <code>vision.ui.check</code>.')}
-${ep('POST', '/api/v1/alt', 'Pics, Media and Chat send an image for short editable alt text. Capability: <code>vision.alt</code>.')}
-${ep('POST', '/api/v1/index; GET or POST /api/v1/find', 'Media and Search index an owner image and find it by words or image. Requires the owner user token. Capability: <code>vision.find</code>.')}
-${ep('POST', '/api/v1/receipt', 'Food sends a receipt image; Tesseract reads store, items and total first. Capability: <code>vision.receipt</code>.')}
-${ep('POST', '/api/v1/product', 'Deals sends a product image for description, CLIP vector and configured offer candidates. Capability: <code>vision.product</code>.')}
-${ep('POST', '/api/v1/watch', 'Cam sends an owner-provided still and watched classes. Requires the owner user token. Capability: <code>vision.watch</code>.')}
-${ep('POST', '/api/v1/moments', 'Live or Media sends timestamped frames for scenes and candidate moments. Requires the owner user token. Capability: <code>vision.moments</code>.')}
+${ep('POST', '/api/v1/alt', 'Pics, Media and Chat send an image for short editable alt text. Capability: <code>vision.alt.write</code>.')}
+${ep('POST', '/api/v1/index; GET or POST /api/v1/find', 'Media and Search index an owner image and find it by words or image. Requires the owner user token. Capability: <code>vision.media.find</code>.')}
+${ep('POST', '/api/v1/receipt', 'Food sends a receipt image; Tesseract reads store, items and total first. Capability: <code>vision.receipt.read</code>.')}
+${ep('POST', '/api/v1/product', 'Deals sends a product image for description, CLIP vector and configured offer candidates. Capability: <code>vision.product.match</code>.')}
+${ep('POST', '/api/v1/watch', 'Cam sends an owner-provided still and watched classes. Requires the owner user token. Capability: <code>vision.camera.watch</code>.')}
+${ep('POST', '/api/v1/moments', 'Live or Media sends timestamped frames for scenes and candidate moments. Requires the owner user token. Capability: <code>vision.video.moments</code>.')}
${ep('GET', '/api/v1/answers/:id', 'One saved answer, for its owner or a service holding the route capability.')}
${ep('GET', '/api/v1/answers/:id', 'One saved answer, for its owner or a service holding the route capability.')}
${ep('GET', '/api/v1/ui-checks/:id', 'One saved UI check, for its owner or a service holding <code>vision.ui.check</code>.')}
${ep('GET', '/api/v1/ui-checks/:id', 'One saved UI check, for its owner or a service holding <code>vision.ui.check</code>.')}
${ep('GET', '/api/v1/models', 'The AI route\'s offered provider:model hints and the local engine. No subject data.')}
${ep('GET', '/api/v1/models', 'The AI route\'s offered provider:model hints and the local engine. No subject data.')}
@@ -272,7 +272,7 @@ ${ep('POST', '/api/v1/faces/library/correct', 'A person\'s own group correction:
@@ -272,7 +272,7 @@ ${ep('POST', '/api/v1/faces/library/correct', 'A person\'s own group correction:
</dl>
</dl>
<p class="vi-fine"><a href="/people">Manage choices</a> · <a href="/people/privacy">Retention and privacy</a></p></section>
<p class="vi-fine"><a href="/people">Manage choices</a> · <a href="/people/privacy">Retention and privacy</a></p></section>
<section id="vi-capabilities"><h2>Capabilities and grants</h2>
<section id="vi-capabilities"><h2>Capabilities and grants</h2>
-<p class="vi-fine">Vision needs no new AI capability: outgoing calls are <code>ai.run.create</code> to OpenVibe.AI, audience <code>openvibe.ai</code>, namespace <code>ai.*</code>. The capabilities Vision offers are <code>vision.ask</code>, <code>vision.ocr</code>, <code>vision.compare</code>, <code>vision.ui.check</code> and <code>vision.face.cast.match</code>; the main session registers the manifests and grants them to Design (the Arena), Live (the cast roster) and any other caller.</p>
+<p class="vi-fine">Vision calls OpenVibe.AI with <code>ai.run.create</code>, audience <code>openvibe.ai</code> and namespace <code>ai.*</code>. The twenty Vision capability manifests cover image questions, text, comparison, UI checks, network tools and consent-scoped face operations. A service token needs the capability named on its route; a person manages only their own face consent. Vision also offers eleven tools on <code>/mcp</code>, with Network token discovery at <code>/.well-known/oauth-protected-resource</code>.</p>
<pre class="vi-code">POST /api/v1/ui-check
<pre class="vi-code">POST /api/v1/ui-check
Authorization: Bearer <Network service token for openvibe.vision>
Authorization: Bearer <Network service token for openvibe.vision>
Content-Type: application/json
Content-Type: application/json
@@ -103,7 +103,7 @@ function createFaces({ s, config, model }) {
@@ -103,7 +103,7 @@ function createFaces({ s, config, model }) {
async function template(subject, feature, scope = '') {
async function template(subject, feature, scope = '') {
requireKey(); await requireConsent(subject, feature, scope);
requireKey(); await requireConsent(subject, feature, scope);
const row = await db.maybe('SELECT ciphertext FROM face_templates WHERE subject=$1 AND feature=$2 AND scope=$3', [subject, feature, scope]);
const row = await db.maybe('SELECT ciphertext FROM face_templates WHERE subject=$1 AND feature=$2 AND scope=$3', [subject, feature, scope]);
- if (!row) deny('enrolment_required', `Enrol in ${feature} first.`);
+ if (!row) deny('consent_required', `Enable ${feature} consent first.`);
return open(key, row.ciphertext);
return open(key, row.ciphertext);
}
}
async function touch(subject, feature, scope = '') {
async function touch(subject, feature, scope = '') {
@@ -101,14 +101,14 @@ const { boot, check, done } = require('./helpers/boot');
@@ -101,14 +101,14 @@ const { boot, check, done } = require('./helpers/boot');
const good = t.network.serviceToken({ sub: 'svc:design', cap: ['vision.ui.check'] });
const good = t.network.serviceToken({ sub: 'svc:design', cap: ['vision.ui.check'] });
const ok = await t.get('/api/v1/ui-check', { bearer: good, json: { html: '<div style="width:900px">x</div>', viewport: { w: 390, h: 844 } } });
const ok = await t.get('/api/v1/ui-check', { bearer: good, json: { html: '<div style="width:900px">x</div>', viewport: { w: 390, h: 844 } } });
assert.strictEqual(ok.status, 201, ok.text);
assert.strictEqual(ok.status, 201, ok.text);
- const bad = t.network.serviceToken({ sub: 'svc:design', cap: ['vision.ask'] });
+ const bad = t.network.serviceToken({ sub: 'svc:design', cap: ['vision.image.ask'] });
const no = await t.get('/api/v1/ui-check', { bearer: bad, json: { html: '<p>x</p>' } });
const no = await t.get('/api/v1/ui-check', { bearer: bad, json: { html: '<p>x</p>' } });
assert.strictEqual(no.status, 403);
assert.strictEqual(no.status, 403);
assert.strictEqual(no.json().code, 'capability.denied');
assert.strictEqual(no.json().code, 'capability.denied');
});
});
await check('a service token with vision.ask can ask; the answer has no subject', async () => {
await check('a service token with vision.ask can ask; the answer has no subject', async () => {
- const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.ask'] });
+ const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.image.ask'] });
const r = await t.get('/api/v1/ask', { bearer: tok, multipart: { fields: { task: 'describe', question: 'What is this?' }, image: png } });
const r = await t.get('/api/v1/ask', { bearer: tok, multipart: { fields: { task: 'describe', question: 'What is this?' }, image: png } });
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.json().answer.answer.engine, 'ai');
assert.strictEqual(r.json().answer.answer.engine, 'ai');
@@ -122,7 +122,7 @@ const { boot, check, done } = require('./helpers/boot');
@@ -122,7 +122,7 @@ const { boot, check, done } = require('./helpers/boot');
const id = r.json().answer.id;
const id = r.json().answer.id;
assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { as: user })).status, 200);
assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { as: user })).status, 200);
assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { as: other })).status, 404);
assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { as: other })).status, 404);
- const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.ask'] });
+ const tok = t.network.serviceToken({ sub: 'svc:today', cap: ['vision.image.ask'] });
assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { bearer: tok })).status, 200);
assert.strictEqual((await t.get(`/api/v1/answers/${id}`, { bearer: tok })).status, 200);
});
});
new file mode 100644
new file mode 100644
@@ -0,0 +1,76 @@
@@ -0,0 +1,76 @@
+'use strict';
+
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const { createPublisher } = require('../server/events/publisher');
+
+(async () => {
+ const sent = [];
+ const config = {
+ drill: false, networkInternalUrl: 'https://network.example',
+ oauth: { clientId: 'vision', clientSecret: 'test-secret' },
+ events: { url: 'https://events.example', scope: 'events.event.publish' },
+ };
+ const fetchImpl = async (url, options) => {
+ if (String(url).endsWith('/oauth/token')) return Response.json({ access_token: 'test-token', expires_in: 300 });
+ if (String(url).endsWith('/api/v1/events')) {
+ sent.push(JSON.parse(options.body));
+ return Response.json({ event_id: sent.at(-1).event_id, cursor: 'cursor' });
+ }
+ throw new Error('unexpected URL');
+ };
+ const publisher = createPublisher({ config, fetchImpl, log: { warn() { throw new Error('event publish failed'); } } });
+ const principal = { kind: 'user', subject: 'usr_01J8Z3Q4R5S6T7V8W9X0Y1Z2A4' };
+ const req = (path, body) => ({ path, body, principal, ov: {} });
+
+ await check('ready event carries provenance but no answer text or image', async () => {
+ await publisher.result(req('/ask', { image: 'data:image/png;base64,AAAA' }), {
+ answer: { id: 'vis_01J8Z3Q4R5S6T7V8W9X0Y1Z2A4', text: 'A red bicycle.', model: 'model', provider: 'ai', route: 'routed', cost_usd: 0.01, latency_ms: 120 },
+ });
+ assert.strictEqual(sent.length, 1);
+ assert.strictEqual(sent[0].event_type, 'vision.result.ready');
+ assert.deepStrictEqual(sent[0].subject, { type: 'user', id: principal.subject });
+ assert.strictEqual(sent[0].payload.task, 'ask');
+ assert.strictEqual(sent[0].payload.source, 'data');
+ assert.ok(!JSON.stringify(sent[0]).includes('red bicycle'));
+ assert.ok(!JSON.stringify(sent[0]).includes('AAAA'));
+ });
+
+ await check('watch event names only the owner-granted source and detected classes', async () => {
+ await publisher.alert(req('/watch', { source_ref: 'cam:front-door' }), { answer: { detections: [{ class: 'animal', confidence: 0.92 }] } });
+ assert.strictEqual(sent[1].event_type, 'vision.alert');
+ assert.strictEqual(sent[1].payload.source_ref, 'cam:front-door');
+ assert.deepStrictEqual(sent[1].payload.classes, ['animal']);
+ await publisher.alert(req('/watch', {}), { answer: { detections: [{ class: 'animal', confidence: 0.92 }] } });
+ assert.strictEqual(sent.length, 2);
+ });
+
+ await check('anonymous results create no subject event', async () => {
+ await publisher.result({ ...req('/ocr', {}), principal: { kind: 'anonymous' } }, { answer: { text: 'x' } });
+ assert.strictEqual(sent.length, 2);
+ });
+
+ await check('successful API results call the publisher with the authenticated owner', async () => {
+ const calls = [];
+ const t = await boot({ eventPublisher: {
+ result: async (request, out) => calls.push({ path: request.path, subject: request.principal.subject, id: out.check && out.check.id }),
+ alert: async () => {},
+ } });
+ try {
+ const user = t.network.addUser('event-owner');
+ const r = await t.get('/api/v1/ui-check', { as: user, json: { html: '<p>hello</p>' } });
+ assert.strictEqual(r.status, 201, r.text);
+ assert.deepStrictEqual(calls, [{ path: '/ui-check', subject: user.subject, id: r.json().check.id }]);
+ const mcp = await t.get('/mcp', { bearer: t.network.userToken(user), json: {
+ jsonrpc: '2.0', id: 1, method: 'tools/call',
+ params: { name: 'vision.check_ui', arguments: { html: '<p>hello</p>' } },
+ } });
+ assert.strictEqual(mcp.status, 200, mcp.text);
+ assert.strictEqual(mcp.json().result.isError, false, mcp.text);
+ assert.strictEqual(calls.length, 2);
+ assert.strictEqual(calls[1].path, '/ui-check');
+ assert.strictEqual(calls[1].subject, user.subject);
+ } finally { await t.close(); }
+ });
+ done();
+})();
@@ -135,6 +135,10 @@ function liveToken(t, channel) {
@@ -135,6 +135,10 @@ function liveToken(t, channel) {
assert.strictEqual(row.signature, 'Test Person');
assert.strictEqual(row.signature, 'Test Person');
assert.strictEqual(row.revoked_at, null);
assert.strictEqual(row.revoked_at, null);
assert.ok(Date.parse(row.granted_at));
assert.ok(Date.parse(row.granted_at));
+ const waitingForEnrolment = await t.get('/api/v1/faces/verify', { as: ana, multipart: { image: png } });
+ assert.strictEqual(waitingForEnrolment.status, 403);
+ assert.strictEqual(waitingForEnrolment.json().code, 'faces.consent_required');
+ assert.strictEqual(waitingForEnrolment.json().detail, 'Enable verify consent first.');
assert.strictEqual((await consent(ana, 'find')).status, 200);
assert.strictEqual((await consent(ana, 'find')).status, 200);
assert.strictEqual((await consent(ana, 'cast', { channel: 'live:one', adult: true })).status, 200);
assert.strictEqual((await consent(ana, 'cast', { channel: 'live:one', adult: true })).status, 200);
@@ -46,7 +46,7 @@ async function boot(opts = {}) {
@@ -46,7 +46,7 @@ async function boot(opts = {}) {
const { createStore } = require('../../server/db');
const { createStore } = require('../../server/db');
const testdb = await require('./db').testDb();
const testdb = await require('./db').testDb();
const store = createStore(testdb.db, { now: opts.now });
const store = createStore(testdb.db, { now: opts.now });
- const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, accountSend: opts.accountSend, ocr: opts.ocr, ai: opts.aiClient, facesModel: opts.facesModel, clip: opts.clip, detector: opts.detector });
+ const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, accountSend: opts.accountSend, eventPublisher: opts.eventPublisher, ocr: opts.ocr, ai: opts.aiClient, facesModel: opts.facesModel, clip: opts.clip, detector: opts.detector });
await built.ctx.keys.ensure();
await built.ctx.keys.ensure();
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
new file mode 100644
new file mode 100644
@@ -0,0 +1,72 @@
@@ -0,0 +1,72 @@
+'use strict';
+
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+
+const rpc = (method, params = {}, id = 1) => ({ jsonrpc: '2.0', id, method, params });
+
+(async () => {
+ const t = await boot();
+ const owner = t.network.addUser('mcp-owner');
+ try {
+ await check('MCP advertises RFC 9728 metadata and its supported tools', async () => {
+ const meta = await t.get('/.well-known/oauth-protected-resource');
+ assert.strictEqual(meta.status, 200);
+ assert.strictEqual(meta.json().resource, 'https://openvibe.vision/mcp');
+ assert.deepStrictEqual(meta.json().authorization_servers, ['http://' + new URL(t.network.url).host]);
+ const init = await t.get('/mcp', { json: rpc('initialize', { protocolVersion: '2025-06-18' }) });
+ assert.strictEqual(init.status, 200, init.text);
+ assert.strictEqual(init.json().result.serverInfo.name, 'openvibe-vision');
+ const list = await t.get('/mcp', { json: rpc('tools/list') });
+ assert.strictEqual(list.json().result.tools.length, 11);
+ assert.ok(list.json().result.tools.some((x) => x.name === 'vision.blur_faces'));
+ assert.ok(!list.json().result.tools.some((x) => /enrol|verify|cast/.test(x.name)));
+ });
+
+ await check('private tools challenge anonymous callers with resource metadata', async () => {
+ const r = await t.get('/mcp', { json: rpc('tools/call', { name: 'vision.search_my_media', arguments: { query: 'desk' } }) });
+ assert.strictEqual(r.status, 401, r.text);
+ assert.match(r.headers.get('www-authenticate'), /resource_metadata="https:\/\/openvibe.vision\/\.well-known\/oauth-protected-resource"/);
+ assert.strictEqual(r.json().result.isError, true);
+ });
+
+ await check('a service token needs the exact tool capability', async () => {
+ const denied = await t.get('/mcp', { bearer: t.network.serviceToken({ cap: ['vision.ui.check'] }), json: rpc('tools/call', { name: 'vision.compare_models', arguments: {} }) });
+ assert.strictEqual(denied.status, 200);
+ assert.strictEqual(denied.json().result.isError, true);
+ assert.match(denied.json().result.content[0].text, /vision.model.compare/);
+ });
+
+ await check('private image tools refuse a remote source before fetching it', async () => {
+ const r = await t.get('/mcp', { bearer: t.network.userToken(owner), json: rpc('tools/call', { name: 'vision.find_similar', arguments: { image_url: t.images.url + '/image.png' } }) });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.json().result.isError, true);
+ assert.match(r.json().result.content[0].text, /image data URL only/);
+ });
+
+ await check('an owner may read their own UI check, while another owner cannot', async () => {
+ const saved = await t.get('/api/v1/ui-check', { as: owner, json: { html: '<div>hello</div>' } });
+ assert.strictEqual(saved.status, 201, saved.text);
+ const id = saved.json().check.id;
+ const own = await t.get('/mcp', { bearer: t.network.userToken(owner), json: rpc('tools/call', { name: 'vision.get_job', arguments: { id } }) });
+ assert.strictEqual(own.json().result.isError, false, own.text);
+ assert.strictEqual(own.json().result.structuredContent.check.id, id);
+ const other = t.network.addUser('mcp-other');
+ const refused = await t.get('/mcp', { bearer: t.network.userToken(other), json: rpc('tools/call', { name: 'vision.get_job', arguments: { id } }) });
+ assert.strictEqual(refused.json().result.isError, true);
+ assert.strictEqual(refused.json().result.content[0].text, 'No such result.');
+ });
+
+ await check('notifications, batches and parse errors follow JSON-RPC', async () => {
+ const batch = await t.get('/mcp', { json: [rpc('ping', {}, 1), rpc('tools/list', {}, 2)] });
+ assert.strictEqual(batch.status, 200);
+ assert.strictEqual(batch.json().length, 2);
+ const notification = await t.get('/mcp', { json: { jsonrpc: '2.0', method: 'notifications/initialized' } });
+ assert.strictEqual(notification.status, 202);
+ const malformed = await t.get('/mcp', { body: '{', headers: { 'content-type': 'application/json' } });
+ assert.strictEqual(malformed.status, 400);
+ assert.strictEqual(malformed.json().error.code, -32700);
+ });
+ } finally { await t.close(); }
+ done();
+})();
@@ -23,7 +23,7 @@ const { parseReceipt } = require('../server/vision/network-tools');
@@ -23,7 +23,7 @@ const { parseReceipt } = require('../server/vision/network-tools');
const theirs = await t.get('/api/v1/find?q=bike', { as: other });
const theirs = await t.get('/api/v1/find?q=bike', { as: other });
assert.strictEqual(theirs.status, 200);
assert.strictEqual(theirs.status, 200);
assert.deepStrictEqual(theirs.json().answer.results, []);
assert.deepStrictEqual(theirs.json().answer.results, []);
- const svc = t.network.serviceToken({ sub: 'svc:media', cap: ['vision.find'] });
+ const svc = t.network.serviceToken({ sub: 'svc:media', cap: ['vision.media.find'] });
assert.strictEqual((await t.get('/api/v1/find?q=bike', { bearer: svc })).status, 403);
assert.strictEqual((await t.get('/api/v1/find?q=bike', { bearer: svc })).status, 403);
const byImage = await t.get('/api/v1/find', { as: owner, json: { image } });
const byImage = await t.get('/api/v1/find', { as: owner, json: { image } });
assert.strictEqual(byImage.json().answer.results[0].item_id, 'photo-1');
assert.strictEqual(byImage.json().answer.results[0].item_id, 'photo-1');
@@ -54,7 +54,7 @@ const { parseReceipt } = require('../server/vision/network-tools');
@@ -54,7 +54,7 @@ const { parseReceipt } = require('../server/vision/network-tools');
assert.strictEqual(m.status, 201, m.text);
assert.strictEqual(m.status, 201, m.text);
assert.ok(m.json().answer.receipt);
assert.ok(m.json().answer.receipt);
assert.strictEqual(m.json().answer.candidate_moments[0].timestamp_ms, 1000);
assert.strictEqual(m.json().answer.candidate_moments[0].timestamp_ms, 1000);
- const svc = t.network.serviceToken({ sub: 'svc:cam', cap: ['vision.watch'] });
+ const svc = t.network.serviceToken({ sub: 'svc:cam', cap: ['vision.camera.watch'] });
assert.strictEqual((await t.get('/api/v1/watch', { bearer: svc, json: { image, classes: ['cat'] } })).status, 403);
assert.strictEqual((await t.get('/api/v1/watch', { bearer: svc, json: { image, classes: ['cat'] } })).status, 403);
});
});
await check('product has embedding and receipt even if offer APIs are not configured', async () => {
await check('product has embedding and receipt even if offer APIs are not configured', async () => {