openvibers/openvibe.tools Commits
Uniform run API, jobs facade, deprecation headers and the Origin check (S6, ADR-027). POST /api/v1/tools/:id/run takes tools.run-request@1 as JSON or multipart exactly as openvibe-sdk v0.6.0 sends it (uploads as file parts; input, files (the references' JSON), wait_ms and idempotency_key as text parts; ?wait_ms= too; the Idempotency-Key header wins) and answers tools.run@1: 200 succeeded or failed (the tool's own problem), 202 + Location while a job tool is still going after wait_ms, 200 + Idempotent-Replayed for a replay. Refusals are problem+json with the contract's codes: 400 tools.run.invalid, 401 token.missing/token.invalid/tools.session_required, 403 capability.denied/tools.origin.refused, 404 tools.tool.not_found/tools.tool.not_runnable (api false: yt, regex…)/tools.run.file_not_found, 405, 409 tools.job.idempotency_conflict, 413 tools.file.too_large/tools.input.too_large, 415 (bytes sniffed), 422 tools.input.invalid with errors[] (the descriptor's input schema, Ajv from openvibe-contracts), 429 tools.quota.exceeded, 503 tools.tool.unavailable (descriptor status) vs tools.unavailable (the engine found a program missing), 503 tools.busy. Authorization is always enforced: a token needs the descriptor's auth.capability, network probes run only for tokens with tools.net.probe, auth.anonymous false needs a session, sign-in or token; quotas are the tool's class and cost per caller tier. The gateway answers for every tool: dev and text engines run the pages' code in a worker pool (terminated at the descriptor's timeout), net tools and Open Graph call their existing route in process (myip is the caller), both cached per tool where safe (spec cacheTtlMs: engines 10 min except random/time-dependent ones, slow lookups 1-60 min, never probes or live checks), inline Idempotency-Key replays for 15 min; img, audio and docs tools are streamed to their satellite, which runs them as jobs (job input { ...input, ...preset, tool: operation }; jobs now record the tool), with { job_id, index } references read across satellites under the owner's own credentials and { media_id } from the owner's Media-stored results. The gateway's /api/v1/jobs facade routes submits by type prefix (JSON, the multipart type part read from the stream head, or ?type=) and everything else by job id (remembered, else asked on loopback via GET /api/internal/jobs/:id, direct callers only and never limited), streaming events and files. /api/process…, /api/net/* and /api/dev/* keep their shapes and add Deprecation: true, Sunset: Thu, 31 Dec 2026 23:59:59 GMT and Link: </api/v1/tools/{id}/run>; rel="successor-version". The guard checks Origin (else Referer) on cookie-carried mutations (run API, job writes, /api/process…, /api/info, /api/probe, webhook bin create/delete): first-party *.openvibe.tools, the tool's own hosts and the same host pass, Bearer calls and cookie-less calls need nothing; 403 tools.origin.refused in enforce mode, recorded in report mode. Run and jobs routes give other sites' pages CORS without credentials. On /api/v1 the older limiters answer problem+json with Retry-After; img and docs let untyped upload parts through to the guard's sniffing (as audio did); refused uploads are deleted before the answer. Pool fix: a worker that errors (heap limit) leaves the pool at once, so no run is handed to it before its exit. nginx copy: a location for the run and jobs routes (512m uploads, no buffering, 600 s). Tests: run-api.test.js (gateway, img, docs, a stand-in Network: every answer checked against tools.run@1), jobs-facade.test.js, sdk-client.test.js (openvibe-sdk v0.6.0 createToolsClient against the real gateway: registry, sync run, job run + wait + file, waitMs, replays, a job reference img → docs, retry, a probe token; skips without an SDK checkout), guard Origin cases; the audio poll and the tools-api ETag checks no longer race on a busy machine
Changes
@@@ -99,6 +99,75 @@ answers for every tool; each satellite answers the same routes for its own toolslibrary or isolation (XML minify, HTML to Markdown, the JS minifier/beautifier, the regex tester, the canvasgraphics makers) say `api: false`, as does the YouTube downloader (page-only by decision).++## Run API (ADR-027)++++`POST /api/v1/tools/:id/run` runs any tool whose descriptor says `api: true` (capability `tools.tool.run`;++`tools.net.probe` for the network probes). The gateway (`https://openvibe.tools`) answers for every tool; img,++audio and docs answer the same route for their own tools. `openvibe-sdk/tools` (v0.6.0) is its client.++++- **Request** (`tools.run-request@1`): JSON `{ input, files, wait_ms, idempotency_key }`, or++ `multipart/form-data` with each upload as a `file` part and the text parts `input` (JSON), `files` (the++ file references' JSON), `wait_ms`, `idempotency_key` — exactly what the SDK sends. `?wait_ms=` does the same++ as the field; the `Idempotency-Key` header wins over `idempotency_key`. A file reference is++ `{ job_id, index }` (a result file of one of your own jobs, on any satellite: one tool's output feeds the++ next) or `{ media_id }` (a result of one of your own jobs stored in OpenVibe.Media). Uploads come first,++ then references, in order; the count must fit the descriptor's `files`.++- **Answers** (`tools.run@1`): `200 { state: succeeded, tool, result: { data | text | files }, took_ms[, job] }`;++ `200 { state: failed | cancelled, tool, error, took_ms[, job] }` when the tool itself fails (its++ problem+json: `tools.job.failed`, `tools.pdf.wrong_password`, `tools.net.target_not_public`, …);++ `202 { state: queued | running, tool, job, location }` + `Location: /api/v1/jobs/:id` when a job tool has not++ finished within `wait_ms` (default 0); `200 + Idempotent-Replayed: true` for a replay.++- **Refusals** (problem+json, before the tool runs): `400 tools.run.invalid`, `401 token.missing |++ token.invalid | tools.session_required`, `403 capability.denied | tools.origin.refused`,++ `404 tools.tool.not_found | tools.tool.not_runnable` (`api: false`: the YouTube downloader, the regex tester,++ the canvas makers…) `| tools.run.file_not_found`, `405`, `409 tools.job.idempotency_conflict`,++ `413 tools.file.too_large | tools.input.too_large` (over `limits.maxInputBytes`), `415++ tools.file.unsupported_type` (the bytes are checked), `422 tools.input.invalid` (`errors[]` with JSON++ pointers, against the descriptor's input schema), `429 tools.quota.exceeded` (`Retry-After`), `503++ tools.tool.unavailable` (the descriptor's status is `unavailable`: traceroute, or Protect PDF while qpdf is++ missing), `503 tools.unavailable` (the engine found a program it needs missing: e.g. an HEIC upload without++ libheif), `503 tools.busy` (`Retry-After`).++- **Who.** Always enforced (authorization, not a quota): an app or service token needs the descriptor's++ `auth.capability`; the network probes (port, ping, latency) run only for tokens holding `tools.net.probe`++ (anonymous `401`, a person `403`: people use the probe pages); `auth.anonymous: false` tools (audio, PDF, SMTP)++ need a browser session, a sign-in or a token; a Bearer token that does not verify is `401 token.invalid`, not++ anonymity. Then the guard: the tool's `quotaClass` × the caller's tier, weighted by `cost` (mode-dependent),++ the per-target throttle for egress tools and upload sniffing (hard).++- **Where it runs.** Dev and text tools with a server engine: the page's own engine code in the gateway's++ worker pool (`TOOLS_WORKERS_GATEWAY`, default 2, 256 MB each; the descriptor's `timeoutMs` terminates a++ runaway input). Net tools and Open Graph: the same `/api/net` and `/api/dev` route the page calls, in++ process, with the run's input as its query (`myip` is the caller's address). Job tools (img, audio, docs): a++ job of the satellite that owns the tool (`{ ...input, ...preset, tool: operation }`), the run streamed there++ by the gateway, uploads included; the job's `tool` names the tool.++- **Cache and replays.** An inline answer is reused per tool and input where that is safe (spec `cacheTtlMs`:++ engines 10 minutes unless random or time-dependent — uuid, lorem, timestamp, cron, jwt, zalgo, case, sort, bio,++ nickname never; DNS 1 minute, IP 10, RDAP/whois 60, a few other lookups 5; probes, headers, redirects, uptime,++ SMTP and myip never), answered with `X-OV-Cache: hit` and without touching the target. An inline run's++ `Idempotency-Key` replays its first answer for 15 minutes (per caller); a job tool's key is its job's.++- **CORS.** First-party pages keep the credentialed rules; any other site's page gets `Access-Control-Allow-Origin: *`++ without credentials on the run and jobs routes (so its calls carry a token, never our cookies).++- `GET /api/health` on the gateway shows `run` (counts, cache, engine pool) and `jobs` (the facade).++++## Jobs facade (gateway)++++`https://openvibe.tools/api/v1/jobs…` fronts every satellite's job routes (below), so one origin serves the++run API's `Location`, `openvibe-sdk/jobs` without a `baseUrl`, and anything else. A submit is routed by its job++type's prefix (`img.process` → img …): `?type=`, the JSON body's `type`, or the multipart `type` part (read from++the first 64 KB of the stream — send it before the files; the uploads then stream through untouched). Every other++route (`GET`/`DELETE /:id`, `/:id/events`, `/:id/files/:n`, `/:id/retry`, `/:id/references/:ref`) is routed by the++job id: the satellite that answered its submit or run (remembered, 50,000 ids), else the one that says it holds++the job when asked on loopback (`GET /api/internal/jobs/:id`, direct loopback callers only, never rate++limited). Events (SSE) and files stream through. Owners, quotas, the Origin check and idempotency stay the++satellite's: the request reaches it with the caller's credentials and address.++++## Older endpoints (deprecated, still answered)++++`/api/process`, `/api/process/direct`, `/api/process/multi` (img, audio, docs), `/api/net/*` and `/api/dev/*`++keep their response shapes and add `Deprecation: true`, `Sunset: Thu, 31 Dec 2026 23:59:59 GMT` and++`Link: </api/v1/tools/{id}/run>; rel="successor-version"` (the host's own tool; `/api/net/tools` and++`/api/dev/tools` point at `/api/v1/tools?family=…`). The webhook bins (`/api/dev/webhook/…`, page-only) have no++successor and are not marked.++## Jobs (Img, Audio, Docs)Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same@@@ -227,9 +296,17 @@ One module used by the gateway and every satellite, driven by each tool's descri`tools_guard_enforcing`, `tools_guard_sync{kind}`.- **Challenge hook.** `createGuard({ challenge })` takes a person-check provider (`required`, `verify`; Turnstilelater); the default challenges nobody.++- **Cross-site requests (CSRF).** A mutation that rides on the browser's cookies (`ov_token`, the jobs session)++ must come from an OpenVibe.Tools page: `Origin` (else `Referer`) `https://openvibe.tools`,++ `https://*.openvibe.tools`, one of the tool's own hosts (custom domains included) or the request's own host.++ Calls with a Bearer token, calls without those cookies and requests with neither header pass. Checked on the++ run API, the job routes' writes, `/api/process…`, `/api/info`, `/api/probe` and the webhook bins' create and++ delete; `403 tools.origin.refused`, recorded as reason `origin`.- **Mode.** `TOOLS_GUARD=report` (default) records and counts what it would refuse and refuses nothing, and theapps' older express-rate-limit limiters stay in force (now keyed by the resolved caller, after sign-in is read).-- `TOOLS_GUARD=enforce` refuses and the older limiters step aside. Hard limits apply in both modes: the pixel limit,++ `TOOLS_GUARD=enforce` refuses and the older limiters step aside. (On `/api/v1/…` an older limiter's refusal is++ problem+json `tools.quota.exceeded`, `scope: "legacy"`, with `Retry-After`; job runs through the run API pass the++ same burst and processing limiters as job submits.) Hard limits apply in both modes: the pixel limit,ffmpeg's whitelists and duration cap, upload sniffing, the port-scan cap and the per-target throttle.Default quotas (units = descriptor `cost` per run; `perMinute` / `burst` / `perDay`, 0 = none):@@@ -298,7 +375,14 @@ hosts on every satellite, the registry-driven catalog, and `registry-consistencytool resolves to an operation, endpoint or page that exists, or is marked unavailable. `descriptors.test.js`holds every descriptor to the contracts and to the code (a job's operation, validation, files and limits; a synctool's route), `engines.test.js` runs every browser tool's server engine in plain Node against its output schema,--and `tools-api.test.js` checks the three routes on the gateway and all seven satellites. The audio tests need++and `tools-api.test.js` checks the three routes on the gateway and all seven satellites. `run-api.test.js` runs++the run API on the gateway, img and docs (with a stand-in Network for tokens, `apps/_shared/test/network.js`):++every answer against `tools.run@1`, the refusal codes, probes, quotas, idempotency, job tools and references++across satellites, the Origin check and the deprecation headers; `jobs-facade.test.js` routes jobs through the++gateway (by type, by id after a restart, events, files, retry, cancel); `sdk-client.test.js` drives++openvibe-sdk v0.6.0's `createToolsClient` against the real gateway (it requires a checkout of OpenVibe.SDK next to++this repo, or `OV_SDK_DIR`, and says it skipped without one); `pool.test.js` (shared and docs) hold the worker++pool to its limits. The audio tests need`ffmpeg` and skip without it. The docs and img tests always check the 503 path with qpdf, poppler and libheifpointed at nothing, and also run the real encrypt/decrypt, page rendering and HEIC decoding when those programsare installed (or `QPDF_PATH` / `HEIF_DEC_PATH` point at them). Install first with `npm run install:all`.@@@ -25,6 +25,10 @@// HMAC(address, today's salt), principal or user id, tool, reason — 30 days, no raw// address anywhere; metric tools_guard_refused_total{reason,tool}// challenge a hook for a person-check (Turnstile later; ./challenge.js, a no-op now)++// origin a mutation that rides on the browser's cookies (ov_token, the jobs session) must come++// from an OpenVibe.Tools page: https://openvibe.tools, https://*.openvibe.tools, the tool's++// own hosts (custom domains included) or the request's own host (CSRF); calls with a++// Bearer token and calls without those cookies do not need it. 403 tools.origin.refused//// Mode: TOOLS_GUARD=report (default) logs and counts what it would refuse and refuses nothing, except// the hard limits that apply in both modes: the image pixel limit, ffmpeg's protocol and format@@@ -70,9 +74,9 @@ function uploadsOf(req) {return req.file ? [req.file] : [];}--/** Delete the files multer wrote to disk for a request that is refused. */++/** Delete the files multer wrote to disk for a request that is refused (before the answer goes out). */function discardUploads(req) {-- for (const f of uploadsOf(req)) if (f.path) fs.unlink(f.path, () => {});++ for (const f of uploadsOf(req)) if (f.path) { try { fs.unlinkSync(f.path); } catch { /* already gone */ } }}/**@@@ -223,7 +227,9 @@ function createGuard(o = {}) {}// Liveness and readiness probes (the gateway polls every satellite's) are never counted.const PROBES = new Set(['/api/health', '/api/ready']);-- const isProbe = (req) => req.method === 'GET' && PROBES.has(`${req.baseUrl || ''}${req.path}`);++ // …nor the gateway's loopback question "which satellite holds this job?" (GET /api/internal/jobs/:id).++ const directLoopback = (req) => ip.isLoopback(req.socket && req.socket.remoteAddress) && !req.headers['x-forwarded-for'] && !req.headers['x-real-ip'];++ const isProbe = (req) => req.method === 'GET' && (PROBES.has(`${req.baseUrl || ''}${req.path}`) || (`${req.baseUrl || ''}${req.path}`.startsWith('/api/internal/jobs/') && directLoopback(req)));/*** Middleware: charge the tool's own class and cost (its descriptor). toolOf(req) → a tool id. Records@@@ -349,6 +355,47 @@ function createGuard(o = {}) {return !refuse(req, res, { status: 503, code: 'tools.busy', detail: busy.detail, reason: `busy.${busy.reason}`, tool: toolId, retryAfter: busy.retryAfter || 30 });}++ // ── Cross-site requests (CSRF) ───────────────────────────++ const SAFE = new Set(['GET', 'HEAD', 'OPTIONS']);++ const FIRST_PARTY = /^([a-z0-9-]+\.)*openvibe\.tools$/;++ const production = String(env.NODE_ENV || '') === 'production';++ /** May a page at `origin` use this request's cookies? First-party, the tool's hosts, the same host. */++ function allowedOrigin(origin, req, hosts) {++ let u;++ try { u = new URL(origin); } catch { return false; }++ const hostname = u.hostname.toLowerCase();++ if (u.protocol === 'https:' && FIRST_PARTY.test(hostname)) return true;++ if (u.protocol === 'https:' && (hosts || []).some(h => String(h).toLowerCase() === hostname)) return true;++ const own = String(req.headers['x-forwarded-host'] || req.headers.host || '').toLowerCase();++ if (own && u.host.toLowerCase() === own && (u.protocol === 'https:' || !production)) return true; // same origin++ if (!production && (hostname === 'localhost' || hostname === '127.0.0.1')) return true;++ return false;++ }++ /**++ * A mutation carried by the browser's cookies must come from one of our pages (Origin, else++ * Referer; a request with neither is not a browser's cross-site request). Bearer calls and calls++ * without our cookies pass. Refused in enforce mode (403 tools.origin.refused), recorded in report++ * mode. → ok++ */++ function originOk(req, res, { hosts = [], tool: toolId = null } = {}) {++ if (SAFE.has(req.method)) return true;++ if (/^Bearer\s+\S/i.test(String(req.headers.authorization || ''))) return true;++ const cookies = req.cookies || {};++ if (!cookies.ov_token && !cookies.token && !cookies[callers.cookieName]) return true;++ let origin = req.headers.origin;++ if (origin == null || origin === '') {++ const ref = req.headers.referer;++ if (!ref) return true;++ try { origin = new URL(String(ref)).origin; } catch { origin = 'null'; }++ }++ if (origin !== 'null' && allowedOrigin(String(origin), req, hosts)) return true;++ return !refuse(req, res, { status: 403, code: 'tools.origin.refused', reason: 'origin', tool: toolId, detail: 'This request came from another site with your OpenVibe.Tools cookies. Send it from an OpenVibe.Tools page, or with an API token.', extra: { origin: String(origin).slice(0, 200) } });++ }++ /** Middleware form: hostsOf(req) → the tool's own hosts. */++ function originCheck(hostsOf, toolOf) {++ return (req, res, next) => (originOk(req, res, { hosts: hostsOf ? hostsOf(req) : [], tool: toolOf ? toolOf(req) : null }) ? next() : undefined);++ }++// ── Egress ───────────────────────────────────────────────/** Per-target throttle (hard). → ok */function target(req, res, { tool: toolId, target: t, perMinute }) {@@@ -391,6 +438,18 @@ function createGuard(o = {}) {keyGenerator: (req) => { const c = caller(req); return signed(c) || c.tier === 'sandbox' ? c.key : c.ipKey; },skip: (req) => enforcing || isProbe(req), // liveness and readiness probes are never limitedmessage: { error: message },++ // The platform API (/api/v1/…) answers like the guard does: problem+json tools.quota.exceeded++ // with Retry-After; the older routes keep their { error } shape.++ handler: (req, res, _next, options) => {++ const reset = req.rateLimit && req.rateLimit.resetTime ? new Date(req.rateLimit.resetTime).getTime() : 0;++ const retry = Math.max(1, Math.ceil(((reset || (now() + windowMs)) - now()) / 1000));++ res.setHeader('Retry-After', String(retry));++ if (/^\/api\/v1\//.test(String(req.originalUrl || req.url || ''))) {++ const c = caller(req);++ return sendProblem(req, res, 429, 'tools.quota.exceeded', message, { scope: 'legacy', tier: c.tier, retry_after: retry }, o.contracts);++ }++ return res.status(options.statusCode).json(options.message);++ },});}/** The three limiters an upload app had (LEGACY numbers in ./limits.js). */@@@ -440,6 +499,7 @@ function createGuard(o = {}) {heavy, sync, jobsBusy,target, portScan, targets, ports, normalizeTarget,legacyLimiter, legacyLimiters, toolRefused,++ originOk, originCheck, allowedOrigin, tokens,attachMetrics, prune,close() { if (pruneTimer) clearInterval(pruneTimer); if (keys.stop) keys.stop(); store.close(); },};@@@ -18,6 +18,8 @@// only what came after it; 204 once there is nothing left to say// GET /api/v1/jobs/:id/files/:n a result file (attachment; ?inline=1 for previews)// GET /js/ov-jobs.js the browser helper (submit, watch, reattach)++// GET /api/internal/jobs/:id direct loopback callers only (the gateway's jobs facade asking++// which satellite holds a job): 200 { id, service } | 404//// Owner scoping: a job is visible only to whoever created it —// a Network user (user:usr_…, from the ov_token subject_id), a service/app principal presenting a@@@ -31,6 +33,7 @@ const path = require('path');const { Readable } = require('stream');const { createCallerResolver, jobOwnerResolver, JOB_CAPS: CAPS } = require('../guard/caller');++const { isLoopback } = require('../guard/ip');const ID_RE = /^job_[0-9A-HJKMNP-TV-Z]{26}$/;const TERMINAL = new Set(['succeeded', 'failed', 'cancelled']);@@@ -77,6 +80,8 @@ function createOwnerResolver(o) {* without it a busy store answers 503 tools.busy* @param {(req, res, full) => boolean} [o.onAddressFull] a browser session's address already has its* unfinished jobs (system.addressFull): true = go on (report mode); without it 429++ * @param {Function} [o.originCheck] middleware for the writes (submit, cancel, retry, references): the++ * guard's Origin check on cookie-authenticated calls (CSRF)*/function mountJobRoutes(app, o) {const { system, contracts } = o;@@@ -99,6 +104,19 @@ function mountJobRoutes(app, o) {return row;}++ const origin = o.originCheck || ((_req, _res, next) => next());++++ // Which satellite holds a job? Only for a caller on this host with no proxy in between (the++ // gateway's facade); through nginx or the gateway's own proxy it is not found. Says nothing but++ // whether the id exists here.++ app.get('/api/internal/jobs/:id', (req, res) => {++ res.set('Cache-Control', 'no-store');++ const direct = isLoopback(req.socket && req.socket.remoteAddress) && !req.headers['x-forwarded-for'] && !req.headers['x-real-ip'];++ const id = String(req.params.id || '');++ if (!direct || !ID_RE.test(id) || !system.get(id)) return send(res, 404, 'tools.job.not_found', 'No such job');++ return res.json({ id, service: system.service || null });++ });++app.get('/js/ov-jobs.js', (_req, res) => {res.set({ 'Content-Type': 'application/javascript; charset=utf-8', 'Cache-Control': 'no-cache' });res.sendFile(CLIENT_JS);@@@ -126,7 +144,7 @@ function mountJobRoutes(app, o) {return send(res, 503, 'tools.busy', busy.detail);};-- app.post('/api/v1/jobs', ...(o.limiters || []), notBusy, multipartOnly, async (req, res) => {++ app.post('/api/v1/jobs', origin, ...(o.limiters || []), notBusy, multipartOnly, async (req, res) => {noStore(res);const w = who(req, res, { create: true, action: 'create' });if (!w) return;@@@ -173,7 +191,7 @@ function mountJobRoutes(app, o) {if (row) res.json(system.view(row));});-- app.delete('/api/v1/jobs/:id', (req, res) => {++ app.delete('/api/v1/jobs/:id', origin, (req, res) => {noStore(res);const row = load(req, res, 'cancel');if (!row) return;@@@ -183,7 +201,7 @@ function mountJobRoutes(app, o) {});// Retry and references are writes: a principal needs tools.job.create, like a submit.-- app.post('/api/v1/jobs/:id/retry', ...(o.limiters || []), notBusy, (req, res) => {++ app.post('/api/v1/jobs/:id/retry', origin, ...(o.limiters || []), notBusy, (req, res) => {noStore(res);const row = load(req, res, 'create');if (!row) return;@@@ -214,8 +232,8 @@ function mountJobRoutes(app, o) {}};}-- app.put('/api/v1/jobs/:id/references/:ref', referenceRoute((id, ref) => system.reference(id, ref)));-- app.delete('/api/v1/jobs/:id/references/:ref', referenceRoute((id, ref) => system.unreference(id, ref)));++ app.put('/api/v1/jobs/:id/references/:ref', origin, referenceRoute((id, ref) => system.reference(id, ref)));++ app.delete('/api/v1/jobs/:id/references/:ref', origin, referenceRoute((id, ref) => system.unreference(id, ref)));app.get('/api/v1/jobs/:id/events', (req, res) => {const row = load(req, res, 'read');@@@ -110,6 +110,7 @@ function setupJobs(o) {mountJobRoutes(o.app, {system, contracts: o.contracts, resolveOwner, receive: o.receive, limiters: o.limiters, defaults: o.defaults,...(g && {++ originCheck: g.originCheck(),admit: (req, res, job) => g.admitJob(req, res, { ...job, toolId: o.jobTool ? o.jobTool(req, job.type, job.input) : null }),onBusy: (req, res, busy) => g.jobsBusy(req, res, busy, o.jobTool ? o.jobTool(req, null, {}) : null),onAddressFull: (req, res, full) => !g.refuse(req, res, { status: 429, code: 'tools.job.too_many_active', reason: 'jobs.address', tool: o.jobTool ? o.jobTool(req, null, {}) : null, retryAfter: 30, detail: `At most ${full.limit} unfinished jobs from one address at a time; wait for one to finish.`, extra: { scope: 'address' } }),@@@ -96,9 +96,11 @@ function createPool(o) {counts.started++;w.worker.on('message', (msg) => settle(w, msg));w.worker.on('error', (err) => {-- // An uncaught throw or the heap limit: the worker is gone either way.++ // An uncaught throw or the heap limit: the worker is gone either way. It leaves the pool now++ // (its 'exit' may come later, or not at all), so no new run is ever handed to it.const t = w.task;w.task = null;++ retire(w);if (!t) return;if (err && err.code === 'ERR_WORKER_OUT_OF_MEMORY') {counts.out_of_memory++;@@@ -108,21 +110,35 @@ function createPool(o) {}});w.worker.on('exit', () => {-- workers.delete(w);-- clearTimeout(w.idleTimer);-- if (w.task) { const t = w.task; w.task = null; finish(t, fail('The tool stopped unexpectedly while working on this file.', { status: 500, code: 'tools.job.failed' })); }-- dispatch();++ const t = w.task;++ w.task = null;++ retire(w);++ if (t) finish(t, fail('The tool stopped unexpectedly while working on this file.', { status: 500, code: 'tools.job.failed' }));++ });++ w.worker.on('messageerror', () => {++ const t = w.task;++ w.task = null;++ retire(w);++ if (t) finish(t, fail('The tool\'s answer could not be read.', { status: 500, code: 'tools.job.failed' }));});workers.add(w);return w;}-- function kill(w) {-- w.task = null;++ /** Take a worker out of the pool for good and make sure its thread ends. */++ function retire(w) {++ if (w.dead) return;++ w.dead = true;workers.delete(w);clearTimeout(w.idleTimer);-- counts.killed++;w.worker.terminate().catch(() => {});++ setImmediate(dispatch);++ }++++ function kill(w) {++ w.task = null;++ counts.killed++;++ retire(w);}function finish(t, err, result) {@@@ -151,7 +167,7 @@ function createPool(o) {clearTimeout(w.idleTimer);w.worker.unref();if (idleMs > 0) {-- w.idleTimer = setTimeout(() => { if (!w.task && !queue.length) { workers.delete(w); w.worker.terminate().catch(() => {}); } }, idleMs);++ w.idleTimer = setTimeout(() => { if (!w.task && !queue.length) retire(w); }, idleMs);if (w.idleTimer.unref) w.idleTimer.unref();}}@@@ -186,7 +202,7 @@ function createPool(o) {function dispatch() {if (closed) return;while (queue.length) {-- let w = [...workers].find(x => !x.task);++ let w = [...workers].find(x => !x.task && !x.dead);if (!w) {if (workers.size >= size) return;w = spawn();@@@ -239,7 +255,7 @@ function createPool(o) {async function close() {closed = true;for (const t of queue.splice(0)) finish(t, fail('The worker pool is shutting down', { status: 503, code: 'tools.busy', retryAfter: 5 }));-- await Promise.all([...workers].map(w => { const t = w.task; w.task = null; if (t) finish(t, fail('The worker pool is shutting down', { status: 503, code: 'tools.busy', retryAfter: 5 })); clearTimeout(w.idleTimer); return w.worker.terminate().catch(() => {}); }));++ await Promise.all([...workers].map(w => { const t = w.task; w.task = null; w.dead = true; if (t) finish(t, fail('The worker pool is shutting down', { status: 503, code: 'tools.busy', retryAfter: 5 })); clearTimeout(w.idleTimer); return w.worker.terminate().catch(() => {}); }));workers.clear();}@@@ -67,6 +67,7 @@ const ADDED_COLUMNS = [['retried_by', 'TEXT'], // set on a failed job once it has been retried: the retry's id['ip_key', 'TEXT'], // a browser session's job: the guard's hashed address key (never the address), so// unfinished jobs are also bounded per address (a new cookie is no new allowance)++ ['tool', 'TEXT'], // the tool (tools.tool@1 id) whose run created the job (POST /api/v1/tools/:id/run)];const parse = (s, fallback) => { if (s == null) return fallback; try { return JSON.parse(s); } catch { return fallback; } };@@@ -80,11 +81,13 @@ function createStore(db) {db.exec("CREATE INDEX IF NOT EXISTS tool_jobs_ip ON tool_jobs(ip_key, state) WHERE ip_key IS NOT NULL");const q = {-- insert: db.prepare(`INSERT INTO tool_jobs (id, type, type_version, owner, state, input_json, files_json, idempotency_key, request_hash, max_attempts, ttl_ms, created_at, updated_at, env, retry_of, ip_key)-- VALUES (@id, @type, @type_version, @owner, 'queued', @input_json, @files_json, @idempotency_key, @request_hash, @max_attempts, @ttl_ms, @now, @now, @env, @retry_of, @ip_key)`),++ insert: db.prepare(`INSERT INTO tool_jobs (id, type, type_version, owner, state, input_json, files_json, idempotency_key, request_hash, max_attempts, ttl_ms, created_at, updated_at, env, retry_of, ip_key, tool)++ VALUES (@id, @type, @type_version, @owner, 'queued', @input_json, @files_json, @idempotency_key, @request_hash, @max_attempts, @ttl_ms, @now, @now, @env, @retry_of, @ip_key, @tool)`),markRetried: db.prepare("UPDATE tool_jobs SET retried_by = @next, updated_at = @now WHERE id = @id AND state = 'failed' AND retried_by IS NULL"),get: db.prepare('SELECT * FROM tool_jobs WHERE id = ?'),byIdem: db.prepare('SELECT * FROM tool_jobs WHERE owner = ? AND idempotency_key = ?'),++ // A result file stored in Media, by its media_id (a run's { media_id } reference); the owner's only.++ byResultMedia: db.prepare("SELECT * FROM tool_jobs WHERE owner = ? AND state = 'succeeded' AND result_json LIKE ? ORDER BY id DESC LIMIT 5"),nextQueued: db.prepare("SELECT * FROM tool_jobs WHERE state = 'queued' ORDER BY id LIMIT ?"),claim: db.prepare("UPDATE tool_jobs SET state = 'running', attempts = attempts + 1, started_at = @now, updated_at = @now, progress = NULL, progress_message = NULL WHERE id = @id AND state = 'queued'"),running: db.prepare("SELECT * FROM tool_jobs WHERE state = 'running'"),@@@ -124,7 +127,8 @@ function createStore(db) {return {db,TERMINAL, STATES,-- insert(row) { q.insert.run({ retry_of: null, ip_key: null, ...row }); },++ insert(row) { q.insert.run({ retry_of: null, ip_key: null, tool: null, ...row }); },++ byResultMedia: (owner, mediaId) => q.byResultMedia.all(owner, `%"media_id":"${String(mediaId).replace(/[%"\\]/g, '')}"%`),markRetried: (id, next, now = Date.now()) => q.markRetried.run({ id, next, now }).changes === 1,get: (id) => q.get.get(id) || null,byIdempotencyKey: (owner, key) => q.byIdem.get(owner, key) || null,@@@ -151,6 +151,7 @@ function createJobSystem(o) {id: row.id,object: 'tools.job',service: o.service,++ ...(row.tool && { tool: row.tool }),type: row.type,type_version: row.type_version,state: row.state,@@@ -202,7 +203,7 @@ function createJobSystem(o) {// Developer-app sandbox jobs: fewer at a time, kept briefly, results never leave this server.const SANDBOX_MAX_ACTIVE = Math.max(1, o.sandboxMaxActivePerOwner || 2);const SANDBOX_TTL_MS = 30 * 60 * 1000;-- async function submit({ owner, type, input = {}, files = [], idempotencyKey = null, ttlMs = HOUR, env = 'production', ipKey = null }) {++ async function submit({ owner, type, input = {}, files = [], idempotencyKey = null, ttlMs = HOUR, env = 'production', ipKey = null, tool = null }) {if (env !== 'sandbox') env = 'production';if (env === 'sandbox') ttlMs = Math.min(ttlMs, SANDBOX_TTL_MS);if (stopped) throw new JobError(503, 'tools.job.unavailable', 'The job system is shutting down');@@@ -252,6 +253,7 @@ function createJobSystem(o) {idempotency_key: idempotencyKey == null ? null : String(idempotencyKey), request_hash: requestHash,max_attempts: def.maxAttempts, ttl_ms: ttlMs, now: Date.now(), env,ip_key: String(owner).startsWith('session:') && ipKey ? String(ipKey) : null,++ tool: tool && /^[a-z][a-z0-9-]{0,39}$/.test(String(tool)) ? String(tool) : null,});announce.created(store.get(id));})();@@@ -474,7 +476,7 @@ function createJobSystem(o) {store.insert({id: next, type: row.type, type_version: def.version, owner: row.owner, input_json: row.input_json, files_json: JSON.stringify(stored),idempotency_key: null, request_hash: row.request_hash, max_attempts: def.maxAttempts, ttl_ms: row.ttl_ms, now: Date.now(),-- env: row.env, retry_of: row.id, ip_key: row.ip_key || null,++ env: row.env, retry_of: row.id, ip_key: row.ip_key || null, tool: row.tool || null,});announce.created(store.get(next));})();@@@ -664,6 +666,16 @@ function createJobSystem(o) {/** Run fn when the system stops; → a function that removes it. */function onStop(fn) { stopHooks.add(fn); return () => stopHooks.delete(fn); }++ /** The owner's succeeded job with a result file stored as Media object `mediaId` → { row, index } or null. */++ function findResultMedia(owner, mediaId) {++ for (const row of store.byResultMedia(owner, mediaId)) {++ const files = (store.parse(row.result_json, null) || {}).files || [];++ const index = files.findIndex(f => f.media && f.media.media_id === mediaId);++ if (index >= 0) return { row, index };++ }++ return null;++ }++/** Where a result file's bytes are: { local: path } or { media: MediaRef }. */function resultFile(row, n) {const result = store.parse(row.result_json, null);@@@ -673,9 +685,10 @@ function createJobSystem(o) {}const api = {-- define, submit, cancel, retry, reference, unreference, subscribe, start, stop, onStop, prune, recover, view, resultFile,++ define, submit, cancel, retry, reference, unreference, subscribe, start, stop, onStop, prune, recover, view, resultFile, findResultMedia,busy, refreshDisk, addressFull, bounds: { maxQueued, diskBudgetBytes: diskBudget, maxActivePerAddress },get: (id) => store.get(id),++ service: o.service,eventsAfter: (id, seq) => store.eventsAfter(id, seq),lastSeq: (id) => store.lastSeq(id),types: () => [...types.keys()],@@@ -352,6 +352,54 @@ async function serve(app) {g.close();}++ // ── Cross-site requests (CSRF): cookie-carried mutations must come from our pages ──++ {++ const res = () => ({ statusCode: 200, headers: {}, body: null, headersSent: false, setHeader(k, v) { this.headers[k.toLowerCase()] = v; }, end(b) { this.body = JSON.parse(b); } });++ const post = (headers, cookies = { ov_tools_jobs: 'a'.repeat(32) }, method = 'POST') => ({ method, ip: '198.51.100.60', headers: { host: 'png.openvibe.tools', ...headers }, cookies });++ for (const mode of ['report', 'enforce']) {++ const g = makeGuard({ mode });++ const ok = (req, hosts) => g.originOk(req, res(), { hosts, tool: 'png' });++ // Allowed: our pages, the tool's own hosts, the same host, no cookies, a Bearer token, safe methods, no Origin at all.++ for (const o of ['https://openvibe.tools', 'https://png.openvibe.tools', 'https://img.openvibe.tools', 'https://a.b.openvibe.tools']) assert.ok(ok(post({ origin: o })), `${mode}: ${o}`);++ assert.ok(ok(post({ origin: 'https://bestpngconverter.example' }), ['bestpngconverter.example']), `${mode}: the tool's own custom domain`);++ assert.ok(ok(post({ origin: 'https://mydomain.example', host: 'mydomain.example' })), `${mode}: the same host`);++ assert.ok(ok(post({ origin: 'https://evil.example' }, {})), `${mode}: no cookies, nothing to forge`);++ assert.ok(ok(post({ origin: 'https://evil.example', authorization: 'Bearer x.y.z' })), `${mode}: a token, not a cookie`);++ assert.ok(ok(post({ origin: 'https://evil.example' }, undefined, 'GET')), `${mode}: GET is not a mutation`);++ assert.ok(ok(post({})), `${mode}: neither Origin nor Referer: not a browser's cross-site request`);++ // Refused (enforce) or only recorded (report).++ const cases = [++ post({ origin: 'https://evil.example' }),++ post({ origin: 'http://img.openvibe.tools' }), // not https++ post({ origin: 'https://openvibe.tools.evil.example' }),++ post({ origin: 'null' }), // a sandboxed frame, a data: page++ post({ referer: 'https://evil.example/form' }), // no Origin, a foreign Referer++ post({ origin: 'https://evil.example' }, { ov_token: 'x.y.z' }), // the sign-in cookie++ ];++ for (const req of cases) {++ const out = res();++ const allowed = g.originOk(req, out, { tool: 'png' });++ if (mode === 'enforce') {++ assert.strictEqual(allowed, false, JSON.stringify(req.headers));++ assert.deepStrictEqual([out.statusCode, out.body.code], [403, 'tools.origin.refused']);++ } else {++ assert.strictEqual(allowed, true, `report mode records ${JSON.stringify(req.headers)}`);++ assert.strictEqual(out.body, null);++ }++ }++ const logged = g.store.abuseRows().filter(x => x.reason === 'origin').reduce((n, x) => n + x.count, 0);++ assert.strictEqual(logged, cases.length, `${mode}: every refusal is in the abuse log`);++ g.close();++ }++ // localhost pages outside production (dev servers), never in production.++ const dev = makeGuard({ mode: 'enforce', env: { NODE_ENV: 'development' } });++ assert.ok(dev.originOk(post({ origin: 'http://localhost:5173' }), res()));++ dev.close();++ const prod = makeGuard({ mode: 'enforce', env: { NODE_ENV: 'production' } });++ assert.strictEqual(prod.originOk(post({ origin: 'http://localhost:5173' }), res()), false);++ prod.close();++ }++// ── The sync semaphore holds heavy calls to the cap ──{const sem = createSemaphore({ max: 2, queue: 3, waitMs: 200 });@@@ -466,5 +514,5 @@ async function serve(app) {system.stop(); db.close(); fs.rmSync(dataDir, { recursive: true, force: true });}-- console.log('guard: callers (/64, one loopback hop, services by token, aud-checked users, sessions, sandbox), bucket math + cost, session sharing, report vs enforce, headers + problem+json, day counters across restarts, sniffing, target throttle, port cap, abuse log + prune + metric, semaphore, job bounds (queue, disk, per address), challenge hook: all checks passed');++ console.log('guard: callers (/64, one loopback hop, services by token, aud-checked users, sessions, sandbox), bucket math + cost, session sharing, report vs enforce, headers + problem+json, day counters across restarts, sniffing, target throttle, port cap, abuse log + prune + metric, semaphore, job bounds (queue, disk, per address), challenge hook, Origin check (CSRF): all checks passed');})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,161 @@++'use strict';++// The gateway's /api/v1/jobs facade (apps/gateway/server/run/jobs-facade.js) on real processes: the++// gateway in front of img and docs, a stand-in Network for tokens.++// • submits are routed by the job type's prefix: JSON, a multipart `type` part read from the head of++// the stream (uploads stream on), or ?type=; a type it cannot find or does not know is refused++// • everything else is routed by the job id: the satellite that answered (remembered), else the one++// that says it holds the job on loopback (after a gateway restart); reads, cancel, retry (the new++// job is remembered), references, SSE events and result files all stream through++// • owners stay the satellite's business (another person gets 404), unknown ids are 404++// • the loopback probe (GET /api/internal/jobs/:id) answers direct callers only and is never limited++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const { startApp, freePort } = require('./spawn');++const { startNetwork } = require('./network');++++const APPS = path.join(__dirname, '..', '..');++const sharp = require(require.resolve('sharp', { paths: [path.join(APPS, 'img')] }));++const sleep = (ms) => new Promise(r => setTimeout(r, ms));++++(async () => {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-jobs-facade-'));++ const net = await startNetwork();++ const procs = [];++ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };++ try {++ const ports = { img: await freePort(), docs: await freePort(), gw: await freePort() };++ const env = { OV_NETWORK_URL: net.url, OV_NETWORK_INTERNAL_URL: net.url, TOOLS_SATELLITE_PORTS: `img=${ports.img},docs=${ports.docs}` };++ const img = await start('img', { ...env, DATA_DIR: path.join(tmp, 'img'), UPLOADS_DIR: path.join(tmp, 'img', 'up'), OUTPUT_DIR: path.join(tmp, 'img', 'out') }, ports.img);++ // docs never starts a job here: its jobs stay queued (cancel).++ const docs = await start('docs', { ...env, DATA_DIR: path.join(tmp, 'docs'), UPLOADS_DIR: path.join(tmp, 'docs', 'up'), OUTPUT_DIR: path.join(tmp, 'docs', 'out'), TOOLS_JOBS_CONCURRENCY_DOCS: '0' }, ports.docs);++ const gwEnv = { ...env, DATA_DIR: path.join(tmp, 'gw'), OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' };++ let gw = await start('gateway', gwEnv, ports.gw);++ const G = gw.base;++ const me = { Authorization: `Bearer ${net.user()}` };++ const other = { Authorization: `Bearer ${net.user()}` };++ const req = async (p, { method = 'GET', headers = {}, body, base = G } = {}) => {++ const r = await fetch(`${base}${p}`, { method, headers, body });++ const text = await r.text();++ let json = null; try { json = text ? JSON.parse(text) : null; } catch { /* not JSON */ }++ return { status: r.status, h: r.headers, text, body: json };++ };++ const png = await sharp({ create: { width: 6, height: 6, channels: 3, background: '#00f' } }).png().toBuffer();++ const form = (parts) => {++ const fd = new FormData();++ for (const [k, v] of parts) { if (v instanceof Blob) fd.append(k, v, 'x.png'); else fd.append(k, String(v)); }++ return fd;++ };++ const finished = async (id, headers) => {++ for (let i = 0; i < 100; i++) {++ const r = await req(`/api/v1/jobs/${id}`, { headers });++ if (['succeeded', 'failed', 'cancelled'].includes(r.body.state)) return r.body;++ await sleep(50);++ }++ throw new Error(`${id} did not finish`);++ };++++ // ── Submit, routed by the multipart type part ──++ let r = await req('/api/v1/jobs', { method: 'POST', headers: me, body: form([['type', 'img.process'], ['input', JSON.stringify({ tool: 'convert', format: 'webp' })], ['file', new Blob([png])]]) });++ assert.strictEqual(r.status, 202, r.text);++ assert.strictEqual(r.body.service, 'img');++ assert.strictEqual(r.h.get('location'), `/api/v1/jobs/${r.body.id}`);++ const job = r.body.id;++ const done = await finished(job, me);++ assert.strictEqual(done.state, 'succeeded', JSON.stringify(done.error));++ // Events stream through (SSE, ids, the terminal event).++ {++ const res = await fetch(`${G}/api/v1/jobs/${job}/events`, { headers: { ...me, Accept: 'text/event-stream' } });++ // Finished and nothing newer than Last-Event-ID 0? It replays the whole log first.++ assert.strictEqual(res.status, 200);++ assert.match(res.headers.get('content-type'), /text\/event-stream/);++ const text = await res.text();++ assert.match(text, /^retry: 3000/);++ assert.match(text, /id: \d+\nevent: job\.succeeded\n/);++ const last = Math.max(...[...text.matchAll(/^id: (\d+)$/gm)].map(m => Number(m[1])));++ const again = await fetch(`${G}/api/v1/jobs/${job}/events`, { headers: { ...me, 'Last-Event-ID': String(last) } });++ assert.strictEqual(again.status, 204, 'nothing after the last event: 204');++ }++ // The result file streams through.++ r = await fetch(`${G}/api/v1/jobs/${job}/files/0`, { headers: me });++ assert.strictEqual(r.status, 200);++ assert.match(r.headers.get('content-disposition'), /attachment; filename="x\.webp"/);++ assert.strictEqual((await sharp(Buffer.from(await r.arrayBuffer())).metadata()).format, 'webp');++ // References through the facade.++ r = await req(`/api/v1/jobs/${job}/references/community:paste:p_1`, { method: 'PUT', headers: me });++ assert.strictEqual(r.status, 201, r.text);++ assert.strictEqual(r.body.expires_at, null);++ r = await req(`/api/v1/jobs/${job}/references/community:paste:p_1`, { method: 'DELETE', headers: me });++ assert.strictEqual(r.status, 200);++ assert.ok(r.body.expires_at);++ // Somebody else: the satellite's 404, as for a job that does not exist.++ r = await req(`/api/v1/jobs/${job}`, { headers: other });++ assert.deepStrictEqual([r.status, r.body.code], [404, 'tools.job.not_found']);++ r = await req('/api/v1/jobs/job_01J00000000000000000000000', { headers: me });++ assert.deepStrictEqual([r.status, r.body.code], [404, 'tools.job.not_found']);++ assert.strictEqual((await req('/api/v1/jobs/not-a-job', { headers: me })).status, 404);++++ // ── A failed job, retried through the facade (the new job is remembered) ──++ const broken = Buffer.concat([png.subarray(0, 40), Buffer.alloc(200, 7)]); // a PNG header, then noise++ r = await req('/api/v1/jobs', { method: 'POST', headers: me, body: form([['type', 'img.process'], ['input', JSON.stringify({ tool: 'convert', format: 'png' })], ['file', new Blob([broken])]]) });++ assert.strictEqual(r.status, 202, r.text);++ const failed = await finished(r.body.id, me);++ assert.strictEqual(failed.state, 'failed');++ r = await req(`/api/v1/jobs/${failed.id}/retry`, { method: 'POST', headers: me });++ assert.strictEqual(r.status, 202, r.text);++ assert.strictEqual(r.body.retry_of, failed.id);++ const retried = r.body.id;++ assert.strictEqual((await finished(retried, me)).state, 'failed');++++ // ── docs: JSON body type; ?type= when the type part comes late; cancel a queued job ──++ r = await req('/api/v1/jobs', { method: 'POST', headers: { ...me, 'Content-Type': 'application/json' }, body: JSON.stringify({ type: 'docs.process', input: { tool: 'merge' } }) });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.invalid'], 'routed to docs, which wants files');++ assert.match(r.body.detail, /at least 2 files|needs a file/);++ const big = Buffer.alloc(1024 * 1024, 1);++ const pdfish = Buffer.concat([Buffer.from('%PDF-1.4\n'), big]);++ r = await req('/api/v1/jobs', { method: 'POST', headers: me, body: form([['file', new Blob([pdfish])], ['type', 'docs.process'], ['input', JSON.stringify({ tool: 'compress' })]]) });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.invalid'], 'a type part after the first 64 KB of the stream is not looked for');++ r = await req('/api/v1/jobs?type=docs.process', { method: 'POST', headers: me, body: form([['file', new Blob([pdfish])], ['type', 'docs.process'], ['input', JSON.stringify({ tool: 'compress' })]]) });++ assert.strictEqual(r.status, 202, r.text);++ assert.strictEqual(r.body.service, 'docs');++ const queued = r.body.id;++ r = await req(`/api/v1/jobs/${queued}`, { method: 'DELETE', headers: me });++ assert.deepStrictEqual([r.status, r.body.state], [200, 'cancelled']);++ r = await req('/api/v1/jobs', { method: 'POST', headers: { ...me, 'Content-Type': 'application/json' }, body: JSON.stringify({ type: 'video.process' }) });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.unknown_type']);++ r = await req('/api/v1/jobs', { method: 'POST', headers: { ...me, 'Content-Type': 'application/json' }, body: '{"type":' });++ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.invalid']);++ assert.strictEqual((await req('/api/v1/jobs', { headers: me })).status, 405);++++ // ── After a gateway restart: nothing remembered, the satellites are asked ──++ const before = (await req('/api/health')).body.jobs;++ assert.ok(before.known >= 4, JSON.stringify(before));++ await gw.kill('SIGTERM');++ procs.splice(procs.indexOf(gw), 1);++ gw = await start('gateway', gwEnv, ports.gw);++ for (const [id, want] of [[job, 'succeeded'], [retried, 'failed'], [queued, 'cancelled']]) {++ r = await req(`/api/v1/jobs/${id}`, { headers: me });++ assert.deepStrictEqual([r.status, r.body.state], [200, want], `${id} found again`);++ }++ const after = (await req('/api/health')).body.jobs;++ assert.strictEqual(after.probed, 3, 'each asked once');++ await req(`/api/v1/jobs/${job}`, { headers: me });++ assert.strictEqual((await req('/api/health')).body.jobs.probed, 3, 'then remembered');++++ // ── The loopback probe: direct callers only, never limited ──++ r = await req(`/api/internal/jobs/${job}`, { base: img.base });++ assert.deepStrictEqual([r.status, r.body], [200, { id: job, service: 'img' }]);++ assert.strictEqual((await req(`/api/internal/jobs/${job}`, { base: docs.base })).status, 404, 'not on docs');++ assert.strictEqual((await req(`/api/internal/jobs/${job}`, { base: img.base, headers: { 'X-Forwarded-For': '203.0.113.5' } })).status, 404, 'through a proxy: nothing to see');++ const statuses = await Promise.all(Array.from({ length: 150 }, () => req(`/api/internal/jobs/${job}`, { base: img.base }).then(x => x.status)));++ assert.ok(statuses.every(s => s === 200), 'the facade\'s probes are never rate limited');++ // Through the gateway's host proxy it does not exist either.++ r = await req(`/api/internal/jobs/${job}`, { headers: { 'X-Forwarded-For': '203.0.113.5' } });++ assert.strictEqual(r.status, 404);++ } finally {++ for (const p of procs) await p.kill('SIGTERM');++ await net.close();++ }++ console.log('jobs facade: submit routed by type (multipart head, JSON, ?type=), by id (remembered, then probed after a restart), events/files/references/retry/cancel through, owner 404s, loopback probe: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,45 @@++'use strict';++// A stand-in OpenVibe.Network for tests that run real Tools processes: it serves the JWKS endpoint++// (GET /api/.well-known/jwks → { public_key }) and signs the tokens the Network would issue — a++// person's browser sign-in (aud openvibe.tools, subject_id usr_…) and service or app principals++// (client credentials: aud openvibe.tools, cap[]). Point OV_NETWORK_URL and OV_NETWORK_INTERNAL_URL++// at `url`; the issuer is `url` too.++const crypto = require('crypto');++const http = require('http');++++const b64 = (o) => Buffer.from(JSON.stringify(o)).toString('base64url');++++async function startNetwork() {++ const { publicKey, privateKey } = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });++ const srv = http.createServer((req, res) => {++ if (req.url.startsWith('/api/.well-known/jwks')) { res.setHeader('Content-Type', 'application/json'); return res.end(JSON.stringify({ public_key: publicKey, algorithm: 'RS256' })); }++ res.statusCode = 404; return res.end('{}');++ });++ await new Promise(r => srv.listen(0, '127.0.0.1', r));++ const url = `http://127.0.0.1:${srv.address().port}`;++ const now = () => Math.floor(Date.now() / 1000);++ const sign = (claims) => {++ const head = b64({ alg: 'RS256', typ: 'JWT' });++ const body = b64(claims);++ const sig = crypto.sign('RSA-SHA256', Buffer.from(`${head}.${body}`), privateKey).toString('base64url');++ return `${head}.${body}.${sig}`;++ };++ let n = 0;++ const ulid = () => `01J${crypto.randomBytes(16).toString('hex').toUpperCase().replace(/[^0-9A-HJKMNP-TV-Z]/g, 'Z').slice(0, 23)}`;++ return {++ url, issuer: url, publicKey,++ /** A signed-in person (what the Network puts in ov_token). */++ user(extra = {}) {++ n++;++ return sign({ iss: url, sub: String(1000 + n), id: 1000 + n, subject_id: `usr_${ulid()}`, username: `person${n}`, aud: ['openvibe.live', 'openvibe.tools', 'openvibe.network'], iat: now(), exp: now() + 3600, ...extra });++ },++ /** A service principal's client-credentials token with these capabilities. */++ service(cap, extra = {}) {++ return sign({ iss: url, sub: `svc:${extra.name || 'partner'}`, actor_type: 'service', aud: ['openvibe.tools'], cap, ns: [], iat: now(), exp: now() + 900, jti: `tok_${crypto.randomBytes(12).toString('hex')}`, ...extra, name: undefined });++ },++ sign,++ close: () => new Promise(r => srv.close(r)),++ };++}++++module.exports = { startNetwork };nnew file mode 100644@@@ -0,0 +1,32 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// The older tool endpoints (/api/process…, /api/net/*, /api/dev/*) keep working with their own++// response shapes, and say what replaces them (RFC 9745 Deprecation, RFC 8594 Sunset, RFC 8288 Link):++//++// Deprecation: true++// Sunset: Thu, 31 Dec 2026 23:59:59 GMT++// Link: </api/v1/tools/{id}/run>; rel="successor-version"++// ═══════════════════════════════════════════════════════════════++++const SUNSET = 'Thu, 31 Dec 2026 23:59:59 GMT';++++/** Mark an answer deprecated; `successor` is a path (the tool's run route, or a registry listing). */++function deprecate(res, successor) {++ if (res.headersSent) return;++ res.setHeader('Deprecation', 'true');++ res.setHeader('Sunset', SUNSET);++ if (successor) res.setHeader('Link', `<${successor}>; rel="successor-version"`);++}++++/** Middleware: successorOf(req) → a path or null (null: deprecated without a named successor is not said). */++function deprecated(successorOf) {++ return (req, res, next) => {++ const s = successorOf(req);++ if (s) deprecate(res, s);++ next();++ };++}++++const runPath = (id) => (id ? `/api/v1/tools/${id}/run` : null);++++module.exports = { deprecate, deprecated, runPath, SUNSET };@@@ -14,13 +14,15 @@// requires programs the tool needs on the host ('qpdf', 'heif-dec', …); the satellite marks it// unavailable while one is missing// example { input } that runs: used by the tests, later by the docs++// cacheTtlMs how long the run API may reuse an inline answer (0: never — random or time-dependent++// output; unset: engines 10 minutes, routes never)//// Like the rest of apps/_shared this file has no dependencies.// ═══════════════════════════════════════════════════════════════const SITE = 'https://openvibe.tools';const JSON_SCHEMA = 'https://json-schema.org/draft/2020-12/schema';--const INTERNAL = new Set(['id', 'engine', 'route', 'requires', 'example', 'job', 'legacy', 'status', 'statusReason']);++const INTERNAL = new Set(['id', 'engine', 'route', 'requires', 'example', 'job', 'legacy', 'status', 'statusReason', 'cacheTtlMs']);const ANYONE = Object.freeze({ anonymous: true, capability: 'tools.tool.run' });const runPath = (id) => `/api/v1/tools/${id}/run`;@@@ -33,12 +33,38 @@ function openCors(res) {res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');}--/** Wrap an app's CORS middleware: registry paths skip it and are open to every origin instead. */--function exceptRegistry(corsMiddleware) {++// The run API and the jobs API take calls from any site's page with a token (openvibe-sdk in a++// browser): such an origin gets CORS without credentials (so the browser never sends our cookies++// along); first-party pages keep the app's own credentialed CORS.++const RUN_OR_JOBS_RE = /^\/api\/v1\/(?:tools\/[a-z][a-z0-9-]{0,39}\/run|jobs(?:\/.*)?)\/?$/;++const isPublicApiPath = (p) => RUN_OR_JOBS_RE.test(String(p || ''));++const API_HEADERS = 'Authorization, Content-Type, Idempotency-Key, Last-Event-ID, Cache-Control, traceparent, X-OpenVibe-Request-Id';++const API_EXPOSE = 'Location, Idempotent-Replayed, Retry-After, RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, X-OpenVibe-Request-Id, traceparent, Content-Disposition, Deprecation, Sunset, Link';++++/**++ * Wrap an app's CORS middleware: registry paths skip it and are open to every origin instead; run and++ * jobs paths from an origin the app does not know (isFirstParty false) get CORS without credentials.++ * @param {Function} corsMiddleware++ * @param {{ isFirstParty?: (origin) => boolean }} [opts]++ */++function exceptRegistry(corsMiddleware, opts = {}) {return function corsExceptToolRegistry(req, res, next) {-- if (!isRegistryPath(req.path)) return corsMiddleware(req, res, next);-- openCors(res);-- return next();++ if (isRegistryPath(req.path)) { openCors(res); return next(); }++ const origin = req.headers.origin;++ if (opts.isFirstParty && origin && isPublicApiPath(req.path) && !opts.isFirstParty(origin)) {++ res.setHeader('Access-Control-Allow-Origin', '*');++ res.setHeader('Access-Control-Expose-Headers', API_EXPOSE);++ res.setHeader('Vary', 'Origin');++ if (req.method === 'OPTIONS') {++ res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');++ res.setHeader('Access-Control-Allow-Headers', API_HEADERS);++ res.setHeader('Access-Control-Max-Age', '600');++ res.statusCode = 204;++ return res.end();++ }++ return next();++ }++ return corsMiddleware(req, res, next);};}@@@ -158,4 +184,4 @@ function createToolsApi(o) {};}--module.exports = { createToolsApi, exceptRegistry, isRegistryPath, parseFilters, PATH_RE };++module.exports = { createToolsApi, exceptRegistry, isRegistryPath, isPublicApiPath, parseFilters, PATH_RE };nnew file mode 100644@@@ -0,0 +1,96 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// Forward one API request to a Tools app on loopback and stream the answer back (the gateway's run++// API for satellite tools and its /api/v1/jobs facade). Uploads, SSE event streams and result files++// pass through without being buffered. The caller's own address goes along as the one trusted hop++// (X-Forwarded-For = req.ip, as the host proxy does), with its credentials, Idempotency-Key,++// Last-Event-ID and trace headers. Small JSON answers can be read on the way (onJson), e.g. to learn++// which satellite holds a job. The gateway's CORS headers are the ones the browser sees.++// No dependencies (node:http).++// ═══════════════════════════════════════════════════════════════++++const http = require('http');++++const HOP = new Set(['connection', 'keep-alive', 'proxy-authenticate', 'proxy-authorization', 'te', 'trailer', 'transfer-encoding', 'upgrade', 'host']);++const JSON_MAX = 1024 * 1024;++++function outHeaders(req, extra = {}) {++ const headers = {};++ for (const [k, v] of Object.entries(req.headers)) if (!HOP.has(k) && !k.startsWith('x-ov-')) headers[k] = v;++ const client = String(req.ip || (req.socket && req.socket.remoteAddress) || '');++ headers['x-forwarded-for'] = client;++ headers['x-real-ip'] = client;++ headers['cf-connecting-ip'] = client;++ headers['x-forwarded-host'] = String(req.headers['x-forwarded-host'] || req.headers.host || '');++ headers['x-forwarded-proto'] = 'https';++ // The satellite sees the public host (same-origin checks, canonical links) as the gateway did.++ if (req.headers.host) headers.host = req.headers.host;++ return Object.assign(headers, extra);++}++++/**++ * @param {object} req @param {object} res++ * @param {object} o++ * @param {number} o.port++ * @param {string} [o.path] default req.originalUrl++ * @param {string} [o.method] default req.method++ * @param {Buffer} [o.body] the whole body (already read); else `head` + the rest of req++ * @param {Buffer} [o.head] bytes of req already read (a peek), sent before the rest++ * @param {(body, status, headers) => void} [o.onJson] sees a JSON answer (≤ 1 MB) before it is sent++ * @param {number} [o.timeoutMs=600000]++ * @param {() => void} [o.onError] the satellite could not be reached (default: 502 problem)++ * @returns {Promise<void>} settles when the answer has been handed over++ */++function forward(req, res, o) {++ return new Promise((resolve) => {++ const extra = {};++ if (o.body) { extra['content-length'] = String(o.body.length); delete req.headers['transfer-encoding']; }++ const up = http.request({ host: '127.0.0.1', port: o.port, method: o.method || req.method, path: o.path || req.originalUrl || req.url, headers: outHeaders(req, extra) }, (r) => {++ const headers = {};++ for (const [k, v] of Object.entries(r.headers)) if (!HOP.has(k) && !k.startsWith('access-control-')) headers[k] = v;++ const type = String(r.headers['content-type'] || '');++ const len = Number(r.headers['content-length'] || 0);++ if (o.onJson && /json/.test(type) && len <= JSON_MAX) {++ const chunks = [];++ let size = 0;++ r.on('data', (c) => { size += c.length; if (size <= JSON_MAX) chunks.push(c); });++ r.on('end', () => {++ const buf = Buffer.concat(chunks);++ if (size <= JSON_MAX) { try { o.onJson(JSON.parse(buf.toString('utf8')), r.statusCode, r.headers); } catch { /* not JSON after all */ } }++ headers['content-length'] = String(buf.length);++ if (!res.headersSent) res.writeHead(r.statusCode || 502, headers);++ res.end(buf);++ resolve();++ });++ r.on('error', () => { res.destroy(); resolve(); });++ return;++ }++ res.writeHead(r.statusCode || 502, headers);++ if (typeof res.flushHeaders === 'function' && /event-stream/.test(type)) res.flushHeaders();++ r.pipe(res);++ r.on('end', resolve);++ r.on('error', () => { res.destroy(); resolve(); });++ });++ up.setTimeout(o.timeoutMs || 10 * 60_000, () => up.destroy(new Error('upstream timeout')));++ up.on('error', () => {++ if (!res.headersSent) {++ if (o.onError) o.onError();++ else {++ res.statusCode = 502;++ res.setHeader('Content-Type', 'application/problem+json');++ res.setHeader('Retry-After', '5');++ res.end(JSON.stringify({ type: 'https://openvibe.network/problems/tools.unavailable', title: 'Bad Gateway', status: 502, code: 'tools.unavailable', detail: 'This tool is restarting. Try again in a moment.', error: 'This tool is restarting. Try again in a moment.' }));++ }++ } else res.destroy();++ resolve();++ });++ res.on('close', () => { if (!res.writableFinished) up.destroy(); });++ if (o.body) up.end(o.body);++ else {++ if (o.head && o.head.length) up.write(o.head);++ req.pipe(up);++ }++ });++}++++module.exports = { forward, outHeaders };nnew file mode 100644@@@ -0,0 +1,601 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// The uniform run API (ADR-027; openvibe-contracts tools.run-request@1 → tools.run@1):++//++// POST /api/v1/tools/:id/run++//++// Body: JSON { input, files, wait_ms, idempotency_key } (tools.run-request@1), or multipart/form-data++// with the uploads as `file` (or `files`) file parts and the other fields as text parts: `input` (JSON++// text), `files` (the file references' JSON), `wait_ms`, `idempotency_key` — exactly what++// openvibe-sdk/tools v0.6.0 sends. ?wait_ms= on the query does the same as the field; the++// Idempotency-Key header wins over idempotency_key.++//++// Answers (tools.run@1):++// 200 { state: succeeded, tool, result: { data | text | files }, took_ms[, job] }++// 200 { state: failed | cancelled, tool, error (problem+json), took_ms[, job] } the tool's own failure++// 202 { state: queued | running, tool, job, location } + Location a job still going after wait_ms++// 200 + Idempotent-Replayed: true the same key and request again++// Refusals before the tool runs are problem+json: 400 tools.run.invalid, 401 token.missing |++// tools.session_required, 403 capability.denied | tools.origin.refused, 404 tools.tool.not_found |++// tools.tool.not_runnable (api false, e.g. the YouTube downloader) | tools.run.file_not_found,++// 409 tools.job.idempotency_conflict, 413 tools.file.too_large | tools.input.too_large,++// 415 tools.file.unsupported_type, 422 tools.input.invalid (errors[] with JSON pointers),++// 429 tools.quota.exceeded (Retry-After), 503 tools.tool.unavailable (the descriptor's status is++// unavailable), 503 tools.unavailable (a program the tool needs is missing), 503 tools.busy.++//++// Who may run what (always enforced, whatever TOOLS_GUARD says, because it is authorization, not a++// quota): an app or service token needs the tool's auth.capability (tools.tool.run, or tools.net.probe++// for the network probes, which nobody else runs through the API: people keep the probe pages); a++// tool with auth.anonymous false needs a browser session, a sign-in or a token. Quotas (the tool's++// quotaClass × the caller's tier, weighted by its cost), the Origin check on cookie-authenticated++// calls and the job bounds follow the guard's mode; the per-target throttle and upload sniffing are++// hard limits in both.++//++// Where a tool runs is the app's business (o.runs): inline (in this process — the gateway's net and++// dev routes, and the dev and text engines in a worker pool), as a job of this satellite's job system,++// or forwarded (o.proxy: the gateway streams a job tool's run to the satellite that owns it).++// Inline answers are cached per tool where that is safe (o.cacheTtl: pure transforms and slow-moving++// lookups; never a probe, a random generator or the caller's own address). An inline run's++// Idempotency-Key replays its first answer for 15 minutes; a job tool's key is the job's.++//++// Like the rest of apps/_shared this has no dependencies: the app passes its guard, contracts, an++// input validator (createInputValidator with the Ajv it has) and, on a satellite, multer.++// ═══════════════════════════════════════════════════════════════++++const crypto = require('crypto');++const fs = require('fs');++const fsp = require('fs/promises');++const path = require('path');++const { canonical } = require('../jobs/system');++++const RUN_RE = /^\/api\/v1\/tools\/([a-z][a-z0-9-]{0,39})\/run\/?$/;++const FIELDS = new Set(['input', 'files', 'wait_ms', 'idempotency_key']);++const KEY_RE = /^[\x21-\x7e]{8,200}$/;++const MEDIA_ID_RE = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;++const JOB_ID_RE = /^job_[0-9A-HJKMNP-TV-Z]{26}$/;++const MAX_WAIT_MS = 60_000;++const TERMINAL = new Set(['succeeded', 'failed', 'cancelled']);++const REPLAY_TTL_MS = 15 * 60 * 1000;++++const isRunPath = (p) => RUN_RE.test(String(p || ''));++++/** Run Express middleware in order (the apps' older limiters) → true when every one called next(). */++async function passes(list, req, res) {++ for (const mw of list || []) {++ const ok = await new Promise((resolve) => {++ let settled = false;++ const done = (v) => { if (!settled) { settled = true; res.removeListener('finish', onFinish); resolve(v); } };++ const onFinish = () => done(false);++ res.once('finish', onFinish);++ try { mw(req, res, (err) => done(!err)); } catch { done(false); }++ });++ if (!ok || res.headersSent) return false;++ }++ return true;++}++++/** A refusal before the tool ran: answered as problem+json, not as a run. */++class Refusal extends Error {++ constructor(status, code, detail, extra, headers) { super(detail || code); this.status = status; this.code = code; this.detail = detail; this.extra = extra; this.headers = headers; this.refusal = true; }++}++++// ── Input validation (the tool's own JSON Schema) ────────────++++/**++ * @param {Function} Ajv ajv/dist/2020 (from the app: openvibe-contracts brings it)++ * @param {Function} [addFormats] ajv-formats++ * @returns {(d, input) => { path, message }[]}++ */++function createInputValidator(Ajv, addFormats) {++ const ajv = new Ajv({ allErrors: true, strict: false, allowUnionTypes: true, logger: false });++ if (addFormats) addFormats(ajv);++ const compiled = new Map(); // id → { sig, fn, schema }++ return function validate(d, input) {++ if (!d.input || typeof d.input !== 'object') return [];++ const sig = JSON.stringify(d.input);++ let v = compiled.get(d.id);++ if (!v || v.sig !== sig) {++ if (v && v.schema) { try { ajv.removeSchema(v.schema); } catch { /* not registered */ } }++ const schema = { ...d.input };++ delete schema.$id;++ v = { sig, schema, fn: ajv.compile(schema) };++ compiled.set(d.id, v);++ }++ if (v.fn(input)) return [];++ return (v.fn.errors || []).slice(0, 20).map((e) => {++ if (e.keyword === 'additionalProperties') return { path: `${e.instancePath}/${e.params.additionalProperty}`, message: 'is not a field of this tool' };++ if (e.keyword === 'required') return { path: `${e.instancePath}/${e.params.missingProperty}`, message: 'is required' };++ return { path: e.instancePath || '/', message: e.message || 'is not valid' };++ });++ };++}++++// ── Answers kept for a while ─────────────────────────────────++++/** A TTL + LRU map bounded by entries and bytes (the inline cache, inline Idempotency-Key replays). */++function createTtlStore({ maxEntries = 2000, maxBytes = 32 * 1024 * 1024, now = Date.now } = {}) {++ const map = new Map(); // key → { value, size, until }++ let bytes = 0, hits = 0, misses = 0;++ const drop = (k) => { const e = map.get(k); if (e) { bytes -= e.size; map.delete(k); } };++ return {++ get(k) {++ const e = map.get(k);++ if (!e) { misses++; return undefined; }++ if (e.until <= now()) { drop(k); misses++; return undefined; }++ map.delete(k); map.set(k, e); // most recently used last++ hits++;++ return e.value;++ },++ set(k, value, ttlMs, size) {++ if (!(ttlMs > 0)) return false;++ const n = size || Buffer.byteLength(JSON.stringify(value));++ if (n > maxBytes / 8) return false; // one answer never takes a big share++ drop(k);++ map.set(k, { value, size: n, until: now() + ttlMs });++ bytes += n;++ while (map.size > maxEntries || bytes > maxBytes) drop(map.keys().next().value);++ return true;++ },++ stats: () => ({ entries: map.size, bytes, hits, misses }),++ clear() { map.clear(); bytes = 0; },++ };++}++++// ── Request parsing ──────────────────────────────────────────++++function toInt(v) {++ if (v === undefined || v === null || v === '') return undefined;++ const n = typeof v === 'number' ? v : Number(String(v).trim());++ return Number.isInteger(n) ? n : NaN;++}++++/** tools.run-request@1 from req.body (JSON, or multipart text fields) and the query → fields, or throws Refusal. */++function readRequest(req, multipart) {++ const b = req.body && typeof req.body === 'object' && !Array.isArray(req.body) ? req.body : null;++ if (!b && req.body != null && req.body !== '') throw new Refusal(400, 'tools.run.invalid', 'The body is a JSON object (tools.run-request@1), or multipart/form-data.');++ const body = b || {};++ for (const k of Object.keys(body)) if (!FIELDS.has(k)) throw new Refusal(400, 'tools.run.invalid', `Unknown field "${k}": a run request has input, files, wait_ms and idempotency_key.`);++ const parseJson = (name, v, dflt) => {++ if (!multipart || typeof v !== 'string') return v === undefined ? dflt : v;++ if (!v.trim()) return dflt;++ try { return JSON.parse(v); } catch { throw new Refusal(400, 'tools.run.invalid', `The ${name} part must be JSON.`); }++ };++ const input = parseJson('input', body.input, {});++ if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Refusal(400, 'tools.run.invalid', 'input is a JSON object.');++ const refs = parseJson('files', body.files, []);++ if (!Array.isArray(refs) || refs.length > 50) throw new Refusal(400, 'tools.run.invalid', 'files is an array of at most 50 references ({ media_id } or { job_id, index }).');++ for (const r of refs) {++ const okMedia = r && typeof r === 'object' && Object.keys(r).length === 1 && MEDIA_ID_RE.test(String(r.media_id || ''));++ const okJob = r && typeof r === 'object' && Object.keys(r).length === 2 && JOB_ID_RE.test(String(r.job_id || '')) && Number.isInteger(r.index) && r.index >= 0 && r.index <= 99;++ if (!okMedia && !okJob) throw new Refusal(400, 'tools.run.invalid', 'A files reference is { media_id: "med_…" } or { job_id: "job_…", index: 0..99 }.');++ }++ let wait = toInt(body.wait_ms);++ if (wait === undefined) wait = toInt(req.query && req.query.wait_ms);++ if (wait === undefined) wait = 0;++ if (!Number.isInteger(wait) || wait < 0 || wait > MAX_WAIT_MS) throw new Refusal(400, 'tools.run.invalid', `wait_ms is an integer, 0..${MAX_WAIT_MS}.`);++ const header = req.headers['idempotency-key'];++ const key = header != null && header !== '' ? String(header) : body.idempotency_key != null && body.idempotency_key !== '' ? String(body.idempotency_key) : null;++ if (key != null && !KEY_RE.test(key)) throw new Refusal(400, 'tools.run.invalid', 'Idempotency-Key is 8-200 printable ASCII characters.');++ return { input, refs, wait, key };++}++++const sha = (s) => crypto.createHash('sha256').update(s).digest('hex');++++/**++ * @param {object} o++ * @param {object} o.guard apps/_shared/guard++ * @param {object} o.contracts openvibe-contracts++ * @param {() => object} o.snapshot the registry: { tools: [descriptor], gone: Map }++ * @param {(d, input) => object[]} o.validate++ * @param {(d) => 'inline'|'job'|null} o.runs what this app runs itself++ * @param {(d, input, ctx) => Promise<{data}|{text}>} [o.inline] ctx: { req, res, caller, signal, timeoutMs }++ * @param {(d) => number} [o.cacheTtl] ms an inline answer may be reused (0 = never)++ * @param {(d, input) => string|null} [o.targetOf] the host an egress tool reaches (per-target throttle)++ * @param {(d, req, res) => boolean} [o.proxy] forwards the run elsewhere (the gateway)++ * @param {Function} [o.parseJson] JSON body parser middleware, when the app has not run one++ * @param {object} [o.jobs] a satellite's jobs: { system: () => system, receive: (d) => middleware,++ * fetchSibling: (ref, req) → file | null }++ * @param {object} [o.log]++ */++function createRunApi(o) {++ const { guard, contracts } = o;++ const log = o.log || console;++ const cache = createTtlStore({ maxEntries: 5000, maxBytes: 64 * 1024 * 1024 });++ const replays = createTtlStore({ maxEntries: 20000, maxBytes: 32 * 1024 * 1024 });++ const counts = { runs: 0, inline: 0, jobs: 0, proxied: 0, refused: 0, failed: 0, cached: 0, replayed: 0 };++++ function problem(req, res, status, code, detail, extra, headers) {++ counts.refused++;++ if (headers) for (const [k, v] of Object.entries(headers)) res.setHeader(k, v);++ res.setHeader('Cache-Control', 'no-store');++ return contracts.http.sendProblem(res, status, code, { detail, extra, ctx: req.ov });++ }++ const send = (res, status, body, headers = {}) => {++ res.setHeader('Cache-Control', 'no-store');++ for (const [k, v] of Object.entries(headers)) res.setHeader(k, v);++ res.status(status).json(body);++ };++ const toolProblem = (req, err, fallbackStatus = 422) => {++ const status = Number.isInteger(err.status) && err.status >= 400 && err.status <= 599 ? err.status : fallbackStatus;++ const code = /^tools\.[a-z0-9_.]+$/.test(String(err.code || '')) ? err.code : 'tools.job.failed';++ const detail = String(err.detail || err.message || 'The tool failed').replace(/(?:\/[\w.-]+){2,}/g, '[file]').slice(0, 500);++ return contracts.http.problem(status, code, { detail, ctx: req.ov });++ };++++ // ── Who may run it (always enforced) ─────────────────────++ function authorize(req, res, d) {++ const c = guard.caller(req, res);++ const cap = (d.auth && d.auth.capability) || 'tools.tool.run';++ const probe = cap === 'tools.net.probe';++ // A token that does not verify (expired, another audience) is an error here, not anonymity:++ // the client refreshes it instead of running as nobody.++ if (/^Bearer\s+\S/i.test(String(req.headers.authorization || '')) && c.kind !== 'principal' && c.kind !== 'user') {++ throw new Refusal(401, 'token.invalid', 'The Bearer token was not accepted (expired, not for openvibe.tools, or not signed by the Network).');++ }++ if (c.kind === 'principal') {++ if (!guard.tokens.hasCapability(c.claims, cap, contracts)) {++ guard.record(req, c, { reason: 'capability', tool: d.id, enforced: true });++ throw new Refusal(403, 'capability.denied', `${cap} not granted to this token`);++ }++ return c;++ }++ if (probe) {++ guard.record(req, c, { reason: 'capability', tool: d.id, enforced: true });++ if (c.kind === 'user') throw new Refusal(403, 'capability.denied', `Network probes run through the API only for apps and services holding ${cap}; people use the tool's page.`);++ throw new Refusal(401, 'token.missing', `${d.id} runs through the API only with a token holding ${cap}; people use the tool's page.`);++ }++ if (d.auth && d.auth.anonymous === false && c.kind === 'anonymous') {++ guard.record(req, c, { reason: 'session', tool: d.id, enforced: true });++ throw new Refusal(401, 'tools.session_required', `${d.id} needs a browser session (open its page first), a sign-in or an API token.`);++ }++ return c;++ }++++ function answerRefusal(req, res, err) {++ if (res.headersSent) return undefined;++ return problem(req, res, err.status, err.code, err.detail, err.extra, err.headers);++ }++++ // ── Inline tools ─────────────────────────────────────────++ async function runInline(req, res, d, c, fields, t0) {++ counts.inline++;++ const { input, key } = fields;++ const reqHash = sha(canonical({ tool: d.id, input }));++ const replayKey = key ? `${c.key}|${key}` : null;++ if (replayKey) {++ const prior = replays.get(replayKey);++ if (prior) {++ if (prior.hash !== reqHash) throw new Refusal(409, 'tools.job.idempotency_conflict', 'This Idempotency-Key was used for a different request');++ counts.replayed++;++ return send(res, 200, prior.body, { 'Idempotent-Replayed': 'true' });++ }++ }++ const ttl = o.cacheTtl ? o.cacheTtl(d) : 0;++ const cacheKey = ttl > 0 ? `${d.id}|${reqHash}` : null;++ let result = cacheKey ? cache.get(cacheKey) : undefined;++ let body;++ if (result !== undefined) {++ counts.cached++;++ res.setHeader('X-OV-Cache', 'hit');++ body = { state: 'succeeded', tool: d.id, result, took_ms: Date.now() - t0 };++ } else {++ // A tool that reaches a host the caller chose: that host's minute is shared by everyone (hard).++ const target = o.targetOf ? o.targetOf(d, input) : null;++ if (target && d.limits && d.limits.perTargetPerMinute && !guard.target(req, res, { tool: d.id, target })) return undefined;++ const ac = new AbortController();++ const onClose = () => { if (!res.writableFinished) ac.abort(); };++ res.on('close', onClose);++ const timeoutMs = (d.limits && d.limits.timeoutMs) || 15000;++ let timer;++ try {++ const out = await Promise.race([++ o.inline(d, input, { req, res, caller: c, signal: ac.signal, timeoutMs }),++ new Promise((_, reject) => { timer = setTimeout(() => { ac.abort(); reject(Object.assign(new Error(`The tool ran longer than ${Math.round(timeoutMs / 1000)} s`), { status: 504, code: 'tools.run.timeout' })); }, timeoutMs + 250); }),++ ]);++ result = out && out.text !== undefined ? { text: String(out.text) } : { data: out && out.data && typeof out.data === 'object' ? out.data : {} };++ if (cacheKey) cache.set(cacheKey, result, ttl);++ body = { state: 'succeeded', tool: d.id, result, took_ms: Date.now() - t0 };++ } catch (err) {++ if (err && err.refusal) throw err;++ if (res.headersSent) return undefined; // a guard refusal inside the tool already answered++ counts.failed++;++ body = { state: 'failed', tool: d.id, error: toolProblem(req, err), took_ms: Date.now() - t0 };++ } finally {++ clearTimeout(timer);++ res.removeListener('close', onClose);++ }++ }++ if (replayKey) replays.set(replayKey, { hash: reqHash, body }, REPLAY_TTL_MS);++ return send(res, 200, body);++ }++++ // ── Job tools (a satellite's own job system) ─────────────++ const uploadsOf = (req) => (Array.isArray(req.files) ? req.files : req.files && typeof req.files === 'object' ? Object.values(req.files).flat() : []);++++ async function tempCopy(from, name, mime, dir) {++ const to = path.join(dir, `run-${crypto.randomBytes(12).toString('hex')}${path.extname(String(name || '')).slice(0, 9)}`);++ await fsp.copyFile(from, to);++ return { path: to, name, mime, size: (await fsp.stat(to)).size };++ }++++ /** A file reference → a temporary input file the job will own, or throws 404 tools.run.file_not_found. */++ async function resolveRef(req, ref, owner, dir) {++ const system = o.jobs.system();++ const notFound = () => new Refusal(404, 'tools.run.file_not_found', `${ref.media_id || `${ref.job_id} file ${ref.index}`} is not a file you can read here: a result of one of your own Tools jobs that has not expired.`);++ let row = null, index = ref.index;++ if (ref.job_id) {++ row = system.get(ref.job_id);++ if (!row && o.jobs.fetchSibling) {++ const got = await o.jobs.fetchSibling(ref, req, dir);++ if (got) return got;++ throw notFound();++ }++ } else {++ const hit = system.findResultMedia(owner, ref.media_id);++ if (hit) { row = hit.row; index = hit.index; }++ }++ if (!row || row.owner !== owner || row.state !== 'succeeded') throw notFound();++ const f = system.resultFile(row, index);++ if (!f) throw notFound();++ if (f.path && fs.existsSync(f.path)) return tempCopy(f.path, f.name, f.mime, dir);++ if (f.storage === 'media' && f.media && system.media) {++ const signed = await system.media.downloadUrl(f.media.media_id);++ const up = await fetch(signed.internal_url, { signal: AbortSignal.timeout(120_000) });++ if (!up.ok) throw notFound();++ const to = path.join(dir, `run-${crypto.randomBytes(12).toString('hex')}`);++ await fsp.writeFile(to, Buffer.from(await up.arrayBuffer()));++ return { path: to, name: f.name, mime: f.mime, size: (await fsp.stat(to)).size };++ }++ throw notFound();++ }++++ function waitFor(system, id, ms, res) {++ return new Promise((resolve) => {++ const row = system.get(id);++ if (!row || TERMINAL.has(row.state) || ms <= 0) return resolve();++ let done = false;++ const finish = () => { if (done) return; done = true; clearTimeout(timer); off(); res.removeListener('close', finish); resolve(); };++ const off = system.subscribe(id, (e) => { if (TERMINAL.has(e.data.state)) finish(); });++ const timer = setTimeout(finish, ms);++ res.on('close', finish);++ // It may have finished between the first read and the subscription.++ const again = system.get(id);++ if (!again || TERMINAL.has(again.state)) finish();++ });++ }++++ function runView(system, d, row) {++ const job = system.view(row);++ const took = row.finished_at ? Math.max(0, row.finished_at - row.created_at) : 0;++ if (row.state === 'succeeded') return { state: 'succeeded', tool: d.id, result: { data: (job.result && job.result.data) || {}, files: (job.result && job.result.files) || [] }, took_ms: took, job };++ if (row.state === 'failed' || row.state === 'cancelled') return { state: row.state, tool: d.id, error: job.error || contracts.http.problem(500, 'tools.job.failed', { detail: 'The job failed' }), took_ms: took, job };++ return { state: row.state, tool: d.id, job, location: job.links.self };++ }++++ async function runJob(req, res, d, c, fields, uploads) {++ counts.jobs++;++ const system = o.jobs.system();++ const { input, refs, wait, key } = fields;++ const dir = o.jobs.uploadsDir;++ const files = uploads.map(f => ({ path: f.path, buffer: f.path ? undefined : f.buffer, name: f.originalname, mime: f.mimetype, size: f.size, originalname: f.originalname, mimetype: f.mimetype }));++ const cleanup = () => { for (const f of files) if (f.path) fs.unlink(f.path, () => {}); };++ try {++ // The owner: a person, a principal, or this browser's session (started now when there is none).++ const who = guard.resolveCaller(req, res, { create: true });++ const owner = who.owner;++ if (!owner) throw new Refusal(401, 'tools.session_required', `${d.id} needs a browser session, a sign-in or an API token.`);++ for (const ref of refs) files.push(await resolveRef(req, ref, owner, dir));++ const spec = d.files;++ if (!spec && files.length) throw new Refusal(400, 'tools.run.invalid', `${d.id} takes no files.`);++ if (spec && (files.length < spec.min || files.length > spec.max)) {++ throw new Refusal(400, 'tools.run.invalid', `${d.id} takes ${spec.min === spec.max ? spec.min : `${spec.min} to ${spec.max}`} file${spec.max === 1 ? '' : 's'}; this request has ${files.length}.`);++ }++ if (spec && spec.maxBytes) {++ const big = files.find(f => (f.size || 0) > spec.maxBytes);++ if (big) throw new Refusal(413, 'tools.file.too_large', `${big.name} is over ${Math.round(spec.maxBytes / 1024 / 1024)} MB, the limit for ${d.id}.`);++ }++ // The bytes against files.accept (hard; a misleading name or type is corrected).++ if (spec && !guard.checkFiles(req, res, files, spec.accept, d.id)) return undefined;++ // Sessions from one address share a bound on unfinished jobs; the store's queue and disk bounds.++ const full = system.addressFull ? system.addressFull(owner, who.ipKey) : null;++ if (full && guard.refuse(req, res, { status: 429, code: 'tools.job.too_many_active', reason: 'jobs.address', tool: d.id, retryAfter: 30, detail: `At most ${full.limit} unfinished jobs from one address at a time; wait for one to finish.`, extra: { scope: 'address' } })) return undefined;++ const busy = system.busy ? system.busy() : null;++ if (busy && !guard.jobsBusy(req, res, busy, d.id)) return undefined;++ let row, replayed;++ try {++ const out = await system.submit({++ owner, type: d.run.job.type, input: contracts.tools.jobInput(d, input),++ files: files.map(f => ({ path: f.path, buffer: f.buffer, name: f.originalname || f.name, mime: f.mimetype || f.mime, size: f.size })),++ idempotencyKey: key, ttlMs: who.kind === 'session' ? 60 * 60 * 1000 : 24 * 60 * 60 * 1000,++ env: who.env || 'production', ipKey: who.ipKey, tool: d.id,++ });++ row = out.job; replayed = out.replayed;++ } catch (err) {++ if (err instanceof system.JobError) throw new Refusal(err.status, err.code, err.detail, err.extra);++ throw err;++ }++ files.length = 0; // the job owns them now++ if (wait > 0) await waitFor(system, row.id, wait, res);++ const fresh = system.get(row.id) || row;++ const body = runView(system, d, fresh);++ const headers = replayed ? { 'Idempotent-Replayed': 'true' } : {};++ if (body.location) headers.Location = body.location;++ return send(res, replayed || !body.location ? 200 : 202, body, headers);++ } finally {++ cleanup();++ }++ }++++ // ── The route ────────────────────────────────────────────++ async function handle(req, res, next) {++ const m = RUN_RE.exec(req.path);++ if (!m) return next();++ if (req.method === 'OPTIONS') return next(); // the app's CORS answers preflights++ if (req.method !== 'POST') { res.setHeader('Allow', 'POST'); return problem(req, res, 405, 'method_not_allowed', 'POST a tools.run-request@1 to run a tool.'); }++ const t0 = Date.now();++ counts.runs++;++ const snap = o.snapshot();++ const id = m[1];++ const d = snap.tools.find(t => t.id === id);++ if (!d) return problem(req, res, 404, 'tools.tool.not_found', (snap.gone && snap.gone.get(id)) || `No tool is called "${id}". GET /api/v1/tools lists them all.`);++ if (d.status === 'unavailable') return problem(req, res, 503, 'tools.tool.unavailable', d.statusReason || `${d.id} is not available right now.`);++ if (!d.api || !d.run) return problem(req, res, 404, 'tools.tool.not_runnable', `${d.id} has no run API; use its page${d.hosts && d.hosts[0] ? ` (https://${d.hosts[0]})` : ''}.`);++ if (o.proxy && o.proxy(d, req, res)) { counts.proxied++; return undefined; }++ const mode = o.runs(d);++ if (!mode) return problem(req, res, 404, 'tools.tool.not_runnable', `${d.id} is not run here; POST it to https://openvibe.tools/api/v1/tools/${d.id}/run.`);++ try {++ // Cookies from another site's page (CSRF): refused in enforce mode, recorded in report mode.++ if (!guard.originOk(req, res, { hosts: d.hosts, tool: d.id })) return undefined;++ const c = authorize(req, res, d);++ // The tool's quota class and cost (recorded for the job admission, so it is not charged twice).++ req._ovCharged = { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id };++ if (!(await guard.charge(req, res, { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id }))) return undefined;++ // A satellite's older per-route limiters count job runs like its job submits (report mode: in force).++ if (mode === 'job' && o.jobs.limiters && !(await passes(o.jobs.limiters, req, res))) return undefined;++ const multipart = req.is && req.is('multipart/form-data');++ if (multipart && mode !== 'job') throw new Refusal(400, 'tools.run.invalid', `${d.id} takes no files: send JSON (tools.run-request@1).`);++ if (!multipart && o.parseJson && req.body === undefined) {++ const bad = await new Promise((resolve) => o.parseJson(req, res, (err) => resolve(err || null)));++ if (bad) throw new Refusal(bad.status === 413 ? 413 : 400, bad.status === 413 ? 'tools.input.too_large' : 'tools.run.invalid', bad.status === 413 ? 'The request body is larger than 1 MB.' : 'The body is not valid JSON.');++ }++ let uploads = [];++ if (multipart) {++ const err = await new Promise((resolve) => o.jobs.receive(d)(req, res, (e) => resolve(e || null)));++ uploads = uploadsOf(req);++ if (err) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw err.refusal ? err : new Refusal(400, 'tools.run.invalid', err.message); }++ }++ let fields;++ try { fields = readRequest(req, multipart); } catch (err) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw err; }++ const bytes = Buffer.byteLength(JSON.stringify(fields.input));++ const limit = d.limits && d.limits.maxInputBytes;++ if (limit && bytes > limit) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw new Refusal(413, 'tools.input.too_large', `input is ${bytes} bytes as JSON; ${d.id} takes at most ${limit}.`); }++ const errors = o.validate(d, fields.input);++ if (errors.length) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw new Refusal(422, 'tools.input.invalid', `The input does not match ${d.id}'s input schema (GET /api/v1/tools/${d.id}/schema).`, { errors }); }++ if (mode === 'inline') return await runInline(req, res, d, c, fields, t0);++ return await runJob(req, res, d, c, fields, uploads);++ } catch (err) {++ if (err && err.refusal) return answerRefusal(req, res, err);++ log.error(`[Run] ${id}:`, err && err.message);++ if (!res.headersSent) return problem(req, res, 500, 'tools.run.failed', 'The run could not be started.');++ return undefined;++ }++ }++++ return {++ handle: (req, res, next) => { handle(req, res, next).catch(next); },++ stats: () => ({ ...counts, cache: cache.stats(), replays: replays.stats() }),++ cache, replays,++ };++}++++/**++ * A satellite's upload receiver for runs: `file` and `files` file parts on disk (never in memory), the++ * descriptor's file count and size, any declared type (the guard sniffs the bytes). The text parts++ * (input, files, wait_ms, idempotency_key) land in req.body.++ * @param {Function} multer the app's multer++ * @param {string} dir where uploads wait (the app's uploads directory)++ */++function createRunUploads(multer, dir) {++ fs.mkdirSync(dir, { recursive: true });++ const storage = multer.diskStorage({++ destination: (_req, _file, cb) => cb(null, dir),++ filename: (_req, file, cb) => {++ const ext = (path.extname(file.originalname || '').toLowerCase().match(/^\.[a-z0-9]{1,8}$/) || ['.bin'])[0];++ cb(null, `run-${crypto.randomBytes(16).toString('hex')}${ext}`);++ },++ });++ const byTool = new Map();++ return function receive(d) {++ const max = (d.files && d.files.max) || 1;++ const maxBytes = (d.files && d.files.maxBytes) || 50 * 1024 * 1024;++ const k = `${max}|${maxBytes}`;++ if (!byTool.has(k)) {++ byTool.set(k, multer({ storage, limits: { fileSize: maxBytes, files: max, fields: 8, fieldSize: 1024 * 1024 } })++ .fields([{ name: 'file', maxCount: max }, { name: 'files', maxCount: max }]));++ }++ const mw = byTool.get(k);++ return (req, res, next) => mw(req, res, (err) => {++ if (!err) return next();++ if (err.code === 'LIMIT_FILE_SIZE') return next(new Refusal(413, 'tools.file.too_large', `A file is over ${Math.round(maxBytes / 1024 / 1024)} MB, the limit for ${d.id}.`));++ if (err.code === 'LIMIT_FILE_COUNT' || err.code === 'LIMIT_UNEXPECTED_FILE') return next(new Refusal(400, 'tools.run.invalid', `${d.id} takes at most ${max} file${max === 1 ? '' : 's'}, as "file" parts.`));++ return next(new Refusal(400, 'tools.run.invalid', err.message));++ });++ };++}++++/**++ * A run's { job_id, index } reference to a job another satellite on this host holds: ask it which one++ * (the facade's loopback probe), then read the file there with the caller's own credentials and++ * address, so the owner check is that satellite's. → { path, name, mime, size } or null.++ */++function createSiblingFetch({ self, ports, fetchImpl = fetch }) {++ const { JOB_SATELLITES } = require('./satellites');++ return async function fetchSibling(ref, req, dir) {++ const all = ports();++ const others = JOB_SATELLITES.filter(s => s !== self && all[s]);++ let port = null;++ for (const sat of others) {++ try {++ const r = await fetchImpl(`http://127.0.0.1:${all[sat]}/api/internal/jobs/${ref.job_id}`, { signal: AbortSignal.timeout(2500) });++ if (r.status === 200) { port = all[sat]; break; }++ } catch { /* not that one */ }++ }++ if (!port) return null;++ const headers = { 'x-forwarded-for': String(req.ip || ''), 'x-real-ip': String(req.ip || '') };++ if (req.headers.authorization) headers.authorization = req.headers.authorization;++ if (req.headers.cookie) headers.cookie = req.headers.cookie;++ const r = await fetchImpl(`http://127.0.0.1:${port}/api/v1/jobs/${ref.job_id}/files/${ref.index}`, { headers, signal: AbortSignal.timeout(120_000) });++ if (r.status !== 200 || !r.body) return null;++ const cd = String(r.headers.get('content-disposition') || '');++ const star = /filename\*=UTF-8''([^;]+)/i.exec(cd);++ const plain = /filename="([^"]+)"/i.exec(cd);++ let name = 'file';++ try { name = star ? decodeURIComponent(star[1]) : plain ? plain[1] : 'file'; } catch { name = plain ? plain[1] : 'file'; }++ const to = path.join(dir, `run-${crypto.randomBytes(12).toString('hex')}${path.extname(name).slice(0, 9)}`);++ await fsp.writeFile(to, Buffer.from(await r.arrayBuffer()));++ return { path: to, name, mime: String(r.headers.get('content-type') || 'application/octet-stream').split(';')[0], size: (await fsp.stat(to)).size };++ };++}++++/**++ * The run API of a satellite that runs job tools (img, audio, docs): each of its job tools runs as a++ * job of its own job system; file references to other satellites' jobs are read from them.++ * @param {object} o++ * @param {string} o.app 'img' | 'audio' | 'docs'++ * @param {object} o.guard @param {object} o.contracts++ * @param {() => object} o.snapshot the satellite's own registry++ * @param {() => object} o.system its job system (set up after the routes)++ * @param {Function} o.multer @param {string} o.uploadsDir++ * @param {Function} o.Ajv @param {Function} [o.addFormats]++ * @param {() => object} o.ports satellite ports (./satellites.js)++ * @param {Function[]} [o.limiters] the app's older burst and processing limiters (as on its job submits)++ */++function satelliteRunApi(o) {++ const uploadsDir = path.resolve(o.uploadsDir);++ return createRunApi({++ guard: o.guard, contracts: o.contracts, snapshot: o.snapshot, log: o.log,++ validate: createInputValidator(o.Ajv, o.addFormats),++ runs: (d) => (d.execution === 'job' && d.run && d.run.job ? 'job' : null),++ jobs: {++ system: o.system,++ limiters: o.limiters || [],++ receive: createRunUploads(o.multer, uploadsDir),++ uploadsDir,++ fetchSibling: createSiblingFetch({ self: o.app, ports: o.ports }),++ },++ });++}++++/**++ * Ajv (and ajv-formats) from the app's own openvibe-contracts, which depends on them (apps/_shared has++ * no node_modules). appRequire: the app's `require`.++ */++function ajvFrom(appRequire) {++ const base = path.dirname(appRequire.resolve('openvibe-contracts'));++ const Ajv = appRequire(appRequire.resolve('ajv/dist/2020', { paths: [base] }));++ let addFormats = null;++ try { addFormats = appRequire(appRequire.resolve('ajv-formats', { paths: [base] })); } catch { /* formats are optional */ }++ return { Ajv: Ajv.default || Ajv, addFormats: addFormats && (addFormats.default || addFormats) };++}++++module.exports = { ajvFrom, createRunApi, satelliteRunApi, createRunUploads, createSiblingFetch, createInputValidator, createTtlStore, readRequest, isRunPath, Refusal, RUN_RE };nnew file mode 100644@@@ -0,0 +1,30 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// Where the Tools apps listen on this host, for the loopback calls between them (the gateway's++// readiness checks, registry polling, run API and jobs facade; a satellite reading another's job++// result for a run's { job_id, index } reference).++//++// TOOLS_SATELLITE_PORTS="img=5012,yt=5013" overrides single entries (tests, a moved unit).++// Job types are routed by their prefix: img.process → img, audio.process → audio, docs.process → docs.++// ═══════════════════════════════════════════════════════════════++++const DEFAULT_PORTS = Object.freeze({ maps: 4010, food: 4011, img: 4012, yt: 4013, audio: 4014, text: 4015, docs: 4016 });++const JOB_SATELLITES = Object.freeze(['img', 'audio', 'docs']);++++/** { maps: 4010, …, docs: 4016 } with the environment's overrides. */++function satellitePorts(env = process.env) {++ const out = { ...DEFAULT_PORTS };++ for (const pair of String(env.TOOLS_SATELLITE_PORTS || '').split(',')) {++ const [name, port] = pair.split('=').map(x => String(x || '').trim());++ if (out[name] && /^\d{2,5}$/.test(port)) out[name] = Number(port);++ }++ return out;++}++++/** The satellite that runs a job type ('img.process' → 'img'), or null. */++function satelliteForType(type) {++ const prefix = String(type || '').split('.')[0];++ return JOB_SATELLITES.includes(prefix) ? prefix : null;++}++++module.exports = { DEFAULT_PORTS, JOB_SATELLITES, satellitePorts, satelliteForType };@@@ -24,6 +24,9 @@ const { probe, getDuration, cleanTmp } = require('./tools/ffmpeg-helper');const { buildOptions, describe, defineJobs, runSync, jobContext, limitsFor } = require('./process');const { hostGuard, ownHost } = require('../../_shared/host-role');const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { satelliteRunApi, ajvFrom } = require('../../_shared/tools/run');++const { satellitePorts } = require('../../_shared/tools/satellites');++const { deprecated, runPath } = require('../../_shared/tools/deprecation');const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const jobsRuntime = require('../../_shared/jobs');const { createGuard, TRUST_PROXY } = require('../../_shared/guard');@@@ -115,7 +118,7 @@ app.use(exceptRegistry(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--})));++}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──app.use(guard.identify);@@@ -177,7 +180,7 @@ app.get('/api/tools', (_req, res) => {});// ── Probe / Metadata Endpoint ────────────────────────────────--app.post('/api/probe', burstLimiter, processLimiter, guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {++app.post('/api/probe', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {try {// Hardened like every run: local files only, audio demuxers only.const { info, meta } = await jobContext.run(limitsFor('metadata'), async () => ({ info: await probe(req.file.path), meta: await readMetadata(req.file.path) }));@@@ -190,7 +193,7 @@ app.post('/api/probe', burstLimiter, processLimiter, guard.toolQuota('metadata')});// ── Main Processing Endpoint ─────────────────────────────────--app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {++app.post('/api/process', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {try {const toolId = req.body.tool || req.ctx.defaultOp || 'convert';const tool = getTool(toolId);@@@ -231,7 +234,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)});// ── Multi-File Processing Endpoint (merge) ───────────────────--app.post('/api/process/multi', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {++app.post('/api/process/multi', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {const paths = req.files.map(f => f.path);try {const toolId = req.body.tool || req.ctx.defaultOp;@@@ -274,6 +277,17 @@ const jobs = jobsRuntime.setupJobs({},});++// ── Run API (ADR-027): POST /api/v1/tools/:id/run for this app's tools ──++// tools.run-request@1 (JSON, or multipart with `file` parts and the text parts input, files, wait_ms)++// → tools.run@1: each tool runs as a job of this satellite (the preset and operation from its++// descriptor), answered finished within wait_ms or 202 with the job. The gateway streams runs here.++const runApi = satelliteRunApi({++ app: 'audio', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,++ multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),++ limiters: [burstLimiter, processLimiter],++});++app.use(runApi.handle);++// ── File Download ────────────────────────────────────────────app.get('/api/download/:id', (req, res) => {const entry = retention.getFile(req.params.id);@@@ -22,7 +22,7 @@ async function waitFor(app, id, cookie, pred, what, ms = 30000) {const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();if (pred(job)) return job;if (Date.now() - t0 > ms) throw new Error(`timed out waiting for ${what}: ${JSON.stringify(job)}`);-- await sleep(50);++ await sleep(150); // the older /api/ limiter allows an address 60 requests a minute}}@@@ -24,6 +24,9 @@ const { buildOptions, defineJobs, runTool } = require('./process');const { poolFromEnv } = require('../../_shared/jobs/pool');const { hostGuard, ownHost } = require('../../_shared/host-role');const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { satelliteRunApi, ajvFrom } = require('../../_shared/tools/run');++const { satellitePorts } = require('../../_shared/tools/satellites');++const { deprecated, runPath } = require('../../_shared/tools/deprecation');const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const jobsRuntime = require('../../_shared/jobs');const { createGuard, TRUST_PROXY } = require('../../_shared/guard');@@@ -131,7 +134,7 @@ app.use(exceptRegistry(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--})));++}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──app.use(guard.identify);@@@ -206,7 +209,7 @@ app.get('/api/tools', (_req, res) => {});// ── PDF Info (no file mutation) ──────────────────────────────--app.post('/api/info', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {++app.post('/api/info', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {try {const tool = getTool('metadata');const [buffer] = await readUploads([req.file]);@@@ -218,7 +221,7 @@ app.post('/api/info', burstLimiter, processLimiter, guard.toolQuota(hostTool), u});// ── Main Processing Endpoint (single file) ───────────────────--app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {++app.post('/api/process', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {try {const toolId = req.body.tool || req.ctx.defaultOp;if (!toolId) { fs.unlink(req.file.path, () => {}); return res.status(400).json({ error: 'No tool specified.' }); }@@@ -265,7 +268,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)});// ── Multi-File Processing Endpoint (merge, img2pdf) ──────────--app.post('/api/process/multi', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {++app.post('/api/process/multi', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {try {const toolId = req.body.tool || req.ctx.defaultOp;const tool = toolId ? getTool(toolId) : null;@@@ -325,6 +328,17 @@ const jobs = jobsRuntime.setupJobs({},});++// ── Run API (ADR-027): POST /api/v1/tools/:id/run for this app's tools ──++// tools.run-request@1 (JSON, or multipart with `file` parts and the text parts input, files, wait_ms)++// → tools.run@1: each tool runs as a job of this satellite (the preset and operation from its++// descriptor), answered finished within wait_ms or 202 with the job. The gateway streams runs here.++const runApi = satelliteRunApi({++ app: 'docs', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,++ multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),++ limiters: [burstLimiter, processLimiter],++});++app.use(runApi.handle);++// ── File Download ────────────────────────────────────────────app.get('/api/download/:id', (req, res) => {const entry = retention.getFile(req.params.id);@@@ -34,7 +34,9 @@ const upload = multer({files: 50, // Allow up to 50 files for merge},fileFilter(_req, file, cb) {-- if (ALLOWED_MIMES.has(file.mimetype)) {++ // A part without a type of its own (application/octet-stream: API clients, openvibe-sdk) is let++ // through: the guard checks the bytes of every upload against the tool's accepted types.++ if (ALLOWED_MIMES.has(file.mimetype) || file.mimetype === 'application/octet-stream') {cb(null, true);} else {cb(new Error(`Unsupported file type: ${file.mimetype}. Accepted: ${config.upload.allowedMimes.join(', ')}`));@@@ -21,7 +21,7 @@ async function bigPdf(pages) {}async function finished(app, id, cookie) {-- for (let i = 0; i < 300; i++) {++ for (let i = 0; i < 1200; i++) { // up to two minutes on a busy machineconst job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();if (['succeeded', 'failed', 'cancelled'].includes(job.state)) return job;await sleep(100);@@@ -68,7 +68,9 @@ function mergeForm(files, job) {assert.ok(took > 700, `the merge is large enough to measure (${took} ms)`);assert.ok(latencies.length >= 5, `health was asked during the merge (${latencies.length} times)`);const worst = Math.max(...latencies);-- assert.ok(worst < 250, `/api/health stayed fast during a ${took} ms merge (worst ${worst} ms)`);++ // On the event loop the merge would hold every request for its whole length; in a worker thread++ // health answers in milliseconds (the bound leaves room for a busy test machine).++ assert.ok(worst < Math.max(250, took / 3), `/api/health stayed fast during a ${took} ms merge (worst ${worst} ms)`);const health = await (await fetch(`${app.base}/api/health`)).json();assert.strictEqual(health.workers.completed, 1);assert.strictEqual(health.workers.busy, 0);@@@ -57,6 +57,25 @@ server {proxy_set_header X-Forwarded-Proto $scheme;}++ # ── Run API and jobs facade (ADR-027): uploads stream to the satellites, SSE streams back ──++ # POST /api/v1/tools/:id/run takes files like the satellites do (docs: 100 MB each); wait_ms holds a++ # run open up to 60 s; /api/v1/jobs/:id/events is a long-lived event stream.++ location ~ ^/api/v1/(tools/[a-z0-9-]+/run|jobs)(/|$) {++ limit_req zone=ovtools_api burst=30 nodelay;++ client_max_body_size 512m;++ proxy_request_buffering off;++ proxy_buffering off;++ proxy_read_timeout 600s;++ proxy_send_timeout 600s;++ proxy_pass http://127.0.0.1:4001;++ proxy_http_version 1.1;++ proxy_set_header Host $host;++ proxy_set_header X-Real-IP $remote_addr;++ proxy_set_header X-Forwarded-For $remote_addr;++ proxy_set_header CF-Connecting-IP $remote_addr;++ proxy_set_header X-Forwarded-Proto $scheme;++ }++# ── API (moderate rate limit) ───────────────────────────location /api/ {limit_req zone=ovtools_api burst=30 nodelay;@@@ -103,4 +103,12 @@ const SPECS = [engine('md2html', obj({ text: text() }), TEXT, { text: '# Hi\n\n- one' }),];++// The run API reuses an engine's answer for 10 minutes (same tool, same input): not for these, whose++// output is random or depends on the time. Open Graph (a page's tags) may be reused for 10 minutes.++const FRESH = new Set(['jwt', 'uuid', 'timestamp', 'cron', 'lorem']);++for (const spec of SPECS) {++ if (FRESH.has(spec.id)) spec.cacheTtlMs = 0;++ if (spec.id === 'opengraph') spec.cacheTtlMs = 10 * 60 * 1000;++}++module.exports = { SPECS };@@@ -33,13 +33,9 @@ const app = express();// What this deploy runs (ADR-016); the shared navbar's release-watch polls it on every tool host.const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });--// The satellites on this host (ports), for readiness and the analytics roll-up.--// TOOLS_SATELLITE_PORTS="img=5012,yt=5013" overrides single entries (tests, a moved unit).--const SATELLITES = { maps: 4010, food: 4011, img: 4012, yt: 4013, audio: 4014, text: 4015, docs: 4016 };--for (const pair of String(process.env.TOOLS_SATELLITE_PORTS || '').split(',')) {-- const [name, port] = pair.split('=').map(x => String(x || '').trim());-- if (SATELLITES[name] && /^\d{2,5}$/.test(port)) SATELLITES[name] = Number(port);--}++// The satellites on this host (ports), for readiness, the analytics roll-up, the run API and the++// jobs facade. TOOLS_SATELLITE_PORTS="img=5012,yt=5013" overrides single entries (tests, a moved unit).++const SATELLITES = require('../../_shared/tools/satellites').satellitePorts();const SATELLITE_BY_PORT = new Map(Object.entries(SATELLITES).map(([name, port]) => [port, name]));// Metrics (GET /metrics, direct loopback callers only) and readiness (roadmap Track O). The metrics@@@ -132,8 +128,15 @@ app.use(hostRoles({app.get('/api/ready', obs.readiness.handler);app.use(cookieParser());--app.use(express.json({ limit: '1mb' }));--app.use(express.urlencoded({ extended: true }));++// The run API and the jobs facade read their own bodies: uploads and SSE stream through to the++// satellites untouched, and a local run parses its JSON itself (run/index.js).++const { isRunPath } = require('../../_shared/tools/run');++const { isJobsPath } = require('./run/jobs-facade');++const jsonBody = express.json({ limit: '1mb' });++const formBody = express.urlencoded({ extended: true });++const streamed = (req) => isRunPath(req.path) || isJobsPath(req.path);++app.use((req, res, next) => (streamed(req) ? next() : jsonBody(req, res, next)));++app.use((req, res, next) => (streamed(req) ? next() : formBody(req, res, next)));// ── CORS ─────────────────────────────────────────────────────// Every *.openvibe.tools subdomain (satellites included) may call the@@@ -162,6 +165,8 @@ function isAllowedOrigin(origin) {app.use('/auth/fedcm', fedcmCors);// The tool registry (/api/v1/tools…) is public and read-only: open to every origin, no credentials.++// The run API and the jobs facade take any site's page with a token: CORS without credentials for an++// origin that is not ours (so no cookie rides along), the credentialed rules below for ours.app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true); // curl / server-to-server@@@ -169,7 +174,8 @@ app.use(exceptRegistry(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--})));++ exposedHeaders: ['Location', 'Idempotent-Replayed', 'Retry-After', 'RateLimit-Limit', 'RateLimit-Remaining', 'RateLimit-Reset', 'X-OpenVibe-Request-Id', 'Deprecation', 'Sunset', 'Link'],++}), { isFirstParty: isAllowedOrigin }));// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──app.use(guard.identify);@@@ -182,6 +188,49 @@ app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, keyGenerator: (req// the /api/ limiter above. /api/catalog.json stays as it is (Network reads it).app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Net.OpenVibe and Dev.OpenVibe routes (the pages' API; the run API calls the same handlers) ──++const netRouter = createNetRoutes(null, requireAuth, { guard });++const devRouter = createDevRoutes(null, requireAuth, { guard });++++// ── Run API and jobs facade (ADR-027) ────────────────────────++// POST /api/v1/tools/:id/run (tools.run-request@1 → tools.run@1) for every tool with an API: net and++// Open Graph through their routes, dev and text engines in a worker pool, img/audio/docs streamed to++// their satellite. /api/v1/jobs… fronts every satellite's job routes (routed by type, then by job id).++const { createGatewayRun } = require('./run');++const { createJobsFacade, createJobIndex } = require('./run/jobs-facade');++const { ajvFrom } = require('../../_shared/tools/run');++const jobIndex = createJobIndex();++const runApi = createGatewayRun({++ guard, contracts: require('openvibe-contracts'), toolRegistry, routers: { net: netRouter, dev: devRouter },++ ports: () => SATELLITES, parseJson: jsonBody, jobIndex, ...ajvFrom(require),++});++const jobsFacade = createJobsFacade({ ports: () => SATELLITES, index: jobIndex, contracts: require('openvibe-contracts') });++app.use(runApi.handle);++app.use((req, res, next) => { jobsFacade.handle(req, res, next); });++++// ── The older tool endpoints: still answered, and they say what replaces them ──++const { deprecated, runPath } = require('../../_shared/tools/deprecation');++const NET_BY_ENDPOINT = new Map();++for (const sp of require('./net/descriptors').SPECS) {++ if (!sp.route || !sp.api) continue;++ const ep = sp.route.path.replace(/^\/api\/net/, '');++ (NET_BY_ENDPOINT.get(ep) || NET_BY_ENDPOINT.set(ep, []).get(ep)).push(sp.id);++}++/** /api/net/<endpoint>… → the run route of the tool it serves here (the host's own, else the one named after it). */++function netSuccessor(req) {++ const ep = `/${String(req.path || '').split('/')[1] || ''}`;++ if (ep === '/tools') return '/api/v1/tools?family=net';++ const ids = NET_BY_ENDPOINT.get(ep);++ if (!ids) return null;++ const host = req.ovHost && req.ovHost.tool;++ return runPath(ids.find(i => i === host) || ids.find(i => i === ep.slice(1)) || ids[0]);++}++function devSuccessor(req) {++ if (req.path === '/opengraph') return runPath('opengraph');++ if (req.path === '/tools') return '/api/v1/tools?family=dev';++ return null; // webhook bins: page-only (api false), no successor++}++// ── Auth (OAuth2 client of OpenVibe.Network) ─────────────────auth = createAuthClient(config);app.locals.auth = auth;@@@ -202,7 +251,7 @@ async function requireAuth(req, res, next) {// ── Basic API ────────────────────────────────────────────────app.get('/api/health', (_req, res) => {-- res.json({ status: 'ok', service: 'openvibe-tools-gateway', version: '2.0.0', registry: registry.services.status() });++ res.json({ status: 'ok', service: 'openvibe-tools-gateway', version: '2.0.0', registry: registry.services.status(), run: runApi.stats(), jobs: jobsFacade.stats() });});app.get('/api/brand', (_req, res) => res.json(BRAND));@@@ -239,10 +288,13 @@ app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonym});// ── Net.OpenVibe — Network Tools API ─────────────────────────--app.use('/api/net', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), createNetRoutes(null, requireAuth, { guard }));++app.use('/api/net', deprecated(netSuccessor), guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), netRouter);// ── Dev.OpenVibe — Developer & SEO Tools API ─────────────────--app.use('/api/dev', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), createDevRoutes(null, requireAuth, { guard }));++// Webhook bins are created and deleted with the browser's session cookie: those writes must come from++// our pages (the /in URL a bin receives on is anyone's to call).++const devOrigin = guard.originCheck();++app.use('/api/dev', deprecated(devSuccessor), (req, res, next) => (/^\/webhook\/bins\/[^/]+\/in(\/|$)/.test(req.path) ? next() : devOrigin(req, res, next)), guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), devRouter);// ── Host-header subdomain routing ────────────────────────────function subdomainOf(req) {@@@ -27,6 +27,16 @@ const TLS_PORTS = [443, 465, 636, 853, 990, 993, 995, 2083, 2087, 5061, 8443];const PROBES = new Set(['port', 'ping', 'latency', 'traceroute', 'mtr']);const KiB = 1024;++const MIN = 60 * 1000;++// How long the run API may reuse a lookup's answer (same tool, same input). Records and registration++// data move slowly (and the routes cache their upstreams anyway); live checks (ping, ports, headers,++// redirects, uptime, SMTP) and myip (the caller's own address) are never reused.++const CACHE_TTL = {++ dns: MIN, dig: MIN, nslookup: MIN, mx: MIN, txt: MIN, ns: MIN, spf: MIN, dkim: MIN, dmarc: MIN, dnspropagation: MIN,++ ip: 10 * MIN, geoip: 10 * MIN, isp: 10 * MIN, asn: 10 * MIN, ipv4: 10 * MIN, ipv6: 10 * MIN,++ hostname: 5 * MIN, rdns: 5 * MIN, whois: 60 * MIN, rdap: 60 * MIN,++ blacklist: 5 * MIN, ssl: 5 * MIN, robots: 5 * MIN, sitemap: 5 * MIN, lookup: 5 * MIN,++};const target = (description) => ({ type: 'string', minLength: 1, maxLength: 253 + 2048, description });const HOST = 'A domain or an IP address';const URL_OR_DOMAIN = 'A URL, or a domain (https:// is assumed)';@@@ -130,6 +140,7 @@ function spec(id) {route: { method: 'GET', path: `/api/net${endpoint}`, ...(t.query && { query: t.query }), ...(t.targetTemplate && { target: t.targetTemplate }) },}),example: { input: endpoint === '/myip' ? {} : { target: 'example.com' } },++ cacheTtlMs: CACHE_TTL[id] || 0,};}nnew file mode 100644@@@ -0,0 +1,15 @@++'use strict';++// A worker thread of the gateway's engine pool (apps/_shared/jobs/pool.js): runs one browser tool's++// server engine (apps/gateway/server/dev/engines.js, apps/text/server/engines.js) on a run's input,++// off the event loop and bounded (the descriptor's timeoutMs terminates it; a heap limit applies).++const loaded = new Map();++++module.exports = {++ /** { module, name, input } → { data } | { text } */++ async run({ module, name, input }) {++ if (!loaded.has(module)) loaded.set(module, require(module));++ const fn = (loaded.get(module).ENGINES || {})[name];++ if (typeof fn !== 'function') throw Object.assign(new Error(`No server engine ${name}`), { status: 500 });++ return fn(input || {});++ },++};nnew file mode 100644@@@ -0,0 +1,188 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// The gateway's run API: POST /api/v1/tools/:id/run for every tool (ADR-027; the protocol is++// apps/_shared/tools/run.js). Where each tool runs:++//++// engines dev and text tools with a server engine (execution client, api true): in the gateway's++// engine pool (worker threads: the descriptor's timeoutMs terminates a runaway input, a heap++// limit applies), the same code the pages run. Cached for 10 minutes unless the spec says++// cacheTtlMs: 0 (random or time-dependent output).++// routes the net tools and Open Graph (execution sync): the /api/net and /api/dev route that already++// answers the page, called in this process with the run's input as its query (spec.route),++// so both give the same answer. The per-target throttle applies; answers are cached only++// where the spec gives a cacheTtlMs (slow-moving lookups, never probes or myip).++// jobs img, audio and docs tools: the run is streamed to the satellite that owns the tool++// (POST /api/v1/tools/:id/run there, uploads and all) and its answer streamed back; the job++// id is remembered for the /api/v1/jobs facade.++// ═══════════════════════════════════════════════════════════════++++const path = require('path');++const { createRunApi, createInputValidator, Refusal } = require('../../../_shared/tools/run');++const { forward } = require('../../../_shared/tools/proxy');++const { poolFromEnv } = require('../../../_shared/jobs/pool');++++const ENGINE_TTL_MS = 10 * 60 * 1000;++++/** A response stand-in that captures what an Express route handler answers. */++function capture() {++ let settle;++ const done = new Promise((r) => { settle = r; });++ const headers = {};++ const res = {++ statusCode: 200, headersSent: false, locals: {}, writableFinished: false,++ status(n) { this.statusCode = n; return this; },++ set(k, v) { if (k && typeof k === 'object') for (const [a, b] of Object.entries(k)) headers[a.toLowerCase()] = b; else headers[String(k).toLowerCase()] = v; return this; },++ header(k, v) { return this.set(k, v); },++ setHeader(k, v) { headers[String(k).toLowerCase()] = v; },++ getHeader(k) { return headers[String(k).toLowerCase()]; },++ removeHeader(k) { delete headers[String(k).toLowerCase()]; },++ cookie() { return this; },++ on() { return this; }, once() { return this; }, removeListener() { return this; },++ json(body) { this.headersSent = true; settle({ status: this.statusCode, headers, body }); return this; },++ send(body) { let b = body; if (typeof b === 'string' || Buffer.isBuffer(b)) { try { b = JSON.parse(String(b)); } catch { /* text */ } } return this.json(b); },++ end(body) { let b = body; try { b = body == null ? null : JSON.parse(String(body)); } catch { b = String(body); } this.headersSent = true; settle({ status: this.statusCode, headers, body: b }); },++ };++ return { res, done, settle };++}++++/** The route handler a router serves for `method path` (path relative to the router, e.g. /dns). */++function routeHandler(router, method, rel) {++ for (const layer of router.stack) {++ if (!layer.route || !layer.route.methods[method.toLowerCase()]) continue;++ const p = layer.route.path;++ if (p === rel || p === `${rel}/:target?`) return layer.route.stack[layer.route.stack.length - 1].handle;++ }++ return null;++}++++/** '{target}' templates from the input, falling back to the input schema's defaults. */++function fill(template, input, schema) {++ return String(template).replace(/\{([a-z_]+)\}/gi, (_, k) => {++ const v = input[k] !== undefined ? input[k] : schema && schema.properties && schema.properties[k] && schema.properties[k].default;++ return v === undefined || v === null ? '' : String(v);++ });++}++++/** What a legacy route answered → the run's data, or a thrown tool error / refusal. */++function fromRoute(out) {++ if (out.error) throw out.error;++ const body = out.body && typeof out.body === 'object' ? out.body : {};++ if (/problem\+json/.test(String(out.headers['content-type'] || ''))) {++ // A guard refusal inside the route (the port checker's caps): the answer to the request itself.++ const { type, title, status, code, detail, error, request_id, trace_id, ...extra } = body;++ const headers = out.headers['retry-after'] ? { 'Retry-After': String(out.headers['retry-after']) } : undefined;++ throw new Refusal(out.status, code || 'tools.run.invalid', detail || error, extra, headers);++ }++ if (out.status < 400 && body.ok !== false) {++ const { ok, ...data } = body;++ return { data };++ }++ const own = /^tools\.[a-z0-9_.]+$/.test(String(body.code || '')) ? body.code : null;++ const status = out.status === 400 ? 422 : out.status || 500;++ const code = own || (out.status === 400 ? 'tools.input.invalid' : 'tools.job.failed');++ throw Object.assign(new Error(body.error || 'The tool failed'), { status, code });++}++++/**++ * @param {object} o++ * @param {object} o.guard @param {object} o.contracts++ * @param {object} o.toolRegistry ../registry/descriptors (snapshot, internal)++ * @param {object} o.routers { net: express.Router, dev: express.Router }++ * @param {() => object} o.ports satellite ports++ * @param {Function} o.parseJson express.json() for local runs++ * @param {object} o.jobIndex job id → satellite (the facade's)++ * @param {object} [o.Ajv] [o.addFormats]++ */++function createGatewayRun(o) {++ const internal = (id) => o.toolRegistry.internal(id);++ const pool = o.pool || poolFromEnv('gateway', path.join(__dirname, 'engine-worker.js'), { size: 2, memoryMb: 256 });++ const handlers = new Map();++++ function kind(d) {++ const s = internal(d.id);++ if (!s) return null;++ if (s.engines && s.spec.engine) return 'engine';++ if ((s.app === 'net' || s.app === 'dev') && s.spec.route) return 'route';++ if (s.satellite && d.execution === 'job') return 'satellite';++ return null;++ }++++ function handlerFor(d, s) {++ if (handlers.has(d.id)) return handlers.get(d.id);++ const router = o.routers[s.app];++ const base = s.app === 'net' ? '/api/net' : '/api/dev';++ const rel = s.spec.route.path.slice(base.length);++ const h = router ? routeHandler(router, s.spec.route.method, rel) : null;++ handlers.set(d.id, h);++ return h;++ }++++ /** Call the tool's route in this process with the run's input as its request. */++ async function viaRoute(d, s, input, ctx) {++ const h = handlerFor(d, s);++ if (!h) throw Object.assign(new Error(`${d.id} has no route here`), { status: 500 });++ const r = s.spec.route;++ const query = {};++ for (const [k, v] of Object.entries(input)) if (k !== 'target') query[k] = Array.isArray(v) ? v.join(',') : String(v);++ for (const [k, v] of Object.entries(r.query || {})) query[k] = fill(v, input, d.input);++ const target = input.target === undefined ? undefined : r.target ? fill(r.target, input, d.input) : String(input.target);++ if (target !== undefined) query.target = target;++ const rel = r.path.replace(/^\/api\/(net|dev)/, '');++ const fake = Object.create(ctx.req);++ Object.defineProperty(fake, 'path', { value: target !== undefined ? `${rel}/${encodeURIComponent(target)}` : rel, enumerable: true });++ fake.params = target !== undefined ? { target } : {};++ fake.query = query;++ fake.method = r.method;++ fake.netTool = d.id;++ fake.body = {};++ const cap = capture();++ Promise.resolve(h(fake, cap.res, (err) => cap.settle({ error: err || new Error('not handled'), headers: {} }))).catch(err => cap.settle({ error: err, headers: {} }));++ return fromRoute(await cap.done);++ }++++ const api = createRunApi({++ guard: o.guard, contracts: o.contracts,++ snapshot: () => o.toolRegistry.snapshot(),++ validate: createInputValidator(o.Ajv, o.addFormats),++ parseJson: o.parseJson,++ runs(d) {++ const k = kind(d);++ return k === 'engine' || k === 'route' ? 'inline' : null;++ },++ async inline(d, input, ctx) {++ const s = internal(d.id);++ if (kind(d) === 'engine') {++ try {++ return await pool.run('run', { module: s.engines, name: s.spec.engine, input }, { timeoutMs: ctx.timeoutMs, signal: ctx.signal });++ } catch (err) {++ if (err.code === 'tools.busy') throw new Refusal(503, 'tools.busy', err.message, undefined, { 'Retry-After': String(err.retryAfter || 10) });++ throw err;++ }++ }++ return viaRoute(d, s, input, ctx);++ },++ cacheTtl(d) {++ const s = internal(d.id);++ if (s.spec.cacheTtlMs !== undefined) return s.spec.cacheTtlMs;++ return kind(d) === 'engine' ? ENGINE_TTL_MS : 0;++ },++ // Open Graph throttles its own target inside the route; the net routes' throttle is a router++ // middleware, which a run does not pass through, so it is applied here.++ targetOf: (d, input) => (internal(d.id).app === 'net' && d.egress ? (input.target || null) : null),++ proxy(d, req, res) {++ if (kind(d) !== 'satellite') return false;++ const s = internal(d.id);++ const port = o.ports()[s.satellite];++ if (!port) return false;++ forward(req, res, {++ port,++ onJson: (body) => { const id = body && body.job && body.job.id; if (id) o.jobIndex.set(id, s.satellite); },++ });++ return true;++ },++ });++++ return { handle: api.handle, stats: () => ({ ...api.stats(), engines: pool.stats() }), pool, api, kind };++}++++module.exports = { createGatewayRun, capture, routeHandler, fromRoute, fill };nnew file mode 100644@@@ -0,0 +1,166 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// The gateway's jobs facade (ADR-027): https://openvibe.tools/api/v1/jobs… fronts every satellite's++// job routes, so a client (openvibe-sdk/jobs without a baseUrl, the run API's Location) needs one origin.++//++// POST /api/v1/jobs routed by the job type's prefix (img.process → img, …): from++// ?type=, the JSON body, or the multipart `type` part (read from++// the head of the stream; the uploads then stream through)++// GET /api/v1/jobs/:id routed by the job id: the satellite that answered its submit or++// DELETE /api/v1/jobs/:id run (remembered here), else the one that says it holds the job++// GET /api/v1/jobs/:id/events when asked on loopback (GET /api/internal/jobs/:id, direct++// GET /api/v1/jobs/:id/files/:n callers only). Events (SSE) and files stream through.++// POST /api/v1/jobs/:id/retry++// PUT /api/v1/jobs/:id/references/:ref++// DELETE /api/v1/jobs/:id/references/:ref++//++// The satellite does everything else (owner scoping, quotas, the Origin check, idempotency): the++// request reaches it with the caller's credentials and address, as through the host proxy.++// ═══════════════════════════════════════════════════════════════++++const { forward } = require('../../../_shared/tools/proxy');++const { JOB_SATELLITES, satelliteForType } = require('../../../_shared/tools/satellites');++++const ID_RE = /^job_[0-9A-HJKMNP-TV-Z]{26}$/;++const PATH_RE = /^\/api\/v1\/jobs(?:\/([^/]+)(\/.*)?)?\/?$/;++const PEEK_MAX = 64 * 1024;++const JSON_MAX = 1024 * 1024;++const TYPE_RE = /^[a-z][a-z0-9]*\.[a-z0-9_.]+$/;++++const isJobsPath = (p) => PATH_RE.test(String(p || ''));++++/** Job id → satellite, most recent first, bounded. */++function createJobIndex(max = 50_000) {++ const map = new Map();++ return {++ get: (id) => map.get(id) || null,++ set(id, sat) { if (!ID_RE.test(String(id)) || !JOB_SATELLITES.includes(sat)) return; map.delete(id); map.set(id, sat); if (map.size > max) map.delete(map.keys().next().value); },++ size: () => map.size,++ };++}++++/** Read the start of a multipart body until the `type` part → { type, head, ended }. */++function peekMultipartType(req) {++ return new Promise((resolve) => {++ const chunks = [];++ let size = 0, finished = false;++ const re = /content-disposition:\s*form-data;\s*name="type"(?:;[^\r\n]*)?\r\n(?:[^\r\n]+\r\n)*\r\n([^\r\n]*)\r\n/i;++ const done = (type, ended) => {++ if (finished) return;++ finished = true;++ req.removeListener('data', onData); req.removeListener('end', onEnd); req.removeListener('error', onEnd);++ if (!ended) req.pause();++ resolve({ type, head: Buffer.concat(chunks), ended });++ };++ const onData = (c) => {++ chunks.push(c); size += c.length;++ const m = re.exec(Buffer.concat(chunks).toString('latin1'));++ if (m) return done(m[1].trim(), false);++ if (size > PEEK_MAX) return done(null, false);++ return undefined;++ };++ const onEnd = () => { const m = re.exec(Buffer.concat(chunks).toString('latin1')); done(m ? m[1].trim() : null, true); };++ req.on('data', onData);++ req.on('end', onEnd);++ req.on('error', onEnd);++ });++}++++/** The whole (small) body → Buffer, or null past `max`. */++function readAll(req, max) {++ return new Promise((resolve) => {++ const chunks = [];++ let size = 0;++ req.on('data', (c) => { size += c.length; if (size <= max) chunks.push(c); });++ req.on('end', () => resolve(size <= max ? Buffer.concat(chunks) : null));++ req.on('error', () => resolve(null));++ });++}++++/**++ * @param {object} o++ * @param {() => object} o.ports satellite ports++ * @param {object} o.index createJobIndex()++ * @param {object} o.contracts++ * @param {Function} [o.fetchImpl]++ */++function createJobsFacade(o) {++ const fetchImpl = o.fetchImpl || fetch;++ const counts = { submitted: 0, routed: 0, probed: 0, not_found: 0 };++ const problem = (req, res, status, code, detail) => {++ res.setHeader('Cache-Control', 'no-store');++ // A body left unread (a refused upload): read and discard it, so the connection stays usable.++ if (!req.readableEnded) req.resume();++ return o.contracts.http.sendProblem(res, status, code, { detail, ctx: req.ov });++ };++ const remember = (sat) => (body) => { if (body && body.id && body.object === 'tools.job') o.index.set(body.id, sat); };++++ /** Which satellite holds job `id`: remembered, else asked (in parallel, on loopback). */++ async function locate(id) {++ const known = o.index.get(id);++ if (known) return known;++ counts.probed++;++ const ports = o.ports();++ const answers = await Promise.all(JOB_SATELLITES.filter(s => ports[s]).map(async (sat) => {++ try {++ const r = await fetchImpl(`http://127.0.0.1:${ports[sat]}/api/internal/jobs/${id}`, { signal: AbortSignal.timeout(2500) });++ return r.status === 200 ? sat : null;++ } catch { return null; }++ }));++ const sat = answers.find(Boolean) || null;++ if (sat) o.index.set(id, sat);++ return sat;++ }++++ async function submit(req, res) {++ let type = typeof req.query.type === 'string' ? req.query.type : null;++ let body = null, head = null;++ if (!type) {++ if (req.is('multipart/form-data')) {++ const p = await peekMultipartType(req);++ type = p.type;++ if (p.ended) body = p.head; else head = p.head;++ } else {++ body = await readAll(req, JSON_MAX);++ if (!body) return problem(req, res, 413, 'tools.job.too_large', 'A JSON job request is at most 1 MB; send files as multipart.');++ try { const j = JSON.parse(body.toString('utf8') || '{}'); type = j && typeof j.type === 'string' ? j.type : null; } catch {++ return problem(req, res, 400, 'tools.job.invalid', 'The body is not valid JSON.');++ }++ }++ }++ if (!type || !TYPE_RE.test(type)) return problem(req, res, 400, 'tools.job.invalid', 'Name the job type (img.process, audio.process or docs.process): the `type` field, as the first multipart part, or ?type=.');++ const sat = satelliteForType(type);++ const port = sat && o.ports()[sat];++ if (!port) return problem(req, res, 400, 'tools.job.unknown_type', `Unknown job type "${type}". Known: ${JOB_SATELLITES.map(s => `${s}.process`).join(', ')}`);++ counts.submitted++;++ return forward(req, res, { port, ...(body ? { body } : { head }), onJson: remember(sat) });++ }++++ return {++ async handle(req, res, next) {++ const m = PATH_RE.exec(req.path);++ if (!m) return next();++ if (req.method === 'OPTIONS') return next();++ try {++ const [, id, rest] = m;++ if (!id) {++ if (req.method !== 'POST') { res.setHeader('Allow', 'POST'); return problem(req, res, 405, 'method_not_allowed', 'POST a job; GET /api/v1/jobs/:id reads one.'); }++ return await submit(req, res);++ }++ if (!ID_RE.test(id)) return problem(req, res, 404, 'tools.job.not_found', 'No such job');++ const sat = await locate(id);++ if (!sat) { counts.not_found++; return problem(req, res, 404, 'tools.job.not_found', 'No such job'); }++ counts.routed++;++ const port = o.ports()[sat];++ const learns = req.method === 'POST' && rest === '/retry';++ return await forward(req, res, { port, ...(learns && { onJson: remember(sat) }) });++ } catch (err) {++ return next(err);++ }++ },++ stats: () => ({ ...counts, known: o.index.size() }),++ locate,++ };++}++++module.exports = { createJobsFacade, createJobIndex, peekMultipartType, isJobsPath, PATH_RE };nnew file mode 100644@@@ -0,0 +1,300 @@++'use strict';++// POST /api/v1/tools/:id/run (ADR-027, tools.run-request@1 → tools.run@1) on real processes: the++// gateway, img and docs, with a stand-in Network for tokens, the guard enforcing.++// • inline tools: dev and text engines (the page's code, in the gateway's worker pool; cached when++// safe, fresh when random), net lookups through their own routes (myip is the caller's address)++// • every answer is a tools.run@1 document; refusals are problem+json with the contract's codes++// (400 tools.run.invalid, 401/403 auth, 404 not_found / not_runnable, 405, 409, 413, 422 with++// errors[], 429 with Retry-After, 503 tools.tool.unavailable vs tools.unavailable)++// • probes need a token with tools.net.probe; tools that need a session refuse the anonymous++// • job tools stream through the gateway to their satellite: multipart uploads as `file` parts,++// wait_ms (200 finished) or not (202 + Location), Idempotency-Key replays (same job) and++// conflicts, file references ({ job_id, index }) across satellites and their owner check++// • the Origin check on cookie-authenticated calls (CSRF) and CORS for other sites' pages++// • the older endpoints answer as before with Deprecation, Sunset and a successor Link++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const contracts = require('openvibe-contracts');++const { startApp, freePort } = require('../../_shared/test/spawn');++const { startNetwork } = require('../../_shared/test/network');++++const APPS = path.join(__dirname, '..', '..');++const sharp = require(require.resolve('sharp', { paths: [path.join(APPS, 'img')] }));++const { PDFDocument } = require(require.resolve('pdf-lib', { paths: [path.join(APPS, 'docs')] }));++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-run-api-'));++const MISSING = path.join(tmp, 'not-installed');++let ipN = 1;++const freshIp = () => `198.51.100.${ipN++}`;++++function checkRun(body, where) {++ const r = contracts.validate('tools.run@1', body);++ assert.ok(r.valid, `${where}: tools.run@1 ${JSON.stringify(r.errors && r.errors.slice(0, 3))} ${JSON.stringify(body).slice(0, 300)}`);++}++function checkProblem(r, status, code, where) {++ assert.strictEqual(r.status, status, `${where}: ${r.text.slice(0, 300)}`);++ assert.match(r.h.get('content-type'), /application\/problem\+json/, where);++ assert.strictEqual(r.body.code, code, `${where}: ${r.text.slice(0, 300)}`);++}++++(async () => {++ const net = await startNetwork();++ const procs = [];++ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };++ try {++ // Every app knows where the others listen (a docs run reads an img job's result on loopback).++ const ports = { img: await freePort(), docs: await freePort() };++ const auth = { OV_NETWORK_URL: net.url, OV_NETWORK_INTERNAL_URL: net.url, TOOLS_GUARD: 'enforce', TOOLS_SATELLITE_PORTS: `img=${ports.img},docs=${ports.docs}` };++ const img = await start('img', { ...auth, DATA_DIR: path.join(tmp, 'img'), UPLOADS_DIR: path.join(tmp, 'img', 'up'), OUTPUT_DIR: path.join(tmp, 'img', 'out'), HEIF_DEC_PATH: MISSING }, ports.img);++ const docs = await start('docs', { ...auth, DATA_DIR: path.join(tmp, 'docs'), UPLOADS_DIR: path.join(tmp, 'docs', 'up'), OUTPUT_DIR: path.join(tmp, 'docs', 'out'), QPDF_PATH: MISSING }, ports.docs);++ const gw = await start('gateway', {++ ...auth, DATA_DIR: path.join(tmp, 'gw'),++ OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry',++ TOOLS_GUARD_LIMITS: JSON.stringify({ 'tools-run': { anonymous: { perMinute: 40, burst: 40 } } }),++ });++ const G = gw.base;++ const call = async (p, { method = 'POST', json, form, headers = {}, base = G } = {}) => {++ const h = { 'X-Forwarded-For': headers['X-Forwarded-For'] || freshIp(), ...headers };++ let body;++ if (json !== undefined) { h['Content-Type'] = 'application/json'; body = typeof json === 'string' ? json : JSON.stringify(json); }++ if (form) body = form;++ const r = await fetch(`${base}${p}`, { method, headers: h, body });++ const text = await r.text();++ let parsed = null; try { parsed = text ? JSON.parse(text) : null; } catch { /* not JSON */ }++ return { status: r.status, h: r.headers, text, body: parsed };++ };++ const run = (id, json, opts = {}) => call(`/api/v1/tools/${id}/run`, { json, ...opts });++ // The gateway reads each satellite's own status on boot (protectpdf without qpdf).++ for (let i = 0; i < 50 && (await call('/api/v1/tools/protectpdf', { method: 'GET' })).body.status !== 'unavailable'; i++) await new Promise(r => setTimeout(r, 100));++++ // ── Inline: a dev engine (the page's own code) ──++ let r = await run('jsonminify', { input: { text: '{ "a": [1, 2] }' } });++ assert.strictEqual(r.status, 200, r.text);++ checkRun(r.body, 'jsonminify');++ assert.deepStrictEqual([r.body.state, r.body.tool, r.body.result], ['succeeded', 'jsonminify', { text: '{"a":[1,2]}' }]);++ assert.strictEqual(r.h.get('cache-control'), 'no-store');++ assert.ok(Number.isInteger(r.body.took_ms));++ r = await run('jsonminify', { input: { text: '{ "a": [1, 2] }' } });++ assert.strictEqual(r.h.get('x-ov-cache'), 'hit', 'a pure transform is answered from the cache');++ // A text engine; the JSON a tool answers as data.++ r = await run('count', { input: { text: 'two words' } });++ checkRun(r.body, 'count');++ assert.strictEqual(r.body.result.data.words, 2);++ // Random output is never reused.++ const a = await run('uuid', { input: { count: 1 } }), b = await run('uuid', { input: { count: 1 } });++ assert.notStrictEqual(a.body.result.data.uuids[0], b.body.result.data.uuids[0]);++ assert.ok(!b.h.get('x-ov-cache'));++ // The tool's own failure is a failed run (200), with its problem.++ r = await run('jsonminify', { input: { text: '{ nope' } });++ assert.strictEqual(r.status, 200);++ checkRun(r.body, 'jsonminify failed');++ assert.deepStrictEqual([r.body.state, r.body.error.status, r.body.error.code], ['failed', 422, 'tools.job.failed']);++ r = await run('compare', { input: { a: 'x\n'.repeat(2500), b: 'y' } });++ assert.deepStrictEqual([r.body.state, r.body.error.code], ['failed', 'tools.input.invalid'], 'an engine\'s own code is kept');++++ // ── Refusals before the tool runs ──++ r = await run('jsonminify', { input: { text: 'x', nope: 1 } });++ checkProblem(r, 422, 'tools.input.invalid', 'schema');++ assert.deepStrictEqual(r.body.errors, [{ path: '/nope', message: 'is not a field of this tool' }]);++ r = await run('jsonminify', { input: {} });++ assert.deepStrictEqual(r.body.errors, [{ path: '/text', message: 'is required' }]);++ checkProblem(await run('jsonminify', { input: { text: 'x' }, extra: 1 }), 400, 'tools.run.invalid', 'unknown field');++ checkProblem(await run('jsonminify', { input: { text: 'x' }, wait_ms: 60001 }), 400, 'tools.run.invalid', 'wait_ms');++ checkProblem(await run('jsonminify', '{"input":'), 400, 'tools.run.invalid', 'not JSON');++ checkProblem(await run('jsonminify', { input: [] }), 400, 'tools.run.invalid', 'input array');++ checkProblem(await run('jsonminify', { input: { text: 'x' } }, { headers: { 'Idempotency-Key': 'short' } }), 400, 'tools.run.invalid', 'bad key');++ checkProblem(await run('dns', { input: { target: 'a'.repeat(4500) } }), 413, 'tools.input.too_large', 'maxInputBytes');++ checkProblem(await run('nosuchtool', {}), 404, 'tools.tool.not_found', 'unknown');++ checkProblem(await run('yt', {}), 404, 'tools.tool.not_runnable', 'yt: api false');++ checkProblem(await run('regex', { input: {} }), 404, 'tools.tool.not_runnable', 'regex: page only');++ r = await run('traceroute', {});++ checkProblem(r, 503, 'tools.tool.unavailable', 'descriptor status unavailable');++ assert.match(r.body.detail, /raw network access/);++ checkProblem(await run('protectpdf', { input: { password: 'x' } }), 503, 'tools.tool.unavailable', 'qpdf missing on docs');++ checkProblem(await call('/api/v1/tools/jsonminify/run', { method: 'GET' }), 405, 'method_not_allowed', 'GET');++++ // ── Net tools through their routes ──++ r = await run('myip', { input: {} }, { headers: { 'X-Forwarded-For': '203.0.113.77' } });++ checkRun(r.body, 'myip');++ assert.deepStrictEqual(r.body.result.data, { ip: '203.0.113.77', version: 4 }, 'the caller\'s own address');++ r = await run('ipv4', { input: { target: '10.1.2.0/24' } });++ checkRun(r.body, 'ipv4');++ assert.strictEqual(r.body.result.data.cidr.usable, 254);++ r = await run('ipv4', { input: { target: '10.1.2.0/24' } });++ assert.strictEqual(r.h.get('x-ov-cache'), 'hit', 'lookups with a cacheTtlMs are reused');++ // A target the SSRF guard refuses: the tool ran and failed with its own code.++ r = await run('headers', { input: { target: 'http://127.0.0.1:1/' } });++ checkRun(r.body, 'headers refused');++ assert.deepStrictEqual([r.body.state, r.body.error.status, r.body.error.code], ['failed', 403, 'tools.net.target_not_public']);++++ // ── Probes: a token with tools.net.probe, nobody else ──++ checkProblem(await run('ping', { input: { target: '127.0.0.1' } }), 401, 'token.missing', 'probe, anonymous');++ checkProblem(await run('ping', { input: { target: '127.0.0.1' } }, { headers: { Authorization: `Bearer ${net.user()}` } }), 403, 'capability.denied', 'probe, a person');++ checkProblem(await run('ping', { input: { target: '127.0.0.1' } }, { headers: { Authorization: `Bearer ${net.service(['tools.tool.run'])}` } }), 403, 'capability.denied', 'probe, runner token');++ r = await run('ping', { input: { target: '127.0.0.1', count: 1 } }, { headers: { Authorization: `Bearer ${net.service(['tools.net.probe'], { name: 'prober' })}` } });++ checkRun(r.body, 'ping with tools.net.probe');++ assert.strictEqual(r.body.error.code, 'tools.net.target_not_public', 'authorized: it ran (and the SSRF guard kept it off loopback)');++ checkProblem(await run('jsonminify', { input: { text: '1' } }, { headers: { Authorization: `Bearer ${net.service(['tools.job.read'])}` } }), 403, 'capability.denied', 'a token without tools.tool.run');++ checkProblem(await run('jsonminify', { input: { text: '1' } }, { headers: { Authorization: 'Bearer not.a.token' } }), 401, 'token.invalid', 'a token that does not verify');++ r = await run('jsonminify', { input: { text: '1' } }, { headers: { Authorization: `Bearer ${net.service(['tools.tool.run'])}` } });++ assert.strictEqual(r.body.state, 'succeeded', 'a runner token runs');++++ // ── Quotas: the tool's class, per caller ──++ {++ const ip = freshIp();++ const codes = [];++ for (let i = 0; i < 80 && !codes.includes(429); i++) {++ r = await run('slug', { input: { text: `q ${i}` } }, { headers: { 'X-Forwarded-For': ip } });++ codes.push(r.status);++ }++ assert.ok(codes.slice(0, 40).every(c => c === 200), JSON.stringify(codes));++ checkProblem(r, 429, 'tools.quota.exceeded', 'tools-run, anonymous: a burst of 40, then 40 a minute');++ assert.ok(Number(r.h.get('retry-after')) >= 1);++ assert.strictEqual(r.body.quota_class, 'tools-run');++ }++++ // ── Inline Idempotency-Key: the same run again, or a conflict ──++ // (anonymous callers are keyed by address: the same one each time)++ const same = await run('lorem', { input: { unit: 'words', count: 4 } }, { headers: { 'Idempotency-Key': 'inline-key-0002', 'X-Forwarded-For': '192.0.2.9' } });++ const replay = await run('lorem', { input: { unit: 'words', count: 4 } }, { headers: { 'Idempotency-Key': 'inline-key-0002', 'X-Forwarded-For': '192.0.2.9' } });++ assert.strictEqual(replay.h.get('idempotent-replayed'), 'true');++ assert.deepStrictEqual(replay.body, same.body, 'a random generator gives the first answer again');++ checkProblem(await run('lorem', { input: { unit: 'words', count: 5 } }, { headers: { 'Idempotency-Key': 'inline-key-0002', 'X-Forwarded-For': '192.0.2.9' } }), 409, 'tools.job.idempotency_conflict', 'same key, other input');++++ // ── Job tools through the gateway ──++ const png = await sharp({ create: { width: 8, height: 6, channels: 3, background: '#0a0' } }).png().toBuffer();++ const upload = (fields, files) => {++ const fd = new FormData();++ for (const [k, v] of Object.entries(fields)) fd.append(k, typeof v === 'string' ? v : JSON.stringify(v));++ for (const f of files) fd.append('file', new Blob([f.data], f.type ? { type: f.type } : {}), f.name);++ return fd;++ };++ const person = net.user();++ const as = { Authorization: `Bearer ${person}` };++ // Finished within wait_ms: 200 with the result files.++ r = await call('/api/v1/tools/webp/run', { form: upload({ input: { quality: 70 }, wait_ms: '10000' }, [{ name: 'photo.png', data: png }]), headers: as });++ assert.strictEqual(r.status, 200, r.text);++ checkRun(r.body, 'webp waited');++ assert.strictEqual(r.body.state, 'succeeded');++ assert.deepStrictEqual([r.body.job.tool, r.body.job.service, r.body.job.type], ['webp', 'img', 'img.process']);++ assert.strictEqual(r.body.result.files[0].mime, 'image/webp');++ assert.strictEqual(r.body.result.files[0].name, 'photo.webp');++ assert.strictEqual(r.body.result.data.output.ext, 'webp');++ const webpJob = r.body.job;++ let f = await fetch(`${G}${r.body.result.files[0].url}`, { headers: as });++ assert.strictEqual(f.status, 200);++ assert.strictEqual((await sharp(Buffer.from(await f.arrayBuffer())).metadata()).format, 'webp', 'the file, through the gateway\'s facade');++ // Not waiting: 202 + Location, then the job.++ const k1 = 'job-key-00000001';++ r = await call('/api/v1/tools/png/run', { form: upload({ input: {} }, [{ name: 'x.png', data: png }]), headers: { ...as, 'Idempotency-Key': k1 } });++ assert.strictEqual(r.status, 202, r.text);++ checkRun(r.body, 'png 202');++ assert.ok(['queued', 'running'].includes(r.body.state));++ assert.strictEqual(r.h.get('location'), `/api/v1/jobs/${r.body.job.id}`);++ assert.strictEqual(r.body.location, r.h.get('location'));++ const pngJob = r.body.job.id;++ // The same key and request: the same job (200 + Idempotent-Replayed).++ r = await call('/api/v1/tools/png/run', { form: upload({ input: {} }, [{ name: 'x.png', data: png }]), headers: { ...as, 'Idempotency-Key': k1 } });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.h.get('idempotent-replayed'), 'true');++ assert.strictEqual(r.body.job.id, pngJob);++ checkRun(r.body, 'png replay');++ // Another request under that key: 409.++ r = await call('/api/v1/tools/png/run', { form: upload({ input: { quality: 5 } }, [{ name: 'x.png', data: png }]), headers: { ...as, 'Idempotency-Key': k1 } });++ checkProblem(r, 409, 'tools.job.idempotency_conflict', 'job key reused');++ // A declared type the bytes contradict is corrected; bytes that are no image are refused (415).++ r = await call('/api/v1/tools/png/run', { form: upload({ wait_ms: '10000' }, [{ name: 'x.bin', data: png, type: 'application/octet-stream' }]), headers: as });++ assert.strictEqual(r.body.state, 'succeeded', 'sniffed as PNG whatever the part said');++ r = await call('/api/v1/tools/png/run', { form: upload({}, [{ name: 'x.png', data: Buffer.from('not an image at all') }]), headers: as });++ checkProblem(r, 415, 'tools.file.unsupported_type', 'sniffing');++ r = await call('/api/v1/tools/png/run', { form: upload({}, [{ name: 'a.png', data: png }, { name: 'b.png', data: png }]), headers: as });++ checkProblem(r, 400, 'tools.run.invalid', 'two files for a one-file tool');++ checkProblem(await run('png', { input: {} }, { headers: as }), 400, 'tools.run.invalid', 'no file');++ r = await call('/api/v1/tools/png/run', { form: upload({ input: { quality: 500 } }, [{ name: 'x.png', data: png }]), headers: as });++ checkProblem(r, 422, 'tools.input.invalid', 'job input schema');++ // A program the engine needs is missing (HEIC without libheif): the run fails with tools.unavailable.++ const heic = Buffer.concat([Buffer.from([0, 0, 0, 0x18]), Buffer.from('ftypheic'), Buffer.alloc(4), Buffer.from('mif1heic'), Buffer.alloc(64)]);++ r = await call('/api/v1/tools/convert/run', { form: upload({ input: { format: 'png' }, wait_ms: '10000' }, [{ name: 'p.heic', data: heic }]), headers: as });++ checkRun(r.body, 'heic');++ assert.deepStrictEqual([r.body.state, r.body.error.status, r.body.error.code], ['failed', 503, 'tools.unavailable'], JSON.stringify(r.body.error));++ checkProblem(await run('heic', { input: {} }, { headers: as }), 503, 'tools.tool.unavailable', 'the heic tool itself');++++ // Docs: a session, a sign-in or a token (auth.anonymous false).++ const pdf = async (pages) => { const d = await PDFDocument.create(); for (let i = 0; i < pages; i++) d.addPage([100, 100]); return Buffer.from(await d.save()); };++ const p1 = await pdf(1), p2 = await pdf(2);++ r = await call('/api/v1/tools/mergepdf/run', { form: upload({}, [{ name: 'a.pdf', data: p1 }, { name: 'b.pdf', data: p2 }]) });++ checkProblem(r, 401, 'tools.session_required', 'merge, anonymous');++ r = await call('/api/v1/tools/mergepdf/run', { form: upload({ input: { order: [1, 0] }, wait_ms: '15000' }, [{ name: 'a.pdf', data: p1 }, { name: 'b.pdf', data: p2 }]), headers: as });++ checkRun(r.body, 'merge');++ assert.deepStrictEqual([r.body.state, r.body.result.data.pageCount, r.body.job.service], ['succeeded', 3, 'docs']);++ // A file reference: the webp result (img) into Image to PDF (docs), same owner.++ r = await run('image2pdf', { input: { pageSize: 'fit' }, files: [{ job_id: webpJob.id, index: 0 }], wait_ms: 15000 }, { headers: as });++ checkRun(r.body, 'image2pdf from a job reference');++ assert.strictEqual(r.body.state, 'succeeded', JSON.stringify(r.body.error || r.body));++ assert.strictEqual(r.body.result.data.pageCount, 1);++ // Somebody else's job, or no such file: 404 tools.run.file_not_found.++ checkProblem(await run('image2pdf', { files: [{ job_id: webpJob.id, index: 0 }] }, { headers: { Authorization: `Bearer ${net.user()}` } }), 404, 'tools.run.file_not_found', 'another owner\'s job');++ checkProblem(await run('image2pdf', { files: [{ job_id: webpJob.id, index: 3 }] }, { headers: as }), 404, 'tools.run.file_not_found', 'no such index');++ checkProblem(await run('image2pdf', { files: [{ media_id: 'med_01J00000000000000000000000' }] }, { headers: as }), 404, 'tools.run.file_not_found', 'unknown media id');++ checkProblem(await run('image2pdf', { files: [{ job_id: 'nope' }] }, { headers: as }), 400, 'tools.run.invalid', 'a malformed reference');++++ // ── Origin check (CSRF): cookies from another site's page ──++ const session = (await call('/api/context', { method: 'GET', base: img.base })).h.get('set-cookie').split(';')[0];++ const evil = { Cookie: session, Origin: 'https://evil.example' };++ r = await run('slug', { input: { text: 'x' } }, { headers: evil });++ checkProblem(r, 403, 'tools.origin.refused', 'cookie + foreign origin');++ assert.strictEqual(r.h.get('access-control-allow-origin'), '*', 'another site gets CORS without credentials');++ assert.ok(!r.h.get('access-control-allow-credentials'));++ assert.strictEqual((await run('slug', { input: { text: 'x' } }, { headers: { Cookie: session, Origin: 'https://slug.openvibe.tools' } })).status, 200, 'our own pages');++ assert.strictEqual((await run('slug', { input: { text: 'x' } }, { headers: { Origin: 'https://evil.example' } })).status, 200, 'no cookies: nothing to forge');++ assert.strictEqual((await run('slug', { input: { text: 'x' } }, { headers: { ...evil, Authorization: `Bearer ${net.service(['tools.tool.run'])}` } })).status, 200, 'a Bearer token is not a cookie');++ r = await call('/api/v1/tools/png/run', { form: upload({}, [{ name: 'x.png', data: png }]), headers: evil });++ checkProblem(r, 403, 'tools.origin.refused', 'a job tool, checked on its satellite');++ // The older endpoints' CORS already turns a foreign Origin away; a foreign Referer alone is refused here.++ r = await call('/api/process', { base: img.base, form: upload({ tool: 'convert', format: 'webp' }, [{ name: 'x.png', data: png, type: 'image/png' }]), headers: { Cookie: session, Referer: 'https://evil.example/page' } });++ checkProblem(r, 403, 'tools.origin.refused', 'the older endpoints too');++ r = await call('/api/v1/jobs', { base: img.base, form: upload({ type: 'img.process', input: { tool: 'convert' } }, [{ name: 'x.png', data: png, type: 'image/png' }]), headers: { Cookie: session, Origin: 'https://evil.example' } });++ assert.strictEqual(r.status, 403, 'and the job routes');++ // Preflight from another site's page.++ r = await call('/api/v1/tools/slug/run', { method: 'OPTIONS', headers: { Origin: 'https://app.example', 'Access-Control-Request-Method': 'POST', 'Access-Control-Request-Headers': 'authorization,content-type,idempotency-key' } });++ assert.strictEqual(r.status, 204);++ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');++ assert.match(r.h.get('access-control-allow-headers'), /Idempotency-Key/);++++ // ── The older endpoints: same answers, and what replaces them ──++ r = await call('/api/net/myip', { method: 'GET' });++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.body.ok, true, 'the legacy shape');++ assert.strictEqual(r.h.get('deprecation'), 'true');++ assert.strictEqual(r.h.get('sunset'), 'Thu, 31 Dec 2026 23:59:59 GMT');++ assert.strictEqual(r.h.get('link'), '</api/v1/tools/myip/run>; rel="successor-version"');++ assert.strictEqual((await call('/api/net/dns/example.invalid?types=MX', { method: 'GET' })).h.get('link'), '</api/v1/tools/dns/run>; rel="successor-version"');++ assert.strictEqual((await call('/api/dev/tools', { method: 'GET' })).h.get('link'), '</api/v1/tools?family=dev>; rel="successor-version"');++ r = await call('/api/process', { base: img.base, form: upload({ tool: 'convert', format: 'webp' }, [{ name: 'x.png', data: png, type: 'image/png' }]) });++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.body.success, true, 'the legacy shape');++ assert.strictEqual(r.h.get('deprecation'), 'true');++ assert.strictEqual(r.h.get('link'), '</api/v1/tools/convert/run>; rel="successor-version"');++ {++ // On png.openvibe.tools the successor is png's run route (fetch cannot set Host: node:http).++ const req = new Request('http://x/', { method: 'POST', body: upload({ tool: 'convert' }, [{ name: 'x.png', data: png, type: 'image/png' }]) });++ const bodyBuf = Buffer.from(await req.arrayBuffer());++ const link = await new Promise((resolve, reject) => {++ const q = require('http').request({ host: '127.0.0.1', port: img.port, method: 'POST', path: '/api/process', headers: { Host: 'png.openvibe.tools', 'Content-Type': req.headers.get('content-type'), 'Content-Length': bodyBuf.length, 'X-Forwarded-For': freshIp() } }, (res) => { res.resume(); resolve(res.headers.link); });++ q.on('error', reject); q.end(bodyBuf);++ });++ assert.strictEqual(link, '</api/v1/tools/png/run>; rel="successor-version"', 'the host\'s own tool');++ }++++ // ── Health shows the run API ──++ const health = (await call('/api/health', { method: 'GET' })).body;++ assert.ok(health.run.runs > 50 && health.run.cached >= 2 && health.run.proxied >= 5, JSON.stringify(health.run));++ assert.ok(health.run.engines.completed >= 5, 'engines ran in worker threads');++ } finally {++ for (const p of procs) await p.kill('SIGTERM');++ await net.close();++ }++ console.log('run API: inline engines (cache, fresh), net routes (myip, SSRF), refusals and contract codes, probes need tools.net.probe, quotas, inline and job idempotency, job tools via the gateway (wait, 202, sniffing, schema), tools.unavailable vs tools.tool.unavailable, session rule, cross-satellite job references, CSRF origin check + CORS, deprecation headers: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,127 @@++'use strict';++// openvibe-sdk v0.6.0's Tools client (createToolsClient: registry, run API, jobs facade) against the++// real gateway, img and docs, exactly as a developer uses it: no baseUrl (the `tools` origin), the++// registry read without a token, a sync tool run anonymously, a job tool with an upload (202, then++// wait() over the facade's SSE, then the file), waitMs, Idempotency-Key replays, a job's result fed++// to the next tool by reference, ToolRunError for a failed run, OpenVibeError for a refusal, and a++// probe with a tools.net.probe token.++//++// The SDK is required from a checkout next to this repo (../../../../OpenVibe.SDK, or OV_SDK_DIR);++// without one this test says so and passes (it is not a dependency of any app).++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const { startApp, freePort } = require('../../_shared/test/spawn');++const { startNetwork } = require('../../_shared/test/network');++++const SDK = process.env.OV_SDK_DIR || path.join(__dirname, '..', '..', '..', '..', 'OpenVibe.SDK');++if (!fs.existsSync(path.join(SDK, 'src', 'tools.js'))) {++ console.log(`sdk client: skipped (no OpenVibe.SDK checkout with src/tools.js at ${SDK}; set OV_SDK_DIR)`);++ process.exit(0);++}++const sdkVersion = require(path.join(SDK, 'package.json')).version;++const { createClient, isOpenVibeError } = require(path.join(SDK, 'src', 'core'));++const { createToolsClient, isToolRunError } = require(path.join(SDK, 'src', 'tools'));++++const APPS = path.join(__dirname, '..', '..');++const sharp = require(require.resolve('sharp', { paths: [path.join(APPS, 'img')] }));++const rejects = (p) => p.then((v) => { throw new Error(`expected a rejection, got ${JSON.stringify(v).slice(0, 200)}`); }, (err) => err);++++(async () => {++ assert.ok(/^0\.(6|[7-9])\.|^[1-9]/.test(sdkVersion), `the SDK is v0.6.0 or later (${sdkVersion})`);++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-sdk-client-'));++ const net = await startNetwork();++ const procs = [];++ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };++ try {++ const ports = { img: await freePort(), docs: await freePort() };++ const env = { OV_NETWORK_URL: net.url, OV_NETWORK_INTERNAL_URL: net.url, TOOLS_SATELLITE_PORTS: `img=${ports.img},docs=${ports.docs}` };++ await start('img', { ...env, DATA_DIR: path.join(tmp, 'img'), UPLOADS_DIR: path.join(tmp, 'img', 'up'), OUTPUT_DIR: path.join(tmp, 'img', 'out') }, ports.img);++ await start('docs', { ...env, DATA_DIR: path.join(tmp, 'docs'), UPLOADS_DIR: path.join(tmp, 'docs', 'up'), OUTPUT_DIR: path.join(tmp, 'docs', 'out') }, ports.docs);++ const gw = await start('gateway', { ...env, DATA_DIR: path.join(tmp, 'gw'), OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' });++++ // The `tools` service origin is the gateway (what discovery gives in production).++ const client = (token) => createClient({ baseUrls: { tools: gw.base }, autoDiscover: false, ...(token && { token }), onWarning: () => {} });++ const anon = createToolsClient(client());++ const person = createToolsClient(client(net.user()));++++ // ── Registry ──++ const list = await anon.list({ family: 'dev', api: true });++ assert.ok(list.count >= 20 && list.tools.every(t => t.family === 'dev' && t.api), 'filtered list');++ const png = await anon.get('png');++ assert.deepStrictEqual([png.id, png.execution, png.run.job.type], ['png', 'job', 'img.process']);++ assert.strictEqual(await anon.get('nosuchtool'), null, 'no such tool → null');++ const schema = await anon.schema('jsonminify');++ assert.deepStrictEqual(schema.$defs.input.required, ['text']);++++ // ── A sync tool, anonymously ──++ const out = await anon.run('jsonminify', { text: '{\n "a": [1, 2]\n}' });++ assert.deepStrictEqual([out.state, out.tool, out.result.text, out.replayed], ['succeeded', 'jsonminify', '{"a":[1,2]}', false]);++ assert.match(out.idempotencyKey, /^[!-~]{8,200}$/, 'the SDK sent an Idempotency-Key');++ assert.strictEqual(await out.wait(), out, 'a finished run waits for nothing');++ const again = await anon.run('jsonminify', { text: '{\n "a": [1, 2]\n}' }, { idempotencyKey: out.idempotencyKey });++ assert.strictEqual(again.replayed, true, 'the same key again: the same run');++ const lookup = await anon.run('ipv4', { target: '192.0.2.0/30' });++ assert.strictEqual(lookup.result.data.cidr.usable, 2);++ // The tool's own failure → ToolRunError; a refusal → OpenVibeError with the problem's code.++ let err = await rejects(anon.run('jsonminify', { text: '{ bad' }));++ assert.ok(isToolRunError(err), err.message);++ assert.deepStrictEqual([err.state, err.code, err.status, err.tool], ['failed', 'tools.job.failed', 422, 'jsonminify']);++ err = await rejects(anon.run('yt', {}));++ assert.ok(isOpenVibeError(err) && !isToolRunError(err));++ assert.deepStrictEqual([err.status, err.code], [404, 'tools.tool.not_runnable']);++ err = await rejects(anon.run('jsonminify', { text: 'x', nope: true }));++ assert.deepStrictEqual([err.status, err.code], [422, 'tools.input.invalid']);++ assert.ok(Array.isArray(err.errors) && err.errors[0].path === '/nope');++ err = await rejects(anon.run('ping', { target: '192.0.2.1' }));++ assert.deepStrictEqual([err.status, err.code], [401, 'token.missing'], 'probes need a token');++++ // ── A job tool: 202, wait() over the facade's events, the file ──++ const image = await sharp({ create: { width: 10, height: 8, channels: 3, background: '#c00' } }).png().toBuffer();++ const queued = await person.run('webp', { quality: 60 }, { files: [{ name: 'photo.png', data: image, type: 'image/png' }] });++ assert.ok(['queued', 'running', 'succeeded'].includes(queued.state), queued.state);++ assert.ok(queued.job && /^\/api\/v1\/jobs\/job_/.test(queued.location));++ const seen = [];++ const done = await queued.wait({ onEvent: (e) => { seen.push(e.event); } });++ assert.deepStrictEqual([done.state, done.tool, done.job.tool], ['succeeded', 'webp', 'webp']);++ assert.ok(queued.state === 'succeeded' || seen.includes('job.succeeded'), `events: ${seen.join(', ')}`);++ const file = done.result.files[0];++ assert.deepStrictEqual([file.name, file.mime], ['photo.webp', 'image/webp']);++ const res = await person.jobs.file(done.job.id, 0);++ assert.strictEqual((await sharp(Buffer.from(await res.arrayBuffer())).metadata()).format, 'webp', 'the file through the facade');++ assert.strictEqual((await person.jobs.get(done.job.id)).state, 'succeeded', 'jobs.get through the facade');++++ // waitMs: answered finished (200); an upload without a declared type is sniffed.++ const quick = await person.run('png', {}, { files: [{ name: 'again.bin', data: image }], waitMs: 15000 });++ assert.strictEqual(quick.state, 'succeeded');++ assert.strictEqual(quick.result.files[0].mime, 'image/png');++ // The same key and request again: the same job.++ const key = 'sdk-job-key-0001';++ const k1 = await person.run('jpg', {}, { files: [{ name: 'k.png', data: image, type: 'image/png' }], idempotencyKey: key });++ const k2 = await person.run('jpg', {}, { files: [{ name: 'k.png', data: image, type: 'image/png' }], idempotencyKey: key });++ assert.strictEqual(k2.replayed, true);++ assert.strictEqual(k2.job.id, k1.job.id);++ // One tool's result feeds the next (img → docs), by reference.++ const pdf = await person.run('image2pdf', { pageSize: 'fit' }, { files: [{ job_id: done.job.id, index: 0 }], waitMs: 15000 });++ assert.deepStrictEqual([pdf.state, pdf.result.data.pageCount, pdf.job.service], ['succeeded', 1, 'docs']);++ // A job that fails: ToolRunError with the job attached (retry through the facade).++ const broken = Buffer.concat([image.subarray(0, 40), Buffer.alloc(100, 3)]);++ err = await rejects(person.run('png', {}, { files: [{ name: 'broken.png', data: broken }], waitMs: 15000 }));++ assert.ok(isToolRunError(err), err.message);++ assert.strictEqual(err.state, 'failed');++ assert.ok(err.job && err.job.id);++ const retried = await person.jobs.retry(err.job.id);++ assert.strictEqual(retried.job.retry_of, err.job.id, 'jobs.retry through the facade');++++ // ── A probe with tools.net.probe: authorized (the SSRF guard keeps it off this host) ──++ const prober = createToolsClient(client(net.service(['tools.net.probe'], { name: 'prober' })));++ err = await rejects(prober.run('ping', { target: '127.0.0.1', count: 1 }));++ assert.ok(isToolRunError(err));++ assert.strictEqual(err.code, 'tools.net.target_not_public');++ } finally {++ for (const p of procs) await p.kill('SIGTERM');++ await net.close();++ }++ console.log(`sdk client (openvibe-sdk ${sdkVersion}): registry, sync run + replay, ToolRunError vs refusals, job run + wait() + file via the facade, waitMs, idempotent job, job reference chain img → docs, retry, probe token: all checks passed`);++})().catch((err) => { console.error(err); process.exit(1); });@@@ -44,6 +44,14 @@ const get = async (url, headers = {}, method = 'GET') => {const ports = Object.entries(sat).map(([n, p]) => `${n}=${p.port}`).join(',');const gw = await start('gateway', { TOOLS_SATELLITE_PORTS: ports, OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' });const G = gw.base;++ // The gateway reads every satellite's own status on boot: wait for all seven, so the list does++ // not change between the ETag checks below (a busy machine answers slower than the first request).++ for (let i = 0; i < 200; i++) {++ const rd = (await get(`${G}/api/ready`)).body;++ const d = rd && rd.checks && rd.checks.tool_registry && rd.checks.tool_registry.detail;++ if (d && d.satellites_polled >= 7) break;++ await new Promise(res => setTimeout(res, 100));++ }// ── The list: a tools.tool-list@1 document with every tool ──let r = await get(`${G}/api/v1/tools`, { Origin: 'https://someone-elses-site.example' });@@@ -24,6 +24,9 @@ const { poolFromEnv } = require('../../_shared/jobs/pool');const codec = require('./tools/codec');const { hostGuard, ownHost } = require('../../_shared/host-role');const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { satelliteRunApi, ajvFrom } = require('../../_shared/tools/run');++const { satellitePorts } = require('../../_shared/tools/satellites');++const { deprecated, runPath } = require('../../_shared/tools/deprecation');const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const jobsRuntime = require('../../_shared/jobs');const { createGuard, TRUST_PROXY } = require('../../_shared/guard');@@@ -138,7 +141,7 @@ app.use(exceptRegistry(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--})));++}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──app.use(guard.identify);@@@ -202,7 +205,7 @@ app.get('/api/tools', (_req, res) => {});// ── Main Processing Endpoint ─────────────────────────────────--app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {++app.post('/api/process', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {try {const toolId = req.body.tool || req.ctx.defaultOp || 'convert';const tool = getTool(toolId);@@@ -229,7 +232,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)});// ── Direct Download (inline preview) ─────────────────────────--app.post('/api/process/direct', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {++app.post('/api/process/direct', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {try {const toolId = req.body.tool || req.ctx.defaultOp || 'convert';const tool = getTool(toolId);@@@ -267,6 +270,17 @@ const jobs = jobsRuntime.setupJobs({},});++// ── Run API (ADR-027): POST /api/v1/tools/:id/run for this app's tools ──++// tools.run-request@1 (JSON, or multipart with `file` parts and the text parts input, files, wait_ms)++// → tools.run@1: each tool runs as a job of this satellite (the preset and operation from its++// descriptor), answered finished within wait_ms or 202 with the job. The gateway streams runs here.++const runApi = satelliteRunApi({++ app: 'img', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,++ multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),++ limiters: [burstLimiter, processLimiter],++});++app.use(runApi.handle);++// ── File Download ────────────────────────────────────────────app.get('/api/download/:id', (req, res) => {const entry = retention.getFile(req.params.id);@@@ -19,7 +19,9 @@ const upload = multer({files: 1,},fileFilter(_req, file, cb) {-- if (ALLOWED_MIMES.has(file.mimetype)) {++ // A part without a type of its own (application/octet-stream: API clients, openvibe-sdk) is let++ // through: the guard checks the bytes of every upload against the tool's accepted types.++ if (ALLOWED_MIMES.has(file.mimetype) || file.mimetype === 'application/octet-stream') {cb(null, true);} else {cb(new Error(`Unsupported file type: ${file.mimetype}. Accepted: ${config.upload.allowedMimes.join(', ')}`));@@@ -128,4 +128,9 @@ const SPECS = [page('peso', { kind: 'text' }), page('spanish', { kind: 'text' }), page('slang', { kind: 'text' }), page('ice', { kind: 'text' }),];++// The run API reuses an engine's answer for 10 minutes (same tool, same input): not for these, whose++// output is random (zalgo, the mocking case, shuffle, the bio and nickname generators).++const FRESH = new Set(['zalgo', 'case', 'sort', 'bio', 'nickname']);++for (const spec of SPECS) if (FRESH.has(spec.id)) spec.cacheTtlMs = 0;++module.exports = { SPECS };
@@ -99,6 +99,75 @@ answers for every tool; each satellite answers the same routes for its own tools
@@ -99,6 +99,75 @@ answers for every tool; each satellite answers the same routes for its own tools
library or isolation (XML minify, HTML to Markdown, the JS minifier/beautifier, the regex tester, the canvas
library or isolation (XML minify, HTML to Markdown, the JS minifier/beautifier, the regex tester, the canvas
graphics makers) say `api: false`, as does the YouTube downloader (page-only by decision).
graphics makers) say `api: false`, as does the YouTube downloader (page-only by decision).
+## Run API (ADR-027)
+
+`POST /api/v1/tools/:id/run` runs any tool whose descriptor says `api: true` (capability `tools.tool.run`;
+`tools.net.probe` for the network probes). The gateway (`https://openvibe.tools`) answers for every tool; img,
+audio and docs answer the same route for their own tools. `openvibe-sdk/tools` (v0.6.0) is its client.
+
+- **Request** (`tools.run-request@1`): JSON `{ input, files, wait_ms, idempotency_key }`, or
+ `multipart/form-data` with each upload as a `file` part and the text parts `input` (JSON), `files` (the
+ file references' JSON), `wait_ms`, `idempotency_key` — exactly what the SDK sends. `?wait_ms=` does the same
+ as the field; the `Idempotency-Key` header wins over `idempotency_key`. A file reference is
+ `{ job_id, index }` (a result file of one of your own jobs, on any satellite: one tool's output feeds the
+ next) or `{ media_id }` (a result of one of your own jobs stored in OpenVibe.Media). Uploads come first,
+ then references, in order; the count must fit the descriptor's `files`.
+- **Answers** (`tools.run@1`): `200 { state: succeeded, tool, result: { data | text | files }, took_ms[, job] }`;
+ `200 { state: failed | cancelled, tool, error, took_ms[, job] }` when the tool itself fails (its
+ problem+json: `tools.job.failed`, `tools.pdf.wrong_password`, `tools.net.target_not_public`, …);
+ `202 { state: queued | running, tool, job, location }` + `Location: /api/v1/jobs/:id` when a job tool has not
+ finished within `wait_ms` (default 0); `200 + Idempotent-Replayed: true` for a replay.
+- **Refusals** (problem+json, before the tool runs): `400 tools.run.invalid`, `401 token.missing |
+ token.invalid | tools.session_required`, `403 capability.denied | tools.origin.refused`,
+ `404 tools.tool.not_found | tools.tool.not_runnable` (`api: false`: the YouTube downloader, the regex tester,
+ the canvas makers…) `| tools.run.file_not_found`, `405`, `409 tools.job.idempotency_conflict`,
+ `413 tools.file.too_large | tools.input.too_large` (over `limits.maxInputBytes`), `415
+ tools.file.unsupported_type` (the bytes are checked), `422 tools.input.invalid` (`errors[]` with JSON
+ pointers, against the descriptor's input schema), `429 tools.quota.exceeded` (`Retry-After`), `503
+ tools.tool.unavailable` (the descriptor's status is `unavailable`: traceroute, or Protect PDF while qpdf is
+ missing), `503 tools.unavailable` (the engine found a program it needs missing: e.g. an HEIC upload without
+ libheif), `503 tools.busy` (`Retry-After`).
+- **Who.** Always enforced (authorization, not a quota): an app or service token needs the descriptor's
+ `auth.capability`; the network probes (port, ping, latency) run only for tokens holding `tools.net.probe`
+ (anonymous `401`, a person `403`: people use the probe pages); `auth.anonymous: false` tools (audio, PDF, SMTP)
+ need a browser session, a sign-in or a token; a Bearer token that does not verify is `401 token.invalid`, not
+ anonymity. Then the guard: the tool's `quotaClass` × the caller's tier, weighted by `cost` (mode-dependent),
+ the per-target throttle for egress tools and upload sniffing (hard).
+- **Where it runs.** Dev and text tools with a server engine: the page's own engine code in the gateway's
+ worker pool (`TOOLS_WORKERS_GATEWAY`, default 2, 256 MB each; the descriptor's `timeoutMs` terminates a
+ runaway input). Net tools and Open Graph: the same `/api/net` and `/api/dev` route the page calls, in
+ process, with the run's input as its query (`myip` is the caller's address). Job tools (img, audio, docs): a
+ job of the satellite that owns the tool (`{ ...input, ...preset, tool: operation }`), the run streamed there
+ by the gateway, uploads included; the job's `tool` names the tool.
+- **Cache and replays.** An inline answer is reused per tool and input where that is safe (spec `cacheTtlMs`:
+ engines 10 minutes unless random or time-dependent — uuid, lorem, timestamp, cron, jwt, zalgo, case, sort, bio,
+ nickname never; DNS 1 minute, IP 10, RDAP/whois 60, a few other lookups 5; probes, headers, redirects, uptime,
+ SMTP and myip never), answered with `X-OV-Cache: hit` and without touching the target. An inline run's
+ `Idempotency-Key` replays its first answer for 15 minutes (per caller); a job tool's key is its job's.
+- **CORS.** First-party pages keep the credentialed rules; any other site's page gets `Access-Control-Allow-Origin: *`
+ without credentials on the run and jobs routes (so its calls carry a token, never our cookies).
+- `GET /api/health` on the gateway shows `run` (counts, cache, engine pool) and `jobs` (the facade).
+
+## Jobs facade (gateway)
+
+`https://openvibe.tools/api/v1/jobs…` fronts every satellite's job routes (below), so one origin serves the
+run API's `Location`, `openvibe-sdk/jobs` without a `baseUrl`, and anything else. A submit is routed by its job
+type's prefix (`img.process` → img …): `?type=`, the JSON body's `type`, or the multipart `type` part (read from
+the first 64 KB of the stream — send it before the files; the uploads then stream through untouched). Every other
+route (`GET`/`DELETE /:id`, `/:id/events`, `/:id/files/:n`, `/:id/retry`, `/:id/references/:ref`) is routed by the
+job id: the satellite that answered its submit or run (remembered, 50,000 ids), else the one that says it holds
+the job when asked on loopback (`GET /api/internal/jobs/:id`, direct loopback callers only, never rate
+limited). Events (SSE) and files stream through. Owners, quotas, the Origin check and idempotency stay the
+satellite's: the request reaches it with the caller's credentials and address.
+
+## Older endpoints (deprecated, still answered)
+
+`/api/process`, `/api/process/direct`, `/api/process/multi` (img, audio, docs), `/api/net/*` and `/api/dev/*`
+keep their response shapes and add `Deprecation: true`, `Sunset: Thu, 31 Dec 2026 23:59:59 GMT` and
+`Link: </api/v1/tools/{id}/run>; rel="successor-version"` (the host's own tool; `/api/net/tools` and
+`/api/dev/tools` point at `/api/v1/tools?family=…`). The webhook bins (`/api/dev/webhook/…`, page-only) have no
+successor and are not marked.
+
## Jobs (Img, Audio, Docs)
## Jobs (Img, Audio, Docs)
Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same
Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same
@@ -227,9 +296,17 @@ One module used by the gateway and every satellite, driven by each tool's descri
@@ -227,9 +296,17 @@ One module used by the gateway and every satellite, driven by each tool's descri
`tools_guard_enforcing`, `tools_guard_sync{kind}`.
`tools_guard_enforcing`, `tools_guard_sync{kind}`.
- **Challenge hook.** `createGuard({ challenge })` takes a person-check provider (`required`, `verify`; Turnstile
- **Challenge hook.** `createGuard({ challenge })` takes a person-check provider (`required`, `verify`; Turnstile
later); the default challenges nobody.
later); the default challenges nobody.
+- **Cross-site requests (CSRF).** A mutation that rides on the browser's cookies (`ov_token`, the jobs session)
+ must come from an OpenVibe.Tools page: `Origin` (else `Referer`) `https://openvibe.tools`,
+ `https://*.openvibe.tools`, one of the tool's own hosts (custom domains included) or the request's own host.
+ Calls with a Bearer token, calls without those cookies and requests with neither header pass. Checked on the
+ run API, the job routes' writes, `/api/process…`, `/api/info`, `/api/probe` and the webhook bins' create and
+ delete; `403 tools.origin.refused`, recorded as reason `origin`.
- **Mode.** `TOOLS_GUARD=report` (default) records and counts what it would refuse and refuses nothing, and the
- **Mode.** `TOOLS_GUARD=report` (default) records and counts what it would refuse and refuses nothing, and the
apps' older express-rate-limit limiters stay in force (now keyed by the resolved caller, after sign-in is read).
apps' older express-rate-limit limiters stay in force (now keyed by the resolved caller, after sign-in is read).
- `TOOLS_GUARD=enforce` refuses and the older limiters step aside. Hard limits apply in both modes: the pixel limit,
+ `TOOLS_GUARD=enforce` refuses and the older limiters step aside. (On `/api/v1/…` an older limiter's refusal is
+ problem+json `tools.quota.exceeded`, `scope: "legacy"`, with `Retry-After`; job runs through the run API pass the
+ same burst and processing limiters as job submits.) Hard limits apply in both modes: the pixel limit,
ffmpeg's whitelists and duration cap, upload sniffing, the port-scan cap and the per-target throttle.
ffmpeg's whitelists and duration cap, upload sniffing, the port-scan cap and the per-target throttle.
Default quotas (units = descriptor `cost` per run; `perMinute` / `burst` / `perDay`, 0 = none):
Default quotas (units = descriptor `cost` per run; `perMinute` / `burst` / `perDay`, 0 = none):
@@ -298,7 +375,14 @@ hosts on every satellite, the registry-driven catalog, and `registry-consistency
@@ -298,7 +375,14 @@ hosts on every satellite, the registry-driven catalog, and `registry-consistency
tool resolves to an operation, endpoint or page that exists, or is marked unavailable. `descriptors.test.js`
tool resolves to an operation, endpoint or page that exists, or is marked unavailable. `descriptors.test.js`
holds every descriptor to the contracts and to the code (a job's operation, validation, files and limits; a sync
holds every descriptor to the contracts and to the code (a job's operation, validation, files and limits; a sync
tool's route), `engines.test.js` runs every browser tool's server engine in plain Node against its output schema,
tool's route), `engines.test.js` runs every browser tool's server engine in plain Node against its output schema,
-and `tools-api.test.js` checks the three routes on the gateway and all seven satellites. The audio tests need
+and `tools-api.test.js` checks the three routes on the gateway and all seven satellites. `run-api.test.js` runs
+the run API on the gateway, img and docs (with a stand-in Network for tokens, `apps/_shared/test/network.js`):
+every answer against `tools.run@1`, the refusal codes, probes, quotas, idempotency, job tools and references
+across satellites, the Origin check and the deprecation headers; `jobs-facade.test.js` routes jobs through the
+gateway (by type, by id after a restart, events, files, retry, cancel); `sdk-client.test.js` drives
+openvibe-sdk v0.6.0's `createToolsClient` against the real gateway (it requires a checkout of OpenVibe.SDK next to
+this repo, or `OV_SDK_DIR`, and says it skipped without one); `pool.test.js` (shared and docs) hold the worker
+pool to its limits. The audio tests need
`ffmpeg` and skip without it. The docs and img tests always check the 503 path with qpdf, poppler and libheif
`ffmpeg` and skip without it. The docs and img tests always check the 503 path with qpdf, poppler and libheif
pointed at nothing, and also run the real encrypt/decrypt, page rendering and HEIC decoding when those programs
pointed at nothing, and also run the real encrypt/decrypt, page rendering and HEIC decoding when those programs
are installed (or `QPDF_PATH` / `HEIF_DEC_PATH` point at them). Install first with `npm run install:all`.
are installed (or `QPDF_PATH` / `HEIF_DEC_PATH` point at them). Install first with `npm run install:all`.
@@ -25,6 +25,10 @@
@@ -25,6 +25,10 @@
// HMAC(address, today's salt), principal or user id, tool, reason — 30 days, no raw
// HMAC(address, today's salt), principal or user id, tool, reason — 30 days, no raw
// address anywhere; metric tools_guard_refused_total{reason,tool}
// address anywhere; metric tools_guard_refused_total{reason,tool}
// challenge a hook for a person-check (Turnstile later; ./challenge.js, a no-op now)
// challenge a hook for a person-check (Turnstile later; ./challenge.js, a no-op now)
+// origin a mutation that rides on the browser's cookies (ov_token, the jobs session) must come
+// from an OpenVibe.Tools page: https://openvibe.tools, https://*.openvibe.tools, the tool's
+// own hosts (custom domains included) or the request's own host (CSRF); calls with a
+// Bearer token and calls without those cookies do not need it. 403 tools.origin.refused
//
//
// Mode: TOOLS_GUARD=report (default) logs and counts what it would refuse and refuses nothing, except
// Mode: TOOLS_GUARD=report (default) logs and counts what it would refuse and refuses nothing, except
// the hard limits that apply in both modes: the image pixel limit, ffmpeg's protocol and format
// the hard limits that apply in both modes: the image pixel limit, ffmpeg's protocol and format
@@ -70,9 +74,9 @@ function uploadsOf(req) {
@@ -70,9 +74,9 @@ function uploadsOf(req) {
return req.file ? [req.file] : [];
return req.file ? [req.file] : [];
}
}
-/** Delete the files multer wrote to disk for a request that is refused. */
+/** Delete the files multer wrote to disk for a request that is refused (before the answer goes out). */
function discardUploads(req) {
function discardUploads(req) {
- for (const f of uploadsOf(req)) if (f.path) fs.unlink(f.path, () => {});
+ for (const f of uploadsOf(req)) if (f.path) { try { fs.unlinkSync(f.path); } catch { /* already gone */ } }
}
}
/**
/**
@@ -223,7 +227,9 @@ function createGuard(o = {}) {
@@ -223,7 +227,9 @@ function createGuard(o = {}) {
}
}
// Liveness and readiness probes (the gateway polls every satellite's) are never counted.
// Liveness and readiness probes (the gateway polls every satellite's) are never counted.
const PROBES = new Set(['/api/health', '/api/ready']);
const PROBES = new Set(['/api/health', '/api/ready']);
- const isProbe = (req) => req.method === 'GET' && PROBES.has(`${req.baseUrl || ''}${req.path}`);
+ // …nor the gateway's loopback question "which satellite holds this job?" (GET /api/internal/jobs/:id).
+ const directLoopback = (req) => ip.isLoopback(req.socket && req.socket.remoteAddress) && !req.headers['x-forwarded-for'] && !req.headers['x-real-ip'];
+ const isProbe = (req) => req.method === 'GET' && (PROBES.has(`${req.baseUrl || ''}${req.path}`) || (`${req.baseUrl || ''}${req.path}`.startsWith('/api/internal/jobs/') && directLoopback(req)));
/**
/**
* Middleware: charge the tool's own class and cost (its descriptor). toolOf(req) → a tool id. Records
* Middleware: charge the tool's own class and cost (its descriptor). toolOf(req) → a tool id. Records
@@ -349,6 +355,47 @@ function createGuard(o = {}) {
@@ -349,6 +355,47 @@ function createGuard(o = {}) {
return !refuse(req, res, { status: 503, code: 'tools.busy', detail: busy.detail, reason: `busy.${busy.reason}`, tool: toolId, retryAfter: busy.retryAfter || 30 });
return !refuse(req, res, { status: 503, code: 'tools.busy', detail: busy.detail, reason: `busy.${busy.reason}`, tool: toolId, retryAfter: busy.retryAfter || 30 });
}
}
+ // ── Cross-site requests (CSRF) ───────────────────────────
+ const SAFE = new Set(['GET', 'HEAD', 'OPTIONS']);
+ const FIRST_PARTY = /^([a-z0-9-]+\.)*openvibe\.tools$/;
+ const production = String(env.NODE_ENV || '') === 'production';
+ /** May a page at `origin` use this request's cookies? First-party, the tool's hosts, the same host. */
+ function allowedOrigin(origin, req, hosts) {
+ let u;
+ try { u = new URL(origin); } catch { return false; }
+ const hostname = u.hostname.toLowerCase();
+ if (u.protocol === 'https:' && FIRST_PARTY.test(hostname)) return true;
+ if (u.protocol === 'https:' && (hosts || []).some(h => String(h).toLowerCase() === hostname)) return true;
+ const own = String(req.headers['x-forwarded-host'] || req.headers.host || '').toLowerCase();
+ if (own && u.host.toLowerCase() === own && (u.protocol === 'https:' || !production)) return true; // same origin
+ if (!production && (hostname === 'localhost' || hostname === '127.0.0.1')) return true;
+ return false;
+ }
+ /**
+ * A mutation carried by the browser's cookies must come from one of our pages (Origin, else
+ * Referer; a request with neither is not a browser's cross-site request). Bearer calls and calls
+ * without our cookies pass. Refused in enforce mode (403 tools.origin.refused), recorded in report
+ * mode. → ok
+ */
+ function originOk(req, res, { hosts = [], tool: toolId = null } = {}) {
+ if (SAFE.has(req.method)) return true;
+ if (/^Bearer\s+\S/i.test(String(req.headers.authorization || ''))) return true;
+ const cookies = req.cookies || {};
+ if (!cookies.ov_token && !cookies.token && !cookies[callers.cookieName]) return true;
+ let origin = req.headers.origin;
+ if (origin == null || origin === '') {
+ const ref = req.headers.referer;
+ if (!ref) return true;
+ try { origin = new URL(String(ref)).origin; } catch { origin = 'null'; }
+ }
+ if (origin !== 'null' && allowedOrigin(String(origin), req, hosts)) return true;
+ return !refuse(req, res, { status: 403, code: 'tools.origin.refused', reason: 'origin', tool: toolId, detail: 'This request came from another site with your OpenVibe.Tools cookies. Send it from an OpenVibe.Tools page, or with an API token.', extra: { origin: String(origin).slice(0, 200) } });
+ }
+ /** Middleware form: hostsOf(req) → the tool's own hosts. */
+ function originCheck(hostsOf, toolOf) {
+ return (req, res, next) => (originOk(req, res, { hosts: hostsOf ? hostsOf(req) : [], tool: toolOf ? toolOf(req) : null }) ? next() : undefined);
+ }
+
// ── Egress ───────────────────────────────────────────────
// ── Egress ───────────────────────────────────────────────
/** Per-target throttle (hard). → ok */
/** Per-target throttle (hard). → ok */
function target(req, res, { tool: toolId, target: t, perMinute }) {
function target(req, res, { tool: toolId, target: t, perMinute }) {
@@ -391,6 +438,18 @@ function createGuard(o = {}) {
@@ -391,6 +438,18 @@ function createGuard(o = {}) {
keyGenerator: (req) => { const c = caller(req); return signed(c) || c.tier === 'sandbox' ? c.key : c.ipKey; },
keyGenerator: (req) => { const c = caller(req); return signed(c) || c.tier === 'sandbox' ? c.key : c.ipKey; },
skip: (req) => enforcing || isProbe(req), // liveness and readiness probes are never limited
skip: (req) => enforcing || isProbe(req), // liveness and readiness probes are never limited
message: { error: message },
message: { error: message },
+ // The platform API (/api/v1/…) answers like the guard does: problem+json tools.quota.exceeded
+ // with Retry-After; the older routes keep their { error } shape.
+ handler: (req, res, _next, options) => {
+ const reset = req.rateLimit && req.rateLimit.resetTime ? new Date(req.rateLimit.resetTime).getTime() : 0;
+ const retry = Math.max(1, Math.ceil(((reset || (now() + windowMs)) - now()) / 1000));
+ res.setHeader('Retry-After', String(retry));
+ if (/^\/api\/v1\//.test(String(req.originalUrl || req.url || ''))) {
+ const c = caller(req);
+ return sendProblem(req, res, 429, 'tools.quota.exceeded', message, { scope: 'legacy', tier: c.tier, retry_after: retry }, o.contracts);
+ }
+ return res.status(options.statusCode).json(options.message);
+ },
});
});
}
}
/** The three limiters an upload app had (LEGACY numbers in ./limits.js). */
/** The three limiters an upload app had (LEGACY numbers in ./limits.js). */
@@ -440,6 +499,7 @@ function createGuard(o = {}) {
@@ -440,6 +499,7 @@ function createGuard(o = {}) {
heavy, sync, jobsBusy,
heavy, sync, jobsBusy,
target, portScan, targets, ports, normalizeTarget,
target, portScan, targets, ports, normalizeTarget,
legacyLimiter, legacyLimiters, toolRefused,
legacyLimiter, legacyLimiters, toolRefused,
+ originOk, originCheck, allowedOrigin, tokens,
attachMetrics, prune,
attachMetrics, prune,
close() { if (pruneTimer) clearInterval(pruneTimer); if (keys.stop) keys.stop(); store.close(); },
close() { if (pruneTimer) clearInterval(pruneTimer); if (keys.stop) keys.stop(); store.close(); },
};
};
@@ -18,6 +18,8 @@
@@ -18,6 +18,8 @@
// only what came after it; 204 once there is nothing left to say
// only what came after it; 204 once there is nothing left to say
// GET /api/v1/jobs/:id/files/:n a result file (attachment; ?inline=1 for previews)
// GET /api/v1/jobs/:id/files/:n a result file (attachment; ?inline=1 for previews)
// GET /js/ov-jobs.js the browser helper (submit, watch, reattach)
// GET /js/ov-jobs.js the browser helper (submit, watch, reattach)
+// GET /api/internal/jobs/:id direct loopback callers only (the gateway's jobs facade asking
+// which satellite holds a job): 200 { id, service } | 404
//
//
// Owner scoping: a job is visible only to whoever created it —
// Owner scoping: a job is visible only to whoever created it —
// a Network user (user:usr_…, from the ov_token subject_id), a service/app principal presenting a
// a Network user (user:usr_…, from the ov_token subject_id), a service/app principal presenting a
@@ -31,6 +33,7 @@ const path = require('path');
@@ -31,6 +33,7 @@ const path = require('path');
const { Readable } = require('stream');
const { Readable } = require('stream');
const { createCallerResolver, jobOwnerResolver, JOB_CAPS: CAPS } = require('../guard/caller');
const { createCallerResolver, jobOwnerResolver, JOB_CAPS: CAPS } = require('../guard/caller');
+const { isLoopback } = require('../guard/ip');
const ID_RE = /^job_[0-9A-HJKMNP-TV-Z]{26}$/;
const ID_RE = /^job_[0-9A-HJKMNP-TV-Z]{26}$/;
const TERMINAL = new Set(['succeeded', 'failed', 'cancelled']);
const TERMINAL = new Set(['succeeded', 'failed', 'cancelled']);
@@ -77,6 +80,8 @@ function createOwnerResolver(o) {
@@ -77,6 +80,8 @@ function createOwnerResolver(o) {
* without it a busy store answers 503 tools.busy
* without it a busy store answers 503 tools.busy
* @param {(req, res, full) => boolean} [o.onAddressFull] a browser session's address already has its
* @param {(req, res, full) => boolean} [o.onAddressFull] a browser session's address already has its
* unfinished jobs (system.addressFull): true = go on (report mode); without it 429
* unfinished jobs (system.addressFull): true = go on (report mode); without it 429
+ * @param {Function} [o.originCheck] middleware for the writes (submit, cancel, retry, references): the
+ * guard's Origin check on cookie-authenticated calls (CSRF)
*/
*/
function mountJobRoutes(app, o) {
function mountJobRoutes(app, o) {
const { system, contracts } = o;
const { system, contracts } = o;
@@ -99,6 +104,19 @@ function mountJobRoutes(app, o) {
@@ -99,6 +104,19 @@ function mountJobRoutes(app, o) {
return row;
return row;
}
}
+ const origin = o.originCheck || ((_req, _res, next) => next());
+
+ // Which satellite holds a job? Only for a caller on this host with no proxy in between (the
+ // gateway's facade); through nginx or the gateway's own proxy it is not found. Says nothing but
+ // whether the id exists here.
+ app.get('/api/internal/jobs/:id', (req, res) => {
+ res.set('Cache-Control', 'no-store');
+ const direct = isLoopback(req.socket && req.socket.remoteAddress) && !req.headers['x-forwarded-for'] && !req.headers['x-real-ip'];
+ const id = String(req.params.id || '');
+ if (!direct || !ID_RE.test(id) || !system.get(id)) return send(res, 404, 'tools.job.not_found', 'No such job');
+ return res.json({ id, service: system.service || null });
+ });
+
app.get('/js/ov-jobs.js', (_req, res) => {
app.get('/js/ov-jobs.js', (_req, res) => {
res.set({ 'Content-Type': 'application/javascript; charset=utf-8', 'Cache-Control': 'no-cache' });
res.set({ 'Content-Type': 'application/javascript; charset=utf-8', 'Cache-Control': 'no-cache' });
res.sendFile(CLIENT_JS);
res.sendFile(CLIENT_JS);
@@ -126,7 +144,7 @@ function mountJobRoutes(app, o) {
@@ -126,7 +144,7 @@ function mountJobRoutes(app, o) {
return send(res, 503, 'tools.busy', busy.detail);
return send(res, 503, 'tools.busy', busy.detail);
};
};
- app.post('/api/v1/jobs', ...(o.limiters || []), notBusy, multipartOnly, async (req, res) => {
+ app.post('/api/v1/jobs', origin, ...(o.limiters || []), notBusy, multipartOnly, async (req, res) => {
noStore(res);
noStore(res);
const w = who(req, res, { create: true, action: 'create' });
const w = who(req, res, { create: true, action: 'create' });
if (!w) return;
if (!w) return;
@@ -173,7 +191,7 @@ function mountJobRoutes(app, o) {
@@ -173,7 +191,7 @@ function mountJobRoutes(app, o) {
if (row) res.json(system.view(row));
if (row) res.json(system.view(row));
});
});
- app.delete('/api/v1/jobs/:id', (req, res) => {
+ app.delete('/api/v1/jobs/:id', origin, (req, res) => {
noStore(res);
noStore(res);
const row = load(req, res, 'cancel');
const row = load(req, res, 'cancel');
if (!row) return;
if (!row) return;
@@ -183,7 +201,7 @@ function mountJobRoutes(app, o) {
@@ -183,7 +201,7 @@ function mountJobRoutes(app, o) {
});
});
// Retry and references are writes: a principal needs tools.job.create, like a submit.
// Retry and references are writes: a principal needs tools.job.create, like a submit.
- app.post('/api/v1/jobs/:id/retry', ...(o.limiters || []), notBusy, (req, res) => {
+ app.post('/api/v1/jobs/:id/retry', origin, ...(o.limiters || []), notBusy, (req, res) => {
noStore(res);
noStore(res);
const row = load(req, res, 'create');
const row = load(req, res, 'create');
if (!row) return;
if (!row) return;
@@ -214,8 +232,8 @@ function mountJobRoutes(app, o) {
@@ -214,8 +232,8 @@ function mountJobRoutes(app, o) {
}
}
};
};
}
}
- app.put('/api/v1/jobs/:id/references/:ref', referenceRoute((id, ref) => system.reference(id, ref)));
- app.delete('/api/v1/jobs/:id/references/:ref', referenceRoute((id, ref) => system.unreference(id, ref)));
+ app.put('/api/v1/jobs/:id/references/:ref', origin, referenceRoute((id, ref) => system.reference(id, ref)));
+ app.delete('/api/v1/jobs/:id/references/:ref', origin, referenceRoute((id, ref) => system.unreference(id, ref)));
app.get('/api/v1/jobs/:id/events', (req, res) => {
app.get('/api/v1/jobs/:id/events', (req, res) => {
const row = load(req, res, 'read');
const row = load(req, res, 'read');
@@ -110,6 +110,7 @@ function setupJobs(o) {
@@ -110,6 +110,7 @@ function setupJobs(o) {
mountJobRoutes(o.app, {
mountJobRoutes(o.app, {
system, contracts: o.contracts, resolveOwner, receive: o.receive, limiters: o.limiters, defaults: o.defaults,
system, contracts: o.contracts, resolveOwner, receive: o.receive, limiters: o.limiters, defaults: o.defaults,
...(g && {
...(g && {
+ originCheck: g.originCheck(),
admit: (req, res, job) => g.admitJob(req, res, { ...job, toolId: o.jobTool ? o.jobTool(req, job.type, job.input) : null }),
admit: (req, res, job) => g.admitJob(req, res, { ...job, toolId: o.jobTool ? o.jobTool(req, job.type, job.input) : null }),
onBusy: (req, res, busy) => g.jobsBusy(req, res, busy, o.jobTool ? o.jobTool(req, null, {}) : null),
onBusy: (req, res, busy) => g.jobsBusy(req, res, busy, o.jobTool ? o.jobTool(req, null, {}) : null),
onAddressFull: (req, res, full) => !g.refuse(req, res, { status: 429, code: 'tools.job.too_many_active', reason: 'jobs.address', tool: o.jobTool ? o.jobTool(req, null, {}) : null, retryAfter: 30, detail: `At most ${full.limit} unfinished jobs from one address at a time; wait for one to finish.`, extra: { scope: 'address' } }),
onAddressFull: (req, res, full) => !g.refuse(req, res, { status: 429, code: 'tools.job.too_many_active', reason: 'jobs.address', tool: o.jobTool ? o.jobTool(req, null, {}) : null, retryAfter: 30, detail: `At most ${full.limit} unfinished jobs from one address at a time; wait for one to finish.`, extra: { scope: 'address' } }),
@@ -96,9 +96,11 @@ function createPool(o) {
@@ -96,9 +96,11 @@ function createPool(o) {
counts.started++;
counts.started++;
w.worker.on('message', (msg) => settle(w, msg));
w.worker.on('message', (msg) => settle(w, msg));
w.worker.on('error', (err) => {
w.worker.on('error', (err) => {
- // An uncaught throw or the heap limit: the worker is gone either way.
+ // An uncaught throw or the heap limit: the worker is gone either way. It leaves the pool now
+ // (its 'exit' may come later, or not at all), so no new run is ever handed to it.
const t = w.task;
const t = w.task;
w.task = null;
w.task = null;
+ retire(w);
if (!t) return;
if (!t) return;
if (err && err.code === 'ERR_WORKER_OUT_OF_MEMORY') {
if (err && err.code === 'ERR_WORKER_OUT_OF_MEMORY') {
counts.out_of_memory++;
counts.out_of_memory++;
@@ -108,21 +110,35 @@ function createPool(o) {
@@ -108,21 +110,35 @@ function createPool(o) {
}
}
});
});
w.worker.on('exit', () => {
w.worker.on('exit', () => {
- workers.delete(w);
- clearTimeout(w.idleTimer);
- if (w.task) { const t = w.task; w.task = null; finish(t, fail('The tool stopped unexpectedly while working on this file.', { status: 500, code: 'tools.job.failed' })); }
- dispatch();
+ const t = w.task;
+ w.task = null;
+ retire(w);
+ if (t) finish(t, fail('The tool stopped unexpectedly while working on this file.', { status: 500, code: 'tools.job.failed' }));
+ });
+ w.worker.on('messageerror', () => {
+ const t = w.task;
+ w.task = null;
+ retire(w);
+ if (t) finish(t, fail('The tool\'s answer could not be read.', { status: 500, code: 'tools.job.failed' }));
});
});
workers.add(w);
workers.add(w);
return w;
return w;
}
}
- function kill(w) {
- w.task = null;
+ /** Take a worker out of the pool for good and make sure its thread ends. */
+ function retire(w) {
+ if (w.dead) return;
+ w.dead = true;
workers.delete(w);
workers.delete(w);
clearTimeout(w.idleTimer);
clearTimeout(w.idleTimer);
- counts.killed++;
w.worker.terminate().catch(() => {});
w.worker.terminate().catch(() => {});
+ setImmediate(dispatch);
+ }
+
+ function kill(w) {
+ w.task = null;
+ counts.killed++;
+ retire(w);
}
}
function finish(t, err, result) {
function finish(t, err, result) {
@@ -151,7 +167,7 @@ function createPool(o) {
@@ -151,7 +167,7 @@ function createPool(o) {
clearTimeout(w.idleTimer);
clearTimeout(w.idleTimer);
w.worker.unref();
w.worker.unref();
if (idleMs > 0) {
if (idleMs > 0) {
- w.idleTimer = setTimeout(() => { if (!w.task && !queue.length) { workers.delete(w); w.worker.terminate().catch(() => {}); } }, idleMs);
+ w.idleTimer = setTimeout(() => { if (!w.task && !queue.length) retire(w); }, idleMs);
if (w.idleTimer.unref) w.idleTimer.unref();
if (w.idleTimer.unref) w.idleTimer.unref();
}
}
}
}
@@ -186,7 +202,7 @@ function createPool(o) {
@@ -186,7 +202,7 @@ function createPool(o) {
function dispatch() {
function dispatch() {
if (closed) return;
if (closed) return;
while (queue.length) {
while (queue.length) {
- let w = [...workers].find(x => !x.task);
+ let w = [...workers].find(x => !x.task && !x.dead);
if (!w) {
if (!w) {
if (workers.size >= size) return;
if (workers.size >= size) return;
w = spawn();
w = spawn();
@@ -239,7 +255,7 @@ function createPool(o) {
@@ -239,7 +255,7 @@ function createPool(o) {
async function close() {
async function close() {
closed = true;
closed = true;
for (const t of queue.splice(0)) finish(t, fail('The worker pool is shutting down', { status: 503, code: 'tools.busy', retryAfter: 5 }));
for (const t of queue.splice(0)) finish(t, fail('The worker pool is shutting down', { status: 503, code: 'tools.busy', retryAfter: 5 }));
- await Promise.all([...workers].map(w => { const t = w.task; w.task = null; if (t) finish(t, fail('The worker pool is shutting down', { status: 503, code: 'tools.busy', retryAfter: 5 })); clearTimeout(w.idleTimer); return w.worker.terminate().catch(() => {}); }));
+ await Promise.all([...workers].map(w => { const t = w.task; w.task = null; w.dead = true; if (t) finish(t, fail('The worker pool is shutting down', { status: 503, code: 'tools.busy', retryAfter: 5 })); clearTimeout(w.idleTimer); return w.worker.terminate().catch(() => {}); }));
workers.clear();
workers.clear();
}
}
@@ -67,6 +67,7 @@ const ADDED_COLUMNS = [
@@ -67,6 +67,7 @@ const ADDED_COLUMNS = [
['retried_by', 'TEXT'], // set on a failed job once it has been retried: the retry's id
['retried_by', 'TEXT'], // set on a failed job once it has been retried: the retry's id
['ip_key', 'TEXT'], // a browser session's job: the guard's hashed address key (never the address), so
['ip_key', 'TEXT'], // a browser session's job: the guard's hashed address key (never the address), so
// unfinished jobs are also bounded per address (a new cookie is no new allowance)
// unfinished jobs are also bounded per address (a new cookie is no new allowance)
+ ['tool', 'TEXT'], // the tool (tools.tool@1 id) whose run created the job (POST /api/v1/tools/:id/run)
];
];
const parse = (s, fallback) => { if (s == null) return fallback; try { return JSON.parse(s); } catch { return fallback; } };
const parse = (s, fallback) => { if (s == null) return fallback; try { return JSON.parse(s); } catch { return fallback; } };
@@ -80,11 +81,13 @@ function createStore(db) {
@@ -80,11 +81,13 @@ function createStore(db) {
db.exec("CREATE INDEX IF NOT EXISTS tool_jobs_ip ON tool_jobs(ip_key, state) WHERE ip_key IS NOT NULL");
db.exec("CREATE INDEX IF NOT EXISTS tool_jobs_ip ON tool_jobs(ip_key, state) WHERE ip_key IS NOT NULL");
const q = {
const q = {
- insert: db.prepare(`INSERT INTO tool_jobs (id, type, type_version, owner, state, input_json, files_json, idempotency_key, request_hash, max_attempts, ttl_ms, created_at, updated_at, env, retry_of, ip_key)
- VALUES (@id, @type, @type_version, @owner, 'queued', @input_json, @files_json, @idempotency_key, @request_hash, @max_attempts, @ttl_ms, @now, @now, @env, @retry_of, @ip_key)`),
+ insert: db.prepare(`INSERT INTO tool_jobs (id, type, type_version, owner, state, input_json, files_json, idempotency_key, request_hash, max_attempts, ttl_ms, created_at, updated_at, env, retry_of, ip_key, tool)
+ VALUES (@id, @type, @type_version, @owner, 'queued', @input_json, @files_json, @idempotency_key, @request_hash, @max_attempts, @ttl_ms, @now, @now, @env, @retry_of, @ip_key, @tool)`),
markRetried: db.prepare("UPDATE tool_jobs SET retried_by = @next, updated_at = @now WHERE id = @id AND state = 'failed' AND retried_by IS NULL"),
markRetried: db.prepare("UPDATE tool_jobs SET retried_by = @next, updated_at = @now WHERE id = @id AND state = 'failed' AND retried_by IS NULL"),
get: db.prepare('SELECT * FROM tool_jobs WHERE id = ?'),
get: db.prepare('SELECT * FROM tool_jobs WHERE id = ?'),
byIdem: db.prepare('SELECT * FROM tool_jobs WHERE owner = ? AND idempotency_key = ?'),
byIdem: db.prepare('SELECT * FROM tool_jobs WHERE owner = ? AND idempotency_key = ?'),
+ // A result file stored in Media, by its media_id (a run's { media_id } reference); the owner's only.
+ byResultMedia: db.prepare("SELECT * FROM tool_jobs WHERE owner = ? AND state = 'succeeded' AND result_json LIKE ? ORDER BY id DESC LIMIT 5"),
nextQueued: db.prepare("SELECT * FROM tool_jobs WHERE state = 'queued' ORDER BY id LIMIT ?"),
nextQueued: db.prepare("SELECT * FROM tool_jobs WHERE state = 'queued' ORDER BY id LIMIT ?"),
claim: db.prepare("UPDATE tool_jobs SET state = 'running', attempts = attempts + 1, started_at = @now, updated_at = @now, progress = NULL, progress_message = NULL WHERE id = @id AND state = 'queued'"),
claim: db.prepare("UPDATE tool_jobs SET state = 'running', attempts = attempts + 1, started_at = @now, updated_at = @now, progress = NULL, progress_message = NULL WHERE id = @id AND state = 'queued'"),
running: db.prepare("SELECT * FROM tool_jobs WHERE state = 'running'"),
running: db.prepare("SELECT * FROM tool_jobs WHERE state = 'running'"),
@@ -124,7 +127,8 @@ function createStore(db) {
@@ -124,7 +127,8 @@ function createStore(db) {
return {
return {
db,
db,
TERMINAL, STATES,
TERMINAL, STATES,
- insert(row) { q.insert.run({ retry_of: null, ip_key: null, ...row }); },
+ insert(row) { q.insert.run({ retry_of: null, ip_key: null, tool: null, ...row }); },
+ byResultMedia: (owner, mediaId) => q.byResultMedia.all(owner, `%"media_id":"${String(mediaId).replace(/[%"\\]/g, '')}"%`),
markRetried: (id, next, now = Date.now()) => q.markRetried.run({ id, next, now }).changes === 1,
markRetried: (id, next, now = Date.now()) => q.markRetried.run({ id, next, now }).changes === 1,
get: (id) => q.get.get(id) || null,
get: (id) => q.get.get(id) || null,
byIdempotencyKey: (owner, key) => q.byIdem.get(owner, key) || null,
byIdempotencyKey: (owner, key) => q.byIdem.get(owner, key) || null,
@@ -151,6 +151,7 @@ function createJobSystem(o) {
@@ -151,6 +151,7 @@ function createJobSystem(o) {
id: row.id,
id: row.id,
object: 'tools.job',
object: 'tools.job',
service: o.service,
service: o.service,
+ ...(row.tool && { tool: row.tool }),
type: row.type,
type: row.type,
type_version: row.type_version,
type_version: row.type_version,
state: row.state,
state: row.state,
@@ -202,7 +203,7 @@ function createJobSystem(o) {
@@ -202,7 +203,7 @@ function createJobSystem(o) {
// Developer-app sandbox jobs: fewer at a time, kept briefly, results never leave this server.
// Developer-app sandbox jobs: fewer at a time, kept briefly, results never leave this server.
const SANDBOX_MAX_ACTIVE = Math.max(1, o.sandboxMaxActivePerOwner || 2);
const SANDBOX_MAX_ACTIVE = Math.max(1, o.sandboxMaxActivePerOwner || 2);
const SANDBOX_TTL_MS = 30 * 60 * 1000;
const SANDBOX_TTL_MS = 30 * 60 * 1000;
- async function submit({ owner, type, input = {}, files = [], idempotencyKey = null, ttlMs = HOUR, env = 'production', ipKey = null }) {
+ async function submit({ owner, type, input = {}, files = [], idempotencyKey = null, ttlMs = HOUR, env = 'production', ipKey = null, tool = null }) {
if (env !== 'sandbox') env = 'production';
if (env !== 'sandbox') env = 'production';
if (env === 'sandbox') ttlMs = Math.min(ttlMs, SANDBOX_TTL_MS);
if (env === 'sandbox') ttlMs = Math.min(ttlMs, SANDBOX_TTL_MS);
if (stopped) throw new JobError(503, 'tools.job.unavailable', 'The job system is shutting down');
if (stopped) throw new JobError(503, 'tools.job.unavailable', 'The job system is shutting down');
@@ -252,6 +253,7 @@ function createJobSystem(o) {
@@ -252,6 +253,7 @@ function createJobSystem(o) {
idempotency_key: idempotencyKey == null ? null : String(idempotencyKey), request_hash: requestHash,
idempotency_key: idempotencyKey == null ? null : String(idempotencyKey), request_hash: requestHash,
max_attempts: def.maxAttempts, ttl_ms: ttlMs, now: Date.now(), env,
max_attempts: def.maxAttempts, ttl_ms: ttlMs, now: Date.now(), env,
ip_key: String(owner).startsWith('session:') && ipKey ? String(ipKey) : null,
ip_key: String(owner).startsWith('session:') && ipKey ? String(ipKey) : null,
+ tool: tool && /^[a-z][a-z0-9-]{0,39}$/.test(String(tool)) ? String(tool) : null,
});
});
announce.created(store.get(id));
announce.created(store.get(id));
})();
})();
@@ -474,7 +476,7 @@ function createJobSystem(o) {
@@ -474,7 +476,7 @@ function createJobSystem(o) {
store.insert({
store.insert({
id: next, type: row.type, type_version: def.version, owner: row.owner, input_json: row.input_json, files_json: JSON.stringify(stored),
id: next, type: row.type, type_version: def.version, owner: row.owner, input_json: row.input_json, files_json: JSON.stringify(stored),
idempotency_key: null, request_hash: row.request_hash, max_attempts: def.maxAttempts, ttl_ms: row.ttl_ms, now: Date.now(),
idempotency_key: null, request_hash: row.request_hash, max_attempts: def.maxAttempts, ttl_ms: row.ttl_ms, now: Date.now(),
- env: row.env, retry_of: row.id, ip_key: row.ip_key || null,
+ env: row.env, retry_of: row.id, ip_key: row.ip_key || null, tool: row.tool || null,
});
});
announce.created(store.get(next));
announce.created(store.get(next));
})();
})();
@@ -664,6 +666,16 @@ function createJobSystem(o) {
@@ -664,6 +666,16 @@ function createJobSystem(o) {
/** Run fn when the system stops; → a function that removes it. */
/** Run fn when the system stops; → a function that removes it. */
function onStop(fn) { stopHooks.add(fn); return () => stopHooks.delete(fn); }
function onStop(fn) { stopHooks.add(fn); return () => stopHooks.delete(fn); }
+ /** The owner's succeeded job with a result file stored as Media object `mediaId` → { row, index } or null. */
+ function findResultMedia(owner, mediaId) {
+ for (const row of store.byResultMedia(owner, mediaId)) {
+ const files = (store.parse(row.result_json, null) || {}).files || [];
+ const index = files.findIndex(f => f.media && f.media.media_id === mediaId);
+ if (index >= 0) return { row, index };
+ }
+ return null;
+ }
+
/** Where a result file's bytes are: { local: path } or { media: MediaRef }. */
/** Where a result file's bytes are: { local: path } or { media: MediaRef }. */
function resultFile(row, n) {
function resultFile(row, n) {
const result = store.parse(row.result_json, null);
const result = store.parse(row.result_json, null);
@@ -673,9 +685,10 @@ function createJobSystem(o) {
@@ -673,9 +685,10 @@ function createJobSystem(o) {
}
}
const api = {
const api = {
- define, submit, cancel, retry, reference, unreference, subscribe, start, stop, onStop, prune, recover, view, resultFile,
+ define, submit, cancel, retry, reference, unreference, subscribe, start, stop, onStop, prune, recover, view, resultFile, findResultMedia,
busy, refreshDisk, addressFull, bounds: { maxQueued, diskBudgetBytes: diskBudget, maxActivePerAddress },
busy, refreshDisk, addressFull, bounds: { maxQueued, diskBudgetBytes: diskBudget, maxActivePerAddress },
get: (id) => store.get(id),
get: (id) => store.get(id),
+ service: o.service,
eventsAfter: (id, seq) => store.eventsAfter(id, seq),
eventsAfter: (id, seq) => store.eventsAfter(id, seq),
lastSeq: (id) => store.lastSeq(id),
lastSeq: (id) => store.lastSeq(id),
types: () => [...types.keys()],
types: () => [...types.keys()],
@@ -352,6 +352,54 @@ async function serve(app) {
@@ -352,6 +352,54 @@ async function serve(app) {
g.close();
g.close();
}
}
+ // ── Cross-site requests (CSRF): cookie-carried mutations must come from our pages ──
+ {
+ const res = () => ({ statusCode: 200, headers: {}, body: null, headersSent: false, setHeader(k, v) { this.headers[k.toLowerCase()] = v; }, end(b) { this.body = JSON.parse(b); } });
+ const post = (headers, cookies = { ov_tools_jobs: 'a'.repeat(32) }, method = 'POST') => ({ method, ip: '198.51.100.60', headers: { host: 'png.openvibe.tools', ...headers }, cookies });
+ for (const mode of ['report', 'enforce']) {
+ const g = makeGuard({ mode });
+ const ok = (req, hosts) => g.originOk(req, res(), { hosts, tool: 'png' });
+ // Allowed: our pages, the tool's own hosts, the same host, no cookies, a Bearer token, safe methods, no Origin at all.
+ for (const o of ['https://openvibe.tools', 'https://png.openvibe.tools', 'https://img.openvibe.tools', 'https://a.b.openvibe.tools']) assert.ok(ok(post({ origin: o })), `${mode}: ${o}`);
+ assert.ok(ok(post({ origin: 'https://bestpngconverter.example' }), ['bestpngconverter.example']), `${mode}: the tool's own custom domain`);
+ assert.ok(ok(post({ origin: 'https://mydomain.example', host: 'mydomain.example' })), `${mode}: the same host`);
+ assert.ok(ok(post({ origin: 'https://evil.example' }, {})), `${mode}: no cookies, nothing to forge`);
+ assert.ok(ok(post({ origin: 'https://evil.example', authorization: 'Bearer x.y.z' })), `${mode}: a token, not a cookie`);
+ assert.ok(ok(post({ origin: 'https://evil.example' }, undefined, 'GET')), `${mode}: GET is not a mutation`);
+ assert.ok(ok(post({})), `${mode}: neither Origin nor Referer: not a browser's cross-site request`);
+ // Refused (enforce) or only recorded (report).
+ const cases = [
+ post({ origin: 'https://evil.example' }),
+ post({ origin: 'http://img.openvibe.tools' }), // not https
+ post({ origin: 'https://openvibe.tools.evil.example' }),
+ post({ origin: 'null' }), // a sandboxed frame, a data: page
+ post({ referer: 'https://evil.example/form' }), // no Origin, a foreign Referer
+ post({ origin: 'https://evil.example' }, { ov_token: 'x.y.z' }), // the sign-in cookie
+ ];
+ for (const req of cases) {
+ const out = res();
+ const allowed = g.originOk(req, out, { tool: 'png' });
+ if (mode === 'enforce') {
+ assert.strictEqual(allowed, false, JSON.stringify(req.headers));
+ assert.deepStrictEqual([out.statusCode, out.body.code], [403, 'tools.origin.refused']);
+ } else {
+ assert.strictEqual(allowed, true, `report mode records ${JSON.stringify(req.headers)}`);
+ assert.strictEqual(out.body, null);
+ }
+ }
+ const logged = g.store.abuseRows().filter(x => x.reason === 'origin').reduce((n, x) => n + x.count, 0);
+ assert.strictEqual(logged, cases.length, `${mode}: every refusal is in the abuse log`);
+ g.close();
+ }
+ // localhost pages outside production (dev servers), never in production.
+ const dev = makeGuard({ mode: 'enforce', env: { NODE_ENV: 'development' } });
+ assert.ok(dev.originOk(post({ origin: 'http://localhost:5173' }), res()));
+ dev.close();
+ const prod = makeGuard({ mode: 'enforce', env: { NODE_ENV: 'production' } });
+ assert.strictEqual(prod.originOk(post({ origin: 'http://localhost:5173' }), res()), false);
+ prod.close();
+ }
+
// ── The sync semaphore holds heavy calls to the cap ──
// ── The sync semaphore holds heavy calls to the cap ──
{
{
const sem = createSemaphore({ max: 2, queue: 3, waitMs: 200 });
const sem = createSemaphore({ max: 2, queue: 3, waitMs: 200 });
@@ -466,5 +514,5 @@ async function serve(app) {
@@ -466,5 +514,5 @@ async function serve(app) {
system.stop(); db.close(); fs.rmSync(dataDir, { recursive: true, force: true });
system.stop(); db.close(); fs.rmSync(dataDir, { recursive: true, force: true });
}
}
- console.log('guard: callers (/64, one loopback hop, services by token, aud-checked users, sessions, sandbox), bucket math + cost, session sharing, report vs enforce, headers + problem+json, day counters across restarts, sniffing, target throttle, port cap, abuse log + prune + metric, semaphore, job bounds (queue, disk, per address), challenge hook: all checks passed');
+ console.log('guard: callers (/64, one loopback hop, services by token, aud-checked users, sessions, sandbox), bucket math + cost, session sharing, report vs enforce, headers + problem+json, day counters across restarts, sniffing, target throttle, port cap, abuse log + prune + metric, semaphore, job bounds (queue, disk, per address), challenge hook, Origin check (CSRF): all checks passed');
})().catch((err) => { console.error(err); process.exit(1); });
})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,161 @@
@@ -0,0 +1,161 @@
+'use strict';
+// The gateway's /api/v1/jobs facade (apps/gateway/server/run/jobs-facade.js) on real processes: the
+// gateway in front of img and docs, a stand-in Network for tokens.
+// • submits are routed by the job type's prefix: JSON, a multipart `type` part read from the head of
+// the stream (uploads stream on), or ?type=; a type it cannot find or does not know is refused
+// • everything else is routed by the job id: the satellite that answered (remembered), else the one
+// that says it holds the job on loopback (after a gateway restart); reads, cancel, retry (the new
+// job is remembered), references, SSE events and result files all stream through
+// • owners stay the satellite's business (another person gets 404), unknown ids are 404
+// • the loopback probe (GET /api/internal/jobs/:id) answers direct callers only and is never limited
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const { startApp, freePort } = require('./spawn');
+const { startNetwork } = require('./network');
+
+const APPS = path.join(__dirname, '..', '..');
+const sharp = require(require.resolve('sharp', { paths: [path.join(APPS, 'img')] }));
+const sleep = (ms) => new Promise(r => setTimeout(r, ms));
+
+(async () => {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-jobs-facade-'));
+ const net = await startNetwork();
+ const procs = [];
+ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };
+ try {
+ const ports = { img: await freePort(), docs: await freePort(), gw: await freePort() };
+ const env = { OV_NETWORK_URL: net.url, OV_NETWORK_INTERNAL_URL: net.url, TOOLS_SATELLITE_PORTS: `img=${ports.img},docs=${ports.docs}` };
+ const img = await start('img', { ...env, DATA_DIR: path.join(tmp, 'img'), UPLOADS_DIR: path.join(tmp, 'img', 'up'), OUTPUT_DIR: path.join(tmp, 'img', 'out') }, ports.img);
+ // docs never starts a job here: its jobs stay queued (cancel).
+ const docs = await start('docs', { ...env, DATA_DIR: path.join(tmp, 'docs'), UPLOADS_DIR: path.join(tmp, 'docs', 'up'), OUTPUT_DIR: path.join(tmp, 'docs', 'out'), TOOLS_JOBS_CONCURRENCY_DOCS: '0' }, ports.docs);
+ const gwEnv = { ...env, DATA_DIR: path.join(tmp, 'gw'), OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' };
+ let gw = await start('gateway', gwEnv, ports.gw);
+ const G = gw.base;
+ const me = { Authorization: `Bearer ${net.user()}` };
+ const other = { Authorization: `Bearer ${net.user()}` };
+ const req = async (p, { method = 'GET', headers = {}, body, base = G } = {}) => {
+ const r = await fetch(`${base}${p}`, { method, headers, body });
+ const text = await r.text();
+ let json = null; try { json = text ? JSON.parse(text) : null; } catch { /* not JSON */ }
+ return { status: r.status, h: r.headers, text, body: json };
+ };
+ const png = await sharp({ create: { width: 6, height: 6, channels: 3, background: '#00f' } }).png().toBuffer();
+ const form = (parts) => {
+ const fd = new FormData();
+ for (const [k, v] of parts) { if (v instanceof Blob) fd.append(k, v, 'x.png'); else fd.append(k, String(v)); }
+ return fd;
+ };
+ const finished = async (id, headers) => {
+ for (let i = 0; i < 100; i++) {
+ const r = await req(`/api/v1/jobs/${id}`, { headers });
+ if (['succeeded', 'failed', 'cancelled'].includes(r.body.state)) return r.body;
+ await sleep(50);
+ }
+ throw new Error(`${id} did not finish`);
+ };
+
+ // ── Submit, routed by the multipart type part ──
+ let r = await req('/api/v1/jobs', { method: 'POST', headers: me, body: form([['type', 'img.process'], ['input', JSON.stringify({ tool: 'convert', format: 'webp' })], ['file', new Blob([png])]]) });
+ assert.strictEqual(r.status, 202, r.text);
+ assert.strictEqual(r.body.service, 'img');
+ assert.strictEqual(r.h.get('location'), `/api/v1/jobs/${r.body.id}`);
+ const job = r.body.id;
+ const done = await finished(job, me);
+ assert.strictEqual(done.state, 'succeeded', JSON.stringify(done.error));
+ // Events stream through (SSE, ids, the terminal event).
+ {
+ const res = await fetch(`${G}/api/v1/jobs/${job}/events`, { headers: { ...me, Accept: 'text/event-stream' } });
+ // Finished and nothing newer than Last-Event-ID 0? It replays the whole log first.
+ assert.strictEqual(res.status, 200);
+ assert.match(res.headers.get('content-type'), /text\/event-stream/);
+ const text = await res.text();
+ assert.match(text, /^retry: 3000/);
+ assert.match(text, /id: \d+\nevent: job\.succeeded\n/);
+ const last = Math.max(...[...text.matchAll(/^id: (\d+)$/gm)].map(m => Number(m[1])));
+ const again = await fetch(`${G}/api/v1/jobs/${job}/events`, { headers: { ...me, 'Last-Event-ID': String(last) } });
+ assert.strictEqual(again.status, 204, 'nothing after the last event: 204');
+ }
+ // The result file streams through.
+ r = await fetch(`${G}/api/v1/jobs/${job}/files/0`, { headers: me });
+ assert.strictEqual(r.status, 200);
+ assert.match(r.headers.get('content-disposition'), /attachment; filename="x\.webp"/);
+ assert.strictEqual((await sharp(Buffer.from(await r.arrayBuffer())).metadata()).format, 'webp');
+ // References through the facade.
+ r = await req(`/api/v1/jobs/${job}/references/community:paste:p_1`, { method: 'PUT', headers: me });
+ assert.strictEqual(r.status, 201, r.text);
+ assert.strictEqual(r.body.expires_at, null);
+ r = await req(`/api/v1/jobs/${job}/references/community:paste:p_1`, { method: 'DELETE', headers: me });
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.body.expires_at);
+ // Somebody else: the satellite's 404, as for a job that does not exist.
+ r = await req(`/api/v1/jobs/${job}`, { headers: other });
+ assert.deepStrictEqual([r.status, r.body.code], [404, 'tools.job.not_found']);
+ r = await req('/api/v1/jobs/job_01J00000000000000000000000', { headers: me });
+ assert.deepStrictEqual([r.status, r.body.code], [404, 'tools.job.not_found']);
+ assert.strictEqual((await req('/api/v1/jobs/not-a-job', { headers: me })).status, 404);
+
+ // ── A failed job, retried through the facade (the new job is remembered) ──
+ const broken = Buffer.concat([png.subarray(0, 40), Buffer.alloc(200, 7)]); // a PNG header, then noise
+ r = await req('/api/v1/jobs', { method: 'POST', headers: me, body: form([['type', 'img.process'], ['input', JSON.stringify({ tool: 'convert', format: 'png' })], ['file', new Blob([broken])]]) });
+ assert.strictEqual(r.status, 202, r.text);
+ const failed = await finished(r.body.id, me);
+ assert.strictEqual(failed.state, 'failed');
+ r = await req(`/api/v1/jobs/${failed.id}/retry`, { method: 'POST', headers: me });
+ assert.strictEqual(r.status, 202, r.text);
+ assert.strictEqual(r.body.retry_of, failed.id);
+ const retried = r.body.id;
+ assert.strictEqual((await finished(retried, me)).state, 'failed');
+
+ // ── docs: JSON body type; ?type= when the type part comes late; cancel a queued job ──
+ r = await req('/api/v1/jobs', { method: 'POST', headers: { ...me, 'Content-Type': 'application/json' }, body: JSON.stringify({ type: 'docs.process', input: { tool: 'merge' } }) });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.invalid'], 'routed to docs, which wants files');
+ assert.match(r.body.detail, /at least 2 files|needs a file/);
+ const big = Buffer.alloc(1024 * 1024, 1);
+ const pdfish = Buffer.concat([Buffer.from('%PDF-1.4\n'), big]);
+ r = await req('/api/v1/jobs', { method: 'POST', headers: me, body: form([['file', new Blob([pdfish])], ['type', 'docs.process'], ['input', JSON.stringify({ tool: 'compress' })]]) });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.invalid'], 'a type part after the first 64 KB of the stream is not looked for');
+ r = await req('/api/v1/jobs?type=docs.process', { method: 'POST', headers: me, body: form([['file', new Blob([pdfish])], ['type', 'docs.process'], ['input', JSON.stringify({ tool: 'compress' })]]) });
+ assert.strictEqual(r.status, 202, r.text);
+ assert.strictEqual(r.body.service, 'docs');
+ const queued = r.body.id;
+ r = await req(`/api/v1/jobs/${queued}`, { method: 'DELETE', headers: me });
+ assert.deepStrictEqual([r.status, r.body.state], [200, 'cancelled']);
+ r = await req('/api/v1/jobs', { method: 'POST', headers: { ...me, 'Content-Type': 'application/json' }, body: JSON.stringify({ type: 'video.process' }) });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.unknown_type']);
+ r = await req('/api/v1/jobs', { method: 'POST', headers: { ...me, 'Content-Type': 'application/json' }, body: '{"type":' });
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.job.invalid']);
+ assert.strictEqual((await req('/api/v1/jobs', { headers: me })).status, 405);
+
+ // ── After a gateway restart: nothing remembered, the satellites are asked ──
+ const before = (await req('/api/health')).body.jobs;
+ assert.ok(before.known >= 4, JSON.stringify(before));
+ await gw.kill('SIGTERM');
+ procs.splice(procs.indexOf(gw), 1);
+ gw = await start('gateway', gwEnv, ports.gw);
+ for (const [id, want] of [[job, 'succeeded'], [retried, 'failed'], [queued, 'cancelled']]) {
+ r = await req(`/api/v1/jobs/${id}`, { headers: me });
+ assert.deepStrictEqual([r.status, r.body.state], [200, want], `${id} found again`);
+ }
+ const after = (await req('/api/health')).body.jobs;
+ assert.strictEqual(after.probed, 3, 'each asked once');
+ await req(`/api/v1/jobs/${job}`, { headers: me });
+ assert.strictEqual((await req('/api/health')).body.jobs.probed, 3, 'then remembered');
+
+ // ── The loopback probe: direct callers only, never limited ──
+ r = await req(`/api/internal/jobs/${job}`, { base: img.base });
+ assert.deepStrictEqual([r.status, r.body], [200, { id: job, service: 'img' }]);
+ assert.strictEqual((await req(`/api/internal/jobs/${job}`, { base: docs.base })).status, 404, 'not on docs');
+ assert.strictEqual((await req(`/api/internal/jobs/${job}`, { base: img.base, headers: { 'X-Forwarded-For': '203.0.113.5' } })).status, 404, 'through a proxy: nothing to see');
+ const statuses = await Promise.all(Array.from({ length: 150 }, () => req(`/api/internal/jobs/${job}`, { base: img.base }).then(x => x.status)));
+ assert.ok(statuses.every(s => s === 200), 'the facade\'s probes are never rate limited');
+ // Through the gateway's host proxy it does not exist either.
+ r = await req(`/api/internal/jobs/${job}`, { headers: { 'X-Forwarded-For': '203.0.113.5' } });
+ assert.strictEqual(r.status, 404);
+ } finally {
+ for (const p of procs) await p.kill('SIGTERM');
+ await net.close();
+ }
+ console.log('jobs facade: submit routed by type (multipart head, JSON, ?type=), by id (remembered, then probed after a restart), events/files/references/retry/cancel through, owner 404s, loopback probe: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
+'use strict';
+// A stand-in OpenVibe.Network for tests that run real Tools processes: it serves the JWKS endpoint
+// (GET /api/.well-known/jwks → { public_key }) and signs the tokens the Network would issue — a
+// person's browser sign-in (aud openvibe.tools, subject_id usr_…) and service or app principals
+// (client credentials: aud openvibe.tools, cap[]). Point OV_NETWORK_URL and OV_NETWORK_INTERNAL_URL
+// at `url`; the issuer is `url` too.
+const crypto = require('crypto');
+const http = require('http');
+
+const b64 = (o) => Buffer.from(JSON.stringify(o)).toString('base64url');
+
+async function startNetwork() {
+ const { publicKey, privateKey } = crypto.generateKeyPairSync('rsa', { modulusLength: 2048, publicKeyEncoding: { type: 'spki', format: 'pem' }, privateKeyEncoding: { type: 'pkcs8', format: 'pem' } });
+ const srv = http.createServer((req, res) => {
+ if (req.url.startsWith('/api/.well-known/jwks')) { res.setHeader('Content-Type', 'application/json'); return res.end(JSON.stringify({ public_key: publicKey, algorithm: 'RS256' })); }
+ res.statusCode = 404; return res.end('{}');
+ });
+ await new Promise(r => srv.listen(0, '127.0.0.1', r));
+ const url = `http://127.0.0.1:${srv.address().port}`;
+ const now = () => Math.floor(Date.now() / 1000);
+ const sign = (claims) => {
+ const head = b64({ alg: 'RS256', typ: 'JWT' });
+ const body = b64(claims);
+ const sig = crypto.sign('RSA-SHA256', Buffer.from(`${head}.${body}`), privateKey).toString('base64url');
+ return `${head}.${body}.${sig}`;
+ };
+ let n = 0;
+ const ulid = () => `01J${crypto.randomBytes(16).toString('hex').toUpperCase().replace(/[^0-9A-HJKMNP-TV-Z]/g, 'Z').slice(0, 23)}`;
+ return {
+ url, issuer: url, publicKey,
+ /** A signed-in person (what the Network puts in ov_token). */
+ user(extra = {}) {
+ n++;
+ return sign({ iss: url, sub: String(1000 + n), id: 1000 + n, subject_id: `usr_${ulid()}`, username: `person${n}`, aud: ['openvibe.live', 'openvibe.tools', 'openvibe.network'], iat: now(), exp: now() + 3600, ...extra });
+ },
+ /** A service principal's client-credentials token with these capabilities. */
+ service(cap, extra = {}) {
+ return sign({ iss: url, sub: `svc:${extra.name || 'partner'}`, actor_type: 'service', aud: ['openvibe.tools'], cap, ns: [], iat: now(), exp: now() + 900, jti: `tok_${crypto.randomBytes(12).toString('hex')}`, ...extra, name: undefined });
+ },
+ sign,
+ close: () => new Promise(r => srv.close(r)),
+ };
+}
+
+module.exports = { startNetwork };
new file mode 100644
new file mode 100644
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// The older tool endpoints (/api/process…, /api/net/*, /api/dev/*) keep working with their own
+// response shapes, and say what replaces them (RFC 9745 Deprecation, RFC 8594 Sunset, RFC 8288 Link):
+//
+// Deprecation: true
+// Sunset: Thu, 31 Dec 2026 23:59:59 GMT
+// Link: </api/v1/tools/{id}/run>; rel="successor-version"
+// ═══════════════════════════════════════════════════════════════
+
+const SUNSET = 'Thu, 31 Dec 2026 23:59:59 GMT';
+
+/** Mark an answer deprecated; `successor` is a path (the tool's run route, or a registry listing). */
+function deprecate(res, successor) {
+ if (res.headersSent) return;
+ res.setHeader('Deprecation', 'true');
+ res.setHeader('Sunset', SUNSET);
+ if (successor) res.setHeader('Link', `<${successor}>; rel="successor-version"`);
+}
+
+/** Middleware: successorOf(req) → a path or null (null: deprecated without a named successor is not said). */
+function deprecated(successorOf) {
+ return (req, res, next) => {
+ const s = successorOf(req);
+ if (s) deprecate(res, s);
+ next();
+ };
+}
+
+const runPath = (id) => (id ? `/api/v1/tools/${id}/run` : null);
+
+module.exports = { deprecate, deprecated, runPath, SUNSET };
@@ -14,13 +14,15 @@
@@ -14,13 +14,15 @@
// requires programs the tool needs on the host ('qpdf', 'heif-dec', …); the satellite marks it
// requires programs the tool needs on the host ('qpdf', 'heif-dec', …); the satellite marks it
// unavailable while one is missing
// unavailable while one is missing
// example { input } that runs: used by the tests, later by the docs
// example { input } that runs: used by the tests, later by the docs
+// cacheTtlMs how long the run API may reuse an inline answer (0: never — random or time-dependent
+// output; unset: engines 10 minutes, routes never)
//
//
// Like the rest of apps/_shared this file has no dependencies.
// Like the rest of apps/_shared this file has no dependencies.
// ═══════════════════════════════════════════════════════════════
// ═══════════════════════════════════════════════════════════════
const SITE = 'https://openvibe.tools';
const SITE = 'https://openvibe.tools';
const JSON_SCHEMA = 'https://json-schema.org/draft/2020-12/schema';
const JSON_SCHEMA = 'https://json-schema.org/draft/2020-12/schema';
-const INTERNAL = new Set(['id', 'engine', 'route', 'requires', 'example', 'job', 'legacy', 'status', 'statusReason']);
+const INTERNAL = new Set(['id', 'engine', 'route', 'requires', 'example', 'job', 'legacy', 'status', 'statusReason', 'cacheTtlMs']);
const ANYONE = Object.freeze({ anonymous: true, capability: 'tools.tool.run' });
const ANYONE = Object.freeze({ anonymous: true, capability: 'tools.tool.run' });
const runPath = (id) => `/api/v1/tools/${id}/run`;
const runPath = (id) => `/api/v1/tools/${id}/run`;
@@ -33,12 +33,38 @@ function openCors(res) {
@@ -33,12 +33,38 @@ function openCors(res) {
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
}
}
-/** Wrap an app's CORS middleware: registry paths skip it and are open to every origin instead. */
-function exceptRegistry(corsMiddleware) {
+// The run API and the jobs API take calls from any site's page with a token (openvibe-sdk in a
+// browser): such an origin gets CORS without credentials (so the browser never sends our cookies
+// along); first-party pages keep the app's own credentialed CORS.
+const RUN_OR_JOBS_RE = /^\/api\/v1\/(?:tools\/[a-z][a-z0-9-]{0,39}\/run|jobs(?:\/.*)?)\/?$/;
+const isPublicApiPath = (p) => RUN_OR_JOBS_RE.test(String(p || ''));
+const API_HEADERS = 'Authorization, Content-Type, Idempotency-Key, Last-Event-ID, Cache-Control, traceparent, X-OpenVibe-Request-Id';
+const API_EXPOSE = 'Location, Idempotent-Replayed, Retry-After, RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, X-OpenVibe-Request-Id, traceparent, Content-Disposition, Deprecation, Sunset, Link';
+
+/**
+ * Wrap an app's CORS middleware: registry paths skip it and are open to every origin instead; run and
+ * jobs paths from an origin the app does not know (isFirstParty false) get CORS without credentials.
+ * @param {Function} corsMiddleware
+ * @param {{ isFirstParty?: (origin) => boolean }} [opts]
+ */
+function exceptRegistry(corsMiddleware, opts = {}) {
return function corsExceptToolRegistry(req, res, next) {
return function corsExceptToolRegistry(req, res, next) {
- if (!isRegistryPath(req.path)) return corsMiddleware(req, res, next);
- openCors(res);
- return next();
+ if (isRegistryPath(req.path)) { openCors(res); return next(); }
+ const origin = req.headers.origin;
+ if (opts.isFirstParty && origin && isPublicApiPath(req.path) && !opts.isFirstParty(origin)) {
+ res.setHeader('Access-Control-Allow-Origin', '*');
+ res.setHeader('Access-Control-Expose-Headers', API_EXPOSE);
+ res.setHeader('Vary', 'Origin');
+ if (req.method === 'OPTIONS') {
+ res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
+ res.setHeader('Access-Control-Allow-Headers', API_HEADERS);
+ res.setHeader('Access-Control-Max-Age', '600');
+ res.statusCode = 204;
+ return res.end();
+ }
+ return next();
+ }
+ return corsMiddleware(req, res, next);
};
};
}
}
@@ -158,4 +184,4 @@ function createToolsApi(o) {
@@ -158,4 +184,4 @@ function createToolsApi(o) {
};
};
}
}
-module.exports = { createToolsApi, exceptRegistry, isRegistryPath, parseFilters, PATH_RE };
+module.exports = { createToolsApi, exceptRegistry, isRegistryPath, isPublicApiPath, parseFilters, PATH_RE };
new file mode 100644
new file mode 100644
@@ -0,0 +1,96 @@
@@ -0,0 +1,96 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// Forward one API request to a Tools app on loopback and stream the answer back (the gateway's run
+// API for satellite tools and its /api/v1/jobs facade). Uploads, SSE event streams and result files
+// pass through without being buffered. The caller's own address goes along as the one trusted hop
+// (X-Forwarded-For = req.ip, as the host proxy does), with its credentials, Idempotency-Key,
+// Last-Event-ID and trace headers. Small JSON answers can be read on the way (onJson), e.g. to learn
+// which satellite holds a job. The gateway's CORS headers are the ones the browser sees.
+// No dependencies (node:http).
+// ═══════════════════════════════════════════════════════════════
+
+const http = require('http');
+
+const HOP = new Set(['connection', 'keep-alive', 'proxy-authenticate', 'proxy-authorization', 'te', 'trailer', 'transfer-encoding', 'upgrade', 'host']);
+const JSON_MAX = 1024 * 1024;
+
+function outHeaders(req, extra = {}) {
+ const headers = {};
+ for (const [k, v] of Object.entries(req.headers)) if (!HOP.has(k) && !k.startsWith('x-ov-')) headers[k] = v;
+ const client = String(req.ip || (req.socket && req.socket.remoteAddress) || '');
+ headers['x-forwarded-for'] = client;
+ headers['x-real-ip'] = client;
+ headers['cf-connecting-ip'] = client;
+ headers['x-forwarded-host'] = String(req.headers['x-forwarded-host'] || req.headers.host || '');
+ headers['x-forwarded-proto'] = 'https';
+ // The satellite sees the public host (same-origin checks, canonical links) as the gateway did.
+ if (req.headers.host) headers.host = req.headers.host;
+ return Object.assign(headers, extra);
+}
+
+/**
+ * @param {object} req @param {object} res
+ * @param {object} o
+ * @param {number} o.port
+ * @param {string} [o.path] default req.originalUrl
+ * @param {string} [o.method] default req.method
+ * @param {Buffer} [o.body] the whole body (already read); else `head` + the rest of req
+ * @param {Buffer} [o.head] bytes of req already read (a peek), sent before the rest
+ * @param {(body, status, headers) => void} [o.onJson] sees a JSON answer (≤ 1 MB) before it is sent
+ * @param {number} [o.timeoutMs=600000]
+ * @param {() => void} [o.onError] the satellite could not be reached (default: 502 problem)
+ * @returns {Promise<void>} settles when the answer has been handed over
+ */
+function forward(req, res, o) {
+ return new Promise((resolve) => {
+ const extra = {};
+ if (o.body) { extra['content-length'] = String(o.body.length); delete req.headers['transfer-encoding']; }
+ const up = http.request({ host: '127.0.0.1', port: o.port, method: o.method || req.method, path: o.path || req.originalUrl || req.url, headers: outHeaders(req, extra) }, (r) => {
+ const headers = {};
+ for (const [k, v] of Object.entries(r.headers)) if (!HOP.has(k) && !k.startsWith('access-control-')) headers[k] = v;
+ const type = String(r.headers['content-type'] || '');
+ const len = Number(r.headers['content-length'] || 0);
+ if (o.onJson && /json/.test(type) && len <= JSON_MAX) {
+ const chunks = [];
+ let size = 0;
+ r.on('data', (c) => { size += c.length; if (size <= JSON_MAX) chunks.push(c); });
+ r.on('end', () => {
+ const buf = Buffer.concat(chunks);
+ if (size <= JSON_MAX) { try { o.onJson(JSON.parse(buf.toString('utf8')), r.statusCode, r.headers); } catch { /* not JSON after all */ } }
+ headers['content-length'] = String(buf.length);
+ if (!res.headersSent) res.writeHead(r.statusCode || 502, headers);
+ res.end(buf);
+ resolve();
+ });
+ r.on('error', () => { res.destroy(); resolve(); });
+ return;
+ }
+ res.writeHead(r.statusCode || 502, headers);
+ if (typeof res.flushHeaders === 'function' && /event-stream/.test(type)) res.flushHeaders();
+ r.pipe(res);
+ r.on('end', resolve);
+ r.on('error', () => { res.destroy(); resolve(); });
+ });
+ up.setTimeout(o.timeoutMs || 10 * 60_000, () => up.destroy(new Error('upstream timeout')));
+ up.on('error', () => {
+ if (!res.headersSent) {
+ if (o.onError) o.onError();
+ else {
+ res.statusCode = 502;
+ res.setHeader('Content-Type', 'application/problem+json');
+ res.setHeader('Retry-After', '5');
+ res.end(JSON.stringify({ type: 'https://openvibe.network/problems/tools.unavailable', title: 'Bad Gateway', status: 502, code: 'tools.unavailable', detail: 'This tool is restarting. Try again in a moment.', error: 'This tool is restarting. Try again in a moment.' }));
+ }
+ } else res.destroy();
+ resolve();
+ });
+ res.on('close', () => { if (!res.writableFinished) up.destroy(); });
+ if (o.body) up.end(o.body);
+ else {
+ if (o.head && o.head.length) up.write(o.head);
+ req.pipe(up);
+ }
+ });
+}
+
+module.exports = { forward, outHeaders };
new file mode 100644
new file mode 100644
@@ -0,0 +1,601 @@
@@ -0,0 +1,601 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// The uniform run API (ADR-027; openvibe-contracts tools.run-request@1 → tools.run@1):
+//
+// POST /api/v1/tools/:id/run
+//
+// Body: JSON { input, files, wait_ms, idempotency_key } (tools.run-request@1), or multipart/form-data
+// with the uploads as `file` (or `files`) file parts and the other fields as text parts: `input` (JSON
+// text), `files` (the file references' JSON), `wait_ms`, `idempotency_key` — exactly what
+// openvibe-sdk/tools v0.6.0 sends. ?wait_ms= on the query does the same as the field; the
+// Idempotency-Key header wins over idempotency_key.
+//
+// Answers (tools.run@1):
+// 200 { state: succeeded, tool, result: { data | text | files }, took_ms[, job] }
+// 200 { state: failed | cancelled, tool, error (problem+json), took_ms[, job] } the tool's own failure
+// 202 { state: queued | running, tool, job, location } + Location a job still going after wait_ms
+// 200 + Idempotent-Replayed: true the same key and request again
+// Refusals before the tool runs are problem+json: 400 tools.run.invalid, 401 token.missing |
+// tools.session_required, 403 capability.denied | tools.origin.refused, 404 tools.tool.not_found |
+// tools.tool.not_runnable (api false, e.g. the YouTube downloader) | tools.run.file_not_found,
+// 409 tools.job.idempotency_conflict, 413 tools.file.too_large | tools.input.too_large,
+// 415 tools.file.unsupported_type, 422 tools.input.invalid (errors[] with JSON pointers),
+// 429 tools.quota.exceeded (Retry-After), 503 tools.tool.unavailable (the descriptor's status is
+// unavailable), 503 tools.unavailable (a program the tool needs is missing), 503 tools.busy.
+//
+// Who may run what (always enforced, whatever TOOLS_GUARD says, because it is authorization, not a
+// quota): an app or service token needs the tool's auth.capability (tools.tool.run, or tools.net.probe
+// for the network probes, which nobody else runs through the API: people keep the probe pages); a
+// tool with auth.anonymous false needs a browser session, a sign-in or a token. Quotas (the tool's
+// quotaClass × the caller's tier, weighted by its cost), the Origin check on cookie-authenticated
+// calls and the job bounds follow the guard's mode; the per-target throttle and upload sniffing are
+// hard limits in both.
+//
+// Where a tool runs is the app's business (o.runs): inline (in this process — the gateway's net and
+// dev routes, and the dev and text engines in a worker pool), as a job of this satellite's job system,
+// or forwarded (o.proxy: the gateway streams a job tool's run to the satellite that owns it).
+// Inline answers are cached per tool where that is safe (o.cacheTtl: pure transforms and slow-moving
+// lookups; never a probe, a random generator or the caller's own address). An inline run's
+// Idempotency-Key replays its first answer for 15 minutes; a job tool's key is the job's.
+//
+// Like the rest of apps/_shared this has no dependencies: the app passes its guard, contracts, an
+// input validator (createInputValidator with the Ajv it has) and, on a satellite, multer.
+// ═══════════════════════════════════════════════════════════════
+
+const crypto = require('crypto');
+const fs = require('fs');
+const fsp = require('fs/promises');
+const path = require('path');
+const { canonical } = require('../jobs/system');
+
+const RUN_RE = /^\/api\/v1\/tools\/([a-z][a-z0-9-]{0,39})\/run\/?$/;
+const FIELDS = new Set(['input', 'files', 'wait_ms', 'idempotency_key']);
+const KEY_RE = /^[\x21-\x7e]{8,200}$/;
+const MEDIA_ID_RE = /^med_[0-9A-HJKMNP-TV-Z]{26}$/;
+const JOB_ID_RE = /^job_[0-9A-HJKMNP-TV-Z]{26}$/;
+const MAX_WAIT_MS = 60_000;
+const TERMINAL = new Set(['succeeded', 'failed', 'cancelled']);
+const REPLAY_TTL_MS = 15 * 60 * 1000;
+
+const isRunPath = (p) => RUN_RE.test(String(p || ''));
+
+/** Run Express middleware in order (the apps' older limiters) → true when every one called next(). */
+async function passes(list, req, res) {
+ for (const mw of list || []) {
+ const ok = await new Promise((resolve) => {
+ let settled = false;
+ const done = (v) => { if (!settled) { settled = true; res.removeListener('finish', onFinish); resolve(v); } };
+ const onFinish = () => done(false);
+ res.once('finish', onFinish);
+ try { mw(req, res, (err) => done(!err)); } catch { done(false); }
+ });
+ if (!ok || res.headersSent) return false;
+ }
+ return true;
+}
+
+/** A refusal before the tool ran: answered as problem+json, not as a run. */
+class Refusal extends Error {
+ constructor(status, code, detail, extra, headers) { super(detail || code); this.status = status; this.code = code; this.detail = detail; this.extra = extra; this.headers = headers; this.refusal = true; }
+}
+
+// ── Input validation (the tool's own JSON Schema) ────────────
+
+/**
+ * @param {Function} Ajv ajv/dist/2020 (from the app: openvibe-contracts brings it)
+ * @param {Function} [addFormats] ajv-formats
+ * @returns {(d, input) => { path, message }[]}
+ */
+function createInputValidator(Ajv, addFormats) {
+ const ajv = new Ajv({ allErrors: true, strict: false, allowUnionTypes: true, logger: false });
+ if (addFormats) addFormats(ajv);
+ const compiled = new Map(); // id → { sig, fn, schema }
+ return function validate(d, input) {
+ if (!d.input || typeof d.input !== 'object') return [];
+ const sig = JSON.stringify(d.input);
+ let v = compiled.get(d.id);
+ if (!v || v.sig !== sig) {
+ if (v && v.schema) { try { ajv.removeSchema(v.schema); } catch { /* not registered */ } }
+ const schema = { ...d.input };
+ delete schema.$id;
+ v = { sig, schema, fn: ajv.compile(schema) };
+ compiled.set(d.id, v);
+ }
+ if (v.fn(input)) return [];
+ return (v.fn.errors || []).slice(0, 20).map((e) => {
+ if (e.keyword === 'additionalProperties') return { path: `${e.instancePath}/${e.params.additionalProperty}`, message: 'is not a field of this tool' };
+ if (e.keyword === 'required') return { path: `${e.instancePath}/${e.params.missingProperty}`, message: 'is required' };
+ return { path: e.instancePath || '/', message: e.message || 'is not valid' };
+ });
+ };
+}
+
+// ── Answers kept for a while ─────────────────────────────────
+
+/** A TTL + LRU map bounded by entries and bytes (the inline cache, inline Idempotency-Key replays). */
+function createTtlStore({ maxEntries = 2000, maxBytes = 32 * 1024 * 1024, now = Date.now } = {}) {
+ const map = new Map(); // key → { value, size, until }
+ let bytes = 0, hits = 0, misses = 0;
+ const drop = (k) => { const e = map.get(k); if (e) { bytes -= e.size; map.delete(k); } };
+ return {
+ get(k) {
+ const e = map.get(k);
+ if (!e) { misses++; return undefined; }
+ if (e.until <= now()) { drop(k); misses++; return undefined; }
+ map.delete(k); map.set(k, e); // most recently used last
+ hits++;
+ return e.value;
+ },
+ set(k, value, ttlMs, size) {
+ if (!(ttlMs > 0)) return false;
+ const n = size || Buffer.byteLength(JSON.stringify(value));
+ if (n > maxBytes / 8) return false; // one answer never takes a big share
+ drop(k);
+ map.set(k, { value, size: n, until: now() + ttlMs });
+ bytes += n;
+ while (map.size > maxEntries || bytes > maxBytes) drop(map.keys().next().value);
+ return true;
+ },
+ stats: () => ({ entries: map.size, bytes, hits, misses }),
+ clear() { map.clear(); bytes = 0; },
+ };
+}
+
+// ── Request parsing ──────────────────────────────────────────
+
+function toInt(v) {
+ if (v === undefined || v === null || v === '') return undefined;
+ const n = typeof v === 'number' ? v : Number(String(v).trim());
+ return Number.isInteger(n) ? n : NaN;
+}
+
+/** tools.run-request@1 from req.body (JSON, or multipart text fields) and the query → fields, or throws Refusal. */
+function readRequest(req, multipart) {
+ const b = req.body && typeof req.body === 'object' && !Array.isArray(req.body) ? req.body : null;
+ if (!b && req.body != null && req.body !== '') throw new Refusal(400, 'tools.run.invalid', 'The body is a JSON object (tools.run-request@1), or multipart/form-data.');
+ const body = b || {};
+ for (const k of Object.keys(body)) if (!FIELDS.has(k)) throw new Refusal(400, 'tools.run.invalid', `Unknown field "${k}": a run request has input, files, wait_ms and idempotency_key.`);
+ const parseJson = (name, v, dflt) => {
+ if (!multipart || typeof v !== 'string') return v === undefined ? dflt : v;
+ if (!v.trim()) return dflt;
+ try { return JSON.parse(v); } catch { throw new Refusal(400, 'tools.run.invalid', `The ${name} part must be JSON.`); }
+ };
+ const input = parseJson('input', body.input, {});
+ if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Refusal(400, 'tools.run.invalid', 'input is a JSON object.');
+ const refs = parseJson('files', body.files, []);
+ if (!Array.isArray(refs) || refs.length > 50) throw new Refusal(400, 'tools.run.invalid', 'files is an array of at most 50 references ({ media_id } or { job_id, index }).');
+ for (const r of refs) {
+ const okMedia = r && typeof r === 'object' && Object.keys(r).length === 1 && MEDIA_ID_RE.test(String(r.media_id || ''));
+ const okJob = r && typeof r === 'object' && Object.keys(r).length === 2 && JOB_ID_RE.test(String(r.job_id || '')) && Number.isInteger(r.index) && r.index >= 0 && r.index <= 99;
+ if (!okMedia && !okJob) throw new Refusal(400, 'tools.run.invalid', 'A files reference is { media_id: "med_…" } or { job_id: "job_…", index: 0..99 }.');
+ }
+ let wait = toInt(body.wait_ms);
+ if (wait === undefined) wait = toInt(req.query && req.query.wait_ms);
+ if (wait === undefined) wait = 0;
+ if (!Number.isInteger(wait) || wait < 0 || wait > MAX_WAIT_MS) throw new Refusal(400, 'tools.run.invalid', `wait_ms is an integer, 0..${MAX_WAIT_MS}.`);
+ const header = req.headers['idempotency-key'];
+ const key = header != null && header !== '' ? String(header) : body.idempotency_key != null && body.idempotency_key !== '' ? String(body.idempotency_key) : null;
+ if (key != null && !KEY_RE.test(key)) throw new Refusal(400, 'tools.run.invalid', 'Idempotency-Key is 8-200 printable ASCII characters.');
+ return { input, refs, wait, key };
+}
+
+const sha = (s) => crypto.createHash('sha256').update(s).digest('hex');
+
+/**
+ * @param {object} o
+ * @param {object} o.guard apps/_shared/guard
+ * @param {object} o.contracts openvibe-contracts
+ * @param {() => object} o.snapshot the registry: { tools: [descriptor], gone: Map }
+ * @param {(d, input) => object[]} o.validate
+ * @param {(d) => 'inline'|'job'|null} o.runs what this app runs itself
+ * @param {(d, input, ctx) => Promise<{data}|{text}>} [o.inline] ctx: { req, res, caller, signal, timeoutMs }
+ * @param {(d) => number} [o.cacheTtl] ms an inline answer may be reused (0 = never)
+ * @param {(d, input) => string|null} [o.targetOf] the host an egress tool reaches (per-target throttle)
+ * @param {(d, req, res) => boolean} [o.proxy] forwards the run elsewhere (the gateway)
+ * @param {Function} [o.parseJson] JSON body parser middleware, when the app has not run one
+ * @param {object} [o.jobs] a satellite's jobs: { system: () => system, receive: (d) => middleware,
+ * fetchSibling: (ref, req) → file | null }
+ * @param {object} [o.log]
+ */
+function createRunApi(o) {
+ const { guard, contracts } = o;
+ const log = o.log || console;
+ const cache = createTtlStore({ maxEntries: 5000, maxBytes: 64 * 1024 * 1024 });
+ const replays = createTtlStore({ maxEntries: 20000, maxBytes: 32 * 1024 * 1024 });
+ const counts = { runs: 0, inline: 0, jobs: 0, proxied: 0, refused: 0, failed: 0, cached: 0, replayed: 0 };
+
+ function problem(req, res, status, code, detail, extra, headers) {
+ counts.refused++;
+ if (headers) for (const [k, v] of Object.entries(headers)) res.setHeader(k, v);
+ res.setHeader('Cache-Control', 'no-store');
+ return contracts.http.sendProblem(res, status, code, { detail, extra, ctx: req.ov });
+ }
+ const send = (res, status, body, headers = {}) => {
+ res.setHeader('Cache-Control', 'no-store');
+ for (const [k, v] of Object.entries(headers)) res.setHeader(k, v);
+ res.status(status).json(body);
+ };
+ const toolProblem = (req, err, fallbackStatus = 422) => {
+ const status = Number.isInteger(err.status) && err.status >= 400 && err.status <= 599 ? err.status : fallbackStatus;
+ const code = /^tools\.[a-z0-9_.]+$/.test(String(err.code || '')) ? err.code : 'tools.job.failed';
+ const detail = String(err.detail || err.message || 'The tool failed').replace(/(?:\/[\w.-]+){2,}/g, '[file]').slice(0, 500);
+ return contracts.http.problem(status, code, { detail, ctx: req.ov });
+ };
+
+ // ── Who may run it (always enforced) ─────────────────────
+ function authorize(req, res, d) {
+ const c = guard.caller(req, res);
+ const cap = (d.auth && d.auth.capability) || 'tools.tool.run';
+ const probe = cap === 'tools.net.probe';
+ // A token that does not verify (expired, another audience) is an error here, not anonymity:
+ // the client refreshes it instead of running as nobody.
+ if (/^Bearer\s+\S/i.test(String(req.headers.authorization || '')) && c.kind !== 'principal' && c.kind !== 'user') {
+ throw new Refusal(401, 'token.invalid', 'The Bearer token was not accepted (expired, not for openvibe.tools, or not signed by the Network).');
+ }
+ if (c.kind === 'principal') {
+ if (!guard.tokens.hasCapability(c.claims, cap, contracts)) {
+ guard.record(req, c, { reason: 'capability', tool: d.id, enforced: true });
+ throw new Refusal(403, 'capability.denied', `${cap} not granted to this token`);
+ }
+ return c;
+ }
+ if (probe) {
+ guard.record(req, c, { reason: 'capability', tool: d.id, enforced: true });
+ if (c.kind === 'user') throw new Refusal(403, 'capability.denied', `Network probes run through the API only for apps and services holding ${cap}; people use the tool's page.`);
+ throw new Refusal(401, 'token.missing', `${d.id} runs through the API only with a token holding ${cap}; people use the tool's page.`);
+ }
+ if (d.auth && d.auth.anonymous === false && c.kind === 'anonymous') {
+ guard.record(req, c, { reason: 'session', tool: d.id, enforced: true });
+ throw new Refusal(401, 'tools.session_required', `${d.id} needs a browser session (open its page first), a sign-in or an API token.`);
+ }
+ return c;
+ }
+
+ function answerRefusal(req, res, err) {
+ if (res.headersSent) return undefined;
+ return problem(req, res, err.status, err.code, err.detail, err.extra, err.headers);
+ }
+
+ // ── Inline tools ─────────────────────────────────────────
+ async function runInline(req, res, d, c, fields, t0) {
+ counts.inline++;
+ const { input, key } = fields;
+ const reqHash = sha(canonical({ tool: d.id, input }));
+ const replayKey = key ? `${c.key}|${key}` : null;
+ if (replayKey) {
+ const prior = replays.get(replayKey);
+ if (prior) {
+ if (prior.hash !== reqHash) throw new Refusal(409, 'tools.job.idempotency_conflict', 'This Idempotency-Key was used for a different request');
+ counts.replayed++;
+ return send(res, 200, prior.body, { 'Idempotent-Replayed': 'true' });
+ }
+ }
+ const ttl = o.cacheTtl ? o.cacheTtl(d) : 0;
+ const cacheKey = ttl > 0 ? `${d.id}|${reqHash}` : null;
+ let result = cacheKey ? cache.get(cacheKey) : undefined;
+ let body;
+ if (result !== undefined) {
+ counts.cached++;
+ res.setHeader('X-OV-Cache', 'hit');
+ body = { state: 'succeeded', tool: d.id, result, took_ms: Date.now() - t0 };
+ } else {
+ // A tool that reaches a host the caller chose: that host's minute is shared by everyone (hard).
+ const target = o.targetOf ? o.targetOf(d, input) : null;
+ if (target && d.limits && d.limits.perTargetPerMinute && !guard.target(req, res, { tool: d.id, target })) return undefined;
+ const ac = new AbortController();
+ const onClose = () => { if (!res.writableFinished) ac.abort(); };
+ res.on('close', onClose);
+ const timeoutMs = (d.limits && d.limits.timeoutMs) || 15000;
+ let timer;
+ try {
+ const out = await Promise.race([
+ o.inline(d, input, { req, res, caller: c, signal: ac.signal, timeoutMs }),
+ new Promise((_, reject) => { timer = setTimeout(() => { ac.abort(); reject(Object.assign(new Error(`The tool ran longer than ${Math.round(timeoutMs / 1000)} s`), { status: 504, code: 'tools.run.timeout' })); }, timeoutMs + 250); }),
+ ]);
+ result = out && out.text !== undefined ? { text: String(out.text) } : { data: out && out.data && typeof out.data === 'object' ? out.data : {} };
+ if (cacheKey) cache.set(cacheKey, result, ttl);
+ body = { state: 'succeeded', tool: d.id, result, took_ms: Date.now() - t0 };
+ } catch (err) {
+ if (err && err.refusal) throw err;
+ if (res.headersSent) return undefined; // a guard refusal inside the tool already answered
+ counts.failed++;
+ body = { state: 'failed', tool: d.id, error: toolProblem(req, err), took_ms: Date.now() - t0 };
+ } finally {
+ clearTimeout(timer);
+ res.removeListener('close', onClose);
+ }
+ }
+ if (replayKey) replays.set(replayKey, { hash: reqHash, body }, REPLAY_TTL_MS);
+ return send(res, 200, body);
+ }
+
+ // ── Job tools (a satellite's own job system) ─────────────
+ const uploadsOf = (req) => (Array.isArray(req.files) ? req.files : req.files && typeof req.files === 'object' ? Object.values(req.files).flat() : []);
+
+ async function tempCopy(from, name, mime, dir) {
+ const to = path.join(dir, `run-${crypto.randomBytes(12).toString('hex')}${path.extname(String(name || '')).slice(0, 9)}`);
+ await fsp.copyFile(from, to);
+ return { path: to, name, mime, size: (await fsp.stat(to)).size };
+ }
+
+ /** A file reference → a temporary input file the job will own, or throws 404 tools.run.file_not_found. */
+ async function resolveRef(req, ref, owner, dir) {
+ const system = o.jobs.system();
+ const notFound = () => new Refusal(404, 'tools.run.file_not_found', `${ref.media_id || `${ref.job_id} file ${ref.index}`} is not a file you can read here: a result of one of your own Tools jobs that has not expired.`);
+ let row = null, index = ref.index;
+ if (ref.job_id) {
+ row = system.get(ref.job_id);
+ if (!row && o.jobs.fetchSibling) {
+ const got = await o.jobs.fetchSibling(ref, req, dir);
+ if (got) return got;
+ throw notFound();
+ }
+ } else {
+ const hit = system.findResultMedia(owner, ref.media_id);
+ if (hit) { row = hit.row; index = hit.index; }
+ }
+ if (!row || row.owner !== owner || row.state !== 'succeeded') throw notFound();
+ const f = system.resultFile(row, index);
+ if (!f) throw notFound();
+ if (f.path && fs.existsSync(f.path)) return tempCopy(f.path, f.name, f.mime, dir);
+ if (f.storage === 'media' && f.media && system.media) {
+ const signed = await system.media.downloadUrl(f.media.media_id);
+ const up = await fetch(signed.internal_url, { signal: AbortSignal.timeout(120_000) });
+ if (!up.ok) throw notFound();
+ const to = path.join(dir, `run-${crypto.randomBytes(12).toString('hex')}`);
+ await fsp.writeFile(to, Buffer.from(await up.arrayBuffer()));
+ return { path: to, name: f.name, mime: f.mime, size: (await fsp.stat(to)).size };
+ }
+ throw notFound();
+ }
+
+ function waitFor(system, id, ms, res) {
+ return new Promise((resolve) => {
+ const row = system.get(id);
+ if (!row || TERMINAL.has(row.state) || ms <= 0) return resolve();
+ let done = false;
+ const finish = () => { if (done) return; done = true; clearTimeout(timer); off(); res.removeListener('close', finish); resolve(); };
+ const off = system.subscribe(id, (e) => { if (TERMINAL.has(e.data.state)) finish(); });
+ const timer = setTimeout(finish, ms);
+ res.on('close', finish);
+ // It may have finished between the first read and the subscription.
+ const again = system.get(id);
+ if (!again || TERMINAL.has(again.state)) finish();
+ });
+ }
+
+ function runView(system, d, row) {
+ const job = system.view(row);
+ const took = row.finished_at ? Math.max(0, row.finished_at - row.created_at) : 0;
+ if (row.state === 'succeeded') return { state: 'succeeded', tool: d.id, result: { data: (job.result && job.result.data) || {}, files: (job.result && job.result.files) || [] }, took_ms: took, job };
+ if (row.state === 'failed' || row.state === 'cancelled') return { state: row.state, tool: d.id, error: job.error || contracts.http.problem(500, 'tools.job.failed', { detail: 'The job failed' }), took_ms: took, job };
+ return { state: row.state, tool: d.id, job, location: job.links.self };
+ }
+
+ async function runJob(req, res, d, c, fields, uploads) {
+ counts.jobs++;
+ const system = o.jobs.system();
+ const { input, refs, wait, key } = fields;
+ const dir = o.jobs.uploadsDir;
+ const files = uploads.map(f => ({ path: f.path, buffer: f.path ? undefined : f.buffer, name: f.originalname, mime: f.mimetype, size: f.size, originalname: f.originalname, mimetype: f.mimetype }));
+ const cleanup = () => { for (const f of files) if (f.path) fs.unlink(f.path, () => {}); };
+ try {
+ // The owner: a person, a principal, or this browser's session (started now when there is none).
+ const who = guard.resolveCaller(req, res, { create: true });
+ const owner = who.owner;
+ if (!owner) throw new Refusal(401, 'tools.session_required', `${d.id} needs a browser session, a sign-in or an API token.`);
+ for (const ref of refs) files.push(await resolveRef(req, ref, owner, dir));
+ const spec = d.files;
+ if (!spec && files.length) throw new Refusal(400, 'tools.run.invalid', `${d.id} takes no files.`);
+ if (spec && (files.length < spec.min || files.length > spec.max)) {
+ throw new Refusal(400, 'tools.run.invalid', `${d.id} takes ${spec.min === spec.max ? spec.min : `${spec.min} to ${spec.max}`} file${spec.max === 1 ? '' : 's'}; this request has ${files.length}.`);
+ }
+ if (spec && spec.maxBytes) {
+ const big = files.find(f => (f.size || 0) > spec.maxBytes);
+ if (big) throw new Refusal(413, 'tools.file.too_large', `${big.name} is over ${Math.round(spec.maxBytes / 1024 / 1024)} MB, the limit for ${d.id}.`);
+ }
+ // The bytes against files.accept (hard; a misleading name or type is corrected).
+ if (spec && !guard.checkFiles(req, res, files, spec.accept, d.id)) return undefined;
+ // Sessions from one address share a bound on unfinished jobs; the store's queue and disk bounds.
+ const full = system.addressFull ? system.addressFull(owner, who.ipKey) : null;
+ if (full && guard.refuse(req, res, { status: 429, code: 'tools.job.too_many_active', reason: 'jobs.address', tool: d.id, retryAfter: 30, detail: `At most ${full.limit} unfinished jobs from one address at a time; wait for one to finish.`, extra: { scope: 'address' } })) return undefined;
+ const busy = system.busy ? system.busy() : null;
+ if (busy && !guard.jobsBusy(req, res, busy, d.id)) return undefined;
+ let row, replayed;
+ try {
+ const out = await system.submit({
+ owner, type: d.run.job.type, input: contracts.tools.jobInput(d, input),
+ files: files.map(f => ({ path: f.path, buffer: f.buffer, name: f.originalname || f.name, mime: f.mimetype || f.mime, size: f.size })),
+ idempotencyKey: key, ttlMs: who.kind === 'session' ? 60 * 60 * 1000 : 24 * 60 * 60 * 1000,
+ env: who.env || 'production', ipKey: who.ipKey, tool: d.id,
+ });
+ row = out.job; replayed = out.replayed;
+ } catch (err) {
+ if (err instanceof system.JobError) throw new Refusal(err.status, err.code, err.detail, err.extra);
+ throw err;
+ }
+ files.length = 0; // the job owns them now
+ if (wait > 0) await waitFor(system, row.id, wait, res);
+ const fresh = system.get(row.id) || row;
+ const body = runView(system, d, fresh);
+ const headers = replayed ? { 'Idempotent-Replayed': 'true' } : {};
+ if (body.location) headers.Location = body.location;
+ return send(res, replayed || !body.location ? 200 : 202, body, headers);
+ } finally {
+ cleanup();
+ }
+ }
+
+ // ── The route ────────────────────────────────────────────
+ async function handle(req, res, next) {
+ const m = RUN_RE.exec(req.path);
+ if (!m) return next();
+ if (req.method === 'OPTIONS') return next(); // the app's CORS answers preflights
+ if (req.method !== 'POST') { res.setHeader('Allow', 'POST'); return problem(req, res, 405, 'method_not_allowed', 'POST a tools.run-request@1 to run a tool.'); }
+ const t0 = Date.now();
+ counts.runs++;
+ const snap = o.snapshot();
+ const id = m[1];
+ const d = snap.tools.find(t => t.id === id);
+ if (!d) return problem(req, res, 404, 'tools.tool.not_found', (snap.gone && snap.gone.get(id)) || `No tool is called "${id}". GET /api/v1/tools lists them all.`);
+ if (d.status === 'unavailable') return problem(req, res, 503, 'tools.tool.unavailable', d.statusReason || `${d.id} is not available right now.`);
+ if (!d.api || !d.run) return problem(req, res, 404, 'tools.tool.not_runnable', `${d.id} has no run API; use its page${d.hosts && d.hosts[0] ? ` (https://${d.hosts[0]})` : ''}.`);
+ if (o.proxy && o.proxy(d, req, res)) { counts.proxied++; return undefined; }
+ const mode = o.runs(d);
+ if (!mode) return problem(req, res, 404, 'tools.tool.not_runnable', `${d.id} is not run here; POST it to https://openvibe.tools/api/v1/tools/${d.id}/run.`);
+ try {
+ // Cookies from another site's page (CSRF): refused in enforce mode, recorded in report mode.
+ if (!guard.originOk(req, res, { hosts: d.hosts, tool: d.id })) return undefined;
+ const c = authorize(req, res, d);
+ // The tool's quota class and cost (recorded for the job admission, so it is not charged twice).
+ req._ovCharged = { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id };
+ if (!(await guard.charge(req, res, { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id }))) return undefined;
+ // A satellite's older per-route limiters count job runs like its job submits (report mode: in force).
+ if (mode === 'job' && o.jobs.limiters && !(await passes(o.jobs.limiters, req, res))) return undefined;
+ const multipart = req.is && req.is('multipart/form-data');
+ if (multipart && mode !== 'job') throw new Refusal(400, 'tools.run.invalid', `${d.id} takes no files: send JSON (tools.run-request@1).`);
+ if (!multipart && o.parseJson && req.body === undefined) {
+ const bad = await new Promise((resolve) => o.parseJson(req, res, (err) => resolve(err || null)));
+ if (bad) throw new Refusal(bad.status === 413 ? 413 : 400, bad.status === 413 ? 'tools.input.too_large' : 'tools.run.invalid', bad.status === 413 ? 'The request body is larger than 1 MB.' : 'The body is not valid JSON.');
+ }
+ let uploads = [];
+ if (multipart) {
+ const err = await new Promise((resolve) => o.jobs.receive(d)(req, res, (e) => resolve(e || null)));
+ uploads = uploadsOf(req);
+ if (err) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw err.refusal ? err : new Refusal(400, 'tools.run.invalid', err.message); }
+ }
+ let fields;
+ try { fields = readRequest(req, multipart); } catch (err) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw err; }
+ const bytes = Buffer.byteLength(JSON.stringify(fields.input));
+ const limit = d.limits && d.limits.maxInputBytes;
+ if (limit && bytes > limit) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw new Refusal(413, 'tools.input.too_large', `input is ${bytes} bytes as JSON; ${d.id} takes at most ${limit}.`); }
+ const errors = o.validate(d, fields.input);
+ if (errors.length) { for (const f of uploads) if (f.path) fs.unlink(f.path, () => {}); throw new Refusal(422, 'tools.input.invalid', `The input does not match ${d.id}'s input schema (GET /api/v1/tools/${d.id}/schema).`, { errors }); }
+ if (mode === 'inline') return await runInline(req, res, d, c, fields, t0);
+ return await runJob(req, res, d, c, fields, uploads);
+ } catch (err) {
+ if (err && err.refusal) return answerRefusal(req, res, err);
+ log.error(`[Run] ${id}:`, err && err.message);
+ if (!res.headersSent) return problem(req, res, 500, 'tools.run.failed', 'The run could not be started.');
+ return undefined;
+ }
+ }
+
+ return {
+ handle: (req, res, next) => { handle(req, res, next).catch(next); },
+ stats: () => ({ ...counts, cache: cache.stats(), replays: replays.stats() }),
+ cache, replays,
+ };
+}
+
+/**
+ * A satellite's upload receiver for runs: `file` and `files` file parts on disk (never in memory), the
+ * descriptor's file count and size, any declared type (the guard sniffs the bytes). The text parts
+ * (input, files, wait_ms, idempotency_key) land in req.body.
+ * @param {Function} multer the app's multer
+ * @param {string} dir where uploads wait (the app's uploads directory)
+ */
+function createRunUploads(multer, dir) {
+ fs.mkdirSync(dir, { recursive: true });
+ const storage = multer.diskStorage({
+ destination: (_req, _file, cb) => cb(null, dir),
+ filename: (_req, file, cb) => {
+ const ext = (path.extname(file.originalname || '').toLowerCase().match(/^\.[a-z0-9]{1,8}$/) || ['.bin'])[0];
+ cb(null, `run-${crypto.randomBytes(16).toString('hex')}${ext}`);
+ },
+ });
+ const byTool = new Map();
+ return function receive(d) {
+ const max = (d.files && d.files.max) || 1;
+ const maxBytes = (d.files && d.files.maxBytes) || 50 * 1024 * 1024;
+ const k = `${max}|${maxBytes}`;
+ if (!byTool.has(k)) {
+ byTool.set(k, multer({ storage, limits: { fileSize: maxBytes, files: max, fields: 8, fieldSize: 1024 * 1024 } })
+ .fields([{ name: 'file', maxCount: max }, { name: 'files', maxCount: max }]));
+ }
+ const mw = byTool.get(k);
+ return (req, res, next) => mw(req, res, (err) => {
+ if (!err) return next();
+ if (err.code === 'LIMIT_FILE_SIZE') return next(new Refusal(413, 'tools.file.too_large', `A file is over ${Math.round(maxBytes / 1024 / 1024)} MB, the limit for ${d.id}.`));
+ if (err.code === 'LIMIT_FILE_COUNT' || err.code === 'LIMIT_UNEXPECTED_FILE') return next(new Refusal(400, 'tools.run.invalid', `${d.id} takes at most ${max} file${max === 1 ? '' : 's'}, as "file" parts.`));
+ return next(new Refusal(400, 'tools.run.invalid', err.message));
+ });
+ };
+}
+
+/**
+ * A run's { job_id, index } reference to a job another satellite on this host holds: ask it which one
+ * (the facade's loopback probe), then read the file there with the caller's own credentials and
+ * address, so the owner check is that satellite's. → { path, name, mime, size } or null.
+ */
+function createSiblingFetch({ self, ports, fetchImpl = fetch }) {
+ const { JOB_SATELLITES } = require('./satellites');
+ return async function fetchSibling(ref, req, dir) {
+ const all = ports();
+ const others = JOB_SATELLITES.filter(s => s !== self && all[s]);
+ let port = null;
+ for (const sat of others) {
+ try {
+ const r = await fetchImpl(`http://127.0.0.1:${all[sat]}/api/internal/jobs/${ref.job_id}`, { signal: AbortSignal.timeout(2500) });
+ if (r.status === 200) { port = all[sat]; break; }
+ } catch { /* not that one */ }
+ }
+ if (!port) return null;
+ const headers = { 'x-forwarded-for': String(req.ip || ''), 'x-real-ip': String(req.ip || '') };
+ if (req.headers.authorization) headers.authorization = req.headers.authorization;
+ if (req.headers.cookie) headers.cookie = req.headers.cookie;
+ const r = await fetchImpl(`http://127.0.0.1:${port}/api/v1/jobs/${ref.job_id}/files/${ref.index}`, { headers, signal: AbortSignal.timeout(120_000) });
+ if (r.status !== 200 || !r.body) return null;
+ const cd = String(r.headers.get('content-disposition') || '');
+ const star = /filename\*=UTF-8''([^;]+)/i.exec(cd);
+ const plain = /filename="([^"]+)"/i.exec(cd);
+ let name = 'file';
+ try { name = star ? decodeURIComponent(star[1]) : plain ? plain[1] : 'file'; } catch { name = plain ? plain[1] : 'file'; }
+ const to = path.join(dir, `run-${crypto.randomBytes(12).toString('hex')}${path.extname(name).slice(0, 9)}`);
+ await fsp.writeFile(to, Buffer.from(await r.arrayBuffer()));
+ return { path: to, name, mime: String(r.headers.get('content-type') || 'application/octet-stream').split(';')[0], size: (await fsp.stat(to)).size };
+ };
+}
+
+/**
+ * The run API of a satellite that runs job tools (img, audio, docs): each of its job tools runs as a
+ * job of its own job system; file references to other satellites' jobs are read from them.
+ * @param {object} o
+ * @param {string} o.app 'img' | 'audio' | 'docs'
+ * @param {object} o.guard @param {object} o.contracts
+ * @param {() => object} o.snapshot the satellite's own registry
+ * @param {() => object} o.system its job system (set up after the routes)
+ * @param {Function} o.multer @param {string} o.uploadsDir
+ * @param {Function} o.Ajv @param {Function} [o.addFormats]
+ * @param {() => object} o.ports satellite ports (./satellites.js)
+ * @param {Function[]} [o.limiters] the app's older burst and processing limiters (as on its job submits)
+ */
+function satelliteRunApi(o) {
+ const uploadsDir = path.resolve(o.uploadsDir);
+ return createRunApi({
+ guard: o.guard, contracts: o.contracts, snapshot: o.snapshot, log: o.log,
+ validate: createInputValidator(o.Ajv, o.addFormats),
+ runs: (d) => (d.execution === 'job' && d.run && d.run.job ? 'job' : null),
+ jobs: {
+ system: o.system,
+ limiters: o.limiters || [],
+ receive: createRunUploads(o.multer, uploadsDir),
+ uploadsDir,
+ fetchSibling: createSiblingFetch({ self: o.app, ports: o.ports }),
+ },
+ });
+}
+
+/**
+ * Ajv (and ajv-formats) from the app's own openvibe-contracts, which depends on them (apps/_shared has
+ * no node_modules). appRequire: the app's `require`.
+ */
+function ajvFrom(appRequire) {
+ const base = path.dirname(appRequire.resolve('openvibe-contracts'));
+ const Ajv = appRequire(appRequire.resolve('ajv/dist/2020', { paths: [base] }));
+ let addFormats = null;
+ try { addFormats = appRequire(appRequire.resolve('ajv-formats', { paths: [base] })); } catch { /* formats are optional */ }
+ return { Ajv: Ajv.default || Ajv, addFormats: addFormats && (addFormats.default || addFormats) };
+}
+
+module.exports = { ajvFrom, createRunApi, satelliteRunApi, createRunUploads, createSiblingFetch, createInputValidator, createTtlStore, readRequest, isRunPath, Refusal, RUN_RE };
new file mode 100644
new file mode 100644
@@ -0,0 +1,30 @@
@@ -0,0 +1,30 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// Where the Tools apps listen on this host, for the loopback calls between them (the gateway's
+// readiness checks, registry polling, run API and jobs facade; a satellite reading another's job
+// result for a run's { job_id, index } reference).
+//
+// TOOLS_SATELLITE_PORTS="img=5012,yt=5013" overrides single entries (tests, a moved unit).
+// Job types are routed by their prefix: img.process → img, audio.process → audio, docs.process → docs.
+// ═══════════════════════════════════════════════════════════════
+
+const DEFAULT_PORTS = Object.freeze({ maps: 4010, food: 4011, img: 4012, yt: 4013, audio: 4014, text: 4015, docs: 4016 });
+const JOB_SATELLITES = Object.freeze(['img', 'audio', 'docs']);
+
+/** { maps: 4010, …, docs: 4016 } with the environment's overrides. */
+function satellitePorts(env = process.env) {
+ const out = { ...DEFAULT_PORTS };
+ for (const pair of String(env.TOOLS_SATELLITE_PORTS || '').split(',')) {
+ const [name, port] = pair.split('=').map(x => String(x || '').trim());
+ if (out[name] && /^\d{2,5}$/.test(port)) out[name] = Number(port);
+ }
+ return out;
+}
+
+/** The satellite that runs a job type ('img.process' → 'img'), or null. */
+function satelliteForType(type) {
+ const prefix = String(type || '').split('.')[0];
+ return JOB_SATELLITES.includes(prefix) ? prefix : null;
+}
+
+module.exports = { DEFAULT_PORTS, JOB_SATELLITES, satellitePorts, satelliteForType };
@@ -24,6 +24,9 @@ const { probe, getDuration, cleanTmp } = require('./tools/ffmpeg-helper');
@@ -24,6 +24,9 @@ const { probe, getDuration, cleanTmp } = require('./tools/ffmpeg-helper');
const { buildOptions, describe, defineJobs, runSync, jobContext, limitsFor } = require('./process');
const { buildOptions, describe, defineJobs, runSync, jobContext, limitsFor } = require('./process');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { satelliteRunApi, ajvFrom } = require('../../_shared/tools/run');
+const { satellitePorts } = require('../../_shared/tools/satellites');
+const { deprecated, runPath } = require('../../_shared/tools/deprecation');
const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const jobsRuntime = require('../../_shared/jobs');
const jobsRuntime = require('../../_shared/jobs');
const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
@@ -115,7 +118,7 @@ app.use(exceptRegistry(cors({
@@ -115,7 +118,7 @@ app.use(exceptRegistry(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-})));
+}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
app.use(guard.identify);
app.use(guard.identify);
@@ -177,7 +180,7 @@ app.get('/api/tools', (_req, res) => {
@@ -177,7 +180,7 @@ app.get('/api/tools', (_req, res) => {
});
});
// ── Probe / Metadata Endpoint ────────────────────────────────
// ── Probe / Metadata Endpoint ────────────────────────────────
-app.post('/api/probe', burstLimiter, processLimiter, guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {
+app.post('/api/probe', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {
try {
try {
// Hardened like every run: local files only, audio demuxers only.
// Hardened like every run: local files only, audio demuxers only.
const { info, meta } = await jobContext.run(limitsFor('metadata'), async () => ({ info: await probe(req.file.path), meta: await readMetadata(req.file.path) }));
const { info, meta } = await jobContext.run(limitsFor('metadata'), async () => ({ info: await probe(req.file.path), meta: await readMetadata(req.file.path) }));
@@ -190,7 +193,7 @@ app.post('/api/probe', burstLimiter, processLimiter, guard.toolQuota('metadata')
@@ -190,7 +193,7 @@ app.post('/api/probe', burstLimiter, processLimiter, guard.toolQuota('metadata')
});
});
// ── Main Processing Endpoint ─────────────────────────────────
// ── Main Processing Endpoint ─────────────────────────────────
-app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
+app.post('/api/process', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
try {
try {
const toolId = req.body.tool || req.ctx.defaultOp || 'convert';
const toolId = req.body.tool || req.ctx.defaultOp || 'convert';
const tool = getTool(toolId);
const tool = getTool(toolId);
@@ -231,7 +234,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)
@@ -231,7 +234,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)
});
});
// ── Multi-File Processing Endpoint (merge) ───────────────────
// ── Multi-File Processing Endpoint (merge) ───────────────────
-app.post('/api/process/multi', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
+app.post('/api/process/multi', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
const paths = req.files.map(f => f.path);
const paths = req.files.map(f => f.path);
try {
try {
const toolId = req.body.tool || req.ctx.defaultOp;
const toolId = req.body.tool || req.ctx.defaultOp;
@@ -274,6 +277,17 @@ const jobs = jobsRuntime.setupJobs({
@@ -274,6 +277,17 @@ const jobs = jobsRuntime.setupJobs({
},
},
});
});
+// ── Run API (ADR-027): POST /api/v1/tools/:id/run for this app's tools ──
+// tools.run-request@1 (JSON, or multipart with `file` parts and the text parts input, files, wait_ms)
+// → tools.run@1: each tool runs as a job of this satellite (the preset and operation from its
+// descriptor), answered finished within wait_ms or 202 with the job. The gateway streams runs here.
+const runApi = satelliteRunApi({
+ app: 'audio', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
+ multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
+ limiters: [burstLimiter, processLimiter],
+});
+app.use(runApi.handle);
+
// ── File Download ────────────────────────────────────────────
// ── File Download ────────────────────────────────────────────
app.get('/api/download/:id', (req, res) => {
app.get('/api/download/:id', (req, res) => {
const entry = retention.getFile(req.params.id);
const entry = retention.getFile(req.params.id);
@@ -22,7 +22,7 @@ async function waitFor(app, id, cookie, pred, what, ms = 30000) {
@@ -22,7 +22,7 @@ async function waitFor(app, id, cookie, pred, what, ms = 30000) {
const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();
const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();
if (pred(job)) return job;
if (pred(job)) return job;
if (Date.now() - t0 > ms) throw new Error(`timed out waiting for ${what}: ${JSON.stringify(job)}`);
if (Date.now() - t0 > ms) throw new Error(`timed out waiting for ${what}: ${JSON.stringify(job)}`);
- await sleep(50);
+ await sleep(150); // the older /api/ limiter allows an address 60 requests a minute
}
}
}
}
@@ -24,6 +24,9 @@ const { buildOptions, defineJobs, runTool } = require('./process');
@@ -24,6 +24,9 @@ const { buildOptions, defineJobs, runTool } = require('./process');
const { poolFromEnv } = require('../../_shared/jobs/pool');
const { poolFromEnv } = require('../../_shared/jobs/pool');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { satelliteRunApi, ajvFrom } = require('../../_shared/tools/run');
+const { satellitePorts } = require('../../_shared/tools/satellites');
+const { deprecated, runPath } = require('../../_shared/tools/deprecation');
const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const jobsRuntime = require('../../_shared/jobs');
const jobsRuntime = require('../../_shared/jobs');
const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
@@ -131,7 +134,7 @@ app.use(exceptRegistry(cors({
@@ -131,7 +134,7 @@ app.use(exceptRegistry(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-})));
+}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
app.use(guard.identify);
app.use(guard.identify);
@@ -206,7 +209,7 @@ app.get('/api/tools', (_req, res) => {
@@ -206,7 +209,7 @@ app.get('/api/tools', (_req, res) => {
});
});
// ── PDF Info (no file mutation) ──────────────────────────────
// ── PDF Info (no file mutation) ──────────────────────────────
-app.post('/api/info', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {
+app.post('/api/info', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {
try {
try {
const tool = getTool('metadata');
const tool = getTool('metadata');
const [buffer] = await readUploads([req.file]);
const [buffer] = await readUploads([req.file]);
@@ -218,7 +221,7 @@ app.post('/api/info', burstLimiter, processLimiter, guard.toolQuota(hostTool), u
@@ -218,7 +221,7 @@ app.post('/api/info', burstLimiter, processLimiter, guard.toolQuota(hostTool), u
});
});
// ── Main Processing Endpoint (single file) ───────────────────
// ── Main Processing Endpoint (single file) ───────────────────
-app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
+app.post('/api/process', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
try {
try {
const toolId = req.body.tool || req.ctx.defaultOp;
const toolId = req.body.tool || req.ctx.defaultOp;
if (!toolId) { fs.unlink(req.file.path, () => {}); return res.status(400).json({ error: 'No tool specified.' }); }
if (!toolId) { fs.unlink(req.file.path, () => {}); return res.status(400).json({ error: 'No tool specified.' }); }
@@ -265,7 +268,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)
@@ -265,7 +268,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)
});
});
// ── Multi-File Processing Endpoint (merge, img2pdf) ──────────
// ── Multi-File Processing Endpoint (merge, img2pdf) ──────────
-app.post('/api/process/multi', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
+app.post('/api/process/multi', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadMultiple, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
try {
try {
const toolId = req.body.tool || req.ctx.defaultOp;
const toolId = req.body.tool || req.ctx.defaultOp;
const tool = toolId ? getTool(toolId) : null;
const tool = toolId ? getTool(toolId) : null;
@@ -325,6 +328,17 @@ const jobs = jobsRuntime.setupJobs({
@@ -325,6 +328,17 @@ const jobs = jobsRuntime.setupJobs({
},
},
});
});
+// ── Run API (ADR-027): POST /api/v1/tools/:id/run for this app's tools ──
+// tools.run-request@1 (JSON, or multipart with `file` parts and the text parts input, files, wait_ms)
+// → tools.run@1: each tool runs as a job of this satellite (the preset and operation from its
+// descriptor), answered finished within wait_ms or 202 with the job. The gateway streams runs here.
+const runApi = satelliteRunApi({
+ app: 'docs', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
+ multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
+ limiters: [burstLimiter, processLimiter],
+});
+app.use(runApi.handle);
+
// ── File Download ────────────────────────────────────────────
// ── File Download ────────────────────────────────────────────
app.get('/api/download/:id', (req, res) => {
app.get('/api/download/:id', (req, res) => {
const entry = retention.getFile(req.params.id);
const entry = retention.getFile(req.params.id);
@@ -34,7 +34,9 @@ const upload = multer({
@@ -34,7 +34,9 @@ const upload = multer({
files: 50, // Allow up to 50 files for merge
files: 50, // Allow up to 50 files for merge
},
},
fileFilter(_req, file, cb) {
fileFilter(_req, file, cb) {
- if (ALLOWED_MIMES.has(file.mimetype)) {
+ // A part without a type of its own (application/octet-stream: API clients, openvibe-sdk) is let
+ // through: the guard checks the bytes of every upload against the tool's accepted types.
+ if (ALLOWED_MIMES.has(file.mimetype) || file.mimetype === 'application/octet-stream') {
cb(null, true);
cb(null, true);
} else {
} else {
cb(new Error(`Unsupported file type: ${file.mimetype}. Accepted: ${config.upload.allowedMimes.join(', ')}`));
cb(new Error(`Unsupported file type: ${file.mimetype}. Accepted: ${config.upload.allowedMimes.join(', ')}`));
@@ -21,7 +21,7 @@ async function bigPdf(pages) {
@@ -21,7 +21,7 @@ async function bigPdf(pages) {
}
}
async function finished(app, id, cookie) {
async function finished(app, id, cookie) {
- for (let i = 0; i < 300; i++) {
+ for (let i = 0; i < 1200; i++) { // up to two minutes on a busy machine
const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();
const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();
if (['succeeded', 'failed', 'cancelled'].includes(job.state)) return job;
if (['succeeded', 'failed', 'cancelled'].includes(job.state)) return job;
await sleep(100);
await sleep(100);
@@ -68,7 +68,9 @@ function mergeForm(files, job) {
@@ -68,7 +68,9 @@ function mergeForm(files, job) {
assert.ok(took > 700, `the merge is large enough to measure (${took} ms)`);
assert.ok(took > 700, `the merge is large enough to measure (${took} ms)`);
assert.ok(latencies.length >= 5, `health was asked during the merge (${latencies.length} times)`);
assert.ok(latencies.length >= 5, `health was asked during the merge (${latencies.length} times)`);
const worst = Math.max(...latencies);
const worst = Math.max(...latencies);
- assert.ok(worst < 250, `/api/health stayed fast during a ${took} ms merge (worst ${worst} ms)`);
+ // On the event loop the merge would hold every request for its whole length; in a worker thread
+ // health answers in milliseconds (the bound leaves room for a busy test machine).
+ assert.ok(worst < Math.max(250, took / 3), `/api/health stayed fast during a ${took} ms merge (worst ${worst} ms)`);
const health = await (await fetch(`${app.base}/api/health`)).json();
const health = await (await fetch(`${app.base}/api/health`)).json();
assert.strictEqual(health.workers.completed, 1);
assert.strictEqual(health.workers.completed, 1);
assert.strictEqual(health.workers.busy, 0);
assert.strictEqual(health.workers.busy, 0);
@@ -57,6 +57,25 @@ server {
@@ -57,6 +57,25 @@ server {
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
+ # ── Run API and jobs facade (ADR-027): uploads stream to the satellites, SSE streams back ──
+ # POST /api/v1/tools/:id/run takes files like the satellites do (docs: 100 MB each); wait_ms holds a
+ # run open up to 60 s; /api/v1/jobs/:id/events is a long-lived event stream.
+ location ~ ^/api/v1/(tools/[a-z0-9-]+/run|jobs)(/|$) {
+ limit_req zone=ovtools_api burst=30 nodelay;
+ client_max_body_size 512m;
+ proxy_request_buffering off;
+ proxy_buffering off;
+ proxy_read_timeout 600s;
+ proxy_send_timeout 600s;
+ proxy_pass http://127.0.0.1:4001;
+ proxy_http_version 1.1;
+ proxy_set_header Host $host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $remote_addr;
+ proxy_set_header CF-Connecting-IP $remote_addr;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ }
+
# ── API (moderate rate limit) ───────────────────────────
# ── API (moderate rate limit) ───────────────────────────
location /api/ {
location /api/ {
limit_req zone=ovtools_api burst=30 nodelay;
limit_req zone=ovtools_api burst=30 nodelay;
@@ -103,4 +103,12 @@ const SPECS = [
@@ -103,4 +103,12 @@ const SPECS = [
engine('md2html', obj({ text: text() }), TEXT, { text: '# Hi\n\n- one' }),
engine('md2html', obj({ text: text() }), TEXT, { text: '# Hi\n\n- one' }),
];
];
+// The run API reuses an engine's answer for 10 minutes (same tool, same input): not for these, whose
+// output is random or depends on the time. Open Graph (a page's tags) may be reused for 10 minutes.
+const FRESH = new Set(['jwt', 'uuid', 'timestamp', 'cron', 'lorem']);
+for (const spec of SPECS) {
+ if (FRESH.has(spec.id)) spec.cacheTtlMs = 0;
+ if (spec.id === 'opengraph') spec.cacheTtlMs = 10 * 60 * 1000;
+}
+
module.exports = { SPECS };
module.exports = { SPECS };
@@ -33,13 +33,9 @@ const app = express();
@@ -33,13 +33,9 @@ const app = express();
// What this deploy runs (ADR-016); the shared navbar's release-watch polls it on every tool host.
// What this deploy runs (ADR-016); the shared navbar's release-watch polls it on every tool host.
const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });
const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });
-// The satellites on this host (ports), for readiness and the analytics roll-up.
-// TOOLS_SATELLITE_PORTS="img=5012,yt=5013" overrides single entries (tests, a moved unit).
-const SATELLITES = { maps: 4010, food: 4011, img: 4012, yt: 4013, audio: 4014, text: 4015, docs: 4016 };
-for (const pair of String(process.env.TOOLS_SATELLITE_PORTS || '').split(',')) {
- const [name, port] = pair.split('=').map(x => String(x || '').trim());
- if (SATELLITES[name] && /^\d{2,5}$/.test(port)) SATELLITES[name] = Number(port);
-}
+// The satellites on this host (ports), for readiness, the analytics roll-up, the run API and the
+// jobs facade. TOOLS_SATELLITE_PORTS="img=5012,yt=5013" overrides single entries (tests, a moved unit).
+const SATELLITES = require('../../_shared/tools/satellites').satellitePorts();
const SATELLITE_BY_PORT = new Map(Object.entries(SATELLITES).map(([name, port]) => [port, name]));
const SATELLITE_BY_PORT = new Map(Object.entries(SATELLITES).map(([name, port]) => [port, name]));
// Metrics (GET /metrics, direct loopback callers only) and readiness (roadmap Track O). The metrics
// Metrics (GET /metrics, direct loopback callers only) and readiness (roadmap Track O). The metrics
@@ -132,8 +128,15 @@ app.use(hostRoles({
@@ -132,8 +128,15 @@ app.use(hostRoles({
app.get('/api/ready', obs.readiness.handler);
app.get('/api/ready', obs.readiness.handler);
app.use(cookieParser());
app.use(cookieParser());
-app.use(express.json({ limit: '1mb' }));
-app.use(express.urlencoded({ extended: true }));
+// The run API and the jobs facade read their own bodies: uploads and SSE stream through to the
+// satellites untouched, and a local run parses its JSON itself (run/index.js).
+const { isRunPath } = require('../../_shared/tools/run');
+const { isJobsPath } = require('./run/jobs-facade');
+const jsonBody = express.json({ limit: '1mb' });
+const formBody = express.urlencoded({ extended: true });
+const streamed = (req) => isRunPath(req.path) || isJobsPath(req.path);
+app.use((req, res, next) => (streamed(req) ? next() : jsonBody(req, res, next)));
+app.use((req, res, next) => (streamed(req) ? next() : formBody(req, res, next)));
// ── CORS ─────────────────────────────────────────────────────
// ── CORS ─────────────────────────────────────────────────────
// Every *.openvibe.tools subdomain (satellites included) may call the
// Every *.openvibe.tools subdomain (satellites included) may call the
@@ -162,6 +165,8 @@ function isAllowedOrigin(origin) {
@@ -162,6 +165,8 @@ function isAllowedOrigin(origin) {
app.use('/auth/fedcm', fedcmCors);
app.use('/auth/fedcm', fedcmCors);
// The tool registry (/api/v1/tools…) is public and read-only: open to every origin, no credentials.
// The tool registry (/api/v1/tools…) is public and read-only: open to every origin, no credentials.
+// The run API and the jobs facade take any site's page with a token: CORS without credentials for an
+// origin that is not ours (so no cookie rides along), the credentialed rules below for ours.
app.use(exceptRegistry(cors({
app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true); // curl / server-to-server
if (!origin) return callback(null, true); // curl / server-to-server
@@ -169,7 +174,8 @@ app.use(exceptRegistry(cors({
@@ -169,7 +174,8 @@ app.use(exceptRegistry(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-})));
+ exposedHeaders: ['Location', 'Idempotent-Replayed', 'Retry-After', 'RateLimit-Limit', 'RateLimit-Remaining', 'RateLimit-Reset', 'X-OpenVibe-Request-Id', 'Deprecation', 'Sunset', 'Link'],
+}), { isFirstParty: isAllowedOrigin }));
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
app.use(guard.identify);
app.use(guard.identify);
@@ -182,6 +188,49 @@ app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, keyGenerator: (req
@@ -182,6 +188,49 @@ app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, keyGenerator: (req
// the /api/ limiter above. /api/catalog.json stays as it is (Network reads it).
// the /api/ limiter above. /api/catalog.json stays as it is (Network reads it).
app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+// ── Net.OpenVibe and Dev.OpenVibe routes (the pages' API; the run API calls the same handlers) ──
+const netRouter = createNetRoutes(null, requireAuth, { guard });
+const devRouter = createDevRoutes(null, requireAuth, { guard });
+
+// ── Run API and jobs facade (ADR-027) ────────────────────────
+// POST /api/v1/tools/:id/run (tools.run-request@1 → tools.run@1) for every tool with an API: net and
+// Open Graph through their routes, dev and text engines in a worker pool, img/audio/docs streamed to
+// their satellite. /api/v1/jobs… fronts every satellite's job routes (routed by type, then by job id).
+const { createGatewayRun } = require('./run');
+const { createJobsFacade, createJobIndex } = require('./run/jobs-facade');
+const { ajvFrom } = require('../../_shared/tools/run');
+const jobIndex = createJobIndex();
+const runApi = createGatewayRun({
+ guard, contracts: require('openvibe-contracts'), toolRegistry, routers: { net: netRouter, dev: devRouter },
+ ports: () => SATELLITES, parseJson: jsonBody, jobIndex, ...ajvFrom(require),
+});
+const jobsFacade = createJobsFacade({ ports: () => SATELLITES, index: jobIndex, contracts: require('openvibe-contracts') });
+app.use(runApi.handle);
+app.use((req, res, next) => { jobsFacade.handle(req, res, next); });
+
+// ── The older tool endpoints: still answered, and they say what replaces them ──
+const { deprecated, runPath } = require('../../_shared/tools/deprecation');
+const NET_BY_ENDPOINT = new Map();
+for (const sp of require('./net/descriptors').SPECS) {
+ if (!sp.route || !sp.api) continue;
+ const ep = sp.route.path.replace(/^\/api\/net/, '');
+ (NET_BY_ENDPOINT.get(ep) || NET_BY_ENDPOINT.set(ep, []).get(ep)).push(sp.id);
+}
+/** /api/net/<endpoint>… → the run route of the tool it serves here (the host's own, else the one named after it). */
+function netSuccessor(req) {
+ const ep = `/${String(req.path || '').split('/')[1] || ''}`;
+ if (ep === '/tools') return '/api/v1/tools?family=net';
+ const ids = NET_BY_ENDPOINT.get(ep);
+ if (!ids) return null;
+ const host = req.ovHost && req.ovHost.tool;
+ return runPath(ids.find(i => i === host) || ids.find(i => i === ep.slice(1)) || ids[0]);
+}
+function devSuccessor(req) {
+ if (req.path === '/opengraph') return runPath('opengraph');
+ if (req.path === '/tools') return '/api/v1/tools?family=dev';
+ return null; // webhook bins: page-only (api false), no successor
+}
+
// ── Auth (OAuth2 client of OpenVibe.Network) ─────────────────
// ── Auth (OAuth2 client of OpenVibe.Network) ─────────────────
auth = createAuthClient(config);
auth = createAuthClient(config);
app.locals.auth = auth;
app.locals.auth = auth;
@@ -202,7 +251,7 @@ async function requireAuth(req, res, next) {
@@ -202,7 +251,7 @@ async function requireAuth(req, res, next) {
// ── Basic API ────────────────────────────────────────────────
// ── Basic API ────────────────────────────────────────────────
app.get('/api/health', (_req, res) => {
app.get('/api/health', (_req, res) => {
- res.json({ status: 'ok', service: 'openvibe-tools-gateway', version: '2.0.0', registry: registry.services.status() });
+ res.json({ status: 'ok', service: 'openvibe-tools-gateway', version: '2.0.0', registry: registry.services.status(), run: runApi.stats(), jobs: jobsFacade.stats() });
});
});
app.get('/api/brand', (_req, res) => res.json(BRAND));
app.get('/api/brand', (_req, res) => res.json(BRAND));
@@ -239,10 +288,13 @@ app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonym
@@ -239,10 +288,13 @@ app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonym
});
});
// ── Net.OpenVibe — Network Tools API ─────────────────────────
// ── Net.OpenVibe — Network Tools API ─────────────────────────
-app.use('/api/net', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), createNetRoutes(null, requireAuth, { guard }));
+app.use('/api/net', deprecated(netSuccessor), guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), netRouter);
// ── Dev.OpenVibe — Developer & SEO Tools API ─────────────────
// ── Dev.OpenVibe — Developer & SEO Tools API ─────────────────
-app.use('/api/dev', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), createDevRoutes(null, requireAuth, { guard }));
+// Webhook bins are created and deleted with the browser's session cookie: those writes must come from
+// our pages (the /in URL a bin receives on is anyone's to call).
+const devOrigin = guard.originCheck();
+app.use('/api/dev', deprecated(devSuccessor), (req, res, next) => (/^\/webhook\/bins\/[^/]+\/in(\/|$)/.test(req.path) ? next() : devOrigin(req, res, next)), guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), devRouter);
// ── Host-header subdomain routing ────────────────────────────
// ── Host-header subdomain routing ────────────────────────────
function subdomainOf(req) {
function subdomainOf(req) {
@@ -27,6 +27,16 @@ const TLS_PORTS = [443, 465, 636, 853, 990, 993, 995, 2083, 2087, 5061, 8443];
@@ -27,6 +27,16 @@ const TLS_PORTS = [443, 465, 636, 853, 990, 993, 995, 2083, 2087, 5061, 8443];
const PROBES = new Set(['port', 'ping', 'latency', 'traceroute', 'mtr']);
const PROBES = new Set(['port', 'ping', 'latency', 'traceroute', 'mtr']);
const KiB = 1024;
const KiB = 1024;
+const MIN = 60 * 1000;
+// How long the run API may reuse a lookup's answer (same tool, same input). Records and registration
+// data move slowly (and the routes cache their upstreams anyway); live checks (ping, ports, headers,
+// redirects, uptime, SMTP) and myip (the caller's own address) are never reused.
+const CACHE_TTL = {
+ dns: MIN, dig: MIN, nslookup: MIN, mx: MIN, txt: MIN, ns: MIN, spf: MIN, dkim: MIN, dmarc: MIN, dnspropagation: MIN,
+ ip: 10 * MIN, geoip: 10 * MIN, isp: 10 * MIN, asn: 10 * MIN, ipv4: 10 * MIN, ipv6: 10 * MIN,
+ hostname: 5 * MIN, rdns: 5 * MIN, whois: 60 * MIN, rdap: 60 * MIN,
+ blacklist: 5 * MIN, ssl: 5 * MIN, robots: 5 * MIN, sitemap: 5 * MIN, lookup: 5 * MIN,
+};
const target = (description) => ({ type: 'string', minLength: 1, maxLength: 253 + 2048, description });
const target = (description) => ({ type: 'string', minLength: 1, maxLength: 253 + 2048, description });
const HOST = 'A domain or an IP address';
const HOST = 'A domain or an IP address';
const URL_OR_DOMAIN = 'A URL, or a domain (https:// is assumed)';
const URL_OR_DOMAIN = 'A URL, or a domain (https:// is assumed)';
@@ -130,6 +140,7 @@ function spec(id) {
@@ -130,6 +140,7 @@ function spec(id) {
route: { method: 'GET', path: `/api/net${endpoint}`, ...(t.query && { query: t.query }), ...(t.targetTemplate && { target: t.targetTemplate }) },
route: { method: 'GET', path: `/api/net${endpoint}`, ...(t.query && { query: t.query }), ...(t.targetTemplate && { target: t.targetTemplate }) },
}),
}),
example: { input: endpoint === '/myip' ? {} : { target: 'example.com' } },
example: { input: endpoint === '/myip' ? {} : { target: 'example.com' } },
+ cacheTtlMs: CACHE_TTL[id] || 0,
};
};
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,15 @@
@@ -0,0 +1,15 @@
+'use strict';
+// A worker thread of the gateway's engine pool (apps/_shared/jobs/pool.js): runs one browser tool's
+// server engine (apps/gateway/server/dev/engines.js, apps/text/server/engines.js) on a run's input,
+// off the event loop and bounded (the descriptor's timeoutMs terminates it; a heap limit applies).
+const loaded = new Map();
+
+module.exports = {
+ /** { module, name, input } → { data } | { text } */
+ async run({ module, name, input }) {
+ if (!loaded.has(module)) loaded.set(module, require(module));
+ const fn = (loaded.get(module).ENGINES || {})[name];
+ if (typeof fn !== 'function') throw Object.assign(new Error(`No server engine ${name}`), { status: 500 });
+ return fn(input || {});
+ },
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,188 @@
@@ -0,0 +1,188 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// The gateway's run API: POST /api/v1/tools/:id/run for every tool (ADR-027; the protocol is
+// apps/_shared/tools/run.js). Where each tool runs:
+//
+// engines dev and text tools with a server engine (execution client, api true): in the gateway's
+// engine pool (worker threads: the descriptor's timeoutMs terminates a runaway input, a heap
+// limit applies), the same code the pages run. Cached for 10 minutes unless the spec says
+// cacheTtlMs: 0 (random or time-dependent output).
+// routes the net tools and Open Graph (execution sync): the /api/net and /api/dev route that already
+// answers the page, called in this process with the run's input as its query (spec.route),
+// so both give the same answer. The per-target throttle applies; answers are cached only
+// where the spec gives a cacheTtlMs (slow-moving lookups, never probes or myip).
+// jobs img, audio and docs tools: the run is streamed to the satellite that owns the tool
+// (POST /api/v1/tools/:id/run there, uploads and all) and its answer streamed back; the job
+// id is remembered for the /api/v1/jobs facade.
+// ═══════════════════════════════════════════════════════════════
+
+const path = require('path');
+const { createRunApi, createInputValidator, Refusal } = require('../../../_shared/tools/run');
+const { forward } = require('../../../_shared/tools/proxy');
+const { poolFromEnv } = require('../../../_shared/jobs/pool');
+
+const ENGINE_TTL_MS = 10 * 60 * 1000;
+
+/** A response stand-in that captures what an Express route handler answers. */
+function capture() {
+ let settle;
+ const done = new Promise((r) => { settle = r; });
+ const headers = {};
+ const res = {
+ statusCode: 200, headersSent: false, locals: {}, writableFinished: false,
+ status(n) { this.statusCode = n; return this; },
+ set(k, v) { if (k && typeof k === 'object') for (const [a, b] of Object.entries(k)) headers[a.toLowerCase()] = b; else headers[String(k).toLowerCase()] = v; return this; },
+ header(k, v) { return this.set(k, v); },
+ setHeader(k, v) { headers[String(k).toLowerCase()] = v; },
+ getHeader(k) { return headers[String(k).toLowerCase()]; },
+ removeHeader(k) { delete headers[String(k).toLowerCase()]; },
+ cookie() { return this; },
+ on() { return this; }, once() { return this; }, removeListener() { return this; },
+ json(body) { this.headersSent = true; settle({ status: this.statusCode, headers, body }); return this; },
+ send(body) { let b = body; if (typeof b === 'string' || Buffer.isBuffer(b)) { try { b = JSON.parse(String(b)); } catch { /* text */ } } return this.json(b); },
+ end(body) { let b = body; try { b = body == null ? null : JSON.parse(String(body)); } catch { b = String(body); } this.headersSent = true; settle({ status: this.statusCode, headers, body: b }); },
+ };
+ return { res, done, settle };
+}
+
+/** The route handler a router serves for `method path` (path relative to the router, e.g. /dns). */
+function routeHandler(router, method, rel) {
+ for (const layer of router.stack) {
+ if (!layer.route || !layer.route.methods[method.toLowerCase()]) continue;
+ const p = layer.route.path;
+ if (p === rel || p === `${rel}/:target?`) return layer.route.stack[layer.route.stack.length - 1].handle;
+ }
+ return null;
+}
+
+/** '{target}' templates from the input, falling back to the input schema's defaults. */
+function fill(template, input, schema) {
+ return String(template).replace(/\{([a-z_]+)\}/gi, (_, k) => {
+ const v = input[k] !== undefined ? input[k] : schema && schema.properties && schema.properties[k] && schema.properties[k].default;
+ return v === undefined || v === null ? '' : String(v);
+ });
+}
+
+/** What a legacy route answered → the run's data, or a thrown tool error / refusal. */
+function fromRoute(out) {
+ if (out.error) throw out.error;
+ const body = out.body && typeof out.body === 'object' ? out.body : {};
+ if (/problem\+json/.test(String(out.headers['content-type'] || ''))) {
+ // A guard refusal inside the route (the port checker's caps): the answer to the request itself.
+ const { type, title, status, code, detail, error, request_id, trace_id, ...extra } = body;
+ const headers = out.headers['retry-after'] ? { 'Retry-After': String(out.headers['retry-after']) } : undefined;
+ throw new Refusal(out.status, code || 'tools.run.invalid', detail || error, extra, headers);
+ }
+ if (out.status < 400 && body.ok !== false) {
+ const { ok, ...data } = body;
+ return { data };
+ }
+ const own = /^tools\.[a-z0-9_.]+$/.test(String(body.code || '')) ? body.code : null;
+ const status = out.status === 400 ? 422 : out.status || 500;
+ const code = own || (out.status === 400 ? 'tools.input.invalid' : 'tools.job.failed');
+ throw Object.assign(new Error(body.error || 'The tool failed'), { status, code });
+}
+
+/**
+ * @param {object} o
+ * @param {object} o.guard @param {object} o.contracts
+ * @param {object} o.toolRegistry ../registry/descriptors (snapshot, internal)
+ * @param {object} o.routers { net: express.Router, dev: express.Router }
+ * @param {() => object} o.ports satellite ports
+ * @param {Function} o.parseJson express.json() for local runs
+ * @param {object} o.jobIndex job id → satellite (the facade's)
+ * @param {object} [o.Ajv] [o.addFormats]
+ */
+function createGatewayRun(o) {
+ const internal = (id) => o.toolRegistry.internal(id);
+ const pool = o.pool || poolFromEnv('gateway', path.join(__dirname, 'engine-worker.js'), { size: 2, memoryMb: 256 });
+ const handlers = new Map();
+
+ function kind(d) {
+ const s = internal(d.id);
+ if (!s) return null;
+ if (s.engines && s.spec.engine) return 'engine';
+ if ((s.app === 'net' || s.app === 'dev') && s.spec.route) return 'route';
+ if (s.satellite && d.execution === 'job') return 'satellite';
+ return null;
+ }
+
+ function handlerFor(d, s) {
+ if (handlers.has(d.id)) return handlers.get(d.id);
+ const router = o.routers[s.app];
+ const base = s.app === 'net' ? '/api/net' : '/api/dev';
+ const rel = s.spec.route.path.slice(base.length);
+ const h = router ? routeHandler(router, s.spec.route.method, rel) : null;
+ handlers.set(d.id, h);
+ return h;
+ }
+
+ /** Call the tool's route in this process with the run's input as its request. */
+ async function viaRoute(d, s, input, ctx) {
+ const h = handlerFor(d, s);
+ if (!h) throw Object.assign(new Error(`${d.id} has no route here`), { status: 500 });
+ const r = s.spec.route;
+ const query = {};
+ for (const [k, v] of Object.entries(input)) if (k !== 'target') query[k] = Array.isArray(v) ? v.join(',') : String(v);
+ for (const [k, v] of Object.entries(r.query || {})) query[k] = fill(v, input, d.input);
+ const target = input.target === undefined ? undefined : r.target ? fill(r.target, input, d.input) : String(input.target);
+ if (target !== undefined) query.target = target;
+ const rel = r.path.replace(/^\/api\/(net|dev)/, '');
+ const fake = Object.create(ctx.req);
+ Object.defineProperty(fake, 'path', { value: target !== undefined ? `${rel}/${encodeURIComponent(target)}` : rel, enumerable: true });
+ fake.params = target !== undefined ? { target } : {};
+ fake.query = query;
+ fake.method = r.method;
+ fake.netTool = d.id;
+ fake.body = {};
+ const cap = capture();
+ Promise.resolve(h(fake, cap.res, (err) => cap.settle({ error: err || new Error('not handled'), headers: {} }))).catch(err => cap.settle({ error: err, headers: {} }));
+ return fromRoute(await cap.done);
+ }
+
+ const api = createRunApi({
+ guard: o.guard, contracts: o.contracts,
+ snapshot: () => o.toolRegistry.snapshot(),
+ validate: createInputValidator(o.Ajv, o.addFormats),
+ parseJson: o.parseJson,
+ runs(d) {
+ const k = kind(d);
+ return k === 'engine' || k === 'route' ? 'inline' : null;
+ },
+ async inline(d, input, ctx) {
+ const s = internal(d.id);
+ if (kind(d) === 'engine') {
+ try {
+ return await pool.run('run', { module: s.engines, name: s.spec.engine, input }, { timeoutMs: ctx.timeoutMs, signal: ctx.signal });
+ } catch (err) {
+ if (err.code === 'tools.busy') throw new Refusal(503, 'tools.busy', err.message, undefined, { 'Retry-After': String(err.retryAfter || 10) });
+ throw err;
+ }
+ }
+ return viaRoute(d, s, input, ctx);
+ },
+ cacheTtl(d) {
+ const s = internal(d.id);
+ if (s.spec.cacheTtlMs !== undefined) return s.spec.cacheTtlMs;
+ return kind(d) === 'engine' ? ENGINE_TTL_MS : 0;
+ },
+ // Open Graph throttles its own target inside the route; the net routes' throttle is a router
+ // middleware, which a run does not pass through, so it is applied here.
+ targetOf: (d, input) => (internal(d.id).app === 'net' && d.egress ? (input.target || null) : null),
+ proxy(d, req, res) {
+ if (kind(d) !== 'satellite') return false;
+ const s = internal(d.id);
+ const port = o.ports()[s.satellite];
+ if (!port) return false;
+ forward(req, res, {
+ port,
+ onJson: (body) => { const id = body && body.job && body.job.id; if (id) o.jobIndex.set(id, s.satellite); },
+ });
+ return true;
+ },
+ });
+
+ return { handle: api.handle, stats: () => ({ ...api.stats(), engines: pool.stats() }), pool, api, kind };
+}
+
+module.exports = { createGatewayRun, capture, routeHandler, fromRoute, fill };
new file mode 100644
new file mode 100644
@@ -0,0 +1,166 @@
@@ -0,0 +1,166 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// The gateway's jobs facade (ADR-027): https://openvibe.tools/api/v1/jobs… fronts every satellite's
+// job routes, so a client (openvibe-sdk/jobs without a baseUrl, the run API's Location) needs one origin.
+//
+// POST /api/v1/jobs routed by the job type's prefix (img.process → img, …): from
+// ?type=, the JSON body, or the multipart `type` part (read from
+// the head of the stream; the uploads then stream through)
+// GET /api/v1/jobs/:id routed by the job id: the satellite that answered its submit or
+// DELETE /api/v1/jobs/:id run (remembered here), else the one that says it holds the job
+// GET /api/v1/jobs/:id/events when asked on loopback (GET /api/internal/jobs/:id, direct
+// GET /api/v1/jobs/:id/files/:n callers only). Events (SSE) and files stream through.
+// POST /api/v1/jobs/:id/retry
+// PUT /api/v1/jobs/:id/references/:ref
+// DELETE /api/v1/jobs/:id/references/:ref
+//
+// The satellite does everything else (owner scoping, quotas, the Origin check, idempotency): the
+// request reaches it with the caller's credentials and address, as through the host proxy.
+// ═══════════════════════════════════════════════════════════════
+
+const { forward } = require('../../../_shared/tools/proxy');
+const { JOB_SATELLITES, satelliteForType } = require('../../../_shared/tools/satellites');
+
+const ID_RE = /^job_[0-9A-HJKMNP-TV-Z]{26}$/;
+const PATH_RE = /^\/api\/v1\/jobs(?:\/([^/]+)(\/.*)?)?\/?$/;
+const PEEK_MAX = 64 * 1024;
+const JSON_MAX = 1024 * 1024;
+const TYPE_RE = /^[a-z][a-z0-9]*\.[a-z0-9_.]+$/;
+
+const isJobsPath = (p) => PATH_RE.test(String(p || ''));
+
+/** Job id → satellite, most recent first, bounded. */
+function createJobIndex(max = 50_000) {
+ const map = new Map();
+ return {
+ get: (id) => map.get(id) || null,
+ set(id, sat) { if (!ID_RE.test(String(id)) || !JOB_SATELLITES.includes(sat)) return; map.delete(id); map.set(id, sat); if (map.size > max) map.delete(map.keys().next().value); },
+ size: () => map.size,
+ };
+}
+
+/** Read the start of a multipart body until the `type` part → { type, head, ended }. */
+function peekMultipartType(req) {
+ return new Promise((resolve) => {
+ const chunks = [];
+ let size = 0, finished = false;
+ const re = /content-disposition:\s*form-data;\s*name="type"(?:;[^\r\n]*)?\r\n(?:[^\r\n]+\r\n)*\r\n([^\r\n]*)\r\n/i;
+ const done = (type, ended) => {
+ if (finished) return;
+ finished = true;
+ req.removeListener('data', onData); req.removeListener('end', onEnd); req.removeListener('error', onEnd);
+ if (!ended) req.pause();
+ resolve({ type, head: Buffer.concat(chunks), ended });
+ };
+ const onData = (c) => {
+ chunks.push(c); size += c.length;
+ const m = re.exec(Buffer.concat(chunks).toString('latin1'));
+ if (m) return done(m[1].trim(), false);
+ if (size > PEEK_MAX) return done(null, false);
+ return undefined;
+ };
+ const onEnd = () => { const m = re.exec(Buffer.concat(chunks).toString('latin1')); done(m ? m[1].trim() : null, true); };
+ req.on('data', onData);
+ req.on('end', onEnd);
+ req.on('error', onEnd);
+ });
+}
+
+/** The whole (small) body → Buffer, or null past `max`. */
+function readAll(req, max) {
+ return new Promise((resolve) => {
+ const chunks = [];
+ let size = 0;
+ req.on('data', (c) => { size += c.length; if (size <= max) chunks.push(c); });
+ req.on('end', () => resolve(size <= max ? Buffer.concat(chunks) : null));
+ req.on('error', () => resolve(null));
+ });
+}
+
+/**
+ * @param {object} o
+ * @param {() => object} o.ports satellite ports
+ * @param {object} o.index createJobIndex()
+ * @param {object} o.contracts
+ * @param {Function} [o.fetchImpl]
+ */
+function createJobsFacade(o) {
+ const fetchImpl = o.fetchImpl || fetch;
+ const counts = { submitted: 0, routed: 0, probed: 0, not_found: 0 };
+ const problem = (req, res, status, code, detail) => {
+ res.setHeader('Cache-Control', 'no-store');
+ // A body left unread (a refused upload): read and discard it, so the connection stays usable.
+ if (!req.readableEnded) req.resume();
+ return o.contracts.http.sendProblem(res, status, code, { detail, ctx: req.ov });
+ };
+ const remember = (sat) => (body) => { if (body && body.id && body.object === 'tools.job') o.index.set(body.id, sat); };
+
+ /** Which satellite holds job `id`: remembered, else asked (in parallel, on loopback). */
+ async function locate(id) {
+ const known = o.index.get(id);
+ if (known) return known;
+ counts.probed++;
+ const ports = o.ports();
+ const answers = await Promise.all(JOB_SATELLITES.filter(s => ports[s]).map(async (sat) => {
+ try {
+ const r = await fetchImpl(`http://127.0.0.1:${ports[sat]}/api/internal/jobs/${id}`, { signal: AbortSignal.timeout(2500) });
+ return r.status === 200 ? sat : null;
+ } catch { return null; }
+ }));
+ const sat = answers.find(Boolean) || null;
+ if (sat) o.index.set(id, sat);
+ return sat;
+ }
+
+ async function submit(req, res) {
+ let type = typeof req.query.type === 'string' ? req.query.type : null;
+ let body = null, head = null;
+ if (!type) {
+ if (req.is('multipart/form-data')) {
+ const p = await peekMultipartType(req);
+ type = p.type;
+ if (p.ended) body = p.head; else head = p.head;
+ } else {
+ body = await readAll(req, JSON_MAX);
+ if (!body) return problem(req, res, 413, 'tools.job.too_large', 'A JSON job request is at most 1 MB; send files as multipart.');
+ try { const j = JSON.parse(body.toString('utf8') || '{}'); type = j && typeof j.type === 'string' ? j.type : null; } catch {
+ return problem(req, res, 400, 'tools.job.invalid', 'The body is not valid JSON.');
+ }
+ }
+ }
+ if (!type || !TYPE_RE.test(type)) return problem(req, res, 400, 'tools.job.invalid', 'Name the job type (img.process, audio.process or docs.process): the `type` field, as the first multipart part, or ?type=.');
+ const sat = satelliteForType(type);
+ const port = sat && o.ports()[sat];
+ if (!port) return problem(req, res, 400, 'tools.job.unknown_type', `Unknown job type "${type}". Known: ${JOB_SATELLITES.map(s => `${s}.process`).join(', ')}`);
+ counts.submitted++;
+ return forward(req, res, { port, ...(body ? { body } : { head }), onJson: remember(sat) });
+ }
+
+ return {
+ async handle(req, res, next) {
+ const m = PATH_RE.exec(req.path);
+ if (!m) return next();
+ if (req.method === 'OPTIONS') return next();
+ try {
+ const [, id, rest] = m;
+ if (!id) {
+ if (req.method !== 'POST') { res.setHeader('Allow', 'POST'); return problem(req, res, 405, 'method_not_allowed', 'POST a job; GET /api/v1/jobs/:id reads one.'); }
+ return await submit(req, res);
+ }
+ if (!ID_RE.test(id)) return problem(req, res, 404, 'tools.job.not_found', 'No such job');
+ const sat = await locate(id);
+ if (!sat) { counts.not_found++; return problem(req, res, 404, 'tools.job.not_found', 'No such job'); }
+ counts.routed++;
+ const port = o.ports()[sat];
+ const learns = req.method === 'POST' && rest === '/retry';
+ return await forward(req, res, { port, ...(learns && { onJson: remember(sat) }) });
+ } catch (err) {
+ return next(err);
+ }
+ },
+ stats: () => ({ ...counts, known: o.index.size() }),
+ locate,
+ };
+}
+
+module.exports = { createJobsFacade, createJobIndex, peekMultipartType, isJobsPath, PATH_RE };
new file mode 100644
new file mode 100644
@@ -0,0 +1,300 @@
@@ -0,0 +1,300 @@
+'use strict';
+// POST /api/v1/tools/:id/run (ADR-027, tools.run-request@1 → tools.run@1) on real processes: the
+// gateway, img and docs, with a stand-in Network for tokens, the guard enforcing.
+// • inline tools: dev and text engines (the page's code, in the gateway's worker pool; cached when
+// safe, fresh when random), net lookups through their own routes (myip is the caller's address)
+// • every answer is a tools.run@1 document; refusals are problem+json with the contract's codes
+// (400 tools.run.invalid, 401/403 auth, 404 not_found / not_runnable, 405, 409, 413, 422 with
+// errors[], 429 with Retry-After, 503 tools.tool.unavailable vs tools.unavailable)
+// • probes need a token with tools.net.probe; tools that need a session refuse the anonymous
+// • job tools stream through the gateway to their satellite: multipart uploads as `file` parts,
+// wait_ms (200 finished) or not (202 + Location), Idempotency-Key replays (same job) and
+// conflicts, file references ({ job_id, index }) across satellites and their owner check
+// • the Origin check on cookie-authenticated calls (CSRF) and CORS for other sites' pages
+// • the older endpoints answer as before with Deprecation, Sunset and a successor Link
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const contracts = require('openvibe-contracts');
+const { startApp, freePort } = require('../../_shared/test/spawn');
+const { startNetwork } = require('../../_shared/test/network');
+
+const APPS = path.join(__dirname, '..', '..');
+const sharp = require(require.resolve('sharp', { paths: [path.join(APPS, 'img')] }));
+const { PDFDocument } = require(require.resolve('pdf-lib', { paths: [path.join(APPS, 'docs')] }));
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-run-api-'));
+const MISSING = path.join(tmp, 'not-installed');
+let ipN = 1;
+const freshIp = () => `198.51.100.${ipN++}`;
+
+function checkRun(body, where) {
+ const r = contracts.validate('tools.run@1', body);
+ assert.ok(r.valid, `${where}: tools.run@1 ${JSON.stringify(r.errors && r.errors.slice(0, 3))} ${JSON.stringify(body).slice(0, 300)}`);
+}
+function checkProblem(r, status, code, where) {
+ assert.strictEqual(r.status, status, `${where}: ${r.text.slice(0, 300)}`);
+ assert.match(r.h.get('content-type'), /application\/problem\+json/, where);
+ assert.strictEqual(r.body.code, code, `${where}: ${r.text.slice(0, 300)}`);
+}
+
+(async () => {
+ const net = await startNetwork();
+ const procs = [];
+ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };
+ try {
+ // Every app knows where the others listen (a docs run reads an img job's result on loopback).
+ const ports = { img: await freePort(), docs: await freePort() };
+ const auth = { OV_NETWORK_URL: net.url, OV_NETWORK_INTERNAL_URL: net.url, TOOLS_GUARD: 'enforce', TOOLS_SATELLITE_PORTS: `img=${ports.img},docs=${ports.docs}` };
+ const img = await start('img', { ...auth, DATA_DIR: path.join(tmp, 'img'), UPLOADS_DIR: path.join(tmp, 'img', 'up'), OUTPUT_DIR: path.join(tmp, 'img', 'out'), HEIF_DEC_PATH: MISSING }, ports.img);
+ const docs = await start('docs', { ...auth, DATA_DIR: path.join(tmp, 'docs'), UPLOADS_DIR: path.join(tmp, 'docs', 'up'), OUTPUT_DIR: path.join(tmp, 'docs', 'out'), QPDF_PATH: MISSING }, ports.docs);
+ const gw = await start('gateway', {
+ ...auth, DATA_DIR: path.join(tmp, 'gw'),
+ OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry',
+ TOOLS_GUARD_LIMITS: JSON.stringify({ 'tools-run': { anonymous: { perMinute: 40, burst: 40 } } }),
+ });
+ const G = gw.base;
+ const call = async (p, { method = 'POST', json, form, headers = {}, base = G } = {}) => {
+ const h = { 'X-Forwarded-For': headers['X-Forwarded-For'] || freshIp(), ...headers };
+ let body;
+ if (json !== undefined) { h['Content-Type'] = 'application/json'; body = typeof json === 'string' ? json : JSON.stringify(json); }
+ if (form) body = form;
+ const r = await fetch(`${base}${p}`, { method, headers: h, body });
+ const text = await r.text();
+ let parsed = null; try { parsed = text ? JSON.parse(text) : null; } catch { /* not JSON */ }
+ return { status: r.status, h: r.headers, text, body: parsed };
+ };
+ const run = (id, json, opts = {}) => call(`/api/v1/tools/${id}/run`, { json, ...opts });
+ // The gateway reads each satellite's own status on boot (protectpdf without qpdf).
+ for (let i = 0; i < 50 && (await call('/api/v1/tools/protectpdf', { method: 'GET' })).body.status !== 'unavailable'; i++) await new Promise(r => setTimeout(r, 100));
+
+ // ── Inline: a dev engine (the page's own code) ──
+ let r = await run('jsonminify', { input: { text: '{ "a": [1, 2] }' } });
+ assert.strictEqual(r.status, 200, r.text);
+ checkRun(r.body, 'jsonminify');
+ assert.deepStrictEqual([r.body.state, r.body.tool, r.body.result], ['succeeded', 'jsonminify', { text: '{"a":[1,2]}' }]);
+ assert.strictEqual(r.h.get('cache-control'), 'no-store');
+ assert.ok(Number.isInteger(r.body.took_ms));
+ r = await run('jsonminify', { input: { text: '{ "a": [1, 2] }' } });
+ assert.strictEqual(r.h.get('x-ov-cache'), 'hit', 'a pure transform is answered from the cache');
+ // A text engine; the JSON a tool answers as data.
+ r = await run('count', { input: { text: 'two words' } });
+ checkRun(r.body, 'count');
+ assert.strictEqual(r.body.result.data.words, 2);
+ // Random output is never reused.
+ const a = await run('uuid', { input: { count: 1 } }), b = await run('uuid', { input: { count: 1 } });
+ assert.notStrictEqual(a.body.result.data.uuids[0], b.body.result.data.uuids[0]);
+ assert.ok(!b.h.get('x-ov-cache'));
+ // The tool's own failure is a failed run (200), with its problem.
+ r = await run('jsonminify', { input: { text: '{ nope' } });
+ assert.strictEqual(r.status, 200);
+ checkRun(r.body, 'jsonminify failed');
+ assert.deepStrictEqual([r.body.state, r.body.error.status, r.body.error.code], ['failed', 422, 'tools.job.failed']);
+ r = await run('compare', { input: { a: 'x\n'.repeat(2500), b: 'y' } });
+ assert.deepStrictEqual([r.body.state, r.body.error.code], ['failed', 'tools.input.invalid'], 'an engine\'s own code is kept');
+
+ // ── Refusals before the tool runs ──
+ r = await run('jsonminify', { input: { text: 'x', nope: 1 } });
+ checkProblem(r, 422, 'tools.input.invalid', 'schema');
+ assert.deepStrictEqual(r.body.errors, [{ path: '/nope', message: 'is not a field of this tool' }]);
+ r = await run('jsonminify', { input: {} });
+ assert.deepStrictEqual(r.body.errors, [{ path: '/text', message: 'is required' }]);
+ checkProblem(await run('jsonminify', { input: { text: 'x' }, extra: 1 }), 400, 'tools.run.invalid', 'unknown field');
+ checkProblem(await run('jsonminify', { input: { text: 'x' }, wait_ms: 60001 }), 400, 'tools.run.invalid', 'wait_ms');
+ checkProblem(await run('jsonminify', '{"input":'), 400, 'tools.run.invalid', 'not JSON');
+ checkProblem(await run('jsonminify', { input: [] }), 400, 'tools.run.invalid', 'input array');
+ checkProblem(await run('jsonminify', { input: { text: 'x' } }, { headers: { 'Idempotency-Key': 'short' } }), 400, 'tools.run.invalid', 'bad key');
+ checkProblem(await run('dns', { input: { target: 'a'.repeat(4500) } }), 413, 'tools.input.too_large', 'maxInputBytes');
+ checkProblem(await run('nosuchtool', {}), 404, 'tools.tool.not_found', 'unknown');
+ checkProblem(await run('yt', {}), 404, 'tools.tool.not_runnable', 'yt: api false');
+ checkProblem(await run('regex', { input: {} }), 404, 'tools.tool.not_runnable', 'regex: page only');
+ r = await run('traceroute', {});
+ checkProblem(r, 503, 'tools.tool.unavailable', 'descriptor status unavailable');
+ assert.match(r.body.detail, /raw network access/);
+ checkProblem(await run('protectpdf', { input: { password: 'x' } }), 503, 'tools.tool.unavailable', 'qpdf missing on docs');
+ checkProblem(await call('/api/v1/tools/jsonminify/run', { method: 'GET' }), 405, 'method_not_allowed', 'GET');
+
+ // ── Net tools through their routes ──
+ r = await run('myip', { input: {} }, { headers: { 'X-Forwarded-For': '203.0.113.77' } });
+ checkRun(r.body, 'myip');
+ assert.deepStrictEqual(r.body.result.data, { ip: '203.0.113.77', version: 4 }, 'the caller\'s own address');
+ r = await run('ipv4', { input: { target: '10.1.2.0/24' } });
+ checkRun(r.body, 'ipv4');
+ assert.strictEqual(r.body.result.data.cidr.usable, 254);
+ r = await run('ipv4', { input: { target: '10.1.2.0/24' } });
+ assert.strictEqual(r.h.get('x-ov-cache'), 'hit', 'lookups with a cacheTtlMs are reused');
+ // A target the SSRF guard refuses: the tool ran and failed with its own code.
+ r = await run('headers', { input: { target: 'http://127.0.0.1:1/' } });
+ checkRun(r.body, 'headers refused');
+ assert.deepStrictEqual([r.body.state, r.body.error.status, r.body.error.code], ['failed', 403, 'tools.net.target_not_public']);
+
+ // ── Probes: a token with tools.net.probe, nobody else ──
+ checkProblem(await run('ping', { input: { target: '127.0.0.1' } }), 401, 'token.missing', 'probe, anonymous');
+ checkProblem(await run('ping', { input: { target: '127.0.0.1' } }, { headers: { Authorization: `Bearer ${net.user()}` } }), 403, 'capability.denied', 'probe, a person');
+ checkProblem(await run('ping', { input: { target: '127.0.0.1' } }, { headers: { Authorization: `Bearer ${net.service(['tools.tool.run'])}` } }), 403, 'capability.denied', 'probe, runner token');
+ r = await run('ping', { input: { target: '127.0.0.1', count: 1 } }, { headers: { Authorization: `Bearer ${net.service(['tools.net.probe'], { name: 'prober' })}` } });
+ checkRun(r.body, 'ping with tools.net.probe');
+ assert.strictEqual(r.body.error.code, 'tools.net.target_not_public', 'authorized: it ran (and the SSRF guard kept it off loopback)');
+ checkProblem(await run('jsonminify', { input: { text: '1' } }, { headers: { Authorization: `Bearer ${net.service(['tools.job.read'])}` } }), 403, 'capability.denied', 'a token without tools.tool.run');
+ checkProblem(await run('jsonminify', { input: { text: '1' } }, { headers: { Authorization: 'Bearer not.a.token' } }), 401, 'token.invalid', 'a token that does not verify');
+ r = await run('jsonminify', { input: { text: '1' } }, { headers: { Authorization: `Bearer ${net.service(['tools.tool.run'])}` } });
+ assert.strictEqual(r.body.state, 'succeeded', 'a runner token runs');
+
+ // ── Quotas: the tool's class, per caller ──
+ {
+ const ip = freshIp();
+ const codes = [];
+ for (let i = 0; i < 80 && !codes.includes(429); i++) {
+ r = await run('slug', { input: { text: `q ${i}` } }, { headers: { 'X-Forwarded-For': ip } });
+ codes.push(r.status);
+ }
+ assert.ok(codes.slice(0, 40).every(c => c === 200), JSON.stringify(codes));
+ checkProblem(r, 429, 'tools.quota.exceeded', 'tools-run, anonymous: a burst of 40, then 40 a minute');
+ assert.ok(Number(r.h.get('retry-after')) >= 1);
+ assert.strictEqual(r.body.quota_class, 'tools-run');
+ }
+
+ // ── Inline Idempotency-Key: the same run again, or a conflict ──
+ // (anonymous callers are keyed by address: the same one each time)
+ const same = await run('lorem', { input: { unit: 'words', count: 4 } }, { headers: { 'Idempotency-Key': 'inline-key-0002', 'X-Forwarded-For': '192.0.2.9' } });
+ const replay = await run('lorem', { input: { unit: 'words', count: 4 } }, { headers: { 'Idempotency-Key': 'inline-key-0002', 'X-Forwarded-For': '192.0.2.9' } });
+ assert.strictEqual(replay.h.get('idempotent-replayed'), 'true');
+ assert.deepStrictEqual(replay.body, same.body, 'a random generator gives the first answer again');
+ checkProblem(await run('lorem', { input: { unit: 'words', count: 5 } }, { headers: { 'Idempotency-Key': 'inline-key-0002', 'X-Forwarded-For': '192.0.2.9' } }), 409, 'tools.job.idempotency_conflict', 'same key, other input');
+
+ // ── Job tools through the gateway ──
+ const png = await sharp({ create: { width: 8, height: 6, channels: 3, background: '#0a0' } }).png().toBuffer();
+ const upload = (fields, files) => {
+ const fd = new FormData();
+ for (const [k, v] of Object.entries(fields)) fd.append(k, typeof v === 'string' ? v : JSON.stringify(v));
+ for (const f of files) fd.append('file', new Blob([f.data], f.type ? { type: f.type } : {}), f.name);
+ return fd;
+ };
+ const person = net.user();
+ const as = { Authorization: `Bearer ${person}` };
+ // Finished within wait_ms: 200 with the result files.
+ r = await call('/api/v1/tools/webp/run', { form: upload({ input: { quality: 70 }, wait_ms: '10000' }, [{ name: 'photo.png', data: png }]), headers: as });
+ assert.strictEqual(r.status, 200, r.text);
+ checkRun(r.body, 'webp waited');
+ assert.strictEqual(r.body.state, 'succeeded');
+ assert.deepStrictEqual([r.body.job.tool, r.body.job.service, r.body.job.type], ['webp', 'img', 'img.process']);
+ assert.strictEqual(r.body.result.files[0].mime, 'image/webp');
+ assert.strictEqual(r.body.result.files[0].name, 'photo.webp');
+ assert.strictEqual(r.body.result.data.output.ext, 'webp');
+ const webpJob = r.body.job;
+ let f = await fetch(`${G}${r.body.result.files[0].url}`, { headers: as });
+ assert.strictEqual(f.status, 200);
+ assert.strictEqual((await sharp(Buffer.from(await f.arrayBuffer())).metadata()).format, 'webp', 'the file, through the gateway\'s facade');
+ // Not waiting: 202 + Location, then the job.
+ const k1 = 'job-key-00000001';
+ r = await call('/api/v1/tools/png/run', { form: upload({ input: {} }, [{ name: 'x.png', data: png }]), headers: { ...as, 'Idempotency-Key': k1 } });
+ assert.strictEqual(r.status, 202, r.text);
+ checkRun(r.body, 'png 202');
+ assert.ok(['queued', 'running'].includes(r.body.state));
+ assert.strictEqual(r.h.get('location'), `/api/v1/jobs/${r.body.job.id}`);
+ assert.strictEqual(r.body.location, r.h.get('location'));
+ const pngJob = r.body.job.id;
+ // The same key and request: the same job (200 + Idempotent-Replayed).
+ r = await call('/api/v1/tools/png/run', { form: upload({ input: {} }, [{ name: 'x.png', data: png }]), headers: { ...as, 'Idempotency-Key': k1 } });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.h.get('idempotent-replayed'), 'true');
+ assert.strictEqual(r.body.job.id, pngJob);
+ checkRun(r.body, 'png replay');
+ // Another request under that key: 409.
+ r = await call('/api/v1/tools/png/run', { form: upload({ input: { quality: 5 } }, [{ name: 'x.png', data: png }]), headers: { ...as, 'Idempotency-Key': k1 } });
+ checkProblem(r, 409, 'tools.job.idempotency_conflict', 'job key reused');
+ // A declared type the bytes contradict is corrected; bytes that are no image are refused (415).
+ r = await call('/api/v1/tools/png/run', { form: upload({ wait_ms: '10000' }, [{ name: 'x.bin', data: png, type: 'application/octet-stream' }]), headers: as });
+ assert.strictEqual(r.body.state, 'succeeded', 'sniffed as PNG whatever the part said');
+ r = await call('/api/v1/tools/png/run', { form: upload({}, [{ name: 'x.png', data: Buffer.from('not an image at all') }]), headers: as });
+ checkProblem(r, 415, 'tools.file.unsupported_type', 'sniffing');
+ r = await call('/api/v1/tools/png/run', { form: upload({}, [{ name: 'a.png', data: png }, { name: 'b.png', data: png }]), headers: as });
+ checkProblem(r, 400, 'tools.run.invalid', 'two files for a one-file tool');
+ checkProblem(await run('png', { input: {} }, { headers: as }), 400, 'tools.run.invalid', 'no file');
+ r = await call('/api/v1/tools/png/run', { form: upload({ input: { quality: 500 } }, [{ name: 'x.png', data: png }]), headers: as });
+ checkProblem(r, 422, 'tools.input.invalid', 'job input schema');
+ // A program the engine needs is missing (HEIC without libheif): the run fails with tools.unavailable.
+ const heic = Buffer.concat([Buffer.from([0, 0, 0, 0x18]), Buffer.from('ftypheic'), Buffer.alloc(4), Buffer.from('mif1heic'), Buffer.alloc(64)]);
+ r = await call('/api/v1/tools/convert/run', { form: upload({ input: { format: 'png' }, wait_ms: '10000' }, [{ name: 'p.heic', data: heic }]), headers: as });
+ checkRun(r.body, 'heic');
+ assert.deepStrictEqual([r.body.state, r.body.error.status, r.body.error.code], ['failed', 503, 'tools.unavailable'], JSON.stringify(r.body.error));
+ checkProblem(await run('heic', { input: {} }, { headers: as }), 503, 'tools.tool.unavailable', 'the heic tool itself');
+
+ // Docs: a session, a sign-in or a token (auth.anonymous false).
+ const pdf = async (pages) => { const d = await PDFDocument.create(); for (let i = 0; i < pages; i++) d.addPage([100, 100]); return Buffer.from(await d.save()); };
+ const p1 = await pdf(1), p2 = await pdf(2);
+ r = await call('/api/v1/tools/mergepdf/run', { form: upload({}, [{ name: 'a.pdf', data: p1 }, { name: 'b.pdf', data: p2 }]) });
+ checkProblem(r, 401, 'tools.session_required', 'merge, anonymous');
+ r = await call('/api/v1/tools/mergepdf/run', { form: upload({ input: { order: [1, 0] }, wait_ms: '15000' }, [{ name: 'a.pdf', data: p1 }, { name: 'b.pdf', data: p2 }]), headers: as });
+ checkRun(r.body, 'merge');
+ assert.deepStrictEqual([r.body.state, r.body.result.data.pageCount, r.body.job.service], ['succeeded', 3, 'docs']);
+ // A file reference: the webp result (img) into Image to PDF (docs), same owner.
+ r = await run('image2pdf', { input: { pageSize: 'fit' }, files: [{ job_id: webpJob.id, index: 0 }], wait_ms: 15000 }, { headers: as });
+ checkRun(r.body, 'image2pdf from a job reference');
+ assert.strictEqual(r.body.state, 'succeeded', JSON.stringify(r.body.error || r.body));
+ assert.strictEqual(r.body.result.data.pageCount, 1);
+ // Somebody else's job, or no such file: 404 tools.run.file_not_found.
+ checkProblem(await run('image2pdf', { files: [{ job_id: webpJob.id, index: 0 }] }, { headers: { Authorization: `Bearer ${net.user()}` } }), 404, 'tools.run.file_not_found', 'another owner\'s job');
+ checkProblem(await run('image2pdf', { files: [{ job_id: webpJob.id, index: 3 }] }, { headers: as }), 404, 'tools.run.file_not_found', 'no such index');
+ checkProblem(await run('image2pdf', { files: [{ media_id: 'med_01J00000000000000000000000' }] }, { headers: as }), 404, 'tools.run.file_not_found', 'unknown media id');
+ checkProblem(await run('image2pdf', { files: [{ job_id: 'nope' }] }, { headers: as }), 400, 'tools.run.invalid', 'a malformed reference');
+
+ // ── Origin check (CSRF): cookies from another site's page ──
+ const session = (await call('/api/context', { method: 'GET', base: img.base })).h.get('set-cookie').split(';')[0];
+ const evil = { Cookie: session, Origin: 'https://evil.example' };
+ r = await run('slug', { input: { text: 'x' } }, { headers: evil });
+ checkProblem(r, 403, 'tools.origin.refused', 'cookie + foreign origin');
+ assert.strictEqual(r.h.get('access-control-allow-origin'), '*', 'another site gets CORS without credentials');
+ assert.ok(!r.h.get('access-control-allow-credentials'));
+ assert.strictEqual((await run('slug', { input: { text: 'x' } }, { headers: { Cookie: session, Origin: 'https://slug.openvibe.tools' } })).status, 200, 'our own pages');
+ assert.strictEqual((await run('slug', { input: { text: 'x' } }, { headers: { Origin: 'https://evil.example' } })).status, 200, 'no cookies: nothing to forge');
+ assert.strictEqual((await run('slug', { input: { text: 'x' } }, { headers: { ...evil, Authorization: `Bearer ${net.service(['tools.tool.run'])}` } })).status, 200, 'a Bearer token is not a cookie');
+ r = await call('/api/v1/tools/png/run', { form: upload({}, [{ name: 'x.png', data: png }]), headers: evil });
+ checkProblem(r, 403, 'tools.origin.refused', 'a job tool, checked on its satellite');
+ // The older endpoints' CORS already turns a foreign Origin away; a foreign Referer alone is refused here.
+ r = await call('/api/process', { base: img.base, form: upload({ tool: 'convert', format: 'webp' }, [{ name: 'x.png', data: png, type: 'image/png' }]), headers: { Cookie: session, Referer: 'https://evil.example/page' } });
+ checkProblem(r, 403, 'tools.origin.refused', 'the older endpoints too');
+ r = await call('/api/v1/jobs', { base: img.base, form: upload({ type: 'img.process', input: { tool: 'convert' } }, [{ name: 'x.png', data: png, type: 'image/png' }]), headers: { Cookie: session, Origin: 'https://evil.example' } });
+ assert.strictEqual(r.status, 403, 'and the job routes');
+ // Preflight from another site's page.
+ r = await call('/api/v1/tools/slug/run', { method: 'OPTIONS', headers: { Origin: 'https://app.example', 'Access-Control-Request-Method': 'POST', 'Access-Control-Request-Headers': 'authorization,content-type,idempotency-key' } });
+ assert.strictEqual(r.status, 204);
+ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');
+ assert.match(r.h.get('access-control-allow-headers'), /Idempotency-Key/);
+
+ // ── The older endpoints: same answers, and what replaces them ──
+ r = await call('/api/net/myip', { method: 'GET' });
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.body.ok, true, 'the legacy shape');
+ assert.strictEqual(r.h.get('deprecation'), 'true');
+ assert.strictEqual(r.h.get('sunset'), 'Thu, 31 Dec 2026 23:59:59 GMT');
+ assert.strictEqual(r.h.get('link'), '</api/v1/tools/myip/run>; rel="successor-version"');
+ assert.strictEqual((await call('/api/net/dns/example.invalid?types=MX', { method: 'GET' })).h.get('link'), '</api/v1/tools/dns/run>; rel="successor-version"');
+ assert.strictEqual((await call('/api/dev/tools', { method: 'GET' })).h.get('link'), '</api/v1/tools?family=dev>; rel="successor-version"');
+ r = await call('/api/process', { base: img.base, form: upload({ tool: 'convert', format: 'webp' }, [{ name: 'x.png', data: png, type: 'image/png' }]) });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.body.success, true, 'the legacy shape');
+ assert.strictEqual(r.h.get('deprecation'), 'true');
+ assert.strictEqual(r.h.get('link'), '</api/v1/tools/convert/run>; rel="successor-version"');
+ {
+ // On png.openvibe.tools the successor is png's run route (fetch cannot set Host: node:http).
+ const req = new Request('http://x/', { method: 'POST', body: upload({ tool: 'convert' }, [{ name: 'x.png', data: png, type: 'image/png' }]) });
+ const bodyBuf = Buffer.from(await req.arrayBuffer());
+ const link = await new Promise((resolve, reject) => {
+ const q = require('http').request({ host: '127.0.0.1', port: img.port, method: 'POST', path: '/api/process', headers: { Host: 'png.openvibe.tools', 'Content-Type': req.headers.get('content-type'), 'Content-Length': bodyBuf.length, 'X-Forwarded-For': freshIp() } }, (res) => { res.resume(); resolve(res.headers.link); });
+ q.on('error', reject); q.end(bodyBuf);
+ });
+ assert.strictEqual(link, '</api/v1/tools/png/run>; rel="successor-version"', 'the host\'s own tool');
+ }
+
+ // ── Health shows the run API ──
+ const health = (await call('/api/health', { method: 'GET' })).body;
+ assert.ok(health.run.runs > 50 && health.run.cached >= 2 && health.run.proxied >= 5, JSON.stringify(health.run));
+ assert.ok(health.run.engines.completed >= 5, 'engines ran in worker threads');
+ } finally {
+ for (const p of procs) await p.kill('SIGTERM');
+ await net.close();
+ }
+ console.log('run API: inline engines (cache, fresh), net routes (myip, SSRF), refusals and contract codes, probes need tools.net.probe, quotas, inline and job idempotency, job tools via the gateway (wait, 202, sniffing, schema), tools.unavailable vs tools.tool.unavailable, session rule, cross-satellite job references, CSRF origin check + CORS, deprecation headers: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,127 @@
@@ -0,0 +1,127 @@
+'use strict';
+// openvibe-sdk v0.6.0's Tools client (createToolsClient: registry, run API, jobs facade) against the
+// real gateway, img and docs, exactly as a developer uses it: no baseUrl (the `tools` origin), the
+// registry read without a token, a sync tool run anonymously, a job tool with an upload (202, then
+// wait() over the facade's SSE, then the file), waitMs, Idempotency-Key replays, a job's result fed
+// to the next tool by reference, ToolRunError for a failed run, OpenVibeError for a refusal, and a
+// probe with a tools.net.probe token.
+//
+// The SDK is required from a checkout next to this repo (../../../../OpenVibe.SDK, or OV_SDK_DIR);
+// without one this test says so and passes (it is not a dependency of any app).
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const { startApp, freePort } = require('../../_shared/test/spawn');
+const { startNetwork } = require('../../_shared/test/network');
+
+const SDK = process.env.OV_SDK_DIR || path.join(__dirname, '..', '..', '..', '..', 'OpenVibe.SDK');
+if (!fs.existsSync(path.join(SDK, 'src', 'tools.js'))) {
+ console.log(`sdk client: skipped (no OpenVibe.SDK checkout with src/tools.js at ${SDK}; set OV_SDK_DIR)`);
+ process.exit(0);
+}
+const sdkVersion = require(path.join(SDK, 'package.json')).version;
+const { createClient, isOpenVibeError } = require(path.join(SDK, 'src', 'core'));
+const { createToolsClient, isToolRunError } = require(path.join(SDK, 'src', 'tools'));
+
+const APPS = path.join(__dirname, '..', '..');
+const sharp = require(require.resolve('sharp', { paths: [path.join(APPS, 'img')] }));
+const rejects = (p) => p.then((v) => { throw new Error(`expected a rejection, got ${JSON.stringify(v).slice(0, 200)}`); }, (err) => err);
+
+(async () => {
+ assert.ok(/^0\.(6|[7-9])\.|^[1-9]/.test(sdkVersion), `the SDK is v0.6.0 or later (${sdkVersion})`);
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-sdk-client-'));
+ const net = await startNetwork();
+ const procs = [];
+ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };
+ try {
+ const ports = { img: await freePort(), docs: await freePort() };
+ const env = { OV_NETWORK_URL: net.url, OV_NETWORK_INTERNAL_URL: net.url, TOOLS_SATELLITE_PORTS: `img=${ports.img},docs=${ports.docs}` };
+ await start('img', { ...env, DATA_DIR: path.join(tmp, 'img'), UPLOADS_DIR: path.join(tmp, 'img', 'up'), OUTPUT_DIR: path.join(tmp, 'img', 'out') }, ports.img);
+ await start('docs', { ...env, DATA_DIR: path.join(tmp, 'docs'), UPLOADS_DIR: path.join(tmp, 'docs', 'up'), OUTPUT_DIR: path.join(tmp, 'docs', 'out') }, ports.docs);
+ const gw = await start('gateway', { ...env, DATA_DIR: path.join(tmp, 'gw'), OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' });
+
+ // The `tools` service origin is the gateway (what discovery gives in production).
+ const client = (token) => createClient({ baseUrls: { tools: gw.base }, autoDiscover: false, ...(token && { token }), onWarning: () => {} });
+ const anon = createToolsClient(client());
+ const person = createToolsClient(client(net.user()));
+
+ // ── Registry ──
+ const list = await anon.list({ family: 'dev', api: true });
+ assert.ok(list.count >= 20 && list.tools.every(t => t.family === 'dev' && t.api), 'filtered list');
+ const png = await anon.get('png');
+ assert.deepStrictEqual([png.id, png.execution, png.run.job.type], ['png', 'job', 'img.process']);
+ assert.strictEqual(await anon.get('nosuchtool'), null, 'no such tool → null');
+ const schema = await anon.schema('jsonminify');
+ assert.deepStrictEqual(schema.$defs.input.required, ['text']);
+
+ // ── A sync tool, anonymously ──
+ const out = await anon.run('jsonminify', { text: '{\n "a": [1, 2]\n}' });
+ assert.deepStrictEqual([out.state, out.tool, out.result.text, out.replayed], ['succeeded', 'jsonminify', '{"a":[1,2]}', false]);
+ assert.match(out.idempotencyKey, /^[!-~]{8,200}$/, 'the SDK sent an Idempotency-Key');
+ assert.strictEqual(await out.wait(), out, 'a finished run waits for nothing');
+ const again = await anon.run('jsonminify', { text: '{\n "a": [1, 2]\n}' }, { idempotencyKey: out.idempotencyKey });
+ assert.strictEqual(again.replayed, true, 'the same key again: the same run');
+ const lookup = await anon.run('ipv4', { target: '192.0.2.0/30' });
+ assert.strictEqual(lookup.result.data.cidr.usable, 2);
+ // The tool's own failure → ToolRunError; a refusal → OpenVibeError with the problem's code.
+ let err = await rejects(anon.run('jsonminify', { text: '{ bad' }));
+ assert.ok(isToolRunError(err), err.message);
+ assert.deepStrictEqual([err.state, err.code, err.status, err.tool], ['failed', 'tools.job.failed', 422, 'jsonminify']);
+ err = await rejects(anon.run('yt', {}));
+ assert.ok(isOpenVibeError(err) && !isToolRunError(err));
+ assert.deepStrictEqual([err.status, err.code], [404, 'tools.tool.not_runnable']);
+ err = await rejects(anon.run('jsonminify', { text: 'x', nope: true }));
+ assert.deepStrictEqual([err.status, err.code], [422, 'tools.input.invalid']);
+ assert.ok(Array.isArray(err.errors) && err.errors[0].path === '/nope');
+ err = await rejects(anon.run('ping', { target: '192.0.2.1' }));
+ assert.deepStrictEqual([err.status, err.code], [401, 'token.missing'], 'probes need a token');
+
+ // ── A job tool: 202, wait() over the facade's events, the file ──
+ const image = await sharp({ create: { width: 10, height: 8, channels: 3, background: '#c00' } }).png().toBuffer();
+ const queued = await person.run('webp', { quality: 60 }, { files: [{ name: 'photo.png', data: image, type: 'image/png' }] });
+ assert.ok(['queued', 'running', 'succeeded'].includes(queued.state), queued.state);
+ assert.ok(queued.job && /^\/api\/v1\/jobs\/job_/.test(queued.location));
+ const seen = [];
+ const done = await queued.wait({ onEvent: (e) => { seen.push(e.event); } });
+ assert.deepStrictEqual([done.state, done.tool, done.job.tool], ['succeeded', 'webp', 'webp']);
+ assert.ok(queued.state === 'succeeded' || seen.includes('job.succeeded'), `events: ${seen.join(', ')}`);
+ const file = done.result.files[0];
+ assert.deepStrictEqual([file.name, file.mime], ['photo.webp', 'image/webp']);
+ const res = await person.jobs.file(done.job.id, 0);
+ assert.strictEqual((await sharp(Buffer.from(await res.arrayBuffer())).metadata()).format, 'webp', 'the file through the facade');
+ assert.strictEqual((await person.jobs.get(done.job.id)).state, 'succeeded', 'jobs.get through the facade');
+
+ // waitMs: answered finished (200); an upload without a declared type is sniffed.
+ const quick = await person.run('png', {}, { files: [{ name: 'again.bin', data: image }], waitMs: 15000 });
+ assert.strictEqual(quick.state, 'succeeded');
+ assert.strictEqual(quick.result.files[0].mime, 'image/png');
+ // The same key and request again: the same job.
+ const key = 'sdk-job-key-0001';
+ const k1 = await person.run('jpg', {}, { files: [{ name: 'k.png', data: image, type: 'image/png' }], idempotencyKey: key });
+ const k2 = await person.run('jpg', {}, { files: [{ name: 'k.png', data: image, type: 'image/png' }], idempotencyKey: key });
+ assert.strictEqual(k2.replayed, true);
+ assert.strictEqual(k2.job.id, k1.job.id);
+ // One tool's result feeds the next (img → docs), by reference.
+ const pdf = await person.run('image2pdf', { pageSize: 'fit' }, { files: [{ job_id: done.job.id, index: 0 }], waitMs: 15000 });
+ assert.deepStrictEqual([pdf.state, pdf.result.data.pageCount, pdf.job.service], ['succeeded', 1, 'docs']);
+ // A job that fails: ToolRunError with the job attached (retry through the facade).
+ const broken = Buffer.concat([image.subarray(0, 40), Buffer.alloc(100, 3)]);
+ err = await rejects(person.run('png', {}, { files: [{ name: 'broken.png', data: broken }], waitMs: 15000 }));
+ assert.ok(isToolRunError(err), err.message);
+ assert.strictEqual(err.state, 'failed');
+ assert.ok(err.job && err.job.id);
+ const retried = await person.jobs.retry(err.job.id);
+ assert.strictEqual(retried.job.retry_of, err.job.id, 'jobs.retry through the facade');
+
+ // ── A probe with tools.net.probe: authorized (the SSRF guard keeps it off this host) ──
+ const prober = createToolsClient(client(net.service(['tools.net.probe'], { name: 'prober' })));
+ err = await rejects(prober.run('ping', { target: '127.0.0.1', count: 1 }));
+ assert.ok(isToolRunError(err));
+ assert.strictEqual(err.code, 'tools.net.target_not_public');
+ } finally {
+ for (const p of procs) await p.kill('SIGTERM');
+ await net.close();
+ }
+ console.log(`sdk client (openvibe-sdk ${sdkVersion}): registry, sync run + replay, ToolRunError vs refusals, job run + wait() + file via the facade, waitMs, idempotent job, job reference chain img → docs, retry, probe token: all checks passed`);
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -44,6 +44,14 @@ const get = async (url, headers = {}, method = 'GET') => {
@@ -44,6 +44,14 @@ const get = async (url, headers = {}, method = 'GET') => {
const ports = Object.entries(sat).map(([n, p]) => `${n}=${p.port}`).join(',');
const ports = Object.entries(sat).map(([n, p]) => `${n}=${p.port}`).join(',');
const gw = await start('gateway', { TOOLS_SATELLITE_PORTS: ports, OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' });
const gw = await start('gateway', { TOOLS_SATELLITE_PORTS: ports, OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' });
const G = gw.base;
const G = gw.base;
+ // The gateway reads every satellite's own status on boot: wait for all seven, so the list does
+ // not change between the ETag checks below (a busy machine answers slower than the first request).
+ for (let i = 0; i < 200; i++) {
+ const rd = (await get(`${G}/api/ready`)).body;
+ const d = rd && rd.checks && rd.checks.tool_registry && rd.checks.tool_registry.detail;
+ if (d && d.satellites_polled >= 7) break;
+ await new Promise(res => setTimeout(res, 100));
+ }
// ── The list: a tools.tool-list@1 document with every tool ──
// ── The list: a tools.tool-list@1 document with every tool ──
let r = await get(`${G}/api/v1/tools`, { Origin: 'https://someone-elses-site.example' });
let r = await get(`${G}/api/v1/tools`, { Origin: 'https://someone-elses-site.example' });
@@ -24,6 +24,9 @@ const { poolFromEnv } = require('../../_shared/jobs/pool');
@@ -24,6 +24,9 @@ const { poolFromEnv } = require('../../_shared/jobs/pool');
const codec = require('./tools/codec');
const codec = require('./tools/codec');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { satelliteRunApi, ajvFrom } = require('../../_shared/tools/run');
+const { satellitePorts } = require('../../_shared/tools/satellites');
+const { deprecated, runPath } = require('../../_shared/tools/deprecation');
const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const jobsRuntime = require('../../_shared/jobs');
const jobsRuntime = require('../../_shared/jobs');
const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
@@ -138,7 +141,7 @@ app.use(exceptRegistry(cors({
@@ -138,7 +141,7 @@ app.use(exceptRegistry(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-})));
+}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
app.use(guard.identify);
app.use(guard.identify);
@@ -202,7 +205,7 @@ app.get('/api/tools', (_req, res) => {
@@ -202,7 +205,7 @@ app.get('/api/tools', (_req, res) => {
});
});
// ── Main Processing Endpoint ─────────────────────────────────
// ── Main Processing Endpoint ─────────────────────────────────
-app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
+app.post('/api/process', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
try {
try {
const toolId = req.body.tool || req.ctx.defaultOp || 'convert';
const toolId = req.body.tool || req.ctx.defaultOp || 'convert';
const tool = getTool(toolId);
const tool = getTool(toolId);
@@ -229,7 +232,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)
@@ -229,7 +232,7 @@ app.post('/api/process', burstLimiter, processLimiter, guard.toolQuota(hostTool)
});
});
// ── Direct Download (inline preview) ─────────────────────────
// ── Direct Download (inline preview) ─────────────────────────
-app.post('/api/process/direct', burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
+app.post('/api/process/direct', deprecated((req) => runPath(hostTool(req))), guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(toolOf), guard.heavy(toolOf), async (req, res) => {
try {
try {
const toolId = req.body.tool || req.ctx.defaultOp || 'convert';
const toolId = req.body.tool || req.ctx.defaultOp || 'convert';
const tool = getTool(toolId);
const tool = getTool(toolId);
@@ -267,6 +270,17 @@ const jobs = jobsRuntime.setupJobs({
@@ -267,6 +270,17 @@ const jobs = jobsRuntime.setupJobs({
},
},
});
});
+// ── Run API (ADR-027): POST /api/v1/tools/:id/run for this app's tools ──
+// tools.run-request@1 (JSON, or multipart with `file` parts and the text parts input, files, wait_ms)
+// → tools.run@1: each tool runs as a job of this satellite (the preset and operation from its
+// descriptor), answered finished within wait_ms or 202 with the job. The gateway streams runs here.
+const runApi = satelliteRunApi({
+ app: 'img', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
+ multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
+ limiters: [burstLimiter, processLimiter],
+});
+app.use(runApi.handle);
+
// ── File Download ────────────────────────────────────────────
// ── File Download ────────────────────────────────────────────
app.get('/api/download/:id', (req, res) => {
app.get('/api/download/:id', (req, res) => {
const entry = retention.getFile(req.params.id);
const entry = retention.getFile(req.params.id);
@@ -19,7 +19,9 @@ const upload = multer({
@@ -19,7 +19,9 @@ const upload = multer({
files: 1,
files: 1,
},
},
fileFilter(_req, file, cb) {
fileFilter(_req, file, cb) {
- if (ALLOWED_MIMES.has(file.mimetype)) {
+ // A part without a type of its own (application/octet-stream: API clients, openvibe-sdk) is let
+ // through: the guard checks the bytes of every upload against the tool's accepted types.
+ if (ALLOWED_MIMES.has(file.mimetype) || file.mimetype === 'application/octet-stream') {
cb(null, true);
cb(null, true);
} else {
} else {
cb(new Error(`Unsupported file type: ${file.mimetype}. Accepted: ${config.upload.allowedMimes.join(', ')}`));
cb(new Error(`Unsupported file type: ${file.mimetype}. Accepted: ${config.upload.allowedMimes.join(', ')}`));
@@ -128,4 +128,9 @@ const SPECS = [
@@ -128,4 +128,9 @@ const SPECS = [
page('peso', { kind: 'text' }), page('spanish', { kind: 'text' }), page('slang', { kind: 'text' }), page('ice', { kind: 'text' }),
page('peso', { kind: 'text' }), page('spanish', { kind: 'text' }), page('slang', { kind: 'text' }), page('ice', { kind: 'text' }),
];
];
+// The run API reuses an engine's answer for 10 minutes (same tool, same input): not for these, whose
+// output is random (zalgo, the mocking case, shuffle, the bio and nickname generators).
+const FRESH = new Set(['zalgo', 'case', 'sort', 'bio', 'nickname']);
+for (const spec of SPECS) if (FRESH.has(spec.id)) spec.cacheTtlMs = 0;
+
module.exports = { SPECS };
module.exports = { SPECS };