openvibers/openvibe.tools Commits
Tool registry API (capability tools.tool.read, ADR-027): GET /api/v1/tools answers tools.tool-list@1 with schemas as $refs (filters ?family= ?execution= ?api= ?status= ?q=, comma lists; a bad value is 400 tools.query.invalid), GET /api/v1/tools/:id tools.tool@1 with its schemas embedded, GET /api/v1/tools/:id/schema { $schema, $id, $defs: { input, output } } that the $refs point at. Public and cacheable: Access-Control-Allow-Origin * for any origin (each app's own CORS rules are skipped for these paths, and even a 429 is readable), OPTIONS preflight, an ETag with 304 on If-None-Match, Cache-Control public, max-age=300, HEAD; an unknown id is 404 problem+json tools.tool.not_found, and a planned placeholder or a mirror says so. Counted by each app's existing /api/ limiter (the gateway's 120 a minute). The gateway answers for all 169 tools and adds a tool_registry readiness row (degraded, never a crash, when a descriptor fails the contracts); img, audio, docs, text, yt, maps and food answer the same routes for their own tools with their own live status (apps/_shared/tools/local.js: catalogue copy, code-default hosts, a tool unavailable while a program it needs is missing, checked every 30 s), which the gateway reads every minute. /api/catalog.json is unchanged. tools-api.test.js runs the gateway and all seven satellites: contract-valid list, tool and schema documents (the $ref resolves with a JSON Schema validator), ETag/304, CORS, preflight, 404 problems, filters, the limiter, the catalogue untouched, each satellite's own list and status (qpdf, poppler, libheif, ffmpeg and yt-dlp pointed at nothing) and the gateway picking those statuses up. README documents the routes, where descriptors live, how status is decided and the server engines
Changes
@@@ -62,6 +62,42 @@ Public, cacheable outputs on the apex: `/` (every tool by family, planned toolsDeploy with `deploy/scripts/deploy.sh` (it refreshes the copied shared package inside each app).++## Tool registry API (ADR-027)++++Every catalogue tool has one descriptor (`openvibe-contracts` `tools.tool@1`): how it runs (`execution`:++`client` in the browser, `sync` answered inline, `job`), whether the run API exposes it (`api`, `run`), its input++JSON Schema, files, output, limits, `auth` (anonymous or not; `tools.net.probe` for network probes), `quotaClass`,++`cost`, `egress`, hosts and docs page. Public, read-only, cacheable (capability `tools.tool.read`):++++| Route | Answer |++|---|---|++| `GET /api/v1/tools` | `tools.tool-list@1`, schemas as `{ $ref }`. Filters `?family=` `?execution=client\|sync\|job` `?api=true\|false` `?status=` `?q=` (comma lists allowed); a bad value → 400 `tools.query.invalid` |++| `GET /api/v1/tools/:id` | `tools.tool@1` with the schemas embedded |++| `GET /api/v1/tools/:id/schema` | `{ $schema, $id, $defs: { input, output } }` (what the list's `$ref`s point at) |++++All three answer `Access-Control-Allow-Origin: *` (the apps' own CORS rules do not apply to them), an ETag++(`If-None-Match` → 304) and `Cache-Control: public, max-age=300`, and count against the `/api/` limiter. An unknown++id is 404 problem+json `tools.tool.not_found` (planned placeholders and mirrors say what they are). The gateway++answers for every tool; each satellite answers the same routes for its own tools, with its own live status.++`/api/catalog.json` is unchanged.++++- **Specs next to the code.** `apps/<app>/server/descriptors.js` (img, audio, docs, text, yt; maps holds maps and++ food) and `apps/gateway/server/{net,dev}/descriptors.js` are pure data: the operation, job type and preset come++ from the app's domain map, net endpoints from `net/config.js`; `apps/_shared/tools/descriptor.js` joins a spec++ with the catalogue's family, name, summary and hosts (owner overrides included).++- **Checked.** `apps/gateway/server/registry/descriptors.js` merges them and checks every build with++ `contracts.validate('tools.tool@1')`, `contracts.tools.checkDescriptor` and `contracts.tools.checkList`. A failing++ descriptor is left out and `/api/ready` lists it (`tool_registry`, degraded), never a crash.++- **Status.** Unavailable when the catalogue says so (net tools without an implementation) or when the satellite++ lacks a program the tool needs (`requires`: qpdf, pdftoppm/pdfinfo, heif-dec, ffmpeg, yt-dlp). The gateway reads++ each satellite's `GET /api/v1/tools` on loopback every minute and keeps its unavailable marks.++- **Server engines for browser tools.** The pure transforms behind the dev and text pages live in++ `apps/gateway/public/js/dev-engine.js`, `apps/text/public/js/text-engine.js` and `format-engine.js`, which the++ pages load; `apps/gateway/server/dev/engines.js` and `apps/text/server/engines.js` run the same code in Node, so++ those tools say `execution: client, api: true` (the pages stay browser-only). Tools that need a DOM, a CDN++ library or isolation (XML minify, HTML to Markdown, the JS minifier/beautifier, the regex tester, the canvas++ graphics makers) say `api: false`, as does the YouTube downloader (page-only by decision).++## Jobs (Img, Audio, Docs)Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same@@@ -160,7 +196,7 @@ Every server (gateway and satellites) mounts `apps/_shared/observe.js` with `ope| Server | Required | Optional (degraded when failing) ||---|---|---|--| gateway (4001) | `catalog` | `network_key`, `service_directory` (Network registry vs fallback list), `community` (`/api/ready`), `satellite_<app>` for all seven (`/api/ready`, cached 15 s) |++| gateway (4001) | `catalog` | `network_key`, `service_directory` (Network registry vs fallback list), `community` (`/api/ready`), `satellite_<app>` for all seven (`/api/ready`, cached 15 s), `tool_registry` (every catalogue tool has a descriptor that meets the contracts) || img, docs (4012, 4016) | `jobs_db` (query), `job_runtime` (worker started), `data_dir`, `uploads_dir`, `output_dir` (write test) | `analytics_db`, `network_key`, `media_results` (only with `TOOLS_JOB_RESULTS=media`); img: `heif_decoder`; docs: `qpdf`, `pdftoppm`, `pdfinfo` || audio (4014) | same as img | same as img, plus `ffmpeg` on PATH || yt (4013) | `downloads_dir` | `analytics_db`, `yt_dlp`, `ffmpeg`, `yt_cookies` (when `YT_COOKIES_FILE` is set), `network_key` |@@@ -175,7 +211,10 @@ Every server (gateway and satellites) mounts `apps/_shared/observe.js` with `opeprocess: the job runtime end to end (restart, reattach, cancel, idempotency, owner scoping, SSE resume, pruning,Media results with a stand-in Media), img/audio/docs as real processes killed with SIGKILL mid-job, canonicalhosts on every satellite, the registry-driven catalog, and `registry-consistency.test.js`: every catalogue--tool resolves to an operation, endpoint or page that exists, or is marked unavailable. The audio tests need++tool resolves to an operation, endpoint or page that exists, or is marked unavailable. `descriptors.test.js`++holds every descriptor to the contracts and to the code (a job's operation, validation, files and limits; a sync++tool's route), `engines.test.js` runs every browser tool's server engine in plain Node against its output schema,++and `tools-api.test.js` checks the three routes on the gateway and all seven satellites. The audio tests need`ffmpeg` and skip without it. The docs and img tests always check the 503 path with qpdf, poppler and libheifpointed at nothing, and also run the real encrypt/decrypt, page rendering and HEIC decoding when those programsare installed (or `QPDF_PATH` / `HEIF_DEC_PATH` point at them). Install first with `npm run install:all`.nnew file mode 100644@@@ -0,0 +1,161 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// The tool registry routes (capability tools.tool.read, ADR-027), the same on the gateway and on++// every satellite:++//++// GET /api/v1/tools tools.tool-list@1: every descriptor, schemas as { $ref }++// ?family= ?execution=client|sync|job ?api=true|false ?status= ?q=++// GET /api/v1/tools/:id tools.tool@1 with its schemas embedded++// GET /api/v1/tools/:id/schema { $schema, $id, $defs: { input, output } }++//++// Public and cacheable: Access-Control-Allow-Origin *, an ETag (If-None-Match → 304) and++// Cache-Control: public, max-age=300. An unknown id is 404 problem+json tools.tool.not_found; a bad++// filter value is 400 tools.query.invalid. Framework-free: (req, res, next) with Express's req.path.++//++// The app's CORS middleware must not refuse these paths (exceptRegistry wraps it, and puts the++// open CORS header on anything answered earlier, a rate limiter's 429 included).++// ═══════════════════════════════════════════════════════════════++++const crypto = require('crypto');++const { refForm, schemaDoc } = require('./descriptor');++++const PATH_RE = /^\/api\/v1\/tools(?:\/([a-z][a-z0-9-]{0,39})(\/schema)?)?\/?$/;++const EXECUTIONS = new Set(['client', 'sync', 'job']);++const STATUSES = new Set(['stable', 'beta', 'preview', 'unavailable']);++const TITLES = { 400: 'Bad Request', 404: 'Not Found' };++const CACHE_MAX = 200;++++const isRegistryPath = (p) => PATH_RE.test(String(p || ''));++++function openCors(res) {++ res.setHeader('Access-Control-Allow-Origin', '*');++ res.setHeader('Access-Control-Expose-Headers', 'ETag');++ res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');++}++++/** Wrap an app's CORS middleware: registry paths skip it and are open to every origin instead. */++function exceptRegistry(corsMiddleware) {++ return function corsExceptToolRegistry(req, res, next) {++ if (!isRegistryPath(req.path)) return corsMiddleware(req, res, next);++ openCors(res);++ return next();++ };++}++++/** Does If-None-Match name this ETag (weak or strong, one of a list, or *)? */++function matches(header, etag) {++ if (!header) return false;++ const want = etag.replace(/^W\//, '');++ return String(header).split(',').some(t => { const v = t.trim(); return v === '*' || v.replace(/^W\//, '') === want; });++}++++/** Filters from the query string → { key (cache key), test(d, meta) } or { error }. */++function parseFilters(url) {++ const sp = new URL(url, 'http://registry.invalid').searchParams;++ const list = (k) => [...new Set(sp.getAll(k).flatMap(v => String(v).split(',')).map(v => v.trim().toLowerCase()).filter(Boolean))].sort();++ const family = list('family');++ const execution = list('execution');++ const status = list('status');++ const apiRaw = sp.get('api');++ const q = String(sp.get('q') || '').trim().toLowerCase().slice(0, 100);++ const bad = execution.find(e => !EXECUTIONS.has(e)) || status.find(s => !STATUSES.has(s));++ if (bad) return { error: `Unknown filter value "${bad}": execution is client, sync or job; status is stable, beta, preview or unavailable.` };++ if (apiRaw != null && apiRaw !== '' && !['true', 'false'].includes(apiRaw)) return { error: 'api is true or false.' };++ const api = apiRaw === 'true' ? true : apiRaw === 'false' ? false : null;++ const words = q.split(/\s+/).filter(Boolean);++ return {++ key: JSON.stringify([family, execution, status, api, words]),++ test(d, meta) {++ if (family.length && !family.includes(d.family)) return false;++ if (execution.length && !execution.includes(d.execution)) return false;++ if (status.length && !status.includes(d.status)) return false;++ if (api !== null && d.api !== api) return false;++ if (words.length) {++ const hay = [d.id, d.name, d.summary, ...(meta.keywords || [])].join(' ').toLowerCase();++ if (!words.every(w => hay.includes(w))) return false;++ }++ return true;++ },++ };++}++++/**++ * @param {object} o++ * @param {() => object} o.snapshot the registry now: { version, updatedAt, tools: [descriptor (embedded)],++ * meta: Map(id → { keywords }), families: Map(id → name),++ * gone: Map(id → why it is not a tool, e.g. planned) }++ */++function createToolsApi(o) {++ const cache = new Map();++ let cacheVersion = null;++++ function entry(version, key, build) {++ if (cacheVersion !== version) { cache.clear(); cacheVersion = version; }++ let e = cache.get(key);++ if (!e) {++ const body = JSON.stringify(build());++ e = { body, etag: `"${crypto.createHash('sha1').update(body).digest('base64url').slice(0, 27)}"` };++ if (cache.size >= CACHE_MAX) cache.delete(cache.keys().next().value);++ cache.set(key, e);++ }++ return e;++ }++++ function send(req, res, e, type) {++ openCors(res);++ res.setHeader('Cache-Control', 'public, max-age=300');++ res.setHeader('ETag', e.etag);++ if (matches(req.headers['if-none-match'], e.etag)) { res.statusCode = 304; return res.end(); }++ res.statusCode = 200;++ res.setHeader('Content-Type', type);++ res.setHeader('Content-Length', Buffer.byteLength(e.body));++ return res.end(req.method === 'HEAD' ? undefined : e.body);++ }++++ function problem(req, res, status, code, detail) {++ const body = JSON.stringify({ type: `https://openvibe.network/problems/${code}`, title: TITLES[status], status, code, detail, error: detail });++ openCors(res);++ res.statusCode = status;++ res.setHeader('Content-Type', 'application/problem+json');++ res.setHeader('Cache-Control', status === 404 ? 'public, max-age=60' : 'no-store');++ return res.end(req.method === 'HEAD' ? undefined : body);++ }++++ return function toolRegistry(req, res, next) {++ const m = PATH_RE.exec(req.path);++ if (!m) return next();++ if (req.method === 'OPTIONS') {++ openCors(res);++ res.setHeader('Access-Control-Allow-Methods', 'GET, HEAD, OPTIONS');++ res.setHeader('Access-Control-Allow-Headers', 'If-None-Match, Accept, Authorization');++ res.setHeader('Access-Control-Max-Age', '86400');++ res.statusCode = 204;++ return res.end();++ }++ if (req.method !== 'GET' && req.method !== 'HEAD') return next();++ const snap = o.snapshot();++ const [, id, schema] = m;++++ if (!id) {++ const f = parseFilters(req.originalUrl || req.url);++ if (f.error) return problem(req, res, 400, 'tools.query.invalid', f.error);++ const e = entry(snap.version, `list:${f.key}`, () => {++ const tools = snap.tools.filter(d => f.test(d, snap.meta.get(d.id) || {}));++ const counts = new Map();++ for (const d of tools) counts.set(d.family, (counts.get(d.family) || 0) + 1);++ return {++ tools: tools.map(refForm), count: tools.length, updated_at: snap.updatedAt,++ families: [...counts].map(([fid, count]) => ({ id: fid, name: snap.families.get(fid) || fid, count })),++ };++ });++ return send(req, res, e, 'application/json; charset=utf-8');++ }++++ const d = snap.tools.find(t => t.id === id);++ if (!d) return problem(req, res, 404, 'tools.tool.not_found', snap.gone.get(id) || `No tool is called "${id}". GET /api/v1/tools lists them all.`);++ if (schema) return send(req, res, entry(snap.version, `schema:${id}`, () => schemaDoc(d)), 'application/schema+json; charset=utf-8');++ return send(req, res, entry(snap.version, `tool:${id}`, () => d), 'application/json; charset=utf-8');++ };++}++++module.exports = { createToolsApi, exceptRegistry, isRegistryPath, parseFilters, PATH_RE };nnew file mode 100644@@@ -0,0 +1,78 @@++'use strict';++// ═══════════════════════════════════════════════════════════════++// A satellite's own tool registry: the descriptors of the tools it runs, for its GET /api/v1/tools.++//++// Specs come from the app's server/descriptors.js; name, summary, keywords and family from the++// gateway's catalogue copy (pure data files, read by path like the rest of apps/_shared); hosts are++// the code defaults (registry/hosts.js). The satellite's own live state decides status: a tool whose++// program is missing on this host (spec.requires, checked by statusOf) is 'unavailable' here first.++// The gateway's registry is the complete one: it adds the owner's domain overrides and every family.++// ═══════════════════════════════════════════════════════════════++++const crypto = require('crypto');++const path = require('path');++const { compose, summaryOf } = require('./descriptor');++++const REGISTRY = path.join(__dirname, '..', '..', 'gateway', 'server', 'registry');++const { SATELLITE_COPY } = require(path.join(REGISTRY, 'copy-satellites'));++const { FAMILIES } = require(path.join(REGISTRY, 'families'));++const { defaultHosts } = require(path.join(REGISTRY, 'hosts'));++++/** Catalogue copy of a satellite tool: { id, family, name, tagline, description, keywords } or null. */++function copyOf(id) {++ return SATELLITE_COPY.find(r => r.id === id) || null;++}++++/**++ * @param {object} o++ * @param {object[]} o.specs the app's specs (server/descriptors.js)++ * @param {(spec) => object|null} [o.statusOf] live status: { status, statusReason } or null (the spec's own)++ * @param {number} [o.liveTtlMs] how long a statusOf answer is kept (default 30 s)++ */++function createLocalRegistry(o) {++ const families = new Map(FAMILIES.map(f => [f.id, f.name]));++ const rows = o.specs.map(spec => {++ const c = copyOf(spec.id);++ if (!c) throw new Error(`tool ${spec.id} has no catalogue copy (apps/gateway/server/registry/copy-satellites.js)`);++ return { spec, c };++ });++ let built = null;++ let live = null, liveAt = 0;++ const ttl = o.liveTtlMs == null ? 30_000 : o.liveTtlMs;++++ function snapshot() {++ if (!live || Date.now() - liveAt >= ttl) { live = rows.map(({ spec }) => (o.statusOf && o.statusOf(spec)) || null); liveAt = Date.now(); }++ const sig = JSON.stringify(live);++ if (built && built.sig === sig) return built;++ const tools = rows.map(({ spec, c }, i) => {++ const h = defaultHosts(spec.id);++ return compose(spec, {++ family: c.family, name: c.name, summary: summaryOf(c),++ hosts: [h.canonical, h.short].filter(Boolean),++ ...(live[i] && { status: live[i].status, statusReason: live[i].statusReason }),++ });++ });++ const version = crypto.createHash('sha1').update(JSON.stringify(tools)).digest('hex').slice(0, 16);++ built = {++ sig, version, tools, families, gone: new Map(),++ updatedAt: built && built.version === version ? built.updatedAt : new Date().toISOString(),++ meta: new Map(rows.map(({ spec, c }) => [spec.id, { keywords: c.keywords || [] }])),++ };++ return built;++ }++++ return { snapshot };++}++++/**++ * statusOf for specs that name programs (spec.requires): unavailable while one of them is missing.++ * @param {(program: string) => boolean} available++ */++function requiresStatus(available) {++ return (spec) => {++ const missing = (spec.requires || []).filter(p => !available(p));++ return missing.length ? { status: 'unavailable', statusReason: `This tool is being set up on the server (it needs ${missing.join(' and ')}) and is not available yet.` } : null;++ };++}++++module.exports = { createLocalRegistry, requiresStatus, copyOf };@@@ -25,6 +25,8 @@ const retention = require('./retention/manager');const { probe, getDuration, cleanTmp } = require('./tools/ffmpeg-helper');const { buildOptions, describe, defineJobs } = require('./process');const { hostGuard, ownHost } = require('../../_shared/host-role');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const jobsRuntime = require('../../_shared/jobs');const contracts = require('openvibe-contracts');const sdk = require('openvibe-sdk');@@@ -44,7 +46,7 @@ const app = express();const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });// Metrics (GET /metrics, direct loopback callers only) and GET /api/ready from this server's real// dependencies (roadmap Track O). First, so the HTTP metrics see every request.--const { observe, checks: ready } = require('../../_shared/observe');++const { observe, checks: ready, which } = require('../../_shared/observe');observe({app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),service: 'tools-audio', release: release.release,@@@ -89,7 +91,7 @@ app.use(express.json({ limit: '1mb' }));app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response// ── CORS ─────────────────────────────────────────────────────--app.use(cors({++app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true);if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);@@@ -97,11 +99,17 @@ app.use(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--}));++})));// ── Rate Limiting ────────────────────────────────────────────app.use('/api/', apiLimiter);++// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's audio tools ──++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.++const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'ffmpeg' ? !!which(process.env.FFMPEG_PATH || 'ffmpeg') : true)) });++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Analytics Middleware ─────────────────────────────────────app.use(analytics.middleware());@@@ -23,6 +23,8 @@ const { apiLimiter, processLimiter, burstLimiter } = require('./middleware/rate-const retention = require('./retention/manager');const { buildOptions, defineJobs } = require('./process');const { hostGuard, ownHost } = require('../../_shared/host-role');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const jobsRuntime = require('../../_shared/jobs');const contracts = require('openvibe-contracts');const sdk = require('openvibe-sdk');@@@ -86,7 +88,7 @@ app.use(express.json({ limit: '1mb' }));app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response// ── CORS ─────────────────────────────────────────────────────--app.use(cors({++app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true);if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);@@@ -94,11 +96,18 @@ app.use(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--}));++})));// ── Rate Limiting ────────────────────────────────────────────app.use('/api/', apiLimiter);++// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's PDF tools ──++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.++const pdfPrograms = require('./tools/pdf');++const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (pdfPrograms[p] ? pdfPrograms[p].available() : true)) });++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Analytics Middleware ─────────────────────────────────────app.use(analytics.middleware());@@@ -17,6 +17,8 @@ const Database = require('better-sqlite3');const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recordedconst { internalOk } = require('../../_shared/internal-auth');const { hostGuard, stampedPage } = require('../../_shared/host-role');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const analyticsDbPath = path.join(__dirname, '..', 'data', 'analytics.db');fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });const analyticsDb = new Database(analyticsDbPath);@@@ -49,7 +51,7 @@ app.get(['/terms', '/privacy', '/dmca', '/tos'], (req, res) => res.redirect(301,// Only loopback hops (the host's nginx, the gateway) are proxies: req.ip is the first address before// them, which is what maps is told below. A client's own X-Forwarded-For is never believed.app.set('trust proxy', 'loopback');--app.use(cors({++app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true);if (/^https:\/\/[a-z0-9-]+\.openvibe\.tools$/.test(origin)) return callback(null, true);@@@ -58,7 +60,7 @@ app.use(cors({return callback(new Error('Origin not allowed'));},credentials: true,--}));++})));app.use(helmet({contentSecurityPolicy: {directives: {@@@ -73,6 +75,13 @@ app.use(helmet({},}));app.use(rateLimit({ windowMs: 60000, max: 60 }));++++// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's food finder ──++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.++// The food finder is built on the maps backend; its descriptor lives with the maps app's.++const toolRegistry = createLocalRegistry({ specs: require('../../maps/server/descriptors').SPECS.filter(s => s.id === 'food') });++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));app.use(cookieParser());// ── Analytics Middleware ─────────────────────────────────────@@@ -46,6 +46,10 @@ const SATELLITE_BY_PORT = new Map(Object.entries(SATELLITES).map(([name, port])// the host router, so on a satellite's host it is the satellite's own.const { observe, checks: ready } = require('../../_shared/observe');const registry = require('./registry');++// Every tool's descriptor (tools.tool@1, ADR-027) for GET /api/v1/tools[/:id[/schema]]; checked with++// openvibe-contracts on every build, a failure is the tool_registry readiness row, never a crash.++const toolRegistry = require('./registry/descriptors');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');let auth = null; // created below; the key check reads it at request timeconst obs = observe({app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),@@@ -68,6 +72,7 @@ const obs = observe({check: () => { const s = registry.services.status(); return s.source === 'registry' ? { ok: true, detail: { as_of: s.as_of, last_ok: s.last_ok } } : `using the built-in fallback list${s.last_error ? ` (${s.last_error})` : ''}`; },},ready.upstream('community', `${config.communityUrl}/api/ready`, { description: 'paste API proxy (pastes.openvibe.tools)' }),++ toolRegistry.readyCheck(),...Object.entries(SATELLITES).map(([name, port]) => ready.upstream(`satellite_${name}`, `http://127.0.0.1:${port}/api/ready`, { description: `${name} satellite (port ${port}); its hosts fail without it` })),],});@@@ -140,19 +145,26 @@ function isAllowedOrigin(origin) {// preflight before the wider allow-list below gets a chance to.app.use('/auth/fedcm', fedcmCors);--app.use(cors({++// The tool registry (/api/v1/tools…) is public and read-only: open to every origin, no credentials.++app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true); // curl / server-to-serverif (isAllowedOrigin(origin)) return callback(null, true);return callback(new Error('Origin not allowed by CORS'));},credentials: true,--}));++})));// ── Rate Limiting ────────────────────────────────────────────app.use('/api/', rateLimit({ windowMs: 60_000, max: 120 }));app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60 }));++// ── Tool registry (ADR-027, capability tools.tool.read) ──────++// GET /api/v1/tools (tools.tool-list@1), /api/v1/tools/:id (tools.tool@1), /api/v1/tools/:id/schema.++// Every tool, every family; satellite hosts answer the same routes for their own tools. Counted by++// the /api/ limiter above. /api/catalog.json stays as it is (Network reads it).++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Auth (OAuth2 client of OpenVibe.Network) ─────────────────auth = createAuthClient(config);app.locals.auth = auth;@@@ -347,6 +359,9 @@ app.get('*', (req, res) => {});// ── Start ────────────────────────────────────────────────────++// What each satellite says about its own tools (a program missing on the host → unavailable here too).++toolRegistry.startSatellitePolling(SATELLITES);++app.listen(config.port, config.host, () => {console.log(`\n╔════════════════════════════════════════════╗`);console.log(`║ 🔧 openvibe.tools — Tools Gateway ║`);nnew file mode 100644@@@ -0,0 +1,203 @@++'use strict';++// GET /api/v1/tools, /api/v1/tools/:id and /api/v1/tools/:id/schema (ADR-027, tools.tool.read), on++// real processes: the gateway and all seven satellites.++// • the answers are contract documents: tools.tool-list@1 (checkList), tools.tool@1 (checkDescriptor)++// with embedded schemas, and a schema document whose $defs the list's $refs point at++// • open and cacheable: Access-Control-Allow-Origin * for any origin (the gateway's own CORS never++// refuses them), OPTIONS preflight, ETag with 304 on If-None-Match, Cache-Control public 300 s, HEAD++// • unknown ids are 404 problem+json tools.tool.not_found (planned and mirrors say so); a bad filter 400++// • counted by the gateway's /api/ limiter; /api/catalog.json is unchanged++// • each satellite answers for its own tools, with its own live status (a missing program → unavailable),++// and the gateway picks that status up++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const contracts = require('openvibe-contracts');++const Ajv2020 = require('ajv/dist/2020');++const addFormats = require('ajv-formats');++const { startApp } = require('../../_shared/test/spawn');++++const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-tools-api-'));++const MISSING = path.join(tmp, 'not-installed');++const get = async (url, headers = {}, method = 'GET') => {++ const r = await fetch(url, { method, headers });++ const text = await r.text();++ let body = null; try { body = text ? JSON.parse(text) : null; } catch { /* not JSON */ }++ return { status: r.status, h: r.headers, text, body };++};++++(async () => {++ const procs = [];++ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };++ try {++ // Satellites, each missing the program one of its tools needs.++ const sat = {++ img: await start('img', { DATA_DIR: path.join(tmp, 'img'), HEIF_DEC_PATH: MISSING }),++ docs: await start('docs', { DATA_DIR: path.join(tmp, 'docs'), QPDF_PATH: MISSING, PDFTOPPM_PATH: MISSING, PDFINFO_PATH: MISSING }),++ audio: await start('audio', { DATA_DIR: path.join(tmp, 'audio'), FFMPEG_PATH: MISSING }),++ yt: await start('yt', { DATA_DIR: path.join(tmp, 'yt'), DOWNLOADS_DIR: path.join(tmp, 'yt', 'dl'), YTDLP_PATH: MISSING }),++ text: await start('text', { DATA_DIR: path.join(tmp, 'text') }),++ maps: await start('maps', { DATA_DIR: path.join(tmp, 'maps') }, undefined, { readyPath: '/api/ready' }),++ };++ sat.food = await start('food', { DATA_DIR: path.join(tmp, 'food'), MAPS_API: sat.maps.base }, undefined, { readyPath: '/api/ready' });++ const ports = Object.entries(sat).map(([n, p]) => `${n}=${p.port}`).join(',');++ const gw = await start('gateway', { TOOLS_SATELLITE_PORTS: ports, OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' });++ const G = gw.base;++++ // ── The list: a tools.tool-list@1 document with every tool ──++ let r = await get(`${G}/api/v1/tools`, { Origin: 'https://someone-elses-site.example' });++ assert.strictEqual(r.status, 200, r.text.slice(0, 300));++ assert.match(r.h.get('content-type'), /^application\/json/);++ assert.strictEqual(r.h.get('access-control-allow-origin'), '*', 'open to any origin');++ assert.ok(!r.h.get('access-control-allow-credentials'), 'no credentials');++ assert.strictEqual(r.h.get('cache-control'), 'public, max-age=300');++ assert.ok(r.h.get('x-ratelimit-limit') || r.h.get('ratelimit-limit') || r.h.get('ratelimit-policy'), 'counted by the gateway\'s /api/ limiter');++ const etag = r.h.get('etag');++ assert.match(etag, /^"[A-Za-z0-9_-]+"$/);++ const list = r.body;++ const lc = contracts.tools.checkList(list);++ assert.ok(lc.valid, JSON.stringify(lc.errors.slice(0, 5)));++ assert.strictEqual(list.count, 169);++ assert.ok(list.tools.every(d => !d.input || d.input.$ref), 'inputs are $refs in the list');++ assert.ok(!Number.isNaN(Date.parse(list.updated_at)));++ assert.deepStrictEqual(list.families.map(f => f.id).sort(), ['audio', 'dev', 'docs', 'img', 'media', 'net', 'pastes', 'places', 'text']);++++ // ETag → 304 (strong, weak, in a list); HEAD; preflight.++ assert.strictEqual((await get(`${G}/api/v1/tools`, { 'If-None-Match': etag })).status, 304);++ assert.strictEqual((await get(`${G}/api/v1/tools`, { 'If-None-Match': `"x", W/${etag}` })).status, 304);++ r = await get(`${G}/api/v1/tools`, {}, 'HEAD');++ assert.deepStrictEqual([r.status, r.text, r.h.get('etag')], [200, '', etag]);++ r = await get(`${G}/api/v1/tools/png`, { Origin: 'https://a.example', 'Access-Control-Request-Method': 'GET', 'Access-Control-Request-Headers': 'if-none-match' }, 'OPTIONS');++ assert.strictEqual(r.status, 204);++ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');++ assert.match(r.h.get('access-control-allow-headers'), /If-None-Match/);++++ // Filters.++ const count = async (q) => (await get(`${G}/api/v1/tools?${q}`)).body;++ let l = await count('family=img');++ assert.strictEqual(l.count, 15); assert.deepStrictEqual(l.families, [{ id: 'img', name: 'Image Tools', count: 15 }]);++ assert.ok(contracts.tools.checkList(l).valid);++ l = await count('execution=client&api=true');++ assert.ok(l.tools.length >= 40 && l.tools.every(d => d.execution === 'client' && d.api));++ l = await count('api=false');++ assert.ok(l.tools.some(d => d.id === 'yt') && l.tools.every(d => !d.api && d.run === null));++ l = await count('q=json%20minify');++ assert.ok(l.tools.some(d => d.id === 'jsonminify'), 'text search');++ l = await count('family=net,dev&execution=sync');++ assert.ok(l.tools.every(d => ['net', 'dev'].includes(d.family) && d.execution === 'sync') && l.tools.some(d => d.id === 'opengraph'));++ r = await get(`${G}/api/v1/tools?execution=later`);++ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.query.invalid']);++ assert.match(r.h.get('content-type'), /^application\/problem\+json/);++ r = await get(`${G}/api/v1/tools?api=yes`);++ assert.strictEqual(r.status, 400);++++ // ── One tool: tools.tool@1 with its schemas embedded ──++ r = await get(`${G}/api/v1/tools/png`);++ assert.strictEqual(r.status, 200);++ const png = r.body;++ assert.ok(contracts.validate('tools.tool@1', png).valid);++ assert.ok(contracts.tools.checkDescriptor(png).valid);++ assert.strictEqual(png.input.type, 'object', 'embedded');++ assert.deepStrictEqual(png.run.job, { type: 'img.process', operation: 'convert', preset: { defaultFormat: 'png' } });++ assert.strictEqual((await get(`${G}/api/v1/tools/png`, { 'If-None-Match': r.h.get('etag') })).status, 304);++ for (const id of ['dns', 'port', 'jsonminify', 'yt', 'logo', 'mergepdf', 'pastes', 'maps']) {++ const d = (await get(`${G}/api/v1/tools/${id}`)).body;++ assert.ok(contracts.tools.checkDescriptor(d).valid, `${id}: ${JSON.stringify(contracts.tools.checkDescriptor(d).errors)}`);++ }++++ // ── The schema document the $refs point at ──++ const ref = list.tools.find(d => d.id === 'dns').input.$ref;++ assert.strictEqual(ref, 'https://openvibe.tools/api/v1/tools/dns/schema#/$defs/input');++ r = await get(`${G}${new URL(ref).pathname}`);++ assert.strictEqual(r.status, 200);++ assert.match(r.h.get('content-type'), /^application\/schema\+json/);++ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');++ const doc = r.body;++ assert.strictEqual(doc.$schema, 'https://json-schema.org/draft/2020-12/schema');++ assert.strictEqual(doc.$id, 'https://openvibe.tools/api/v1/tools/dns/schema');++ const dns = (await get(`${G}/api/v1/tools/dns`)).body;++ assert.deepStrictEqual(doc.$defs.input, dns.input);++ assert.deepStrictEqual(doc.$defs.output, dns.output.schema);++ // The $ref resolves with a JSON Schema validator, as a client would use it.++ const ajv = new Ajv2020({ strict: false }); addFormats(ajv);++ ajv.addSchema(doc);++ const v = ajv.compile({ $ref: ref });++ assert.ok(v({ target: 'example.com', types: ['MX'] }));++ assert.ok(!v({ types: ['MX'] }), 'target is required');++ assert.strictEqual((await get(`${G}/api/v1/tools/yt/schema`)).body.$defs.input, false, 'no API, no input');++++ // ── Not tools ──++ for (const [id, why] of [['no-such-tool', /No tool is called/], ['qr', /planned/], ['jsonfmt', /mirror of json|second build of json/]]) {++ r = await get(`${G}/api/v1/tools/${id}`);++ assert.strictEqual(r.status, 404, id);++ assert.match(r.h.get('content-type'), /^application\/problem\+json/);++ assert.strictEqual(r.body.code, 'tools.tool.not_found');++ assert.strictEqual(r.body.status, 404);++ assert.match(r.body.detail, why);++ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');++ assert.strictEqual((await get(`${G}/api/v1/tools/${id}/schema`)).status, 404);++ }++ assert.strictEqual((await get(`${G}/api/v1/tools/Bad_Id`)).status, 404);++++ // ── /api/catalog.json is unchanged (Network reads it) ──++ const catalog = (await get(`${G}/api/catalog.json`)).body;++ assert.strictEqual(catalog.tools.length, 169);++ assert.ok(catalog.tools.every(t => ['available', 'unavailable'].includes(t.status) && t.hosts && t.hosts.canonical));++ assert.ok(!('api' in catalog.tools[0]) && !('execution' in catalog.tools[0]), 'no descriptor fields leak into the catalogue');++++ // ── Each satellite answers for its own tools, with its own status ──++ const own = { img: 'img', docs: 'docs', audio: 'audio', yt: 'media', text: 'text', maps: 'places', food: 'places' };++ const counts = {};++ for (const [name, p] of Object.entries(sat)) {++ r = await get(`${p.base}/api/v1/tools`, { Origin: 'https://elsewhere.example' });++ assert.strictEqual(r.status, 200, `${name}: ${r.text.slice(0, 200)}`);++ assert.strictEqual(r.h.get('access-control-allow-origin'), '*', `${name}: open CORS`);++ assert.ok(r.h.get('etag'), `${name}: ETag`);++ assert.strictEqual((await get(`${p.base}/api/v1/tools`, { 'If-None-Match': r.h.get('etag') })).status, 304, `${name}: 304`);++ const c = contracts.tools.checkList(r.body);++ assert.ok(c.valid, `${name}: ${JSON.stringify(c.errors.slice(0, 3))}`);++ assert.ok(r.body.tools.length && r.body.tools.every(d => d.family === own[name]), `${name}: only its own tools`);++ counts[name] = r.body.count;++ const first = r.body.tools[0].id;++ const one = await get(`${p.base}/api/v1/tools/${first}`);++ assert.ok(contracts.tools.checkDescriptor(one.body).valid, `${name}: /api/v1/tools/${first}`);++ assert.strictEqual((await get(`${p.base}/api/v1/tools/${first}/schema`)).status, 200);++ assert.strictEqual((await get(`${p.base}/api/v1/tools/png-nope`)).status, 404);++ }++ assert.deepStrictEqual(counts, { img: 15, docs: 10, audio: 36, yt: 1, text: 33, maps: 1, food: 1 });++ const status = async (base, id) => (await get(`${base}/api/v1/tools/${id}`)).body;++ let d = await status(sat.docs.base, 'protectpdf');++ assert.strictEqual(d.status, 'unavailable'); assert.match(d.statusReason, /qpdf/);++ assert.strictEqual((await status(sat.docs.base, 'pdf2jpg')).status, 'unavailable');++ assert.strictEqual((await status(sat.docs.base, 'mergepdf')).status, 'stable');++ assert.strictEqual((await status(sat.img.base, 'heic')).status, 'unavailable', 'HEIC without libheif');++ assert.strictEqual((await status(sat.img.base, 'png')).status, 'stable');++ assert.strictEqual((await status(sat.audio.base, 'mp3')).status, 'unavailable', 'audio without ffmpeg');++ assert.strictEqual((await status(sat.yt.base, 'yt')).status, 'unavailable', 'yt without yt-dlp');++ assert.strictEqual((await status(sat.text.base, 'logo')).status, 'stable');++++ // The gateway polled them at boot: the same statuses in the full registry.++ for (let i = 0; i < 50 && (await status(G, 'protectpdf')).status !== 'unavailable'; i++) await new Promise(res => setTimeout(res, 100));++ for (const id of ['protectpdf', 'unlockpdf', 'pdf2jpg', 'heic', 'mp3', 'merge', 'yt']) assert.strictEqual((await status(G, id)).status, 'unavailable', `gateway: ${id}`);++ for (const id of ['png', 'mergepdf', 'fancy', 'dns']) assert.strictEqual((await status(G, id)).status, 'stable', `gateway: ${id}`);++ l = await count('status=unavailable');++ assert.ok(l.tools.length >= 3 + 36 + 5 && contracts.tools.checkList(l).valid, 'status filter');++++ // Readiness: the registry is checked, and it passes.++ const ready = (await get(`${G}/api/ready`)).body;++ assert.strictEqual(ready.checks.tool_registry.status, 'ok', JSON.stringify(ready.checks.tool_registry));++ assert.strictEqual(ready.checks.tool_registry.required, false);++++ // ── The gateway's /api/ limiter counts registry reads (120 a minute per address) ──++ let limited = null;++ for (let i = 0; i < 130 && !limited; i++) { const x = await fetch(`${G}/api/v1/tools/png`); if (x.status === 429) limited = x; }++ assert.ok(limited, 'the 121st request in a minute is refused');++ assert.strictEqual(limited.headers.get('access-control-allow-origin'), '*', 'even a 429 is readable cross-origin');++++ console.log(`tools API: gateway ${list.count} tools (list, one, schema, ETag/304, CORS *, 404 problem, filters, limiter); satellites ${Object.entries(counts).map(([n, c]) => `${n} ${c}`).join(', ')} with live status`);++ } finally {++ await Promise.all(procs.map(p => p.kill()));++ fs.rmSync(tmp, { recursive: true, force: true });++ }++})().catch((err) => { console.error(err); process.exit(1); });@@@ -24,6 +24,8 @@ const retention = require('./retention/manager');const { buildOptions, describe, processBuffer, defineJobs } = require('./process');const codec = require('./tools/codec');const { hostGuard, ownHost } = require('../../_shared/host-role');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const jobsRuntime = require('../../_shared/jobs');const contracts = require('openvibe-contracts');const sdk = require('openvibe-sdk');@@@ -87,7 +89,7 @@ app.use(express.json({ limit: '1mb' }));app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response// ── CORS ─────────────────────────────────────────────────────--app.use(cors({++app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true);// Allow any *.openvibe.tools subdomain + openvibe.tools itself@@@ -96,11 +98,17 @@ app.use(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--}));++})));// ── Rate Limiting ────────────────────────────────────────────app.use('/api/', apiLimiter);++// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's image tools ──++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.++const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'heif-dec' ? codec.heif.available() : true)) });++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Analytics Middleware ─────────────────────────────────────app.use(analytics.middleware());@@@ -22,6 +22,8 @@ const Database = require('better-sqlite3');const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recordedconst { internalOk } = require('../../_shared/internal-auth');const { hostGuard, stampedPage } = require('../../_shared/host-role');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const analyticsDbPath = path.join(__dirname, '..', 'data', 'analytics.db');fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });const analyticsDb = new Database(analyticsDbPath);@@@ -67,7 +69,7 @@ app.use(helmet({},},}));--app.use(cors({ origin: ['https://maps.openvibe.tools', 'https://food.openvibe.tools', 'https://openvibe.network', 'http://localhost:4010', 'http://localhost:4011'] }));++app.use(exceptRegistry(cors({ origin: ['https://maps.openvibe.tools', 'https://food.openvibe.tools', 'https://openvibe.network', 'http://localhost:4010', 'http://localhost:4011'] })));app.use(cookieParser());app.use(express.json());@@@ -79,6 +81,12 @@ const apiLimiter = rateLimit({});app.use('/api/', apiLimiter);++// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's survival map ──++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.++const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS.filter(s => s.id === 'maps') });++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Analytics Middleware ─────────────────────────────────────app.use(analytics.middleware());@@@ -15,6 +15,8 @@ const Database = require('better-sqlite3');const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recordedconst { internalOk } = require('../../_shared/internal-auth');const { hostGuard } = require('../../_shared/host-role');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');const analyticsDbPath = path.resolve(__dirname, '..', process.env.DATA_DIR || 'data', 'analytics.db');fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });const analyticsDb = new Database(analyticsDbPath);@@@ -60,7 +62,7 @@ app.use(cookieParser());app.use(express.json({ limit: '256kb' }));// ── CORS ─────────────────────────────────────────────────────--app.use(cors({++app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true);if (/^https:\/\/[a-z0-9-]+\.openvibe\.tools$/.test(origin)) return callback(null, true);@@@ -69,11 +71,17 @@ app.use(cors({return callback(new Error('Origin not allowed'));},credentials: true,--}));++})));// ── Rate Limiting ────────────────────────────────────────────app.use(rateLimit({ windowMs: 60_000, max: 200 }));++// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's text and logo tools ──++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.++const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS });++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Analytics Middleware ─────────────────────────────────────app.use(analytics.middleware());@@@ -18,6 +18,8 @@ const { optionalAuth } = require('./auth');const downloader = require('./downloader');const seo = require('./seo');const { hostGuard } = require('../../_shared/host-role');++const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');++const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');// ── Analytics ────────────────────────────────────────────────const Database = require('better-sqlite3');@@@ -34,7 +36,7 @@ const app = express();const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });// Metrics (GET /metrics, direct loopback callers only) and GET /api/ready from this server's real// dependencies (roadmap Track O). First, so the HTTP metrics see every request.--const { observe, checks: ready } = require('../../_shared/observe');++const { observe, checks: ready, which } = require('../../_shared/observe');observe({app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),service: 'tools-yt', release: release.release,@@@ -78,7 +80,7 @@ app.use(cookieParser());app.use(express.json({ limit: '1mb' }));// ── CORS ─────────────────────────────────────────────────────--app.use(cors({++app.use(exceptRegistry(cors({origin(origin, callback) {if (!origin) return callback(null, true);if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);@@@ -86,7 +88,7 @@ app.use(cors({return callback(new Error('Origin not allowed by CORS'));},credentials: true,--}));++})));// ── Rate Limiting ────────────────────────────────────────────// Progress is one SSE connection, or a poll every 1–3 s when SSE is unavailable; the poll must@@@ -96,6 +98,12 @@ app.use('/api/', rateLimit({ windowMs: 60_000, max: 60, skip: isStatusRoute }));app.use('/api/status/', rateLimit({ windowMs: 60_000, max: 240 }));app.use('/api/', (_req, res, next) => { res.setHeader('Cache-Control', 'no-store'); next(); });++// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's YouTube downloader ──++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.++const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'yt-dlp' ? !!which(config.ytdlpPath) : true)) });++app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));++// ── Analytics Middleware ─────────────────────────────────────app.use(analytics.middleware());
@@ -62,6 +62,42 @@ Public, cacheable outputs on the apex: `/` (every tool by family, planned tools
@@ -62,6 +62,42 @@ Public, cacheable outputs on the apex: `/` (every tool by family, planned tools
Deploy with `deploy/scripts/deploy.sh` (it refreshes the copied shared package inside each app).
Deploy with `deploy/scripts/deploy.sh` (it refreshes the copied shared package inside each app).
+## Tool registry API (ADR-027)
+
+Every catalogue tool has one descriptor (`openvibe-contracts` `tools.tool@1`): how it runs (`execution`:
+`client` in the browser, `sync` answered inline, `job`), whether the run API exposes it (`api`, `run`), its input
+JSON Schema, files, output, limits, `auth` (anonymous or not; `tools.net.probe` for network probes), `quotaClass`,
+`cost`, `egress`, hosts and docs page. Public, read-only, cacheable (capability `tools.tool.read`):
+
+| Route | Answer |
+|---|---|
+| `GET /api/v1/tools` | `tools.tool-list@1`, schemas as `{ $ref }`. Filters `?family=` `?execution=client\|sync\|job` `?api=true\|false` `?status=` `?q=` (comma lists allowed); a bad value → 400 `tools.query.invalid` |
+| `GET /api/v1/tools/:id` | `tools.tool@1` with the schemas embedded |
+| `GET /api/v1/tools/:id/schema` | `{ $schema, $id, $defs: { input, output } }` (what the list's `$ref`s point at) |
+
+All three answer `Access-Control-Allow-Origin: *` (the apps' own CORS rules do not apply to them), an ETag
+(`If-None-Match` → 304) and `Cache-Control: public, max-age=300`, and count against the `/api/` limiter. An unknown
+id is 404 problem+json `tools.tool.not_found` (planned placeholders and mirrors say what they are). The gateway
+answers for every tool; each satellite answers the same routes for its own tools, with its own live status.
+`/api/catalog.json` is unchanged.
+
+- **Specs next to the code.** `apps/<app>/server/descriptors.js` (img, audio, docs, text, yt; maps holds maps and
+ food) and `apps/gateway/server/{net,dev}/descriptors.js` are pure data: the operation, job type and preset come
+ from the app's domain map, net endpoints from `net/config.js`; `apps/_shared/tools/descriptor.js` joins a spec
+ with the catalogue's family, name, summary and hosts (owner overrides included).
+- **Checked.** `apps/gateway/server/registry/descriptors.js` merges them and checks every build with
+ `contracts.validate('tools.tool@1')`, `contracts.tools.checkDescriptor` and `contracts.tools.checkList`. A failing
+ descriptor is left out and `/api/ready` lists it (`tool_registry`, degraded), never a crash.
+- **Status.** Unavailable when the catalogue says so (net tools without an implementation) or when the satellite
+ lacks a program the tool needs (`requires`: qpdf, pdftoppm/pdfinfo, heif-dec, ffmpeg, yt-dlp). The gateway reads
+ each satellite's `GET /api/v1/tools` on loopback every minute and keeps its unavailable marks.
+- **Server engines for browser tools.** The pure transforms behind the dev and text pages live in
+ `apps/gateway/public/js/dev-engine.js`, `apps/text/public/js/text-engine.js` and `format-engine.js`, which the
+ pages load; `apps/gateway/server/dev/engines.js` and `apps/text/server/engines.js` run the same code in Node, so
+ those tools say `execution: client, api: true` (the pages stay browser-only). Tools that need a DOM, a CDN
+ library or isolation (XML minify, HTML to Markdown, the JS minifier/beautifier, the regex tester, the canvas
+ graphics makers) say `api: false`, as does the YouTube downloader (page-only by decision).
+
## Jobs (Img, Audio, Docs)
## Jobs (Img, Audio, Docs)
Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same
Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same
@@ -160,7 +196,7 @@ Every server (gateway and satellites) mounts `apps/_shared/observe.js` with `ope
@@ -160,7 +196,7 @@ Every server (gateway and satellites) mounts `apps/_shared/observe.js` with `ope
| Server | Required | Optional (degraded when failing) |
| Server | Required | Optional (degraded when failing) |
|---|---|---|
|---|---|---|
-| gateway (4001) | `catalog` | `network_key`, `service_directory` (Network registry vs fallback list), `community` (`/api/ready`), `satellite_<app>` for all seven (`/api/ready`, cached 15 s) |
+| gateway (4001) | `catalog` | `network_key`, `service_directory` (Network registry vs fallback list), `community` (`/api/ready`), `satellite_<app>` for all seven (`/api/ready`, cached 15 s), `tool_registry` (every catalogue tool has a descriptor that meets the contracts) |
| img, docs (4012, 4016) | `jobs_db` (query), `job_runtime` (worker started), `data_dir`, `uploads_dir`, `output_dir` (write test) | `analytics_db`, `network_key`, `media_results` (only with `TOOLS_JOB_RESULTS=media`); img: `heif_decoder`; docs: `qpdf`, `pdftoppm`, `pdfinfo` |
| img, docs (4012, 4016) | `jobs_db` (query), `job_runtime` (worker started), `data_dir`, `uploads_dir`, `output_dir` (write test) | `analytics_db`, `network_key`, `media_results` (only with `TOOLS_JOB_RESULTS=media`); img: `heif_decoder`; docs: `qpdf`, `pdftoppm`, `pdfinfo` |
| audio (4014) | same as img | same as img, plus `ffmpeg` on PATH |
| audio (4014) | same as img | same as img, plus `ffmpeg` on PATH |
| yt (4013) | `downloads_dir` | `analytics_db`, `yt_dlp`, `ffmpeg`, `yt_cookies` (when `YT_COOKIES_FILE` is set), `network_key` |
| yt (4013) | `downloads_dir` | `analytics_db`, `yt_dlp`, `ffmpeg`, `yt_cookies` (when `YT_COOKIES_FILE` is set), `network_key` |
@@ -175,7 +211,10 @@ Every server (gateway and satellites) mounts `apps/_shared/observe.js` with `ope
@@ -175,7 +211,10 @@ Every server (gateway and satellites) mounts `apps/_shared/observe.js` with `ope
process: the job runtime end to end (restart, reattach, cancel, idempotency, owner scoping, SSE resume, pruning,
process: the job runtime end to end (restart, reattach, cancel, idempotency, owner scoping, SSE resume, pruning,
Media results with a stand-in Media), img/audio/docs as real processes killed with SIGKILL mid-job, canonical
Media results with a stand-in Media), img/audio/docs as real processes killed with SIGKILL mid-job, canonical
hosts on every satellite, the registry-driven catalog, and `registry-consistency.test.js`: every catalogue
hosts on every satellite, the registry-driven catalog, and `registry-consistency.test.js`: every catalogue
-tool resolves to an operation, endpoint or page that exists, or is marked unavailable. The audio tests need
+tool resolves to an operation, endpoint or page that exists, or is marked unavailable. `descriptors.test.js`
+holds every descriptor to the contracts and to the code (a job's operation, validation, files and limits; a sync
+tool's route), `engines.test.js` runs every browser tool's server engine in plain Node against its output schema,
+and `tools-api.test.js` checks the three routes on the gateway and all seven satellites. The audio tests need
`ffmpeg` and skip without it. The docs and img tests always check the 503 path with qpdf, poppler and libheif
`ffmpeg` and skip without it. The docs and img tests always check the 503 path with qpdf, poppler and libheif
pointed at nothing, and also run the real encrypt/decrypt, page rendering and HEIC decoding when those programs
pointed at nothing, and also run the real encrypt/decrypt, page rendering and HEIC decoding when those programs
are installed (or `QPDF_PATH` / `HEIF_DEC_PATH` point at them). Install first with `npm run install:all`.
are installed (or `QPDF_PATH` / `HEIF_DEC_PATH` point at them). Install first with `npm run install:all`.
new file mode 100644
new file mode 100644
@@ -0,0 +1,161 @@
@@ -0,0 +1,161 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// The tool registry routes (capability tools.tool.read, ADR-027), the same on the gateway and on
+// every satellite:
+//
+// GET /api/v1/tools tools.tool-list@1: every descriptor, schemas as { $ref }
+// ?family= ?execution=client|sync|job ?api=true|false ?status= ?q=
+// GET /api/v1/tools/:id tools.tool@1 with its schemas embedded
+// GET /api/v1/tools/:id/schema { $schema, $id, $defs: { input, output } }
+//
+// Public and cacheable: Access-Control-Allow-Origin *, an ETag (If-None-Match → 304) and
+// Cache-Control: public, max-age=300. An unknown id is 404 problem+json tools.tool.not_found; a bad
+// filter value is 400 tools.query.invalid. Framework-free: (req, res, next) with Express's req.path.
+//
+// The app's CORS middleware must not refuse these paths (exceptRegistry wraps it, and puts the
+// open CORS header on anything answered earlier, a rate limiter's 429 included).
+// ═══════════════════════════════════════════════════════════════
+
+const crypto = require('crypto');
+const { refForm, schemaDoc } = require('./descriptor');
+
+const PATH_RE = /^\/api\/v1\/tools(?:\/([a-z][a-z0-9-]{0,39})(\/schema)?)?\/?$/;
+const EXECUTIONS = new Set(['client', 'sync', 'job']);
+const STATUSES = new Set(['stable', 'beta', 'preview', 'unavailable']);
+const TITLES = { 400: 'Bad Request', 404: 'Not Found' };
+const CACHE_MAX = 200;
+
+const isRegistryPath = (p) => PATH_RE.test(String(p || ''));
+
+function openCors(res) {
+ res.setHeader('Access-Control-Allow-Origin', '*');
+ res.setHeader('Access-Control-Expose-Headers', 'ETag');
+ res.setHeader('Cross-Origin-Resource-Policy', 'cross-origin');
+}
+
+/** Wrap an app's CORS middleware: registry paths skip it and are open to every origin instead. */
+function exceptRegistry(corsMiddleware) {
+ return function corsExceptToolRegistry(req, res, next) {
+ if (!isRegistryPath(req.path)) return corsMiddleware(req, res, next);
+ openCors(res);
+ return next();
+ };
+}
+
+/** Does If-None-Match name this ETag (weak or strong, one of a list, or *)? */
+function matches(header, etag) {
+ if (!header) return false;
+ const want = etag.replace(/^W\//, '');
+ return String(header).split(',').some(t => { const v = t.trim(); return v === '*' || v.replace(/^W\//, '') === want; });
+}
+
+/** Filters from the query string → { key (cache key), test(d, meta) } or { error }. */
+function parseFilters(url) {
+ const sp = new URL(url, 'http://registry.invalid').searchParams;
+ const list = (k) => [...new Set(sp.getAll(k).flatMap(v => String(v).split(',')).map(v => v.trim().toLowerCase()).filter(Boolean))].sort();
+ const family = list('family');
+ const execution = list('execution');
+ const status = list('status');
+ const apiRaw = sp.get('api');
+ const q = String(sp.get('q') || '').trim().toLowerCase().slice(0, 100);
+ const bad = execution.find(e => !EXECUTIONS.has(e)) || status.find(s => !STATUSES.has(s));
+ if (bad) return { error: `Unknown filter value "${bad}": execution is client, sync or job; status is stable, beta, preview or unavailable.` };
+ if (apiRaw != null && apiRaw !== '' && !['true', 'false'].includes(apiRaw)) return { error: 'api is true or false.' };
+ const api = apiRaw === 'true' ? true : apiRaw === 'false' ? false : null;
+ const words = q.split(/\s+/).filter(Boolean);
+ return {
+ key: JSON.stringify([family, execution, status, api, words]),
+ test(d, meta) {
+ if (family.length && !family.includes(d.family)) return false;
+ if (execution.length && !execution.includes(d.execution)) return false;
+ if (status.length && !status.includes(d.status)) return false;
+ if (api !== null && d.api !== api) return false;
+ if (words.length) {
+ const hay = [d.id, d.name, d.summary, ...(meta.keywords || [])].join(' ').toLowerCase();
+ if (!words.every(w => hay.includes(w))) return false;
+ }
+ return true;
+ },
+ };
+}
+
+/**
+ * @param {object} o
+ * @param {() => object} o.snapshot the registry now: { version, updatedAt, tools: [descriptor (embedded)],
+ * meta: Map(id → { keywords }), families: Map(id → name),
+ * gone: Map(id → why it is not a tool, e.g. planned) }
+ */
+function createToolsApi(o) {
+ const cache = new Map();
+ let cacheVersion = null;
+
+ function entry(version, key, build) {
+ if (cacheVersion !== version) { cache.clear(); cacheVersion = version; }
+ let e = cache.get(key);
+ if (!e) {
+ const body = JSON.stringify(build());
+ e = { body, etag: `"${crypto.createHash('sha1').update(body).digest('base64url').slice(0, 27)}"` };
+ if (cache.size >= CACHE_MAX) cache.delete(cache.keys().next().value);
+ cache.set(key, e);
+ }
+ return e;
+ }
+
+ function send(req, res, e, type) {
+ openCors(res);
+ res.setHeader('Cache-Control', 'public, max-age=300');
+ res.setHeader('ETag', e.etag);
+ if (matches(req.headers['if-none-match'], e.etag)) { res.statusCode = 304; return res.end(); }
+ res.statusCode = 200;
+ res.setHeader('Content-Type', type);
+ res.setHeader('Content-Length', Buffer.byteLength(e.body));
+ return res.end(req.method === 'HEAD' ? undefined : e.body);
+ }
+
+ function problem(req, res, status, code, detail) {
+ const body = JSON.stringify({ type: `https://openvibe.network/problems/${code}`, title: TITLES[status], status, code, detail, error: detail });
+ openCors(res);
+ res.statusCode = status;
+ res.setHeader('Content-Type', 'application/problem+json');
+ res.setHeader('Cache-Control', status === 404 ? 'public, max-age=60' : 'no-store');
+ return res.end(req.method === 'HEAD' ? undefined : body);
+ }
+
+ return function toolRegistry(req, res, next) {
+ const m = PATH_RE.exec(req.path);
+ if (!m) return next();
+ if (req.method === 'OPTIONS') {
+ openCors(res);
+ res.setHeader('Access-Control-Allow-Methods', 'GET, HEAD, OPTIONS');
+ res.setHeader('Access-Control-Allow-Headers', 'If-None-Match, Accept, Authorization');
+ res.setHeader('Access-Control-Max-Age', '86400');
+ res.statusCode = 204;
+ return res.end();
+ }
+ if (req.method !== 'GET' && req.method !== 'HEAD') return next();
+ const snap = o.snapshot();
+ const [, id, schema] = m;
+
+ if (!id) {
+ const f = parseFilters(req.originalUrl || req.url);
+ if (f.error) return problem(req, res, 400, 'tools.query.invalid', f.error);
+ const e = entry(snap.version, `list:${f.key}`, () => {
+ const tools = snap.tools.filter(d => f.test(d, snap.meta.get(d.id) || {}));
+ const counts = new Map();
+ for (const d of tools) counts.set(d.family, (counts.get(d.family) || 0) + 1);
+ return {
+ tools: tools.map(refForm), count: tools.length, updated_at: snap.updatedAt,
+ families: [...counts].map(([fid, count]) => ({ id: fid, name: snap.families.get(fid) || fid, count })),
+ };
+ });
+ return send(req, res, e, 'application/json; charset=utf-8');
+ }
+
+ const d = snap.tools.find(t => t.id === id);
+ if (!d) return problem(req, res, 404, 'tools.tool.not_found', snap.gone.get(id) || `No tool is called "${id}". GET /api/v1/tools lists them all.`);
+ if (schema) return send(req, res, entry(snap.version, `schema:${id}`, () => schemaDoc(d)), 'application/schema+json; charset=utf-8');
+ return send(req, res, entry(snap.version, `tool:${id}`, () => d), 'application/json; charset=utf-8');
+ };
+}
+
+module.exports = { createToolsApi, exceptRegistry, isRegistryPath, parseFilters, PATH_RE };
new file mode 100644
new file mode 100644
@@ -0,0 +1,78 @@
@@ -0,0 +1,78 @@
+'use strict';
+// ═══════════════════════════════════════════════════════════════
+// A satellite's own tool registry: the descriptors of the tools it runs, for its GET /api/v1/tools.
+//
+// Specs come from the app's server/descriptors.js; name, summary, keywords and family from the
+// gateway's catalogue copy (pure data files, read by path like the rest of apps/_shared); hosts are
+// the code defaults (registry/hosts.js). The satellite's own live state decides status: a tool whose
+// program is missing on this host (spec.requires, checked by statusOf) is 'unavailable' here first.
+// The gateway's registry is the complete one: it adds the owner's domain overrides and every family.
+// ═══════════════════════════════════════════════════════════════
+
+const crypto = require('crypto');
+const path = require('path');
+const { compose, summaryOf } = require('./descriptor');
+
+const REGISTRY = path.join(__dirname, '..', '..', 'gateway', 'server', 'registry');
+const { SATELLITE_COPY } = require(path.join(REGISTRY, 'copy-satellites'));
+const { FAMILIES } = require(path.join(REGISTRY, 'families'));
+const { defaultHosts } = require(path.join(REGISTRY, 'hosts'));
+
+/** Catalogue copy of a satellite tool: { id, family, name, tagline, description, keywords } or null. */
+function copyOf(id) {
+ return SATELLITE_COPY.find(r => r.id === id) || null;
+}
+
+/**
+ * @param {object} o
+ * @param {object[]} o.specs the app's specs (server/descriptors.js)
+ * @param {(spec) => object|null} [o.statusOf] live status: { status, statusReason } or null (the spec's own)
+ * @param {number} [o.liveTtlMs] how long a statusOf answer is kept (default 30 s)
+ */
+function createLocalRegistry(o) {
+ const families = new Map(FAMILIES.map(f => [f.id, f.name]));
+ const rows = o.specs.map(spec => {
+ const c = copyOf(spec.id);
+ if (!c) throw new Error(`tool ${spec.id} has no catalogue copy (apps/gateway/server/registry/copy-satellites.js)`);
+ return { spec, c };
+ });
+ let built = null;
+ let live = null, liveAt = 0;
+ const ttl = o.liveTtlMs == null ? 30_000 : o.liveTtlMs;
+
+ function snapshot() {
+ if (!live || Date.now() - liveAt >= ttl) { live = rows.map(({ spec }) => (o.statusOf && o.statusOf(spec)) || null); liveAt = Date.now(); }
+ const sig = JSON.stringify(live);
+ if (built && built.sig === sig) return built;
+ const tools = rows.map(({ spec, c }, i) => {
+ const h = defaultHosts(spec.id);
+ return compose(spec, {
+ family: c.family, name: c.name, summary: summaryOf(c),
+ hosts: [h.canonical, h.short].filter(Boolean),
+ ...(live[i] && { status: live[i].status, statusReason: live[i].statusReason }),
+ });
+ });
+ const version = crypto.createHash('sha1').update(JSON.stringify(tools)).digest('hex').slice(0, 16);
+ built = {
+ sig, version, tools, families, gone: new Map(),
+ updatedAt: built && built.version === version ? built.updatedAt : new Date().toISOString(),
+ meta: new Map(rows.map(({ spec, c }) => [spec.id, { keywords: c.keywords || [] }])),
+ };
+ return built;
+ }
+
+ return { snapshot };
+}
+
+/**
+ * statusOf for specs that name programs (spec.requires): unavailable while one of them is missing.
+ * @param {(program: string) => boolean} available
+ */
+function requiresStatus(available) {
+ return (spec) => {
+ const missing = (spec.requires || []).filter(p => !available(p));
+ return missing.length ? { status: 'unavailable', statusReason: `This tool is being set up on the server (it needs ${missing.join(' and ')}) and is not available yet.` } : null;
+ };
+}
+
+module.exports = { createLocalRegistry, requiresStatus, copyOf };
@@ -25,6 +25,8 @@ const retention = require('./retention/manager');
@@ -25,6 +25,8 @@ const retention = require('./retention/manager');
const { probe, getDuration, cleanTmp } = require('./tools/ffmpeg-helper');
const { probe, getDuration, cleanTmp } = require('./tools/ffmpeg-helper');
const { buildOptions, describe, defineJobs } = require('./process');
const { buildOptions, describe, defineJobs } = require('./process');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { hostGuard, ownHost } = require('../../_shared/host-role');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const jobsRuntime = require('../../_shared/jobs');
const jobsRuntime = require('../../_shared/jobs');
const contracts = require('openvibe-contracts');
const contracts = require('openvibe-contracts');
const sdk = require('openvibe-sdk');
const sdk = require('openvibe-sdk');
@@ -44,7 +46,7 @@ const app = express();
@@ -44,7 +46,7 @@ const app = express();
const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });
const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });
// Metrics (GET /metrics, direct loopback callers only) and GET /api/ready from this server's real
// Metrics (GET /metrics, direct loopback callers only) and GET /api/ready from this server's real
// dependencies (roadmap Track O). First, so the HTTP metrics see every request.
// dependencies (roadmap Track O). First, so the HTTP metrics see every request.
-const { observe, checks: ready } = require('../../_shared/observe');
+const { observe, checks: ready, which } = require('../../_shared/observe');
observe({
observe({
app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),
app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),
service: 'tools-audio', release: release.release,
service: 'tools-audio', release: release.release,
@@ -89,7 +91,7 @@ app.use(express.json({ limit: '1mb' }));
@@ -89,7 +91,7 @@ app.use(express.json({ limit: '1mb' }));
app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response
app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response
// ── CORS ─────────────────────────────────────────────────────
// ── CORS ─────────────────────────────────────────────────────
-app.use(cors({
+app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true);
if (!origin) return callback(null, true);
if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);
if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);
@@ -97,11 +99,17 @@ app.use(cors({
@@ -97,11 +99,17 @@ app.use(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-}));
+})));
// ── Rate Limiting ────────────────────────────────────────────
// ── Rate Limiting ────────────────────────────────────────────
app.use('/api/', apiLimiter);
app.use('/api/', apiLimiter);
+// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's audio tools ──
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
+const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'ffmpeg' ? !!which(process.env.FFMPEG_PATH || 'ffmpeg') : true)) });
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
app.use(analytics.middleware());
app.use(analytics.middleware());
@@ -23,6 +23,8 @@ const { apiLimiter, processLimiter, burstLimiter } = require('./middleware/rate-
@@ -23,6 +23,8 @@ const { apiLimiter, processLimiter, burstLimiter } = require('./middleware/rate-
const retention = require('./retention/manager');
const retention = require('./retention/manager');
const { buildOptions, defineJobs } = require('./process');
const { buildOptions, defineJobs } = require('./process');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { hostGuard, ownHost } = require('../../_shared/host-role');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const jobsRuntime = require('../../_shared/jobs');
const jobsRuntime = require('../../_shared/jobs');
const contracts = require('openvibe-contracts');
const contracts = require('openvibe-contracts');
const sdk = require('openvibe-sdk');
const sdk = require('openvibe-sdk');
@@ -86,7 +88,7 @@ app.use(express.json({ limit: '1mb' }));
@@ -86,7 +88,7 @@ app.use(express.json({ limit: '1mb' }));
app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response
app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response
// ── CORS ─────────────────────────────────────────────────────
// ── CORS ─────────────────────────────────────────────────────
-app.use(cors({
+app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true);
if (!origin) return callback(null, true);
if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);
if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);
@@ -94,11 +96,18 @@ app.use(cors({
@@ -94,11 +96,18 @@ app.use(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-}));
+})));
// ── Rate Limiting ────────────────────────────────────────────
// ── Rate Limiting ────────────────────────────────────────────
app.use('/api/', apiLimiter);
app.use('/api/', apiLimiter);
+// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's PDF tools ──
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
+const pdfPrograms = require('./tools/pdf');
+const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (pdfPrograms[p] ? pdfPrograms[p].available() : true)) });
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
app.use(analytics.middleware());
app.use(analytics.middleware());
@@ -17,6 +17,8 @@ const Database = require('better-sqlite3');
@@ -17,6 +17,8 @@ const Database = require('better-sqlite3');
const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recorded
const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recorded
const { internalOk } = require('../../_shared/internal-auth');
const { internalOk } = require('../../_shared/internal-auth');
const { hostGuard, stampedPage } = require('../../_shared/host-role');
const { hostGuard, stampedPage } = require('../../_shared/host-role');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const analyticsDbPath = path.join(__dirname, '..', 'data', 'analytics.db');
const analyticsDbPath = path.join(__dirname, '..', 'data', 'analytics.db');
fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });
fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });
const analyticsDb = new Database(analyticsDbPath);
const analyticsDb = new Database(analyticsDbPath);
@@ -49,7 +51,7 @@ app.get(['/terms', '/privacy', '/dmca', '/tos'], (req, res) => res.redirect(301,
@@ -49,7 +51,7 @@ app.get(['/terms', '/privacy', '/dmca', '/tos'], (req, res) => res.redirect(301,
// Only loopback hops (the host's nginx, the gateway) are proxies: req.ip is the first address before
// Only loopback hops (the host's nginx, the gateway) are proxies: req.ip is the first address before
// them, which is what maps is told below. A client's own X-Forwarded-For is never believed.
// them, which is what maps is told below. A client's own X-Forwarded-For is never believed.
app.set('trust proxy', 'loopback');
app.set('trust proxy', 'loopback');
-app.use(cors({
+app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true);
if (!origin) return callback(null, true);
if (/^https:\/\/[a-z0-9-]+\.openvibe\.tools$/.test(origin)) return callback(null, true);
if (/^https:\/\/[a-z0-9-]+\.openvibe\.tools$/.test(origin)) return callback(null, true);
@@ -58,7 +60,7 @@ app.use(cors({
@@ -58,7 +60,7 @@ app.use(cors({
return callback(new Error('Origin not allowed'));
return callback(new Error('Origin not allowed'));
},
},
credentials: true,
credentials: true,
-}));
+})));
app.use(helmet({
app.use(helmet({
contentSecurityPolicy: {
contentSecurityPolicy: {
directives: {
directives: {
@@ -73,6 +75,13 @@ app.use(helmet({
@@ -73,6 +75,13 @@ app.use(helmet({
},
},
}));
}));
app.use(rateLimit({ windowMs: 60000, max: 60 }));
app.use(rateLimit({ windowMs: 60000, max: 60 }));
+
+// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's food finder ──
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
+// The food finder is built on the maps backend; its descriptor lives with the maps app's.
+const toolRegistry = createLocalRegistry({ specs: require('../../maps/server/descriptors').SPECS.filter(s => s.id === 'food') });
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
app.use(cookieParser());
app.use(cookieParser());
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
@@ -46,6 +46,10 @@ const SATELLITE_BY_PORT = new Map(Object.entries(SATELLITES).map(([name, port])
@@ -46,6 +46,10 @@ const SATELLITE_BY_PORT = new Map(Object.entries(SATELLITES).map(([name, port])
// the host router, so on a satellite's host it is the satellite's own.
// the host router, so on a satellite's host it is the satellite's own.
const { observe, checks: ready } = require('../../_shared/observe');
const { observe, checks: ready } = require('../../_shared/observe');
const registry = require('./registry');
const registry = require('./registry');
+// Every tool's descriptor (tools.tool@1, ADR-027) for GET /api/v1/tools[/:id[/schema]]; checked with
+// openvibe-contracts on every build, a failure is the tool_registry readiness row, never a crash.
+const toolRegistry = require('./registry/descriptors');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
let auth = null; // created below; the key check reads it at request time
let auth = null; // created below; the key check reads it at request time
const obs = observe({
const obs = observe({
app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),
app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),
@@ -68,6 +72,7 @@ const obs = observe({
@@ -68,6 +72,7 @@ const obs = observe({
check: () => { const s = registry.services.status(); return s.source === 'registry' ? { ok: true, detail: { as_of: s.as_of, last_ok: s.last_ok } } : `using the built-in fallback list${s.last_error ? ` (${s.last_error})` : ''}`; },
check: () => { const s = registry.services.status(); return s.source === 'registry' ? { ok: true, detail: { as_of: s.as_of, last_ok: s.last_ok } } : `using the built-in fallback list${s.last_error ? ` (${s.last_error})` : ''}`; },
},
},
ready.upstream('community', `${config.communityUrl}/api/ready`, { description: 'paste API proxy (pastes.openvibe.tools)' }),
ready.upstream('community', `${config.communityUrl}/api/ready`, { description: 'paste API proxy (pastes.openvibe.tools)' }),
+ toolRegistry.readyCheck(),
...Object.entries(SATELLITES).map(([name, port]) => ready.upstream(`satellite_${name}`, `http://127.0.0.1:${port}/api/ready`, { description: `${name} satellite (port ${port}); its hosts fail without it` })),
...Object.entries(SATELLITES).map(([name, port]) => ready.upstream(`satellite_${name}`, `http://127.0.0.1:${port}/api/ready`, { description: `${name} satellite (port ${port}); its hosts fail without it` })),
],
],
});
});
@@ -140,19 +145,26 @@ function isAllowedOrigin(origin) {
@@ -140,19 +145,26 @@ function isAllowedOrigin(origin) {
// preflight before the wider allow-list below gets a chance to.
// preflight before the wider allow-list below gets a chance to.
app.use('/auth/fedcm', fedcmCors);
app.use('/auth/fedcm', fedcmCors);
-app.use(cors({
+// The tool registry (/api/v1/tools…) is public and read-only: open to every origin, no credentials.
+app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true); // curl / server-to-server
if (!origin) return callback(null, true); // curl / server-to-server
if (isAllowedOrigin(origin)) return callback(null, true);
if (isAllowedOrigin(origin)) return callback(null, true);
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-}));
+})));
// ── Rate Limiting ────────────────────────────────────────────
// ── Rate Limiting ────────────────────────────────────────────
app.use('/api/', rateLimit({ windowMs: 60_000, max: 120 }));
app.use('/api/', rateLimit({ windowMs: 60_000, max: 120 }));
app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60 }));
app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60 }));
+// ── Tool registry (ADR-027, capability tools.tool.read) ──────
+// GET /api/v1/tools (tools.tool-list@1), /api/v1/tools/:id (tools.tool@1), /api/v1/tools/:id/schema.
+// Every tool, every family; satellite hosts answer the same routes for their own tools. Counted by
+// the /api/ limiter above. /api/catalog.json stays as it is (Network reads it).
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+
// ── Auth (OAuth2 client of OpenVibe.Network) ─────────────────
// ── Auth (OAuth2 client of OpenVibe.Network) ─────────────────
auth = createAuthClient(config);
auth = createAuthClient(config);
app.locals.auth = auth;
app.locals.auth = auth;
@@ -347,6 +359,9 @@ app.get('*', (req, res) => {
@@ -347,6 +359,9 @@ app.get('*', (req, res) => {
});
});
// ── Start ────────────────────────────────────────────────────
// ── Start ────────────────────────────────────────────────────
+// What each satellite says about its own tools (a program missing on the host → unavailable here too).
+toolRegistry.startSatellitePolling(SATELLITES);
+
app.listen(config.port, config.host, () => {
app.listen(config.port, config.host, () => {
console.log(`\n╔════════════════════════════════════════════╗`);
console.log(`\n╔════════════════════════════════════════════╗`);
console.log(`║ 🔧 openvibe.tools — Tools Gateway ║`);
console.log(`║ 🔧 openvibe.tools — Tools Gateway ║`);
new file mode 100644
new file mode 100644
@@ -0,0 +1,203 @@
@@ -0,0 +1,203 @@
+'use strict';
+// GET /api/v1/tools, /api/v1/tools/:id and /api/v1/tools/:id/schema (ADR-027, tools.tool.read), on
+// real processes: the gateway and all seven satellites.
+// • the answers are contract documents: tools.tool-list@1 (checkList), tools.tool@1 (checkDescriptor)
+// with embedded schemas, and a schema document whose $defs the list's $refs point at
+// • open and cacheable: Access-Control-Allow-Origin * for any origin (the gateway's own CORS never
+// refuses them), OPTIONS preflight, ETag with 304 on If-None-Match, Cache-Control public 300 s, HEAD
+// • unknown ids are 404 problem+json tools.tool.not_found (planned and mirrors say so); a bad filter 400
+// • counted by the gateway's /api/ limiter; /api/catalog.json is unchanged
+// • each satellite answers for its own tools, with its own live status (a missing program → unavailable),
+// and the gateway picks that status up
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const contracts = require('openvibe-contracts');
+const Ajv2020 = require('ajv/dist/2020');
+const addFormats = require('ajv-formats');
+const { startApp } = require('../../_shared/test/spawn');
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-tools-api-'));
+const MISSING = path.join(tmp, 'not-installed');
+const get = async (url, headers = {}, method = 'GET') => {
+ const r = await fetch(url, { method, headers });
+ const text = await r.text();
+ let body = null; try { body = text ? JSON.parse(text) : null; } catch { /* not JSON */ }
+ return { status: r.status, h: r.headers, text, body };
+};
+
+(async () => {
+ const procs = [];
+ const start = async (...a) => { const p = await startApp(...a); procs.push(p); return p; };
+ try {
+ // Satellites, each missing the program one of its tools needs.
+ const sat = {
+ img: await start('img', { DATA_DIR: path.join(tmp, 'img'), HEIF_DEC_PATH: MISSING }),
+ docs: await start('docs', { DATA_DIR: path.join(tmp, 'docs'), QPDF_PATH: MISSING, PDFTOPPM_PATH: MISSING, PDFINFO_PATH: MISSING }),
+ audio: await start('audio', { DATA_DIR: path.join(tmp, 'audio'), FFMPEG_PATH: MISSING }),
+ yt: await start('yt', { DATA_DIR: path.join(tmp, 'yt'), DOWNLOADS_DIR: path.join(tmp, 'yt', 'dl'), YTDLP_PATH: MISSING }),
+ text: await start('text', { DATA_DIR: path.join(tmp, 'text') }),
+ maps: await start('maps', { DATA_DIR: path.join(tmp, 'maps') }, undefined, { readyPath: '/api/ready' }),
+ };
+ sat.food = await start('food', { DATA_DIR: path.join(tmp, 'food'), MAPS_API: sat.maps.base }, undefined, { readyPath: '/api/ready' });
+ const ports = Object.entries(sat).map(([n, p]) => `${n}=${p.port}`).join(',');
+ const gw = await start('gateway', { TOOLS_SATELLITE_PORTS: ports, OV_DOMAINS_URL: 'http://127.0.0.1:9/api/domains', OV_REGISTRY_URL: 'http://127.0.0.1:9/registry' });
+ const G = gw.base;
+
+ // ── The list: a tools.tool-list@1 document with every tool ──
+ let r = await get(`${G}/api/v1/tools`, { Origin: 'https://someone-elses-site.example' });
+ assert.strictEqual(r.status, 200, r.text.slice(0, 300));
+ assert.match(r.h.get('content-type'), /^application\/json/);
+ assert.strictEqual(r.h.get('access-control-allow-origin'), '*', 'open to any origin');
+ assert.ok(!r.h.get('access-control-allow-credentials'), 'no credentials');
+ assert.strictEqual(r.h.get('cache-control'), 'public, max-age=300');
+ assert.ok(r.h.get('x-ratelimit-limit') || r.h.get('ratelimit-limit') || r.h.get('ratelimit-policy'), 'counted by the gateway\'s /api/ limiter');
+ const etag = r.h.get('etag');
+ assert.match(etag, /^"[A-Za-z0-9_-]+"$/);
+ const list = r.body;
+ const lc = contracts.tools.checkList(list);
+ assert.ok(lc.valid, JSON.stringify(lc.errors.slice(0, 5)));
+ assert.strictEqual(list.count, 169);
+ assert.ok(list.tools.every(d => !d.input || d.input.$ref), 'inputs are $refs in the list');
+ assert.ok(!Number.isNaN(Date.parse(list.updated_at)));
+ assert.deepStrictEqual(list.families.map(f => f.id).sort(), ['audio', 'dev', 'docs', 'img', 'media', 'net', 'pastes', 'places', 'text']);
+
+ // ETag → 304 (strong, weak, in a list); HEAD; preflight.
+ assert.strictEqual((await get(`${G}/api/v1/tools`, { 'If-None-Match': etag })).status, 304);
+ assert.strictEqual((await get(`${G}/api/v1/tools`, { 'If-None-Match': `"x", W/${etag}` })).status, 304);
+ r = await get(`${G}/api/v1/tools`, {}, 'HEAD');
+ assert.deepStrictEqual([r.status, r.text, r.h.get('etag')], [200, '', etag]);
+ r = await get(`${G}/api/v1/tools/png`, { Origin: 'https://a.example', 'Access-Control-Request-Method': 'GET', 'Access-Control-Request-Headers': 'if-none-match' }, 'OPTIONS');
+ assert.strictEqual(r.status, 204);
+ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');
+ assert.match(r.h.get('access-control-allow-headers'), /If-None-Match/);
+
+ // Filters.
+ const count = async (q) => (await get(`${G}/api/v1/tools?${q}`)).body;
+ let l = await count('family=img');
+ assert.strictEqual(l.count, 15); assert.deepStrictEqual(l.families, [{ id: 'img', name: 'Image Tools', count: 15 }]);
+ assert.ok(contracts.tools.checkList(l).valid);
+ l = await count('execution=client&api=true');
+ assert.ok(l.tools.length >= 40 && l.tools.every(d => d.execution === 'client' && d.api));
+ l = await count('api=false');
+ assert.ok(l.tools.some(d => d.id === 'yt') && l.tools.every(d => !d.api && d.run === null));
+ l = await count('q=json%20minify');
+ assert.ok(l.tools.some(d => d.id === 'jsonminify'), 'text search');
+ l = await count('family=net,dev&execution=sync');
+ assert.ok(l.tools.every(d => ['net', 'dev'].includes(d.family) && d.execution === 'sync') && l.tools.some(d => d.id === 'opengraph'));
+ r = await get(`${G}/api/v1/tools?execution=later`);
+ assert.deepStrictEqual([r.status, r.body.code], [400, 'tools.query.invalid']);
+ assert.match(r.h.get('content-type'), /^application\/problem\+json/);
+ r = await get(`${G}/api/v1/tools?api=yes`);
+ assert.strictEqual(r.status, 400);
+
+ // ── One tool: tools.tool@1 with its schemas embedded ──
+ r = await get(`${G}/api/v1/tools/png`);
+ assert.strictEqual(r.status, 200);
+ const png = r.body;
+ assert.ok(contracts.validate('tools.tool@1', png).valid);
+ assert.ok(contracts.tools.checkDescriptor(png).valid);
+ assert.strictEqual(png.input.type, 'object', 'embedded');
+ assert.deepStrictEqual(png.run.job, { type: 'img.process', operation: 'convert', preset: { defaultFormat: 'png' } });
+ assert.strictEqual((await get(`${G}/api/v1/tools/png`, { 'If-None-Match': r.h.get('etag') })).status, 304);
+ for (const id of ['dns', 'port', 'jsonminify', 'yt', 'logo', 'mergepdf', 'pastes', 'maps']) {
+ const d = (await get(`${G}/api/v1/tools/${id}`)).body;
+ assert.ok(contracts.tools.checkDescriptor(d).valid, `${id}: ${JSON.stringify(contracts.tools.checkDescriptor(d).errors)}`);
+ }
+
+ // ── The schema document the $refs point at ──
+ const ref = list.tools.find(d => d.id === 'dns').input.$ref;
+ assert.strictEqual(ref, 'https://openvibe.tools/api/v1/tools/dns/schema#/$defs/input');
+ r = await get(`${G}${new URL(ref).pathname}`);
+ assert.strictEqual(r.status, 200);
+ assert.match(r.h.get('content-type'), /^application\/schema\+json/);
+ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');
+ const doc = r.body;
+ assert.strictEqual(doc.$schema, 'https://json-schema.org/draft/2020-12/schema');
+ assert.strictEqual(doc.$id, 'https://openvibe.tools/api/v1/tools/dns/schema');
+ const dns = (await get(`${G}/api/v1/tools/dns`)).body;
+ assert.deepStrictEqual(doc.$defs.input, dns.input);
+ assert.deepStrictEqual(doc.$defs.output, dns.output.schema);
+ // The $ref resolves with a JSON Schema validator, as a client would use it.
+ const ajv = new Ajv2020({ strict: false }); addFormats(ajv);
+ ajv.addSchema(doc);
+ const v = ajv.compile({ $ref: ref });
+ assert.ok(v({ target: 'example.com', types: ['MX'] }));
+ assert.ok(!v({ types: ['MX'] }), 'target is required');
+ assert.strictEqual((await get(`${G}/api/v1/tools/yt/schema`)).body.$defs.input, false, 'no API, no input');
+
+ // ── Not tools ──
+ for (const [id, why] of [['no-such-tool', /No tool is called/], ['qr', /planned/], ['jsonfmt', /mirror of json|second build of json/]]) {
+ r = await get(`${G}/api/v1/tools/${id}`);
+ assert.strictEqual(r.status, 404, id);
+ assert.match(r.h.get('content-type'), /^application\/problem\+json/);
+ assert.strictEqual(r.body.code, 'tools.tool.not_found');
+ assert.strictEqual(r.body.status, 404);
+ assert.match(r.body.detail, why);
+ assert.strictEqual(r.h.get('access-control-allow-origin'), '*');
+ assert.strictEqual((await get(`${G}/api/v1/tools/${id}/schema`)).status, 404);
+ }
+ assert.strictEqual((await get(`${G}/api/v1/tools/Bad_Id`)).status, 404);
+
+ // ── /api/catalog.json is unchanged (Network reads it) ──
+ const catalog = (await get(`${G}/api/catalog.json`)).body;
+ assert.strictEqual(catalog.tools.length, 169);
+ assert.ok(catalog.tools.every(t => ['available', 'unavailable'].includes(t.status) && t.hosts && t.hosts.canonical));
+ assert.ok(!('api' in catalog.tools[0]) && !('execution' in catalog.tools[0]), 'no descriptor fields leak into the catalogue');
+
+ // ── Each satellite answers for its own tools, with its own status ──
+ const own = { img: 'img', docs: 'docs', audio: 'audio', yt: 'media', text: 'text', maps: 'places', food: 'places' };
+ const counts = {};
+ for (const [name, p] of Object.entries(sat)) {
+ r = await get(`${p.base}/api/v1/tools`, { Origin: 'https://elsewhere.example' });
+ assert.strictEqual(r.status, 200, `${name}: ${r.text.slice(0, 200)}`);
+ assert.strictEqual(r.h.get('access-control-allow-origin'), '*', `${name}: open CORS`);
+ assert.ok(r.h.get('etag'), `${name}: ETag`);
+ assert.strictEqual((await get(`${p.base}/api/v1/tools`, { 'If-None-Match': r.h.get('etag') })).status, 304, `${name}: 304`);
+ const c = contracts.tools.checkList(r.body);
+ assert.ok(c.valid, `${name}: ${JSON.stringify(c.errors.slice(0, 3))}`);
+ assert.ok(r.body.tools.length && r.body.tools.every(d => d.family === own[name]), `${name}: only its own tools`);
+ counts[name] = r.body.count;
+ const first = r.body.tools[0].id;
+ const one = await get(`${p.base}/api/v1/tools/${first}`);
+ assert.ok(contracts.tools.checkDescriptor(one.body).valid, `${name}: /api/v1/tools/${first}`);
+ assert.strictEqual((await get(`${p.base}/api/v1/tools/${first}/schema`)).status, 200);
+ assert.strictEqual((await get(`${p.base}/api/v1/tools/png-nope`)).status, 404);
+ }
+ assert.deepStrictEqual(counts, { img: 15, docs: 10, audio: 36, yt: 1, text: 33, maps: 1, food: 1 });
+ const status = async (base, id) => (await get(`${base}/api/v1/tools/${id}`)).body;
+ let d = await status(sat.docs.base, 'protectpdf');
+ assert.strictEqual(d.status, 'unavailable'); assert.match(d.statusReason, /qpdf/);
+ assert.strictEqual((await status(sat.docs.base, 'pdf2jpg')).status, 'unavailable');
+ assert.strictEqual((await status(sat.docs.base, 'mergepdf')).status, 'stable');
+ assert.strictEqual((await status(sat.img.base, 'heic')).status, 'unavailable', 'HEIC without libheif');
+ assert.strictEqual((await status(sat.img.base, 'png')).status, 'stable');
+ assert.strictEqual((await status(sat.audio.base, 'mp3')).status, 'unavailable', 'audio without ffmpeg');
+ assert.strictEqual((await status(sat.yt.base, 'yt')).status, 'unavailable', 'yt without yt-dlp');
+ assert.strictEqual((await status(sat.text.base, 'logo')).status, 'stable');
+
+ // The gateway polled them at boot: the same statuses in the full registry.
+ for (let i = 0; i < 50 && (await status(G, 'protectpdf')).status !== 'unavailable'; i++) await new Promise(res => setTimeout(res, 100));
+ for (const id of ['protectpdf', 'unlockpdf', 'pdf2jpg', 'heic', 'mp3', 'merge', 'yt']) assert.strictEqual((await status(G, id)).status, 'unavailable', `gateway: ${id}`);
+ for (const id of ['png', 'mergepdf', 'fancy', 'dns']) assert.strictEqual((await status(G, id)).status, 'stable', `gateway: ${id}`);
+ l = await count('status=unavailable');
+ assert.ok(l.tools.length >= 3 + 36 + 5 && contracts.tools.checkList(l).valid, 'status filter');
+
+ // Readiness: the registry is checked, and it passes.
+ const ready = (await get(`${G}/api/ready`)).body;
+ assert.strictEqual(ready.checks.tool_registry.status, 'ok', JSON.stringify(ready.checks.tool_registry));
+ assert.strictEqual(ready.checks.tool_registry.required, false);
+
+ // ── The gateway's /api/ limiter counts registry reads (120 a minute per address) ──
+ let limited = null;
+ for (let i = 0; i < 130 && !limited; i++) { const x = await fetch(`${G}/api/v1/tools/png`); if (x.status === 429) limited = x; }
+ assert.ok(limited, 'the 121st request in a minute is refused');
+ assert.strictEqual(limited.headers.get('access-control-allow-origin'), '*', 'even a 429 is readable cross-origin');
+
+ console.log(`tools API: gateway ${list.count} tools (list, one, schema, ETag/304, CORS *, 404 problem, filters, limiter); satellites ${Object.entries(counts).map(([n, c]) => `${n} ${c}`).join(', ')} with live status`);
+ } finally {
+ await Promise.all(procs.map(p => p.kill()));
+ fs.rmSync(tmp, { recursive: true, force: true });
+ }
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -24,6 +24,8 @@ const retention = require('./retention/manager');
@@ -24,6 +24,8 @@ const retention = require('./retention/manager');
const { buildOptions, describe, processBuffer, defineJobs } = require('./process');
const { buildOptions, describe, processBuffer, defineJobs } = require('./process');
const codec = require('./tools/codec');
const codec = require('./tools/codec');
const { hostGuard, ownHost } = require('../../_shared/host-role');
const { hostGuard, ownHost } = require('../../_shared/host-role');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const jobsRuntime = require('../../_shared/jobs');
const jobsRuntime = require('../../_shared/jobs');
const contracts = require('openvibe-contracts');
const contracts = require('openvibe-contracts');
const sdk = require('openvibe-sdk');
const sdk = require('openvibe-sdk');
@@ -87,7 +89,7 @@ app.use(express.json({ limit: '1mb' }));
@@ -87,7 +89,7 @@ app.use(express.json({ limit: '1mb' }));
app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response
app.use(contracts.http.middleware()); // traceparent + X-OpenVibe-Request-Id on every response
// ── CORS ─────────────────────────────────────────────────────
// ── CORS ─────────────────────────────────────────────────────
-app.use(cors({
+app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true);
if (!origin) return callback(null, true);
// Allow any *.openvibe.tools subdomain + openvibe.tools itself
// Allow any *.openvibe.tools subdomain + openvibe.tools itself
@@ -96,11 +98,17 @@ app.use(cors({
@@ -96,11 +98,17 @@ app.use(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-}));
+})));
// ── Rate Limiting ────────────────────────────────────────────
// ── Rate Limiting ────────────────────────────────────────────
app.use('/api/', apiLimiter);
app.use('/api/', apiLimiter);
+// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's image tools ──
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
+const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'heif-dec' ? codec.heif.available() : true)) });
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
app.use(analytics.middleware());
app.use(analytics.middleware());
@@ -22,6 +22,8 @@ const Database = require('better-sqlite3');
@@ -22,6 +22,8 @@ const Database = require('better-sqlite3');
const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recorded
const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recorded
const { internalOk } = require('../../_shared/internal-auth');
const { internalOk } = require('../../_shared/internal-auth');
const { hostGuard, stampedPage } = require('../../_shared/host-role');
const { hostGuard, stampedPage } = require('../../_shared/host-role');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const analyticsDbPath = path.join(__dirname, '..', 'data', 'analytics.db');
const analyticsDbPath = path.join(__dirname, '..', 'data', 'analytics.db');
fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });
fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });
const analyticsDb = new Database(analyticsDbPath);
const analyticsDb = new Database(analyticsDbPath);
@@ -67,7 +69,7 @@ app.use(helmet({
@@ -67,7 +69,7 @@ app.use(helmet({
},
},
},
},
}));
}));
-app.use(cors({ origin: ['https://maps.openvibe.tools', 'https://food.openvibe.tools', 'https://openvibe.network', 'http://localhost:4010', 'http://localhost:4011'] }));
+app.use(exceptRegistry(cors({ origin: ['https://maps.openvibe.tools', 'https://food.openvibe.tools', 'https://openvibe.network', 'http://localhost:4010', 'http://localhost:4011'] })));
app.use(cookieParser());
app.use(cookieParser());
app.use(express.json());
app.use(express.json());
@@ -79,6 +81,12 @@ const apiLimiter = rateLimit({
@@ -79,6 +81,12 @@ const apiLimiter = rateLimit({
});
});
app.use('/api/', apiLimiter);
app.use('/api/', apiLimiter);
+// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's survival map ──
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
+const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS.filter(s => s.id === 'maps') });
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
app.use(analytics.middleware());
app.use(analytics.middleware());
@@ -15,6 +15,8 @@ const Database = require('better-sqlite3');
@@ -15,6 +15,8 @@ const Database = require('better-sqlite3');
const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recorded
const { AnalyticsTracker } = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, raw rows pruned after 30 days, Sec-GPC/DNT not recorded
const { internalOk } = require('../../_shared/internal-auth');
const { internalOk } = require('../../_shared/internal-auth');
const { hostGuard } = require('../../_shared/host-role');
const { hostGuard } = require('../../_shared/host-role');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
const analyticsDbPath = path.resolve(__dirname, '..', process.env.DATA_DIR || 'data', 'analytics.db');
const analyticsDbPath = path.resolve(__dirname, '..', process.env.DATA_DIR || 'data', 'analytics.db');
fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });
fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });
const analyticsDb = new Database(analyticsDbPath);
const analyticsDb = new Database(analyticsDbPath);
@@ -60,7 +62,7 @@ app.use(cookieParser());
@@ -60,7 +62,7 @@ app.use(cookieParser());
app.use(express.json({ limit: '256kb' }));
app.use(express.json({ limit: '256kb' }));
// ── CORS ─────────────────────────────────────────────────────
// ── CORS ─────────────────────────────────────────────────────
-app.use(cors({
+app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true);
if (!origin) return callback(null, true);
if (/^https:\/\/[a-z0-9-]+\.openvibe\.tools$/.test(origin)) return callback(null, true);
if (/^https:\/\/[a-z0-9-]+\.openvibe\.tools$/.test(origin)) return callback(null, true);
@@ -69,11 +71,17 @@ app.use(cors({
@@ -69,11 +71,17 @@ app.use(cors({
return callback(new Error('Origin not allowed'));
return callback(new Error('Origin not allowed'));
},
},
credentials: true,
credentials: true,
-}));
+})));
// ── Rate Limiting ────────────────────────────────────────────
// ── Rate Limiting ────────────────────────────────────────────
app.use(rateLimit({ windowMs: 60_000, max: 200 }));
app.use(rateLimit({ windowMs: 60_000, max: 200 }));
+// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's text and logo tools ──
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
+const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS });
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
app.use(analytics.middleware());
app.use(analytics.middleware());
@@ -18,6 +18,8 @@ const { optionalAuth } = require('./auth');
@@ -18,6 +18,8 @@ const { optionalAuth } = require('./auth');
const downloader = require('./downloader');
const downloader = require('./downloader');
const seo = require('./seo');
const seo = require('./seo');
const { hostGuard } = require('../../_shared/host-role');
const { hostGuard } = require('../../_shared/host-role');
+const { createToolsApi, exceptRegistry } = require('../../_shared/tools/http');
+const { createLocalRegistry, requiresStatus } = require('../../_shared/tools/local');
// ── Analytics ────────────────────────────────────────────────
// ── Analytics ────────────────────────────────────────────────
const Database = require('better-sqlite3');
const Database = require('better-sqlite3');
@@ -34,7 +36,7 @@ const app = express();
@@ -34,7 +36,7 @@ const app = express();
const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });
const release = require('openvibe-shared/release').createRelease({ service: 'tools', root: require('path').join(__dirname, '..', '..', '..') });
// Metrics (GET /metrics, direct loopback callers only) and GET /api/ready from this server's real
// Metrics (GET /metrics, direct loopback callers only) and GET /api/ready from this server's real
// dependencies (roadmap Track O). First, so the HTTP metrics see every request.
// dependencies (roadmap Track O). First, so the HTTP metrics see every request.
-const { observe, checks: ready } = require('../../_shared/observe');
+const { observe, checks: ready, which } = require('../../_shared/observe');
observe({
observe({
app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),
app, metrics: require('openvibe-shared/metrics'), ready: require('openvibe-shared/ready'),
service: 'tools-yt', release: release.release,
service: 'tools-yt', release: release.release,
@@ -78,7 +80,7 @@ app.use(cookieParser());
@@ -78,7 +80,7 @@ app.use(cookieParser());
app.use(express.json({ limit: '1mb' }));
app.use(express.json({ limit: '1mb' }));
// ── CORS ─────────────────────────────────────────────────────
// ── CORS ─────────────────────────────────────────────────────
-app.use(cors({
+app.use(exceptRegistry(cors({
origin(origin, callback) {
origin(origin, callback) {
if (!origin) return callback(null, true);
if (!origin) return callback(null, true);
if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);
if (/^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin)) return callback(null, true);
@@ -86,7 +88,7 @@ app.use(cors({
@@ -86,7 +88,7 @@ app.use(cors({
return callback(new Error('Origin not allowed by CORS'));
return callback(new Error('Origin not allowed by CORS'));
},
},
credentials: true,
credentials: true,
-}));
+})));
// ── Rate Limiting ────────────────────────────────────────────
// ── Rate Limiting ────────────────────────────────────────────
// Progress is one SSE connection, or a poll every 1–3 s when SSE is unavailable; the poll must
// Progress is one SSE connection, or a poll every 1–3 s when SSE is unavailable; the poll must
@@ -96,6 +98,12 @@ app.use('/api/', rateLimit({ windowMs: 60_000, max: 60, skip: isStatusRoute }));
@@ -96,6 +98,12 @@ app.use('/api/', rateLimit({ windowMs: 60_000, max: 60, skip: isStatusRoute }));
app.use('/api/status/', rateLimit({ windowMs: 60_000, max: 240 }));
app.use('/api/status/', rateLimit({ windowMs: 60_000, max: 240 }));
app.use('/api/', (_req, res, next) => { res.setHeader('Cache-Control', 'no-store'); next(); });
app.use('/api/', (_req, res, next) => { res.setHeader('Cache-Control', 'no-store'); next(); });
+// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's YouTube downloader ──
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
+const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'yt-dlp' ? !!which(config.ytdlpPath) : true)) });
+app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
+
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
app.use(analytics.middleware());
app.use(analytics.middleware());