openvibers/openvibe.tools README.md
README.md
| 1 |  |
| 2 | |
| 3 | # OpenVibe.Tools |
| 4 | |
| 5 | **Free online tools** — [openvibe.tools](https://openvibe.tools). Hundreds of tools that each do one job well (images, text, audio, docs, maps and video), open by API. |
| 6 | |
| 7 | **Live site:** https://openvibe.tools · **API docs:** https://openvibe.services · **For AI:** https://openvibe.tools/llms.txt |
| 8 | **Run it:** `npm ci && npm run install:all && npm run start:gateway` (Node 22 or newer) · **Tests:** `npm test` |
| 9 | |
| 10 | Part of the [OpenVibe network](https://openvibe.network); [help](https://openvibe.help) and the [developer API](https://openvibe.services). |
| 11 | |
| 12 | ## Purpose |
| 13 | |
| 14 | The tools suite of the OpenVibe network — `openvibe.tools` plus dozens of |
| 15 | tool subdomains under `*.openvibe.tools`. |
| 16 | |
| 17 | Identity is provided by **OpenVibe.Network** (`https://openvibe.network`). |
| 18 | This repo contains no identity provider: the gateway is an OAuth2 **client** |
| 19 | (`client_id: tools`) that sets the shared `ov_token` cookie on |
| 20 | `.openvibe.tools`; every satellite reads that cookie and verifies the JWT |
| 21 | offline against the Network's public key (JWKS). |
| 22 | |
| 23 | ## Owns |
| 24 | |
| 25 | - the tool registry (families, tools, descriptors `tools.tool@1`, canonical, short and alias hosts), |
| 26 | the catalog and every tool page on `openvibe.tools` and `*.openvibe.tools` |
| 27 | - the run API and the jobs runtime (img, audio, docs jobs with their files and SSE progress), the |
| 28 | guard (callers, quotas, sniffing, egress throttles, PostgreSQL abuse log) |
| 29 | - the `tools.*` events, Tools' Search documents and the `tools.usage` user module |
| 30 | |
| 31 | ## Does not own |
| 32 | |
| 33 | - identity and sign-in (OpenVibe.Network), pastes (OpenVibe.Community: `pastes.openvibe.tools` hands over), |
| 34 | result storage (OpenVibe.Media keeps job results), custom-domain records (Network's domains admin) |
| 35 | |
| 36 | ## Depends on |
| 37 | |
| 38 | - OpenVibe.Network (SSO as OAuth client `tools`, JWKS, client-credentials tokens, the domains registry), |
| 39 | OpenVibe.Media (job results), OpenVibe.Events (job and usage events, revocation cutoffs), |
| 40 | OpenVibe.Search (tool documents), OpenVibe.Community (the paste hand-over), OpenVibe.Billing (job usage |
| 41 | readings) |
| 42 | - host programs some tools need (ffmpeg, yt-dlp, qpdf, poppler, libheif; [Host packages](#host-packages-some-tools-need)) |
| 43 | - in each app: `openvibe-contracts` v0.92.0 (img, audio, docs, gateway), `openvibe-sdk` v0.12.0 and `openvibe-shared` v4.179.0, pinned |
| 44 | by release tarball, and `apps/_shared` (openvibe-tools-shared) by relative path |
| 45 | |
| 46 | ## Capabilities |
| 47 | |
| 48 | Implemented here (the service manifest's `capabilities`, audience `openvibe.tools`): `tools.tool.read` |
| 49 | (the public registry API), `tools.tool.run` (the run API), `tools.job.create`, `tools.job.read` and |
| 50 | `tools.job.cancel` (jobs), and `tools.net.probe` (network probes). |
| 51 | |
| 52 | Called elsewhere, as the service principal `tools` (the OAuth client `tools`): |
| 53 | |
| 54 | | Service | Grant | Why | |
| 55 | |---|---|---| |
| 56 | | OpenVibe.Media | `media.object.upload`, `media.object.read` | job results | |
| 57 | | OpenVibe.Events | `events.event.publish`, `events.subscription.manage` | job and usage events; the `network.user.token_valid_after` subscription | |
| 58 | | OpenVibe.Search | `search.document.write` | tool documents for the network index | |
| 59 | | OpenVibe.Billing | `billing.usage.record` (audience `openvibe.billing`) | one usage reading per ended job | |
| 60 | |
| 61 | ## Layout |
| 62 | |
| 63 | ``` |
| 64 | apps/ |
| 65 | ├── gateway # apex (server-rendered index, family + tool pages, catalog.json), the tool registry |
| 66 | │ # and host roles, Net.OpenVibe + Dev.OpenVibe tools, proxy to the satellites |
| 67 | ├── maps # Maps.OpenVibe — survival map for North America |
| 68 | ├── food # the food finder API (proxies maps backend); its page moved to OpenVibe.Food (301 to openvibe.food) |
| 69 | ├── img # Img.OpenVibe — image converter & processing tools |
| 70 | ├── yt # YT.OpenVibe — YouTube downloader |
| 71 | ├── audio # Audio.OpenVibe — audio converter & effects |
| 72 | ├── text # Text.OpenVibe + Logo.OpenVibe — text generators & logo makers |
| 73 | └── docs # Docs.OpenVibe — PDF & document tools |
| 74 | apps/_shared # openvibe-tools-shared, a versioned package (package.json, CHANGELOG.md): each app declares the |
| 75 | # range it needs ("openvibeToolsShared": "^1.0.0") and refuses to boot when this checkout does not |
| 76 | # satisfy it (version.js). The apps require it by relative path: host roles, internal auth, the job runtime (jobs/), |
| 77 | # the guard (guard/: callers, quotas, sniffing, ffmpeg hardening, egress throttles, abuse log), |
| 78 | # the SSRF guard (egress.js) that every tool reaching a visitor-chosen host or URL goes |
| 79 | # through (public addresses only, checked after DNS and dialled as checked, redirect hops |
| 80 | # re-checked). Visit analytics come from openvibe-shared/analytics. |
| 81 | scripts/analytics-prune.js # operator CLI: PostgreSQL raw analytics retention (dry run by default) |
| 82 | (openvibe-shared is a pinned OpenVibe.Shared release in each app's package.json; no vendor/ copy) |
| 83 | ``` |
| 84 | |
| 85 | ## Ports |
| 86 | |
| 87 | | App | Domain | Port | |
| 88 | |---|---|---| |
| 89 | | gateway | openvibe.tools + net./dev./pastes. + tool aliases | **4001** | |
| 90 | | maps | maps.openvibe.tools | **4010** | |
| 91 | | food | food.openvibe.tools | **4011** | |
| 92 | | img | img.openvibe.tools + format aliases | **4012** | |
| 93 | | yt | yt.openvibe.tools | **4013** | |
| 94 | | audio | audio.openvibe.tools + effect aliases | **4014** | |
| 95 | | text | text.openvibe.tools + logo.openvibe.tools + aliases | **4015** | |
| 96 | | docs | docs.openvibe.tools + pdf.openvibe.tools | **4016** | |
| 97 | |
| 98 | Related services: Network **4000** (SSO/JWKS/themes/shared assets), |
| 99 | Media **4100**, Live **3000**, Community **4200** (pastes live on openvibe.community; |
| 100 | `pastes.openvibe.tools` hands over to it). |
| 101 | |
| 102 | ## Registry, hosts and domains |
| 103 | |
| 104 | `apps/gateway/server/registry` is the one list of tools and families. Each has a **canonical** host |
| 105 | (links, sitemaps, `rel=canonical`), an optional **short** host people type, and **aliases** that 301 |
| 106 | to the short host. Unknown `*.openvibe.tools` hosts 301 to the index; unknown custom domains 404. |
| 107 | The owner can attach hosts (including bought domains) to a tool in `openvibe.network/admin` → |
| 108 | Domains; the gateway merges those overrides every minute. New custom domain on the host: |
| 109 | `sudo deploy/scripts/add-custom-domain.sh <domain>`. |
| 110 | |
| 111 | Public, cacheable outputs on the apex: `/` (every tool by family, planned tools as placeholders), |
| 112 | `/<family>-tools`, `/all-tools`, `/search?q=`, `/sitemap.xml`, `/robots.txt` (and `/tool/<id>`, which 301s to the tool's primary host), |
| 113 | `/llms.txt`, `/api/catalog.json`, `/terms`, `/privacy`, `/dmca`. All of it renders without JavaScript. |
| 114 | |
| 115 | Deploy with `deploy/scripts/deploy.sh`: it runs `ovhost deploy tools` (OpenVibe.Host, strategy |
| 116 | `multi-app`; `--wait-idle`, `--restart`, `--rollback`, `DRY_RUN=1` for `ovhost plan tools`). ovhost pulls |
| 117 | as the checkout owner, installs in each app whose dependencies changed (apps/_* are packages, not apps), |
| 118 | checks every dependency resolves and that the jobs runtime and the guard load, restarts every |
| 119 | `openvibe-tools*` unit, waits for the gateway's `/api/ready` and its `/release.json` to name the new sha |
| 120 | with every unit active, rolls back if they do not, and announces the release (`host.release.published` |
| 121 | to OpenVibe.Events, so open tabs check `/release.json` within seconds). If ovhost is missing, the wrapper exits with an error. |
| 122 | |
| 123 | ## Tool registry API (ADR-027) |
| 124 | |
| 125 | Every catalogue tool has one descriptor (`openvibe-contracts` `tools.tool@1`): how it runs (`execution`: |
| 126 | `client` in the browser, `sync` answered inline, `job`), whether the run API exposes it (`api`, `run`), its input |
| 127 | JSON Schema, files, output, limits, `auth` (anonymous or not; `tools.net.probe` for network probes), `quotaClass`, |
| 128 | `cost`, `egress`, hosts and docs page. Public, read-only, cacheable (capability `tools.tool.read`): |
| 129 | |
| 130 | | Route | Answer | |
| 131 | |---|---| |
| 132 | | `GET /api/v1/tools` | `tools.tool-list@1`, schemas as `{ $ref }`. Filters `?family=` `?execution=client\|sync\|job` `?api=true\|false` `?status=` `?q=` (comma lists allowed); a bad value → 400 `tools.query.invalid` | |
| 133 | | `GET /api/v1/tools/:id` | `tools.tool@1` with the schemas embedded | |
| 134 | | `GET /api/v1/tools/:id/schema` | `{ $schema, $id, $defs: { input, output } }` (what the list's `$ref`s point at) | |
| 135 | |
| 136 | All three answer `Access-Control-Allow-Origin: *` (the apps' own CORS rules do not apply to them), an ETag |
| 137 | (`If-None-Match` → 304) and `Cache-Control: public, max-age=300`, and count against the `/api/` limiter. An unknown |
| 138 | id is 404 problem+json `tools.tool.not_found` (planned placeholders and mirrors say what they are). The gateway |
| 139 | answers for every tool; each satellite answers the same routes for its own tools, with its own live status. |
| 140 | `/api/catalog.json` is unchanged. |
| 141 | |
| 142 | - **Specs next to the code.** `apps/<app>/server/descriptors.js` (img, audio, docs, text, yt; maps holds maps and |
| 143 | food) and `apps/gateway/server/{net,dev}/descriptors.js` are pure data: the operation, job type and preset come |
| 144 | from the app's domain map, net endpoints from `net/config.js`; `apps/_shared/tools/descriptor.js` joins a spec |
| 145 | with the catalogue's family, name, summary and hosts (owner overrides included). |
| 146 | - **Checked.** `apps/gateway/server/registry/descriptors.js` merges them and checks every build with |
| 147 | `contracts.validate('tools.tool@1')`, `contracts.tools.checkDescriptor` and `contracts.tools.checkList`. A failing |
| 148 | descriptor is left out and `/api/ready` lists it (`tool_registry`, degraded), never a crash. |
| 149 | - **Status.** Unavailable when the catalogue says so (net tools without an implementation) or when the satellite |
| 150 | lacks a program the tool needs (`requires`: qpdf, pdftoppm/pdfinfo, heif-dec, ffmpeg, yt-dlp). The gateway reads |
| 151 | each satellite's `GET /api/v1/tools` on loopback every minute and keeps its unavailable marks. |
| 152 | - **Server engines for browser tools.** The pure transforms behind the dev and text pages live in |
| 153 | `apps/gateway/public/js/dev-engine.js`, `apps/text/public/js/text-engine.js` and `format-engine.js`, which the |
| 154 | pages load; `apps/gateway/server/dev/engines.js` and `apps/text/server/engines.js` run the same code in Node, so |
| 155 | those tools say `execution: client, api: true` (the pages stay browser-only). Tools that need a DOM, a CDN |
| 156 | library or isolation (XML minify, HTML to Markdown, the JS minifier/beautifier, the regex tester, the canvas |
| 157 | graphics makers) say `api: false`, as does the YouTube downloader (page-only by decision). |
| 158 | |
| 159 | ## Run API (ADR-027) |
| 160 | |
| 161 | `POST /api/v1/tools/:id/run` runs any tool whose descriptor says `api: true` (capability `tools.tool.run`; |
| 162 | `tools.net.probe` for the network probes). The gateway (`https://openvibe.tools`) answers for every tool; img, |
| 163 | audio and docs answer the same route for their own tools. `openvibe-sdk/tools` (v0.6.0) is its client. |
| 164 | |
| 165 | - **Request** (`tools.run-request@1`): JSON `{ input, files, wait_ms, idempotency_key }`, or |
| 166 | `multipart/form-data` with each upload as a `file` part and the text parts `input` (JSON), `files` (the |
| 167 | file references' JSON), `wait_ms`, `idempotency_key` — exactly what the SDK sends. `?wait_ms=` does the same |
| 168 | as the field; the `Idempotency-Key` header wins over `idempotency_key`. A file reference is |
| 169 | `{ job_id, index }` (a result file of one of your own jobs, on any satellite: one tool's output feeds the |
| 170 | next) or `{ media_id }` (a result of one of your own jobs stored in OpenVibe.Media). Uploads come first, |
| 171 | then references, in order; the count must fit the descriptor's `files`. |
| 172 | - **Answers** (`tools.run@1`): `200 { state: succeeded, tool, result: { data | text | files }, took_ms[, job] }`; |
| 173 | `200 { state: failed | cancelled, tool, error, took_ms[, job] }` when the tool itself fails (its |
| 174 | problem+json: `tools.job.failed`, `tools.pdf.wrong_password`, `tools.net.target_not_public`, …); |
| 175 | `202 { state: queued | running, tool, job, location }` + `Location: /api/v1/jobs/:id` when a job tool has not |
| 176 | finished within `wait_ms` (default 0); `200 + Idempotent-Replayed: true` for a replay. |
| 177 | - **Refusals** (problem+json, before the tool runs): `400 tools.run.invalid`, `401 token.missing | |
| 178 | token.invalid | tools.session_required`, `403 capability.denied | tools.origin.refused`, |
| 179 | `404 tools.tool.not_found | tools.tool.not_runnable` (`api: false`: the YouTube downloader, the regex tester, |
| 180 | the canvas makers…) `| tools.run.file_not_found`, `405`, `409 tools.job.idempotency_conflict`, |
| 181 | `413 tools.file.too_large | tools.input.too_large` (over `limits.maxInputBytes`), `415 |
| 182 | tools.file.unsupported_type` (the bytes are checked), `422 tools.input.invalid` (`errors[]` with JSON |
| 183 | pointers, against the descriptor's input schema), `429 tools.quota.exceeded` (`Retry-After`), `503 |
| 184 | tools.tool.unavailable` (the descriptor's status is `unavailable`: Protect PDF while qpdf is |
| 185 | missing), `503 tools.unavailable` (the engine found a program it needs missing: e.g. an HEIC upload without |
| 186 | libheif), `503 tools.busy` (`Retry-After`). |
| 187 | - **Who.** Always enforced (authorization, not a quota): an app or service token needs the descriptor's |
| 188 | `auth.capability`; the network probes (port, ping, latency) run only for tokens holding `tools.net.probe` |
| 189 | (anonymous `401`, a person `403`: people use the probe pages); `auth.anonymous: false` tools (audio, PDF, SMTP) |
| 190 | need a browser session, a sign-in or a token; a Bearer token that does not verify is `401 token.invalid`, not |
| 191 | anonymity. Then the guard: the tool's `quotaClass` × the caller's tier, weighted by `cost` (mode-dependent), |
| 192 | the per-target throttle for egress tools and upload sniffing (hard). |
| 193 | - **Where it runs.** Dev and text tools with a server engine: the page's own engine code in the gateway's |
| 194 | worker pool (`TOOLS_WORKERS_GATEWAY`, default 2, 256 MB each; the descriptor's `timeoutMs` terminates a |
| 195 | runaway input). Net tools and Open Graph: the gateway's net and dev handlers (once `/api/net` and |
| 196 | `/api/dev`, now reachable only through the run API), in process, with the run's input as their query |
| 197 | (`myip` is the caller's address). Job tools (img, audio, docs): a |
| 198 | job of the satellite that owns the tool (`{ ...input, ...preset, tool: operation }`), the run streamed there |
| 199 | by the gateway, uploads included; the job's `tool` names the tool. |
| 200 | - **Cache and replays.** An inline answer is reused per tool and input where that is safe (spec `cacheTtlMs`: |
| 201 | engines 10 minutes unless random or time-dependent — uuid, lorem, timestamp, cron, jwt, zalgo, case, sort, bio, |
| 202 | nickname never; DNS 1 minute, IP 10, RDAP/whois 60, a few other lookups 5; probes, headers, redirects, uptime, |
| 203 | SMTP and myip never), answered with `X-OV-Cache: hit` and without touching the target. An inline run's |
| 204 | `Idempotency-Key` replays its first answer for 15 minutes (per caller); a job tool's key is its job's. |
| 205 | - **CORS.** First-party pages keep the credentialed rules; any other site's page gets `Access-Control-Allow-Origin: *` |
| 206 | without credentials on the run and jobs routes (so its calls carry a token, never our cookies). |
| 207 | - `GET /api/health` on the gateway shows `run` (counts, cache, engine pool) and `jobs` (the facade). |
| 208 | |
| 209 | ## Jobs facade (gateway) |
| 210 | |
| 211 | `https://openvibe.tools/api/v1/jobs…` fronts every satellite's job routes (below), so one origin serves the |
| 212 | run API's `Location`, `openvibe-sdk/jobs` without a `baseUrl`, and anything else. A submit is routed by its job |
| 213 | type's prefix (`img.process` → img …): `?type=`, the JSON body's `type`, or the multipart `type` part (read from |
| 214 | the first 64 KB of the stream — send it before the files; the uploads then stream through untouched). Every other |
| 215 | route (`GET`/`DELETE /:id`, `/:id/events`, `/:id/files/:n`, `/:id/retry`, `/:id/references/:ref`) is routed by the |
| 216 | job id: the satellite that answered its submit or run (remembered, 50,000 ids), else the one that says it holds |
| 217 | the job when asked on loopback (`GET /api/internal/jobs/:id`, direct loopback callers only, never rate |
| 218 | limited). Events (SSE) and files stream through. Owners, quotas, the Origin check and idempotency stay the |
| 219 | satellite's: the request reaches it with the caller's credentials and address. |
| 220 | |
| 221 | ## Removed endpoints (plan T8, phase 6) |
| 222 | |
| 223 | `/api/process`, `/api/process/direct`, `/api/process/multi` (img, audio, docs), `/api/net/*` and `/api/dev/*` |
| 224 | (with the `/api/net/tools` and `/api/dev/tools` lists) are gone; nothing answers them any more. Their |
| 225 | successors: `POST /api/v1/tools/{id}/run` for any tool (or the job API below for img, audio and docs), and |
| 226 | `GET /api/v1/tools?family=…` for the lists. The webhook bins (`/api/dev/webhook/…`, page-only) keep |
| 227 | their URL on their own router. |
| 228 | |
| 229 | ## Jobs (Img, Audio, Docs) |
| 230 | |
| 231 | Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same |
| 232 | routes answer on every host of the img, audio and docs satellites: |
| 233 | |
| 234 | | Route | What it does | |
| 235 | |---|---| |
| 236 | | `POST /api/v1/jobs` | Submit: multipart `type`, `input` (JSON text), `file` / `files`, or JSON `{ type, input }`. `Idempotency-Key` header (8–200 chars). → `202` + `Location`; a repeat with the same key and request → `200` + `Idempotent-Replayed: true` (same job); same key, different request → `409 tools.job.idempotency_conflict`. | |
| 237 | | `GET /api/v1/jobs/:id` | The job: `state` (`queued`, `running`, `succeeded`, `failed`, `cancelled`), `progress {percent, message}`, `attempts`, `result {files, data}`, `error` (problem+json), `retryable`, `expires_at`. | |
| 238 | | `DELETE /api/v1/jobs/:id` | Cancel: queued → `200` cancelled at once; running → `202` (signalled; ffmpeg is killed, a sharp/pdf-lib worker thread is terminated) and ends `cancelled`; finished → `409 tools.job.already_finished`. | |
| 239 | | `GET /api/v1/jobs/:id/events` | SSE: `job.queued`, `job.running`, `job.progress`, `job.cancel_requested`, `job.succeeded`, `job.failed`, `job.cancelled`, each with an `id`. Reconnecting with `Last-Event-ID` (or `?last_event_id=`) replays only later events; a finished job with nothing newer answers `204`. | |
| 240 | | `GET /api/v1/jobs/:id/files/:n` | A result file (attachment); `?inline=1` for previews. | |
| 241 | | `POST /api/v1/jobs/:id/retry` | Retry a **failed** job: a new job with the same type, input and files (`retry_of` → the failed one, which gets `retried_by`) → `202` + `Location`. Idempotent: asking again returns that same retry with `200` + `Idempotent-Replayed: true`. Not failed → `409 tools.job.not_failed`; inputs gone → `410 tools.job.inputs_gone`. A failed job keeps its input files until it expires. | |
| 242 | | `PUT /api/v1/jobs/:id/references/:ref` | Keep a **succeeded** job's result while `<ref>` (`<service>:<kind>:<id>`, e.g. `community:paste:p_123`) points at it → `201` (`200` if already there). The job then shows `references` and `expires_at: null`. At most 50 per job; not for sandbox jobs. | |
| 243 | | `DELETE /api/v1/jobs/:id/references/:ref` | Drop it → `200`; after the last one the job expires one ttl later at the earliest. | |
| 244 | |
| 245 | Job types: `img.process` (input `{ tool: convert|compress|resize|crop, format, quality, width, … }`, one image), |
| 246 | `audio.process` (`{ tool, …options }`, one audio/video file; progress from ffmpeg), |
| 247 | `docs.process` (`{ tool, …options }`, one PDF, or several files for `merge` / `img2pdf`). A format host fills |
| 248 | in its format (`webp.openvibe.tools` converts to WebP). The run API (`POST /api/v1/tools/{id}/run`) submits the |
| 249 | same job and waits for it; the pages use jobs and keep the job id in the address (`?job=`) and in sessionStorage, so a |
| 250 | reload reattaches. |
| 251 | |
| 252 | - **Durable.** A job and its uploaded input are in the `tools` database (`tool_jobs`) and `data/jobs/<id>/` before the `202` is sent. |
| 253 | After a restart, queued jobs run; jobs that were running are re-queued (all three types, up to their attempt |
| 254 | limit) — a type can instead declare `onRestart: 'fail'`, which fails them with `retryable: true`. |
| 255 | - **Owner-scoped.** A job is visible only to whoever created it: a signed-in person (`user:usr_…` from the |
| 256 | `ov_token` subject), an app/service principal (Network client-credentials token, audience `openvibe.tools`, |
| 257 | capabilities `tools.job.create|read|cancel`), or else this browser's `ov_tools_jobs` cookie (only its hash is |
| 258 | stored). Anyone else gets `404 tools.job.not_found`. |
| 259 | - **Worker threads.** sharp (img) and pdf-lib (docs) never run on the event loop: `apps/_shared/jobs/pool.js` |
| 260 | runs them in `worker_threads`, `TOOLS_WORKERS` at once per satellite (default 2; `TOOLS_WORKERS_<APP>`), shared |
| 261 | by the jobs and the synchronous calls that remain (docs' `/api/info`; they submit to the pool and wait), with |
| 262 | `TOOLS_WORKER_QUEUE` (64) more waiting before `503 tools.busy`. Each worker has a V8 heap limit |
| 263 | (`TOOLS_WORKER_MEMORY_MB`, 512; `TOOLS_WORKER_MEMORY_MB_<APP>`): a run that needs more fails with |
| 264 | `413 tools.input.too_large` and its worker is replaced. A run past its timeout (the job's, or the descriptor's |
| 265 | `limits.timeoutMs` on the sync endpoints: `504 tools.run.timeout`), a cancelled job or a sync client that went |
| 266 | away has its worker terminated at once. Idle workers stop after `TOOLS_WORKER_IDLE_MS` (60 s). qpdf and poppler |
| 267 | tools stay on the main thread (their child processes have their own timeouts). `GET /api/health` shows `workers`. |
| 268 | - **Bounded.** `TOOLS_JOBS_CONCURRENCY` jobs run at once per satellite (default 2; `TOOLS_JOBS_CONCURRENCY_<APP>` |
| 269 | overrides), `TOOLS_JOBS_MAX_ACTIVE` unfinished jobs per owner (default 10, then `429`), and all browser sessions |
| 270 | from one address together `TOOLS_JOBS_MAX_ACTIVE_PER_ADDRESS` (default 3 × that; jobs keep only the guard's hashed |
| 271 | address key), at most |
| 272 | `TOOLS_JOBS_MAX_QUEUED` queued jobs in all (default 200) and `TOOLS_DISK_BUDGET_MB` under `data/jobs` (default |
| 273 | 8192) before a submit answers `503 tools.busy` with `Retry-After` (through the guard, below), plus the guard's |
| 274 | quotas on submit. |
| 275 | - **Retention.** Finished jobs expire after 1 hour (browser sessions) or 24 hours (signed-in people, principals); |
| 276 | the pruner deletes the row, its events, its files and its Media objects. It never touches a job that has a |
| 277 | reference (above). If Media will not delete a result object (a retention hold, `409 media.object.held`), cannot |
| 278 | be reached, or is no longer configured here, the job is kept, record and all, and looked at again 24 hours |
| 279 | later, so no object is left without the record that would delete it. |
| 280 | - **Results in OpenVibe.Media** when `TOOLS_JOB_RESULTS=media`: each result file becomes a private Media object |
| 281 | (v2 object API, namespace `tools`, owner `X-OV-Subject` for signed-in people) and the job's result carries its |
| 282 | `media.media_id`; previews redirect to a short-lived signed Media URL, downloads stream through the satellite. |
| 283 | This needs `OV_OAUTH_CLIENT_SECRET` and Network grants for the `tools` client: |
| 284 | `media.object.upload` and `media.object.read` for audience `openvibe.media`, namespace `tools`, and a `tools` |
| 285 | tenant in Media. Without them (or if an upload fails) results stay on local disk and the file says |
| 286 | `storage: "local"`. Default: `local`. |
| 287 | - **Lifecycle events to OpenVibe.Events** when `EVENTS_URL` is set (`apps/_shared/jobs/events.js`): |
| 288 | `tools.job.created` (submit or retry; actor the owner), `tools.job.started` (a worker claimed it; a job |
| 289 | re-queued after a restart is announced again when it starts), `tools.job.succeeded` and `tools.job.failed` |
| 290 | (payloads `openvibe-contracts` `tools.job.*@1`, validated before they are queued; subject `job <id>`, visibility |
| 291 | `internal`, source `tools`). Each is written to the shared PostgreSQL `event_outbox` table (openvibe-sdk |
| 292 | `createPgOutbox`) in the same transaction as the state change, so an event exists exactly when its |
| 293 | transition committed, and relayed to `EVENTS_URL/api/v1/events` (at least once; Events dedupes on `event_id`). |
| 294 | Payloads carry ids, type, owner, state, attempts, times, where result files are (index, mime, size, sha256, |
| 295 | `local` or a Media `media_id`) and the error (`status`, `code`, `detail` with server paths and the job's file |
| 296 | names taken out) — never the input, file names, output data or a browser session: a session's job has |
| 297 | `owner: null`. Cancelled jobs, progress and sandbox app jobs are not announced; an Idempotency-Key replay |
| 298 | creates no job and no event. The relay uses the `tools` client (`OV_OAUTH_CLIENT_ID`, `OV_OAUTH_CLIENT_SECRET`) |
| 299 | with `events.event.publish` on audience `openvibe.events`. Without `EVENTS_URL` (or with `EVENTS_PUBLISH=off`, or |
| 300 | no client secret) nothing is written or sent. `GET /api/health` shows `jobs.events` (pending, rejected). |
| 301 | - **Project usage to OpenVibe.Events** on the same outbox (`apps/_shared/jobs/usage.js`, roadmap WS-N task 4): a |
| 302 | developer app's job (it carries a project) that ends is counted in the transaction that records its end, per |
| 303 | project, environment, capability (`tools.tool.run` for a tool run, `tools.job.create` for `POST /api/v1/jobs`), |
| 304 | job type or tool and UTC hour: the jobs that ended (succeeded or failed; cancelled ones are not counted) and the |
| 305 | failed ones by problem code, with the last ten failures' job id, status and the trace id of the request that |
| 306 | submitted the job (`tool_jobs.trace_id`). A minute after an hour closes, the pruner's timer writes each rollup |
| 307 | once as `tools.usage.recorded` (`common.usage-recorded@1`, openvibe-contracts 0.63.0; subject the project, |
| 308 | visibility `internal`, priority `low`) in the transaction that marks it sent. Sandbox jobs count under |
| 309 | `env: sandbox`. No owner, session, address, input, file name or output leaves. OpenVibe.Network adds the rollups |
| 310 | up for the project's usage page on openvibe.services. Off without `EVENTS_URL`, like the job events; |
| 311 | `GET /api/health` shows `jobs.usage` (pending, invalid). |
| 312 | - **Usage readings to OpenVibe.Billing** (`apps/_shared/billing.js`, `apps/_shared/jobs/usage.js`, plan T5 step 7): |
| 313 | every job that succeeds or fails (anyone's, not only a project's; a cancelled one is not billed) gets one |
| 314 | `platform.usage-sample@1` reading — id `tools-job-<id>`, `idempotency_key` `tools:job:<id>`, service `tools` (the |
| 315 | Contracts manifest id, so Billing keys ratings and budgets on the reading) and source `openvibe.tools`, operation |
| 316 | `tools.tool.run` (a tool run) or `tools.job.create`, resource the tool or job type, |
| 317 | quantity 1, unit `jobs`, `at` the job's end, plus `project` and `trace_id` when the job has them and `subject` |
| 318 | (`user:usr_…`) only for a signed-in person's job, never an app or a browser session. It is stored in |
| 319 | `tool_job_billing_readings` in the transaction that records the end (no network call there); the pruner's timer |
| 320 | posts the unsent ones (500 at most a tick) to `POST /api/v1/usage` and marks each sent. A refused post is logged, |
| 321 | never thrown: the reading keeps its error and attempt count and is posted again on the next tick (Billing answers |
| 322 | a replay with 200), and an outage, a 401/403, a 429 or a refused token stops that tick's batch. The hourly `tools.usage.recorded` |
| 323 | rollups above are unchanged and stay subject-less. Off without `OV_BILLING_URL` (nothing is stored). The token is |
| 324 | Network `client_credentials` for the `tools` client on audience `OV_BILLING_AUDIENCE` (default |
| 325 | `openvibe.billing`); Network must grant that client `billing.usage.record` on it — until the grant exists every |
| 326 | token request is refused (400 invalid_scope) and only logged, and the readings wait in the table. `GET /api/health` shows |
| 327 | `jobs.billing` (pending, posted, refused, invalid, last_error) when it is on. |
| 328 | |
| 329 | Environment (all in `/etc/openvibe/tools.env`): `TOOLS_WORKERS`, `TOOLS_WORKERS_<APP>`, `TOOLS_WORKER_MEMORY_MB`, |
| 330 | `TOOLS_WORKER_MEMORY_MB_<APP>`, `TOOLS_WORKER_QUEUE`, `TOOLS_WORKER_IDLE_MS`, `TOOLS_JOBS_CONCURRENCY`, `TOOLS_JOBS_CONCURRENCY_<APP>`, |
| 331 | `TOOLS_JOBS_MAX_ACTIVE`, `TOOLS_JOB_RESULTS`, `TOOLS_MEDIA_NAMESPACE`, `OV_MEDIA_INTERNAL_URL`, `OV_MEDIA_URL`, |
| 332 | `OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID`, `OV_OAUTH_CLIENT_SECRET`, `EVENTS_URL`, `EVENTS_PUBLISH`, |
| 333 | `EVENTS_RELAY_INTERVAL_MS`, `OV_BILLING_URL`, `OV_BILLING_AUDIENCE`. |
| 334 | |
| 335 | Not done yet: quotas for external developer apps are the guard's service and sandbox tiers, not a Codes-issued quota. |
| 336 | |
| 337 | ## Guard (anti-abuse, `apps/_shared/guard`) |
| 338 | |
| 339 | One module used by the gateway and every satellite, driven by each tool's descriptor (`quotaClass`, `cost`, `auth`, |
| 340 | `files.accept`, `limits`) and the numbers in `apps/_shared/guard/limits.js` (the one place they live). |
| 341 | |
| 342 | - **Who is asking** (`caller.js`, replacing the four copied `auth.js` files): a **service or app principal** |
| 343 | (Bearer client-credentials token, audience `openvibe.tools`; a developer app's sandbox token is tier `sandbox`), a |
| 344 | **user** (the `ov_token` cookie or a Bearer Network token, verified offline, issuer **and audience |
| 345 | `openvibe.tools`** — the Network puts that audience on every browser sign-in; its `/internal/issue-token` tokens |
| 346 | carry none and are not believed), a **session** (this browser's `ov_tools_jobs` cookie; the page's `/api/context` |
| 347 | call and the first job or webhook bin start one) or else **anonymous**, counted by address (IPv6 by its /64). |
| 348 | Tiers: anonymous < session < user < service. |
| 349 | - **One address source.** Every app sets `trust proxy` to `TRUST_PROXY`: exactly one hop, and only on loopback (the |
| 350 | host's nginx, which sets `X-Forwarded-For` from `$remote_addr` after Cloudflare's real IP). `req.ip` is the only |
| 351 | address anything reads. The gateway's proxy to a satellite passes its own `req.ip` as that one hop; food passes its |
| 352 | visitor's to maps. A first-party service calling on loopback is identified by its service token (each has its own |
| 353 | bucket at the service tier); without one it is anonymous, counted by the address it forwarded or its own. |
| 354 | - **Quotas.** A token bucket per quota class × tier (`perMinute`, `burst`) and a UTC-day allowance in |
| 355 | Valkey when configured (process-local otherwise), both counting the tool's `cost`. Browser sessions from one address share |
| 356 | 3 × one session's allowance, so dropping the cookie never resets anything. Counted answers carry |
| 357 | `RateLimit-Limit`/`-Remaining`/`-Reset`; a refusal is `429` problem+json `tools.quota.exceeded` with `Retry-After` |
| 358 | and `quota_class`, `scope`, `tier`. |
| 359 | - **Heavy work.** A synchronous heavy call (docs' `/api/info`) holds one of `TOOLS_SYNC_CONCURRENCY` slots |
| 360 | (default 2; `TOOLS_SYNC_QUEUE` may wait, 8, for `TOOLS_SYNC_WAIT_MS`, 30 s), else `503 tools.busy`. Every sync |
| 361 | path is stopped at the descriptor's `timeoutMs` (`504 tools.run.timeout`) or when the client goes away: audio's |
| 362 | ffmpeg is killed, img's and docs' worker thread is terminated (Jobs → Worker threads). Liveness and readiness |
| 363 | probes (`GET /api/health`, `/api/ready`) are never rate limited. |
| 364 | - **Uploads.** The bytes are checked against the descriptor's `files.accept` (`415 tools.file.unsupported_type`; |
| 365 | a misleading extension is corrected); docs uploads go to disk. Tools whose descriptor says `auth.anonymous: false` |
| 366 | (audio, PDF) need a session, a sign-in or a token (`401 tools.session_required`). Images: no input over |
| 367 | `TOOLS_MAX_INPUT_PIXELS` (40 MP, as Media) is decoded (header first, sharp `limitInputPixels` behind it). |
| 368 | - **ffmpeg.** Every input and probe: `-protocol_whitelist file,pipe`, `-format_whitelist` of the operation's |
| 369 | demuxers (never hls, concat, image2, lavfi…), `-t` = the descriptor's `limits.maxDurationSec` (3 h); inputs are |
| 370 | probed first and a longer one is refused (`413`). |
| 371 | - **Egress.** A per-target throttle across all callers and tools (`limits.perTargetPerMinute`; a host's minute is |
| 372 | shared by headers, SSL, ping…); the port checker takes at most 20 ports a request and 100 ports / 10 hosts per |
| 373 | caller in 10 minutes (sessions counted by address). Probes through a token need `tools.net.probe`. Webhook bins |
| 374 | belong to their maker (session, sign-in or token; anyone else gets 404), at most 5 per owner and 10 per address. |
| 375 | YouTube downloads belong to whoever started them. Nominatim is paced to one request a second; Overpass waits are |
| 376 | capped. |
| 377 | - **Abuse log.** Every refusal, and every would-be refusal in report mode, goes to PostgreSQL `guard_abuse`: time, `HMAC-SHA256` |
| 378 | of the address (or /64) with a random daily salt (only today's is kept, so older hashes cannot be tied to |
| 379 | anything), principal or user id, tool, reason, enforced or not; repeats within a minute are one row with a count; |
| 380 | kept 30 days. No raw address is stored anywhere (pseudonymous, ADR-021). Metrics: `tools_guard_refused_total{reason,tool}`, |
| 381 | `tools_guard_enforcing`, `tools_guard_sync{kind}`. |
| 382 | - **Challenge hook.** `createGuard({ challenge })` takes a person-check provider (`required`, `verify`; Turnstile |
| 383 | later); the default challenges nobody. |
| 384 | - **Cross-site requests (CSRF).** A mutation that rides on the browser's cookies (`ov_token`, the jobs session) |
| 385 | must come from an OpenVibe.Tools page: `Origin` (else `Referer`) `https://openvibe.tools`, |
| 386 | `https://*.openvibe.tools`, one of the tool's own hosts (custom domains included) or the request's own host. |
| 387 | Calls with a Bearer token, calls without those cookies and requests with neither header pass. Checked on the |
| 388 | run API, the job routes' writes, `/api/info`, `/api/probe` and the webhook bins' create and |
| 389 | delete; `403 tools.origin.refused`, recorded as reason `origin`. |
| 390 | - **Mode.** `TOOLS_GUARD=enforce` (default, production) refuses what is past a limit with `429` |
| 391 | problem+json `tools.quota.exceeded`. `TOOLS_GUARD=report` (local debugging) records and counts what it |
| 392 | would refuse and refuses nothing. Hard limits apply in both modes: the pixel limit, ffmpeg's whitelists |
| 393 | and duration cap, upload sniffing, the port-scan cap and the per-target throttle. |
| 394 | |
| 395 | Default quotas (units = descriptor `cost` per run; `perMinute` / `burst` / `perDay`, 0 = none): |
| 396 | |
| 397 | | Class | anonymous | session | user | service | sandbox | |
| 398 | |---|---|---|---|---|---| |
| 399 | | `tools-api` (every `/api/` call, cost 1) | 120 / 60 / – | 180 / 90 / – | 480 / 240 / – | 2400 / 1200 / – | 60 / 30 / – | |
| 400 | | `tools-run` (text, dev engines) | 120 / 60 / 5000 | 180 / 90 / 10000 | 480 / 240 / 50000 | 2400 / 1200 / 500000 | 60 / 30 / 1000 | |
| 401 | | `tools-fetch` (lookups, Open Graph) | 60 / 30 / 2000 | 90 / 45 / 4000 | 240 / 120 / 20000 | 1200 / 600 / 200000 | 30 / 15 / 500 | |
| 402 | | `tools-probe` (port, ping, latency) | 30 / 15 / 600 | 45 / 25 / 1000 | 120 / 60 / 5000 | 600 / 300 / 50000 | 15 / 10 / 200 | |
| 403 | | `tools-job` (img, audio, docs) | 60 / 30 / 1500 | 90 / 45 / 3000 | 240 / 120 / 20000 | 1200 / 600 / 200000 | 30 / 15 / 300 | |
| 404 | | `tools-map` (maps, food) | 60 / 40 / 6000 | 90 / 60 / 8000 | 240 / 120 / 20000 | 1200 / 600 / 200000 | 30 / 20 / 500 | |
| 405 | | `tools-download` (yt, cost 50) | 50 / 100 / 1500 | 75 / 150 / 2500 | 150 / 300 / 5000 | 300 / 600 / 10000 | 25 / 50 / 100 | |
| 406 | |
| 407 | Environment: `TOOLS_GUARD`, `TOOLS_GUARD_LIMITS` (JSON merged into the table, e.g. |
| 408 | `{"tools-job":{"anonymous":{"perDay":800}}}`), `TOOLS_SYNC_CONCURRENCY`, `TOOLS_SYNC_QUEUE`, `TOOLS_SYNC_WAIT_MS`, |
| 409 | `TOOLS_JOBS_MAX_QUEUED`, `TOOLS_JOBS_MAX_ACTIVE_PER_ADDRESS`, `TOOLS_DISK_BUDGET_MB`, `TOOLS_MAX_INPUT_PIXELS`, `TOOLS_PORTS_PER_CALLER`, |
| 410 | `TOOLS_PORT_TARGETS_PER_CALLER`, `AUDIO_MAX_DURATION`, `OV_TOOLS_AUDIENCE` (white-label installs). |
| 411 | |
| 412 | ## Per-actor limits (`apps/_shared/actor-limits.js`) |
| 413 | |
| 414 | Roadmap WS-R task 4, openvibe-sdk/limits, one limiter per app, only where the guard has no per-caller |
| 415 | limit of its own. The guard, its quotas and `TOOLS_GUARD` are unchanged; these limits |
| 416 | read the guard's resolved caller (`guard.caller`): a person as `user:usr_…`, a service or app by its |
| 417 | principal, a browser session or nobody by the guard's hashed address key (`ip:<HMAC>`, never a raw |
| 418 | address). Past a limit: `429` problem+json `rate_limited` with `Retry-After`, one `[Limits]` log line and |
| 419 | `tools_rate_limited_total{limit,window}`. |
| 420 | |
| 421 | | Where | Routes | Per caller, a minute / an hour | |
| 422 | |---|---|---| |
| 423 | | every app | Registry reads `GET`/`HEAD /api/v1/tools[/:id[/schema]]` by a signed-in person or a third-party principal (`app:…`, `mod:…`) | `TOOLS_LIMITS_MINUTE` / `TOOLS_LIMITS_HOUR` (120 / 3000) | |
| 424 | | gateway, img, audio, docs | Backstop on job submits and retries (`POST /api/v1/jobs`, `…/:id/retry`) and runs (`POST /api/v1/tools/:id/run`) | a person or an address 600 / 20 000; a principal 3000 / 100 000 | |
| 425 | | gateway | `GET /api/internal/analytics` (the Network admin's analytics page, with a Network service token) | 30 / 300 | |
| 426 | |
| 427 | Signed-out registry reads keep only the per-address `/api/` limit (many visitors share a carrier or campus |
| 428 | address); a first-party service (`svc:…`) reading for itself is not counted. The backstop sits above the |
| 429 | guard's highest allowances (tools-run: a person 480 a minute, a service 2400, one address's sessions |
| 430 | together 540; the day allowances average at most about 2100 and 21 000 an hour), so the guard still |
| 431 | decides every normal caller's allowance, and a runaway caller is stopped even while the guard only |
| 432 | reports. A run the gateway streams to a satellite is counted there again. Never limited: `/api/health`, |
| 433 | `/api/ready`, `/release.json`, `/metrics`, the pages, `/api/internal/jobs/:id` and the signed |
| 434 | `/internal/events` deliveries. Tests: `apps/_shared/test/actor-limits.test.js` (fixed clock) and |
| 435 | `apps/gateway/test/actor-limits.test.js` (the real gateway). |
| 436 | |
| 437 | ## Canonical hosts and the service registry |
| 438 | |
| 439 | Every satellite honours the gateway's `X-OV-Tool` / `X-OV-Host-Role` / `X-OV-Canonical-Host` / `X-OV-Short-Host` |
| 440 | headers (`apps/_shared/host-role.js`): canonical links, `og:url` and JSON-LD use the canonical host (a custom |
| 441 | domain included), a custom domain gets its tool's page through `X-OV-Tool`, aliases the gateway missed are |
| 442 | redirected, and a host a satellite does not serve goes to the tools index. Reached directly, each host is its |
| 443 | own canonical as before. |
| 444 | |
| 445 | The gateway resolves the other services' origins (Community for pastes, the "elsewhere on OpenVibe" links) |
| 446 | through Network's registry, `GET /api/v1/registry/services` (fetched on boot and every 10 minutes; the last good |
| 447 | answer is kept). Until it answers, a local list is used and `/api/catalog.json` says `services.source: "fallback"`. |
| 448 | Services the registry marks `placeholder` or `retired` are not linked. `OV_REGISTRY_URL` overrides the URL. |
| 449 | |
| 450 | ## Metrics and readiness |
| 451 | |
| 452 | Every server (gateway and satellites) mounts `apps/_shared/observe.js` with `openvibe-shared/metrics` and |
| 453 | `openvibe-shared/ready`: |
| 454 | |
| 455 | - `GET /metrics` — Prometheus text for **direct loopback callers only** (127.0.0.1/::1 with no |
| 456 | `X-Forwarded-For`/`X-Real-IP`); anything through nginx or the gateway gets 404, and each nginx server block |
| 457 | also answers `location = /metrics` with 404. HTTP golden signals are labelled by route template |
| 458 | (`/api/v1/jobs/:id`), proxied satellite traffic on the gateway by satellite (`proxy:img`), SPA pages and |
| 459 | static files by a fixed label — never the raw URL. `release_info{service="tools"|"tools-<app>"}`, process |
| 460 | metrics, and on img/audio/docs `tools_jobs{app,state}` (a group-by on the shared job store) and |
| 461 | `tools_jobs_executing{app,kind="executing"|"limit"}`. |
| 462 | - `GET /api/ready` — named checks with `status`, `required`, `latency_ms`, `checked_at`; HTTP 503 only when a |
| 463 | required check fails, otherwise 200 with failed optional checks listed in `degraded`. `/api/health` is |
| 464 | unchanged. |
| 465 | - **Request telemetry** — counted by route template, method and status class, not raw URL; `/shared`, static assets, |
| 466 | health, readiness and `/metrics` do not count. Once a minute one `http.request` sample per key plus the active- |
| 467 | requests, p95 latency and event-loop-lag gauges goes to `telemetry_samples` (openvibe-sdk 0.46, |
| 468 | `platform.telemetry-sample@1`, `migrations/0006_telemetry.sql`); the gateway alone deletes rows older than 14 days |
| 469 | daily. Wired by `apps/_shared/observe.js`; each `server/index.js` calls `telemetry.init()` with its service id |
| 470 | (`tools`, `tools-img`, …) into the one `tools` database. |
| 471 | |
| 472 | | Server | Required | Optional (degraded when failing) | |
| 473 | |---|---|---| |
| 474 | | gateway (4001) | `catalog`, `tools_db` | `network_key`, `service_directory`, `community`, `satellite_<app>`, `tool_registry` | |
| 475 | | img, docs (4012, 4016) | `tools_db`, `job_runtime`, `data_dir`, `uploads_dir`, `output_dir` | `network_key`, `media_results` (when enabled); img: `heif_decoder`; docs: `qpdf`, `pdftoppm`, `pdfinfo` | |
| 476 | | audio (4014) | same as img | same as img, plus `ffmpeg` | |
| 477 | | yt (4013) | `tools_db`, `downloads_dir` | `yt_dlp`, `ffmpeg`, `yt_cookies` (when set), `network_key` | |
| 478 | | food (4011) | `tools_db`, `maps` | — | |
| 479 | | maps, text (4010, 4015) | `tools_db` | — | |
| 480 | |
| 481 | `TOOLS_SATELLITE_PORTS="img=5012,…"` overrides the gateway's satellite ports (tests, a moved unit); every app reads |
| 482 | it (`apps/_shared/tools/satellites.js`), so set it for all units if a port moves. |
| 483 | |
| 484 | **Releases (ADR-016).** Every app mounts openvibe-shared's `release.mount` (`apps/_shared/release.js`): `GET |
| 485 | /release.json` and `POST /release-metrics` (the shared navbar's release-watch outcomes → |
| 486 | `release_client_updates_total` on `/metrics`). The gateway, img, audio and docs (which pin openvibe-contracts) serve |
| 487 | manifest 1.1.0 components: `shell` (what their pages are made of: `public/`, the server files that stamp the pages, |
| 488 | the job helper) and `server` (the rest), so a release that changes only API code no longer reloads open tabs. Maps, |
| 489 | food, text and yt serve the 1.0.0 fields. |
| 490 | |
| 491 | ## Tests |
| 492 | |
| 493 | `npm test` (Node 22) syntax-checks every server file and runs every `apps/*/test/*.test.js`, each in its own |
| 494 | process: the job runtime end to end (restart, reattach, cancel, idempotency, owner scoping, SSE resume, pruning, |
| 495 | Media results with a stand-in Media), img/audio/docs as real processes killed with SIGKILL mid-job, canonical |
| 496 | hosts on every satellite, the registry-driven catalog, and `registry-consistency.test.js`: every catalogue |
| 497 | tool resolves to an operation, endpoint or page that exists, or is marked unavailable. `descriptors.test.js` |
| 498 | holds every descriptor to the contracts and to the code (a job's operation, validation, files and limits; a sync |
| 499 | tool's route), `engines.test.js` runs every browser tool's server engine in plain Node against its output schema, |
| 500 | and `tools-api.test.js` checks the three routes on the gateway and all seven satellites. `run-api.test.js` runs |
| 501 | the run API on the gateway, img and docs (with a stand-in Network for tokens, `apps/_shared/test/network.js`): |
| 502 | every answer against `tools.run@1`, the refusal codes, probes, quotas, idempotency, job tools and references |
| 503 | across satellites, the Origin check and the deprecation headers; `jobs-facade.test.js` routes jobs through the |
| 504 | gateway (by type, by id after a restart, events, files, retry, cancel); `sdk-client.test.js` drives |
| 505 | the pinned openvibe-sdk's `createToolsClient` against the real gateway (it requires a checkout of OpenVibe.SDK next to |
| 506 | this repo, or `OV_SDK_DIR`, and says it skipped without one); `pool.test.js` (shared and docs) hold the worker |
| 507 | pool to its limits. The audio tests need |
| 508 | `ffmpeg` and skip without it. The docs and img tests always check the 503 path with qpdf, poppler and libheif |
| 509 | pointed at nothing, and also run the real encrypt/decrypt, page rendering and HEIC decoding when those programs |
| 510 | are installed (or `QPDF_PATH` / `HEIF_DEC_PATH` point at them). Install first with `npm run install:all`. |
| 511 | |
| 512 | ## Analytics (ADR-021) |
| 513 | |
| 514 | All eight apps store visit analytics in the shared PostgreSQL database through |
| 515 | `openvibe-shared/analytics/pg` (ADR-021). Raw events include the service, route template, |
| 516 | method, status, response time, session identifier and coarse visitor metadata. GPC and DNT opt-outs |
| 517 | are excluded. Daily and hourly rollups remain after raw events expire. |
| 518 | |
| 519 | - **Retention:** the application prunes raw rows older than 30 days; rollups stay. |
| 520 | - **CLI:** `node scripts/analytics-prune.js` counts old rows by service. Pass `--days N` to choose |
| 521 | retention and `--apply` to delete them. It requires `DATABASE_URL` and applies to every service. |
| 522 | |
| 523 | ## Host packages some tools need |
| 524 | |
| 525 | A few operations run a program that is not part of Node. Each satellite looks for it at boot (on `PATH`, |
| 526 | or at the path in its environment variable) and again every minute while it is missing; until it is |
| 527 | there, the operation answers **503 problem `tools.unavailable`** ("this tool is being set up") on |
| 528 | the run API and at job submit, its page says so instead of offering the upload, and `/api/ready` |
| 529 | lists it under `degraded`. Nothing pretends to work. |
| 530 | |
| 531 | | Package (Ubuntu) | Program | Used by | Override | |
| 532 | |---|---|---|---| |
| 533 | | `qpdf` | `qpdf` | Protect PDF (AES-256), Unlock PDF | `QPDF_PATH` | |
| 534 | | `poppler-utils` | `pdftoppm`, `pdfinfo` | PDF to image | `PDFTOPPM_PATH`, `PDFINFO_PATH` | |
| 535 | | `libheif-examples` + `libheif-plugin-libde265` | `heif-dec` (or `heif-convert`) with an HEVC decoder | HEIC (iPhone) photos on every image tool | `HEIF_DEC_PATH` | |
| 536 | | `ffmpeg` | `ffmpeg`, `ffprobe` | every audio tool | `FFMPEG_PATH` | |
| 537 | | `mtr-tiny` (0.95+) | `mtr` | Traceroute, MTR (on the gateway, which answers 502 `trace_unavailable` without it) | `NET_MTR_BIN` | |
| 538 | |
| 539 | sharp's prebuilt libvips cannot decode HEIC (HEVC) at all, and it cannot read or write BMP or read ICO: |
| 540 | BMP and ICO are handled in JavaScript (`apps/img/server/tools/codec.js`), HEIC by libheif's CLI. |
| 541 | |
| 542 | Limits (all in `/etc/openvibe/tools.env`): `PDF_MAX_PAGES` (500 pages per document, every PDF tool), |
| 543 | `PDF2IMG_MAX_PAGES` (50 pages per conversion up to 150 dpi; 40 % of that up to 300 dpi, 10 % above), |
| 544 | `YT_MAX_DURATION` (seconds, default 3 hours), `YT_MAX_FILESIZE_MB` (per downloaded part, default 2048), |
| 545 | `YT_INFO_CONCURRENCY` (yt-dlp lookups at once for `/api/info`, default 3; 20 more wait, then 503). |
| 546 | Net tools: `NET_IPINFO_TOKEN` makes IP lookups use ipinfo.io over HTTPS instead of ip-api.com's free tier |
| 547 | (HTTP only); ip-api/ipinfo, RDAP and DNS-over-HTTPS answers are cached either way. `NET_GLOBALPING_TOKEN` |
| 548 | is read but not used yet. Traceroute and MTR run `mtr` in report mode without raw sockets: its `mtr-packet` |
| 549 | sends ICMP echo through Linux's unprivileged ICMP sockets (`net.ipv4.ping_group_range`), on the address the SSRF guard |
| 550 | checked, at most two runs at once (`apps/gateway/server/net/trace.js`, `NET_MTR_BIN`). Domain Reputation is one report |
| 551 | from public signals (blacklists, SPF, DMARC, MX, HTTPS, domain age, reverse DNS), each finding with its cost. Ping and |
| 552 | latency are timed TCP connections to port 443 from this server. |
| 553 | |
| 554 | ## YouTube downloader: when YouTube refuses the server |
| 555 | |
| 556 | YouTube blocks some datacentre addresses outright ("Sign in to confirm you're not a bot"). The app |
| 557 | checks every 30 minutes (`GET /api/health` → `youtube: ok | blocked | unknown`) and shows a banner |
| 558 | while blocked. Player-client switches, IPv6, proof-of-origin tokens, the nightly yt-dlp and Cloudflare |
| 559 | WARP were all tried on 2026-09-18 and all refused; what works is a different network identity, set in |
| 560 | `/etc/openvibe/tools.env` and followed by `sudo systemctl restart openvibe-tools-yt`: |
| 561 | |
| 562 | - `YT_PROXY=socks5://user:pass@host:port` — a residential or mobile proxy (http, https, socks4/5). |
| 563 | - `YT_COOKIES_FILE=/etc/openvibe/yt-cookies.txt` — a Netscape cookies.txt exported from a throwaway |
| 564 | YouTube account, readable by the service user. Accounts used this way can be suspended. |
| 565 | |
| 566 | ## Dev quickstart |
| 567 | |
| 568 | ```bash |
| 569 | # install every app's dependencies (openvibe-shared comes from the pinned release) |
| 570 | npm run install:all |
| 571 | |
| 572 | # gateway — http://localhost:4001 |
| 573 | cd apps/gateway && cp .env.example .env # set OV_OAUTH_CLIENT_SECRET |
| 574 | npm run start:gateway |
| 575 | |
| 576 | # any satellite, e.g. maps on http://localhost:4010 |
| 577 | npm run start:maps |
| 578 | ``` |
| 579 | |
| 580 | Host-header routing without DNS: |
| 581 | |
| 582 | ```bash |
| 583 | curl -H 'Host: net.openvibe.tools' http://localhost:4001/ |
| 584 | ``` |
| 585 | |
| 586 | The full OAuth round-trip needs OpenVibe.Network running on port 4000 |
| 587 | (`BOOTSTRAP_PROFILE=local-dev` seeds a `tools` client with a |
| 588 | `http://localhost:4001/auth/callback` redirect). |
| 589 | |
| 590 | ## Auth model |
| 591 | |
| 592 | - `GET /auth/login` on the gateway → Network `/oauth/authorize` |
| 593 | (client_id `tools`, scope `profile theme`, state cookie). |
| 594 | - `GET /auth/callback` → server-side code exchange → sets `ov_token` |
| 595 | (JS-readable, `Domain=.openvibe.tools`, SameSite=Lax, Secure) and |
| 596 | `ov_refresh` (httpOnly, Path=/auth). |
| 597 | - `GET /auth/login?silent=1&next=…` adds `prompt=none`: the shared navbar |
| 598 | uses it for one silent sign-in attempt per tab when the browser carries |
| 599 | `ov_sso_hint=account`. No Network session → straight back to `next` with |
| 600 | `?sso=none`. A successful callback sets `ov_sso_hint=account` (1 year, |
| 601 | JS-readable, `Domain=.openvibe.tools`); `/auth/logout` sets it to `guest`. |
| 602 | - `next` must be a relative path, an `https://*.openvibe.tools` URL or an |
| 603 | `https://openvibe.network/...` URL (the Network's sign-in/sign-out |
| 604 | everywhere chain hops through the gateway). |
| 605 | - `GET /auth/me`, `POST /auth/refresh`, `GET /auth/logout`. |
| 606 | - Satellites never talk OAuth — they read `ov_token` and verify offline |
| 607 | via the Network JWKS (`GET /api/.well-known/jwks`). |
| 608 | - Browser pages load the shared files (theme-loader, navbar, …) from their |
| 609 | own app's pinned copy at `/shared/` (`apps/_shared/release.js`, |
| 610 | `openvibe-shared/serve`), not from openvibe.network. |
| 611 | |
| 612 | ## Security |
| 613 | |
| 614 | Reporting a vulnerability: [SECURITY.md](SECURITY.md). The rules are in [Guard](#guard-anti-abuse-apps_sharedguard) |
| 615 | and [Auth model](#auth-model); in short: every caller is identified (principal, user with audience |
| 616 | `openvibe.tools`, session or address) and held to per-tier quotas and per-actor limits; uploads are |
| 617 | sniffed against the descriptor; every tool that reaches a visitor-chosen host goes through the SSRF guard |
| 618 | (`apps/_shared/egress.js`: public addresses only, checked after DNS, redirect hops re-checked); `trust proxy` |
| 619 | is one loopback hop; sign-in `next` accepts only relative paths, `*.openvibe.tools` and `openvibe.network`. |
| 620 | Secrets (`OV_OAUTH_CLIENT_SECRET`, `TOOLS_EVENTS_SECRET`, `NET_IPINFO_TOKEN`, `RIDB_API_KEY`, `NPS_API_KEY`, |
| 621 | `OPEN_CHARGE_MAP_KEY`, `YT_COOKIES_FILE`) live in `/etc/openvibe/tools.env` |
| 622 | (0600), by name only. |
| 623 | |
| 624 | ## Deploy |
| 625 | |
| 626 | Production deploys with `sudo ovhost deploy tools` (`deploy/scripts/deploy.sh` is the wrapper; strategy |
| 627 | `multi-app`, described under [Registry, hosts and domains](#registry-hosts-and-domains)). Rollback: automatic |
| 628 | when the gateway's `/api/ready` or `/release.json` does not come up with every unit active; afterwards |
| 629 | `sudo ovhost rollback tools --to <sha>`. Nothing blocks a rollback: the schema code only adds. |
| 630 | |
| 631 | - Production path: `/opt/openvibe.tools` (apps under `apps/<name>`) |
| 632 | - Env file: `/etc/openvibe/tools.env` (0600) — shared by all units; per-app |
| 633 | `PORT` is set in each systemd unit, so tools.env must NOT define PORT. |
| 634 | - Units: `openvibe-tools.service` (gateway) and |
| 635 | `openvibe-tools-<name>.service` per satellite |
| 636 | (`apps/<name>/deploy/systemd/`). Resource bounds: `MemoryMax=2G` (img, audio, |
| 637 | docs), `1G` (yt), `768M` (gateway, text, maps, food); `TasksMax=256`; `Nice=5` |
| 638 | for img, audio, docs and yt. The gateway data directory remains writable for local runtime files. |
| 639 | - Nginx: satellites have specific `server_name` blocks; the gateway's |
| 640 | wildcard `*.openvibe.tools` block catches everything else. TLS via |
| 641 | `/etc/letsencrypt/live/openvibe.tools/`. |
| 642 | |
| 643 | ## Database and guard state |
| 644 | |
| 645 | All eight apps use one PostgreSQL database through `openvibe-sdk/db`. Production requires |
| 646 | `DATABASE_URL`; `DATABASE_DIRECT_URL` lets the owner apply the immutable SQL files in `migrations/` |
| 647 | at boot. Development and tests use embedded PGlite through the same API. Job records, abuse logs, |
| 648 | revocations, analytics, and the event outbox live in PostgreSQL. The guard's transient throttle |
| 649 | counters and salt use Valkey through `VALKEY_URL`, with process-local state when it is unset. |
| 650 | |
| 651 | <!-- versions:start --> |
| 652 | - openvibe-contracts: v0.422.0 |
| 653 | - openvibe-sdk: v0.161.0 |
| 654 | - openvibe-shared: v4.241.0 |
| 655 | <!-- versions:end --> |
| 656 |