openvibers/openvibe.tools Commits
Docs: jobs API, durability, owner scoping, Media results and their grants, canonical hosts, registry catalog, tests; tools.job.* capability and service manifest proposals
Changes
@@@ -22,6 +22,7 @@ apps/├── audio # Audio.OpenVibe — audio converter & effects├── text # Text.OpenVibe + Logo.OpenVibe — text generators & logo makers└── docs # Docs.OpenVibe — PDF & document tools++apps/_shared # code the apps require by relative path: host roles, internal auth, the job runtime (jobs/)(openvibe-shared is a pinned OpenVibe.Shared release in each app's package.json; no vendor/ copy)```@@@ -57,6 +58,75 @@ Public, cacheable outputs on the apex: `/` (every tool by family, planned toolsDeploy with `deploy/scripts/deploy.sh` (it refreshes the copied shared package inside each app).++## Jobs (Img, Audio, Docs)++++Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same++routes answer on every host of the img, audio and docs satellites:++++| Route | What it does |++|---|---|++| `POST /api/v1/jobs` | Submit: multipart `type`, `input` (JSON text), `file` / `files`, or JSON `{ type, input }`. `Idempotency-Key` header (8–200 chars). → `202` + `Location`; a repeat with the same key and request → `200` + `Idempotent-Replayed: true` (same job); same key, different request → `409 tools.job.idempotency_conflict`. |++| `GET /api/v1/jobs/:id` | The job: `state` (`queued`, `running`, `succeeded`, `failed`, `cancelled`), `progress {percent, message}`, `attempts`, `result {files, data}`, `error` (problem+json), `retryable`, `expires_at`. |++| `DELETE /api/v1/jobs/:id` | Cancel: queued → `200` cancelled at once; running → `202` (signalled; ffmpeg is killed, sharp/pdf-lib work finishes and is discarded) and ends `cancelled`; finished → `409 tools.job.already_finished`. |++| `GET /api/v1/jobs/:id/events` | SSE: `job.queued`, `job.running`, `job.progress`, `job.cancel_requested`, `job.succeeded`, `job.failed`, `job.cancelled`, each with an `id`. Reconnecting with `Last-Event-ID` (or `?last_event_id=`) replays only later events; a finished job with nothing newer answers `204`. |++| `GET /api/v1/jobs/:id/files/:n` | A result file (attachment); `?inline=1` for previews. |++++Job types: `img.process` (input `{ tool: convert|compress|resize|crop, format, quality, width, … }`, one image),++`audio.process` (`{ tool, …options }` as `/api/process` takes them, one audio/video file; progress from ffmpeg),++`docs.process` (`{ tool, …options }`, one PDF, or several files for `merge` / `img2pdf`). A format host fills++in its format (`webp.openvibe.tools` converts to WebP). The synchronous `/api/process` endpoints still work and++run the same code; the pages use jobs and keep the job id in the address (`?job=`) and in sessionStorage, so a++reload reattaches.++++- **Durable.** A job and its uploaded input are in `data/jobs.db` and `data/jobs/<id>/` before the `202` is sent.++ After a restart, queued jobs run; jobs that were running are re-queued (all three types, up to their attempt++ limit) — a type can instead declare `onRestart: 'fail'`, which fails them with `retryable: true`.++- **Owner-scoped.** A job is visible only to whoever created it: a signed-in person (`user:usr_…` from the++ `ov_token` subject), an app/service principal (Network client-credentials token, audience `openvibe.tools`,++ capabilities `tools.job.create|read|cancel`), or else this browser's `ov_tools_jobs` cookie (only its hash is++ stored). Anyone else gets `404 tools.job.not_found`.++- **Bounded.** `TOOLS_JOBS_CONCURRENCY` jobs run at once per satellite (default 2; `TOOLS_JOBS_CONCURRENCY_<APP>`++ overrides), `TOOLS_JOBS_MAX_ACTIVE` unfinished jobs per owner (default 10, then `429`), plus the satellites'++ existing burst and processing rate limits on submit.++- **Retention.** Finished jobs expire after 1 hour (browser sessions) or 24 hours (signed-in people, principals);++ the pruner deletes the row, its events, its files and its Media objects.++- **Results in OpenVibe.Media** when `TOOLS_JOB_RESULTS=media`: each result file becomes a private Media object++ (v2 object API, namespace `tools`, owner `X-OV-Subject` for signed-in people) and the job's result carries its++ `media.media_id`; previews redirect to a short-lived signed Media URL, downloads stream through the satellite.++ This needs `OV_OAUTH_CLIENT_SECRET` and Network grants for the `tools` client:++ `media.object.upload` and `media.object.read` for audience `openvibe.media`, namespace `tools`, and a `tools`++ tenant in Media. Without them (or if an upload fails) results stay on local disk and the file says++ `storage: "local"`. Default: `local`.++++Environment (all in `/etc/openvibe/tools.env`): `TOOLS_JOBS_CONCURRENCY`, `TOOLS_JOBS_CONCURRENCY_<APP>`,++`TOOLS_JOBS_MAX_ACTIVE`, `TOOLS_JOB_RESULTS`, `TOOLS_MEDIA_NAMESPACE`, `OV_MEDIA_INTERNAL_URL`, `OV_MEDIA_URL`,++`OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID`, `OV_OAUTH_CLIENT_SECRET`.++++Not done yet: job lifecycle events are not published to OpenVibe.Events; there is no retry endpoint (a failed++retryable job is submitted again); quotas for external developer apps are the per-owner limits above, not a++Codes-issued quota.++++## Canonical hosts and the service registry++++Every satellite honours the gateway's `X-OV-Tool` / `X-OV-Host-Role` / `X-OV-Canonical-Host` / `X-OV-Short-Host`++headers (`apps/_shared/host-role.js`): canonical links, `og:url` and JSON-LD use the canonical host (a custom++domain included), a custom domain gets its tool's page through `X-OV-Tool`, aliases the gateway missed are++redirected, and a host a satellite does not serve goes to the tools index. Reached directly, each host is its++own canonical as before.++++The gateway resolves the other services' origins (Community for pastes, the "elsewhere on OpenVibe" links)++through Network's registry, `GET /api/v1/registry/services` (fetched on boot and every 10 minutes; the last good++answer is kept). Until it answers, a local list is used and `/api/catalog.json` says `services.source: "fallback"`.++Services the registry marks `placeholder` or `retired` are not linked. `OV_REGISTRY_URL` overrides the URL.++++## Tests++++`npm test` (Node 22) syntax-checks every server file and runs every `apps/*/test/*.test.js`, each in its own++process: the job runtime end to end (restart, reattach, cancel, idempotency, owner scoping, SSE resume, pruning,++Media results with a stand-in Media), img/audio/docs as real processes killed with SIGKILL mid-job, canonical++hosts on every satellite, and the registry-driven catalog. The audio test needs `ffmpeg` and skips without it.++Install first with `npm run install:all`.++## YouTube downloader: when YouTube refuses the serverYouTube blocks some datacentre addresses outright ("Sign in to confirm you're not a bot"). The app@@@ -29,3 +29,17 @@ MEDIA_PUBLIC_URL=https://openvibe.media# Internal API key for Network /internal/* calls (if ever needed)INTERNAL_API_KEY=change-me-to-a-random-64-char-string++++# Service registry (the catalog's service origins); defaults to OV_NETWORK_INTERNAL_URL + /api/v1/registry/services++#OV_REGISTRY_URL=http://127.0.0.1:4000/api/v1/registry/services++++# ── Satellites: jobs (img, audio, docs read the same tools.env) ──++#TOOLS_JOBS_CONCURRENCY=2++#TOOLS_JOBS_CONCURRENCY_AUDIO=1++#TOOLS_JOBS_MAX_ACTIVE=10++# local (default) | media — media needs OV_OAUTH_CLIENT_SECRET and the Network grants++# [tools, media.object.upload, openvibe.media] + [tools, media.object.read, openvibe.media] (namespace tools)++#TOOLS_JOB_RESULTS=local++#TOOLS_MEDIA_NAMESPACE=tools++#OV_MEDIA_INTERNAL_URL=http://127.0.0.1:4100++#OV_MEDIA_URL=https://openvibe.mediannew file mode 100644@@@ -0,0 +1,19 @@++{++ "id": "tools.job.cancel",++ "version": "1.0",++ "owner": "tools",++ "status": "active",++ "visibility": "public",++ "description": "Cancel a queued or running job the calling principal created.",++ "permissions": [++ "tools:jobs:write"++ ],++ "resourceConstraints": [++ "owner"++ ],++ "quotaClass": "tools-job",++ "events": [],++ "implementedBy": [++ "DELETE /api/v1/jobs/:id"++ ]++}nnew file mode 100644@@@ -0,0 +1,19 @@++{++ "id": "tools.job.create",++ "version": "1.0",++ "owner": "tools",++ "status": "active",++ "visibility": "public",++ "description": "Submit an asynchronous tool job (img.process, audio.process, docs.process) with its input files; the job is owned by the calling principal. Idempotency-Key supported.",++ "permissions": [++ "tools:jobs:write"++ ],++ "resourceConstraints": [++ "owner"++ ],++ "quotaClass": "tools-job",++ "events": [],++ "implementedBy": [++ "POST /api/v1/jobs (Tools satellites: img, audio, docs)"++ ]++}nnew file mode 100644@@@ -0,0 +1,21 @@++{++ "id": "tools.job.read",++ "version": "1.0",++ "owner": "tools",++ "status": "active",++ "visibility": "public",++ "description": "Read a job the calling principal created: state, progress, result (Media object references or local files), SSE progress with Last-Event-ID resume, result files.",++ "permissions": [++ "tools:jobs:read"++ ],++ "resourceConstraints": [++ "owner"++ ],++ "quotaClass": "tools-read",++ "events": [],++ "implementedBy": [++ "GET /api/v1/jobs/:id",++ "GET /api/v1/jobs/:id/events",++ "GET /api/v1/jobs/:id/files/:n"++ ]++}nnew file mode 100644@@@ -0,0 +1,26 @@++{++ "version": "0.2.0",++ "contractRanges": {++ "openvibe-contracts": ">=0.7.0 <1.0.0"++ },++ "id": "tools",++ "name": "OpenVibe.Tools",++ "status": "stable",++ "repository": "OpenVibers/OpenVibe.Tools",++ "domains": [++ "openvibe.tools"++ ],++ "publicOrigin": "https://openvibe.tools",++ "health": "/api/health",++ "capabilities": [++ "tools.job.create",++ "tools.job.read",++ "tools.job.cancel"++ ],++ "eventsProduced": [],++ "eventsConsumed": [],++ "namespacesOwned": [++ "tools.*"++ ],++ "notes": "Wave 11: asynchronous jobs on the img, audio and docs satellites (POST/GET/DELETE /api/v1/jobs, SSE events). Job lifecycle is not yet published to OpenVibe.Events, so eventsProduced stays empty. Tools consumes media.object.upload and media.object.read (namespace tools) when TOOLS_JOB_RESULTS=media."++}
@@ -22,6 +22,7 @@ apps/
@@ -22,6 +22,7 @@ apps/
├── audio # Audio.OpenVibe — audio converter & effects
├── audio # Audio.OpenVibe — audio converter & effects
├── text # Text.OpenVibe + Logo.OpenVibe — text generators & logo makers
├── text # Text.OpenVibe + Logo.OpenVibe — text generators & logo makers
└── docs # Docs.OpenVibe — PDF & document tools
└── docs # Docs.OpenVibe — PDF & document tools
+apps/_shared # code the apps require by relative path: host roles, internal auth, the job runtime (jobs/)
(openvibe-shared is a pinned OpenVibe.Shared release in each app's package.json; no vendor/ copy)
(openvibe-shared is a pinned OpenVibe.Shared release in each app's package.json; no vendor/ copy)
```
```
@@ -57,6 +58,75 @@ Public, cacheable outputs on the apex: `/` (every tool by family, planned tools
@@ -57,6 +58,75 @@ Public, cacheable outputs on the apex: `/` (every tool by family, planned tools
Deploy with `deploy/scripts/deploy.sh` (it refreshes the copied shared package inside each app).
Deploy with `deploy/scripts/deploy.sh` (it refreshes the copied shared package inside each app).
+## Jobs (Img, Audio, Docs)
+
+Heavy operations run as durable asynchronous jobs (`apps/_shared/jobs`, roadmap Wave 11). The same
+routes answer on every host of the img, audio and docs satellites:
+
+| Route | What it does |
+|---|---|
+| `POST /api/v1/jobs` | Submit: multipart `type`, `input` (JSON text), `file` / `files`, or JSON `{ type, input }`. `Idempotency-Key` header (8–200 chars). → `202` + `Location`; a repeat with the same key and request → `200` + `Idempotent-Replayed: true` (same job); same key, different request → `409 tools.job.idempotency_conflict`. |
+| `GET /api/v1/jobs/:id` | The job: `state` (`queued`, `running`, `succeeded`, `failed`, `cancelled`), `progress {percent, message}`, `attempts`, `result {files, data}`, `error` (problem+json), `retryable`, `expires_at`. |
+| `DELETE /api/v1/jobs/:id` | Cancel: queued → `200` cancelled at once; running → `202` (signalled; ffmpeg is killed, sharp/pdf-lib work finishes and is discarded) and ends `cancelled`; finished → `409 tools.job.already_finished`. |
+| `GET /api/v1/jobs/:id/events` | SSE: `job.queued`, `job.running`, `job.progress`, `job.cancel_requested`, `job.succeeded`, `job.failed`, `job.cancelled`, each with an `id`. Reconnecting with `Last-Event-ID` (or `?last_event_id=`) replays only later events; a finished job with nothing newer answers `204`. |
+| `GET /api/v1/jobs/:id/files/:n` | A result file (attachment); `?inline=1` for previews. |
+
+Job types: `img.process` (input `{ tool: convert|compress|resize|crop, format, quality, width, … }`, one image),
+`audio.process` (`{ tool, …options }` as `/api/process` takes them, one audio/video file; progress from ffmpeg),
+`docs.process` (`{ tool, …options }`, one PDF, or several files for `merge` / `img2pdf`). A format host fills
+in its format (`webp.openvibe.tools` converts to WebP). The synchronous `/api/process` endpoints still work and
+run the same code; the pages use jobs and keep the job id in the address (`?job=`) and in sessionStorage, so a
+reload reattaches.
+
+- **Durable.** A job and its uploaded input are in `data/jobs.db` and `data/jobs/<id>/` before the `202` is sent.
+ After a restart, queued jobs run; jobs that were running are re-queued (all three types, up to their attempt
+ limit) — a type can instead declare `onRestart: 'fail'`, which fails them with `retryable: true`.
+- **Owner-scoped.** A job is visible only to whoever created it: a signed-in person (`user:usr_…` from the
+ `ov_token` subject), an app/service principal (Network client-credentials token, audience `openvibe.tools`,
+ capabilities `tools.job.create|read|cancel`), or else this browser's `ov_tools_jobs` cookie (only its hash is
+ stored). Anyone else gets `404 tools.job.not_found`.
+- **Bounded.** `TOOLS_JOBS_CONCURRENCY` jobs run at once per satellite (default 2; `TOOLS_JOBS_CONCURRENCY_<APP>`
+ overrides), `TOOLS_JOBS_MAX_ACTIVE` unfinished jobs per owner (default 10, then `429`), plus the satellites'
+ existing burst and processing rate limits on submit.
+- **Retention.** Finished jobs expire after 1 hour (browser sessions) or 24 hours (signed-in people, principals);
+ the pruner deletes the row, its events, its files and its Media objects.
+- **Results in OpenVibe.Media** when `TOOLS_JOB_RESULTS=media`: each result file becomes a private Media object
+ (v2 object API, namespace `tools`, owner `X-OV-Subject` for signed-in people) and the job's result carries its
+ `media.media_id`; previews redirect to a short-lived signed Media URL, downloads stream through the satellite.
+ This needs `OV_OAUTH_CLIENT_SECRET` and Network grants for the `tools` client:
+ `media.object.upload` and `media.object.read` for audience `openvibe.media`, namespace `tools`, and a `tools`
+ tenant in Media. Without them (or if an upload fails) results stay on local disk and the file says
+ `storage: "local"`. Default: `local`.
+
+Environment (all in `/etc/openvibe/tools.env`): `TOOLS_JOBS_CONCURRENCY`, `TOOLS_JOBS_CONCURRENCY_<APP>`,
+`TOOLS_JOBS_MAX_ACTIVE`, `TOOLS_JOB_RESULTS`, `TOOLS_MEDIA_NAMESPACE`, `OV_MEDIA_INTERNAL_URL`, `OV_MEDIA_URL`,
+`OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID`, `OV_OAUTH_CLIENT_SECRET`.
+
+Not done yet: job lifecycle events are not published to OpenVibe.Events; there is no retry endpoint (a failed
+retryable job is submitted again); quotas for external developer apps are the per-owner limits above, not a
+Codes-issued quota.
+
+## Canonical hosts and the service registry
+
+Every satellite honours the gateway's `X-OV-Tool` / `X-OV-Host-Role` / `X-OV-Canonical-Host` / `X-OV-Short-Host`
+headers (`apps/_shared/host-role.js`): canonical links, `og:url` and JSON-LD use the canonical host (a custom
+domain included), a custom domain gets its tool's page through `X-OV-Tool`, aliases the gateway missed are
+redirected, and a host a satellite does not serve goes to the tools index. Reached directly, each host is its
+own canonical as before.
+
+The gateway resolves the other services' origins (Community for pastes, the "elsewhere on OpenVibe" links)
+through Network's registry, `GET /api/v1/registry/services` (fetched on boot and every 10 minutes; the last good
+answer is kept). Until it answers, a local list is used and `/api/catalog.json` says `services.source: "fallback"`.
+Services the registry marks `placeholder` or `retired` are not linked. `OV_REGISTRY_URL` overrides the URL.
+
+## Tests
+
+`npm test` (Node 22) syntax-checks every server file and runs every `apps/*/test/*.test.js`, each in its own
+process: the job runtime end to end (restart, reattach, cancel, idempotency, owner scoping, SSE resume, pruning,
+Media results with a stand-in Media), img/audio/docs as real processes killed with SIGKILL mid-job, canonical
+hosts on every satellite, and the registry-driven catalog. The audio test needs `ffmpeg` and skips without it.
+Install first with `npm run install:all`.
+
## YouTube downloader: when YouTube refuses the server
## YouTube downloader: when YouTube refuses the server
YouTube blocks some datacentre addresses outright ("Sign in to confirm you're not a bot"). The app
YouTube blocks some datacentre addresses outright ("Sign in to confirm you're not a bot"). The app
@@ -29,3 +29,17 @@ MEDIA_PUBLIC_URL=https://openvibe.media
@@ -29,3 +29,17 @@ MEDIA_PUBLIC_URL=https://openvibe.media
# Internal API key for Network /internal/* calls (if ever needed)
# Internal API key for Network /internal/* calls (if ever needed)
INTERNAL_API_KEY=change-me-to-a-random-64-char-string
INTERNAL_API_KEY=change-me-to-a-random-64-char-string
+
+# Service registry (the catalog's service origins); defaults to OV_NETWORK_INTERNAL_URL + /api/v1/registry/services
+#OV_REGISTRY_URL=http://127.0.0.1:4000/api/v1/registry/services
+
+# ── Satellites: jobs (img, audio, docs read the same tools.env) ──
+#TOOLS_JOBS_CONCURRENCY=2
+#TOOLS_JOBS_CONCURRENCY_AUDIO=1
+#TOOLS_JOBS_MAX_ACTIVE=10
+# local (default) | media — media needs OV_OAUTH_CLIENT_SECRET and the Network grants
+# [tools, media.object.upload, openvibe.media] + [tools, media.object.read, openvibe.media] (namespace tools)
+#TOOLS_JOB_RESULTS=local
+#TOOLS_MEDIA_NAMESPACE=tools
+#OV_MEDIA_INTERNAL_URL=http://127.0.0.1:4100
+#OV_MEDIA_URL=https://openvibe.media
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+{
+ "id": "tools.job.cancel",
+ "version": "1.0",
+ "owner": "tools",
+ "status": "active",
+ "visibility": "public",
+ "description": "Cancel a queued or running job the calling principal created.",
+ "permissions": [
+ "tools:jobs:write"
+ ],
+ "resourceConstraints": [
+ "owner"
+ ],
+ "quotaClass": "tools-job",
+ "events": [],
+ "implementedBy": [
+ "DELETE /api/v1/jobs/:id"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+{
+ "id": "tools.job.create",
+ "version": "1.0",
+ "owner": "tools",
+ "status": "active",
+ "visibility": "public",
+ "description": "Submit an asynchronous tool job (img.process, audio.process, docs.process) with its input files; the job is owned by the calling principal. Idempotency-Key supported.",
+ "permissions": [
+ "tools:jobs:write"
+ ],
+ "resourceConstraints": [
+ "owner"
+ ],
+ "quotaClass": "tools-job",
+ "events": [],
+ "implementedBy": [
+ "POST /api/v1/jobs (Tools satellites: img, audio, docs)"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+{
+ "id": "tools.job.read",
+ "version": "1.0",
+ "owner": "tools",
+ "status": "active",
+ "visibility": "public",
+ "description": "Read a job the calling principal created: state, progress, result (Media object references or local files), SSE progress with Last-Event-ID resume, result files.",
+ "permissions": [
+ "tools:jobs:read"
+ ],
+ "resourceConstraints": [
+ "owner"
+ ],
+ "quotaClass": "tools-read",
+ "events": [],
+ "implementedBy": [
+ "GET /api/v1/jobs/:id",
+ "GET /api/v1/jobs/:id/events",
+ "GET /api/v1/jobs/:id/files/:n"
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+{
+ "version": "0.2.0",
+ "contractRanges": {
+ "openvibe-contracts": ">=0.7.0 <1.0.0"
+ },
+ "id": "tools",
+ "name": "OpenVibe.Tools",
+ "status": "stable",
+ "repository": "OpenVibers/OpenVibe.Tools",
+ "domains": [
+ "openvibe.tools"
+ ],
+ "publicOrigin": "https://openvibe.tools",
+ "health": "/api/health",
+ "capabilities": [
+ "tools.job.create",
+ "tools.job.read",
+ "tools.job.cancel"
+ ],
+ "eventsProduced": [],
+ "eventsConsumed": [],
+ "namespacesOwned": [
+ "tools.*"
+ ],
+ "notes": "Wave 11: asynchronous jobs on the img, audio and docs satellites (POST/GET/DELETE /api/v1/jobs, SSE events). Job lifecycle is not yet published to OpenVibe.Events, so eventsProduced stays empty. Tools consumes media.object.upload and media.object.read (namespace tools) when TOOLS_JOB_RESULTS=media."
+}