openvibers/openvibe.tools AGENTS.md
AGENTS.md
| 1 | # AGENTS.md — OpenVibe.Tools |
| 2 | |
| 3 | ## Project overview |
| 4 | |
| 5 | OpenVibe.Tools is the network's tools suite: hundreds of single-purpose tools (images, text, audio, docs, |
| 6 | maps, video) at `openvibe.tools` and `*.openvibe.tools`, each open by API. It is a multi-app monorepo — |
| 7 | a gateway plus seven satellites — that share a registry, a run API, a job runtime and an anti-abuse |
| 8 | guard. |
| 9 | |
| 10 | ``` |
| 11 | apps/gateway openvibe.tools, net./dev./pastes. and the tool aliases :4001 |
| 12 | apps/maps maps.openvibe.tools :4010 |
| 13 | apps/food food.openvibe.tools (the page moved to OpenVibe.Food) :4011 |
| 14 | apps/img img.openvibe.tools + format aliases :4012 |
| 15 | apps/yt yt.openvibe.tools :4013 |
| 16 | apps/audio audio.openvibe.tools + effect aliases :4014 |
| 17 | apps/text text./logo.openvibe.tools + aliases :4015 |
| 18 | apps/docs docs./pdf.openvibe.tools :4016 |
| 19 | apps/_shared openvibe-tools-shared: host roles, internal auth, jobs, guard, egress |
| 20 | ``` |
| 21 | |
| 22 | - **Registry:** `apps/gateway/server/registry` is the one list of tools/families; each has a canonical |
| 23 | host, an optional short host and aliases that 301. Descriptors are `tools.tool@1`. |
| 24 | - **Run API:** `POST /api/v1/tools/:id/run` on the gateway and on img/audio/docs for their tools |
| 25 | (`tools.run-request@1`, JSON or multipart, `wait_ms`, `idempotency_key`); SDK `openvibe-sdk/tools`. |
| 26 | - **Jobs:** `apps/_shared/jobs/` plus the gateway jobs facade and each satellite's job type; job |
| 27 | records, abuse logs, revocations, analytics and the outbox are in PostgreSQL. |
| 28 | - **Guard:** `apps/_shared/guard/` (callers, quotas, sniffing, ffmpeg hardening, egress throttles, |
| 29 | abuse log) and `apps/_shared/egress.js` (SSRF-safe fetch; public addresses only, checked after DNS, |
| 30 | redirect hops re-checked). |
| 31 | - **Pages:** all server-rendered (the gateway index/family/tool pages, catalogue, sitemap/llms); |
| 32 | browser tools ship their own JS, no bundler at the service level. `openvibe-shared` is pinned per app. |
| 33 | |
| 34 | ## Owns, does not own |
| 35 | |
| 36 | Owns the tool registry (families, tools, descriptors, canonical/short/alias hosts), the catalog and |
| 37 | every tool page on `openvibe.tools` and `*.openvibe.tools`, the run API and jobs runtime (files, SSE |
| 38 | progress), the guard, the `tools.*` events, Tools' Search documents and the `tools.usage` user module. |
| 39 | |
| 40 | Does not own: identity/sign-in (Network), pastes (Community; `pastes.openvibe.tools` hands over), |
| 41 | result storage (Media keeps job results), custom-domain records (Network's domains admin — the gateway |
| 42 | merges the owner's host overrides every minute). |
| 43 | |
| 44 | ## Who calls it, what it calls |
| 45 | |
| 46 | People use the pages and the API; services and developers use `openvibe-sdk/tools` (`list`, `get`, |
| 47 | `schema`, `run`, `jobs`). The run API is the public capability (`tools.tool.run`; `tools.net.probe` for |
| 48 | network probes; `tools.tool.read` for the registry), with per-tier quotas and per-actor limits. Tools |
| 49 | calls Network (SSO, JWKS, registry/domains admin, notification/usage), Media (job results), Community |
| 50 | (pastes hand-over), Events (outbox and `host.release.published`) and the outside hosts a tool needs |
| 51 | through the SSRF guard. Billing receives Tools' job usage readings. |
| 52 | |
| 53 | ## Auth model |
| 54 | |
| 55 | The gateway is the OAuth client (`client_id: tools`): `/auth/login` → Network authorize, `/auth/callback` |
| 56 | → server-side code exchange → sets `ov_token` (JS-readable, `Domain=.openvibe.tools`, SameSite=Lax, |
| 57 | Secure) and `ov_refresh` (httpOnly, `Path=/auth`). A satellite never talks OAuth: it reads `ov_token` |
| 58 | and verifies offline with the Network JWKS. `next` accepts only a relative path, `https://*.openvibe.tools` |
| 59 | or `https://openvibe.network/...`. Every caller is identified (service principal, user, session or |
| 60 | address) and held to quotas/limits; every tool reaching a visitor-chosen host or URL goes through the |
| 61 | SSRF guard. `trust proxy` is one loopback hop. |
| 62 | |
| 63 | ## Database and migrations |
| 64 | |
| 65 | One PostgreSQL database for all eight apps through `openvibe-sdk/db`; `DATABASE_DIRECT_URL` applies |
| 66 | the immutable `migrations/` at boot, `DATABASE_URL` serves. Development/tests use PGlite through the |
| 67 | same API. Transient guard throttle counters and the salt live in Valkey (`VALKEY_URL`), with |
| 68 | process-local state when unset. Migrations only add — never edit one that has run, and nothing may |
| 69 | block a rollback. |
| 70 | |
| 71 | ## Deploy |
| 72 | |
| 73 | `sudo ovhost deploy tools` via `deploy/scripts/deploy.sh` (strategy `multi-app`; flags `--wait-idle`, |
| 74 | `--restart`, `--rollback`, `DRY_RUN=1`). The wrapper pulls as the checkout owner, installs in changed |
| 75 | apps (`apps/_*` are packages, not apps), checks every dependency resolves and the job runtime/guard |
| 76 | load, restarts every `openvibe-tools*` unit, waits for the gateway `/api/ready` and `/release.json` to |
| 77 | name the new sha with every unit active, rolls back otherwise, then announces the release. Production |
| 78 | path `/opt/openvibe.tools`; env `/etc/openvibe/tools.env` (0600, shared by all units; per-app `PORT` is |
| 79 | in each systemd unit, so tools.env must NOT define PORT). Units: `openvibe-tools.service` and |
| 80 | `openvibe-tools-<name>.service` (`apps/<name>/deploy/systemd/`); nginx satellites have specific |
| 81 | `server_name` blocks and the gateway's wildcard `*.openvibe.tools` catches the rest. New custom domain: |
| 82 | `sudo deploy/scripts/add-custom-domain.sh <domain>`. |
| 83 | |
| 84 | ## Commands and tests |
| 85 | |
| 86 | ```bash |
| 87 | npm ci && npm run install:all |
| 88 | npm run start:gateway # http://localhost:4001 |
| 89 | npm run start:img # any satellite, e.g. :4012 |
| 90 | npm test # syntax-check every server file + every apps/*/test/*.test.js |
| 91 | npm run test:pg # the suite against PostgreSQL + Valkey |
| 92 | npm run test:discovery # catalog/hosts/discovery/run-api/perf subset |
| 93 | npm run test:worker # the worker runtime tests |
| 94 | ``` |
| 95 | |
| 96 | `npm test` runs each file in its own process and covers the job runtime end to end (restart, reattach, |
| 97 | cancel, idempotency, owner scoping, SSE resume, pruning, Media results), real img/audio/docs processes |
| 98 | killed mid-job, canonical hosts, the registry-driven catalog, `registry-consistency.test.js` (every |
| 99 | catalogue tool resolves to a real operation/page or is marked unavailable), descriptors against the |
| 100 | contracts and code, every browser tool's engine against its output schema, the run API on gateway/img/ |
| 101 | docs, the jobs facade and the pinned SDK client. Audio tests skip without `ffmpeg`; docs/img check the |
| 102 | 503 path and run real qpdf/poppler/libheif work when installed. `sdk-client.test.js` needs an |
| 103 | OpenVibe.SDK checkout (`OV_SDK_DIR`) and says it skipped without one. |
| 104 | |
| 105 | ## Conventions and pitfalls |
| 106 | |
| 107 | - **The registry is the source of truth:** adding a tool means a descriptor, a registry entry, a route/ |
| 108 | operation and a page that resolve to each other — `registry-consistency.test.js` fails otherwise. |
| 109 | Never hand-write a catalog page that the registry does not know. |
| 110 | - **Descriptor first:** `execution` (`client`/`sync`/`job`), `api`, `run`, input schema, files, output, |
| 111 | limits, auth, `quotaClass`, `cost`, `egress`, hosts and docs are one contract; the code must match it. |
| 112 | - **Every outbound fetch of a visitor-chosen URL uses `apps/_shared/egress.js`.** Uploads are sniffed |
| 113 | against the descriptor before a job sees them; ffmpeg runs hardened. A new tool must not call `fetch` |
| 114 | directly. |
| 115 | - **Idempotency:** a run with the same key replays; a different request under the key is a conflict. |
| 116 | Jobs are owner-scoped and survive reattach after a restart. |
| 117 | - **One shared env file:** a `PORT` in `tools.env` will fight the per-unit ports. Secrets |
| 118 | (`OV_OAUTH_CLIENT_SECRET`, `TOOLS_EVENTS_SECRET`, NET_IPINFO_TOKEN, RIDB/NPS keys) live only there. |
| 119 | - **YouTube refusal is infrastructure, not code:** `/api/health` reports `youtube: ok|blocked|unknown` |
| 120 | and the page shows a banner; a proxy is set in `tools.env` and applied by restarting |
| 121 | `openvibe-tools-yt`. Do not add a bypass that ignores the block. |
| 122 | - **`apps/_shared` is a versioned package:** every app declares the range it needs and refuses to boot |
| 123 | when the checkout does not satisfy it. |
| 124 |