openvibers/openvibe.host AGENTS.md
AGENTS.md
| 1 | # AGENTS.md — OpenVibe.Host |
| 2 | |
| 3 | ## Project Overview |
| 4 | |
| 5 | OpenVibe.Host is two things in one repository: |
| 6 | |
| 7 | 1. **The network's operator plane, `ovhost`** — a root-run CLI/library that deploys, rolls back, validates, probes, |
| 8 | backs up and restore-drills every service on the host. It is installed at `/usr/local/lib/openvibe-host` |
| 9 | (linked `/usr/local/bin/ovhost`) with the inventory `/etc/openvibe/host.json`; it is updated with |
| 10 | `sudo ovhost self-update`, not by the catalog deploy. |
| 11 | 2. **The `openvibe.host` catalog service** — hosting compared honestly (cloud/VPS/dedicated plans, offers, domain |
| 12 | prices, benchmarks, price history, staff entries) and the console where a person connects their own hosting |
| 13 | accounts. It runs at port 4910. |
| 14 | |
| 15 | - **Domain:** `openvibe.host` (catalog) · **Port:** 4910 · **Service id:** `host` · **Env prefix:** `HOST`/`OV_` |
| 16 | - **Runtime:** Node 22, Express 4, PostgreSQL 18 through `openvibe-sdk/db` (PGlite in development/tests), Valkey 9; |
| 17 | plain systemd/nginx/git/ffmpeg-free host tools for the CLI. |
| 18 | - **Read first:** [README.md](README.md) (very long and authoritative), [docs/deploy-strategies.md](docs/deploy-strategies.md), |
| 19 | [docs/backups.md](docs/backups.md), [docs/restore-drills.md](docs/restore-drills.md), |
| 20 | [docs/data-provisioning.md](docs/data-provisioning.md), [host.example.json](host.example.json). |
| 21 | |
| 22 | **Network context:** the catalog uses Network SSO/staff (`host.catalog.manage`), the JWKS, client-credentials for the |
| 23 | outbox relay and `ovhost alerts relay`, and `network.status.incident` for incidents/maintenance. The catalog's pages |
| 24 | are OpenVibe.Search documents; discussion is OpenVibe.Community (thread refs only); prices come from OpenVibe.Sources |
| 25 | (optional ingest) and keyed provider APIs. `ovhost` calls only the host's own services, Network, Events, Prometheus and |
| 26 | the backup bucket. |
| 27 | |
| 28 | ## Commands |
| 29 | |
| 30 | ```bash |
| 31 | npm ci # install (contracts/sdk/shared/publishing pinned by tag) |
| 32 | npm run dev # http://localhost:4910, PGlite under data/ |
| 33 | npm start # node server/index.js |
| 34 | npm test # every test/*.test.js; system calls go to test/fake-host.js |
| 35 | npm run test:pg # the same through PostgreSQL + PgBouncer |
| 36 | npm test -- catalog # only test files whose name contains the word |
| 37 | node --check <file.js> # syntax check (no linter configured) |
| 38 | sudo ovhost --version # the installed CLI (not this checkout, unless run as ./bin/ovhost) |
| 39 | sudo ovhost status [<service>...] # units, sha, readiness, protected sessions |
| 40 | sudo ovhost validate <service> [--manifest <file>] # env NAMES, units, port, vhost+nginx -t, deps, lifecycle |
| 41 | sudo ovhost plan <service> [--to <sha>] [--no-fetch] [--restart] |
| 42 | sudo ovhost deploy <service> [--wait-idle] [--force] [--restart] [--to <sha>] [--install-units] |
| 43 | sudo ovhost rollback <service> [--to <sha|release>] [--wait-idle] [--force] |
| 44 | sudo ovhost backup --all [--offsite] [--logical] | sudo ovhost drill <service> [--keep] |
| 45 | sudo ovhost incident|maintenance|freeze ... | sudo ovhost alerts relay | sudo ovhost self-update |
| 46 | ``` |
| 47 | |
| 48 | Every command takes `--json` and `--inventory <file>`. Exit codes: `0` ok · `1` usage/precondition (held lock) · `2` |
| 49 | validation failed, nothing restarted · `3` not ready, rolled back and serving · `4` rollback failed, **manual |
| 50 | intervention** · `5` protected sessions active (or `--wait-idle` gave up) · `6` frozen (`--force` goes through; |
| 51 | rollbacks are never frozen). |
| 52 | |
| 53 | ## Architecture at a Glance |
| 54 | |
| 55 | - **CLI entry:** [bin/ovhost](bin/ovhost) → [lib/cli.js](lib/cli.js), commands under [lib/commands/](lib/commands/) |
| 56 | (`backup.js`, `data.js`, `drill.js`, `self-update.js`, `service-add.js`, `snapshot.js`, `status.js`, `validate.js`). |
| 57 | - **Operator core:** [lib/inventory.js](lib/inventory.js) (the inventory model and its refusals), |
| 58 | [lib/executor.js](lib/executor.js) (deploy/rollback orchestration), [lib/systemd.js](lib/systemd.js) (the socket |
| 59 | guard and `setpriv` drops), [lib/readiness.js](lib/readiness.js), [lib/probes.js](lib/probes.js) (protected |
| 60 | sessions, `http-json-count`/`postgresql-count`/`sum` and `addFields` semantics), [lib/nginx.js](lib/nginx.js) |
| 61 | (render + transactional install), [lib/releases.js](lib/releases.js), [lib/release-layout.js](lib/release-layout.js), |
| 62 | [lib/deps.js](lib/deps.js), [lib/deploy-guard.js](lib/deploy-guard.js), [lib/lock.js](lib/lock.js), |
| 63 | [lib/envfile.js](lib/envfile.js) (names/emptiness only), [lib/lifecycle.js](lib/lifecycle.js), |
| 64 | [lib/dbengine.js](lib/dbengine.js), [lib/backup-crypto.js](lib/backup-crypto.js), [lib/offsite.js](lib/offsite.js), |
| 65 | [lib/retention.js](lib/retention.js), [lib/alerts.js](lib/alerts.js), [lib/announce.js](lib/announce.js), |
| 66 | [lib/incidents.js](lib/incidents.js), [lib/nodes.js](lib/nodes.js), [lib/media.js](lib/media.js), |
| 67 | [lib/browser-watch.js](lib/browser-watch.js), [lib/dns.js](lib/dns.js), [lib/certs.js](lib/certs.js), |
| 68 | [lib/commands/snapshot.js](lib/commands/snapshot.js), [lib/tar.js](lib/tar.js), [lib/archive.js](lib/archive.js). |
| 69 | - **Inventory:** [host.example.json](host.example.json) is the model (checkout/owner/units/socket/unitSources, env |
| 70 | required names, port, loopback ready URL/headers, packages/install/build hooks, `noRestartPaths`, `protected` |
| 71 | probes, drain, databases, backupOnChange, nginx, layout, workerUnits, auxiliaryUnits, objects, strategy). The real |
| 72 | `/etc/openvibe/host.json` is **never committed**. |
| 73 | - **Catalog service:** [server/app.js](server/app.js), [server/index.js](server/index.js), [server/config.js](server/config.js), |
| 74 | [server/db.js](server/db.js), [server/catalog/](server/catalog/) (`mapping.js`, `ingest.js`, `keyed-feeds.js`, |
| 75 | `offers.js`, `domains.js`, `domain-check.js`, `rdap.js`, `benchmarks.js`, `community.js`, `pricing.js`, `trend.js`, |
| 76 | `search-index.js`), [server/console/console.js](server/console/console.js), [server/http/](server/http/), |
| 77 | [server/auth/](server/auth/) (capabilities, forms, SSO, viewer), [server/render/](server/render/), |
| 78 | [server/data-sources-config.js](server/data-sources-config.js), [server/account-data.js](server/account-data.js), |
| 79 | [server/worker.js](server/worker.js). |
| 80 | - **Database:** [server/db.js](server/db.js) and [migrations/](migrations/): catalog providers/plans/promos/ |
| 81 | price changes/offers/domain prices/benchmarks, `host_console_accounts`/`host_console_actions`, `config_keys`/ |
| 82 | `config_snapshots`, `event_outbox`, `account_data_events`, `telemetry_samples`. Migration 0008 dropped the old |
| 83 | static-hosting engine tables on purpose. |
| 84 | - **Deploy:** [deploy/systemd/openvibe-host.service](deploy/systemd/openvibe-host.service) (127.0.0.1:4910, |
| 85 | `/etc/openvibe/host.env`, checkout `/opt/openvibe.host`, state `/var/lib/openvibe-host-api`) and the operator |
| 86 | timers in [deploy/systemd/](deploy/systemd/) (`openvibe-backup`, `-alerts`, `-browsercheck`, `-authhealth`, |
| 87 | `-readyhealth`, `-devpath`, `-toolsjob`); nginx files in [deploy/nginx/](deploy/nginx/). `sudo ovhost deploy host` |
| 88 | (strategy `git-checkout`); `sudo ovhost rollback host --to <sha>`. |
| 89 | |
| 90 | ## Ownership and Boundaries |
| 91 | |
| 92 | Host owns (a) the operator plane: inventory, env-name validation, deploy strategies, readiness/drain, release log, |
| 93 | certificate inventory, vhost rendering/install, config snapshots, backups/offsite/retention, restore drills, incidents, |
| 94 | alerts relay and release announcements; (b) the catalog: providers/plans/promotions/price history, offers, registrar |
| 95 | prices and domain checks, benchmarks and their one-time links, the console's sealed provider tokens and action log, the |
| 96 | catalog Search documents; (c) the data role and its `add-service.sh`. It does **not** own: product business logic |
| 97 | (calls hooks and `/api/ready`, `/api/streams`), secrets (reads env **names** and emptiness only; the sole exception is |
| 98 | its own root-only `/etc/openvibe/backup.env`), certificate issuance (certbot; inventory only), platform-wide credentials |
| 99 | for hosted code, or static hosting (moved to OpenVibe.Space on 2026-10-12; `/hosting`, `/projects*`, `/sites*`, |
| 100 | `/deploys*`, `/domains*`, `/preview*`, `*.openvibe.host` and the old APIs answer **410** with a link to Space). |
| 101 | |
| 102 | ## Catalog and Console |
| 103 | |
| 104 | - **Catalog data comes only from named sources**: Sources items (`HOST_CATALOG_INGEST`, `HOST_CATALOG_SOURCES_URL`), |
| 105 | keyed provider feeds (DigitalOcean/Hetzner read every six hours with a sealed read-only token from |
| 106 | `host.data_sources`; env fallback `HOST_DIGITALOCEAN_TOKEN`/`HOST_HETZNER_TOKEN`), and staff entries with a source |
| 107 | URL. **Nothing is estimated**: a value the source does not state is "Unknown". LightSail monthly = hourly × 744, |
| 108 | IPv6-only bundles show 0 IPv4, OVH addons are joined to their product, Vultr is disabled (its API host forbids |
| 109 | robots). `POST /internal/config/host.data_sources/test/:key` makes one read-only call and never returns the token. |
| 110 | - **Offers** from LowEndSpirit's public feed; LowEndTalk/LowEndBox feeds are off (403 to the production host). Keep |
| 111 | headline/poster/date/link and a stated monthly or yearly price with its monthly equivalent; editorials/giveaways are |
| 112 | not offers; a removed post is hidden; >120 days drops off; `sort=price` needs a currency and converts nothing. |
| 113 | - **Domains** from registrars' public price lists (Porkbun keyless; Handshake names out) with append-only change |
| 114 | history; a name check asks the ending's RDAP (IANA bootstrap, https only, no redirects, 6 s, 10 min cache, 30/min |
| 115 | per address) and prices the name at popular endings; each registrar's search is a plain deep link. |
| 116 | - **Benchmarks**: a signed-in person gets a one-time YABS link (24 h, 10/day); Host parses the JSON field by field |
| 117 | (never the server's address), stores it, and never shows the submitter. They or staff may remove a result; export |
| 118 | carries results and links, erasure deletes the links and leaves results anonymous. |
| 119 | - **Order** comes only from the visitor's sort; affiliate links are `rel="sponsored nofollow noopener"` with the |
| 120 | provider's disclosure beside them and never change the order. |
| 121 | - **Console**: each provider token is checked when added, sealed AES-256-GCM under `HOST_CREDENTIALS_KEY` (32 bytes, |
| 122 | hex/base64), never shown again (only the last four clear), opened only to call that provider, deleted on disconnect. |
| 123 | Power on/off/reboot reach the provider and are logged. With no key the console connects nothing and says so. |
| 124 | Account export/deletion (ADR-033) carries accounts (never tokens) + action log and erases them. |
| 125 | - **Discussion** uses Community thread refs `{service:'host', type:'provider'|'plan', id}` only; hidden providers and |
| 126 | hidden/gone plans have no page, and status changes hide/reopen threads after commit. With no |
| 127 | `HOST_COMMUNITY_URL`, no Discussion block appears. |
| 128 | |
| 129 | ## Operator Rules (the ones that page you) |
| 130 | |
| 131 | - **Socket units are never stopped/restarted** except the release-layout rebind when the unit file changed or pid 1 |
| 132 | does not hold the port. Every `systemctl` call goes through the guard. |
| 133 | - **Protected sessions:** Live's `/api/streams`, Media's `/api/ready` counters, Events SSE, Chat sockets, Games players, |
| 134 | OpenRestream ingest, Tools jobs. A probe that **cannot answer while the service runs counts as active**, never zero. |
| 135 | `--wait-idle` waits for two idle checks in a row; `--force` proceeds with a warning. `null` from a service is |
| 136 | unknown, so the restart is held; an `addFields` field the service does not report at all is zero (older service). |
| 137 | - **Tracked local changes block a deploy**; they are never discarded. One operation per service at a time (pid lock |
| 138 | with stale recovery). Every attempt is in the release log, refusals included. |
| 139 | - **Readiness:** poll `/api/ready`; on failure restore the previous sha, reinstall dependencies if they differed, |
| 140 | restart, and exit 3 (or 4 + MANUAL INTERVENTION if that fails). |
| 141 | - **Dependencies** must resolve (`node_modules/<dep>/package.json` parses and names the package) before anything |
| 142 | restarts; install only when the lockfile/deps changed and restore `package-lock.json` if npm rewrote it. |
| 143 | - **nginx** changes are transactional: render, `nginx -t`, install, reload; a failed test restores the previous files |
| 144 | and does not reload. Certificates are inventory only; no key file is read. |
| 145 | - **Secrets** never leave an env file: no value in text/JSON output, logs, snapshots or backup manifests. Backups are |
| 146 | root-only (0700/0600), off-host copies AES-256-GCM encrypted and authenticated; a failed verification is `failed`, |
| 147 | never `skipped`. |
| 148 | - **Restore drills** check the dump, restore into a scratch database/role, start a sandboxed second instance, compare |
| 149 | declared HTTP paths and row counts, then stop it and drop the scratch objects. `drill` exits 0 passed, 1 refused, |
| 150 | 2 failed; run as root only. |
| 151 | - **Inventory** must be root-owned and not group/world-writable when `ovhost` runs as root; relative paths, non-loopback |
| 152 | probe URLs, non-SELECT probe queries and vhost names with a path are refused. `workerUnits` are never started/stopped |
| 153 | by ovhost; `auxiliaryUnits` restart only when their `restartOn` path changed. |
| 154 | - **Lifecycle:** each service's Contracts manifest (≥0.55.0) supplies liveness/shutdown/drains/workers/rollback/ |
| 155 | contracts/leases; the inventory block may override. `validate` fails on a missing field or a deadline longer than a |
| 156 | unit's stop timeout, and on a different `KillSignal`. |
| 157 | - **Self-update** refuses a dirty tree, a branch other than `main` or a divergence; it restarts nothing. |
| 158 | |
| 159 | ## Key Pitfalls |
| 160 | |
| 161 | - **This repository deploys two things differently:** `sudo ovhost deploy host` replaces the catalog checkout at |
| 162 | `/opt/openvibe.host`, but **not** the CLI install at `/usr/local/lib/openvibe-host`; after a merge to `main` run |
| 163 | `sudo ovhost self-update` or the CLI silently runs behind. |
| 164 | - **Never commit `host.json`** or a real env value. `host.example.json` is the model; `service add` copies an entry |
| 165 | and backs up the live inventory. |
| 166 | - **Stage C is not started** and Stage B's engine is gone. Do not resurrect `/projects`, `/sites`, `/deploys`, |
| 167 | `/domains` or `*.openvibe.host`; they answer 410 for a reason. |
| 168 | - **Console tokens:** never log, return, export or snapshot a provider token; only the last four clear characters may |
| 169 | appear. The export carries accounts without tokens. |
| 170 | - **Catalog honesty:** no estimated price, no converted currency, no affiliate-driven order, no invented benchmark |
| 171 | field. A source that does not state a value is Unknown. |
| 172 | - **IndexNow** is off without `INDEXNOW_KEY`; a re-check at the same price sends nothing; key served at `/<key>.txt`. |
| 173 | - **Freeze** (`ovhost freeze`) makes deploys exit 6; rollbacks are never frozen and `--force` goes through. |
| 174 | - **Browser check** is report only: it never changes the exit code or the release record. |
| 175 | - **Boot order drop-ins** (`openvibe-.service.d/`, PgBouncer after PostgreSQL) are `unitSources` of the host service; |
| 176 | `sudo ovhost validate host` reports drift and `sudo ovhost deploy host --install-units` installs them. |
| 177 | |
| 178 | ## Testing |
| 179 | |
| 180 | `npm test` runs every file in [test/](test/) with every system call against [test/fake-host.js](test/fake-host.js): |
| 181 | acceptance, deploy/rollback/guard, release layouts and strategies, probes, readiness, socket guard, env |
| 182 | validation, nginx/certs, backups (pgBackRest/logical/offsite), drills, data provisioning/role, boot-order drop-ins, |
| 183 | alerts/announce/incidents, catalog (ingest, keyed feeds, offers, domains/RDAP, benchmarks, discussion, search, trends, |
| 184 | staff), console, account data, actor limits, CSP, engine-moved 410s, browser check/watch, DNS, nodes, inference planner, |
| 185 | and the deploy wrapper. `npm run test:pg` for PostgreSQL. Run the narrowest file first, then the suite; always |
| 186 | `node --check` changed files. |
| 187 | |
| 188 | ## Documentation |
| 189 | |
| 190 | - [README.md](README.md) — catalog, console, static hosting moved, purpose/rules, owns, does-not-own, depends on, |
| 191 | capabilities, stages, acceptance, using ovhost, inventory, deploy, installing, security, development, launch rule. |
| 192 | - [docs/deploy-strategies.md](docs/deploy-strategies.md), [docs/backups.md](docs/backups.md), |
| 193 | [docs/restore-drills.md](docs/restore-drills.md), [docs/data-provisioning.md](docs/data-provisioning.md), |
| 194 | [docs/db-inventory.md](docs/db-inventory.md), [docs/browser-check.md](docs/browser-check.md), |
| 195 | [docs/release-notifications.md](docs/release-notifications.md), [docs/operator-alerts.md](docs/operator-alerts.md), |
| 196 | [docs/incidents-and-dns.md](docs/incidents-and-dns.md), [docs/no-502.md](docs/no-502.md), |
| 197 | [docs/ready-health.md](docs/ready-health.md), [docs/host-firewall.md](docs/host-firewall.md), |
| 198 | [docs/threat-review.md](docs/threat-review.md), [docs/launch.md](docs/launch.md), [docs/developer-path.md](docs/developer-path.md). |
| 199 | - [SECURITY.md](SECURITY.md), [host.example.json](host.example.json). |
| 200 |