openvibers/openvibe.host Commits
Host: delete the migration era (SQLite worker, backup/drill/probe branches, data switch, legacy deploy fallback) (#55)
Changes
@@@ -22,17 +22,15 @@ HOST_SITES_DOMAIN=openvibe.host# says hosting is not open yet. Set to on at launch.HOST_PUBLIC_HOSTING=off--# SQLite database and content-addressed objects (the production unit sets both under++# PostgreSQL database and content-addressed objects (the production unit sets both under# /var/lib/openvibe-host-api)# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh host: the runtime role through PgBouncer,--# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).++# the owner on a direct connection (migrations), and Valkey (shared limit counters).# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.DATABASE_URL=DATABASE_DIRECT_URL=VALKEY_URL=VALKEY_PREFIX=--# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.--HOST_DB_PATH=./data/host.dbHOST_STORAGE_DIR=./data/objects# Object storage. Unset/empty: local disk only (data/objects), exactly as before. `media`: every@@@ -2,27 +2,27 @@> The network's deployment/control plane first; then isolated hosting for community sites, bots and mods.--**Status:** alpha. Stage A (operator plane) is a CLI, `ovhost`, tested against a fake host. It is installed on the production host (2026-09-23, `/usr/local/bin/ovhost`, inventory `/etc/openvibe/host.json`) and it deploys 20 services (`ovhost deploy <svc>`: Network, Events, Chat, Billing, Tips, VIP, AI, Search, Sources, Wiki, Blog, News, Reviews, Deals, Coupons, Trade, Codes, Host, Media, Community; 70+ releases since 2026-09-23, with automatic rollback on a failed readiness check) and runs the nightly backups and the restore drills of 19 services. Live, Tools, Sites, Games and OpenRestream moved to it in the 2026-09-27 cutover (roadmap WS-N task 11) through the deploy strategies `release-layout`, `multi-app`, `static-build` and `pnpm-build`, so every service now deploys with `sudo ovhost deploy <svc>` (Live with `--wait-idle`); each of those repositories keeps `deploy/scripts/deploy.sh` as a thin wrapper that hands over to ovhost or falls back to its old script: [docs/deploy-strategies.md](docs/deploy-strategies.md). Stage B (tenant static hosting) is a service, written and tested, and **launched on 2026-10-07**: `openvibe.host` serves the dashboard, API and front page, and `*.openvibe.host` serves tenant sites (vhosts from `ovhost nginx tenants host --install`; the Sites placeholder and the pending 404 vhost are gone; certificate lifecycle on `openvibe-certs.timer`; an nginx catch-all refuses hosts no site names). The launch record, verification and rollback: [docs/launch.md](docs/launch.md); the threat review: [docs/threat-review.md](docs/threat-review.md). Stage C (sandboxed user code) is **not started**.++**Status:** alpha. Stage A (operator plane) is a CLI, `ovhost`, tested against a fake host. It is installed on the production host at `/usr/local/bin/ovhost` with inventory `/etc/openvibe/host.json`; it deploys services, runs nightly backups, and performs restore drills. Repository deploy scripts are thin wrappers around `ovhost deploy|rollback|plan`: [docs/deploy-strategies.md](docs/deploy-strategies.md). Stage B (tenant static hosting) serves the dashboard, API, front page, and tenant sites. See [docs/launch.md](docs/launch.md) and [docs/threat-review.md](docs/threat-review.md). Stage C (sandboxed user code) is **not started**.**Domain:** `openvibe.host` (dashboard and API) and `*.openvibe.host` (tenant sites). Launched on 2026-10-07 (see [Launch rule](#launch-rule)); OpenVibe.Sites no longer serves it.**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §15.3 and §15.17; roadmap Wave 21.**License:** AGPL-3.0 (same as every OpenVibe service).## Purpose--Every repository ships its own deploy script, and each one encodes rules learned from an outage. Stage A pulls those rules into one tool that works the same way for every service on the host:++Repository deploy scripts call ovhost. Stage A applies the deploy rules for every service on the host:| Rule | Where it came from | How `ovhost` applies it ||---|---|---|| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. || Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |--| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`sqlite-count` of a file, or `postgresql-count` of a database in the cluster: Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only **as the owning user** (the service user, or the postgres OS user for the cluster). |++| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`postgresql-count` of Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only as the postgres OS user. || Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. || Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. || Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. || Build, install vhosts, `nginx -t`, reload | Sites `deploy.sh` | `build` hooks and `nginx.installOnDeploy`. Vhost installs are transactional: if `nginx -t` fails, the previous files come back and nginx is not reloaded. || Build a release while the old one serves; switch a symlink; roll back to a release with its own `node_modules`; pid 1 must hold the socket | Live `deploy.sh` (release layout), OpenRestream `deploy.sh release`/`api` | Strategy `release-layout` ([docs/deploy-strategies.md](docs/deploy-strategies.md)). The socket unit is restarted only when its unit file changed or systemd does not hold the port. || Per-app installs (not `apps/_shared`), jobs runtime and guard load before the restart, every `openvibe-tools*` unit | Tools `deploy.sh` | Strategy `multi-app`: `skipPackages`, `preflight`, `unitsMatch`, `ready.release`. |--| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm, builds, SQLite queries and backups all run as the declared owner/service user (`runuser` from root, `sudo -u` otherwise); PostgreSQL dumps and probes run as the postgres OS user. |++| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm and builds run as the checkout owner; PostgreSQL dumps and probes run as the postgres OS user. |Stage A adds a few safety rules of its own:@@@ -33,10 +33,10 @@ Stage A adds a few safety rules of its own:## Owns--- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled backups (online SQLite copies, pgBackRest verification and logical PostgreSQL dumps) with retention and encrypted off-host copies, restore drills.++- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled backups (pgBackRest verification and logical PostgreSQL dumps) with retention and encrypted off-host copies, restore drills.- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.--- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user. `ovhost data provision <svc>` and `ovhost data switch <svc> [--sqlite <file>]` run `add-service.sh` and `switch-service.sh` from the installed CLI, redacting every URL and password: [docs/data-provisioning.md](docs/data-provisioning.md).++- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user. `ovhost data provision <svc>` runs `add-service.sh` from the installed CLI, redacting every URL and password: [docs/data-provisioning.md](docs/data-provisioning.md).## Does not own@@@ -82,11 +82,11 @@ credentials from `/etc/openvibe/host.env`):| Stage | Scope | State ||---|---|---|--| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup, restore drill. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp SQLite/HTTP/git). Installed on the host: deploys every service, runs the nightly backups and the restore drills. |++| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup, restore drill. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp HTTP/git). Installed on the host: deploys every service, runs the nightly backups and the restore drills. || **B: tenant static hosting** | The Host API service (port 4910): projects, sites, immutable content-addressed deploys, activation and rollback, `<site>.openvibe.host` and TXT-verified custom domains, quotas, upload logs, events, a server-rendered dashboard; tenant vhosts via `ovhost nginx tenants`. | **alpha**: written and tested (`npm test`, against a temp database, a mock Network and a DNS table). Runs on the host on `:4910` behind nginx; launched at openvibe.host on 2026-10-07. || **C: sandboxed user code** | Isolation profiles, CPU/memory/time/network/storage budgets, secret references, outbound policy, metering, kill/revoke without touching platform services. | **not started**, deliberately. Blocked until isolation and metering are proven. Nothing in Stage B runs tenant code. |--Not in Stage A yet (from the charter/roadmap list): container adapters, DNS adapters, certificate **renewal**, logs/metrics links, the release-manifest/active-client work in §15.18, and the Wave 22 cutover runbook. (Incident and maintenance controls, `ovhost incident|maintenance|freeze`, and release notifications, `host.release.published`, exist now.) Restore drills (`ovhost drill`) ran on the host for 19 services on 2026-09-23 ([docs/restore-drills.md](docs/restore-drills.md)).++Stage A also provides incident and maintenance controls (`ovhost incident|maintenance|freeze`) and release notifications (`host.release.published`). Restore drills are described in [docs/restore-drills.md](docs/restore-drills.md).## Acceptance (Stage A)@@@ -100,7 +100,7 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak- The socket unit is **never stopped or restarted**, in any flow, except the release-layout rebind: only when its unit file changed or systemd does not hold the port (`test/strategy-release-layout.test.js`).- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.--- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok` (a SQLite copy) or `pg_restore --list` on the `.dump` of a PostgreSQL service. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.++- A **restore drill** checks a PostgreSQL dump with `pg_restore --list`, restores it into a scratch database and role, starts a sandboxed second instance, compares declared HTTP paths and row counts, then stops it and drops the scratch database and role.- **Backups:**- `backup --all` backs up every service with databases. One failing service leaves the others backed up and exits 2.@@@ -130,7 +130,6 @@ ovhost capabilities [<service>] key=value lines the deploy wrappers proovhost self-update [--dry-run] update the CLI install (/usr/local/lib/openvibe-host) itself: fetch origin main, --ff-only merge, npm ciovhost service add <id> [--dry-run] first setup: inventory entry, checkout, 0600 env file, database if DATABASE_URL is declaredovhost data provision <service> [--dry-run] run roles/data/add-service.sh from the install: PostgreSQL database, roles and Valkey user (names only)--ovhost data switch <service> [--sqlite <file>] [--dry-run] run roles/data/switch-service.sh: stop, back up, import, deploy (docs/data-provisioning.md)ovhost --versionovhost announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run] release notificationovhost releases <service> [--limit <n>]@@@ -150,11 +149,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe`service add` runs as root against `/etc/openvibe/host.json`. It copies `services.<id>` from the installed CLI's `host.example.json`, drops `_note`, and backs up the live inventory before changing it. The example entry may set `cloneUrl` to a credential-free GitHub HTTPS URL; otherwise the repository is `https://github.com/OpenVibers/OpenVibe.<Capitalized id>.git`. The command refuses an existing entry, occupied checkout or env file. It holds the inventory lock (`<stateDir>/locks/_inventory.lock`) from reading the inventory to the end, and refuses a symlinked `.env.example`, which it reads with the checkout owner's permissions. It clones as the checkout owner, creates the entry's `envFile` (`/etc/openvibe/<id>.env`, 0600) as a copy of the checkout's `.env.example` with production `NODE_ENV`, `HOST` and `PORT`. It sets `BASE_URL` when the inventory vhost, origin or env example supplies a public HTTPS origin; otherwise it leaves an existing empty `BASE_URL` for the operator and does not add one to services without it. An explicitly required `BASE_URL` needs an inventory vhost or origin before setup begins. The tracked `.env.example` is never changed. It runs `data provision` if that example declares `DATABASE_URL`. Output lists env names but never values. `--dry-run` shows the plan without changing files. It then lists the env names still empty (leaving out the production overrides and, after provisioning, `DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`, `VALKEY_PREFIX`). When the example declares `OV_OAUTH_CLIENT_ID` or `OV_OAUTH_CLIENT_SECRET`, it prints one root command for the Network service principal. The command runs `server/setup/service-principal.js` through `systemd-run` with Network's env file (`services.network.envFile`, default `/etc/openvibe/network.env`) in Network's checkout (`services.network.repo`, default `/opt/openvibe.network`). It uses `rotate` when `list` already shows the id and `create` otherwise, so the principal never lands in a development database (the secret is never printed). It ends with `then: ovhost validate <id>; ovhost deploy <id> --restart`; ovhost runs neither. If a later step fails, the inventory backup path is printed; completed setup steps are not automatically undone.--**Deploy strategies** (roadmap WS-N task 11). The inventory entry's `strategy` picks how a service deploys: `git-checkout` (the default: an in-place checkout, every service above), `multi-app` (Tools), `static-build` (Sites), `pnpm-build` (Games) or `release-layout` (Live, OpenRestream: `releases/<id>` behind a `current` symlink, prepared while the old release serves, `--prepare-only` to stop after preparing). Each strategy carries the rules of the script it replaces: preflight checks, build output restored from git, untracked lockfiles, `unitsMatch`, `/release.json` naming the new sha, the socket rule. [docs/deploy-strategies.md](docs/deploy-strategies.md) has the engines, every inventory field, the rule-by-rule mapping from each script, what was not ported and why, the production cutover checklist and the proposed inventory diff.++**Deploy strategies** (roadmap WS-N task 11). The inventory entry's `strategy` picks how a service deploys: `git-checkout`, `multi-app` (Tools), `static-build` (Sites), `pnpm-build` (Games) or `release-layout` (Live, OpenRestream). Each strategy handles its service's preflight checks, build output, units and readiness rules. See [docs/deploy-strategies.md](docs/deploy-strategies.md).`deploy --browser-check` checks the service's public site in headless Chrome after a deploy that went through. It runs [`scripts/browser-check.js`](docs/browser-check.md) for that one site. The check is report only: it never changes the exit code or the release record. It needs Chrome where ovhost runs.--**Release notifications** (WS-P task 9). A deploy or rollback that went live is announced to OpenVibe.Events as `host.release.published`. The event is public, has subject `release` and carries the service, its `/release.json` release id, commit and origin. Open tabs (openvibe-shared release-watch 1.17.0 and later) then check `/release.json` within seconds instead of at their next poll. `ovhost announce <service>` does the same for services deployed by their own scripts (Live, Tools and Sites call it at the end of theirs). It is best effort: a few seconds at most, never a failed deploy, one event per release. The credentials are Host's service principal from `/etc/openvibe/host.env`. [docs/release-notifications.md](docs/release-notifications.md) covers the payload, the credentials, provisioning and an end-to-end check.++**Release notifications** (WS-P task 9). A deploy or rollback that went live is announced to OpenVibe.Events as `host.release.published`. The event is public, has subject `release` and carries the service, its `/release.json` release id, commit and origin. Open tabs (openvibe-shared release-watch 1.17.0 and later) then check `/release.json` within seconds instead of at their next poll. `ovhost announce <service>` can announce the currently served release manually. It is best effort: a few seconds at most, never a failed deploy, one event per release. The credentials are Host's service principal from `/etc/openvibe/host.env`. [docs/release-notifications.md](docs/release-notifications.md) covers the payload, the credentials, provisioning and an end-to-end check.### Browser check (all public sites)@@@ -180,11 +179,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe`sudo ovhost backup --all --offsite` runs daily from `deploy/systemd/openvibe-backup.timer` (03:30 UTC). It works like this:- It backs up every service that declares databases. A failing service never stops the others; exit `2` if anything failed.--- A **SQLite** database is copied online (as the service user). A **PostgreSQL** database is verified, not copied: every run checks pgBackRest for the whole cluster and records `verified`, and a logical `pg_dump -Fc` per database is taken when the last good one is over 7 days old or on any run with `--logical`. A failed verification is `failed`, never `skipped`.++- A **PostgreSQL** database is verified on every run: every run checks pgBackRest for the whole cluster and records `verified`, and a logical `pg_dump -Fc` per database is taken when the last good one is over 7 days old or on any run with `--logical`. A failed verification is `failed`, never `skipped`.- It keeps the last 7 daily and 4 weekly good backups per service.- It writes a JSON summary to `<stateDir>/backup-runs/<run>.json`.- It encrypts every copy on the host (AES-256-GCM, Node crypto, with a key that stays on the host) and uploads it to S3-compatible storage (Backblaze B2). Off-host copies are kept for 30 days.--- Backups are `root:root`: directories 0700, files 0600, no `-wal`/`-shm`. PostgreSQL dumps are made as the postgres OS user over peer auth; ovhost reads no service env value.++- Backups are `root:root`: directories 0700, files 0600. PostgreSQL dumps are made as the postgres OS user over peer auth; ovhost reads no service env value.`ovhost restore-download` fetches, verifies and decrypts a run into a new root-only directory, and never touches a live database. See [docs/backups.md](docs/backups.md) for the install steps, the env names (`/etc/openvibe/backup.env`), the encryption format and the restore procedure.@@@ -195,8 +194,8 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passeEach inventory entry's `drill` block declares:- `port`: the service port + 10000 in the example;--- `databases`: the env var that points the service at each restored copy, or `{ "url": "<NAME>", "directUrl": "<NAME>" }` for a PostgreSQL database (the drill creates a scratch database and role, restores the `.dump` into it, and sets both vars to it);--- `env`: overrides that turn side effects off. Values may use `{tmp}`, `{port}` and `{db:<name>}`;++- `databases`: `{ "url": "<NAME>", "directUrl": "<NAME>" }` for each PostgreSQL database (the drill creates a scratch database and role, restores the `.dump` into it, and sets both vars to it);++- `env`: overrides that turn side effects off. Values may use `{tmp}` and `{port}`;- `dirs` to create inside the drill directory;- `ready`: the readiness path;- `compare`: paths, plus volatile `ignore` keys where needed;@@@ -204,17 +203,17 @@ Each inventory entry's `drill` block declares:- `countsTolerance` (default `0`): how many rows a busy table may differ by, because production keeps writing between the backup read and the drill;- `supported: false` with a `reason` when a second instance cannot run without side effects.--In `host.example.json`, 25 services have drill blocks, and 24 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`) or a PostgreSQL database's two URL env vars, `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. A PostgreSQL drill always overrides `DATABASE_URL`, `DATABASE_DIRECT_URL` and `VALKEY_URL` and drops its scratch database and role in a `finally` ([docs/restore-drills.md](docs/restore-drills.md#a-postgresql-drill)). The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.++In `host.example.json`, 25 services have drill blocks, and 24 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries name the PostgreSQL database's two URL env vars, `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. A PostgreSQL drill always overrides `DATABASE_URL`, `DATABASE_DIRECT_URL` and `VALKEY_URL` and drops its scratch database and role in a `finally` ([docs/restore-drills.md](docs/restore-drills.md#a-postgresql-drill)). The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.### Inventory--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `sqlite-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a SQLite file `{ "name", "path" }` or a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.--`games` deploys with the `pnpm-build` strategy (a pnpm workspace with a TypeScript build: `pnpm install --frozen-lockfile`, `pnpm build`, the tracked `dist-types/` restored from git before the merge). An entry with `layout: "release"` and no `strategy` stays unmanaged: `deploy` and `rollback` refuse it and the repository's own script deploys it. The deploy-strategy fields (`strategy`, `release`, `preflight`, `skipPackages`, `removeUntrackedLockfiles`, `generated`, `installUnits`, `unitsMatch`, `announce.releaseFiles`, `install.lockfile`, `install.workspace`, `ready.release`, `ready.allUnits`) are listed in [docs/deploy-strategies.md](docs/deploy-strategies.md#inventory-fields).++`games` deploys with the `pnpm-build` strategy (a pnpm workspace with a TypeScript build: `pnpm install --frozen-lockfile`, `pnpm build`, the tracked `dist-types/` restored from git before the merge). A release layout requires `strategy: "release-layout"`. The deploy-strategy fields (`strategy`, `release`, `preflight`, `skipPackages`, `removeUntrackedLockfiles`, `generated`, `installUnits`, `unitsMatch`, `announce.releaseFiles`, `install.lockfile`, `install.workspace`, `ready.release`, `ready.allUnits`) are listed in [docs/deploy-strategies.md](docs/deploy-strategies.md#inventory-fields).## Deploy@@@ -243,17 +242,17 @@ This was done on 2026-09-23 (the CLI and inventory are installed); the steps stacd /usr/local/lib/openvibe-host && sudo npm ci --omit=dev --no-audit --no-fundsudo ln -s /usr/local/lib/openvibe-host/bin/ovhost /usr/local/bin/ovhost```-- The tree must stay world-readable (default 0755/0644). The SQLite worker runs as `ubuntu` from this directory.++ The tree must stay world-readable (default 0755/0644).2. **Inventory:** `sudo install -o root -g root -m 0640 host.example.json /etc/openvibe/host.json`, then correct it against the machine. Check the Community database path, the Tools unit list, and Events, which was not deployed as of 22 Sep.--3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots, drill logs and backup run summaries. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe` and everything in it is root-only (directories 0700, files 0600). The backup worker writes as the service user into its own directory under `/var/backups/openvibe.staging` (root, 0711), and root takes each copy over from there. All of these are created on first use.--4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm/SQLite (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.++3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots, drill logs and backup run summaries. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe` and everything in it is root-only (directories 0700, files 0600). The postgres OS user writes logical dumps into its own directory under `/var/backups/openvibe.staging` (root, 0711), and root takes each copy over from there. All of these are created on first use.++4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.5. **First run:** `sudo ovhost status`, then `sudo ovhost validate <each service>`, then `sudo ovhost plan live`, all read-only apart from `git fetch`. Adopt `deploy` one service at a time, starting with one that has no protected sessions.6. **Scheduled backups:** see [docs/backups.md](docs/backups.md#installing-on-the-host-operator) (bucket, key, `/etc/openvibe/backup.env`, `openvibe-backup.timer`).7. **Updating the CLI itself:** a deploy of this repository replaces `/opt/openvibe.host` (the Stage B service), not the root-owned install at `/usr/local/lib/openvibe-host`, so after a merge to `main` run `sudo ovhost self-update`. It finds its own install directory from its real path (`/usr/local/bin/ovhost` → `/usr/local/lib/openvibe-host/bin/ovhost`), fetches `origin main`, refuses a dirty tree, a branch other than `main` or a divergence, merges `--ff-only` and runs `npm ci --omit=dev --no-audit --no-fund` there; `--dry-run` prints the plan first. It restarts nothing — the next `ovhost` invocation runs the new code. Without it the installed CLI silently runs behind `main` (on 2026-10-02 it was 16 commits behind: no `env-names --set`, no `logs`, no Bot vhost).## Stage B: tenant static hosting--A Node service (`server/`, Express 4, better-sqlite3, port **4910**, service id `host`) that hosts static sites for OpenVibe projects. Every request is dispatched on its `Host` header first:++A Node service (`server/`, Express 4, PostgreSQL, port **4910**, service id `host`) that hosts static sites for OpenVibe projects. Every request is dispatched on its `Host` header first:| Host | What answers ||---|---|@@@ -516,7 +515,7 @@ records: [docs/threat-review.md](docs/threat-review.md). The rules the code keepstores, prints or snapshots a value. The one exception is its own root-only `/etc/openvibe/backup.env`.Backups are root-only (0700/0600) and off-host copies are AES-256-GCM encrypted and authenticated.- **Operator plane.** `ovhost` runs as root and drops to the checkout owner for git, npm, builds and-- SQLite; it never stops or restarts a socket unit except the release-layout rebind; a protected-session++ PostgreSQL; it never stops or restarts a socket unit except the release-layout rebind; a protected-sessionprobe that cannot answer counts as sessions active; tracked local changes block a deploy.- **Tenants (Stage B).** Static files only: no tenant code runs, no tenant secret is stored, uploads arevalidated and content-addressed, tenant hosts get no CORS grant, staff takedowns answer 451. Custom@@@ -535,7 +534,7 @@ npm run dev # Stage B service on http://localhost:4910 (HOST_SITStage B tests (`test/host-*.test.js`) boot the service on a temp database and object store with a controllable clock, a mock Network (real RS256 keys) and a DNS table, and talk to it over raw HTTP so they can send any `Host` header and request target.--Stage A: everything that touches the system goes through the executor (`lib/executor.js`). The tests replace it with `test/fake-host.js`, an in-memory host with git repos, systemd units, npm, nginx, ss, HTTP endpoints and SQLite. No test runs systemctl, git, npm, nginx or curl against the real machine. `test/executor.test.js` covers the real executor only where that is safe: SQLite on a temp database, HTTP to a local server, and temp files.++Stage A: everything that touches the system goes through the executor (`lib/executor.js`). The tests replace it with `test/fake-host.js`, an in-memory host with git repos, systemd units, npm, nginx, ss, HTTP endpoints and PostgreSQL. No test runs systemctl, git, npm, nginx or curl against the real machine. `test/executor.test.js` covers the real executor only where that is safe: HTTP to a local server and temp files.## Launch rule@@@ -56,7 +56,7 @@"deploy strategies: git-checkout, multi-app, static-build, pnpm-build, release-layout","protected-session probes and drain policies (--wait-idle)","certificate inventory and transactional nginx vhost install",-- "scheduled SQLite backups with retention and encrypted off-host copies",++ "scheduled PostgreSQL backup verification, logical dumps, retention and encrypted off-host copies","restore drills","release notifications (host.release.published), incidents, maintenance windows and freezes","Prometheus alerts relay to Network",@@@ -46,17 +46,6 @@ groups:labels: { severity: page }annotations:summary: "No successful nightly backup for more than 30 hours (or no backup metrics at all)"-- # A service whose databases are gone (the 2026-09 SQLite→PostgreSQL switch archived the .db files-- # and dropped the path from the inventory) is reported `skipped` by backup --all. openvibe_backup_last_run_ok-- # stays 1 (a skip is not a failure) and no other backup alert fires, so production can sit with no backups-- # at all. 30 h is longer than the 24 h between runs, so ONE skipped run never reaches `for` and TWO-- # consecutive ones always do.-- - alert: OpenVibeBackupSkipped-- expr: openvibe_backup_services{status="skipped"} > 0-- for: 30h-- labels: { severity: ticket }-- annotations:-- summary: "A service has been skipped by two nightly backups in a row ({{ with query \"openvibe_backup_services{status='skipped'}\" }}{{ . | first | value }}{{ end }} service(s)): its databases are gone, so it has no backups"# Production drift (OpenVibe.Network registry/deploy-drift.js, hourly against GitHub; WS-S task 7).# max_over_time: Network restarts drop the series for a minute, which reset `for` and made the alert# resolve and fire again every few hours (2026-09-28). Documentation-only drift is not counted.@@@ -1,13 +1,12 @@[Unit]--Description=OpenVibe backups: verify pgBackRest for the PostgreSQL services, dump SQLite databases and the scheduled logical PostgreSQL dumps, then an encrypted off-host copy++Description=OpenVibe backups: verify pgBackRest, create scheduled logical PostgreSQL dumps, then an encrypted off-host copyDocumentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.mdAfter=network-online.targetWants=network-online.target[Service]Type=oneshot--# Root: ovhost drops to each service user for the SQLite backup and to the postgres OS user for the--# PostgreSQL dumps and the pgBackRest verification (runuser) and is the only reader of++# Root: ovhost uses the postgres OS user for dumps and pgBackRest verification and is the only reader of# /etc/openvibe/backup.env, the encryption key and the root-only backups. The env file is read by# ovhost itself, not loaded into this unit's environment.User=root@@@ -19,8 +18,7 @@ UMask=0077Nice=10IOSchedulingClass=best-effortIOSchedulingPriority=7--# The PostgreSQL logical dumps are the large ones now (they run as the postgres OS user), and the--# SQLite worker's own limit is 6 h per database; a pg_dump is given the same. 6 h covers a whole run.++# A pg_dump has a six-hour limit, which also covers a whole run.TimeoutStartSec=6hPrivateTmp=trueNoNewPrivileges=true@@@ -34,7 +34,6 @@ LockPersonality=true# root-owned state directory (release logs, locks, snapshots).StateDirectory=openvibe-host-apiStateDirectoryMode=0750--Environment=HOST_DB_PATH=/var/lib/openvibe-host-api/host.dbEnvironment=HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects[Install]@@@ -2,9 +2,7 @@`ovhost backup --all --offsite` runs once a day from a systemd timer. It backs up every databasethe inventory declares, keeps a week of daily and a month of weekly copies on the host, and uploads--an encrypted copy of each run to S3-compatible object storage (Backblaze B2). A SQLite database is--copied; a PostgreSQL service is covered by the cluster's pgBackRest backups, which every run verifies,--with a per-service logical dump on a slower cadence. This page covers what it does, how to install it,++an encrypted copy of each run to S3-compatible object storage (Backblaze B2). The cluster is covered by pgBackRest, which every run verifies, with a per-service logical dump on a slower cadence. This page covers what it does, how to install it,and how to restore from the off-host copy.Restore drills (a second instance started from a backup and compared with production) are in@@@ -13,81 +11,11 @@ Restore drills (a second instance started from a backup and compared with produc## What is backed up--`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has--a `databases` or `objects` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):--SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster--(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:----| Service | Engine | Databases |--|---|---|---|--| network | SQLite | `network.db` |--| chat | SQLite | `chat.db` |--| tools | SQLite | seven `apps/*/data/analytics.db`, and `jobs.db` for img, audio and docs |--| games | SQLite | `world.db` |--| live | SQLite | `live.db` (includes the money ledger while `BILLING_AUTHORITY=live`), `rs-companion.db`. `data/analytics.db` is left out on purpose ([storage-state.md](storage-state.md): analytics, hard-coded path, not canonical). |--| tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media, openre | PostgreSQL | one database each in the cluster, named `ov_<name>` (for example `ov_trade`); there is no local file. The media files themselves are in B2/R2, not covered here. |----The 19 PostgreSQL services each run the every-night pgBackRest verification and a scheduled logical--dump; the five SQLite services keep the online copy described below. [PostgreSQL--services](#postgresql-services) covers what that means.----A database that does not exist yet (a service that is not deployed) is reported as `skipped` and--does not fail the run. Adding a database to the inventory is all it takes to include it.----**Object directories.** A service may also declare `objects`: a list of `{ name, path }` directories--whose files are content-addressed, each named by its own sha256. OpenVibe.Host's tenant object store--(`HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects`) is the first. `ovhost backup` archives each--directory as `<name>.tar.gz` (root:root 0600, like every other copy); the off-host code encrypts and--uploads it with the rest, and a [drill](restore-drills.md) extracts it into its own scratch directory--and checks every blob's sha256.++Every inventory database is PostgreSQL. `ovhost backup --all` verifies the cluster's pgBackRest backups and WAL archiver on every run. It makes a per-service `pg_dump -Fc` when the last good logical dump is over seven days old or when `--logical` is given. Declared content-addressed object directories are archived as `.tar.gz` files. Restore drills use the logical dumps and object archives.## One run--For each service, one after another:----1. **Copy.** The SQLite online backup API (`better-sqlite3` `.backup()`) makes a consistent copy while-- the service keeps running. The copy is made by a worker running **as the service user**: root never-- opens a service database, because a root-owned `-wal`/`-shm` file would stop the service from-- writing its own database.--2. **Check.** The worker switches the copy to `journal_mode=DELETE`, which checkpoints it and removes-- its `-wal`/`-shm`, and runs `PRAGMA quick_check`. A copy that does not answer `ok` fails.--3. **Take over.** The worker writes into `/var/backups/openvibe.staging/<service>-<stamp>/`, which-- belongs to the service user. Root then takes that directory back (`root:root 0700`), so the service-- user can no longer reach it. Root checks that each copy is a plain file with one link, and makes it-- `root:root 0600`. It then renames the directory to `/var/backups/openvibe/<service>/<stamp>/`.--4. **Record.** The backup is appended to `/var/lib/openvibe-host/backups/<service>.jsonl`, with-- `ok: false` when any database failed.--5. **Prune** (see [Retention](#retention)).----Steps 1-3 are the **SQLite** path. A PostgreSQL service does not copy a file at all: every run first--verifies pgBackRest, and a logical dump is taken only when the last good one is old enough (see--[PostgreSQL services](#postgresql-services)). `drill` uses the same `.dump` artifacts.----**Permissions.** Backups hold user data. Every directory under `/var/backups/openvibe` is--`root:root 0700` and every copy is `root:root 0600`, with no `-wal`/`-shm` next to it. Directories left--by older versions (`ubuntu`, 0750) are locked down on the next run. The staging parent--`/var/backups/openvibe.staging` is `root:root 0711`, and each run removes its own staging directory.----**Failures.** A failing database never stops the other databases of the service, and a failing--service never stops the other services. The run's exit code is `2` when anything failed and `0`--otherwise. If a service fails halfway (one of two databases), the good copy is kept but the backup is--marked `ok: false`. Such a backup is never picked by a drill, and it is pruned once a newer good backup--exists. A PostgreSQL service whose pgBackRest verification fails is reported `failed`, **never--`skipped`**: a broken backup layer is a failure, not an absent database.----**Summary.** Each run writes `/var/lib/openvibe-host/backup-runs/<run>.json` (`root`, 0640). It holds--per-service status, files, sizes, errors, what was pruned, and the off-host result: objects, bytes,--failures, and the key id. A PostgreSQL service that only verified pgBackRest is reported `verified`.--The summary never holds a secret. The run id is the start time in UTC, for example `20260924-033412`.----```--sudo ovhost backup --all # local only--sudo ovhost backup --all --offsite # what the timer runs--sudo ovhost backup --all --json # the summary on stdout--sudo ovhost backup --all --logical # force the logical PostgreSQL dumps too--sudo ovhost backup --all --keep-daily 14 --keep-weekly 8--sudo ovhost backup --all --no-prune--```++Run `sudo ovhost backup --all --offsite`. A failed pgBackRest or WAL check is recorded as a failure. A run with no dump due and no object directories records successful verification without creating an artifact directory. Backup artifacts are root-owned, with 0700 directories and 0600 files; `pg_dump` writes into a postgres-owned staging directory before root takes ownership. The off-site uploader encrypts artifacts before upload.## PostgreSQL services@@@ -120,7 +48,7 @@ layout, one file per database:/var/backups/openvibe/<service>/<stamp>/<name>.dump # pg_dump -Fc, owned root:root 0600```--The off-site code then encrypts and uploads each `.dump` exactly as it does a `.db` copy, so a `.dump`++The off-site code encrypts and uploads each `.dump`, so the archivereaches the bucket as `<service>/<name>.dump.ovbk` and is restored by `restore-download` unchanged.`pg_restore --list` proves a `.dump` is a readable archive.@@@ -137,11 +65,7 @@ node_exporter's textfile collector:- `DataBackupStale` fires when `openvibe_pgbackrest_last_backup_age_seconds` is over **30 hours**, or themetric is absent (no run has written it). It is the cluster staleness alert; the per-service`openvibe_backup_pg_verified` says which service's verification failed.--- `OpenVibeBackupSkipped` fires when a service has been `skipped` for **30 hours** — two nightly runs in a-- row. A service whose databases are gone (for example one whose SQLite file was archived after the-- 2026-09 switch to PostgreSQL) is `skipped`, `openvibe_backup_last_run_ok` stays `1` and nothing else-- fires, so this is what catches "production with no backups". The rules live in-- [`deploy/prometheus/openvibe-rules.yml`](../deploy/prometheus/openvibe-rules.yml).++The rules live in [`deploy/prometheus/openvibe-rules.yml`](../deploy/prometheus/openvibe-rules.yml).### pgBackRest is the durability layer@@@ -174,7 +98,6 @@ To leave deletion to a bucket lifecycle rule instead, set `BACKUP_OFFSITE_PRUNE=### Layout```--s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.db.ovbk # a SQLite copys3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.dump.ovbk # a PostgreSQL dumps3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/manifest.json```@@@ -312,8 +235,7 @@ sudo jq '{ok, services: [.services[] | {service, status, error}], offsite: {ok:The monitoring stack alerts on it (all in `deploy/prometheus/openvibe-rules.yml`):`OpenVibeBackupFailed` pages when the last run failed, `OpenVibeBackupMissed` when no run has succeeded--for 30 hours, `OpenVibeBackupSkipped` when a service has no databases left to back up, and--`DataBackupStale` when pgBackRest itself is stale. The unit also fails (exit 2) and the summary says++for 30 hours, and `DataBackupStale` when pgBackRest itself is stale. The unit also fails (exit 2) and the summary says`"ok": false`.### Bucket (Backblaze B2)@@@ -365,38 +287,14 @@ For each file, the command:- checks the manifest's HMAC;- streams the object through GCM decryption;- checks both SHA-256 hashes against the manifest;--- runs `PRAGMA quick_check` on the result of a `.db` copy, or `pg_restore --list` on a `.dump`.++- runs `pg_restore --list` on each `.dump`.Any failure removes that file and exits non-zero. The output lists each file with its size, its hash,its check and the production path it came from.--### 3. Put it in place (manual, one service at a time)----Take a fresh local backup of the current state first, even a broken one, so the restore can be undone.--Then stop the service, move its current database aside, install the copy as the service user's file,--and start the service again. For Live, check protected sessions first (`ovhost status live`). The--socket unit stays up, and only the service unit stops.++### 3. Restore a service database--```--sudo ovhost backup live # the state you are about to replace--sudo systemctl stop openvibe-live.service # never the .socket unit--cd /opt/openvibe.live/data--sudo mv live.db live.db.replaced-$(date -u +%Y%m%d-%H%M%S)--sudo rm -f live.db-wal live.db-shm # only after the service has stopped--sudo install -o ubuntu -g ubuntu -m 0640 /var/lib/openvibe-restore/live-20260924-033412/live.db live.db--sudo systemctl start openvibe-live.service--curl -s 127.0.0.1:3000/api/ready | jq .status--```----- Use the production path from the restore output (`was …`). The owner and mode must match the other-- files in that directory (`ls -l`).--- For services with a `StateDirectory` (`/var/lib/openvibe-*`), the directory is owned by the service-- user.--- Remove `/var/lib/openvibe-restore/<service>-<run>` when you are done: it holds user data.--- **Money.** `live.db` holds the ledger while `BILLING_AUTHORITY=live`. A restore loses every-- transaction made after the backup. Before any money action resumes, reconcile against the payment-- provider for the gap (see [cutover-runbook.md](cutover-runbook.md)). Consider setting the owner-only-- `money_writes_frozen` flag first.++Use `pg_restore` from the downloaded `.dump` into a new PostgreSQL database, then point the service's `DATABASE_URL` and `DATABASE_DIRECT_URL` at it. Check the application before changing production settings. `ovhost drill` automates a scratch restore and comparison without changing the production database.### Disaster recovery: a new host@@@ -427,27 +325,7 @@ sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-communitysudo rm -rf /var/lib/openvibe-restore/drill-community```--The drill:--- restores each database. A SQLite copy goes into the sandbox as the service user (`install -o <user>-- -m 0600`) and is checked with `integrity_check`; a PostgreSQL `.dump` is restored with `pg_restore`-- into a scratch database created for the drill ([restore-drills.md](restore-drills.md));--- starts a sandboxed second instance and compares it with production;--- stops it and cleans up (a PostgreSQL drill always drops its scratch database and role).----Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.----For a service without a drill (ai), or for live and media until the release with their drill--switch is deployed (`LIVE_DRILL`, `MEDIA_DRILL`; `drill.requires` refuses an older checkout), download the--copy and check it by hand:----```--sudo ovhost restore-download live latest --out /var/lib/openvibe-restore/check-live--sudo sqlite3 -readonly /var/lib/openvibe-restore/check-live/live.db 'PRAGMA integrity_check; SELECT count(*) FROM users;'--sudo rm -rf /var/lib/openvibe-restore/check-live--```----(`sqlite3` is the Debian package of the same name. `restore-download` has already run `quick_check`;--this adds a full `integrity_check` and a look at the data.)++The drill verifies each PostgreSQL dump with `pg_restore --list`, restores it into a scratch database, compares the second instance with production, and drops the scratch database and role. For a service without a supported drill, verify the downloaded `.dump` with `pg_restore --list` before a manual scratch restore.Do this at least once after installing, and again after any change to the key or the bucket.@@@ -455,7 +333,7 @@ Do this at least once after installing, and again after any change to the key or| Command | What it does | Exit ||---|---|---|--| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases or object directories (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`, archive object directories); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |++| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases or object directories (verify pgBackRest, dump PostgreSQL when due or `--logical`, archive object directories); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure || `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure || `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error || `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |@@@ -134,22 +134,6 @@ estate now carries a service token with the one capability it performs (Contract| C-84 | **Live deploy notice over the Chat bridge** (`chat-remote.js` `deployNotice`) next to the `live.release.deployed` event (Live `e73a92d`). | Live `server/chat/deploy-notice.js`, `server/chat/chat-remote.js` | Chat + Live | *done 2026-09-23*: Chat subscribes to `live.release.deployed` (Chat `ed90461`, table `deploy_releases` keyed by head) and folds by `subject.id`; the 22:14 UTC deploy produced one card via both paths (`duplicate:release`), and since Live `5324f40` (22:18 UTC deploy: `first_via = events`, folded) the bridge hop runs only while Live's Events publishing is off | 2026-10-14 | Keep the bridge call || C-85 | **Live's own go-live follower push** (`server/streaming/golive-notify.js`) next to Network's `live.stream.started` consumer (subscription since 2026-09-23 19:55 UTC; Live `GET /internal/followers`). A shared per-streamer rate limit in Network stops double notifications. | Live `golive-notify.js` + call sites in `routes.js`, `rtmp-server.js`, `whip-handler.js`, `openre/mirror.js`; Network `server/notifications/stream-live.js` | Live + Network | *documented* (Network commit 2a95548): Network's log shows `outcome: notified` for real go-lives; then delete Live's push and retire Network's `POST /internal/events/stream-live`. | 2026-10-07 | Unsubscribe Network from `live.stream.started` |--## PostgreSQL switches (ADR-035, roadmap WS-X2)----Each service that moves keeps its SQLite file read-only for the 7-day N-1 window, as the rollback (the previous release reads it). One row per service, added at its switch.----| # | What | Where | Owner | Removal condition | Target | Rollback lever until then |--|---|---|---|---|---|---|---|--| C-89 | **Closed 2026-09-29 (plan T1):** the SQLite files of the 19 services that moved to PostgreSQL on 2026-09-28 (post-switch files and pre-switch backups: tips, wiki, blog, news, deals, coupons, trade, reviews, search, sources, codes, vip, ai, host, events, community, billing, media, openre), Community's pre-c24/pre-live-comments backups and Billing's `live-snapshot-202609230121.db` are archived off-host (`ovhost archive` `20260929-124813`: 44 files, 805 MB encrypted; a restore of `codes.db` verified byte for byte) and deleted from the host. The rollback is `ovhost archive restore 20260929-124813 <path> --out <dir>` plus a release older than the switch. | the archive | Host | — | done | `ovhost archive restore` |----**Archive step unblocked (2026-09-28).** `ovhost archive push` refuses a path the inventory declares as a--live SQLite database (`lib/archive.js`: it refuses a file that is any non-postgresql `databases[].path`).--Until the switch, that rule covered these frozen `.db` files, so the "archive both files, then delete"--step of C-89 could not run. The 19 services above now declare `engine: postgresql` with no `path`, so their--sqlite entries no longer name a file and archive push accepts the frozen `.db` files. The archive-then-delete--step of each C-89 row can proceed once its 7-day window closes; the removal condition and target of every--row stand unchanged.--## Kept by design (not shims)These match the search terms but are permanent. They are listed so the next review does not re-open them.ddeleted file mode 100644@@@ -1,379 +0,0 @@--# Final cutover runbook (roadmap Wave 22)----The Wave 22 exit criterion is: "The final cutover runbook, with rollback, is absorbed into OpenVibe.Host."--This page gathers every service cutover of the realignment into one ordered runbook. Each section gives the--cutover's production state and its rollback, and links the per-repository document that stays--authoritative for the details. When a step here and its original disagree, the original wins; fix this--page.----- Host: `ssh openvibe-ovh` (single host). The public IPv4 is in `~/OpenVibers/RUNBOOK.md` on the-- operator workstation, which is not in a repository.--- Env files: `/etc/openvibe/<service>.env`. Units: `openvibe-<service>.service`. Checkouts:-- `/opt/openvibe.<service>`.--- Live paths are from `~/orca/workspaces/OpenVibe.Live/seadragon` (`ae07553`).--- Production state is from the operator notes of 2026-09-23.----State markers:--- **DONE**: executed on production, with the date.--- **IN PROGRESS**: partly on production.--- **BLOCKED (owner)**: waits for something only the owner can do. The action is spelled out.--- **READY**: nothing blocks it.--- **NOT STARTED**----## Overview----| # | Cutover | State | Per-repo original | Rollback in one line |--|---|---|---|---|---|--| A | Pastes → OpenVibe.Community (Wave 5) | **DONE** 2026-09-22 ~23:35 UTC | ADR-011; Community `README.md:16-21,354,387-390` | `PASTES_AUTHORITY=live` in Community and Live, remove Media's two flags, restart all three |--| B | Chat → OpenVibe.Chat (Wave 6) | **DONE** 2026-09-23 02:03 UTC | Chat [`docs/cutover.md`](https://github.com/OpenVibers/OpenVibe.Chat/blob/main/docs/cutover.md) | Remove the nginx include, drain the mirror, unset `CHAT_AUTHORITY`, restart Live |--| C | AI → OpenVibe.AI (Wave 13) | **DONE** 2026-09-23 ~15:40 UTC (`AI_SERVICE=remote`) | AI [`docs/migration.md`](https://github.com/OpenVibers/OpenVibe.AI/blob/main/docs/migration.md) | Unset `AI_SERVICE`, restart Live |--| D | Events delivery signature v2 | **IN PROGRESS**: Events sends v2; consumer deploys unknown; v1 still sent | Events [`docs/replay-window-rollout.md`](https://github.com/OpenVibers/OpenVibe.Events/blob/main/docs/replay-window-rollout.md) | The consumer redeploys its previous release; replay the DLQ |--| E | Money → OpenVibe.Billing | **BLOCKED (owner)**: PowerChat dashboard. Billing runs in shadow. | Billing [`docs/live-cutover.md`](https://github.com/OpenVibers/OpenVibe.Billing/blob/main/docs/live-cutover.md) | Depends on the step reached. After step 9 there is **no tool** |--| F | RTMP ingest → OpenRestream | **BLOCKED (owner)**: broadcaster windows and the per-slot rehearsal. DNS `ingest.openre.stream` is done, port 1936 is open (2026-09-23) and OpenRestream is deployed. | OpenRestream [`docs/cutover.md`](https://github.com/OpenVibers/OpenRestream/blob/main/docs/cutover.md) (steps, checks and rollback; `scripts/cutover-preflight.js`) | Per slot `{"authority":"live"}` + `set-definition-state.js --state disabled`. All slots: unset `OPENRE_URL` |--| G | Retire the shims | **NOT STARTED**. Dated in [compatibility-register.md](compatibility-register.md) | Network [`docs/retirement.md`](https://github.com/OpenVibers/OpenVibe.Network/blob/main/docs/retirement.md) | per entry |----The remaining order is D → E → F → G.--- D first: nothing depends on it, and v1 can stop only when every consumer requires v2.--- E before F: the Billing switch restarts Live once, and it is best done before any stream depends on-- OpenRestream.--- G follows each entry's date in the register.----## What the owner must do----These actions need the owner's accounts or physical presence. Nothing in this runbook can do them.----1. **PowerChat dashboard, for step E4.** In the PowerChat developer dashboard, for OpenVibe.Live's app,-- change the webhook URL:-- - from `https://openvibe.live/api/powerchat/webhook`-- - to **`https://billing.openvibe.network/webhooks/powerchat`**---- Keep the **same signing secret** (Billing's `POWERCHAT_WEBHOOK_SECRET` equals Live's-- `powerchat_webhook_secret`). Do this only at step E4 of the sequence, **not before**: Live freezes-- first, and Billing is frozen to hold deliveries. A 410 from Live makes PowerChat disable the endpoint-- (Billing `docs/live-cutover.md:131-134, 170-175`).--2. **Cloudflare DNS, for step F.** Create an A record `ingest.openre.stream` pointing at the host's IP,-- **DNS only (grey cloud)**. RTMP cannot go through Cloudflare's proxy (OpenRestream `README.md:145`,-- `deploy/nginx/openre.stream.conf:15-17`).--3. **Port 1936/tcp, for step F.** Open it in the host firewall **and** at the provider edge (OpenRestream-- `README.md:145,160`). *Done 2026-09-23: 1936 answers from outside since 19:24 UTC.*--3a. **DNS changes in a cutover (any step that moves a hostname).** A day before: lower the record's TTL to 60 s-- and write down the current records (`dig +noall +answer <name> A AAAA CNAME` into the step's notes). At the-- switch: change the record, then watch the new target for 30 minutes (readiness, error rate, the step's own-- checks) before calling the step done. Keep the old target running until the old TTL has passed twice over;-- only then decommission it, and put the TTL back.--4. **Broadcaster windows, for step F.** For each slot, agree a maintenance window with its broadcaster.-- The slot is offline, and the broadcaster regenerates the key and pastes the new server into OBS-- (ADR-009).--5. **Cloudflare browser cache TTL** (ADR-016, not a cutover, but it affects every deploy). Set the zone-- to "Respect existing headers". Today Cloudflare raises `/shared/*` `max-age=300` to 4 h.----## 0. Deploy rules that apply to every step----These rules are learned from outages. `ovhost` encodes them (see [README](../README.md#purpose)). As of--2026-09-23 it is used read-only on the host, so the per-repo scripts are still what runs. Since WS-N task 11--(2026-09-27) those scripts are wrappers around `ovhost deploy <svc>` that fall back to the old script--(`deploy/scripts/deploy-legacy.sh`) until the host's ovhost and inventory are cut over--([deploy-strategies.md](deploy-strategies.md)); the rules below hold either way.----- **Live**: `sudo /opt/openvibe.live/deploy/scripts/deploy.sh [--wait-idle|--restart|--force]`-- (`deploy/scripts/deploy.sh:7-12`).-- - Production uses the legacy in-place layout (Live `docs/deploy.md:31`). `--rollback` is refused-- there (`deploy.sh:347`), and a failed-readiness rollback resets the checkout without restoring-- `node_modules`. The release layout (`deploy/scripts/migrate-to-releases.sh`) is register entry C-76.-- - **`openvibe-live.socket` survives restarts.** It has no `PartOf=`-- (`deploy/systemd/openvibe-live.socket:13-23`). The script makes sure it is active and restarts only-- `openvibe-live` (`deploy.sh:205-222`). HTTP queues on the socket during boot. WebSocket, WHIP, WebRTC-- and RTMP sessions drop and reconnect. **Never stop or restart the socket.**-- - Readiness: polls `/api/ready` every 1 s for up to 90 s. If it is not ready, the script rolls back and-- exits 3; if the rollback also fails, it exits 4 (**manual intervention**) (`deploy.sh:31-32,139-147`).-- - `--wait-idle` polls `/api/streams` every 60 s and needs 2 idle checks in a row, for up to 8 h-- (`deploy.sh:120-137`). The script treats a failed `curl` as zero streams (`deploy.sh:121-125`).-- Check by hand when it matters.-- - `public/`, `docs/` and tests deploy without a restart. Schema changes take an online backup to-- `data/backups/live-<ts>.db` first (`deploy.sh:192-203`). `deploy/nginx/` is **never** installed by-- the script.-- - Owner decision, 2026-09-23: restarts while streamers are live are permitted "for now". Prefer-- `--wait-idle` when it costs nothing, and say when a restart hit a live stream.--- **Media**: there is **no deploy script** in the repo. Pull, `npm ci --omit=dev` if dependencies-- changed, then `systemctl restart openvibe-media`.-- - **Do not restart Media while it is recording** (hazard H3,-- `OpenVibe.Network/docs/roadmap-baseline/10-hazards.md:11`). Check first:-- `curl -s 127.0.0.1:4100/api/ready | jq .recordings_in_progress`, or ovhost's probe-- `SELECT count(*) FROM vods WHERE is_recording = 1` run read-only as the service user.-- - If a restart does cut a recording: 5 s after boot, Media finalizes every `is_recording = 1` row that-- has no ffmpeg. It merges chunks, remuxes, falls back to `.master.mkv`, and marks very short results-- `needs_review` (`server/index.js:197-210`, `server/vod/finalize.js`). Interrupted clips are marked-- failed.-- - **Known gap:** the unit's `TimeoutStopSec=30` (`deploy/systemd/openvibe-media.service:16`) is-- shorter than the recorder's ffmpeg stop grace (`VOD_STOP_GRACE_MS`, default 60 000 ms,-- `server/vod/recorder.js:35`). systemd therefore SIGKILLs a stop that is still flushing a recording,-- and the boot sweep has to recover it.--- **Tools**: `deploy/scripts/deploy.sh` (run from the checkout).-- 1. Fetch, and drop untracked lockfiles that the incoming release tracks.-- 2. `git pull --ff-only`, and `npm install --omit=dev` in apps whose `package.json` changed.-- 3. Reinstall any dependency that does not resolve. If it still does not, **ABORT, nothing restarted**.-- 4. Check that img, audio and docs load the jobs runtime.-- 5. Restart **all** `openvibe-tools*` units at once, then curl the gateway's `/api/health`.---- (`deploy/scripts/deploy.sh:1-42`.) It has **no automatic rollback** and does not check `/api/ready`.-- `apps/gateway/deploy/scripts/deploy.sh` is the superseded per-app script. Do not use it.--- **OpenRestream**: `sudo bash /opt/openre.stream/repo/deploy/scripts/deploy.sh release|api|workers|status|prune`.-- - `api` restarts the API and coordinator only, polls `/api/ready` for 60 s, and rolls back on failure-- (exit 2).-- - `workers` starts a new generation. The old ones drain for up to 30 min.-- - **Never `systemctl restart` a worker instance during a broadcast.** (OpenRestream `README.md:147-160`.)--- **Everything else** (Network, Community, Events, Chat, Billing, and the loopback services): pull,-- install if the lockfile changed, restart the unit, then poll `/api/ready`. Network's own-- `deploy.sh` and Sites' `deploy.sh` exist (Host `README.md`, rules table).--- **Always**:-- - Stage explicit paths when committing on the host. Never `git add -A`: on 2026-09-23 02:09 it-- committed half-done work and took Network down for two minutes.-- - Production WebSocket probes send `join` only, never `chat` or `dm` frames.-- - Take `sudo ovhost backup <service>` before any data move.----## A. Pastes → OpenVibe.Community (Wave 5): DONE 2026-09-22----Community owns pastes and comments (ADR-011). Production values:----| Service | Setting | Effect |--|---|---|---|--| Community | `PASTES_AUTHORITY=community` in `community.env` | Serves pastes natively. The database is `/var/lib/openvibe-community/community.db`. |--| Live | `PASTES_AUTHORITY=community` in `live.env` | `/api/pastes` forwards to Community with a service token (`server/pastes-client.js`, `server/media-proxy/pastes.js:250-319`). |--| Media | `PASTES_FROZEN_APPS=live` | Paste writes answer 410 (`server/pastes/routes.js:47-55`). |--| Media | `PASTES_MOVED_TO=https://openvibe.community` | 301 for `/p/:slug` and raw text (`server/public/routes.js:446-494`). |--| Tools | none | Forwards `/api/pastes/*` to Community with the visitor's token, and has no switch (`apps/gateway/server/index.js:183-207`). |----Import: 889+2 pastes, 60 comments, 48 likes. Held: 1 paste and 4 comments, `ambiguous_owner`. Shadow--parity before the flip was 878/878 (ADR-011 §Migration consequences).----**Verify:** `curl -s 127.0.0.1:4200/api/ready | jq .pastes_authority` must print `"community"`.----**Rollback** (ADR-011 §Rollback):--1. Export the writes made in Community since the cutover. **No export tool exists.** Write one before you-- need it (register C-10).--2. Set `PASTES_AUTHORITY=live` in `community.env` and `live.env`.--3. Remove `PASTES_FROZEN_APPS` and `PASTES_MOVED_TO` from `media.env`.--4. Restart Community, Live and Media.---- Tools has no switch: revert `fd5d318` in Tools and deploy. Community deletes are soft and never-- remove Media bytes.----## B. Chat → OpenVibe.Chat (Wave 6): DONE 2026-09-23 02:03 UTC----Chat serves `/ws/chat`, `/api/chat/`, `/api/dm/`, `/api/tts/` and `/api/sounds`. It runs on--127.0.0.1:4400 (`openvibe-chat`) with `/var/lib/openvibe-chat/chat.db`, `LIVE_MIRROR=1` and `EVENTS_URL`--set. Traffic reaches it through `include /opt/openvibe.chat/deploy/nginx/*.locations.conf;` in--openvibe.live's vhost. Live runs `CHAT_AUTHORITY=chat` and `OV_CHAT_INTERNAL_URL`. Its chat tables are a--read mirror kept current by Chat (register C-01–C-03).----Record of the run (Chat `docs/cutover.md`):--- Prerequisites `:51-84`: Network principal `chat` and grants; Live patch deployed without the flag;-- Chat installed and not started.--- Rehearsal `:86-123`.--- Cutover `:125-167`:-- 1. Start Chat and import snapshot 1.-- 2. Set `CHAT_AUTHORITY=chat`, restart Live, add the include, `nginx -t && systemctl reload nginx`.-- 3. Import pass 2. `held` must be 0.-- 4. Check that `mirror.pending` and `chat_bridge_outbox` are near 0, and that no `ChatRemote` refusals-- appear.--- Imports: pass 1 had 70,860 messages, held 0. Parity was 15/15 read paths (ADR-010 §Acceptance tests).--- The pre-cutover nginx backup is `/root/openvibe.live.conf.pre-chat-<ts>` on the host.----**Verify:** `curl -s 127.0.0.1:4400/ready | jq '.live, .mirror'`.--- `mirror.pending` should be near 0.--- `live.failures` should not grow.----`/ready` stays 200 while Live syncs fail (see [storage-state.md](storage-state.md) finding 1), so read the--body, not just the status.----**Rollback** (Chat `docs/cutover.md:169-184`, ADR-010 §Rollback). The order matters:--1. Remove the include from openvibe.live's vhost, then `sudo nginx -t && sudo systemctl reload nginx`.-- Chat traffic returns to Live, which answers 503 until step 3.--2. Make sure everything Chat wrote has reached Live. `curl -s 127.0.0.1:4400/ready | jq .mirror.pending`-- must be `0`. If Chat is down, run `scripts/mirror-flush.js` in the Chat checkout; exit 0 means it is-- drained. This must happen **before** step 3, because Live accepts mirror writes only while-- `CHAT_AUTHORITY=chat`.--3. Remove `CHAT_AUTHORITY` from `/etc/openvibe/live.env` and restart Live-- (`deploy.sh --restart`). At boot, `drainToLocal()` applies `chat_bridge_outbox` to Live's own tables-- (`server/chat/chat-remote.js:351-372`). A write Chat applied but never acknowledged is applied twice-- (`:354`).--4. `sudo systemctl stop openvibe-chat`. Keep `/var/lib/openvibe-chat/chat.db`.----## C. AI → OpenVibe.AI (Wave 13): DONE 2026-09-23----OpenVibe.AI runs on 127.0.0.1:4700 as principal `ai`, with `/var/lib/openvibe-ai/ai.db`. Live runs--`AI_SERVICE=remote` (`server/ai/ai-service.js:39`). With it, `llm.complete`, translation, paste/frame,--overview, media and recap calls go through AI workflows. There is **no fallback to Live's own key** when--AI returns nothing (`server/ai/llm.js:299-313`). Streamer BYO providers bypass AI.----Steps from AI `docs/migration.md:72-99`:--- Steps 1-6 (deploy, import, grants, Live `AI_SERVICE=remote`, verify) are **DONE**.--- Step 7 is **DONE**: Network calls `network.site_copy` itself (Network `422f8e9`). Live's-- `/internal/ai/site-copy` remains as Network's fallback (register C-21).--- The Live grant names its namespaces (`live.*`, `network.site_copy`; Network `988330b`), and AI fails-- closed on a token with no `ns` (register C-22/C-23).----**Verify:**--- `curl -s 127.0.0.1:4700/api/ready`.--- `GET /api/v1/runs?workflow=live.translate` shows recent succeeded runs.--- `journalctl -u openvibe-live | grep '\[AI service\]'` is empty.----**Rollback** (AI `docs/migration.md:101-103`, ADR-015 §Rollback):--1. Unset `AI_SERVICE` in `live.env` and restart Live. Live then uses its own provider settings. **Keep the-- shared key in Live's settings** until C-20's removal date. Without it, this rollback has no provider.--2. Namespace problems only: set `AI_NS_FALLBACK` back to its default, or set `AI_NS_REQUIRED=false`, in-- `ai.env`, then restart AI.----Stopping OpenVibe.AI touches no product data.----## D. Events delivery signature v2: IN PROGRESS----Events signs every attempt with v1 (`X-OpenVibe-Signature`) **and** v2 (`X-OpenVibe-Timestamp`,--`X-OpenVibe-Signature-V2` = HMAC of `"<t>.<raw body>"`, ±300 s) (Events `server/worker.js:56,66-68`).--v1 cannot be switched off by env.----| Step (Events `docs/replay-window-rollout.md:22-34`) | State |--|---|---|--| 1. Tag openvibe-sdk `v0.4.0` | **DONE**: the tag exists. The doc's status line (`:3`) is stale. |--| 2. Deploy Events. Confirm one real delivery carries the v2 headers. Compare `date +%s` with a trusted clock. | **DONE**: Events sends v2 (operator notes). |--| 3. Codes docs | **DONE** in the repo (Codes `a9371c3`). The tester still checks v1 only (register C-61). |--| 4. Consumers, one at a time, in this order: Live (`/internal/openre-events`), Search, News, Deals, Trade, Reviews, then Tips and VIP after a clean day, then Examples | **Committed in every repo**: Live `ae07553`, Search `5a7e59b`, News `1802e7d`, Deals `e72aeec`, Trade `9a55644`, Reviews `3151c05`, Tips `435f32b`, VIP `dc98781`, Examples `6ea9cfd`. **Deployed: unknown.** Check `git -C /opt/openvibe.<svc> log -1` for each. |--| 5. Stop sending v1 (optional; only after every consumer requires v2) | **NOT STARTED** (register C-60, target 2026-10-07) |----For each consumer deploy:--- Pin `openvibe-sdk` v0.4.0 (Search has no SDK), run `npm test` on Node 22, and deploy.--- Watch that service for `401 *.bad_signature`.--- Watch Events with `GET /api/v1/deliveries?status=failed&subscription_id=…`.----**Rollback** (rollout doc `:36`, ADR-004): redeploy the consumer's previous release, which unsets--`requireV2`. Events keeps retrying for about 2 h 13 min (1 s, 5 s, 30 s, 2 min, 10 min, 1 h, 1 h), then--moves the delivery to the DLQ. Requeue with `POST /api/v1/deliveries/replay { subscription_id, event_ids }`--or `{ subscription_id, from_seq }`. Replays are signed afresh, and the consumer inbox dedupes on--`event_id`. Rolling back Events itself is safe only while no consumer requires v2.----## E. Money → OpenVibe.Billing: BLOCKED on the owner (PowerChat dashboard)----Billing runs in shadow on 127.0.0.1:4600 (principal `billing`), and its staff console is public at--`https://billing.openvibe.network`. Live is the authority: `BILLING_AUTHORITY` unset = `live`--(`server/monetization/money-authority.js:26-31`). Billing's own status is "prepared, **not executed**"--(Billing `docs/live-cutover.md:3`).----**Before the day** (Billing `docs/live-cutover.md:56-102`):----| Prerequisite | State |--|---|---|--| 1. `billing.env`: `POWERCHAT_WEBHOOK_SECRET` equal to Live's, `POWERCHAT_SITE_USERNAME`, `POWERCHAT_ALLOW_TEST_FULFILLMENT` off, `BILLING_*` rates equal to Live's | `POWERCHAT_SITE_USERNAME` set 2026-09-23 (operator notes). The rest: unknown. Check with `sudo ovhost validate billing`, which prints names only. |--| 2. Network grants for client `live`: `billing.intent.create`, `billing.transfer.create`, `billing.balance.read`, `billing.cashout.request`, `billing.subscription.manage`, `billing.entitlement.check`, plus `identity.subject.resolve`. **Not** `billing.cashout.manage` or `billing.ledger.admin`. | **DONE** (operator notes: Live granted `billing.*` except `cashout.manage` and `ledger.admin`) |--| 3. Staff console: redirect URI, `BILLING_STAFF_SUBJECTS`, `BILLING_SESSION_SECRET`, vhost, sign-in works | **DONE** (console live; staff subject is the owner only) |--| 4. Live patch deployed with `BILLING_AUTHORITY` unset; `/api/admin/money` shows `authority: "live"` | **DONE** (Live `c384787`) |--| 5. `identity-legacy-sync` has run since the last new account; a fresh shadow import has no holds for anyone with a balance | Shadow import done 2026-09-23: 322 transactions, reconcile OK. Re-run it on the day. |----**The sequence** (Billing `docs/live-cutover.md:116-168`). Keep the order: it guarantees that every--PowerChat delivery lands in exactly one ledger.----0. Check who is live: `curl -s https://openvibe.live/api/streams`. Step 6 restarts Live.--1. Freeze Live money writes: `POST /api/admin/money/freeze {on:true, reason:'Billing cutover'}` (owner).-- `GET /api/admin/money` must show `frozen: true`.--2. Wait **at least 60 minutes**, because minted PowerChat links live for an hour. Then run-- `POST /api/powerchat/reconcile`. List `payment_orders` that are `pending` or `paid` from the last 3-- days. A `paid` row is an anomaly; fix it first.--3. Back up Billing to `/var/lib/openvibe-billing/billing-pre-cutover.db`, **keep it for rollback**. Then-- run `node scripts/freeze.js on 'Live cutover'`.--4. **Owner: re-point the PowerChat webhook** (see "What the owner must do", item 1). From here on, Billing-- stores and holds every delivery (202).--5. Take the final snapshot to `/var/backups/openvibe/live-cutover-<date>.db` and **keep it**. Run-- `import-live.js --dry-run`, then the import, then `reconcile.js`. `holds` must be empty for anyone with-- money, and reconciliation must print **OK**.--6. Add `BILLING_AUTHORITY=billing`, `OV_BILLING_INTERNAL_URL=http://127.0.0.1:4600` and-- `OV_BILLING_PUBLIC_URL=https://billing.openvibe.network` to `live.env`. Run-- `sudo /opt/openvibe.live/deploy/scripts/deploy.sh --wait-idle`.-- `GET /api/admin/money` must show `authority: "billing"`, `reachable: true` and `drift: []`.--7. Unfreeze Billing with `node scripts/freeze.js off`. Held deliveries settle once. Deliveries that Live-- already credited are rejected (`billing.intent_settled_in_live`). Then `reconcile.js` must print OK.--8. Verify the reads while Live is still frozen: balances match, and `billing.actions.attention: []`.--9. Unfreeze Live: `POST /api/admin/money/freeze {on:false}`.--10. Make a real 100-Vibes PowerChat purchase. Expect one settled event, +100 exactly once, reconcile OK,-- and Live's legacy columns unchanged.----**Rollback** (Billing `docs/live-cutover.md:177-200`):--- **Before step 6:**-- 1. Re-point PowerChat back to `https://openvibe.live/api/powerchat/webhook`.-- 2. Leave Billing **frozen**.-- 3. Unfreeze Live. Live's reconciler (every 15 min) backfills payments made in the window.--- **After step 6, before step 9:**-- 1. Remove `BILLING_AUTHORITY` and run `deploy.sh --wait-idle`.-- 2. Re-point PowerChat to Live and unfreeze Live.-- 3. Stop Billing, restore `billing-pre-cutover.db` (delete its `-wal`/`-shm` files), freeze it, and start-- it.--- **After step 9:** ADR-012's rollback re-derives Live's columns from Billing's journal. **No tool for this-- exists.** It is manual and lossy. Prefer fixing forward.----**What the switch does not preserve** (`:202-239`):--- PowerChat side effects that arrived on Live's webhook (chat lines, alerts, goals) stop at the re-point.--- Displays that read Live's `transactions` table freeze.--- Renewal notices stop.--- The SPA sends no `Idempotency-Key` yet.----Read the full list before choosing the day. VIP checkout stays closed until this cutover.----## F. RTMP ingest → OpenRestream: BLOCKED on the owner (rehearsal and broadcaster windows; DNS and port 1936 are done)----OpenRestream is deployed. The API is `openre-api` on 127.0.0.1:4500. The other units are--`openre-session-coordinator`, `openre-rtmp-ingest@<sha>` and `openre-restream-worker@<sha>`. The database--is `/var/lib/openre/openre.db`. RTMP is bound to 127.0.0.1:1936 (`OPENRE_RTMP_BIND`). A loopback rehearsal--with an ffmpeg test pattern passed: the session went live→ended, a wrong key was refused, `openre.session.*`--reached Events, and the key never appeared in logs.----Live carries the integration (`f0ca18b`). It is inert until `OPENRE_URL` is set. `ingest_authority`--defaults to `live` on every slot (`server/openre/schema.js:5-7`), and only RTMP moves--(`server/openre/authority.js:18`). WHIP, JSMPEG and SFU are **not ported** (OpenRestream `README.md:207-209`).----**Once, before the first slot.** OpenRestream's [`docs/cutover.md`](https://github.com/OpenVibers/OpenRestream/blob/main/docs/cutover.md) supersedes the table below: it has the exact commands, the owner and agent steps (A1–A7), and a read-only preflight whose output on 2026-09-23 settles the "Unknown" rows (no subscription, no `OPENRE_*` in `live.env`).----| Step | State |--|---|---|--| 1. Network client `openre` and its grants; release the contract proposals | **DONE** (operator notes: principal `openre` in `openre.env`; openre capabilities released by contracts v0.19.0) |--| 2. Deploy OpenRestream; `curl 127.0.0.1:4500/api/ready` shows ready, with one ingest worker, one restream worker and a valid coordinator lease | **DONE** |--| — Owner: DNS-only `ingest.openre.stream`; 1936/tcp open at the host and provider edge; RTMP bound publicly (`OPENRE_RTMP_BIND`) | **BLOCKED (owner)** |--| 3. Events subscription `openre.session.*` → `http://127.0.0.1:3000/internal/openre-events`; its secret in Live as `OPENRE_EVENTS_SECRET` | Unknown. Check `GET /api/v1/subscriptions` on Events and `ovhost validate live`. |--| 4. Live: `OPENRE_URL=http://127.0.0.1:4500`, `OPENRE_PUBLIC_URL`, then `deploy.sh --wait-idle`. With no slot switched this changes nothing. | Unknown. Check `GET /api/admin/openre/status`. |--| 5. Rehearsal with a test account from OBS to `rtmp://ingest.openre.stream:1936/live`. Deploy Live and `deploy.sh api` during the broadcast, and check that nothing drops. | Needs the DNS and port first |----**Per slot** (`README.md:196-203`; `migrate-from-live.js --checklist` prints the same list):--1. Agree a window with the broadcaster. The slot must be offline.--2. Run `node scripts/migrate-from-live.js --live-db <fresh snapshot> --apply --slots <id>`. Review held-- destinations on openre.stream.--3. `PUT /api/admin/openre/managed/<id>/ingest-authority {"authority":"openre"}` on Live. From then on,-- Live refuses the old key and has rotated its own copy (`server/openre/authority.js:103-146`).--4. The broadcaster presses **Regenerate stream key** on Go Live (it is shown once) and sets OBS to-- `rtmp://ingest.openre.stream:1936/live`.--5. Rotate the broadcaster's personal Live key (`users.stream_key`). ADR-009 requires every pre-cutover-- key to be rotated.--6. Test a broadcast. Watch `deploy.sh status`, `/api/admin/openre/status` and output health.----**Rollback:**--- Per slot: `{"authority":"live"}`. The broadcaster regenerates the key on Live and points OBS back at the-- Live RTMP URL.--- All slots: unset `OPENRE_URL` in `live.env` and restart Live (`README.md:205`,-- `server/openre/authority.js:47-49`).--- Live's ingest code stays until the last protocol has run on OpenRestream for two weeks (ADR-009 §Rollback).--- Port 1935 moves to OpenRestream (`OPENRE_RTMP_EXTRA_PORTS=1935`) only after Live stops listening-- (`README.md:139-145`).----## G. Retiring the shims: NOT STARTED----Every remaining shim has an owner, a removal condition, a date and a rollback lever in--[compatibility-register.md](compatibility-register.md). The ones on the critical path:----- **The shared internal key** (C-50–C-58), per Network `docs/retirement.md`:-- 1. Move each caller to a service token by 2026-10-09.-- 2. Remove each key path after 14 days with no legacy use. Check with the `principal_usage` query in-- that doc.-- 3. Delete `INTERNAL_API_KEY` from every env file by 2026-11-06.--- **The hard-coded `openvibe-internal-2026` secret** in Live cosmetics (C-59): remove it by 2026-10-09.--- **Events v1 signatures** (C-60): stop them after every consumer is deployed with `requireV2`.--- **Chat's Live mirror** (C-01–C-03), and Live's frozen legacy tables and their readers (C-73): target-- 2026-11-06.----Before removing a shim, take `sudo ovhost backup <service>` for any service whose data it touches. Update--the register row in the same change.@@@ -1,58 +1,5 @@--# Provisioning a service's PostgreSQL database (`ovhost data`)++# Provisioning a service database--The data role ([roles/data/README.md](../roles/data/README.md), ADR-035) is configured from code:--`add-service.sh` gives one service its database, roles and Valkey user, and `switch-service.sh` moves a--service from SQLite to PostgreSQL. The agent pipeline may run only `ovhost` through its broker, so both--scripts are reachable from the CLI:++`sudo ovhost data provision <service> [--dry-run]` runs `roles/data/add-service.sh` from the installed ovhost tree. The command requires a service listed in the host inventory and root privileges for a real run. Dry run prints the command without changing the host.--```--sudo ovhost data provision <service> [--dry-run]--sudo ovhost data switch <service> [--sqlite <file>] [--dry-run]--```----Both run the script **from the ovhost install** (`<install>/roles/data/`), exactly as--`sudo ovhost self-update` maintains it, and **pass the script's own exit code through** (a `switch` that--gives up on protected sessions exits the same code the script did). `--dry-run` prints what would run and--changes nothing. A service the inventory does not list is refused before anything runs. Both must run as--root (`sudo ovhost data …`; `--dry-run` need not): the scripts need it, and an inner `sudo` would reset--`SWITCH_UNITS`/`SWITCH_DIR` and switch the default unit and directory instead.----- **`data provision`** runs `roles/data/add-service.sh <service>` with no arguments of its own. It is-- idempotent: passwords are generated once into `/etc/openvibe/data.env`, the roles, database and Valkey-- user are created if missing, and the four connection settings are replaced in place in the service's-- env file. The script prints **names only** — `DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`,-- `VALKEY_PREFIX` — never a value.--- **`data switch`** runs `roles/data/switch-service.sh <service> [<file>]`. It takes `SWITCH_UNITS` and-- `SWITCH_DIR` from the inventory entry (`units` plus `workerUnits`, a template worker shown as a glob-- such as `openre-rtmp-ingest@*.service`, plus the `unitsMatch` glob when set; `repo`), so a release-layout service such as OpenRestream needs no-- hand-set environment. `--sqlite <file>` names the SQLite file when it is not-- `/var/lib/openvibe-<service>/<service>.db` (Host: `/var/lib/openvibe-host-api/host.db`). Before running-- it, the service's postgres branch must be merged with green CI and its auto-deploy frozen-- (`sudo ovhost freeze <service> --reason …`); the script unfreezes and deploys at the end. Its output-- is collected and printed when the script exits, not streamed: while it waits (up to 12 hours) for-- protected sessions to go idle, no `[switch] waiting…` line appears, so give the caller (the broker) a-- timeout longer than that wait, or switch when the service is idle.----**`switch` needs the service's `scripts/migrate-to-postgres.js`.** The script clones `main`, installs it--and runs that file (`openvibe-sdk` `runSqliteMigration`: migrate, import, verify counts and checksums)--with the unit's own environment. A service without it cannot be switched this way.----**No URL or password appears in the output.** `add-service.sh` prints names only, and every line either--command writes out (stdout, stderr, `--json`, and the `--dry-run` command line) passes through a redact--pass: a `postgres://`, `postgresql://`, `redis://` or `valkey://` URL becomes `<redacted-url>` and a--`…PASSWORD=value` becomes `…PASSWORD=<redacted>`. This is asserted by `test/data-provision.test.js`.----## Precedent: the 2026-10-02 manual runs----On 2026-10-02 Tools, Chat and Network were provisioned and switched **by hand** with exactly these--scripts, before the CLI wrappers existed:----| Service | Switch |--|---|---|--| Tools | `scripts/migrate-to-postgres.js` over `apps/*/data` |--| Chat | `scripts/import-sqlite-to-pg.js` |--| Network | its T2 runbook |----Those runs are the reason `data provision` takes no inventory-specific flags and `data switch` reads the--units and directory from the inventory: the scripts already encode the per-service rules, and the CLI only--supplies what the entry knows.++The role creates the PostgreSQL database and roles, a Valkey user, and sets `DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL` and `VALKEY_PREFIX` in the service env file. It is idempotent and prints setting names rather than values. ovhost redacts connection URLs and password assignments from command output.@@@ -1,123 +1,15 @@--# Databases: the inventory model, and the files left on the host (WS-S task 6, hazard H14)++# PostgreSQL databases in the host inventory--`host.example.json` declares each service's databases as a `databases[]` list. An item is a union of two--**engines** ([lib/inventory.js](../lib/inventory.js) normalises them, [lib/dbengine.js](../lib/dbengine.js)--holds the per-engine helpers):++Every `services.<id>.databases[]` entry names a database in the host PostgreSQL cluster:--- **SQLite** — a file the service user's worker copies:-- ```json-- { "name": "live", "path": "/opt/openvibe.live/data/live.db" }-- ```--- **PostgreSQL** — a database in the host's cluster, named, with **no `path`**:-- ```json-- { "name": "trade", "engine": "postgresql", "database": "ov_trade" }-- ```++```json++{ "name": "trade", "engine": "postgresql", "database": "ov_trade" }++```--`engine` defaults to `sqlite`. A PostgreSQL `database` must match `/^ov_[a-z0-9_]+$/`; a PostgreSQL entry--has no `path`, and a SQLite entry has no `database`. The data lives in the cluster, so ovhost never opens--it by copy: pgBackRest backs the whole cluster up, the nightly run verifies that, and a per-service--logical `.dump` feeds the drills ([backups.md](backups.md#postgresql-services)).++`engine` must be `postgresql`, `database` must match `ov_[a-z0-9_]+`, and the entry cannot have `path`. `ovhost backup` verifies pgBackRest and periodically makes a `pg_dump -Fc` archive. A restore drill maps each database name to the service's URL settings:--**Which engine.** With `host.example.json`:++```json++{ "trade": { "url": "DATABASE_URL", "directUrl": "DATABASE_DIRECT_URL" } }++```--| Engine | Services |--|---|---|--| PostgreSQL | tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, bot, host, events, community, billing, media, openre (20) |--| SQLite | network, chat, tools, games, live (5) |----**The two engines appear in four more places in the inventory:**----- `drill.databases`. A SQLite database keeps a string (the env var that points the service at the restored-- file) or `{ "env": "DATA_DIR", "dir": true }` for a service that takes a data directory; a PostgreSQL-- database is `{ "trade": { "url": "DATABASE_URL", "directUrl": "DATABASE_DIRECT_URL" } }`, and the drill-- creates a scratch database and role and points both vars at it ([restore-drills.md](restore-drills.md)).--- `drill.countsTolerance` — a whole number of rows, default `0`. Production keeps writing between the read-- that makes the backup and the dump, so a busy table may differ by up to this many rows without failing-- the drill.--- `nginx.tenants.database` — the Stage B tenant database (the Host API is on PostgreSQL): a SQLite path-- keeps its string form, and PostgreSQL is the object `{ "engine": "postgresql", "database": "ov_host" }`.--- the `protected` probe kind `postgresql-count` — `{ "kind": "postgresql-count", "database": "ov_openre",-- "sql": "SELECT count(*) …", "label": "…" }`: a read-only SELECT against a database in the cluster, run as-- the postgres OS user. OpenRestream's protected probe (its active ingest sessions) is one, on `ov_openre`. The-- other kinds are `http-json-count`, `sqlite-count` and `sum` ([lib/probes.js](../lib/probes.js)).------The rest of this page is the 2026-09-26 sweep of SQLite **files**: every file under /opt, /var/lib, /root,--/home and /srv that no process had open. 42 open files (the live databases) are not listed. Paths, sizes--and dates only; nothing was read. A PostgreSQL service has no row here: its rows are in the cluster.----`ovhost archive push <file>...` encrypts files with the backup key and copies them to `<BACKUP_S3_PREFIX>-archive/<host>/<stamp>/`, outside the backup runs so retention never prunes them; `ovhost archive list` and `ovhost archive restore <stamp> <path> --out <dir>` read them back. It never deletes a local file.----| Class | File | Size | Modified | Recommended action |--|---|---|---|---|---|--| legacy | `/opt/backups/hobo.quest-2026-08-16/data/world.db` | 0.1 MB | 2026-08-16 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/backups/legacy-quest-20260817/hobo-quest.db` | 69.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/backups/pre-openvibe-20260817/hobo-tools-final.db` | 731.0 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/backups/pre-openvibe-20260817/hobo-tools.db` | 731.0 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/backups/pre-openvibe-20260817/hobostreamer-final.db` | 177.7 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/backups/pre-openvibe-20260817/hobostreamer.db` | 177.7 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/backups/pre-openvibe-20260817/world-final.db` | 0.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/hobo.quest-old-20260816/data/world.db` | 0.1 MB | 2026-08-16 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/hobo.quest/data/world.db` | 0.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/hobo/hobo-quest/data/hobo-quest.db` | 69.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/hobo/hobo-tools/data/hobo-tools.db` | 731.0 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/hobo/hobo-tools/data/hobotools.db` | 0.0 MB | 2026-03-22 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/hobostreamer/data/hobostreamer.db` | 177.7 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| legacy | `/opt/hobostreamer/hobostreamer.db` | 0.0 MB | 2026-08-11 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |--| live-backup | `/opt/openvibe.live/shared/data/backups/live-20260926-072748.db` | 270.9 MB | 2026-09-26 | delete (owner OK) once its migration is a release old; archive first if wanted |--| live-backup | `/opt/openvibe.live/shared/data/backups/live-pre-643a2e6.db` | 252.5 MB | 2026-09-17 | delete (owner OK) once its migration is a release old; archive first if wanted |--| live-backup | `/opt/openvibe.live/shared/data/live-before-rs-slots-20260924T0153Z.db` | 266.2 MB | 2026-09-24 | delete (owner OK) once its migration is a release old; archive first if wanted |--| live-backup | `/opt/openvibe.live/shared/data/live.db.bak-preAIstate` | 175.9 MB | 2026-08-19 | delete (owner OK) once its migration is a release old; archive first if wanted |--| rehearsal | `/var/lib/openvibe-billing/live-snapshot-202609230121.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |--| rehearsal | `/var/lib/openvibe-chat/live-snap-a.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |--| rehearsal | `/var/lib/openvibe-chat/live-snap-b.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |--| rehearsal | `/var/lib/openvibe-chat/live-snapshot-0159.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |--| rehearsal | `/var/lib/openvibe-chat/rehearsal.db` | 0.3 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |--| migration-backup | `/var/lib/openvibe-community/community.pre-c24-20260925T185117Z.db` | 2.1 MB | 2026-09-25 | delete (owner OK); the migrations are verified |--| migration-backup | `/var/lib/openvibe-community/community.pre-c24b-20260925T185400Z.db` | 2.1 MB | 2026-09-25 | delete (owner OK); the migrations are verified |--| migration-backup | `/var/lib/openvibe-community/community.pre-c24c-20260925T185425Z.db` | 2.1 MB | 2026-09-25 | delete (owner OK); the migrations are verified |--| migration-backup | `/var/lib/openvibe-community/community.pre-live-comments-20260923T191345Z.db` | 1.9 MB | 2026-09-23 | delete (owner OK); the migrations are verified |--| ip-bearing | `/opt/hobo/hobo-audio/data/analytics.db` | 24.7 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/hobo/hobo-docs/data/analytics.db` | 1.4 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/hobo/hobo-food/data/analytics.db` | 0.5 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/hobo/hobo-img/data/analytics.db` | 3.3 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/hobo/hobo-maps/data/analytics.db` | 1.0 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/hobo/hobo-text/data/analytics.db` | 6.2 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/hobo/hobo-yt/data/analytics.db` | 0.5 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.live/shared/data/backups/analytics-pre-adr021-2026-09-23.db` | 1682.4 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/audio.analytics.db` | 57.5 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/docs.analytics.db` | 1.6 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/food.analytics.db` | 0.4 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/img.analytics.db` | 5.4 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/maps.analytics.db` | 0.6 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/text.analytics.db` | 9.1 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/yt.analytics.db` | 0.9 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |--| secrets | `/opt/openvibe.network/data/network-pre-refresh-hash-2026-09-24.db` | 266.2 MB | 2026-09-24 | delete after the rollback window (WS-S task 5), never archive |--| secrets | `/opt/openvibe.network/data/network-pre-secrets-2026-09-24.db` | 266.2 MB | 2026-09-24 | delete after the rollback window (WS-S task 5), never archive |--| stale | `/opt/openvibe.blog/data/blog.db` | 0.4 MB | 2026-09-24 | delete (owner OK) |--| stale | `/opt/openvibe.community/data/community.db` | 0.2 MB | 2026-09-25 | delete (owner OK) |--| stale | `/opt/openvibe.games/data/legacy-import/world-before-20260923T184930Z.db` | 0.2 MB | 2026-09-23 | delete (owner OK) |--| stale | `/opt/openvibe.live/shared/data/hobo.db` | 0.0 MB | 2026-03-13 | delete (owner OK) |--| stale | `/opt/openvibe.live/shared/data/openvibe.db` | 0.0 MB | 2026-08-18 | delete (owner OK) |--| stale | `/opt/openvibe.live/shared/data/rs-companion.db` | 8.2 MB | 2026-03-10 | delete (owner OK) |--| migration-backup | `/opt/openvibe.media/data/backups/owner-subject-20260923.media.db` | 9.1 MB | 2026-09-23 | delete (owner OK); the owner-subject migration is verified |--| migration-backup | `/opt/openvibe.media/data/reports/h15-repair-applied-20260925.media.db` | 10.8 MB | 2026-09-25 | delete (owner OK) once the H15 repair is a release old |--| stale | `/opt/openvibe.tips/data/tips.db` | 0.0 MB | 2026-09-23 | delete (owner OK) |--| in use | `/opt/openvibe.tools/apps/gateway/data/token-revocations.db` | 0.0 MB | 2026-09-25 | keep: Tools' guard opens it per request (apps/_shared/guard/revocations.js) |--| stale | `/root/blog-stale-db-before-revert-20260924T174243.db` | 0.4 MB | 2026-09-24 | delete (owner OK) |--| system | `/var/lib/PackageKit/transactions.db` | 0.0 MB | 2026-05-11 | leave |--| system | `/var/lib/command-not-found/commands.db` | 3.6 MB | 2026-07-27 | leave |--| system | `/var/lib/fwupd/pending.db` | 0.0 MB | 2026-07-27 | leave |----Classes:--- **legacy**: the pre-OpenVibe sites' databases; the only copies, so they are archived before anything else happens.--- **live-backup**, **rehearsal**, **migration-backup**: copies made around a migration; the nightly encrypted backups cover the live data.--- **ip-bearing**, **secrets**: dated deletions in WS-S task 5 (section 10); archiving them would defeat the point.--- **stale**: files an older layout left behind (H14 names the Blog one).--- **system**: Ubuntu's own.--- **in use**: closed at the moment of the inventory but opened on demand; never archive or delete.----**Archived 2026-09-26:** the 14 legacy files (2.8 GB) as `20260926-094703` (`ovhost archive list`); a restore of `/opt/hobo.quest/data/world.db` came back byte-identical.----Deleting anything here is the owner's call (WS-S task 6). The files a deletion may touch are exactly the rows above.++The drill creates a scratch database and role, restores the archive, and points both settings at the scratch database. `nginx.tenants.database` uses `{ "engine": "postgresql", "database": "ov_host" }`. A `postgresql-count` protected-session probe runs a read-only SELECT through the postgres OS user.@@@ -4,18 +4,14 @@Live (release layout), Tools, Sites, OpenRestream and Network, and Games (which deployed by hand). Each serviceentry in the inventory names a **strategy**: an engine plus the defaults the script it replaces encoded.Every repository keeps `deploy/scripts/deploy.sh` as a thin wrapper that maps its old flags onto ovhost,--with the old script body kept as `deploy/scripts/deploy-legacy.sh` for the fallback.++and invokes ovhost directly.--**State (2026-09-28):** in production since the 2026-09-27 cutover (the checklist below was run then).--Every service, including the six here, deploys with `sudo ovhost deploy <svc>` and rolls back with `sudo ovhost rollback--<svc>`; the strategies are tested against the fake host (`test/strategy-release-layout.test.js`,--`test/strategy-in-place.test.js`) and in each repository (the wrapper tests). The wrappers' legacy fallback is only--reached where an ovhost without `capabilities` answers, which no longer happens on the production host.++Production deploys and rollbacks use `ovhost deploy <svc>` and `ovhost rollback <svc>`. The strategy tests exercise each implementation against the fake host.| Service | Strategy | Replaces | Wrapper (`deploy/scripts/deploy.sh`) ||---|---|---|---|| live | `release-layout` | Live `deploy.sh` (release layout) | `--wait-idle`, `--restart`, `--rollback` → `ovhost rollback live`, `DRY_RUN=1` → `ovhost plan live`; `--force` refused (see below) |--| openre | `release-layout` | OpenRestream `deploy.sh release` + `api` | `deploy`, `release [<ref>]` → `--prepare-only`, `api [<sha>]` → `deploy --to <sha>`, `rollback`, `plan`; `workers`, `status`, `prune` stay in the legacy script |++| openre | `release-layout` | OpenRestream `deploy.sh release` + `api` | `deploy`, `release [<ref>]` → `--prepare-only`, `api [<sha>]` → `deploy --to <sha>`, `rollback`, `plan` || tools | `multi-app` | Tools `deploy.sh` | `--wait-idle`, `--restart`, `--force`, `--rollback`, `DRY_RUN=1` || sites | `static-build` | Sites `deploy.sh` (after a manual pull) | always `ovhost deploy sites --restart` (rebuild every run, as before); `DRY_RUN=1` || games | `pnpm-build` | the manual procedure | `--wait-idle`, `--restart`, `--force`, `--rollback`, `DRY_RUN=1` |@@@ -23,41 +19,9 @@ reached where an ovhost without `capabilities` answers, which no longer happensEvery other service keeps the default strategy, `git-checkout`, which is what `ovhost deploy` did before.--## The wrappers and their fallback++## Deploy wrappers--Each wrapper first asks `ovhost capabilities <svc>` (run as root: `sudo` when the wrapper is not root). It--hands over to ovhost only when every line it needs is there:----```--ovhost=0.3.0--deploy-api=1--strategies=git-checkout,multi-app,static-build,pnpm-build,release-layout--commands=plan,deploy,rollback,announce,capabilities--deploy-flags=--wait-idle,--force,--restart,--to,--install-units,--ready-timeout,--prepare-only,--no-announce--service=live--strategy=release-layout--managed=yes--layout=release--```----`deploy-api` is at least 1, `strategy=` is the one the wrapper was written for, and `managed=yes`.--Otherwise it prints why (`ovhost not found`, `no 'capabilities' (too old)`, `deploy-api is 0`, `the host--inventory does not deploy live with strategy release-layout (none)`, …) and runs `deploy-legacy.sh` with the--original arguments. So an old ovhost, a missing one, or an inventory that has not been changed yet never--breaks a deploy. `OVHOST_LEGACY=1` forces the fallback; `OVHOST=<path>` picks another ovhost. Where the legacy--script cannot do what was asked (Tools, Network and Games have no `--rollback`; none of them has `DRY_RUN`--except Live), the wrapper refuses instead of doing something else. The Sites fallback pulls first (it restores--`dist/` and runs `git pull --ff-only` as the checkout owner), then runs the pulled legacy script. OpenRestream's--fallback runs `release` then `api` for `deploy`.----Exit codes are ovhost's: `0` ok · `1` usage/precondition · `2` validation failed, nothing restarted · `3` not--ready, rolled back and serving · `4` rollback failed, **manual intervention** · `5` protected sessions active--· `6` frozen.----**Live `--force`.** The old script's `--force` meant "discard local tracked changes" (legacy in-place layout--only). ovhost never discards changes, and its `--force` drops live streams and passes a freeze. The Live--wrapper therefore refuses `--force`; run `sudo ovhost deploy live --force` yourself when that is what you--mean. (In the fallback, `--force` goes to the legacy script as before.)++Repository wrappers call `ovhost plan`, `ovhost deploy`, or `ovhost rollback` for their service. If ovhost is missing, the wrapper exits with an error. The `capabilities` command remains available for installed wrappers that still query it.## The two engines@@@ -117,7 +81,7 @@ overrides them.| Field | Meaning | Default ||---|---|---|--| `strategy` | `git-checkout`, `multi-app`, `static-build`, `pnpm-build` or `release-layout` | `git-checkout`; an entry with `layout: "release"` and no strategy stays unmanaged (deployed by its own script), as before |++| `strategy` | `git-checkout`, `multi-app`, `static-build`, `pnpm-build` or `release-layout` | `git-checkout`; `layout: "release"` requires `release-layout` || `release.git` | the clone releases are made from (relative to `repo`, or absolute inside it) | `repo` || `release.releases`, `release.current` | the releases directory and the link | `releases`, `current` || `release.id` | `time-sha8` (`<UTC yyyymmdd-HHMMSS>-<sha8>`, Live; the old script used the host's local time, the same on a UTC host) or `sha12` (OpenRestream; one release per commit, reused) | `time-sha8` |@@@ -190,129 +154,13 @@ log and a freeze refuses.operator's environment): `sudo` does not pass `ADMIN_TOKEN`, so it said "chat notification skipped" inpractice, and Live posts its own deploy notice when it boots on new commits (`server/chat/deploy-notice.js`,through Events).--- **Live's in-place (legacy) layout branch** of `deploy.sh`: production runs the release layout since-- 2026-09-24; an in-place Live would be `strategy: git-checkout`.- **Live `--force`** (discard local tracked changes): ovhost never discards changes (see above).- **Network's "restart anyway" when nothing is new**: only with `--restart` now.- **OpenRestream `workers`, `status`, `prune`**: they stay in the repository's script. Starting a workergeneration is not a deploy ovhost makes (it never starts, stops or restarts a worker unit); ovhost pruneson its own and keeps every release a worker runs from.- **Games `sudo git`**: ovhost runs git as the checkout owner. If earlier `sudo git pull`s left root-owned-- objects in `.git`, fix the ownership once (checklist below).----## Production cutover checklist (for the operator)----Run on 2026-09-27 (the cutover); kept as the record of what was done and as the procedure for a new host. Each step is--read-only until step 5.----1. **Update ovhost.** `cd /usr/local/lib/openvibe-host && sudo git pull --ff-only && sudo npm ci --omit=dev-- --no-audit --no-fund`, then `ovhost --version` (0.3.0) and `sudo ovhost capabilities` (`deploy-api=1`).-- Until step 3 every wrapper still falls back (the inventory has no strategies).--2. **Check the checkouts.**-- - `sudo find /opt/openvibe.games/.git ! -user ubuntu | head`: if anything is listed,-- `sudo chown -R ubuntu:ubuntu /opt/openvibe.games/.git` (earlier `sudo git` pulls).-- - `sudo -u ubuntu bash -lc 'command -v pnpm && pnpm --version'`: ovhost runs pnpm as ubuntu with the-- PATH it was started with; if pnpm lives elsewhere, set `install.command`/`build` to its absolute path.-- - `ls -ld /opt/openvibe.live/repo /opt/openre.stream/repo` (root-owned clones, `owner: root`).-- - `sudo ss -Hltnp 'sport = :3000'` shows `"systemd",pid=1` (else the first Live deploy rebinds the socket).--3. **Edit the inventory.** `sudo cp /etc/openvibe/host.json /etc/openvibe/host.json.pre-ws-n-11`, apply the-- [proposed diff](#proposed-production-inventory-diff), then `sudo ovhost show live openre tools sites games-- network` and `sudo ovhost validate <svc>` for each (no new errors).--4. **Compare each plan with the old script** (`sudo ovhost plan <svc>` only fetches):-- - live: `sudo ovhost plan live` against `cd /opt/openvibe.live/current && sudo DRY_RUN=1 bash-- deploy/scripts/deploy-legacy.sh` (the old script; `deploy.sh` before the wrapper is deployed): same target sha and change classes (restart yes/no),-- `node_modules` hard-linked vs `npm ci` the same way, `socket held by systemd on :3000`,-- `strategy release-layout (release layout, keep 5)`.-- - openre: `sudo ovhost plan openre` against `sudo bash /opt/openre.stream/repo/deploy/scripts/deploy.sh-- status`: current release = the `current` link, restart = `openre-api.service,-- openre-session-coordinator.service` only, ingest sessions counted.-- - tools: `sudo ovhost plan tools`: the installs match the apps whose `package.json`/lockfile changed-- (`git -C /opt/openvibe.tools diff --name-only HEAD origin/main -- 'apps/*/package*.json'`), no-- `apps/_shared` line, the restart lists every `systemctl list-unit-files 'openvibe-tools*'` unit (no-- "not in the inventory" warning), the preflight lists the jobs and guard checks.-- - sites: `sudo ovhost plan sites --restart`: build output restored, `npm ci`, `node build.js`, repo vhosts,-- one notification per placeholder, no tracked changes.-- - games: `sudo ovhost plan games`: tracked changes none (dist-types restored), `install (always)`, build,-- preflight better-sqlite3, restart `openvibe-games.service`, players online counted (drain report).-- - network: `sudo ovhost plan network`: nothing unexpected (Network already deploys with ovhost; the-- entry only gains `installUnits`).--5. **Pull the wrappers.** Tools, Sites, Network and Games get theirs with their next pull (the old Tools and-- Network scripts pull themselves, so the deploy that brings the wrapper is still run by the old script).-- Live's arrives in the release that contains it; OpenRestream's with `git -C /opt/openre.stream/repo pull`.--6. **Deploy through the wrapper**, one service at a time, lowest risk first: network, sites, games (quiet-- hour or `--wait-idle`), tools, openre (no ingest session, or `--wait-idle`), live (`--wait-idle`). Each-- wrapper prints `ovhost deploy <svc> …`; if it prints "running deploy-legacy.sh", read why.--7. **Verify.** `sudo ovhost releases <svc>` (strategy, from/to, result), `sudo ovhost status`, and for Live-- `sudo ss -Hltnp 'sport = :3000'` still shows `"systemd",pid=1`.--8. **Undo** (any step): `OVHOST_LEGACY=1 sudo deploy/scripts/deploy.sh` runs the old script for one deploy;-- restoring `/etc/openvibe/host.json.pre-ws-n-11` (or removing a `strategy`) makes the wrapper fall back-- for good.----## Proposed production inventory diff----`/etc/openvibe/host.json` is not changed by this work. These are the fields each entry needs (merge them--into the existing entries; `host.example.json` in this repository has the full entries). JSON has no--comments: the `//` notes below are for reading only.----```jsonc--"live": {-- "repo": "/opt/openvibe.live", // the release root (".../current" is read the same way)-- "owner": "root", // git runs as root in the root-owned clone-- "runAs": "ubuntu",-- "strategy": "release-layout",-- "release": { "git": "repo", "id": "time-sha8", "links": { "data": "../../shared/data" }, "keep": 5 },-- // MUST be the release-layout units: the in-place ones (deploy/systemd/*.service) run from-- // /opt/openvibe.live, and installing them would point Live at the stale checkout.-- "unitSources": {-- "openvibe-live.service": "deploy/systemd/release/openvibe-live.service",-- "openvibe-live.socket": "deploy/systemd/release/openvibe-live.socket",-- "openvibe-live.service.d/socket.conf": "deploy/systemd/release/openvibe-live.service.d/socket.conf"-- },-- "ready": { "url": "http://127.0.0.1:3000/api/ready", "timeoutSeconds": 90, "release": true },-- "preflight": { "syntaxCheck": true }-- // unchanged: units, socketUnit, port, noRestartPaths, protected, drain (refuse, 28800 s), databases,-- // backupOnChange, nginx (repoVhost is never installed by a deploy; the plan names a change)--},--"openre": {-- "repo": "/opt/openre.stream", // the release root; the clone is /opt/openre.stream/repo-- "strategy": "release-layout", // replaces "layout": "release"-- "owner": "root",-- "runAs": "ubuntu",-- "release": { "git": "repo", "id": "sha12", "reuseModules": false, "chown": "ubuntu:ubuntu", "keep": 5 },-- "ready": { "url": "http://127.0.0.1:4500/api/ready", "timeoutSeconds": 60 }-- // unchanged: units (API + coordinator), workerUnits, protected (ingest sessions), drain--},--"tools": {-- "strategy": "multi-app", // skipPackages apps/_*, removeUntrackedLockfiles-- "unitsMatch": "openvibe-tools*.service",-- "ready": { "url": "http://127.0.0.1:4001/api/ready", "headers": { "Host": "openvibe.tools" },-- "timeoutSeconds": 60, "release": true, "allUnits": true },-- "preflight": {-- "dirs": ["data"],-- "checks": [-- { "label": "jobs runtime loads", "packages": ["apps/img", "apps/audio", "apps/docs"],-- "argv": ["node", "-e", "const D=require('better-sqlite3'); new D(':memory:').close(); require('openvibe-contracts'); require('openvibe-sdk'); require('../_shared/jobs')"] },-- { "label": "guard loads", "packages": "*",-- "argv": ["node", "-e", "const D=require('better-sqlite3'); new D(':memory:').close(); require('../_shared/guard')"] }-- ]-- }--},--"sites": {-- "strategy": "static-build" // npm ci, node build.js, generated dist/, repo vhosts-- // behind nginx -t, one announce per dist/*/release.json--},--"games": {-- // remove "managed": false and "unmanagedReason"-- "strategy": "pnpm-build",-- "generated": ["apps/client/dist-types/", "apps/server/dist-types/"],-- "preflight": { "checks": [ { "label": "better-sqlite3 loads under this Node", "packages": ["apps/server"],-- "argv": ["node", "-e", "new (require('better-sqlite3'))(':memory:').close()"] } ] }-- // unchanged: units, ready, protected (players online), drain (report), databases, drill--},--"network": {-- "installUnits": true // the unit file is installed whenever it differs--}--```++ objects in `.git`, fix the ownership once.## The deploy controller: `ovhost reconcile` (roadmap WS-X11 phase 1, D72)@@@ -258,7 +258,7 @@ Each launch-rule item, checked against the code at this commit:| 1 | Owning runtime with health/readiness and observability | `server/index.js` + `deploy/systemd/openvibe-host.service`. `GET /api/health`, `/api/ready` (`server/observability.js`: required `db`, `storage`; optional `disk_headroom`, `network_jwks`, `events_relay`, `domain_checks`), `/release.json`, `/metrics` (loopback only; HTTP golden signals, process, release, `host_sites`). Running on loopback `:4910` with ready 200 (audit, 2026-09-23). `host-secrets-dashboard` "machine endpoints". | **met** || 2 | Canonical identity (Network subjects, scoped principals) | Network SSO OAuth client `host` (Network `server/db/database.js:516`); users are `usr_` subjects from Network JWTs verified against the JWKS (`server/auth/viewer.js`); services present client-credentials tokens for audience `openvibe.host` with one capability per route; `X-OV-Subject` for first-party services only; app principals act as themselves; sandbox tokens refused on production. Principal `svc:host` has `events.event.publish` (Network `server/identity/principals.js:100`). `host-quota-auth` tests. Projects are Host's own until Network has projects (ADR-014, documented). | **met** || 3 | SSR/static public routes useful without JS | Every dashboard page is server-rendered; every action is a plain form, including folder upload. Shared chrome with `<noscript>` nav and SSR footer. Legal pages. Tenant sites are static by definition. `host-secrets-dashboard` "signed-out home page … useful without JavaScript". | **met** |--| 4 | Real persistence and end-to-end workflows | SQLite `host.db` (WAL) plus the content-addressed object store; upload → validate → store → activate → serve → rollback → delete, custom domains through verification and lapse, all over HTTP in the tests (`host-lifecycle`, `host-rollback`, `host-domains`). `host.db` is in `ovhost backup` and the restore drill passed on the host (2026-09-23). Objects are not backed up (accepted for alpha, threat review §6). | **met** (objects backup: follow-up) |++| 4 | Real persistence and end-to-end workflows | PostgreSQL (`ov_host`) plus the content-addressed object store; upload → validate → store → activate → serve → rollback → delete, custom domains through verification and lapse, all over HTTP in the tests (`host-lifecycle`, `host-rollback`, `host-domains`). The database is covered by pgBackRest and logical dumps; object directories are archived by `ovhost backup`. | **met** || 5 | Capability and event registration against Contracts | `host.site.manage`, `host.deploy.create` and `host.domain.manage` and the `host` service manifest (with all four `host.*` events in `eventsProduced`) are released in `openvibe-contracts` **v0.24.0**, which Host pins, and are identical in **v0.30.1** (v0.83.0 adds `host.site.config`). `npx openvibe-contracts-check --service host` passes, and CI runs it as a blocking step. No `host.*` payload schemas: optional, most services have none. | **met** || 6a | Migration/seed strategy | Bootstrap, not migration: Host replaces no legacy data (roadmap §8 register). The schema is created idempotently at every boot (`server/db.js`, `CREATE … IF NOT EXISTS`); `host_takedowns` is additive and appears on the next restart. There is no seed: a new database is an empty, valid Host. The pointer and data survive a restart (`host-rollback`). A future non-additive change needs a migration ledger, which Host does not have yet. | **met** || 6b | Security/threat review | [threat-review.md](threat-review.md): tenant isolation, uploads, custom domains, cookies, the PSL decision, quotas, abuse and takedown, Host's own surface. Seven gaps were fixed in `1e48cb2` and `0ff29d5`: takedowns, the upload concurrency cap, the CA-validation path refusal, the CDN cache cap, the realip templates, the disk floor with the owner project cap, and Clear-Site-Data. | **met** |@@@ -88,7 +88,7 @@ against the 2 s the deploy took to report ready).| Gate | What | Evidence | Budget | Where the number comes from ||---|---|---|---|---|| 1a | a styles-only release is switched without restarting Live, streams live | Host `test/strategy-release-layout.test.js` | Live restarts (dropped streams) ≤ 0 | asserted: the test asserts host.restarts() is [] and the record says restarted: false |--| 1b | the same through Live's own deploy script on a simulated host | Live `test/deploy-sim.test.js` | Live restarts ≤ 0 | asserted: the test compares the process id before and after the switch |++| 1b | the same through Live's deploy wrapper on a simulated host | Live `test/deploy-sim.test.js` | Live restarts ≤ 0 | asserted: the test compares the process id before and after the switch || 1c | a tab broadcasting (live camera/mic) or playing a stream is never reloaded | Shared `test/release.test.js` | reloads while capturing or playing ≤ 0 | asserted: asserts [reloads, metrics] = [0, { deferred: { capture: 1 } }] with a live track, and the same for a playing <video> || 1d | a style change is applied in place: only the changed stylesheet moves | Shared `test/release-update.test.js` | reloads + prompts for a styles-only release ≤ 0 | asserted: asserts [p.reloads, p.toasts.length] = [0, 0] after the update || 1e | Live's manifest declares style components, so its open tabs take styles in place | `<base>/release.json` (--base) | style components in /release.json ≥ 1 | GET <base>/release.json |@@@ -28,14 +28,7 @@ visibility `internal`, its own payload). Browsers never receive that one.- A `static-build` service with `announce.releaseFiles` (Sites) sends one notification per`dist/<domain>/release.json`, with that placeholder's service, release and `https://<domain>` origin,and stops at the first failure.--- Since WS-N task 11 each repository's `deploy/scripts/deploy.sh` is a wrapper around `ovhost deploy <svc>`-- ([deploy-strategies.md](deploy-strategies.md)), so the deploy announces. Their fallback scripts-- (`deploy/scripts/deploy-legacy.sh`) still call `ovhost announce <service>` at the end:-- - Live, Tools, Games: `ovhost announce live|tools|games`.-- - Sites: one call per placeholder with `--release` and `--origin`.---- They do nothing if `ovhost` is missing or has no `announce`, and a failed announcement never fails the-- deploy.++- Repository deploy wrappers call `ovhost deploy <service>`, which announces a deployed release.`ovhost announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run] [--events-env <file>] [--json]`@@@ -106,7 +99,7 @@ All commands run on the production host. None of them prints a secret.the Network admin, `GET https://openvibe.network/api/admin/grants?client=host`, where it shows asactive. Or check the database directly:```-- cd /opt/openvibe.network && sudo sqlite3 data/network.db "SELECT capability, audience, revoked_at FROM principal_grants WHERE client_id='host'"++ sudo -u postgres psql ov_network -c "SELECT capability, audience, revoked_at FROM principal_grants WHERE client_id='host'"```Expect `events.event.publish|openvibe.events|` (not revoked). If it is missing, grant it in theadmin: `POST /api/admin/grants { client_id: "host", capability: "events.event.publish", reason: "release notifications (WS-P 9)" }`.@@@ -4,35 +4,7 @@ A drill restores a service's database from its latest `ovhost backup`, starts aservice on a spare loopback port against the restored copy, and compares its answers with production.The production instance is never touched.--| Date (UTC) | Service | Backup | Checks | Result |--|---|---|---|---|---|--| 2026-09-23 03:44 | community | `/var/backups/openvibe/community/20260923-034341/` (`ovhost backup community`) | `pragma integrity_check` = ok; restored instance `/api/ready` 200; `/api/pastes?limit=5`, `/api/v1/spaces`, `/api/v1/pulse?limit=5` byte-identical to production; pastes 892 = 892 | passed |--| 2026-09-23 05:22 | community | `/var/backups/openvibe/community/20260923-052231/` (`ovhost drill community`) | `pragma integrity_check` (community) = ok; drill instance `/api/ready` 200 after 1s; `/api/pastes?limit=5`, `/api/v1/spaces`, `/api/v1/pulse?limit=5` identical to production; pastes 892 = 892; spaces 3 = 3; threads 0 = 0; posts 0 = 0 | passed |--| 2026-09-23 05:22 | network | `/var/backups/openvibe/network/20260923-052241/` (`ovhost drill network`) | `pragma integrity_check` (network) = ok; drill instance `/api/ready` 200 after 1s; `/api/health`, `/api/.well-known/jwks`, `/api/brand`, `/api/themes?limit=500`, `/api/domains`, `/api/v1/registry/capabilities` identical to production; users 393 = 393; oauth_clients 22 = 22; wallets 30 = 30; coin_transactions 34 = 34 | passed |--| 2026-09-23 05:22 | events | `/var/backups/openvibe/events/20260923-052251/` (`ovhost drill events`) | `pragma integrity_check` (events) = ok; drill instance `/api/ready` 200 after 1s; events 69 = 69; subscriptions 8 = 8; deliveries 20 = 20; consumer_checkpoints 0 = 0 | passed |--| 2026-09-23 05:22 | billing | `/var/backups/openvibe/billing/20260923-052254/` (`ovhost drill billing`) | `pragma integrity_check` (billing) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/rates` identical to production; accounts 22 = 22; transactions 322 = 322; ledger_entries 996 = 996; subscriptions 0 = 0; payment_intents 5 = 5 | passed |--| 2026-09-23 05:22 | chat | `/var/backups/openvibe/chat/20260923-052257/` (`ovhost drill chat`) | `pragma integrity_check` (chat) = ok; drill instance `/ready` 200 after 1s; `/api/chat/filters/friendly` identical to production; dm_conversations 11 = 11; dm_messages 1522 = 1522; emotes 42 = 42; moderation_actions 264 = 264 | passed |--| 2026-09-23 05:23 | search | `/var/backups/openvibe/search/20260923-052300/` (`ovhost drill search`) | `pragma integrity_check` (search) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/search?q=openvibe&limit=10`, `/api/v1/suggest?q=open&limit=10` identical to production; documents 10 = 10; doc_acl 0 = 0; doc_facets 0 = 0 | passed |--| 2026-09-23 05:23 | sources | `/var/backups/openvibe/sources/20260923-052303/` (`ovhost drill sources`) | `pragma integrity_check` (sources) = ok; drill instance `/api/ready` 200 after 1s; sources 6 = 6; items 0 = 0; item_revisions 0 = 0; fetch_runs 0 = 0 | passed |--| 2026-09-23 05:23 | wiki | `/var/backups/openvibe/wiki/20260923-052306/` (`ovhost drill wiki`) | `pragma integrity_check` (wiki) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/spaces`, `/feed.json` identical to production; wiki_spaces 1 = 1; wiki_pages 10 = 10; wiki_page_revisions 10 = 10 | passed |--| 2026-09-23 05:23 | blog | `/var/backups/openvibe/blog/20260923-052309/` (`ovhost drill blog`) | `pragma integrity_check` (blog) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/blogs/openvibe/posts?limit=10`, `/feed.json` identical to production; blogs 1 = 1; blog_posts 1 = 1; blog_post_revisions 1 = 1 | passed |--| 2026-09-23 05:23 | tips | `/var/backups/openvibe/tips/20260923-052312/` (`ovhost drill tips`) | `pragma integrity_check` (tips) = ok; drill instance `/api/ready` 200 after 1s; `/sitemap.xml` identical to production; creator_tip_profiles 0 = 0; tip_interactions 0 = 0; tip_goals 0 = 0 | passed |--| 2026-09-23 05:23 | vip | `/var/backups/openvibe/vip/20260923-052315/` (`ovhost drill vip`) | `pragma integrity_check` (vip) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/creators/network`, `/api/v1/plans?creator=network`, `/api/v1/perks?creator=network` identical to production; vip_creators 1 = 1; vip_plans 0 = 0; vip_memberships 0 = 0 | passed |--| 2026-09-23 05:23 | news | `/var/backups/openvibe/news/20260923-052318/` (`ovhost drill news`) | `pragma integrity_check` (news) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/stories?limit=10`, `/api/v1/topics` identical to production; news_topics 12 = 12; news_stories 0 = 0; news_source_items 0 = 0 | passed |--| 2026-09-23 05:23 | reviews | `/var/backups/openvibe/reviews/20260923-052321/` (`ovhost drill reviews`) | `pragma integrity_check` (reviews) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/entities?limit=10`, `/feed.json` identical to production; review_entities 0 = 0; review_signals 0 = 0; review_summaries 0 = 0 | passed |--| 2026-09-23 05:23 | trade | `/var/backups/openvibe/trade/20260923-052323/` (`ovhost drill trade`) | `pragma integrity_check` (trade) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/instruments?limit=10`, `/feed.json` identical to production; trade_instruments 0 = 0; trade_market_observations 0 = 0; trade_source_documents 0 = 0 | passed |--| 2026-09-23 05:23 | deals | `/var/backups/openvibe/deals/20260923-052326/` (`ovhost drill deals`) | `pragma integrity_check` (deals) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/offers?sort=new&page=1` identical to production (JSON without volatile keys where declared); deal_offers 0 = 0; deal_products 0 = 0; deal_votes 0 = 0 | passed |--| 2026-09-23 05:23 | coupons | `/var/backups/openvibe/coupons/20260923-052329/` (`ovhost drill coupons`) | `pragma integrity_check` (coupons) = ok; drill instance `/api/ready` 200 after 1s; `/sitemaps/merchants.xml`, `/feed.json` identical to production; coupons 0 = 0; coupon_merchants 0 = 0; coupon_merchant_domains 0 = 0 | passed |--| 2026-09-23 05:23 | codes | `/var/backups/openvibe/codes/20260923-052332/` (`ovhost drill codes`) | `pragma integrity_check` (codes) = ok; drill instance `/api/ready` 200 after 1s; `/sitemap.xml` identical to production; manifests 0 = 0; releases 0 = 0; trust 0 = 0 | passed |--| 2026-09-23 05:23 | host | `/var/backups/openvibe/host/20260923-052335/` (`ovhost drill host`) | `pragma integrity_check` (host) = ok; drill instance `/api/ready` 200 after 1s; host_projects 0 = 0; host_sites 0 = 0; host_deploys 0 = 0; host_domains 0 = 0 | passed |--| 2026-09-23 19:24 | openre | `/var/backups/openvibe/openre/20260923-192308/` (`ovhost drill openre`) | `pragma integrity_check` (openre) = ok; drill instance `/api/ready` 200 after 1s; `/api/health`, `/robots.txt` identical to production; stream_definitions 1 = 1; ingest_keys 1 = 1; destinations 0 = 0; migration_map 0 = 0 | passed |--| 2026-09-23 23:07 | live | `/var/backups/openvibe/live/20260923-230748/` (`ovhost drill live`) | `pragma integrity_check` (live) = ok; drill instance `/api/ready` 200 after 1s; `/api/themes`, `/api/emotes/global`, `/api/streams`, `/api/streams/recently-online?limit=20`, `/api/streams/channel/japaneseoldguy/live` identical to production (JSON without volatile keys where declared); users 352 = 352; channels 123 = 123; managed_streams 103 = 103; streams 2477 = 2477; follows 62 = 62; chat_messages 70864 = 70864 | passed |--| 2026-09-23 23:07 | media | `/var/backups/openvibe/media/20260923-230757/` (`ovhost drill media`) | `pragma integrity_check` (media) = ok; drill instance `/api/ready` 200 after 1s; `/browse?tab=videos`, `/browse?tab=clips` identical to production; media_objects 3056 = 3056; vods 906 = 906; clips 315 = 315; apps 7 = 7 | passed |--| 2026-09-24 05:40 | ai | `/var/backups/openvibe/ai/20260924-054015/` (`ovhost drill ai`) | `pragma integrity_check` (ai) = ok; drill instance `/api/ready` 200 after 1s; `/api/health` identical to production; providers 3 = 3; models 9 = 9; routes 19 = 19; templates 26 = 26; workflows 47 = 47; quotas 5 = 5; citations 0 = 0 | passed |--| 2026-09-24 05:40 | games | `/var/backups/openvibe/games/20260924-054054/` (`ovhost drill games`) | `pragma integrity_check` (world) = ok; drill instance `/api/ready` 200 after 2s; `/map.json`, `/api/v1/mods` identical to production; players 28 = 28; mods 0 = 0 | passed |--| 2026-09-24 05:41 | tools | `/var/backups/openvibe/tools/20260924-054102/` (`ovhost drill tools`) | `pragma integrity_check` (tools-docs-analytics) = ok; `pragma integrity_check` (tools-docs-jobs) = ok; drill instance `/api/ready` 200 after 1s; `/release.json` identical to production (JSON without volatile keys where declared); tool_jobs 0 = 0 | passed |--| 2026-09-24 05:54 | live | `/var/backups/openvibe/live/20260924-055447/` (`ovhost drill live`) | `pragma integrity_check` (live) = ok; drill instance `/api/ready` 200 after 1s; `/api/themes`, `/api/emotes/global`, `/api/streams`, `/api/streams/recently-online?limit=20`, `/api/streams/channel/japaneseoldguy/live` identical to production (JSON without volatile keys where declared); users 353 = 353; channels 123 = 123; managed_streams 103 = 103; streams 2477 = 2477; follows 62 = 62; chat_messages 70875 = 70875; first drill in the release layout | passed |--| 2026-09-25 23:00 | sources | `/var/backups/openvibe/sources/20260925-212055/` (`ovhost drill sources`) | `pragma integrity_check` (sources) = ok; drill instance `/api/ready` 200 after 1s; sources 6 = 6; items 0 = 0; item_revisions 0 = 0; fetch_runs 0 = 0; acceptance `/api/ready`, `/api/health` | passed |++The inventory `drill` block maps each database to its `DATABASE_URL` and `DATABASE_DIRECT_URL` setting. A drill selects the newest good logical dump, checks it with `pg_restore --list`, restores it into a scratch database under a temporary role, and starts a sandboxed second instance. It compares declared HTTP paths and row counts, stops the instance, and drops the scratch database and role.## R2 eviction drills (Media's hot cache)@@@ -42,22 +14,6 @@ The production instance is never touched.|---|---|---|---|| 2026-09-25 18:55 | 2178 (3.5 MB, `--pick --max-mb 200`, real `GET /v/2178?raw=1`) | before: served from R2 · evict: R2 copy removed, row → b2 · from B2: served from the canonical copy with the same first MiB · re-warm: R2 copy restored from B2 (HEAD verified), row → r2 · after: served from R2 again; `media_locations` b2 and r2 present. Artifact `/opt/openvibe.media/data/drills/r2-eviction-2178-2026-09-25T18-55-13-212Z.json` | passed |--## How a drill was run (community)----1. `sudo ovhost backup community`--2. Copy `<backup>/community.db` to a temp directory owned by the service user; `pragma integrity_check`.--3. Start `node server/index.js` from the production checkout with the service's env file, overriding-- `PORT` (spare loopback port), `COMMUNITY_DB_PATH` (the copy), and turning off side effects-- (`DISCORD_RELAY_ENABLED=0`, `EVENTS_URL=` empty) so the drill instance never talks to the world.--4. Compare responses of read endpoints with production; compare row counts.--5. Stop the drill instance by its port's pid (not `pkill -f`, which can match the operator's own shell),-- delete the temp directory.----Looking back at that manual run: `EVENTS_URL=` had no effect, because Community has no events relay. The--instance could also still reach production Network over loopback (JWKS, service tokens, identity--resolve-batch). The inventory's `community` drill block now also empties `OV_OAUTH_CLIENT_SECRET` and--points `OV_LIVE_INTERNAL_URL` and `OV_MEDIA_INTERNAL_URL` at a closed port.--## Drilling an off-host copyScheduled backups and their encrypted off-host copies are described in [backups.md](backups.md).@@@ -70,68 +26,15 @@ sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-communitysudo rm -rf /var/lib/openvibe-restore/drill-community```--Write "off-host" and the run id in the Backup column of the table above.++Record the run id and that the copy came from off-host storage.## Running a drill with `ovhost drill`--```--sudo ovhost backup community # a fresh backup keeps the comparison meaningful--sudo ovhost drill community # restore, start, compare, stop, clean up, log--sudo ovhost drill community --keep # keep the drill directory (copy, drill.env, drill.log)--sudo ovhost drill community --backup /var/backups/openvibe/community/20260923-034341--```----The command does the steps above automatically for any service whose inventory entry has a `drill`--block (see `host.example.json`):----1. **Restore.** The command picks the latest `ovhost backup` from `<stateDir>/backups/<service>.jsonl`,-- or the `--backup` directory. It copies every database named in `drill.databases` into-- `/var/lib/openvibe-drills/<service>-<stamp>/db/`, a directory owned by the service user (mode 0700).-- Backups are root-only (directories 0700, files 0600, see [backups.md](backups.md)), so root makes-- the copy with `install -o <service user> -m 0600`, and the copy belongs to the service user.-- `PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;-- otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into a-- scratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).-- Every `objects` directory the service declares is restored as well: its archive is extracted into-- `<tmp>/<name>`, owned by the service user and never the live directory, and every blob is checked-- against the sha256 in its file name.--2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the-- service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production-- unit's non-secret `Environment=` values and the production env file. After that it loads-- `drill.env`, which holds the drill port, the restored database paths and the switches that turn-- side effects off. `drill.env` holds inventory values only and is loaded last, so its values win.-- ovhost never reads the env file's values.--3. **Sandbox.** The instance runs with:-- - `ProtectSystem=strict`, with only the drill directory writable;-- - `PrivateTmp`, `ProtectHome=read-only` and `NoNewPrivileges`;-- - binding allowed only on its drill port (`SocketBindAllow`);-- - no addresses beyond loopback (`IPAddressDeny=any`, `IPAddressAllow=localhost`);-- - `RuntimeMaxSec`, which stops it even if ovhost dies.---- Loopback stays open because services fetch Network's public keys there. Production services-- are therefore still reachable on loopback, and the inventory's overrides are what keep the drill-- away from them.--4. **Compare.** Once the instance answers its readiness path, every `drill.compare` path is fetched-- from production and from the drill instance. Bodies must match byte for byte, or as JSON with the-- listed volatile keys removed. Statuses must match, and production must answer 2xx. Each-- `drill.counts` table is counted in production and in the restored copy.--5. **Stop and record.** The instance's own MainPID gets SIGTERM, then SIGKILL after 20 s. ovhost-- never uses `pkill -f` and never signals a production pid. The drill directory is removed unless-- `--keep` is set, or unless the instance could not be confirmed stopped. The result is appended to-- `/var/lib/openvibe-host/drills/<service>.jsonl`, and a row for the table above is printed.----Exit codes are `0` passed, `1` refused, and `2` failed. A refusal happens when ovhost is not run as--root, the port is in use, there is no backup, or the service's drill is unsupported. A failure is--integrity, readiness, a mismatch, or a stop that did not complete. Every failure is still logged and--cleaned up.----A drill compares the backup with production *now*. If writes landed after the backup, row counts and--list endpoints differ, and the drill reports that honestly. Take the backup right before the drill.++Run `sudo ovhost drill <service>` or pass `--backup <absolute directory>` to select a copy. The service must have a supported drill block and an unused loopback port. PostgreSQL drills require URL mappings for every declared database and a `VALKEY_URL` override so the drill cannot write into production.### A PostgreSQL drill--Step 1 above is the SQLite path. A service whose `databases[]` entry is `engine: postgresql`--([db-inventory.md](db-inventory.md)) restores a **logical dump** instead of a file copy:++A service whose `databases[]` entry is `engine: postgresql` ([db-inventory.md](db-inventory.md)) restores a **logical dump**:1. **Restore.** The drill takes the service's newest `<name>.dump` (`pg_dump -Fc`, from`<stateDir>/backups/<service>.jsonl`, or the `--backup <dir>` directory) and creates a scratch@@@ -157,8 +60,7 @@ Step 1 above is the SQLite path. A service whose `databases[]` entry is `engine:the second instance write locks, caches and queues into production's keyspace.- **`DATABASE_URL` un-overridden would serve production data.** The drilled instance would read andwrite the live database instead of the restored copy, and every comparison would be vacuous.--3. **Compare.** `counts` are read from production and from the scratch database over `psql` (the SQLite-- drills compare files). `countsTolerance` in the drill block (default `0`) is the number of rows that++3. **Compare.** `counts` are read from production and from the scratch database over `psql`. `countsTolerance` in the drill block (default `0`) is the number of rows thatmay differ: production keeps writing between the read that makes the backup and the read that countsit, so a busy table is allowed to drift by that many rows instead of failing the drill.@@@ -176,177 +78,4 @@ The scratch database and its role are dropped in a `finally`, together with stopsuccess and on failure. A PostgreSQL drill always cleans up after itself; `--keep` keeps the drilldirectory (the copy, `drill.env`, `drill.log`) but never the scratch database or role.--For such a service the **Checks** column in the table above reports `pg_restore --list` (that the dump is--a readable archive) rather than `pragma integrity_check`, and each count is read over `psql` from--production and from the scratch database.----### Services----Supported with declared overrides (the table above records which have passed on the host):----- network--- community--- events--- host (the Host API: the `ov_host` database and the tenant object store, whose blobs are sha256-checked; counts only)--- billing--- chat--- search--- sources--- wiki--- blog--- tips--- vip--- news--- reviews--- deals--- coupons--- trade--- codes--- openre (not run yet). Only openre-api starts (`drill.command`), with `OPENRE_DRILL=1`. From OpenRestream-- `5e86ea7` on, that makes it read-only: writes, `/play/` and sign-in answer 503. The coordinator-- and every transport worker refuse to start in a drill, and the event relay and Media calls are-- off. The API never runs a transport or the relay in any case. Empty `EVENTS_URL` and-- `OV_OAUTH_CLIENT_SECRET`, plus `MEDIA_URL` on the closed port, cover older releases. The workers-- stay out of every ovhost command: they are `workerUnits`, which are listed and never restarted.-- Run it after the OpenRestream release with `OPENRE_DRILL` is deployed (OpenRestream `docs/cutover.md`-- phase C).--- live (not run yet). Needs OpenVibe.Live `8a58aea` or later; `drill.requires` refuses an older-- checkout. The production unit's ExecStart starts with `LIVE_DRILL=1`, `DB_PATH` on the copy and-- `DATA_DIR={tmp}/data`. In that mode (Live `server/drill.js`, `docs/deploy.md#restore-drills`) Live-- refuses to start unless `DB_PATH` and `DATA_DIR` are outside the checkout and `/opt/openvibe.live`,-- `HOST` is loopback, `PORT` is not 3000 and no socket was handed over by systemd. It writes only-- under `DATA_DIR` (a fresh `analytics.db`, empty data directories; the per-location `*_PATH` values-- in `live.env` are ignored) and the copy. It starts only its HTTP server:-- - no RTMP, SRT, WHIP, SFU or JSMPEG listener, and no TURN credential;-- - no WebSocket: upgrades get 403;-- - no jobs loop, restream or relay resume, AI job, Media reconciler, Events outbox, chat bridge,-- identity sync, deploy notice, star job or registry refresh.---- Every method but GET, HEAD and OPTIONS answers 403. Outbound connections, `fetch`, programs other-- than `git`, UDP sockets and other listeners are refused in-process, so Media, Community and-- Network look down to it. `rs-companion.db` is not restored: the server never opens it. Compared:-- - `/api/themes` and `/api/emotes/global`, byte for byte;-- - `/api/streams` and `/api/streams/channel/japaneseoldguy/live`, without the values production-- changes while someone is live (viewer counts, heartbeats, live thumbnails, recording state);-- - `/api/streams/recently-online?limit=20`, without `vod_thumbnail`, the one value it asks Media for.---- Counts: `users`, `channels`, `managed_streams`, `streams`, `follows`, `chat_messages`. Not-- compared: `/release.json`, which production computes at its own boot, so a `public/`-only deploy-- without a restart makes it differ; and the home statistics, which are cached for 30 s and windowed-- by the clock. Any existing channel can replace `japaneseoldguy`.--- media (not run yet). Needs OpenVibe.Media `6e74eb0` or later (`drill.requires`). `MEDIA_DRILL=1`-- with `DB_PATH` on the copy. Media then refuses to start unless `DB_PATH` is outside the checkout-- and `/opt/openvibe.media`, `HOST` is loopback and `PORT` is not 4100. It starts only its HTTP-- server:-- - no app seeding or JWKS refresh;-- - no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant-- scans, disk guardian, thumbnail cleanup, object purge, backfill or orphan-recording finalize;-- - no Events relay and no webhook (the `apps` table's URLs are production's).---- It writes no file and creates no directory. It never opens a file path from the database: every-- byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`, `/o`, paste screenshots, live frames) answers-- 503, while watch pages and `/browse` render from the copy. Every method but GET, HEAD and OPTIONS-- answers 403 and `/auth/*` 503. Connections, `fetch`, programs other than `git` (no ffmpeg or-- ffprobe), UDP and other listeners are refused in-process. `/api/ready` drops the storage and-- remote-tier checks and reports `"mode": "drill"`. The storage paths from `media.env` are moved-- under `{tmp}/storage`, which is never created, as a second switch. `THUMBNAILS_PATH` stays-- production's, read-only in the sandbox, because `/browse` counts that directory in its tab bar;-- the drill lists it and serves none of it. Compared byte for byte: `/browse?tab=videos` and-- `/browse?tab=clips`. Counts: `media_objects`, `vods`, `clips`, `apps`. Production caches the tab-- counts for 60 s, so take the backup right before the drill.----Services that are not deployed yet (everything except network, live, media, tools, community and--games as of 22 Sep) have entries built from their repositories' `deploy/` directories.----- ai (passed 2026-09-24 05:40 UTC): AI starts with AI_ENABLED=0 and every provider key, base URL-- and the HTTP seam blanked, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and-- EVENTS_URL are blanked (no service calls, no events) and WHISPER_BIN too (no transcription). Boot-- marks interrupted runs failed in the restored copy only. `/api/health` is compared; the counts cover-- configuration and citations, because runs, requests and usage grow by the minute (integrity_check-- covers them).--- tools (passed 2026-09-24 05:41 UTC): the **docs** app alone. Tools runs eight units; `drill.unit` picks-- `openvibe-tools-docs.service`, whose ExecStart, WorkingDirectory and Environment= the drill copies,-- and `drill.productionPort` (4016) is what the comparison reads, not the gateway's 4001. docs takes a-- data DIRECTORY (`DATA_DIR`), so both restored copies land in `{tmp}/data/` under their production-- names (`analytics.db`, `jobs.db`). Existing Tools switches make it side-effect-free:-- `TOOLS_JOB_RESULTS=local` (the job pruner would otherwise delete production Media objects for-- expired jobs in the copy), `EVENTS_PUBLISH=off` and an empty `EVENTS_URL` (the copy's pending outbox-- rows would be republished), `TOOLS_JOBS_CONCURRENCY=0` (no job runs), `UPLOADS_DIR`/`OUTPUT_DIR`-- under `{tmp}`, an empty `OV_OAUTH_CLIENT_SECRET`. Job recovery and pruning, analytics aggregation-- and upload retention still run, on the copy only. `drill.requires` refuses the drill unless the-- deployed checkout has those switches. Compares `/release.json`; counts `tool_jobs`.--- games (passed 2026-09-24 05:40 UTC): Reviewed against OpenVibe.Games `7863fc6`. One HTTP listener; `/ws` and-- `/editor-ws` are upgrades on it. tsx runs `src/main.ts` directly, with no build. The unit sets-- `DB_PATH` with `Environment=`, and that value never reaches the drill because the drill sets-- `DB_PATH` itself. `EVENTS_PUBLISH=off` stops the outbox relay, `MEDIA_MIRROR=off` stops the-- map-asset mirror, and an empty `OV_OAUTH_CLIENT_SECRET` stops every service call. The 30 Hz tick-- loop keeps running because readiness needs recent ticks, and it flushes the world into the copy-- only. `map.json` stays production's file, which is read-only in the sandbox. `drill.requires`-- checks that `apps/server/src/config.ts` has both switches. The drill compares `/map.json` and-- `/api/v1/mods` and counts `players` and `mods`, which the simulation does not write. There is no-- Games backup yet, so run `sudo ovhost backup games` first.----### What the drill block can declare for multi-process and checkout-relative services----- `unit`: which of the service's units to start (its ExecStart, WorkingDirectory and non-secret-- Environment=). It is required when there are several units and no `command`.--- `productionPort`: the production port the compare paths are read from. The default is the service's-- port.--- `databases.<name>: { "env": "DATA_DIR", "dir": true }`, or `"dir": "{tmp}/<name>"`: the copy keeps-- its production file name inside that directory, and the env var, if one is given, points at the-- directory. For a `postgresql` database the entry is instead `{ "url": "<NAME>", "directUrl":-- "<NAME>" }`: the two env vars that point the drilled service at a database URL, which the drill sets-- to the scratch database (see [A PostgreSQL drill](#a-postgresql-drill)).--- `bind: [{ "from": "{tmp}/…", "to": "<path inside the checkout>" }]`: `BindPaths=` in the drill-- unit's own mount namespace. Use it for files a service opens relative to its checkout with no env-- override: the restored copy appears at the production path for the drill instance only. Tools yt,-- maps and food open `apps/<app>/data/analytics.db` this way. Drill one of them with-- `unit: openvibe-tools-<app>.service`, a `databases` entry with `"dir": "{tmp}/app-data"`, and-- `bind: [{ "from": "{tmp}/app-data", "to": "/opt/openvibe.tools/apps/<app>/data" }]`. For yt, also-- set `YT_PROXY=`. Before first use, check on the host that `BindPaths=` works together with-- `ProtectSystem=strict` in a `systemd-run` transient unit.--- `requires: [{ "file": "<path in the checkout>", "contains": "<text>" }]`: the drill refuses before-- it creates anything unless the deployed checkout has every switch its overrides rely on. An older-- release would ignore the override and run its side effects.----### What Tools and Games should add (product side, not Host)----These make the drills stricter and quieter. Neither is needed for the drills above to be--side-effect-free.----- **Tools `TOOLS_DRILL=1`:**-- - `_shared/jobs/system.js:576-585`: skip `recover()`, `prune()`, the prune interval and `kick()`.-- - `_shared/jobs/index.js:83-105`: force local results and no outbox.-- - `_shared/analytics/tracker.js:81,105-118`: no startup `DELETE` of rate tracking, no-- flush/aggregate/prune timers, and a no-op middleware.-- - The `retention.startCleanup()` calls (img `:273`, docs `:323`, audio `:307`), and yt's-- `startCleanup()` and `startUpstreamProbe()` (`:280-281`, which spawns yt-dlp against YouTube).-- - `_shared/observe.js:95-105`: no `writableDir` probe writes.-- - A `DATA_DIR` override in yt `index.js:26`, maps `:25` and food `:20`, so those apps no longer-- need `bind`.---- With these, the restored copies stay byte-stable during a drill, and row counts other than-- `tool_jobs` become comparable.--- **Games `GAMES_DRILL=1`:**-- - `config.ts:60-99`: force the client secret, OAuth, `networkAuthUrl`, `eventsUrl` and `mediaUrl`-- to null.-- - `config.ts:77-79`: refuse to start unless `HOST` is loopback and `DB_PATH` is not the production-- path.-- - `main.ts:252`: refuse `/ws` and `/editor-ws` upgrades.-- - Return 403 for map, map-asset and mod writes (`httpServer.ts:161,231` and the `routes.ts` write-- branches), and for `/auth/*` (`httpServer.ts:333`).-- - Optionally skip the periodic flush (`gameServer.ts:2320,2502`) and the prune timer-- (`main.ts:148`), so world tables can be counted too.----Marked `drill: { supported: false }`, with the reason in the inventory:----- **ai:** a drill must start AI with every provider key blanked, or queued and new runs would call-- paid providers. Backups work.----Until 23 Sep 2026 live and media were unsupported too:--- **live:** `data/analytics.db` was opened relative to the checkout at boot. RTMP always bound.-- Restreams, relays, AI jobs and Media reconcilers resumed from rows in the copy.--- **media:** webhooks to Live took their URL from the `apps` table. The health job, junk sweep and-- clip jobs ran ffmpeg against production files through absolute paths stored in the database.----`LIVE_DRILL` and `MEDIA_DRILL` (above) replace those reasons.++Each database check uses `pg_restore --list`, and row counts are read with `psql` from production and the scratch database.ddeleted file mode 100644@@@ -1,172 +0,0 @@--# Storage state (roadmap Wave 22)----This page covers two Wave 22 exit criteria:--- "No service still queries SQLite for canonical data unless that is its documented and reviewed state."--- "Each service's /ready tells the truth about its actual backing store."----It has one row per deployed service. Each row gives the service's backing store, whether that store is--canonical, its status under ADR-007, what its readiness handler actually checks (read from the handler,--not from docs), and its backup and drill status from [restore-drills.md](restore-drills.md).----ADR-007 (`OpenVibe.Contracts/docs/adr/ADR-007-data-ownership.md`) decides:--- "SQLite remains the documented store per service while the platform is single-host; each service's-- `/ready` reports its actual store."--- A service moves to PostgreSQL individually when it needs concurrent writers across processes, a second-- host, or sizes SQLite handles poorly.--- Money paths use SERIALIZABLE transactions and row locks when they move.----As reviewed on 2026-09-23 every service ran on SQLite (better-sqlite3), with one database per service.--Since 2026-09-28 nineteen of them run on the host's **PostgreSQL cluster** (ADR-035), and five stay on--SQLite: network, chat, tools, games and live. The per-service table below is the Wave 22 review of the--SQLite state; where a service switched, its SQLite file is now frozen read-only for the 7-day rollback--window (compatibility-register.md, C-89), and the live rows are in the cluster. **This page remains the--Wave 22 review of that state.**----## Where the data lives now----- **PostgreSQL services** (tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes,-- vip, ai, host, events, community, billing, media, openre) keep their rows in the cluster on the data-- role (`roles/data/`, ADR-035): PostgreSQL 18 on loopback, one database per service named `ov_<name>`-- (`ov_trade`, `ov_media`, …). The service reaches it through **PgBouncer** on `127.0.0.1:6432`, which-- fronts the cluster for the app roles (and carries the app role's SCRAM password). Their durability is-- **pgBackRest**, off-host: a full cluster backup weekly, a differential nightly and the WAL stream, which-- the nightly `ovhost backup` verifies rather than copies ([backups.md](backups.md#postgresql-services)).--- **What stays on the local filesystem** for those services is the rest of the data: object and media-- files under `*_PATH` and the object stores (Media's B2/R2), blobs (`HOST_STORAGE_DIR`), uploads and-- outputs, plus the frozen pre-switch SQLite file kept read-only for rollback. Those files are not in the-- cluster and are backed up (or not) exactly as before.--- **SQLite services** are unchanged: one file per service, copied by the online backup API as the service-- user ([backups.md](backups.md)).----`openvibe-shared/ready` is still what each service's `/ready` reports ([below](#per-service)); a switched--service's `db` check reads the cluster as its `DATABASE_URL`/`DATABASE_DIRECT_URL` do.----Sources:--- Each repository's `server/config.js`, `.env.example` and `deploy/systemd/*.service`.--- The ovhost inventory example [`host.example.json`](../host.example.json), which lists production paths.--- The operator notes of 2026-09-23, for what is deployed.--- Live was read at `~/orca/workspaces/OpenVibe.Live/seadragon` (`ae07553`).----`openvibe-shared/ready` (v1.3.0, `node_modules/openvibe-shared/ready.js:43-124`) behaves like this:--- It has no checks of its own.--- A service passes named checks, and each check is required unless it is marked optional.--- It answers 503 only when a required check fails.--- A failed optional check gives 200 with `status: "degraded"`.--- The default timeout is 2 s.----## Findings----1. **Readiness that does not check the store.**-- - *(Fixed 2026-09-23, Games `f11e21c`: `/api/ready` checks `world.db` and the tick; host-only.)* **Games** had no `/api/ready`. `/healthz` returns `{ ok: true, tick }` without touching `world.db`-- (`apps/server/src/net/httpServer.ts:596-600`).-- - *(Fixed 2026-09-23, Chat `45f2b1f`: required `db` read of `chat_messages`, optional `live_sync` degraded after 60 s without a clean pass.)* **Chat** `/ready` was 200 as soon as `ctx.stats.lastSyncAt` was set (`server/app.js:77-86`). That-- timestamp is set even when every Live sync step fails, because `tick()` and `sync()` catch each-- step's error and set it anyway (`server/live-context.js:665-683`). The only database access is the-- `mirror.pending` count (`SELECT COUNT(*) FROM live_mirror_outbox`, `server/bridge/live-mirror.js:34`,-- production runs `LIVE_MIRROR=1`). A broken `chat.db` would therefore surface as a thrown 500, not as a-- named check.--2. **Readiness that checks the main store but not every store.**-- - **Live** runs `SELECT 1` on `live.db`. It does not check `data/analytics.db` or `data/rs-companion.db`.-- - The **Tools** satellites maps, text, yt and food only check `analytics.db`, and only as an optional-- check. By design, that is their only database, and the tools work without it.-- - The **Tools** gateway reports each satellite as an optional upstream. So a failed `jobs.db` on img,-- audio or docs makes the gateway *degraded* (still 200), while that satellite's own `/api/ready`-- answers 503.--3. **ovhost's post-deploy readiness URL is not the truthful endpoint for four services.**-- `host.example.json` polls:-- - Network `/api/health`, which is static (`server/index.js:392-394`), although Network's `/api/ready`-- checks the database and the signing key;-- - Media `/healthz`, which is liveness only, although Media's `/api/ready` checks the database and-- storage;-- - Tools gateway `/api/health`;-- - Games `/healthz` (the inventory now uses `/api/ready`, 2026-09-23; Network, Media and Tools too).---- The inventory should point at `/api/ready` where one exists. Games has none.--4. *(Fixed: the inventory has had AI and OpenRestream entries since `f0dbd0c`, and both are in the scheduled-- `ovhost backup --all` with off-host copies, [backups.md](backups.md). Their drills are still unsupported.)*-- **Two services have no ovhost inventory entry, so no `ovhost backup` and no drill**: AI (`ai.db`) and-- OpenRestream (`openre.db`). Whether they are backed up any other way is unknown. Check the host's cron-- and timers.--5. **The inventory's Tools backup list is incomplete.** It names the seven `analytics.db` files only.-- The job databases `apps/{img,audio,docs}/data/jobs.db` (table `tool_jobs`,-- `apps/_shared/jobs/index.js:77`) are missing. Tools W11 jobs have been in production since-- 2026-09-23.--6. **SQLite reads outside the documented state.** These go against exit criterion 1, and each has an entry-- in [compatibility-register.md](compatibility-register.md):-- - **Live reads its frozen legacy `vods`/`pastes` tables for live pages**, although Media and Community-- hold the canonical rows (C-73):-- - `server/ai/chat-ai-routes.js:337`-- - `server/admin/routes.js:91-92`-- - `server/arena/arena-service.js:436`-- - `server/streaming/routes.js:1547`-- - **Network may open Live's `live.db`** read-only at boot (`syncLinkedLiveRoles`,-- `server/db/database.js:28-99`), which ADR-007 forbids (C-58). None of its default paths is Live's-- production path. Whether `OPENVIBELIVE_DB_PATH` is set is unknown: check `/etc/openvibe/network.env`.-- - **Chat reads Live's files** at `/opt/openvibe.live/data/sounds`. These are files, not a database-- (`deploy/systemd/openvibe-chat.service:43`).--7. **Canonical money is still in Live.** Billing's ledger (was `billing.db`, now the cluster database-- `ov_billing` after the 2026-09-28 switch) is a shadow (Billing `STATUS.json: "authoritative": false`)-- until the cutover in [cutover-runbook.md](cutover-runbook.md) step B.----## Per service----"Drill" is the latest result in [restore-drills.md](restore-drills.md), all 2026-09-23. "Deployed" is from--the operator notes unless cited. Where a repo's `STATUS.json` disagrees, the operator notes and the drill--log win.----**The rows record the 2026-09-23 SQLite state.** For a service that has since switched to PostgreSQL —--tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events,--community, billing, media, openre (C-89) — the "Backing store" path below is now the **frozen pre-switch--SQLite file** kept read-only for the 7-day rollback, and the live store is the cluster database--`ov_<name>` reached through PgBouncer, as in [Where the data lives now](#where-the-data-lives-now). The--five SQLite rows (network, chat, tools, games, live) are current. The handler citations are the ones--reviewed on 2026-09-23.----| Service | Backing store (production path; env, default) | Canonical? | ADR-007 status | Readiness checks (handler) | Store checked? | Backup / drill |--|---|---|---|---|---|---|---|--| **Network** | `/opt/openvibe.network/data/network.db`; `DB_PATH`, default `./data/network.db` (`server/config.js:55`). Analytics lives in the same file (`README.md:306-307`). | Yes: identity, OAuth clients, subjects, OpenCoins wallets, themes, notifications, analytics | SQLite, the reviewed single-host state. `docs/platform-plan.md:74,111` defers PostgreSQL "until a room needs it". | `/api/ready` (`server/observability.js:58-96`). Required: `db` (`SELECT COUNT(*) FROM oauth_clients`), `signing_key` in production (signs and verifies an RS256 probe; fails on the HS256 dev key). Optional: `registry_poll`, `discord_bot`. | **Yes** | Drill passed 05:22. ovhost's deploy readiness polls the static `/api/health` (finding 3). |--| **Live** | `/opt/openvibe.live/data/live.db` (legacy in-place layout, "current production", Live `docs/deploy.md:31`); `DB_PATH`, default `./data/live.db` (`server/config.js:228`). Also `data/rs-companion.db`, and `data/analytics.db`, whose path was hard-coded with no env (`server/index.js:361-363`); from Live `8a58aea` it is `<DATA_DIR>/analytics.db` (`ANALYTICS_DB_PATH` overrides it; `server/paths.js`). | Yes: streams, channels, follows, Live accounts, restreams, **money** (`BILLING_AUTHORITY` unset = `live`), AI state (`vod_ai_state`/`clip_ai_state`), analytics. **Not canonical:** the 12 chat mirror tables (Chat is, since 02:03 UTC) and the frozen `vods`/`clips`/`pastes` (Media and Community are). | SQLite, the reviewed state. No PostgreSQL plan. Money moves to Billing, not to PostgreSQL. | `/api/ready` (`server/index.js:748-756`, `server/web/observability.js:79-106`). Required: `boot`, `db` (`SELECT 1` on `live.db`). Optional: `sfu`, `media` (`MEDIA_URL/healthz`, cached 30 s), `network_key` (PEM loaded, no call). | **Yes** for `live.db`; not `analytics.db` or `rs-companion.db` | `ovhost backup` covers `live.db` and `rs-companion.db`. `analytics.db` is left out on purpose (`host.example.json` `_databasesNote`). Drill **supported** from Live `8a58aea` (`LIVE_DRILL=1`: reads only, files under `DATA_DIR`, no listeners but HTTP, no jobs; not run yet). Before that release it was unsupported: `analytics.db` was opened relative to the checkout, RTMP always bound, and restreams, relays and AI jobs resumed from the copy. |--| **Media** | `/opt/openvibe.media/data/media.db`; `DB_PATH`, default `./data/media.db` (`server/config.js:24`). Local `VOD_PATH`, `CLIPS_PATH`, `PASTES_PATH`, `THUMBNAILS_PATH`, `FILES_PATH`, `OBJECTS_PATH` (`config.js:27-58`), plus B2 and R2 object storage. | Yes: media objects, VODs, clips, files, thumbnails, avatars. Paste writes are frozen (Community is canonical). Media still serves paste screenshot bytes. | SQLite, the reviewed state. No PostgreSQL plan. | `/api/ready` (`server/observability.js:72-107`). Required: `db` (`SELECT COUNT(*) FROM apps`), and `storage_{vods,clips,pastes,thumbnails,files,objects}` (writes and unlinks a probe). Optional: `network_jwks`, `remote_b2`/`remote_r2` (HeadBucket, cached 60 s), `events_outbox` (>1000 pending). Also reports `recordings_in_progress`. | **Yes** (database and disk; object storage optional) | `ovhost backup` covers `media.db`. Drill **supported** from Media `6e74eb0` (`MEDIA_DRILL=1`: reads only, no webhooks, jobs or ffmpeg, byte routes 503; not run yet). Before that release it was unsupported: webhooks to Live come from the `apps` table, and ffmpeg jobs run against production paths. ovhost's deploy readiness polls `/healthz` (finding 3). |--| **Tools** | The gateway has no database. Each satellite has `apps/<app>/data/analytics.db` (relative to `/opt/openvibe.tools`). img, audio and docs also have `data/jobs.db` (`apps/_shared/jobs/index.js:77`), `data/uploads` and `data/output`; yt has `data/downloads`. Job results go to Media when `TOOLS_JOB_RESULTS=media` (default `local`, `apps/_shared/jobs/index.js:32-36`). | Yes: jobs (img, audio, docs) and per-app analytics | SQLite, the reviewed state. One file per process, so ADR-007's "concurrent writers across processes" trigger does not apply. | `/api/ready` on every unit (`apps/_shared/observe.js:71-162`). Gateway: `catalog` required; `network_key`, `service_directory`, `community`, and each `satellite_<name>` optional (`apps/gateway/server/index.js:60-72`). img, docs, audio: `jobs_db` (`SELECT COUNT(*) FROM tool_jobs`), `job_runtime` and data directories required. yt: `downloads_dir`. food: `maps` upstream. maps, text: none required (`analytics_db` optional). | **Partly**: `jobs.db` yes on its satellites; analytics optional by design; the gateway reports satellites as optional (finding 2) | `ovhost backup` covers the seven `analytics.db` files only; **`jobs.db` is missing** (finding 5). Drill **unsupported**: eight units, relative `analytics.db` paths, job runtimes re-queue at start. |--| **Games** | `/opt/openvibe.games/data/world.db`; `DB_PATH`, default `data/world.db` (`apps/server/src/config.ts:79`). No `.env.example`. | Yes: world, accounts, inventory | SQLite, the reviewed state. No PostgreSQL plan. | `/api/ready` (since `f11e21c`): required `db` (`meta.schema_version` in `world.db`) and `tick` (last step < 5 s ago); 404 on the public vhosts. `/healthz` unchanged. | **Yes** | `ovhost backup` covers `world.db`. Drill **unsupported** (runtime side effects not reviewed). Games is `managed: false` in the inventory. |--| **Community** | `/var/lib/openvibe-community/community.db` (`StateDirectory`, `deploy/systemd/openvibe-community.service:26-27`). `/opt/openvibe.community/data/community.db` is the legacy location, and the inventory keeps whichever of the two exists. `COMMUNITY_DB_PATH`, default `./data/community.db` (`server/config.js:56`). | Yes: pastes (`PASTES_AUTHORITY=community` since 2026-09-22), comments, spaces, threads, Pulse | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:29-66`). Required: `db` (`SELECT COUNT(*) FROM sqlite_master WHERE type='table'`, fails on 0). Optional: `network_jwks`, `media` (community mode) or `live` (live mode). Reports `pastes_authority`. | **Yes** | Drill passed 03:44 (manual) and 05:22. |--| **Events** | `/var/lib/openvibe-events/events.db` (`deploy/systemd/openvibe-events.service:30-31`); `EVENTS_DB_PATH`, default `./data/events.db` (`server/config.js:67`). | Yes: the durable event store, subscriptions, deliveries, DLQ | SQLite, the reviewed state. `README.md:163`: "the plan's PostgreSQL + Redis fanout is a later step". No dated plan. | `/api/ready` (`server/app.js:41-66`). Required: `db` (`store.ping()` = `SELECT 1`), `network_jwks`, `delivery_worker` (unless `EVENTS_WORKER=off`). Optional: `dlq` (degraded at more than `EVENTS_DLQ_DEGRADED_AT`, default 100). | **Yes** | Drill passed 05:22. |--| **Chat** | `/var/lib/openvibe-chat/chat.db` (`deploy/systemd/openvibe-chat.service:38-39`); `CHAT_DB_PATH`, default `./data/chat.db` (`server/config.js:34`). It also reads Live's `/opt/openvibe.live/data/sounds` (`:43`). | Yes, since 2026-09-23 02:03 UTC: the 12 chat tables (`table_authority = 'chat'`). Six staged tables are still Live's. | SQLite, the reviewed state. | `/ready` (loopback). Since `45f2b1f`: required `db` (`SELECT MAX(id) FROM chat_messages`, 503 on failure); optional `live_sync` (degraded when no clean Live sync pass for `LIVE_SYNC_STALE_MS`, 60 s, or one step is late). `mirror.pending` is still reported only. | **Yes** | Drill passed 05:22. |--| **OpenRestream** | `/var/lib/openre/openre.db` (all four units); default `./data/openre.db`. | Stream definitions, sessions and destinations for switched slots. **No slot is switched**, so Live is still the ingest authority for every stream. | SQLite, the reviewed state. `README.md:66` names PostgreSQL only as a future ADR-007 trigger. | `/api/ready` (`server/app.js:31-51`, own handler). Required: `db` (`SELECT 1`), `key` (Network public key). Reported but not required: workers, coordinator lease, store, events. Not checked: RTMP reachability, Media, Events. | **Yes** | **Not in the ovhost inventory: no `ovhost backup`, no drill** (finding 4). |--| **Billing** | `/var/lib/openvibe-billing/billing.db` (`deploy/systemd/openvibe-billing.service:23-25`, `StateDirectoryMode=0700`); `BILLING_DB_PATH`, default `./data/billing.db` (`server/config.js:62`). The journal is append-only (triggers refuse UPDATE and DELETE, `README.md:45-50`). | **Not yet.** It is a shadow: `STATUS.json` `"authoritative": false`. Live holds the money until the cutover. | SQLite, the reviewed state. ADR-007's SERIALIZABLE and row-lock rule applies when money moves to PostgreSQL. No plan yet. | `/api/ready` (`server/app.js:54-60`, own handler). `SELECT 1 FROM settings WHERE id = 1` and Network public key loaded, otherwise 503 `service.not_ready`. `/api/health` reports frozen state and providers. | **Yes** | Drill passed 05:22. |--| **Tips** | `/var/lib/openvibe-tips/tips.db` (`deploy/systemd/openvibe-tips.service:25`); `TIPS_DB_PATH` (`server/config.js:31`). | Yes: creator tip profiles, interactions, goals | SQLite, the reviewed state. | `/api/ready` (`server/app.js:74-79`, own handler): `SELECT 1 FROM settings WHERE id = 1` and key loaded. | **Yes** | Drill passed 05:23. |--| **VIP** | `/var/lib/openvibe-vip/vip.db` (`deploy/systemd/openvibe-vip.service:25`); `VIP_DB_PATH` (`server/config.js:31`). | Yes: creators, plans, memberships, perks (checkout closed until the Billing cutover) | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:20-48`). Required: `db` (`COUNT(*) FROM vip_creators >= 1`). Optional: `network_jwks`, `billing`. | **Yes** | Drill passed 05:23. |--| **AI** | `/var/lib/openvibe-ai/ai.db` (`deploy/systemd/openvibe-ai.service:30-31`); `AI_DB_PATH`, default `./data/ai.db` (`server/config.js:81`). | Yes: workflows, runs, providers, quotas, usage (Live keeps its AI product state, AI `docs/migration.md:30-35`) | SQLite, the reviewed state. | `/api/ready` (`server/app.js:26-33`, own handler). Required: `db` (`SELECT 1`), `key`, `workflows` (at least one registered). | **Yes** | **Not in the ovhost inventory: no `ovhost backup`, no drill** (finding 4). |--| **Search** | `/var/lib/openvibe-search/search.db` (`deploy/systemd/openvibe-search.service:28`); `SEARCH_DB_PATH` (`server/config.js:49`). | **No**: a derived FTS5 index, rebuildable from its owners (sources, wiki, blog, news, reviews) | SQLite FTS5. `docs/adr-engine.md` plans PostgreSQL FTS "once a PostgreSQL instance exists" (`server/engine/fts5.js:16` keeps the engine behind an interface). | `/api/ready` (`server/app.js:30-43`): `SELECT 1` and key loaded. | **Yes** | Drill passed 05:23. |--| **Sources** | `/var/lib/openvibe-sources/sources.db` (`deploy/systemd/openvibe-sources.service:29`); `SOURCES_DB_PATH` (`server/config.js:41`). | Yes: source registry, items, revisions, fetch runs | SQLite, the reviewed state. | `/api/ready` (`server/app.js:25+`): `SELECT 1`, key, and the worker when enabled. | **Yes** | Drill passed 05:23. |--| **Wiki** | `/var/lib/openvibe-wiki/wiki.db` (`deploy/systemd/openvibe-wiki.service:29`); `WIKI_DB_PATH` (`server/config.js:27`). | Yes: spaces, pages, revisions | SQLite, the reviewed state. | `/api/ready` (`server/app.js:63-75`). Required: `db` (`wiki_pages` exists). Optional: jwks, events_relay, community, sources, media. | **Yes** | Drill passed 05:23. |--| **Blog** | `/var/lib/openvibe-blog/blog.db` (`deploy/systemd/openvibe-blog.service:28`); `BLOG_DB_PATH` (`server/config.js:32`). | Yes: blogs, posts, revisions | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:19-58`). Required: `db` (charter tables exist). Optional: jwks, events_relay, scheduler. | **Yes** | Drill passed 05:23. |--| **News** | `/var/lib/openvibe-news/news.db` (`deploy/systemd/openvibe-news.service:29`); `NEWS_DB_PATH` (`server/config.js:32`). | Yes: topics, stories, source items | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:21-65`). Required: `db` (charter tables). Optional: jwks, events_relay, events_webhook, sources_pull. | **Yes** | Drill passed 05:23. |--| **Reviews** | `/var/lib/openvibe-reviews/reviews.db` (`deploy/systemd/openvibe-reviews.service:30`); `REVIEWS_DB_PATH` (`server/config.js:28`). | Yes: entities, signals, summaries | SQLite, the reviewed state. | `/api/ready` (`server/app.js:82-99`). Required: `db` (three review tables). Optional: jwks, sources, events relay and webhook, community, editors. | **Yes** | Drill passed 05:23. |--| **Deals** | `/var/lib/openvibe-deals/deals.db` (`deploy/systemd/openvibe-deals.service:28`); `DEALS_DB_PATH` (`server/config.js:34`). | Yes: offers, products, votes | SQLite, the reviewed state. | `/api/ready` (`server/observability.js`). Required: `db` (tables exist, `:26-28`). Optional: jwks, events_relay, sources_import, worker, abuse_keys. | **Yes** | Drill passed 05:23. |--| **Coupons** | `/var/lib/openvibe-coupons/coupons.db` (`deploy/systemd/openvibe-coupons.service:29`); `COUPONS_DB_PATH` (`server/config.js:32`). | Yes: coupons, merchants, domains | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:24-26`). Required: `db`. Optional: jwks, events_relay, sweep, sources_import, reporter_key. | **Yes** | Drill passed 05:23. |--| **Trade** | `/var/lib/openvibe-trade/trade.db` (`deploy/systemd/openvibe-trade.service:29`); `TRADE_DB_PATH` (`server/config.js:32`). | Yes: instruments, observations, source documents (informational, ADR-025) | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:23-25`, mounted `server/app.js:121`). Required: `db`. Optional: jwks, events_relay, sources_sync, freshness. | **Yes** | Drill passed 05:23. |--| **Codes** | `/var/lib/openvibe-codes/codes.db` (`deploy/systemd/openvibe-codes.service:29`); `CODES_DB_PATH` (`server/config.js:37`). | Yes: manifests, releases, trust | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:24-32`). Required: `db`, `docs`. Optional: jwks, network, oauth_client, events_relay. | **Yes** | Drill passed 05:23. |--| **Host** | Stage B: `/var/lib/openvibe-host-api/host.db` and `HOST_STORAGE_DIR` blobs (`deploy/systemd/openvibe-host.service:37-38`; `server/config.js:47-49`). Stage A (`ovhost`) keeps JSONL logs in `/var/lib/openvibe-host` and has no database. | Yes: tenant projects, sites, deploys, domains | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:22-29`). Required: `db`, `storage` (`blobs.writable()`, `server/storage.js:98-102`). Optional: jwks, events_relay, domain_checks. | **Yes** (database and blob directory) | Drill passed 05:23 (`host.db`, counts only). `README.md` and `STATUS.json` say Stage B is **not deployed**, while the drill counted a production `host.db`. Whether `openvibe-host.service` is running is unknown: check `systemctl status openvibe-host`. |----## Deployed-state notes----- **Loopback only**, not on their domains yet (operator notes): Tips, VIP, Search, Sources, News, Reviews,-- Codes, AI and OpenRestream's API. **Public**: Wiki and Blog (since ~03:00 UTC). **Staff console public**:-- Billing. Deals, Coupons and Trade passed drills against production, so they run on the host. Whether-- they are public is unknown: check nginx `sites-enabled`. Sites still has placeholders for-- openvibe.deals, .coupons and .trade.--- Several `STATUS.json` files are stale against the operator notes and the drill log. Chat, Events, OpenRestream-- and AI still say `"deployed": false`, and so do Tips, Search, Sources, News, Reviews, Deals, Coupons,-- Trade and Codes. `STATUS.json` stays each service's own record to fix. This page does not change it.----## Re-checking----```--sudo ovhost status <service> # unit, port, readiness as ovhost sees it--curl -s 127.0.0.1:<port>/api/ready | jq .checks # the named checks and their state--sudo ovhost backup <service> && sudo ovhost drill <service>--```----To re-verify a row, read the handler cited in the row. A readiness change is a code change in that--service, and this page is updated in the same week.@@@ -1,6 +1,6 @@{"_about": "Example ovhost inventory for the single OpenVibe host (openvibe-oregon). Copy to /etc/openvibe/host.json (root:root, 0640) and correct anything that differs on the machine; the real file is never committed. Paths, units, ports and env file locations come from OpenVibe.Network docs/roadmap-baseline/data/services.json and prod-snapshot.json (22 Sep 2026) and each repository's deploy/ directory. env.required lists NAMES only.",-- "_databases": "Databases (lib/inventory.js): each services.<id>.databases[] item is one of two engines. SQLite (the default) is a file the service user's worker copies: { \"name\": \"live\", \"path\": \"/opt/openvibe.live/data/live.db\" }. PostgreSQL is a database in the cluster: { \"name\": \"trade\", \"engine\": \"postgresql\", \"database\": \"ov_trade\" } \u2014 the name must match ov_[a-z0-9_]+, and there is no path (ovhost reaches it as the postgres OS user over peer auth, and pgBackRest, not ovhost, copies it). tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, bot, host, events, community, billing, media, openre and games are PostgreSQL; network, chat, tools and live are still SQLite. A drill block follows the engine: a PostgreSQL database maps to { \"url\": \"DATABASE_URL\", \"directUrl\": \"DATABASE_DIRECT_URL\" } and the drill creates a scratch database and role to restore into.",++ "_databases": "Every databases entry names a PostgreSQL database in the cluster: { \"name\": \"trade\", \"engine\": \"postgresql\", \"database\": \"ov_trade\" }. A drill maps it to { \"url\": \"DATABASE_URL\", \"directUrl\": \"DATABASE_DIRECT_URL\" } and restores a scratch database and role.","host": "openvibe-oregon","stateDir": "/var/lib/openvibe-host","backupDir": "/var/backups/openvibe",@@@ -39,7 +39,8 @@"BASE_URL","JWT_PRIVATE_KEY","JWT_PUBLIC_KEY",-- "DB_PATH"++ "DATABASE_URL",++ "DATABASE_DIRECT_URL"]},"port": 4000,@@@ -55,7 +56,8 @@"databases": [{"name": "network",-- "path": "/opt/openvibe.network/data/network.db"++ "engine": "postgresql",++ "database": "ov_network"}],"nginx": {@@@ -67,7 +69,10 @@"port": 14000,"ready": "/api/ready","databases": {-- "network": "DB_PATH"++ "network": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -75,7 +80,8 @@"OV_EVENTS_INTERNAL_URL": "","INTERNAL_API_KEY": "","OV_TOOLS_INTERNAL_URL": "http://127.0.0.1:9",-- "AVATAR_PATH": "{tmp}/avatars"++ "AVATAR_PATH": "{tmp}/avatars",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"dirs": ["{tmp}/avatars"@@@ -105,8 +111,7 @@"db": "network","table": "coin_transactions"}-- ],-- "_notes": "OV_EVENTS_INTERNAL_URL empty stops the dev_audit relay; INTERNAL_API_KEY empty falls back to the placeholder key, which makes the chrome site-copy job (paid LLM call through Live), avatar import and Live analytics calls return early. No switch: the ecosystem poller GETs every service readiness and release.json on loopback every 60 s (read-only). Discord bot login and the Resend email queue are gated by settings in the copy and reach only remote hosts, which the sandbox blocks. JWT_PRIVATE_KEY/JWT_PUBLIC_KEY come from the production env file and are only read."++ ]}},"live": {@@@ -145,7 +150,9 @@"OV_OAUTH_CLIENT_SECRET","MEDIA_URL","MEDIA_API_KEY",-- "MEDIA_WEBHOOK_SECRET"++ "MEDIA_WEBHOOK_SECRET",++ "DATABASE_URL",++ "DATABASE_DIRECT_URL"]},"port": 3000,@@@ -183,11 +190,8 @@"databases": [{"name": "live",-- "path": "/opt/openvibe.live/data/live.db"-- },-- {-- "name": "rs-companion",-- "path": "/opt/openvibe.live/data/rs-companion.db"++ "engine": "postgresql",++ "database": "ov_live"}],"_databasesNote": "analytics.db (1.7 GB, append-only pageviews) is left out of routine backups on purpose; add it here to include it.",@@@ -208,10 +212,12 @@"maxBody": "3m"},"drill": {-- "_notes": "Needs OpenVibe.Live 8a58aea or later (LIVE_DRILL, server/drill.js; requires checks the deployed checkout). LIVE_DRILL=1 refuses to start unless DB_PATH and DATA_DIR are outside the checkout and /opt/openvibe.live, HOST is loopback and PORT is not 3000, and never takes over the systemd socket. It then serves HTTP only: no RTMP/SRT/WHIP/SFU/JSMPEG listener, no WebSocket (upgrades get 403), no TURN credential, no jobs loop, restream or relay resume, AI job, Media reconciler, Events outbox, chat bridge, identity sync, deploy notice, star job or registry refresh. Every method but GET/HEAD/OPTIONS answers 403; net, fetch, child_process (git only), dgram and listen are guarded, so every other service looks down to it. All its files go to DATA_DIR ({tmp}/data: a fresh analytics.db and empty data directories; the per-location *_PATH values in live.env are ignored in a drill). rs-companion.db is not restored: the server never opens it. Compared: DB-only public reads; ignore lists drop what production changes in memory or every few seconds while someone is live (viewer counts, heartbeats, live thumbnails, recording state) and the one Media lookup (recently-online vod_thumbnail). /api/streams/channel/japaneseoldguy/live is the star channel: any channel that exists works. /release.json is not compared: production computes it at its own boot, so a public/-only deploy without a restart would make it differ. EVENTS_URL, the client secret and MEDIA_URL are blanked as a second switch. Not run on the host yet.","port": 13000,"databases": {-- "live": "DB_PATH"++ "live": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -220,7 +226,8 @@"DATA_DIR": "{tmp}/data","EVENTS_URL": "","OV_OAUTH_CLIENT_SECRET": "",-- "MEDIA_URL": "http://127.0.0.1:9"++ "MEDIA_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"dirs": ["{tmp}/data"@@@ -436,7 +443,9 @@"OV_NETWORK_URL","OV_OAUTH_CLIENT_ID","OV_OAUTH_CLIENT_SECRET",-- "MEDIA_URL"++ "MEDIA_URL",++ "DATABASE_URL",++ "DATABASE_DIRECT_URL"]},"metrics": {@@@ -515,59 +524,19 @@},"databases": [{-- "name": "tools-audio-analytics",-- "path": "/opt/openvibe.tools/apps/audio/data/analytics.db"-- },-- {-- "name": "tools-docs-analytics",-- "path": "/opt/openvibe.tools/apps/docs/data/analytics.db"-- },-- {-- "name": "tools-food-analytics",-- "path": "/opt/openvibe.tools/apps/food/data/analytics.db"-- },-- {-- "name": "tools-img-analytics",-- "path": "/opt/openvibe.tools/apps/img/data/analytics.db"-- },-- {-- "name": "tools-maps-analytics",-- "path": "/opt/openvibe.tools/apps/maps/data/analytics.db"-- },-- {-- "name": "tools-text-analytics",-- "path": "/opt/openvibe.tools/apps/text/data/analytics.db"-- },-- {-- "name": "tools-yt-analytics",-- "path": "/opt/openvibe.tools/apps/yt/data/analytics.db"-- },-- {-- "name": "tools-audio-jobs",-- "path": "/opt/openvibe.tools/apps/audio/data/jobs.db"-- },-- {-- "name": "tools-docs-jobs",-- "path": "/opt/openvibe.tools/apps/docs/data/jobs.db"-- },-- {-- "name": "tools-img-jobs",-- "path": "/opt/openvibe.tools/apps/img/data/jobs.db"++ "name": "tools",++ "engine": "postgresql",++ "database": "ov_tools"}],"drill": {-- "_notes": "Drills ONE app: docs (both database kinds; spawns no external program; every path has an env override). Its unit's ExecStart/WorkingDirectory come from drill.unit, and the comparison runs against docs' own port. DATA_DIR takes a directory, so both copies land in {tmp}/data under their production names. TOOLS_JOB_RESULTS=local keeps the job pruner from deleting production Media objects for expired jobs in the copy; EVENTS_PUBLISH=off keeps the copy's pending outbox rows from being republished; TOOLS_JOBS_CONCURRENCY=0 runs no job. What still runs, on the copy only: job recovery and pruning, analytics aggregation, upload retention (all under {tmp}). yt, maps and food open data/analytics.db relative to the checkout: drill one with drill.unit + a {tmp} data dir + bind over apps/<app>/data (plus YT_PROXY= for yt) until Tools adds DATA_DIR there. Not run on the host yet.","port": 14016,"unit": "openvibe-tools-docs.service","productionPort": 4016,"databases": {-- "tools-docs-analytics": {-- "env": "DATA_DIR",-- "dir": true-- },-- "tools-docs-jobs": {-- "env": "DATA_DIR",-- "dir": true++ "tools": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"}},"env": {@@@ -580,7 +549,8 @@"EVENTS_PUBLISH": "off","EVENTS_URL": "","OV_OAUTH_CLIENT_SECRET": "",-- "MEDIA_URL": "http://127.0.0.1:9"++ "MEDIA_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"dirs": ["{tmp}/uploads",@@@ -611,31 +581,16 @@],"counts": [{-- "db": "tools-docs-jobs",++ "db": "tools","table": "tool_jobs"}]},"protected": {-- "kind": "sum",-- "label": "running tool jobs",-- "probes": [-- {-- "kind": "sqlite-count",-- "db": "/opt/openvibe.tools/apps/img/data/jobs.db",-- "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'"-- },-- {-- "kind": "sqlite-count",-- "db": "/opt/openvibe.tools/apps/audio/data/jobs.db",-- "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'"-- },-- {-- "kind": "sqlite-count",-- "db": "/opt/openvibe.tools/apps/docs/data/jobs.db",-- "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'"-- }-- ]++ "kind": "postgresql-count",++ "database": "ov_tools",++ "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'",++ "label": "running tool jobs"},"drain": {"policy": "report"@@@ -676,7 +631,6 @@"database": "ov_community"}],-- "_databasesNote": "Community is one PostgreSQL database (ov_community) now. Before the switch it declared two SQLite paths (the repo unit's /var/lib/openvibe-community/community.db and the /opt/openvibe.community/data/community.db the 22 Sep production snapshot found); neither is declared any more, and the frozen files are for `ovhost archive push`.","nginx": {"vhost": "openvibe.community.conf","repoVhost": "deploy/nginx/openvibe.community.conf",@@@ -721,8 +675,7 @@"db": "community","table": "posts"}-- ],-- "_notes": "Community is one PostgreSQL database (ov_community) now: the second, legacy SQLite path it used to declare is gone. An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."++ ]}},"events": {@@@ -1145,13 +1098,17 @@"databases": [{"name": "chat",-- "path": "/var/lib/openvibe-chat/chat.db"++ "engine": "postgresql",++ "database": "ov_chat"}],"drill": {"port": 14400,"databases": {-- "chat": "CHAT_DB_PATH"++ "chat": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1161,7 +1118,8 @@"OV_OAUTH_CLIENT_SECRET": "","OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9","CHAT_CACHE_DIR": "{tmp}/cache",-- "SOUNDS_PATH": "{tmp}/sounds"++ "SOUNDS_PATH": "{tmp}/sounds",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"dirs": ["{tmp}/cache",@@@ -1187,8 +1145,7 @@"db": "chat","table": "moderation_actions"}-- ],-- "_notes": "An empty client secret and a dead Live URL stop the 2 s Live projection sync and the effects POSTed to Live. History endpoints are not compared: cosmetics come from Live (cut off here) and expired messages are swept at boot, so chat_messages is not counted either."++ ]},"protected": {"kind": "http-json-count",@@@ -2056,7 +2013,7 @@}],"drill": {-- "_notes": "Every provider key, base URL and the HTTP seam are blanked and AI_ENABLED=0, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and EVENTS_URL are blanked (no service calls, no events); WHISPER_BIN is blanked (no transcription). Boot marks interrupted runs failed in the restored copy only. Counts cover configuration and citations only: runs, requests and usage grow by the minute, so they are checked by integrity_check, not by an equal count. Passed on the host 2026-09-24 05:40 UTC.",++ "_notes": "Every provider key, base URL and the HTTP seam are blanked and AI_ENABLED=0, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and EVENTS_URL are blanked (no service calls, no events); WHISPER_BIN is blanked (no transcription). Boot marks interrupted runs failed in the restored copy only. Counts cover configuration and citations only: runs, requests and usage grow by the minute, so the dump is checked by pg_restore --list; these rows are not compared by count. Passed on the host 2026-09-24 05:40 UTC.","port": 14700,"databases": {"ai": {@@@ -2119,7 +2076,7 @@}},"bot": {-- "_note": "Deployed; migrations 0001 and 0002 applied. Bot owns openvibe.bot.conf: deploy bot installs its proxy vhost after Sites skips that filename. BOT_WHIP_BASE (in /etc/openvibe/bot.env) is the switch that points Bot's video at https://ingest.openre.stream/whip. Use ovhost deploy bot --restart if the checkout is already current so it installs and enables the service unit. Keep Sites on its ovhost deploy path (its legacy fallback still installs every vhost).",++ "_note": "Deployed; migrations 0001 and 0002 applied. Bot owns openvibe.bot.conf: deploy bot installs its proxy vhost after Sites skips that filename. BOT_WHIP_BASE (in /etc/openvibe/bot.env) points Bot's video at https://ingest.openre.stream/whip. Use ovhost deploy bot --restart if the checkout is already current so it installs and enables the service unit. Keep Sites on its ovhost deploy path so it installs every vhost.","repo": "/opt/openvibe.bot","installUnits": true,"enableUnits": true,@@@ -159,7 +159,7 @@ const SCENARIOS = [source: 'the test asserts host.restarts() is [] and the record says restarted: false',},{-- id: '1b', what: 'the same through Live\'s own deploy script on a simulated host', repo: 'live', files: ['test/deploy-sim.test.js'], timeoutMs: 300000,++ id: '1b', what: 'the same through Live\'s deploy wrapper on a simulated host', repo: 'live', files: ['test/deploy-sim.test.js'], timeoutMs: 300000,expect: [ok('public/-only change: current switches, process is NOT restarted')],budget: { name: 'Live restarts', max: 0 }, measure: asserted(), category: 'affected sessions',source: 'the test compares the process id before and after the switch',@@@ -7,8 +7,7 @@* poll.** ovhost deploy|rollback <service> announces after a deploy or rollback that went live-- * ovhost announce <service> for services deployed by their own scripts (Live, Tools, Sites,-- * OpenRestream, Games)++ * ovhost announce <service> manually announce a service's currently served release** Best effort: it never fails or holds up a deploy. The token request and the publish each have a* short timeout and the whole announcement a budget of a few seconds; a failure is logged and the@@@ -1,6 +1,6 @@'use strict';/**-- * ovhost archive (roadmap WS-S task 6, hazard H14): dead and backup database files on the host are++ * ovhost archive: retired data files on the host are* encrypted and copied off-host before anyone deletes them. This never deletes a local file: that* waits for the owner, with the archive's manifest in hand.*@@@ -42,11 +42,6 @@ async function refusal(exec, inv, file) {if (typeof file !== 'string' || !path.isAbsolute(file) || path.normalize(file) !== file) return 'not a normalised absolute path';if (!ALLOWED_ROOTS.some((r) => file.startsWith(r))) return `outside ${ALLOWED_ROOTS.join(', ')}`;if (!NAME_RE.test(objectName(file))) return 'the path has characters an object name cannot carry';-- // A live SQLite database is a file the inventory declares, and archiving it is refused. A-- // PostgreSQL database has no file (its own entry carries no path), so the C-89 switch — archive-- // the frozen SQLite file, then delete it — is no longer blocked by the inventory that moved on.-- const declared = Object.values(inv.services || {}).flatMap((s) => (s.databases || []).filter((d) => d.engine !== 'postgresql').map((d) => d.path));-- if (declared.includes(file)) return 'a database the inventory declares (a live database)';const st = await exec.stat(file);if (!st) return 'no such file';if (st.isSymlink || !st.isFile) return 'not a regular file';@@@ -10,7 +10,7 @@ const { SocketGuardError } = require('./systemd');const { runBrowserCheck } = require('./browser-check-hook');const VERSION = require('../package.json').version;--// What deploy wrappers probe (`ovhost capabilities <service>`): bump when the deploy contract changes.++// Keep the capabilities command for deployed wrappers that still call it.const DEPLOY_API = 1; // 1: strategies, capabilities, deploy --prepare-onlyconst USAGE = `ovhost — OpenVibe.Host operator plane (Stage A)@@@ -54,12 +54,6 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]data provision <service> [--dry-run] run roles/data/add-service.sh from the ovhost install: theservice's PostgreSQL database, roles and Valkey user(idempotent; prints names only, never values)-- data switch <service> [--sqlite <file>] [--dry-run]-- run roles/data/switch-service.sh from the ovhost install: stop,-- back up the SQLite file, import and verify, deploy; SWITCH_UNITS/-- SWITCH_DIR come from the inventory, and the script's exit code is-- passed through. Needs the service's scripts/migrate-to-postgres.js-- (docs/data-provisioning.md)version | --version ovhost's versionannounce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run]publish host.release.published to OpenVibe.Events (public), so open@@@ -142,7 +136,7 @@ self-update: 0 updated, up to date or dry run · 1 precondition (not root, dirtybackup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed`;--const FLAGS_WITH_VALUE = new Set(['prometheus', 'sites', 'token-file', 'ttl', 'title', 'services', 'severity', 'message', 'state', 'from', 'until', 'reason', 'incident', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note', 'tail', 'sqlite']);++const FLAGS_WITH_VALUE = new Set(['prometheus', 'sites', 'token-file', 'ttl', 'title', 'services', 'severity', 'message', 'state', 'from', 'until', 'reason', 'incident', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note', 'tail']);function parseArgs(argv) {const pos = [];@@@ -214,8 +208,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procconst ctx = { exec, inv, log };switch (cmd) {case 'capabilities': {-- // For the per-repository deploy wrappers (deploy/scripts/deploy.sh): they use ovhost only when-- // it knows the strategy they need and the inventory manages the service with it.++ // Older per-repository deploy wrappers still query this command before invoking ovhost.const lines = [`ovhost=${VERSION}`, `deploy-api=${DEPLOY_API}`, `strategies=${lib.inventory.STRATEGIES.join(',')}`, 'commands=plan,deploy,rollback,announce,capabilities', 'deploy-flags=--wait-idle,--force,--restart,--to,--install-units,--ready-timeout,--prepare-only,--no-announce'];const row = { ovhost: VERSION, deployApi: DEPLOY_API, strategies: lib.inventory.STRATEGIES };if (pos[1]) {@@@ -255,11 +248,11 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc// The data role's own scripts, run from ovhost's install (ADR-035): the agent pipeline// reaches ovhost, not the shell. Output is redacted and the script's exit code passes through.const sub = pos[1];-- if (sub !== 'provision' && sub !== 'switch') throw new UsageError('usage: ovhost data provision <service> [--dry-run] | ovhost data switch <service> [--sqlite <file>] [--dry-run]');++ if (sub !== 'provision') throw new UsageError('usage: ovhost data provision <service> [--dry-run]');const id = pos[2];if (!id) throw new UsageError(`data ${sub} needs a <service>`);-- const o = { selfPath: selfPath || process.argv[1], dryRun: !!opts['dry-run'], sqlite: opts.sqlite };-- const r = sub === 'provision' ? await lib.data.provision(ctx, id, o) : await lib.data.switch(ctx, id, o);++ const o = { selfPath: selfPath || process.argv[1], dryRun: !!opts['dry-run'] };++ const r = await lib.data.provision(ctx, id, o);if (json) { emit(r); return r.code === null ? 0 : r.code; }if (r.dryRun) { out(`would run: ${r.command}`); return 0; }for (const l of String(r.stdout).split('\n')) if (l) out(l);@@@ -515,7 +508,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procfor (const s of r.services) {const copied = s.files.filter((f) => f.dest && !f.error);const size = copied.reduce((a, f) => a + (f.bytes || 0), 0);-- const detail = s.status === 'failed' ? s.error : s.status === 'skipped' ? 'no database found' : s.status === 'verified' ? 'pgBackRest verified (no dump due)' : `${copied.length} file(s), ${bytes(size)}`;++ const detail = s.status === 'failed' ? s.error : s.status === 'verified' ? 'pgBackRest verified (no dump due)' : `${copied.length} file(s), ${bytes(size)}`;out(`${s.status.toUpperCase().padEnd(8)} ${s.service.padEnd(10)} ${detail}${s.pruned.length ? `; pruned ${s.pruned.length}` : ''}`);}if (r.offsite) {@@@ -723,7 +716,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procconst cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });const r = await lib.offsite.restoreDownload(ctx, id, pos[2], { cfg, client: s3(cfg), out: opts.out, host: opts['from-host'] });if (json) { emit(r); return 0; }-- for (const f of r.files) out(`${f.file} ${bytes(f.bytes)} sha256 ${f.sha256.slice(0, 16)}… quick_check ${f.quickCheck} (was ${f.source})`);++ for (const f of r.files) out(`${f.file} ${bytes(f.bytes)} sha256 ${f.sha256.slice(0, 16)}… check ${f.check} (was ${f.source})`);out(`restored ${r.service} from ${r.host}/${r.run} into ${r.dir} (root-only). Nothing live was touched;`);out('see docs/backups.md "Restoring from an off-host copy" to put a copy in place.');return 0;@@@ -1,41 +1,30 @@'use strict';/**-- * ovhost backup <service>: an online copy of every database the inventory declares for the service,-- * and a tar archive of every content-addressed object directory it declares, into-- * <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in-- * <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root-- * never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.-- * PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups-- * and its WAL (roles/data/provision.sh), so a run that finds a PostgreSQL database first VERIFIES-- * that layer (dbengine.verifyPgBackRest: a stanza that is ok and whose newest backup is under 26 h-- * old; dbengine.walFailedCount: pg_stat_archiver.failed_count has not risen since the last run) and-- * only then, on a slower cadence (LOGICAL_DUMP_INTERVAL_DAYS, or --logical), takes a per-service-- * pg_dump -Fc archive — the artifact the off-site code uploads and a drill can restore.++ * ovhost backup <service>: every database the inventory declares for the service is in the PostgreSQL++ * cluster, and every content-addressed object directory it declares is archived, into++ * <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in <stateDir>/backups/<service>.jsonl.++ * PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups and++ * its WAL (roles/data/provision.sh), so a run first VERIFIES that layer (dbengine.verifyPgBackRest: a++ * stanza that is ok and whose newest backup is under 26 h old; dbengine.walFailedCount:++ * pg_stat_archiver.failed_count has not risen since the last run) and only then, on a slower cadence++ * (LOGICAL_DUMP_INTERVAL_DAYS, or --logical), takes a per-service pg_dump -Fc archive: the artifact the++ * off-site code uploads and a drill restores.*-- * A PostgreSQL service's record always carries `verified` (type + stop + walOk when it passed, an-- * error when it did not), and a run with nothing to dump is verification-only: ok: true, files: [],-- * and no directory at all. A failed verification (or a logical dump that failed) is recorded-- * (ok: false) and then throws a BackupError carrying the record, so backup --all reports the service-- * failed — never skipped.++ * A record always carries `verified` (type + stop + walOk when it passed, an error when it did not), and a++ * run with nothing to dump or archive is verification-only: ok: true, files: [], and no directory at all.++ * A failed verification (or a dump or archive that failed) is recorded (ok: false) and then throws a++ * BackupError carrying the record, so backup --all reports the service failed, never skipped.*-- * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600, with-- * no -wal/-shm beside it. SQLite's worker writes into <backupStagingDir>/<service>-<stamp> (its own,-- * 0700, under a root 0711 parent) as the service user. A service with any PostgreSQL database gets a-- * staging directory owned by postgres instead, because pg_dump runs as that OS user. In both cases-- * root then takes that directory over (lockDown, root:root 0700) BEFORE anything inside it is-- * checked — the service user (or postgres) can no longer reach it while root validates the copies,-- * chowns and chmods them and renames the directory into place, which is what keeps the property.++ * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600. pg_dump++ * runs as the postgres OS user, so a service with a database gets a staging directory owned by postgres++ * under a root 0711 parent (an objects-only service's stays root's). Root takes that directory over++ * (lockDown, root:root 0700) BEFORE anything inside it is checked, then validates, chowns and chmods the++ * copies and renames the directory into place.*-- * Every declared database is attempted even when an earlier one fails. A backup with any failed-- * database is still recorded (ok: false, so a drill and retention never treat it as good) and then-- * throws a BackupError carrying the record. A SQLite backup that found none of its databases is-- * recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.-- *-- * ovhost backup --all (backupAll): every service that declares databases or object directories, one-- * after another. A failing service never stops the others. Afterwards each service's backups are-- * pruned to the last 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run-- * is written to <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded-- * (lib/offsite.js).++ * ovhost backup --all (backupAll): every service that declares databases or object directories, one after++ * another. A failing service never stops the others. Afterwards each service's backups are pruned to the++ * last 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run is written to++ * <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded (lib/offsite.js).*/const path = require('path');const { service } = require('../inventory');@@@ -148,7 +137,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {if (!svc.databases.length && !svc.objects.length) throw new Error(`${id} declares no databases or object directories`);const postgres = svc.databases.filter((d) => d.engine === 'postgresql');const hasPostgres = postgres.length > 0;-- const engine = hasPostgres ? 'postgresql' : 'sqlite';++ const engine = hasPostgres ? 'postgresql' : null;// (a) Verify the cluster's own backup layer BEFORE copying anything. A failed verification is// itself a record (ok: false): backupAll then marks the service failed, never skipped.@@@ -171,9 +160,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {const at = last === null ? NaN : Date.parse(last);if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);}-- // A verification-only run (PostgreSQL, no dump due, no SQLite database, no objects) leaves++ // A verification-only run (no dump due and no objects) leaves// nothing on disk.-- if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite') && !svc.objects.length) {++ if (!dump.length && !svc.objects.length) {const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);return record;@@@ -183,8 +172,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {const dir = path.join(inv.backupDir, id, st);if (await exec.stat(dir)) throw new Error(`${dir} already exists; a backup never overwrites another`);// Backups hold user data: every directory under backupDir is root:root 0700 and every copy is-- // root:root 0600. SQLite's worker runs as the service user (root never opens a service database),-- // so it writes into a staging directory of its own, which root then takes over before the move.++ // root:root 0600. The postgres OS user writes dumps into staging; root takes over before the move.await privateDir(exec, inv.backupDir);await privateDir(exec, path.join(inv.backupDir, id));await exec.mkdir(inv.backupStagingDir, { owner: 'root', mode: 0o711 });@@@ -192,29 +180,14 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {const stage = path.join(inv.backupStagingDir, `${id}-${st}`);if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });// pg_dump runs as the postgres OS user, so a service with any PostgreSQL database needs a staging-- // directory postgres can write into (SQLite keeps the service user). The lockDown(stage, 0o700)++ // directory postgres can write into. The lockDown(stage, 0o700)// below takes the whole directory back to root:root BEFORE anything inside it is checked, so// neither the service user nor postgres can reach it while root validates and moves it.-- await exec.mkdir(stage, { owner: hasPostgres ? dbengine.PG_USER : svc.runAs, mode: 0o700 });++ await exec.mkdir(stage, { owner: hasPostgres ? dbengine.PG_USER : 'root', mode: 0o700 });const files = [];try {for (const db of svc.databases) {const file = dbengine.fileName(db);-- if (db.engine === 'sqlite') {-- if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }-- log(`backing up ${db.path} → ${path.join(dir, file)} (as ${svc.runAs})`);-- const t0 = exec.now();-- try {-- // The worker runs PRAGMA quick_check on the copy and fails unless it answers ok.-- await exec.sqliteBackup(db.path, path.join(stage, file), { as: svc.runAs });-- } catch (err) {-- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), error: err.message });-- log(`FAILED ${db.path}: ${err.message}`);-- continue;-- }-- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), file, check: 'quick_check ok', seconds: Math.round((exec.now() - t0) / 1000) });-- continue;-- }if (!dump.includes(db)) { log(`${db.database} is covered by pgBackRest today; no logical dump due`); continue; }const t0 = exec.now();try {@@@ -227,9 +200,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {}files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });}-- // Take the staging directory back FIRST: while the service user (or postgres) owns it, it could++ // Take the staging directory back FIRST: while postgres owns it, that user could// plant a symlink at the path root is about to write the object archive through, and root's-- // `tar -czf` would follow it. Once root:root 0700, neither can reach it, and everything written++ // `tar -czf` would follow it. Once root:root 0700, postgres cannot reach it, and everything written// below is checked before the move.await lockDown(exec, stage, 0o700);// Content-addressed object directories: one tar archive each, created by root AFTER the stage@@@ -253,7 +226,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {continue;}}-- // Clear whatever might sit at the archive's path — a link planted while the service user++ // Clear whatever might sit at the archive's path — a link planted while postgres// owned the stage would survive lockDown — before root's tar opens it for writing.await exec.run('rm', ['-rf', '--', archive], { privileged: true });log(`backing up object directory ${obj.path} → ${path.join(dir, file)}`);@@@ -275,7 +248,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {await lockDown(exec, p, 0o600);f.bytes = fst.size;}-- // Anything else (a stray -wal/-shm) is not part of the backup.++ // Anything else in staging is not part of the backup.const wanted = new Set(files.filter((x) => x.file && !x.error).map((x) => x.file));for (const e of (await exec.readdir(stage)) || []) {if (!wanted.has(e.name)) await exec.run('rm', ['-rf', '--', path.join(stage, e.name)], { privileged: true });@@@ -293,9 +266,6 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {// `file` is kept on each entry: the weekly cadence reads it back to find the last good dump.const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: !failed.length && copied.length > 0, dir, files };if (verified) record.verified = verified;-- // Nothing existed to back up (a SQLite service not deployed yet): no directory is left, so an-- // empty backup can never stand in for a real one in retention or a drill. PostgreSQL never empty.-- if (!copied.length && !failed.length && !hasPostgres) record.empty = true;await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);if (failed.length) throw new BackupError(failed.map((f) => `${f.name}: ${f.error}`).join('; '), record);return record;@@@ -357,13 +327,12 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, ofsummary.services.push(entry);try {const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}`, logical });-- entry.dir = rec.empty ? null : (rec.dir || null);++ entry.dir = rec.dir || null;entry.files = rec.files;if (rec.verified) entry.verified = rec.verified;// A PostgreSQL service with nothing to dump this run was still verified, not backed// up: it has its own status so the summary distinguishes the two.-- if (rec.empty) entry.status = 'skipped';-- else if (rec.engine === 'postgresql' && !rec.dir) entry.status = 'verified';++ if (rec.engine === 'postgresql' && !rec.dir) entry.status = 'verified';} catch (err) {entry.status = 'failed';entry.error = err.message;@@@ -411,7 +380,7 @@ const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';* openvibe_backup_last_run_timestamp_seconds when the last run finished* openvibe_backup_last_run_ok 1 when every service (and the off-site copy) succeeded* openvibe_backup_last_success_timestamp_seconds the last run that succeeded (read back from backup-runs)-- * openvibe_backup_services{status} services ok, failed, skipped and verified in the last run++ * openvibe_backup_services{status} services ok, failed and verified in the last run* openvibe_backup_pg_verified{service} 1 when that service's PostgreSQL verification passed* openvibe_pgbackrest_last_backup_age_seconds age of the newest pgBackRest backup, at write time* openvibe_backup_offsite_ok 1/0 for the last run's off-site copy, when it made one@@@ -444,7 +413,7 @@ async function writeMetrics(ctx, summary) {`openvibe_backup_last_run_ok ${summary.ok ? 1 : 0}`,'# HELP openvibe_backup_services Services by outcome in the last run.','# TYPE openvibe_backup_services gauge',-- ...['ok', 'failed', 'skipped', 'verified'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),++ ...['ok', 'failed', 'verified'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),];if (verified.length) {lines.push(@@@ -1,18 +1,5 @@'use strict';--/**-- * ovhost data provision | switch — run the data role's own scripts from the ovhost install-- * (roles/data/add-service.sh, roles/data/switch-service.sh; ADR-035). The agent pipeline may run-- * only ovhost through its broker, so giving a service its PostgreSQL database and switching it-- * over from SQLite has to go through here too.-- *-- * ovhost data provision <service> -> roles/data/add-service.sh <service>-- * ovhost data switch <service> [--sqlite <file>] -> roles/data/switch-service.sh <service> [<file>],-- * with SWITCH_UNITS/SWITCH_DIR from the inventory-- *-- * add-service.sh prints names only. Whatever the scripts say, their output passes through redact():-- * a connection URL (postgres://, redis://, valkey://) or a NAME=value whose name carries PASSWORD-- * never reaches a terminal, a log or --json. The scripts' own exit code is passed through.-- */++/** ovhost data provision runs the data role from this installation. */const path = require('path');const { service: inventoryService } = require('../inventory');const { OpError, EXIT } = require('../release-ops');@@@ -33,27 +20,6 @@ function dataDirFor(selfPath) {return path.join(dir, 'roles', 'data');}--/** A template worker unit is stopped as a glob: [email protected] -> openre-rtmp-ingest@*.service. */--function unitPattern(unit) {-- return unit.endsWith('@.service') ? `${unit.slice(0, -'.service'.length)}*.service` : unit;--}----/** The SWITCH_UNITS/SWITCH_DIR the script would otherwise default to (roles/data/switch-service.sh). */--function switchEnv(svc) {-- const env = {};-- const units = [...svc.units, ...svc.workerUnits].map(unitPattern);-- // Deploys restart every unit matching unitsMatch (release-ops), so the switch stops them too.-- if (svc.unitsMatch && !units.includes(svc.unitsMatch)) units.push(svc.unitsMatch);-- // A socket-activated service comes straight back on the next request while its socket listens: the old-- // release would write the database during the import (2026-10-08, Live: analytics rows moved under the verify).-- // The socket is stopped after the service (the first unit stays the one whose environment runs the import); the-- // deploy's restart starts it again through the service's Requires=.-- if (svc.socketUnit && !units.includes(svc.socketUnit)) units.push(svc.socketUnit);-- if (units.length) env.SWITCH_UNITS = units.join(' ');-- env.SWITCH_DIR = svc.repo;-- return env;--}--function printable(script, args, env) {const assigns = Object.entries(env || {}).map(([k, v]) => `${k}=${JSON.stringify(v)}`).join(' ');return redact(`${assigns ? `${assigns} ` : ''}${script}${args.map((a) => ` ${JSON.stringify(a)}`).join('')}`);@@@ -71,8 +37,7 @@ async function locate(exec, selfPath, script, action, id) {async function run(ctx, svc, { action, script, args, env = {}, dryRun = false }) {const result = { service: svc.id, action, script, args, env, dryRun };if (dryRun) return { ...result, command: printable(script, args, env), code: null, ok: true, stdout: '', stderr: '' };-- // Both scripts need root (add-service.sh exits otherwise; switch uses systemctl, setpriv, chown),-- // and sudo's env_reset would drop SWITCH_UNITS/SWITCH_DIR, silently switching the wrong units/dir.++ // The data role requires root.if (!(await ctx.exec.isRoot())) throw new OpError(`data ${action} must run as root (sudo ovhost data ${action} ${svc.id})`, EXIT.USAGE);const r = await ctx.exec.run(script, args, { privileged: true, env });return { ...result, code: r.code, ok: r.code === 0, stdout: redact(r.stdout), stderr: redact(r.stderr) };@@@ -84,12 +49,4 @@ async function provision(ctx, id, { selfPath, dryRun = false } = {}) {return run(ctx, svc, { action: 'provision', script, args: [svc.id], dryRun });}--async function switchService(ctx, id, { selfPath, sqlite, dryRun = false } = {}) {-- const svc = inventoryService(ctx.inv, id);-- const script = await locate(ctx.exec, selfPath, 'switch-service.sh', 'switch', svc.id);-- const args = [svc.id];-- if (sqlite) args.push(sqlite);-- return run(ctx, svc, { action: 'switch', script, args, env: switchEnv(svc), dryRun });--}----module.exports = { provision, switch: switchService, redact, dataDirFor, switchEnv, unitPattern };++module.exports = { provision, redact, dataDirFor };@@@ -5,20 +5,17 @@* production does:** 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Every database the drill-- * block names is restored, per engine (lib/dbengine.js):-- * SQLite copy the file into a fresh directory under <drillDir> owned by the service user-- * and run PRAGMA integrity_check on the copy. The result must be `ok`.-- * PostgreSQL check free space first, copy the pg_dump -Fc archive into <tmp>/pg owned by-- * postgres, verify it with `pg_restore --list`, then create a scratch database and-- * a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the-- * archive into the scratch database. Both are dropped again in a finally.++ * block names is restored from its pg_dump -Fc archive (lib/dbengine.js): check free space first,++ * copy the archive into <tmp>/pg owned by postgres, verify it with `pg_restore --list`, then create a++ * scratch database and a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the++ * archive into the scratch database. Both are dropped again in a finally.* Every object directory the service declares is also restored: its archive is extracted into* <tmp>/<name> (owned by the service user, never the live directory) and every blob is checked* against the sha256 in its file name.* 2. Start a second instance from the production checkout, as the service user, through* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production-- * env file, and a second env file with the drill's overrides: its port, the restored database-- * paths (or the scratch database's URLs), and the switches that turn side effects off. The++ * env file, and a second env file with the drill's overrides: its port, the scratch database++ * URLs, and the switches that turn side effects off. The* instance runs sandboxed: ProtectSystem=strict with only the drill directory writable, a* private /tmp, binding only its drill port, and no addresses beyond loopback unless the* inventory says otherwise. RuntimeMaxSec stops it even if ovhost dies. Loopback stays open@@@ -70,10 +67,9 @@ const STOP_GRACE_MS = 20000;// ── helpers ──────────────────────────────────────────────────────────────────function expand(value, vars) {-- return value.replace(/\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g, (_, k) => {++ return value.replace(/\{(tmp|port)\}/g, (_, k) => {if (k === 'tmp') return vars.tmp;if (k === 'port') return String(vars.port);-- return vars.db[k.slice(3)];});}@@@ -145,13 +141,6 @@ function compareBodies(entry, prod, drill) {function countSql(table) { return `SELECT count(*) AS n FROM "${table}"`; }--async function countRows(exec, db, table, as) {-- const rows = await exec.sqlite(db, countSql(table), { as });-- const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);-- if (!Number.isFinite(n)) throw new Error(`count of ${table} returned no number`);-- return n;--}--// A content-addressed blob is named by its own sha256: that name is the whole integrity check.const BLOB_RE = /^[0-9a-f]{64}$/;@@@ -230,7 +219,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {let takenAt = null;let record = null;const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));-- const pgTargets = restored.filter((d) => d.engine === 'postgresql');++ const pgTargets = restored;if (dir) {if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');dir = path.normalize(dir);@@@ -255,11 +244,10 @@ async function resolveBackup(exec, inv, svc, dirArg) {for (const name of Object.keys(svc.drill.databases)) {const db = svc.databases.find((d) => d.name === name);const target = svc.drill.databases[name];-- const file = path.join(dir, db.engine === 'postgresql' ? dbengine.fileName(db) : `${name}${path.extname(db.path) || '.db'}`);++ const file = path.join(dir, dbengine.fileName(db));const fst = await exec.stat(file);if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);-- if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });-- else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });++ files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });}// Object directories are restored from the same backup: the archive holds the declared// directory's contents, so the drill must not run without it (there would be nothing to verify).@@@ -327,7 +315,7 @@ async function missingRequirements(exec, svc) {async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const { exec, inv, log } = ctx;const svc = service(inv, id);-- if (!(await exec.isRoot())) throw new DrillError('ovhost drill must run as root (sudo ovhost drill …): it copies databases as the service user and starts the drill instance through systemd-run');++ if (!(await exec.isRoot())) throw new DrillError('ovhost drill must run as root (sudo ovhost drill …): it starts the drill instance through systemd-run');if (!svc.drill) throw new DrillError(`${id} has no drill block in the inventory`);if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);const d = svc.drill;@@@ -425,18 +413,13 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);await exec.mkdir(inv.drillDir, { mode: 0o711 });// postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;-- // every file inside keeps its own restrictive mode). A SQLite-only drill keeps <tmp> at 0700.-- await exec.mkdir(tmp, { owner: svc.runAs, mode: pgFiles.length ? 0o711 : 0o700 });++ // every file inside keeps its own restrictive mode).++ await exec.mkdir(tmp, { owner: svc.runAs, mode: 0o711 });created = true;-- await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });// postgres (not the service user) reads a PostgreSQL dump, so it gets its own directory; the// copy inside is 0600 postgres (the backup itself is root-only, which postgres cannot read).if (pgFiles.length) await exec.mkdir(path.join(tmp, 'pg'), { owner: dbengine.PG_USER, mode: 0o700 });-- const vars = { tmp, port: d.port, db: {} };-- // A data-directory database keeps its production file name: the service finds it by name.-- const dirOf = (f) => (f.dir ? expand(f.dir, vars) : null);-- for (const f of bk.files) { if (f.engine === 'postgresql') continue; vars.db[f.name] = f.dir ? path.join(dirOf(f), path.basename(f.production)) : path.join(tmp, 'db', `${f.name}${f.ext}`); }-- for (const dir of new Set(bk.files.filter((f) => f.dir).map(dirOf))) await exec.mkdir(dir, { owner: svc.runAs, mode: 0o700 });++ const vars = { tmp, port: d.port };for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });for (const b of d.bind) if (!(await exec.stat(expand(b.from, vars)))) await exec.mkdir(expand(b.from, vars), { owner: svc.runAs, mode: 0o700 });@@@ -444,46 +427,27 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const sid = id.replace(/[^a-z0-9_]/g, '_');const scratchFor = (name) => scratches.find((s) => s.name === name);for (const f of bk.files) {-- if (f.engine === 'postgresql') {-- // A PostgreSQL restore replaces the file copy: root copies the root-only archive into-- // <tmp>/pg owned by postgres, verifies it, then creates a scratch database and role and-- // restores the archive into them (--no-owner, --role: the archive names production's-- // role, and every object must end up owned by the drill role the instance connects as).-- const copy = path.join(tmp, 'pg', `${f.name}.dump`);-- log(`restoring ${f.file} → ${copy} (owned by ${dbengine.PG_USER})`);-- const cp = await exec.run('install', ['-o', dbengine.PG_USER, '-m', '0600', '-T', '--', f.file, copy], { privileged: true });-- if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);-- const st = await exec.stat(copy);-- const entry = { name: f.name, source: f.file, copy, bytes: st ? st.size : null, integrity: null, engine: 'postgresql' };-- record.databases.push(entry);-- try { await dbengine.verifyDump(exec, copy, { as: dbengine.PG_USER }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }-- entry.integrity = 'pg_restore --list ok';-- log(`pg_restore --list ${f.name}: ok`);-- // The role and the database share the same name (separate namespaces); dbengine always-- // quotes the identifier, so the stamp's hyphen is fine.-- const name = `ov_${sid}_drill_${at}${pgFiles.length > 1 ? `_${pgFiles.indexOf(f)}` : ''}`;-- const { password } = await dbengine.createRole(exec, name, {});-- const scratch = { name: f.name, database: name, role: name, password, dropped: null, error: null };-- scratches.push(scratch);-- if (!record.scratch) record.scratch = { database: name, role: name };-- await dbengine.createDatabase(exec, name, { owner: name });-- await dbengine.restore(exec, f.production, { dump: copy, database: name, role: name, log });-- continue;-- }-- const dest = vars.db[f.name];-- log(`restoring ${f.file} → ${dest} (owned by ${svc.runAs})`);-- // Backups are root-only (0600), so root copies; install unlinks whatever is at dest-- // before writing, so nothing planted in the drill directory is followed.-- const cp = await exec.run('install', ['-o', svc.runAs, '-m', '0600', '-T', '--', f.file, dest], { privileged: true });++ // Root copies the archive into <tmp>/pg owned by postgres, verifies it, then restores++ // it into a scratch database and role. Every object ends up owned by the drill role.++ const copy = path.join(tmp, 'pg', `${f.name}.dump`);++ log(`restoring ${f.file} → ${copy} (owned by ${dbengine.PG_USER})`);++ const cp = await exec.run('install', ['-o', dbengine.PG_USER, '-m', '0600', '-T', '--', f.file, copy], { privileged: true });if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);-- const st = await exec.stat(dest);-- const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null, engine: 'sqlite' };++ const st = await exec.stat(copy);++ const entry = { name: f.name, source: f.file, copy, bytes: st ? st.size : null, integrity: null, engine: 'postgresql' };record.databases.push(entry);-- let rows;-- try { rows = await exec.sqlite(dest, 'PRAGMA integrity_check', { as: svc.runAs }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }-- entry.integrity = (rows || []).map((r) => String(Object.values(r)[0])).join('; ') || 'no result';-- if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);-- log(`integrity_check ${f.name}: ok`);++ try { await dbengine.verifyDump(exec, copy, { as: dbengine.PG_USER }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }++ entry.integrity = 'pg_restore --list ok';++ log(`pg_restore --list ${f.name}: ok`);++ // The role and the database share the same name (separate namespaces); dbengine always++ // quotes the identifier, so the stamp's hyphen is fine.++ const name = `ov_${sid}_drill_${at}${pgFiles.length > 1 ? `_${pgFiles.indexOf(f)}` : ''}`;++ const { password } = await dbengine.createRole(exec, name, {});++ const scratch = { name: f.name, database: name, role: name, password, dropped: null, error: null };++ scratches.push(scratch);++ if (!record.scratch) record.scratch = { database: name, role: name };++ await dbengine.createDatabase(exec, name, { owner: name });++ await dbengine.restore(exec, f.production, { dump: copy, database: name, role: name, log });}// 1b. restore every declared object directory and check every blob's sha256. The archive is// root-only, so root extracts it into the drill's own scratch directory — never over the live@@@ -510,10 +474,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {}const restoredCounts = [];for (const c of d.counts) {-- const rdb = svc.databases.find((x) => x.name === c.db);-- // PostgreSQL counts come from the scratch database over psql, SQLite from the restored file.-- if (rdb.engine === 'postgresql') restoredCounts.push(await dbengine.countRows(exec, { engine: 'postgresql', database: scratchFor(c.db).database }, countSql(c.table)));-- else restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));++ restoredCounts.push(await dbengine.countRows(exec, { database: scratchFor(c.db).database }, countSql(c.table)));}// 2. start the drill instance@@@ -529,7 +490,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const url = dbengine.databaseUrl({ database: s.database, role: s.role, password: s.password });dbEnv.set(f.url, url);dbEnv.set(f.directUrl, url);-- } else if (f.envName) dbEnv.set(f.envName, f.dir ? dirOf(f) : vars.db[f.name]);++ }}for (const [k, v] of dbEnv) lines.push(envLine(k, v));const envPath = path.join(tmp, 'drill.env');@@@ -606,9 +567,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const db = svc.databases.find((x) => x.name === c.db);let production = null;let error = null;-- // PostgreSQL counts come from the cluster over psql (the inventory entry itself, never-- // .path, which a PostgreSQL entry does not have); SQLite from the production file.-- try { production = db.engine === 'postgresql' ? await dbengine.countRows(exec, db, countSql(c.table)) : await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }++ try { production = await dbengine.countRows(exec, db, countSql(c.table)); } catch (err) { error = err.message; }const tol = d.countsTolerance || 0;// A busy table drifts between the backup and now: countsTolerance (default 0) allows it.const match = production != null && restoredCounts[i] != null && Math.abs(production - restoredCounts[i]) <= tol;@@@ -709,9 +668,7 @@ function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }/** A row for the table in docs/restore-drills.md. */function markdownRow(r) {const checks = [];-- for (const db of r.databases) checks.push(db.engine === 'postgresql'-- ? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`-- : `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);++ for (const db of r.databases) checks.push(`\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`: `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);@@@ -64,11 +64,11 @@ async function validate(ctx, id, opts = {}) {try {const g = git(exec, svc);const [branch, head, dirty] = [await g.branch(), await g.head(), await g.dirty()];-- // A checkout reached through <dir>/current -> <dir>/releases/<id> is a deployed release (Live's-- // own deploy script): detached by design, so the branch is not checked; changes still are.++ // A checkout reached through <dir>/current -> <dir>/releases/<id> is a deployed release:++ // detached by design, so the branch is not checked; changes still are.const link = branch === 'HEAD' && svc.repo.endsWith('/current') ? await exec.readlink(svc.repo) : null;const release = link && path.resolve(path.dirname(svc.repo), link).startsWith(`${path.dirname(svc.repo)}/releases/`);-- if (release) add('ok', 'checkout', `release ${link} (detached; deployed by the repository's own script)`);++ if (release) add('ok', 'checkout', `release ${link} (detached)`);else if (branch !== svc.branch) add('error', 'checkout', `on branch "${branch}", expected "${svc.branch}"`);const own = dirty.filter((f) => !require('../release-ops').matchesPath(f, svc.generated));if (own.length < dirty.length) add('info', 'checkout', `${dirty.length - own.length} tracked build output file(s) differ from git (${svc.generated.join(', ')}); a deploy restores them first`);@@@ -1,14 +1,8 @@'use strict';/**-- * Per-engine database helpers. lib/inventory.js normalises every entry in `databases[]` to an-- * `engine` of "sqlite" (the default: a file) or "postgresql" (a database in the cluster), so no-- * caller has to know which commands an engine needs.-- *-- * SQLite keeps the existing worker (lib/sqlite-worker.js): every file operation runs as the service-- * user, and root never opens a service database. PostgreSQL runs as the postgres OS user over peer-- * auth (`local postgres peer` in roles/data/files/pg_hba.conf) — the same property: ovhost reads no-- * service env value, and never holds a database password except the one it generates itself for a-- * restore drill.++ * PostgreSQL database helpers (lib/inventory.js accepts no other engine). Every command runs as the postgres++ * OS user over peer auth (`local postgres peer` in roles/data/files/pg_hba.conf): ovhost reads no service env++ * value and never holds a database password except the one it generates itself for a restore drill's role.** pgBackRest, not ovhost, is the durability layer for the cluster (roles/data/provision.sh: one* stanza `openvibe`, full weekly + differential nightly + WAL to object storage). What ovhost does@@@ -16,7 +10,6 @@* a per-service logical dump (dump) that the off-site code encrypts and uploads unchanged.*/const crypto = require('crypto');--const path = require('path');// The OS user PostgreSQL's own tools run as (peer auth), the stanza provision.sh creates, and the// direct connection a drill must use: 5432 is the cluster, 6432 is PgBouncer.@@@ -36,17 +29,16 @@ const scrubbed = (r) => ((r.stderr || r.stdout || '').trim().slice(0, 300) || `e/** What a log line and a record call this database. */function label(db) {-- return db.engine === 'postgresql' ? db.database : db.path;++ return db.database;}/** The file name this database's artifact gets inside a backup directory. */function fileName(db) {-- return db.engine === 'postgresql' ? `${db.name}.dump` : `${db.name}${path.extname(db.path) || '.db'}`;++ return `${db.name}.dump`;}--/** Whether the database is still there (SQLite: the file; PostgreSQL: the database in the cluster). */++/** Whether the database is still there in the cluster. */async function exists(exec, db) {-- if (db.engine === 'sqlite') return !!(await exec.stat(db.path));const rows = await exec.psql('postgres', `SELECT 1 AS present FROM pg_database WHERE datname = ${literal(db.database)}`);return rows.length > 0;}@@@ -57,7 +49,6 @@ async function exists(exec, db) {* the caller's staging directory is owned by postgres when any database is PostgreSQL.*/async function dump(exec, db, dest, { log } = {}) {-- if (db.engine !== 'postgresql') throw new Error('sqlite dumps go through exec.sqliteBackup (the worker runs as the service user)');// -w: peer auth needs no password, and a prompt in the nightly unit would hang until the timeout.const r = await exec.run('pg_dump', ['-w', '-Fc', '-f', dest, '-d', db.database], { as: PG_USER, timeoutMs: WORK_TIMEOUT_MS });if (r.code !== 0) throw new Error(`pg_dump ${db.database} failed: ${scrubbed(r)}`);@@@ -114,7 +105,6 @@ async function dropRole(exec, role) {* owned by the drill role, so the drilled service can read and write it.*/async function restore(exec, db, { dump: dumpFile, database, role, log } = {}) {-- if (db.engine !== 'postgresql') throw new Error('sqlite restore is a file copy, not pg_restore');// --no-privileges: the archive's GRANTs name production roles; the scratch copy must not hand them anything.const args = ['-w', '--no-owner', '--no-privileges'];if (role) args.push(`--role=${role}`);@@@ -137,9 +127,9 @@ async function scratchLeftovers(exec, prefix) {return { databases: names(databases), roles: names(roles) };}--/** A row count from either engine. `sql` must return the number in its first column. */--async function countRows(exec, db, sql, { as } = {}) {-- const rows = db.engine === 'postgresql' ? await exec.psql(db.database, sql) : await exec.sqlite(db.path, sql, { as });++/** A row count from PostgreSQL. `sql` must return the number in its first column. */++async function countRows(exec, db, sql) {++ const rows = await exec.psql(db.database, sql);const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);if (!Number.isFinite(n)) throw new Error(`count of ${label(db)} returned no number`);return n;@@@ -1,12 +1,11 @@'use strict';/**-- * The system executor: the ONLY module in ovhost that touches the machine (processes, files, HTTP,-- * SQLite). Every command and library function takes an executor, so the tests swap this for the-- * fake host in test/fake-host.js and nothing in the suite ever runs systemctl, git, npm or curl.++ * The system executor: the only module in ovhost that touches the machine. Tests replace it with++ * the fake host in test/fake-host.js.** Interface (all async):* run(cmd, args, { as, privileged, cwd, timeoutMs, input }) -> { code, stdout, stderr }-- * as run as this user (git/npm as the checkout owner, sqlite as the service user);++ * as run as this user (git/npm as the checkout owner);* root uses runuser with the user's HOME, the same user runs directly, anyone else* goes through `sudo -n -H -u`.* privileged run as root (systemctl, nginx -t, reading 0600 env files); `sudo -n` unless root.@@@ -22,11 +21,9 @@* request(url, { method, headers, body, timeoutMs }) -> { status, body } | { status: 0, error }* any method, http:// or https:// (release notifications: Network's token endpoint and Events'* publish API). Headers and body may carry a secret: nothing here logs them.-- * sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })* psql(database, sql, { as }) -> rows. PostgreSQL is reached as the postgres OS user over peer-- * auth (`local postgres peer`, roles/data/files/pg_hba.conf) — the same no-secret property the-- * SQLite worker gives the service user. Everything else PostgreSQL (pg_dump, pg_restore,-- * createdb, dropdb, pgbackrest) goes through run() with { as: 'postgres' }.++ * auth (`local postgres peer`, roles/data/files/pg_hba.conf). PostgreSQL tools also go++ * through run() with { as: 'postgres' }.* statfs(path) -> { free, size } bytes, for the free-space check before a restore* listeners(port) -> [{ pid, process }]* kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)@@@ -39,7 +36,6 @@ const httpsLib = require('https');const os = require('os');const path = require('path');--const WORKER = path.join(__dirname, 'sqlite-worker.js');function passwdEntry(user) {try {@@@ -229,14 +225,6 @@ function createSystemExecutor({ env = process.env } = {}) {});}-- async function sqliteCall(payload, as) {-- const r = await run(process.execPath, [WORKER, JSON.stringify(payload)], { as, timeoutMs: 6 * 3600 * 1000 });-- let out;-- try { out = JSON.parse(r.stdout); } catch { throw new Error(`sqlite worker failed: ${(r.stderr || r.stdout).trim().slice(0, 300)}`); }-- if (!out.ok) throw new Error(out.error);-- return out;-- }--/** `psql --csv`: a header row and one line per row, so a caller reads columns by name. */function parseCsv(text) {const rows = [];@@@ -291,8 +279,6 @@ function createSystemExecutor({ env = process.env } = {}) {pidAlive: async (pid) => pidAlive(pid),http,request,-- sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,-- sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },psql: async (database, sql, { as = 'postgres' } = {}) => {const r = await run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '--csv', '-d', database, '-c', sql], { as, timeoutMs: 6 * 3600 * 1000 });if (r.code !== 0) throw new Error(`psql ${database} failed: ${(r.stderr || r.stdout).trim().slice(0, 300)}`);@@@ -20,7 +20,7 @@ const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;const DRAIN_POLICIES = ['refuse', 'wait', 'report'];--const PROBE_KINDS = ['http-json-count', 'sqlite-count', 'postgresql-count', 'sum'];++const PROBE_KINDS = ['http-json-count', 'postgresql-count', 'sum'];// A PostgreSQL database an ovhost service owns: one per service in the cluster pgBackRest backs up// (roles/data/provision.sh), reached as the postgres OS user over peer auth (no secret is ever read).const PG_DATABASE_RE = /^ov_[a-z0-9_]+$/;@@@ -166,9 +166,9 @@ function normaliseObjects(raw, where) {function normaliseService(id, raw, defaults) {if (!ID_RE.test(id)) throw new InventoryError(`service id "${id}" must match ${ID_RE}`);const where = `services.${id}`;-- const strategy = raw.strategy == null ? (raw.layout === 'release' ? null : 'git-checkout') : raw.strategy;-- if (strategy !== null && !STRATEGIES.includes(strategy)) throw new InventoryError(`${where}.strategy must be one of ${STRATEGIES.join(', ')}`);-- if (strategy && strategy !== 'release-layout' && raw.layout === 'release') throw new InventoryError(`${where}: layout "release" needs strategy "release-layout"`);++ const strategy = raw.strategy == null ? 'git-checkout' : raw.strategy;++ if (!STRATEGIES.includes(strategy)) throw new InventoryError(`${where}.strategy must be one of ${STRATEGIES.join(', ')}`);++ if (strategy !== 'release-layout' && raw.layout === 'release') throw new InventoryError(`${where}: layout "release" needs strategy "release-layout"`);const s = withPreset({ ...defaults, ...raw }, strategy);const releaseLayout = strategy === 'release-layout' || s.layout === 'release';let repo = abs(s.repo, `${where}.repo`);@@@ -190,16 +190,15 @@ function normaliseService(id, raw, defaults) {strategy,release,// 'git' (a checkout ovhost can move) or 'release' (releases/<id> behind a `current` symlink).-- // A release-layout entry without strategy "release-layout" is deployed by its own script.layout: releaseLayout ? 'release' : 'git',-- managed: strategy === null ? false : s.managed !== false,++ managed: s.managed !== false,// The deploy controller (ovhost reconcile, WS-X11): auto deploys main at green CI; gated and manual wait for a person.deploy: (() => {const d = s.deploy || {};if (d.policy != null && !['auto', 'gated', 'manual'].includes(d.policy)) throw new InventoryError(`${where}.deploy.policy must be auto, gated or manual`);return { policy: d.policy || 'manual' };})(),-- unmanagedReason: s.unmanagedReason || (strategy === null ? 'release layout without strategy "release-layout": deployed by its own deploy script' : null),++ unmanagedReason: s.unmanagedReason || null,skipPackages: s.skipPackages || [],removeUntrackedLockfiles: s.removeUntrackedLockfiles === true,generated: s.generated || [],@@@ -233,28 +232,16 @@ function normaliseService(id, raw, defaults) {noRestartPaths: s.noRestartPaths || [],protected: s.protected || null,drain: { policy: 'refuse', waitMaxSeconds: 8 * 3600, pollSeconds: 60, quietChecks: 2, ...(s.drain || {}) },-- // A database is one of two engines. SQLite (the default) is a file the service user's worker-- // copies; PostgreSQL is a database in the cluster, named, and backed up by pgBackRest as a-- // whole — ovhost never opens it directly. Nothing outside the engine's own code may assume-- // `.path` exists.++ // Each database is in the PostgreSQL cluster.databases: (s.databases || []).map((d, i) => {const at = `${where}.databases[${i}]`;-- const engine = d.engine == null ? 'sqlite' : d.engine;-- if (engine === 'sqlite') {-- if (d.database != null) throw new InventoryError(`${at}: a sqlite database is a path, not a database name`);-- const name = d.name || path.basename(String(d.path), '.db');-- if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);-- return { name, engine, path: abs(d.path, `${at}.path`) };-- }-- if (engine === 'postgresql') {-- if (d.path != null) throw new InventoryError(`${at}: a postgresql database has no path (it lives in the cluster, named by "database")`);-- if (!PG_DATABASE_RE.test(d.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_trade)`);-- const name = d.name || String(d.database).slice(3);-- if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);-- if (d.role != null && !/^[a-z_][a-z0-9_-]*$/.test(String(d.role))) throw new InventoryError(`${at}.role must be a database role name`);-- return { name, engine, database: d.database, role: d.role == null ? null : String(d.role) };-- }-- throw new InventoryError(`${at}.engine must be sqlite or postgresql`);++ if (!d || d.engine !== 'postgresql') throw new InventoryError(`${at}.engine must be "postgresql"; use { name, engine: "postgresql", database: "ov_<service>" }`);++ if (d.path != null) throw new InventoryError(`${at}: a postgresql database has no path (it lives in the cluster, named by "database")`);++ if (!PG_DATABASE_RE.test(d.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_trade)`);++ const name = d.name || String(d.database).slice(3);++ if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);++ if (d.role != null && !/^[a-z_][a-z0-9_-]*$/.test(String(d.role))) throw new InventoryError(`${at}.role must be a database role name`);++ return { name, engine: 'postgresql', database: d.database, role: d.role == null ? null : String(d.role) };}),// Content-addressed object directories alongside the databases (tenant objects, Host's// HOST_STORAGE_DIR). Copied as one archive per directory; never opened as a database.@@@ -314,10 +301,6 @@ function normaliseService(id, raw, defaults) {const kinds = nested ? PROBE_KINDS.filter((k) => k !== 'sum') : PROBE_KINDS;if (!kinds.includes(p.kind)) throw new InventoryError(`${at}.kind must be one of ${kinds.join(', ')}`);if (p.kind === 'http-json-count' && !/^http:\/\/(127\.0\.0\.1|localhost|\[::1\])[:/]/.test(p.url || '')) throw new InventoryError(`${at}.url must be a loopback http:// URL`);-- if (p.kind === 'sqlite-count') {-- abs(p.db, `${at}.db`);-- if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);-- }if (p.kind === 'postgresql-count') {if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);@@@ -342,9 +325,8 @@ function normaliseService(id, raw, defaults) {if (n.tenants && n.tenants.database != null) {const db = n.tenants.database;let tenantDb;-- if (typeof db === 'string') tenantDb = { engine: 'sqlite', path: abs(db, `${where}.nginx.tenants.database`) };-- else if (db && typeof db === 'object' && !Array.isArray(db) && db.engine === 'postgresql' && PG_DATABASE_RE.test(db.database || '')) tenantDb = { engine: 'postgresql', database: db.database };-- else throw new InventoryError(`${where}.nginx.tenants.database must be a path or { "engine": "postgresql", "database": "ov_host" }`);++ if (db && typeof db === 'object' && !Array.isArray(db) && db.engine === 'postgresql' && PG_DATABASE_RE.test(db.database || '')) tenantDb = { engine: 'postgresql', database: db.database };++ else throw new InventoryError(`${where}.nginx.tenants.database must be { "engine": "postgresql", "database": "ov_host" }`);// A fresh object every time: normalise() must never rewrite the document it was handed// (the example inventory is a shared require() cache, and a drill test normalises it twice).out.nginx = { ...n, tenants: { ...n.tenants, database: tenantDb } };@@@ -358,7 +340,6 @@ function normaliseService(id, raw, defaults) {const ENV_NAME_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;const TABLE_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;--const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;/*** The `drill` block (Wave 22 restore drills): how to start a second, side-effect-free instance of@@@ -366,19 +347,11 @@ const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;** supported false (with a reason) when a second instance cannot run safely* port spare loopback port for the drill instance (never the service's own port)-- * databases { <database name>: <ENV VAR that points the service at that file> }, or-- * { <database name>: { "env": "DATA_DIR", "dir": true } } when the service takes a data-- * DIRECTORY: the copy lands in {tmp}/data/ (or "dir": "{tmp}/<name>") under its-- * production file name and the env var (optional) points at that directory. Without-- * "env", pair the directory with `bind`.-- * A database with engine postgresql instead names the two env vars that point the-- * service at a database URL: { "url": "DATABASE_URL", "directUrl":-- * "DATABASE_DIRECT_URL" }. The drill creates a scratch database and a scratch role,-- * restores the dump into it, and writes both vars (and VALKEY_URL, which the drill-- * must neutralise itself) into drill.env — the service's own env file is loaded first,-- * so any name the drill does not override keeps its production value.++ * databases { <database name>: { "url": "DATABASE_URL", "directUrl":++ * "DATABASE_DIRECT_URL" } }. The drill restores into a scratch database++ * and role, then points both env vars at it.* env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}-- * and {db:<name>}++ * in the scratch directory.* dirs directories to create (owned by the service user) inside {tmp} before the start* command argv to start the service; default: the production unit's ExecStart* unit which of the service's units to start (its ExecStart, WorkingDirectory and@@@ -412,54 +385,27 @@ function normaliseDrill(raw, svc, where) {if (!svc.databases.length) throw new InventoryError(`${w}: a drill restores databases, and ${svc.id} declares none`);const dbNames = svc.databases.map((d) => d.name);const rawDatabases = raw.databases || {};-- if (typeof rawDatabases !== 'object' || Array.isArray(rawDatabases) || !Object.keys(rawDatabases).length) throw new InventoryError(`${w}.databases must map at least one database name to an env var name`);++ if (typeof rawDatabases !== 'object' || Array.isArray(rawDatabases) || !Object.keys(rawDatabases).length) throw new InventoryError(`${w}.databases must map at least one database name to its URL env var names`);const databases = {};-- const inDataDir = new Map();for (const [name, target] of Object.entries(rawDatabases)) {if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);-- const defined = svc.databases.find((d) => d.name === name);-- if (defined.engine === 'postgresql') {-- if (!target || typeof target !== 'object' || Array.isArray(target)) throw new InventoryError(`${w}.databases.${name} must be { "url": "<NAME>", "directUrl": "<NAME>" } for a postgresql database`);-- if (!ENV_NAME_RE.test(target.url || '')) throw new InventoryError(`${w}.databases.${name}.url must be an env var name`);-- if (!ENV_NAME_RE.test(target.directUrl || '')) throw new InventoryError(`${w}.databases.${name}.directUrl must be an env var name`);-- if (target.url === target.directUrl) throw new InventoryError(`${w}.databases.${name}: url and directUrl must be different env vars`);-- databases[name] = { engine: 'postgresql', url: target.url, directUrl: target.directUrl };-- continue;-- }-- if (typeof target === 'string') {-- if (!ENV_NAME_RE.test(target)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);-- databases[name] = { engine: 'sqlite', env: target, dir: false };-- continue;-- }-- const dir = target && target.dir === true ? '{tmp}/data' : target && target.dir;-- if (!target || typeof target !== 'object' || typeof dir !== 'string') throw new InventoryError(`${w}.databases.${name} must be an env var name or { "env": "<NAME>", "dir": true }`);-- if (!/^\{tmp\}\/[A-Za-z0-9._-]+$/.test(dir)) throw new InventoryError(`${w}.databases.${name}.dir must be true or "{tmp}/<name>"`);-- if (target.env != null && !ENV_NAME_RE.test(target.env)) throw new InventoryError(`${w}.databases.${name}.env must be an env var name`);-- const dest = `${dir}/${path.basename(defined.path)}`;-- if (inDataDir.has(dest)) throw new InventoryError(`${w}.databases: ${name} and ${inDataDir.get(dest)} would both be ${dest}`);-- inDataDir.set(dest, name);-- databases[name] = { engine: 'sqlite', env: target.env || null, dir };-- }-- const dirEnvs = new Map();-- for (const t of Object.values(databases)) {-- if (!t.env) continue;-- const key = t.dir || `file:${t.env}`;-- if (dirEnvs.has(t.env) && dirEnvs.get(t.env) !== key) throw new InventoryError(`${w}.databases: ${t.env} would point at two different places`);-- dirEnvs.set(t.env, key);++ if (!target || typeof target !== 'object' || Array.isArray(target)) throw new InventoryError(`${w}.databases.${name} must be { "url": "<NAME>", "directUrl": "<NAME>" } for a postgresql database`);++ if (!ENV_NAME_RE.test(target.url || '')) throw new InventoryError(`${w}.databases.${name}.url must be an env var name`);++ if (!ENV_NAME_RE.test(target.directUrl || '')) throw new InventoryError(`${w}.databases.${name}.directUrl must be an env var name`);++ if (target.url === target.directUrl) throw new InventoryError(`${w}.databases.${name}: url and directUrl must be different env vars`);++ databases[name] = { engine: 'postgresql', url: target.url, directUrl: target.directUrl };}const env = raw.env || {};for (const [k, v] of Object.entries(env)) {if (!ENV_NAME_RE.test(k)) throw new InventoryError(`${w}.env: "${k}" is not an env var name`);if (typeof v !== 'string' || /[\n\r\0]/.test(v)) throw new InventoryError(`${w}.env.${k} must be a one-line string`);-- for (const m of v.matchAll(PLACEHOLDER_RE)) {-- if (m[1].startsWith('db:') && !Object.keys(rawDatabases).includes(m[1].slice(3))) throw new InventoryError(`${w}.env.${k}: {${m[1]}} is not a restored database`);-- }++ if (/\{db:[^}]+\}/.test(v)) throw new InventoryError(`${w}.env.${k}: database file placeholders are unsupported; use drill.databases URL settings`);}if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);for (const t of Object.values(databases)) {// The drill writes these itself; declaring one in drill.env as well would put two lines with// the same name in the file, and which wins is not something an operator should have to know.-- const owned = t.engine === 'postgresql' ? [t.url, t.directUrl] : t.env ? [t.env] : [];++ const owned = [t.url, t.directUrl];for (const name of owned) if (name in env) throw new InventoryError(`${w}.env.${name} is already set by drill.databases`);}const dirs = raw.dirs || [];@@@ -567,7 +513,7 @@ function normalise(raw) {host: raw.host || null,stateDir: abs(raw.stateDir || '/var/lib/openvibe-host', 'stateDir'),backupDir: abs(raw.backupDir || '/var/backups/openvibe', 'backupDir'),-- // Where the service user's backup worker writes before root takes the copy over (see++ // Where postgres writes logical dumps before root takes the copies over (see// commands/backup.js). The same filesystem as backupDir, so the final move is a rename.backupStagingDir: abs(raw.backupStagingDir || `${path.normalize(raw.backupDir || '/var/backups/openvibe').replace(/\/+$/, '')}.staging`, 'backupStagingDir'),drillDir: abs(raw.drillDir || '/var/lib/openvibe-drills', 'drillDir'),@@@ -255,14 +255,12 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {/*** Verified custom domains from the Host database + certificate presence. A PostgreSQL database is* read as the postgres OS user over peer auth (Host moved to PostgreSQL; its service role's password-- * is not ovhost's to read), a SQLite file as the service user.++ * is not ovhost's to read).*/async function tenantDomains(exec, svc) {const t = tenantsConfig(svc);if (!t.database) throw new Error(`${svc.id}: nginx.tenants.database is not set in the inventory`);-- const rows = t.database.engine === 'postgresql'-- ? await exec.psql(t.database.database, VERIFIED_SQL)-- : await exec.sqlite(t.database.path, VERIFIED_SQL, { as: svc.runAs });++ const rows = await exec.psql(t.database.database, VERIFIED_SQL);const out = [];for (const r of rows || []) {const h = String(r.hostname || '').toLowerCase();@@@ -322,15 +322,9 @@ async function readManifest(client, cfg, master, host, run) {return m;}--/** Directories restore-download must never write into: anything holding a declared database or object directory. */++/** Directories restore-download must never write into. */function unsafeOut(inv, out) {for (const svc of Object.values(inv.services)) {-- for (const db of svc.databases) {-- // A PostgreSQL database is in the cluster, not at a path: it has no directory to avoid.-- if (!db.path) continue;-- const dir = path.dirname(db.path);-- if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;-- }// An object directory is itself a directory the live service writes: never restore over it.for (const obj of svc.objects || []) {const dir = obj.path;@@@ -344,8 +338,8 @@ function unsafeOut(inv, out) {/*** Download one service's databases from an off-host run, verify and decrypt them into a new-- * directory (root:root 0700, files 0600), and quick_check each. It never touches a live database: the directory-- * must not exist yet (or be empty), must not overlap any service's checkout or database directory,++ * directory (root:root 0700, files 0600), and check each dump. It never touches a live database: the directory++ * must not exist yet (or be empty), must not overlap any service's checkout,* and every file is created exclusively. Putting a copy in place is a separate, manual step.*/async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host: hostArg = null } = {}) {@@@ -393,27 +387,15 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:await exec.removeFile(dest);throw new OffsiteError(`${f.file}: the download does not match the manifest's hashes`);}-- // A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which-- // needs no database and is run as the caller — the copy is root-only 0600, and postgres could-- // not read it); a SQLite copy is quick_checked through the worker. An object-directory archive-- // is neither: its own sha256 has already matched the manifest, and a drill is what extracts it-- // and checks every blob.-- let check;-- const isDump = path.extname(f.file) === '.dump';++ // A database artifact is a pg_dump archive; object archives are checked by a drill.const isObjects = f.kind === 'objects' || f.file.endsWith('.tar.gz');-- if (isObjects) {-- check = 'object archive (verify with a drill)';-- } else {-- try {-- if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }-- else {-- const rows = await exec.sqlite(dest, 'PRAGMA quick_check');-- check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';-- }-- } catch (err) { check = `error: ${err.message}`; }++ let check = 'object archive (verify with a drill)';++ if (!isObjects) {++ try { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }++ catch (err) { check = `error: ${err.message}`; }}-- record.files.push({ name: f.name, kind: isObjects ? 'objects' : 'database', file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: isObjects ? null : check });-- if (!isObjects && (isDump ? check.startsWith('error') : check !== 'ok')) throw new OffsiteError(`${dest}: ${check}`);++ record.files.push({ name: f.name, kind: isObjects ? 'objects' : 'database', file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, check: isObjects ? null : check });++ if (!isObjects && check.startsWith('error')) throw new OffsiteError(`${dest}: ${check}`);}return record;}@@@ -10,7 +10,7 @@* unit whose ReadWritePaths is missing)* checks an argv run in each named package directory as the service user; non-zero stops the* deploy (Tools: the jobs runtime loads in img/audio/docs and the guard in every app, so-- * a native-module ABI mismatch shows up here, not in a crash loop; Games: better-sqlite3++ * a native-module ABI mismatch shows up here, not in a crash loop* loads under this Node)** run() never restarts or changes a unit. -> { syntaxChecked, dirsMade, checksRun, problems: [text] }@@@ -5,13 +5,10 @@* http-json-count GET a loopback URL and count: `array` (dotted path) filtered by `where`* (key: value equality), or a numeric `field` (dotted path).* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.-- * sqlite-count a read-only SELECT against the service's database, run as the service user.-- * Media: SELECT count(*) FROM vods WHERE is_recording = 1.* postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS* user over peer auth (no secret is read, and a PostgreSQL service's database has-- * no file for a sqlite-count to open). OpenRestream: ingest sessions in flight.-- * sum the total of `probes` (each an http-json-count, sqlite-count or-- * postgresql-count): Tools' running++ * no file to open). OpenRestream: ingest sessions in flight.++ * sum the total of `probes` (each an http-json-count or postgresql-count): Tools' running* jobs in each satellite's jobs.db. Unknown when any part is.** A probe that cannot answer while the service is running is UNKNOWN, never zero: "could not ask"@@@ -61,7 +58,7 @@ async function countOne(exec, svc, p, label) {const arr = at(body, p.array);if (!Array.isArray(arr)) return { count: null, label, unknown: `no array at "${p.array}"` };const where = p.where || {};-- // `true` matches any truthy value (SQLite rows say is_live: 1).++ // `true` matches any truthy value.const matches = (x) => Object.entries(where).every(([k, v]) => {const actual = x == null ? undefined : x[k];return v === true ? Boolean(actual) && actual !== '0' : actual === v;@@@ -72,13 +69,6 @@ async function countOne(exec, svc, p, label) {if (!Number.isFinite(n)) return { count: null, label, unknown: `no number at "${p.field}"` };return { count: n, label };}-- if (p.kind === 'sqlite-count') {-- const rows = await exec.sqlite(p.db, p.sql, { as: svc.runAs });-- const first = rows && rows[0] ? Object.values(rows[0])[0] : undefined;-- const n = Number(first);-- if (!Number.isFinite(n)) return { count: null, label, unknown: 'query returned no number' };-- return { count: n, label };-- }if (p.kind === 'postgresql-count') {// As the postgres OS user over peer auth: the service's own role needs a password, which// ovhost must never read.@@@ -436,7 +436,7 @@ async function apply(ctx, svc, mode, opts = {}) {if (opts.prepareOnly) throw new OpError(`--prepare-only needs a release-layout service; ${svc.id} is ${svc.strategy}`, EXIT.USAGE);// An in-place strategy never guesses its way through a releases/ + current layout.if (await exec.readlink(path.join(svc.repo, 'current'))) {-- throw new OpError(`${svc.repo} uses the releases/current layout; set strategy "release-layout" for ${svc.id} (docs/deploy-strategies.md) or use the repository's own deploy script`, EXIT.USAGE);++ throw new OpError(`${svc.repo} uses the releases/current layout; set strategy "release-layout" for ${svc.id} (docs/deploy-strategies.md)`, EXIT.USAGE);}const held = await lock.acquire(exec, inv, svc.id);if (held.recoveredStale) log(`removed a stale lock (${held.file})`);ddeleted file mode 100644@@@ -1,64 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * Runs one SQLite operation in its own process so the executor can start it AS THE SERVICE USER-- * (runuser/sudo -u). Root never opens a service's database: a root-owned -wal/-shm file left under-- * /opt would stop the service from writing its own database.-- *-- * node sqlite-worker.js '{"op":"query","db":"/opt/x/data/x.db","sql":"SELECT count(*) AS n FROM t"}'-- * node sqlite-worker.js '{"op":"backup","db":"/opt/x/data/x.db","dest":"/var/backups/openvibe/x/…/x.db"}'-- *-- * The source database is always opened read-only (fileMustExist); a query that tries to write fails.-- * Prints one JSON line: { ok: true, rows } | { ok: true, bytes } | { ok: false, error }.-- *-- * Backup copies hold user data, so the worker runs with umask 077 and leaves exactly one file,-- * mode 0600: the copy is switched to journal_mode=DELETE (a WAL source gives a WAL-mode copy), which-- * checkpoints and removes its -wal/-shm, and any side file still present is deleted.-- */--const fs = require('fs');----process.umask(0o077);----function done(obj, code = 0) {-- process.stdout.write(`${JSON.stringify(obj)}\n`);-- process.exit(code);--}----let req;--try { req = JSON.parse(process.argv[2] || ''); } catch { done({ ok: false, error: 'bad request' }, 2); }----let Database;--try { Database = require('better-sqlite3'); } catch (err) { done({ ok: false, error: `better-sqlite3 unavailable: ${err.message}` }, 2); }----let db;--try {-- db = new Database(req.db, { readonly: true, fileMustExist: true });-- db.pragma('busy_timeout = 5000');--} catch (err) {-- done({ ok: false, error: `cannot open ${req.db}: ${err.message}` }, 1);--}----if (req.op === 'query') {-- try {-- const stmt = db.prepare(req.sql);-- if (!stmt.reader) done({ ok: false, error: 'only read-only statements are allowed' }, 1);-- done({ ok: true, rows: stmt.all() });-- } catch (err) {-- done({ ok: false, error: err.message }, 1);-- }--} else if (req.op === 'backup') {-- if (!req.dest) done({ ok: false, error: 'dest required' }, 2);-- if (fs.existsSync(req.dest)) done({ ok: false, error: `refusing to overwrite ${req.dest}` }, 1);-- db.backup(req.dest).then(() => {-- const check = new Database(req.dest);-- check.pragma('journal_mode = DELETE');-- const result = check.pragma('quick_check', { simple: true });-- check.close();-- for (const side of ['-wal', '-shm', '-journal']) fs.rmSync(`${req.dest}${side}`, { force: true });-- fs.chmodSync(req.dest, 0o600);-- if (result !== 'ok') done({ ok: false, error: `backup failed quick_check: ${result}` }, 1);-- done({ ok: true, bytes: fs.statSync(req.dest).size });-- }, (err) => done({ ok: false, error: err.message }, 1));--} else {-- done({ ok: false, error: `unknown op ${req.op}` }, 2);--}@@@ -10,7 +10,6 @@"license": "AGPL-3.0-only","dependencies": {"@aws-sdk/client-s3": "^3.1138.0",-- "better-sqlite3": "^11.10.0","busboy": "^1.6.0","cookie-parser": "^1.4.7","dotenv": "^16.4.7",@@@ -487,57 +486,6 @@"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==","license": "MIT"},-- "node_modules/base64-js": {-- "version": "1.5.1",-- "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",-- "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",-- "funding": [-- {-- "type": "github",-- "url": "https://github.com/sponsors/feross"-- },-- {-- "type": "patreon",-- "url": "https://www.patreon.com/feross"-- },-- {-- "type": "consulting",-- "url": "https://feross.org/support"-- }-- ],-- "license": "MIT"-- },-- "node_modules/better-sqlite3": {-- "version": "11.10.0",-- "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-11.10.0.tgz",-- "integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==",-- "hasInstallScript": true,-- "license": "MIT",-- "dependencies": {-- "bindings": "^1.5.0",-- "prebuild-install": "^7.1.1"-- }-- },-- "node_modules/bindings": {-- "version": "1.5.0",-- "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz",-- "integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==",-- "license": "MIT",-- "dependencies": {-- "file-uri-to-path": "1.0.0"-- }-- },-- "node_modules/bl": {-- "version": "4.1.0",-- "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",-- "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",-- "license": "MIT",-- "dependencies": {-- "buffer": "^5.5.0",-- "inherits": "^2.0.4",-- "readable-stream": "^3.4.0"-- }-- },"node_modules/body-parser": {"version": "1.20.8","resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",@@@ -568,30 +516,6 @@"integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==","license": "MIT"},-- "node_modules/buffer": {-- "version": "5.7.1",-- "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",-- "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",-- "funding": [-- {-- "type": "github",-- "url": "https://github.com/sponsors/feross"-- },-- {-- "type": "patreon",-- "url": "https://www.patreon.com/feross"-- },-- {-- "type": "consulting",-- "url": "https://feross.org/support"-- }-- ],-- "license": "MIT",-- "dependencies": {-- "base64-js": "^1.3.1",-- "ieee754": "^1.1.13"-- }-- },"node_modules/buffer-equal-constant-time": {"version": "1.0.1","resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",@@@ -647,12 +571,6 @@"url": "https://github.com/sponsors/ljharb"}},-- "node_modules/chownr": {-- "version": "1.1.4",-- "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",-- "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",-- "license": "ISC"-- },"node_modules/cluster-key-slot": {"version": "1.1.2","resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",@@@ -720,30 +638,6 @@"ms": "2.0.0"}},-- "node_modules/decompress-response": {-- "version": "6.0.0",-- "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",-- "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",-- "license": "MIT",-- "dependencies": {-- "mimic-response": "^3.1.0"-- },-- "engines": {-- "node": ">=10"-- },-- "funding": {-- "url": "https://github.com/sponsors/sindresorhus"-- }-- },-- "node_modules/deep-extend": {-- "version": "0.6.0",-- "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",-- "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",-- "license": "MIT",-- "engines": {-- "node": ">=4.0.0"-- }-- },"node_modules/denque": {"version": "2.1.0","resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",@@@ -772,15 +666,6 @@"npm": "1.2.8000 || >= 1.4.16"}},-- "node_modules/detect-libc": {-- "version": "2.1.2",-- "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",-- "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",-- "license": "Apache-2.0",-- "engines": {-- "node": ">=8"-- }-- },"node_modules/dotenv": {"version": "16.6.1","resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz",@@@ -831,15 +716,6 @@"node": ">= 0.8"}},-- "node_modules/end-of-stream": {-- "version": "1.4.5",-- "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",-- "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",-- "license": "MIT",-- "dependencies": {-- "once": "^1.4.0"-- }-- },"node_modules/es-define-property": {"version": "1.0.1","resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",@@@ -885,15 +761,6 @@"node": ">= 0.6"}},-- "node_modules/expand-template": {-- "version": "2.0.3",-- "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",-- "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",-- "license": "(MIT OR WTFPL)",-- "engines": {-- "node": ">=6"-- }-- },"node_modules/express": {"version": "4.22.3","resolved": "https://registry.npmjs.org/express/-/express-4.22.3.tgz",@@@ -977,12 +844,6 @@],"license": "BSD-3-Clause"},-- "node_modules/file-uri-to-path": {-- "version": "1.0.0",-- "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz",-- "integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==",-- "license": "MIT"-- },"node_modules/finalhandler": {"version": "1.3.2","resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz",@@@ -1019,12 +880,6 @@"node": ">= 0.6"}},-- "node_modules/fs-constants": {-- "version": "1.0.0",-- "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",-- "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",-- "license": "MIT"-- },"node_modules/function-bind": {"version": "1.1.2","resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",@@@ -1071,12 +926,6 @@"node": ">= 0.4"}},-- "node_modules/github-from-package": {-- "version": "0.0.0",-- "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",-- "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",-- "license": "MIT"-- },"node_modules/gopd": {"version": "1.2.0","resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",@@@ -1145,38 +994,12 @@"node": ">=0.10.0"}},-- "node_modules/ieee754": {-- "version": "1.2.1",-- "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",-- "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",-- "funding": [-- {-- "type": "github",-- "url": "https://github.com/sponsors/feross"-- },-- {-- "type": "patreon",-- "url": "https://www.patreon.com/feross"-- },-- {-- "type": "consulting",-- "url": "https://feross.org/support"-- }-- ],-- "license": "BSD-3-Clause"-- },"node_modules/inherits": {"version": "2.0.4","resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz","integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==","license": "ISC"},-- "node_modules/ini": {-- "version": "1.3.8",-- "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",-- "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",-- "license": "ISC"-- },"node_modules/iovalkey": {"version": "0.4.0","resolved": "https://registry.npmjs.org/iovalkey/-/iovalkey-0.4.0.tgz",@@@ -1407,45 +1230,12 @@"node": ">= 0.6"}},-- "node_modules/mimic-response": {-- "version": "3.1.0",-- "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",-- "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",-- "license": "MIT",-- "engines": {-- "node": ">=10"-- },-- "funding": {-- "url": "https://github.com/sponsors/sindresorhus"-- }-- },-- "node_modules/minimist": {-- "version": "1.2.8",-- "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",-- "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",-- "license": "MIT",-- "funding": {-- "url": "https://github.com/sponsors/ljharb"-- }-- },-- "node_modules/mkdirp-classic": {-- "version": "0.5.3",-- "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",-- "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",-- "license": "MIT"-- },"node_modules/ms": {"version": "2.0.0","resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz","integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==","license": "MIT"},-- "node_modules/napi-build-utils": {-- "version": "2.0.0",-- "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",-- "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",-- "license": "MIT"-- },"node_modules/negotiator": {"version": "0.6.3","resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",@@@ -1455,18 +1245,6 @@"node": ">= 0.6"}},-- "node_modules/node-abi": {-- "version": "3.96.0",-- "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz",-- "integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==",-- "license": "MIT",-- "dependencies": {-- "semver": "^7.3.5"-- },-- "engines": {-- "node": ">=10"-- }-- },"node_modules/object-inspect": {"version": "1.13.4","resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",@@@ -1491,15 +1269,6 @@"node": ">= 0.8"}},-- "node_modules/once": {-- "version": "1.4.0",-- "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",-- "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",-- "license": "ISC",-- "dependencies": {-- "wrappy": "1"-- }-- },"node_modules/openvibe-contracts": {"version": "0.127.0","resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.127.0",@@@ -1700,33 +1469,6 @@"node": ">=0.10.0"}},-- "node_modules/prebuild-install": {-- "version": "7.1.3",-- "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",-- "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",-- "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",-- "license": "MIT",-- "dependencies": {-- "detect-libc": "^2.0.0",-- "expand-template": "^2.0.3",-- "github-from-package": "0.0.0",-- "minimist": "^1.2.3",-- "mkdirp-classic": "^0.5.3",-- "napi-build-utils": "^2.0.0",-- "node-abi": "^3.3.0",-- "pump": "^3.0.0",-- "rc": "^1.2.7",-- "simple-get": "^4.0.0",-- "tar-fs": "^2.0.0",-- "tunnel-agent": "^0.6.0"-- },-- "bin": {-- "prebuild-install": "bin.js"-- },-- "engines": {-- "node": ">=10"-- }-- },"node_modules/proxy-addr": {"version": "2.0.8","resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",@@@ -1744,16 +1486,6 @@"url": "https://opencollective.com/express"}},-- "node_modules/pump": {-- "version": "3.0.4",-- "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",-- "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",-- "license": "MIT",-- "dependencies": {-- "end-of-stream": "^1.1.0",-- "once": "^1.3.1"-- }-- },"node_modules/qs": {"version": "6.16.0","resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",@@@ -1794,35 +1526,6 @@"node": ">= 0.8"}},-- "node_modules/rc": {-- "version": "1.2.8",-- "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",-- "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",-- "license": "(BSD-2-Clause OR MIT OR Apache-2.0)",-- "dependencies": {-- "deep-extend": "^0.6.0",-- "ini": "~1.3.0",-- "minimist": "^1.2.0",-- "strip-json-comments": "~2.0.1"-- },-- "bin": {-- "rc": "cli.js"-- }-- },-- "node_modules/readable-stream": {-- "version": "3.6.2",-- "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",-- "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",-- "license": "MIT",-- "dependencies": {-- "inherits": "^2.0.3",-- "string_decoder": "^1.1.1",-- "util-deprecate": "^1.0.1"-- },-- "engines": {-- "node": ">= 6"-- }-- },"node_modules/redis-errors": {"version": "1.2.0","resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",@@@ -2014,51 +1717,6 @@"url": "https://github.com/sponsors/ljharb"}},-- "node_modules/simple-concat": {-- "version": "1.0.1",-- "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz",-- "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==",-- "funding": [-- {-- "type": "github",-- "url": "https://github.com/sponsors/feross"-- },-- {-- "type": "patreon",-- "url": "https://www.patreon.com/feross"-- },-- {-- "type": "consulting",-- "url": "https://feross.org/support"-- }-- ],-- "license": "MIT"-- },-- "node_modules/simple-get": {-- "version": "4.0.1",-- "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz",-- "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==",-- "funding": [-- {-- "type": "github",-- "url": "https://github.com/sponsors/feross"-- },-- {-- "type": "patreon",-- "url": "https://www.patreon.com/feross"-- },-- {-- "type": "consulting",-- "url": "https://feross.org/support"-- }-- ],-- "license": "MIT",-- "dependencies": {-- "decompress-response": "^6.0.0",-- "once": "^1.3.1",-- "simple-concat": "^1.0.0"-- }-- },"node_modules/split2": {"version": "4.2.0","resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",@@@ -2091,52 +1749,6 @@"node": ">=10.0.0"}},-- "node_modules/string_decoder": {-- "version": "1.3.0",-- "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",-- "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",-- "license": "MIT",-- "dependencies": {-- "safe-buffer": "~5.2.0"-- }-- },-- "node_modules/strip-json-comments": {-- "version": "2.0.1",-- "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",-- "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",-- "license": "MIT",-- "engines": {-- "node": ">=0.10.0"-- }-- },-- "node_modules/tar-fs": {-- "version": "2.1.5",-- "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",-- "integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",-- "license": "MIT",-- "dependencies": {-- "chownr": "^1.1.1",-- "mkdirp-classic": "^0.5.2",-- "pump": "^3.0.0",-- "tar-stream": "^2.1.4"-- }-- },-- "node_modules/tar-stream": {-- "version": "2.2.0",-- "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",-- "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",-- "license": "MIT",-- "dependencies": {-- "bl": "^4.0.3",-- "end-of-stream": "^1.4.1",-- "fs-constants": "^1.0.0",-- "inherits": "^2.0.3",-- "readable-stream": "^3.1.1"-- },-- "engines": {-- "node": ">=6"-- }-- },"node_modules/toidentifier": {"version": "1.0.1","resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",@@@ -2152,18 +1764,6 @@"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==","license": "0BSD"},-- "node_modules/tunnel-agent": {-- "version": "0.6.0",-- "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",-- "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",-- "license": "Apache-2.0",-- "dependencies": {-- "safe-buffer": "^5.0.1"-- },-- "engines": {-- "node": "*"-- }-- },"node_modules/type-is": {"version": "1.6.18","resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",@@@ -2186,12 +1786,6 @@"node": ">= 0.8"}},-- "node_modules/util-deprecate": {-- "version": "1.0.2",-- "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",-- "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",-- "license": "MIT"-- },"node_modules/utils-merge": {"version": "1.0.1","resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",@@@ -2210,12 +1804,6 @@"node": ">= 0.8"}},-- "node_modules/wrappy": {-- "version": "1.0.2",-- "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",-- "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",-- "license": "ISC"-- },"node_modules/xtend": {"version": "4.0.2","resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",@@@ -26,7 +26,6 @@"repository": "github:OpenVibers/OpenVibe.Host","dependencies": {"@aws-sdk/client-s3": "^3.1138.0",-- "better-sqlite3": "^11.10.0","busboy": "^1.6.0","cookie-parser": "^1.4.7","dotenv": "^16.4.7",@@@ -17,21 +17,9 @@ sudo roles/data/provision.sh --check # what would changesudo roles/data/provision.sh # apply (idempotent)sudo roles/data/add-service.sh wiki # database ov_wiki, roles ov_wiki / ov_wiki_app, Valkey user ov_svc_wiki,# and DATABASE_URL, DATABASE_DIRECT_URL, VALKEY_URL, VALKEY_PREFIX in /etc/openvibe/wiki.env--sudo roles/data/switch-service.sh news # the production switch from SQLite: stop, back up, import and verify, deploy-- # (after the postgres branch is merged with green CI and auto-deploy is frozen)```--The same two steps from the installed CLI (the agent pipeline may run only `ovhost` through its broker;--both run these scripts from the ovhost install, pass their exit code through and redact every URL and--password; [docs/data-provisioning.md](../../docs/data-provisioning.md)):----```bash--sudo ovhost data provision wiki # == sudo roles/data/add-service.sh wiki--sudo ovhost data switch news [--sqlite <file>] # == sudo roles/data/switch-service.sh news [<file>]--```----`switch-service.sh` runs the deployed release's `scripts/migrate-to-postgres.js`: a service without that--file cannot be switched this way.++The installed CLI runs `add-service.sh` through `sudo ovhost data provision wiki`. It passes the role's exit code and redacts URLs and passwords; see [data provisioning](../../docs/data-provisioning.md).Generated passwords live only in `/etc/openvibe/data.env` (root, 0600). PostgreSQL receives SCRAM secrets andValkey receives SHA-256 hashes, so no plaintext is ever in SQL, logs, a process list or the scripts' output.ddeleted file mode 100755@@@ -1,62 +0,0 @@--#!/usr/bin/env bash--# roles/data/switch-service.sh — the production switch of one service from its SQLite file to PostgreSQL (ADR-035).--#--# sudo roles/data/switch-service.sh <service> [<sqlite file>]--#--# Before: the service's postgres branch is merged to main with green CI, its auto-deploy is frozen--# (ovhost freeze <svc> --reason …), and add-service.sh <svc> has written its database settings.--# It clones main as ubuntu into /var/tmp/pg-<svc> and installs it; waits until the service's protected sessions are--# idle (two checks a minute apart); stops the unit; keeps a read-only--# <svc>.pre-postgres-<time>.db backup next to the file; runs that release's scripts/migrate-to-postgres.js--# (openvibe-sdk runSqliteMigration: migrate, import, verify counts and checksums) with the unit's--# environment, as ubuntu; on any failure it starts the old release again and stops. Then it unfreezes and--# deploys (ovhost deploy), and makes the SQLite file read-only: it is the 7-day rollback (compatibility--# register C-89: sudo ovhost rollback <svc> to the release before, which reads the file).--#--# A service that is not one unit in /opt/openvibe.<svc> (OpenRestream: two units plus per-release transport workers, a--# release layout) sets, in the environment:--# SWITCH_UNITS="a.service b.service 'worker@*.service'" every unit to stop (the first one's environment runs the--# import); a pattern stops every running instance--# SWITCH_DIR=/opt/openre.stream the checkout (a release layout's origin is in repo/)--# SWITCH_AFTER="/opt/openre.stream/current/deploy/scripts/deploy.sh workers" run after the deploy (as root)--set -euo pipefail--SVC=$1; UNIT=openvibe-$SVC.service; DIR=${SWITCH_DIR:-/opt/openvibe.$SVC}; DATA=/var/lib/openvibe-$SVC; DB=$DATA/$SVC.db--read -r -a UNITS <<< "${SWITCH_UNITS:-$UNIT}"; UNIT=${UNITS[0]}--# A second argument names the SQLite file when it is not /var/lib/openvibe-<svc>/<svc>.db (Host: /var/lib/openvibe-host-api/host.db).--if [ -n "${2:-}" ]; then DB=$2; DATA=$(dirname "$DB"); fi--WORK=/var/tmp/pg-$SVC; UB=$(id -u ubuntu); GB=$(id -g ubuntu)--[ -f "$DB" ] || { echo "no $DB"; exit 1; }--rm -rf "$WORK"; install -d -o ubuntu -g ubuntu "$WORK"--GITDIR=$DIR; [ -d "$DIR/repo/.git" ] || [ -f "$DIR/repo/HEAD" ] && GITDIR=$DIR/repo--REMOTE=$(git -C "$GITDIR" remote get-url origin)--setpriv --reuid=$UB --regid=$GB --init-groups env HOME=/home/ubuntu bash -c "cd $WORK && git clone -q --depth 1 --branch main '$REMOTE' src && cd src && npm ci --omit=dev --no-audit --no-fund 2>&1 | tail -1 && git log --oneline -1"--# Protected sessions (the inventory's probe: Media's recordings, Live's streams) would be cut by the stop: wait for two--# idle checks a minute apart, as a deploy's --wait-idle does, for up to 12 hours. A probe that cannot answer counts as busy.--quiet=0--for _ in $(seq 1 720); do-- n=$(ovhost status "$SVC" --json | node -e 'const r = JSON.parse(require("fs").readFileSync(0, "utf8"))[0] || {}; const p = r.protected; process.stdout.write(!p ? "none" : p.count == null ? "unknown" : String(p.count));')-- [ "$n" = none ] && break-- if [ "$n" = 0 ]; then quiet=$((quiet + 1)); [ $quiet -ge 2 ] && break; else quiet=0; echo "[switch] waiting: $n protected session(s)"; fi-- sleep 60--done--[ "$n" = none ] || [ $quiet -ge 2 ] || { echo "[switch] still busy after 12 hours: not switching"; rm -rf "$WORK"; exit 1; }--echo "[switch] stopping ${UNITS[*]}"; systemctl stop "${UNITS[@]}"--sqlite3 "$DB" ".backup $WORK/source.db"; chown ubuntu:ubuntu "$WORK/source.db"--TS=$(date -u +%Y%m%dT%H%M%SZ); cp "$WORK/source.db" "$DATA/$SVC.pre-postgres-$TS.db"; chmod 0400 "$DATA/$SVC.pre-postgres-$TS.db"; echo "[switch] backup $DATA/$SVC.pre-postgres-$TS.db"--echo "[switch] importing"--# The unit's own environment, parsed by systemd (EnvironmentFile= and Environment=): sourcing the file with bash would--# mangle values systemd reads fine (JSON, quotes). As ubuntu, in the release just installed.--SETENV=(); while IFS= read -r kv; do [ -n "$kv" ] && SETENV+=(-E "$kv"); done < <(systemctl show "$UNIT" -p Environment --value | tr ' ' '\n')--ENVFILES=(); for f in $(systemctl show "$UNIT" -p EnvironmentFiles --value | grep -o '^[^ ]*\|; [^ ]*' | tr -d '; '); do ENVFILES+=(-p "EnvironmentFile=$f"); done--[ ${#ENVFILES[@]} -gt 0 ] || ENVFILES=(-p "EnvironmentFile=/etc/openvibe/$SVC.env")--if ! systemd-run --quiet --wait --pipe --collect --uid="$UB" --gid="$GB" -p WorkingDirectory="$WORK/src" -E HOME=/home/ubuntu "${ENVFILES[@]}" "${SETENV[@]}" \-- "$(command -v node)" scripts/migrate-to-postgres.js --sqlite "$WORK/source.db" 2>&1 | grep -v " 0 rows"; then-- echo "[switch] IMPORT FAILED: starting the old release again"; systemctl start "${UNITS[@]}" 2>/dev/null || systemctl start "$UNIT"; exit 1--fi--echo "[switch] deploying"--ovhost unfreeze "$SVC" >/dev/null--ovhost deploy "$SVC" 2>&1 | tail -3--if [ -n "${SWITCH_AFTER:-}" ]; then echo "[switch] $SWITCH_AFTER"; bash -c "$SWITCH_AFTER" 2>&1 | tail -5; fi--chmod 0400 "$DB"; [ -f "$DB-wal" ] && chmod 0400 "$DB-wal"; [ -f "$DB-shm" ] && chmod 0400 "$DB-shm"--rm -rf "$WORK"--echo "[switch] done"ddeleted file mode 100644@@@ -1,26 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * The one-time move of Host's SQLite database (HOST_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is-- * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).-- *-- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]-- *-- * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts-- * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its-- * name and columns.-- */--require('dotenv').config();--const { runSqliteMigration } = require('openvibe-sdk/db');--const configLib = require('../server/config');--const { MIGRATIONS } = require('../server/db');----const TABLES = {};----if (require.main === module) {-- const config = (configLib.load || configLib.loadConfig)();-- runSqliteMigration({ service: 'host', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })-- .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });--}----module.exports = { TABLES };@@@ -69,7 +69,7 @@ const DASHBOARD_CSP = {const DASHBOARD_CSP_HEADER = Object.entries(DASHBOARD_CSP).map(([k, v]) => `${k} ${v.join(' ')}`).join('; ');/**-- * opts: config, store | dbPath, now (clock), fetchImpl, auth (an openvibe-sdk/sso client),++ * opts: config, store, now (clock), fetchImpl, auth (an openvibe-sdk/sso client),* resolver ({ resolveTxt }), log*/async function createApp(opts = {}) {@@@ -49,8 +49,6 @@ function load(env = process.env) {// point it at a temp directory so a spawned server gets a database of its own.db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.HOST_PGLITE_DIR || '' },valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:host:' },-- // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.-- dbPath: env.HOST_DB_PATH || './data/host.db',// Content-addressed deploy objects, one directory per project (tenancy keyed by project id).storageDir: env.HOST_STORAGE_DIR || './data/objects',@@@ -55,7 +55,7 @@ async function openDb(config, { log = console, registry } = {}) {}/**-- * Every store on a migrated database handle. opts.now — injectable clock (epoch ms), so tests and replays are++ * Every store on a database handle. opts.now — injectable clock (epoch ms), so tests and replays are* deterministic. store.tx(fn) is a transaction; inside it, plain db calls join it (ambient).*/function createStore(db, { now = () => Date.now() } = {}) {@@@ -15,7 +15,7 @@ const { app, ctx } = await createApp();const { config } = ctx;const server = app.listen(config.port, config.host, () => {-- console.log(`[Host] ${config.nodeEnv} on http://${config.host}:${config.port} → dashboard ${config.baseUrl}, sites *.${config.sitesDomain} (db ${config.dbPath}, objects ${ctx.blobs.root})`);++ console.log(`[Host] ${config.nodeEnv} on http://${config.host}:${config.port} → dashboard ${config.baseUrl}, sites *.${config.sitesDomain} (db ${config.db.url ? 'PostgreSQL' : 'embedded PGlite'}, objects ${ctx.blobs.root})`);console.log(`[Host] events relay ${ctx.outbox.enabled ? `on → ${config.events.url}` : 'off (events wait in event_outbox)'}; worker ${config.worker.enabled ? 'on' : 'off'}`);});server.keepAliveTimeout = 65_000;@@@ -2,7 +2,7 @@/*** Truthful readiness for GET /api/ready (openvibe-shared/ready, Track O).*-- * db required a real query on Host's SQLite (the charter tables answer)++ * db required a real query on Host's PostgreSQL database (the charter tables answer)* storage required the object store directory is writable (uploads and serving need it)* network_jwks optional the Network signing key has loaded; without it tenant sites still* serve, but nobody can sign in and service tokens are refused (503)@@@ -1,8 +1,7 @@'use strict';/**-- * ovhost archive (WS-S task 6, H14): dead and backup database files are encrypted and copied off-host-- * under <prefix>-archive/, checked, listed and restored, and never deleted locally. Refused: live-- * (inventory) databases, open files, symlinks, paths outside the allowed roots, relative paths.++ * ovhost archive: retired data files are encrypted and copied off-host++ * under <prefix>-archive/, checked, listed and restored, and never deleted locally. Refused: open files, symlinks, paths outside the allowed roots, relative paths.* Nothing leaves the process (an in-memory S3).*/const assert = require('assert');@@@ -11,8 +10,6 @@ const { scenario, test, runTests, SECRET } = require('./helpers');const { main } = require('../lib/cli');const { mockS3 } = require('./s3-mock');const { refusal } = require('../lib/archive');--const offsite = require('../lib/offsite');--const { normalise } = require('../lib/inventory');const KEY_HEX = crypto.createHash('sha256').update('archive test key').digest('hex');const S3_SECRET = `${SECRET}-s3`;@@@ -29,11 +26,11 @@ function archiveHost() {'',].join('\n'), { mode: 0o600, owner: 'root' });host.put('/etc/openvibe/backup.key', `${KEY_HEX}\n`, { mode: 0o600, owner: 'root' });-- host.put('/opt/backups/old/hobostreamer.db', Buffer.from(`SQLite format 3\0 old site ${'x'.repeat(9000)}`), { owner: 'root' });-- host.put('/var/lib/openvibe-chat/live-snap-a.db', Buffer.from(`SQLite format 3\0 snapshot ${'y'.repeat(3000)}`), { owner: 'root' });-- host.put('/var/lib/openvibe-chat/in-use.db', 'SQLite format 3\0 open', { owner: 'root' });-- host.put('/etc/passwd-copy.db', 'nope', { owner: 'root' });-- host.openFiles = new Set(['/var/lib/openvibe-chat/in-use.db']);++ host.put('/opt/backups/old/hobostreamer.tar', Buffer.from(`old site ${'x'.repeat(9000)}`), { owner: 'root' });++ host.put('/var/lib/openvibe-chat/live-snap-a.tar', Buffer.from(`snapshot ${'y'.repeat(3000)}`), { owner: 'root' });++ host.put('/var/lib/openvibe-chat/in-use.tar', 'open', { owner: 'root' });++ host.put('/etc/passwd-copy.tar', 'nope', { owner: 'root' });++ host.openFiles = new Set(['/var/lib/openvibe-chat/in-use.tar']);host.s3 = mockS3();host.cli = async (...argv) => {const lines = [];@@@ -48,19 +45,19 @@ const noSecrets = (text) => { for (const s of [S3_SECRET, SECRET, KEY_HEX]) asserunTests([test('push encrypts, uploads under the archive prefix, checks, records, and deletes nothing', async () => {const host = archiveHost();-- const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.db', '/var/lib/openvibe-chat/live-snap-a.db', '--note', 'H14 first pass');++ const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.tar', '/var/lib/openvibe-chat/live-snap-a.tar', '--note', 'H14 first pass');assert.strictEqual(r.code, 0, r.out);noSecrets(r.out);assert.match(r.out, /ARCHIVE ok: 2 file\(s\) at s3:\/\/openvibe-backups-test\/openvibe-backups-archive\/.+\/\d{8}-\d{6}\/.*Nothing was deleted/);const keys = [...host.s3.objects.keys()];assert.ok(keys.every((k) => k.startsWith('openvibe-backups-archive/')), 'outside the backup runs prefix (retention never prunes it)');-- const obj = keys.find((k) => k.endsWith('opt__backups__old__hobostreamer.db.ovbk'));++ const obj = keys.find((k) => k.endsWith('opt__backups__old__hobostreamer.tar.ovbk'));assert.ok(obj, keys.join(', '));assert.ok(!host.s3.objects.get(obj).includes(Buffer.from('old site')), 'the bytes are encrypted');-- assert.ok(host.read('/opt/backups/old/hobostreamer.db') !== null && host.read('/var/lib/openvibe-chat/live-snap-a.db') !== null, 'the local files stay');++ assert.ok(host.read('/opt/backups/old/hobostreamer.tar') !== null && host.read('/var/lib/openvibe-chat/live-snap-a.tar') !== null, 'the local files stay');const manifestKey = keys.find((k) => k.endsWith('/manifest.json'));const m = JSON.parse(host.s3.objects.get(manifestKey).toString('utf8'));-- assert.deepStrictEqual(m.files.map((f) => f.path), ['/opt/backups/old/hobostreamer.db', '/var/lib/openvibe-chat/live-snap-a.db']);++ assert.deepStrictEqual(m.files.map((f) => f.path), ['/opt/backups/old/hobostreamer.tar', '/var/lib/openvibe-chat/live-snap-a.tar']);assert.strictEqual(m.note, 'H14 first pass');assert.ok(m.hmac && m.files.every((f) => /^[0-9a-f]{64}$/.test(f.plainSha256)));const rec = [...host.files.keys()].find((k) => k.startsWith('/var/lib/openvibe-host/archives/'));@@@ -72,46 +69,22 @@ runTests([assert.match(list.out, /\d{8}-\d{6}\s+2 file\(s\).*H14 first pass/);const st = m.stamp;-- const back = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.db', '--out', '/var/lib/openvibe-restore/h14');++ const back = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.tar', '--out', '/var/lib/openvibe-restore/h14');assert.strictEqual(back.code, 0, back.out);-- assert.strictEqual(host.read('/var/lib/openvibe-restore/h14/hobostreamer.db'), host.read('/opt/backups/old/hobostreamer.db'), 'restored byte for byte');-- const again = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.db', '--out', '/var/lib/openvibe-restore/h14');++ assert.strictEqual(host.read('/var/lib/openvibe-restore/h14/hobostreamer.tar'), host.read('/opt/backups/old/hobostreamer.tar'), 'restored byte for byte');++ const again = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.tar', '--out', '/var/lib/openvibe-restore/h14');assert.notStrictEqual(again.code, 0, 'restore never overwrites');}),-- test('refusals: live databases, open files, symlinks, outside roots, relative paths; nothing uploaded', async () => {++ test('refusals: open files, symlinks, outside roots and relative paths', async () => {const host = archiveHost();-- const liveDb = Object.values(host.inv.services).flatMap((s) => s.databases.filter((d) => d.engine !== 'postgresql').map((d) => d.path))[0];-- assert.ok(liveDb, 'the scenario declares a database');-- host.put(liveDb, 'SQLite format 3\0 live', { owner: 'ubuntu' });-- await host.exec.symlink('/opt/backups/old/hobostreamer.db', '/opt/backups/old/link.db');-- for (const [file, why] of [['/opt/backups/old/link.db', /not a regular file/], [liveDb, /inventory declares/], ['/var/lib/openvibe-chat/in-use.db', /open by a running process/], ['/etc/passwd-copy.db', /outside/], ['relative.db', /absolute/], ['/opt/backups/../etc/x.db', /normalised/], ['/opt/backups/none.db', /no such file/]]) {-- const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.db', file);++ await host.exec.symlink('/opt/backups/old/hobostreamer.tar', '/opt/backups/old/link.tar');++ for (const [file, why] of [['/opt/backups/old/link.tar', /not a regular file/], ['/var/lib/openvibe-chat/in-use.tar', /open by a running process/], ['/etc/passwd-copy.tar', /outside/], ['relative.tar', /absolute/], ['/opt/backups/../etc/x.tar', /normalised/], ['/opt/backups/none.tar', /no such file/]]) {++ const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.tar', file);assert.strictEqual(r.code, 1, `${file}: ${r.out}`);assert.match(r.out, why, file);-- assert.strictEqual(host.s3.objects.size, 0, `${file}: nothing is uploaded when any file is refused`);++ assert.strictEqual(host.s3.objects.size, 0);}}),-- test('a PostgreSQL entry has no path, so it neither blocks archiving the file it replaced nor makes unsafeOut throw', async () => {-- const host = archiveHost();-- const raw = JSON.parse(host.read('/etc/openvibe/host.json'));-- // Media moved to PostgreSQL (C-89): its databases[] entry names a database in the cluster.-- raw.services.media.databases = [{ name: 'media', engine: 'postgresql', database: 'ov_media' }];-- const inv = normalise(raw);-- const frozen = '/opt/openvibe.media/data/media.db'; // Media's old SQLite file, frozen at the switch-- host.put(frozen, 'SQLite format 3\0 frozen', { owner: 'ubuntu' });-- assert.strictEqual(await refusal(host.exec, inv, frozen), null, 'the frozen SQLite file is archivable again');---- // A database the inventory still declares as SQLite is refused, message unchanged.-- const live = inv.services.live.databases[0];-- assert.strictEqual(live.engine, 'sqlite');-- host.put(live.path, 'SQLite format 3\0 live', { owner: 'ubuntu' });-- assert.match(await refusal(host.exec, inv, live.path), /a database the inventory declares \(a live database\)/);---- // unsafeOut only avoids database directories; a database in the cluster has none.-- let out;-- assert.doesNotThrow(() => { out = offsite.unsafeOut(inv, '/var/lib/openvibe-restore/media-2026'); });-- assert.strictEqual(out, null);-- }),]);@@@ -26,12 +26,12 @@ const TENANT_BLOB = 'tenant object bytes\n';function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {const host = scenario();const doc = JSON.parse(host.read('/etc/openvibe/host.json'));-- doc.services.ai = { repo: '/opt/openvibe.ai', units: [], databases: [{ name: 'ai', path: '/var/lib/openvibe-ai/ai.db' }, { name: 'ai-extra', path: '/var/lib/openvibe-ai/extra.db' }] };-- doc.services.openre = { repo: '/opt/openre', units: [], databases: [{ name: 'openre', path: '/var/lib/openre/openre.db' }] };++ doc.services.ai = { repo: '/opt/openvibe.ai', units: [], databases: [{ name: 'ai', engine: 'postgresql', database: 'ov_ai' }, { name: 'ai-extra', engine: 'postgresql', database: 'ov_ai_extra' }] };++ doc.services.openre = { repo: '/opt/openre', units: [], databases: [{ name: 'openre', engine: 'postgresql', database: 'ov_openre' }] };host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });host.inv = normalise(doc);-- host.put('/var/lib/openvibe-ai/ai.db', 'sqlite', { owner: 'ubuntu' });-- host.put('/var/lib/openvibe-ai/extra.db', 'sqlite', { owner: 'ubuntu' });++ for (const db of ['ov_ai', 'ov_ai_extra', 'ov_openre']) host.pgDatabases.add(db);++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];host.put('/etc/openvibe/backup.env', env || ['# off-host backups','BACKUP_S3_ENDPOINT=https://s3.us-west-004.backblazeb2.com',@@@ -43,9 +43,6 @@ function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env ='',].join('\n'), { mode: envMode, owner: envOwner });host.put('/etc/openvibe/backup.key', `${KEY_HEX}\n`, { mode: keyMode, owner: 'root' });-- // Distinct content per database, so a restore can be compared byte for byte.-- host.backupContent = (db) => Buffer.from(`SQLite format 3\0 copy of ${db} ${'x'.repeat(5000)}`);-- host.sqliteHandler = (db, sql) => (/quick_check/.test(sql) ? [{ quick_check: 'ok' }] : [{ n: 0 }]);host.s3 = mockS3();host.s3Configs = [];host.cli = async (...argv) => {@@@ -169,149 +166,32 @@ runTests([assert.deepStrictEqual([...retention.keep(['junk', '20260101-000000'])], ['20260101-000000']);}),-- test('backup --all prunes to 7 daily + 4 weekly per service; failing days never age out good backups', async () => {-- const host = backupHost();-- // A directory ovhost did not record is never touched.-- host.put('/var/backups/openvibe/live/20200101-000000/live.db', 'operator copy', { owner: 'root', mode: 0o600 });-- for (let d = 0; d < 40; d++) {-- const r = await host.cli('backup', '--all');-- assert.strictEqual(r.code, 0, r.out);-- host.advance(DAY);-- }-- // 40 daily runs from 2026-09-22 12:00 to 2026-10-31 12:00.-- const expected = ['20200101-000000', '20261011-120000', '20261018-120000', '20261025-120000', '20261026-120000', '20261027-120000', '20261028-120000', '20261029-120000', '20261030-120000', '20261031-120000'];-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), expected);-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/media'), expected.slice(1));---- // Live fails for 20 days: nothing of Live is pruned, the others carry on.-- host.onSqliteBackup = (db) => (db.includes('openvibe.live') ? new Error('database disk image is malformed') : null);-- for (let d = 0; d < 20; d++) {-- const r = await host.cli('backup', '--all');-- assert.strictEqual(r.code, 2);-- host.advance(DAY);-- }-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), expected);-- assert.strictEqual(dirsOf(host, '/var/backups/openvibe/media').length, 9);-- assert.ok(dirsOf(host, '/var/backups/openvibe/media').includes('20261119-120000'));-- }),---- test('a partly failed backup is kept until a newer good one exists, then pruned; drills skip it', async () => {-- const host = backupHost();-- await host.cli('backup', '--all');-- host.advance(DAY);-- host.onSqliteBackup = (db) => (db.endsWith('extra.db') ? new Error('disk I/O error') : null);-- const bad = await host.cli('backup', '--all', '--json');-- assert.strictEqual(bad.code, 2);-- const s = JSON.parse(bad.out);-- const ai = s.services.find((x) => x.service === 'ai');-- assert.strictEqual(ai.status, 'failed');-- assert.match(ai.error, /ai-extra: disk I\/O error/);-- assert.deepStrictEqual(dirsOf(host, ai.dir), ['ai.db'], 'the good half is kept');-- const log = host.read('/var/lib/openvibe-host/backups/ai.jsonl').trim().split('\n').map((l) => JSON.parse(l));-- assert.strictEqual(log[log.length - 1].ok, false);-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/ai'), ['20260922-120000', '20260923-120000']);-- host.onSqliteBackup = null;-- host.advance(DAY);-- await host.cli('backup', '--all');-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/ai'), ['20260922-120000', '20260924-120000'], 'the failed one went once a newer good one existed');-- }),---- test('backup --all leaves its outcome for Prometheus (textfile collector): last run, ok, last success, per-status counts', async () => {-- const host = backupHost();-- const PROM = '/var/lib/prometheus/node-exporter/openvibe_backup.prom';-- await host.cli('backup', '--all');-- assert.strictEqual(host.files.has(PROM), false, 'no collector directory: nothing written');-- host.put('/var/lib/prometheus/node-exporter/.keep', '');-- host.advance(DAY);-- await host.cli('backup', '--all');-- const metric = (name) => { const m = new RegExp(`^${name} (\\S+)$`, 'm').exec(host.read(PROM)); return m ? Number(m[1]) : null; };-- const goodAt = metric('openvibe_backup_last_run_timestamp_seconds');-- assert.ok(goodAt > 0);-- assert.strictEqual(metric('openvibe_backup_last_run_ok'), 1);-- assert.strictEqual(metric('openvibe_backup_last_success_timestamp_seconds'), goodAt);-- assert.strictEqual(metric('openvibe_backup_services\\{status="failed"\\}'), 0);-- host.advance(DAY);-- host.onSqliteBackup = (db) => (db.endsWith('extra.db') ? new Error('disk I/O error') : null);-- await host.cli('backup', '--all');-- host.onSqliteBackup = null;-- assert.strictEqual(metric('openvibe_backup_last_run_ok'), 0);-- assert.strictEqual(metric('openvibe_backup_services\\{status="failed"\\}'), 1);-- assert.ok(metric('openvibe_backup_last_run_timestamp_seconds') > goodAt);-- assert.strictEqual(metric('openvibe_backup_last_success_timestamp_seconds'), goodAt, 'the last success is still the good run');-- host.files.delete(PROM);-- const r = await host.cli('backup-metrics');-- assert.strictEqual(r.code, 0, r.out);-- assert.strictEqual(metric('openvibe_backup_last_run_ok'), 0, 'rewritten from the latest recorded run');-- assert.strictEqual(metric('openvibe_backup_last_success_timestamp_seconds'), goodAt);-- }),---- // ── failure isolation and the summary ─────────────────────────────────────-- test('one failing service never stops the others; exit code 2; JSON summary per run', async () => {-- const host = backupHost();-- host.onSqliteBackup = (db) => (db.includes('openvibe.media') ? new Error('SQLITE_BUSY: database is locked') : null);-- const r = await host.cli('backup', '--all');-- assert.strictEqual(r.code, 2, r.out);-- assert.match(r.out, /FAILED\s+media\s+media: SQLITE_BUSY/);-- assert.match(r.out, /OK\s+live\s+1 file\(s\)/);-- assert.match(r.out, /OK\s+ai\s+2 file\(s\)/);-- assert.match(r.out, /SKIPPED\s+openre\s+no database found/);-- const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));-- assert.strictEqual(summary.ok, false);-- assert.deepStrictEqual(summary.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'failed'], ['ai', 'ok'], ['openre', 'skipped']]);-- assert.deepStrictEqual(summary.retention, { daily: 7, weekly: 4 });-- // The services after the failing one were still backed up.-- assert.ok(host.files.has('/var/backups/openvibe/ai/20260922-120000/ai-extra.db'));-- assert.strictEqual(host.files.has('/var/backups/openvibe/openre'), true);-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/openre'), [], 'no empty backup directory is left behind');-- assert.strictEqual(host.files.has('/var/lib/openvibe-host/locks/_backup-all.lock'), false, 'lock released');-- const ok = await (host.advance(1000), host.cli('backup', '--all', '--json'));-- host.onSqliteBackup = null;-- assert.strictEqual(JSON.parse(ok.out).ok, false);-- host.advance(1000);-- assert.strictEqual((await host.cli('backup', '--all')).code, 0);-- }),---- test('permissions: backup directories root 0700, copies root 0600, no -wal/-shm; old service-owned directories are locked down', async () => {++ test('PostgreSQL dump copies are root-only and staging is emptied', async () => {const host = backupHost();host.ensureDir('/var/backups/openvibe/live', 'ubuntu', 0o750);-- // The worker leaves a side file behind: it is not kept.-- host.onSqliteBackup = (db, dest) => { host.put(`${dest}-wal`, 'wal', { owner: 'ubuntu', mode: 0o644 }); return null; };const r = await host.cli('backup', '--all', '--offsite');assert.strictEqual(r.code, 0, r.out);-- let dirs = 0;-- let files = 0;-- for (const [p, e] of host.files) {-- if (p !== '/var/backups/openvibe' && !p.startsWith('/var/backups/openvibe/')) continue;-- if (e.type === 'dir') { dirs += 1; assert.deepStrictEqual([e.owner, e.mode], ['root', 0o700], p); }-- else { files += 1; assert.deepStrictEqual([e.owner, e.mode], ['root', 0o600], p); assert.ok(!/-(wal|shm|journal)$/.test(p), `${p} left behind`); }-- }-- assert.ok(dirs >= 8 && files === 4, `${dirs} dirs, ${files} files`);-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe.staging'), [], 'staging emptied');-- assert.deepStrictEqual([host.files.get('/var/backups/openvibe.staging').owner, host.files.get('/var/backups/openvibe.staging').mode], ['root', 0o711]);-- // The worker still ran as the service user, into a directory it owned at the time.-- assert.ok(host.sqliteCalls.filter((c) => c.op === 'backup').every((c) => c.as === 'ubuntu' && c.dest.startsWith('/var/backups/openvibe.staging/')));---- // restore-download: a root-only directory, 0600 files.-- const d = await host.cli('restore-download', 'ai', 'latest');-- assert.strictEqual(d.code, 0, d.out);-- for (const p of ['/var/lib/openvibe-restore', '/var/lib/openvibe-restore/ai-20260922-120000']) assert.deepStrictEqual([host.files.get(p).owner, host.files.get(p).mode], ['root', 0o700], p);-- for (const f of ['ai.db', 'ai-extra.db']) {-- const e = host.files.get(`/var/lib/openvibe-restore/ai-20260922-120000/${f}`);-- assert.deepStrictEqual([e.owner, e.mode], ['root', 0o600], f);++ for (const [file, entry] of host.files) {++ if (!file.startsWith('/var/backups/openvibe/')) continue;++ assert.deepStrictEqual([entry.owner, entry.mode], ['root', entry.type === 'dir' ? 0o700 : 0o600], file);}++ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe.staging'), []);++ assert.ok(host.dumps.every((d) => d.as === 'postgres'));++ const restored = await host.cli('restore-download', 'ai', 'latest');++ assert.strictEqual(restored.code, 0, restored.out);++ for (const file of ['ai.dump', 'ai-extra.dump']) assert.strictEqual(host.files.get(`/var/lib/openvibe-restore/ai-20260922-120000/${file}`).mode, 0o600);}),-- test('a copy that is a link is refused, not chowned', async () => {++ test('a failed PostgreSQL dump does not prevent another service from backing up', async () => {const host = backupHost();-- // The service user hard-links its "copy" to some other file before root takes it over.-- const orig = host.exec.sqliteBackup;-- host.exec.sqliteBackup = async (db, dest, o) => { await orig(db, dest, o); host.files.get(dest).nlink = 2; };-- const r = await host.cli('backup', 'live');-- host.exec.sqliteBackup = orig;-- assert.strictEqual(r.code, 2);-- assert.match(r.out, /not a plain file/);-- assert.ok(!host.calls.some((c) => (c.cmd === 'chown' || c.cmd === 'chmod') && String(c.args[c.args.length - 1]).endsWith('live.db')), 'a linked copy is never chowned');-- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), [], 'and never moved into the backups');++ const orig = host.exec.run;++ host.exec.run = async (cmd, args, opts) => cmd === 'pg_dump' && args.includes('ov_media')++ ? { code: 1, stdout: '', stderr: 'simulated pg_dump failure' } : orig(cmd, args, opts);++ const r = await host.cli('backup', '--all', '--json');++ assert.strictEqual(r.code, 2, r.out);++ const summary = JSON.parse(r.out);++ assert.deepStrictEqual(summary.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'failed'], ['ai', 'ok'], ['openre', 'ok']]);++ assert.ok(host.files.has('/var/backups/openvibe/ai/20260922-120000/ai-extra.dump'));}),// ── off-host upload ────────────────────────────────────────────────────────@@@ -320,7 +200,7 @@ runTests([const r = await host.cli('backup', '--all', '--offsite');assert.strictEqual(r.code, 0, r.out);const base = 'openvibe-backups/fake-host/20260922-120000/';-- assert.deepStrictEqual([...host.s3.objects.keys()].sort(), [`${base}ai/ai-extra.db.ovbk`, `${base}ai/ai.db.ovbk`, `${base}live/live.db.ovbk`, `${base}manifest.json`, `${base}media/media.db.ovbk`]);++ assert.deepStrictEqual([...host.s3.objects.keys()].sort(), [`${base}ai/ai-extra.dump.ovbk`, `${base}ai/ai.dump.ovbk`, `${base}live/live.dump.ovbk`, `${base}manifest.json`, `${base}media/media.dump.ovbk`, `${base}openre/openre.dump.ovbk`]);const key = Buffer.from(KEY_HEX, 'hex');for (const [k, v] of host.s3.objects) {if (k.endsWith('manifest.json')) continue;@@@ -328,12 +208,12 @@ runTests([const plain = await decrypt(key, v);const svc = k.split('/')[3];const file = path.basename(k, '.ovbk');-- assert.ok(plain.equals(host.files.get(`/var/backups/openvibe/${svc}/20260922-120000/${file}`).content), `${k} round trip`);++ assert.ok(plain.equals(Buffer.from(host.files.get(`/var/backups/openvibe/${svc}/20260922-120000/${file}`).content)), `${k} round trip`);}const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));assert.strictEqual(m.hmac, bcrypto.hmacManifest(key, JSON.stringify((({ hmac, ...rest }) => rest)(m))));assert.strictEqual(m.keyId, bcrypto.keyId(key));-- assert.deepStrictEqual(Object.keys(m.services), ['live', 'media', 'ai']);++ assert.deepStrictEqual(Object.keys(m.services), ['live', 'media', 'ai', 'openre']);// Only PutObject for small files; path-style B2 endpoint and region from the endpoint.assert.ok(host.s3.sent.filter((c) => /Put/.test(c.name)).every((c) => c.input.Bucket === 'openvibe-backups-test'));const cfg = host.s3Configs[0];@@@ -346,14 +226,14 @@ runTests([noSecrets(host.s3.objects.get(`${base}manifest.json`), 'the manifest');const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));assert.strictEqual(summary.offsite.ok, true);-- assert.strictEqual(summary.offsite.uploaded.length, 4);++ assert.strictEqual(summary.offsite.uploaded.length, 5);assert.strictEqual(summary.offsite.location, base);}),test('large files go up as a multipart upload; a failed part aborts it', async () => {const host = backupHost();const big = crypto.randomBytes(300 * 1024);-- host.backupContent = (db) => (db.includes('openvibe.live') ? big : Buffer.from('small'));++ host.dumpContent = (db) => db === 'ov_live' ? Buffer.concat([Buffer.from('pg_dump -Fc'), big]) : Buffer.from(`pg_dump -Fc of ${db}`);await host.cli('backup', '--all');const summary = await require('../lib/commands/backup').readRun(host.exec, host.inv, null);const cfg = await offsite.loadConfig(host.exec, {});@@@ -364,8 +244,8 @@ runTests([assert.strictEqual(live.parts, Math.ceil(live.cipherBytes / (64 * 1024)));const partSizes = host.s3.sent.filter((c) => c.name === 'UploadPartCommand').map((c) => c.input.Body);assert.ok(partSizes.slice(0, -1).every((b) => b === `<${64 * 1024} bytes>`), 'every part but the last is exactly partSize');-- assert.strictEqual(live.cipherBytes, bcrypto.encryptedSize(big.length, 16 * 1024));-- assert.ok((await decrypt(Buffer.from(KEY_HEX, 'hex'), host.s3.objects.get(live.object))).equals(big));++ assert.strictEqual(live.cipherBytes, bcrypto.encryptedSize(big.length + 11, 16 * 1024));++ assert.ok((await decrypt(Buffer.from(KEY_HEX, 'hex'), host.s3.objects.get(live.object))).equals(Buffer.concat([Buffer.from('pg_dump -Fc'), big])));assert.ok(host.s3.sent.some((c) => c.name === 'CompleteMultipartUploadCommand'));const s3 = mockS3();@@@ -456,7 +336,7 @@ runTests([test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {const host = backupHost();const put = (run) => {-- host.s3.objects.set(`openvibe-backups/fake-host/${run}/live/live.db.ovbk`, Buffer.from('x'));++ host.s3.objects.set(`openvibe-backups/fake-host/${run}/live/live.dump.ovbk`, Buffer.from('x'));host.s3.objects.set(`openvibe-backups/fake-host/${run}/manifest.json`, Buffer.from('{}'));};// Ten runs 35..44 days old, one 5 days old; another host's and a stray object are never touched.@@@ -466,7 +346,7 @@ runTests([host.s3.objects.set('openvibe-backups/fake-host/README', Buffer.from('keep'));// A failed upload: no pruning at all.-- host.s3.failPut = (k) => k.endsWith('media.db.ovbk');++ host.s3.failPut = (k) => k.endsWith('media.dump.ovbk');const bad = await host.cli('backup', '--all', '--offsite', '--json');assert.strictEqual(bad.code, 2);const badSummary = JSON.parse(bad.out);@@@ -495,10 +375,11 @@ runTests([host.s3 = mockS3({ pageSize: 2 });await host.cli('backup', '--all', '--offsite');host.advance(DAY);-- await host.cli('backup', '--all', '--offsite');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];++ await host.cli('backup', '--all', '--offsite', '--logical');const l = await host.cli('offsite', 'list', 'live', '--json');assert.strictEqual(l.code, 0, l.out);-- assert.deepStrictEqual(JSON.parse(l.out).map((x) => [x.run, x.objects, x.manifest]), [['20260923-120000', 5, true], ['20260922-120000', 5, true]]);++ assert.deepStrictEqual(JSON.parse(l.out).map((x) => [x.run, x.objects, x.manifest]), [['20260923-120000', 6, true], ['20260922-120000', 6, true]]);const c = await host.cli('offsite', 'check');assert.strictEqual(c.code, 0, c.out);assert.match(c.out, /s3:\/\/openvibe-backups-test\/openvibe-backups\/fake-host\/ .*region us-west-004.*key id [0-9a-f]{16}; retention 30 days/);@@@ -509,20 +390,18 @@ runTests([test('restore-download: verified, decrypted copies in a new directory; never overwrites; never near a live database', async () => {const host = backupHost();await host.cli('backup', '--all', '--offsite');-- const original = host.files.get('/var/backups/openvibe/live/20260922-120000/live.db').content;++ const original = host.files.get('/var/backups/openvibe/live/20260922-120000/live.dump').content;const r = await host.cli('restore-download', 'live', '20260922-120000');assert.strictEqual(r.code, 0, r.out);-- const dest = '/var/lib/openvibe-restore/live-20260922-120000/live.db';-- assert.ok(host.files.get(dest).content.equals(original));-- assert.match(r.out, /quick_check ok/);-- assert.ok(host.sqliteCalls.some((c) => c.db === dest && /quick_check/.test(c.sql)));-- // The live database was never written.-- assert.strictEqual(host.read('/opt/openvibe.live/data/live.db'), 'sqlite');++ const dest = '/var/lib/openvibe-restore/live-20260922-120000/live.dump';++ assert.ok(Buffer.from(host.files.get(dest).content).equals(Buffer.from(original)));++ assert.match(r.out, /pg_restore --list ok/);++ assert.ok(host.calls.some((c) => c.cmd === 'pg_restore' && c.args.includes('--list')));const again = await host.cli('restore-download', 'live', '20260922-120000');assert.strictEqual(again.code, 1);assert.match(again.out, /not empty; restore-download never overwrites/);-- for (const out of ['/opt/openvibe.live/data', '/opt/openvibe.live', '/opt', '/var/lib/openvibe-ai', '/var/backups/openvibe/live/x']) {++ for (const out of ['/opt/openvibe.live/data', '/opt/openvibe.live', '/opt', '/var/backups/openvibe/live/x']) {const bad = await host.cli('restore-download', 'live', 'latest', '--out', out);assert.strictEqual(bad.code, 1, out);assert.match(bad.out, /refusing --out/, out);@@@ -536,16 +415,16 @@ runTests([const host = backupHost();await host.cli('backup', '--all', '--offsite');const base = 'openvibe-backups/fake-host/20260922-120000/';-- const obj = Buffer.from(host.s3.objects.get(`${base}media/media.db.ovbk`));++ const obj = Buffer.from(host.s3.objects.get(`${base}media/media.dump.ovbk`));obj[obj.length - 20] ^= 1;-- host.s3.objects.set(`${base}media/media.db.ovbk`, obj);++ host.s3.objects.set(`${base}media/media.dump.ovbk`, obj);const t = await host.cli('restore-download', 'media', 'latest');assert.strictEqual(t.code, 2);assert.match(t.out, /failed authentication/);-- assert.strictEqual(host.files.has('/var/lib/openvibe-restore/media-20260922-120000/media.db'), false, 'a bad copy is removed');++ assert.strictEqual(host.files.has('/var/lib/openvibe-restore/media-20260922-120000/media.dump'), false, 'a bad copy is removed');// Swapping two encrypted files is caught by the manifest's hashes.-- host.s3.objects.set(`${base}ai/ai.db.ovbk`, host.s3.objects.get(`${base}ai/ai-extra.db.ovbk`));++ host.s3.objects.set(`${base}ai/ai.dump.ovbk`, host.s3.objects.get(`${base}ai/ai-extra.dump.ovbk`));const sw = await host.cli('restore-download', 'ai', 'latest', '--out', '/root/r-ai');assert.strictEqual(sw.code, 2);assert.match(sw.out, /does not match the manifest/);@@@ -557,9 +436,6 @@ runTests([assert.strictEqual(f.code, 2);assert.match(f.out, /failed its HMAC check/);-- const none = await host.cli('restore-download', 'openre', 'latest');-- assert.strictEqual(none.code, 1);-- assert.match(none.out, /no off-host run of openre/);}),test('restore-download: a PostgreSQL .dump is verified with pg_restore --list; a corrupt one is refused', async () => {@@@ -595,9 +471,8 @@ runTests([assert.strictEqual(r.code, 0, r.out);const rec = JSON.parse(r.out);const f = rec.files.find((x) => x.file.endsWith('objects.tar.gz'));-- assert.deepStrictEqual([f.kind, f.quickCheck], ['objects', null], 'an object archive is not quick_checked as a database');++ assert.deepStrictEqual([f.kind, f.check], ['objects', null], 'an object archive is checked by a drill');assert.ok(host.files.has('/root/r-host/objects.tar.gz'));-- assert.ok(!host.sqliteCalls.some((c) => c.db === '/root/r-host/objects.tar.gz'));assert.ok(rec.files.some((x) => x.file.endsWith('host.dump') && x.kind === 'database'), 'the database is still checked');}),@@@ -633,7 +508,7 @@ runTests([r = await host.cli('backup', '--all', '--offsite');assert.strictEqual(r.code, 2);assert.match(r.out, /OFFSITE\s+FAILED: encryption key file/);-- assert.ok(host.files.has('/var/backups/openvibe/live/20260922-120000/live.db'));++ assert.ok(host.files.has('/var/backups/openvibe/live/20260922-120000/live.dump'));const { createFakeHost } = require('./fake-host');const plain = createFakeHost({ root: false, user: 'ubuntu' });@@@ -2,7 +2,7 @@/*** The PostgreSQL side of `ovhost backup`: every run verifies pgBackRest (a fresh stanza, and a WAL* archiver failed_count that has not risen), and a per-service pg_dump -Fc only on a slower cadence-- * (the weekly one, or --logical). SQLite services must be untouched. Against the fake host; no++ * (the weekly one, or --logical). All declared databases use PostgreSQL. Against the fake host; no* pg_dump, pgbackrest or psql ever runs for real.*/const assert = require('assert');@@@ -57,7 +57,7 @@ runTests([assert.strictEqual(rec.verified.stop, new Date((Math.floor(host.exec.now() / 1000) - 3600) * 1000).toISOString());// No prior record: a logical dump was due on the first run.assert.ok(rec.dir);-- assert.strictEqual(host.dumps.length, 1);++ assert.strictEqual(host.dumps.length, 3);assert.ok(host.files.has(`${rec.dir}/trade.dump`));const prom = host.read(PROM);@@@ -148,7 +148,7 @@ runTests([assert.strictEqual(host.dumps.length, 2);}),-- test('staging ownership: postgres owns the directory while pg_dump runs; SQLite stays the service user', async () => {++ test('staging ownership: postgres owns the directory while pg_dump runs; all dumps use postgres', async () => {const host = pgHost();const stages = [];const mkdir = host.exec.mkdir;@@@ -159,7 +159,7 @@ runTests([assert.strictEqual(host.dumps[0].as, 'postgres');await host.cli('backup', 'live');-- assert.strictEqual(stages[1][1], 'ubuntu', 'a SQLite service keeps the service user');++ assert.strictEqual(stages[1][1], 'postgres', 'every database dump uses postgres');assert.strictEqual(stages[1][0], 'live-20260922-120000');// Root takes every copy back once the staging directory is back to root:root.@@@ -167,7 +167,7 @@ runTests([assert.deepStrictEqual([copy.owner, copy.mode], ['root', 0o600]);}),-- test('backup --all: SQLite and PostgreSQL services in one run; verified differs from ok and failed', async () => {++ test('backup --all: PostgreSQL services in one run; verified differs from ok and failed', async () => {const host = pgHost();seedDump(host, 3 * DAY); // the weekly dump is not due: trade is verification-only this runconst r = await host.cli('backup', '--all', '--json');@@@ -187,6 +187,6 @@ runTests([assert.strictEqual(bad.code, 2);const s2 = JSON.parse(bad.out);assert.strictEqual(s2.ok, false);-- assert.deepStrictEqual(s2.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'ok'], ['trade', 'failed']]);++ assert.deepStrictEqual(s2.services.map((x) => [x.service, x.status]), [['live', 'failed'], ['media', 'failed'], ['trade', 'failed']]);}),]);@@@ -17,11 +17,12 @@ function withHost(host, rows = []) {units: ['openvibe-host.service'],port: 4910,ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },-- databases: [{ name: 'host', path: '/var/lib/openvibe-host-api/host.db' }],-- nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: '/var/lib/openvibe-host-api/host.db', maxUpload: '110m' } },++ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],++ nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: { engine: 'postgresql', database: 'ov_host' }, maxUpload: '110m' } },};host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });-- host.sqliteHandler = (db, sql) => {++ host.psqlHandler = (db, sql) => {++ assert.strictEqual(db, 'ov_host');assert.match(sql, /^SELECT hostname FROM host_domains WHERE kind = 'custom' AND status = 'verified'/);return rows;};@@@ -21,7 +21,7 @@ runTests([assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');-- // Media on PostgreSQL (ADR-035): recordings in progress come from its /api/ready, not from a SQLite file.++ // Media on PostgreSQL (ADR-035): recordings in progress come from its /api/ready, through PostgreSQL.assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field], ['http-json-count', 'recordings_in_progress']);// Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');@@@ -69,7 +69,7 @@ runTests([assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { vhost: '../../etc/passwd' } } } }), /file name/);assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { skipVhosts: ['../../etc/passwd'] } } } }), /vhost file names/);assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', ready: { url: 'http://evil.example/ready' } } } }), /loopback/);-- assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'sqlite-count', db: '/d.db', sql: 'DELETE FROM vods' } } } }), /SELECT/);++ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'DELETE FROM vods' } } } }), /SELECT/);assert.throws(() => normalise({ services: { 'X Y': { ...base, repo: '/opt/x' } } }), /service id/);assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', unitSources: { 'a.service': '../../etc/shadow' } } } }), /inside the checkout/);}),@@@ -1,7 +1,6 @@'use strict';/**-- * `ovhost data provision|switch`: the data role's scripts (roles/data/add-service.sh,-- * switch-service.sh) run from the ovhost install, driven against the fake host. The scripts are++ * `ovhost data provision`: the data role (roles/data/add-service.sh) run from the ovhost install, driven against the fake host. The scripts are* stubbed at their install path: nothing here touches PostgreSQL, Valkey, systemd or a real* database, and every invocation is recorded so the tests assert the argv, the env and the exit code.*/@@@ -22,7 +21,7 @@ function cli(host, ...argv) {/** Install both scripts at the install path and record every run of them. */function installScripts(host, { stdout = '', stderr = '', code = 0 } = {}) {-- for (const f of ['add-service.sh', 'switch-service.sh']) host.put(`${DIR}/roles/data/${f}`, '#!/usr/bin/env bash\n', { mode: 0o755, owner: 'root' });++ for (const f of ['add-service.sh']) host.put(`${DIR}/roles/data/${f}`, '#!/usr/bin/env bash\n', { mode: 0o755, owner: 'root' });const runs = [];const orig = host.exec.run;host.exec.run = async (cmd, args, opts = {}) => {@@@ -77,96 +76,4 @@ runTests([assert.match(r.out, /add-service\.sh is not in the ovhost install/);}),-- test('switch passes the inventory units (worker templates as a glob) and dir, plus --sqlite, and passes the exit code through', async () => {-- const host = scenario();-- const raw = JSON.parse(host.read('/etc/openvibe/host.json'));-- raw.services.media.workerUnits = ['[email protected]'];-- host.put('/etc/openvibe/host.json', JSON.stringify(raw), { mode: 0o640 });-- const runs = installScripts(host, { code: 3, stderr: '[switch] still busy after 12 hours: not switching\n' });---- const r = await cli(host, 'data', 'switch', 'media', '--sqlite', '/var/lib/openvibe-media/media.db');-- assert.strictEqual(r.code, 3, r.out);-- assert.strictEqual(runs.length, 1);-- assert.strictEqual(runs[0].script, `${DIR}/roles/data/switch-service.sh`);-- assert.deepStrictEqual(runs[0].args, ['media', '/var/lib/openvibe-media/media.db']);-- assert.strictEqual(runs[0].env.SWITCH_DIR, '/opt/openvibe.media');-- assert.strictEqual(runs[0].env.SWITCH_UNITS, 'openvibe-media.service openvibe-media-clip@*.service');-- assert.match(r.out, /still busy after 12 hours/);-- }),---- test('switch without --sqlite passes no file and uses the entries for a default service', async () => {-- const host = scenario();-- const runs = installScripts(host);-- const r = await cli(host, 'data', 'switch', 'live');-- assert.strictEqual(r.code, 0, r.out);-- assert.deepStrictEqual(runs[0].args, ['live']);-- assert.strictEqual(runs[0].env.SWITCH_DIR, '/opt/openvibe.live');-- // Live is socket-activated: its socket is stopped too (after the service), or the next request starts the old-- // release again while the import runs.-- assert.strictEqual(runs[0].env.SWITCH_UNITS, 'openvibe-live.service openvibe-live.socket');-- }),---- test('switch also stops the units matching unitsMatch', async () => {-- const host = scenario();-- const raw = JSON.parse(host.read('/etc/openvibe/host.json'));-- raw.services.live.unitsMatch = 'openvibe-live*.service';-- host.put('/etc/openvibe/host.json', JSON.stringify(raw), { mode: 0o640 });-- const runs = installScripts(host);-- const r = await cli(host, 'data', 'switch', 'live');-- assert.strictEqual(r.code, 0, r.out);-- assert.strictEqual(runs[0].env.SWITCH_UNITS, 'openvibe-live.service openvibe-live*.service openvibe-live.socket');-- }),---- test('without root both are refused and nothing runs (sudo would drop SWITCH_UNITS/SWITCH_DIR)', async () => {-- const host = scenario();-- const runs = installScripts(host);-- host.exec.isRoot = async () => false;-- for (const action of ['provision', 'switch']) {-- const r = await cli(host, 'data', action, 'live');-- assert.strictEqual(r.code, 1, r.out);-- assert.match(r.out, new RegExp(`data ${action} must run as root`));-- }-- assert.deepStrictEqual(runs, []);-- const dry = await cli(host, 'data', 'switch', 'live', '--dry-run');-- assert.strictEqual(dry.code, 0, dry.out);-- }),---- test('--dry-run prints the command and runs nothing', async () => {-- const host = scenario();-- const runs = installScripts(host);---- const p = await cli(host, 'data', 'provision', 'live', '--dry-run');-- assert.strictEqual(p.code, 0, p.out);-- assert.match(p.out, /^would run: \/opt\/ovhost-install\/roles\/data\/add-service\.sh "live"$/m);---- const s = await cli(host, 'data', 'switch', 'live', '--sqlite', '/var/lib/openvibe-live/live.db', '--dry-run');-- assert.strictEqual(s.code, 0, s.out);-- assert.match(s.out, /would run: SWITCH_UNITS="openvibe-live\.service openvibe-live\.socket" SWITCH_DIR="\/opt\/openvibe\.live" \/opt\/ovhost-install\/roles\/data\/switch-service\.sh "live" "\/var\/lib\/openvibe-live\/live\.db"/);-- assert.deepStrictEqual(runs, []);-- }),---- test('--json reports the run and still carries no URL or password', async () => {-- const host = scenario();-- installScripts(host, { stdout: 'DATABASE_URL=postgresql://ov_live_app:[email protected]:6432/ov_live\n' });-- const r = await cli(host, 'data', 'provision', 'live', '--json');-- assert.strictEqual(r.code, 0, r.out);-- const doc = JSON.parse(r.out);-- assert.strictEqual(doc.service, 'live');-- assert.strictEqual(doc.action, 'provision');-- assert.strictEqual(doc.ok, true);-- assert.strictEqual(doc.code, 0);-- assert.strictEqual(doc.stdout, 'DATABASE_URL=<redacted-url>\n');-- assert.doesNotMatch(r.out, /postgres(ql)?:\/\//i);-- assert.doesNotMatch(r.out, /ov_live_app:pw/);-- }),---- test('usage: data needs a known subcommand and a service', async () => {-- const host = scenario();-- const bad = await cli(host, 'data', 'migrate', 'live');-- assert.strictEqual(bad.code, 1, bad.out);-- assert.match(bad.out, /usage: ovhost data provision/);-- const missing = await cli(host, 'data', 'switch');-- assert.strictEqual(missing.code, 1, missing.out);-- assert.match(missing.out, /data switch needs a <service>/);-- }),]);@@@ -120,7 +120,7 @@ runTests([assert.deepStrictEqual(host.restarts(), []);}),-- test('refuses to restart Media while a recording is in progress (read-only query as the service user)', async () => {++ test('refuses to restart Media while a recording is in progress (read-only PostgreSQL query)', async () => {const host = scenario();const from = host.repo('media').head;push(host, 'media', { 'server/index.js': 'media2();' });@@@ -130,9 +130,8 @@ runTests([assert.match(r.out, /1 recordings in progress — refusing/);assert.strictEqual(host.repo('media').head, from);assert.deepStrictEqual(host.restarts(), []);-- const q = host.sqliteCalls.find((c) => c.op === 'query');-- assert.strictEqual(q.as, 'ubuntu');-- assert.strictEqual(q.db, '/opt/openvibe.media/data/media.db');++ const q = host.psqlCalls.find((c) => c.database === 'ov_media');++ assert.strictEqual(q.as, 'postgres');assert.match(q.sql, /^SELECT count\(\*\) AS n FROM vods WHERE is_recording = 1$/);}),@@@ -263,18 +262,18 @@ runTests([assert.deepStrictEqual(host.restarts(), ['openvibe-events.service']);}),-- test('a schema change backs up the declared databases (as the service user) before the restart', async () => {++ test('a schema change backs up the declared PostgreSQL database before the restart', async () => {const host = scenario();push(host, 'live', { 'server/db/schema.sql': 'CREATE TABLE b(y);' });const r = await host.cli('deploy', 'live');assert.strictEqual(r.code, 0, r.out);-- const b = host.sqliteCalls.find((c) => c.op === 'backup');++ const b = host.dumps.find((c) => c.database === 'ov_live');assert.ok(b, 'backup taken');-- assert.strictEqual(b.as, 'ubuntu');-- assert.match(b.dest, /^\/var\/backups\/openvibe\.staging\/live-\d{8}-\d{6}\/live\.db$/, 'the worker writes to its staging directory');-- const dir = [...host.files.keys()].find((k) => /^\/var\/backups\/openvibe\/live\/\d{8}-\d{6}\/live\.db$/.test(k));++ assert.strictEqual(b.as, 'postgres');++ assert.match(b.dest, /^\/var\/backups\/openvibe\.staging\/live-\d{8}-\d{6}\/live\.dump$/);++ const dir = [...host.files.keys()].find((k) => /^\/var\/backups\/openvibe\/live\/\d{8}-\d{6}\/live\.dump$/.test(k));assert.ok(dir, 'the copy ends up under /var/backups/openvibe/live/<stamp>/');-- const backupIdx = host.calls.findIndex((c) => c.cmd === 'sqlite-backup');++ const backupIdx = host.calls.findIndex((c) => c.cmd === 'pg_dump');const restartIdx = host.calls.findIndex((c) => c.cmd === 'systemctl' && c.args[0] === 'restart');assert.ok(backupIdx < restartIdx, 'backup before restart');}),@@@ -2,10 +2,10 @@/*** The example inventory's live and media drills (LIVE_DRILL, OpenVibe.Live 8a58aea; MEDIA_DRILL,* OpenVibe.Media 6e74eb0), run by `ovhost drill` on the fake host:-- * - live: the production unit's ExecStart from its own checkout with LIVE_DRILL=1, the copy as-- * DB_PATH and {tmp}/data as DATA_DIR (created before the start); DB-only reads compared, the++ * - live: the production unit's ExecStart from its own checkout with LIVE_DRILL=1, the scratch database as++ * DATABASE_URL and {tmp}/data as DATA_DIR (created before the start); DB-only reads compared, the* values production changes while someone is live ignored; the socket and the service untouched.-- * - media: MEDIA_DRILL=1, the copy as DB_PATH, every storage path but the thumbnail listing moved++ * - media: MEDIA_DRILL=1, the scratch database as DATABASE_URL, every storage path but the thumbnail listing moved* under {tmp}/storage and never created; the /browse pages compared byte for byte.* - a checkout without the switch is refused before anything is created (an older release would* ignore LIVE_DRILL / MEDIA_DRILL and run its side effects).@@@ -17,7 +17,6 @@ const { scenario, test, runTests, SECRET } = require('./helpers');const { normalise } = require('../lib/inventory');const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));--const LIVE_DB = '/opt/openvibe.live/data/live.db';const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));@@@ -59,14 +58,12 @@ async function liveHost({ switchInCheckout = true, drillBodies = null } = {}) {unit.dropIns = ['/etc/systemd/system/openvibe-live.service.d/socket.conf'];host.alivePids.add(unit.mainPid);host.listeners.set(3000, [{ pid: unit.mainPid, process: 'node' }]);-- host.put(LIVE_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });const prod = liveBodies({ drill: false });host.http.set('http://127.0.0.1:3000/api/ready', () => ({ status: 200, body: { ready: true } }));for (const [p, body] of Object.entries(prod)) host.http.set(`http://127.0.0.1:3000${p}`, () => ({ status: 200, body }));const counts = { users: 393, channels: 180, managed_streams: 140, streams: 5120, follows: 910, chat_messages: 250000 };-- host.sqliteHandler = (db, sql) => {-- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];++ host.psqlHandler = (db, sql) => {const m = /FROM "([a-z_]+)"/.exec(sql);return [{ n: m ? counts[m[1]] ?? 0 : 0 }];};@@@ -138,7 +135,7 @@ runTests([const live = inv.services.live.drill;assert.strictEqual(live.supported, true);assert.strictEqual(live.port, 13000);-- assert.deepStrictEqual(live.databases, { live: { engine: 'sqlite', env: 'DB_PATH', dir: false } });++ assert.deepStrictEqual(live.databases, { live: { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } });assert.strictEqual(live.env.LIVE_DRILL, '1');assert.strictEqual(live.env.DATA_DIR, '{tmp}/data');assert.strictEqual(live.env.HOST, '127.0.0.1');@@@ -169,7 +166,7 @@ runTests([['/api/themes', true], ['/api/emotes/global', true], ['/api/streams', true],['/api/streams/recently-online?limit=20', true], ['/api/streams/channel/japaneseoldguy/live', true],]);-- assert.deepStrictEqual(rec.databases.map((d) => d.name), ['live'], 'rs-companion.db is not restored');++ assert.deepStrictEqual(rec.databases.map((d) => d.name), ['live'], 'only the declared database is restored');assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.match]), [['users', true], ['channels', true], ['managed_streams', true], ['streams', true], ['follows', true], ['chat_messages', true]]);assert.strictEqual(host.systemdRuns.length, 1);@@@ -179,9 +176,12 @@ runTests([assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/live.env', `${rec.dir}/drill.env`]);for (const p of ['SocketBindAllow=tcp:13000', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'ProtectSystem=strict', `ReadWritePaths=${rec.dir}`, 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);assert.ok(!run.props.some((p) => /BindPaths=|ReadWritePaths=\/opt/.test(p)), 'no bind mounts; production\'s data directory is not writable');-- assert.deepStrictEqual(envOf(host.drillEnv), {++ const { DATABASE_URL, DATABASE_DIRECT_URL, ...envRest } = envOf(host.drillEnv);++ assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);++ assert.match(DATABASE_URL, /^postgresql:\/\//);++ assert.deepStrictEqual(envRest, {PORT: '13000', HOST: '127.0.0.1', LIVE_DRILL: '1', DATA_DIR: `${rec.dir}/data`,-- EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', DB_PATH: `${rec.dir}/db/live.db`,++ EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', VALKEY_URL: 'redis://127.0.0.1:9/0',});assert.ok(host.dataDirAtStart, 'DATA_DIR exists before the instance starts');@@@ -190,7 +190,7 @@ runTests([assert.deepStrictEqual(stateChanges(host), [], 'no unit was started, stopped or restarted');assert.deepStrictEqual(host.socketViolations(), []);assert.strictEqual(host.units.get('openvibe-live.socket').active, 'active');-- assert.strictEqual(host.read(LIVE_DB), 'sqlite-production');++ assert.ok(host.pgDatabases.has('ov_live'), 'production database untouched');assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/live.jsonl')).includes(SECRET));}),@@@ -231,7 +231,7 @@ runTests([assert.match(DATABASE_URL, new RegExp(`^postgresql://${scratch}:[0-9a-f]+@127\\.0\\.0\\.1:5432/${scratch}$`));assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);assert.strictEqual(rec.databases[0].integrity, 'pg_restore --list ok');-- assert.deepStrictEqual(host.drillDirAtStart, ['db', 'drill.env', 'pg', 'pg/media.dump'], 'the dump copy and the override file; storage is never created');++ assert.deepStrictEqual(host.drillDirAtStart, ['drill.env', 'pg', 'pg/media.dump'], 'the dump copy and the override file; storage is never created');assert.deepStrictEqual(stateChanges(host), []);assert.ok(host.pgDatabases.has('ov_media'), 'the production database is untouched');assert.ok(host.droppedDatabases.some((d) => d.name === scratch), 'the scratch database is dropped');@@@ -1,7 +1,7 @@'use strict';/*** ovhost drill for a PostgreSQL service (lib/dbengine.js): the artifact is a `.dump` in the backup,-- * not a SQLite file. The drill copies it into <tmp>/pg owned by postgres, verifies it with++ * a PostgreSQL dump. The drill copies it into <tmp>/pg owned by postgres, verifies it with* `pg_restore --list`, creates a scratch database and login role, restores into them, points* drill.env at the scratch database (and at a closed Valkey), and always drops both again in a* finally. Covered: the generated URLs and their password, the refusal when an override is missing,@@@ -177,7 +177,7 @@ runTests([assert.deepStrictEqual(cp.args, ['-o', 'postgres', '-m', '0600', '-T', '--', `${host.backupDir}/trade.dump`, `${tmp}/pg/trade.dump`]);assert.strictEqual(cp.privileged, true);-- // The Markdown row names pg_restore --list, not pragma integrity_check.++ // The Markdown row reports the dump archive check.assert.match(r.out, /`pg_restore --list` \(trade\) = ok/);assert.strictEqual(rec.markdown, r.out.split('\n').pop());}),ddeleted file mode 100644@@@ -1,725 +0,0 @@--'use strict';--/**-- * ovhost drill against the fake host: the full flow, integrity failure, readiness timeout, an-- * instance that dies, mismatch reporting, cleanup on failure, refusals (not root, port in use,-- * unsupported, no backup), and the guarantee that nothing outside the drill directory, the lock and-- * the drill log is written.-- */--const assert = require('assert');--const crypto = require('crypto');--const fs = require('fs');--const path = require('path');--const { scenario, test, runTests, SECRET } = require('./helpers');--const { normalise } = require('../lib/inventory');--const { compareBodies, verifyObjectBlobs } = require('../lib/commands/drill');----const DRILL_DIR = '/var/lib/openvibe-drills';--const PROD_DB = '/var/lib/openvibe-tools/tools.db';--const OBJECTS_DIR = '/var/lib/openvibe-host-api/objects';--const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';----/** A content-addressed blob under the Host object-store layout: <root>/projects/<prj>/<aa>/<sha256>. */--function blob(root, data) {-- const sha = crypto.createHash('sha256').update(data).digest('hex');-- return { sha, path: `${root}/projects/prj_1/${sha.slice(0, 2)}/${sha}` };--}----function toolsEntry(drillOverrides = {}) {-- return {-- repo: '/opt/openvibe.tools',-- units: ['openvibe-tools.service'],-- envFile: '/etc/openvibe/tools.env',-- port: 4200,-- ready: { url: 'http://127.0.0.1:4200/api/ready', timeoutSeconds: 30 },-- databases: [{ name: 'tools', path: PROD_DB }],-- drill: {-- port: 14200,-- databases: { tools: 'TOOLS_DB_PATH' },-- env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words' },-- dirs: ['{tmp}/data'],-- compare: [{ path: '/api/pastes?limit=5' }, { path: '/api/v1/pulse?limit=5', ignore: ['generated_at'] }],-- counts: [{ db: 'tools', table: 'pastes' }],-- ...drillOverrides,-- },-- };--}----/**-- * The standard fake host plus Tools in production, with one `ovhost backup tools` taken.-- * The drill instance's behaviour is controlled through `host.drill`.-- */--async function drillScenario({ drill: drillOverrides, backup = true, objects = null, blobs = [] } = {}) {-- const host = scenario();-- const doc = JSON.parse(host.read('/etc/openvibe/host.json'));-- const tools = toolsEntry(drillOverrides);-- if (objects) tools.objects = objects;-- doc.services.tools = tools;-- host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });---- const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });-- const sha = repo.commit({ 'package.json': '{"name":"openvibe-tools"}', 'server/index.js': 'tools();' }, { message: 'initial' });-- repo.publish(sha);-- repo.checkout(sha);-- host.addUnit('openvibe-tools.service', { mainPid: 2222 });-- host.alivePids.add(2222);-- host.listeners.set(4200, [{ pid: 2222, process: 'node' }]);-- host.put('/etc/systemd/system/openvibe-tools.service', [-- '[Service]',-- 'User=ubuntu',-- 'WorkingDirectory=/opt/openvibe.tools',-- 'ExecStart=/usr/bin/node server/index.js',-- `Environment=NODE_ENV=production "SESSION_SECRET=${SECRET}-unit"`,-- '',-- ].join('\n'));-- host.put('/etc/openvibe/tools.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.tools\n`, { mode: 0o600 });-- host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });---- host.http.set('http://127.0.0.1:4200/api/ready', () => ({ status: 200, body: { status: 'ready' } }));-- host.http.set('http://127.0.0.1:4200/api/pastes?limit=5', () => ({ status: 200, body: PASTES }));-- host.http.set('http://127.0.0.1:4200/api/v1/pulse?limit=5', () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:00.000Z' } }));---- host.drill = {-- integrity: [{ integrity_check: 'ok' }],-- productionCount: 892,-- restoredCount: 892,-- ready: () => ({ status: 200, body: { status: 'ready' } }),-- pastes: () => ({ status: 200, body: PASTES }),-- pulse: () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:07.000Z' } }),-- onStart: null,-- ignoreTerm: false,-- ignoreKill: false,-- };-- host.sqliteHandler = (db, sql) => {-- if (/integrity_check/.test(sql)) return host.drill.integrity;-- if (/count\(\*\) AS n FROM "pastes"/.test(sql)) return [{ n: db === PROD_DB ? host.drill.productionCount : host.drill.restoredCount }];-- if (/is_recording/.test(sql)) return [{ n: host.recording }];-- return [{ n: 0 }];-- };-- host.onSystemdRun = (spec) => {-- if (host.drill.onStart) { const r = host.drill.onStart(spec); if (r) return r; }-- host.listeners.set(14200, [{ pid: spec.pid, process: 'node' }]);-- const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });-- host.http.set('http://127.0.0.1:14200/api/ready', alive(() => host.drill.ready()));-- host.http.set('http://127.0.0.1:14200/api/pastes?limit=5', alive(() => host.drill.pastes()));-- host.http.set('http://127.0.0.1:14200/api/v1/pulse?limit=5', alive(() => host.drill.pulse()));-- return undefined;-- };-- host.onKill = (pid, signal) => {-- if (signal === 'SIGTERM' && host.drill.ignoreTerm) return false;-- if (signal === 'SIGKILL' && host.drill.ignoreKill) return false;-- return undefined;-- };---- for (const b of blobs) host.put(b.path, b.content, { owner: 'ubuntu', mode: 0o644 });---- if (backup) {-- const b = await host.cli('backup', 'tools', '--json');-- assert.strictEqual(b.code, 0, b.out);-- host.backupDir = JSON.parse(b.out).dir;-- }-- host.advance(60 * 1000);-- return host;--}----/** A Tools-like service: a gateway and apps, each app its own unit and port, data directories. */--function appsEntry(drill = {}) {-- const d = {-- port: 14016,-- unit: 'openvibe-apps-docs.service',-- productionPort: 4016,-- databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'docs-jobs': { env: 'DATA_DIR', dir: true }, 'yt-analytics': { dir: '{tmp}/yt-data' } },-- env: { PORT: '{port}', HOST: '127.0.0.1', TOOLS_JOB_RESULTS: 'local', EVENTS_PUBLISH: 'off' },-- bind: [{ from: '{tmp}/yt-data', to: '/opt/openvibe.apps/apps/yt/data' }],-- requires: [{ file: 'apps/_shared/jobs/index.js', contains: 'TOOLS_JOB_RESULTS' }],-- ready: '/api/ready',-- compare: ['/release.json'],-- counts: [{ db: 'docs-jobs', table: 'tool_jobs' }],-- ...drill,-- };-- for (const [k, v] of Object.entries(d)) if (v === null) delete d[k];-- return {-- owner: 'ubuntu',-- repo: '/opt/openvibe.apps',-- units: ['openvibe-apps.service', 'openvibe-apps-docs.service', 'openvibe-apps-yt.service'],-- envFile: '/etc/openvibe/apps.env',-- port: 4001,-- ready: { url: 'http://127.0.0.1:4001/api/ready', timeoutSeconds: 30 },-- databases: [-- { name: 'docs-analytics', path: '/opt/openvibe.apps/apps/docs/data/analytics.db' },-- { name: 'docs-jobs', path: '/opt/openvibe.apps/apps/docs/data/jobs.db' },-- { name: 'yt-analytics', path: '/opt/openvibe.apps/apps/yt/data/analytics.db' },-- ],-- drill: d,-- };--}----async function appsScenario({ switchInCheckout = true, drill } = {}) {-- const host = scenario();-- const doc = JSON.parse(host.read('/etc/openvibe/host.json'));-- doc.services.apps = appsEntry(drill);-- host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });-- const repo = host.createRepo('/opt/openvibe.apps', { owner: 'ubuntu' });-- const sha = repo.commit({-- 'package.json': '{"name":"apps"}',-- 'apps/_shared/jobs/index.js': switchInCheckout ? "const want = env.TOOLS_JOB_RESULTS || 'local';" : 'jobs();',-- }, { message: 'initial' });-- repo.publish(sha);-- repo.checkout(sha);-- const unit = (name, dir, port) => {-- host.addUnit(name, { mainPid: 3000 + port });-- host.alivePids.add(3000 + port);-- host.listeners.set(port, [{ pid: 3000 + port, process: 'node' }]);-- host.put(`/etc/systemd/system/${name}`, ['[Service]', 'User=ubuntu', `WorkingDirectory=/opt/openvibe.apps/apps/${dir}`, 'Environment=NODE_ENV=production', `Environment=PORT=${port}`, `ExecStart=/usr/bin/env node /opt/openvibe.apps/apps/${dir}/server/index.js`, ''].join('\n'));-- };-- unit('openvibe-apps.service', 'gateway', 4001);-- unit('openvibe-apps-docs.service', 'docs', 4016);-- unit('openvibe-apps-yt.service', 'yt', 4013);-- host.put('/etc/openvibe/apps.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\n`, { mode: 0o600 });-- for (const db of appsEntry().databases) host.put(db.path, `sqlite-${db.name}`, { owner: 'ubuntu', mode: 0o640 });-- host.http.set('http://127.0.0.1:4001/release.json', () => ({ status: 200, body: '{"service":"gateway"}' }));-- host.http.set('http://127.0.0.1:4016/release.json', () => ({ status: 200, body: '{"service":"docs"}' }));-- host.sqliteHandler = (db, sql) => {-- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];-- if (/count\(\*\)/.test(sql)) return [{ n: 3 }];-- return [{ n: 0 }];-- };-- host.onSystemdRun = (spec) => {-- const envFile = spec.envFiles[spec.envFiles.length - 1];-- host.drillEnv = host.read(envFile);-- host.listeners.set(14016, [{ pid: spec.pid, process: 'node' }]);-- const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });-- host.http.set('http://127.0.0.1:14016/api/ready', alive(() => ({ status: 200, body: { status: 'ready' } })));-- host.http.set('http://127.0.0.1:14016/release.json', alive(() => ({ status: 200, body: '{"service":"docs"}' })));-- return undefined;-- };-- const b = await host.cli('backup', 'apps', '--json');-- assert.strictEqual(b.code, 0, b.out);-- host.advance(60 * 1000);-- return host;--}----/** Every file entry as "type:owner:mode:content", to diff the fake filesystem. */--function fsSnapshot(host) {-- const m = new Map();-- for (const [k, v] of host.files) m.set(k, `${v.type}:${v.owner}:${v.mode}:${v.type === 'file' ? String(v.content) : v.target || ''}`);-- return m;--}----function changedPaths(before, after) {-- const out = [];-- for (const [k, v] of after) if (before.get(k) !== v) out.push(k);-- for (const k of before.keys()) if (!after.has(k)) out.push(k);-- return out.sort();--}----function lastLog(host) {-- const text = host.read('/var/lib/openvibe-host/drills/tools.jsonl');-- const lines = text.trim().split('\n');-- return JSON.parse(lines[lines.length - 1]);--}----const allowedWrite = (p, tmp) => p === DRILL_DIR || p === tmp || p.startsWith(`${tmp}/`)-- || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/tools.jsonl'-- || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/tools-drill.lock';----runTests([-- test('full flow: restore, integrity, sandboxed second instance, identical answers, stop by pid, cleanup, log row', async () => {-- const host = await drillScenario();-- const before = fsSnapshot(host);-- const callsBefore = host.calls.length;-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 0, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.result, 'passed');-- assert.strictEqual(rec.failure, null);-- assert.strictEqual(rec.backup, host.backupDir);-- const tmp = rec.dir;-- assert.match(tmp, /^\/var\/lib\/openvibe-drills\/tools-\d{8}-\d{6}$/);-- assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['tools', 'ok', `${tmp}/db/tools.db`]]);-- assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/pastes?limit=5', true], ['/api/v1/pulse?limit=5', true]]);-- assert.deepStrictEqual(rec.counts, [{ db: 'tools', table: 'pastes', production: 892, restored: 892, match: true }]);-- assert.strictEqual(rec.ready.ok, true);-- assert.strictEqual(rec.stop.ok, true);-- assert.strictEqual(rec.stop.signal, 'SIGTERM');---- const calls = host.calls.slice(callsBefore);-- // The copy is made as the service user; integrity and counts run as the service user.-- // Backups are root-only: root copies, and the copy belongs to the service user, 0600.-- const cp = calls.find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/db/tools.db`));-- assert.strictEqual(cp.privileged, true);-- assert.deepStrictEqual(cp.args, ['-o', 'ubuntu', '-m', '0600', '-T', '--', `${host.backupDir}/tools.db`, `${tmp}/db/tools.db`]);-- const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));-- assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/tools.db`, 'ubuntu']);---- // systemd-run: production ExecStart/WorkingDirectory, production env file then the override file,-- // sandbox properties, loopback-only networking, the service user.-- assert.strictEqual(host.systemdRuns.length, 1);-- const run = host.systemdRuns[0];-- assert.strictEqual(run.uid, 'ubuntu');-- assert.strictEqual(run.cwd, '/opt/openvibe.tools');-- assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js']);-- assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/tools.env', `${tmp}/drill.env`]);-- for (const p of ['ProtectSystem=strict', `ReadWritePaths=${tmp}`, 'PrivateTmp=yes', 'NoNewPrivileges=yes', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'SocketBindAllow=tcp:14200', 'SocketBindDeny=any', 'Environment=NODE_ENV=production']) {-- assert.ok(run.props.includes(p), `systemd-run -p ${p}`);-- }-- assert.ok(!run.props.some((p) => p.includes('SESSION_SECRET')), 'a secret-looking unit Environment= is not passed on');-- const runCall = calls.find((c) => c.cmd === 'systemd-run');-- assert.strictEqual(runCall.privileged, true);---- // Stopped by its own pid with SIGTERM; production's pid is never signalled; no pkill.-- const kills = calls.filter((c) => c.cmd === 'kill');-- assert.deepStrictEqual(kills.map((c) => c.args), [['SIGTERM', run.pid]]);-- assert.ok(!calls.some((c) => /pkill|killall/.test(c.cmd)));-- assert.ok(host.alivePids.has(2222), 'production still running');-- assert.strictEqual(host.units.get('openvibe-tools.service').restarts, 0);-- assert.ok(!host.restarts().length, 'nothing restarted');---- // Only the drill directory (now removed), the lock (released) and the drill log changed.-- const after = fsSnapshot(host);-- const changed = changedPaths(before, after);-- for (const p of changed) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);-- assert.ok(!host.files.has(tmp), 'temp dir removed');-- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'), 'lock released');---- // Secrets: nothing ovhost printed, logged or passed on carries a value.-- const everything = [r.out, host.read('/var/lib/openvibe-host/drills/tools.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');-- assert.ok(!everything.includes(SECRET), 'a secret value leaked');-- assert.ok(!host.reads.includes('/etc/openvibe/tools.env'), 'ovhost never reads the env file for a drill');---- // Markdown row ready for docs/restore-drills.md.-- assert.match(r.out, /\| 2026-09-22 12:01 \| tools \| `\/var\/backups\/openvibe\/tools\/20260922-120000\/` \(`ovhost drill tools`\) \| `pragma integrity_check` \(tools\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);-- assert.strictEqual(rec.markdown, r.out.split('\n').pop());-- }),---- test('--keep leaves the drill directory; the override file holds inventory values only', async () => {-- const host = await drillScenario();-- const before = fsSnapshot(host);-- const r = await host.cli('drill', 'tools', '--keep');-- assert.strictEqual(r.code, 0, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.kept, true);-- const tmp = rec.dir;-- assert.strictEqual(host.files.get(tmp).owner, 'ubuntu');-- assert.strictEqual(host.files.get(tmp).mode, 0o700);-- assert.strictEqual(host.files.get(`${tmp}/db/tools.db`).owner, 'ubuntu');-- assert.strictEqual(host.read(`${tmp}/db/tools.db`), host.read(`${host.backupDir}/tools.db`));-- assert.strictEqual(host.files.get(`${tmp}/data`).owner, 'ubuntu');-- const env = host.read(`${tmp}/drill.env`);-- for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `TOOLS_DB_PATH=${tmp}/db/tools.db`]) {-- assert.ok(env.split('\n').includes(line), `drill.env has ${line}`);-- }-- assert.ok(!env.includes(SECRET));-- for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);-- // The production database and the backup are untouched.-- assert.strictEqual(host.read(PROD_DB), 'sqlite-production');-- }),---- test('integrity failure: nothing is started, the copy is removed, the log says failed', async () => {-- const host = await drillScenario();-- host.drill.integrity = [{ integrity_check: '*** in database main ***' }, { integrity_check: 'Page 5: btree page has a bad cell' }];-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.result, 'failed');-- assert.strictEqual(rec.failure.stage, 'integrity');-- assert.match(rec.failure.message, /bad cell/);-- assert.strictEqual(host.systemdRuns.length, 0);-- assert.ok(!host.files.has(rec.dir));-- assert.match(r.out, /\| failed \|$/);-- }),---- test('readiness timeout: the instance is stopped by its pid and the directory removed', async () => {-- const host = await drillScenario();-- host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'ready');-- assert.match(rec.failure.message, /not ready after 30s \(503\)/);-- assert.strictEqual(rec.compare.length, 0, 'nothing compared');-- const pid = host.systemdRuns[0].pid;-- assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', pid]]);-- assert.ok(!host.alivePids.has(pid));-- assert.ok(!host.files.has(rec.dir));-- assert.ok(!host.listeners.has(14200));-- }),---- test('an instance that exits before it is ready fails the drill without signalling anything', async () => {-- const host = await drillScenario();-- let probes = 0;-- host.drill.ready = () => { probes += 1; if (probes === 2) host.alivePids.delete(host.systemdRuns[0].pid); return { status: 503 }; };-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'ready');-- assert.match(rec.failure.message, /exited before it was ready/);-- assert.ok(!host.calls.some((c) => c.cmd === 'kill'));-- assert.ok(!host.files.has(rec.dir));-- }),---- test('mismatches are reported per path and per table; volatile keys are ignored only where declared', async () => {-- const host = await drillScenario();-- host.drill.pastes = () => ({ status: 200, body: PASTES.replace('world', 'World') });-- host.drill.pulse = () => ({ status: 200, body: { items: [{ id: 2 }], generated_at: 'different, but ignored' } });-- host.drill.productionCount = 900;-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'compare');-- const [pastes, pulse] = rec.compare;-- assert.strictEqual(pastes.match, false);-- assert.match(pastes.detail, /first difference at byte 55/);-- assert.strictEqual(pulse.match, false);-- assert.match(pulse.detail, /\$\.items\[0\]\.id/);-- assert.ok(!/generated_at/.test(pulse.detail));-- assert.deepStrictEqual(rec.counts[0], { db: 'tools', table: 'pastes', production: 900, restored: 892, match: false });-- assert.match(rec.markdown, /`\/api\/pastes\?limit=5` differs \(bodies differ/);-- assert.match(rec.markdown, /pastes 900 ≠ 892 \| failed \|$/);-- // Still stopped and cleaned up.-- assert.ok(!host.alivePids.has(host.systemdRuns[0].pid));-- assert.ok(!host.files.has(rec.dir));-- }),---- test('acceptance: the restored instance alone answers each declared GET with its status and JSON keys (WS-S task 2)', async () => {-- const accept = [{ path: '/api/ready', keys: ['status'] }, { path: '/api/v1/pulse?limit=5', keys: ['items', 'generated_at'] }];-- let host = await drillScenario({ drill: { acceptance: accept } });-- let r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 0, r.out);-- let rec = lastLog(host);-- assert.deepStrictEqual(rec.acceptance, [{ path: '/api/ready', ok: true }, { path: '/api/v1/pulse?limit=5', ok: true }]);-- assert.match(rec.markdown, /acceptance `\/api\/ready`, `\/api\/v1\/pulse\?limit=5`/);---- host = await drillScenario({ drill: { acceptance: [{ path: '/api/v1/pulse?limit=5', keys: ['items', 'cursor'] }, { path: '/api/ready', status: 204 }] } });-- r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'acceptance');-- assert.deepStrictEqual(rec.acceptance.map((a) => [a.ok, a.detail || null]), [[false, 'missing cursor'], [false, 'status 200, expected 204']]);-- assert.match(rec.markdown, /acceptance failed: /);-- assert.ok(!host.files.has(rec.dir), 'still cleaned up');-- }),---- test('acceptance entries are validated in the inventory', async () => {-- const host = await drillScenario({ drill: { acceptance: [{ path: 'no-slash' }] }, backup: false });-- const r = await host.cli('drill', 'tools');-- assert.notStrictEqual(r.code, 0);-- assert.match(r.out, /acceptance\[0\]\.path must be a path/);-- }),---- test('compare: a status mismatch or a failing production answer is never a match', () => {-- const e = { path: '/x', ignore: [] };-- assert.strictEqual(compareBodies(e, { status: 500, body: 'e' }, { status: 500, body: 'e' }).match, false);-- assert.match(compareBodies(e, { status: 200, body: 'a' }, { status: 404, body: 'a' }).detail, /status 404/);-- assert.strictEqual(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '{"a":1,"t":1}' }, { status: 200, body: '{"a":1,"t":2}' }).match, true);-- assert.match(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '<html>' }, { status: 200, body: '<html>' }).detail, /not JSON/);-- }),---- test('cleanup on failure: systemd-run fails → directory removed, lock released, row logged', async () => {-- const host = await drillScenario();-- const before = fsSnapshot(host);-- host.drill.onStart = () => ({ code: 1, stderr: 'Failed to start transient service unit: Unit already exists.' });-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'start');-- assert.match(rec.failure.message, /systemd-run failed/);-- assert.ok(!host.files.has(rec.dir));-- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'));-- for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, rec.dir), `unexpected write: ${p}`);-- // A second drill can run straight after.-- host.drill.onStart = null;-- host.advance(1000);-- assert.strictEqual((await host.cli('drill', 'tools')).code, 0);-- }),---- test('an instance that ignores SIGTERM gets SIGKILL; one that survives SIGKILL keeps its directory and fails', async () => {-- let host = await drillScenario();-- host.drill.ignoreTerm = true;-- let r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 0, r.out);-- let rec = lastLog(host);-- assert.strictEqual(rec.stop.signal, 'SIGKILL');-- assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args[0]), ['SIGTERM', 'SIGKILL']);---- host = await drillScenario();-- host.drill.ignoreTerm = true;-- host.drill.ignoreKill = true;-- r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'stop');-- assert.strictEqual(rec.kept, true);-- assert.ok(host.files.has(rec.dir), 'directory kept while the instance may still hold it');-- assert.ok(host.alivePids.has(2222), 'production untouched');-- assert.ok(!host.calls.some((c) => c.cmd === 'kill' && c.args[1] === 2222));-- }),---- test('refusals: not root, port in use, unsupported service, no drill block, no backup, missing database copy', async () => {-- let host = await drillScenario();-- host.exec.isRoot = async () => false;-- let r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 1);-- assert.match(r.out, /must run as root/);-- assert.strictEqual(host.systemdRuns.length, 0);---- host = await drillScenario();-- host.listeners.set(14200, [{ pid: 777, process: 'python3' }]);-- const before = fsSnapshot(host);-- r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 1);-- assert.match(r.out, /port 14200 is already in use \(python3 pid 777\)/);-- assert.strictEqual(host.systemdRuns.length, 0);-- assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');---- host = await drillScenario({ drill: { supported: false, reason: 'workers have no off switch' } });-- r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 1);-- assert.match(r.out, /does not support restore drills: workers have no off switch/);---- host = await drillScenario();-- r = await host.cli('drill', 'media');-- assert.strictEqual(r.code, 1);-- assert.match(r.out, /media has no drill block/);---- host = await drillScenario({ backup: false });-- r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 1);-- assert.match(r.out, /no good ovhost backup recorded for tools/);---- host = await drillScenario({ backup: false });-- host.ensureDir('/srv/old-backup');-- r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup');-- assert.strictEqual(r.code, 1);-- assert.match(r.out, /has no copy of the tools database/);-- host.put('/srv/old-backup/tools.db', 'older sqlite', { owner: 'ubuntu' });-- r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup', '--json');-- assert.strictEqual(r.code, 0, r.out);-- assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');-- r = await host.cli('drill', 'tools', '--backup', 'relative/dir');-- assert.strictEqual(r.code, 1);-- }),---- test('one app of a multi-unit service: drill.unit, data-directory databases, a bind mount, its own production port', async () => {-- const host = await appsScenario();-- const r = await host.cli('drill', 'apps');-- assert.strictEqual(r.code, 0, r.out);-- const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/apps.jsonl').trim().split('\n').pop());-- assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));-- const tmp = rec.dir;-- // Data-directory databases keep their production file names under {tmp}/data.-- assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.copy]).sort(), [['docs-analytics', `${tmp}/data/analytics.db`], ['docs-jobs', `${tmp}/data/jobs.db`], ['yt-analytics', `${tmp}/yt-data/analytics.db`]].sort());-- const run = host.systemdRuns[0];-- // The docs unit's ExecStart and WorkingDirectory, not the gateway's.-- assert.strictEqual(run.cwd, '/opt/openvibe.apps/apps/docs');-- assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.apps/apps/docs/server/index.js']);-- assert.ok(run.props.includes('Environment=NODE_ENV=production'));-- assert.ok(!run.props.some((p) => p.startsWith('Environment=PORT=')), 'the unit\'s PORT never reaches the drill');-- assert.ok(run.props.includes(`BindPaths=${tmp}/yt-data:/opt/openvibe.apps/apps/yt/data`), run.props.join('\n'));-- const env = host.drillEnv;-- assert.match(env, new RegExp(`^DATA_DIR=${tmp}/data$`, 'm'));-- assert.match(env, /^PORT=14016$/m);-- assert.ok(!/analytics\.db/.test(env), 'a data-directory database is found by its file name, not an env var');-- // Compared with the docs app's production port, not the gateway's.-- assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/release.json', true]]);-- assert.ok(host.calls.some((c) => c.cmd === 'curl' && c.args[0] === 'http://127.0.0.1:4016/release.json'));-- assert.ok(!host.calls.some((c) => c.cmd === 'curl' && c.args[0].startsWith('http://127.0.0.1:4001/')));-- assert.ok(!host.restarts().length, 'nothing restarted');-- }),---- test('requires: a checkout without the drill switch is refused before anything is created', async () => {-- const host = await appsScenario({ switchInCheckout: false });-- const before = fsSnapshot(host);-- const r = await host.cli('drill', 'apps');-- assert.strictEqual(r.code, 1, r.out);-- assert.match(r.out, /TOOLS_JOB_RESULTS in apps\/_shared\/jobs\/index\.js/);-- assert.match(r.out, /Deploy a release with the drill switch first/);-- assert.strictEqual(host.systemdRuns.length, 0);-- assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), []);-- }),---- test('inventory: unit, productionPort, bind, requires and data-directory databases are validated', () => {-- const entry = appsEntry();-- const inv = (drill) => normalise({ services: { apps: { ...entry, drill: { ...entry.drill, ...drill } } } });-- assert.strictEqual(inv({}).services.apps.drill.unit, 'openvibe-apps-docs.service');-- assert.throws(() => inv({ unit: 'openvibe-live.service' }), /unit must be one of apps's units/);-- assert.throws(() => inv({ productionPort: 14016 }), /productionPort must differ/);-- assert.throws(() => inv({ bind: [{ from: '/tmp/x', to: '/opt/openvibe.apps/apps/yt/data' }] }), /bind\[0\]\.from/);-- assert.throws(() => inv({ bind: [{ from: '{tmp}/x', to: '/etc/openvibe' }] }), /inside the checkout/);-- assert.throws(() => inv({ bind: [{ from: '{tmp}/x', to: '/opt/openvibe.apps/../etc' }] }), /plain absolute path/);-- assert.throws(() => inv({ requires: [{ file: '../etc/passwd', contains: 'x' }] }), /inside the checkout/);-- assert.throws(() => inv({ requires: [{ file: 'a.js' }] }), /contains/);-- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR' } } }), /"dir": true/);-- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'yt-analytics': { dir: true } } }), /would both be \{tmp\}\/data\/analytics\.db/);-- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: '/var/lib/x' } } }), /dir must be true or/);-- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'docs-jobs': { env: 'DATA_DIR', dir: '{tmp}/other' } } }), /two different places/);-- const multi = { ...entry, drill: { ...entry.drill } };-- delete multi.drill.unit;-- normalise({ services: { apps: multi } }); // valid inventory; the drill itself refuses (below)-- }),---- test('objects: the declared object directory is archived, restored into scratch, and every blob sha256 checked', async () => {-- const a = 'tenant page a\n';-- const b = 'tenant asset b\n';-- const pa = blob(OBJECTS_DIR, a);-- const pb = blob(OBJECTS_DIR, b);-- const host = await drillScenario({-- objects: [{ name: 'objects', path: OBJECTS_DIR }],-- blobs: [{ path: pa.path, content: a }, { path: pb.path, content: b }],-- drill: {-- env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words', HOST_STORAGE_DIR: '{tmp}/objects' },-- dirs: ['{tmp}/data', '{tmp}/objects'],-- },-- });-- const archive = `${host.backupDir}/objects.tar.gz`;-- assert.deepStrictEqual([host.files.get(archive).owner, host.files.get(archive).mode], ['root', 0o600], 'the archive is an ordinary root-only copy');-- const r = await host.cli('drill', 'tools', '--keep');-- assert.strictEqual(r.code, 0, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.kept, true);-- assert.deepStrictEqual(rec.objects.map((o) => [o.name, o.blobs, o.skipped, o.integrity]), [['objects', 2, 0, 'sha256 ok']]);-- const tmp = rec.dir;-- // The archive is listed first, then extracted without restoring production's ownership or modes.-- assert.ok(host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-tvzf' && c.args[1] === archive), 'listed before extraction');-- const extract = host.calls.find((c) => c.cmd === 'tar' && c.args[0] === '-xzf');-- assert.deepStrictEqual(extract.args, ['-xzf', archive, '-C', `${tmp}/objects`, '--no-same-owner', '--no-same-permissions']);-- assert.strictEqual(extract.privileged, true);-- // Restored into the drill's own scratch directory, never the live object directory.-- assert.strictEqual(rec.objects[0].copy, `${tmp}/objects`);-- assert.ok(rec.objects[0].copy.startsWith(`${tmp}/`));-- assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pa.sha.slice(0, 2)}/${pa.sha}`), a);-- assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pb.sha.slice(0, 2)}/${pb.sha}`), b);-- // The instance is pointed at the restored directory, not production's.-- assert.ok(host.read(`${tmp}/drill.env`).split('\n').includes(`HOST_STORAGE_DIR=${tmp}/objects`));-- assert.match(rec.markdown, /object store `objects` 2 blob\(s\) sha256 ok/);-- // The live objects were never written or moved.-- assert.strictEqual(host.read(pa.path), a);-- assert.strictEqual(host.read(pb.path), b);-- }),---- test('objects: a blob whose bytes do not match the sha256 in its name fails the drill before anything starts', async () => {-- const good = 'good blob\n';-- const p = blob(OBJECTS_DIR, good);-- const wrongName = 'a'.repeat(64);-- const host = await drillScenario({-- objects: [{ name: 'objects', path: OBJECTS_DIR }],-- blobs: [{ path: p.path, content: good }, { path: `${OBJECTS_DIR}/projects/prj_1/aa/${wrongName}`, content: 'not its name\n' }],-- });-- const before = fsSnapshot(host);-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'objects');-- assert.match(rec.failure.message, /blob\(s\) do not match their name/);-- assert.match(rec.failure.message, /aa\/a{64}/);-- assert.strictEqual(host.systemdRuns.length, 0, 'nothing is started');-- assert.ok(!host.files.has(rec.dir));-- assert.strictEqual(host.read(p.path), good, 'the live object is untouched');-- for (const changed of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(changed, rec.dir), `unexpected write: ${changed}`);-- }),---- test('objects: a service declaring an object directory refuses to drill from a backup without its archive', async () => {-- // The directory does not exist, so the backup records the object directory as skipped.-- const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [] });-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 1);-- assert.match(r.out, /has no copy of the objects object directory \(objects\.tar\.gz\)/);-- assert.strictEqual(host.systemdRuns.length, 0);-- }),---- test('objects: a symlink or special-file member is refused before root extracts the archive', async () => {-- for (const member of [{ type: 'l', name: './evil', target: '/' }, { type: 'p', name: './fifo' }]) {-- const a = 'tenant page a\n';-- const pa = blob(OBJECTS_DIR, a);-- const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });-- host.tarMembers = [member];-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, `${member.type}: ${r.out}`);-- const rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'objects');-- assert.match(rec.failure.message, /not a regular file or directory/);-- assert.ok(!host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-xzf'), `${member.type}: never extracted`);-- assert.ok(!host.files.has(rec.dir), 'cleaned up');-- }-- }),---- test('objects: a symlinked directory already in the restored tree is refused, not walked as root', async () => {-- const host = scenario();-- const dir = '/var/lib/openvibe-drills/tools-20260922-120100/objects';-- const good = 'a good blob\n';-- const sha = crypto.createHash('sha256').update(good).digest('hex');-- host.put(`${dir}/projects/prj_1/${sha.slice(0, 2)}/${sha}`, good, { owner: 'ubuntu' });-- // A file whose name IS a sha256 but whose bytes are not, behind a symlinked directory: if the-- // walk followed the link it would hash this and report a mismatch.-- const outsideSha = 'b'.repeat(64);-- host.put(`/var/lib/openvibe-secret/${outsideSha}`, 'not its name\n', { owner: 'root', mode: 0o600 });-- await host.exec.symlink('/var/lib/openvibe-secret', `${dir}/evil`);-- const entry = { blobs: 0, skipped: 0, bytes: 0, mismatches: 0 };-- await assert.rejects(-- verifyObjectBlobs(host.exec, dir, entry),-- (err) => /evil \(symlink or special file\)/.test(err.message) && !err.message.includes(outsideSha) && !err.message.includes('/var/lib/openvibe-secret'),-- );-- assert.strictEqual(entry.blobs, 1, 'the real blob was still checked');-- assert.strictEqual(entry.mismatches, 1);-- }),---- test('objects: a drill with no room to extract the archive fails before root restores it', async () => {-- const a = 'tenant page a\n';-- const pa = blob(OBJECTS_DIR, a);-- const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });-- const callsBefore = host.calls.length;-- host.statfsFree = 1024;-- const r = await host.cli('drill', 'tools');-- assert.strictEqual(r.code, 2, r.out);-- const rec = lastLog(host);-- assert.strictEqual(rec.failure.stage, 'space');-- assert.match(rec.failure.message, /not enough free space under \/var\/lib\/openvibe-drills/);-- assert.ok(!host.calls.slice(callsBefore).some((c) => c.cmd === 'tar'), 'nothing was tarred or extracted');-- assert.ok(!host.files.has(rec.dir));-- }),---- test('inventory: object directories are validated', () => {-- const base = { owner: 'ubuntu', repo: '/opt/openvibe.host', units: [], databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }] };-- const svc = (objects) => normalise({ services: { h: { ...base, objects } } });-- assert.deepStrictEqual(svc([{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]).services.h.objects, [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]);-- assert.deepStrictEqual(svc(undefined).services.h.objects, []);-- assert.throws(() => svc([{ name: 'a/b', path: '/x' }]), /name must be a plain name/);-- assert.throws(() => svc([{ name: 'objects', path: 'relative' }]), /absolute path/);-- assert.throws(() => svc([{ name: 'objects', path: '/x' }, { name: 'objects', path: '/y' }]), /listed twice/);-- assert.throws(() => svc('nope'), /must be a list of directories/);-- }),---- test('several units and neither drill.unit nor drill.command: refused', async () => {-- const host = await appsScenario({ drill: { unit: null } });-- const r = await host.cli('drill', 'apps');-- assert.strictEqual(r.code, 1, r.out);-- assert.match(r.out, /set drill\.unit/);-- }),---- test('inventory: drill blocks are validated', () => {-- const base = { owner: 'ubuntu' };-- const svc = (drill, extra = {}) => normalise({ services: { tools: { ...base, ...toolsEntry(), ...extra, drill: { ...toolsEntry().drill, ...drill } } } });-- assert.throws(() => svc({ port: 4200 }), /must differ from the service's own port/);-- assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of tools's databases/);-- assert.throws(() => svc({ env: { HOST: '127.0.0.1' } }), /\{port\}/);-- assert.throws(() => svc({ env: { PORT: '{port}', TOOLS_DB_PATH: '/x' } }), /already set by drill.databases/);-- assert.throws(() => svc({ env: { PORT: '{port}\nEVIL=1' } }), /one-line/);-- assert.throws(() => svc({ dirs: ['/opt/openvibe.tools/data'] }), /under \{tmp\}/);-- assert.throws(() => svc({ counts: [{ db: 'tools', table: 'x; drop table y' }] }), /table name/);-- assert.throws(() => svc({ compare: [{ path: 'http://evil/' }] }), /path/);-- assert.throws(() => svc({ supported: false }), /reason is required/);-- assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, tools: { ...base, ...toolsEntry() } } }), /14200 is a's port/);-- assert.strictEqual(svc({}).services.tools.drill.ready, '/api/ready');-- }),---- test('host.example.json: every service with a database has a drill block; unsupported ones give a reason', () => {-- const raw = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));-- const inv = normalise(raw);-- for (const s of Object.values(inv.services)) {-- if (!s.databases.length) continue;-- assert.ok(s.drill, `${s.id} has a drill block`);-- if (!s.drill.supported) { assert.ok(s.drill.reason.length > 20, `${s.id} explains why`); continue; }-- assert.ok(s.drill.compare.length || s.drill.counts.length);-- assert.ok(Object.keys(s.drill.env).length >= 2, `${s.id} overrides at least its port and one side effect`);-- }-- }),--]);@@@ -1,15 +1,12 @@'use strict';/**-- * The real executor, exercised only where that is safe on any machine: the SQLite worker against a-- * temp database, HTTP against a local server, files in a temp directory, and run() as the current-- * user. No systemctl/git/npm/nginx command is run.++ * The real executor, exercised with HTTP, files in a temp directory, and run() as the current user. No systemctl/git/npm/nginx command is run.*/const assert = require('assert');const fs = require('fs');const http = require('http');const os = require('os');const path = require('path');--const Database = require('better-sqlite3');const { createSystemExecutor } = require('../lib/executor');const { test, runTests } = require('./helpers');@@@ -18,72 +15,6 @@ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ovhost-exec-'));process.on('exit', () => fs.rmSync(dir, { recursive: true, force: true }));runTests([-- test('sqlite: read-only count as the current user; writes are refused', async () => {-- const dbPath = path.join(dir, 'media.db');-- const db = new Database(dbPath);-- db.pragma('journal_mode = WAL');-- db.exec('CREATE TABLE vods (id INTEGER PRIMARY KEY, is_recording INTEGER); INSERT INTO vods (is_recording) VALUES (1), (0), (1);');-- const me = await exec.userName();-- const rows = await exec.sqlite(dbPath, 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', { as: me });-- assert.deepStrictEqual(rows, [{ n: 2 }]);-- await assert.rejects(exec.sqlite(dbPath, 'DELETE FROM vods', { as: me }), /read-only|readonly/);-- assert.strictEqual(db.prepare('SELECT count(*) AS n FROM vods').get().n, 3);-- await assert.rejects(exec.sqlite(path.join(dir, 'missing.db'), 'SELECT 1'), /cannot open/);-- db.close();-- }),---- test('sqlite backup: consistent copy that passes quick_check; never overwrites', async () => {-- const src = path.join(dir, 'src.db');-- const db = new Database(src);-- db.exec('CREATE TABLE t (x); INSERT INTO t VALUES (1), (2);');-- const dest = path.join(dir, 'backup.db');-- await exec.sqliteBackup(src, dest);-- const copy = new Database(dest, { readonly: true });-- assert.strictEqual(copy.prepare('SELECT count(*) AS n FROM t').get().n, 2);-- copy.close();-- await assert.rejects(exec.sqliteBackup(src, dest), /refusing to overwrite/);-- db.close();-- }),---- test('sqlite backup of a WAL database: one 0600 file in rollback-journal mode, no -wal/-shm left', async () => {-- const src = path.join(dir, 'wal-src.db');-- const db = new Database(src);-- db.pragma('journal_mode = WAL');-- db.exec('CREATE TABLE t (x); INSERT INTO t VALUES (1), (2), (3);');-- const out = fs.mkdtempSync(path.join(dir, 'bk-'));-- const dest = path.join(out, 'wal.db');-- await exec.sqliteBackup(src, dest);-- db.close();-- assert.deepStrictEqual(fs.readdirSync(out), ['wal.db']);-- assert.strictEqual(fs.statSync(dest).mode & 0o777, 0o600);-- const copy = new Database(dest, { readonly: true });-- assert.strictEqual(copy.pragma('journal_mode', { simple: true }), 'delete');-- assert.strictEqual(copy.prepare('SELECT count(*) AS n FROM t').get().n, 3);-- copy.close();-- assert.deepStrictEqual(fs.readdirSync(out), ['wal.db'], 'reading the copy leaves no side files');-- }),---- test('drill primitives: integrity_check through the worker, kill() signals exactly one pid', async () => {-- const dbPath = path.join(dir, 'restored.db');-- const db = new Database(dbPath);-- db.pragma('journal_mode = WAL');-- db.exec('CREATE TABLE pastes (id INTEGER PRIMARY KEY); INSERT INTO pastes DEFAULT VALUES;');-- db.close();-- const me = await exec.userName();-- assert.deepStrictEqual(await exec.sqlite(dbPath, 'PRAGMA integrity_check', { as: me }), [{ integrity_check: 'ok' }]);-- assert.deepStrictEqual(await exec.sqlite(dbPath, 'SELECT count(*) AS n FROM "pastes"', { as: me }), [{ n: 1 }]);---- const { spawn } = require('child_process');-- const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' });-- const exited = new Promise((r) => child.on('exit', (code, signal) => r(signal)));-- assert.strictEqual(await exec.pidAlive(child.pid), true);-- assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), true);-- assert.strictEqual(await exited, 'SIGTERM');-- assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), false, 'a pid that is gone');-- await assert.rejects(exec.kill(1, 'SIGTERM'), /refusing/);-- await assert.rejects(exec.kill(0, 'SIGTERM'), /refusing/);-- }),--test('http, files, locks and run() behave as the fake host assumes', async () => {const server = http.createServer((req, res) => { res.writeHead(req.url === '/ok' ? 200 : 503); res.end(JSON.stringify({ host: req.headers.host })); });await new Promise((r) => server.listen(0, '127.0.0.1', r));@@@ -1,7 +1,7 @@'use strict';/*** A fake host implementing the executor interface (lib/executor.js) in memory: a filesystem, git-- * repositories, systemd units, npm, nginx, ss, HTTP endpoints, SQLite and PostgreSQL. Nothing here++ * repositories, systemd units, npm, nginx, ss, HTTP endpoints and PostgreSQL. Nothing here* touches the real machine. Every run() call is recorded in `calls` so tests can assert what would* have been executed (and as whom).*@@@ -31,8 +31,6 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stnginxTest: () => ({ code: 0, stderr: 'nginx: configuration file /etc/nginx/nginx.conf test is successful' }),npm: null, // (cwd, argv, as) -> undefined | { code, stderr } ; default installs package.json depsnpmRewritesLockfile: true,-- sqliteHandler: () => [{ n: 0 }],-- sqliteCalls: [],// ── PostgreSQL ──// The cluster's databases and roles, the archiver's failed WAL count, the stanza's backups// and its status. A test drives them directly (host.pgDatabases.add('ov_trade')) or through@@@ -749,20 +747,6 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stif (!get(p)) { const err = new Error(`ENOENT: no such file or directory, statfs '${p}'`); err.code = 'ENOENT'; throw err; }return { free: host.statfsFree, size: host.statfsSize };},-- async sqlite(db, sql, { as } = {}) {-- host.sqliteCalls.push({ op: 'query', db, sql, as });-- calls.push({ cmd: 'sqlite', args: [db, sql], as: as || null, privileged: false, cwd: null });-- return host.sqliteHandler(db, sql, as);-- },-- async sqliteBackup(db, dest, { as } = {}) {-- host.sqliteCalls.push({ op: 'backup', db, dest, as });-- calls.push({ cmd: 'sqlite-backup', args: [db, dest], as: as || null, privileged: false, cwd: null });-- if (files.has(path.resolve(dest))) throw new Error(`refusing to overwrite ${dest}`);-- const parent = files.get(path.dirname(path.resolve(dest)));-- if (as && parent && parent.owner !== as) throw new Error(`cannot open ${dest}: permission denied (directory owned by ${parent.owner}, worker runs as ${as})`);-- if (host.onSqliteBackup) { const r = host.onSqliteBackup(db, dest, as); if (r instanceof Error) throw r; }-- put(dest, host.backupContent ? host.backupContent(db) : `backup of ${db}`, { owner: as || user, mode: 0o600 });-- },async listeners(port) { return listeners.get(port) || []; },async kill(pid, signal = 'SIGTERM') {calls.push({ cmd: 'kill', args: [signal, pid], as: null, privileged: true, cwd: null });@@@ -1,7 +1,7 @@'use strict';/*** A standard fake host with the shapes of the real estate: Live (socket-activated, /api/streams),-- * Media (recordings counted in SQLite), Tools (several packages and units), Sites (static build +++ * Media (recordings counted in PostgreSQL), Tools (several packages and units), Sites (static build +* nginx vhosts, no units) and Events (SSE, drain policy "report").*/const path = require('path');@@@ -48,7 +48,7 @@ function inventoryDoc() {noRestartPaths: ['public/', 'docs/', '*.md'],protected: { kind: 'http-json-count', url: 'http://127.0.0.1:3000/api/streams', array: 'streams', where: { is_live: true }, label: 'live streams' },drain: { policy: 'refuse', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },-- databases: [{ name: 'live', path: '/opt/openvibe.live/data/live.db' }],++ databases: [{ name: 'live', engine: 'postgresql', database: 'ov_live' }],backupOnChange: ['server/db/schema.sql'],nginx: { vhost: 'openvibe.live.conf', variant: 'websocket', wsPaths: ['/ws/'] },lifecycle: lifecycleDoc(),@@@ -60,8 +60,8 @@ function inventoryDoc() {env: { required: 'from-example' },port: 4100,ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },-- protected: { kind: 'sqlite-count', db: '/opt/openvibe.media/data/media.db', sql: 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', label: 'recordings in progress' },-- databases: [{ name: 'media', path: '/opt/openvibe.media/data/media.db' }],++ protected: { kind: 'postgresql-count', database: 'ov_media', sql: 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', label: 'recordings in progress' },++ databases: [{ name: 'media', engine: 'postgresql', database: 'ov_media' }],lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),},tools: {@@@ -126,7 +126,7 @@ function scenario() {}service('live', {-- 'package.json': pkgJson('openvibe-live', ['express', 'better-sqlite3', 'openvibe-shared']),++ 'package.json': pkgJson('openvibe-live', ['express', 'openvibe-sdk', 'openvibe-shared']),'package-lock.json': '{"lockfileVersion":3,"v":1}','server/index.js': 'console.log(1);','server/db/schema.sql': 'CREATE TABLE a(x);',@@@ -137,16 +137,14 @@ function scenario() {host.addUnit('openvibe-live.socket', { sub: 'listening', partOf: ['openvibe-live.service'], mainPid: 0 });host.put('/etc/systemd/system/openvibe-live.service', '[Service]\nExecStart=node server/index.js\nEnvironment=NODE_ENV=production\n');host.put('/etc/openvibe/live.env', `JWT_SECRET=${SECRET}\nBASE_URL=https://openvibe.live\nPAYPAL_CLIENT_SECRET="${SECRET}-paypal"\nOV_OAUTH_CLIENT_SECRET=${SECRET}-oauth\n`, { mode: 0o600 });-- host.put('/opt/openvibe.live/data/live.db', 'sqlite', { owner: 'ubuntu' });service('media', {-- 'package.json': pkgJson('openvibe-media', ['express', 'better-sqlite3']),++ 'package.json': pkgJson('openvibe-media', ['express', 'openvibe-sdk']),'package-lock.json': '{"lockfileVersion":3,"v":1}','server/index.js': 'media();','.env.example': 'PORT=4100\nMEDIA_B2_APP_KEY=\nMEDIA_R2_SECRET_ACCESS_KEY=\n',}, ['.']);host.put('/etc/openvibe/media.env', `PORT=4100\nMEDIA_B2_APP_KEY=${SECRET}-b2\nMEDIA_R2_SECRET_ACCESS_KEY=${SECRET}-r2\n`, { mode: 0o600 });-- host.put('/opt/openvibe.media/data/media.db', 'sqlite', { owner: 'ubuntu' });service('tools', {'apps/gateway/package.json': pkgJson('tools-gateway', ['express', 'openvibe-shared']),@@@ -185,7 +183,10 @@ function scenario() {host.http.set('http://127.0.0.1:4100/healthz', readyFor('openvibe-media.service'));host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));host.http.set('http://127.0.0.1:4300/api/ready', readyFor('openvibe-events.service'));-- host.sqliteHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : [{ n: 0 }]);++ host.psqlHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : undefined);++ host.pgDatabases.add('ov_live');++ host.pgDatabases.add('ov_media');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];host.repo = (id) => repoFor[id];host.ctx = () => ({ exec: host.exec, inv, log: (s) => host.logs.push(s) });@@@ -17,14 +17,13 @@ function withHost(host, rows) {units: ['openvibe-host.service'],port: 4910,ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },-- databases: [{ name: 'host', path: '/var/lib/openvibe-host-api/host.db' }],-- nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: '/var/lib/openvibe-host-api/host.db', maxUpload: '110m' } },++ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],++ nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: { engine: 'postgresql', database: 'ov_host' }, maxUpload: '110m' } },};host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });-- host.sqliteHandler = (db, sql, as) => {-- assert.strictEqual(db, '/var/lib/openvibe-host-api/host.db');++ host.psqlHandler = (db, sql) => {++ assert.strictEqual(db, 'ov_host');assert.match(sql, /^SELECT hostname FROM host_domains WHERE kind = 'custom' AND status = 'verified'/);-- assert.strictEqual(as, 'ubuntu', 'queried as the service user');return rows;};host.put('/etc/letsencrypt/live/www.tenant-a.org/fullchain.pem', 'CERT', { mode: 0o644 });@@@ -54,7 +53,7 @@ runTests([assert.strictEqual(host.read('/etc/nginx/sites-available/openvibe.host.conf'), null, 'render alone writes nothing');}),-- test('a PostgreSQL tenant database reads the verified domains over psql, never a SQLite file', async () => {++ test('a PostgreSQL tenant database reads the verified domains over psql, as PostgreSQL', async () => {const host = scenario();const doc = JSON.parse(host.read('/etc/openvibe/host.json'));// Host moved to PostgreSQL (ADR-035): its tenants database is a database in the cluster, not@@@ -76,7 +75,6 @@ runTests([const domains = await nginx.tenantDomains(host.exec, normalise(doc).services.host);assert.deepStrictEqual(domains, [{ hostname: 'www.tenant-a.org', hasCert: true }, { hostname: 'docs.tenant-b.net', hasCert: false }], 'the same [{ hostname, hasCert }] shape');assert.ok(host.psqlCalls.some((c) => c.database === 'ov_host' && c.sql === nginx.VERIFIED_SQL), 'the verified domains come from psql');-- assert.strictEqual(host.sqliteCalls.length, 0, 'a database in the cluster is never opened as a file');const r = await host.cli('nginx', 'tenants', 'host');assert.strictEqual(r.code, 0, r.out);assert.match(r.out, /server_name www\.tenant-a\.org;/);@@@ -1,6 +1,6 @@'use strict';/**-- * The two-engine inventory (ADR-035): a `databases[]` entry is SQLite (the default, a file) or++ * The PostgreSQL inventory: a `databases[]` entry is* PostgreSQL (a database in the cluster, named — its object carries NO `path`). The same engine* choice reaches the drill block ({ url, directUrl }), nginx tenants (read over psql, not a file)* and the protected `postgresql-count` probe. These are the normalised shapes the rest of ovhost@@@ -60,15 +60,9 @@ runTests([for (const database of ['trade', 'ov-Trade', 'public']) {assert.throws(() => first(withDatabases([{ engine: 'postgresql', database }])), /database must match/, database);}-- // An engine that is neither.-- assert.throws(() => first(withDatabases([{ engine: 'mysql', database: 'ov_trade' }])), /engine must be sqlite or postgresql/);-- // A SQLite entry cannot name a database, and must have a path.-- assert.throws(() => first(withDatabases([{ path: '/x/live.db', database: 'ov_live' }])), /a sqlite database is a path, not a database name/);-- assert.throws(() => first(withDatabases([{ name: 'live' }])), /path must be an absolute path/);-- }),---- test('the SQLite default is unchanged', () => {-- assert.deepStrictEqual(first(withDatabases([{ name: 'live', path: '/x/live.db' }])), { name: 'live', engine: 'sqlite', path: '/x/live.db' });++ for (const entry of [{ engine: 'mysql', database: 'ov_trade' }, { path: '/x/live.db', database: 'ov_live' }, { name: 'live' }]) {++ assert.throws(() => first(withDatabases([entry])), /engine must be "postgresql"; use/);++ }}),test('drill.databases for a PostgreSQL database is { engine, url, directUrl }', () => {@@@ -76,7 +70,7 @@ runTests([assert.deepStrictEqual(d.databases.trade, { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' });}),-- test('drill.databases rejects the SQLite shape and bad env var names for a PostgreSQL database', () => {++ test('drill.databases rejects a file database shape and bad env var names for a PostgreSQL database', () => {const drill = (databases) => pgDrill({ databases });assert.throws(() => normalise(drill({ trade: 'DATABASE_URL' })), /for a postgresql database/);assert.throws(() => normalise(drill({ trade: { url: 'DATABASE_URL' } })), /directUrl must be an env var name/);@@@ -92,16 +86,15 @@ runTests([assert.throws(() => normalise(pgDrill({ countsTolerance: 1.5 })), /countsTolerance must be a whole number of rows/);}),-- test('nginx.tenants.database accepts a path (SQLite) or a PostgreSQL database object', () => {++ test('nginx.tenants.database accepts a PostgreSQL database object', () => {const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } }).services.host.nginx.tenants.database;-- assert.deepStrictEqual(tenants('/var/lib/openvibe-host-api/host.db'), { engine: 'sqlite', path: '/var/lib/openvibe-host-api/host.db' });assert.deepStrictEqual(tenants({ engine: 'postgresql', database: 'ov_host' }), { engine: 'postgresql', database: 'ov_host' });}),test('nginx.tenants.database rejects anything else', () => {const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } });for (const database of [{ engine: 'mysql', database: 'ov_host' }, { engine: 'postgresql', database: 'host' }, { database: 'ov_host' }, 42, []]) {-- assert.throws(() => tenants(database), /nginx\.tenants\.database must be a path or/, JSON.stringify(database));++ assert.throws(() => tenants(database), /nginx\.tenants\.database must be/, JSON.stringify(database));}}),@@@ -1,56 +1,39 @@'use strict';--// Protected probes that add up parts (kind "sum", WS-P task 2): Tools' running jobs sit in each--// satellite's jobs.db, so the probe is the total of one sqlite-count per satellite. Unknown when any part--// cannot answer (never read as zero); validation checks every part and refuses a sum inside a sum.const assert = require('assert');const { countProtected } = require('../lib/probes');const { normalise } = require('../lib/inventory');const { createFakeHost } = require('./fake-host');--const counts = { '/opt/t/apps/img/data/jobs.db': 2, '/opt/t/apps/audio/data/jobs.db': 0, '/opt/t/apps/docs/data/jobs.db': 1 };++const counts = { ov_tools_img: 2, ov_tools_audio: 0, ov_tools_docs: 1 };const exec = {run: async () => ({ code: 0, stdout: 'LoadState=loaded\nActiveState=active\nSubState=running\n', stderr: '' }),-- sqlite: async (db) => { if (!(db in counts)) throw new Error(`no such database ${db}`); return [{ n: counts[db] }]; },++ psql: async (database) => { if (!(database in counts)) throw new Error(`no such database ${database}`); return [{ n: counts[database] }]; },http: async () => ({ status: 200, body: JSON.stringify({ connections: 4 }) }),};--const part = (app) => ({ kind: 'sqlite-count', db: `/opt/t/apps/${app}/data/jobs.db`, sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" });--const svc = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), part('audio'), part('docs')] } };++const part = (app) => ({ kind: 'postgresql-count', database: `ov_tools_${app}`, sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" });++const svc = { units: ['openvibe-tools.service'], protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), part('audio'), part('docs')] } };(async () => {assert.deepStrictEqual(await countProtected(exec, svc), { count: 3, label: 'running tool jobs' });const mixed = { ...svc, protected: { kind: 'sum', label: 'x', probes: [part('img'), { kind: 'http-json-count', url: 'http://127.0.0.1:4400/ready', field: 'connections' }] } };-- assert.strictEqual((await countProtected(exec, mixed)).count, 6, 'parts of different kinds add up');++ assert.strictEqual((await countProtected(exec, mixed)).count, 6);const broken = { ...svc, protected: { ...svc.protected, probes: [part('img'), part('video')] } };const r = await countProtected(exec, broken);-- assert.strictEqual(r.count, null, 'one part that cannot answer makes the total unknown, never a smaller number');++ assert.strictEqual(r.count, null);assert.match(r.unknown, /^part 2: no such database/);-- // A postgresql-count part asks the cluster over psql, as the postgres OS user (peer auth); the-- // number it answers is the count, and a psql failure is unknown — never zero.const host = createFakeHost();host.addUnit('openvibe-tools.service');-- host.sqliteHandler = () => [{ n: 2 }];host.psqlHandler = (database, sql) => { assert.strictEqual(database, 'ov_tools'); assert.match(sql, /^SELECT/); return [{ n: 7 }]; };-- const pgPart = { kind: 'postgresql-count', database: 'ov_tools', sql: 'SELECT count(*) AS n FROM tool_jobs', label: 'running tool jobs' };-- assert.deepStrictEqual(await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart }), { count: 7, label: 'running tool jobs' });-- assert.deepStrictEqual(host.psqlCalls.at(-1), { database: 'ov_tools', sql: pgPart.sql, as: 'postgres' });-- const pgSum = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), pgPart] } };-- assert.strictEqual((await countProtected(host.exec, pgSum)).count, 9, 'a postgresql-count part adds to the sum');++ const pgPart = { kind: 'postgresql-count', database: 'ov_tools', sql: 'SELECT count(*) AS n FROM tool_jobs' };++ assert.deepStrictEqual(await countProtected(host.exec, { units: ['openvibe-tools.service'], protected: pgPart }), { count: 7, label: 'protected sessions' });host.psqlHandler = () => { throw new Error('psql: could not connect to server'); };-- const down = await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart });-- assert.strictEqual(down.count, null, 'a probe that cannot ask is unknown');-- assert.notStrictEqual(down.count, 0, 'never read as zero');-- assert.match(down.unknown, /could not connect/);-- const downSum = await countProtected(host.exec, pgSum);-- assert.strictEqual(downSum.count, null);-- assert.match(downSum.unknown, /^part 2: psql: could not connect/);++ assert.match((await countProtected(host.exec, { units: ['openvibe-tools.service'], protected: pgPart })).unknown, /could not connect/);-- // The inventory checks each part.const base = { owner: 'ubuntu', repo: '/opt/t', units: ['openvibe-tools.service'] };assert.doesNotThrow(() => normalise({ services: { tools: { ...base, protected: svc.protected } } }));assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [] } } } }), /probes must be a non-empty array/);assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ ...part('img'), sql: 'DELETE FROM tool_jobs' }] } } } }), /probes\[0\]\.sql must be a SELECT/);-- assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ kind: 'sum', probes: [part('img')] }] } } } }), /probes\[0\]\.kind must be one of http-json-count, sqlite-count/);-- assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ kind: 'http-json-count', url: 'https://example.com/ready', field: 'n' }] } } } }), /loopback/);++ assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ kind: 'sum', probes: [part('img')] }] } } } }), /probes\[0\]\.kind must be one of http-json-count, postgresql-count/);console.log('probes sum: all checks passed');})().catch((err) => { console.error(err); process.exitCode = 1; });nnew file mode 100644@@@ -0,0 +1,16 @@++'use strict';++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++const { openDb } = require('../server/db');++const { test, runTests } = require('./helpers');++++runTests([++ test('production requires DATABASE_URL before creating a PGlite directory', async () => {++ const dir = path.join(os.tmpdir(), `ovhost-no-fallback-${process.pid}`);++ assert.strictEqual(fs.existsSync(dir), false);++ await assert.rejects(openDb({ isProduction: true, db: { url: '', directUrl: '', pgliteDir: dir } }), /DATABASE_URL is not set: production serves from PostgreSQL/);++ assert.strictEqual(fs.existsSync(dir), false);++ }),++]);@@@ -2,7 +2,7 @@/*** Runs every test in test/ — the files named *.test.js — each in its own process, and fails if any* of them fails. Every test drives ovhost against the in-memory fake host (test/fake-host.js):-- * no systemctl, git, npm, nginx, curl or SQLite call reaches the real machine.++ * no systemctl, git, npm, nginx or curl call reaches the real machine.** npm test # everything* npm test -- deploy certs # only files whose name contains one of the words@@@ -35,30 +35,6 @@ runTests([assert.ok(meta.lockfiles['.']);}),-- test('backup: sqlite .backup of each declared database, run as the service user, into a dated directory', async () => {-- const host = scenario();-- const r = await host.cli('backup', 'media', '--json');-- assert.strictEqual(r.code, 0, r.out);-- const res = JSON.parse(r.out);-- assert.match(res.dir, /^\/var\/backups\/openvibe\/media\/20260922-120000$/);-- assert.strictEqual(res.files[0].dest, `${res.dir}/media.db`);-- const call = host.sqliteCalls.find((c) => c.op === 'backup');-- assert.strictEqual(call.as, 'ubuntu');-- assert.strictEqual(call.db, '/opt/openvibe.media/data/media.db');-- assert.strictEqual(call.dest, '/var/backups/openvibe.staging/media-20260922-120000/media.db', 'the worker writes into its staging directory');-- // Backups hold user data: root:root, directories 0700, files 0600.-- for (const d of ['/var/backups/openvibe', '/var/backups/openvibe/media', res.dir]) {-- assert.deepStrictEqual([host.files.get(d).owner, host.files.get(d).mode], ['root', 0o700], d);-- }-- assert.deepStrictEqual([host.files.get(res.files[0].dest).owner, host.files.get(res.files[0].dest).mode], ['root', 0o600]);-- assert.strictEqual(host.files.has('/var/backups/openvibe.staging/media-20260922-120000'), false, 'staging directory removed');-- const log = host.read('/var/lib/openvibe-host/backups/media.jsonl');-- assert.strictEqual(JSON.parse(log.trim()).files[0].dest, res.files[0].dest);-- // A second backup in the same second never overwrites the first.-- const again = await host.cli('backup', 'media');-- assert.notStrictEqual(again.code, 0);-- }),--test('backup refuses a service without declared databases', async () => {const host = scenario();const r = await host.cli('backup', 'tools');@@@ -45,11 +45,11 @@ async function boot(opts = {}) {};const env = {NODE_ENV: 'test', PORT: '0', BASE_URL: `https://${DASHBOARD}`, TRUST_PROXY: '1',-- HOST_DB_PATH: path.join(dir, 'host.db'), HOST_STORAGE_DIR: path.join(dir, 'objects'),HOST_SITES_DOMAIN: 'openvibe.host',OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,OV_OAUTH_CLIENT_ID: 'host', OV_OAUTH_CLIENT_SECRET: SECRET, COOKIE_SECURE: 'false',HOST_FORM_SECRET: 'test-form-secret', HOST_WORKER: 'off', HOST_MIN_FREE_BYTES: '0',++ HOST_STORAGE_DIR: path.join(dir, 'objects'),HOST_PUBLIC_HOSTING: 'on', // the suites exercise the engine as launched; test/host-launch-switch.test.js covers off...(opts.env || {}),};@@@ -28,6 +28,8 @@ function base(services) {function finish(host, doc) {host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });host.inv = normalise(doc);++ for (const svc of Object.values(host.inv.services)) for (const db of svc.databases) host.pgDatabases.add(db.database);++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];host.ensureDir('/etc/nginx/sites-available');host.ensureDir('/etc/nginx/sites-enabled');host.badShas = new Set();@@@ -69,7 +71,7 @@ function liveEntry(overrides = {}) {noRestartPaths: ['public/', 'docs/', '**/*.md'],protected: { kind: 'http-json-count', url: 'http://127.0.0.1:3000/api/streams', array: 'streams', where: { is_live: true }, label: 'live streams' },drain: { policy: 'refuse', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },-- databases: [{ name: 'live', path: '/opt/openvibe.live/shared/data/live.db' }],++ databases: [{ name: 'live', engine: 'postgresql', database: 'ov_live' }],backupOnChange: ['server/db/schema.sql'],preflight: { syntaxCheck: true },nginx: { vhost: 'openvibe.live.conf', repoVhost: 'deploy/nginx/openvibe.live.conf', variant: 'websocket' },@@@ -121,7 +123,7 @@ async function releaseLayoutHost({ id, entry, files, initialId, unitFiles = {},}const LIVE_FILES = {-- 'package.json': pkgJson('openvibe-live', ['express', 'better-sqlite3']),++ 'package.json': pkgJson('openvibe-live', ['express', 'openvibe-sdk']),'package-lock.json': '{"lockfileVersion":3,"v":1}','server/index.js': 'console.log(1);','server/db/schema.sql': 'CREATE TABLE a(x);',@@@ -138,7 +140,6 @@ async function liveHost(overrides = {}) {host.addUnit('openvibe-live.socket', { sub: 'listening', mainPid: 0 });host.listeners.set(3000, [{ pid: 1, process: 'systemd' }]);host.put('/etc/openvibe/live.env', `JWT_SECRET=${SECRET}\n`, { mode: 0o600 });-- host.put('/opt/openvibe.live/shared/data/live.db', 'sqlite', { owner: 'ubuntu' });host.liveStreams = [];host.http.set('http://127.0.0.1:3000/api/ready', host.readyFor('openvibe-live.service'));host.http.set('http://127.0.0.1:3000/api/streams', () => ({ status: 200, body: { streams: host.liveStreams } }));@@@ -166,8 +167,8 @@ function openreEntry(overrides = {}) {port: 4500,ready: { url: 'http://127.0.0.1:4500/api/ready', timeoutSeconds: 60 },noRestartPaths: ['docs/', 'test/', '**/*.md'],-- databases: [{ name: 'openre', path: '/var/lib/openre/openre.db' }],-- protected: { kind: 'sqlite-count', db: '/var/lib/openre/openre.db', sql: "SELECT count(*) AS n FROM ingest_sessions WHERE state IN ('starting', 'live', 'ending')", label: 'ingest sessions' },++ databases: [{ name: 'openre', engine: 'postgresql', database: 'ov_openre' }],++ protected: { kind: 'postgresql-count', database: 'ov_openre', sql: "SELECT count(*) AS n FROM ingest_sessions WHERE state IN ('starting', 'live', 'ending')", label: 'ingest sessions' },drain: { policy: 'refuse' },lifecycle: lifecycleDoc(),...overrides,@@@ -175,7 +176,7 @@ function openreEntry(overrides = {}) {}const OPENRE_FILES = {-- 'package.json': pkgJson('openre-stream', ['express', 'better-sqlite3']),++ 'package.json': pkgJson('openre-stream', ['express', 'openvibe-sdk']),'package-lock.json': '{"lockfileVersion":3,"v":1}','server/index.js': 'api();','docs/README.md': '# OpenRe',@@@ -184,12 +185,11 @@ const OPENRE_FILES = {async function openreHost(overrides = {}) {const host = await releaseLayoutHost({ id: 'openre', entry: openreEntry(overrides), files: OPENRE_FILES, initialId: (sha) => sha.slice(0, 12) });host.put('/etc/openvibe/openre.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\n`, { mode: 0o600 });-- host.put('/var/lib/openre/openre.db', 'sqlite', { owner: 'ubuntu' });host.addUnit(`openre-rtmp-ingest@${host.firstRelease}.service`, { mainPid: 3102 });host.addUnit(`openre-restream-worker@${host.firstRelease}.service`, { mainPid: 3103 });host.addUnit(`openre-jsmpeg@${host.firstRelease}.service`, { mainPid: 3105 });host.sessions = 0;-- host.sqliteHandler = () => [{ n: host.sessions }];++ host.psqlHandler = () => [{ n: host.sessions }];host.http.set('http://127.0.0.1:4500/api/ready', host.readyFor('openre-api.service'));host.calls.length = 0;return host;@@@ -233,11 +233,11 @@ function toolsEntry(overrides = {}) {preflight: {dirs: ['data'],checks: [-- { label: 'jobs runtime loads', packages: ['apps/img'], argv: ['node', '-e', "const D=require('better-sqlite3'); new D(':memory:').close(); require('openvibe-contracts'); require('openvibe-sdk'); require('../_shared/jobs')"] },-- { label: 'guard loads', packages: '*', argv: ['node', '-e', "const D=require('better-sqlite3'); new D(':memory:').close(); require('../_shared/guard')"] },++ { label: 'jobs runtime loads', packages: ['apps/img'], argv: ['node', '-e', "require('openvibe-sdk/db'); require('openvibe-contracts'); require('openvibe-sdk'); require('../_shared/jobs')"] },++ { label: 'guard loads', packages: '*', argv: ['node', '-e', "require('openvibe-sdk/db'); require('../_shared/guard')"] },],},-- protected: { kind: 'sum', label: 'running tool jobs', probes: [{ kind: 'sqlite-count', db: '/opt/openvibe.tools/apps/img/data/jobs.db', sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" }] },++ protected: { kind: 'sum', label: 'running tool jobs', probes: [{ kind: 'postgresql-count', database: 'ov_tools', sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" }] },drain: { policy: 'report', waitMaxSeconds: 900, pollSeconds: 60, quietChecks: 2 },lifecycle: lifecycleDoc(),...overrides,@@@ -245,13 +245,13 @@ function toolsEntry(overrides = {}) {}const TOOLS_FILES = {-- 'apps/gateway/package.json': pkgJson('tools-gateway', ['express', 'better-sqlite3']),++ 'apps/gateway/package.json': pkgJson('tools-gateway', ['express', 'openvibe-sdk']),'apps/gateway/package-lock.json': '{"v":1}','apps/gateway/server/index.js': 'gw();',-- 'apps/img/package.json': pkgJson('tools-img', ['express', 'better-sqlite3', 'openvibe-contracts', 'openvibe-sdk']),++ 'apps/img/package.json': pkgJson('tools-img', ['express', 'openvibe-sdk', 'openvibe-contracts', 'openvibe-sdk']),'apps/img/package-lock.json': '{"v":1}','apps/img/server/index.js': 'img();',-- 'apps/maps/package.json': pkgJson('tools-maps', ['express', 'better-sqlite3']),++ 'apps/maps/package.json': pkgJson('tools-maps', ['express', 'openvibe-sdk']),'apps/maps/server/index.js': 'maps();','apps/_shared/package.json': JSON.stringify({ name: 'openvibe-tools-shared', version: '1.1.0', private: true }),'apps/_shared/guard/index.js': 'guard();',@@@ -270,7 +270,7 @@ function toolsHost(overrides = {}) {return { status: 200, body: { service: 'tools', release: host.gatewayRelease || String(u.runningSha).slice(0, 12) } };});host.jobs = 0;-- host.sqliteHandler = () => [{ n: host.jobs }];++ host.psqlHandler = () => [{ n: host.jobs }];return host;}@@@ -318,10 +318,10 @@ function gamesEntry(overrides = {}) {port: 8000,ready: { url: 'http://127.0.0.1:8000/api/ready', timeoutSeconds: 60 },generated: ['apps/client/dist-types/', 'apps/server/dist-types/'],-- preflight: { checks: [{ label: 'better-sqlite3 loads under this Node', packages: ['apps/server'], argv: ['node', '-e', "new (require('better-sqlite3'))(':memory:').close()"] }] },++ preflight: { checks: [{ label: 'database SDK loads under this Node', packages: ['apps/server'], argv: ['node', '-e', "require('openvibe-sdk/db')"] }] },protected: { kind: 'http-json-count', url: 'http://127.0.0.1:8000/api/ready', field: 'checks.sessions.detail.online', label: 'players online' },drain: { policy: 'report', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },-- databases: [{ name: 'world', path: '/opt/openvibe.games/data/world.db' }],++ databases: [{ name: 'world', engine: 'postgresql', database: 'ov_world' }],lifecycle: lifecycleDoc(),...overrides,};@@@ -331,7 +331,7 @@ const GAMES_FILES = {'package.json': JSON.stringify({ name: 'openvibe-games', private: true, scripts: { build: 'pnpm -r build' } }),'pnpm-lock.yaml': 'lockfileVersion: 9.0\nv: 1\n','pnpm-workspace.yaml': 'packages:\n - packages/*\n - apps/*\n',-- 'apps/server/package.json': pkgJson('@openvibe/server', ['@openvibe/shared', 'better-sqlite3', 'tsx']),++ 'apps/server/package.json': pkgJson('@openvibe/server', ['@openvibe/shared', 'openvibe-sdk', 'tsx']),'apps/server/src/main.ts': 'main();','apps/server/dist-types/main.d.ts': 'export {};\n','apps/client/package.json': pkgJson('@openvibe/client', ['@openvibe/shared']),@@@ -357,7 +357,6 @@ function gamesHost(overrides = {}) {host.put(path.join(cwd, 'apps/client/dist-types/main.d.ts'), `export {}; // built ${host.repo.head.slice(0, 7)}\n`, { owner: as });return { code: 0 };};-- host.put('/opt/openvibe.games/data/world.db', 'sqlite', { owner: 'ubuntu' });return host;}@@@ -43,7 +43,7 @@ runTests([host.addUnit('openvibe-tools-docs.service', { runningSha: host.repo.head });// A timer's oneshot job the glob also matches (production's openvibe-toolsjob.service): never restarted or waited on.host.addUnit('openvibe-toolsjob.service', { type: 'oneshot', active: 'inactive', sub: 'dead' });-- const to = host.push({ 'apps/img/package.json': JSON.stringify({ name: 'tools-img', version: '1.0.1', dependencies: { express: '^1', 'better-sqlite3': '^1', 'openvibe-contracts': '^1', 'openvibe-sdk': '^1', sharp: '^1' } }), 'apps/_shared/guard/index.js': 'guard2();' }, 'img: sharp');++ const to = host.push({ 'apps/img/package.json': JSON.stringify({ name: 'tools-img', version: '1.0.1', dependencies: { express: '^1', 'openvibe-sdk': '^1', 'openvibe-contracts': '^1', 'openvibe-sdk': '^1', sharp: '^1' } }), 'apps/_shared/guard/index.js': 'guard2();' }, 'img: sharp');const r = await host.cli('deploy', 'tools');assert.strictEqual(r.code, 0, r.out);assert.strictEqual(host.repo.head, to);@@@ -146,10 +146,10 @@ runTests([const host = toolsHost();const from = host.repo.head;host.push({ 'apps/img/server/index.js': 'img2();' }, 'img');-- host.onNode = (args, opts) => (/jobs/.test(args[1]) && opts.cwd.endsWith('apps/img') ? { code: 1, stderr: "Error: The module 'better_sqlite3.node' was compiled against a different Node.js version" } : undefined);++ host.onNode = (args, opts) => (/jobs/.test(args[1]) && opts.cwd.endsWith('apps/img') ? { code: 1, stderr: "Error: The module 'sharp.node' was compiled against a different Node.js version" } : undefined);const r = await host.cli('deploy', 'tools');assert.strictEqual(r.code, 2, r.out);-- assert.match(r.out, /preflight failed: jobs runtime loads failed in apps\/img: Error: The module 'better_sqlite3\.node'/);++ assert.match(r.out, /preflight failed: jobs runtime loads failed in apps\/img: Error: The module 'sharp\.node'/);assert.match(r.out, /checkout restored/);assert.strictEqual(host.repo.head, from);assert.deepStrictEqual(host.restarts(), []);@@@ -188,7 +188,7 @@ runTests([const host2 = toolsHost();host2.push({ 'apps/img/server/index.js': 'img2();' }, 'img');let polls = 0;-- host2.sqliteHandler = () => { polls += 1; return [{ n: polls < 3 ? 1 : 0 }]; };++ host2.psqlHandler = () => { polls += 1; return [{ n: polls < 3 ? 1 : 0 }]; };const w = await host2.cli('deploy', 'tools', '--wait-idle');assert.strictEqual(w.code, 0, w.out);assert.match(w.out, /--wait-idle: holding the restart/);@@@ -453,7 +453,7 @@ runTests([host.onNode = () => ({ code: 1, stderr: 'Error: Could not locate the bindings file' });const r = await host.cli('deploy', 'games');assert.strictEqual(r.code, 2, r.out);-- assert.match(r.out, /better-sqlite3 loads under this Node failed in apps\/server/);++ assert.match(r.out, /database SDK loads under this Node failed in apps\/server/);assert.strictEqual(host.repo.head, from);assert.deepStrictEqual(host.restarts(), []);assert.strictEqual(host.builds, 2, 'built for the new sha, rebuilt for the restored one');@@@ -536,7 +536,7 @@ runTests([assert.match(c.out, /^strategy=pnpm-build$/m);assert.match(c.out, /^managed=yes$/m);const unknown = await host.cli('capabilities', 'nope');-- assert.strictEqual(unknown.code, 1, 'an unknown service: the wrapper falls back');++ assert.strictEqual(unknown.code, 1, 'an unknown service is an error');const json = JSON.parse((await host.cli('capabilities', 'games', '--json')).out);assert.strictEqual(json.strategy, 'pnpm-build');const bare = await host.cli('capabilities', '--inventory', '/nope.json');@@@ -30,7 +30,6 @@ runTests([assert.ok(host.calls.some((c) => c.cmd === 'cp' && c.args[0] === '-al' && c.args[1].endsWith(`${first}/node_modules`)), 'cp -al of node_modules');assert.ok(!host.calls.some((c) => c.cmd === 'npm'), 'no install without a dependency change');assert.strictEqual(await host.exec.readlink(`/opt/openvibe.live/releases/${now}/data`), '../../shared/data');-- assert.strictEqual(host.read(`/opt/openvibe.live/current/data/live.db`), 'sqlite', 'shared data reached through the new release');// Live's syntax check: every changed .js file, in the new release, before anything restarts.const check = host.nodeChecks().find((c) => c.args[0] === '--check');assert.ok(check && check.args[1] === `/opt/openvibe.live/releases/${now}/server/index.js`);@@@ -265,11 +264,11 @@ runTests([host.push({ 'server/db/schema.sql': 'CREATE TABLE b(y);' }, 'schema');const r = await host.cli('deploy', 'live');assert.strictEqual(r.code, 0, r.out);-- const b = host.calls.findIndex((c) => c.cmd === 'sqlite-backup');++ const b = host.calls.findIndex((c) => c.cmd === 'pg_dump');const mv = host.calls.findIndex((c) => c.cmd === 'mv' && c.args.includes('/opt/openvibe.live/current'));const restart = host.calls.findIndex((c) => c.cmd === 'systemctl' && c.args[0] === 'restart');assert.ok(b >= 0 && b < mv && mv < restart, 'backup, then switch, then restart');-- assert.strictEqual(host.sqliteCalls.find((c) => c.op === 'backup').as, 'ubuntu');++ assert.strictEqual(host.dumps.find((c) => c.database === 'ov_live').as, 'postgres');}),test('live: plan shows the new release, node_modules reuse, the socket and the restart; changes nothing', async () => {@@@ -314,8 +313,7 @@ runTests([assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'release-layout', release: { links: { data: '../../../../etc' } } } } }), /must point inside/);assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'tarball' } } }), /strategy must be one of/);assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'git-checkout', layout: 'release' } } }), /needs strategy "release-layout"/);-- const legacy = normalise({ services: { x: { repo: '/opt/x', owner: 'root', layout: 'release' } } });-- assert.strictEqual(legacy.services.x.managed, false, 'a release layout without the strategy stays with its own script');++ assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', layout: 'release' } } }), /needs strategy "release-layout"/);}),// ── OpenRestream ──@@@ -335,8 +333,10 @@ runTests([assert.ok(!host.calls.some((c) => c.cmd === 'systemctl' && /openre-(rtmp-ingest|restream-worker|jsmpeg)@/.test(String(c.args[1])) && c.args[0] !== 'show'), 'no worker unit is started, stopped or restarted');assert.ok((await host.releaseIds()).includes(first), 'the release the workers run from is kept');// A second deploy prunes to keep (2), but never the release a worker instance runs from.++ host.advance(1000);host.push({ 'server/index.js': 'api3();' }, 'api3');assert.strictEqual((await host.cli('deploy', 'openre')).code, 0);++ host.advance(1000);host.push({ 'server/index.js': 'api4();' }, 'api4');assert.strictEqual((await host.cli('deploy', 'openre')).code, 0);const ids = await host.releaseIds();@@@ -9,11 +9,12 @@ function noSecrets(text) {}runTests([-- test('a release-layout service is checked by its current release, not as a git checkout, and is never managed', async () => {++ test('a release-layout service is checked by its current release and strategy clone', async () => {const host = scenario();const raw = JSON.parse(host.read('/etc/openvibe/host.json'));-- raw.services.rel = { repo: '/opt/rel.stream', layout: 'release', units: [], lifecycle: lifecycleDoc() };++ raw.services.rel = { repo: '/opt/rel.stream', strategy: 'release-layout', units: [], lifecycle: lifecycleDoc() };host.put('/etc/openvibe/host.json', JSON.stringify(raw), { mode: 0o640 });++ host.createRepo('/opt/rel.stream/repo');host.put('/opt/rel.stream/releases/655b98a10aaa/package.json', '{}');await host.exec.symlink('/opt/rel.stream/releases/655b98a10aaa', '/opt/rel.stream/current');let res = JSON.parse((await host.cli('validate', 'rel', '--json')).out);@@@ -21,7 +22,7 @@ runTests([assert.ok(res.findings.some((f) => f.area === 'checkout' && /current -> \/opt\/rel\.stream\/releases\/655b98a10aaa/.test(f.message)));const st = await host.cli('status', '--json');assert.match(st.out, /655b98a10aaa/);-- assert.match(st.out, /"managed":\s*false/);++ assert.match(st.out, /"managed":\s*true/);// current must name a release under releases/, owned by root or the checkout owner.host.put('/opt/elsewhere/package.json', '{}');host.files.delete('/opt/rel.stream/current');
@@ -22,17 +22,15 @@ HOST_SITES_DOMAIN=openvibe.host
@@ -22,17 +22,15 @@ HOST_SITES_DOMAIN=openvibe.host
# says hosting is not open yet. Set to on at launch.
# says hosting is not open yet. Set to on at launch.
HOST_PUBLIC_HOSTING=off
HOST_PUBLIC_HOSTING=off
-# SQLite database and content-addressed objects (the production unit sets both under
+# PostgreSQL database and content-addressed objects (the production unit sets both under
# /var/lib/openvibe-host-api)
# /var/lib/openvibe-host-api)
# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh host: the runtime role through PgBouncer,
# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh host: the runtime role through PgBouncer,
-# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).
+# the owner on a direct connection (migrations), and Valkey (shared limit counters).
# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
DATABASE_URL=
DATABASE_URL=
DATABASE_DIRECT_URL=
DATABASE_DIRECT_URL=
VALKEY_URL=
VALKEY_URL=
VALKEY_PREFIX=
VALKEY_PREFIX=
-# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.
-HOST_DB_PATH=./data/host.db
HOST_STORAGE_DIR=./data/objects
HOST_STORAGE_DIR=./data/objects
# Object storage. Unset/empty: local disk only (data/objects), exactly as before. `media`: every
# Object storage. Unset/empty: local disk only (data/objects), exactly as before. `media`: every
@@ -2,27 +2,27 @@
@@ -2,27 +2,27 @@
> The network's deployment/control plane first; then isolated hosting for community sites, bots and mods.
> The network's deployment/control plane first; then isolated hosting for community sites, bots and mods.
-**Status:** alpha. Stage A (operator plane) is a CLI, `ovhost`, tested against a fake host. It is installed on the production host (2026-09-23, `/usr/local/bin/ovhost`, inventory `/etc/openvibe/host.json`) and it deploys 20 services (`ovhost deploy <svc>`: Network, Events, Chat, Billing, Tips, VIP, AI, Search, Sources, Wiki, Blog, News, Reviews, Deals, Coupons, Trade, Codes, Host, Media, Community; 70+ releases since 2026-09-23, with automatic rollback on a failed readiness check) and runs the nightly backups and the restore drills of 19 services. Live, Tools, Sites, Games and OpenRestream moved to it in the 2026-09-27 cutover (roadmap WS-N task 11) through the deploy strategies `release-layout`, `multi-app`, `static-build` and `pnpm-build`, so every service now deploys with `sudo ovhost deploy <svc>` (Live with `--wait-idle`); each of those repositories keeps `deploy/scripts/deploy.sh` as a thin wrapper that hands over to ovhost or falls back to its old script: [docs/deploy-strategies.md](docs/deploy-strategies.md). Stage B (tenant static hosting) is a service, written and tested, and **launched on 2026-10-07**: `openvibe.host` serves the dashboard, API and front page, and `*.openvibe.host` serves tenant sites (vhosts from `ovhost nginx tenants host --install`; the Sites placeholder and the pending 404 vhost are gone; certificate lifecycle on `openvibe-certs.timer`; an nginx catch-all refuses hosts no site names). The launch record, verification and rollback: [docs/launch.md](docs/launch.md); the threat review: [docs/threat-review.md](docs/threat-review.md). Stage C (sandboxed user code) is **not started**.
+**Status:** alpha. Stage A (operator plane) is a CLI, `ovhost`, tested against a fake host. It is installed on the production host at `/usr/local/bin/ovhost` with inventory `/etc/openvibe/host.json`; it deploys services, runs nightly backups, and performs restore drills. Repository deploy scripts are thin wrappers around `ovhost deploy|rollback|plan`: [docs/deploy-strategies.md](docs/deploy-strategies.md). Stage B (tenant static hosting) serves the dashboard, API, front page, and tenant sites. See [docs/launch.md](docs/launch.md) and [docs/threat-review.md](docs/threat-review.md). Stage C (sandboxed user code) is **not started**.
**Domain:** `openvibe.host` (dashboard and API) and `*.openvibe.host` (tenant sites). Launched on 2026-10-07 (see [Launch rule](#launch-rule)); OpenVibe.Sites no longer serves it.
**Domain:** `openvibe.host` (dashboard and API) and `*.openvibe.host` (tenant sites). Launched on 2026-10-07 (see [Launch rule](#launch-rule)); OpenVibe.Sites no longer serves it.
**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §15.3 and §15.17; roadmap Wave 21.
**Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §15.3 and §15.17; roadmap Wave 21.
**License:** AGPL-3.0 (same as every OpenVibe service).
**License:** AGPL-3.0 (same as every OpenVibe service).
## Purpose
## Purpose
-Every repository ships its own deploy script, and each one encodes rules learned from an outage. Stage A pulls those rules into one tool that works the same way for every service on the host:
+Repository deploy scripts call ovhost. Stage A applies the deploy rules for every service on the host:
| Rule | Where it came from | How `ovhost` applies it |
| Rule | Where it came from | How `ovhost` applies it |
|---|---|---|
|---|---|---|
| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
| Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
| Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
-| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`sqlite-count` of a file, or `postgresql-count` of a database in the cluster: Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only **as the owning user** (the service user, or the postgres OS user for the cluster). |
+| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`postgresql-count` of Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only as the postgres OS user. |
| Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
| Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
| Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
| Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
| Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
| Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
| Build, install vhosts, `nginx -t`, reload | Sites `deploy.sh` | `build` hooks and `nginx.installOnDeploy`. Vhost installs are transactional: if `nginx -t` fails, the previous files come back and nginx is not reloaded. |
| Build, install vhosts, `nginx -t`, reload | Sites `deploy.sh` | `build` hooks and `nginx.installOnDeploy`. Vhost installs are transactional: if `nginx -t` fails, the previous files come back and nginx is not reloaded. |
| Build a release while the old one serves; switch a symlink; roll back to a release with its own `node_modules`; pid 1 must hold the socket | Live `deploy.sh` (release layout), OpenRestream `deploy.sh release`/`api` | Strategy `release-layout` ([docs/deploy-strategies.md](docs/deploy-strategies.md)). The socket unit is restarted only when its unit file changed or systemd does not hold the port. |
| Build a release while the old one serves; switch a symlink; roll back to a release with its own `node_modules`; pid 1 must hold the socket | Live `deploy.sh` (release layout), OpenRestream `deploy.sh release`/`api` | Strategy `release-layout` ([docs/deploy-strategies.md](docs/deploy-strategies.md)). The socket unit is restarted only when its unit file changed or systemd does not hold the port. |
| Per-app installs (not `apps/_shared`), jobs runtime and guard load before the restart, every `openvibe-tools*` unit | Tools `deploy.sh` | Strategy `multi-app`: `skipPackages`, `preflight`, `unitsMatch`, `ready.release`. |
| Per-app installs (not `apps/_shared`), jobs runtime and guard load before the restart, every `openvibe-tools*` unit | Tools `deploy.sh` | Strategy `multi-app`: `skipPackages`, `preflight`, `unitsMatch`, `ready.release`. |
-| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm, builds, SQLite queries and backups all run as the declared owner/service user (`runuser` from root, `sudo -u` otherwise); PostgreSQL dumps and probes run as the postgres OS user. |
+| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm and builds run as the checkout owner; PostgreSQL dumps and probes run as the postgres OS user. |
Stage A adds a few safety rules of its own:
Stage A adds a few safety rules of its own:
@@ -33,10 +33,10 @@ Stage A adds a few safety rules of its own:
@@ -33,10 +33,10 @@ Stage A adds a few safety rules of its own:
## Owns
## Owns
-- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled backups (online SQLite copies, pgBackRest verification and logical PostgreSQL dumps) with retention and encrypted off-host copies, restore drills.
+- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled backups (pgBackRest verification and logical PostgreSQL dumps) with retention and encrypted off-host copies, restore drills.
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
-- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user. `ovhost data provision <svc>` and `ovhost data switch <svc> [--sqlite <file>]` run `add-service.sh` and `switch-service.sh` from the installed CLI, redacting every URL and password: [docs/data-provisioning.md](docs/data-provisioning.md).
+- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user. `ovhost data provision <svc>` runs `add-service.sh` from the installed CLI, redacting every URL and password: [docs/data-provisioning.md](docs/data-provisioning.md).
## Does not own
## Does not own
@@ -82,11 +82,11 @@ credentials from `/etc/openvibe/host.env`):
@@ -82,11 +82,11 @@ credentials from `/etc/openvibe/host.env`):
| Stage | Scope | State |
| Stage | Scope | State |
|---|---|---|
|---|---|---|
-| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup, restore drill. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp SQLite/HTTP/git). Installed on the host: deploys every service, runs the nightly backups and the restore drills. |
+| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup, restore drill. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp HTTP/git). Installed on the host: deploys every service, runs the nightly backups and the restore drills. |
| **B: tenant static hosting** | The Host API service (port 4910): projects, sites, immutable content-addressed deploys, activation and rollback, `<site>.openvibe.host` and TXT-verified custom domains, quotas, upload logs, events, a server-rendered dashboard; tenant vhosts via `ovhost nginx tenants`. | **alpha**: written and tested (`npm test`, against a temp database, a mock Network and a DNS table). Runs on the host on `:4910` behind nginx; launched at openvibe.host on 2026-10-07. |
| **B: tenant static hosting** | The Host API service (port 4910): projects, sites, immutable content-addressed deploys, activation and rollback, `<site>.openvibe.host` and TXT-verified custom domains, quotas, upload logs, events, a server-rendered dashboard; tenant vhosts via `ovhost nginx tenants`. | **alpha**: written and tested (`npm test`, against a temp database, a mock Network and a DNS table). Runs on the host on `:4910` behind nginx; launched at openvibe.host on 2026-10-07. |
| **C: sandboxed user code** | Isolation profiles, CPU/memory/time/network/storage budgets, secret references, outbound policy, metering, kill/revoke without touching platform services. | **not started**, deliberately. Blocked until isolation and metering are proven. Nothing in Stage B runs tenant code. |
| **C: sandboxed user code** | Isolation profiles, CPU/memory/time/network/storage budgets, secret references, outbound policy, metering, kill/revoke without touching platform services. | **not started**, deliberately. Blocked until isolation and metering are proven. Nothing in Stage B runs tenant code. |
-Not in Stage A yet (from the charter/roadmap list): container adapters, DNS adapters, certificate **renewal**, logs/metrics links, the release-manifest/active-client work in §15.18, and the Wave 22 cutover runbook. (Incident and maintenance controls, `ovhost incident|maintenance|freeze`, and release notifications, `host.release.published`, exist now.) Restore drills (`ovhost drill`) ran on the host for 19 services on 2026-09-23 ([docs/restore-drills.md](docs/restore-drills.md)).
+Stage A also provides incident and maintenance controls (`ovhost incident|maintenance|freeze`) and release notifications (`host.release.published`). Restore drills are described in [docs/restore-drills.md](docs/restore-drills.md).
## Acceptance (Stage A)
## Acceptance (Stage A)
@@ -100,7 +100,7 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak
@@ -100,7 +100,7 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak
- The socket unit is **never stopped or restarted**, in any flow, except the release-layout rebind: only when its unit file changed or systemd does not hold the port (`test/strategy-release-layout.test.js`).
- The socket unit is **never stopped or restarted**, in any flow, except the release-layout rebind: only when its unit file changed or systemd does not hold the port (`test/strategy-release-layout.test.js`).
- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.
- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.
- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.
- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.
-- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok` (a SQLite copy) or `pg_restore --list` on the `.dump` of a PostgreSQL service. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.
+- A **restore drill** checks a PostgreSQL dump with `pg_restore --list`, restores it into a scratch database and role, starts a sandboxed second instance, compares declared HTTP paths and row counts, then stops it and drops the scratch database and role.
- **Backups:**
- **Backups:**
- `backup --all` backs up every service with databases. One failing service leaves the others backed up and exits 2.
- `backup --all` backs up every service with databases. One failing service leaves the others backed up and exits 2.
@@ -130,7 +130,6 @@ ovhost capabilities [<service>] key=value lines the deploy wrappers pro
@@ -130,7 +130,6 @@ ovhost capabilities [<service>] key=value lines the deploy wrappers pro
ovhost self-update [--dry-run] update the CLI install (/usr/local/lib/openvibe-host) itself: fetch origin main, --ff-only merge, npm ci
ovhost self-update [--dry-run] update the CLI install (/usr/local/lib/openvibe-host) itself: fetch origin main, --ff-only merge, npm ci
ovhost service add <id> [--dry-run] first setup: inventory entry, checkout, 0600 env file, database if DATABASE_URL is declared
ovhost service add <id> [--dry-run] first setup: inventory entry, checkout, 0600 env file, database if DATABASE_URL is declared
ovhost data provision <service> [--dry-run] run roles/data/add-service.sh from the install: PostgreSQL database, roles and Valkey user (names only)
ovhost data provision <service> [--dry-run] run roles/data/add-service.sh from the install: PostgreSQL database, roles and Valkey user (names only)
-ovhost data switch <service> [--sqlite <file>] [--dry-run] run roles/data/switch-service.sh: stop, back up, import, deploy (docs/data-provisioning.md)
ovhost --version
ovhost --version
ovhost announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run] release notification
ovhost announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run] release notification
ovhost releases <service> [--limit <n>]
ovhost releases <service> [--limit <n>]
@@ -150,11 +149,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
@@ -150,11 +149,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
`service add` runs as root against `/etc/openvibe/host.json`. It copies `services.<id>` from the installed CLI's `host.example.json`, drops `_note`, and backs up the live inventory before changing it. The example entry may set `cloneUrl` to a credential-free GitHub HTTPS URL; otherwise the repository is `https://github.com/OpenVibers/OpenVibe.<Capitalized id>.git`. The command refuses an existing entry, occupied checkout or env file. It holds the inventory lock (`<stateDir>/locks/_inventory.lock`) from reading the inventory to the end, and refuses a symlinked `.env.example`, which it reads with the checkout owner's permissions. It clones as the checkout owner, creates the entry's `envFile` (`/etc/openvibe/<id>.env`, 0600) as a copy of the checkout's `.env.example` with production `NODE_ENV`, `HOST` and `PORT`. It sets `BASE_URL` when the inventory vhost, origin or env example supplies a public HTTPS origin; otherwise it leaves an existing empty `BASE_URL` for the operator and does not add one to services without it. An explicitly required `BASE_URL` needs an inventory vhost or origin before setup begins. The tracked `.env.example` is never changed. It runs `data provision` if that example declares `DATABASE_URL`. Output lists env names but never values. `--dry-run` shows the plan without changing files. It then lists the env names still empty (leaving out the production overrides and, after provisioning, `DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`, `VALKEY_PREFIX`). When the example declares `OV_OAUTH_CLIENT_ID` or `OV_OAUTH_CLIENT_SECRET`, it prints one root command for the Network service principal. The command runs `server/setup/service-principal.js` through `systemd-run` with Network's env file (`services.network.envFile`, default `/etc/openvibe/network.env`) in Network's checkout (`services.network.repo`, default `/opt/openvibe.network`). It uses `rotate` when `list` already shows the id and `create` otherwise, so the principal never lands in a development database (the secret is never printed). It ends with `then: ovhost validate <id>; ovhost deploy <id> --restart`; ovhost runs neither. If a later step fails, the inventory backup path is printed; completed setup steps are not automatically undone.
`service add` runs as root against `/etc/openvibe/host.json`. It copies `services.<id>` from the installed CLI's `host.example.json`, drops `_note`, and backs up the live inventory before changing it. The example entry may set `cloneUrl` to a credential-free GitHub HTTPS URL; otherwise the repository is `https://github.com/OpenVibers/OpenVibe.<Capitalized id>.git`. The command refuses an existing entry, occupied checkout or env file. It holds the inventory lock (`<stateDir>/locks/_inventory.lock`) from reading the inventory to the end, and refuses a symlinked `.env.example`, which it reads with the checkout owner's permissions. It clones as the checkout owner, creates the entry's `envFile` (`/etc/openvibe/<id>.env`, 0600) as a copy of the checkout's `.env.example` with production `NODE_ENV`, `HOST` and `PORT`. It sets `BASE_URL` when the inventory vhost, origin or env example supplies a public HTTPS origin; otherwise it leaves an existing empty `BASE_URL` for the operator and does not add one to services without it. An explicitly required `BASE_URL` needs an inventory vhost or origin before setup begins. The tracked `.env.example` is never changed. It runs `data provision` if that example declares `DATABASE_URL`. Output lists env names but never values. `--dry-run` shows the plan without changing files. It then lists the env names still empty (leaving out the production overrides and, after provisioning, `DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`, `VALKEY_PREFIX`). When the example declares `OV_OAUTH_CLIENT_ID` or `OV_OAUTH_CLIENT_SECRET`, it prints one root command for the Network service principal. The command runs `server/setup/service-principal.js` through `systemd-run` with Network's env file (`services.network.envFile`, default `/etc/openvibe/network.env`) in Network's checkout (`services.network.repo`, default `/opt/openvibe.network`). It uses `rotate` when `list` already shows the id and `create` otherwise, so the principal never lands in a development database (the secret is never printed). It ends with `then: ovhost validate <id>; ovhost deploy <id> --restart`; ovhost runs neither. If a later step fails, the inventory backup path is printed; completed setup steps are not automatically undone.
-**Deploy strategies** (roadmap WS-N task 11). The inventory entry's `strategy` picks how a service deploys: `git-checkout` (the default: an in-place checkout, every service above), `multi-app` (Tools), `static-build` (Sites), `pnpm-build` (Games) or `release-layout` (Live, OpenRestream: `releases/<id>` behind a `current` symlink, prepared while the old release serves, `--prepare-only` to stop after preparing). Each strategy carries the rules of the script it replaces: preflight checks, build output restored from git, untracked lockfiles, `unitsMatch`, `/release.json` naming the new sha, the socket rule. [docs/deploy-strategies.md](docs/deploy-strategies.md) has the engines, every inventory field, the rule-by-rule mapping from each script, what was not ported and why, the production cutover checklist and the proposed inventory diff.
+**Deploy strategies** (roadmap WS-N task 11). The inventory entry's `strategy` picks how a service deploys: `git-checkout`, `multi-app` (Tools), `static-build` (Sites), `pnpm-build` (Games) or `release-layout` (Live, OpenRestream). Each strategy handles its service's preflight checks, build output, units and readiness rules. See [docs/deploy-strategies.md](docs/deploy-strategies.md).
`deploy --browser-check` checks the service's public site in headless Chrome after a deploy that went through. It runs [`scripts/browser-check.js`](docs/browser-check.md) for that one site. The check is report only: it never changes the exit code or the release record. It needs Chrome where ovhost runs.
`deploy --browser-check` checks the service's public site in headless Chrome after a deploy that went through. It runs [`scripts/browser-check.js`](docs/browser-check.md) for that one site. The check is report only: it never changes the exit code or the release record. It needs Chrome where ovhost runs.
-**Release notifications** (WS-P task 9). A deploy or rollback that went live is announced to OpenVibe.Events as `host.release.published`. The event is public, has subject `release` and carries the service, its `/release.json` release id, commit and origin. Open tabs (openvibe-shared release-watch 1.17.0 and later) then check `/release.json` within seconds instead of at their next poll. `ovhost announce <service>` does the same for services deployed by their own scripts (Live, Tools and Sites call it at the end of theirs). It is best effort: a few seconds at most, never a failed deploy, one event per release. The credentials are Host's service principal from `/etc/openvibe/host.env`. [docs/release-notifications.md](docs/release-notifications.md) covers the payload, the credentials, provisioning and an end-to-end check.
+**Release notifications** (WS-P task 9). A deploy or rollback that went live is announced to OpenVibe.Events as `host.release.published`. The event is public, has subject `release` and carries the service, its `/release.json` release id, commit and origin. Open tabs (openvibe-shared release-watch 1.17.0 and later) then check `/release.json` within seconds instead of at their next poll. `ovhost announce <service>` can announce the currently served release manually. It is best effort: a few seconds at most, never a failed deploy, one event per release. The credentials are Host's service principal from `/etc/openvibe/host.env`. [docs/release-notifications.md](docs/release-notifications.md) covers the payload, the credentials, provisioning and an end-to-end check.
### Browser check (all public sites)
### Browser check (all public sites)
@@ -180,11 +179,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
@@ -180,11 +179,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
`sudo ovhost backup --all --offsite` runs daily from `deploy/systemd/openvibe-backup.timer` (03:30 UTC). It works like this:
`sudo ovhost backup --all --offsite` runs daily from `deploy/systemd/openvibe-backup.timer` (03:30 UTC). It works like this:
- It backs up every service that declares databases. A failing service never stops the others; exit `2` if anything failed.
- It backs up every service that declares databases. A failing service never stops the others; exit `2` if anything failed.
-- A **SQLite** database is copied online (as the service user). A **PostgreSQL** database is verified, not copied: every run checks pgBackRest for the whole cluster and records `verified`, and a logical `pg_dump -Fc` per database is taken when the last good one is over 7 days old or on any run with `--logical`. A failed verification is `failed`, never `skipped`.
+- A **PostgreSQL** database is verified on every run: every run checks pgBackRest for the whole cluster and records `verified`, and a logical `pg_dump -Fc` per database is taken when the last good one is over 7 days old or on any run with `--logical`. A failed verification is `failed`, never `skipped`.
- It keeps the last 7 daily and 4 weekly good backups per service.
- It keeps the last 7 daily and 4 weekly good backups per service.
- It writes a JSON summary to `<stateDir>/backup-runs/<run>.json`.
- It writes a JSON summary to `<stateDir>/backup-runs/<run>.json`.
- It encrypts every copy on the host (AES-256-GCM, Node crypto, with a key that stays on the host) and uploads it to S3-compatible storage (Backblaze B2). Off-host copies are kept for 30 days.
- It encrypts every copy on the host (AES-256-GCM, Node crypto, with a key that stays on the host) and uploads it to S3-compatible storage (Backblaze B2). Off-host copies are kept for 30 days.
-- Backups are `root:root`: directories 0700, files 0600, no `-wal`/`-shm`. PostgreSQL dumps are made as the postgres OS user over peer auth; ovhost reads no service env value.
+- Backups are `root:root`: directories 0700, files 0600. PostgreSQL dumps are made as the postgres OS user over peer auth; ovhost reads no service env value.
`ovhost restore-download` fetches, verifies and decrypts a run into a new root-only directory, and never touches a live database. See [docs/backups.md](docs/backups.md) for the install steps, the env names (`/etc/openvibe/backup.env`), the encryption format and the restore procedure.
`ovhost restore-download` fetches, verifies and decrypts a run into a new root-only directory, and never touches a live database. See [docs/backups.md](docs/backups.md) for the install steps, the env names (`/etc/openvibe/backup.env`), the encryption format and the restore procedure.
@@ -195,8 +194,8 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
@@ -195,8 +194,8 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
Each inventory entry's `drill` block declares:
Each inventory entry's `drill` block declares:
- `port`: the service port + 10000 in the example;
- `port`: the service port + 10000 in the example;
-- `databases`: the env var that points the service at each restored copy, or `{ "url": "<NAME>", "directUrl": "<NAME>" }` for a PostgreSQL database (the drill creates a scratch database and role, restores the `.dump` into it, and sets both vars to it);
-- `env`: overrides that turn side effects off. Values may use `{tmp}`, `{port}` and `{db:<name>}`;
+- `databases`: `{ "url": "<NAME>", "directUrl": "<NAME>" }` for each PostgreSQL database (the drill creates a scratch database and role, restores the `.dump` into it, and sets both vars to it);
+- `env`: overrides that turn side effects off. Values may use `{tmp}` and `{port}`;
- `dirs` to create inside the drill directory;
- `dirs` to create inside the drill directory;
- `ready`: the readiness path;
- `ready`: the readiness path;
- `compare`: paths, plus volatile `ignore` keys where needed;
- `compare`: paths, plus volatile `ignore` keys where needed;
@@ -204,17 +203,17 @@ Each inventory entry's `drill` block declares:
@@ -204,17 +203,17 @@ Each inventory entry's `drill` block declares:
- `countsTolerance` (default `0`): how many rows a busy table may differ by, because production keeps writing between the backup read and the drill;
- `countsTolerance` (default `0`): how many rows a busy table may differ by, because production keeps writing between the backup read and the drill;
- `supported: false` with a `reason` when a second instance cannot run without side effects.
- `supported: false` with a `reason` when a second instance cannot run without side effects.
-In `host.example.json`, 25 services have drill blocks, and 24 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`) or a PostgreSQL database's two URL env vars, `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. A PostgreSQL drill always overrides `DATABASE_URL`, `DATABASE_DIRECT_URL` and `VALKEY_URL` and drops its scratch database and role in a `finally` ([docs/restore-drills.md](docs/restore-drills.md#a-postgresql-drill)). The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.
+In `host.example.json`, 25 services have drill blocks, and 24 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries name the PostgreSQL database's two URL env vars, `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. A PostgreSQL drill always overrides `DATABASE_URL`, `DATABASE_DIRECT_URL` and `VALKEY_URL` and drops its scratch database and role in a `finally` ([docs/restore-drills.md](docs/restore-drills.md#a-postgresql-drill)). The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
### Inventory
### Inventory
-`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `sqlite-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a SQLite file `{ "name", "path" }` or a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
+`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
-`games` deploys with the `pnpm-build` strategy (a pnpm workspace with a TypeScript build: `pnpm install --frozen-lockfile`, `pnpm build`, the tracked `dist-types/` restored from git before the merge). An entry with `layout: "release"` and no `strategy` stays unmanaged: `deploy` and `rollback` refuse it and the repository's own script deploys it. The deploy-strategy fields (`strategy`, `release`, `preflight`, `skipPackages`, `removeUntrackedLockfiles`, `generated`, `installUnits`, `unitsMatch`, `announce.releaseFiles`, `install.lockfile`, `install.workspace`, `ready.release`, `ready.allUnits`) are listed in [docs/deploy-strategies.md](docs/deploy-strategies.md#inventory-fields).
+`games` deploys with the `pnpm-build` strategy (a pnpm workspace with a TypeScript build: `pnpm install --frozen-lockfile`, `pnpm build`, the tracked `dist-types/` restored from git before the merge). A release layout requires `strategy: "release-layout"`. The deploy-strategy fields (`strategy`, `release`, `preflight`, `skipPackages`, `removeUntrackedLockfiles`, `generated`, `installUnits`, `unitsMatch`, `announce.releaseFiles`, `install.lockfile`, `install.workspace`, `ready.release`, `ready.allUnits`) are listed in [docs/deploy-strategies.md](docs/deploy-strategies.md#inventory-fields).
## Deploy
## Deploy
@@ -243,17 +242,17 @@ This was done on 2026-09-23 (the CLI and inventory are installed); the steps sta
@@ -243,17 +242,17 @@ This was done on 2026-09-23 (the CLI and inventory are installed); the steps sta
cd /usr/local/lib/openvibe-host && sudo npm ci --omit=dev --no-audit --no-fund
cd /usr/local/lib/openvibe-host && sudo npm ci --omit=dev --no-audit --no-fund
sudo ln -s /usr/local/lib/openvibe-host/bin/ovhost /usr/local/bin/ovhost
sudo ln -s /usr/local/lib/openvibe-host/bin/ovhost /usr/local/bin/ovhost
```
```
- The tree must stay world-readable (default 0755/0644). The SQLite worker runs as `ubuntu` from this directory.
+ The tree must stay world-readable (default 0755/0644).
2. **Inventory:** `sudo install -o root -g root -m 0640 host.example.json /etc/openvibe/host.json`, then correct it against the machine. Check the Community database path, the Tools unit list, and Events, which was not deployed as of 22 Sep.
2. **Inventory:** `sudo install -o root -g root -m 0640 host.example.json /etc/openvibe/host.json`, then correct it against the machine. Check the Community database path, the Tools unit list, and Events, which was not deployed as of 22 Sep.
-3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots, drill logs and backup run summaries. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe` and everything in it is root-only (directories 0700, files 0600). The backup worker writes as the service user into its own directory under `/var/backups/openvibe.staging` (root, 0711), and root takes each copy over from there. All of these are created on first use.
-4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm/SQLite (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.
+3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots, drill logs and backup run summaries. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe` and everything in it is root-only (directories 0700, files 0600). The postgres OS user writes logical dumps into its own directory under `/var/backups/openvibe.staging` (root, 0711), and root takes each copy over from there. All of these are created on first use.
+4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.
5. **First run:** `sudo ovhost status`, then `sudo ovhost validate <each service>`, then `sudo ovhost plan live`, all read-only apart from `git fetch`. Adopt `deploy` one service at a time, starting with one that has no protected sessions.
5. **First run:** `sudo ovhost status`, then `sudo ovhost validate <each service>`, then `sudo ovhost plan live`, all read-only apart from `git fetch`. Adopt `deploy` one service at a time, starting with one that has no protected sessions.
6. **Scheduled backups:** see [docs/backups.md](docs/backups.md#installing-on-the-host-operator) (bucket, key, `/etc/openvibe/backup.env`, `openvibe-backup.timer`).
6. **Scheduled backups:** see [docs/backups.md](docs/backups.md#installing-on-the-host-operator) (bucket, key, `/etc/openvibe/backup.env`, `openvibe-backup.timer`).
7. **Updating the CLI itself:** a deploy of this repository replaces `/opt/openvibe.host` (the Stage B service), not the root-owned install at `/usr/local/lib/openvibe-host`, so after a merge to `main` run `sudo ovhost self-update`. It finds its own install directory from its real path (`/usr/local/bin/ovhost` → `/usr/local/lib/openvibe-host/bin/ovhost`), fetches `origin main`, refuses a dirty tree, a branch other than `main` or a divergence, merges `--ff-only` and runs `npm ci --omit=dev --no-audit --no-fund` there; `--dry-run` prints the plan first. It restarts nothing — the next `ovhost` invocation runs the new code. Without it the installed CLI silently runs behind `main` (on 2026-10-02 it was 16 commits behind: no `env-names --set`, no `logs`, no Bot vhost).
7. **Updating the CLI itself:** a deploy of this repository replaces `/opt/openvibe.host` (the Stage B service), not the root-owned install at `/usr/local/lib/openvibe-host`, so after a merge to `main` run `sudo ovhost self-update`. It finds its own install directory from its real path (`/usr/local/bin/ovhost` → `/usr/local/lib/openvibe-host/bin/ovhost`), fetches `origin main`, refuses a dirty tree, a branch other than `main` or a divergence, merges `--ff-only` and runs `npm ci --omit=dev --no-audit --no-fund` there; `--dry-run` prints the plan first. It restarts nothing — the next `ovhost` invocation runs the new code. Without it the installed CLI silently runs behind `main` (on 2026-10-02 it was 16 commits behind: no `env-names --set`, no `logs`, no Bot vhost).
## Stage B: tenant static hosting
## Stage B: tenant static hosting
-A Node service (`server/`, Express 4, better-sqlite3, port **4910**, service id `host`) that hosts static sites for OpenVibe projects. Every request is dispatched on its `Host` header first:
+A Node service (`server/`, Express 4, PostgreSQL, port **4910**, service id `host`) that hosts static sites for OpenVibe projects. Every request is dispatched on its `Host` header first:
| Host | What answers |
| Host | What answers |
|---|---|
|---|---|
@@ -516,7 +515,7 @@ records: [docs/threat-review.md](docs/threat-review.md). The rules the code keep
@@ -516,7 +515,7 @@ records: [docs/threat-review.md](docs/threat-review.md). The rules the code keep
stores, prints or snapshots a value. The one exception is its own root-only `/etc/openvibe/backup.env`.
stores, prints or snapshots a value. The one exception is its own root-only `/etc/openvibe/backup.env`.
Backups are root-only (0700/0600) and off-host copies are AES-256-GCM encrypted and authenticated.
Backups are root-only (0700/0600) and off-host copies are AES-256-GCM encrypted and authenticated.
- **Operator plane.** `ovhost` runs as root and drops to the checkout owner for git, npm, builds and
- **Operator plane.** `ovhost` runs as root and drops to the checkout owner for git, npm, builds and
- SQLite; it never stops or restarts a socket unit except the release-layout rebind; a protected-session
+ PostgreSQL; it never stops or restarts a socket unit except the release-layout rebind; a protected-session
probe that cannot answer counts as sessions active; tracked local changes block a deploy.
probe that cannot answer counts as sessions active; tracked local changes block a deploy.
- **Tenants (Stage B).** Static files only: no tenant code runs, no tenant secret is stored, uploads are
- **Tenants (Stage B).** Static files only: no tenant code runs, no tenant secret is stored, uploads are
validated and content-addressed, tenant hosts get no CORS grant, staff takedowns answer 451. Custom
validated and content-addressed, tenant hosts get no CORS grant, staff takedowns answer 451. Custom
@@ -535,7 +534,7 @@ npm run dev # Stage B service on http://localhost:4910 (HOST_SIT
@@ -535,7 +534,7 @@ npm run dev # Stage B service on http://localhost:4910 (HOST_SIT
Stage B tests (`test/host-*.test.js`) boot the service on a temp database and object store with a controllable clock, a mock Network (real RS256 keys) and a DNS table, and talk to it over raw HTTP so they can send any `Host` header and request target.
Stage B tests (`test/host-*.test.js`) boot the service on a temp database and object store with a controllable clock, a mock Network (real RS256 keys) and a DNS table, and talk to it over raw HTTP so they can send any `Host` header and request target.
-Stage A: everything that touches the system goes through the executor (`lib/executor.js`). The tests replace it with `test/fake-host.js`, an in-memory host with git repos, systemd units, npm, nginx, ss, HTTP endpoints and SQLite. No test runs systemctl, git, npm, nginx or curl against the real machine. `test/executor.test.js` covers the real executor only where that is safe: SQLite on a temp database, HTTP to a local server, and temp files.
+Stage A: everything that touches the system goes through the executor (`lib/executor.js`). The tests replace it with `test/fake-host.js`, an in-memory host with git repos, systemd units, npm, nginx, ss, HTTP endpoints and PostgreSQL. No test runs systemctl, git, npm, nginx or curl against the real machine. `test/executor.test.js` covers the real executor only where that is safe: HTTP to a local server and temp files.
## Launch rule
## Launch rule
@@ -56,7 +56,7 @@
@@ -56,7 +56,7 @@
"deploy strategies: git-checkout, multi-app, static-build, pnpm-build, release-layout",
"deploy strategies: git-checkout, multi-app, static-build, pnpm-build, release-layout",
"protected-session probes and drain policies (--wait-idle)",
"protected-session probes and drain policies (--wait-idle)",
"certificate inventory and transactional nginx vhost install",
"certificate inventory and transactional nginx vhost install",
- "scheduled SQLite backups with retention and encrypted off-host copies",
+ "scheduled PostgreSQL backup verification, logical dumps, retention and encrypted off-host copies",
"restore drills",
"restore drills",
"release notifications (host.release.published), incidents, maintenance windows and freezes",
"release notifications (host.release.published), incidents, maintenance windows and freezes",
"Prometheus alerts relay to Network",
"Prometheus alerts relay to Network",
@@ -46,17 +46,6 @@ groups:
@@ -46,17 +46,6 @@ groups:
labels: { severity: page }
labels: { severity: page }
annotations:
annotations:
summary: "No successful nightly backup for more than 30 hours (or no backup metrics at all)"
summary: "No successful nightly backup for more than 30 hours (or no backup metrics at all)"
- # A service whose databases are gone (the 2026-09 SQLite→PostgreSQL switch archived the .db files
- # and dropped the path from the inventory) is reported `skipped` by backup --all. openvibe_backup_last_run_ok
- # stays 1 (a skip is not a failure) and no other backup alert fires, so production can sit with no backups
- # at all. 30 h is longer than the 24 h between runs, so ONE skipped run never reaches `for` and TWO
- # consecutive ones always do.
- - alert: OpenVibeBackupSkipped
- expr: openvibe_backup_services{status="skipped"} > 0
- for: 30h
- labels: { severity: ticket }
- annotations:
- summary: "A service has been skipped by two nightly backups in a row ({{ with query \"openvibe_backup_services{status='skipped'}\" }}{{ . | first | value }}{{ end }} service(s)): its databases are gone, so it has no backups"
# Production drift (OpenVibe.Network registry/deploy-drift.js, hourly against GitHub; WS-S task 7).
# Production drift (OpenVibe.Network registry/deploy-drift.js, hourly against GitHub; WS-S task 7).
# max_over_time: Network restarts drop the series for a minute, which reset `for` and made the alert
# max_over_time: Network restarts drop the series for a minute, which reset `for` and made the alert
# resolve and fire again every few hours (2026-09-28). Documentation-only drift is not counted.
# resolve and fire again every few hours (2026-09-28). Documentation-only drift is not counted.
@@ -1,13 +1,12 @@
@@ -1,13 +1,12 @@
[Unit]
[Unit]
-Description=OpenVibe backups: verify pgBackRest for the PostgreSQL services, dump SQLite databases and the scheduled logical PostgreSQL dumps, then an encrypted off-host copy
+Description=OpenVibe backups: verify pgBackRest, create scheduled logical PostgreSQL dumps, then an encrypted off-host copy
Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.md
Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.md
After=network-online.target
After=network-online.target
Wants=network-online.target
Wants=network-online.target
[Service]
[Service]
Type=oneshot
Type=oneshot
-# Root: ovhost drops to each service user for the SQLite backup and to the postgres OS user for the
-# PostgreSQL dumps and the pgBackRest verification (runuser) and is the only reader of
+# Root: ovhost uses the postgres OS user for dumps and pgBackRest verification and is the only reader of
# /etc/openvibe/backup.env, the encryption key and the root-only backups. The env file is read by
# /etc/openvibe/backup.env, the encryption key and the root-only backups. The env file is read by
# ovhost itself, not loaded into this unit's environment.
# ovhost itself, not loaded into this unit's environment.
User=root
User=root
@@ -19,8 +18,7 @@ UMask=0077
@@ -19,8 +18,7 @@ UMask=0077
Nice=10
Nice=10
IOSchedulingClass=best-effort
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOSchedulingPriority=7
-# The PostgreSQL logical dumps are the large ones now (they run as the postgres OS user), and the
-# SQLite worker's own limit is 6 h per database; a pg_dump is given the same. 6 h covers a whole run.
+# A pg_dump has a six-hour limit, which also covers a whole run.
TimeoutStartSec=6h
TimeoutStartSec=6h
PrivateTmp=true
PrivateTmp=true
NoNewPrivileges=true
NoNewPrivileges=true
@@ -34,7 +34,6 @@ LockPersonality=true
@@ -34,7 +34,6 @@ LockPersonality=true
# root-owned state directory (release logs, locks, snapshots).
# root-owned state directory (release logs, locks, snapshots).
StateDirectory=openvibe-host-api
StateDirectory=openvibe-host-api
StateDirectoryMode=0750
StateDirectoryMode=0750
-Environment=HOST_DB_PATH=/var/lib/openvibe-host-api/host.db
Environment=HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects
Environment=HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects
[Install]
[Install]
@@ -2,9 +2,7 @@
@@ -2,9 +2,7 @@
`ovhost backup --all --offsite` runs once a day from a systemd timer. It backs up every database
`ovhost backup --all --offsite` runs once a day from a systemd timer. It backs up every database
the inventory declares, keeps a week of daily and a month of weekly copies on the host, and uploads
the inventory declares, keeps a week of daily and a month of weekly copies on the host, and uploads
-an encrypted copy of each run to S3-compatible object storage (Backblaze B2). A SQLite database is
-copied; a PostgreSQL service is covered by the cluster's pgBackRest backups, which every run verifies,
-with a per-service logical dump on a slower cadence. This page covers what it does, how to install it,
+an encrypted copy of each run to S3-compatible object storage (Backblaze B2). The cluster is covered by pgBackRest, which every run verifies, with a per-service logical dump on a slower cadence. This page covers what it does, how to install it,
and how to restore from the off-host copy.
and how to restore from the off-host copy.
Restore drills (a second instance started from a backup and compared with production) are in
Restore drills (a second instance started from a backup and compared with production) are in
@@ -13,81 +11,11 @@ Restore drills (a second instance started from a backup and compared with produc
@@ -13,81 +11,11 @@ Restore drills (a second instance started from a backup and compared with produc
## What is backed up
## What is backed up
-`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has
-a `databases` or `objects` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):
-SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster
-(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:
-
-| Service | Engine | Databases |
-|---|---|---|
-| network | SQLite | `network.db` |
-| chat | SQLite | `chat.db` |
-| tools | SQLite | seven `apps/*/data/analytics.db`, and `jobs.db` for img, audio and docs |
-| games | SQLite | `world.db` |
-| live | SQLite | `live.db` (includes the money ledger while `BILLING_AUTHORITY=live`), `rs-companion.db`. `data/analytics.db` is left out on purpose ([storage-state.md](storage-state.md): analytics, hard-coded path, not canonical). |
-| tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media, openre | PostgreSQL | one database each in the cluster, named `ov_<name>` (for example `ov_trade`); there is no local file. The media files themselves are in B2/R2, not covered here. |
-
-The 19 PostgreSQL services each run the every-night pgBackRest verification and a scheduled logical
-dump; the five SQLite services keep the online copy described below. [PostgreSQL
-services](#postgresql-services) covers what that means.
-
-A database that does not exist yet (a service that is not deployed) is reported as `skipped` and
-does not fail the run. Adding a database to the inventory is all it takes to include it.
-
-**Object directories.** A service may also declare `objects`: a list of `{ name, path }` directories
-whose files are content-addressed, each named by its own sha256. OpenVibe.Host's tenant object store
-(`HOST_STORAGE_DIR=/var/lib/openvibe-host-api/objects`) is the first. `ovhost backup` archives each
-directory as `<name>.tar.gz` (root:root 0600, like every other copy); the off-host code encrypts and
-uploads it with the rest, and a [drill](restore-drills.md) extracts it into its own scratch directory
-and checks every blob's sha256.
+Every inventory database is PostgreSQL. `ovhost backup --all` verifies the cluster's pgBackRest backups and WAL archiver on every run. It makes a per-service `pg_dump -Fc` when the last good logical dump is over seven days old or when `--logical` is given. Declared content-addressed object directories are archived as `.tar.gz` files. Restore drills use the logical dumps and object archives.
## One run
## One run
-For each service, one after another:
-
-1. **Copy.** The SQLite online backup API (`better-sqlite3` `.backup()`) makes a consistent copy while
- the service keeps running. The copy is made by a worker running **as the service user**: root never
- opens a service database, because a root-owned `-wal`/`-shm` file would stop the service from
- writing its own database.
-2. **Check.** The worker switches the copy to `journal_mode=DELETE`, which checkpoints it and removes
- its `-wal`/`-shm`, and runs `PRAGMA quick_check`. A copy that does not answer `ok` fails.
-3. **Take over.** The worker writes into `/var/backups/openvibe.staging/<service>-<stamp>/`, which
- belongs to the service user. Root then takes that directory back (`root:root 0700`), so the service
- user can no longer reach it. Root checks that each copy is a plain file with one link, and makes it
- `root:root 0600`. It then renames the directory to `/var/backups/openvibe/<service>/<stamp>/`.
-4. **Record.** The backup is appended to `/var/lib/openvibe-host/backups/<service>.jsonl`, with
- `ok: false` when any database failed.
-5. **Prune** (see [Retention](#retention)).
-
-Steps 1-3 are the **SQLite** path. A PostgreSQL service does not copy a file at all: every run first
-verifies pgBackRest, and a logical dump is taken only when the last good one is old enough (see
-[PostgreSQL services](#postgresql-services)). `drill` uses the same `.dump` artifacts.
-
-**Permissions.** Backups hold user data. Every directory under `/var/backups/openvibe` is
-`root:root 0700` and every copy is `root:root 0600`, with no `-wal`/`-shm` next to it. Directories left
-by older versions (`ubuntu`, 0750) are locked down on the next run. The staging parent
-`/var/backups/openvibe.staging` is `root:root 0711`, and each run removes its own staging directory.
-
-**Failures.** A failing database never stops the other databases of the service, and a failing
-service never stops the other services. The run's exit code is `2` when anything failed and `0`
-otherwise. If a service fails halfway (one of two databases), the good copy is kept but the backup is
-marked `ok: false`. Such a backup is never picked by a drill, and it is pruned once a newer good backup
-exists. A PostgreSQL service whose pgBackRest verification fails is reported `failed`, **never
-`skipped`**: a broken backup layer is a failure, not an absent database.
-
-**Summary.** Each run writes `/var/lib/openvibe-host/backup-runs/<run>.json` (`root`, 0640). It holds
-per-service status, files, sizes, errors, what was pruned, and the off-host result: objects, bytes,
-failures, and the key id. A PostgreSQL service that only verified pgBackRest is reported `verified`.
-The summary never holds a secret. The run id is the start time in UTC, for example `20260924-033412`.
-
-```
-sudo ovhost backup --all # local only
-sudo ovhost backup --all --offsite # what the timer runs
-sudo ovhost backup --all --json # the summary on stdout
-sudo ovhost backup --all --logical # force the logical PostgreSQL dumps too
-sudo ovhost backup --all --keep-daily 14 --keep-weekly 8
-sudo ovhost backup --all --no-prune
-```
+Run `sudo ovhost backup --all --offsite`. A failed pgBackRest or WAL check is recorded as a failure. A run with no dump due and no object directories records successful verification without creating an artifact directory. Backup artifacts are root-owned, with 0700 directories and 0600 files; `pg_dump` writes into a postgres-owned staging directory before root takes ownership. The off-site uploader encrypts artifacts before upload.
## PostgreSQL services
## PostgreSQL services
@@ -120,7 +48,7 @@ layout, one file per database:
@@ -120,7 +48,7 @@ layout, one file per database:
/var/backups/openvibe/<service>/<stamp>/<name>.dump # pg_dump -Fc, owned root:root 0600
/var/backups/openvibe/<service>/<stamp>/<name>.dump # pg_dump -Fc, owned root:root 0600
```
```
-The off-site code then encrypts and uploads each `.dump` exactly as it does a `.db` copy, so a `.dump`
+The off-site code encrypts and uploads each `.dump`, so the archive
reaches the bucket as `<service>/<name>.dump.ovbk` and is restored by `restore-download` unchanged.
reaches the bucket as `<service>/<name>.dump.ovbk` and is restored by `restore-download` unchanged.
`pg_restore --list` proves a `.dump` is a readable archive.
`pg_restore --list` proves a `.dump` is a readable archive.
@@ -137,11 +65,7 @@ node_exporter's textfile collector:
@@ -137,11 +65,7 @@ node_exporter's textfile collector:
- `DataBackupStale` fires when `openvibe_pgbackrest_last_backup_age_seconds` is over **30 hours**, or the
- `DataBackupStale` fires when `openvibe_pgbackrest_last_backup_age_seconds` is over **30 hours**, or the
metric is absent (no run has written it). It is the cluster staleness alert; the per-service
metric is absent (no run has written it). It is the cluster staleness alert; the per-service
`openvibe_backup_pg_verified` says which service's verification failed.
`openvibe_backup_pg_verified` says which service's verification failed.
-- `OpenVibeBackupSkipped` fires when a service has been `skipped` for **30 hours** — two nightly runs in a
- row. A service whose databases are gone (for example one whose SQLite file was archived after the
- 2026-09 switch to PostgreSQL) is `skipped`, `openvibe_backup_last_run_ok` stays `1` and nothing else
- fires, so this is what catches "production with no backups". The rules live in
- [`deploy/prometheus/openvibe-rules.yml`](../deploy/prometheus/openvibe-rules.yml).
+The rules live in [`deploy/prometheus/openvibe-rules.yml`](../deploy/prometheus/openvibe-rules.yml).
### pgBackRest is the durability layer
### pgBackRest is the durability layer
@@ -174,7 +98,6 @@ To leave deletion to a bucket lifecycle rule instead, set `BACKUP_OFFSITE_PRUNE=
@@ -174,7 +98,6 @@ To leave deletion to a bucket lifecycle rule instead, set `BACKUP_OFFSITE_PRUNE=
### Layout
### Layout
```
```
-s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.db.ovbk # a SQLite copy
s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.dump.ovbk # a PostgreSQL dump
s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.dump.ovbk # a PostgreSQL dump
s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/manifest.json
s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/manifest.json
```
```
@@ -312,8 +235,7 @@ sudo jq '{ok, services: [.services[] | {service, status, error}], offsite: {ok:
@@ -312,8 +235,7 @@ sudo jq '{ok, services: [.services[] | {service, status, error}], offsite: {ok:
The monitoring stack alerts on it (all in `deploy/prometheus/openvibe-rules.yml`):
The monitoring stack alerts on it (all in `deploy/prometheus/openvibe-rules.yml`):
`OpenVibeBackupFailed` pages when the last run failed, `OpenVibeBackupMissed` when no run has succeeded
`OpenVibeBackupFailed` pages when the last run failed, `OpenVibeBackupMissed` when no run has succeeded
-for 30 hours, `OpenVibeBackupSkipped` when a service has no databases left to back up, and
-`DataBackupStale` when pgBackRest itself is stale. The unit also fails (exit 2) and the summary says
+for 30 hours, and `DataBackupStale` when pgBackRest itself is stale. The unit also fails (exit 2) and the summary says
`"ok": false`.
`"ok": false`.
### Bucket (Backblaze B2)
### Bucket (Backblaze B2)
@@ -365,38 +287,14 @@ For each file, the command:
@@ -365,38 +287,14 @@ For each file, the command:
- checks the manifest's HMAC;
- checks the manifest's HMAC;
- streams the object through GCM decryption;
- streams the object through GCM decryption;
- checks both SHA-256 hashes against the manifest;
- checks both SHA-256 hashes against the manifest;
-- runs `PRAGMA quick_check` on the result of a `.db` copy, or `pg_restore --list` on a `.dump`.
+- runs `pg_restore --list` on each `.dump`.
Any failure removes that file and exits non-zero. The output lists each file with its size, its hash,
Any failure removes that file and exits non-zero. The output lists each file with its size, its hash,
its check and the production path it came from.
its check and the production path it came from.
-### 3. Put it in place (manual, one service at a time)
-
-Take a fresh local backup of the current state first, even a broken one, so the restore can be undone.
-Then stop the service, move its current database aside, install the copy as the service user's file,
-and start the service again. For Live, check protected sessions first (`ovhost status live`). The
-socket unit stays up, and only the service unit stops.
+### 3. Restore a service database
-```
-sudo ovhost backup live # the state you are about to replace
-sudo systemctl stop openvibe-live.service # never the .socket unit
-cd /opt/openvibe.live/data
-sudo mv live.db live.db.replaced-$(date -u +%Y%m%d-%H%M%S)
-sudo rm -f live.db-wal live.db-shm # only after the service has stopped
-sudo install -o ubuntu -g ubuntu -m 0640 /var/lib/openvibe-restore/live-20260924-033412/live.db live.db
-sudo systemctl start openvibe-live.service
-curl -s 127.0.0.1:3000/api/ready | jq .status
-```
-
-- Use the production path from the restore output (`was …`). The owner and mode must match the other
- files in that directory (`ls -l`).
-- For services with a `StateDirectory` (`/var/lib/openvibe-*`), the directory is owned by the service
- user.
-- Remove `/var/lib/openvibe-restore/<service>-<run>` when you are done: it holds user data.
-- **Money.** `live.db` holds the ledger while `BILLING_AUTHORITY=live`. A restore loses every
- transaction made after the backup. Before any money action resumes, reconcile against the payment
- provider for the gap (see [cutover-runbook.md](cutover-runbook.md)). Consider setting the owner-only
- `money_writes_frozen` flag first.
+Use `pg_restore` from the downloaded `.dump` into a new PostgreSQL database, then point the service's `DATABASE_URL` and `DATABASE_DIRECT_URL` at it. Check the application before changing production settings. `ovhost drill` automates a scratch restore and comparison without changing the production database.
### Disaster recovery: a new host
### Disaster recovery: a new host
@@ -427,27 +325,7 @@ sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-community
@@ -427,27 +325,7 @@ sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-community
sudo rm -rf /var/lib/openvibe-restore/drill-community
sudo rm -rf /var/lib/openvibe-restore/drill-community
```
```
-The drill:
-- restores each database. A SQLite copy goes into the sandbox as the service user (`install -o <user>
- -m 0600`) and is checked with `integrity_check`; a PostgreSQL `.dump` is restored with `pg_restore`
- into a scratch database created for the drill ([restore-drills.md](restore-drills.md));
-- starts a sandboxed second instance and compares it with production;
-- stops it and cleans up (a PostgreSQL drill always drops its scratch database and role).
-
-Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
-
-For a service without a drill (ai), or for live and media until the release with their drill
-switch is deployed (`LIVE_DRILL`, `MEDIA_DRILL`; `drill.requires` refuses an older checkout), download the
-copy and check it by hand:
-
-```
-sudo ovhost restore-download live latest --out /var/lib/openvibe-restore/check-live
-sudo sqlite3 -readonly /var/lib/openvibe-restore/check-live/live.db 'PRAGMA integrity_check; SELECT count(*) FROM users;'
-sudo rm -rf /var/lib/openvibe-restore/check-live
-```
-
-(`sqlite3` is the Debian package of the same name. `restore-download` has already run `quick_check`;
-this adds a full `integrity_check` and a look at the data.)
+The drill verifies each PostgreSQL dump with `pg_restore --list`, restores it into a scratch database, compares the second instance with production, and drops the scratch database and role. For a service without a supported drill, verify the downloaded `.dump` with `pg_restore --list` before a manual scratch restore.
Do this at least once after installing, and again after any change to the key or the bucket.
Do this at least once after installing, and again after any change to the key or the bucket.
@@ -455,7 +333,7 @@ Do this at least once after installing, and again after any change to the key or
@@ -455,7 +333,7 @@ Do this at least once after installing, and again after any change to the key or
| Command | What it does | Exit |
| Command | What it does | Exit |
|---|---|---|
|---|---|---|
-| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases or object directories (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`, archive object directories); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
+| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases or object directories (verify pgBackRest, dump PostgreSQL when due or `--logical`, archive object directories); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
@@ -134,22 +134,6 @@ estate now carries a service token with the one capability it performs (Contract
@@ -134,22 +134,6 @@ estate now carries a service token with the one capability it performs (Contract
| C-84 | **Live deploy notice over the Chat bridge** (`chat-remote.js` `deployNotice`) next to the `live.release.deployed` event (Live `e73a92d`). | Live `server/chat/deploy-notice.js`, `server/chat/chat-remote.js` | Chat + Live | *done 2026-09-23*: Chat subscribes to `live.release.deployed` (Chat `ed90461`, table `deploy_releases` keyed by head) and folds by `subject.id`; the 22:14 UTC deploy produced one card via both paths (`duplicate:release`), and since Live `5324f40` (22:18 UTC deploy: `first_via = events`, folded) the bridge hop runs only while Live's Events publishing is off | 2026-10-14 | Keep the bridge call |
| C-84 | **Live deploy notice over the Chat bridge** (`chat-remote.js` `deployNotice`) next to the `live.release.deployed` event (Live `e73a92d`). | Live `server/chat/deploy-notice.js`, `server/chat/chat-remote.js` | Chat + Live | *done 2026-09-23*: Chat subscribes to `live.release.deployed` (Chat `ed90461`, table `deploy_releases` keyed by head) and folds by `subject.id`; the 22:14 UTC deploy produced one card via both paths (`duplicate:release`), and since Live `5324f40` (22:18 UTC deploy: `first_via = events`, folded) the bridge hop runs only while Live's Events publishing is off | 2026-10-14 | Keep the bridge call |
| C-85 | **Live's own go-live follower push** (`server/streaming/golive-notify.js`) next to Network's `live.stream.started` consumer (subscription since 2026-09-23 19:55 UTC; Live `GET /internal/followers`). A shared per-streamer rate limit in Network stops double notifications. | Live `golive-notify.js` + call sites in `routes.js`, `rtmp-server.js`, `whip-handler.js`, `openre/mirror.js`; Network `server/notifications/stream-live.js` | Live + Network | *documented* (Network commit 2a95548): Network's log shows `outcome: notified` for real go-lives; then delete Live's push and retire Network's `POST /internal/events/stream-live`. | 2026-10-07 | Unsubscribe Network from `live.stream.started` |
| C-85 | **Live's own go-live follower push** (`server/streaming/golive-notify.js`) next to Network's `live.stream.started` consumer (subscription since 2026-09-23 19:55 UTC; Live `GET /internal/followers`). A shared per-streamer rate limit in Network stops double notifications. | Live `golive-notify.js` + call sites in `routes.js`, `rtmp-server.js`, `whip-handler.js`, `openre/mirror.js`; Network `server/notifications/stream-live.js` | Live + Network | *documented* (Network commit 2a95548): Network's log shows `outcome: notified` for real go-lives; then delete Live's push and retire Network's `POST /internal/events/stream-live`. | 2026-10-07 | Unsubscribe Network from `live.stream.started` |
-## PostgreSQL switches (ADR-035, roadmap WS-X2)
-
-Each service that moves keeps its SQLite file read-only for the 7-day N-1 window, as the rollback (the previous release reads it). One row per service, added at its switch.
-
-| # | What | Where | Owner | Removal condition | Target | Rollback lever until then |
-|---|---|---|---|---|---|---|
-| C-89 | **Closed 2026-09-29 (plan T1):** the SQLite files of the 19 services that moved to PostgreSQL on 2026-09-28 (post-switch files and pre-switch backups: tips, wiki, blog, news, deals, coupons, trade, reviews, search, sources, codes, vip, ai, host, events, community, billing, media, openre), Community's pre-c24/pre-live-comments backups and Billing's `live-snapshot-202609230121.db` are archived off-host (`ovhost archive` `20260929-124813`: 44 files, 805 MB encrypted; a restore of `codes.db` verified byte for byte) and deleted from the host. The rollback is `ovhost archive restore 20260929-124813 <path> --out <dir>` plus a release older than the switch. | the archive | Host | — | done | `ovhost archive restore` |
-
-**Archive step unblocked (2026-09-28).** `ovhost archive push` refuses a path the inventory declares as a
-live SQLite database (`lib/archive.js`: it refuses a file that is any non-postgresql `databases[].path`).
-Until the switch, that rule covered these frozen `.db` files, so the "archive both files, then delete"
-step of C-89 could not run. The 19 services above now declare `engine: postgresql` with no `path`, so their
-sqlite entries no longer name a file and archive push accepts the frozen `.db` files. The archive-then-delete
-step of each C-89 row can proceed once its 7-day window closes; the removal condition and target of every
-row stand unchanged.
-
## Kept by design (not shims)
## Kept by design (not shims)
These match the search terms but are permanent. They are listed so the next review does not re-open them.
These match the search terms but are permanent. They are listed so the next review does not re-open them.
deleted file mode 100644
deleted file mode 100644
@@ -1,379 +0,0 @@
@@ -1,379 +0,0 @@
-# Final cutover runbook (roadmap Wave 22)
-
-The Wave 22 exit criterion is: "The final cutover runbook, with rollback, is absorbed into OpenVibe.Host."
-This page gathers every service cutover of the realignment into one ordered runbook. Each section gives the
-cutover's production state and its rollback, and links the per-repository document that stays
-authoritative for the details. When a step here and its original disagree, the original wins; fix this
-page.
-
-- Host: `ssh openvibe-ovh` (single host). The public IPv4 is in `~/OpenVibers/RUNBOOK.md` on the
- operator workstation, which is not in a repository.
-- Env files: `/etc/openvibe/<service>.env`. Units: `openvibe-<service>.service`. Checkouts:
- `/opt/openvibe.<service>`.
-- Live paths are from `~/orca/workspaces/OpenVibe.Live/seadragon` (`ae07553`).
-- Production state is from the operator notes of 2026-09-23.
-
-State markers:
-- **DONE**: executed on production, with the date.
-- **IN PROGRESS**: partly on production.
-- **BLOCKED (owner)**: waits for something only the owner can do. The action is spelled out.
-- **READY**: nothing blocks it.
-- **NOT STARTED**
-
-## Overview
-
-| # | Cutover | State | Per-repo original | Rollback in one line |
-|---|---|---|---|---|
-| A | Pastes → OpenVibe.Community (Wave 5) | **DONE** 2026-09-22 ~23:35 UTC | ADR-011; Community `README.md:16-21,354,387-390` | `PASTES_AUTHORITY=live` in Community and Live, remove Media's two flags, restart all three |
-| B | Chat → OpenVibe.Chat (Wave 6) | **DONE** 2026-09-23 02:03 UTC | Chat [`docs/cutover.md`](https://github.com/OpenVibers/OpenVibe.Chat/blob/main/docs/cutover.md) | Remove the nginx include, drain the mirror, unset `CHAT_AUTHORITY`, restart Live |
-| C | AI → OpenVibe.AI (Wave 13) | **DONE** 2026-09-23 ~15:40 UTC (`AI_SERVICE=remote`) | AI [`docs/migration.md`](https://github.com/OpenVibers/OpenVibe.AI/blob/main/docs/migration.md) | Unset `AI_SERVICE`, restart Live |
-| D | Events delivery signature v2 | **IN PROGRESS**: Events sends v2; consumer deploys unknown; v1 still sent | Events [`docs/replay-window-rollout.md`](https://github.com/OpenVibers/OpenVibe.Events/blob/main/docs/replay-window-rollout.md) | The consumer redeploys its previous release; replay the DLQ |
-| E | Money → OpenVibe.Billing | **BLOCKED (owner)**: PowerChat dashboard. Billing runs in shadow. | Billing [`docs/live-cutover.md`](https://github.com/OpenVibers/OpenVibe.Billing/blob/main/docs/live-cutover.md) | Depends on the step reached. After step 9 there is **no tool** |
-| F | RTMP ingest → OpenRestream | **BLOCKED (owner)**: broadcaster windows and the per-slot rehearsal. DNS `ingest.openre.stream` is done, port 1936 is open (2026-09-23) and OpenRestream is deployed. | OpenRestream [`docs/cutover.md`](https://github.com/OpenVibers/OpenRestream/blob/main/docs/cutover.md) (steps, checks and rollback; `scripts/cutover-preflight.js`) | Per slot `{"authority":"live"}` + `set-definition-state.js --state disabled`. All slots: unset `OPENRE_URL` |
-| G | Retire the shims | **NOT STARTED**. Dated in [compatibility-register.md](compatibility-register.md) | Network [`docs/retirement.md`](https://github.com/OpenVibers/OpenVibe.Network/blob/main/docs/retirement.md) | per entry |
-
-The remaining order is D → E → F → G.
-- D first: nothing depends on it, and v1 can stop only when every consumer requires v2.
-- E before F: the Billing switch restarts Live once, and it is best done before any stream depends on
- OpenRestream.
-- G follows each entry's date in the register.
-
-## What the owner must do
-
-These actions need the owner's accounts or physical presence. Nothing in this runbook can do them.
-
-1. **PowerChat dashboard, for step E4.** In the PowerChat developer dashboard, for OpenVibe.Live's app,
- change the webhook URL:
- - from `https://openvibe.live/api/powerchat/webhook`
- - to **`https://billing.openvibe.network/webhooks/powerchat`**
-
- Keep the **same signing secret** (Billing's `POWERCHAT_WEBHOOK_SECRET` equals Live's
- `powerchat_webhook_secret`). Do this only at step E4 of the sequence, **not before**: Live freezes
- first, and Billing is frozen to hold deliveries. A 410 from Live makes PowerChat disable the endpoint
- (Billing `docs/live-cutover.md:131-134, 170-175`).
-2. **Cloudflare DNS, for step F.** Create an A record `ingest.openre.stream` pointing at the host's IP,
- **DNS only (grey cloud)**. RTMP cannot go through Cloudflare's proxy (OpenRestream `README.md:145`,
- `deploy/nginx/openre.stream.conf:15-17`).
-3. **Port 1936/tcp, for step F.** Open it in the host firewall **and** at the provider edge (OpenRestream
- `README.md:145,160`). *Done 2026-09-23: 1936 answers from outside since 19:24 UTC.*
-3a. **DNS changes in a cutover (any step that moves a hostname).** A day before: lower the record's TTL to 60 s
- and write down the current records (`dig +noall +answer <name> A AAAA CNAME` into the step's notes). At the
- switch: change the record, then watch the new target for 30 minutes (readiness, error rate, the step's own
- checks) before calling the step done. Keep the old target running until the old TTL has passed twice over;
- only then decommission it, and put the TTL back.
-4. **Broadcaster windows, for step F.** For each slot, agree a maintenance window with its broadcaster.
- The slot is offline, and the broadcaster regenerates the key and pastes the new server into OBS
- (ADR-009).
-5. **Cloudflare browser cache TTL** (ADR-016, not a cutover, but it affects every deploy). Set the zone
- to "Respect existing headers". Today Cloudflare raises `/shared/*` `max-age=300` to 4 h.
-
-## 0. Deploy rules that apply to every step
-
-These rules are learned from outages. `ovhost` encodes them (see [README](../README.md#purpose)). As of
-2026-09-23 it is used read-only on the host, so the per-repo scripts are still what runs. Since WS-N task 11
-(2026-09-27) those scripts are wrappers around `ovhost deploy <svc>` that fall back to the old script
-(`deploy/scripts/deploy-legacy.sh`) until the host's ovhost and inventory are cut over
-([deploy-strategies.md](deploy-strategies.md)); the rules below hold either way.
-
-- **Live**: `sudo /opt/openvibe.live/deploy/scripts/deploy.sh [--wait-idle|--restart|--force]`
- (`deploy/scripts/deploy.sh:7-12`).
- - Production uses the legacy in-place layout (Live `docs/deploy.md:31`). `--rollback` is refused
- there (`deploy.sh:347`), and a failed-readiness rollback resets the checkout without restoring
- `node_modules`. The release layout (`deploy/scripts/migrate-to-releases.sh`) is register entry C-76.
- - **`openvibe-live.socket` survives restarts.** It has no `PartOf=`
- (`deploy/systemd/openvibe-live.socket:13-23`). The script makes sure it is active and restarts only
- `openvibe-live` (`deploy.sh:205-222`). HTTP queues on the socket during boot. WebSocket, WHIP, WebRTC
- and RTMP sessions drop and reconnect. **Never stop or restart the socket.**
- - Readiness: polls `/api/ready` every 1 s for up to 90 s. If it is not ready, the script rolls back and
- exits 3; if the rollback also fails, it exits 4 (**manual intervention**) (`deploy.sh:31-32,139-147`).
- - `--wait-idle` polls `/api/streams` every 60 s and needs 2 idle checks in a row, for up to 8 h
- (`deploy.sh:120-137`). The script treats a failed `curl` as zero streams (`deploy.sh:121-125`).
- Check by hand when it matters.
- - `public/`, `docs/` and tests deploy without a restart. Schema changes take an online backup to
- `data/backups/live-<ts>.db` first (`deploy.sh:192-203`). `deploy/nginx/` is **never** installed by
- the script.
- - Owner decision, 2026-09-23: restarts while streamers are live are permitted "for now". Prefer
- `--wait-idle` when it costs nothing, and say when a restart hit a live stream.
-- **Media**: there is **no deploy script** in the repo. Pull, `npm ci --omit=dev` if dependencies
- changed, then `systemctl restart openvibe-media`.
- - **Do not restart Media while it is recording** (hazard H3,
- `OpenVibe.Network/docs/roadmap-baseline/10-hazards.md:11`). Check first:
- `curl -s 127.0.0.1:4100/api/ready | jq .recordings_in_progress`, or ovhost's probe
- `SELECT count(*) FROM vods WHERE is_recording = 1` run read-only as the service user.
- - If a restart does cut a recording: 5 s after boot, Media finalizes every `is_recording = 1` row that
- has no ffmpeg. It merges chunks, remuxes, falls back to `.master.mkv`, and marks very short results
- `needs_review` (`server/index.js:197-210`, `server/vod/finalize.js`). Interrupted clips are marked
- failed.
- - **Known gap:** the unit's `TimeoutStopSec=30` (`deploy/systemd/openvibe-media.service:16`) is
- shorter than the recorder's ffmpeg stop grace (`VOD_STOP_GRACE_MS`, default 60 000 ms,
- `server/vod/recorder.js:35`). systemd therefore SIGKILLs a stop that is still flushing a recording,
- and the boot sweep has to recover it.
-- **Tools**: `deploy/scripts/deploy.sh` (run from the checkout).
- 1. Fetch, and drop untracked lockfiles that the incoming release tracks.
- 2. `git pull --ff-only`, and `npm install --omit=dev` in apps whose `package.json` changed.
- 3. Reinstall any dependency that does not resolve. If it still does not, **ABORT, nothing restarted**.
- 4. Check that img, audio and docs load the jobs runtime.
- 5. Restart **all** `openvibe-tools*` units at once, then curl the gateway's `/api/health`.
-
- (`deploy/scripts/deploy.sh:1-42`.) It has **no automatic rollback** and does not check `/api/ready`.
- `apps/gateway/deploy/scripts/deploy.sh` is the superseded per-app script. Do not use it.
-- **OpenRestream**: `sudo bash /opt/openre.stream/repo/deploy/scripts/deploy.sh release|api|workers|status|prune`.
- - `api` restarts the API and coordinator only, polls `/api/ready` for 60 s, and rolls back on failure
- (exit 2).
- - `workers` starts a new generation. The old ones drain for up to 30 min.
- - **Never `systemctl restart` a worker instance during a broadcast.** (OpenRestream `README.md:147-160`.)
-- **Everything else** (Network, Community, Events, Chat, Billing, and the loopback services): pull,
- install if the lockfile changed, restart the unit, then poll `/api/ready`. Network's own
- `deploy.sh` and Sites' `deploy.sh` exist (Host `README.md`, rules table).
-- **Always**:
- - Stage explicit paths when committing on the host. Never `git add -A`: on 2026-09-23 02:09 it
- committed half-done work and took Network down for two minutes.
- - Production WebSocket probes send `join` only, never `chat` or `dm` frames.
- - Take `sudo ovhost backup <service>` before any data move.
-
-## A. Pastes → OpenVibe.Community (Wave 5): DONE 2026-09-22
-
-Community owns pastes and comments (ADR-011). Production values:
-
-| Service | Setting | Effect |
-|---|---|---|
-| Community | `PASTES_AUTHORITY=community` in `community.env` | Serves pastes natively. The database is `/var/lib/openvibe-community/community.db`. |
-| Live | `PASTES_AUTHORITY=community` in `live.env` | `/api/pastes` forwards to Community with a service token (`server/pastes-client.js`, `server/media-proxy/pastes.js:250-319`). |
-| Media | `PASTES_FROZEN_APPS=live` | Paste writes answer 410 (`server/pastes/routes.js:47-55`). |
-| Media | `PASTES_MOVED_TO=https://openvibe.community` | 301 for `/p/:slug` and raw text (`server/public/routes.js:446-494`). |
-| Tools | none | Forwards `/api/pastes/*` to Community with the visitor's token, and has no switch (`apps/gateway/server/index.js:183-207`). |
-
-Import: 889+2 pastes, 60 comments, 48 likes. Held: 1 paste and 4 comments, `ambiguous_owner`. Shadow
-parity before the flip was 878/878 (ADR-011 §Migration consequences).
-
-**Verify:** `curl -s 127.0.0.1:4200/api/ready | jq .pastes_authority` must print `"community"`.
-
-**Rollback** (ADR-011 §Rollback):
-1. Export the writes made in Community since the cutover. **No export tool exists.** Write one before you
- need it (register C-10).
-2. Set `PASTES_AUTHORITY=live` in `community.env` and `live.env`.
-3. Remove `PASTES_FROZEN_APPS` and `PASTES_MOVED_TO` from `media.env`.
-4. Restart Community, Live and Media.
-
- Tools has no switch: revert `fd5d318` in Tools and deploy. Community deletes are soft and never
- remove Media bytes.
-
-## B. Chat → OpenVibe.Chat (Wave 6): DONE 2026-09-23 02:03 UTC
-
-Chat serves `/ws/chat`, `/api/chat/`, `/api/dm/`, `/api/tts/` and `/api/sounds`. It runs on
-127.0.0.1:4400 (`openvibe-chat`) with `/var/lib/openvibe-chat/chat.db`, `LIVE_MIRROR=1` and `EVENTS_URL`
-set. Traffic reaches it through `include /opt/openvibe.chat/deploy/nginx/*.locations.conf;` in
-openvibe.live's vhost. Live runs `CHAT_AUTHORITY=chat` and `OV_CHAT_INTERNAL_URL`. Its chat tables are a
-read mirror kept current by Chat (register C-01–C-03).
-
-Record of the run (Chat `docs/cutover.md`):
-- Prerequisites `:51-84`: Network principal `chat` and grants; Live patch deployed without the flag;
- Chat installed and not started.
-- Rehearsal `:86-123`.
-- Cutover `:125-167`:
- 1. Start Chat and import snapshot 1.
- 2. Set `CHAT_AUTHORITY=chat`, restart Live, add the include, `nginx -t && systemctl reload nginx`.
- 3. Import pass 2. `held` must be 0.
- 4. Check that `mirror.pending` and `chat_bridge_outbox` are near 0, and that no `ChatRemote` refusals
- appear.
-- Imports: pass 1 had 70,860 messages, held 0. Parity was 15/15 read paths (ADR-010 §Acceptance tests).
-- The pre-cutover nginx backup is `/root/openvibe.live.conf.pre-chat-<ts>` on the host.
-
-**Verify:** `curl -s 127.0.0.1:4400/ready | jq '.live, .mirror'`.
-- `mirror.pending` should be near 0.
-- `live.failures` should not grow.
-
-`/ready` stays 200 while Live syncs fail (see [storage-state.md](storage-state.md) finding 1), so read the
-body, not just the status.
-
-**Rollback** (Chat `docs/cutover.md:169-184`, ADR-010 §Rollback). The order matters:
-1. Remove the include from openvibe.live's vhost, then `sudo nginx -t && sudo systemctl reload nginx`.
- Chat traffic returns to Live, which answers 503 until step 3.
-2. Make sure everything Chat wrote has reached Live. `curl -s 127.0.0.1:4400/ready | jq .mirror.pending`
- must be `0`. If Chat is down, run `scripts/mirror-flush.js` in the Chat checkout; exit 0 means it is
- drained. This must happen **before** step 3, because Live accepts mirror writes only while
- `CHAT_AUTHORITY=chat`.
-3. Remove `CHAT_AUTHORITY` from `/etc/openvibe/live.env` and restart Live
- (`deploy.sh --restart`). At boot, `drainToLocal()` applies `chat_bridge_outbox` to Live's own tables
- (`server/chat/chat-remote.js:351-372`). A write Chat applied but never acknowledged is applied twice
- (`:354`).
-4. `sudo systemctl stop openvibe-chat`. Keep `/var/lib/openvibe-chat/chat.db`.
-
-## C. AI → OpenVibe.AI (Wave 13): DONE 2026-09-23
-
-OpenVibe.AI runs on 127.0.0.1:4700 as principal `ai`, with `/var/lib/openvibe-ai/ai.db`. Live runs
-`AI_SERVICE=remote` (`server/ai/ai-service.js:39`). With it, `llm.complete`, translation, paste/frame,
-overview, media and recap calls go through AI workflows. There is **no fallback to Live's own key** when
-AI returns nothing (`server/ai/llm.js:299-313`). Streamer BYO providers bypass AI.
-
-Steps from AI `docs/migration.md:72-99`:
-- Steps 1-6 (deploy, import, grants, Live `AI_SERVICE=remote`, verify) are **DONE**.
-- Step 7 is **DONE**: Network calls `network.site_copy` itself (Network `422f8e9`). Live's
- `/internal/ai/site-copy` remains as Network's fallback (register C-21).
-- The Live grant names its namespaces (`live.*`, `network.site_copy`; Network `988330b`), and AI fails
- closed on a token with no `ns` (register C-22/C-23).
-
-**Verify:**
-- `curl -s 127.0.0.1:4700/api/ready`.
-- `GET /api/v1/runs?workflow=live.translate` shows recent succeeded runs.
-- `journalctl -u openvibe-live | grep '\[AI service\]'` is empty.
-
-**Rollback** (AI `docs/migration.md:101-103`, ADR-015 §Rollback):
-1. Unset `AI_SERVICE` in `live.env` and restart Live. Live then uses its own provider settings. **Keep the
- shared key in Live's settings** until C-20's removal date. Without it, this rollback has no provider.
-2. Namespace problems only: set `AI_NS_FALLBACK` back to its default, or set `AI_NS_REQUIRED=false`, in
- `ai.env`, then restart AI.
-
-Stopping OpenVibe.AI touches no product data.
-
-## D. Events delivery signature v2: IN PROGRESS
-
-Events signs every attempt with v1 (`X-OpenVibe-Signature`) **and** v2 (`X-OpenVibe-Timestamp`,
-`X-OpenVibe-Signature-V2` = HMAC of `"<t>.<raw body>"`, ±300 s) (Events `server/worker.js:56,66-68`).
-v1 cannot be switched off by env.
-
-| Step (Events `docs/replay-window-rollout.md:22-34`) | State |
-|---|---|
-| 1. Tag openvibe-sdk `v0.4.0` | **DONE**: the tag exists. The doc's status line (`:3`) is stale. |
-| 2. Deploy Events. Confirm one real delivery carries the v2 headers. Compare `date +%s` with a trusted clock. | **DONE**: Events sends v2 (operator notes). |
-| 3. Codes docs | **DONE** in the repo (Codes `a9371c3`). The tester still checks v1 only (register C-61). |
-| 4. Consumers, one at a time, in this order: Live (`/internal/openre-events`), Search, News, Deals, Trade, Reviews, then Tips and VIP after a clean day, then Examples | **Committed in every repo**: Live `ae07553`, Search `5a7e59b`, News `1802e7d`, Deals `e72aeec`, Trade `9a55644`, Reviews `3151c05`, Tips `435f32b`, VIP `dc98781`, Examples `6ea9cfd`. **Deployed: unknown.** Check `git -C /opt/openvibe.<svc> log -1` for each. |
-| 5. Stop sending v1 (optional; only after every consumer requires v2) | **NOT STARTED** (register C-60, target 2026-10-07) |
-
-For each consumer deploy:
-- Pin `openvibe-sdk` v0.4.0 (Search has no SDK), run `npm test` on Node 22, and deploy.
-- Watch that service for `401 *.bad_signature`.
-- Watch Events with `GET /api/v1/deliveries?status=failed&subscription_id=…`.
-
-**Rollback** (rollout doc `:36`, ADR-004): redeploy the consumer's previous release, which unsets
-`requireV2`. Events keeps retrying for about 2 h 13 min (1 s, 5 s, 30 s, 2 min, 10 min, 1 h, 1 h), then
-moves the delivery to the DLQ. Requeue with `POST /api/v1/deliveries/replay { subscription_id, event_ids }`
-or `{ subscription_id, from_seq }`. Replays are signed afresh, and the consumer inbox dedupes on
-`event_id`. Rolling back Events itself is safe only while no consumer requires v2.
-
-## E. Money → OpenVibe.Billing: BLOCKED on the owner (PowerChat dashboard)
-
-Billing runs in shadow on 127.0.0.1:4600 (principal `billing`), and its staff console is public at
-`https://billing.openvibe.network`. Live is the authority: `BILLING_AUTHORITY` unset = `live`
-(`server/monetization/money-authority.js:26-31`). Billing's own status is "prepared, **not executed**"
-(Billing `docs/live-cutover.md:3`).
-
-**Before the day** (Billing `docs/live-cutover.md:56-102`):
-
-| Prerequisite | State |
-|---|---|
-| 1. `billing.env`: `POWERCHAT_WEBHOOK_SECRET` equal to Live's, `POWERCHAT_SITE_USERNAME`, `POWERCHAT_ALLOW_TEST_FULFILLMENT` off, `BILLING_*` rates equal to Live's | `POWERCHAT_SITE_USERNAME` set 2026-09-23 (operator notes). The rest: unknown. Check with `sudo ovhost validate billing`, which prints names only. |
-| 2. Network grants for client `live`: `billing.intent.create`, `billing.transfer.create`, `billing.balance.read`, `billing.cashout.request`, `billing.subscription.manage`, `billing.entitlement.check`, plus `identity.subject.resolve`. **Not** `billing.cashout.manage` or `billing.ledger.admin`. | **DONE** (operator notes: Live granted `billing.*` except `cashout.manage` and `ledger.admin`) |
-| 3. Staff console: redirect URI, `BILLING_STAFF_SUBJECTS`, `BILLING_SESSION_SECRET`, vhost, sign-in works | **DONE** (console live; staff subject is the owner only) |
-| 4. Live patch deployed with `BILLING_AUTHORITY` unset; `/api/admin/money` shows `authority: "live"` | **DONE** (Live `c384787`) |
-| 5. `identity-legacy-sync` has run since the last new account; a fresh shadow import has no holds for anyone with a balance | Shadow import done 2026-09-23: 322 transactions, reconcile OK. Re-run it on the day. |
-
-**The sequence** (Billing `docs/live-cutover.md:116-168`). Keep the order: it guarantees that every
-PowerChat delivery lands in exactly one ledger.
-
-0. Check who is live: `curl -s https://openvibe.live/api/streams`. Step 6 restarts Live.
-1. Freeze Live money writes: `POST /api/admin/money/freeze {on:true, reason:'Billing cutover'}` (owner).
- `GET /api/admin/money` must show `frozen: true`.
-2. Wait **at least 60 minutes**, because minted PowerChat links live for an hour. Then run
- `POST /api/powerchat/reconcile`. List `payment_orders` that are `pending` or `paid` from the last 3
- days. A `paid` row is an anomaly; fix it first.
-3. Back up Billing to `/var/lib/openvibe-billing/billing-pre-cutover.db`, **keep it for rollback**. Then
- run `node scripts/freeze.js on 'Live cutover'`.
-4. **Owner: re-point the PowerChat webhook** (see "What the owner must do", item 1). From here on, Billing
- stores and holds every delivery (202).
-5. Take the final snapshot to `/var/backups/openvibe/live-cutover-<date>.db` and **keep it**. Run
- `import-live.js --dry-run`, then the import, then `reconcile.js`. `holds` must be empty for anyone with
- money, and reconciliation must print **OK**.
-6. Add `BILLING_AUTHORITY=billing`, `OV_BILLING_INTERNAL_URL=http://127.0.0.1:4600` and
- `OV_BILLING_PUBLIC_URL=https://billing.openvibe.network` to `live.env`. Run
- `sudo /opt/openvibe.live/deploy/scripts/deploy.sh --wait-idle`.
- `GET /api/admin/money` must show `authority: "billing"`, `reachable: true` and `drift: []`.
-7. Unfreeze Billing with `node scripts/freeze.js off`. Held deliveries settle once. Deliveries that Live
- already credited are rejected (`billing.intent_settled_in_live`). Then `reconcile.js` must print OK.
-8. Verify the reads while Live is still frozen: balances match, and `billing.actions.attention: []`.
-9. Unfreeze Live: `POST /api/admin/money/freeze {on:false}`.
-10. Make a real 100-Vibes PowerChat purchase. Expect one settled event, +100 exactly once, reconcile OK,
- and Live's legacy columns unchanged.
-
-**Rollback** (Billing `docs/live-cutover.md:177-200`):
-- **Before step 6:**
- 1. Re-point PowerChat back to `https://openvibe.live/api/powerchat/webhook`.
- 2. Leave Billing **frozen**.
- 3. Unfreeze Live. Live's reconciler (every 15 min) backfills payments made in the window.
-- **After step 6, before step 9:**
- 1. Remove `BILLING_AUTHORITY` and run `deploy.sh --wait-idle`.
- 2. Re-point PowerChat to Live and unfreeze Live.
- 3. Stop Billing, restore `billing-pre-cutover.db` (delete its `-wal`/`-shm` files), freeze it, and start
- it.
-- **After step 9:** ADR-012's rollback re-derives Live's columns from Billing's journal. **No tool for this
- exists.** It is manual and lossy. Prefer fixing forward.
-
-**What the switch does not preserve** (`:202-239`):
-- PowerChat side effects that arrived on Live's webhook (chat lines, alerts, goals) stop at the re-point.
-- Displays that read Live's `transactions` table freeze.
-- Renewal notices stop.
-- The SPA sends no `Idempotency-Key` yet.
-
-Read the full list before choosing the day. VIP checkout stays closed until this cutover.
-
-## F. RTMP ingest → OpenRestream: BLOCKED on the owner (rehearsal and broadcaster windows; DNS and port 1936 are done)
-
-OpenRestream is deployed. The API is `openre-api` on 127.0.0.1:4500. The other units are
-`openre-session-coordinator`, `openre-rtmp-ingest@<sha>` and `openre-restream-worker@<sha>`. The database
-is `/var/lib/openre/openre.db`. RTMP is bound to 127.0.0.1:1936 (`OPENRE_RTMP_BIND`). A loopback rehearsal
-with an ffmpeg test pattern passed: the session went live→ended, a wrong key was refused, `openre.session.*`
-reached Events, and the key never appeared in logs.
-
-Live carries the integration (`f0ca18b`). It is inert until `OPENRE_URL` is set. `ingest_authority`
-defaults to `live` on every slot (`server/openre/schema.js:5-7`), and only RTMP moves
-(`server/openre/authority.js:18`). WHIP, JSMPEG and SFU are **not ported** (OpenRestream `README.md:207-209`).
-
-**Once, before the first slot.** OpenRestream's [`docs/cutover.md`](https://github.com/OpenVibers/OpenRestream/blob/main/docs/cutover.md) supersedes the table below: it has the exact commands, the owner and agent steps (A1–A7), and a read-only preflight whose output on 2026-09-23 settles the "Unknown" rows (no subscription, no `OPENRE_*` in `live.env`).
-
-| Step | State |
-|---|---|
-| 1. Network client `openre` and its grants; release the contract proposals | **DONE** (operator notes: principal `openre` in `openre.env`; openre capabilities released by contracts v0.19.0) |
-| 2. Deploy OpenRestream; `curl 127.0.0.1:4500/api/ready` shows ready, with one ingest worker, one restream worker and a valid coordinator lease | **DONE** |
-| — Owner: DNS-only `ingest.openre.stream`; 1936/tcp open at the host and provider edge; RTMP bound publicly (`OPENRE_RTMP_BIND`) | **BLOCKED (owner)** |
-| 3. Events subscription `openre.session.*` → `http://127.0.0.1:3000/internal/openre-events`; its secret in Live as `OPENRE_EVENTS_SECRET` | Unknown. Check `GET /api/v1/subscriptions` on Events and `ovhost validate live`. |
-| 4. Live: `OPENRE_URL=http://127.0.0.1:4500`, `OPENRE_PUBLIC_URL`, then `deploy.sh --wait-idle`. With no slot switched this changes nothing. | Unknown. Check `GET /api/admin/openre/status`. |
-| 5. Rehearsal with a test account from OBS to `rtmp://ingest.openre.stream:1936/live`. Deploy Live and `deploy.sh api` during the broadcast, and check that nothing drops. | Needs the DNS and port first |
-
-**Per slot** (`README.md:196-203`; `migrate-from-live.js --checklist` prints the same list):
-1. Agree a window with the broadcaster. The slot must be offline.
-2. Run `node scripts/migrate-from-live.js --live-db <fresh snapshot> --apply --slots <id>`. Review held
- destinations on openre.stream.
-3. `PUT /api/admin/openre/managed/<id>/ingest-authority {"authority":"openre"}` on Live. From then on,
- Live refuses the old key and has rotated its own copy (`server/openre/authority.js:103-146`).
-4. The broadcaster presses **Regenerate stream key** on Go Live (it is shown once) and sets OBS to
- `rtmp://ingest.openre.stream:1936/live`.
-5. Rotate the broadcaster's personal Live key (`users.stream_key`). ADR-009 requires every pre-cutover
- key to be rotated.
-6. Test a broadcast. Watch `deploy.sh status`, `/api/admin/openre/status` and output health.
-
-**Rollback:**
-- Per slot: `{"authority":"live"}`. The broadcaster regenerates the key on Live and points OBS back at the
- Live RTMP URL.
-- All slots: unset `OPENRE_URL` in `live.env` and restart Live (`README.md:205`,
- `server/openre/authority.js:47-49`).
-- Live's ingest code stays until the last protocol has run on OpenRestream for two weeks (ADR-009 §Rollback).
-- Port 1935 moves to OpenRestream (`OPENRE_RTMP_EXTRA_PORTS=1935`) only after Live stops listening
- (`README.md:139-145`).
-
-## G. Retiring the shims: NOT STARTED
-
-Every remaining shim has an owner, a removal condition, a date and a rollback lever in
-[compatibility-register.md](compatibility-register.md). The ones on the critical path:
-
-- **The shared internal key** (C-50–C-58), per Network `docs/retirement.md`:
- 1. Move each caller to a service token by 2026-10-09.
- 2. Remove each key path after 14 days with no legacy use. Check with the `principal_usage` query in
- that doc.
- 3. Delete `INTERNAL_API_KEY` from every env file by 2026-11-06.
-- **The hard-coded `openvibe-internal-2026` secret** in Live cosmetics (C-59): remove it by 2026-10-09.
-- **Events v1 signatures** (C-60): stop them after every consumer is deployed with `requireV2`.
-- **Chat's Live mirror** (C-01–C-03), and Live's frozen legacy tables and their readers (C-73): target
- 2026-11-06.
-
-Before removing a shim, take `sudo ovhost backup <service>` for any service whose data it touches. Update
-the register row in the same change.
@@ -1,58 +1,5 @@
@@ -1,58 +1,5 @@
-# Provisioning a service's PostgreSQL database (`ovhost data`)
+# Provisioning a service database
-The data role ([roles/data/README.md](../roles/data/README.md), ADR-035) is configured from code:
-`add-service.sh` gives one service its database, roles and Valkey user, and `switch-service.sh` moves a
-service from SQLite to PostgreSQL. The agent pipeline may run only `ovhost` through its broker, so both
-scripts are reachable from the CLI:
+`sudo ovhost data provision <service> [--dry-run]` runs `roles/data/add-service.sh` from the installed ovhost tree. The command requires a service listed in the host inventory and root privileges for a real run. Dry run prints the command without changing the host.
-```
-sudo ovhost data provision <service> [--dry-run]
-sudo ovhost data switch <service> [--sqlite <file>] [--dry-run]
-```
-
-Both run the script **from the ovhost install** (`<install>/roles/data/`), exactly as
-`sudo ovhost self-update` maintains it, and **pass the script's own exit code through** (a `switch` that
-gives up on protected sessions exits the same code the script did). `--dry-run` prints what would run and
-changes nothing. A service the inventory does not list is refused before anything runs. Both must run as
-root (`sudo ovhost data …`; `--dry-run` need not): the scripts need it, and an inner `sudo` would reset
-`SWITCH_UNITS`/`SWITCH_DIR` and switch the default unit and directory instead.
-
-- **`data provision`** runs `roles/data/add-service.sh <service>` with no arguments of its own. It is
- idempotent: passwords are generated once into `/etc/openvibe/data.env`, the roles, database and Valkey
- user are created if missing, and the four connection settings are replaced in place in the service's
- env file. The script prints **names only** — `DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`,
- `VALKEY_PREFIX` — never a value.
-- **`data switch`** runs `roles/data/switch-service.sh <service> [<file>]`. It takes `SWITCH_UNITS` and
- `SWITCH_DIR` from the inventory entry (`units` plus `workerUnits`, a template worker shown as a glob
- such as `openre-rtmp-ingest@*.service`, plus the `unitsMatch` glob when set; `repo`), so a release-layout service such as OpenRestream needs no
- hand-set environment. `--sqlite <file>` names the SQLite file when it is not
- `/var/lib/openvibe-<service>/<service>.db` (Host: `/var/lib/openvibe-host-api/host.db`). Before running
- it, the service's postgres branch must be merged with green CI and its auto-deploy frozen
- (`sudo ovhost freeze <service> --reason …`); the script unfreezes and deploys at the end. Its output
- is collected and printed when the script exits, not streamed: while it waits (up to 12 hours) for
- protected sessions to go idle, no `[switch] waiting…` line appears, so give the caller (the broker) a
- timeout longer than that wait, or switch when the service is idle.
-
-**`switch` needs the service's `scripts/migrate-to-postgres.js`.** The script clones `main`, installs it
-and runs that file (`openvibe-sdk` `runSqliteMigration`: migrate, import, verify counts and checksums)
-with the unit's own environment. A service without it cannot be switched this way.
-
-**No URL or password appears in the output.** `add-service.sh` prints names only, and every line either
-command writes out (stdout, stderr, `--json`, and the `--dry-run` command line) passes through a redact
-pass: a `postgres://`, `postgresql://`, `redis://` or `valkey://` URL becomes `<redacted-url>` and a
-`…PASSWORD=value` becomes `…PASSWORD=<redacted>`. This is asserted by `test/data-provision.test.js`.
-
-## Precedent: the 2026-10-02 manual runs
-
-On 2026-10-02 Tools, Chat and Network were provisioned and switched **by hand** with exactly these
-scripts, before the CLI wrappers existed:
-
-| Service | Switch |
-|---|---|
-| Tools | `scripts/migrate-to-postgres.js` over `apps/*/data` |
-| Chat | `scripts/import-sqlite-to-pg.js` |
-| Network | its T2 runbook |
-
-Those runs are the reason `data provision` takes no inventory-specific flags and `data switch` reads the
-units and directory from the inventory: the scripts already encode the per-service rules, and the CLI only
-supplies what the entry knows.
+The role creates the PostgreSQL database and roles, a Valkey user, and sets `DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL` and `VALKEY_PREFIX` in the service env file. It is idempotent and prints setting names rather than values. ovhost redacts connection URLs and password assignments from command output.
@@ -1,123 +1,15 @@
@@ -1,123 +1,15 @@
-# Databases: the inventory model, and the files left on the host (WS-S task 6, hazard H14)
+# PostgreSQL databases in the host inventory
-`host.example.json` declares each service's databases as a `databases[]` list. An item is a union of two
-**engines** ([lib/inventory.js](../lib/inventory.js) normalises them, [lib/dbengine.js](../lib/dbengine.js)
-holds the per-engine helpers):
+Every `services.<id>.databases[]` entry names a database in the host PostgreSQL cluster:
-- **SQLite** — a file the service user's worker copies:
- ```json
- { "name": "live", "path": "/opt/openvibe.live/data/live.db" }
- ```
-- **PostgreSQL** — a database in the host's cluster, named, with **no `path`**:
- ```json
- { "name": "trade", "engine": "postgresql", "database": "ov_trade" }
- ```
+```json
+{ "name": "trade", "engine": "postgresql", "database": "ov_trade" }
+```
-`engine` defaults to `sqlite`. A PostgreSQL `database` must match `/^ov_[a-z0-9_]+$/`; a PostgreSQL entry
-has no `path`, and a SQLite entry has no `database`. The data lives in the cluster, so ovhost never opens
-it by copy: pgBackRest backs the whole cluster up, the nightly run verifies that, and a per-service
-logical `.dump` feeds the drills ([backups.md](backups.md#postgresql-services)).
+`engine` must be `postgresql`, `database` must match `ov_[a-z0-9_]+`, and the entry cannot have `path`. `ovhost backup` verifies pgBackRest and periodically makes a `pg_dump -Fc` archive. A restore drill maps each database name to the service's URL settings:
-**Which engine.** With `host.example.json`:
+```json
+{ "trade": { "url": "DATABASE_URL", "directUrl": "DATABASE_DIRECT_URL" } }
+```
-| Engine | Services |
-|---|---|
-| PostgreSQL | tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, bot, host, events, community, billing, media, openre (20) |
-| SQLite | network, chat, tools, games, live (5) |
-
-**The two engines appear in four more places in the inventory:**
-
-- `drill.databases`. A SQLite database keeps a string (the env var that points the service at the restored
- file) or `{ "env": "DATA_DIR", "dir": true }` for a service that takes a data directory; a PostgreSQL
- database is `{ "trade": { "url": "DATABASE_URL", "directUrl": "DATABASE_DIRECT_URL" } }`, and the drill
- creates a scratch database and role and points both vars at it ([restore-drills.md](restore-drills.md)).
-- `drill.countsTolerance` — a whole number of rows, default `0`. Production keeps writing between the read
- that makes the backup and the dump, so a busy table may differ by up to this many rows without failing
- the drill.
-- `nginx.tenants.database` — the Stage B tenant database (the Host API is on PostgreSQL): a SQLite path
- keeps its string form, and PostgreSQL is the object `{ "engine": "postgresql", "database": "ov_host" }`.
-- the `protected` probe kind `postgresql-count` — `{ "kind": "postgresql-count", "database": "ov_openre",
- "sql": "SELECT count(*) …", "label": "…" }`: a read-only SELECT against a database in the cluster, run as
- the postgres OS user. OpenRestream's protected probe (its active ingest sessions) is one, on `ov_openre`. The
- other kinds are `http-json-count`, `sqlite-count` and `sum` ([lib/probes.js](../lib/probes.js)).
-
-
-The rest of this page is the 2026-09-26 sweep of SQLite **files**: every file under /opt, /var/lib, /root,
-/home and /srv that no process had open. 42 open files (the live databases) are not listed. Paths, sizes
-and dates only; nothing was read. A PostgreSQL service has no row here: its rows are in the cluster.
-
-`ovhost archive push <file>...` encrypts files with the backup key and copies them to `<BACKUP_S3_PREFIX>-archive/<host>/<stamp>/`, outside the backup runs so retention never prunes them; `ovhost archive list` and `ovhost archive restore <stamp> <path> --out <dir>` read them back. It never deletes a local file.
-
-| Class | File | Size | Modified | Recommended action |
-|---|---|---|---|---|
-| legacy | `/opt/backups/hobo.quest-2026-08-16/data/world.db` | 0.1 MB | 2026-08-16 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/backups/legacy-quest-20260817/hobo-quest.db` | 69.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/backups/pre-openvibe-20260817/hobo-tools-final.db` | 731.0 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/backups/pre-openvibe-20260817/hobo-tools.db` | 731.0 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/backups/pre-openvibe-20260817/hobostreamer-final.db` | 177.7 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/backups/pre-openvibe-20260817/hobostreamer.db` | 177.7 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/backups/pre-openvibe-20260817/world-final.db` | 0.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/hobo.quest-old-20260816/data/world.db` | 0.1 MB | 2026-08-16 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/hobo.quest/data/world.db` | 0.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/hobo/hobo-quest/data/hobo-quest.db` | 69.1 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/hobo/hobo-tools/data/hobo-tools.db` | 731.0 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/hobo/hobo-tools/data/hobotools.db` | 0.0 MB | 2026-03-22 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/hobostreamer/data/hobostreamer.db` | 177.7 MB | 2026-08-17 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| legacy | `/opt/hobostreamer/hobostreamer.db` | 0.0 MB | 2026-08-11 | archived 2026-09-26 (`20260926-094703`); delete locally with the owner's OK |
-| live-backup | `/opt/openvibe.live/shared/data/backups/live-20260926-072748.db` | 270.9 MB | 2026-09-26 | delete (owner OK) once its migration is a release old; archive first if wanted |
-| live-backup | `/opt/openvibe.live/shared/data/backups/live-pre-643a2e6.db` | 252.5 MB | 2026-09-17 | delete (owner OK) once its migration is a release old; archive first if wanted |
-| live-backup | `/opt/openvibe.live/shared/data/live-before-rs-slots-20260924T0153Z.db` | 266.2 MB | 2026-09-24 | delete (owner OK) once its migration is a release old; archive first if wanted |
-| live-backup | `/opt/openvibe.live/shared/data/live.db.bak-preAIstate` | 175.9 MB | 2026-08-19 | delete (owner OK) once its migration is a release old; archive first if wanted |
-| rehearsal | `/var/lib/openvibe-billing/live-snapshot-202609230121.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |
-| rehearsal | `/var/lib/openvibe-chat/live-snap-a.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |
-| rehearsal | `/var/lib/openvibe-chat/live-snap-b.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |
-| rehearsal | `/var/lib/openvibe-chat/live-snapshot-0159.db` | 266.2 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |
-| rehearsal | `/var/lib/openvibe-chat/rehearsal.db` | 0.3 MB | 2026-09-23 | delete (owner OK); the nightly backups cover Live |
-| migration-backup | `/var/lib/openvibe-community/community.pre-c24-20260925T185117Z.db` | 2.1 MB | 2026-09-25 | delete (owner OK); the migrations are verified |
-| migration-backup | `/var/lib/openvibe-community/community.pre-c24b-20260925T185400Z.db` | 2.1 MB | 2026-09-25 | delete (owner OK); the migrations are verified |
-| migration-backup | `/var/lib/openvibe-community/community.pre-c24c-20260925T185425Z.db` | 2.1 MB | 2026-09-25 | delete (owner OK); the migrations are verified |
-| migration-backup | `/var/lib/openvibe-community/community.pre-live-comments-20260923T191345Z.db` | 1.9 MB | 2026-09-23 | delete (owner OK); the migrations are verified |
-| ip-bearing | `/opt/hobo/hobo-audio/data/analytics.db` | 24.7 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/hobo/hobo-docs/data/analytics.db` | 1.4 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/hobo/hobo-food/data/analytics.db` | 0.5 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/hobo/hobo-img/data/analytics.db` | 3.3 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/hobo/hobo-maps/data/analytics.db` | 1.0 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/hobo/hobo-text/data/analytics.db` | 6.2 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/hobo/hobo-yt/data/analytics.db` | 0.5 MB | 2026-08-17 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.live/shared/data/backups/analytics-pre-adr021-2026-09-23.db` | 1682.4 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/audio.analytics.db` | 57.5 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/docs.analytics.db` | 1.6 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/food.analytics.db` | 0.4 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/img.analytics.db` | 5.4 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/maps.analytics.db` | 0.6 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/text.analytics.db` | 9.1 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| ip-bearing | `/opt/openvibe.tools/backups/analytics-2026-09-23/yt.analytics.db` | 0.9 MB | 2026-09-23 | delete after 2026-10-07 (WS-S task 5), never archive |
-| secrets | `/opt/openvibe.network/data/network-pre-refresh-hash-2026-09-24.db` | 266.2 MB | 2026-09-24 | delete after the rollback window (WS-S task 5), never archive |
-| secrets | `/opt/openvibe.network/data/network-pre-secrets-2026-09-24.db` | 266.2 MB | 2026-09-24 | delete after the rollback window (WS-S task 5), never archive |
-| stale | `/opt/openvibe.blog/data/blog.db` | 0.4 MB | 2026-09-24 | delete (owner OK) |
-| stale | `/opt/openvibe.community/data/community.db` | 0.2 MB | 2026-09-25 | delete (owner OK) |
-| stale | `/opt/openvibe.games/data/legacy-import/world-before-20260923T184930Z.db` | 0.2 MB | 2026-09-23 | delete (owner OK) |
-| stale | `/opt/openvibe.live/shared/data/hobo.db` | 0.0 MB | 2026-03-13 | delete (owner OK) |
-| stale | `/opt/openvibe.live/shared/data/openvibe.db` | 0.0 MB | 2026-08-18 | delete (owner OK) |
-| stale | `/opt/openvibe.live/shared/data/rs-companion.db` | 8.2 MB | 2026-03-10 | delete (owner OK) |
-| migration-backup | `/opt/openvibe.media/data/backups/owner-subject-20260923.media.db` | 9.1 MB | 2026-09-23 | delete (owner OK); the owner-subject migration is verified |
-| migration-backup | `/opt/openvibe.media/data/reports/h15-repair-applied-20260925.media.db` | 10.8 MB | 2026-09-25 | delete (owner OK) once the H15 repair is a release old |
-| stale | `/opt/openvibe.tips/data/tips.db` | 0.0 MB | 2026-09-23 | delete (owner OK) |
-| in use | `/opt/openvibe.tools/apps/gateway/data/token-revocations.db` | 0.0 MB | 2026-09-25 | keep: Tools' guard opens it per request (apps/_shared/guard/revocations.js) |
-| stale | `/root/blog-stale-db-before-revert-20260924T174243.db` | 0.4 MB | 2026-09-24 | delete (owner OK) |
-| system | `/var/lib/PackageKit/transactions.db` | 0.0 MB | 2026-05-11 | leave |
-| system | `/var/lib/command-not-found/commands.db` | 3.6 MB | 2026-07-27 | leave |
-| system | `/var/lib/fwupd/pending.db` | 0.0 MB | 2026-07-27 | leave |
-
-Classes:
-- **legacy**: the pre-OpenVibe sites' databases; the only copies, so they are archived before anything else happens.
-- **live-backup**, **rehearsal**, **migration-backup**: copies made around a migration; the nightly encrypted backups cover the live data.
-- **ip-bearing**, **secrets**: dated deletions in WS-S task 5 (section 10); archiving them would defeat the point.
-- **stale**: files an older layout left behind (H14 names the Blog one).
-- **system**: Ubuntu's own.
-- **in use**: closed at the moment of the inventory but opened on demand; never archive or delete.
-
-**Archived 2026-09-26:** the 14 legacy files (2.8 GB) as `20260926-094703` (`ovhost archive list`); a restore of `/opt/hobo.quest/data/world.db` came back byte-identical.
-
-Deleting anything here is the owner's call (WS-S task 6). The files a deletion may touch are exactly the rows above.
+The drill creates a scratch database and role, restores the archive, and points both settings at the scratch database. `nginx.tenants.database` uses `{ "engine": "postgresql", "database": "ov_host" }`. A `postgresql-count` protected-session probe runs a read-only SELECT through the postgres OS user.
@@ -4,18 +4,14 @@
@@ -4,18 +4,14 @@
Live (release layout), Tools, Sites, OpenRestream and Network, and Games (which deployed by hand). Each service
Live (release layout), Tools, Sites, OpenRestream and Network, and Games (which deployed by hand). Each service
entry in the inventory names a **strategy**: an engine plus the defaults the script it replaces encoded.
entry in the inventory names a **strategy**: an engine plus the defaults the script it replaces encoded.
Every repository keeps `deploy/scripts/deploy.sh` as a thin wrapper that maps its old flags onto ovhost,
Every repository keeps `deploy/scripts/deploy.sh` as a thin wrapper that maps its old flags onto ovhost,
-with the old script body kept as `deploy/scripts/deploy-legacy.sh` for the fallback.
+and invokes ovhost directly.
-**State (2026-09-28):** in production since the 2026-09-27 cutover (the checklist below was run then).
-Every service, including the six here, deploys with `sudo ovhost deploy <svc>` and rolls back with `sudo ovhost rollback
-<svc>`; the strategies are tested against the fake host (`test/strategy-release-layout.test.js`,
-`test/strategy-in-place.test.js`) and in each repository (the wrapper tests). The wrappers' legacy fallback is only
-reached where an ovhost without `capabilities` answers, which no longer happens on the production host.
+Production deploys and rollbacks use `ovhost deploy <svc>` and `ovhost rollback <svc>`. The strategy tests exercise each implementation against the fake host.
| Service | Strategy | Replaces | Wrapper (`deploy/scripts/deploy.sh`) |
| Service | Strategy | Replaces | Wrapper (`deploy/scripts/deploy.sh`) |
|---|---|---|---|
|---|---|---|---|
| live | `release-layout` | Live `deploy.sh` (release layout) | `--wait-idle`, `--restart`, `--rollback` → `ovhost rollback live`, `DRY_RUN=1` → `ovhost plan live`; `--force` refused (see below) |
| live | `release-layout` | Live `deploy.sh` (release layout) | `--wait-idle`, `--restart`, `--rollback` → `ovhost rollback live`, `DRY_RUN=1` → `ovhost plan live`; `--force` refused (see below) |
-| openre | `release-layout` | OpenRestream `deploy.sh release` + `api` | `deploy`, `release [<ref>]` → `--prepare-only`, `api [<sha>]` → `deploy --to <sha>`, `rollback`, `plan`; `workers`, `status`, `prune` stay in the legacy script |
+| openre | `release-layout` | OpenRestream `deploy.sh release` + `api` | `deploy`, `release [<ref>]` → `--prepare-only`, `api [<sha>]` → `deploy --to <sha>`, `rollback`, `plan` |
| tools | `multi-app` | Tools `deploy.sh` | `--wait-idle`, `--restart`, `--force`, `--rollback`, `DRY_RUN=1` |
| tools | `multi-app` | Tools `deploy.sh` | `--wait-idle`, `--restart`, `--force`, `--rollback`, `DRY_RUN=1` |
| sites | `static-build` | Sites `deploy.sh` (after a manual pull) | always `ovhost deploy sites --restart` (rebuild every run, as before); `DRY_RUN=1` |
| sites | `static-build` | Sites `deploy.sh` (after a manual pull) | always `ovhost deploy sites --restart` (rebuild every run, as before); `DRY_RUN=1` |
| games | `pnpm-build` | the manual procedure | `--wait-idle`, `--restart`, `--force`, `--rollback`, `DRY_RUN=1` |
| games | `pnpm-build` | the manual procedure | `--wait-idle`, `--restart`, `--force`, `--rollback`, `DRY_RUN=1` |
@@ -23,41 +19,9 @@ reached where an ovhost without `capabilities` answers, which no longer happens
@@ -23,41 +19,9 @@ reached where an ovhost without `capabilities` answers, which no longer happens
Every other service keeps the default strategy, `git-checkout`, which is what `ovhost deploy` did before.
Every other service keeps the default strategy, `git-checkout`, which is what `ovhost deploy` did before.
-## The wrappers and their fallback
+## Deploy wrappers
-Each wrapper first asks `ovhost capabilities <svc>` (run as root: `sudo` when the wrapper is not root). It
-hands over to ovhost only when every line it needs is there:
-
-```
-ovhost=0.3.0
-deploy-api=1
-strategies=git-checkout,multi-app,static-build,pnpm-build,release-layout
-commands=plan,deploy,rollback,announce,capabilities
-deploy-flags=--wait-idle,--force,--restart,--to,--install-units,--ready-timeout,--prepare-only,--no-announce
-service=live
-strategy=release-layout
-managed=yes
-layout=release
-```
-
-`deploy-api` is at least 1, `strategy=` is the one the wrapper was written for, and `managed=yes`.
-Otherwise it prints why (`ovhost not found`, `no 'capabilities' (too old)`, `deploy-api is 0`, `the host
-inventory does not deploy live with strategy release-layout (none)`, …) and runs `deploy-legacy.sh` with the
-original arguments. So an old ovhost, a missing one, or an inventory that has not been changed yet never
-breaks a deploy. `OVHOST_LEGACY=1` forces the fallback; `OVHOST=<path>` picks another ovhost. Where the legacy
-script cannot do what was asked (Tools, Network and Games have no `--rollback`; none of them has `DRY_RUN`
-except Live), the wrapper refuses instead of doing something else. The Sites fallback pulls first (it restores
-`dist/` and runs `git pull --ff-only` as the checkout owner), then runs the pulled legacy script. OpenRestream's
-fallback runs `release` then `api` for `deploy`.
-
-Exit codes are ovhost's: `0` ok · `1` usage/precondition · `2` validation failed, nothing restarted · `3` not
-ready, rolled back and serving · `4` rollback failed, **manual intervention** · `5` protected sessions active
-· `6` frozen.
-
-**Live `--force`.** The old script's `--force` meant "discard local tracked changes" (legacy in-place layout
-only). ovhost never discards changes, and its `--force` drops live streams and passes a freeze. The Live
-wrapper therefore refuses `--force`; run `sudo ovhost deploy live --force` yourself when that is what you
-mean. (In the fallback, `--force` goes to the legacy script as before.)
+Repository wrappers call `ovhost plan`, `ovhost deploy`, or `ovhost rollback` for their service. If ovhost is missing, the wrapper exits with an error. The `capabilities` command remains available for installed wrappers that still query it.
## The two engines
## The two engines
@@ -117,7 +81,7 @@ overrides them.
@@ -117,7 +81,7 @@ overrides them.
| Field | Meaning | Default |
| Field | Meaning | Default |
|---|---|---|
|---|---|---|
-| `strategy` | `git-checkout`, `multi-app`, `static-build`, `pnpm-build` or `release-layout` | `git-checkout`; an entry with `layout: "release"` and no strategy stays unmanaged (deployed by its own script), as before |
+| `strategy` | `git-checkout`, `multi-app`, `static-build`, `pnpm-build` or `release-layout` | `git-checkout`; `layout: "release"` requires `release-layout` |
| `release.git` | the clone releases are made from (relative to `repo`, or absolute inside it) | `repo` |
| `release.git` | the clone releases are made from (relative to `repo`, or absolute inside it) | `repo` |
| `release.releases`, `release.current` | the releases directory and the link | `releases`, `current` |
| `release.releases`, `release.current` | the releases directory and the link | `releases`, `current` |
| `release.id` | `time-sha8` (`<UTC yyyymmdd-HHMMSS>-<sha8>`, Live; the old script used the host's local time, the same on a UTC host) or `sha12` (OpenRestream; one release per commit, reused) | `time-sha8` |
| `release.id` | `time-sha8` (`<UTC yyyymmdd-HHMMSS>-<sha8>`, Live; the old script used the host's local time, the same on a UTC host) or `sha12` (OpenRestream; one release per commit, reused) | `time-sha8` |
@@ -190,129 +154,13 @@ log and a freeze refuses.
@@ -190,129 +154,13 @@ log and a freeze refuses.
operator's environment): `sudo` does not pass `ADMIN_TOKEN`, so it said "chat notification skipped" in
operator's environment): `sudo` does not pass `ADMIN_TOKEN`, so it said "chat notification skipped" in
practice, and Live posts its own deploy notice when it boots on new commits (`server/chat/deploy-notice.js`,
practice, and Live posts its own deploy notice when it boots on new commits (`server/chat/deploy-notice.js`,
through Events).
through Events).
-- **Live's in-place (legacy) layout branch** of `deploy.sh`: production runs the release layout since
- 2026-09-24; an in-place Live would be `strategy: git-checkout`.
- **Live `--force`** (discard local tracked changes): ovhost never discards changes (see above).
- **Live `--force`** (discard local tracked changes): ovhost never discards changes (see above).
- **Network's "restart anyway" when nothing is new**: only with `--restart` now.
- **Network's "restart anyway" when nothing is new**: only with `--restart` now.
- **OpenRestream `workers`, `status`, `prune`**: they stay in the repository's script. Starting a worker
- **OpenRestream `workers`, `status`, `prune`**: they stay in the repository's script. Starting a worker
generation is not a deploy ovhost makes (it never starts, stops or restarts a worker unit); ovhost prunes
generation is not a deploy ovhost makes (it never starts, stops or restarts a worker unit); ovhost prunes
on its own and keeps every release a worker runs from.
on its own and keeps every release a worker runs from.
- **Games `sudo git`**: ovhost runs git as the checkout owner. If earlier `sudo git pull`s left root-owned
- **Games `sudo git`**: ovhost runs git as the checkout owner. If earlier `sudo git pull`s left root-owned
- objects in `.git`, fix the ownership once (checklist below).
-
-## Production cutover checklist (for the operator)
-
-Run on 2026-09-27 (the cutover); kept as the record of what was done and as the procedure for a new host. Each step is
-read-only until step 5.
-
-1. **Update ovhost.** `cd /usr/local/lib/openvibe-host && sudo git pull --ff-only && sudo npm ci --omit=dev
- --no-audit --no-fund`, then `ovhost --version` (0.3.0) and `sudo ovhost capabilities` (`deploy-api=1`).
- Until step 3 every wrapper still falls back (the inventory has no strategies).
-2. **Check the checkouts.**
- - `sudo find /opt/openvibe.games/.git ! -user ubuntu | head`: if anything is listed,
- `sudo chown -R ubuntu:ubuntu /opt/openvibe.games/.git` (earlier `sudo git` pulls).
- - `sudo -u ubuntu bash -lc 'command -v pnpm && pnpm --version'`: ovhost runs pnpm as ubuntu with the
- PATH it was started with; if pnpm lives elsewhere, set `install.command`/`build` to its absolute path.
- - `ls -ld /opt/openvibe.live/repo /opt/openre.stream/repo` (root-owned clones, `owner: root`).
- - `sudo ss -Hltnp 'sport = :3000'` shows `"systemd",pid=1` (else the first Live deploy rebinds the socket).
-3. **Edit the inventory.** `sudo cp /etc/openvibe/host.json /etc/openvibe/host.json.pre-ws-n-11`, apply the
- [proposed diff](#proposed-production-inventory-diff), then `sudo ovhost show live openre tools sites games
- network` and `sudo ovhost validate <svc>` for each (no new errors).
-4. **Compare each plan with the old script** (`sudo ovhost plan <svc>` only fetches):
- - live: `sudo ovhost plan live` against `cd /opt/openvibe.live/current && sudo DRY_RUN=1 bash
- deploy/scripts/deploy-legacy.sh` (the old script; `deploy.sh` before the wrapper is deployed): same target sha and change classes (restart yes/no),
- `node_modules` hard-linked vs `npm ci` the same way, `socket held by systemd on :3000`,
- `strategy release-layout (release layout, keep 5)`.
- - openre: `sudo ovhost plan openre` against `sudo bash /opt/openre.stream/repo/deploy/scripts/deploy.sh
- status`: current release = the `current` link, restart = `openre-api.service,
- openre-session-coordinator.service` only, ingest sessions counted.
- - tools: `sudo ovhost plan tools`: the installs match the apps whose `package.json`/lockfile changed
- (`git -C /opt/openvibe.tools diff --name-only HEAD origin/main -- 'apps/*/package*.json'`), no
- `apps/_shared` line, the restart lists every `systemctl list-unit-files 'openvibe-tools*'` unit (no
- "not in the inventory" warning), the preflight lists the jobs and guard checks.
- - sites: `sudo ovhost plan sites --restart`: build output restored, `npm ci`, `node build.js`, repo vhosts,
- one notification per placeholder, no tracked changes.
- - games: `sudo ovhost plan games`: tracked changes none (dist-types restored), `install (always)`, build,
- preflight better-sqlite3, restart `openvibe-games.service`, players online counted (drain report).
- - network: `sudo ovhost plan network`: nothing unexpected (Network already deploys with ovhost; the
- entry only gains `installUnits`).
-5. **Pull the wrappers.** Tools, Sites, Network and Games get theirs with their next pull (the old Tools and
- Network scripts pull themselves, so the deploy that brings the wrapper is still run by the old script).
- Live's arrives in the release that contains it; OpenRestream's with `git -C /opt/openre.stream/repo pull`.
-6. **Deploy through the wrapper**, one service at a time, lowest risk first: network, sites, games (quiet
- hour or `--wait-idle`), tools, openre (no ingest session, or `--wait-idle`), live (`--wait-idle`). Each
- wrapper prints `ovhost deploy <svc> …`; if it prints "running deploy-legacy.sh", read why.
-7. **Verify.** `sudo ovhost releases <svc>` (strategy, from/to, result), `sudo ovhost status`, and for Live
- `sudo ss -Hltnp 'sport = :3000'` still shows `"systemd",pid=1`.
-8. **Undo** (any step): `OVHOST_LEGACY=1 sudo deploy/scripts/deploy.sh` runs the old script for one deploy;
- restoring `/etc/openvibe/host.json.pre-ws-n-11` (or removing a `strategy`) makes the wrapper fall back
- for good.
-
-## Proposed production inventory diff
-
-`/etc/openvibe/host.json` is not changed by this work. These are the fields each entry needs (merge them
-into the existing entries; `host.example.json` in this repository has the full entries). JSON has no
-comments: the `//` notes below are for reading only.
-
-```jsonc
-"live": {
- "repo": "/opt/openvibe.live", // the release root (".../current" is read the same way)
- "owner": "root", // git runs as root in the root-owned clone
- "runAs": "ubuntu",
- "strategy": "release-layout",
- "release": { "git": "repo", "id": "time-sha8", "links": { "data": "../../shared/data" }, "keep": 5 },
- // MUST be the release-layout units: the in-place ones (deploy/systemd/*.service) run from
- // /opt/openvibe.live, and installing them would point Live at the stale checkout.
- "unitSources": {
- "openvibe-live.service": "deploy/systemd/release/openvibe-live.service",
- "openvibe-live.socket": "deploy/systemd/release/openvibe-live.socket",
- "openvibe-live.service.d/socket.conf": "deploy/systemd/release/openvibe-live.service.d/socket.conf"
- },
- "ready": { "url": "http://127.0.0.1:3000/api/ready", "timeoutSeconds": 90, "release": true },
- "preflight": { "syntaxCheck": true }
- // unchanged: units, socketUnit, port, noRestartPaths, protected, drain (refuse, 28800 s), databases,
- // backupOnChange, nginx (repoVhost is never installed by a deploy; the plan names a change)
-},
-"openre": {
- "repo": "/opt/openre.stream", // the release root; the clone is /opt/openre.stream/repo
- "strategy": "release-layout", // replaces "layout": "release"
- "owner": "root",
- "runAs": "ubuntu",
- "release": { "git": "repo", "id": "sha12", "reuseModules": false, "chown": "ubuntu:ubuntu", "keep": 5 },
- "ready": { "url": "http://127.0.0.1:4500/api/ready", "timeoutSeconds": 60 }
- // unchanged: units (API + coordinator), workerUnits, protected (ingest sessions), drain
-},
-"tools": {
- "strategy": "multi-app", // skipPackages apps/_*, removeUntrackedLockfiles
- "unitsMatch": "openvibe-tools*.service",
- "ready": { "url": "http://127.0.0.1:4001/api/ready", "headers": { "Host": "openvibe.tools" },
- "timeoutSeconds": 60, "release": true, "allUnits": true },
- "preflight": {
- "dirs": ["data"],
- "checks": [
- { "label": "jobs runtime loads", "packages": ["apps/img", "apps/audio", "apps/docs"],
- "argv": ["node", "-e", "const D=require('better-sqlite3'); new D(':memory:').close(); require('openvibe-contracts'); require('openvibe-sdk'); require('../_shared/jobs')"] },
- { "label": "guard loads", "packages": "*",
- "argv": ["node", "-e", "const D=require('better-sqlite3'); new D(':memory:').close(); require('../_shared/guard')"] }
- ]
- }
-},
-"sites": {
- "strategy": "static-build" // npm ci, node build.js, generated dist/, repo vhosts
- // behind nginx -t, one announce per dist/*/release.json
-},
-"games": {
- // remove "managed": false and "unmanagedReason"
- "strategy": "pnpm-build",
- "generated": ["apps/client/dist-types/", "apps/server/dist-types/"],
- "preflight": { "checks": [ { "label": "better-sqlite3 loads under this Node", "packages": ["apps/server"],
- "argv": ["node", "-e", "new (require('better-sqlite3'))(':memory:').close()"] } ] }
- // unchanged: units, ready, protected (players online), drain (report), databases, drill
-},
-"network": {
- "installUnits": true // the unit file is installed whenever it differs
-}
-```
+ objects in `.git`, fix the ownership once.
## The deploy controller: `ovhost reconcile` (roadmap WS-X11 phase 1, D72)
## The deploy controller: `ovhost reconcile` (roadmap WS-X11 phase 1, D72)
@@ -258,7 +258,7 @@ Each launch-rule item, checked against the code at this commit:
@@ -258,7 +258,7 @@ Each launch-rule item, checked against the code at this commit:
| 1 | Owning runtime with health/readiness and observability | `server/index.js` + `deploy/systemd/openvibe-host.service`. `GET /api/health`, `/api/ready` (`server/observability.js`: required `db`, `storage`; optional `disk_headroom`, `network_jwks`, `events_relay`, `domain_checks`), `/release.json`, `/metrics` (loopback only; HTTP golden signals, process, release, `host_sites`). Running on loopback `:4910` with ready 200 (audit, 2026-09-23). `host-secrets-dashboard` "machine endpoints". | **met** |
| 1 | Owning runtime with health/readiness and observability | `server/index.js` + `deploy/systemd/openvibe-host.service`. `GET /api/health`, `/api/ready` (`server/observability.js`: required `db`, `storage`; optional `disk_headroom`, `network_jwks`, `events_relay`, `domain_checks`), `/release.json`, `/metrics` (loopback only; HTTP golden signals, process, release, `host_sites`). Running on loopback `:4910` with ready 200 (audit, 2026-09-23). `host-secrets-dashboard` "machine endpoints". | **met** |
| 2 | Canonical identity (Network subjects, scoped principals) | Network SSO OAuth client `host` (Network `server/db/database.js:516`); users are `usr_` subjects from Network JWTs verified against the JWKS (`server/auth/viewer.js`); services present client-credentials tokens for audience `openvibe.host` with one capability per route; `X-OV-Subject` for first-party services only; app principals act as themselves; sandbox tokens refused on production. Principal `svc:host` has `events.event.publish` (Network `server/identity/principals.js:100`). `host-quota-auth` tests. Projects are Host's own until Network has projects (ADR-014, documented). | **met** |
| 2 | Canonical identity (Network subjects, scoped principals) | Network SSO OAuth client `host` (Network `server/db/database.js:516`); users are `usr_` subjects from Network JWTs verified against the JWKS (`server/auth/viewer.js`); services present client-credentials tokens for audience `openvibe.host` with one capability per route; `X-OV-Subject` for first-party services only; app principals act as themselves; sandbox tokens refused on production. Principal `svc:host` has `events.event.publish` (Network `server/identity/principals.js:100`). `host-quota-auth` tests. Projects are Host's own until Network has projects (ADR-014, documented). | **met** |
| 3 | SSR/static public routes useful without JS | Every dashboard page is server-rendered; every action is a plain form, including folder upload. Shared chrome with `<noscript>` nav and SSR footer. Legal pages. Tenant sites are static by definition. `host-secrets-dashboard` "signed-out home page … useful without JavaScript". | **met** |
| 3 | SSR/static public routes useful without JS | Every dashboard page is server-rendered; every action is a plain form, including folder upload. Shared chrome with `<noscript>` nav and SSR footer. Legal pages. Tenant sites are static by definition. `host-secrets-dashboard` "signed-out home page … useful without JavaScript". | **met** |
-| 4 | Real persistence and end-to-end workflows | SQLite `host.db` (WAL) plus the content-addressed object store; upload → validate → store → activate → serve → rollback → delete, custom domains through verification and lapse, all over HTTP in the tests (`host-lifecycle`, `host-rollback`, `host-domains`). `host.db` is in `ovhost backup` and the restore drill passed on the host (2026-09-23). Objects are not backed up (accepted for alpha, threat review §6). | **met** (objects backup: follow-up) |
+| 4 | Real persistence and end-to-end workflows | PostgreSQL (`ov_host`) plus the content-addressed object store; upload → validate → store → activate → serve → rollback → delete, custom domains through verification and lapse, all over HTTP in the tests (`host-lifecycle`, `host-rollback`, `host-domains`). The database is covered by pgBackRest and logical dumps; object directories are archived by `ovhost backup`. | **met** |
| 5 | Capability and event registration against Contracts | `host.site.manage`, `host.deploy.create` and `host.domain.manage` and the `host` service manifest (with all four `host.*` events in `eventsProduced`) are released in `openvibe-contracts` **v0.24.0**, which Host pins, and are identical in **v0.30.1** (v0.83.0 adds `host.site.config`). `npx openvibe-contracts-check --service host` passes, and CI runs it as a blocking step. No `host.*` payload schemas: optional, most services have none. | **met** |
| 5 | Capability and event registration against Contracts | `host.site.manage`, `host.deploy.create` and `host.domain.manage` and the `host` service manifest (with all four `host.*` events in `eventsProduced`) are released in `openvibe-contracts` **v0.24.0**, which Host pins, and are identical in **v0.30.1** (v0.83.0 adds `host.site.config`). `npx openvibe-contracts-check --service host` passes, and CI runs it as a blocking step. No `host.*` payload schemas: optional, most services have none. | **met** |
| 6a | Migration/seed strategy | Bootstrap, not migration: Host replaces no legacy data (roadmap §8 register). The schema is created idempotently at every boot (`server/db.js`, `CREATE … IF NOT EXISTS`); `host_takedowns` is additive and appears on the next restart. There is no seed: a new database is an empty, valid Host. The pointer and data survive a restart (`host-rollback`). A future non-additive change needs a migration ledger, which Host does not have yet. | **met** |
| 6a | Migration/seed strategy | Bootstrap, not migration: Host replaces no legacy data (roadmap §8 register). The schema is created idempotently at every boot (`server/db.js`, `CREATE … IF NOT EXISTS`); `host_takedowns` is additive and appears on the next restart. There is no seed: a new database is an empty, valid Host. The pointer and data survive a restart (`host-rollback`). A future non-additive change needs a migration ledger, which Host does not have yet. | **met** |
| 6b | Security/threat review | [threat-review.md](threat-review.md): tenant isolation, uploads, custom domains, cookies, the PSL decision, quotas, abuse and takedown, Host's own surface. Seven gaps were fixed in `1e48cb2` and `0ff29d5`: takedowns, the upload concurrency cap, the CA-validation path refusal, the CDN cache cap, the realip templates, the disk floor with the owner project cap, and Clear-Site-Data. | **met** |
| 6b | Security/threat review | [threat-review.md](threat-review.md): tenant isolation, uploads, custom domains, cookies, the PSL decision, quotas, abuse and takedown, Host's own surface. Seven gaps were fixed in `1e48cb2` and `0ff29d5`: takedowns, the upload concurrency cap, the CA-validation path refusal, the CDN cache cap, the realip templates, the disk floor with the owner project cap, and Clear-Site-Data. | **met** |
@@ -88,7 +88,7 @@ against the 2 s the deploy took to report ready).
@@ -88,7 +88,7 @@ against the 2 s the deploy took to report ready).
| Gate | What | Evidence | Budget | Where the number comes from |
| Gate | What | Evidence | Budget | Where the number comes from |
|---|---|---|---|---|
|---|---|---|---|---|
| 1a | a styles-only release is switched without restarting Live, streams live | Host `test/strategy-release-layout.test.js` | Live restarts (dropped streams) ≤ 0 | asserted: the test asserts host.restarts() is [] and the record says restarted: false |
| 1a | a styles-only release is switched without restarting Live, streams live | Host `test/strategy-release-layout.test.js` | Live restarts (dropped streams) ≤ 0 | asserted: the test asserts host.restarts() is [] and the record says restarted: false |
-| 1b | the same through Live's own deploy script on a simulated host | Live `test/deploy-sim.test.js` | Live restarts ≤ 0 | asserted: the test compares the process id before and after the switch |
+| 1b | the same through Live's deploy wrapper on a simulated host | Live `test/deploy-sim.test.js` | Live restarts ≤ 0 | asserted: the test compares the process id before and after the switch |
| 1c | a tab broadcasting (live camera/mic) or playing a stream is never reloaded | Shared `test/release.test.js` | reloads while capturing or playing ≤ 0 | asserted: asserts [reloads, metrics] = [0, { deferred: { capture: 1 } }] with a live track, and the same for a playing <video> |
| 1c | a tab broadcasting (live camera/mic) or playing a stream is never reloaded | Shared `test/release.test.js` | reloads while capturing or playing ≤ 0 | asserted: asserts [reloads, metrics] = [0, { deferred: { capture: 1 } }] with a live track, and the same for a playing <video> |
| 1d | a style change is applied in place: only the changed stylesheet moves | Shared `test/release-update.test.js` | reloads + prompts for a styles-only release ≤ 0 | asserted: asserts [p.reloads, p.toasts.length] = [0, 0] after the update |
| 1d | a style change is applied in place: only the changed stylesheet moves | Shared `test/release-update.test.js` | reloads + prompts for a styles-only release ≤ 0 | asserted: asserts [p.reloads, p.toasts.length] = [0, 0] after the update |
| 1e | Live's manifest declares style components, so its open tabs take styles in place | `<base>/release.json` (--base) | style components in /release.json ≥ 1 | GET <base>/release.json |
| 1e | Live's manifest declares style components, so its open tabs take styles in place | `<base>/release.json` (--base) | style components in /release.json ≥ 1 | GET <base>/release.json |
@@ -28,14 +28,7 @@ visibility `internal`, its own payload). Browsers never receive that one.
@@ -28,14 +28,7 @@ visibility `internal`, its own payload). Browsers never receive that one.
- A `static-build` service with `announce.releaseFiles` (Sites) sends one notification per
- A `static-build` service with `announce.releaseFiles` (Sites) sends one notification per
`dist/<domain>/release.json`, with that placeholder's service, release and `https://<domain>` origin,
`dist/<domain>/release.json`, with that placeholder's service, release and `https://<domain>` origin,
and stops at the first failure.
and stops at the first failure.
-- Since WS-N task 11 each repository's `deploy/scripts/deploy.sh` is a wrapper around `ovhost deploy <svc>`
- ([deploy-strategies.md](deploy-strategies.md)), so the deploy announces. Their fallback scripts
- (`deploy/scripts/deploy-legacy.sh`) still call `ovhost announce <service>` at the end:
- - Live, Tools, Games: `ovhost announce live|tools|games`.
- - Sites: one call per placeholder with `--release` and `--origin`.
-
- They do nothing if `ovhost` is missing or has no `announce`, and a failed announcement never fails the
- deploy.
+- Repository deploy wrappers call `ovhost deploy <service>`, which announces a deployed release.
`ovhost announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run] [--events-env <file>] [--json]`
`ovhost announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run] [--events-env <file>] [--json]`
@@ -106,7 +99,7 @@ All commands run on the production host. None of them prints a secret.
@@ -106,7 +99,7 @@ All commands run on the production host. None of them prints a secret.
the Network admin, `GET https://openvibe.network/api/admin/grants?client=host`, where it shows as
the Network admin, `GET https://openvibe.network/api/admin/grants?client=host`, where it shows as
active. Or check the database directly:
active. Or check the database directly:
```
```
- cd /opt/openvibe.network && sudo sqlite3 data/network.db "SELECT capability, audience, revoked_at FROM principal_grants WHERE client_id='host'"
+ sudo -u postgres psql ov_network -c "SELECT capability, audience, revoked_at FROM principal_grants WHERE client_id='host'"
```
```
Expect `events.event.publish|openvibe.events|` (not revoked). If it is missing, grant it in the
Expect `events.event.publish|openvibe.events|` (not revoked). If it is missing, grant it in the
admin: `POST /api/admin/grants { client_id: "host", capability: "events.event.publish", reason: "release notifications (WS-P 9)" }`.
admin: `POST /api/admin/grants { client_id: "host", capability: "events.event.publish", reason: "release notifications (WS-P 9)" }`.
@@ -4,35 +4,7 @@ A drill restores a service's database from its latest `ovhost backup`, starts a
@@ -4,35 +4,7 @@ A drill restores a service's database from its latest `ovhost backup`, starts a
service on a spare loopback port against the restored copy, and compares its answers with production.
service on a spare loopback port against the restored copy, and compares its answers with production.
The production instance is never touched.
The production instance is never touched.
-| Date (UTC) | Service | Backup | Checks | Result |
-|---|---|---|---|---|
-| 2026-09-23 03:44 | community | `/var/backups/openvibe/community/20260923-034341/` (`ovhost backup community`) | `pragma integrity_check` = ok; restored instance `/api/ready` 200; `/api/pastes?limit=5`, `/api/v1/spaces`, `/api/v1/pulse?limit=5` byte-identical to production; pastes 892 = 892 | passed |
-| 2026-09-23 05:22 | community | `/var/backups/openvibe/community/20260923-052231/` (`ovhost drill community`) | `pragma integrity_check` (community) = ok; drill instance `/api/ready` 200 after 1s; `/api/pastes?limit=5`, `/api/v1/spaces`, `/api/v1/pulse?limit=5` identical to production; pastes 892 = 892; spaces 3 = 3; threads 0 = 0; posts 0 = 0 | passed |
-| 2026-09-23 05:22 | network | `/var/backups/openvibe/network/20260923-052241/` (`ovhost drill network`) | `pragma integrity_check` (network) = ok; drill instance `/api/ready` 200 after 1s; `/api/health`, `/api/.well-known/jwks`, `/api/brand`, `/api/themes?limit=500`, `/api/domains`, `/api/v1/registry/capabilities` identical to production; users 393 = 393; oauth_clients 22 = 22; wallets 30 = 30; coin_transactions 34 = 34 | passed |
-| 2026-09-23 05:22 | events | `/var/backups/openvibe/events/20260923-052251/` (`ovhost drill events`) | `pragma integrity_check` (events) = ok; drill instance `/api/ready` 200 after 1s; events 69 = 69; subscriptions 8 = 8; deliveries 20 = 20; consumer_checkpoints 0 = 0 | passed |
-| 2026-09-23 05:22 | billing | `/var/backups/openvibe/billing/20260923-052254/` (`ovhost drill billing`) | `pragma integrity_check` (billing) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/rates` identical to production; accounts 22 = 22; transactions 322 = 322; ledger_entries 996 = 996; subscriptions 0 = 0; payment_intents 5 = 5 | passed |
-| 2026-09-23 05:22 | chat | `/var/backups/openvibe/chat/20260923-052257/` (`ovhost drill chat`) | `pragma integrity_check` (chat) = ok; drill instance `/ready` 200 after 1s; `/api/chat/filters/friendly` identical to production; dm_conversations 11 = 11; dm_messages 1522 = 1522; emotes 42 = 42; moderation_actions 264 = 264 | passed |
-| 2026-09-23 05:23 | search | `/var/backups/openvibe/search/20260923-052300/` (`ovhost drill search`) | `pragma integrity_check` (search) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/search?q=openvibe&limit=10`, `/api/v1/suggest?q=open&limit=10` identical to production; documents 10 = 10; doc_acl 0 = 0; doc_facets 0 = 0 | passed |
-| 2026-09-23 05:23 | sources | `/var/backups/openvibe/sources/20260923-052303/` (`ovhost drill sources`) | `pragma integrity_check` (sources) = ok; drill instance `/api/ready` 200 after 1s; sources 6 = 6; items 0 = 0; item_revisions 0 = 0; fetch_runs 0 = 0 | passed |
-| 2026-09-23 05:23 | wiki | `/var/backups/openvibe/wiki/20260923-052306/` (`ovhost drill wiki`) | `pragma integrity_check` (wiki) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/spaces`, `/feed.json` identical to production; wiki_spaces 1 = 1; wiki_pages 10 = 10; wiki_page_revisions 10 = 10 | passed |
-| 2026-09-23 05:23 | blog | `/var/backups/openvibe/blog/20260923-052309/` (`ovhost drill blog`) | `pragma integrity_check` (blog) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/blogs/openvibe/posts?limit=10`, `/feed.json` identical to production; blogs 1 = 1; blog_posts 1 = 1; blog_post_revisions 1 = 1 | passed |
-| 2026-09-23 05:23 | tips | `/var/backups/openvibe/tips/20260923-052312/` (`ovhost drill tips`) | `pragma integrity_check` (tips) = ok; drill instance `/api/ready` 200 after 1s; `/sitemap.xml` identical to production; creator_tip_profiles 0 = 0; tip_interactions 0 = 0; tip_goals 0 = 0 | passed |
-| 2026-09-23 05:23 | vip | `/var/backups/openvibe/vip/20260923-052315/` (`ovhost drill vip`) | `pragma integrity_check` (vip) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/creators/network`, `/api/v1/plans?creator=network`, `/api/v1/perks?creator=network` identical to production; vip_creators 1 = 1; vip_plans 0 = 0; vip_memberships 0 = 0 | passed |
-| 2026-09-23 05:23 | news | `/var/backups/openvibe/news/20260923-052318/` (`ovhost drill news`) | `pragma integrity_check` (news) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/stories?limit=10`, `/api/v1/topics` identical to production; news_topics 12 = 12; news_stories 0 = 0; news_source_items 0 = 0 | passed |
-| 2026-09-23 05:23 | reviews | `/var/backups/openvibe/reviews/20260923-052321/` (`ovhost drill reviews`) | `pragma integrity_check` (reviews) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/entities?limit=10`, `/feed.json` identical to production; review_entities 0 = 0; review_signals 0 = 0; review_summaries 0 = 0 | passed |
-| 2026-09-23 05:23 | trade | `/var/backups/openvibe/trade/20260923-052323/` (`ovhost drill trade`) | `pragma integrity_check` (trade) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/instruments?limit=10`, `/feed.json` identical to production; trade_instruments 0 = 0; trade_market_observations 0 = 0; trade_source_documents 0 = 0 | passed |
-| 2026-09-23 05:23 | deals | `/var/backups/openvibe/deals/20260923-052326/` (`ovhost drill deals`) | `pragma integrity_check` (deals) = ok; drill instance `/api/ready` 200 after 1s; `/api/v1/offers?sort=new&page=1` identical to production (JSON without volatile keys where declared); deal_offers 0 = 0; deal_products 0 = 0; deal_votes 0 = 0 | passed |
-| 2026-09-23 05:23 | coupons | `/var/backups/openvibe/coupons/20260923-052329/` (`ovhost drill coupons`) | `pragma integrity_check` (coupons) = ok; drill instance `/api/ready` 200 after 1s; `/sitemaps/merchants.xml`, `/feed.json` identical to production; coupons 0 = 0; coupon_merchants 0 = 0; coupon_merchant_domains 0 = 0 | passed |
-| 2026-09-23 05:23 | codes | `/var/backups/openvibe/codes/20260923-052332/` (`ovhost drill codes`) | `pragma integrity_check` (codes) = ok; drill instance `/api/ready` 200 after 1s; `/sitemap.xml` identical to production; manifests 0 = 0; releases 0 = 0; trust 0 = 0 | passed |
-| 2026-09-23 05:23 | host | `/var/backups/openvibe/host/20260923-052335/` (`ovhost drill host`) | `pragma integrity_check` (host) = ok; drill instance `/api/ready` 200 after 1s; host_projects 0 = 0; host_sites 0 = 0; host_deploys 0 = 0; host_domains 0 = 0 | passed |
-| 2026-09-23 19:24 | openre | `/var/backups/openvibe/openre/20260923-192308/` (`ovhost drill openre`) | `pragma integrity_check` (openre) = ok; drill instance `/api/ready` 200 after 1s; `/api/health`, `/robots.txt` identical to production; stream_definitions 1 = 1; ingest_keys 1 = 1; destinations 0 = 0; migration_map 0 = 0 | passed |
-| 2026-09-23 23:07 | live | `/var/backups/openvibe/live/20260923-230748/` (`ovhost drill live`) | `pragma integrity_check` (live) = ok; drill instance `/api/ready` 200 after 1s; `/api/themes`, `/api/emotes/global`, `/api/streams`, `/api/streams/recently-online?limit=20`, `/api/streams/channel/japaneseoldguy/live` identical to production (JSON without volatile keys where declared); users 352 = 352; channels 123 = 123; managed_streams 103 = 103; streams 2477 = 2477; follows 62 = 62; chat_messages 70864 = 70864 | passed |
-| 2026-09-23 23:07 | media | `/var/backups/openvibe/media/20260923-230757/` (`ovhost drill media`) | `pragma integrity_check` (media) = ok; drill instance `/api/ready` 200 after 1s; `/browse?tab=videos`, `/browse?tab=clips` identical to production; media_objects 3056 = 3056; vods 906 = 906; clips 315 = 315; apps 7 = 7 | passed |
-| 2026-09-24 05:40 | ai | `/var/backups/openvibe/ai/20260924-054015/` (`ovhost drill ai`) | `pragma integrity_check` (ai) = ok; drill instance `/api/ready` 200 after 1s; `/api/health` identical to production; providers 3 = 3; models 9 = 9; routes 19 = 19; templates 26 = 26; workflows 47 = 47; quotas 5 = 5; citations 0 = 0 | passed |
-| 2026-09-24 05:40 | games | `/var/backups/openvibe/games/20260924-054054/` (`ovhost drill games`) | `pragma integrity_check` (world) = ok; drill instance `/api/ready` 200 after 2s; `/map.json`, `/api/v1/mods` identical to production; players 28 = 28; mods 0 = 0 | passed |
-| 2026-09-24 05:41 | tools | `/var/backups/openvibe/tools/20260924-054102/` (`ovhost drill tools`) | `pragma integrity_check` (tools-docs-analytics) = ok; `pragma integrity_check` (tools-docs-jobs) = ok; drill instance `/api/ready` 200 after 1s; `/release.json` identical to production (JSON without volatile keys where declared); tool_jobs 0 = 0 | passed |
-| 2026-09-24 05:54 | live | `/var/backups/openvibe/live/20260924-055447/` (`ovhost drill live`) | `pragma integrity_check` (live) = ok; drill instance `/api/ready` 200 after 1s; `/api/themes`, `/api/emotes/global`, `/api/streams`, `/api/streams/recently-online?limit=20`, `/api/streams/channel/japaneseoldguy/live` identical to production (JSON without volatile keys where declared); users 353 = 353; channels 123 = 123; managed_streams 103 = 103; streams 2477 = 2477; follows 62 = 62; chat_messages 70875 = 70875; first drill in the release layout | passed |
-| 2026-09-25 23:00 | sources | `/var/backups/openvibe/sources/20260925-212055/` (`ovhost drill sources`) | `pragma integrity_check` (sources) = ok; drill instance `/api/ready` 200 after 1s; sources 6 = 6; items 0 = 0; item_revisions 0 = 0; fetch_runs 0 = 0; acceptance `/api/ready`, `/api/health` | passed |
+The inventory `drill` block maps each database to its `DATABASE_URL` and `DATABASE_DIRECT_URL` setting. A drill selects the newest good logical dump, checks it with `pg_restore --list`, restores it into a scratch database under a temporary role, and starts a sandboxed second instance. It compares declared HTTP paths and row counts, stops the instance, and drops the scratch database and role.
## R2 eviction drills (Media's hot cache)
## R2 eviction drills (Media's hot cache)
@@ -42,22 +14,6 @@ The production instance is never touched.
@@ -42,22 +14,6 @@ The production instance is never touched.
|---|---|---|---|
|---|---|---|---|
| 2026-09-25 18:55 | 2178 (3.5 MB, `--pick --max-mb 200`, real `GET /v/2178?raw=1`) | before: served from R2 · evict: R2 copy removed, row → b2 · from B2: served from the canonical copy with the same first MiB · re-warm: R2 copy restored from B2 (HEAD verified), row → r2 · after: served from R2 again; `media_locations` b2 and r2 present. Artifact `/opt/openvibe.media/data/drills/r2-eviction-2178-2026-09-25T18-55-13-212Z.json` | passed |
| 2026-09-25 18:55 | 2178 (3.5 MB, `--pick --max-mb 200`, real `GET /v/2178?raw=1`) | before: served from R2 · evict: R2 copy removed, row → b2 · from B2: served from the canonical copy with the same first MiB · re-warm: R2 copy restored from B2 (HEAD verified), row → r2 · after: served from R2 again; `media_locations` b2 and r2 present. Artifact `/opt/openvibe.media/data/drills/r2-eviction-2178-2026-09-25T18-55-13-212Z.json` | passed |
-## How a drill was run (community)
-
-1. `sudo ovhost backup community`
-2. Copy `<backup>/community.db` to a temp directory owned by the service user; `pragma integrity_check`.
-3. Start `node server/index.js` from the production checkout with the service's env file, overriding
- `PORT` (spare loopback port), `COMMUNITY_DB_PATH` (the copy), and turning off side effects
- (`DISCORD_RELAY_ENABLED=0`, `EVENTS_URL=` empty) so the drill instance never talks to the world.
-4. Compare responses of read endpoints with production; compare row counts.
-5. Stop the drill instance by its port's pid (not `pkill -f`, which can match the operator's own shell),
- delete the temp directory.
-
-Looking back at that manual run: `EVENTS_URL=` had no effect, because Community has no events relay. The
-instance could also still reach production Network over loopback (JWKS, service tokens, identity
-resolve-batch). The inventory's `community` drill block now also empties `OV_OAUTH_CLIENT_SECRET` and
-points `OV_LIVE_INTERNAL_URL` and `OV_MEDIA_INTERNAL_URL` at a closed port.
-
## Drilling an off-host copy
## Drilling an off-host copy
Scheduled backups and their encrypted off-host copies are described in [backups.md](backups.md).
Scheduled backups and their encrypted off-host copies are described in [backups.md](backups.md).
@@ -70,68 +26,15 @@ sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-community
@@ -70,68 +26,15 @@ sudo ovhost drill community --backup /var/lib/openvibe-restore/drill-community
sudo rm -rf /var/lib/openvibe-restore/drill-community
sudo rm -rf /var/lib/openvibe-restore/drill-community
```
```
-Write "off-host" and the run id in the Backup column of the table above.
+Record the run id and that the copy came from off-host storage.
## Running a drill with `ovhost drill`
## Running a drill with `ovhost drill`
-```
-sudo ovhost backup community # a fresh backup keeps the comparison meaningful
-sudo ovhost drill community # restore, start, compare, stop, clean up, log
-sudo ovhost drill community --keep # keep the drill directory (copy, drill.env, drill.log)
-sudo ovhost drill community --backup /var/backups/openvibe/community/20260923-034341
-```
-
-The command does the steps above automatically for any service whose inventory entry has a `drill`
-block (see `host.example.json`):
-
-1. **Restore.** The command picks the latest `ovhost backup` from `<stateDir>/backups/<service>.jsonl`,
- or the `--backup` directory. It copies every database named in `drill.databases` into
- `/var/lib/openvibe-drills/<service>-<stamp>/db/`, a directory owned by the service user (mode 0700).
- Backups are root-only (directories 0700, files 0600, see [backups.md](backups.md)), so root makes
- the copy with `install -o <service user> -m 0600`, and the copy belongs to the service user.
- `PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;
- otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into a
- scratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).
- Every `objects` directory the service declares is restored as well: its archive is extracted into
- `<tmp>/<name>`, owned by the service user and never the live directory, and every blob is checked
- against the sha256 in its file name.
-2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
- service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
- unit's non-secret `Environment=` values and the production env file. After that it loads
- `drill.env`, which holds the drill port, the restored database paths and the switches that turn
- side effects off. `drill.env` holds inventory values only and is loaded last, so its values win.
- ovhost never reads the env file's values.
-3. **Sandbox.** The instance runs with:
- - `ProtectSystem=strict`, with only the drill directory writable;
- - `PrivateTmp`, `ProtectHome=read-only` and `NoNewPrivileges`;
- - binding allowed only on its drill port (`SocketBindAllow`);
- - no addresses beyond loopback (`IPAddressDeny=any`, `IPAddressAllow=localhost`);
- - `RuntimeMaxSec`, which stops it even if ovhost dies.
-
- Loopback stays open because services fetch Network's public keys there. Production services
- are therefore still reachable on loopback, and the inventory's overrides are what keep the drill
- away from them.
-4. **Compare.** Once the instance answers its readiness path, every `drill.compare` path is fetched
- from production and from the drill instance. Bodies must match byte for byte, or as JSON with the
- listed volatile keys removed. Statuses must match, and production must answer 2xx. Each
- `drill.counts` table is counted in production and in the restored copy.
-5. **Stop and record.** The instance's own MainPID gets SIGTERM, then SIGKILL after 20 s. ovhost
- never uses `pkill -f` and never signals a production pid. The drill directory is removed unless
- `--keep` is set, or unless the instance could not be confirmed stopped. The result is appended to
- `/var/lib/openvibe-host/drills/<service>.jsonl`, and a row for the table above is printed.
-
-Exit codes are `0` passed, `1` refused, and `2` failed. A refusal happens when ovhost is not run as
-root, the port is in use, there is no backup, or the service's drill is unsupported. A failure is
-integrity, readiness, a mismatch, or a stop that did not complete. Every failure is still logged and
-cleaned up.
-
-A drill compares the backup with production *now*. If writes landed after the backup, row counts and
-list endpoints differ, and the drill reports that honestly. Take the backup right before the drill.
+Run `sudo ovhost drill <service>` or pass `--backup <absolute directory>` to select a copy. The service must have a supported drill block and an unused loopback port. PostgreSQL drills require URL mappings for every declared database and a `VALKEY_URL` override so the drill cannot write into production.
### A PostgreSQL drill
### A PostgreSQL drill
-Step 1 above is the SQLite path. A service whose `databases[]` entry is `engine: postgresql`
-([db-inventory.md](db-inventory.md)) restores a **logical dump** instead of a file copy:
+A service whose `databases[]` entry is `engine: postgresql` ([db-inventory.md](db-inventory.md)) restores a **logical dump**:
1. **Restore.** The drill takes the service's newest `<name>.dump` (`pg_dump -Fc`, from
1. **Restore.** The drill takes the service's newest `<name>.dump` (`pg_dump -Fc`, from
`<stateDir>/backups/<service>.jsonl`, or the `--backup <dir>` directory) and creates a scratch
`<stateDir>/backups/<service>.jsonl`, or the `--backup <dir>` directory) and creates a scratch
@@ -157,8 +60,7 @@ Step 1 above is the SQLite path. A service whose `databases[]` entry is `engine:
@@ -157,8 +60,7 @@ Step 1 above is the SQLite path. A service whose `databases[]` entry is `engine:
the second instance write locks, caches and queues into production's keyspace.
the second instance write locks, caches and queues into production's keyspace.
- **`DATABASE_URL` un-overridden would serve production data.** The drilled instance would read and
- **`DATABASE_URL` un-overridden would serve production data.** The drilled instance would read and
write the live database instead of the restored copy, and every comparison would be vacuous.
write the live database instead of the restored copy, and every comparison would be vacuous.
-3. **Compare.** `counts` are read from production and from the scratch database over `psql` (the SQLite
- drills compare files). `countsTolerance` in the drill block (default `0`) is the number of rows that
+3. **Compare.** `counts` are read from production and from the scratch database over `psql`. `countsTolerance` in the drill block (default `0`) is the number of rows that
may differ: production keeps writing between the read that makes the backup and the read that counts
may differ: production keeps writing between the read that makes the backup and the read that counts
it, so a busy table is allowed to drift by that many rows instead of failing the drill.
it, so a busy table is allowed to drift by that many rows instead of failing the drill.
@@ -176,177 +78,4 @@ The scratch database and its role are dropped in a `finally`, together with stop
@@ -176,177 +78,4 @@ The scratch database and its role are dropped in a `finally`, together with stop
success and on failure. A PostgreSQL drill always cleans up after itself; `--keep` keeps the drill
success and on failure. A PostgreSQL drill always cleans up after itself; `--keep` keeps the drill
directory (the copy, `drill.env`, `drill.log`) but never the scratch database or role.
directory (the copy, `drill.env`, `drill.log`) but never the scratch database or role.
-For such a service the **Checks** column in the table above reports `pg_restore --list` (that the dump is
-a readable archive) rather than `pragma integrity_check`, and each count is read over `psql` from
-production and from the scratch database.
-
-### Services
-
-Supported with declared overrides (the table above records which have passed on the host):
-
-- network
-- community
-- events
-- host (the Host API: the `ov_host` database and the tenant object store, whose blobs are sha256-checked; counts only)
-- billing
-- chat
-- search
-- sources
-- wiki
-- blog
-- tips
-- vip
-- news
-- reviews
-- deals
-- coupons
-- trade
-- codes
-- openre (not run yet). Only openre-api starts (`drill.command`), with `OPENRE_DRILL=1`. From OpenRestream
- `5e86ea7` on, that makes it read-only: writes, `/play/` and sign-in answer 503. The coordinator
- and every transport worker refuse to start in a drill, and the event relay and Media calls are
- off. The API never runs a transport or the relay in any case. Empty `EVENTS_URL` and
- `OV_OAUTH_CLIENT_SECRET`, plus `MEDIA_URL` on the closed port, cover older releases. The workers
- stay out of every ovhost command: they are `workerUnits`, which are listed and never restarted.
- Run it after the OpenRestream release with `OPENRE_DRILL` is deployed (OpenRestream `docs/cutover.md`
- phase C).
-- live (not run yet). Needs OpenVibe.Live `8a58aea` or later; `drill.requires` refuses an older
- checkout. The production unit's ExecStart starts with `LIVE_DRILL=1`, `DB_PATH` on the copy and
- `DATA_DIR={tmp}/data`. In that mode (Live `server/drill.js`, `docs/deploy.md#restore-drills`) Live
- refuses to start unless `DB_PATH` and `DATA_DIR` are outside the checkout and `/opt/openvibe.live`,
- `HOST` is loopback, `PORT` is not 3000 and no socket was handed over by systemd. It writes only
- under `DATA_DIR` (a fresh `analytics.db`, empty data directories; the per-location `*_PATH` values
- in `live.env` are ignored) and the copy. It starts only its HTTP server:
- - no RTMP, SRT, WHIP, SFU or JSMPEG listener, and no TURN credential;
- - no WebSocket: upgrades get 403;
- - no jobs loop, restream or relay resume, AI job, Media reconciler, Events outbox, chat bridge,
- identity sync, deploy notice, star job or registry refresh.
-
- Every method but GET, HEAD and OPTIONS answers 403. Outbound connections, `fetch`, programs other
- than `git`, UDP sockets and other listeners are refused in-process, so Media, Community and
- Network look down to it. `rs-companion.db` is not restored: the server never opens it. Compared:
- - `/api/themes` and `/api/emotes/global`, byte for byte;
- - `/api/streams` and `/api/streams/channel/japaneseoldguy/live`, without the values production
- changes while someone is live (viewer counts, heartbeats, live thumbnails, recording state);
- - `/api/streams/recently-online?limit=20`, without `vod_thumbnail`, the one value it asks Media for.
-
- Counts: `users`, `channels`, `managed_streams`, `streams`, `follows`, `chat_messages`. Not
- compared: `/release.json`, which production computes at its own boot, so a `public/`-only deploy
- without a restart makes it differ; and the home statistics, which are cached for 30 s and windowed
- by the clock. Any existing channel can replace `japaneseoldguy`.
-- media (not run yet). Needs OpenVibe.Media `6e74eb0` or later (`drill.requires`). `MEDIA_DRILL=1`
- with `DB_PATH` on the copy. Media then refuses to start unless `DB_PATH` is outside the checkout
- and `/opt/openvibe.media`, `HOST` is loopback and `PORT` is not 4100. It starts only its HTTP
- server:
- - no app seeding or JWKS refresh;
- - no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant
- scans, disk guardian, thumbnail cleanup, object purge, backfill or orphan-recording finalize;
- - no Events relay and no webhook (the `apps` table's URLs are production's).
-
- It writes no file and creates no directory. It never opens a file path from the database: every
- byte route (`/v` and `/c` bytes, `/t`, `/a`, `/f`, `/o`, paste screenshots, live frames) answers
- 503, while watch pages and `/browse` render from the copy. Every method but GET, HEAD and OPTIONS
- answers 403 and `/auth/*` 503. Connections, `fetch`, programs other than `git` (no ffmpeg or
- ffprobe), UDP and other listeners are refused in-process. `/api/ready` drops the storage and
- remote-tier checks and reports `"mode": "drill"`. The storage paths from `media.env` are moved
- under `{tmp}/storage`, which is never created, as a second switch. `THUMBNAILS_PATH` stays
- production's, read-only in the sandbox, because `/browse` counts that directory in its tab bar;
- the drill lists it and serves none of it. Compared byte for byte: `/browse?tab=videos` and
- `/browse?tab=clips`. Counts: `media_objects`, `vods`, `clips`, `apps`. Production caches the tab
- counts for 60 s, so take the backup right before the drill.
-
-Services that are not deployed yet (everything except network, live, media, tools, community and
-games as of 22 Sep) have entries built from their repositories' `deploy/` directories.
-
-- ai (passed 2026-09-24 05:40 UTC): AI starts with AI_ENABLED=0 and every provider key, base URL
- and the HTTP seam blanked, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and
- EVENTS_URL are blanked (no service calls, no events) and WHISPER_BIN too (no transcription). Boot
- marks interrupted runs failed in the restored copy only. `/api/health` is compared; the counts cover
- configuration and citations, because runs, requests and usage grow by the minute (integrity_check
- covers them).
-- tools (passed 2026-09-24 05:41 UTC): the **docs** app alone. Tools runs eight units; `drill.unit` picks
- `openvibe-tools-docs.service`, whose ExecStart, WorkingDirectory and Environment= the drill copies,
- and `drill.productionPort` (4016) is what the comparison reads, not the gateway's 4001. docs takes a
- data DIRECTORY (`DATA_DIR`), so both restored copies land in `{tmp}/data/` under their production
- names (`analytics.db`, `jobs.db`). Existing Tools switches make it side-effect-free:
- `TOOLS_JOB_RESULTS=local` (the job pruner would otherwise delete production Media objects for
- expired jobs in the copy), `EVENTS_PUBLISH=off` and an empty `EVENTS_URL` (the copy's pending outbox
- rows would be republished), `TOOLS_JOBS_CONCURRENCY=0` (no job runs), `UPLOADS_DIR`/`OUTPUT_DIR`
- under `{tmp}`, an empty `OV_OAUTH_CLIENT_SECRET`. Job recovery and pruning, analytics aggregation
- and upload retention still run, on the copy only. `drill.requires` refuses the drill unless the
- deployed checkout has those switches. Compares `/release.json`; counts `tool_jobs`.
-- games (passed 2026-09-24 05:40 UTC): Reviewed against OpenVibe.Games `7863fc6`. One HTTP listener; `/ws` and
- `/editor-ws` are upgrades on it. tsx runs `src/main.ts` directly, with no build. The unit sets
- `DB_PATH` with `Environment=`, and that value never reaches the drill because the drill sets
- `DB_PATH` itself. `EVENTS_PUBLISH=off` stops the outbox relay, `MEDIA_MIRROR=off` stops the
- map-asset mirror, and an empty `OV_OAUTH_CLIENT_SECRET` stops every service call. The 30 Hz tick
- loop keeps running because readiness needs recent ticks, and it flushes the world into the copy
- only. `map.json` stays production's file, which is read-only in the sandbox. `drill.requires`
- checks that `apps/server/src/config.ts` has both switches. The drill compares `/map.json` and
- `/api/v1/mods` and counts `players` and `mods`, which the simulation does not write. There is no
- Games backup yet, so run `sudo ovhost backup games` first.
-
-### What the drill block can declare for multi-process and checkout-relative services
-
-- `unit`: which of the service's units to start (its ExecStart, WorkingDirectory and non-secret
- Environment=). It is required when there are several units and no `command`.
-- `productionPort`: the production port the compare paths are read from. The default is the service's
- port.
-- `databases.<name>: { "env": "DATA_DIR", "dir": true }`, or `"dir": "{tmp}/<name>"`: the copy keeps
- its production file name inside that directory, and the env var, if one is given, points at the
- directory. For a `postgresql` database the entry is instead `{ "url": "<NAME>", "directUrl":
- "<NAME>" }`: the two env vars that point the drilled service at a database URL, which the drill sets
- to the scratch database (see [A PostgreSQL drill](#a-postgresql-drill)).
-- `bind: [{ "from": "{tmp}/…", "to": "<path inside the checkout>" }]`: `BindPaths=` in the drill
- unit's own mount namespace. Use it for files a service opens relative to its checkout with no env
- override: the restored copy appears at the production path for the drill instance only. Tools yt,
- maps and food open `apps/<app>/data/analytics.db` this way. Drill one of them with
- `unit: openvibe-tools-<app>.service`, a `databases` entry with `"dir": "{tmp}/app-data"`, and
- `bind: [{ "from": "{tmp}/app-data", "to": "/opt/openvibe.tools/apps/<app>/data" }]`. For yt, also
- set `YT_PROXY=`. Before first use, check on the host that `BindPaths=` works together with
- `ProtectSystem=strict` in a `systemd-run` transient unit.
-- `requires: [{ "file": "<path in the checkout>", "contains": "<text>" }]`: the drill refuses before
- it creates anything unless the deployed checkout has every switch its overrides rely on. An older
- release would ignore the override and run its side effects.
-
-### What Tools and Games should add (product side, not Host)
-
-These make the drills stricter and quieter. Neither is needed for the drills above to be
-side-effect-free.
-
-- **Tools `TOOLS_DRILL=1`:**
- - `_shared/jobs/system.js:576-585`: skip `recover()`, `prune()`, the prune interval and `kick()`.
- - `_shared/jobs/index.js:83-105`: force local results and no outbox.
- - `_shared/analytics/tracker.js:81,105-118`: no startup `DELETE` of rate tracking, no
- flush/aggregate/prune timers, and a no-op middleware.
- - The `retention.startCleanup()` calls (img `:273`, docs `:323`, audio `:307`), and yt's
- `startCleanup()` and `startUpstreamProbe()` (`:280-281`, which spawns yt-dlp against YouTube).
- - `_shared/observe.js:95-105`: no `writableDir` probe writes.
- - A `DATA_DIR` override in yt `index.js:26`, maps `:25` and food `:20`, so those apps no longer
- need `bind`.
-
- With these, the restored copies stay byte-stable during a drill, and row counts other than
- `tool_jobs` become comparable.
-- **Games `GAMES_DRILL=1`:**
- - `config.ts:60-99`: force the client secret, OAuth, `networkAuthUrl`, `eventsUrl` and `mediaUrl`
- to null.
- - `config.ts:77-79`: refuse to start unless `HOST` is loopback and `DB_PATH` is not the production
- path.
- - `main.ts:252`: refuse `/ws` and `/editor-ws` upgrades.
- - Return 403 for map, map-asset and mod writes (`httpServer.ts:161,231` and the `routes.ts` write
- branches), and for `/auth/*` (`httpServer.ts:333`).
- - Optionally skip the periodic flush (`gameServer.ts:2320,2502`) and the prune timer
- (`main.ts:148`), so world tables can be counted too.
-
-Marked `drill: { supported: false }`, with the reason in the inventory:
-
-- **ai:** a drill must start AI with every provider key blanked, or queued and new runs would call
- paid providers. Backups work.
-
-Until 23 Sep 2026 live and media were unsupported too:
-- **live:** `data/analytics.db` was opened relative to the checkout at boot. RTMP always bound.
- Restreams, relays, AI jobs and Media reconcilers resumed from rows in the copy.
-- **media:** webhooks to Live took their URL from the `apps` table. The health job, junk sweep and
- clip jobs ran ffmpeg against production files through absolute paths stored in the database.
-
-`LIVE_DRILL` and `MEDIA_DRILL` (above) replace those reasons.
+Each database check uses `pg_restore --list`, and row counts are read with `psql` from production and the scratch database.
deleted file mode 100644
deleted file mode 100644
@@ -1,172 +0,0 @@
@@ -1,172 +0,0 @@
-# Storage state (roadmap Wave 22)
-
-This page covers two Wave 22 exit criteria:
-- "No service still queries SQLite for canonical data unless that is its documented and reviewed state."
-- "Each service's /ready tells the truth about its actual backing store."
-
-It has one row per deployed service. Each row gives the service's backing store, whether that store is
-canonical, its status under ADR-007, what its readiness handler actually checks (read from the handler,
-not from docs), and its backup and drill status from [restore-drills.md](restore-drills.md).
-
-ADR-007 (`OpenVibe.Contracts/docs/adr/ADR-007-data-ownership.md`) decides:
-- "SQLite remains the documented store per service while the platform is single-host; each service's
- `/ready` reports its actual store."
-- A service moves to PostgreSQL individually when it needs concurrent writers across processes, a second
- host, or sizes SQLite handles poorly.
-- Money paths use SERIALIZABLE transactions and row locks when they move.
-
-As reviewed on 2026-09-23 every service ran on SQLite (better-sqlite3), with one database per service.
-Since 2026-09-28 nineteen of them run on the host's **PostgreSQL cluster** (ADR-035), and five stay on
-SQLite: network, chat, tools, games and live. The per-service table below is the Wave 22 review of the
-SQLite state; where a service switched, its SQLite file is now frozen read-only for the 7-day rollback
-window (compatibility-register.md, C-89), and the live rows are in the cluster. **This page remains the
-Wave 22 review of that state.**
-
-## Where the data lives now
-
-- **PostgreSQL services** (tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes,
- vip, ai, host, events, community, billing, media, openre) keep their rows in the cluster on the data
- role (`roles/data/`, ADR-035): PostgreSQL 18 on loopback, one database per service named `ov_<name>`
- (`ov_trade`, `ov_media`, …). The service reaches it through **PgBouncer** on `127.0.0.1:6432`, which
- fronts the cluster for the app roles (and carries the app role's SCRAM password). Their durability is
- **pgBackRest**, off-host: a full cluster backup weekly, a differential nightly and the WAL stream, which
- the nightly `ovhost backup` verifies rather than copies ([backups.md](backups.md#postgresql-services)).
-- **What stays on the local filesystem** for those services is the rest of the data: object and media
- files under `*_PATH` and the object stores (Media's B2/R2), blobs (`HOST_STORAGE_DIR`), uploads and
- outputs, plus the frozen pre-switch SQLite file kept read-only for rollback. Those files are not in the
- cluster and are backed up (or not) exactly as before.
-- **SQLite services** are unchanged: one file per service, copied by the online backup API as the service
- user ([backups.md](backups.md)).
-
-`openvibe-shared/ready` is still what each service's `/ready` reports ([below](#per-service)); a switched
-service's `db` check reads the cluster as its `DATABASE_URL`/`DATABASE_DIRECT_URL` do.
-
-Sources:
-- Each repository's `server/config.js`, `.env.example` and `deploy/systemd/*.service`.
-- The ovhost inventory example [`host.example.json`](../host.example.json), which lists production paths.
-- The operator notes of 2026-09-23, for what is deployed.
-- Live was read at `~/orca/workspaces/OpenVibe.Live/seadragon` (`ae07553`).
-
-`openvibe-shared/ready` (v1.3.0, `node_modules/openvibe-shared/ready.js:43-124`) behaves like this:
-- It has no checks of its own.
-- A service passes named checks, and each check is required unless it is marked optional.
-- It answers 503 only when a required check fails.
-- A failed optional check gives 200 with `status: "degraded"`.
-- The default timeout is 2 s.
-
-## Findings
-
-1. **Readiness that does not check the store.**
- - *(Fixed 2026-09-23, Games `f11e21c`: `/api/ready` checks `world.db` and the tick; host-only.)* **Games** had no `/api/ready`. `/healthz` returns `{ ok: true, tick }` without touching `world.db`
- (`apps/server/src/net/httpServer.ts:596-600`).
- - *(Fixed 2026-09-23, Chat `45f2b1f`: required `db` read of `chat_messages`, optional `live_sync` degraded after 60 s without a clean pass.)* **Chat** `/ready` was 200 as soon as `ctx.stats.lastSyncAt` was set (`server/app.js:77-86`). That
- timestamp is set even when every Live sync step fails, because `tick()` and `sync()` catch each
- step's error and set it anyway (`server/live-context.js:665-683`). The only database access is the
- `mirror.pending` count (`SELECT COUNT(*) FROM live_mirror_outbox`, `server/bridge/live-mirror.js:34`,
- production runs `LIVE_MIRROR=1`). A broken `chat.db` would therefore surface as a thrown 500, not as a
- named check.
-2. **Readiness that checks the main store but not every store.**
- - **Live** runs `SELECT 1` on `live.db`. It does not check `data/analytics.db` or `data/rs-companion.db`.
- - The **Tools** satellites maps, text, yt and food only check `analytics.db`, and only as an optional
- check. By design, that is their only database, and the tools work without it.
- - The **Tools** gateway reports each satellite as an optional upstream. So a failed `jobs.db` on img,
- audio or docs makes the gateway *degraded* (still 200), while that satellite's own `/api/ready`
- answers 503.
-3. **ovhost's post-deploy readiness URL is not the truthful endpoint for four services.**
- `host.example.json` polls:
- - Network `/api/health`, which is static (`server/index.js:392-394`), although Network's `/api/ready`
- checks the database and the signing key;
- - Media `/healthz`, which is liveness only, although Media's `/api/ready` checks the database and
- storage;
- - Tools gateway `/api/health`;
- - Games `/healthz` (the inventory now uses `/api/ready`, 2026-09-23; Network, Media and Tools too).
-
- The inventory should point at `/api/ready` where one exists. Games has none.
-4. *(Fixed: the inventory has had AI and OpenRestream entries since `f0dbd0c`, and both are in the scheduled
- `ovhost backup --all` with off-host copies, [backups.md](backups.md). Their drills are still unsupported.)*
- **Two services have no ovhost inventory entry, so no `ovhost backup` and no drill**: AI (`ai.db`) and
- OpenRestream (`openre.db`). Whether they are backed up any other way is unknown. Check the host's cron
- and timers.
-5. **The inventory's Tools backup list is incomplete.** It names the seven `analytics.db` files only.
- The job databases `apps/{img,audio,docs}/data/jobs.db` (table `tool_jobs`,
- `apps/_shared/jobs/index.js:77`) are missing. Tools W11 jobs have been in production since
- 2026-09-23.
-6. **SQLite reads outside the documented state.** These go against exit criterion 1, and each has an entry
- in [compatibility-register.md](compatibility-register.md):
- - **Live reads its frozen legacy `vods`/`pastes` tables for live pages**, although Media and Community
- hold the canonical rows (C-73):
- - `server/ai/chat-ai-routes.js:337`
- - `server/admin/routes.js:91-92`
- - `server/arena/arena-service.js:436`
- - `server/streaming/routes.js:1547`
- - **Network may open Live's `live.db`** read-only at boot (`syncLinkedLiveRoles`,
- `server/db/database.js:28-99`), which ADR-007 forbids (C-58). None of its default paths is Live's
- production path. Whether `OPENVIBELIVE_DB_PATH` is set is unknown: check `/etc/openvibe/network.env`.
- - **Chat reads Live's files** at `/opt/openvibe.live/data/sounds`. These are files, not a database
- (`deploy/systemd/openvibe-chat.service:43`).
-7. **Canonical money is still in Live.** Billing's ledger (was `billing.db`, now the cluster database
- `ov_billing` after the 2026-09-28 switch) is a shadow (Billing `STATUS.json: "authoritative": false`)
- until the cutover in [cutover-runbook.md](cutover-runbook.md) step B.
-
-## Per service
-
-"Drill" is the latest result in [restore-drills.md](restore-drills.md), all 2026-09-23. "Deployed" is from
-the operator notes unless cited. Where a repo's `STATUS.json` disagrees, the operator notes and the drill
-log win.
-
-**The rows record the 2026-09-23 SQLite state.** For a service that has since switched to PostgreSQL —
-tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events,
-community, billing, media, openre (C-89) — the "Backing store" path below is now the **frozen pre-switch
-SQLite file** kept read-only for the 7-day rollback, and the live store is the cluster database
-`ov_<name>` reached through PgBouncer, as in [Where the data lives now](#where-the-data-lives-now). The
-five SQLite rows (network, chat, tools, games, live) are current. The handler citations are the ones
-reviewed on 2026-09-23.
-
-| Service | Backing store (production path; env, default) | Canonical? | ADR-007 status | Readiness checks (handler) | Store checked? | Backup / drill |
-|---|---|---|---|---|---|---|
-| **Network** | `/opt/openvibe.network/data/network.db`; `DB_PATH`, default `./data/network.db` (`server/config.js:55`). Analytics lives in the same file (`README.md:306-307`). | Yes: identity, OAuth clients, subjects, OpenCoins wallets, themes, notifications, analytics | SQLite, the reviewed single-host state. `docs/platform-plan.md:74,111` defers PostgreSQL "until a room needs it". | `/api/ready` (`server/observability.js:58-96`). Required: `db` (`SELECT COUNT(*) FROM oauth_clients`), `signing_key` in production (signs and verifies an RS256 probe; fails on the HS256 dev key). Optional: `registry_poll`, `discord_bot`. | **Yes** | Drill passed 05:22. ovhost's deploy readiness polls the static `/api/health` (finding 3). |
-| **Live** | `/opt/openvibe.live/data/live.db` (legacy in-place layout, "current production", Live `docs/deploy.md:31`); `DB_PATH`, default `./data/live.db` (`server/config.js:228`). Also `data/rs-companion.db`, and `data/analytics.db`, whose path was hard-coded with no env (`server/index.js:361-363`); from Live `8a58aea` it is `<DATA_DIR>/analytics.db` (`ANALYTICS_DB_PATH` overrides it; `server/paths.js`). | Yes: streams, channels, follows, Live accounts, restreams, **money** (`BILLING_AUTHORITY` unset = `live`), AI state (`vod_ai_state`/`clip_ai_state`), analytics. **Not canonical:** the 12 chat mirror tables (Chat is, since 02:03 UTC) and the frozen `vods`/`clips`/`pastes` (Media and Community are). | SQLite, the reviewed state. No PostgreSQL plan. Money moves to Billing, not to PostgreSQL. | `/api/ready` (`server/index.js:748-756`, `server/web/observability.js:79-106`). Required: `boot`, `db` (`SELECT 1` on `live.db`). Optional: `sfu`, `media` (`MEDIA_URL/healthz`, cached 30 s), `network_key` (PEM loaded, no call). | **Yes** for `live.db`; not `analytics.db` or `rs-companion.db` | `ovhost backup` covers `live.db` and `rs-companion.db`. `analytics.db` is left out on purpose (`host.example.json` `_databasesNote`). Drill **supported** from Live `8a58aea` (`LIVE_DRILL=1`: reads only, files under `DATA_DIR`, no listeners but HTTP, no jobs; not run yet). Before that release it was unsupported: `analytics.db` was opened relative to the checkout, RTMP always bound, and restreams, relays and AI jobs resumed from the copy. |
-| **Media** | `/opt/openvibe.media/data/media.db`; `DB_PATH`, default `./data/media.db` (`server/config.js:24`). Local `VOD_PATH`, `CLIPS_PATH`, `PASTES_PATH`, `THUMBNAILS_PATH`, `FILES_PATH`, `OBJECTS_PATH` (`config.js:27-58`), plus B2 and R2 object storage. | Yes: media objects, VODs, clips, files, thumbnails, avatars. Paste writes are frozen (Community is canonical). Media still serves paste screenshot bytes. | SQLite, the reviewed state. No PostgreSQL plan. | `/api/ready` (`server/observability.js:72-107`). Required: `db` (`SELECT COUNT(*) FROM apps`), and `storage_{vods,clips,pastes,thumbnails,files,objects}` (writes and unlinks a probe). Optional: `network_jwks`, `remote_b2`/`remote_r2` (HeadBucket, cached 60 s), `events_outbox` (>1000 pending). Also reports `recordings_in_progress`. | **Yes** (database and disk; object storage optional) | `ovhost backup` covers `media.db`. Drill **supported** from Media `6e74eb0` (`MEDIA_DRILL=1`: reads only, no webhooks, jobs or ffmpeg, byte routes 503; not run yet). Before that release it was unsupported: webhooks to Live come from the `apps` table, and ffmpeg jobs run against production paths. ovhost's deploy readiness polls `/healthz` (finding 3). |
-| **Tools** | The gateway has no database. Each satellite has `apps/<app>/data/analytics.db` (relative to `/opt/openvibe.tools`). img, audio and docs also have `data/jobs.db` (`apps/_shared/jobs/index.js:77`), `data/uploads` and `data/output`; yt has `data/downloads`. Job results go to Media when `TOOLS_JOB_RESULTS=media` (default `local`, `apps/_shared/jobs/index.js:32-36`). | Yes: jobs (img, audio, docs) and per-app analytics | SQLite, the reviewed state. One file per process, so ADR-007's "concurrent writers across processes" trigger does not apply. | `/api/ready` on every unit (`apps/_shared/observe.js:71-162`). Gateway: `catalog` required; `network_key`, `service_directory`, `community`, and each `satellite_<name>` optional (`apps/gateway/server/index.js:60-72`). img, docs, audio: `jobs_db` (`SELECT COUNT(*) FROM tool_jobs`), `job_runtime` and data directories required. yt: `downloads_dir`. food: `maps` upstream. maps, text: none required (`analytics_db` optional). | **Partly**: `jobs.db` yes on its satellites; analytics optional by design; the gateway reports satellites as optional (finding 2) | `ovhost backup` covers the seven `analytics.db` files only; **`jobs.db` is missing** (finding 5). Drill **unsupported**: eight units, relative `analytics.db` paths, job runtimes re-queue at start. |
-| **Games** | `/opt/openvibe.games/data/world.db`; `DB_PATH`, default `data/world.db` (`apps/server/src/config.ts:79`). No `.env.example`. | Yes: world, accounts, inventory | SQLite, the reviewed state. No PostgreSQL plan. | `/api/ready` (since `f11e21c`): required `db` (`meta.schema_version` in `world.db`) and `tick` (last step < 5 s ago); 404 on the public vhosts. `/healthz` unchanged. | **Yes** | `ovhost backup` covers `world.db`. Drill **unsupported** (runtime side effects not reviewed). Games is `managed: false` in the inventory. |
-| **Community** | `/var/lib/openvibe-community/community.db` (`StateDirectory`, `deploy/systemd/openvibe-community.service:26-27`). `/opt/openvibe.community/data/community.db` is the legacy location, and the inventory keeps whichever of the two exists. `COMMUNITY_DB_PATH`, default `./data/community.db` (`server/config.js:56`). | Yes: pastes (`PASTES_AUTHORITY=community` since 2026-09-22), comments, spaces, threads, Pulse | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:29-66`). Required: `db` (`SELECT COUNT(*) FROM sqlite_master WHERE type='table'`, fails on 0). Optional: `network_jwks`, `media` (community mode) or `live` (live mode). Reports `pastes_authority`. | **Yes** | Drill passed 03:44 (manual) and 05:22. |
-| **Events** | `/var/lib/openvibe-events/events.db` (`deploy/systemd/openvibe-events.service:30-31`); `EVENTS_DB_PATH`, default `./data/events.db` (`server/config.js:67`). | Yes: the durable event store, subscriptions, deliveries, DLQ | SQLite, the reviewed state. `README.md:163`: "the plan's PostgreSQL + Redis fanout is a later step". No dated plan. | `/api/ready` (`server/app.js:41-66`). Required: `db` (`store.ping()` = `SELECT 1`), `network_jwks`, `delivery_worker` (unless `EVENTS_WORKER=off`). Optional: `dlq` (degraded at more than `EVENTS_DLQ_DEGRADED_AT`, default 100). | **Yes** | Drill passed 05:22. |
-| **Chat** | `/var/lib/openvibe-chat/chat.db` (`deploy/systemd/openvibe-chat.service:38-39`); `CHAT_DB_PATH`, default `./data/chat.db` (`server/config.js:34`). It also reads Live's `/opt/openvibe.live/data/sounds` (`:43`). | Yes, since 2026-09-23 02:03 UTC: the 12 chat tables (`table_authority = 'chat'`). Six staged tables are still Live's. | SQLite, the reviewed state. | `/ready` (loopback). Since `45f2b1f`: required `db` (`SELECT MAX(id) FROM chat_messages`, 503 on failure); optional `live_sync` (degraded when no clean Live sync pass for `LIVE_SYNC_STALE_MS`, 60 s, or one step is late). `mirror.pending` is still reported only. | **Yes** | Drill passed 05:22. |
-| **OpenRestream** | `/var/lib/openre/openre.db` (all four units); default `./data/openre.db`. | Stream definitions, sessions and destinations for switched slots. **No slot is switched**, so Live is still the ingest authority for every stream. | SQLite, the reviewed state. `README.md:66` names PostgreSQL only as a future ADR-007 trigger. | `/api/ready` (`server/app.js:31-51`, own handler). Required: `db` (`SELECT 1`), `key` (Network public key). Reported but not required: workers, coordinator lease, store, events. Not checked: RTMP reachability, Media, Events. | **Yes** | **Not in the ovhost inventory: no `ovhost backup`, no drill** (finding 4). |
-| **Billing** | `/var/lib/openvibe-billing/billing.db` (`deploy/systemd/openvibe-billing.service:23-25`, `StateDirectoryMode=0700`); `BILLING_DB_PATH`, default `./data/billing.db` (`server/config.js:62`). The journal is append-only (triggers refuse UPDATE and DELETE, `README.md:45-50`). | **Not yet.** It is a shadow: `STATUS.json` `"authoritative": false`. Live holds the money until the cutover. | SQLite, the reviewed state. ADR-007's SERIALIZABLE and row-lock rule applies when money moves to PostgreSQL. No plan yet. | `/api/ready` (`server/app.js:54-60`, own handler). `SELECT 1 FROM settings WHERE id = 1` and Network public key loaded, otherwise 503 `service.not_ready`. `/api/health` reports frozen state and providers. | **Yes** | Drill passed 05:22. |
-| **Tips** | `/var/lib/openvibe-tips/tips.db` (`deploy/systemd/openvibe-tips.service:25`); `TIPS_DB_PATH` (`server/config.js:31`). | Yes: creator tip profiles, interactions, goals | SQLite, the reviewed state. | `/api/ready` (`server/app.js:74-79`, own handler): `SELECT 1 FROM settings WHERE id = 1` and key loaded. | **Yes** | Drill passed 05:23. |
-| **VIP** | `/var/lib/openvibe-vip/vip.db` (`deploy/systemd/openvibe-vip.service:25`); `VIP_DB_PATH` (`server/config.js:31`). | Yes: creators, plans, memberships, perks (checkout closed until the Billing cutover) | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:20-48`). Required: `db` (`COUNT(*) FROM vip_creators >= 1`). Optional: `network_jwks`, `billing`. | **Yes** | Drill passed 05:23. |
-| **AI** | `/var/lib/openvibe-ai/ai.db` (`deploy/systemd/openvibe-ai.service:30-31`); `AI_DB_PATH`, default `./data/ai.db` (`server/config.js:81`). | Yes: workflows, runs, providers, quotas, usage (Live keeps its AI product state, AI `docs/migration.md:30-35`) | SQLite, the reviewed state. | `/api/ready` (`server/app.js:26-33`, own handler). Required: `db` (`SELECT 1`), `key`, `workflows` (at least one registered). | **Yes** | **Not in the ovhost inventory: no `ovhost backup`, no drill** (finding 4). |
-| **Search** | `/var/lib/openvibe-search/search.db` (`deploy/systemd/openvibe-search.service:28`); `SEARCH_DB_PATH` (`server/config.js:49`). | **No**: a derived FTS5 index, rebuildable from its owners (sources, wiki, blog, news, reviews) | SQLite FTS5. `docs/adr-engine.md` plans PostgreSQL FTS "once a PostgreSQL instance exists" (`server/engine/fts5.js:16` keeps the engine behind an interface). | `/api/ready` (`server/app.js:30-43`): `SELECT 1` and key loaded. | **Yes** | Drill passed 05:23. |
-| **Sources** | `/var/lib/openvibe-sources/sources.db` (`deploy/systemd/openvibe-sources.service:29`); `SOURCES_DB_PATH` (`server/config.js:41`). | Yes: source registry, items, revisions, fetch runs | SQLite, the reviewed state. | `/api/ready` (`server/app.js:25+`): `SELECT 1`, key, and the worker when enabled. | **Yes** | Drill passed 05:23. |
-| **Wiki** | `/var/lib/openvibe-wiki/wiki.db` (`deploy/systemd/openvibe-wiki.service:29`); `WIKI_DB_PATH` (`server/config.js:27`). | Yes: spaces, pages, revisions | SQLite, the reviewed state. | `/api/ready` (`server/app.js:63-75`). Required: `db` (`wiki_pages` exists). Optional: jwks, events_relay, community, sources, media. | **Yes** | Drill passed 05:23. |
-| **Blog** | `/var/lib/openvibe-blog/blog.db` (`deploy/systemd/openvibe-blog.service:28`); `BLOG_DB_PATH` (`server/config.js:32`). | Yes: blogs, posts, revisions | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:19-58`). Required: `db` (charter tables exist). Optional: jwks, events_relay, scheduler. | **Yes** | Drill passed 05:23. |
-| **News** | `/var/lib/openvibe-news/news.db` (`deploy/systemd/openvibe-news.service:29`); `NEWS_DB_PATH` (`server/config.js:32`). | Yes: topics, stories, source items | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:21-65`). Required: `db` (charter tables). Optional: jwks, events_relay, events_webhook, sources_pull. | **Yes** | Drill passed 05:23. |
-| **Reviews** | `/var/lib/openvibe-reviews/reviews.db` (`deploy/systemd/openvibe-reviews.service:30`); `REVIEWS_DB_PATH` (`server/config.js:28`). | Yes: entities, signals, summaries | SQLite, the reviewed state. | `/api/ready` (`server/app.js:82-99`). Required: `db` (three review tables). Optional: jwks, sources, events relay and webhook, community, editors. | **Yes** | Drill passed 05:23. |
-| **Deals** | `/var/lib/openvibe-deals/deals.db` (`deploy/systemd/openvibe-deals.service:28`); `DEALS_DB_PATH` (`server/config.js:34`). | Yes: offers, products, votes | SQLite, the reviewed state. | `/api/ready` (`server/observability.js`). Required: `db` (tables exist, `:26-28`). Optional: jwks, events_relay, sources_import, worker, abuse_keys. | **Yes** | Drill passed 05:23. |
-| **Coupons** | `/var/lib/openvibe-coupons/coupons.db` (`deploy/systemd/openvibe-coupons.service:29`); `COUPONS_DB_PATH` (`server/config.js:32`). | Yes: coupons, merchants, domains | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:24-26`). Required: `db`. Optional: jwks, events_relay, sweep, sources_import, reporter_key. | **Yes** | Drill passed 05:23. |
-| **Trade** | `/var/lib/openvibe-trade/trade.db` (`deploy/systemd/openvibe-trade.service:29`); `TRADE_DB_PATH` (`server/config.js:32`). | Yes: instruments, observations, source documents (informational, ADR-025) | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:23-25`, mounted `server/app.js:121`). Required: `db`. Optional: jwks, events_relay, sources_sync, freshness. | **Yes** | Drill passed 05:23. |
-| **Codes** | `/var/lib/openvibe-codes/codes.db` (`deploy/systemd/openvibe-codes.service:29`); `CODES_DB_PATH` (`server/config.js:37`). | Yes: manifests, releases, trust | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:24-32`). Required: `db`, `docs`. Optional: jwks, network, oauth_client, events_relay. | **Yes** | Drill passed 05:23. |
-| **Host** | Stage B: `/var/lib/openvibe-host-api/host.db` and `HOST_STORAGE_DIR` blobs (`deploy/systemd/openvibe-host.service:37-38`; `server/config.js:47-49`). Stage A (`ovhost`) keeps JSONL logs in `/var/lib/openvibe-host` and has no database. | Yes: tenant projects, sites, deploys, domains | SQLite, the reviewed state. | `/api/ready` (`server/observability.js:22-29`). Required: `db`, `storage` (`blobs.writable()`, `server/storage.js:98-102`). Optional: jwks, events_relay, domain_checks. | **Yes** (database and blob directory) | Drill passed 05:23 (`host.db`, counts only). `README.md` and `STATUS.json` say Stage B is **not deployed**, while the drill counted a production `host.db`. Whether `openvibe-host.service` is running is unknown: check `systemctl status openvibe-host`. |
-
-## Deployed-state notes
-
-- **Loopback only**, not on their domains yet (operator notes): Tips, VIP, Search, Sources, News, Reviews,
- Codes, AI and OpenRestream's API. **Public**: Wiki and Blog (since ~03:00 UTC). **Staff console public**:
- Billing. Deals, Coupons and Trade passed drills against production, so they run on the host. Whether
- they are public is unknown: check nginx `sites-enabled`. Sites still has placeholders for
- openvibe.deals, .coupons and .trade.
-- Several `STATUS.json` files are stale against the operator notes and the drill log. Chat, Events, OpenRestream
- and AI still say `"deployed": false`, and so do Tips, Search, Sources, News, Reviews, Deals, Coupons,
- Trade and Codes. `STATUS.json` stays each service's own record to fix. This page does not change it.
-
-## Re-checking
-
-```
-sudo ovhost status <service> # unit, port, readiness as ovhost sees it
-curl -s 127.0.0.1:<port>/api/ready | jq .checks # the named checks and their state
-sudo ovhost backup <service> && sudo ovhost drill <service>
-```
-
-To re-verify a row, read the handler cited in the row. A readiness change is a code change in that
-service, and this page is updated in the same week.
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
{
{
"_about": "Example ovhost inventory for the single OpenVibe host (openvibe-oregon). Copy to /etc/openvibe/host.json (root:root, 0640) and correct anything that differs on the machine; the real file is never committed. Paths, units, ports and env file locations come from OpenVibe.Network docs/roadmap-baseline/data/services.json and prod-snapshot.json (22 Sep 2026) and each repository's deploy/ directory. env.required lists NAMES only.",
"_about": "Example ovhost inventory for the single OpenVibe host (openvibe-oregon). Copy to /etc/openvibe/host.json (root:root, 0640) and correct anything that differs on the machine; the real file is never committed. Paths, units, ports and env file locations come from OpenVibe.Network docs/roadmap-baseline/data/services.json and prod-snapshot.json (22 Sep 2026) and each repository's deploy/ directory. env.required lists NAMES only.",
- "_databases": "Databases (lib/inventory.js): each services.<id>.databases[] item is one of two engines. SQLite (the default) is a file the service user's worker copies: { \"name\": \"live\", \"path\": \"/opt/openvibe.live/data/live.db\" }. PostgreSQL is a database in the cluster: { \"name\": \"trade\", \"engine\": \"postgresql\", \"database\": \"ov_trade\" } \u2014 the name must match ov_[a-z0-9_]+, and there is no path (ovhost reaches it as the postgres OS user over peer auth, and pgBackRest, not ovhost, copies it). tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, bot, host, events, community, billing, media, openre and games are PostgreSQL; network, chat, tools and live are still SQLite. A drill block follows the engine: a PostgreSQL database maps to { \"url\": \"DATABASE_URL\", \"directUrl\": \"DATABASE_DIRECT_URL\" } and the drill creates a scratch database and role to restore into.",
+ "_databases": "Every databases entry names a PostgreSQL database in the cluster: { \"name\": \"trade\", \"engine\": \"postgresql\", \"database\": \"ov_trade\" }. A drill maps it to { \"url\": \"DATABASE_URL\", \"directUrl\": \"DATABASE_DIRECT_URL\" } and restores a scratch database and role.",
"host": "openvibe-oregon",
"host": "openvibe-oregon",
"stateDir": "/var/lib/openvibe-host",
"stateDir": "/var/lib/openvibe-host",
"backupDir": "/var/backups/openvibe",
"backupDir": "/var/backups/openvibe",
@@ -39,7 +39,8 @@
@@ -39,7 +39,8 @@
"BASE_URL",
"BASE_URL",
"JWT_PRIVATE_KEY",
"JWT_PRIVATE_KEY",
"JWT_PUBLIC_KEY",
"JWT_PUBLIC_KEY",
- "DB_PATH"
+ "DATABASE_URL",
+ "DATABASE_DIRECT_URL"
]
]
},
},
"port": 4000,
"port": 4000,
@@ -55,7 +56,8 @@
@@ -55,7 +56,8 @@
"databases": [
"databases": [
{
{
"name": "network",
"name": "network",
- "path": "/opt/openvibe.network/data/network.db"
+ "engine": "postgresql",
+ "database": "ov_network"
}
}
],
],
"nginx": {
"nginx": {
@@ -67,7 +69,10 @@
@@ -67,7 +69,10 @@
"port": 14000,
"port": 14000,
"ready": "/api/ready",
"ready": "/api/ready",
"databases": {
"databases": {
- "network": "DB_PATH"
+ "network": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -75,7 +80,8 @@
@@ -75,7 +80,8 @@
"OV_EVENTS_INTERNAL_URL": "",
"OV_EVENTS_INTERNAL_URL": "",
"INTERNAL_API_KEY": "",
"INTERNAL_API_KEY": "",
"OV_TOOLS_INTERNAL_URL": "http://127.0.0.1:9",
"OV_TOOLS_INTERNAL_URL": "http://127.0.0.1:9",
- "AVATAR_PATH": "{tmp}/avatars"
+ "AVATAR_PATH": "{tmp}/avatars",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"dirs": [
"dirs": [
"{tmp}/avatars"
"{tmp}/avatars"
@@ -105,8 +111,7 @@
@@ -105,8 +111,7 @@
"db": "network",
"db": "network",
"table": "coin_transactions"
"table": "coin_transactions"
}
}
- ],
- "_notes": "OV_EVENTS_INTERNAL_URL empty stops the dev_audit relay; INTERNAL_API_KEY empty falls back to the placeholder key, which makes the chrome site-copy job (paid LLM call through Live), avatar import and Live analytics calls return early. No switch: the ecosystem poller GETs every service readiness and release.json on loopback every 60 s (read-only). Discord bot login and the Resend email queue are gated by settings in the copy and reach only remote hosts, which the sandbox blocks. JWT_PRIVATE_KEY/JWT_PUBLIC_KEY come from the production env file and are only read."
+ ]
}
}
},
},
"live": {
"live": {
@@ -145,7 +150,9 @@
@@ -145,7 +150,9 @@
"OV_OAUTH_CLIENT_SECRET",
"OV_OAUTH_CLIENT_SECRET",
"MEDIA_URL",
"MEDIA_URL",
"MEDIA_API_KEY",
"MEDIA_API_KEY",
- "MEDIA_WEBHOOK_SECRET"
+ "MEDIA_WEBHOOK_SECRET",
+ "DATABASE_URL",
+ "DATABASE_DIRECT_URL"
]
]
},
},
"port": 3000,
"port": 3000,
@@ -183,11 +190,8 @@
@@ -183,11 +190,8 @@
"databases": [
"databases": [
{
{
"name": "live",
"name": "live",
- "path": "/opt/openvibe.live/data/live.db"
- },
- {
- "name": "rs-companion",
- "path": "/opt/openvibe.live/data/rs-companion.db"
+ "engine": "postgresql",
+ "database": "ov_live"
}
}
],
],
"_databasesNote": "analytics.db (1.7 GB, append-only pageviews) is left out of routine backups on purpose; add it here to include it.",
"_databasesNote": "analytics.db (1.7 GB, append-only pageviews) is left out of routine backups on purpose; add it here to include it.",
@@ -208,10 +212,12 @@
@@ -208,10 +212,12 @@
"maxBody": "3m"
"maxBody": "3m"
},
},
"drill": {
"drill": {
- "_notes": "Needs OpenVibe.Live 8a58aea or later (LIVE_DRILL, server/drill.js; requires checks the deployed checkout). LIVE_DRILL=1 refuses to start unless DB_PATH and DATA_DIR are outside the checkout and /opt/openvibe.live, HOST is loopback and PORT is not 3000, and never takes over the systemd socket. It then serves HTTP only: no RTMP/SRT/WHIP/SFU/JSMPEG listener, no WebSocket (upgrades get 403), no TURN credential, no jobs loop, restream or relay resume, AI job, Media reconciler, Events outbox, chat bridge, identity sync, deploy notice, star job or registry refresh. Every method but GET/HEAD/OPTIONS answers 403; net, fetch, child_process (git only), dgram and listen are guarded, so every other service looks down to it. All its files go to DATA_DIR ({tmp}/data: a fresh analytics.db and empty data directories; the per-location *_PATH values in live.env are ignored in a drill). rs-companion.db is not restored: the server never opens it. Compared: DB-only public reads; ignore lists drop what production changes in memory or every few seconds while someone is live (viewer counts, heartbeats, live thumbnails, recording state) and the one Media lookup (recently-online vod_thumbnail). /api/streams/channel/japaneseoldguy/live is the star channel: any channel that exists works. /release.json is not compared: production computes it at its own boot, so a public/-only deploy without a restart would make it differ. EVENTS_URL, the client secret and MEDIA_URL are blanked as a second switch. Not run on the host yet.",
"port": 13000,
"port": 13000,
"databases": {
"databases": {
- "live": "DB_PATH"
+ "live": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -220,7 +226,8 @@
@@ -220,7 +226,8 @@
"DATA_DIR": "{tmp}/data",
"DATA_DIR": "{tmp}/data",
"EVENTS_URL": "",
"EVENTS_URL": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
- "MEDIA_URL": "http://127.0.0.1:9"
+ "MEDIA_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"dirs": [
"dirs": [
"{tmp}/data"
"{tmp}/data"
@@ -436,7 +443,9 @@
@@ -436,7 +443,9 @@
"OV_NETWORK_URL",
"OV_NETWORK_URL",
"OV_OAUTH_CLIENT_ID",
"OV_OAUTH_CLIENT_ID",
"OV_OAUTH_CLIENT_SECRET",
"OV_OAUTH_CLIENT_SECRET",
- "MEDIA_URL"
+ "MEDIA_URL",
+ "DATABASE_URL",
+ "DATABASE_DIRECT_URL"
]
]
},
},
"metrics": {
"metrics": {
@@ -515,59 +524,19 @@
@@ -515,59 +524,19 @@
},
},
"databases": [
"databases": [
{
{
- "name": "tools-audio-analytics",
- "path": "/opt/openvibe.tools/apps/audio/data/analytics.db"
- },
- {
- "name": "tools-docs-analytics",
- "path": "/opt/openvibe.tools/apps/docs/data/analytics.db"
- },
- {
- "name": "tools-food-analytics",
- "path": "/opt/openvibe.tools/apps/food/data/analytics.db"
- },
- {
- "name": "tools-img-analytics",
- "path": "/opt/openvibe.tools/apps/img/data/analytics.db"
- },
- {
- "name": "tools-maps-analytics",
- "path": "/opt/openvibe.tools/apps/maps/data/analytics.db"
- },
- {
- "name": "tools-text-analytics",
- "path": "/opt/openvibe.tools/apps/text/data/analytics.db"
- },
- {
- "name": "tools-yt-analytics",
- "path": "/opt/openvibe.tools/apps/yt/data/analytics.db"
- },
- {
- "name": "tools-audio-jobs",
- "path": "/opt/openvibe.tools/apps/audio/data/jobs.db"
- },
- {
- "name": "tools-docs-jobs",
- "path": "/opt/openvibe.tools/apps/docs/data/jobs.db"
- },
- {
- "name": "tools-img-jobs",
- "path": "/opt/openvibe.tools/apps/img/data/jobs.db"
+ "name": "tools",
+ "engine": "postgresql",
+ "database": "ov_tools"
}
}
],
],
"drill": {
"drill": {
- "_notes": "Drills ONE app: docs (both database kinds; spawns no external program; every path has an env override). Its unit's ExecStart/WorkingDirectory come from drill.unit, and the comparison runs against docs' own port. DATA_DIR takes a directory, so both copies land in {tmp}/data under their production names. TOOLS_JOB_RESULTS=local keeps the job pruner from deleting production Media objects for expired jobs in the copy; EVENTS_PUBLISH=off keeps the copy's pending outbox rows from being republished; TOOLS_JOBS_CONCURRENCY=0 runs no job. What still runs, on the copy only: job recovery and pruning, analytics aggregation, upload retention (all under {tmp}). yt, maps and food open data/analytics.db relative to the checkout: drill one with drill.unit + a {tmp} data dir + bind over apps/<app>/data (plus YT_PROXY= for yt) until Tools adds DATA_DIR there. Not run on the host yet.",
"port": 14016,
"port": 14016,
"unit": "openvibe-tools-docs.service",
"unit": "openvibe-tools-docs.service",
"productionPort": 4016,
"productionPort": 4016,
"databases": {
"databases": {
- "tools-docs-analytics": {
- "env": "DATA_DIR",
- "dir": true
- },
- "tools-docs-jobs": {
- "env": "DATA_DIR",
- "dir": true
+ "tools": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
}
}
},
},
"env": {
"env": {
@@ -580,7 +549,8 @@
@@ -580,7 +549,8 @@
"EVENTS_PUBLISH": "off",
"EVENTS_PUBLISH": "off",
"EVENTS_URL": "",
"EVENTS_URL": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
- "MEDIA_URL": "http://127.0.0.1:9"
+ "MEDIA_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"dirs": [
"dirs": [
"{tmp}/uploads",
"{tmp}/uploads",
@@ -611,31 +581,16 @@
@@ -611,31 +581,16 @@
],
],
"counts": [
"counts": [
{
{
- "db": "tools-docs-jobs",
+ "db": "tools",
"table": "tool_jobs"
"table": "tool_jobs"
}
}
]
]
},
},
"protected": {
"protected": {
- "kind": "sum",
- "label": "running tool jobs",
- "probes": [
- {
- "kind": "sqlite-count",
- "db": "/opt/openvibe.tools/apps/img/data/jobs.db",
- "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'"
- },
- {
- "kind": "sqlite-count",
- "db": "/opt/openvibe.tools/apps/audio/data/jobs.db",
- "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'"
- },
- {
- "kind": "sqlite-count",
- "db": "/opt/openvibe.tools/apps/docs/data/jobs.db",
- "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'"
- }
- ]
+ "kind": "postgresql-count",
+ "database": "ov_tools",
+ "sql": "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'",
+ "label": "running tool jobs"
},
},
"drain": {
"drain": {
"policy": "report"
"policy": "report"
@@ -676,7 +631,6 @@
@@ -676,7 +631,6 @@
"database": "ov_community"
"database": "ov_community"
}
}
],
],
- "_databasesNote": "Community is one PostgreSQL database (ov_community) now. Before the switch it declared two SQLite paths (the repo unit's /var/lib/openvibe-community/community.db and the /opt/openvibe.community/data/community.db the 22 Sep production snapshot found); neither is declared any more, and the frozen files are for `ovhost archive push`.",
"nginx": {
"nginx": {
"vhost": "openvibe.community.conf",
"vhost": "openvibe.community.conf",
"repoVhost": "deploy/nginx/openvibe.community.conf",
"repoVhost": "deploy/nginx/openvibe.community.conf",
@@ -721,8 +675,7 @@
@@ -721,8 +675,7 @@
"db": "community",
"db": "community",
"table": "posts"
"table": "posts"
}
}
- ],
- "_notes": "Community is one PostgreSQL database (ov_community) now: the second, legacy SQLite path it used to declare is gone. An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."
+ ]
}
}
},
},
"events": {
"events": {
@@ -1145,13 +1098,17 @@
@@ -1145,13 +1098,17 @@
"databases": [
"databases": [
{
{
"name": "chat",
"name": "chat",
- "path": "/var/lib/openvibe-chat/chat.db"
+ "engine": "postgresql",
+ "database": "ov_chat"
}
}
],
],
"drill": {
"drill": {
"port": 14400,
"port": 14400,
"databases": {
"databases": {
- "chat": "CHAT_DB_PATH"
+ "chat": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1161,7 +1118,8 @@
@@ -1161,7 +1118,8 @@
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9",
"OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9",
"CHAT_CACHE_DIR": "{tmp}/cache",
"CHAT_CACHE_DIR": "{tmp}/cache",
- "SOUNDS_PATH": "{tmp}/sounds"
+ "SOUNDS_PATH": "{tmp}/sounds",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"dirs": [
"dirs": [
"{tmp}/cache",
"{tmp}/cache",
@@ -1187,8 +1145,7 @@
@@ -1187,8 +1145,7 @@
"db": "chat",
"db": "chat",
"table": "moderation_actions"
"table": "moderation_actions"
}
}
- ],
- "_notes": "An empty client secret and a dead Live URL stop the 2 s Live projection sync and the effects POSTed to Live. History endpoints are not compared: cosmetics come from Live (cut off here) and expired messages are swept at boot, so chat_messages is not counted either."
+ ]
},
},
"protected": {
"protected": {
"kind": "http-json-count",
"kind": "http-json-count",
@@ -2056,7 +2013,7 @@
@@ -2056,7 +2013,7 @@
}
}
],
],
"drill": {
"drill": {
- "_notes": "Every provider key, base URL and the HTTP seam are blanked and AI_ENABLED=0, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and EVENTS_URL are blanked (no service calls, no events); WHISPER_BIN is blanked (no transcription). Boot marks interrupted runs failed in the restored copy only. Counts cover configuration and citations only: runs, requests and usage grow by the minute, so they are checked by integrity_check, not by an equal count. Passed on the host 2026-09-24 05:40 UTC.",
+ "_notes": "Every provider key, base URL and the HTTP seam are blanked and AI_ENABLED=0, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and EVENTS_URL are blanked (no service calls, no events); WHISPER_BIN is blanked (no transcription). Boot marks interrupted runs failed in the restored copy only. Counts cover configuration and citations only: runs, requests and usage grow by the minute, so the dump is checked by pg_restore --list; these rows are not compared by count. Passed on the host 2026-09-24 05:40 UTC.",
"port": 14700,
"port": 14700,
"databases": {
"databases": {
"ai": {
"ai": {
@@ -2119,7 +2076,7 @@
@@ -2119,7 +2076,7 @@
}
}
},
},
"bot": {
"bot": {
- "_note": "Deployed; migrations 0001 and 0002 applied. Bot owns openvibe.bot.conf: deploy bot installs its proxy vhost after Sites skips that filename. BOT_WHIP_BASE (in /etc/openvibe/bot.env) is the switch that points Bot's video at https://ingest.openre.stream/whip. Use ovhost deploy bot --restart if the checkout is already current so it installs and enables the service unit. Keep Sites on its ovhost deploy path (its legacy fallback still installs every vhost).",
+ "_note": "Deployed; migrations 0001 and 0002 applied. Bot owns openvibe.bot.conf: deploy bot installs its proxy vhost after Sites skips that filename. BOT_WHIP_BASE (in /etc/openvibe/bot.env) points Bot's video at https://ingest.openre.stream/whip. Use ovhost deploy bot --restart if the checkout is already current so it installs and enables the service unit. Keep Sites on its ovhost deploy path so it installs every vhost.",
"repo": "/opt/openvibe.bot",
"repo": "/opt/openvibe.bot",
"installUnits": true,
"installUnits": true,
"enableUnits": true,
"enableUnits": true,
@@ -159,7 +159,7 @@ const SCENARIOS = [
@@ -159,7 +159,7 @@ const SCENARIOS = [
source: 'the test asserts host.restarts() is [] and the record says restarted: false',
source: 'the test asserts host.restarts() is [] and the record says restarted: false',
},
},
{
{
- id: '1b', what: 'the same through Live\'s own deploy script on a simulated host', repo: 'live', files: ['test/deploy-sim.test.js'], timeoutMs: 300000,
+ id: '1b', what: 'the same through Live\'s deploy wrapper on a simulated host', repo: 'live', files: ['test/deploy-sim.test.js'], timeoutMs: 300000,
expect: [ok('public/-only change: current switches, process is NOT restarted')],
expect: [ok('public/-only change: current switches, process is NOT restarted')],
budget: { name: 'Live restarts', max: 0 }, measure: asserted(), category: 'affected sessions',
budget: { name: 'Live restarts', max: 0 }, measure: asserted(), category: 'affected sessions',
source: 'the test compares the process id before and after the switch',
source: 'the test compares the process id before and after the switch',
@@ -7,8 +7,7 @@
@@ -7,8 +7,7 @@
* poll.
* poll.
*
*
* ovhost deploy|rollback <service> announces after a deploy or rollback that went live
* ovhost deploy|rollback <service> announces after a deploy or rollback that went live
- * ovhost announce <service> for services deployed by their own scripts (Live, Tools, Sites,
- * OpenRestream, Games)
+ * ovhost announce <service> manually announce a service's currently served release
*
*
* Best effort: it never fails or holds up a deploy. The token request and the publish each have a
* Best effort: it never fails or holds up a deploy. The token request and the publish each have a
* short timeout and the whole announcement a budget of a few seconds; a failure is logged and the
* short timeout and the whole announcement a budget of a few seconds; a failure is logged and the
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
'use strict';
'use strict';
/**
/**
- * ovhost archive (roadmap WS-S task 6, hazard H14): dead and backup database files on the host are
+ * ovhost archive: retired data files on the host are
* encrypted and copied off-host before anyone deletes them. This never deletes a local file: that
* encrypted and copied off-host before anyone deletes them. This never deletes a local file: that
* waits for the owner, with the archive's manifest in hand.
* waits for the owner, with the archive's manifest in hand.
*
*
@@ -42,11 +42,6 @@ async function refusal(exec, inv, file) {
@@ -42,11 +42,6 @@ async function refusal(exec, inv, file) {
if (typeof file !== 'string' || !path.isAbsolute(file) || path.normalize(file) !== file) return 'not a normalised absolute path';
if (typeof file !== 'string' || !path.isAbsolute(file) || path.normalize(file) !== file) return 'not a normalised absolute path';
if (!ALLOWED_ROOTS.some((r) => file.startsWith(r))) return `outside ${ALLOWED_ROOTS.join(', ')}`;
if (!ALLOWED_ROOTS.some((r) => file.startsWith(r))) return `outside ${ALLOWED_ROOTS.join(', ')}`;
if (!NAME_RE.test(objectName(file))) return 'the path has characters an object name cannot carry';
if (!NAME_RE.test(objectName(file))) return 'the path has characters an object name cannot carry';
- // A live SQLite database is a file the inventory declares, and archiving it is refused. A
- // PostgreSQL database has no file (its own entry carries no path), so the C-89 switch — archive
- // the frozen SQLite file, then delete it — is no longer blocked by the inventory that moved on.
- const declared = Object.values(inv.services || {}).flatMap((s) => (s.databases || []).filter((d) => d.engine !== 'postgresql').map((d) => d.path));
- if (declared.includes(file)) return 'a database the inventory declares (a live database)';
const st = await exec.stat(file);
const st = await exec.stat(file);
if (!st) return 'no such file';
if (!st) return 'no such file';
if (st.isSymlink || !st.isFile) return 'not a regular file';
if (st.isSymlink || !st.isFile) return 'not a regular file';
@@ -10,7 +10,7 @@ const { SocketGuardError } = require('./systemd');
@@ -10,7 +10,7 @@ const { SocketGuardError } = require('./systemd');
const { runBrowserCheck } = require('./browser-check-hook');
const { runBrowserCheck } = require('./browser-check-hook');
const VERSION = require('../package.json').version;
const VERSION = require('../package.json').version;
-// What deploy wrappers probe (`ovhost capabilities <service>`): bump when the deploy contract changes.
+// Keep the capabilities command for deployed wrappers that still call it.
const DEPLOY_API = 1; // 1: strategies, capabilities, deploy --prepare-only
const DEPLOY_API = 1; // 1: strategies, capabilities, deploy --prepare-only
const USAGE = `ovhost — OpenVibe.Host operator plane (Stage A)
const USAGE = `ovhost — OpenVibe.Host operator plane (Stage A)
@@ -54,12 +54,6 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
@@ -54,12 +54,6 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
data provision <service> [--dry-run] run roles/data/add-service.sh from the ovhost install: the
data provision <service> [--dry-run] run roles/data/add-service.sh from the ovhost install: the
service's PostgreSQL database, roles and Valkey user
service's PostgreSQL database, roles and Valkey user
(idempotent; prints names only, never values)
(idempotent; prints names only, never values)
- data switch <service> [--sqlite <file>] [--dry-run]
- run roles/data/switch-service.sh from the ovhost install: stop,
- back up the SQLite file, import and verify, deploy; SWITCH_UNITS/
- SWITCH_DIR come from the inventory, and the script's exit code is
- passed through. Needs the service's scripts/migrate-to-postgres.js
- (docs/data-provisioning.md)
version | --version ovhost's version
version | --version ovhost's version
announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run]
announce <service> [--release <id>] [--commit <sha>] [--origin <url>] [--force] [--dry-run]
publish host.release.published to OpenVibe.Events (public), so open
publish host.release.published to OpenVibe.Events (public), so open
@@ -142,7 +136,7 @@ self-update: 0 updated, up to date or dry run · 1 precondition (not root, dirty
@@ -142,7 +136,7 @@ self-update: 0 updated, up to date or dry run · 1 precondition (not root, dirty
backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
backup --all / offsite: 0 everything succeeded · 1 precondition (config, key, lock) · 2 something failed
`;
`;
-const FLAGS_WITH_VALUE = new Set(['prometheus', 'sites', 'token-file', 'ttl', 'title', 'services', 'severity', 'message', 'state', 'from', 'until', 'reason', 'incident', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note', 'tail', 'sqlite']);
+const FLAGS_WITH_VALUE = new Set(['prometheus', 'sites', 'token-file', 'ttl', 'title', 'services', 'severity', 'message', 'state', 'from', 'until', 'reason', 'incident', 'inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup', 'run', 'keep-daily', 'keep-weekly', 'from-host', 'backup-env', 'release', 'commit', 'origin', 'events-env', 'note', 'tail']);
function parseArgs(argv) {
function parseArgs(argv) {
const pos = [];
const pos = [];
@@ -214,8 +208,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -214,8 +208,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
const ctx = { exec, inv, log };
const ctx = { exec, inv, log };
switch (cmd) {
switch (cmd) {
case 'capabilities': {
case 'capabilities': {
- // For the per-repository deploy wrappers (deploy/scripts/deploy.sh): they use ovhost only when
- // it knows the strategy they need and the inventory manages the service with it.
+ // Older per-repository deploy wrappers still query this command before invoking ovhost.
const lines = [`ovhost=${VERSION}`, `deploy-api=${DEPLOY_API}`, `strategies=${lib.inventory.STRATEGIES.join(',')}`, 'commands=plan,deploy,rollback,announce,capabilities', 'deploy-flags=--wait-idle,--force,--restart,--to,--install-units,--ready-timeout,--prepare-only,--no-announce'];
const lines = [`ovhost=${VERSION}`, `deploy-api=${DEPLOY_API}`, `strategies=${lib.inventory.STRATEGIES.join(',')}`, 'commands=plan,deploy,rollback,announce,capabilities', 'deploy-flags=--wait-idle,--force,--restart,--to,--install-units,--ready-timeout,--prepare-only,--no-announce'];
const row = { ovhost: VERSION, deployApi: DEPLOY_API, strategies: lib.inventory.STRATEGIES };
const row = { ovhost: VERSION, deployApi: DEPLOY_API, strategies: lib.inventory.STRATEGIES };
if (pos[1]) {
if (pos[1]) {
@@ -255,11 +248,11 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -255,11 +248,11 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
// The data role's own scripts, run from ovhost's install (ADR-035): the agent pipeline
// The data role's own scripts, run from ovhost's install (ADR-035): the agent pipeline
// reaches ovhost, not the shell. Output is redacted and the script's exit code passes through.
// reaches ovhost, not the shell. Output is redacted and the script's exit code passes through.
const sub = pos[1];
const sub = pos[1];
- if (sub !== 'provision' && sub !== 'switch') throw new UsageError('usage: ovhost data provision <service> [--dry-run] | ovhost data switch <service> [--sqlite <file>] [--dry-run]');
+ if (sub !== 'provision') throw new UsageError('usage: ovhost data provision <service> [--dry-run]');
const id = pos[2];
const id = pos[2];
if (!id) throw new UsageError(`data ${sub} needs a <service>`);
if (!id) throw new UsageError(`data ${sub} needs a <service>`);
- const o = { selfPath: selfPath || process.argv[1], dryRun: !!opts['dry-run'], sqlite: opts.sqlite };
- const r = sub === 'provision' ? await lib.data.provision(ctx, id, o) : await lib.data.switch(ctx, id, o);
+ const o = { selfPath: selfPath || process.argv[1], dryRun: !!opts['dry-run'] };
+ const r = await lib.data.provision(ctx, id, o);
if (json) { emit(r); return r.code === null ? 0 : r.code; }
if (json) { emit(r); return r.code === null ? 0 : r.code; }
if (r.dryRun) { out(`would run: ${r.command}`); return 0; }
if (r.dryRun) { out(`would run: ${r.command}`); return 0; }
for (const l of String(r.stdout).split('\n')) if (l) out(l);
for (const l of String(r.stdout).split('\n')) if (l) out(l);
@@ -515,7 +508,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -515,7 +508,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
for (const s of r.services) {
for (const s of r.services) {
const copied = s.files.filter((f) => f.dest && !f.error);
const copied = s.files.filter((f) => f.dest && !f.error);
const size = copied.reduce((a, f) => a + (f.bytes || 0), 0);
const size = copied.reduce((a, f) => a + (f.bytes || 0), 0);
- const detail = s.status === 'failed' ? s.error : s.status === 'skipped' ? 'no database found' : s.status === 'verified' ? 'pgBackRest verified (no dump due)' : `${copied.length} file(s), ${bytes(size)}`;
+ const detail = s.status === 'failed' ? s.error : s.status === 'verified' ? 'pgBackRest verified (no dump due)' : `${copied.length} file(s), ${bytes(size)}`;
out(`${s.status.toUpperCase().padEnd(8)} ${s.service.padEnd(10)} ${detail}${s.pruned.length ? `; pruned ${s.pruned.length}` : ''}`);
out(`${s.status.toUpperCase().padEnd(8)} ${s.service.padEnd(10)} ${detail}${s.pruned.length ? `; pruned ${s.pruned.length}` : ''}`);
}
}
if (r.offsite) {
if (r.offsite) {
@@ -723,7 +716,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -723,7 +716,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
const cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });
const cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });
const r = await lib.offsite.restoreDownload(ctx, id, pos[2], { cfg, client: s3(cfg), out: opts.out, host: opts['from-host'] });
const r = await lib.offsite.restoreDownload(ctx, id, pos[2], { cfg, client: s3(cfg), out: opts.out, host: opts['from-host'] });
if (json) { emit(r); return 0; }
if (json) { emit(r); return 0; }
- for (const f of r.files) out(`${f.file} ${bytes(f.bytes)} sha256 ${f.sha256.slice(0, 16)}… quick_check ${f.quickCheck} (was ${f.source})`);
+ for (const f of r.files) out(`${f.file} ${bytes(f.bytes)} sha256 ${f.sha256.slice(0, 16)}… check ${f.check} (was ${f.source})`);
out(`restored ${r.service} from ${r.host}/${r.run} into ${r.dir} (root-only). Nothing live was touched;`);
out(`restored ${r.service} from ${r.host}/${r.run} into ${r.dir} (root-only). Nothing live was touched;`);
out('see docs/backups.md "Restoring from an off-host copy" to put a copy in place.');
out('see docs/backups.md "Restoring from an off-host copy" to put a copy in place.');
return 0;
return 0;
@@ -1,41 +1,30 @@
@@ -1,41 +1,30 @@
'use strict';
'use strict';
/**
/**
- * ovhost backup <service>: an online copy of every database the inventory declares for the service,
- * and a tar archive of every content-addressed object directory it declares, into
- * <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in
- * <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root
- * never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.
- * PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups
- * and its WAL (roles/data/provision.sh), so a run that finds a PostgreSQL database first VERIFIES
- * that layer (dbengine.verifyPgBackRest: a stanza that is ok and whose newest backup is under 26 h
- * old; dbengine.walFailedCount: pg_stat_archiver.failed_count has not risen since the last run) and
- * only then, on a slower cadence (LOGICAL_DUMP_INTERVAL_DAYS, or --logical), takes a per-service
- * pg_dump -Fc archive — the artifact the off-site code uploads and a drill can restore.
+ * ovhost backup <service>: every database the inventory declares for the service is in the PostgreSQL
+ * cluster, and every content-addressed object directory it declares is archived, into
+ * <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in <stateDir>/backups/<service>.jsonl.
+ * PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups and
+ * its WAL (roles/data/provision.sh), so a run first VERIFIES that layer (dbengine.verifyPgBackRest: a
+ * stanza that is ok and whose newest backup is under 26 h old; dbengine.walFailedCount:
+ * pg_stat_archiver.failed_count has not risen since the last run) and only then, on a slower cadence
+ * (LOGICAL_DUMP_INTERVAL_DAYS, or --logical), takes a per-service pg_dump -Fc archive: the artifact the
+ * off-site code uploads and a drill restores.
*
*
- * A PostgreSQL service's record always carries `verified` (type + stop + walOk when it passed, an
- * error when it did not), and a run with nothing to dump is verification-only: ok: true, files: [],
- * and no directory at all. A failed verification (or a logical dump that failed) is recorded
- * (ok: false) and then throws a BackupError carrying the record, so backup --all reports the service
- * failed — never skipped.
+ * A record always carries `verified` (type + stop + walOk when it passed, an error when it did not), and a
+ * run with nothing to dump or archive is verification-only: ok: true, files: [], and no directory at all.
+ * A failed verification (or a dump or archive that failed) is recorded (ok: false) and then throws a
+ * BackupError carrying the record, so backup --all reports the service failed, never skipped.
*
*
- * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600, with
- * no -wal/-shm beside it. SQLite's worker writes into <backupStagingDir>/<service>-<stamp> (its own,
- * 0700, under a root 0711 parent) as the service user. A service with any PostgreSQL database gets a
- * staging directory owned by postgres instead, because pg_dump runs as that OS user. In both cases
- * root then takes that directory over (lockDown, root:root 0700) BEFORE anything inside it is
- * checked — the service user (or postgres) can no longer reach it while root validates the copies,
- * chowns and chmods them and renames the directory into place, which is what keeps the property.
+ * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600. pg_dump
+ * runs as the postgres OS user, so a service with a database gets a staging directory owned by postgres
+ * under a root 0711 parent (an objects-only service's stays root's). Root takes that directory over
+ * (lockDown, root:root 0700) BEFORE anything inside it is checked, then validates, chowns and chmods the
+ * copies and renames the directory into place.
*
*
- * Every declared database is attempted even when an earlier one fails. A backup with any failed
- * database is still recorded (ok: false, so a drill and retention never treat it as good) and then
- * throws a BackupError carrying the record. A SQLite backup that found none of its databases is
- * recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.
- *
- * ovhost backup --all (backupAll): every service that declares databases or object directories, one
- * after another. A failing service never stops the others. Afterwards each service's backups are
- * pruned to the last 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run
- * is written to <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded
- * (lib/offsite.js).
+ * ovhost backup --all (backupAll): every service that declares databases or object directories, one after
+ * another. A failing service never stops the others. Afterwards each service's backups are pruned to the
+ * last 7 daily and 4 weekly good ones (lib/retention.js), and a JSON summary of the run is written to
+ * <stateDir>/backup-runs/<run>.json. With --offsite the run is then uploaded (lib/offsite.js).
*/
*/
const path = require('path');
const path = require('path');
const { service } = require('../inventory');
const { service } = require('../inventory');
@@ -148,7 +137,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -148,7 +137,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
if (!svc.databases.length && !svc.objects.length) throw new Error(`${id} declares no databases or object directories`);
if (!svc.databases.length && !svc.objects.length) throw new Error(`${id} declares no databases or object directories`);
const postgres = svc.databases.filter((d) => d.engine === 'postgresql');
const postgres = svc.databases.filter((d) => d.engine === 'postgresql');
const hasPostgres = postgres.length > 0;
const hasPostgres = postgres.length > 0;
- const engine = hasPostgres ? 'postgresql' : 'sqlite';
+ const engine = hasPostgres ? 'postgresql' : null;
// (a) Verify the cluster's own backup layer BEFORE copying anything. A failed verification is
// (a) Verify the cluster's own backup layer BEFORE copying anything. A failed verification is
// itself a record (ok: false): backupAll then marks the service failed, never skipped.
// itself a record (ok: false): backupAll then marks the service failed, never skipped.
@@ -171,9 +160,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -171,9 +160,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const at = last === null ? NaN : Date.parse(last);
const at = last === null ? NaN : Date.parse(last);
if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);
if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);
}
}
- // A verification-only run (PostgreSQL, no dump due, no SQLite database, no objects) leaves
+ // A verification-only run (no dump due and no objects) leaves
// nothing on disk.
// nothing on disk.
- if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite') && !svc.objects.length) {
+ if (!dump.length && !svc.objects.length) {
const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };
const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
return record;
return record;
@@ -183,8 +172,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -183,8 +172,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const dir = path.join(inv.backupDir, id, st);
const dir = path.join(inv.backupDir, id, st);
if (await exec.stat(dir)) throw new Error(`${dir} already exists; a backup never overwrites another`);
if (await exec.stat(dir)) throw new Error(`${dir} already exists; a backup never overwrites another`);
// Backups hold user data: every directory under backupDir is root:root 0700 and every copy is
// Backups hold user data: every directory under backupDir is root:root 0700 and every copy is
- // root:root 0600. SQLite's worker runs as the service user (root never opens a service database),
- // so it writes into a staging directory of its own, which root then takes over before the move.
+ // root:root 0600. The postgres OS user writes dumps into staging; root takes over before the move.
await privateDir(exec, inv.backupDir);
await privateDir(exec, inv.backupDir);
await privateDir(exec, path.join(inv.backupDir, id));
await privateDir(exec, path.join(inv.backupDir, id));
await exec.mkdir(inv.backupStagingDir, { owner: 'root', mode: 0o711 });
await exec.mkdir(inv.backupStagingDir, { owner: 'root', mode: 0o711 });
@@ -192,29 +180,14 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -192,29 +180,14 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const stage = path.join(inv.backupStagingDir, `${id}-${st}`);
const stage = path.join(inv.backupStagingDir, `${id}-${st}`);
if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
// pg_dump runs as the postgres OS user, so a service with any PostgreSQL database needs a staging
// pg_dump runs as the postgres OS user, so a service with any PostgreSQL database needs a staging
- // directory postgres can write into (SQLite keeps the service user). The lockDown(stage, 0o700)
+ // directory postgres can write into. The lockDown(stage, 0o700)
// below takes the whole directory back to root:root BEFORE anything inside it is checked, so
// below takes the whole directory back to root:root BEFORE anything inside it is checked, so
// neither the service user nor postgres can reach it while root validates and moves it.
// neither the service user nor postgres can reach it while root validates and moves it.
- await exec.mkdir(stage, { owner: hasPostgres ? dbengine.PG_USER : svc.runAs, mode: 0o700 });
+ await exec.mkdir(stage, { owner: hasPostgres ? dbengine.PG_USER : 'root', mode: 0o700 });
const files = [];
const files = [];
try {
try {
for (const db of svc.databases) {
for (const db of svc.databases) {
const file = dbengine.fileName(db);
const file = dbengine.fileName(db);
- if (db.engine === 'sqlite') {
- if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }
- log(`backing up ${db.path} → ${path.join(dir, file)} (as ${svc.runAs})`);
- const t0 = exec.now();
- try {
- // The worker runs PRAGMA quick_check on the copy and fails unless it answers ok.
- await exec.sqliteBackup(db.path, path.join(stage, file), { as: svc.runAs });
- } catch (err) {
- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), error: err.message });
- log(`FAILED ${db.path}: ${err.message}`);
- continue;
- }
- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), file, check: 'quick_check ok', seconds: Math.round((exec.now() - t0) / 1000) });
- continue;
- }
if (!dump.includes(db)) { log(`${db.database} is covered by pgBackRest today; no logical dump due`); continue; }
if (!dump.includes(db)) { log(`${db.database} is covered by pgBackRest today; no logical dump due`); continue; }
const t0 = exec.now();
const t0 = exec.now();
try {
try {
@@ -227,9 +200,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -227,9 +200,9 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
}
}
files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });
files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });
}
}
- // Take the staging directory back FIRST: while the service user (or postgres) owns it, it could
+ // Take the staging directory back FIRST: while postgres owns it, that user could
// plant a symlink at the path root is about to write the object archive through, and root's
// plant a symlink at the path root is about to write the object archive through, and root's
- // `tar -czf` would follow it. Once root:root 0700, neither can reach it, and everything written
+ // `tar -czf` would follow it. Once root:root 0700, postgres cannot reach it, and everything written
// below is checked before the move.
// below is checked before the move.
await lockDown(exec, stage, 0o700);
await lockDown(exec, stage, 0o700);
// Content-addressed object directories: one tar archive each, created by root AFTER the stage
// Content-addressed object directories: one tar archive each, created by root AFTER the stage
@@ -253,7 +226,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -253,7 +226,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
continue;
continue;
}
}
}
}
- // Clear whatever might sit at the archive's path — a link planted while the service user
+ // Clear whatever might sit at the archive's path — a link planted while postgres
// owned the stage would survive lockDown — before root's tar opens it for writing.
// owned the stage would survive lockDown — before root's tar opens it for writing.
await exec.run('rm', ['-rf', '--', archive], { privileged: true });
await exec.run('rm', ['-rf', '--', archive], { privileged: true });
log(`backing up object directory ${obj.path} → ${path.join(dir, file)}`);
log(`backing up object directory ${obj.path} → ${path.join(dir, file)}`);
@@ -275,7 +248,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -275,7 +248,7 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
await lockDown(exec, p, 0o600);
await lockDown(exec, p, 0o600);
f.bytes = fst.size;
f.bytes = fst.size;
}
}
- // Anything else (a stray -wal/-shm) is not part of the backup.
+ // Anything else in staging is not part of the backup.
const wanted = new Set(files.filter((x) => x.file && !x.error).map((x) => x.file));
const wanted = new Set(files.filter((x) => x.file && !x.error).map((x) => x.file));
for (const e of (await exec.readdir(stage)) || []) {
for (const e of (await exec.readdir(stage)) || []) {
if (!wanted.has(e.name)) await exec.run('rm', ['-rf', '--', path.join(stage, e.name)], { privileged: true });
if (!wanted.has(e.name)) await exec.run('rm', ['-rf', '--', path.join(stage, e.name)], { privileged: true });
@@ -293,9 +266,6 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
@@ -293,9 +266,6 @@ async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
// `file` is kept on each entry: the weekly cadence reads it back to find the last good dump.
// `file` is kept on each entry: the weekly cadence reads it back to find the last good dump.
const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: !failed.length && copied.length > 0, dir, files };
const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: !failed.length && copied.length > 0, dir, files };
if (verified) record.verified = verified;
if (verified) record.verified = verified;
- // Nothing existed to back up (a SQLite service not deployed yet): no directory is left, so an
- // empty backup can never stand in for a real one in retention or a drill. PostgreSQL never empty.
- if (!copied.length && !failed.length && !hasPostgres) record.empty = true;
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
if (failed.length) throw new BackupError(failed.map((f) => `${f.name}: ${f.error}`).join('; '), record);
if (failed.length) throw new BackupError(failed.map((f) => `${f.name}: ${f.error}`).join('; '), record);
return record;
return record;
@@ -357,13 +327,12 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, of
@@ -357,13 +327,12 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, of
summary.services.push(entry);
summary.services.push(entry);
try {
try {
const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}`, logical });
const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}`, logical });
- entry.dir = rec.empty ? null : (rec.dir || null);
+ entry.dir = rec.dir || null;
entry.files = rec.files;
entry.files = rec.files;
if (rec.verified) entry.verified = rec.verified;
if (rec.verified) entry.verified = rec.verified;
// A PostgreSQL service with nothing to dump this run was still verified, not backed
// A PostgreSQL service with nothing to dump this run was still verified, not backed
// up: it has its own status so the summary distinguishes the two.
// up: it has its own status so the summary distinguishes the two.
- if (rec.empty) entry.status = 'skipped';
- else if (rec.engine === 'postgresql' && !rec.dir) entry.status = 'verified';
+ if (rec.engine === 'postgresql' && !rec.dir) entry.status = 'verified';
} catch (err) {
} catch (err) {
entry.status = 'failed';
entry.status = 'failed';
entry.error = err.message;
entry.error = err.message;
@@ -411,7 +380,7 @@ const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
@@ -411,7 +380,7 @@ const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
* openvibe_backup_last_run_timestamp_seconds when the last run finished
* openvibe_backup_last_run_timestamp_seconds when the last run finished
* openvibe_backup_last_run_ok 1 when every service (and the off-site copy) succeeded
* openvibe_backup_last_run_ok 1 when every service (and the off-site copy) succeeded
* openvibe_backup_last_success_timestamp_seconds the last run that succeeded (read back from backup-runs)
* openvibe_backup_last_success_timestamp_seconds the last run that succeeded (read back from backup-runs)
- * openvibe_backup_services{status} services ok, failed, skipped and verified in the last run
+ * openvibe_backup_services{status} services ok, failed and verified in the last run
* openvibe_backup_pg_verified{service} 1 when that service's PostgreSQL verification passed
* openvibe_backup_pg_verified{service} 1 when that service's PostgreSQL verification passed
* openvibe_pgbackrest_last_backup_age_seconds age of the newest pgBackRest backup, at write time
* openvibe_pgbackrest_last_backup_age_seconds age of the newest pgBackRest backup, at write time
* openvibe_backup_offsite_ok 1/0 for the last run's off-site copy, when it made one
* openvibe_backup_offsite_ok 1/0 for the last run's off-site copy, when it made one
@@ -444,7 +413,7 @@ async function writeMetrics(ctx, summary) {
@@ -444,7 +413,7 @@ async function writeMetrics(ctx, summary) {
`openvibe_backup_last_run_ok ${summary.ok ? 1 : 0}`,
`openvibe_backup_last_run_ok ${summary.ok ? 1 : 0}`,
'# HELP openvibe_backup_services Services by outcome in the last run.',
'# HELP openvibe_backup_services Services by outcome in the last run.',
'# TYPE openvibe_backup_services gauge',
'# TYPE openvibe_backup_services gauge',
- ...['ok', 'failed', 'skipped', 'verified'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),
+ ...['ok', 'failed', 'verified'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),
];
];
if (verified.length) {
if (verified.length) {
lines.push(
lines.push(
@@ -1,18 +1,5 @@
@@ -1,18 +1,5 @@
'use strict';
'use strict';
-/**
- * ovhost data provision | switch — run the data role's own scripts from the ovhost install
- * (roles/data/add-service.sh, roles/data/switch-service.sh; ADR-035). The agent pipeline may run
- * only ovhost through its broker, so giving a service its PostgreSQL database and switching it
- * over from SQLite has to go through here too.
- *
- * ovhost data provision <service> -> roles/data/add-service.sh <service>
- * ovhost data switch <service> [--sqlite <file>] -> roles/data/switch-service.sh <service> [<file>],
- * with SWITCH_UNITS/SWITCH_DIR from the inventory
- *
- * add-service.sh prints names only. Whatever the scripts say, their output passes through redact():
- * a connection URL (postgres://, redis://, valkey://) or a NAME=value whose name carries PASSWORD
- * never reaches a terminal, a log or --json. The scripts' own exit code is passed through.
- */
+/** ovhost data provision runs the data role from this installation. */
const path = require('path');
const path = require('path');
const { service: inventoryService } = require('../inventory');
const { service: inventoryService } = require('../inventory');
const { OpError, EXIT } = require('../release-ops');
const { OpError, EXIT } = require('../release-ops');
@@ -33,27 +20,6 @@ function dataDirFor(selfPath) {
@@ -33,27 +20,6 @@ function dataDirFor(selfPath) {
return path.join(dir, 'roles', 'data');
return path.join(dir, 'roles', 'data');
}
}
-/** A template worker unit is stopped as a glob: [email protected] -> openre-rtmp-ingest@*.service. */
-function unitPattern(unit) {
- return unit.endsWith('@.service') ? `${unit.slice(0, -'.service'.length)}*.service` : unit;
-}
-
-/** The SWITCH_UNITS/SWITCH_DIR the script would otherwise default to (roles/data/switch-service.sh). */
-function switchEnv(svc) {
- const env = {};
- const units = [...svc.units, ...svc.workerUnits].map(unitPattern);
- // Deploys restart every unit matching unitsMatch (release-ops), so the switch stops them too.
- if (svc.unitsMatch && !units.includes(svc.unitsMatch)) units.push(svc.unitsMatch);
- // A socket-activated service comes straight back on the next request while its socket listens: the old
- // release would write the database during the import (2026-10-08, Live: analytics rows moved under the verify).
- // The socket is stopped after the service (the first unit stays the one whose environment runs the import); the
- // deploy's restart starts it again through the service's Requires=.
- if (svc.socketUnit && !units.includes(svc.socketUnit)) units.push(svc.socketUnit);
- if (units.length) env.SWITCH_UNITS = units.join(' ');
- env.SWITCH_DIR = svc.repo;
- return env;
-}
-
function printable(script, args, env) {
function printable(script, args, env) {
const assigns = Object.entries(env || {}).map(([k, v]) => `${k}=${JSON.stringify(v)}`).join(' ');
const assigns = Object.entries(env || {}).map(([k, v]) => `${k}=${JSON.stringify(v)}`).join(' ');
return redact(`${assigns ? `${assigns} ` : ''}${script}${args.map((a) => ` ${JSON.stringify(a)}`).join('')}`);
return redact(`${assigns ? `${assigns} ` : ''}${script}${args.map((a) => ` ${JSON.stringify(a)}`).join('')}`);
@@ -71,8 +37,7 @@ async function locate(exec, selfPath, script, action, id) {
@@ -71,8 +37,7 @@ async function locate(exec, selfPath, script, action, id) {
async function run(ctx, svc, { action, script, args, env = {}, dryRun = false }) {
async function run(ctx, svc, { action, script, args, env = {}, dryRun = false }) {
const result = { service: svc.id, action, script, args, env, dryRun };
const result = { service: svc.id, action, script, args, env, dryRun };
if (dryRun) return { ...result, command: printable(script, args, env), code: null, ok: true, stdout: '', stderr: '' };
if (dryRun) return { ...result, command: printable(script, args, env), code: null, ok: true, stdout: '', stderr: '' };
- // Both scripts need root (add-service.sh exits otherwise; switch uses systemctl, setpriv, chown),
- // and sudo's env_reset would drop SWITCH_UNITS/SWITCH_DIR, silently switching the wrong units/dir.
+ // The data role requires root.
if (!(await ctx.exec.isRoot())) throw new OpError(`data ${action} must run as root (sudo ovhost data ${action} ${svc.id})`, EXIT.USAGE);
if (!(await ctx.exec.isRoot())) throw new OpError(`data ${action} must run as root (sudo ovhost data ${action} ${svc.id})`, EXIT.USAGE);
const r = await ctx.exec.run(script, args, { privileged: true, env });
const r = await ctx.exec.run(script, args, { privileged: true, env });
return { ...result, code: r.code, ok: r.code === 0, stdout: redact(r.stdout), stderr: redact(r.stderr) };
return { ...result, code: r.code, ok: r.code === 0, stdout: redact(r.stdout), stderr: redact(r.stderr) };
@@ -84,12 +49,4 @@ async function provision(ctx, id, { selfPath, dryRun = false } = {}) {
@@ -84,12 +49,4 @@ async function provision(ctx, id, { selfPath, dryRun = false } = {}) {
return run(ctx, svc, { action: 'provision', script, args: [svc.id], dryRun });
return run(ctx, svc, { action: 'provision', script, args: [svc.id], dryRun });
}
}
-async function switchService(ctx, id, { selfPath, sqlite, dryRun = false } = {}) {
- const svc = inventoryService(ctx.inv, id);
- const script = await locate(ctx.exec, selfPath, 'switch-service.sh', 'switch', svc.id);
- const args = [svc.id];
- if (sqlite) args.push(sqlite);
- return run(ctx, svc, { action: 'switch', script, args, env: switchEnv(svc), dryRun });
-}
-
-module.exports = { provision, switch: switchService, redact, dataDirFor, switchEnv, unitPattern };
+module.exports = { provision, redact, dataDirFor };
@@ -5,20 +5,17 @@
@@ -5,20 +5,17 @@
* production does:
* production does:
*
*
* 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Every database the drill
* 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Every database the drill
- * block names is restored, per engine (lib/dbengine.js):
- * SQLite copy the file into a fresh directory under <drillDir> owned by the service user
- * and run PRAGMA integrity_check on the copy. The result must be `ok`.
- * PostgreSQL check free space first, copy the pg_dump -Fc archive into <tmp>/pg owned by
- * postgres, verify it with `pg_restore --list`, then create a scratch database and
- * a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the
- * archive into the scratch database. Both are dropped again in a finally.
+ * block names is restored from its pg_dump -Fc archive (lib/dbengine.js): check free space first,
+ * copy the archive into <tmp>/pg owned by postgres, verify it with `pg_restore --list`, then create a
+ * scratch database and a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the
+ * archive into the scratch database. Both are dropped again in a finally.
* Every object directory the service declares is also restored: its archive is extracted into
* Every object directory the service declares is also restored: its archive is extracted into
* <tmp>/<name> (owned by the service user, never the live directory) and every blob is checked
* <tmp>/<name> (owned by the service user, never the live directory) and every blob is checked
* against the sha256 in its file name.
* against the sha256 in its file name.
* 2. Start a second instance from the production checkout, as the service user, through
* 2. Start a second instance from the production checkout, as the service user, through
* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
- * env file, and a second env file with the drill's overrides: its port, the restored database
- * paths (or the scratch database's URLs), and the switches that turn side effects off. The
+ * env file, and a second env file with the drill's overrides: its port, the scratch database
+ * URLs, and the switches that turn side effects off. The
* instance runs sandboxed: ProtectSystem=strict with only the drill directory writable, a
* instance runs sandboxed: ProtectSystem=strict with only the drill directory writable, a
* private /tmp, binding only its drill port, and no addresses beyond loopback unless the
* private /tmp, binding only its drill port, and no addresses beyond loopback unless the
* inventory says otherwise. RuntimeMaxSec stops it even if ovhost dies. Loopback stays open
* inventory says otherwise. RuntimeMaxSec stops it even if ovhost dies. Loopback stays open
@@ -70,10 +67,9 @@ const STOP_GRACE_MS = 20000;
@@ -70,10 +67,9 @@ const STOP_GRACE_MS = 20000;
// ── helpers ──────────────────────────────────────────────────────────────────
// ── helpers ──────────────────────────────────────────────────────────────────
function expand(value, vars) {
function expand(value, vars) {
- return value.replace(/\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g, (_, k) => {
+ return value.replace(/\{(tmp|port)\}/g, (_, k) => {
if (k === 'tmp') return vars.tmp;
if (k === 'tmp') return vars.tmp;
if (k === 'port') return String(vars.port);
if (k === 'port') return String(vars.port);
- return vars.db[k.slice(3)];
});
});
}
}
@@ -145,13 +141,6 @@ function compareBodies(entry, prod, drill) {
@@ -145,13 +141,6 @@ function compareBodies(entry, prod, drill) {
function countSql(table) { return `SELECT count(*) AS n FROM "${table}"`; }
function countSql(table) { return `SELECT count(*) AS n FROM "${table}"`; }
-async function countRows(exec, db, table, as) {
- const rows = await exec.sqlite(db, countSql(table), { as });
- const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);
- if (!Number.isFinite(n)) throw new Error(`count of ${table} returned no number`);
- return n;
-}
-
// A content-addressed blob is named by its own sha256: that name is the whole integrity check.
// A content-addressed blob is named by its own sha256: that name is the whole integrity check.
const BLOB_RE = /^[0-9a-f]{64}$/;
const BLOB_RE = /^[0-9a-f]{64}$/;
@@ -230,7 +219,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {
@@ -230,7 +219,7 @@ async function resolveBackup(exec, inv, svc, dirArg) {
let takenAt = null;
let takenAt = null;
let record = null;
let record = null;
const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));
const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));
- const pgTargets = restored.filter((d) => d.engine === 'postgresql');
+ const pgTargets = restored;
if (dir) {
if (dir) {
if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');
if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');
dir = path.normalize(dir);
dir = path.normalize(dir);
@@ -255,11 +244,10 @@ async function resolveBackup(exec, inv, svc, dirArg) {
@@ -255,11 +244,10 @@ async function resolveBackup(exec, inv, svc, dirArg) {
for (const name of Object.keys(svc.drill.databases)) {
for (const name of Object.keys(svc.drill.databases)) {
const db = svc.databases.find((d) => d.name === name);
const db = svc.databases.find((d) => d.name === name);
const target = svc.drill.databases[name];
const target = svc.drill.databases[name];
- const file = path.join(dir, db.engine === 'postgresql' ? dbengine.fileName(db) : `${name}${path.extname(db.path) || '.db'}`);
+ const file = path.join(dir, dbengine.fileName(db));
const fst = await exec.stat(file);
const fst = await exec.stat(file);
if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);
if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);
- if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });
- else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });
+ files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });
}
}
// Object directories are restored from the same backup: the archive holds the declared
// Object directories are restored from the same backup: the archive holds the declared
// directory's contents, so the drill must not run without it (there would be nothing to verify).
// directory's contents, so the drill must not run without it (there would be nothing to verify).
@@ -327,7 +315,7 @@ async function missingRequirements(exec, svc) {
@@ -327,7 +315,7 @@ async function missingRequirements(exec, svc) {
async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const { exec, inv, log } = ctx;
const { exec, inv, log } = ctx;
const svc = service(inv, id);
const svc = service(inv, id);
- if (!(await exec.isRoot())) throw new DrillError('ovhost drill must run as root (sudo ovhost drill …): it copies databases as the service user and starts the drill instance through systemd-run');
+ if (!(await exec.isRoot())) throw new DrillError('ovhost drill must run as root (sudo ovhost drill …): it starts the drill instance through systemd-run');
if (!svc.drill) throw new DrillError(`${id} has no drill block in the inventory`);
if (!svc.drill) throw new DrillError(`${id} has no drill block in the inventory`);
if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);
if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);
const d = svc.drill;
const d = svc.drill;
@@ -425,18 +413,13 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -425,18 +413,13 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
await exec.mkdir(inv.drillDir, { mode: 0o711 });
await exec.mkdir(inv.drillDir, { mode: 0o711 });
// postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;
// postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;
- // every file inside keeps its own restrictive mode). A SQLite-only drill keeps <tmp> at 0700.
- await exec.mkdir(tmp, { owner: svc.runAs, mode: pgFiles.length ? 0o711 : 0o700 });
+ // every file inside keeps its own restrictive mode).
+ await exec.mkdir(tmp, { owner: svc.runAs, mode: 0o711 });
created = true;
created = true;
- await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });
// postgres (not the service user) reads a PostgreSQL dump, so it gets its own directory; the
// postgres (not the service user) reads a PostgreSQL dump, so it gets its own directory; the
// copy inside is 0600 postgres (the backup itself is root-only, which postgres cannot read).
// copy inside is 0600 postgres (the backup itself is root-only, which postgres cannot read).
if (pgFiles.length) await exec.mkdir(path.join(tmp, 'pg'), { owner: dbengine.PG_USER, mode: 0o700 });
if (pgFiles.length) await exec.mkdir(path.join(tmp, 'pg'), { owner: dbengine.PG_USER, mode: 0o700 });
- const vars = { tmp, port: d.port, db: {} };
- // A data-directory database keeps its production file name: the service finds it by name.
- const dirOf = (f) => (f.dir ? expand(f.dir, vars) : null);
- for (const f of bk.files) { if (f.engine === 'postgresql') continue; vars.db[f.name] = f.dir ? path.join(dirOf(f), path.basename(f.production)) : path.join(tmp, 'db', `${f.name}${f.ext}`); }
- for (const dir of new Set(bk.files.filter((f) => f.dir).map(dirOf))) await exec.mkdir(dir, { owner: svc.runAs, mode: 0o700 });
+ const vars = { tmp, port: d.port };
for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });
for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });
for (const b of d.bind) if (!(await exec.stat(expand(b.from, vars)))) await exec.mkdir(expand(b.from, vars), { owner: svc.runAs, mode: 0o700 });
for (const b of d.bind) if (!(await exec.stat(expand(b.from, vars)))) await exec.mkdir(expand(b.from, vars), { owner: svc.runAs, mode: 0o700 });
@@ -444,46 +427,27 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -444,46 +427,27 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const sid = id.replace(/[^a-z0-9_]/g, '_');
const sid = id.replace(/[^a-z0-9_]/g, '_');
const scratchFor = (name) => scratches.find((s) => s.name === name);
const scratchFor = (name) => scratches.find((s) => s.name === name);
for (const f of bk.files) {
for (const f of bk.files) {
- if (f.engine === 'postgresql') {
- // A PostgreSQL restore replaces the file copy: root copies the root-only archive into
- // <tmp>/pg owned by postgres, verifies it, then creates a scratch database and role and
- // restores the archive into them (--no-owner, --role: the archive names production's
- // role, and every object must end up owned by the drill role the instance connects as).
- const copy = path.join(tmp, 'pg', `${f.name}.dump`);
- log(`restoring ${f.file} → ${copy} (owned by ${dbengine.PG_USER})`);
- const cp = await exec.run('install', ['-o', dbengine.PG_USER, '-m', '0600', '-T', '--', f.file, copy], { privileged: true });
- if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
- const st = await exec.stat(copy);
- const entry = { name: f.name, source: f.file, copy, bytes: st ? st.size : null, integrity: null, engine: 'postgresql' };
- record.databases.push(entry);
- try { await dbengine.verifyDump(exec, copy, { as: dbengine.PG_USER }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }
- entry.integrity = 'pg_restore --list ok';
- log(`pg_restore --list ${f.name}: ok`);
- // The role and the database share the same name (separate namespaces); dbengine always
- // quotes the identifier, so the stamp's hyphen is fine.
- const name = `ov_${sid}_drill_${at}${pgFiles.length > 1 ? `_${pgFiles.indexOf(f)}` : ''}`;
- const { password } = await dbengine.createRole(exec, name, {});
- const scratch = { name: f.name, database: name, role: name, password, dropped: null, error: null };
- scratches.push(scratch);
- if (!record.scratch) record.scratch = { database: name, role: name };
- await dbengine.createDatabase(exec, name, { owner: name });
- await dbengine.restore(exec, f.production, { dump: copy, database: name, role: name, log });
- continue;
- }
- const dest = vars.db[f.name];
- log(`restoring ${f.file} → ${dest} (owned by ${svc.runAs})`);
- // Backups are root-only (0600), so root copies; install unlinks whatever is at dest
- // before writing, so nothing planted in the drill directory is followed.
- const cp = await exec.run('install', ['-o', svc.runAs, '-m', '0600', '-T', '--', f.file, dest], { privileged: true });
+ // Root copies the archive into <tmp>/pg owned by postgres, verifies it, then restores
+ // it into a scratch database and role. Every object ends up owned by the drill role.
+ const copy = path.join(tmp, 'pg', `${f.name}.dump`);
+ log(`restoring ${f.file} → ${copy} (owned by ${dbengine.PG_USER})`);
+ const cp = await exec.run('install', ['-o', dbengine.PG_USER, '-m', '0600', '-T', '--', f.file, copy], { privileged: true });
if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
- const st = await exec.stat(dest);
- const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null, engine: 'sqlite' };
+ const st = await exec.stat(copy);
+ const entry = { name: f.name, source: f.file, copy, bytes: st ? st.size : null, integrity: null, engine: 'postgresql' };
record.databases.push(entry);
record.databases.push(entry);
- let rows;
- try { rows = await exec.sqlite(dest, 'PRAGMA integrity_check', { as: svc.runAs }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }
- entry.integrity = (rows || []).map((r) => String(Object.values(r)[0])).join('; ') || 'no result';
- if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);
- log(`integrity_check ${f.name}: ok`);
+ try { await dbengine.verifyDump(exec, copy, { as: dbengine.PG_USER }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }
+ entry.integrity = 'pg_restore --list ok';
+ log(`pg_restore --list ${f.name}: ok`);
+ // The role and the database share the same name (separate namespaces); dbengine always
+ // quotes the identifier, so the stamp's hyphen is fine.
+ const name = `ov_${sid}_drill_${at}${pgFiles.length > 1 ? `_${pgFiles.indexOf(f)}` : ''}`;
+ const { password } = await dbengine.createRole(exec, name, {});
+ const scratch = { name: f.name, database: name, role: name, password, dropped: null, error: null };
+ scratches.push(scratch);
+ if (!record.scratch) record.scratch = { database: name, role: name };
+ await dbengine.createDatabase(exec, name, { owner: name });
+ await dbengine.restore(exec, f.production, { dump: copy, database: name, role: name, log });
}
}
// 1b. restore every declared object directory and check every blob's sha256. The archive is
// 1b. restore every declared object directory and check every blob's sha256. The archive is
// root-only, so root extracts it into the drill's own scratch directory — never over the live
// root-only, so root extracts it into the drill's own scratch directory — never over the live
@@ -510,10 +474,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -510,10 +474,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
}
}
const restoredCounts = [];
const restoredCounts = [];
for (const c of d.counts) {
for (const c of d.counts) {
- const rdb = svc.databases.find((x) => x.name === c.db);
- // PostgreSQL counts come from the scratch database over psql, SQLite from the restored file.
- if (rdb.engine === 'postgresql') restoredCounts.push(await dbengine.countRows(exec, { engine: 'postgresql', database: scratchFor(c.db).database }, countSql(c.table)));
- else restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));
+ restoredCounts.push(await dbengine.countRows(exec, { database: scratchFor(c.db).database }, countSql(c.table)));
}
}
// 2. start the drill instance
// 2. start the drill instance
@@ -529,7 +490,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -529,7 +490,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const url = dbengine.databaseUrl({ database: s.database, role: s.role, password: s.password });
const url = dbengine.databaseUrl({ database: s.database, role: s.role, password: s.password });
dbEnv.set(f.url, url);
dbEnv.set(f.url, url);
dbEnv.set(f.directUrl, url);
dbEnv.set(f.directUrl, url);
- } else if (f.envName) dbEnv.set(f.envName, f.dir ? dirOf(f) : vars.db[f.name]);
+ }
}
}
for (const [k, v] of dbEnv) lines.push(envLine(k, v));
for (const [k, v] of dbEnv) lines.push(envLine(k, v));
const envPath = path.join(tmp, 'drill.env');
const envPath = path.join(tmp, 'drill.env');
@@ -606,9 +567,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -606,9 +567,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const db = svc.databases.find((x) => x.name === c.db);
const db = svc.databases.find((x) => x.name === c.db);
let production = null;
let production = null;
let error = null;
let error = null;
- // PostgreSQL counts come from the cluster over psql (the inventory entry itself, never
- // .path, which a PostgreSQL entry does not have); SQLite from the production file.
- try { production = db.engine === 'postgresql' ? await dbengine.countRows(exec, db, countSql(c.table)) : await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }
+ try { production = await dbengine.countRows(exec, db, countSql(c.table)); } catch (err) { error = err.message; }
const tol = d.countsTolerance || 0;
const tol = d.countsTolerance || 0;
// A busy table drifts between the backup and now: countsTolerance (default 0) allows it.
// A busy table drifts between the backup and now: countsTolerance (default 0) allows it.
const match = production != null && restoredCounts[i] != null && Math.abs(production - restoredCounts[i]) <= tol;
const match = production != null && restoredCounts[i] != null && Math.abs(production - restoredCounts[i]) <= tol;
@@ -709,9 +668,7 @@ function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }
@@ -709,9 +668,7 @@ function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }
/** A row for the table in docs/restore-drills.md. */
/** A row for the table in docs/restore-drills.md. */
function markdownRow(r) {
function markdownRow(r) {
const checks = [];
const checks = [];
- for (const db of r.databases) checks.push(db.engine === 'postgresql'
- ? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`
- : `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
+ for (const db of r.databases) checks.push(`\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'
for (const o of r.objects || []) checks.push(o.integrity === 'sha256 ok'
? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`
? `object store \`${o.name}\` ${o.blobs} blob(s) sha256 ok`
: `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);
: `object store \`${o.name}\`: ${cell(o.integrity).slice(0, 80)}`);
@@ -64,11 +64,11 @@ async function validate(ctx, id, opts = {}) {
@@ -64,11 +64,11 @@ async function validate(ctx, id, opts = {}) {
try {
try {
const g = git(exec, svc);
const g = git(exec, svc);
const [branch, head, dirty] = [await g.branch(), await g.head(), await g.dirty()];
const [branch, head, dirty] = [await g.branch(), await g.head(), await g.dirty()];
- // A checkout reached through <dir>/current -> <dir>/releases/<id> is a deployed release (Live's
- // own deploy script): detached by design, so the branch is not checked; changes still are.
+ // A checkout reached through <dir>/current -> <dir>/releases/<id> is a deployed release:
+ // detached by design, so the branch is not checked; changes still are.
const link = branch === 'HEAD' && svc.repo.endsWith('/current') ? await exec.readlink(svc.repo) : null;
const link = branch === 'HEAD' && svc.repo.endsWith('/current') ? await exec.readlink(svc.repo) : null;
const release = link && path.resolve(path.dirname(svc.repo), link).startsWith(`${path.dirname(svc.repo)}/releases/`);
const release = link && path.resolve(path.dirname(svc.repo), link).startsWith(`${path.dirname(svc.repo)}/releases/`);
- if (release) add('ok', 'checkout', `release ${link} (detached; deployed by the repository's own script)`);
+ if (release) add('ok', 'checkout', `release ${link} (detached)`);
else if (branch !== svc.branch) add('error', 'checkout', `on branch "${branch}", expected "${svc.branch}"`);
else if (branch !== svc.branch) add('error', 'checkout', `on branch "${branch}", expected "${svc.branch}"`);
const own = dirty.filter((f) => !require('../release-ops').matchesPath(f, svc.generated));
const own = dirty.filter((f) => !require('../release-ops').matchesPath(f, svc.generated));
if (own.length < dirty.length) add('info', 'checkout', `${dirty.length - own.length} tracked build output file(s) differ from git (${svc.generated.join(', ')}); a deploy restores them first`);
if (own.length < dirty.length) add('info', 'checkout', `${dirty.length - own.length} tracked build output file(s) differ from git (${svc.generated.join(', ')}); a deploy restores them first`);
@@ -1,14 +1,8 @@
@@ -1,14 +1,8 @@
'use strict';
'use strict';
/**
/**
- * Per-engine database helpers. lib/inventory.js normalises every entry in `databases[]` to an
- * `engine` of "sqlite" (the default: a file) or "postgresql" (a database in the cluster), so no
- * caller has to know which commands an engine needs.
- *
- * SQLite keeps the existing worker (lib/sqlite-worker.js): every file operation runs as the service
- * user, and root never opens a service database. PostgreSQL runs as the postgres OS user over peer
- * auth (`local postgres peer` in roles/data/files/pg_hba.conf) — the same property: ovhost reads no
- * service env value, and never holds a database password except the one it generates itself for a
- * restore drill.
+ * PostgreSQL database helpers (lib/inventory.js accepts no other engine). Every command runs as the postgres
+ * OS user over peer auth (`local postgres peer` in roles/data/files/pg_hba.conf): ovhost reads no service env
+ * value and never holds a database password except the one it generates itself for a restore drill's role.
*
*
* pgBackRest, not ovhost, is the durability layer for the cluster (roles/data/provision.sh: one
* pgBackRest, not ovhost, is the durability layer for the cluster (roles/data/provision.sh: one
* stanza `openvibe`, full weekly + differential nightly + WAL to object storage). What ovhost does
* stanza `openvibe`, full weekly + differential nightly + WAL to object storage). What ovhost does
@@ -16,7 +10,6 @@
@@ -16,7 +10,6 @@
* a per-service logical dump (dump) that the off-site code encrypts and uploads unchanged.
* a per-service logical dump (dump) that the off-site code encrypts and uploads unchanged.
*/
*/
const crypto = require('crypto');
const crypto = require('crypto');
-const path = require('path');
// The OS user PostgreSQL's own tools run as (peer auth), the stanza provision.sh creates, and the
// The OS user PostgreSQL's own tools run as (peer auth), the stanza provision.sh creates, and the
// direct connection a drill must use: 5432 is the cluster, 6432 is PgBouncer.
// direct connection a drill must use: 5432 is the cluster, 6432 is PgBouncer.
@@ -36,17 +29,16 @@ const scrubbed = (r) => ((r.stderr || r.stdout || '').trim().slice(0, 300) || `e
@@ -36,17 +29,16 @@ const scrubbed = (r) => ((r.stderr || r.stdout || '').trim().slice(0, 300) || `e
/** What a log line and a record call this database. */
/** What a log line and a record call this database. */
function label(db) {
function label(db) {
- return db.engine === 'postgresql' ? db.database : db.path;
+ return db.database;
}
}
/** The file name this database's artifact gets inside a backup directory. */
/** The file name this database's artifact gets inside a backup directory. */
function fileName(db) {
function fileName(db) {
- return db.engine === 'postgresql' ? `${db.name}.dump` : `${db.name}${path.extname(db.path) || '.db'}`;
+ return `${db.name}.dump`;
}
}
-/** Whether the database is still there (SQLite: the file; PostgreSQL: the database in the cluster). */
+/** Whether the database is still there in the cluster. */
async function exists(exec, db) {
async function exists(exec, db) {
- if (db.engine === 'sqlite') return !!(await exec.stat(db.path));
const rows = await exec.psql('postgres', `SELECT 1 AS present FROM pg_database WHERE datname = ${literal(db.database)}`);
const rows = await exec.psql('postgres', `SELECT 1 AS present FROM pg_database WHERE datname = ${literal(db.database)}`);
return rows.length > 0;
return rows.length > 0;
}
}
@@ -57,7 +49,6 @@ async function exists(exec, db) {
@@ -57,7 +49,6 @@ async function exists(exec, db) {
* the caller's staging directory is owned by postgres when any database is PostgreSQL.
* the caller's staging directory is owned by postgres when any database is PostgreSQL.
*/
*/
async function dump(exec, db, dest, { log } = {}) {
async function dump(exec, db, dest, { log } = {}) {
- if (db.engine !== 'postgresql') throw new Error('sqlite dumps go through exec.sqliteBackup (the worker runs as the service user)');
// -w: peer auth needs no password, and a prompt in the nightly unit would hang until the timeout.
// -w: peer auth needs no password, and a prompt in the nightly unit would hang until the timeout.
const r = await exec.run('pg_dump', ['-w', '-Fc', '-f', dest, '-d', db.database], { as: PG_USER, timeoutMs: WORK_TIMEOUT_MS });
const r = await exec.run('pg_dump', ['-w', '-Fc', '-f', dest, '-d', db.database], { as: PG_USER, timeoutMs: WORK_TIMEOUT_MS });
if (r.code !== 0) throw new Error(`pg_dump ${db.database} failed: ${scrubbed(r)}`);
if (r.code !== 0) throw new Error(`pg_dump ${db.database} failed: ${scrubbed(r)}`);
@@ -114,7 +105,6 @@ async function dropRole(exec, role) {
@@ -114,7 +105,6 @@ async function dropRole(exec, role) {
* owned by the drill role, so the drilled service can read and write it.
* owned by the drill role, so the drilled service can read and write it.
*/
*/
async function restore(exec, db, { dump: dumpFile, database, role, log } = {}) {
async function restore(exec, db, { dump: dumpFile, database, role, log } = {}) {
- if (db.engine !== 'postgresql') throw new Error('sqlite restore is a file copy, not pg_restore');
// --no-privileges: the archive's GRANTs name production roles; the scratch copy must not hand them anything.
// --no-privileges: the archive's GRANTs name production roles; the scratch copy must not hand them anything.
const args = ['-w', '--no-owner', '--no-privileges'];
const args = ['-w', '--no-owner', '--no-privileges'];
if (role) args.push(`--role=${role}`);
if (role) args.push(`--role=${role}`);
@@ -137,9 +127,9 @@ async function scratchLeftovers(exec, prefix) {
@@ -137,9 +127,9 @@ async function scratchLeftovers(exec, prefix) {
return { databases: names(databases), roles: names(roles) };
return { databases: names(databases), roles: names(roles) };
}
}
-/** A row count from either engine. `sql` must return the number in its first column. */
-async function countRows(exec, db, sql, { as } = {}) {
- const rows = db.engine === 'postgresql' ? await exec.psql(db.database, sql) : await exec.sqlite(db.path, sql, { as });
+/** A row count from PostgreSQL. `sql` must return the number in its first column. */
+async function countRows(exec, db, sql) {
+ const rows = await exec.psql(db.database, sql);
const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);
const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);
if (!Number.isFinite(n)) throw new Error(`count of ${label(db)} returned no number`);
if (!Number.isFinite(n)) throw new Error(`count of ${label(db)} returned no number`);
return n;
return n;
@@ -1,12 +1,11 @@
@@ -1,12 +1,11 @@
'use strict';
'use strict';
/**
/**
- * The system executor: the ONLY module in ovhost that touches the machine (processes, files, HTTP,
- * SQLite). Every command and library function takes an executor, so the tests swap this for the
- * fake host in test/fake-host.js and nothing in the suite ever runs systemctl, git, npm or curl.
+ * The system executor: the only module in ovhost that touches the machine. Tests replace it with
+ * the fake host in test/fake-host.js.
*
*
* Interface (all async):
* Interface (all async):
* run(cmd, args, { as, privileged, cwd, timeoutMs, input }) -> { code, stdout, stderr }
* run(cmd, args, { as, privileged, cwd, timeoutMs, input }) -> { code, stdout, stderr }
- * as run as this user (git/npm as the checkout owner, sqlite as the service user);
+ * as run as this user (git/npm as the checkout owner);
* root uses runuser with the user's HOME, the same user runs directly, anyone else
* root uses runuser with the user's HOME, the same user runs directly, anyone else
* goes through `sudo -n -H -u`.
* goes through `sudo -n -H -u`.
* privileged run as root (systemctl, nginx -t, reading 0600 env files); `sudo -n` unless root.
* privileged run as root (systemctl, nginx -t, reading 0600 env files); `sudo -n` unless root.
@@ -22,11 +21,9 @@
@@ -22,11 +21,9 @@
* request(url, { method, headers, body, timeoutMs }) -> { status, body } | { status: 0, error }
* request(url, { method, headers, body, timeoutMs }) -> { status, body } | { status: 0, error }
* any method, http:// or https:// (release notifications: Network's token endpoint and Events'
* any method, http:// or https:// (release notifications: Network's token endpoint and Events'
* publish API). Headers and body may carry a secret: nothing here logs them.
* publish API). Headers and body may carry a secret: nothing here logs them.
- * sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })
* psql(database, sql, { as }) -> rows. PostgreSQL is reached as the postgres OS user over peer
* psql(database, sql, { as }) -> rows. PostgreSQL is reached as the postgres OS user over peer
- * auth (`local postgres peer`, roles/data/files/pg_hba.conf) — the same no-secret property the
- * SQLite worker gives the service user. Everything else PostgreSQL (pg_dump, pg_restore,
- * createdb, dropdb, pgbackrest) goes through run() with { as: 'postgres' }.
+ * auth (`local postgres peer`, roles/data/files/pg_hba.conf). PostgreSQL tools also go
+ * through run() with { as: 'postgres' }.
* statfs(path) -> { free, size } bytes, for the free-space check before a restore
* statfs(path) -> { free, size } bytes, for the free-space check before a restore
* listeners(port) -> [{ pid, process }]
* listeners(port) -> [{ pid, process }]
* kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)
* kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)
@@ -39,7 +36,6 @@ const httpsLib = require('https');
@@ -39,7 +36,6 @@ const httpsLib = require('https');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const WORKER = path.join(__dirname, 'sqlite-worker.js');
function passwdEntry(user) {
function passwdEntry(user) {
try {
try {
@@ -229,14 +225,6 @@ function createSystemExecutor({ env = process.env } = {}) {
@@ -229,14 +225,6 @@ function createSystemExecutor({ env = process.env } = {}) {
});
});
}
}
- async function sqliteCall(payload, as) {
- const r = await run(process.execPath, [WORKER, JSON.stringify(payload)], { as, timeoutMs: 6 * 3600 * 1000 });
- let out;
- try { out = JSON.parse(r.stdout); } catch { throw new Error(`sqlite worker failed: ${(r.stderr || r.stdout).trim().slice(0, 300)}`); }
- if (!out.ok) throw new Error(out.error);
- return out;
- }
-
/** `psql --csv`: a header row and one line per row, so a caller reads columns by name. */
/** `psql --csv`: a header row and one line per row, so a caller reads columns by name. */
function parseCsv(text) {
function parseCsv(text) {
const rows = [];
const rows = [];
@@ -291,8 +279,6 @@ function createSystemExecutor({ env = process.env } = {}) {
@@ -291,8 +279,6 @@ function createSystemExecutor({ env = process.env } = {}) {
pidAlive: async (pid) => pidAlive(pid),
pidAlive: async (pid) => pidAlive(pid),
http,
http,
request,
request,
- sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,
- sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },
psql: async (database, sql, { as = 'postgres' } = {}) => {
psql: async (database, sql, { as = 'postgres' } = {}) => {
const r = await run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '--csv', '-d', database, '-c', sql], { as, timeoutMs: 6 * 3600 * 1000 });
const r = await run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '--csv', '-d', database, '-c', sql], { as, timeoutMs: 6 * 3600 * 1000 });
if (r.code !== 0) throw new Error(`psql ${database} failed: ${(r.stderr || r.stdout).trim().slice(0, 300)}`);
if (r.code !== 0) throw new Error(`psql ${database} failed: ${(r.stderr || r.stdout).trim().slice(0, 300)}`);
@@ -20,7 +20,7 @@ const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
@@ -20,7 +20,7 @@ const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
-const PROBE_KINDS = ['http-json-count', 'sqlite-count', 'postgresql-count', 'sum'];
+const PROBE_KINDS = ['http-json-count', 'postgresql-count', 'sum'];
// A PostgreSQL database an ovhost service owns: one per service in the cluster pgBackRest backs up
// A PostgreSQL database an ovhost service owns: one per service in the cluster pgBackRest backs up
// (roles/data/provision.sh), reached as the postgres OS user over peer auth (no secret is ever read).
// (roles/data/provision.sh), reached as the postgres OS user over peer auth (no secret is ever read).
const PG_DATABASE_RE = /^ov_[a-z0-9_]+$/;
const PG_DATABASE_RE = /^ov_[a-z0-9_]+$/;
@@ -166,9 +166,9 @@ function normaliseObjects(raw, where) {
@@ -166,9 +166,9 @@ function normaliseObjects(raw, where) {
function normaliseService(id, raw, defaults) {
function normaliseService(id, raw, defaults) {
if (!ID_RE.test(id)) throw new InventoryError(`service id "${id}" must match ${ID_RE}`);
if (!ID_RE.test(id)) throw new InventoryError(`service id "${id}" must match ${ID_RE}`);
const where = `services.${id}`;
const where = `services.${id}`;
- const strategy = raw.strategy == null ? (raw.layout === 'release' ? null : 'git-checkout') : raw.strategy;
- if (strategy !== null && !STRATEGIES.includes(strategy)) throw new InventoryError(`${where}.strategy must be one of ${STRATEGIES.join(', ')}`);
- if (strategy && strategy !== 'release-layout' && raw.layout === 'release') throw new InventoryError(`${where}: layout "release" needs strategy "release-layout"`);
+ const strategy = raw.strategy == null ? 'git-checkout' : raw.strategy;
+ if (!STRATEGIES.includes(strategy)) throw new InventoryError(`${where}.strategy must be one of ${STRATEGIES.join(', ')}`);
+ if (strategy !== 'release-layout' && raw.layout === 'release') throw new InventoryError(`${where}: layout "release" needs strategy "release-layout"`);
const s = withPreset({ ...defaults, ...raw }, strategy);
const s = withPreset({ ...defaults, ...raw }, strategy);
const releaseLayout = strategy === 'release-layout' || s.layout === 'release';
const releaseLayout = strategy === 'release-layout' || s.layout === 'release';
let repo = abs(s.repo, `${where}.repo`);
let repo = abs(s.repo, `${where}.repo`);
@@ -190,16 +190,15 @@ function normaliseService(id, raw, defaults) {
@@ -190,16 +190,15 @@ function normaliseService(id, raw, defaults) {
strategy,
strategy,
release,
release,
// 'git' (a checkout ovhost can move) or 'release' (releases/<id> behind a `current` symlink).
// 'git' (a checkout ovhost can move) or 'release' (releases/<id> behind a `current` symlink).
- // A release-layout entry without strategy "release-layout" is deployed by its own script.
layout: releaseLayout ? 'release' : 'git',
layout: releaseLayout ? 'release' : 'git',
- managed: strategy === null ? false : s.managed !== false,
+ managed: s.managed !== false,
// The deploy controller (ovhost reconcile, WS-X11): auto deploys main at green CI; gated and manual wait for a person.
// The deploy controller (ovhost reconcile, WS-X11): auto deploys main at green CI; gated and manual wait for a person.
deploy: (() => {
deploy: (() => {
const d = s.deploy || {};
const d = s.deploy || {};
if (d.policy != null && !['auto', 'gated', 'manual'].includes(d.policy)) throw new InventoryError(`${where}.deploy.policy must be auto, gated or manual`);
if (d.policy != null && !['auto', 'gated', 'manual'].includes(d.policy)) throw new InventoryError(`${where}.deploy.policy must be auto, gated or manual`);
return { policy: d.policy || 'manual' };
return { policy: d.policy || 'manual' };
})(),
})(),
- unmanagedReason: s.unmanagedReason || (strategy === null ? 'release layout without strategy "release-layout": deployed by its own deploy script' : null),
+ unmanagedReason: s.unmanagedReason || null,
skipPackages: s.skipPackages || [],
skipPackages: s.skipPackages || [],
removeUntrackedLockfiles: s.removeUntrackedLockfiles === true,
removeUntrackedLockfiles: s.removeUntrackedLockfiles === true,
generated: s.generated || [],
generated: s.generated || [],
@@ -233,28 +232,16 @@ function normaliseService(id, raw, defaults) {
@@ -233,28 +232,16 @@ function normaliseService(id, raw, defaults) {
noRestartPaths: s.noRestartPaths || [],
noRestartPaths: s.noRestartPaths || [],
protected: s.protected || null,
protected: s.protected || null,
drain: { policy: 'refuse', waitMaxSeconds: 8 * 3600, pollSeconds: 60, quietChecks: 2, ...(s.drain || {}) },
drain: { policy: 'refuse', waitMaxSeconds: 8 * 3600, pollSeconds: 60, quietChecks: 2, ...(s.drain || {}) },
- // A database is one of two engines. SQLite (the default) is a file the service user's worker
- // copies; PostgreSQL is a database in the cluster, named, and backed up by pgBackRest as a
- // whole — ovhost never opens it directly. Nothing outside the engine's own code may assume
- // `.path` exists.
+ // Each database is in the PostgreSQL cluster.
databases: (s.databases || []).map((d, i) => {
databases: (s.databases || []).map((d, i) => {
const at = `${where}.databases[${i}]`;
const at = `${where}.databases[${i}]`;
- const engine = d.engine == null ? 'sqlite' : d.engine;
- if (engine === 'sqlite') {
- if (d.database != null) throw new InventoryError(`${at}: a sqlite database is a path, not a database name`);
- const name = d.name || path.basename(String(d.path), '.db');
- if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);
- return { name, engine, path: abs(d.path, `${at}.path`) };
- }
- if (engine === 'postgresql') {
- if (d.path != null) throw new InventoryError(`${at}: a postgresql database has no path (it lives in the cluster, named by "database")`);
- if (!PG_DATABASE_RE.test(d.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_trade)`);
- const name = d.name || String(d.database).slice(3);
- if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);
- if (d.role != null && !/^[a-z_][a-z0-9_-]*$/.test(String(d.role))) throw new InventoryError(`${at}.role must be a database role name`);
- return { name, engine, database: d.database, role: d.role == null ? null : String(d.role) };
- }
- throw new InventoryError(`${at}.engine must be sqlite or postgresql`);
+ if (!d || d.engine !== 'postgresql') throw new InventoryError(`${at}.engine must be "postgresql"; use { name, engine: "postgresql", database: "ov_<service>" }`);
+ if (d.path != null) throw new InventoryError(`${at}: a postgresql database has no path (it lives in the cluster, named by "database")`);
+ if (!PG_DATABASE_RE.test(d.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_trade)`);
+ const name = d.name || String(d.database).slice(3);
+ if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);
+ if (d.role != null && !/^[a-z_][a-z0-9_-]*$/.test(String(d.role))) throw new InventoryError(`${at}.role must be a database role name`);
+ return { name, engine: 'postgresql', database: d.database, role: d.role == null ? null : String(d.role) };
}),
}),
// Content-addressed object directories alongside the databases (tenant objects, Host's
// Content-addressed object directories alongside the databases (tenant objects, Host's
// HOST_STORAGE_DIR). Copied as one archive per directory; never opened as a database.
// HOST_STORAGE_DIR). Copied as one archive per directory; never opened as a database.
@@ -314,10 +301,6 @@ function normaliseService(id, raw, defaults) {
@@ -314,10 +301,6 @@ function normaliseService(id, raw, defaults) {
const kinds = nested ? PROBE_KINDS.filter((k) => k !== 'sum') : PROBE_KINDS;
const kinds = nested ? PROBE_KINDS.filter((k) => k !== 'sum') : PROBE_KINDS;
if (!kinds.includes(p.kind)) throw new InventoryError(`${at}.kind must be one of ${kinds.join(', ')}`);
if (!kinds.includes(p.kind)) throw new InventoryError(`${at}.kind must be one of ${kinds.join(', ')}`);
if (p.kind === 'http-json-count' && !/^http:\/\/(127\.0\.0\.1|localhost|\[::1\])[:/]/.test(p.url || '')) throw new InventoryError(`${at}.url must be a loopback http:// URL`);
if (p.kind === 'http-json-count' && !/^http:\/\/(127\.0\.0\.1|localhost|\[::1\])[:/]/.test(p.url || '')) throw new InventoryError(`${at}.url must be a loopback http:// URL`);
- if (p.kind === 'sqlite-count') {
- abs(p.db, `${at}.db`);
- if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
- }
if (p.kind === 'postgresql-count') {
if (p.kind === 'postgresql-count') {
if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);
if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);
if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
@@ -342,9 +325,8 @@ function normaliseService(id, raw, defaults) {
@@ -342,9 +325,8 @@ function normaliseService(id, raw, defaults) {
if (n.tenants && n.tenants.database != null) {
if (n.tenants && n.tenants.database != null) {
const db = n.tenants.database;
const db = n.tenants.database;
let tenantDb;
let tenantDb;
- if (typeof db === 'string') tenantDb = { engine: 'sqlite', path: abs(db, `${where}.nginx.tenants.database`) };
- else if (db && typeof db === 'object' && !Array.isArray(db) && db.engine === 'postgresql' && PG_DATABASE_RE.test(db.database || '')) tenantDb = { engine: 'postgresql', database: db.database };
- else throw new InventoryError(`${where}.nginx.tenants.database must be a path or { "engine": "postgresql", "database": "ov_host" }`);
+ if (db && typeof db === 'object' && !Array.isArray(db) && db.engine === 'postgresql' && PG_DATABASE_RE.test(db.database || '')) tenantDb = { engine: 'postgresql', database: db.database };
+ else throw new InventoryError(`${where}.nginx.tenants.database must be { "engine": "postgresql", "database": "ov_host" }`);
// A fresh object every time: normalise() must never rewrite the document it was handed
// A fresh object every time: normalise() must never rewrite the document it was handed
// (the example inventory is a shared require() cache, and a drill test normalises it twice).
// (the example inventory is a shared require() cache, and a drill test normalises it twice).
out.nginx = { ...n, tenants: { ...n.tenants, database: tenantDb } };
out.nginx = { ...n, tenants: { ...n.tenants, database: tenantDb } };
@@ -358,7 +340,6 @@ function normaliseService(id, raw, defaults) {
@@ -358,7 +340,6 @@ function normaliseService(id, raw, defaults) {
const ENV_NAME_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
const ENV_NAME_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
const TABLE_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
const TABLE_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
-const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;
/**
/**
* The `drill` block (Wave 22 restore drills): how to start a second, side-effect-free instance of
* The `drill` block (Wave 22 restore drills): how to start a second, side-effect-free instance of
@@ -366,19 +347,11 @@ const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;
@@ -366,19 +347,11 @@ const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;
*
*
* supported false (with a reason) when a second instance cannot run safely
* supported false (with a reason) when a second instance cannot run safely
* port spare loopback port for the drill instance (never the service's own port)
* port spare loopback port for the drill instance (never the service's own port)
- * databases { <database name>: <ENV VAR that points the service at that file> }, or
- * { <database name>: { "env": "DATA_DIR", "dir": true } } when the service takes a data
- * DIRECTORY: the copy lands in {tmp}/data/ (or "dir": "{tmp}/<name>") under its
- * production file name and the env var (optional) points at that directory. Without
- * "env", pair the directory with `bind`.
- * A database with engine postgresql instead names the two env vars that point the
- * service at a database URL: { "url": "DATABASE_URL", "directUrl":
- * "DATABASE_DIRECT_URL" }. The drill creates a scratch database and a scratch role,
- * restores the dump into it, and writes both vars (and VALKEY_URL, which the drill
- * must neutralise itself) into drill.env — the service's own env file is loaded first,
- * so any name the drill does not override keeps its production value.
+ * databases { <database name>: { "url": "DATABASE_URL", "directUrl":
+ * "DATABASE_DIRECT_URL" } }. The drill restores into a scratch database
+ * and role, then points both env vars at it.
* env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}
* env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}
- * and {db:<name>}
+ * in the scratch directory.
* dirs directories to create (owned by the service user) inside {tmp} before the start
* dirs directories to create (owned by the service user) inside {tmp} before the start
* command argv to start the service; default: the production unit's ExecStart
* command argv to start the service; default: the production unit's ExecStart
* unit which of the service's units to start (its ExecStart, WorkingDirectory and
* unit which of the service's units to start (its ExecStart, WorkingDirectory and
@@ -412,54 +385,27 @@ function normaliseDrill(raw, svc, where) {
@@ -412,54 +385,27 @@ function normaliseDrill(raw, svc, where) {
if (!svc.databases.length) throw new InventoryError(`${w}: a drill restores databases, and ${svc.id} declares none`);
if (!svc.databases.length) throw new InventoryError(`${w}: a drill restores databases, and ${svc.id} declares none`);
const dbNames = svc.databases.map((d) => d.name);
const dbNames = svc.databases.map((d) => d.name);
const rawDatabases = raw.databases || {};
const rawDatabases = raw.databases || {};
- if (typeof rawDatabases !== 'object' || Array.isArray(rawDatabases) || !Object.keys(rawDatabases).length) throw new InventoryError(`${w}.databases must map at least one database name to an env var name`);
+ if (typeof rawDatabases !== 'object' || Array.isArray(rawDatabases) || !Object.keys(rawDatabases).length) throw new InventoryError(`${w}.databases must map at least one database name to its URL env var names`);
const databases = {};
const databases = {};
- const inDataDir = new Map();
for (const [name, target] of Object.entries(rawDatabases)) {
for (const [name, target] of Object.entries(rawDatabases)) {
if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);
if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);
- const defined = svc.databases.find((d) => d.name === name);
- if (defined.engine === 'postgresql') {
- if (!target || typeof target !== 'object' || Array.isArray(target)) throw new InventoryError(`${w}.databases.${name} must be { "url": "<NAME>", "directUrl": "<NAME>" } for a postgresql database`);
- if (!ENV_NAME_RE.test(target.url || '')) throw new InventoryError(`${w}.databases.${name}.url must be an env var name`);
- if (!ENV_NAME_RE.test(target.directUrl || '')) throw new InventoryError(`${w}.databases.${name}.directUrl must be an env var name`);
- if (target.url === target.directUrl) throw new InventoryError(`${w}.databases.${name}: url and directUrl must be different env vars`);
- databases[name] = { engine: 'postgresql', url: target.url, directUrl: target.directUrl };
- continue;
- }
- if (typeof target === 'string') {
- if (!ENV_NAME_RE.test(target)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);
- databases[name] = { engine: 'sqlite', env: target, dir: false };
- continue;
- }
- const dir = target && target.dir === true ? '{tmp}/data' : target && target.dir;
- if (!target || typeof target !== 'object' || typeof dir !== 'string') throw new InventoryError(`${w}.databases.${name} must be an env var name or { "env": "<NAME>", "dir": true }`);
- if (!/^\{tmp\}\/[A-Za-z0-9._-]+$/.test(dir)) throw new InventoryError(`${w}.databases.${name}.dir must be true or "{tmp}/<name>"`);
- if (target.env != null && !ENV_NAME_RE.test(target.env)) throw new InventoryError(`${w}.databases.${name}.env must be an env var name`);
- const dest = `${dir}/${path.basename(defined.path)}`;
- if (inDataDir.has(dest)) throw new InventoryError(`${w}.databases: ${name} and ${inDataDir.get(dest)} would both be ${dest}`);
- inDataDir.set(dest, name);
- databases[name] = { engine: 'sqlite', env: target.env || null, dir };
- }
- const dirEnvs = new Map();
- for (const t of Object.values(databases)) {
- if (!t.env) continue;
- const key = t.dir || `file:${t.env}`;
- if (dirEnvs.has(t.env) && dirEnvs.get(t.env) !== key) throw new InventoryError(`${w}.databases: ${t.env} would point at two different places`);
- dirEnvs.set(t.env, key);
+ if (!target || typeof target !== 'object' || Array.isArray(target)) throw new InventoryError(`${w}.databases.${name} must be { "url": "<NAME>", "directUrl": "<NAME>" } for a postgresql database`);
+ if (!ENV_NAME_RE.test(target.url || '')) throw new InventoryError(`${w}.databases.${name}.url must be an env var name`);
+ if (!ENV_NAME_RE.test(target.directUrl || '')) throw new InventoryError(`${w}.databases.${name}.directUrl must be an env var name`);
+ if (target.url === target.directUrl) throw new InventoryError(`${w}.databases.${name}: url and directUrl must be different env vars`);
+ databases[name] = { engine: 'postgresql', url: target.url, directUrl: target.directUrl };
}
}
const env = raw.env || {};
const env = raw.env || {};
for (const [k, v] of Object.entries(env)) {
for (const [k, v] of Object.entries(env)) {
if (!ENV_NAME_RE.test(k)) throw new InventoryError(`${w}.env: "${k}" is not an env var name`);
if (!ENV_NAME_RE.test(k)) throw new InventoryError(`${w}.env: "${k}" is not an env var name`);
if (typeof v !== 'string' || /[\n\r\0]/.test(v)) throw new InventoryError(`${w}.env.${k} must be a one-line string`);
if (typeof v !== 'string' || /[\n\r\0]/.test(v)) throw new InventoryError(`${w}.env.${k} must be a one-line string`);
- for (const m of v.matchAll(PLACEHOLDER_RE)) {
- if (m[1].startsWith('db:') && !Object.keys(rawDatabases).includes(m[1].slice(3))) throw new InventoryError(`${w}.env.${k}: {${m[1]}} is not a restored database`);
- }
+ if (/\{db:[^}]+\}/.test(v)) throw new InventoryError(`${w}.env.${k}: database file placeholders are unsupported; use drill.databases URL settings`);
}
}
if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);
if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);
for (const t of Object.values(databases)) {
for (const t of Object.values(databases)) {
// The drill writes these itself; declaring one in drill.env as well would put two lines with
// The drill writes these itself; declaring one in drill.env as well would put two lines with
// the same name in the file, and which wins is not something an operator should have to know.
// the same name in the file, and which wins is not something an operator should have to know.
- const owned = t.engine === 'postgresql' ? [t.url, t.directUrl] : t.env ? [t.env] : [];
+ const owned = [t.url, t.directUrl];
for (const name of owned) if (name in env) throw new InventoryError(`${w}.env.${name} is already set by drill.databases`);
for (const name of owned) if (name in env) throw new InventoryError(`${w}.env.${name} is already set by drill.databases`);
}
}
const dirs = raw.dirs || [];
const dirs = raw.dirs || [];
@@ -567,7 +513,7 @@ function normalise(raw) {
@@ -567,7 +513,7 @@ function normalise(raw) {
host: raw.host || null,
host: raw.host || null,
stateDir: abs(raw.stateDir || '/var/lib/openvibe-host', 'stateDir'),
stateDir: abs(raw.stateDir || '/var/lib/openvibe-host', 'stateDir'),
backupDir: abs(raw.backupDir || '/var/backups/openvibe', 'backupDir'),
backupDir: abs(raw.backupDir || '/var/backups/openvibe', 'backupDir'),
- // Where the service user's backup worker writes before root takes the copy over (see
+ // Where postgres writes logical dumps before root takes the copies over (see
// commands/backup.js). The same filesystem as backupDir, so the final move is a rename.
// commands/backup.js). The same filesystem as backupDir, so the final move is a rename.
backupStagingDir: abs(raw.backupStagingDir || `${path.normalize(raw.backupDir || '/var/backups/openvibe').replace(/\/+$/, '')}.staging`, 'backupStagingDir'),
backupStagingDir: abs(raw.backupStagingDir || `${path.normalize(raw.backupDir || '/var/backups/openvibe').replace(/\/+$/, '')}.staging`, 'backupStagingDir'),
drillDir: abs(raw.drillDir || '/var/lib/openvibe-drills', 'drillDir'),
drillDir: abs(raw.drillDir || '/var/lib/openvibe-drills', 'drillDir'),
@@ -255,14 +255,12 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {
@@ -255,14 +255,12 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {
/**
/**
* Verified custom domains from the Host database + certificate presence. A PostgreSQL database is
* Verified custom domains from the Host database + certificate presence. A PostgreSQL database is
* read as the postgres OS user over peer auth (Host moved to PostgreSQL; its service role's password
* read as the postgres OS user over peer auth (Host moved to PostgreSQL; its service role's password
- * is not ovhost's to read), a SQLite file as the service user.
+ * is not ovhost's to read).
*/
*/
async function tenantDomains(exec, svc) {
async function tenantDomains(exec, svc) {
const t = tenantsConfig(svc);
const t = tenantsConfig(svc);
if (!t.database) throw new Error(`${svc.id}: nginx.tenants.database is not set in the inventory`);
if (!t.database) throw new Error(`${svc.id}: nginx.tenants.database is not set in the inventory`);
- const rows = t.database.engine === 'postgresql'
- ? await exec.psql(t.database.database, VERIFIED_SQL)
- : await exec.sqlite(t.database.path, VERIFIED_SQL, { as: svc.runAs });
+ const rows = await exec.psql(t.database.database, VERIFIED_SQL);
const out = [];
const out = [];
for (const r of rows || []) {
for (const r of rows || []) {
const h = String(r.hostname || '').toLowerCase();
const h = String(r.hostname || '').toLowerCase();
@@ -322,15 +322,9 @@ async function readManifest(client, cfg, master, host, run) {
@@ -322,15 +322,9 @@ async function readManifest(client, cfg, master, host, run) {
return m;
return m;
}
}
-/** Directories restore-download must never write into: anything holding a declared database or object directory. */
+/** Directories restore-download must never write into. */
function unsafeOut(inv, out) {
function unsafeOut(inv, out) {
for (const svc of Object.values(inv.services)) {
for (const svc of Object.values(inv.services)) {
- for (const db of svc.databases) {
- // A PostgreSQL database is in the cluster, not at a path: it has no directory to avoid.
- if (!db.path) continue;
- const dir = path.dirname(db.path);
- if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;
- }
// An object directory is itself a directory the live service writes: never restore over it.
// An object directory is itself a directory the live service writes: never restore over it.
for (const obj of svc.objects || []) {
for (const obj of svc.objects || []) {
const dir = obj.path;
const dir = obj.path;
@@ -344,8 +338,8 @@ function unsafeOut(inv, out) {
@@ -344,8 +338,8 @@ function unsafeOut(inv, out) {
/**
/**
* Download one service's databases from an off-host run, verify and decrypt them into a new
* Download one service's databases from an off-host run, verify and decrypt them into a new
- * directory (root:root 0700, files 0600), and quick_check each. It never touches a live database: the directory
- * must not exist yet (or be empty), must not overlap any service's checkout or database directory,
+ * directory (root:root 0700, files 0600), and check each dump. It never touches a live database: the directory
+ * must not exist yet (or be empty), must not overlap any service's checkout,
* and every file is created exclusively. Putting a copy in place is a separate, manual step.
* and every file is created exclusively. Putting a copy in place is a separate, manual step.
*/
*/
async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host: hostArg = null } = {}) {
async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host: hostArg = null } = {}) {
@@ -393,27 +387,15 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:
@@ -393,27 +387,15 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:
await exec.removeFile(dest);
await exec.removeFile(dest);
throw new OffsiteError(`${f.file}: the download does not match the manifest's hashes`);
throw new OffsiteError(`${f.file}: the download does not match the manifest's hashes`);
}
}
- // A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which
- // needs no database and is run as the caller — the copy is root-only 0600, and postgres could
- // not read it); a SQLite copy is quick_checked through the worker. An object-directory archive
- // is neither: its own sha256 has already matched the manifest, and a drill is what extracts it
- // and checks every blob.
- let check;
- const isDump = path.extname(f.file) === '.dump';
+ // A database artifact is a pg_dump archive; object archives are checked by a drill.
const isObjects = f.kind === 'objects' || f.file.endsWith('.tar.gz');
const isObjects = f.kind === 'objects' || f.file.endsWith('.tar.gz');
- if (isObjects) {
- check = 'object archive (verify with a drill)';
- } else {
- try {
- if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }
- else {
- const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
- check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
- }
- } catch (err) { check = `error: ${err.message}`; }
+ let check = 'object archive (verify with a drill)';
+ if (!isObjects) {
+ try { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }
+ catch (err) { check = `error: ${err.message}`; }
}
}
- record.files.push({ name: f.name, kind: isObjects ? 'objects' : 'database', file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: isObjects ? null : check });
- if (!isObjects && (isDump ? check.startsWith('error') : check !== 'ok')) throw new OffsiteError(`${dest}: ${check}`);
+ record.files.push({ name: f.name, kind: isObjects ? 'objects' : 'database', file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, check: isObjects ? null : check });
+ if (!isObjects && check.startsWith('error')) throw new OffsiteError(`${dest}: ${check}`);
}
}
return record;
return record;
}
}
@@ -10,7 +10,7 @@
@@ -10,7 +10,7 @@
* unit whose ReadWritePaths is missing)
* unit whose ReadWritePaths is missing)
* checks an argv run in each named package directory as the service user; non-zero stops the
* checks an argv run in each named package directory as the service user; non-zero stops the
* deploy (Tools: the jobs runtime loads in img/audio/docs and the guard in every app, so
* deploy (Tools: the jobs runtime loads in img/audio/docs and the guard in every app, so
- * a native-module ABI mismatch shows up here, not in a crash loop; Games: better-sqlite3
+ * a native-module ABI mismatch shows up here, not in a crash loop
* loads under this Node)
* loads under this Node)
*
*
* run() never restarts or changes a unit. -> { syntaxChecked, dirsMade, checksRun, problems: [text] }
* run() never restarts or changes a unit. -> { syntaxChecked, dirsMade, checksRun, problems: [text] }
@@ -5,13 +5,10 @@
@@ -5,13 +5,10 @@
* http-json-count GET a loopback URL and count: `array` (dotted path) filtered by `where`
* http-json-count GET a loopback URL and count: `array` (dotted path) filtered by `where`
* (key: value equality), or a numeric `field` (dotted path).
* (key: value equality), or a numeric `field` (dotted path).
* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
- * sqlite-count a read-only SELECT against the service's database, run as the service user.
- * Media: SELECT count(*) FROM vods WHERE is_recording = 1.
* postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS
* postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS
* user over peer auth (no secret is read, and a PostgreSQL service's database has
* user over peer auth (no secret is read, and a PostgreSQL service's database has
- * no file for a sqlite-count to open). OpenRestream: ingest sessions in flight.
- * sum the total of `probes` (each an http-json-count, sqlite-count or
- * postgresql-count): Tools' running
+ * no file to open). OpenRestream: ingest sessions in flight.
+ * sum the total of `probes` (each an http-json-count or postgresql-count): Tools' running
* jobs in each satellite's jobs.db. Unknown when any part is.
* jobs in each satellite's jobs.db. Unknown when any part is.
*
*
* A probe that cannot answer while the service is running is UNKNOWN, never zero: "could not ask"
* A probe that cannot answer while the service is running is UNKNOWN, never zero: "could not ask"
@@ -61,7 +58,7 @@ async function countOne(exec, svc, p, label) {
@@ -61,7 +58,7 @@ async function countOne(exec, svc, p, label) {
const arr = at(body, p.array);
const arr = at(body, p.array);
if (!Array.isArray(arr)) return { count: null, label, unknown: `no array at "${p.array}"` };
if (!Array.isArray(arr)) return { count: null, label, unknown: `no array at "${p.array}"` };
const where = p.where || {};
const where = p.where || {};
- // `true` matches any truthy value (SQLite rows say is_live: 1).
+ // `true` matches any truthy value.
const matches = (x) => Object.entries(where).every(([k, v]) => {
const matches = (x) => Object.entries(where).every(([k, v]) => {
const actual = x == null ? undefined : x[k];
const actual = x == null ? undefined : x[k];
return v === true ? Boolean(actual) && actual !== '0' : actual === v;
return v === true ? Boolean(actual) && actual !== '0' : actual === v;
@@ -72,13 +69,6 @@ async function countOne(exec, svc, p, label) {
@@ -72,13 +69,6 @@ async function countOne(exec, svc, p, label) {
if (!Number.isFinite(n)) return { count: null, label, unknown: `no number at "${p.field}"` };
if (!Number.isFinite(n)) return { count: null, label, unknown: `no number at "${p.field}"` };
return { count: n, label };
return { count: n, label };
}
}
- if (p.kind === 'sqlite-count') {
- const rows = await exec.sqlite(p.db, p.sql, { as: svc.runAs });
- const first = rows && rows[0] ? Object.values(rows[0])[0] : undefined;
- const n = Number(first);
- if (!Number.isFinite(n)) return { count: null, label, unknown: 'query returned no number' };
- return { count: n, label };
- }
if (p.kind === 'postgresql-count') {
if (p.kind === 'postgresql-count') {
// As the postgres OS user over peer auth: the service's own role needs a password, which
// As the postgres OS user over peer auth: the service's own role needs a password, which
// ovhost must never read.
// ovhost must never read.
@@ -436,7 +436,7 @@ async function apply(ctx, svc, mode, opts = {}) {
@@ -436,7 +436,7 @@ async function apply(ctx, svc, mode, opts = {}) {
if (opts.prepareOnly) throw new OpError(`--prepare-only needs a release-layout service; ${svc.id} is ${svc.strategy}`, EXIT.USAGE);
if (opts.prepareOnly) throw new OpError(`--prepare-only needs a release-layout service; ${svc.id} is ${svc.strategy}`, EXIT.USAGE);
// An in-place strategy never guesses its way through a releases/ + current layout.
// An in-place strategy never guesses its way through a releases/ + current layout.
if (await exec.readlink(path.join(svc.repo, 'current'))) {
if (await exec.readlink(path.join(svc.repo, 'current'))) {
- throw new OpError(`${svc.repo} uses the releases/current layout; set strategy "release-layout" for ${svc.id} (docs/deploy-strategies.md) or use the repository's own deploy script`, EXIT.USAGE);
+ throw new OpError(`${svc.repo} uses the releases/current layout; set strategy "release-layout" for ${svc.id} (docs/deploy-strategies.md)`, EXIT.USAGE);
}
}
const held = await lock.acquire(exec, inv, svc.id);
const held = await lock.acquire(exec, inv, svc.id);
if (held.recoveredStale) log(`removed a stale lock (${held.file})`);
if (held.recoveredStale) log(`removed a stale lock (${held.file})`);
deleted file mode 100644
deleted file mode 100644
@@ -1,64 +0,0 @@
@@ -1,64 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * Runs one SQLite operation in its own process so the executor can start it AS THE SERVICE USER
- * (runuser/sudo -u). Root never opens a service's database: a root-owned -wal/-shm file left under
- * /opt would stop the service from writing its own database.
- *
- * node sqlite-worker.js '{"op":"query","db":"/opt/x/data/x.db","sql":"SELECT count(*) AS n FROM t"}'
- * node sqlite-worker.js '{"op":"backup","db":"/opt/x/data/x.db","dest":"/var/backups/openvibe/x/…/x.db"}'
- *
- * The source database is always opened read-only (fileMustExist); a query that tries to write fails.
- * Prints one JSON line: { ok: true, rows } | { ok: true, bytes } | { ok: false, error }.
- *
- * Backup copies hold user data, so the worker runs with umask 077 and leaves exactly one file,
- * mode 0600: the copy is switched to journal_mode=DELETE (a WAL source gives a WAL-mode copy), which
- * checkpoints and removes its -wal/-shm, and any side file still present is deleted.
- */
-const fs = require('fs');
-
-process.umask(0o077);
-
-function done(obj, code = 0) {
- process.stdout.write(`${JSON.stringify(obj)}\n`);
- process.exit(code);
-}
-
-let req;
-try { req = JSON.parse(process.argv[2] || ''); } catch { done({ ok: false, error: 'bad request' }, 2); }
-
-let Database;
-try { Database = require('better-sqlite3'); } catch (err) { done({ ok: false, error: `better-sqlite3 unavailable: ${err.message}` }, 2); }
-
-let db;
-try {
- db = new Database(req.db, { readonly: true, fileMustExist: true });
- db.pragma('busy_timeout = 5000');
-} catch (err) {
- done({ ok: false, error: `cannot open ${req.db}: ${err.message}` }, 1);
-}
-
-if (req.op === 'query') {
- try {
- const stmt = db.prepare(req.sql);
- if (!stmt.reader) done({ ok: false, error: 'only read-only statements are allowed' }, 1);
- done({ ok: true, rows: stmt.all() });
- } catch (err) {
- done({ ok: false, error: err.message }, 1);
- }
-} else if (req.op === 'backup') {
- if (!req.dest) done({ ok: false, error: 'dest required' }, 2);
- if (fs.existsSync(req.dest)) done({ ok: false, error: `refusing to overwrite ${req.dest}` }, 1);
- db.backup(req.dest).then(() => {
- const check = new Database(req.dest);
- check.pragma('journal_mode = DELETE');
- const result = check.pragma('quick_check', { simple: true });
- check.close();
- for (const side of ['-wal', '-shm', '-journal']) fs.rmSync(`${req.dest}${side}`, { force: true });
- fs.chmodSync(req.dest, 0o600);
- if (result !== 'ok') done({ ok: false, error: `backup failed quick_check: ${result}` }, 1);
- done({ ok: true, bytes: fs.statSync(req.dest).size });
- }, (err) => done({ ok: false, error: err.message }, 1));
-} else {
- done({ ok: false, error: `unknown op ${req.op}` }, 2);
-}
@@ -10,7 +10,6 @@
@@ -10,7 +10,6 @@
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"@aws-sdk/client-s3": "^3.1138.0",
"@aws-sdk/client-s3": "^3.1138.0",
- "better-sqlite3": "^11.10.0",
"busboy": "^1.6.0",
"busboy": "^1.6.0",
"cookie-parser": "^1.4.7",
"cookie-parser": "^1.4.7",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
@@ -487,57 +486,6 @@
@@ -487,57 +486,6 @@
"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==",
"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==",
"license": "MIT"
"license": "MIT"
},
},
- "node_modules/base64-js": {
- "version": "1.5.1",
- "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
- "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
- "funding": [
- {
- "type": "github",
- "url": "https://github.com/sponsors/feross"
- },
- {
- "type": "patreon",
- "url": "https://www.patreon.com/feross"
- },
- {
- "type": "consulting",
- "url": "https://feross.org/support"
- }
- ],
- "license": "MIT"
- },
- "node_modules/better-sqlite3": {
- "version": "11.10.0",
- "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-11.10.0.tgz",
- "integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==",
- "hasInstallScript": true,
- "license": "MIT",
- "dependencies": {
- "bindings": "^1.5.0",
- "prebuild-install": "^7.1.1"
- }
- },
- "node_modules/bindings": {
- "version": "1.5.0",
- "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz",
- "integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==",
- "license": "MIT",
- "dependencies": {
- "file-uri-to-path": "1.0.0"
- }
- },
- "node_modules/bl": {
- "version": "4.1.0",
- "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
- "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
- "license": "MIT",
- "dependencies": {
- "buffer": "^5.5.0",
- "inherits": "^2.0.4",
- "readable-stream": "^3.4.0"
- }
- },
"node_modules/body-parser": {
"node_modules/body-parser": {
"version": "1.20.8",
"version": "1.20.8",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",
@@ -568,30 +516,6 @@
@@ -568,30 +516,6 @@
"integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==",
"integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==",
"license": "MIT"
"license": "MIT"
},
},
- "node_modules/buffer": {
- "version": "5.7.1",
- "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
- "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",
- "funding": [
- {
- "type": "github",
- "url": "https://github.com/sponsors/feross"
- },
- {
- "type": "patreon",
- "url": "https://www.patreon.com/feross"
- },
- {
- "type": "consulting",
- "url": "https://feross.org/support"
- }
- ],
- "license": "MIT",
- "dependencies": {
- "base64-js": "^1.3.1",
- "ieee754": "^1.1.13"
- }
- },
"node_modules/buffer-equal-constant-time": {
"node_modules/buffer-equal-constant-time": {
"version": "1.0.1",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
"resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
@@ -647,12 +571,6 @@
@@ -647,12 +571,6 @@
"url": "https://github.com/sponsors/ljharb"
"url": "https://github.com/sponsors/ljharb"
}
}
},
},
- "node_modules/chownr": {
- "version": "1.1.4",
- "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
- "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
- "license": "ISC"
- },
"node_modules/cluster-key-slot": {
"node_modules/cluster-key-slot": {
"version": "1.1.2",
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",
"resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",
@@ -720,30 +638,6 @@
@@ -720,30 +638,6 @@
"ms": "2.0.0"
"ms": "2.0.0"
}
}
},
},
- "node_modules/decompress-response": {
- "version": "6.0.0",
- "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",
- "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",
- "license": "MIT",
- "dependencies": {
- "mimic-response": "^3.1.0"
- },
- "engines": {
- "node": ">=10"
- },
- "funding": {
- "url": "https://github.com/sponsors/sindresorhus"
- }
- },
- "node_modules/deep-extend": {
- "version": "0.6.0",
- "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",
- "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",
- "license": "MIT",
- "engines": {
- "node": ">=4.0.0"
- }
- },
"node_modules/denque": {
"node_modules/denque": {
"version": "2.1.0",
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",
"resolved": "https://registry.npmjs.org/denque/-/denque-2.1.0.tgz",
@@ -772,15 +666,6 @@
@@ -772,15 +666,6 @@
"npm": "1.2.8000 || >= 1.4.16"
"npm": "1.2.8000 || >= 1.4.16"
}
}
},
},
- "node_modules/detect-libc": {
- "version": "2.1.2",
- "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
- "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
- "license": "Apache-2.0",
- "engines": {
- "node": ">=8"
- }
- },
"node_modules/dotenv": {
"node_modules/dotenv": {
"version": "16.6.1",
"version": "16.6.1",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz",
@@ -831,15 +716,6 @@
@@ -831,15 +716,6 @@
"node": ">= 0.8"
"node": ">= 0.8"
}
}
},
},
- "node_modules/end-of-stream": {
- "version": "1.4.5",
- "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
- "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
- "license": "MIT",
- "dependencies": {
- "once": "^1.4.0"
- }
- },
"node_modules/es-define-property": {
"node_modules/es-define-property": {
"version": "1.0.1",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
"resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
@@ -885,15 +761,6 @@
@@ -885,15 +761,6 @@
"node": ">= 0.6"
"node": ">= 0.6"
}
}
},
},
- "node_modules/expand-template": {
- "version": "2.0.3",
- "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",
- "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",
- "license": "(MIT OR WTFPL)",
- "engines": {
- "node": ">=6"
- }
- },
"node_modules/express": {
"node_modules/express": {
"version": "4.22.3",
"version": "4.22.3",
"resolved": "https://registry.npmjs.org/express/-/express-4.22.3.tgz",
"resolved": "https://registry.npmjs.org/express/-/express-4.22.3.tgz",
@@ -977,12 +844,6 @@
@@ -977,12 +844,6 @@
],
],
"license": "BSD-3-Clause"
"license": "BSD-3-Clause"
},
},
- "node_modules/file-uri-to-path": {
- "version": "1.0.0",
- "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz",
- "integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==",
- "license": "MIT"
- },
"node_modules/finalhandler": {
"node_modules/finalhandler": {
"version": "1.3.2",
"version": "1.3.2",
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz",
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz",
@@ -1019,12 +880,6 @@
@@ -1019,12 +880,6 @@
"node": ">= 0.6"
"node": ">= 0.6"
}
}
},
},
- "node_modules/fs-constants": {
- "version": "1.0.0",
- "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
- "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",
- "license": "MIT"
- },
"node_modules/function-bind": {
"node_modules/function-bind": {
"version": "1.1.2",
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
@@ -1071,12 +926,6 @@
@@ -1071,12 +926,6 @@
"node": ">= 0.4"
"node": ">= 0.4"
}
}
},
},
- "node_modules/github-from-package": {
- "version": "0.0.0",
- "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",
- "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",
- "license": "MIT"
- },
"node_modules/gopd": {
"node_modules/gopd": {
"version": "1.2.0",
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
@@ -1145,38 +994,12 @@
@@ -1145,38 +994,12 @@
"node": ">=0.10.0"
"node": ">=0.10.0"
}
}
},
},
- "node_modules/ieee754": {
- "version": "1.2.1",
- "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
- "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
- "funding": [
- {
- "type": "github",
- "url": "https://github.com/sponsors/feross"
- },
- {
- "type": "patreon",
- "url": "https://www.patreon.com/feross"
- },
- {
- "type": "consulting",
- "url": "https://feross.org/support"
- }
- ],
- "license": "BSD-3-Clause"
- },
"node_modules/inherits": {
"node_modules/inherits": {
"version": "2.0.4",
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
"license": "ISC"
"license": "ISC"
},
},
- "node_modules/ini": {
- "version": "1.3.8",
- "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
- "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
- "license": "ISC"
- },
"node_modules/iovalkey": {
"node_modules/iovalkey": {
"version": "0.4.0",
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/iovalkey/-/iovalkey-0.4.0.tgz",
"resolved": "https://registry.npmjs.org/iovalkey/-/iovalkey-0.4.0.tgz",
@@ -1407,45 +1230,12 @@
@@ -1407,45 +1230,12 @@
"node": ">= 0.6"
"node": ">= 0.6"
}
}
},
},
- "node_modules/mimic-response": {
- "version": "3.1.0",
- "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",
- "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",
- "license": "MIT",
- "engines": {
- "node": ">=10"
- },
- "funding": {
- "url": "https://github.com/sponsors/sindresorhus"
- }
- },
- "node_modules/minimist": {
- "version": "1.2.8",
- "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
- "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",
- "license": "MIT",
- "funding": {
- "url": "https://github.com/sponsors/ljharb"
- }
- },
- "node_modules/mkdirp-classic": {
- "version": "0.5.3",
- "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
- "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",
- "license": "MIT"
- },
"node_modules/ms": {
"node_modules/ms": {
"version": "2.0.0",
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
"license": "MIT"
"license": "MIT"
},
},
- "node_modules/napi-build-utils": {
- "version": "2.0.0",
- "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",
- "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",
- "license": "MIT"
- },
"node_modules/negotiator": {
"node_modules/negotiator": {
"version": "0.6.3",
"version": "0.6.3",
"resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",
"resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",
@@ -1455,18 +1245,6 @@
@@ -1455,18 +1245,6 @@
"node": ">= 0.6"
"node": ">= 0.6"
}
}
},
},
- "node_modules/node-abi": {
- "version": "3.96.0",
- "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz",
- "integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==",
- "license": "MIT",
- "dependencies": {
- "semver": "^7.3.5"
- },
- "engines": {
- "node": ">=10"
- }
- },
"node_modules/object-inspect": {
"node_modules/object-inspect": {
"version": "1.13.4",
"version": "1.13.4",
"resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
"resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
@@ -1491,15 +1269,6 @@
@@ -1491,15 +1269,6 @@
"node": ">= 0.8"
"node": ">= 0.8"
}
}
},
},
- "node_modules/once": {
- "version": "1.4.0",
- "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
- "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
- "license": "ISC",
- "dependencies": {
- "wrappy": "1"
- }
- },
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
"version": "0.127.0",
"version": "0.127.0",
"resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.127.0",
"resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.127.0",
@@ -1700,33 +1469,6 @@
@@ -1700,33 +1469,6 @@
"node": ">=0.10.0"
"node": ">=0.10.0"
}
}
},
},
- "node_modules/prebuild-install": {
- "version": "7.1.3",
- "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
- "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",
- "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",
- "license": "MIT",
- "dependencies": {
- "detect-libc": "^2.0.0",
- "expand-template": "^2.0.3",
- "github-from-package": "0.0.0",
- "minimist": "^1.2.3",
- "mkdirp-classic": "^0.5.3",
- "napi-build-utils": "^2.0.0",
- "node-abi": "^3.3.0",
- "pump": "^3.0.0",
- "rc": "^1.2.7",
- "simple-get": "^4.0.0",
- "tar-fs": "^2.0.0",
- "tunnel-agent": "^0.6.0"
- },
- "bin": {
- "prebuild-install": "bin.js"
- },
- "engines": {
- "node": ">=10"
- }
- },
"node_modules/proxy-addr": {
"node_modules/proxy-addr": {
"version": "2.0.8",
"version": "2.0.8",
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",
@@ -1744,16 +1486,6 @@
@@ -1744,16 +1486,6 @@
"url": "https://opencollective.com/express"
"url": "https://opencollective.com/express"
}
}
},
},
- "node_modules/pump": {
- "version": "3.0.4",
- "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",
- "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",
- "license": "MIT",
- "dependencies": {
- "end-of-stream": "^1.1.0",
- "once": "^1.3.1"
- }
- },
"node_modules/qs": {
"node_modules/qs": {
"version": "6.16.0",
"version": "6.16.0",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",
@@ -1794,35 +1526,6 @@
@@ -1794,35 +1526,6 @@
"node": ">= 0.8"
"node": ">= 0.8"
}
}
},
},
- "node_modules/rc": {
- "version": "1.2.8",
- "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",
- "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",
- "license": "(BSD-2-Clause OR MIT OR Apache-2.0)",
- "dependencies": {
- "deep-extend": "^0.6.0",
- "ini": "~1.3.0",
- "minimist": "^1.2.0",
- "strip-json-comments": "~2.0.1"
- },
- "bin": {
- "rc": "cli.js"
- }
- },
- "node_modules/readable-stream": {
- "version": "3.6.2",
- "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
- "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
- "license": "MIT",
- "dependencies": {
- "inherits": "^2.0.3",
- "string_decoder": "^1.1.1",
- "util-deprecate": "^1.0.1"
- },
- "engines": {
- "node": ">= 6"
- }
- },
"node_modules/redis-errors": {
"node_modules/redis-errors": {
"version": "1.2.0",
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",
"resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",
@@ -2014,51 +1717,6 @@
@@ -2014,51 +1717,6 @@
"url": "https://github.com/sponsors/ljharb"
"url": "https://github.com/sponsors/ljharb"
}
}
},
},
- "node_modules/simple-concat": {
- "version": "1.0.1",
- "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz",
- "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==",
- "funding": [
- {
- "type": "github",
- "url": "https://github.com/sponsors/feross"
- },
- {
- "type": "patreon",
- "url": "https://www.patreon.com/feross"
- },
- {
- "type": "consulting",
- "url": "https://feross.org/support"
- }
- ],
- "license": "MIT"
- },
- "node_modules/simple-get": {
- "version": "4.0.1",
- "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz",
- "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==",
- "funding": [
- {
- "type": "github",
- "url": "https://github.com/sponsors/feross"
- },
- {
- "type": "patreon",
- "url": "https://www.patreon.com/feross"
- },
- {
- "type": "consulting",
- "url": "https://feross.org/support"
- }
- ],
- "license": "MIT",
- "dependencies": {
- "decompress-response": "^6.0.0",
- "once": "^1.3.1",
- "simple-concat": "^1.0.0"
- }
- },
"node_modules/split2": {
"node_modules/split2": {
"version": "4.2.0",
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
@@ -2091,52 +1749,6 @@
@@ -2091,52 +1749,6 @@
"node": ">=10.0.0"
"node": ">=10.0.0"
}
}
},
},
- "node_modules/string_decoder": {
- "version": "1.3.0",
- "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
- "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
- "license": "MIT",
- "dependencies": {
- "safe-buffer": "~5.2.0"
- }
- },
- "node_modules/strip-json-comments": {
- "version": "2.0.1",
- "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
- "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",
- "license": "MIT",
- "engines": {
- "node": ">=0.10.0"
- }
- },
- "node_modules/tar-fs": {
- "version": "2.1.5",
- "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",
- "integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",
- "license": "MIT",
- "dependencies": {
- "chownr": "^1.1.1",
- "mkdirp-classic": "^0.5.2",
- "pump": "^3.0.0",
- "tar-stream": "^2.1.4"
- }
- },
- "node_modules/tar-stream": {
- "version": "2.2.0",
- "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
- "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
- "license": "MIT",
- "dependencies": {
- "bl": "^4.0.3",
- "end-of-stream": "^1.4.1",
- "fs-constants": "^1.0.0",
- "inherits": "^2.0.3",
- "readable-stream": "^3.1.1"
- },
- "engines": {
- "node": ">=6"
- }
- },
"node_modules/toidentifier": {
"node_modules/toidentifier": {
"version": "1.0.1",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
"resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
@@ -2152,18 +1764,6 @@
@@ -2152,18 +1764,6 @@
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"license": "0BSD"
"license": "0BSD"
},
},
- "node_modules/tunnel-agent": {
- "version": "0.6.0",
- "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
- "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",
- "license": "Apache-2.0",
- "dependencies": {
- "safe-buffer": "^5.0.1"
- },
- "engines": {
- "node": "*"
- }
- },
"node_modules/type-is": {
"node_modules/type-is": {
"version": "1.6.18",
"version": "1.6.18",
"resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
"resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
@@ -2186,12 +1786,6 @@
@@ -2186,12 +1786,6 @@
"node": ">= 0.8"
"node": ">= 0.8"
}
}
},
},
- "node_modules/util-deprecate": {
- "version": "1.0.2",
- "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
- "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
- "license": "MIT"
- },
"node_modules/utils-merge": {
"node_modules/utils-merge": {
"version": "1.0.1",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
"resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
@@ -2210,12 +1804,6 @@
@@ -2210,12 +1804,6 @@
"node": ">= 0.8"
"node": ">= 0.8"
}
}
},
},
- "node_modules/wrappy": {
- "version": "1.0.2",
- "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
- "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
- "license": "ISC"
- },
"node_modules/xtend": {
"node_modules/xtend": {
"version": "4.0.2",
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
@@ -26,7 +26,6 @@
@@ -26,7 +26,6 @@
"repository": "github:OpenVibers/OpenVibe.Host",
"repository": "github:OpenVibers/OpenVibe.Host",
"dependencies": {
"dependencies": {
"@aws-sdk/client-s3": "^3.1138.0",
"@aws-sdk/client-s3": "^3.1138.0",
- "better-sqlite3": "^11.10.0",
"busboy": "^1.6.0",
"busboy": "^1.6.0",
"cookie-parser": "^1.4.7",
"cookie-parser": "^1.4.7",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
@@ -17,21 +17,9 @@ sudo roles/data/provision.sh --check # what would change
@@ -17,21 +17,9 @@ sudo roles/data/provision.sh --check # what would change
sudo roles/data/provision.sh # apply (idempotent)
sudo roles/data/provision.sh # apply (idempotent)
sudo roles/data/add-service.sh wiki # database ov_wiki, roles ov_wiki / ov_wiki_app, Valkey user ov_svc_wiki,
sudo roles/data/add-service.sh wiki # database ov_wiki, roles ov_wiki / ov_wiki_app, Valkey user ov_svc_wiki,
# and DATABASE_URL, DATABASE_DIRECT_URL, VALKEY_URL, VALKEY_PREFIX in /etc/openvibe/wiki.env
# and DATABASE_URL, DATABASE_DIRECT_URL, VALKEY_URL, VALKEY_PREFIX in /etc/openvibe/wiki.env
-sudo roles/data/switch-service.sh news # the production switch from SQLite: stop, back up, import and verify, deploy
- # (after the postgres branch is merged with green CI and auto-deploy is frozen)
```
```
-The same two steps from the installed CLI (the agent pipeline may run only `ovhost` through its broker;
-both run these scripts from the ovhost install, pass their exit code through and redact every URL and
-password; [docs/data-provisioning.md](../../docs/data-provisioning.md)):
-
-```bash
-sudo ovhost data provision wiki # == sudo roles/data/add-service.sh wiki
-sudo ovhost data switch news [--sqlite <file>] # == sudo roles/data/switch-service.sh news [<file>]
-```
-
-`switch-service.sh` runs the deployed release's `scripts/migrate-to-postgres.js`: a service without that
-file cannot be switched this way.
+The installed CLI runs `add-service.sh` through `sudo ovhost data provision wiki`. It passes the role's exit code and redacts URLs and passwords; see [data provisioning](../../docs/data-provisioning.md).
Generated passwords live only in `/etc/openvibe/data.env` (root, 0600). PostgreSQL receives SCRAM secrets and
Generated passwords live only in `/etc/openvibe/data.env` (root, 0600). PostgreSQL receives SCRAM secrets and
Valkey receives SHA-256 hashes, so no plaintext is ever in SQL, logs, a process list or the scripts' output.
Valkey receives SHA-256 hashes, so no plaintext is ever in SQL, logs, a process list or the scripts' output.
deleted file mode 100755
deleted file mode 100755
@@ -1,62 +0,0 @@
@@ -1,62 +0,0 @@
-#!/usr/bin/env bash
-# roles/data/switch-service.sh — the production switch of one service from its SQLite file to PostgreSQL (ADR-035).
-#
-# sudo roles/data/switch-service.sh <service> [<sqlite file>]
-#
-# Before: the service's postgres branch is merged to main with green CI, its auto-deploy is frozen
-# (ovhost freeze <svc> --reason …), and add-service.sh <svc> has written its database settings.
-# It clones main as ubuntu into /var/tmp/pg-<svc> and installs it; waits until the service's protected sessions are
-# idle (two checks a minute apart); stops the unit; keeps a read-only
-# <svc>.pre-postgres-<time>.db backup next to the file; runs that release's scripts/migrate-to-postgres.js
-# (openvibe-sdk runSqliteMigration: migrate, import, verify counts and checksums) with the unit's
-# environment, as ubuntu; on any failure it starts the old release again and stops. Then it unfreezes and
-# deploys (ovhost deploy), and makes the SQLite file read-only: it is the 7-day rollback (compatibility
-# register C-89: sudo ovhost rollback <svc> to the release before, which reads the file).
-#
-# A service that is not one unit in /opt/openvibe.<svc> (OpenRestream: two units plus per-release transport workers, a
-# release layout) sets, in the environment:
-# SWITCH_UNITS="a.service b.service 'worker@*.service'" every unit to stop (the first one's environment runs the
-# import); a pattern stops every running instance
-# SWITCH_DIR=/opt/openre.stream the checkout (a release layout's origin is in repo/)
-# SWITCH_AFTER="/opt/openre.stream/current/deploy/scripts/deploy.sh workers" run after the deploy (as root)
-set -euo pipefail
-SVC=$1; UNIT=openvibe-$SVC.service; DIR=${SWITCH_DIR:-/opt/openvibe.$SVC}; DATA=/var/lib/openvibe-$SVC; DB=$DATA/$SVC.db
-read -r -a UNITS <<< "${SWITCH_UNITS:-$UNIT}"; UNIT=${UNITS[0]}
-# A second argument names the SQLite file when it is not /var/lib/openvibe-<svc>/<svc>.db (Host: /var/lib/openvibe-host-api/host.db).
-if [ -n "${2:-}" ]; then DB=$2; DATA=$(dirname "$DB"); fi
-WORK=/var/tmp/pg-$SVC; UB=$(id -u ubuntu); GB=$(id -g ubuntu)
-[ -f "$DB" ] || { echo "no $DB"; exit 1; }
-rm -rf "$WORK"; install -d -o ubuntu -g ubuntu "$WORK"
-GITDIR=$DIR; [ -d "$DIR/repo/.git" ] || [ -f "$DIR/repo/HEAD" ] && GITDIR=$DIR/repo
-REMOTE=$(git -C "$GITDIR" remote get-url origin)
-setpriv --reuid=$UB --regid=$GB --init-groups env HOME=/home/ubuntu bash -c "cd $WORK && git clone -q --depth 1 --branch main '$REMOTE' src && cd src && npm ci --omit=dev --no-audit --no-fund 2>&1 | tail -1 && git log --oneline -1"
-# Protected sessions (the inventory's probe: Media's recordings, Live's streams) would be cut by the stop: wait for two
-# idle checks a minute apart, as a deploy's --wait-idle does, for up to 12 hours. A probe that cannot answer counts as busy.
-quiet=0
-for _ in $(seq 1 720); do
- n=$(ovhost status "$SVC" --json | node -e 'const r = JSON.parse(require("fs").readFileSync(0, "utf8"))[0] || {}; const p = r.protected; process.stdout.write(!p ? "none" : p.count == null ? "unknown" : String(p.count));')
- [ "$n" = none ] && break
- if [ "$n" = 0 ]; then quiet=$((quiet + 1)); [ $quiet -ge 2 ] && break; else quiet=0; echo "[switch] waiting: $n protected session(s)"; fi
- sleep 60
-done
-[ "$n" = none ] || [ $quiet -ge 2 ] || { echo "[switch] still busy after 12 hours: not switching"; rm -rf "$WORK"; exit 1; }
-echo "[switch] stopping ${UNITS[*]}"; systemctl stop "${UNITS[@]}"
-sqlite3 "$DB" ".backup $WORK/source.db"; chown ubuntu:ubuntu "$WORK/source.db"
-TS=$(date -u +%Y%m%dT%H%M%SZ); cp "$WORK/source.db" "$DATA/$SVC.pre-postgres-$TS.db"; chmod 0400 "$DATA/$SVC.pre-postgres-$TS.db"; echo "[switch] backup $DATA/$SVC.pre-postgres-$TS.db"
-echo "[switch] importing"
-# The unit's own environment, parsed by systemd (EnvironmentFile= and Environment=): sourcing the file with bash would
-# mangle values systemd reads fine (JSON, quotes). As ubuntu, in the release just installed.
-SETENV=(); while IFS= read -r kv; do [ -n "$kv" ] && SETENV+=(-E "$kv"); done < <(systemctl show "$UNIT" -p Environment --value | tr ' ' '\n')
-ENVFILES=(); for f in $(systemctl show "$UNIT" -p EnvironmentFiles --value | grep -o '^[^ ]*\|; [^ ]*' | tr -d '; '); do ENVFILES+=(-p "EnvironmentFile=$f"); done
-[ ${#ENVFILES[@]} -gt 0 ] || ENVFILES=(-p "EnvironmentFile=/etc/openvibe/$SVC.env")
-if ! systemd-run --quiet --wait --pipe --collect --uid="$UB" --gid="$GB" -p WorkingDirectory="$WORK/src" -E HOME=/home/ubuntu "${ENVFILES[@]}" "${SETENV[@]}" \
- "$(command -v node)" scripts/migrate-to-postgres.js --sqlite "$WORK/source.db" 2>&1 | grep -v " 0 rows"; then
- echo "[switch] IMPORT FAILED: starting the old release again"; systemctl start "${UNITS[@]}" 2>/dev/null || systemctl start "$UNIT"; exit 1
-fi
-echo "[switch] deploying"
-ovhost unfreeze "$SVC" >/dev/null
-ovhost deploy "$SVC" 2>&1 | tail -3
-if [ -n "${SWITCH_AFTER:-}" ]; then echo "[switch] $SWITCH_AFTER"; bash -c "$SWITCH_AFTER" 2>&1 | tail -5; fi
-chmod 0400 "$DB"; [ -f "$DB-wal" ] && chmod 0400 "$DB-wal"; [ -f "$DB-shm" ] && chmod 0400 "$DB-shm"
-rm -rf "$WORK"
-echo "[switch] done"
deleted file mode 100644
deleted file mode 100644
@@ -1,26 +0,0 @@
@@ -1,26 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * The one-time move of Host's SQLite database (HOST_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
- * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
- *
- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
- *
- * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
- * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
- * name and columns.
- */
-require('dotenv').config();
-const { runSqliteMigration } = require('openvibe-sdk/db');
-const configLib = require('../server/config');
-const { MIGRATIONS } = require('../server/db');
-
-const TABLES = {};
-
-if (require.main === module) {
- const config = (configLib.load || configLib.loadConfig)();
- runSqliteMigration({ service: 'host', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES })
- .then((code) => process.exit(code), (err) => { console.error(`migrate-to-postgres failed: ${err.message}`); process.exit(1); });
-}
-
-module.exports = { TABLES };
@@ -69,7 +69,7 @@ const DASHBOARD_CSP = {
@@ -69,7 +69,7 @@ const DASHBOARD_CSP = {
const DASHBOARD_CSP_HEADER = Object.entries(DASHBOARD_CSP).map(([k, v]) => `${k} ${v.join(' ')}`).join('; ');
const DASHBOARD_CSP_HEADER = Object.entries(DASHBOARD_CSP).map(([k, v]) => `${k} ${v.join(' ')}`).join('; ');
/**
/**
- * opts: config, store | dbPath, now (clock), fetchImpl, auth (an openvibe-sdk/sso client),
+ * opts: config, store, now (clock), fetchImpl, auth (an openvibe-sdk/sso client),
* resolver ({ resolveTxt }), log
* resolver ({ resolveTxt }), log
*/
*/
async function createApp(opts = {}) {
async function createApp(opts = {}) {
@@ -49,8 +49,6 @@ function load(env = process.env) {
@@ -49,8 +49,6 @@ function load(env = process.env) {
// point it at a temp directory so a spawned server gets a database of its own.
// point it at a temp directory so a spawned server gets a database of its own.
db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.HOST_PGLITE_DIR || '' },
db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.HOST_PGLITE_DIR || '' },
valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:host:' },
valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:host:' },
- // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
- dbPath: env.HOST_DB_PATH || './data/host.db',
// Content-addressed deploy objects, one directory per project (tenancy keyed by project id).
// Content-addressed deploy objects, one directory per project (tenancy keyed by project id).
storageDir: env.HOST_STORAGE_DIR || './data/objects',
storageDir: env.HOST_STORAGE_DIR || './data/objects',
@@ -55,7 +55,7 @@ async function openDb(config, { log = console, registry } = {}) {
@@ -55,7 +55,7 @@ async function openDb(config, { log = console, registry } = {}) {
}
}
/**
/**
- * Every store on a migrated database handle. opts.now — injectable clock (epoch ms), so tests and replays are
+ * Every store on a database handle. opts.now — injectable clock (epoch ms), so tests and replays are
* deterministic. store.tx(fn) is a transaction; inside it, plain db calls join it (ambient).
* deterministic. store.tx(fn) is a transaction; inside it, plain db calls join it (ambient).
*/
*/
function createStore(db, { now = () => Date.now() } = {}) {
function createStore(db, { now = () => Date.now() } = {}) {
@@ -15,7 +15,7 @@ const { app, ctx } = await createApp();
@@ -15,7 +15,7 @@ const { app, ctx } = await createApp();
const { config } = ctx;
const { config } = ctx;
const server = app.listen(config.port, config.host, () => {
const server = app.listen(config.port, config.host, () => {
- console.log(`[Host] ${config.nodeEnv} on http://${config.host}:${config.port} → dashboard ${config.baseUrl}, sites *.${config.sitesDomain} (db ${config.dbPath}, objects ${ctx.blobs.root})`);
+ console.log(`[Host] ${config.nodeEnv} on http://${config.host}:${config.port} → dashboard ${config.baseUrl}, sites *.${config.sitesDomain} (db ${config.db.url ? 'PostgreSQL' : 'embedded PGlite'}, objects ${ctx.blobs.root})`);
console.log(`[Host] events relay ${ctx.outbox.enabled ? `on → ${config.events.url}` : 'off (events wait in event_outbox)'}; worker ${config.worker.enabled ? 'on' : 'off'}`);
console.log(`[Host] events relay ${ctx.outbox.enabled ? `on → ${config.events.url}` : 'off (events wait in event_outbox)'}; worker ${config.worker.enabled ? 'on' : 'off'}`);
});
});
server.keepAliveTimeout = 65_000;
server.keepAliveTimeout = 65_000;
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
/**
/**
* Truthful readiness for GET /api/ready (openvibe-shared/ready, Track O).
* Truthful readiness for GET /api/ready (openvibe-shared/ready, Track O).
*
*
- * db required a real query on Host's SQLite (the charter tables answer)
+ * db required a real query on Host's PostgreSQL database (the charter tables answer)
* storage required the object store directory is writable (uploads and serving need it)
* storage required the object store directory is writable (uploads and serving need it)
* network_jwks optional the Network signing key has loaded; without it tenant sites still
* network_jwks optional the Network signing key has loaded; without it tenant sites still
* serve, but nobody can sign in and service tokens are refused (503)
* serve, but nobody can sign in and service tokens are refused (503)
@@ -1,8 +1,7 @@
@@ -1,8 +1,7 @@
'use strict';
'use strict';
/**
/**
- * ovhost archive (WS-S task 6, H14): dead and backup database files are encrypted and copied off-host
- * under <prefix>-archive/, checked, listed and restored, and never deleted locally. Refused: live
- * (inventory) databases, open files, symlinks, paths outside the allowed roots, relative paths.
+ * ovhost archive: retired data files are encrypted and copied off-host
+ * under <prefix>-archive/, checked, listed and restored, and never deleted locally. Refused: open files, symlinks, paths outside the allowed roots, relative paths.
* Nothing leaves the process (an in-memory S3).
* Nothing leaves the process (an in-memory S3).
*/
*/
const assert = require('assert');
const assert = require('assert');
@@ -11,8 +10,6 @@ const { scenario, test, runTests, SECRET } = require('./helpers');
@@ -11,8 +10,6 @@ const { scenario, test, runTests, SECRET } = require('./helpers');
const { main } = require('../lib/cli');
const { main } = require('../lib/cli');
const { mockS3 } = require('./s3-mock');
const { mockS3 } = require('./s3-mock');
const { refusal } = require('../lib/archive');
const { refusal } = require('../lib/archive');
-const offsite = require('../lib/offsite');
-const { normalise } = require('../lib/inventory');
const KEY_HEX = crypto.createHash('sha256').update('archive test key').digest('hex');
const KEY_HEX = crypto.createHash('sha256').update('archive test key').digest('hex');
const S3_SECRET = `${SECRET}-s3`;
const S3_SECRET = `${SECRET}-s3`;
@@ -29,11 +26,11 @@ function archiveHost() {
@@ -29,11 +26,11 @@ function archiveHost() {
'',
'',
].join('\n'), { mode: 0o600, owner: 'root' });
].join('\n'), { mode: 0o600, owner: 'root' });
host.put('/etc/openvibe/backup.key', `${KEY_HEX}\n`, { mode: 0o600, owner: 'root' });
host.put('/etc/openvibe/backup.key', `${KEY_HEX}\n`, { mode: 0o600, owner: 'root' });
- host.put('/opt/backups/old/hobostreamer.db', Buffer.from(`SQLite format 3\0 old site ${'x'.repeat(9000)}`), { owner: 'root' });
- host.put('/var/lib/openvibe-chat/live-snap-a.db', Buffer.from(`SQLite format 3\0 snapshot ${'y'.repeat(3000)}`), { owner: 'root' });
- host.put('/var/lib/openvibe-chat/in-use.db', 'SQLite format 3\0 open', { owner: 'root' });
- host.put('/etc/passwd-copy.db', 'nope', { owner: 'root' });
- host.openFiles = new Set(['/var/lib/openvibe-chat/in-use.db']);
+ host.put('/opt/backups/old/hobostreamer.tar', Buffer.from(`old site ${'x'.repeat(9000)}`), { owner: 'root' });
+ host.put('/var/lib/openvibe-chat/live-snap-a.tar', Buffer.from(`snapshot ${'y'.repeat(3000)}`), { owner: 'root' });
+ host.put('/var/lib/openvibe-chat/in-use.tar', 'open', { owner: 'root' });
+ host.put('/etc/passwd-copy.tar', 'nope', { owner: 'root' });
+ host.openFiles = new Set(['/var/lib/openvibe-chat/in-use.tar']);
host.s3 = mockS3();
host.s3 = mockS3();
host.cli = async (...argv) => {
host.cli = async (...argv) => {
const lines = [];
const lines = [];
@@ -48,19 +45,19 @@ const noSecrets = (text) => { for (const s of [S3_SECRET, SECRET, KEY_HEX]) asse
@@ -48,19 +45,19 @@ const noSecrets = (text) => { for (const s of [S3_SECRET, SECRET, KEY_HEX]) asse
runTests([
runTests([
test('push encrypts, uploads under the archive prefix, checks, records, and deletes nothing', async () => {
test('push encrypts, uploads under the archive prefix, checks, records, and deletes nothing', async () => {
const host = archiveHost();
const host = archiveHost();
- const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.db', '/var/lib/openvibe-chat/live-snap-a.db', '--note', 'H14 first pass');
+ const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.tar', '/var/lib/openvibe-chat/live-snap-a.tar', '--note', 'H14 first pass');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
noSecrets(r.out);
noSecrets(r.out);
assert.match(r.out, /ARCHIVE ok: 2 file\(s\) at s3:\/\/openvibe-backups-test\/openvibe-backups-archive\/.+\/\d{8}-\d{6}\/.*Nothing was deleted/);
assert.match(r.out, /ARCHIVE ok: 2 file\(s\) at s3:\/\/openvibe-backups-test\/openvibe-backups-archive\/.+\/\d{8}-\d{6}\/.*Nothing was deleted/);
const keys = [...host.s3.objects.keys()];
const keys = [...host.s3.objects.keys()];
assert.ok(keys.every((k) => k.startsWith('openvibe-backups-archive/')), 'outside the backup runs prefix (retention never prunes it)');
assert.ok(keys.every((k) => k.startsWith('openvibe-backups-archive/')), 'outside the backup runs prefix (retention never prunes it)');
- const obj = keys.find((k) => k.endsWith('opt__backups__old__hobostreamer.db.ovbk'));
+ const obj = keys.find((k) => k.endsWith('opt__backups__old__hobostreamer.tar.ovbk'));
assert.ok(obj, keys.join(', '));
assert.ok(obj, keys.join(', '));
assert.ok(!host.s3.objects.get(obj).includes(Buffer.from('old site')), 'the bytes are encrypted');
assert.ok(!host.s3.objects.get(obj).includes(Buffer.from('old site')), 'the bytes are encrypted');
- assert.ok(host.read('/opt/backups/old/hobostreamer.db') !== null && host.read('/var/lib/openvibe-chat/live-snap-a.db') !== null, 'the local files stay');
+ assert.ok(host.read('/opt/backups/old/hobostreamer.tar') !== null && host.read('/var/lib/openvibe-chat/live-snap-a.tar') !== null, 'the local files stay');
const manifestKey = keys.find((k) => k.endsWith('/manifest.json'));
const manifestKey = keys.find((k) => k.endsWith('/manifest.json'));
const m = JSON.parse(host.s3.objects.get(manifestKey).toString('utf8'));
const m = JSON.parse(host.s3.objects.get(manifestKey).toString('utf8'));
- assert.deepStrictEqual(m.files.map((f) => f.path), ['/opt/backups/old/hobostreamer.db', '/var/lib/openvibe-chat/live-snap-a.db']);
+ assert.deepStrictEqual(m.files.map((f) => f.path), ['/opt/backups/old/hobostreamer.tar', '/var/lib/openvibe-chat/live-snap-a.tar']);
assert.strictEqual(m.note, 'H14 first pass');
assert.strictEqual(m.note, 'H14 first pass');
assert.ok(m.hmac && m.files.every((f) => /^[0-9a-f]{64}$/.test(f.plainSha256)));
assert.ok(m.hmac && m.files.every((f) => /^[0-9a-f]{64}$/.test(f.plainSha256)));
const rec = [...host.files.keys()].find((k) => k.startsWith('/var/lib/openvibe-host/archives/'));
const rec = [...host.files.keys()].find((k) => k.startsWith('/var/lib/openvibe-host/archives/'));
@@ -72,46 +69,22 @@ runTests([
@@ -72,46 +69,22 @@ runTests([
assert.match(list.out, /\d{8}-\d{6}\s+2 file\(s\).*H14 first pass/);
assert.match(list.out, /\d{8}-\d{6}\s+2 file\(s\).*H14 first pass/);
const st = m.stamp;
const st = m.stamp;
- const back = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.db', '--out', '/var/lib/openvibe-restore/h14');
+ const back = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.tar', '--out', '/var/lib/openvibe-restore/h14');
assert.strictEqual(back.code, 0, back.out);
assert.strictEqual(back.code, 0, back.out);
- assert.strictEqual(host.read('/var/lib/openvibe-restore/h14/hobostreamer.db'), host.read('/opt/backups/old/hobostreamer.db'), 'restored byte for byte');
- const again = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.db', '--out', '/var/lib/openvibe-restore/h14');
+ assert.strictEqual(host.read('/var/lib/openvibe-restore/h14/hobostreamer.tar'), host.read('/opt/backups/old/hobostreamer.tar'), 'restored byte for byte');
+ const again = await host.cli('archive', 'restore', st, '/opt/backups/old/hobostreamer.tar', '--out', '/var/lib/openvibe-restore/h14');
assert.notStrictEqual(again.code, 0, 'restore never overwrites');
assert.notStrictEqual(again.code, 0, 'restore never overwrites');
}),
}),
- test('refusals: live databases, open files, symlinks, outside roots, relative paths; nothing uploaded', async () => {
+ test('refusals: open files, symlinks, outside roots and relative paths', async () => {
const host = archiveHost();
const host = archiveHost();
- const liveDb = Object.values(host.inv.services).flatMap((s) => s.databases.filter((d) => d.engine !== 'postgresql').map((d) => d.path))[0];
- assert.ok(liveDb, 'the scenario declares a database');
- host.put(liveDb, 'SQLite format 3\0 live', { owner: 'ubuntu' });
- await host.exec.symlink('/opt/backups/old/hobostreamer.db', '/opt/backups/old/link.db');
- for (const [file, why] of [['/opt/backups/old/link.db', /not a regular file/], [liveDb, /inventory declares/], ['/var/lib/openvibe-chat/in-use.db', /open by a running process/], ['/etc/passwd-copy.db', /outside/], ['relative.db', /absolute/], ['/opt/backups/../etc/x.db', /normalised/], ['/opt/backups/none.db', /no such file/]]) {
- const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.db', file);
+ await host.exec.symlink('/opt/backups/old/hobostreamer.tar', '/opt/backups/old/link.tar');
+ for (const [file, why] of [['/opt/backups/old/link.tar', /not a regular file/], ['/var/lib/openvibe-chat/in-use.tar', /open by a running process/], ['/etc/passwd-copy.tar', /outside/], ['relative.tar', /absolute/], ['/opt/backups/../etc/x.tar', /normalised/], ['/opt/backups/none.tar', /no such file/]]) {
+ const r = await host.cli('archive', 'push', '/opt/backups/old/hobostreamer.tar', file);
assert.strictEqual(r.code, 1, `${file}: ${r.out}`);
assert.strictEqual(r.code, 1, `${file}: ${r.out}`);
assert.match(r.out, why, file);
assert.match(r.out, why, file);
- assert.strictEqual(host.s3.objects.size, 0, `${file}: nothing is uploaded when any file is refused`);
+ assert.strictEqual(host.s3.objects.size, 0);
}
}
}),
}),
- test('a PostgreSQL entry has no path, so it neither blocks archiving the file it replaced nor makes unsafeOut throw', async () => {
- const host = archiveHost();
- const raw = JSON.parse(host.read('/etc/openvibe/host.json'));
- // Media moved to PostgreSQL (C-89): its databases[] entry names a database in the cluster.
- raw.services.media.databases = [{ name: 'media', engine: 'postgresql', database: 'ov_media' }];
- const inv = normalise(raw);
- const frozen = '/opt/openvibe.media/data/media.db'; // Media's old SQLite file, frozen at the switch
- host.put(frozen, 'SQLite format 3\0 frozen', { owner: 'ubuntu' });
- assert.strictEqual(await refusal(host.exec, inv, frozen), null, 'the frozen SQLite file is archivable again');
-
- // A database the inventory still declares as SQLite is refused, message unchanged.
- const live = inv.services.live.databases[0];
- assert.strictEqual(live.engine, 'sqlite');
- host.put(live.path, 'SQLite format 3\0 live', { owner: 'ubuntu' });
- assert.match(await refusal(host.exec, inv, live.path), /a database the inventory declares \(a live database\)/);
-
- // unsafeOut only avoids database directories; a database in the cluster has none.
- let out;
- assert.doesNotThrow(() => { out = offsite.unsafeOut(inv, '/var/lib/openvibe-restore/media-2026'); });
- assert.strictEqual(out, null);
- }),
]);
]);
@@ -26,12 +26,12 @@ const TENANT_BLOB = 'tenant object bytes\n';
@@ -26,12 +26,12 @@ const TENANT_BLOB = 'tenant object bytes\n';
function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {
function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env = null } = {}) {
const host = scenario();
const host = scenario();
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
- doc.services.ai = { repo: '/opt/openvibe.ai', units: [], databases: [{ name: 'ai', path: '/var/lib/openvibe-ai/ai.db' }, { name: 'ai-extra', path: '/var/lib/openvibe-ai/extra.db' }] };
- doc.services.openre = { repo: '/opt/openre', units: [], databases: [{ name: 'openre', path: '/var/lib/openre/openre.db' }] };
+ doc.services.ai = { repo: '/opt/openvibe.ai', units: [], databases: [{ name: 'ai', engine: 'postgresql', database: 'ov_ai' }, { name: 'ai-extra', engine: 'postgresql', database: 'ov_ai_extra' }] };
+ doc.services.openre = { repo: '/opt/openre', units: [], databases: [{ name: 'openre', engine: 'postgresql', database: 'ov_openre' }] };
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.inv = normalise(doc);
host.inv = normalise(doc);
- host.put('/var/lib/openvibe-ai/ai.db', 'sqlite', { owner: 'ubuntu' });
- host.put('/var/lib/openvibe-ai/extra.db', 'sqlite', { owner: 'ubuntu' });
+ for (const db of ['ov_ai', 'ov_ai_extra', 'ov_openre']) host.pgDatabases.add(db);
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
host.put('/etc/openvibe/backup.env', env || [
host.put('/etc/openvibe/backup.env', env || [
'# off-host backups',
'# off-host backups',
'BACKUP_S3_ENDPOINT=https://s3.us-west-004.backblazeb2.com',
'BACKUP_S3_ENDPOINT=https://s3.us-west-004.backblazeb2.com',
@@ -43,9 +43,6 @@ function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env =
@@ -43,9 +43,6 @@ function backupHost({ envMode = 0o600, keyMode = 0o600, envOwner = 'root', env =
'',
'',
].join('\n'), { mode: envMode, owner: envOwner });
].join('\n'), { mode: envMode, owner: envOwner });
host.put('/etc/openvibe/backup.key', `${KEY_HEX}\n`, { mode: keyMode, owner: 'root' });
host.put('/etc/openvibe/backup.key', `${KEY_HEX}\n`, { mode: keyMode, owner: 'root' });
- // Distinct content per database, so a restore can be compared byte for byte.
- host.backupContent = (db) => Buffer.from(`SQLite format 3\0 copy of ${db} ${'x'.repeat(5000)}`);
- host.sqliteHandler = (db, sql) => (/quick_check/.test(sql) ? [{ quick_check: 'ok' }] : [{ n: 0 }]);
host.s3 = mockS3();
host.s3 = mockS3();
host.s3Configs = [];
host.s3Configs = [];
host.cli = async (...argv) => {
host.cli = async (...argv) => {
@@ -169,149 +166,32 @@ runTests([
@@ -169,149 +166,32 @@ runTests([
assert.deepStrictEqual([...retention.keep(['junk', '20260101-000000'])], ['20260101-000000']);
assert.deepStrictEqual([...retention.keep(['junk', '20260101-000000'])], ['20260101-000000']);
}),
}),
- test('backup --all prunes to 7 daily + 4 weekly per service; failing days never age out good backups', async () => {
- const host = backupHost();
- // A directory ovhost did not record is never touched.
- host.put('/var/backups/openvibe/live/20200101-000000/live.db', 'operator copy', { owner: 'root', mode: 0o600 });
- for (let d = 0; d < 40; d++) {
- const r = await host.cli('backup', '--all');
- assert.strictEqual(r.code, 0, r.out);
- host.advance(DAY);
- }
- // 40 daily runs from 2026-09-22 12:00 to 2026-10-31 12:00.
- const expected = ['20200101-000000', '20261011-120000', '20261018-120000', '20261025-120000', '20261026-120000', '20261027-120000', '20261028-120000', '20261029-120000', '20261030-120000', '20261031-120000'];
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), expected);
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/media'), expected.slice(1));
-
- // Live fails for 20 days: nothing of Live is pruned, the others carry on.
- host.onSqliteBackup = (db) => (db.includes('openvibe.live') ? new Error('database disk image is malformed') : null);
- for (let d = 0; d < 20; d++) {
- const r = await host.cli('backup', '--all');
- assert.strictEqual(r.code, 2);
- host.advance(DAY);
- }
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), expected);
- assert.strictEqual(dirsOf(host, '/var/backups/openvibe/media').length, 9);
- assert.ok(dirsOf(host, '/var/backups/openvibe/media').includes('20261119-120000'));
- }),
-
- test('a partly failed backup is kept until a newer good one exists, then pruned; drills skip it', async () => {
- const host = backupHost();
- await host.cli('backup', '--all');
- host.advance(DAY);
- host.onSqliteBackup = (db) => (db.endsWith('extra.db') ? new Error('disk I/O error') : null);
- const bad = await host.cli('backup', '--all', '--json');
- assert.strictEqual(bad.code, 2);
- const s = JSON.parse(bad.out);
- const ai = s.services.find((x) => x.service === 'ai');
- assert.strictEqual(ai.status, 'failed');
- assert.match(ai.error, /ai-extra: disk I\/O error/);
- assert.deepStrictEqual(dirsOf(host, ai.dir), ['ai.db'], 'the good half is kept');
- const log = host.read('/var/lib/openvibe-host/backups/ai.jsonl').trim().split('\n').map((l) => JSON.parse(l));
- assert.strictEqual(log[log.length - 1].ok, false);
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/ai'), ['20260922-120000', '20260923-120000']);
- host.onSqliteBackup = null;
- host.advance(DAY);
- await host.cli('backup', '--all');
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/ai'), ['20260922-120000', '20260924-120000'], 'the failed one went once a newer good one existed');
- }),
-
- test('backup --all leaves its outcome for Prometheus (textfile collector): last run, ok, last success, per-status counts', async () => {
- const host = backupHost();
- const PROM = '/var/lib/prometheus/node-exporter/openvibe_backup.prom';
- await host.cli('backup', '--all');
- assert.strictEqual(host.files.has(PROM), false, 'no collector directory: nothing written');
- host.put('/var/lib/prometheus/node-exporter/.keep', '');
- host.advance(DAY);
- await host.cli('backup', '--all');
- const metric = (name) => { const m = new RegExp(`^${name} (\\S+)$`, 'm').exec(host.read(PROM)); return m ? Number(m[1]) : null; };
- const goodAt = metric('openvibe_backup_last_run_timestamp_seconds');
- assert.ok(goodAt > 0);
- assert.strictEqual(metric('openvibe_backup_last_run_ok'), 1);
- assert.strictEqual(metric('openvibe_backup_last_success_timestamp_seconds'), goodAt);
- assert.strictEqual(metric('openvibe_backup_services\\{status="failed"\\}'), 0);
- host.advance(DAY);
- host.onSqliteBackup = (db) => (db.endsWith('extra.db') ? new Error('disk I/O error') : null);
- await host.cli('backup', '--all');
- host.onSqliteBackup = null;
- assert.strictEqual(metric('openvibe_backup_last_run_ok'), 0);
- assert.strictEqual(metric('openvibe_backup_services\\{status="failed"\\}'), 1);
- assert.ok(metric('openvibe_backup_last_run_timestamp_seconds') > goodAt);
- assert.strictEqual(metric('openvibe_backup_last_success_timestamp_seconds'), goodAt, 'the last success is still the good run');
- host.files.delete(PROM);
- const r = await host.cli('backup-metrics');
- assert.strictEqual(r.code, 0, r.out);
- assert.strictEqual(metric('openvibe_backup_last_run_ok'), 0, 'rewritten from the latest recorded run');
- assert.strictEqual(metric('openvibe_backup_last_success_timestamp_seconds'), goodAt);
- }),
-
- // ── failure isolation and the summary ─────────────────────────────────────
- test('one failing service never stops the others; exit code 2; JSON summary per run', async () => {
- const host = backupHost();
- host.onSqliteBackup = (db) => (db.includes('openvibe.media') ? new Error('SQLITE_BUSY: database is locked') : null);
- const r = await host.cli('backup', '--all');
- assert.strictEqual(r.code, 2, r.out);
- assert.match(r.out, /FAILED\s+media\s+media: SQLITE_BUSY/);
- assert.match(r.out, /OK\s+live\s+1 file\(s\)/);
- assert.match(r.out, /OK\s+ai\s+2 file\(s\)/);
- assert.match(r.out, /SKIPPED\s+openre\s+no database found/);
- const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));
- assert.strictEqual(summary.ok, false);
- assert.deepStrictEqual(summary.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'failed'], ['ai', 'ok'], ['openre', 'skipped']]);
- assert.deepStrictEqual(summary.retention, { daily: 7, weekly: 4 });
- // The services after the failing one were still backed up.
- assert.ok(host.files.has('/var/backups/openvibe/ai/20260922-120000/ai-extra.db'));
- assert.strictEqual(host.files.has('/var/backups/openvibe/openre'), true);
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/openre'), [], 'no empty backup directory is left behind');
- assert.strictEqual(host.files.has('/var/lib/openvibe-host/locks/_backup-all.lock'), false, 'lock released');
- const ok = await (host.advance(1000), host.cli('backup', '--all', '--json'));
- host.onSqliteBackup = null;
- assert.strictEqual(JSON.parse(ok.out).ok, false);
- host.advance(1000);
- assert.strictEqual((await host.cli('backup', '--all')).code, 0);
- }),
-
- test('permissions: backup directories root 0700, copies root 0600, no -wal/-shm; old service-owned directories are locked down', async () => {
+ test('PostgreSQL dump copies are root-only and staging is emptied', async () => {
const host = backupHost();
const host = backupHost();
host.ensureDir('/var/backups/openvibe/live', 'ubuntu', 0o750);
host.ensureDir('/var/backups/openvibe/live', 'ubuntu', 0o750);
- // The worker leaves a side file behind: it is not kept.
- host.onSqliteBackup = (db, dest) => { host.put(`${dest}-wal`, 'wal', { owner: 'ubuntu', mode: 0o644 }); return null; };
const r = await host.cli('backup', '--all', '--offsite');
const r = await host.cli('backup', '--all', '--offsite');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
- let dirs = 0;
- let files = 0;
- for (const [p, e] of host.files) {
- if (p !== '/var/backups/openvibe' && !p.startsWith('/var/backups/openvibe/')) continue;
- if (e.type === 'dir') { dirs += 1; assert.deepStrictEqual([e.owner, e.mode], ['root', 0o700], p); }
- else { files += 1; assert.deepStrictEqual([e.owner, e.mode], ['root', 0o600], p); assert.ok(!/-(wal|shm|journal)$/.test(p), `${p} left behind`); }
- }
- assert.ok(dirs >= 8 && files === 4, `${dirs} dirs, ${files} files`);
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe.staging'), [], 'staging emptied');
- assert.deepStrictEqual([host.files.get('/var/backups/openvibe.staging').owner, host.files.get('/var/backups/openvibe.staging').mode], ['root', 0o711]);
- // The worker still ran as the service user, into a directory it owned at the time.
- assert.ok(host.sqliteCalls.filter((c) => c.op === 'backup').every((c) => c.as === 'ubuntu' && c.dest.startsWith('/var/backups/openvibe.staging/')));
-
- // restore-download: a root-only directory, 0600 files.
- const d = await host.cli('restore-download', 'ai', 'latest');
- assert.strictEqual(d.code, 0, d.out);
- for (const p of ['/var/lib/openvibe-restore', '/var/lib/openvibe-restore/ai-20260922-120000']) assert.deepStrictEqual([host.files.get(p).owner, host.files.get(p).mode], ['root', 0o700], p);
- for (const f of ['ai.db', 'ai-extra.db']) {
- const e = host.files.get(`/var/lib/openvibe-restore/ai-20260922-120000/${f}`);
- assert.deepStrictEqual([e.owner, e.mode], ['root', 0o600], f);
+ for (const [file, entry] of host.files) {
+ if (!file.startsWith('/var/backups/openvibe/')) continue;
+ assert.deepStrictEqual([entry.owner, entry.mode], ['root', entry.type === 'dir' ? 0o700 : 0o600], file);
}
}
+ assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe.staging'), []);
+ assert.ok(host.dumps.every((d) => d.as === 'postgres'));
+ const restored = await host.cli('restore-download', 'ai', 'latest');
+ assert.strictEqual(restored.code, 0, restored.out);
+ for (const file of ['ai.dump', 'ai-extra.dump']) assert.strictEqual(host.files.get(`/var/lib/openvibe-restore/ai-20260922-120000/${file}`).mode, 0o600);
}),
}),
- test('a copy that is a link is refused, not chowned', async () => {
+ test('a failed PostgreSQL dump does not prevent another service from backing up', async () => {
const host = backupHost();
const host = backupHost();
- // The service user hard-links its "copy" to some other file before root takes it over.
- const orig = host.exec.sqliteBackup;
- host.exec.sqliteBackup = async (db, dest, o) => { await orig(db, dest, o); host.files.get(dest).nlink = 2; };
- const r = await host.cli('backup', 'live');
- host.exec.sqliteBackup = orig;
- assert.strictEqual(r.code, 2);
- assert.match(r.out, /not a plain file/);
- assert.ok(!host.calls.some((c) => (c.cmd === 'chown' || c.cmd === 'chmod') && String(c.args[c.args.length - 1]).endsWith('live.db')), 'a linked copy is never chowned');
- assert.deepStrictEqual(dirsOf(host, '/var/backups/openvibe/live'), [], 'and never moved into the backups');
+ const orig = host.exec.run;
+ host.exec.run = async (cmd, args, opts) => cmd === 'pg_dump' && args.includes('ov_media')
+ ? { code: 1, stdout: '', stderr: 'simulated pg_dump failure' } : orig(cmd, args, opts);
+ const r = await host.cli('backup', '--all', '--json');
+ assert.strictEqual(r.code, 2, r.out);
+ const summary = JSON.parse(r.out);
+ assert.deepStrictEqual(summary.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'failed'], ['ai', 'ok'], ['openre', 'ok']]);
+ assert.ok(host.files.has('/var/backups/openvibe/ai/20260922-120000/ai-extra.dump'));
}),
}),
// ── off-host upload ────────────────────────────────────────────────────────
// ── off-host upload ────────────────────────────────────────────────────────
@@ -320,7 +200,7 @@ runTests([
@@ -320,7 +200,7 @@ runTests([
const r = await host.cli('backup', '--all', '--offsite');
const r = await host.cli('backup', '--all', '--offsite');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
const base = 'openvibe-backups/fake-host/20260922-120000/';
const base = 'openvibe-backups/fake-host/20260922-120000/';
- assert.deepStrictEqual([...host.s3.objects.keys()].sort(), [`${base}ai/ai-extra.db.ovbk`, `${base}ai/ai.db.ovbk`, `${base}live/live.db.ovbk`, `${base}manifest.json`, `${base}media/media.db.ovbk`]);
+ assert.deepStrictEqual([...host.s3.objects.keys()].sort(), [`${base}ai/ai-extra.dump.ovbk`, `${base}ai/ai.dump.ovbk`, `${base}live/live.dump.ovbk`, `${base}manifest.json`, `${base}media/media.dump.ovbk`, `${base}openre/openre.dump.ovbk`]);
const key = Buffer.from(KEY_HEX, 'hex');
const key = Buffer.from(KEY_HEX, 'hex');
for (const [k, v] of host.s3.objects) {
for (const [k, v] of host.s3.objects) {
if (k.endsWith('manifest.json')) continue;
if (k.endsWith('manifest.json')) continue;
@@ -328,12 +208,12 @@ runTests([
@@ -328,12 +208,12 @@ runTests([
const plain = await decrypt(key, v);
const plain = await decrypt(key, v);
const svc = k.split('/')[3];
const svc = k.split('/')[3];
const file = path.basename(k, '.ovbk');
const file = path.basename(k, '.ovbk');
- assert.ok(plain.equals(host.files.get(`/var/backups/openvibe/${svc}/20260922-120000/${file}`).content), `${k} round trip`);
+ assert.ok(plain.equals(Buffer.from(host.files.get(`/var/backups/openvibe/${svc}/20260922-120000/${file}`).content)), `${k} round trip`);
}
}
const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));
const m = JSON.parse(host.s3.objects.get(`${base}manifest.json`));
assert.strictEqual(m.hmac, bcrypto.hmacManifest(key, JSON.stringify((({ hmac, ...rest }) => rest)(m))));
assert.strictEqual(m.hmac, bcrypto.hmacManifest(key, JSON.stringify((({ hmac, ...rest }) => rest)(m))));
assert.strictEqual(m.keyId, bcrypto.keyId(key));
assert.strictEqual(m.keyId, bcrypto.keyId(key));
- assert.deepStrictEqual(Object.keys(m.services), ['live', 'media', 'ai']);
+ assert.deepStrictEqual(Object.keys(m.services), ['live', 'media', 'ai', 'openre']);
// Only PutObject for small files; path-style B2 endpoint and region from the endpoint.
// Only PutObject for small files; path-style B2 endpoint and region from the endpoint.
assert.ok(host.s3.sent.filter((c) => /Put/.test(c.name)).every((c) => c.input.Bucket === 'openvibe-backups-test'));
assert.ok(host.s3.sent.filter((c) => /Put/.test(c.name)).every((c) => c.input.Bucket === 'openvibe-backups-test'));
const cfg = host.s3Configs[0];
const cfg = host.s3Configs[0];
@@ -346,14 +226,14 @@ runTests([
@@ -346,14 +226,14 @@ runTests([
noSecrets(host.s3.objects.get(`${base}manifest.json`), 'the manifest');
noSecrets(host.s3.objects.get(`${base}manifest.json`), 'the manifest');
const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));
const summary = JSON.parse(host.read('/var/lib/openvibe-host/backup-runs/20260922-120000.json'));
assert.strictEqual(summary.offsite.ok, true);
assert.strictEqual(summary.offsite.ok, true);
- assert.strictEqual(summary.offsite.uploaded.length, 4);
+ assert.strictEqual(summary.offsite.uploaded.length, 5);
assert.strictEqual(summary.offsite.location, base);
assert.strictEqual(summary.offsite.location, base);
}),
}),
test('large files go up as a multipart upload; a failed part aborts it', async () => {
test('large files go up as a multipart upload; a failed part aborts it', async () => {
const host = backupHost();
const host = backupHost();
const big = crypto.randomBytes(300 * 1024);
const big = crypto.randomBytes(300 * 1024);
- host.backupContent = (db) => (db.includes('openvibe.live') ? big : Buffer.from('small'));
+ host.dumpContent = (db) => db === 'ov_live' ? Buffer.concat([Buffer.from('pg_dump -Fc'), big]) : Buffer.from(`pg_dump -Fc of ${db}`);
await host.cli('backup', '--all');
await host.cli('backup', '--all');
const summary = await require('../lib/commands/backup').readRun(host.exec, host.inv, null);
const summary = await require('../lib/commands/backup').readRun(host.exec, host.inv, null);
const cfg = await offsite.loadConfig(host.exec, {});
const cfg = await offsite.loadConfig(host.exec, {});
@@ -364,8 +244,8 @@ runTests([
@@ -364,8 +244,8 @@ runTests([
assert.strictEqual(live.parts, Math.ceil(live.cipherBytes / (64 * 1024)));
assert.strictEqual(live.parts, Math.ceil(live.cipherBytes / (64 * 1024)));
const partSizes = host.s3.sent.filter((c) => c.name === 'UploadPartCommand').map((c) => c.input.Body);
const partSizes = host.s3.sent.filter((c) => c.name === 'UploadPartCommand').map((c) => c.input.Body);
assert.ok(partSizes.slice(0, -1).every((b) => b === `<${64 * 1024} bytes>`), 'every part but the last is exactly partSize');
assert.ok(partSizes.slice(0, -1).every((b) => b === `<${64 * 1024} bytes>`), 'every part but the last is exactly partSize');
- assert.strictEqual(live.cipherBytes, bcrypto.encryptedSize(big.length, 16 * 1024));
- assert.ok((await decrypt(Buffer.from(KEY_HEX, 'hex'), host.s3.objects.get(live.object))).equals(big));
+ assert.strictEqual(live.cipherBytes, bcrypto.encryptedSize(big.length + 11, 16 * 1024));
+ assert.ok((await decrypt(Buffer.from(KEY_HEX, 'hex'), host.s3.objects.get(live.object))).equals(Buffer.concat([Buffer.from('pg_dump -Fc'), big])));
assert.ok(host.s3.sent.some((c) => c.name === 'CompleteMultipartUploadCommand'));
assert.ok(host.s3.sent.some((c) => c.name === 'CompleteMultipartUploadCommand'));
const s3 = mockS3();
const s3 = mockS3();
@@ -456,7 +336,7 @@ runTests([
@@ -456,7 +336,7 @@ runTests([
test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {
test('off-host retention: runs older than 30 days are deleted, the newest 7 always stay, and nothing is pruned after a failed upload', async () => {
const host = backupHost();
const host = backupHost();
const put = (run) => {
const put = (run) => {
- host.s3.objects.set(`openvibe-backups/fake-host/${run}/live/live.db.ovbk`, Buffer.from('x'));
+ host.s3.objects.set(`openvibe-backups/fake-host/${run}/live/live.dump.ovbk`, Buffer.from('x'));
host.s3.objects.set(`openvibe-backups/fake-host/${run}/manifest.json`, Buffer.from('{}'));
host.s3.objects.set(`openvibe-backups/fake-host/${run}/manifest.json`, Buffer.from('{}'));
};
};
// Ten runs 35..44 days old, one 5 days old; another host's and a stray object are never touched.
// Ten runs 35..44 days old, one 5 days old; another host's and a stray object are never touched.
@@ -466,7 +346,7 @@ runTests([
@@ -466,7 +346,7 @@ runTests([
host.s3.objects.set('openvibe-backups/fake-host/README', Buffer.from('keep'));
host.s3.objects.set('openvibe-backups/fake-host/README', Buffer.from('keep'));
// A failed upload: no pruning at all.
// A failed upload: no pruning at all.
- host.s3.failPut = (k) => k.endsWith('media.db.ovbk');
+ host.s3.failPut = (k) => k.endsWith('media.dump.ovbk');
const bad = await host.cli('backup', '--all', '--offsite', '--json');
const bad = await host.cli('backup', '--all', '--offsite', '--json');
assert.strictEqual(bad.code, 2);
assert.strictEqual(bad.code, 2);
const badSummary = JSON.parse(bad.out);
const badSummary = JSON.parse(bad.out);
@@ -495,10 +375,11 @@ runTests([
@@ -495,10 +375,11 @@ runTests([
host.s3 = mockS3({ pageSize: 2 });
host.s3 = mockS3({ pageSize: 2 });
await host.cli('backup', '--all', '--offsite');
await host.cli('backup', '--all', '--offsite');
host.advance(DAY);
host.advance(DAY);
- await host.cli('backup', '--all', '--offsite');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
+ await host.cli('backup', '--all', '--offsite', '--logical');
const l = await host.cli('offsite', 'list', 'live', '--json');
const l = await host.cli('offsite', 'list', 'live', '--json');
assert.strictEqual(l.code, 0, l.out);
assert.strictEqual(l.code, 0, l.out);
- assert.deepStrictEqual(JSON.parse(l.out).map((x) => [x.run, x.objects, x.manifest]), [['20260923-120000', 5, true], ['20260922-120000', 5, true]]);
+ assert.deepStrictEqual(JSON.parse(l.out).map((x) => [x.run, x.objects, x.manifest]), [['20260923-120000', 6, true], ['20260922-120000', 6, true]]);
const c = await host.cli('offsite', 'check');
const c = await host.cli('offsite', 'check');
assert.strictEqual(c.code, 0, c.out);
assert.strictEqual(c.code, 0, c.out);
assert.match(c.out, /s3:\/\/openvibe-backups-test\/openvibe-backups\/fake-host\/ .*region us-west-004.*key id [0-9a-f]{16}; retention 30 days/);
assert.match(c.out, /s3:\/\/openvibe-backups-test\/openvibe-backups\/fake-host\/ .*region us-west-004.*key id [0-9a-f]{16}; retention 30 days/);
@@ -509,20 +390,18 @@ runTests([
@@ -509,20 +390,18 @@ runTests([
test('restore-download: verified, decrypted copies in a new directory; never overwrites; never near a live database', async () => {
test('restore-download: verified, decrypted copies in a new directory; never overwrites; never near a live database', async () => {
const host = backupHost();
const host = backupHost();
await host.cli('backup', '--all', '--offsite');
await host.cli('backup', '--all', '--offsite');
- const original = host.files.get('/var/backups/openvibe/live/20260922-120000/live.db').content;
+ const original = host.files.get('/var/backups/openvibe/live/20260922-120000/live.dump').content;
const r = await host.cli('restore-download', 'live', '20260922-120000');
const r = await host.cli('restore-download', 'live', '20260922-120000');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
- const dest = '/var/lib/openvibe-restore/live-20260922-120000/live.db';
- assert.ok(host.files.get(dest).content.equals(original));
- assert.match(r.out, /quick_check ok/);
- assert.ok(host.sqliteCalls.some((c) => c.db === dest && /quick_check/.test(c.sql)));
- // The live database was never written.
- assert.strictEqual(host.read('/opt/openvibe.live/data/live.db'), 'sqlite');
+ const dest = '/var/lib/openvibe-restore/live-20260922-120000/live.dump';
+ assert.ok(Buffer.from(host.files.get(dest).content).equals(Buffer.from(original)));
+ assert.match(r.out, /pg_restore --list ok/);
+ assert.ok(host.calls.some((c) => c.cmd === 'pg_restore' && c.args.includes('--list')));
const again = await host.cli('restore-download', 'live', '20260922-120000');
const again = await host.cli('restore-download', 'live', '20260922-120000');
assert.strictEqual(again.code, 1);
assert.strictEqual(again.code, 1);
assert.match(again.out, /not empty; restore-download never overwrites/);
assert.match(again.out, /not empty; restore-download never overwrites/);
- for (const out of ['/opt/openvibe.live/data', '/opt/openvibe.live', '/opt', '/var/lib/openvibe-ai', '/var/backups/openvibe/live/x']) {
+ for (const out of ['/opt/openvibe.live/data', '/opt/openvibe.live', '/opt', '/var/backups/openvibe/live/x']) {
const bad = await host.cli('restore-download', 'live', 'latest', '--out', out);
const bad = await host.cli('restore-download', 'live', 'latest', '--out', out);
assert.strictEqual(bad.code, 1, out);
assert.strictEqual(bad.code, 1, out);
assert.match(bad.out, /refusing --out/, out);
assert.match(bad.out, /refusing --out/, out);
@@ -536,16 +415,16 @@ runTests([
@@ -536,16 +415,16 @@ runTests([
const host = backupHost();
const host = backupHost();
await host.cli('backup', '--all', '--offsite');
await host.cli('backup', '--all', '--offsite');
const base = 'openvibe-backups/fake-host/20260922-120000/';
const base = 'openvibe-backups/fake-host/20260922-120000/';
- const obj = Buffer.from(host.s3.objects.get(`${base}media/media.db.ovbk`));
+ const obj = Buffer.from(host.s3.objects.get(`${base}media/media.dump.ovbk`));
obj[obj.length - 20] ^= 1;
obj[obj.length - 20] ^= 1;
- host.s3.objects.set(`${base}media/media.db.ovbk`, obj);
+ host.s3.objects.set(`${base}media/media.dump.ovbk`, obj);
const t = await host.cli('restore-download', 'media', 'latest');
const t = await host.cli('restore-download', 'media', 'latest');
assert.strictEqual(t.code, 2);
assert.strictEqual(t.code, 2);
assert.match(t.out, /failed authentication/);
assert.match(t.out, /failed authentication/);
- assert.strictEqual(host.files.has('/var/lib/openvibe-restore/media-20260922-120000/media.db'), false, 'a bad copy is removed');
+ assert.strictEqual(host.files.has('/var/lib/openvibe-restore/media-20260922-120000/media.dump'), false, 'a bad copy is removed');
// Swapping two encrypted files is caught by the manifest's hashes.
// Swapping two encrypted files is caught by the manifest's hashes.
- host.s3.objects.set(`${base}ai/ai.db.ovbk`, host.s3.objects.get(`${base}ai/ai-extra.db.ovbk`));
+ host.s3.objects.set(`${base}ai/ai.dump.ovbk`, host.s3.objects.get(`${base}ai/ai-extra.dump.ovbk`));
const sw = await host.cli('restore-download', 'ai', 'latest', '--out', '/root/r-ai');
const sw = await host.cli('restore-download', 'ai', 'latest', '--out', '/root/r-ai');
assert.strictEqual(sw.code, 2);
assert.strictEqual(sw.code, 2);
assert.match(sw.out, /does not match the manifest/);
assert.match(sw.out, /does not match the manifest/);
@@ -557,9 +436,6 @@ runTests([
@@ -557,9 +436,6 @@ runTests([
assert.strictEqual(f.code, 2);
assert.strictEqual(f.code, 2);
assert.match(f.out, /failed its HMAC check/);
assert.match(f.out, /failed its HMAC check/);
- const none = await host.cli('restore-download', 'openre', 'latest');
- assert.strictEqual(none.code, 1);
- assert.match(none.out, /no off-host run of openre/);
}),
}),
test('restore-download: a PostgreSQL .dump is verified with pg_restore --list; a corrupt one is refused', async () => {
test('restore-download: a PostgreSQL .dump is verified with pg_restore --list; a corrupt one is refused', async () => {
@@ -595,9 +471,8 @@ runTests([
@@ -595,9 +471,8 @@ runTests([
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
const rec = JSON.parse(r.out);
const rec = JSON.parse(r.out);
const f = rec.files.find((x) => x.file.endsWith('objects.tar.gz'));
const f = rec.files.find((x) => x.file.endsWith('objects.tar.gz'));
- assert.deepStrictEqual([f.kind, f.quickCheck], ['objects', null], 'an object archive is not quick_checked as a database');
+ assert.deepStrictEqual([f.kind, f.check], ['objects', null], 'an object archive is checked by a drill');
assert.ok(host.files.has('/root/r-host/objects.tar.gz'));
assert.ok(host.files.has('/root/r-host/objects.tar.gz'));
- assert.ok(!host.sqliteCalls.some((c) => c.db === '/root/r-host/objects.tar.gz'));
assert.ok(rec.files.some((x) => x.file.endsWith('host.dump') && x.kind === 'database'), 'the database is still checked');
assert.ok(rec.files.some((x) => x.file.endsWith('host.dump') && x.kind === 'database'), 'the database is still checked');
}),
}),
@@ -633,7 +508,7 @@ runTests([
@@ -633,7 +508,7 @@ runTests([
r = await host.cli('backup', '--all', '--offsite');
r = await host.cli('backup', '--all', '--offsite');
assert.strictEqual(r.code, 2);
assert.strictEqual(r.code, 2);
assert.match(r.out, /OFFSITE\s+FAILED: encryption key file/);
assert.match(r.out, /OFFSITE\s+FAILED: encryption key file/);
- assert.ok(host.files.has('/var/backups/openvibe/live/20260922-120000/live.db'));
+ assert.ok(host.files.has('/var/backups/openvibe/live/20260922-120000/live.dump'));
const { createFakeHost } = require('./fake-host');
const { createFakeHost } = require('./fake-host');
const plain = createFakeHost({ root: false, user: 'ubuntu' });
const plain = createFakeHost({ root: false, user: 'ubuntu' });
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
/**
/**
* The PostgreSQL side of `ovhost backup`: every run verifies pgBackRest (a fresh stanza, and a WAL
* The PostgreSQL side of `ovhost backup`: every run verifies pgBackRest (a fresh stanza, and a WAL
* archiver failed_count that has not risen), and a per-service pg_dump -Fc only on a slower cadence
* archiver failed_count that has not risen), and a per-service pg_dump -Fc only on a slower cadence
- * (the weekly one, or --logical). SQLite services must be untouched. Against the fake host; no
+ * (the weekly one, or --logical). All declared databases use PostgreSQL. Against the fake host; no
* pg_dump, pgbackrest or psql ever runs for real.
* pg_dump, pgbackrest or psql ever runs for real.
*/
*/
const assert = require('assert');
const assert = require('assert');
@@ -57,7 +57,7 @@ runTests([
@@ -57,7 +57,7 @@ runTests([
assert.strictEqual(rec.verified.stop, new Date((Math.floor(host.exec.now() / 1000) - 3600) * 1000).toISOString());
assert.strictEqual(rec.verified.stop, new Date((Math.floor(host.exec.now() / 1000) - 3600) * 1000).toISOString());
// No prior record: a logical dump was due on the first run.
// No prior record: a logical dump was due on the first run.
assert.ok(rec.dir);
assert.ok(rec.dir);
- assert.strictEqual(host.dumps.length, 1);
+ assert.strictEqual(host.dumps.length, 3);
assert.ok(host.files.has(`${rec.dir}/trade.dump`));
assert.ok(host.files.has(`${rec.dir}/trade.dump`));
const prom = host.read(PROM);
const prom = host.read(PROM);
@@ -148,7 +148,7 @@ runTests([
@@ -148,7 +148,7 @@ runTests([
assert.strictEqual(host.dumps.length, 2);
assert.strictEqual(host.dumps.length, 2);
}),
}),
- test('staging ownership: postgres owns the directory while pg_dump runs; SQLite stays the service user', async () => {
+ test('staging ownership: postgres owns the directory while pg_dump runs; all dumps use postgres', async () => {
const host = pgHost();
const host = pgHost();
const stages = [];
const stages = [];
const mkdir = host.exec.mkdir;
const mkdir = host.exec.mkdir;
@@ -159,7 +159,7 @@ runTests([
@@ -159,7 +159,7 @@ runTests([
assert.strictEqual(host.dumps[0].as, 'postgres');
assert.strictEqual(host.dumps[0].as, 'postgres');
await host.cli('backup', 'live');
await host.cli('backup', 'live');
- assert.strictEqual(stages[1][1], 'ubuntu', 'a SQLite service keeps the service user');
+ assert.strictEqual(stages[1][1], 'postgres', 'every database dump uses postgres');
assert.strictEqual(stages[1][0], 'live-20260922-120000');
assert.strictEqual(stages[1][0], 'live-20260922-120000');
// Root takes every copy back once the staging directory is back to root:root.
// Root takes every copy back once the staging directory is back to root:root.
@@ -167,7 +167,7 @@ runTests([
@@ -167,7 +167,7 @@ runTests([
assert.deepStrictEqual([copy.owner, copy.mode], ['root', 0o600]);
assert.deepStrictEqual([copy.owner, copy.mode], ['root', 0o600]);
}),
}),
- test('backup --all: SQLite and PostgreSQL services in one run; verified differs from ok and failed', async () => {
+ test('backup --all: PostgreSQL services in one run; verified differs from ok and failed', async () => {
const host = pgHost();
const host = pgHost();
seedDump(host, 3 * DAY); // the weekly dump is not due: trade is verification-only this run
seedDump(host, 3 * DAY); // the weekly dump is not due: trade is verification-only this run
const r = await host.cli('backup', '--all', '--json');
const r = await host.cli('backup', '--all', '--json');
@@ -187,6 +187,6 @@ runTests([
@@ -187,6 +187,6 @@ runTests([
assert.strictEqual(bad.code, 2);
assert.strictEqual(bad.code, 2);
const s2 = JSON.parse(bad.out);
const s2 = JSON.parse(bad.out);
assert.strictEqual(s2.ok, false);
assert.strictEqual(s2.ok, false);
- assert.deepStrictEqual(s2.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'ok'], ['trade', 'failed']]);
+ assert.deepStrictEqual(s2.services.map((x) => [x.service, x.status]), [['live', 'failed'], ['media', 'failed'], ['trade', 'failed']]);
}),
}),
]);
]);
@@ -17,11 +17,12 @@ function withHost(host, rows = []) {
@@ -17,11 +17,12 @@ function withHost(host, rows = []) {
units: ['openvibe-host.service'],
units: ['openvibe-host.service'],
port: 4910,
port: 4910,
ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },
ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },
- databases: [{ name: 'host', path: '/var/lib/openvibe-host-api/host.db' }],
- nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: '/var/lib/openvibe-host-api/host.db', maxUpload: '110m' } },
+ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],
+ nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: { engine: 'postgresql', database: 'ov_host' }, maxUpload: '110m' } },
};
};
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
- host.sqliteHandler = (db, sql) => {
+ host.psqlHandler = (db, sql) => {
+ assert.strictEqual(db, 'ov_host');
assert.match(sql, /^SELECT hostname FROM host_domains WHERE kind = 'custom' AND status = 'verified'/);
assert.match(sql, /^SELECT hostname FROM host_domains WHERE kind = 'custom' AND status = 'verified'/);
return rows;
return rows;
};
};
@@ -21,7 +21,7 @@ runTests([
@@ -21,7 +21,7 @@ runTests([
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
- // Media on PostgreSQL (ADR-035): recordings in progress come from its /api/ready, not from a SQLite file.
+ // Media on PostgreSQL (ADR-035): recordings in progress come from its /api/ready, through PostgreSQL.
assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field], ['http-json-count', 'recordings_in_progress']);
assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field], ['http-json-count', 'recordings_in_progress']);
// Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.
// Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.
assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');
assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');
@@ -69,7 +69,7 @@ runTests([
@@ -69,7 +69,7 @@ runTests([
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { vhost: '../../etc/passwd' } } } }), /file name/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { vhost: '../../etc/passwd' } } } }), /file name/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { skipVhosts: ['../../etc/passwd'] } } } }), /vhost file names/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { skipVhosts: ['../../etc/passwd'] } } } }), /vhost file names/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', ready: { url: 'http://evil.example/ready' } } } }), /loopback/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', ready: { url: 'http://evil.example/ready' } } } }), /loopback/);
- assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'sqlite-count', db: '/d.db', sql: 'DELETE FROM vods' } } } }), /SELECT/);
+ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'DELETE FROM vods' } } } }), /SELECT/);
assert.throws(() => normalise({ services: { 'X Y': { ...base, repo: '/opt/x' } } }), /service id/);
assert.throws(() => normalise({ services: { 'X Y': { ...base, repo: '/opt/x' } } }), /service id/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', unitSources: { 'a.service': '../../etc/shadow' } } } }), /inside the checkout/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', unitSources: { 'a.service': '../../etc/shadow' } } } }), /inside the checkout/);
}),
}),
@@ -1,7 +1,6 @@
@@ -1,7 +1,6 @@
'use strict';
'use strict';
/**
/**
- * `ovhost data provision|switch`: the data role's scripts (roles/data/add-service.sh,
- * switch-service.sh) run from the ovhost install, driven against the fake host. The scripts are
+ * `ovhost data provision`: the data role (roles/data/add-service.sh) run from the ovhost install, driven against the fake host. The scripts are
* stubbed at their install path: nothing here touches PostgreSQL, Valkey, systemd or a real
* stubbed at their install path: nothing here touches PostgreSQL, Valkey, systemd or a real
* database, and every invocation is recorded so the tests assert the argv, the env and the exit code.
* database, and every invocation is recorded so the tests assert the argv, the env and the exit code.
*/
*/
@@ -22,7 +21,7 @@ function cli(host, ...argv) {
@@ -22,7 +21,7 @@ function cli(host, ...argv) {
/** Install both scripts at the install path and record every run of them. */
/** Install both scripts at the install path and record every run of them. */
function installScripts(host, { stdout = '', stderr = '', code = 0 } = {}) {
function installScripts(host, { stdout = '', stderr = '', code = 0 } = {}) {
- for (const f of ['add-service.sh', 'switch-service.sh']) host.put(`${DIR}/roles/data/${f}`, '#!/usr/bin/env bash\n', { mode: 0o755, owner: 'root' });
+ for (const f of ['add-service.sh']) host.put(`${DIR}/roles/data/${f}`, '#!/usr/bin/env bash\n', { mode: 0o755, owner: 'root' });
const runs = [];
const runs = [];
const orig = host.exec.run;
const orig = host.exec.run;
host.exec.run = async (cmd, args, opts = {}) => {
host.exec.run = async (cmd, args, opts = {}) => {
@@ -77,96 +76,4 @@ runTests([
@@ -77,96 +76,4 @@ runTests([
assert.match(r.out, /add-service\.sh is not in the ovhost install/);
assert.match(r.out, /add-service\.sh is not in the ovhost install/);
}),
}),
- test('switch passes the inventory units (worker templates as a glob) and dir, plus --sqlite, and passes the exit code through', async () => {
- const host = scenario();
- const raw = JSON.parse(host.read('/etc/openvibe/host.json'));
- raw.services.media.workerUnits = ['[email protected]'];
- host.put('/etc/openvibe/host.json', JSON.stringify(raw), { mode: 0o640 });
- const runs = installScripts(host, { code: 3, stderr: '[switch] still busy after 12 hours: not switching\n' });
-
- const r = await cli(host, 'data', 'switch', 'media', '--sqlite', '/var/lib/openvibe-media/media.db');
- assert.strictEqual(r.code, 3, r.out);
- assert.strictEqual(runs.length, 1);
- assert.strictEqual(runs[0].script, `${DIR}/roles/data/switch-service.sh`);
- assert.deepStrictEqual(runs[0].args, ['media', '/var/lib/openvibe-media/media.db']);
- assert.strictEqual(runs[0].env.SWITCH_DIR, '/opt/openvibe.media');
- assert.strictEqual(runs[0].env.SWITCH_UNITS, 'openvibe-media.service openvibe-media-clip@*.service');
- assert.match(r.out, /still busy after 12 hours/);
- }),
-
- test('switch without --sqlite passes no file and uses the entries for a default service', async () => {
- const host = scenario();
- const runs = installScripts(host);
- const r = await cli(host, 'data', 'switch', 'live');
- assert.strictEqual(r.code, 0, r.out);
- assert.deepStrictEqual(runs[0].args, ['live']);
- assert.strictEqual(runs[0].env.SWITCH_DIR, '/opt/openvibe.live');
- // Live is socket-activated: its socket is stopped too (after the service), or the next request starts the old
- // release again while the import runs.
- assert.strictEqual(runs[0].env.SWITCH_UNITS, 'openvibe-live.service openvibe-live.socket');
- }),
-
- test('switch also stops the units matching unitsMatch', async () => {
- const host = scenario();
- const raw = JSON.parse(host.read('/etc/openvibe/host.json'));
- raw.services.live.unitsMatch = 'openvibe-live*.service';
- host.put('/etc/openvibe/host.json', JSON.stringify(raw), { mode: 0o640 });
- const runs = installScripts(host);
- const r = await cli(host, 'data', 'switch', 'live');
- assert.strictEqual(r.code, 0, r.out);
- assert.strictEqual(runs[0].env.SWITCH_UNITS, 'openvibe-live.service openvibe-live*.service openvibe-live.socket');
- }),
-
- test('without root both are refused and nothing runs (sudo would drop SWITCH_UNITS/SWITCH_DIR)', async () => {
- const host = scenario();
- const runs = installScripts(host);
- host.exec.isRoot = async () => false;
- for (const action of ['provision', 'switch']) {
- const r = await cli(host, 'data', action, 'live');
- assert.strictEqual(r.code, 1, r.out);
- assert.match(r.out, new RegExp(`data ${action} must run as root`));
- }
- assert.deepStrictEqual(runs, []);
- const dry = await cli(host, 'data', 'switch', 'live', '--dry-run');
- assert.strictEqual(dry.code, 0, dry.out);
- }),
-
- test('--dry-run prints the command and runs nothing', async () => {
- const host = scenario();
- const runs = installScripts(host);
-
- const p = await cli(host, 'data', 'provision', 'live', '--dry-run');
- assert.strictEqual(p.code, 0, p.out);
- assert.match(p.out, /^would run: \/opt\/ovhost-install\/roles\/data\/add-service\.sh "live"$/m);
-
- const s = await cli(host, 'data', 'switch', 'live', '--sqlite', '/var/lib/openvibe-live/live.db', '--dry-run');
- assert.strictEqual(s.code, 0, s.out);
- assert.match(s.out, /would run: SWITCH_UNITS="openvibe-live\.service openvibe-live\.socket" SWITCH_DIR="\/opt\/openvibe\.live" \/opt\/ovhost-install\/roles\/data\/switch-service\.sh "live" "\/var\/lib\/openvibe-live\/live\.db"/);
- assert.deepStrictEqual(runs, []);
- }),
-
- test('--json reports the run and still carries no URL or password', async () => {
- const host = scenario();
- installScripts(host, { stdout: 'DATABASE_URL=postgresql://ov_live_app:[email protected]:6432/ov_live\n' });
- const r = await cli(host, 'data', 'provision', 'live', '--json');
- assert.strictEqual(r.code, 0, r.out);
- const doc = JSON.parse(r.out);
- assert.strictEqual(doc.service, 'live');
- assert.strictEqual(doc.action, 'provision');
- assert.strictEqual(doc.ok, true);
- assert.strictEqual(doc.code, 0);
- assert.strictEqual(doc.stdout, 'DATABASE_URL=<redacted-url>\n');
- assert.doesNotMatch(r.out, /postgres(ql)?:\/\//i);
- assert.doesNotMatch(r.out, /ov_live_app:pw/);
- }),
-
- test('usage: data needs a known subcommand and a service', async () => {
- const host = scenario();
- const bad = await cli(host, 'data', 'migrate', 'live');
- assert.strictEqual(bad.code, 1, bad.out);
- assert.match(bad.out, /usage: ovhost data provision/);
- const missing = await cli(host, 'data', 'switch');
- assert.strictEqual(missing.code, 1, missing.out);
- assert.match(missing.out, /data switch needs a <service>/);
- }),
]);
]);
@@ -120,7 +120,7 @@ runTests([
@@ -120,7 +120,7 @@ runTests([
assert.deepStrictEqual(host.restarts(), []);
assert.deepStrictEqual(host.restarts(), []);
}),
}),
- test('refuses to restart Media while a recording is in progress (read-only query as the service user)', async () => {
+ test('refuses to restart Media while a recording is in progress (read-only PostgreSQL query)', async () => {
const host = scenario();
const host = scenario();
const from = host.repo('media').head;
const from = host.repo('media').head;
push(host, 'media', { 'server/index.js': 'media2();' });
push(host, 'media', { 'server/index.js': 'media2();' });
@@ -130,9 +130,8 @@ runTests([
@@ -130,9 +130,8 @@ runTests([
assert.match(r.out, /1 recordings in progress — refusing/);
assert.match(r.out, /1 recordings in progress — refusing/);
assert.strictEqual(host.repo('media').head, from);
assert.strictEqual(host.repo('media').head, from);
assert.deepStrictEqual(host.restarts(), []);
assert.deepStrictEqual(host.restarts(), []);
- const q = host.sqliteCalls.find((c) => c.op === 'query');
- assert.strictEqual(q.as, 'ubuntu');
- assert.strictEqual(q.db, '/opt/openvibe.media/data/media.db');
+ const q = host.psqlCalls.find((c) => c.database === 'ov_media');
+ assert.strictEqual(q.as, 'postgres');
assert.match(q.sql, /^SELECT count\(\*\) AS n FROM vods WHERE is_recording = 1$/);
assert.match(q.sql, /^SELECT count\(\*\) AS n FROM vods WHERE is_recording = 1$/);
}),
}),
@@ -263,18 +262,18 @@ runTests([
@@ -263,18 +262,18 @@ runTests([
assert.deepStrictEqual(host.restarts(), ['openvibe-events.service']);
assert.deepStrictEqual(host.restarts(), ['openvibe-events.service']);
}),
}),
- test('a schema change backs up the declared databases (as the service user) before the restart', async () => {
+ test('a schema change backs up the declared PostgreSQL database before the restart', async () => {
const host = scenario();
const host = scenario();
push(host, 'live', { 'server/db/schema.sql': 'CREATE TABLE b(y);' });
push(host, 'live', { 'server/db/schema.sql': 'CREATE TABLE b(y);' });
const r = await host.cli('deploy', 'live');
const r = await host.cli('deploy', 'live');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
- const b = host.sqliteCalls.find((c) => c.op === 'backup');
+ const b = host.dumps.find((c) => c.database === 'ov_live');
assert.ok(b, 'backup taken');
assert.ok(b, 'backup taken');
- assert.strictEqual(b.as, 'ubuntu');
- assert.match(b.dest, /^\/var\/backups\/openvibe\.staging\/live-\d{8}-\d{6}\/live\.db$/, 'the worker writes to its staging directory');
- const dir = [...host.files.keys()].find((k) => /^\/var\/backups\/openvibe\/live\/\d{8}-\d{6}\/live\.db$/.test(k));
+ assert.strictEqual(b.as, 'postgres');
+ assert.match(b.dest, /^\/var\/backups\/openvibe\.staging\/live-\d{8}-\d{6}\/live\.dump$/);
+ const dir = [...host.files.keys()].find((k) => /^\/var\/backups\/openvibe\/live\/\d{8}-\d{6}\/live\.dump$/.test(k));
assert.ok(dir, 'the copy ends up under /var/backups/openvibe/live/<stamp>/');
assert.ok(dir, 'the copy ends up under /var/backups/openvibe/live/<stamp>/');
- const backupIdx = host.calls.findIndex((c) => c.cmd === 'sqlite-backup');
+ const backupIdx = host.calls.findIndex((c) => c.cmd === 'pg_dump');
const restartIdx = host.calls.findIndex((c) => c.cmd === 'systemctl' && c.args[0] === 'restart');
const restartIdx = host.calls.findIndex((c) => c.cmd === 'systemctl' && c.args[0] === 'restart');
assert.ok(backupIdx < restartIdx, 'backup before restart');
assert.ok(backupIdx < restartIdx, 'backup before restart');
}),
}),
@@ -2,10 +2,10 @@
@@ -2,10 +2,10 @@
/**
/**
* The example inventory's live and media drills (LIVE_DRILL, OpenVibe.Live 8a58aea; MEDIA_DRILL,
* The example inventory's live and media drills (LIVE_DRILL, OpenVibe.Live 8a58aea; MEDIA_DRILL,
* OpenVibe.Media 6e74eb0), run by `ovhost drill` on the fake host:
* OpenVibe.Media 6e74eb0), run by `ovhost drill` on the fake host:
- * - live: the production unit's ExecStart from its own checkout with LIVE_DRILL=1, the copy as
- * DB_PATH and {tmp}/data as DATA_DIR (created before the start); DB-only reads compared, the
+ * - live: the production unit's ExecStart from its own checkout with LIVE_DRILL=1, the scratch database as
+ * DATABASE_URL and {tmp}/data as DATA_DIR (created before the start); DB-only reads compared, the
* values production changes while someone is live ignored; the socket and the service untouched.
* values production changes while someone is live ignored; the socket and the service untouched.
- * - media: MEDIA_DRILL=1, the copy as DB_PATH, every storage path but the thumbnail listing moved
+ * - media: MEDIA_DRILL=1, the scratch database as DATABASE_URL, every storage path but the thumbnail listing moved
* under {tmp}/storage and never created; the /browse pages compared byte for byte.
* under {tmp}/storage and never created; the /browse pages compared byte for byte.
* - a checkout without the switch is refused before anything is created (an older release would
* - a checkout without the switch is refused before anything is created (an older release would
* ignore LIVE_DRILL / MEDIA_DRILL and run its side effects).
* ignore LIVE_DRILL / MEDIA_DRILL and run its side effects).
@@ -17,7 +17,6 @@ const { scenario, test, runTests, SECRET } = require('./helpers');
@@ -17,7 +17,6 @@ const { scenario, test, runTests, SECRET } = require('./helpers');
const { normalise } = require('../lib/inventory');
const { normalise } = require('../lib/inventory');
const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
-const LIVE_DB = '/opt/openvibe.live/data/live.db';
const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));
const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));
@@ -59,14 +58,12 @@ async function liveHost({ switchInCheckout = true, drillBodies = null } = {}) {
@@ -59,14 +58,12 @@ async function liveHost({ switchInCheckout = true, drillBodies = null } = {}) {
unit.dropIns = ['/etc/systemd/system/openvibe-live.service.d/socket.conf'];
unit.dropIns = ['/etc/systemd/system/openvibe-live.service.d/socket.conf'];
host.alivePids.add(unit.mainPid);
host.alivePids.add(unit.mainPid);
host.listeners.set(3000, [{ pid: unit.mainPid, process: 'node' }]);
host.listeners.set(3000, [{ pid: unit.mainPid, process: 'node' }]);
- host.put(LIVE_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
const prod = liveBodies({ drill: false });
const prod = liveBodies({ drill: false });
host.http.set('http://127.0.0.1:3000/api/ready', () => ({ status: 200, body: { ready: true } }));
host.http.set('http://127.0.0.1:3000/api/ready', () => ({ status: 200, body: { ready: true } }));
for (const [p, body] of Object.entries(prod)) host.http.set(`http://127.0.0.1:3000${p}`, () => ({ status: 200, body }));
for (const [p, body] of Object.entries(prod)) host.http.set(`http://127.0.0.1:3000${p}`, () => ({ status: 200, body }));
const counts = { users: 393, channels: 180, managed_streams: 140, streams: 5120, follows: 910, chat_messages: 250000 };
const counts = { users: 393, channels: 180, managed_streams: 140, streams: 5120, follows: 910, chat_messages: 250000 };
- host.sqliteHandler = (db, sql) => {
- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
+ host.psqlHandler = (db, sql) => {
const m = /FROM "([a-z_]+)"/.exec(sql);
const m = /FROM "([a-z_]+)"/.exec(sql);
return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
};
};
@@ -138,7 +135,7 @@ runTests([
@@ -138,7 +135,7 @@ runTests([
const live = inv.services.live.drill;
const live = inv.services.live.drill;
assert.strictEqual(live.supported, true);
assert.strictEqual(live.supported, true);
assert.strictEqual(live.port, 13000);
assert.strictEqual(live.port, 13000);
- assert.deepStrictEqual(live.databases, { live: { engine: 'sqlite', env: 'DB_PATH', dir: false } });
+ assert.deepStrictEqual(live.databases, { live: { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } });
assert.strictEqual(live.env.LIVE_DRILL, '1');
assert.strictEqual(live.env.LIVE_DRILL, '1');
assert.strictEqual(live.env.DATA_DIR, '{tmp}/data');
assert.strictEqual(live.env.DATA_DIR, '{tmp}/data');
assert.strictEqual(live.env.HOST, '127.0.0.1');
assert.strictEqual(live.env.HOST, '127.0.0.1');
@@ -169,7 +166,7 @@ runTests([
@@ -169,7 +166,7 @@ runTests([
['/api/themes', true], ['/api/emotes/global', true], ['/api/streams', true],
['/api/themes', true], ['/api/emotes/global', true], ['/api/streams', true],
['/api/streams/recently-online?limit=20', true], ['/api/streams/channel/japaneseoldguy/live', true],
['/api/streams/recently-online?limit=20', true], ['/api/streams/channel/japaneseoldguy/live', true],
]);
]);
- assert.deepStrictEqual(rec.databases.map((d) => d.name), ['live'], 'rs-companion.db is not restored');
+ assert.deepStrictEqual(rec.databases.map((d) => d.name), ['live'], 'only the declared database is restored');
assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.match]), [['users', true], ['channels', true], ['managed_streams', true], ['streams', true], ['follows', true], ['chat_messages', true]]);
assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.match]), [['users', true], ['channels', true], ['managed_streams', true], ['streams', true], ['follows', true], ['chat_messages', true]]);
assert.strictEqual(host.systemdRuns.length, 1);
assert.strictEqual(host.systemdRuns.length, 1);
@@ -179,9 +176,12 @@ runTests([
@@ -179,9 +176,12 @@ runTests([
assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/live.env', `${rec.dir}/drill.env`]);
assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/live.env', `${rec.dir}/drill.env`]);
for (const p of ['SocketBindAllow=tcp:13000', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'ProtectSystem=strict', `ReadWritePaths=${rec.dir}`, 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);
for (const p of ['SocketBindAllow=tcp:13000', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'ProtectSystem=strict', `ReadWritePaths=${rec.dir}`, 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);
assert.ok(!run.props.some((p) => /BindPaths=|ReadWritePaths=\/opt/.test(p)), 'no bind mounts; production\'s data directory is not writable');
assert.ok(!run.props.some((p) => /BindPaths=|ReadWritePaths=\/opt/.test(p)), 'no bind mounts; production\'s data directory is not writable');
- assert.deepStrictEqual(envOf(host.drillEnv), {
+ const { DATABASE_URL, DATABASE_DIRECT_URL, ...envRest } = envOf(host.drillEnv);
+ assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);
+ assert.match(DATABASE_URL, /^postgresql:\/\//);
+ assert.deepStrictEqual(envRest, {
PORT: '13000', HOST: '127.0.0.1', LIVE_DRILL: '1', DATA_DIR: `${rec.dir}/data`,
PORT: '13000', HOST: '127.0.0.1', LIVE_DRILL: '1', DATA_DIR: `${rec.dir}/data`,
- EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', DB_PATH: `${rec.dir}/db/live.db`,
+ EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', VALKEY_URL: 'redis://127.0.0.1:9/0',
});
});
assert.ok(host.dataDirAtStart, 'DATA_DIR exists before the instance starts');
assert.ok(host.dataDirAtStart, 'DATA_DIR exists before the instance starts');
@@ -190,7 +190,7 @@ runTests([
@@ -190,7 +190,7 @@ runTests([
assert.deepStrictEqual(stateChanges(host), [], 'no unit was started, stopped or restarted');
assert.deepStrictEqual(stateChanges(host), [], 'no unit was started, stopped or restarted');
assert.deepStrictEqual(host.socketViolations(), []);
assert.deepStrictEqual(host.socketViolations(), []);
assert.strictEqual(host.units.get('openvibe-live.socket').active, 'active');
assert.strictEqual(host.units.get('openvibe-live.socket').active, 'active');
- assert.strictEqual(host.read(LIVE_DB), 'sqlite-production');
+ assert.ok(host.pgDatabases.has('ov_live'), 'production database untouched');
assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/live.jsonl')).includes(SECRET));
assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/live.jsonl')).includes(SECRET));
}),
}),
@@ -231,7 +231,7 @@ runTests([
@@ -231,7 +231,7 @@ runTests([
assert.match(DATABASE_URL, new RegExp(`^postgresql://${scratch}:[0-9a-f]+@127\\.0\\.0\\.1:5432/${scratch}$`));
assert.match(DATABASE_URL, new RegExp(`^postgresql://${scratch}:[0-9a-f]+@127\\.0\\.0\\.1:5432/${scratch}$`));
assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);
assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);
assert.strictEqual(rec.databases[0].integrity, 'pg_restore --list ok');
assert.strictEqual(rec.databases[0].integrity, 'pg_restore --list ok');
- assert.deepStrictEqual(host.drillDirAtStart, ['db', 'drill.env', 'pg', 'pg/media.dump'], 'the dump copy and the override file; storage is never created');
+ assert.deepStrictEqual(host.drillDirAtStart, ['drill.env', 'pg', 'pg/media.dump'], 'the dump copy and the override file; storage is never created');
assert.deepStrictEqual(stateChanges(host), []);
assert.deepStrictEqual(stateChanges(host), []);
assert.ok(host.pgDatabases.has('ov_media'), 'the production database is untouched');
assert.ok(host.pgDatabases.has('ov_media'), 'the production database is untouched');
assert.ok(host.droppedDatabases.some((d) => d.name === scratch), 'the scratch database is dropped');
assert.ok(host.droppedDatabases.some((d) => d.name === scratch), 'the scratch database is dropped');
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
'use strict';
'use strict';
/**
/**
* ovhost drill for a PostgreSQL service (lib/dbengine.js): the artifact is a `.dump` in the backup,
* ovhost drill for a PostgreSQL service (lib/dbengine.js): the artifact is a `.dump` in the backup,
- * not a SQLite file. The drill copies it into <tmp>/pg owned by postgres, verifies it with
+ * a PostgreSQL dump. The drill copies it into <tmp>/pg owned by postgres, verifies it with
* `pg_restore --list`, creates a scratch database and login role, restores into them, points
* `pg_restore --list`, creates a scratch database and login role, restores into them, points
* drill.env at the scratch database (and at a closed Valkey), and always drops both again in a
* drill.env at the scratch database (and at a closed Valkey), and always drops both again in a
* finally. Covered: the generated URLs and their password, the refusal when an override is missing,
* finally. Covered: the generated URLs and their password, the refusal when an override is missing,
@@ -177,7 +177,7 @@ runTests([
@@ -177,7 +177,7 @@ runTests([
assert.deepStrictEqual(cp.args, ['-o', 'postgres', '-m', '0600', '-T', '--', `${host.backupDir}/trade.dump`, `${tmp}/pg/trade.dump`]);
assert.deepStrictEqual(cp.args, ['-o', 'postgres', '-m', '0600', '-T', '--', `${host.backupDir}/trade.dump`, `${tmp}/pg/trade.dump`]);
assert.strictEqual(cp.privileged, true);
assert.strictEqual(cp.privileged, true);
- // The Markdown row names pg_restore --list, not pragma integrity_check.
+ // The Markdown row reports the dump archive check.
assert.match(r.out, /`pg_restore --list` \(trade\) = ok/);
assert.match(r.out, /`pg_restore --list` \(trade\) = ok/);
assert.strictEqual(rec.markdown, r.out.split('\n').pop());
assert.strictEqual(rec.markdown, r.out.split('\n').pop());
}),
}),
deleted file mode 100644
deleted file mode 100644
@@ -1,725 +0,0 @@
@@ -1,725 +0,0 @@
-'use strict';
-/**
- * ovhost drill against the fake host: the full flow, integrity failure, readiness timeout, an
- * instance that dies, mismatch reporting, cleanup on failure, refusals (not root, port in use,
- * unsupported, no backup), and the guarantee that nothing outside the drill directory, the lock and
- * the drill log is written.
- */
-const assert = require('assert');
-const crypto = require('crypto');
-const fs = require('fs');
-const path = require('path');
-const { scenario, test, runTests, SECRET } = require('./helpers');
-const { normalise } = require('../lib/inventory');
-const { compareBodies, verifyObjectBlobs } = require('../lib/commands/drill');
-
-const DRILL_DIR = '/var/lib/openvibe-drills';
-const PROD_DB = '/var/lib/openvibe-tools/tools.db';
-const OBJECTS_DIR = '/var/lib/openvibe-host-api/objects';
-const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';
-
-/** A content-addressed blob under the Host object-store layout: <root>/projects/<prj>/<aa>/<sha256>. */
-function blob(root, data) {
- const sha = crypto.createHash('sha256').update(data).digest('hex');
- return { sha, path: `${root}/projects/prj_1/${sha.slice(0, 2)}/${sha}` };
-}
-
-function toolsEntry(drillOverrides = {}) {
- return {
- repo: '/opt/openvibe.tools',
- units: ['openvibe-tools.service'],
- envFile: '/etc/openvibe/tools.env',
- port: 4200,
- ready: { url: 'http://127.0.0.1:4200/api/ready', timeoutSeconds: 30 },
- databases: [{ name: 'tools', path: PROD_DB }],
- drill: {
- port: 14200,
- databases: { tools: 'TOOLS_DB_PATH' },
- env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words' },
- dirs: ['{tmp}/data'],
- compare: [{ path: '/api/pastes?limit=5' }, { path: '/api/v1/pulse?limit=5', ignore: ['generated_at'] }],
- counts: [{ db: 'tools', table: 'pastes' }],
- ...drillOverrides,
- },
- };
-}
-
-/**
- * The standard fake host plus Tools in production, with one `ovhost backup tools` taken.
- * The drill instance's behaviour is controlled through `host.drill`.
- */
-async function drillScenario({ drill: drillOverrides, backup = true, objects = null, blobs = [] } = {}) {
- const host = scenario();
- const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
- const tools = toolsEntry(drillOverrides);
- if (objects) tools.objects = objects;
- doc.services.tools = tools;
- host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
-
- const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });
- const sha = repo.commit({ 'package.json': '{"name":"openvibe-tools"}', 'server/index.js': 'tools();' }, { message: 'initial' });
- repo.publish(sha);
- repo.checkout(sha);
- host.addUnit('openvibe-tools.service', { mainPid: 2222 });
- host.alivePids.add(2222);
- host.listeners.set(4200, [{ pid: 2222, process: 'node' }]);
- host.put('/etc/systemd/system/openvibe-tools.service', [
- '[Service]',
- 'User=ubuntu',
- 'WorkingDirectory=/opt/openvibe.tools',
- 'ExecStart=/usr/bin/node server/index.js',
- `Environment=NODE_ENV=production "SESSION_SECRET=${SECRET}-unit"`,
- '',
- ].join('\n'));
- host.put('/etc/openvibe/tools.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.tools\n`, { mode: 0o600 });
- host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
-
- host.http.set('http://127.0.0.1:4200/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
- host.http.set('http://127.0.0.1:4200/api/pastes?limit=5', () => ({ status: 200, body: PASTES }));
- host.http.set('http://127.0.0.1:4200/api/v1/pulse?limit=5', () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:00.000Z' } }));
-
- host.drill = {
- integrity: [{ integrity_check: 'ok' }],
- productionCount: 892,
- restoredCount: 892,
- ready: () => ({ status: 200, body: { status: 'ready' } }),
- pastes: () => ({ status: 200, body: PASTES }),
- pulse: () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:07.000Z' } }),
- onStart: null,
- ignoreTerm: false,
- ignoreKill: false,
- };
- host.sqliteHandler = (db, sql) => {
- if (/integrity_check/.test(sql)) return host.drill.integrity;
- if (/count\(\*\) AS n FROM "pastes"/.test(sql)) return [{ n: db === PROD_DB ? host.drill.productionCount : host.drill.restoredCount }];
- if (/is_recording/.test(sql)) return [{ n: host.recording }];
- return [{ n: 0 }];
- };
- host.onSystemdRun = (spec) => {
- if (host.drill.onStart) { const r = host.drill.onStart(spec); if (r) return r; }
- host.listeners.set(14200, [{ pid: spec.pid, process: 'node' }]);
- const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
- host.http.set('http://127.0.0.1:14200/api/ready', alive(() => host.drill.ready()));
- host.http.set('http://127.0.0.1:14200/api/pastes?limit=5', alive(() => host.drill.pastes()));
- host.http.set('http://127.0.0.1:14200/api/v1/pulse?limit=5', alive(() => host.drill.pulse()));
- return undefined;
- };
- host.onKill = (pid, signal) => {
- if (signal === 'SIGTERM' && host.drill.ignoreTerm) return false;
- if (signal === 'SIGKILL' && host.drill.ignoreKill) return false;
- return undefined;
- };
-
- for (const b of blobs) host.put(b.path, b.content, { owner: 'ubuntu', mode: 0o644 });
-
- if (backup) {
- const b = await host.cli('backup', 'tools', '--json');
- assert.strictEqual(b.code, 0, b.out);
- host.backupDir = JSON.parse(b.out).dir;
- }
- host.advance(60 * 1000);
- return host;
-}
-
-/** A Tools-like service: a gateway and apps, each app its own unit and port, data directories. */
-function appsEntry(drill = {}) {
- const d = {
- port: 14016,
- unit: 'openvibe-apps-docs.service',
- productionPort: 4016,
- databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'docs-jobs': { env: 'DATA_DIR', dir: true }, 'yt-analytics': { dir: '{tmp}/yt-data' } },
- env: { PORT: '{port}', HOST: '127.0.0.1', TOOLS_JOB_RESULTS: 'local', EVENTS_PUBLISH: 'off' },
- bind: [{ from: '{tmp}/yt-data', to: '/opt/openvibe.apps/apps/yt/data' }],
- requires: [{ file: 'apps/_shared/jobs/index.js', contains: 'TOOLS_JOB_RESULTS' }],
- ready: '/api/ready',
- compare: ['/release.json'],
- counts: [{ db: 'docs-jobs', table: 'tool_jobs' }],
- ...drill,
- };
- for (const [k, v] of Object.entries(d)) if (v === null) delete d[k];
- return {
- owner: 'ubuntu',
- repo: '/opt/openvibe.apps',
- units: ['openvibe-apps.service', 'openvibe-apps-docs.service', 'openvibe-apps-yt.service'],
- envFile: '/etc/openvibe/apps.env',
- port: 4001,
- ready: { url: 'http://127.0.0.1:4001/api/ready', timeoutSeconds: 30 },
- databases: [
- { name: 'docs-analytics', path: '/opt/openvibe.apps/apps/docs/data/analytics.db' },
- { name: 'docs-jobs', path: '/opt/openvibe.apps/apps/docs/data/jobs.db' },
- { name: 'yt-analytics', path: '/opt/openvibe.apps/apps/yt/data/analytics.db' },
- ],
- drill: d,
- };
-}
-
-async function appsScenario({ switchInCheckout = true, drill } = {}) {
- const host = scenario();
- const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
- doc.services.apps = appsEntry(drill);
- host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
- const repo = host.createRepo('/opt/openvibe.apps', { owner: 'ubuntu' });
- const sha = repo.commit({
- 'package.json': '{"name":"apps"}',
- 'apps/_shared/jobs/index.js': switchInCheckout ? "const want = env.TOOLS_JOB_RESULTS || 'local';" : 'jobs();',
- }, { message: 'initial' });
- repo.publish(sha);
- repo.checkout(sha);
- const unit = (name, dir, port) => {
- host.addUnit(name, { mainPid: 3000 + port });
- host.alivePids.add(3000 + port);
- host.listeners.set(port, [{ pid: 3000 + port, process: 'node' }]);
- host.put(`/etc/systemd/system/${name}`, ['[Service]', 'User=ubuntu', `WorkingDirectory=/opt/openvibe.apps/apps/${dir}`, 'Environment=NODE_ENV=production', `Environment=PORT=${port}`, `ExecStart=/usr/bin/env node /opt/openvibe.apps/apps/${dir}/server/index.js`, ''].join('\n'));
- };
- unit('openvibe-apps.service', 'gateway', 4001);
- unit('openvibe-apps-docs.service', 'docs', 4016);
- unit('openvibe-apps-yt.service', 'yt', 4013);
- host.put('/etc/openvibe/apps.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\n`, { mode: 0o600 });
- for (const db of appsEntry().databases) host.put(db.path, `sqlite-${db.name}`, { owner: 'ubuntu', mode: 0o640 });
- host.http.set('http://127.0.0.1:4001/release.json', () => ({ status: 200, body: '{"service":"gateway"}' }));
- host.http.set('http://127.0.0.1:4016/release.json', () => ({ status: 200, body: '{"service":"docs"}' }));
- host.sqliteHandler = (db, sql) => {
- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
- if (/count\(\*\)/.test(sql)) return [{ n: 3 }];
- return [{ n: 0 }];
- };
- host.onSystemdRun = (spec) => {
- const envFile = spec.envFiles[spec.envFiles.length - 1];
- host.drillEnv = host.read(envFile);
- host.listeners.set(14016, [{ pid: spec.pid, process: 'node' }]);
- const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
- host.http.set('http://127.0.0.1:14016/api/ready', alive(() => ({ status: 200, body: { status: 'ready' } })));
- host.http.set('http://127.0.0.1:14016/release.json', alive(() => ({ status: 200, body: '{"service":"docs"}' })));
- return undefined;
- };
- const b = await host.cli('backup', 'apps', '--json');
- assert.strictEqual(b.code, 0, b.out);
- host.advance(60 * 1000);
- return host;
-}
-
-/** Every file entry as "type:owner:mode:content", to diff the fake filesystem. */
-function fsSnapshot(host) {
- const m = new Map();
- for (const [k, v] of host.files) m.set(k, `${v.type}:${v.owner}:${v.mode}:${v.type === 'file' ? String(v.content) : v.target || ''}`);
- return m;
-}
-
-function changedPaths(before, after) {
- const out = [];
- for (const [k, v] of after) if (before.get(k) !== v) out.push(k);
- for (const k of before.keys()) if (!after.has(k)) out.push(k);
- return out.sort();
-}
-
-function lastLog(host) {
- const text = host.read('/var/lib/openvibe-host/drills/tools.jsonl');
- const lines = text.trim().split('\n');
- return JSON.parse(lines[lines.length - 1]);
-}
-
-const allowedWrite = (p, tmp) => p === DRILL_DIR || p === tmp || p.startsWith(`${tmp}/`)
- || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/tools.jsonl'
- || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/tools-drill.lock';
-
-runTests([
- test('full flow: restore, integrity, sandboxed second instance, identical answers, stop by pid, cleanup, log row', async () => {
- const host = await drillScenario();
- const before = fsSnapshot(host);
- const callsBefore = host.calls.length;
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 0, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.result, 'passed');
- assert.strictEqual(rec.failure, null);
- assert.strictEqual(rec.backup, host.backupDir);
- const tmp = rec.dir;
- assert.match(tmp, /^\/var\/lib\/openvibe-drills\/tools-\d{8}-\d{6}$/);
- assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['tools', 'ok', `${tmp}/db/tools.db`]]);
- assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/pastes?limit=5', true], ['/api/v1/pulse?limit=5', true]]);
- assert.deepStrictEqual(rec.counts, [{ db: 'tools', table: 'pastes', production: 892, restored: 892, match: true }]);
- assert.strictEqual(rec.ready.ok, true);
- assert.strictEqual(rec.stop.ok, true);
- assert.strictEqual(rec.stop.signal, 'SIGTERM');
-
- const calls = host.calls.slice(callsBefore);
- // The copy is made as the service user; integrity and counts run as the service user.
- // Backups are root-only: root copies, and the copy belongs to the service user, 0600.
- const cp = calls.find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/db/tools.db`));
- assert.strictEqual(cp.privileged, true);
- assert.deepStrictEqual(cp.args, ['-o', 'ubuntu', '-m', '0600', '-T', '--', `${host.backupDir}/tools.db`, `${tmp}/db/tools.db`]);
- const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));
- assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/tools.db`, 'ubuntu']);
-
- // systemd-run: production ExecStart/WorkingDirectory, production env file then the override file,
- // sandbox properties, loopback-only networking, the service user.
- assert.strictEqual(host.systemdRuns.length, 1);
- const run = host.systemdRuns[0];
- assert.strictEqual(run.uid, 'ubuntu');
- assert.strictEqual(run.cwd, '/opt/openvibe.tools');
- assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js']);
- assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/tools.env', `${tmp}/drill.env`]);
- for (const p of ['ProtectSystem=strict', `ReadWritePaths=${tmp}`, 'PrivateTmp=yes', 'NoNewPrivileges=yes', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'SocketBindAllow=tcp:14200', 'SocketBindDeny=any', 'Environment=NODE_ENV=production']) {
- assert.ok(run.props.includes(p), `systemd-run -p ${p}`);
- }
- assert.ok(!run.props.some((p) => p.includes('SESSION_SECRET')), 'a secret-looking unit Environment= is not passed on');
- const runCall = calls.find((c) => c.cmd === 'systemd-run');
- assert.strictEqual(runCall.privileged, true);
-
- // Stopped by its own pid with SIGTERM; production's pid is never signalled; no pkill.
- const kills = calls.filter((c) => c.cmd === 'kill');
- assert.deepStrictEqual(kills.map((c) => c.args), [['SIGTERM', run.pid]]);
- assert.ok(!calls.some((c) => /pkill|killall/.test(c.cmd)));
- assert.ok(host.alivePids.has(2222), 'production still running');
- assert.strictEqual(host.units.get('openvibe-tools.service').restarts, 0);
- assert.ok(!host.restarts().length, 'nothing restarted');
-
- // Only the drill directory (now removed), the lock (released) and the drill log changed.
- const after = fsSnapshot(host);
- const changed = changedPaths(before, after);
- for (const p of changed) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);
- assert.ok(!host.files.has(tmp), 'temp dir removed');
- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'), 'lock released');
-
- // Secrets: nothing ovhost printed, logged or passed on carries a value.
- const everything = [r.out, host.read('/var/lib/openvibe-host/drills/tools.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');
- assert.ok(!everything.includes(SECRET), 'a secret value leaked');
- assert.ok(!host.reads.includes('/etc/openvibe/tools.env'), 'ovhost never reads the env file for a drill');
-
- // Markdown row ready for docs/restore-drills.md.
- assert.match(r.out, /\| 2026-09-22 12:01 \| tools \| `\/var\/backups\/openvibe\/tools\/20260922-120000\/` \(`ovhost drill tools`\) \| `pragma integrity_check` \(tools\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);
- assert.strictEqual(rec.markdown, r.out.split('\n').pop());
- }),
-
- test('--keep leaves the drill directory; the override file holds inventory values only', async () => {
- const host = await drillScenario();
- const before = fsSnapshot(host);
- const r = await host.cli('drill', 'tools', '--keep');
- assert.strictEqual(r.code, 0, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.kept, true);
- const tmp = rec.dir;
- assert.strictEqual(host.files.get(tmp).owner, 'ubuntu');
- assert.strictEqual(host.files.get(tmp).mode, 0o700);
- assert.strictEqual(host.files.get(`${tmp}/db/tools.db`).owner, 'ubuntu');
- assert.strictEqual(host.read(`${tmp}/db/tools.db`), host.read(`${host.backupDir}/tools.db`));
- assert.strictEqual(host.files.get(`${tmp}/data`).owner, 'ubuntu');
- const env = host.read(`${tmp}/drill.env`);
- for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `TOOLS_DB_PATH=${tmp}/db/tools.db`]) {
- assert.ok(env.split('\n').includes(line), `drill.env has ${line}`);
- }
- assert.ok(!env.includes(SECRET));
- for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);
- // The production database and the backup are untouched.
- assert.strictEqual(host.read(PROD_DB), 'sqlite-production');
- }),
-
- test('integrity failure: nothing is started, the copy is removed, the log says failed', async () => {
- const host = await drillScenario();
- host.drill.integrity = [{ integrity_check: '*** in database main ***' }, { integrity_check: 'Page 5: btree page has a bad cell' }];
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.result, 'failed');
- assert.strictEqual(rec.failure.stage, 'integrity');
- assert.match(rec.failure.message, /bad cell/);
- assert.strictEqual(host.systemdRuns.length, 0);
- assert.ok(!host.files.has(rec.dir));
- assert.match(r.out, /\| failed \|$/);
- }),
-
- test('readiness timeout: the instance is stopped by its pid and the directory removed', async () => {
- const host = await drillScenario();
- host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'ready');
- assert.match(rec.failure.message, /not ready after 30s \(503\)/);
- assert.strictEqual(rec.compare.length, 0, 'nothing compared');
- const pid = host.systemdRuns[0].pid;
- assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', pid]]);
- assert.ok(!host.alivePids.has(pid));
- assert.ok(!host.files.has(rec.dir));
- assert.ok(!host.listeners.has(14200));
- }),
-
- test('an instance that exits before it is ready fails the drill without signalling anything', async () => {
- const host = await drillScenario();
- let probes = 0;
- host.drill.ready = () => { probes += 1; if (probes === 2) host.alivePids.delete(host.systemdRuns[0].pid); return { status: 503 }; };
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'ready');
- assert.match(rec.failure.message, /exited before it was ready/);
- assert.ok(!host.calls.some((c) => c.cmd === 'kill'));
- assert.ok(!host.files.has(rec.dir));
- }),
-
- test('mismatches are reported per path and per table; volatile keys are ignored only where declared', async () => {
- const host = await drillScenario();
- host.drill.pastes = () => ({ status: 200, body: PASTES.replace('world', 'World') });
- host.drill.pulse = () => ({ status: 200, body: { items: [{ id: 2 }], generated_at: 'different, but ignored' } });
- host.drill.productionCount = 900;
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'compare');
- const [pastes, pulse] = rec.compare;
- assert.strictEqual(pastes.match, false);
- assert.match(pastes.detail, /first difference at byte 55/);
- assert.strictEqual(pulse.match, false);
- assert.match(pulse.detail, /\$\.items\[0\]\.id/);
- assert.ok(!/generated_at/.test(pulse.detail));
- assert.deepStrictEqual(rec.counts[0], { db: 'tools', table: 'pastes', production: 900, restored: 892, match: false });
- assert.match(rec.markdown, /`\/api\/pastes\?limit=5` differs \(bodies differ/);
- assert.match(rec.markdown, /pastes 900 ≠ 892 \| failed \|$/);
- // Still stopped and cleaned up.
- assert.ok(!host.alivePids.has(host.systemdRuns[0].pid));
- assert.ok(!host.files.has(rec.dir));
- }),
-
- test('acceptance: the restored instance alone answers each declared GET with its status and JSON keys (WS-S task 2)', async () => {
- const accept = [{ path: '/api/ready', keys: ['status'] }, { path: '/api/v1/pulse?limit=5', keys: ['items', 'generated_at'] }];
- let host = await drillScenario({ drill: { acceptance: accept } });
- let r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 0, r.out);
- let rec = lastLog(host);
- assert.deepStrictEqual(rec.acceptance, [{ path: '/api/ready', ok: true }, { path: '/api/v1/pulse?limit=5', ok: true }]);
- assert.match(rec.markdown, /acceptance `\/api\/ready`, `\/api\/v1\/pulse\?limit=5`/);
-
- host = await drillScenario({ drill: { acceptance: [{ path: '/api/v1/pulse?limit=5', keys: ['items', 'cursor'] }, { path: '/api/ready', status: 204 }] } });
- r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'acceptance');
- assert.deepStrictEqual(rec.acceptance.map((a) => [a.ok, a.detail || null]), [[false, 'missing cursor'], [false, 'status 200, expected 204']]);
- assert.match(rec.markdown, /acceptance failed: /);
- assert.ok(!host.files.has(rec.dir), 'still cleaned up');
- }),
-
- test('acceptance entries are validated in the inventory', async () => {
- const host = await drillScenario({ drill: { acceptance: [{ path: 'no-slash' }] }, backup: false });
- const r = await host.cli('drill', 'tools');
- assert.notStrictEqual(r.code, 0);
- assert.match(r.out, /acceptance\[0\]\.path must be a path/);
- }),
-
- test('compare: a status mismatch or a failing production answer is never a match', () => {
- const e = { path: '/x', ignore: [] };
- assert.strictEqual(compareBodies(e, { status: 500, body: 'e' }, { status: 500, body: 'e' }).match, false);
- assert.match(compareBodies(e, { status: 200, body: 'a' }, { status: 404, body: 'a' }).detail, /status 404/);
- assert.strictEqual(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '{"a":1,"t":1}' }, { status: 200, body: '{"a":1,"t":2}' }).match, true);
- assert.match(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '<html>' }, { status: 200, body: '<html>' }).detail, /not JSON/);
- }),
-
- test('cleanup on failure: systemd-run fails → directory removed, lock released, row logged', async () => {
- const host = await drillScenario();
- const before = fsSnapshot(host);
- host.drill.onStart = () => ({ code: 1, stderr: 'Failed to start transient service unit: Unit already exists.' });
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'start');
- assert.match(rec.failure.message, /systemd-run failed/);
- assert.ok(!host.files.has(rec.dir));
- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'));
- for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, rec.dir), `unexpected write: ${p}`);
- // A second drill can run straight after.
- host.drill.onStart = null;
- host.advance(1000);
- assert.strictEqual((await host.cli('drill', 'tools')).code, 0);
- }),
-
- test('an instance that ignores SIGTERM gets SIGKILL; one that survives SIGKILL keeps its directory and fails', async () => {
- let host = await drillScenario();
- host.drill.ignoreTerm = true;
- let r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 0, r.out);
- let rec = lastLog(host);
- assert.strictEqual(rec.stop.signal, 'SIGKILL');
- assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args[0]), ['SIGTERM', 'SIGKILL']);
-
- host = await drillScenario();
- host.drill.ignoreTerm = true;
- host.drill.ignoreKill = true;
- r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'stop');
- assert.strictEqual(rec.kept, true);
- assert.ok(host.files.has(rec.dir), 'directory kept while the instance may still hold it');
- assert.ok(host.alivePids.has(2222), 'production untouched');
- assert.ok(!host.calls.some((c) => c.cmd === 'kill' && c.args[1] === 2222));
- }),
-
- test('refusals: not root, port in use, unsupported service, no drill block, no backup, missing database copy', async () => {
- let host = await drillScenario();
- host.exec.isRoot = async () => false;
- let r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 1);
- assert.match(r.out, /must run as root/);
- assert.strictEqual(host.systemdRuns.length, 0);
-
- host = await drillScenario();
- host.listeners.set(14200, [{ pid: 777, process: 'python3' }]);
- const before = fsSnapshot(host);
- r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 1);
- assert.match(r.out, /port 14200 is already in use \(python3 pid 777\)/);
- assert.strictEqual(host.systemdRuns.length, 0);
- assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');
-
- host = await drillScenario({ drill: { supported: false, reason: 'workers have no off switch' } });
- r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 1);
- assert.match(r.out, /does not support restore drills: workers have no off switch/);
-
- host = await drillScenario();
- r = await host.cli('drill', 'media');
- assert.strictEqual(r.code, 1);
- assert.match(r.out, /media has no drill block/);
-
- host = await drillScenario({ backup: false });
- r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 1);
- assert.match(r.out, /no good ovhost backup recorded for tools/);
-
- host = await drillScenario({ backup: false });
- host.ensureDir('/srv/old-backup');
- r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup');
- assert.strictEqual(r.code, 1);
- assert.match(r.out, /has no copy of the tools database/);
- host.put('/srv/old-backup/tools.db', 'older sqlite', { owner: 'ubuntu' });
- r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup', '--json');
- assert.strictEqual(r.code, 0, r.out);
- assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');
- r = await host.cli('drill', 'tools', '--backup', 'relative/dir');
- assert.strictEqual(r.code, 1);
- }),
-
- test('one app of a multi-unit service: drill.unit, data-directory databases, a bind mount, its own production port', async () => {
- const host = await appsScenario();
- const r = await host.cli('drill', 'apps');
- assert.strictEqual(r.code, 0, r.out);
- const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/apps.jsonl').trim().split('\n').pop());
- assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));
- const tmp = rec.dir;
- // Data-directory databases keep their production file names under {tmp}/data.
- assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.copy]).sort(), [['docs-analytics', `${tmp}/data/analytics.db`], ['docs-jobs', `${tmp}/data/jobs.db`], ['yt-analytics', `${tmp}/yt-data/analytics.db`]].sort());
- const run = host.systemdRuns[0];
- // The docs unit's ExecStart and WorkingDirectory, not the gateway's.
- assert.strictEqual(run.cwd, '/opt/openvibe.apps/apps/docs');
- assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.apps/apps/docs/server/index.js']);
- assert.ok(run.props.includes('Environment=NODE_ENV=production'));
- assert.ok(!run.props.some((p) => p.startsWith('Environment=PORT=')), 'the unit\'s PORT never reaches the drill');
- assert.ok(run.props.includes(`BindPaths=${tmp}/yt-data:/opt/openvibe.apps/apps/yt/data`), run.props.join('\n'));
- const env = host.drillEnv;
- assert.match(env, new RegExp(`^DATA_DIR=${tmp}/data$`, 'm'));
- assert.match(env, /^PORT=14016$/m);
- assert.ok(!/analytics\.db/.test(env), 'a data-directory database is found by its file name, not an env var');
- // Compared with the docs app's production port, not the gateway's.
- assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/release.json', true]]);
- assert.ok(host.calls.some((c) => c.cmd === 'curl' && c.args[0] === 'http://127.0.0.1:4016/release.json'));
- assert.ok(!host.calls.some((c) => c.cmd === 'curl' && c.args[0].startsWith('http://127.0.0.1:4001/')));
- assert.ok(!host.restarts().length, 'nothing restarted');
- }),
-
- test('requires: a checkout without the drill switch is refused before anything is created', async () => {
- const host = await appsScenario({ switchInCheckout: false });
- const before = fsSnapshot(host);
- const r = await host.cli('drill', 'apps');
- assert.strictEqual(r.code, 1, r.out);
- assert.match(r.out, /TOOLS_JOB_RESULTS in apps\/_shared\/jobs\/index\.js/);
- assert.match(r.out, /Deploy a release with the drill switch first/);
- assert.strictEqual(host.systemdRuns.length, 0);
- assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), []);
- }),
-
- test('inventory: unit, productionPort, bind, requires and data-directory databases are validated', () => {
- const entry = appsEntry();
- const inv = (drill) => normalise({ services: { apps: { ...entry, drill: { ...entry.drill, ...drill } } } });
- assert.strictEqual(inv({}).services.apps.drill.unit, 'openvibe-apps-docs.service');
- assert.throws(() => inv({ unit: 'openvibe-live.service' }), /unit must be one of apps's units/);
- assert.throws(() => inv({ productionPort: 14016 }), /productionPort must differ/);
- assert.throws(() => inv({ bind: [{ from: '/tmp/x', to: '/opt/openvibe.apps/apps/yt/data' }] }), /bind\[0\]\.from/);
- assert.throws(() => inv({ bind: [{ from: '{tmp}/x', to: '/etc/openvibe' }] }), /inside the checkout/);
- assert.throws(() => inv({ bind: [{ from: '{tmp}/x', to: '/opt/openvibe.apps/../etc' }] }), /plain absolute path/);
- assert.throws(() => inv({ requires: [{ file: '../etc/passwd', contains: 'x' }] }), /inside the checkout/);
- assert.throws(() => inv({ requires: [{ file: 'a.js' }] }), /contains/);
- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR' } } }), /"dir": true/);
- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'yt-analytics': { dir: true } } }), /would both be \{tmp\}\/data\/analytics\.db/);
- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: '/var/lib/x' } } }), /dir must be true or/);
- assert.throws(() => inv({ databases: { 'docs-analytics': { env: 'DATA_DIR', dir: true }, 'docs-jobs': { env: 'DATA_DIR', dir: '{tmp}/other' } } }), /two different places/);
- const multi = { ...entry, drill: { ...entry.drill } };
- delete multi.drill.unit;
- normalise({ services: { apps: multi } }); // valid inventory; the drill itself refuses (below)
- }),
-
- test('objects: the declared object directory is archived, restored into scratch, and every blob sha256 checked', async () => {
- const a = 'tenant page a\n';
- const b = 'tenant asset b\n';
- const pa = blob(OBJECTS_DIR, a);
- const pb = blob(OBJECTS_DIR, b);
- const host = await drillScenario({
- objects: [{ name: 'objects', path: OBJECTS_DIR }],
- blobs: [{ path: pa.path, content: a }, { path: pb.path, content: b }],
- drill: {
- env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words', HOST_STORAGE_DIR: '{tmp}/objects' },
- dirs: ['{tmp}/data', '{tmp}/objects'],
- },
- });
- const archive = `${host.backupDir}/objects.tar.gz`;
- assert.deepStrictEqual([host.files.get(archive).owner, host.files.get(archive).mode], ['root', 0o600], 'the archive is an ordinary root-only copy');
- const r = await host.cli('drill', 'tools', '--keep');
- assert.strictEqual(r.code, 0, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.kept, true);
- assert.deepStrictEqual(rec.objects.map((o) => [o.name, o.blobs, o.skipped, o.integrity]), [['objects', 2, 0, 'sha256 ok']]);
- const tmp = rec.dir;
- // The archive is listed first, then extracted without restoring production's ownership or modes.
- assert.ok(host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-tvzf' && c.args[1] === archive), 'listed before extraction');
- const extract = host.calls.find((c) => c.cmd === 'tar' && c.args[0] === '-xzf');
- assert.deepStrictEqual(extract.args, ['-xzf', archive, '-C', `${tmp}/objects`, '--no-same-owner', '--no-same-permissions']);
- assert.strictEqual(extract.privileged, true);
- // Restored into the drill's own scratch directory, never the live object directory.
- assert.strictEqual(rec.objects[0].copy, `${tmp}/objects`);
- assert.ok(rec.objects[0].copy.startsWith(`${tmp}/`));
- assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pa.sha.slice(0, 2)}/${pa.sha}`), a);
- assert.strictEqual(host.read(`${tmp}/objects/projects/prj_1/${pb.sha.slice(0, 2)}/${pb.sha}`), b);
- // The instance is pointed at the restored directory, not production's.
- assert.ok(host.read(`${tmp}/drill.env`).split('\n').includes(`HOST_STORAGE_DIR=${tmp}/objects`));
- assert.match(rec.markdown, /object store `objects` 2 blob\(s\) sha256 ok/);
- // The live objects were never written or moved.
- assert.strictEqual(host.read(pa.path), a);
- assert.strictEqual(host.read(pb.path), b);
- }),
-
- test('objects: a blob whose bytes do not match the sha256 in its name fails the drill before anything starts', async () => {
- const good = 'good blob\n';
- const p = blob(OBJECTS_DIR, good);
- const wrongName = 'a'.repeat(64);
- const host = await drillScenario({
- objects: [{ name: 'objects', path: OBJECTS_DIR }],
- blobs: [{ path: p.path, content: good }, { path: `${OBJECTS_DIR}/projects/prj_1/aa/${wrongName}`, content: 'not its name\n' }],
- });
- const before = fsSnapshot(host);
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'objects');
- assert.match(rec.failure.message, /blob\(s\) do not match their name/);
- assert.match(rec.failure.message, /aa\/a{64}/);
- assert.strictEqual(host.systemdRuns.length, 0, 'nothing is started');
- assert.ok(!host.files.has(rec.dir));
- assert.strictEqual(host.read(p.path), good, 'the live object is untouched');
- for (const changed of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(changed, rec.dir), `unexpected write: ${changed}`);
- }),
-
- test('objects: a service declaring an object directory refuses to drill from a backup without its archive', async () => {
- // The directory does not exist, so the backup records the object directory as skipped.
- const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [] });
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 1);
- assert.match(r.out, /has no copy of the objects object directory \(objects\.tar\.gz\)/);
- assert.strictEqual(host.systemdRuns.length, 0);
- }),
-
- test('objects: a symlink or special-file member is refused before root extracts the archive', async () => {
- for (const member of [{ type: 'l', name: './evil', target: '/' }, { type: 'p', name: './fifo' }]) {
- const a = 'tenant page a\n';
- const pa = blob(OBJECTS_DIR, a);
- const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });
- host.tarMembers = [member];
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, `${member.type}: ${r.out}`);
- const rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'objects');
- assert.match(rec.failure.message, /not a regular file or directory/);
- assert.ok(!host.calls.some((c) => c.cmd === 'tar' && c.args[0] === '-xzf'), `${member.type}: never extracted`);
- assert.ok(!host.files.has(rec.dir), 'cleaned up');
- }
- }),
-
- test('objects: a symlinked directory already in the restored tree is refused, not walked as root', async () => {
- const host = scenario();
- const dir = '/var/lib/openvibe-drills/tools-20260922-120100/objects';
- const good = 'a good blob\n';
- const sha = crypto.createHash('sha256').update(good).digest('hex');
- host.put(`${dir}/projects/prj_1/${sha.slice(0, 2)}/${sha}`, good, { owner: 'ubuntu' });
- // A file whose name IS a sha256 but whose bytes are not, behind a symlinked directory: if the
- // walk followed the link it would hash this and report a mismatch.
- const outsideSha = 'b'.repeat(64);
- host.put(`/var/lib/openvibe-secret/${outsideSha}`, 'not its name\n', { owner: 'root', mode: 0o600 });
- await host.exec.symlink('/var/lib/openvibe-secret', `${dir}/evil`);
- const entry = { blobs: 0, skipped: 0, bytes: 0, mismatches: 0 };
- await assert.rejects(
- verifyObjectBlobs(host.exec, dir, entry),
- (err) => /evil \(symlink or special file\)/.test(err.message) && !err.message.includes(outsideSha) && !err.message.includes('/var/lib/openvibe-secret'),
- );
- assert.strictEqual(entry.blobs, 1, 'the real blob was still checked');
- assert.strictEqual(entry.mismatches, 1);
- }),
-
- test('objects: a drill with no room to extract the archive fails before root restores it', async () => {
- const a = 'tenant page a\n';
- const pa = blob(OBJECTS_DIR, a);
- const host = await drillScenario({ objects: [{ name: 'objects', path: OBJECTS_DIR }], blobs: [{ path: pa.path, content: a }] });
- const callsBefore = host.calls.length;
- host.statfsFree = 1024;
- const r = await host.cli('drill', 'tools');
- assert.strictEqual(r.code, 2, r.out);
- const rec = lastLog(host);
- assert.strictEqual(rec.failure.stage, 'space');
- assert.match(rec.failure.message, /not enough free space under \/var\/lib\/openvibe-drills/);
- assert.ok(!host.calls.slice(callsBefore).some((c) => c.cmd === 'tar'), 'nothing was tarred or extracted');
- assert.ok(!host.files.has(rec.dir));
- }),
-
- test('inventory: object directories are validated', () => {
- const base = { owner: 'ubuntu', repo: '/opt/openvibe.host', units: [], databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }] };
- const svc = (objects) => normalise({ services: { h: { ...base, objects } } });
- assert.deepStrictEqual(svc([{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]).services.h.objects, [{ name: 'objects', path: '/var/lib/openvibe-host-api/objects' }]);
- assert.deepStrictEqual(svc(undefined).services.h.objects, []);
- assert.throws(() => svc([{ name: 'a/b', path: '/x' }]), /name must be a plain name/);
- assert.throws(() => svc([{ name: 'objects', path: 'relative' }]), /absolute path/);
- assert.throws(() => svc([{ name: 'objects', path: '/x' }, { name: 'objects', path: '/y' }]), /listed twice/);
- assert.throws(() => svc('nope'), /must be a list of directories/);
- }),
-
- test('several units and neither drill.unit nor drill.command: refused', async () => {
- const host = await appsScenario({ drill: { unit: null } });
- const r = await host.cli('drill', 'apps');
- assert.strictEqual(r.code, 1, r.out);
- assert.match(r.out, /set drill\.unit/);
- }),
-
- test('inventory: drill blocks are validated', () => {
- const base = { owner: 'ubuntu' };
- const svc = (drill, extra = {}) => normalise({ services: { tools: { ...base, ...toolsEntry(), ...extra, drill: { ...toolsEntry().drill, ...drill } } } });
- assert.throws(() => svc({ port: 4200 }), /must differ from the service's own port/);
- assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of tools's databases/);
- assert.throws(() => svc({ env: { HOST: '127.0.0.1' } }), /\{port\}/);
- assert.throws(() => svc({ env: { PORT: '{port}', TOOLS_DB_PATH: '/x' } }), /already set by drill.databases/);
- assert.throws(() => svc({ env: { PORT: '{port}\nEVIL=1' } }), /one-line/);
- assert.throws(() => svc({ dirs: ['/opt/openvibe.tools/data'] }), /under \{tmp\}/);
- assert.throws(() => svc({ counts: [{ db: 'tools', table: 'x; drop table y' }] }), /table name/);
- assert.throws(() => svc({ compare: [{ path: 'http://evil/' }] }), /path/);
- assert.throws(() => svc({ supported: false }), /reason is required/);
- assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, tools: { ...base, ...toolsEntry() } } }), /14200 is a's port/);
- assert.strictEqual(svc({}).services.tools.drill.ready, '/api/ready');
- }),
-
- test('host.example.json: every service with a database has a drill block; unsupported ones give a reason', () => {
- const raw = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
- const inv = normalise(raw);
- for (const s of Object.values(inv.services)) {
- if (!s.databases.length) continue;
- assert.ok(s.drill, `${s.id} has a drill block`);
- if (!s.drill.supported) { assert.ok(s.drill.reason.length > 20, `${s.id} explains why`); continue; }
- assert.ok(s.drill.compare.length || s.drill.counts.length);
- assert.ok(Object.keys(s.drill.env).length >= 2, `${s.id} overrides at least its port and one side effect`);
- }
- }),
-]);
@@ -1,15 +1,12 @@
@@ -1,15 +1,12 @@
'use strict';
'use strict';
/**
/**
- * The real executor, exercised only where that is safe on any machine: the SQLite worker against a
- * temp database, HTTP against a local server, files in a temp directory, and run() as the current
- * user. No systemctl/git/npm/nginx command is run.
+ * The real executor, exercised with HTTP, files in a temp directory, and run() as the current user. No systemctl/git/npm/nginx command is run.
*/
*/
const assert = require('assert');
const assert = require('assert');
const fs = require('fs');
const fs = require('fs');
const http = require('http');
const http = require('http');
const os = require('os');
const os = require('os');
const path = require('path');
const path = require('path');
-const Database = require('better-sqlite3');
const { createSystemExecutor } = require('../lib/executor');
const { createSystemExecutor } = require('../lib/executor');
const { test, runTests } = require('./helpers');
const { test, runTests } = require('./helpers');
@@ -18,72 +15,6 @@ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ovhost-exec-'));
@@ -18,72 +15,6 @@ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ovhost-exec-'));
process.on('exit', () => fs.rmSync(dir, { recursive: true, force: true }));
process.on('exit', () => fs.rmSync(dir, { recursive: true, force: true }));
runTests([
runTests([
- test('sqlite: read-only count as the current user; writes are refused', async () => {
- const dbPath = path.join(dir, 'media.db');
- const db = new Database(dbPath);
- db.pragma('journal_mode = WAL');
- db.exec('CREATE TABLE vods (id INTEGER PRIMARY KEY, is_recording INTEGER); INSERT INTO vods (is_recording) VALUES (1), (0), (1);');
- const me = await exec.userName();
- const rows = await exec.sqlite(dbPath, 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', { as: me });
- assert.deepStrictEqual(rows, [{ n: 2 }]);
- await assert.rejects(exec.sqlite(dbPath, 'DELETE FROM vods', { as: me }), /read-only|readonly/);
- assert.strictEqual(db.prepare('SELECT count(*) AS n FROM vods').get().n, 3);
- await assert.rejects(exec.sqlite(path.join(dir, 'missing.db'), 'SELECT 1'), /cannot open/);
- db.close();
- }),
-
- test('sqlite backup: consistent copy that passes quick_check; never overwrites', async () => {
- const src = path.join(dir, 'src.db');
- const db = new Database(src);
- db.exec('CREATE TABLE t (x); INSERT INTO t VALUES (1), (2);');
- const dest = path.join(dir, 'backup.db');
- await exec.sqliteBackup(src, dest);
- const copy = new Database(dest, { readonly: true });
- assert.strictEqual(copy.prepare('SELECT count(*) AS n FROM t').get().n, 2);
- copy.close();
- await assert.rejects(exec.sqliteBackup(src, dest), /refusing to overwrite/);
- db.close();
- }),
-
- test('sqlite backup of a WAL database: one 0600 file in rollback-journal mode, no -wal/-shm left', async () => {
- const src = path.join(dir, 'wal-src.db');
- const db = new Database(src);
- db.pragma('journal_mode = WAL');
- db.exec('CREATE TABLE t (x); INSERT INTO t VALUES (1), (2), (3);');
- const out = fs.mkdtempSync(path.join(dir, 'bk-'));
- const dest = path.join(out, 'wal.db');
- await exec.sqliteBackup(src, dest);
- db.close();
- assert.deepStrictEqual(fs.readdirSync(out), ['wal.db']);
- assert.strictEqual(fs.statSync(dest).mode & 0o777, 0o600);
- const copy = new Database(dest, { readonly: true });
- assert.strictEqual(copy.pragma('journal_mode', { simple: true }), 'delete');
- assert.strictEqual(copy.prepare('SELECT count(*) AS n FROM t').get().n, 3);
- copy.close();
- assert.deepStrictEqual(fs.readdirSync(out), ['wal.db'], 'reading the copy leaves no side files');
- }),
-
- test('drill primitives: integrity_check through the worker, kill() signals exactly one pid', async () => {
- const dbPath = path.join(dir, 'restored.db');
- const db = new Database(dbPath);
- db.pragma('journal_mode = WAL');
- db.exec('CREATE TABLE pastes (id INTEGER PRIMARY KEY); INSERT INTO pastes DEFAULT VALUES;');
- db.close();
- const me = await exec.userName();
- assert.deepStrictEqual(await exec.sqlite(dbPath, 'PRAGMA integrity_check', { as: me }), [{ integrity_check: 'ok' }]);
- assert.deepStrictEqual(await exec.sqlite(dbPath, 'SELECT count(*) AS n FROM "pastes"', { as: me }), [{ n: 1 }]);
-
- const { spawn } = require('child_process');
- const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' });
- const exited = new Promise((r) => child.on('exit', (code, signal) => r(signal)));
- assert.strictEqual(await exec.pidAlive(child.pid), true);
- assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), true);
- assert.strictEqual(await exited, 'SIGTERM');
- assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), false, 'a pid that is gone');
- await assert.rejects(exec.kill(1, 'SIGTERM'), /refusing/);
- await assert.rejects(exec.kill(0, 'SIGTERM'), /refusing/);
- }),
-
test('http, files, locks and run() behave as the fake host assumes', async () => {
test('http, files, locks and run() behave as the fake host assumes', async () => {
const server = http.createServer((req, res) => { res.writeHead(req.url === '/ok' ? 200 : 503); res.end(JSON.stringify({ host: req.headers.host })); });
const server = http.createServer((req, res) => { res.writeHead(req.url === '/ok' ? 200 : 503); res.end(JSON.stringify({ host: req.headers.host })); });
await new Promise((r) => server.listen(0, '127.0.0.1', r));
await new Promise((r) => server.listen(0, '127.0.0.1', r));
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
'use strict';
'use strict';
/**
/**
* A fake host implementing the executor interface (lib/executor.js) in memory: a filesystem, git
* A fake host implementing the executor interface (lib/executor.js) in memory: a filesystem, git
- * repositories, systemd units, npm, nginx, ss, HTTP endpoints, SQLite and PostgreSQL. Nothing here
+ * repositories, systemd units, npm, nginx, ss, HTTP endpoints and PostgreSQL. Nothing here
* touches the real machine. Every run() call is recorded in `calls` so tests can assert what would
* touches the real machine. Every run() call is recorded in `calls` so tests can assert what would
* have been executed (and as whom).
* have been executed (and as whom).
*
*
@@ -31,8 +31,6 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -31,8 +31,6 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
nginxTest: () => ({ code: 0, stderr: 'nginx: configuration file /etc/nginx/nginx.conf test is successful' }),
nginxTest: () => ({ code: 0, stderr: 'nginx: configuration file /etc/nginx/nginx.conf test is successful' }),
npm: null, // (cwd, argv, as) -> undefined | { code, stderr } ; default installs package.json deps
npm: null, // (cwd, argv, as) -> undefined | { code, stderr } ; default installs package.json deps
npmRewritesLockfile: true,
npmRewritesLockfile: true,
- sqliteHandler: () => [{ n: 0 }],
- sqliteCalls: [],
// ── PostgreSQL ──
// ── PostgreSQL ──
// The cluster's databases and roles, the archiver's failed WAL count, the stanza's backups
// The cluster's databases and roles, the archiver's failed WAL count, the stanza's backups
// and its status. A test drives them directly (host.pgDatabases.add('ov_trade')) or through
// and its status. A test drives them directly (host.pgDatabases.add('ov_trade')) or through
@@ -749,20 +747,6 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -749,20 +747,6 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
if (!get(p)) { const err = new Error(`ENOENT: no such file or directory, statfs '${p}'`); err.code = 'ENOENT'; throw err; }
if (!get(p)) { const err = new Error(`ENOENT: no such file or directory, statfs '${p}'`); err.code = 'ENOENT'; throw err; }
return { free: host.statfsFree, size: host.statfsSize };
return { free: host.statfsFree, size: host.statfsSize };
},
},
- async sqlite(db, sql, { as } = {}) {
- host.sqliteCalls.push({ op: 'query', db, sql, as });
- calls.push({ cmd: 'sqlite', args: [db, sql], as: as || null, privileged: false, cwd: null });
- return host.sqliteHandler(db, sql, as);
- },
- async sqliteBackup(db, dest, { as } = {}) {
- host.sqliteCalls.push({ op: 'backup', db, dest, as });
- calls.push({ cmd: 'sqlite-backup', args: [db, dest], as: as || null, privileged: false, cwd: null });
- if (files.has(path.resolve(dest))) throw new Error(`refusing to overwrite ${dest}`);
- const parent = files.get(path.dirname(path.resolve(dest)));
- if (as && parent && parent.owner !== as) throw new Error(`cannot open ${dest}: permission denied (directory owned by ${parent.owner}, worker runs as ${as})`);
- if (host.onSqliteBackup) { const r = host.onSqliteBackup(db, dest, as); if (r instanceof Error) throw r; }
- put(dest, host.backupContent ? host.backupContent(db) : `backup of ${db}`, { owner: as || user, mode: 0o600 });
- },
async listeners(port) { return listeners.get(port) || []; },
async listeners(port) { return listeners.get(port) || []; },
async kill(pid, signal = 'SIGTERM') {
async kill(pid, signal = 'SIGTERM') {
calls.push({ cmd: 'kill', args: [signal, pid], as: null, privileged: true, cwd: null });
calls.push({ cmd: 'kill', args: [signal, pid], as: null, privileged: true, cwd: null });
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
'use strict';
'use strict';
/**
/**
* A standard fake host with the shapes of the real estate: Live (socket-activated, /api/streams),
* A standard fake host with the shapes of the real estate: Live (socket-activated, /api/streams),
- * Media (recordings counted in SQLite), Tools (several packages and units), Sites (static build +
+ * Media (recordings counted in PostgreSQL), Tools (several packages and units), Sites (static build +
* nginx vhosts, no units) and Events (SSE, drain policy "report").
* nginx vhosts, no units) and Events (SSE, drain policy "report").
*/
*/
const path = require('path');
const path = require('path');
@@ -48,7 +48,7 @@ function inventoryDoc() {
@@ -48,7 +48,7 @@ function inventoryDoc() {
noRestartPaths: ['public/', 'docs/', '*.md'],
noRestartPaths: ['public/', 'docs/', '*.md'],
protected: { kind: 'http-json-count', url: 'http://127.0.0.1:3000/api/streams', array: 'streams', where: { is_live: true }, label: 'live streams' },
protected: { kind: 'http-json-count', url: 'http://127.0.0.1:3000/api/streams', array: 'streams', where: { is_live: true }, label: 'live streams' },
drain: { policy: 'refuse', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },
drain: { policy: 'refuse', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },
- databases: [{ name: 'live', path: '/opt/openvibe.live/data/live.db' }],
+ databases: [{ name: 'live', engine: 'postgresql', database: 'ov_live' }],
backupOnChange: ['server/db/schema.sql'],
backupOnChange: ['server/db/schema.sql'],
nginx: { vhost: 'openvibe.live.conf', variant: 'websocket', wsPaths: ['/ws/'] },
nginx: { vhost: 'openvibe.live.conf', variant: 'websocket', wsPaths: ['/ws/'] },
lifecycle: lifecycleDoc(),
lifecycle: lifecycleDoc(),
@@ -60,8 +60,8 @@ function inventoryDoc() {
@@ -60,8 +60,8 @@ function inventoryDoc() {
env: { required: 'from-example' },
env: { required: 'from-example' },
port: 4100,
port: 4100,
ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },
ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },
- protected: { kind: 'sqlite-count', db: '/opt/openvibe.media/data/media.db', sql: 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', label: 'recordings in progress' },
- databases: [{ name: 'media', path: '/opt/openvibe.media/data/media.db' }],
+ protected: { kind: 'postgresql-count', database: 'ov_media', sql: 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', label: 'recordings in progress' },
+ databases: [{ name: 'media', engine: 'postgresql', database: 'ov_media' }],
lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),
lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),
},
},
tools: {
tools: {
@@ -126,7 +126,7 @@ function scenario() {
@@ -126,7 +126,7 @@ function scenario() {
}
}
service('live', {
service('live', {
- 'package.json': pkgJson('openvibe-live', ['express', 'better-sqlite3', 'openvibe-shared']),
+ 'package.json': pkgJson('openvibe-live', ['express', 'openvibe-sdk', 'openvibe-shared']),
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'server/index.js': 'console.log(1);',
'server/index.js': 'console.log(1);',
'server/db/schema.sql': 'CREATE TABLE a(x);',
'server/db/schema.sql': 'CREATE TABLE a(x);',
@@ -137,16 +137,14 @@ function scenario() {
@@ -137,16 +137,14 @@ function scenario() {
host.addUnit('openvibe-live.socket', { sub: 'listening', partOf: ['openvibe-live.service'], mainPid: 0 });
host.addUnit('openvibe-live.socket', { sub: 'listening', partOf: ['openvibe-live.service'], mainPid: 0 });
host.put('/etc/systemd/system/openvibe-live.service', '[Service]\nExecStart=node server/index.js\nEnvironment=NODE_ENV=production\n');
host.put('/etc/systemd/system/openvibe-live.service', '[Service]\nExecStart=node server/index.js\nEnvironment=NODE_ENV=production\n');
host.put('/etc/openvibe/live.env', `JWT_SECRET=${SECRET}\nBASE_URL=https://openvibe.live\nPAYPAL_CLIENT_SECRET="${SECRET}-paypal"\nOV_OAUTH_CLIENT_SECRET=${SECRET}-oauth\n`, { mode: 0o600 });
host.put('/etc/openvibe/live.env', `JWT_SECRET=${SECRET}\nBASE_URL=https://openvibe.live\nPAYPAL_CLIENT_SECRET="${SECRET}-paypal"\nOV_OAUTH_CLIENT_SECRET=${SECRET}-oauth\n`, { mode: 0o600 });
- host.put('/opt/openvibe.live/data/live.db', 'sqlite', { owner: 'ubuntu' });
service('media', {
service('media', {
- 'package.json': pkgJson('openvibe-media', ['express', 'better-sqlite3']),
+ 'package.json': pkgJson('openvibe-media', ['express', 'openvibe-sdk']),
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'server/index.js': 'media();',
'server/index.js': 'media();',
'.env.example': 'PORT=4100\nMEDIA_B2_APP_KEY=\nMEDIA_R2_SECRET_ACCESS_KEY=\n',
'.env.example': 'PORT=4100\nMEDIA_B2_APP_KEY=\nMEDIA_R2_SECRET_ACCESS_KEY=\n',
}, ['.']);
}, ['.']);
host.put('/etc/openvibe/media.env', `PORT=4100\nMEDIA_B2_APP_KEY=${SECRET}-b2\nMEDIA_R2_SECRET_ACCESS_KEY=${SECRET}-r2\n`, { mode: 0o600 });
host.put('/etc/openvibe/media.env', `PORT=4100\nMEDIA_B2_APP_KEY=${SECRET}-b2\nMEDIA_R2_SECRET_ACCESS_KEY=${SECRET}-r2\n`, { mode: 0o600 });
- host.put('/opt/openvibe.media/data/media.db', 'sqlite', { owner: 'ubuntu' });
service('tools', {
service('tools', {
'apps/gateway/package.json': pkgJson('tools-gateway', ['express', 'openvibe-shared']),
'apps/gateway/package.json': pkgJson('tools-gateway', ['express', 'openvibe-shared']),
@@ -185,7 +183,10 @@ function scenario() {
@@ -185,7 +183,10 @@ function scenario() {
host.http.set('http://127.0.0.1:4100/healthz', readyFor('openvibe-media.service'));
host.http.set('http://127.0.0.1:4100/healthz', readyFor('openvibe-media.service'));
host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));
host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));
host.http.set('http://127.0.0.1:4300/api/ready', readyFor('openvibe-events.service'));
host.http.set('http://127.0.0.1:4300/api/ready', readyFor('openvibe-events.service'));
- host.sqliteHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : [{ n: 0 }]);
+ host.psqlHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : undefined);
+ host.pgDatabases.add('ov_live');
+ host.pgDatabases.add('ov_media');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
host.repo = (id) => repoFor[id];
host.repo = (id) => repoFor[id];
host.ctx = () => ({ exec: host.exec, inv, log: (s) => host.logs.push(s) });
host.ctx = () => ({ exec: host.exec, inv, log: (s) => host.logs.push(s) });
@@ -17,14 +17,13 @@ function withHost(host, rows) {
@@ -17,14 +17,13 @@ function withHost(host, rows) {
units: ['openvibe-host.service'],
units: ['openvibe-host.service'],
port: 4910,
port: 4910,
ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },
ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },
- databases: [{ name: 'host', path: '/var/lib/openvibe-host-api/host.db' }],
- nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: '/var/lib/openvibe-host-api/host.db', maxUpload: '110m' } },
+ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],
+ nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: { engine: 'postgresql', database: 'ov_host' }, maxUpload: '110m' } },
};
};
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
- host.sqliteHandler = (db, sql, as) => {
- assert.strictEqual(db, '/var/lib/openvibe-host-api/host.db');
+ host.psqlHandler = (db, sql) => {
+ assert.strictEqual(db, 'ov_host');
assert.match(sql, /^SELECT hostname FROM host_domains WHERE kind = 'custom' AND status = 'verified'/);
assert.match(sql, /^SELECT hostname FROM host_domains WHERE kind = 'custom' AND status = 'verified'/);
- assert.strictEqual(as, 'ubuntu', 'queried as the service user');
return rows;
return rows;
};
};
host.put('/etc/letsencrypt/live/www.tenant-a.org/fullchain.pem', 'CERT', { mode: 0o644 });
host.put('/etc/letsencrypt/live/www.tenant-a.org/fullchain.pem', 'CERT', { mode: 0o644 });
@@ -54,7 +53,7 @@ runTests([
@@ -54,7 +53,7 @@ runTests([
assert.strictEqual(host.read('/etc/nginx/sites-available/openvibe.host.conf'), null, 'render alone writes nothing');
assert.strictEqual(host.read('/etc/nginx/sites-available/openvibe.host.conf'), null, 'render alone writes nothing');
}),
}),
- test('a PostgreSQL tenant database reads the verified domains over psql, never a SQLite file', async () => {
+ test('a PostgreSQL tenant database reads the verified domains over psql, as PostgreSQL', async () => {
const host = scenario();
const host = scenario();
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
// Host moved to PostgreSQL (ADR-035): its tenants database is a database in the cluster, not
// Host moved to PostgreSQL (ADR-035): its tenants database is a database in the cluster, not
@@ -76,7 +75,6 @@ runTests([
@@ -76,7 +75,6 @@ runTests([
const domains = await nginx.tenantDomains(host.exec, normalise(doc).services.host);
const domains = await nginx.tenantDomains(host.exec, normalise(doc).services.host);
assert.deepStrictEqual(domains, [{ hostname: 'www.tenant-a.org', hasCert: true }, { hostname: 'docs.tenant-b.net', hasCert: false }], 'the same [{ hostname, hasCert }] shape');
assert.deepStrictEqual(domains, [{ hostname: 'www.tenant-a.org', hasCert: true }, { hostname: 'docs.tenant-b.net', hasCert: false }], 'the same [{ hostname, hasCert }] shape');
assert.ok(host.psqlCalls.some((c) => c.database === 'ov_host' && c.sql === nginx.VERIFIED_SQL), 'the verified domains come from psql');
assert.ok(host.psqlCalls.some((c) => c.database === 'ov_host' && c.sql === nginx.VERIFIED_SQL), 'the verified domains come from psql');
- assert.strictEqual(host.sqliteCalls.length, 0, 'a database in the cluster is never opened as a file');
const r = await host.cli('nginx', 'tenants', 'host');
const r = await host.cli('nginx', 'tenants', 'host');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
assert.match(r.out, /server_name www\.tenant-a\.org;/);
assert.match(r.out, /server_name www\.tenant-a\.org;/);
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
'use strict';
'use strict';
/**
/**
- * The two-engine inventory (ADR-035): a `databases[]` entry is SQLite (the default, a file) or
+ * The PostgreSQL inventory: a `databases[]` entry is
* PostgreSQL (a database in the cluster, named — its object carries NO `path`). The same engine
* PostgreSQL (a database in the cluster, named — its object carries NO `path`). The same engine
* choice reaches the drill block ({ url, directUrl }), nginx tenants (read over psql, not a file)
* choice reaches the drill block ({ url, directUrl }), nginx tenants (read over psql, not a file)
* and the protected `postgresql-count` probe. These are the normalised shapes the rest of ovhost
* and the protected `postgresql-count` probe. These are the normalised shapes the rest of ovhost
@@ -60,15 +60,9 @@ runTests([
@@ -60,15 +60,9 @@ runTests([
for (const database of ['trade', 'ov-Trade', 'public']) {
for (const database of ['trade', 'ov-Trade', 'public']) {
assert.throws(() => first(withDatabases([{ engine: 'postgresql', database }])), /database must match/, database);
assert.throws(() => first(withDatabases([{ engine: 'postgresql', database }])), /database must match/, database);
}
}
- // An engine that is neither.
- assert.throws(() => first(withDatabases([{ engine: 'mysql', database: 'ov_trade' }])), /engine must be sqlite or postgresql/);
- // A SQLite entry cannot name a database, and must have a path.
- assert.throws(() => first(withDatabases([{ path: '/x/live.db', database: 'ov_live' }])), /a sqlite database is a path, not a database name/);
- assert.throws(() => first(withDatabases([{ name: 'live' }])), /path must be an absolute path/);
- }),
-
- test('the SQLite default is unchanged', () => {
- assert.deepStrictEqual(first(withDatabases([{ name: 'live', path: '/x/live.db' }])), { name: 'live', engine: 'sqlite', path: '/x/live.db' });
+ for (const entry of [{ engine: 'mysql', database: 'ov_trade' }, { path: '/x/live.db', database: 'ov_live' }, { name: 'live' }]) {
+ assert.throws(() => first(withDatabases([entry])), /engine must be "postgresql"; use/);
+ }
}),
}),
test('drill.databases for a PostgreSQL database is { engine, url, directUrl }', () => {
test('drill.databases for a PostgreSQL database is { engine, url, directUrl }', () => {
@@ -76,7 +70,7 @@ runTests([
@@ -76,7 +70,7 @@ runTests([
assert.deepStrictEqual(d.databases.trade, { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' });
assert.deepStrictEqual(d.databases.trade, { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' });
}),
}),
- test('drill.databases rejects the SQLite shape and bad env var names for a PostgreSQL database', () => {
+ test('drill.databases rejects a file database shape and bad env var names for a PostgreSQL database', () => {
const drill = (databases) => pgDrill({ databases });
const drill = (databases) => pgDrill({ databases });
assert.throws(() => normalise(drill({ trade: 'DATABASE_URL' })), /for a postgresql database/);
assert.throws(() => normalise(drill({ trade: 'DATABASE_URL' })), /for a postgresql database/);
assert.throws(() => normalise(drill({ trade: { url: 'DATABASE_URL' } })), /directUrl must be an env var name/);
assert.throws(() => normalise(drill({ trade: { url: 'DATABASE_URL' } })), /directUrl must be an env var name/);
@@ -92,16 +86,15 @@ runTests([
@@ -92,16 +86,15 @@ runTests([
assert.throws(() => normalise(pgDrill({ countsTolerance: 1.5 })), /countsTolerance must be a whole number of rows/);
assert.throws(() => normalise(pgDrill({ countsTolerance: 1.5 })), /countsTolerance must be a whole number of rows/);
}),
}),
- test('nginx.tenants.database accepts a path (SQLite) or a PostgreSQL database object', () => {
+ test('nginx.tenants.database accepts a PostgreSQL database object', () => {
const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } }).services.host.nginx.tenants.database;
const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } }).services.host.nginx.tenants.database;
- assert.deepStrictEqual(tenants('/var/lib/openvibe-host-api/host.db'), { engine: 'sqlite', path: '/var/lib/openvibe-host-api/host.db' });
assert.deepStrictEqual(tenants({ engine: 'postgresql', database: 'ov_host' }), { engine: 'postgresql', database: 'ov_host' });
assert.deepStrictEqual(tenants({ engine: 'postgresql', database: 'ov_host' }), { engine: 'postgresql', database: 'ov_host' });
}),
}),
test('nginx.tenants.database rejects anything else', () => {
test('nginx.tenants.database rejects anything else', () => {
const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } });
const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } });
for (const database of [{ engine: 'mysql', database: 'ov_host' }, { engine: 'postgresql', database: 'host' }, { database: 'ov_host' }, 42, []]) {
for (const database of [{ engine: 'mysql', database: 'ov_host' }, { engine: 'postgresql', database: 'host' }, { database: 'ov_host' }, 42, []]) {
- assert.throws(() => tenants(database), /nginx\.tenants\.database must be a path or/, JSON.stringify(database));
+ assert.throws(() => tenants(database), /nginx\.tenants\.database must be/, JSON.stringify(database));
}
}
}),
}),
@@ -1,56 +1,39 @@
@@ -1,56 +1,39 @@
'use strict';
'use strict';
-// Protected probes that add up parts (kind "sum", WS-P task 2): Tools' running jobs sit in each
-// satellite's jobs.db, so the probe is the total of one sqlite-count per satellite. Unknown when any part
-// cannot answer (never read as zero); validation checks every part and refuses a sum inside a sum.
const assert = require('assert');
const assert = require('assert');
const { countProtected } = require('../lib/probes');
const { countProtected } = require('../lib/probes');
const { normalise } = require('../lib/inventory');
const { normalise } = require('../lib/inventory');
const { createFakeHost } = require('./fake-host');
const { createFakeHost } = require('./fake-host');
-const counts = { '/opt/t/apps/img/data/jobs.db': 2, '/opt/t/apps/audio/data/jobs.db': 0, '/opt/t/apps/docs/data/jobs.db': 1 };
+const counts = { ov_tools_img: 2, ov_tools_audio: 0, ov_tools_docs: 1 };
const exec = {
const exec = {
run: async () => ({ code: 0, stdout: 'LoadState=loaded\nActiveState=active\nSubState=running\n', stderr: '' }),
run: async () => ({ code: 0, stdout: 'LoadState=loaded\nActiveState=active\nSubState=running\n', stderr: '' }),
- sqlite: async (db) => { if (!(db in counts)) throw new Error(`no such database ${db}`); return [{ n: counts[db] }]; },
+ psql: async (database) => { if (!(database in counts)) throw new Error(`no such database ${database}`); return [{ n: counts[database] }]; },
http: async () => ({ status: 200, body: JSON.stringify({ connections: 4 }) }),
http: async () => ({ status: 200, body: JSON.stringify({ connections: 4 }) }),
};
};
-const part = (app) => ({ kind: 'sqlite-count', db: `/opt/t/apps/${app}/data/jobs.db`, sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" });
-const svc = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), part('audio'), part('docs')] } };
+const part = (app) => ({ kind: 'postgresql-count', database: `ov_tools_${app}`, sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" });
+const svc = { units: ['openvibe-tools.service'], protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), part('audio'), part('docs')] } };
(async () => {
(async () => {
assert.deepStrictEqual(await countProtected(exec, svc), { count: 3, label: 'running tool jobs' });
assert.deepStrictEqual(await countProtected(exec, svc), { count: 3, label: 'running tool jobs' });
const mixed = { ...svc, protected: { kind: 'sum', label: 'x', probes: [part('img'), { kind: 'http-json-count', url: 'http://127.0.0.1:4400/ready', field: 'connections' }] } };
const mixed = { ...svc, protected: { kind: 'sum', label: 'x', probes: [part('img'), { kind: 'http-json-count', url: 'http://127.0.0.1:4400/ready', field: 'connections' }] } };
- assert.strictEqual((await countProtected(exec, mixed)).count, 6, 'parts of different kinds add up');
+ assert.strictEqual((await countProtected(exec, mixed)).count, 6);
const broken = { ...svc, protected: { ...svc.protected, probes: [part('img'), part('video')] } };
const broken = { ...svc, protected: { ...svc.protected, probes: [part('img'), part('video')] } };
const r = await countProtected(exec, broken);
const r = await countProtected(exec, broken);
- assert.strictEqual(r.count, null, 'one part that cannot answer makes the total unknown, never a smaller number');
+ assert.strictEqual(r.count, null);
assert.match(r.unknown, /^part 2: no such database/);
assert.match(r.unknown, /^part 2: no such database/);
- // A postgresql-count part asks the cluster over psql, as the postgres OS user (peer auth); the
- // number it answers is the count, and a psql failure is unknown — never zero.
const host = createFakeHost();
const host = createFakeHost();
host.addUnit('openvibe-tools.service');
host.addUnit('openvibe-tools.service');
- host.sqliteHandler = () => [{ n: 2 }];
host.psqlHandler = (database, sql) => { assert.strictEqual(database, 'ov_tools'); assert.match(sql, /^SELECT/); return [{ n: 7 }]; };
host.psqlHandler = (database, sql) => { assert.strictEqual(database, 'ov_tools'); assert.match(sql, /^SELECT/); return [{ n: 7 }]; };
- const pgPart = { kind: 'postgresql-count', database: 'ov_tools', sql: 'SELECT count(*) AS n FROM tool_jobs', label: 'running tool jobs' };
- assert.deepStrictEqual(await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart }), { count: 7, label: 'running tool jobs' });
- assert.deepStrictEqual(host.psqlCalls.at(-1), { database: 'ov_tools', sql: pgPart.sql, as: 'postgres' });
- const pgSum = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), pgPart] } };
- assert.strictEqual((await countProtected(host.exec, pgSum)).count, 9, 'a postgresql-count part adds to the sum');
+ const pgPart = { kind: 'postgresql-count', database: 'ov_tools', sql: 'SELECT count(*) AS n FROM tool_jobs' };
+ assert.deepStrictEqual(await countProtected(host.exec, { units: ['openvibe-tools.service'], protected: pgPart }), { count: 7, label: 'protected sessions' });
host.psqlHandler = () => { throw new Error('psql: could not connect to server'); };
host.psqlHandler = () => { throw new Error('psql: could not connect to server'); };
- const down = await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart });
- assert.strictEqual(down.count, null, 'a probe that cannot ask is unknown');
- assert.notStrictEqual(down.count, 0, 'never read as zero');
- assert.match(down.unknown, /could not connect/);
- const downSum = await countProtected(host.exec, pgSum);
- assert.strictEqual(downSum.count, null);
- assert.match(downSum.unknown, /^part 2: psql: could not connect/);
+ assert.match((await countProtected(host.exec, { units: ['openvibe-tools.service'], protected: pgPart })).unknown, /could not connect/);
- // The inventory checks each part.
const base = { owner: 'ubuntu', repo: '/opt/t', units: ['openvibe-tools.service'] };
const base = { owner: 'ubuntu', repo: '/opt/t', units: ['openvibe-tools.service'] };
assert.doesNotThrow(() => normalise({ services: { tools: { ...base, protected: svc.protected } } }));
assert.doesNotThrow(() => normalise({ services: { tools: { ...base, protected: svc.protected } } }));
assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [] } } } }), /probes must be a non-empty array/);
assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [] } } } }), /probes must be a non-empty array/);
assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ ...part('img'), sql: 'DELETE FROM tool_jobs' }] } } } }), /probes\[0\]\.sql must be a SELECT/);
assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ ...part('img'), sql: 'DELETE FROM tool_jobs' }] } } } }), /probes\[0\]\.sql must be a SELECT/);
- assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ kind: 'sum', probes: [part('img')] }] } } } }), /probes\[0\]\.kind must be one of http-json-count, sqlite-count/);
- assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ kind: 'http-json-count', url: 'https://example.com/ready', field: 'n' }] } } } }), /loopback/);
+ assert.throws(() => normalise({ services: { tools: { ...base, protected: { kind: 'sum', probes: [{ kind: 'sum', probes: [part('img')] }] } } } }), /probes\[0\]\.kind must be one of http-json-count, postgresql-count/);
console.log('probes sum: all checks passed');
console.log('probes sum: all checks passed');
})().catch((err) => { console.error(err); process.exitCode = 1; });
})().catch((err) => { console.error(err); process.exitCode = 1; });
new file mode 100644
new file mode 100644
@@ -0,0 +1,16 @@
@@ -0,0 +1,16 @@
+'use strict';
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const { openDb } = require('../server/db');
+const { test, runTests } = require('./helpers');
+
+runTests([
+ test('production requires DATABASE_URL before creating a PGlite directory', async () => {
+ const dir = path.join(os.tmpdir(), `ovhost-no-fallback-${process.pid}`);
+ assert.strictEqual(fs.existsSync(dir), false);
+ await assert.rejects(openDb({ isProduction: true, db: { url: '', directUrl: '', pgliteDir: dir } }), /DATABASE_URL is not set: production serves from PostgreSQL/);
+ assert.strictEqual(fs.existsSync(dir), false);
+ }),
+]);
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
/**
/**
* Runs every test in test/ — the files named *.test.js — each in its own process, and fails if any
* Runs every test in test/ — the files named *.test.js — each in its own process, and fails if any
* of them fails. Every test drives ovhost against the in-memory fake host (test/fake-host.js):
* of them fails. Every test drives ovhost against the in-memory fake host (test/fake-host.js):
- * no systemctl, git, npm, nginx, curl or SQLite call reaches the real machine.
+ * no systemctl, git, npm, nginx or curl call reaches the real machine.
*
*
* npm test # everything
* npm test # everything
* npm test -- deploy certs # only files whose name contains one of the words
* npm test -- deploy certs # only files whose name contains one of the words
@@ -35,30 +35,6 @@ runTests([
@@ -35,30 +35,6 @@ runTests([
assert.ok(meta.lockfiles['.']);
assert.ok(meta.lockfiles['.']);
}),
}),
- test('backup: sqlite .backup of each declared database, run as the service user, into a dated directory', async () => {
- const host = scenario();
- const r = await host.cli('backup', 'media', '--json');
- assert.strictEqual(r.code, 0, r.out);
- const res = JSON.parse(r.out);
- assert.match(res.dir, /^\/var\/backups\/openvibe\/media\/20260922-120000$/);
- assert.strictEqual(res.files[0].dest, `${res.dir}/media.db`);
- const call = host.sqliteCalls.find((c) => c.op === 'backup');
- assert.strictEqual(call.as, 'ubuntu');
- assert.strictEqual(call.db, '/opt/openvibe.media/data/media.db');
- assert.strictEqual(call.dest, '/var/backups/openvibe.staging/media-20260922-120000/media.db', 'the worker writes into its staging directory');
- // Backups hold user data: root:root, directories 0700, files 0600.
- for (const d of ['/var/backups/openvibe', '/var/backups/openvibe/media', res.dir]) {
- assert.deepStrictEqual([host.files.get(d).owner, host.files.get(d).mode], ['root', 0o700], d);
- }
- assert.deepStrictEqual([host.files.get(res.files[0].dest).owner, host.files.get(res.files[0].dest).mode], ['root', 0o600]);
- assert.strictEqual(host.files.has('/var/backups/openvibe.staging/media-20260922-120000'), false, 'staging directory removed');
- const log = host.read('/var/lib/openvibe-host/backups/media.jsonl');
- assert.strictEqual(JSON.parse(log.trim()).files[0].dest, res.files[0].dest);
- // A second backup in the same second never overwrites the first.
- const again = await host.cli('backup', 'media');
- assert.notStrictEqual(again.code, 0);
- }),
-
test('backup refuses a service without declared databases', async () => {
test('backup refuses a service without declared databases', async () => {
const host = scenario();
const host = scenario();
const r = await host.cli('backup', 'tools');
const r = await host.cli('backup', 'tools');
@@ -45,11 +45,11 @@ async function boot(opts = {}) {
@@ -45,11 +45,11 @@ async function boot(opts = {}) {
};
};
const env = {
const env = {
NODE_ENV: 'test', PORT: '0', BASE_URL: `https://${DASHBOARD}`, TRUST_PROXY: '1',
NODE_ENV: 'test', PORT: '0', BASE_URL: `https://${DASHBOARD}`, TRUST_PROXY: '1',
- HOST_DB_PATH: path.join(dir, 'host.db'), HOST_STORAGE_DIR: path.join(dir, 'objects'),
HOST_SITES_DOMAIN: 'openvibe.host',
HOST_SITES_DOMAIN: 'openvibe.host',
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
OV_OAUTH_CLIENT_ID: 'host', OV_OAUTH_CLIENT_SECRET: SECRET, COOKIE_SECURE: 'false',
OV_OAUTH_CLIENT_ID: 'host', OV_OAUTH_CLIENT_SECRET: SECRET, COOKIE_SECURE: 'false',
HOST_FORM_SECRET: 'test-form-secret', HOST_WORKER: 'off', HOST_MIN_FREE_BYTES: '0',
HOST_FORM_SECRET: 'test-form-secret', HOST_WORKER: 'off', HOST_MIN_FREE_BYTES: '0',
+ HOST_STORAGE_DIR: path.join(dir, 'objects'),
HOST_PUBLIC_HOSTING: 'on', // the suites exercise the engine as launched; test/host-launch-switch.test.js covers off
HOST_PUBLIC_HOSTING: 'on', // the suites exercise the engine as launched; test/host-launch-switch.test.js covers off
...(opts.env || {}),
...(opts.env || {}),
};
};
@@ -28,6 +28,8 @@ function base(services) {
@@ -28,6 +28,8 @@ function base(services) {
function finish(host, doc) {
function finish(host, doc) {
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.inv = normalise(doc);
host.inv = normalise(doc);
+ for (const svc of Object.values(host.inv.services)) for (const db of svc.databases) host.pgDatabases.add(db.database);
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
host.ensureDir('/etc/nginx/sites-available');
host.ensureDir('/etc/nginx/sites-available');
host.ensureDir('/etc/nginx/sites-enabled');
host.ensureDir('/etc/nginx/sites-enabled');
host.badShas = new Set();
host.badShas = new Set();
@@ -69,7 +71,7 @@ function liveEntry(overrides = {}) {
@@ -69,7 +71,7 @@ function liveEntry(overrides = {}) {
noRestartPaths: ['public/', 'docs/', '**/*.md'],
noRestartPaths: ['public/', 'docs/', '**/*.md'],
protected: { kind: 'http-json-count', url: 'http://127.0.0.1:3000/api/streams', array: 'streams', where: { is_live: true }, label: 'live streams' },
protected: { kind: 'http-json-count', url: 'http://127.0.0.1:3000/api/streams', array: 'streams', where: { is_live: true }, label: 'live streams' },
drain: { policy: 'refuse', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },
drain: { policy: 'refuse', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },
- databases: [{ name: 'live', path: '/opt/openvibe.live/shared/data/live.db' }],
+ databases: [{ name: 'live', engine: 'postgresql', database: 'ov_live' }],
backupOnChange: ['server/db/schema.sql'],
backupOnChange: ['server/db/schema.sql'],
preflight: { syntaxCheck: true },
preflight: { syntaxCheck: true },
nginx: { vhost: 'openvibe.live.conf', repoVhost: 'deploy/nginx/openvibe.live.conf', variant: 'websocket' },
nginx: { vhost: 'openvibe.live.conf', repoVhost: 'deploy/nginx/openvibe.live.conf', variant: 'websocket' },
@@ -121,7 +123,7 @@ async function releaseLayoutHost({ id, entry, files, initialId, unitFiles = {},
@@ -121,7 +123,7 @@ async function releaseLayoutHost({ id, entry, files, initialId, unitFiles = {},
}
}
const LIVE_FILES = {
const LIVE_FILES = {
- 'package.json': pkgJson('openvibe-live', ['express', 'better-sqlite3']),
+ 'package.json': pkgJson('openvibe-live', ['express', 'openvibe-sdk']),
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'server/index.js': 'console.log(1);',
'server/index.js': 'console.log(1);',
'server/db/schema.sql': 'CREATE TABLE a(x);',
'server/db/schema.sql': 'CREATE TABLE a(x);',
@@ -138,7 +140,6 @@ async function liveHost(overrides = {}) {
@@ -138,7 +140,6 @@ async function liveHost(overrides = {}) {
host.addUnit('openvibe-live.socket', { sub: 'listening', mainPid: 0 });
host.addUnit('openvibe-live.socket', { sub: 'listening', mainPid: 0 });
host.listeners.set(3000, [{ pid: 1, process: 'systemd' }]);
host.listeners.set(3000, [{ pid: 1, process: 'systemd' }]);
host.put('/etc/openvibe/live.env', `JWT_SECRET=${SECRET}\n`, { mode: 0o600 });
host.put('/etc/openvibe/live.env', `JWT_SECRET=${SECRET}\n`, { mode: 0o600 });
- host.put('/opt/openvibe.live/shared/data/live.db', 'sqlite', { owner: 'ubuntu' });
host.liveStreams = [];
host.liveStreams = [];
host.http.set('http://127.0.0.1:3000/api/ready', host.readyFor('openvibe-live.service'));
host.http.set('http://127.0.0.1:3000/api/ready', host.readyFor('openvibe-live.service'));
host.http.set('http://127.0.0.1:3000/api/streams', () => ({ status: 200, body: { streams: host.liveStreams } }));
host.http.set('http://127.0.0.1:3000/api/streams', () => ({ status: 200, body: { streams: host.liveStreams } }));
@@ -166,8 +167,8 @@ function openreEntry(overrides = {}) {
@@ -166,8 +167,8 @@ function openreEntry(overrides = {}) {
port: 4500,
port: 4500,
ready: { url: 'http://127.0.0.1:4500/api/ready', timeoutSeconds: 60 },
ready: { url: 'http://127.0.0.1:4500/api/ready', timeoutSeconds: 60 },
noRestartPaths: ['docs/', 'test/', '**/*.md'],
noRestartPaths: ['docs/', 'test/', '**/*.md'],
- databases: [{ name: 'openre', path: '/var/lib/openre/openre.db' }],
- protected: { kind: 'sqlite-count', db: '/var/lib/openre/openre.db', sql: "SELECT count(*) AS n FROM ingest_sessions WHERE state IN ('starting', 'live', 'ending')", label: 'ingest sessions' },
+ databases: [{ name: 'openre', engine: 'postgresql', database: 'ov_openre' }],
+ protected: { kind: 'postgresql-count', database: 'ov_openre', sql: "SELECT count(*) AS n FROM ingest_sessions WHERE state IN ('starting', 'live', 'ending')", label: 'ingest sessions' },
drain: { policy: 'refuse' },
drain: { policy: 'refuse' },
lifecycle: lifecycleDoc(),
lifecycle: lifecycleDoc(),
...overrides,
...overrides,
@@ -175,7 +176,7 @@ function openreEntry(overrides = {}) {
@@ -175,7 +176,7 @@ function openreEntry(overrides = {}) {
}
}
const OPENRE_FILES = {
const OPENRE_FILES = {
- 'package.json': pkgJson('openre-stream', ['express', 'better-sqlite3']),
+ 'package.json': pkgJson('openre-stream', ['express', 'openvibe-sdk']),
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'package-lock.json': '{"lockfileVersion":3,"v":1}',
'server/index.js': 'api();',
'server/index.js': 'api();',
'docs/README.md': '# OpenRe',
'docs/README.md': '# OpenRe',
@@ -184,12 +185,11 @@ const OPENRE_FILES = {
@@ -184,12 +185,11 @@ const OPENRE_FILES = {
async function openreHost(overrides = {}) {
async function openreHost(overrides = {}) {
const host = await releaseLayoutHost({ id: 'openre', entry: openreEntry(overrides), files: OPENRE_FILES, initialId: (sha) => sha.slice(0, 12) });
const host = await releaseLayoutHost({ id: 'openre', entry: openreEntry(overrides), files: OPENRE_FILES, initialId: (sha) => sha.slice(0, 12) });
host.put('/etc/openvibe/openre.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\n`, { mode: 0o600 });
host.put('/etc/openvibe/openre.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\n`, { mode: 0o600 });
- host.put('/var/lib/openre/openre.db', 'sqlite', { owner: 'ubuntu' });
host.addUnit(`openre-rtmp-ingest@${host.firstRelease}.service`, { mainPid: 3102 });
host.addUnit(`openre-rtmp-ingest@${host.firstRelease}.service`, { mainPid: 3102 });
host.addUnit(`openre-restream-worker@${host.firstRelease}.service`, { mainPid: 3103 });
host.addUnit(`openre-restream-worker@${host.firstRelease}.service`, { mainPid: 3103 });
host.addUnit(`openre-jsmpeg@${host.firstRelease}.service`, { mainPid: 3105 });
host.addUnit(`openre-jsmpeg@${host.firstRelease}.service`, { mainPid: 3105 });
host.sessions = 0;
host.sessions = 0;
- host.sqliteHandler = () => [{ n: host.sessions }];
+ host.psqlHandler = () => [{ n: host.sessions }];
host.http.set('http://127.0.0.1:4500/api/ready', host.readyFor('openre-api.service'));
host.http.set('http://127.0.0.1:4500/api/ready', host.readyFor('openre-api.service'));
host.calls.length = 0;
host.calls.length = 0;
return host;
return host;
@@ -233,11 +233,11 @@ function toolsEntry(overrides = {}) {
@@ -233,11 +233,11 @@ function toolsEntry(overrides = {}) {
preflight: {
preflight: {
dirs: ['data'],
dirs: ['data'],
checks: [
checks: [
- { label: 'jobs runtime loads', packages: ['apps/img'], argv: ['node', '-e', "const D=require('better-sqlite3'); new D(':memory:').close(); require('openvibe-contracts'); require('openvibe-sdk'); require('../_shared/jobs')"] },
- { label: 'guard loads', packages: '*', argv: ['node', '-e', "const D=require('better-sqlite3'); new D(':memory:').close(); require('../_shared/guard')"] },
+ { label: 'jobs runtime loads', packages: ['apps/img'], argv: ['node', '-e', "require('openvibe-sdk/db'); require('openvibe-contracts'); require('openvibe-sdk'); require('../_shared/jobs')"] },
+ { label: 'guard loads', packages: '*', argv: ['node', '-e', "require('openvibe-sdk/db'); require('../_shared/guard')"] },
],
],
},
},
- protected: { kind: 'sum', label: 'running tool jobs', probes: [{ kind: 'sqlite-count', db: '/opt/openvibe.tools/apps/img/data/jobs.db', sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" }] },
+ protected: { kind: 'sum', label: 'running tool jobs', probes: [{ kind: 'postgresql-count', database: 'ov_tools', sql: "SELECT count(*) AS n FROM tool_jobs WHERE state = 'running'" }] },
drain: { policy: 'report', waitMaxSeconds: 900, pollSeconds: 60, quietChecks: 2 },
drain: { policy: 'report', waitMaxSeconds: 900, pollSeconds: 60, quietChecks: 2 },
lifecycle: lifecycleDoc(),
lifecycle: lifecycleDoc(),
...overrides,
...overrides,
@@ -245,13 +245,13 @@ function toolsEntry(overrides = {}) {
@@ -245,13 +245,13 @@ function toolsEntry(overrides = {}) {
}
}
const TOOLS_FILES = {
const TOOLS_FILES = {
- 'apps/gateway/package.json': pkgJson('tools-gateway', ['express', 'better-sqlite3']),
+ 'apps/gateway/package.json': pkgJson('tools-gateway', ['express', 'openvibe-sdk']),
'apps/gateway/package-lock.json': '{"v":1}',
'apps/gateway/package-lock.json': '{"v":1}',
'apps/gateway/server/index.js': 'gw();',
'apps/gateway/server/index.js': 'gw();',
- 'apps/img/package.json': pkgJson('tools-img', ['express', 'better-sqlite3', 'openvibe-contracts', 'openvibe-sdk']),
+ 'apps/img/package.json': pkgJson('tools-img', ['express', 'openvibe-sdk', 'openvibe-contracts', 'openvibe-sdk']),
'apps/img/package-lock.json': '{"v":1}',
'apps/img/package-lock.json': '{"v":1}',
'apps/img/server/index.js': 'img();',
'apps/img/server/index.js': 'img();',
- 'apps/maps/package.json': pkgJson('tools-maps', ['express', 'better-sqlite3']),
+ 'apps/maps/package.json': pkgJson('tools-maps', ['express', 'openvibe-sdk']),
'apps/maps/server/index.js': 'maps();',
'apps/maps/server/index.js': 'maps();',
'apps/_shared/package.json': JSON.stringify({ name: 'openvibe-tools-shared', version: '1.1.0', private: true }),
'apps/_shared/package.json': JSON.stringify({ name: 'openvibe-tools-shared', version: '1.1.0', private: true }),
'apps/_shared/guard/index.js': 'guard();',
'apps/_shared/guard/index.js': 'guard();',
@@ -270,7 +270,7 @@ function toolsHost(overrides = {}) {
@@ -270,7 +270,7 @@ function toolsHost(overrides = {}) {
return { status: 200, body: { service: 'tools', release: host.gatewayRelease || String(u.runningSha).slice(0, 12) } };
return { status: 200, body: { service: 'tools', release: host.gatewayRelease || String(u.runningSha).slice(0, 12) } };
});
});
host.jobs = 0;
host.jobs = 0;
- host.sqliteHandler = () => [{ n: host.jobs }];
+ host.psqlHandler = () => [{ n: host.jobs }];
return host;
return host;
}
}
@@ -318,10 +318,10 @@ function gamesEntry(overrides = {}) {
@@ -318,10 +318,10 @@ function gamesEntry(overrides = {}) {
port: 8000,
port: 8000,
ready: { url: 'http://127.0.0.1:8000/api/ready', timeoutSeconds: 60 },
ready: { url: 'http://127.0.0.1:8000/api/ready', timeoutSeconds: 60 },
generated: ['apps/client/dist-types/', 'apps/server/dist-types/'],
generated: ['apps/client/dist-types/', 'apps/server/dist-types/'],
- preflight: { checks: [{ label: 'better-sqlite3 loads under this Node', packages: ['apps/server'], argv: ['node', '-e', "new (require('better-sqlite3'))(':memory:').close()"] }] },
+ preflight: { checks: [{ label: 'database SDK loads under this Node', packages: ['apps/server'], argv: ['node', '-e', "require('openvibe-sdk/db')"] }] },
protected: { kind: 'http-json-count', url: 'http://127.0.0.1:8000/api/ready', field: 'checks.sessions.detail.online', label: 'players online' },
protected: { kind: 'http-json-count', url: 'http://127.0.0.1:8000/api/ready', field: 'checks.sessions.detail.online', label: 'players online' },
drain: { policy: 'report', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },
drain: { policy: 'report', waitMaxSeconds: 600, pollSeconds: 60, quietChecks: 2 },
- databases: [{ name: 'world', path: '/opt/openvibe.games/data/world.db' }],
+ databases: [{ name: 'world', engine: 'postgresql', database: 'ov_world' }],
lifecycle: lifecycleDoc(),
lifecycle: lifecycleDoc(),
...overrides,
...overrides,
};
};
@@ -331,7 +331,7 @@ const GAMES_FILES = {
@@ -331,7 +331,7 @@ const GAMES_FILES = {
'package.json': JSON.stringify({ name: 'openvibe-games', private: true, scripts: { build: 'pnpm -r build' } }),
'package.json': JSON.stringify({ name: 'openvibe-games', private: true, scripts: { build: 'pnpm -r build' } }),
'pnpm-lock.yaml': 'lockfileVersion: 9.0\nv: 1\n',
'pnpm-lock.yaml': 'lockfileVersion: 9.0\nv: 1\n',
'pnpm-workspace.yaml': 'packages:\n - packages/*\n - apps/*\n',
'pnpm-workspace.yaml': 'packages:\n - packages/*\n - apps/*\n',
- 'apps/server/package.json': pkgJson('@openvibe/server', ['@openvibe/shared', 'better-sqlite3', 'tsx']),
+ 'apps/server/package.json': pkgJson('@openvibe/server', ['@openvibe/shared', 'openvibe-sdk', 'tsx']),
'apps/server/src/main.ts': 'main();',
'apps/server/src/main.ts': 'main();',
'apps/server/dist-types/main.d.ts': 'export {};\n',
'apps/server/dist-types/main.d.ts': 'export {};\n',
'apps/client/package.json': pkgJson('@openvibe/client', ['@openvibe/shared']),
'apps/client/package.json': pkgJson('@openvibe/client', ['@openvibe/shared']),
@@ -357,7 +357,6 @@ function gamesHost(overrides = {}) {
@@ -357,7 +357,6 @@ function gamesHost(overrides = {}) {
host.put(path.join(cwd, 'apps/client/dist-types/main.d.ts'), `export {}; // built ${host.repo.head.slice(0, 7)}\n`, { owner: as });
host.put(path.join(cwd, 'apps/client/dist-types/main.d.ts'), `export {}; // built ${host.repo.head.slice(0, 7)}\n`, { owner: as });
return { code: 0 };
return { code: 0 };
};
};
- host.put('/opt/openvibe.games/data/world.db', 'sqlite', { owner: 'ubuntu' });
return host;
return host;
}
}
@@ -43,7 +43,7 @@ runTests([
@@ -43,7 +43,7 @@ runTests([
host.addUnit('openvibe-tools-docs.service', { runningSha: host.repo.head });
host.addUnit('openvibe-tools-docs.service', { runningSha: host.repo.head });
// A timer's oneshot job the glob also matches (production's openvibe-toolsjob.service): never restarted or waited on.
// A timer's oneshot job the glob also matches (production's openvibe-toolsjob.service): never restarted or waited on.
host.addUnit('openvibe-toolsjob.service', { type: 'oneshot', active: 'inactive', sub: 'dead' });
host.addUnit('openvibe-toolsjob.service', { type: 'oneshot', active: 'inactive', sub: 'dead' });
- const to = host.push({ 'apps/img/package.json': JSON.stringify({ name: 'tools-img', version: '1.0.1', dependencies: { express: '^1', 'better-sqlite3': '^1', 'openvibe-contracts': '^1', 'openvibe-sdk': '^1', sharp: '^1' } }), 'apps/_shared/guard/index.js': 'guard2();' }, 'img: sharp');
+ const to = host.push({ 'apps/img/package.json': JSON.stringify({ name: 'tools-img', version: '1.0.1', dependencies: { express: '^1', 'openvibe-sdk': '^1', 'openvibe-contracts': '^1', 'openvibe-sdk': '^1', sharp: '^1' } }), 'apps/_shared/guard/index.js': 'guard2();' }, 'img: sharp');
const r = await host.cli('deploy', 'tools');
const r = await host.cli('deploy', 'tools');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(host.repo.head, to);
assert.strictEqual(host.repo.head, to);
@@ -146,10 +146,10 @@ runTests([
@@ -146,10 +146,10 @@ runTests([
const host = toolsHost();
const host = toolsHost();
const from = host.repo.head;
const from = host.repo.head;
host.push({ 'apps/img/server/index.js': 'img2();' }, 'img');
host.push({ 'apps/img/server/index.js': 'img2();' }, 'img');
- host.onNode = (args, opts) => (/jobs/.test(args[1]) && opts.cwd.endsWith('apps/img') ? { code: 1, stderr: "Error: The module 'better_sqlite3.node' was compiled against a different Node.js version" } : undefined);
+ host.onNode = (args, opts) => (/jobs/.test(args[1]) && opts.cwd.endsWith('apps/img') ? { code: 1, stderr: "Error: The module 'sharp.node' was compiled against a different Node.js version" } : undefined);
const r = await host.cli('deploy', 'tools');
const r = await host.cli('deploy', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
- assert.match(r.out, /preflight failed: jobs runtime loads failed in apps\/img: Error: The module 'better_sqlite3\.node'/);
+ assert.match(r.out, /preflight failed: jobs runtime loads failed in apps\/img: Error: The module 'sharp\.node'/);
assert.match(r.out, /checkout restored/);
assert.match(r.out, /checkout restored/);
assert.strictEqual(host.repo.head, from);
assert.strictEqual(host.repo.head, from);
assert.deepStrictEqual(host.restarts(), []);
assert.deepStrictEqual(host.restarts(), []);
@@ -188,7 +188,7 @@ runTests([
@@ -188,7 +188,7 @@ runTests([
const host2 = toolsHost();
const host2 = toolsHost();
host2.push({ 'apps/img/server/index.js': 'img2();' }, 'img');
host2.push({ 'apps/img/server/index.js': 'img2();' }, 'img');
let polls = 0;
let polls = 0;
- host2.sqliteHandler = () => { polls += 1; return [{ n: polls < 3 ? 1 : 0 }]; };
+ host2.psqlHandler = () => { polls += 1; return [{ n: polls < 3 ? 1 : 0 }]; };
const w = await host2.cli('deploy', 'tools', '--wait-idle');
const w = await host2.cli('deploy', 'tools', '--wait-idle');
assert.strictEqual(w.code, 0, w.out);
assert.strictEqual(w.code, 0, w.out);
assert.match(w.out, /--wait-idle: holding the restart/);
assert.match(w.out, /--wait-idle: holding the restart/);
@@ -453,7 +453,7 @@ runTests([
@@ -453,7 +453,7 @@ runTests([
host.onNode = () => ({ code: 1, stderr: 'Error: Could not locate the bindings file' });
host.onNode = () => ({ code: 1, stderr: 'Error: Could not locate the bindings file' });
const r = await host.cli('deploy', 'games');
const r = await host.cli('deploy', 'games');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
- assert.match(r.out, /better-sqlite3 loads under this Node failed in apps\/server/);
+ assert.match(r.out, /database SDK loads under this Node failed in apps\/server/);
assert.strictEqual(host.repo.head, from);
assert.strictEqual(host.repo.head, from);
assert.deepStrictEqual(host.restarts(), []);
assert.deepStrictEqual(host.restarts(), []);
assert.strictEqual(host.builds, 2, 'built for the new sha, rebuilt for the restored one');
assert.strictEqual(host.builds, 2, 'built for the new sha, rebuilt for the restored one');
@@ -536,7 +536,7 @@ runTests([
@@ -536,7 +536,7 @@ runTests([
assert.match(c.out, /^strategy=pnpm-build$/m);
assert.match(c.out, /^strategy=pnpm-build$/m);
assert.match(c.out, /^managed=yes$/m);
assert.match(c.out, /^managed=yes$/m);
const unknown = await host.cli('capabilities', 'nope');
const unknown = await host.cli('capabilities', 'nope');
- assert.strictEqual(unknown.code, 1, 'an unknown service: the wrapper falls back');
+ assert.strictEqual(unknown.code, 1, 'an unknown service is an error');
const json = JSON.parse((await host.cli('capabilities', 'games', '--json')).out);
const json = JSON.parse((await host.cli('capabilities', 'games', '--json')).out);
assert.strictEqual(json.strategy, 'pnpm-build');
assert.strictEqual(json.strategy, 'pnpm-build');
const bare = await host.cli('capabilities', '--inventory', '/nope.json');
const bare = await host.cli('capabilities', '--inventory', '/nope.json');
@@ -30,7 +30,6 @@ runTests([
@@ -30,7 +30,6 @@ runTests([
assert.ok(host.calls.some((c) => c.cmd === 'cp' && c.args[0] === '-al' && c.args[1].endsWith(`${first}/node_modules`)), 'cp -al of node_modules');
assert.ok(host.calls.some((c) => c.cmd === 'cp' && c.args[0] === '-al' && c.args[1].endsWith(`${first}/node_modules`)), 'cp -al of node_modules');
assert.ok(!host.calls.some((c) => c.cmd === 'npm'), 'no install without a dependency change');
assert.ok(!host.calls.some((c) => c.cmd === 'npm'), 'no install without a dependency change');
assert.strictEqual(await host.exec.readlink(`/opt/openvibe.live/releases/${now}/data`), '../../shared/data');
assert.strictEqual(await host.exec.readlink(`/opt/openvibe.live/releases/${now}/data`), '../../shared/data');
- assert.strictEqual(host.read(`/opt/openvibe.live/current/data/live.db`), 'sqlite', 'shared data reached through the new release');
// Live's syntax check: every changed .js file, in the new release, before anything restarts.
// Live's syntax check: every changed .js file, in the new release, before anything restarts.
const check = host.nodeChecks().find((c) => c.args[0] === '--check');
const check = host.nodeChecks().find((c) => c.args[0] === '--check');
assert.ok(check && check.args[1] === `/opt/openvibe.live/releases/${now}/server/index.js`);
assert.ok(check && check.args[1] === `/opt/openvibe.live/releases/${now}/server/index.js`);
@@ -265,11 +264,11 @@ runTests([
@@ -265,11 +264,11 @@ runTests([
host.push({ 'server/db/schema.sql': 'CREATE TABLE b(y);' }, 'schema');
host.push({ 'server/db/schema.sql': 'CREATE TABLE b(y);' }, 'schema');
const r = await host.cli('deploy', 'live');
const r = await host.cli('deploy', 'live');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
- const b = host.calls.findIndex((c) => c.cmd === 'sqlite-backup');
+ const b = host.calls.findIndex((c) => c.cmd === 'pg_dump');
const mv = host.calls.findIndex((c) => c.cmd === 'mv' && c.args.includes('/opt/openvibe.live/current'));
const mv = host.calls.findIndex((c) => c.cmd === 'mv' && c.args.includes('/opt/openvibe.live/current'));
const restart = host.calls.findIndex((c) => c.cmd === 'systemctl' && c.args[0] === 'restart');
const restart = host.calls.findIndex((c) => c.cmd === 'systemctl' && c.args[0] === 'restart');
assert.ok(b >= 0 && b < mv && mv < restart, 'backup, then switch, then restart');
assert.ok(b >= 0 && b < mv && mv < restart, 'backup, then switch, then restart');
- assert.strictEqual(host.sqliteCalls.find((c) => c.op === 'backup').as, 'ubuntu');
+ assert.strictEqual(host.dumps.find((c) => c.database === 'ov_live').as, 'postgres');
}),
}),
test('live: plan shows the new release, node_modules reuse, the socket and the restart; changes nothing', async () => {
test('live: plan shows the new release, node_modules reuse, the socket and the restart; changes nothing', async () => {
@@ -314,8 +313,7 @@ runTests([
@@ -314,8 +313,7 @@ runTests([
assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'release-layout', release: { links: { data: '../../../../etc' } } } } }), /must point inside/);
assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'release-layout', release: { links: { data: '../../../../etc' } } } } }), /must point inside/);
assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'tarball' } } }), /strategy must be one of/);
assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'tarball' } } }), /strategy must be one of/);
assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'git-checkout', layout: 'release' } } }), /needs strategy "release-layout"/);
assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', strategy: 'git-checkout', layout: 'release' } } }), /needs strategy "release-layout"/);
- const legacy = normalise({ services: { x: { repo: '/opt/x', owner: 'root', layout: 'release' } } });
- assert.strictEqual(legacy.services.x.managed, false, 'a release layout without the strategy stays with its own script');
+ assert.throws(() => normalise({ services: { x: { repo: '/opt/x', owner: 'root', layout: 'release' } } }), /needs strategy "release-layout"/);
}),
}),
// ── OpenRestream ──
// ── OpenRestream ──
@@ -335,8 +333,10 @@ runTests([
@@ -335,8 +333,10 @@ runTests([
assert.ok(!host.calls.some((c) => c.cmd === 'systemctl' && /openre-(rtmp-ingest|restream-worker|jsmpeg)@/.test(String(c.args[1])) && c.args[0] !== 'show'), 'no worker unit is started, stopped or restarted');
assert.ok(!host.calls.some((c) => c.cmd === 'systemctl' && /openre-(rtmp-ingest|restream-worker|jsmpeg)@/.test(String(c.args[1])) && c.args[0] !== 'show'), 'no worker unit is started, stopped or restarted');
assert.ok((await host.releaseIds()).includes(first), 'the release the workers run from is kept');
assert.ok((await host.releaseIds()).includes(first), 'the release the workers run from is kept');
// A second deploy prunes to keep (2), but never the release a worker instance runs from.
// A second deploy prunes to keep (2), but never the release a worker instance runs from.
+ host.advance(1000);
host.push({ 'server/index.js': 'api3();' }, 'api3');
host.push({ 'server/index.js': 'api3();' }, 'api3');
assert.strictEqual((await host.cli('deploy', 'openre')).code, 0);
assert.strictEqual((await host.cli('deploy', 'openre')).code, 0);
+ host.advance(1000);
host.push({ 'server/index.js': 'api4();' }, 'api4');
host.push({ 'server/index.js': 'api4();' }, 'api4');
assert.strictEqual((await host.cli('deploy', 'openre')).code, 0);
assert.strictEqual((await host.cli('deploy', 'openre')).code, 0);
const ids = await host.releaseIds();
const ids = await host.releaseIds();
@@ -9,11 +9,12 @@ function noSecrets(text) {
@@ -9,11 +9,12 @@ function noSecrets(text) {
}
}
runTests([
runTests([
- test('a release-layout service is checked by its current release, not as a git checkout, and is never managed', async () => {
+ test('a release-layout service is checked by its current release and strategy clone', async () => {
const host = scenario();
const host = scenario();
const raw = JSON.parse(host.read('/etc/openvibe/host.json'));
const raw = JSON.parse(host.read('/etc/openvibe/host.json'));
- raw.services.rel = { repo: '/opt/rel.stream', layout: 'release', units: [], lifecycle: lifecycleDoc() };
+ raw.services.rel = { repo: '/opt/rel.stream', strategy: 'release-layout', units: [], lifecycle: lifecycleDoc() };
host.put('/etc/openvibe/host.json', JSON.stringify(raw), { mode: 0o640 });
host.put('/etc/openvibe/host.json', JSON.stringify(raw), { mode: 0o640 });
+ host.createRepo('/opt/rel.stream/repo');
host.put('/opt/rel.stream/releases/655b98a10aaa/package.json', '{}');
host.put('/opt/rel.stream/releases/655b98a10aaa/package.json', '{}');
await host.exec.symlink('/opt/rel.stream/releases/655b98a10aaa', '/opt/rel.stream/current');
await host.exec.symlink('/opt/rel.stream/releases/655b98a10aaa', '/opt/rel.stream/current');
let res = JSON.parse((await host.cli('validate', 'rel', '--json')).out);
let res = JSON.parse((await host.cli('validate', 'rel', '--json')).out);
@@ -21,7 +22,7 @@ runTests([
@@ -21,7 +22,7 @@ runTests([
assert.ok(res.findings.some((f) => f.area === 'checkout' && /current -> \/opt\/rel\.stream\/releases\/655b98a10aaa/.test(f.message)));
assert.ok(res.findings.some((f) => f.area === 'checkout' && /current -> \/opt\/rel\.stream\/releases\/655b98a10aaa/.test(f.message)));
const st = await host.cli('status', '--json');
const st = await host.cli('status', '--json');
assert.match(st.out, /655b98a10aaa/);
assert.match(st.out, /655b98a10aaa/);
- assert.match(st.out, /"managed":\s*false/);
+ assert.match(st.out, /"managed":\s*true/);
// current must name a release under releases/, owned by root or the checkout owner.
// current must name a release under releases/, owned by root or the checkout owner.
host.put('/opt/elsewhere/package.json', '{}');
host.put('/opt/elsewhere/package.json', '{}');
host.files.delete('/opt/rel.stream/current');
host.files.delete('/opt/rel.stream/current');