openvibers/openvibe.host Commits
ovhost backs up and drills PostgreSQL services natively (every service but network, chat, tools, games and live is on PostgreSQL since 2026-09-28; the old inventory's SQLite paths no longer exist, so backups of those services had silently skipped). databases[] is a union: SQLite { name, path } or PostgreSQL { name, engine: postgresql, database: ov_* }, reached as the postgres OS user over peer auth (lib/dbengine.js; no service env value is read). Backup verifies pgBackRest every run (newest backup within 26 h, stanza ok, WAL archive failures not rising, else the service fails, never skips) and takes a pg_dump -Fc at most weekly (or --logical) for the existing off-site upload; restore-download checks .dump files with pg_restore --list. Drills restore the dump into a scratch database under a freshly generated role (pg_restore --no-owner --no-privileges --role), boot the release with DATABASE_URL and DATABASE_DIRECT_URL on the scratch database (direct port) and VALKEY_URL on a closed port, and refuse to start if any of the three would keep production's value or any PostgreSQL database is left unmapped; the scratch database and role are always dropped (dropdb --force, also after a failed stop), leftovers of a killed drill are swept first, free space is checked, and postgres can traverse the drill directory (0711). Probes gain postgresql-count, nginx tenants read ov_host over psql, metrics openvibe_backup_pg_verified and openvibe_pgbackrest_last_backup_age_seconds, alerts OpenVibeBackupSkipped and DataBackupStale. host.example.json converts the 19 PostgreSQL services (Media's drill also requires DATABASE_URL handling). Built by DeepSeek from the design brief, reviewed independently (FIX) and by Claude; every finding fixed here with tests (unmapped database refused, 0711 directory, leftovers swept, --force and --no-privileges). npm test 45/45. npm test 45/45 test files passed.
Changes
@@@ -15,14 +15,14 @@ Every repository ships its own deploy script, and each one encodes rules learned|---|---|---|| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. || Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |--| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`sqlite-count`: `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only **as the service user**. |++| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`sqlite-count` of a file, or `postgresql-count` of a database in the cluster: Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only **as the owning user** (the service user, or the postgres OS user for the cluster). || Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. || Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. || Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. || Build, install vhosts, `nginx -t`, reload | Sites `deploy.sh` | `build` hooks and `nginx.installOnDeploy`. Vhost installs are transactional: if `nginx -t` fails, the previous files come back and nginx is not reloaded. || Build a release while the old one serves; switch a symlink; roll back to a release with its own `node_modules`; pid 1 must hold the socket | Live `deploy.sh` (release layout), OpenRe `deploy.sh release`/`api` | Strategy `release-layout` ([docs/deploy-strategies.md](docs/deploy-strategies.md)). The socket unit is restarted only when its unit file changed or systemd does not hold the port. || Per-app installs (not `apps/_shared`), jobs runtime and guard load before the restart, every `openvibe-tools*` unit | Tools `deploy.sh` | Strategy `multi-app`: `skipPackages`, `preflight`, `unitsMatch`, `ready.release`. |--| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm, builds, SQLite queries and backups all run as the declared owner/service user (`runuser` from root, `sudo -u` otherwise). |++| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm, builds, SQLite queries and backups all run as the declared owner/service user (`runuser` from root, `sudo -u` otherwise); PostgreSQL dumps and probes run as the postgres OS user. |Stage A adds a few safety rules of its own:@@@ -33,7 +33,7 @@ Stage A adds a few safety rules of its own:## Owns--- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled SQLite backups with retention and encrypted off-host copies, restore drills.++- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled backups (online SQLite copies, pgBackRest verification and logical PostgreSQL dumps) with retention and encrypted off-host copies, restore drills.- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user.@@@ -92,10 +92,11 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak- The socket unit is **never stopped or restarted**, in any flow, except the release-layout rebind: only when its unit file changed or systemd does not hold the port (`test/strategy-release-layout.test.js`).- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.--- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok`. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.++- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok` (a SQLite copy) or `pg_restore --list` on the `.dump` of a PostgreSQL service. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.- **Backups:**- `backup --all` backs up every service with databases. One failing service leaves the others backed up and exits 2.++ - A PostgreSQL service is verified, not copied: every run checks pgBackRest (newest backup within 26 h, stanza ok, `pg_stat_archiver.failed_count` not rising) and reports `verified`; a logical `pg_dump -Fc` per database is taken weekly or with `--logical`, and a **failed verification** is `failed`, never `skipped`.- Retention keeps the newest good backup of each of the last 7 days and 4 ISO weeks. A service whose backups keep failing never loses its last good ones. Directories ovhost did not record are never touched.- Every backup directory is root 0700 and every copy is root 0600, with no `-wal`/`-shm`. A copy that is a link is refused.- Off-host copies round-trip through AES-256-GCM. Tampering, truncation, swapped objects, a forged manifest and another key are all refused.@@@ -124,7 +125,7 @@ ovhost nginx render <service> [--variant http|sse|websocket] [--manifest <file>]ovhost nginx tenants <service> [--wildcard-cert <name|dir>] [--install] Stage B tenant vhostsovhost snapshot <service> [--out <file>]ovhost backup <service>--ovhost backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]++ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]ovhost offsite push [--run <run>] | offsite list [<service>] | offsite checkovhost archive push <file>... [--note <text>] | archive list | archive restore <stamp> <path> --out <dir>ovhost restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]@@@ -163,10 +164,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe`sudo ovhost backup --all --offsite` runs daily from `deploy/systemd/openvibe-backup.timer` (03:30 UTC). It works like this:- It backs up every service that declares databases. A failing service never stops the others; exit `2` if anything failed.++- A **SQLite** database is copied online (as the service user). A **PostgreSQL** database is verified, not copied: every run checks pgBackRest for the whole cluster and records `verified`, and a logical `pg_dump -Fc` per database is taken when the last good one is over 7 days old or on any run with `--logical`. A failed verification is `failed`, never `skipped`.- It keeps the last 7 daily and 4 weekly good backups per service.- It writes a JSON summary to `<stateDir>/backup-runs/<run>.json`.- It encrypts every copy on the host (AES-256-GCM, Node crypto, with a key that stays on the host) and uploads it to S3-compatible storage (Backblaze B2). Off-host copies are kept for 30 days.--- Backups are `root:root`: directories 0700, files 0600, no `-wal`/`-shm`.++- Backups are `root:root`: directories 0700, files 0600, no `-wal`/`-shm`. PostgreSQL dumps are made as the postgres OS user over peer auth; ovhost reads no service env value.`ovhost restore-download` fetches, verifies and decrypts a run into a new root-only directory, and never touches a live database. See [docs/backups.md](docs/backups.md) for the install steps, the env names (`/etc/openvibe/backup.env`), the encryption format and the restore procedure.@@@ -177,21 +179,22 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passeEach inventory entry's `drill` block declares:- `port`: the service port + 10000 in the example;--- `databases`: the env var that points the service at each restored copy;++- `databases`: the env var that points the service at each restored copy, or `{ "url": "<NAME>", "directUrl": "<NAME>" }` for a PostgreSQL database (the drill creates a scratch database and role, restores the `.dump` into it, and sets both vars to it);- `env`: overrides that turn side effects off. Values may use `{tmp}`, `{port}` and `{db:<name>}`;- `dirs` to create inside the drill directory;- `ready`: the readiness path;- `compare`: paths, plus volatile `ignore` keys where needed;- `counts`: `{ db, table }` pairs;++- `countsTolerance` (default `0`): how many rows a busy table may differ by, because production keeps writing between the backup read and the drill;- `supported: false` with a `reason` when a second instance cannot run without side effects.--In `host.example.json`, 24 services have drill blocks, and 23 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`), `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.++In `host.example.json`, 24 services have drill blocks, and 23 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`) or a PostgreSQL database's two URL env vars, `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. A PostgreSQL drill always overrides `DATABASE_URL`, `DATABASE_DIRECT_URL` and `VALKEY_URL` and drops its scratch database and role in a `finally` ([docs/restore-drills.md](docs/restore-drills.md#a-postgresql-drill)). The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.### Inventory--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `sqlite-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRe ingest sessions, Tools running jobs), the `drain` policy, `databases`, `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRe's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `sqlite-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRe ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a SQLite file `{ "name", "path" }` or a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRe's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.@@@ -46,6 +46,17 @@ groups:labels: { severity: page }annotations:summary: "No successful nightly backup for more than 30 hours (or no backup metrics at all)"++ # A service whose databases are gone (the 2026-09 SQLite→PostgreSQL switch archived the .db files++ # and dropped the path from the inventory) is reported `skipped` by backup --all. openvibe_backup_last_run_ok++ # stays 1 (a skip is not a failure) and no other backup alert fires, so production can sit with no backups++ # at all. 30 h is longer than the 24 h between runs, so ONE skipped run never reaches `for` and TWO++ # consecutive ones always do.++ - alert: OpenVibeBackupSkipped++ expr: openvibe_backup_services{status="skipped"} > 0++ for: 30h++ labels: { severity: ticket }++ annotations:++ summary: "A service has been skipped by two nightly backups in a row ({{ with query \"openvibe_backup_services{status='skipped'}\" }}{{ . | first | value }}{{ end }} service(s)): its databases are gone, so it has no backups"# Production drift (OpenVibe.Network registry/deploy-drift.js, hourly against GitHub; WS-S task 7).# max_over_time: Network restarts drop the series for a minute, which reset `for` and made the alert# resolve and fire again every few hours (2026-09-28). Documentation-only drift is not counted.@@@ -149,12 +160,17 @@ groups:labels: { severity: page }annotations:summary: "WAL archiving to the pgBackRest repository is failing: point-in-time recovery is at risk"++ # Cluster-wide pgBackRest staleness. openvibe_pgbackrest_last_backup_age_seconds is written by++ # `ovhost backup --all` (the nightly verify pass reads the stanza and records the age of its newest++ # backup), NOT by the pgbackrest-backup@ unit's own backup-metric.sh: that keeps writing++ # openvibe_pgbackrest_last_success_timestamp_seconds, which stays. absent() covers "no backup --all++ # run has written the age at all" — a host where the nightly verify never ran.- alert: DataBackupStale-- expr: time() - max(openvibe_pgbackrest_last_success_timestamp_seconds) > 26 * 3600++ expr: max(openvibe_pgbackrest_last_backup_age_seconds) > 30 * 3600 or absent(openvibe_pgbackrest_last_backup_age_seconds)for: 30mlabels: { severity: ticket }annotations:-- summary: "No successful pgBackRest backup for more than 26 hours"++ summary: "No pgBackRest backup of the cluster has stopped within 30 hours (point-in-time recovery is at risk)"- alert: DataValkeyMemoryHighexpr: redis_memory_used_bytes / clamp_min(redis_memory_max_bytes, 1) > 0.9for: 15m@@@ -1,12 +1,13 @@[Unit]--Description=OpenVibe backups: every service's SQLite databases, then an encrypted off-host copy++Description=OpenVibe backups: verify pgBackRest for the PostgreSQL services, dump SQLite databases and the scheduled logical PostgreSQL dumps, then an encrypted off-host copyDocumentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.mdAfter=network-online.targetWants=network-online.target[Service]Type=oneshot--# Root: ovhost drops to each service user for the SQLite backup (runuser) and is the only reader of++# Root: ovhost drops to each service user for the SQLite backup and to the postgres OS user for the++# PostgreSQL dumps and the pgBackRest verification (runuser) and is the only reader of# /etc/openvibe/backup.env, the encryption key and the root-only backups. The env file is read by# ovhost itself, not loaded into this unit's environment.User=root@@@ -18,7 +19,8 @@ UMask=0077Nice=10IOSchedulingClass=best-effortIOSchedulingPriority=7--# media.db and live.db are the large ones; the SQLite worker's own limit is 6 h per database.++# The PostgreSQL logical dumps are the large ones now (they run as the postgres OS user), and the++# SQLite worker's own limit is 6 h per database; a pg_dump is given the same. 6 h covers a whole run.TimeoutStartSec=6hPrivateTmp=trueNoNewPrivileges=true@@@ -2,8 +2,10 @@`ovhost backup --all --offsite` runs once a day from a systemd timer. It backs up every databasethe inventory declares, keeps a week of daily and a month of weekly copies on the host, and uploads--an encrypted copy of each run to S3-compatible object storage (Backblaze B2). This page covers what it--does, how to install it, and how to restore from the off-host copy.++an encrypted copy of each run to S3-compatible object storage (Backblaze B2). A SQLite database is++copied; a PostgreSQL service is covered by the cluster's pgBackRest backups, which every run verifies,++with a per-service logical dump on a slower cadence. This page covers what it does, how to install it,++and how to restore from the off-host copy.Restore drills (a second instance started from a backup and compared with production) are in[restore-drills.md](restore-drills.md). A drill works on an off-host copy too: see@@@ -12,18 +14,22 @@ Restore drills (a second instance started from a backup and compared with produc## What is backed up`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has--a `databases` list. With `host.example.json` that means:----| Service | Databases |--|---|---|--| network | `network.db` |--| live | `live.db` (includes the money ledger while `BILLING_AUTHORITY=live`), `rs-companion.db`. `data/analytics.db` is left out on purpose ([storage-state.md](storage-state.md): analytics, hard-coded path, not canonical). |--| media | `media.db` (the media files themselves are in B2/R2, not covered here) |--| tools | seven `apps/*/data/analytics.db`, and `jobs.db` for img, audio and docs |--| games | `world.db` |--| ai | `ai.db` |--| openre | `openre.db` |--| community, events, chat, billing, host, search, sources, wiki, blog, tips, vip, news, reviews, deals, coupons, trade, codes | one database each |++a `databases` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):++SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster++(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:++++| Service | Engine | Databases |++|---|---|---|++| network | SQLite | `network.db` |++| chat | SQLite | `chat.db` |++| tools | SQLite | seven `apps/*/data/analytics.db`, and `jobs.db` for img, audio and docs |++| games | SQLite | `world.db` |++| live | SQLite | `live.db` (includes the money ledger while `BILLING_AUTHORITY=live`), `rs-companion.db`. `data/analytics.db` is left out on purpose ([storage-state.md](storage-state.md): analytics, hard-coded path, not canonical). |++| tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media, openre | PostgreSQL | one database each in the cluster, named `ov_<name>` (for example `ov_trade`); there is no local file. The media files themselves are in B2/R2, not covered here. |++++The 19 PostgreSQL services each run the every-night pgBackRest verification and a scheduled logical++dump; the five SQLite services keep the online copy described below. [PostgreSQL++services](#postgresql-services) covers what that means.A database that does not exist yet (a service that is not deployed) is reported as `skipped` anddoes not fail the run. Adding a database to the inventory is all it takes to include it.@@@ -46,6 +52,10 @@ For each service, one after another:`ok: false` when any database failed.5. **Prune** (see [Retention](#retention)).++Steps 1-3 are the **SQLite** path. A PostgreSQL service does not copy a file at all: every run first++verifies pgBackRest, and a logical dump is taken only when the last good one is old enough (see++[PostgreSQL services](#postgresql-services)). `drill` uses the same `.dump` artifacts.++**Permissions.** Backups hold user data. Every directory under `/var/backups/openvibe` is`root:root 0700` and every copy is `root:root 0600`, with no `-wal`/`-shm` next to it. Directories leftby older versions (`ubuntu`, 0750) are locked down on the next run. The staging parent@@@ -55,21 +65,85 @@ by older versions (`ubuntu`, 0750) are locked down on the next run. The stagingservice never stops the other services. The run's exit code is `2` when anything failed and `0`otherwise. If a service fails halfway (one of two databases), the good copy is kept but the backup ismarked `ok: false`. Such a backup is never picked by a drill, and it is pruned once a newer good backup--exists.++exists. A PostgreSQL service whose pgBackRest verification fails is reported `failed`, **never++`skipped`**: a broken backup layer is a failure, not an absent database.**Summary.** Each run writes `/var/lib/openvibe-host/backup-runs/<run>.json` (`root`, 0640). It holdsper-service status, files, sizes, errors, what was pruned, and the off-host result: objects, bytes,--failures, and the key id. The summary never holds a secret. The run id is the start time in UTC, for--example `20260924-033412`.++failures, and the key id. A PostgreSQL service that only verified pgBackRest is reported `verified`.++The summary never holds a secret. The run id is the start time in UTC, for example `20260924-033412`.```sudo ovhost backup --all # local onlysudo ovhost backup --all --offsite # what the timer runssudo ovhost backup --all --json # the summary on stdout++sudo ovhost backup --all --logical # force the logical PostgreSQL dumps toosudo ovhost backup --all --keep-daily 14 --keep-weekly 8sudo ovhost backup --all --no-prune```++## PostgreSQL services++++A PostgreSQL service's data does not live in a file ovhost can copy. It lives in the host's cluster, so++ovhost does not make the durability layer — [pgBackRest](#pgbackrest-is-the-durability-layer) already++does — it **verifies** that layer on every run, and takes a per-service logical dump on a slower cadence++for restores and drills. Everything runs as the `postgres` OS user over peer auth++(`local postgres peer` in `roles/data/files/pg_hba.conf`), so ovhost still reads no service env value.++++**Every night: verify pgBackRest.** Before anything else, the run asks pgBackRest about the whole-cluster++stanza (`runuser -u postgres -- pgbackrest --stanza=openvibe info --output=json`). Two conditions must++hold:++++- the stanza's newest backup **stopped within 26 hours** and the stanza **status is ok** (newest first,++ whatever its type);++- `pg_stat_archiver.failed_count` has **not increased** since the last run. The last value ovhost saw is++ kept in `<stateDir>/pgbackrest-archiver.json`, and each run compares against it, so a WAL segment that++ failed to archive is caught even though it is not visible in a single night's snapshot.++++The result is recorded per service in `<stateDir>/backups/<svc>.jsonl` as `verified: { type, stop, walOk }`.++A **failed verification** makes that service's status `failed` — never `skipped` — so `ovhost backup --all`++exits `2`, `openvibe_backup_last_run_ok` goes to `0` and `OpenVibeBackupFailed` pages. A run where every++PostgreSQL service only verified is `verified` in the summary.++++**Weekly, or on demand: the logical dump.** A `pg_dump -Fc` of each of the service's databases is taken++when the last good dump is older than **7 days**, or on any run with `--logical`. It lands in the existing++layout, one file per database:++++```++/var/backups/openvibe/<service>/<stamp>/<name>.dump # pg_dump -Fc, owned root:root 0600++```++++The off-site code then encrypts and uploads each `.dump` exactly as it does a `.db` copy, so a `.dump`++reaches the bucket as `<service>/<name>.dump.ovbk` and is restored by `restore-download` unchanged.++`pg_restore --list` proves a `.dump` is a readable archive.++++**Metrics and alerts.** The run rewrites `/var/lib/prometheus/node-exporter/openvibe_backup.prom` for++node_exporter's textfile collector:++++- `openvibe_backup_pg_verified{service}` — `1` when that PostgreSQL service's last verification passed,++ `0` when it failed;++- `openvibe_pgbackrest_last_backup_age_seconds` — the age of the stanza's newest backup (gauge, seconds);++- `openvibe_backup_services{status}` gains the value `verified` (a service with a verification-only run).++++`deploy/prometheus/openvibe-rules.yml` reads those:++++- `DataBackupStale` fires when `openvibe_pgbackrest_last_backup_age_seconds` is over **30 hours**, or the++ metric is absent (no run has written it). It is the cluster staleness alert; the per-service++ `openvibe_backup_pg_verified` says which service's verification failed.++- `OpenVibeBackupSkipped` fires when a service has been `skipped` for **30 hours** — two nightly runs in a++ row. A service whose databases are gone (for example one whose SQLite file was archived after the++ 2026-09 switch to PostgreSQL) is `skipped`, `openvibe_backup_last_run_ok` stays `1` and nothing else++ fires, so this is what catches "production with no backups". The rules live in++ [`deploy/prometheus/openvibe-rules.yml`](../deploy/prometheus/openvibe-rules.yml).++++### pgBackRest is the durability layer++++It backs up the whole cluster — encrypted, off-host, with point-in-time recovery: a full backup every++Sunday, a differential nightly, and the WAL stream. The per-service `.dump` files are **not** the++durability layer; they are smaller, service-scoped, portable restore units (a logical archive survives a++PostgreSQL major-version change, and it is what the drills restore and compare). Losing the cluster is++recovered from pgBackRest; losing one service's rows, or drilling a service, uses its `.dump`.++## Retention**On the host** (per service). Kept: the newest good backup of each of the last **7 days** that have@@@ -93,7 +167,8 @@ To leave deletion to a bucket lifecycle rule instead, set `BACKUP_OFFSITE_PRUNE=### Layout```--s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.db.ovbk++s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.db.ovbk # a SQLite copy++s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.dump.ovbk # a PostgreSQL dumps3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/manifest.json```@@@ -228,8 +303,11 @@ sudo ls -t /var/lib/openvibe-host/backup-runs | head -1sudo jq '{ok, services: [.services[] | {service, status, error}], offsite: {ok: .offsite.ok, failures: .offsite.failures}}' /var/lib/openvibe-host/backup-runs/<run>.json```--Nothing alerts on a failed run yet. The unit fails (exit 2) and the summary says `"ok": false`. Hook--an `OnFailure=` unit or the monitoring stack to that when one exists.++The monitoring stack alerts on it (all in `deploy/prometheus/openvibe-rules.yml`):++`OpenVibeBackupFailed` pages when the last run failed, `OpenVibeBackupMissed` when no run has succeeded++for 30 hours, `OpenVibeBackupSkipped` when a service has no databases left to back up, and++`DataBackupStale` when pgBackRest itself is stale. The unit also fails (exit 2) and the summary says++`"ok": false`.### Bucket (Backblaze B2)@@@ -280,10 +358,10 @@ For each file, the command:- checks the manifest's HMAC;- streams the object through GCM decryption;- checks both SHA-256 hashes against the manifest;--- runs `PRAGMA quick_check` on the result.++- runs `PRAGMA quick_check` on the result of a `.db` copy, or `pg_restore --list` on a `.dump`.Any failure removes that file and exits non-zero. The output lists each file with its size, its hash,--`quick_check ok` and the production path it came from.++its check and the production path it came from.### 3. Put it in place (manual, one service at a time)@@@ -343,10 +421,11 @@ sudo rm -rf /var/lib/openvibe-restore/drill-community```The drill:--- copies the databases into its sandbox as the service user (`install -o <user> -m 0600`);--- runs `integrity_check`;++- restores each database. A SQLite copy goes into the sandbox as the service user (`install -o <user>++ -m 0600`) and is checked with `integrity_check`; a PostgreSQL `.dump` is restored with `pg_restore`++ into a scratch database created for the drill ([restore-drills.md](restore-drills.md));- starts a sandboxed second instance and compares it with production;--- stops it and cleans up.++- stops it and cleans up (a PostgreSQL drill always drops its scratch database and role).Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.@@@ -369,7 +448,7 @@ Do this at least once after installing, and again after any change to the key or| Command | What it does | Exit ||---|---|---|--| `ovhost backup --all [--offsite] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases; prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |++| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure || `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure || `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error || `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |@@@ -163,6 +163,14 @@ Each service that moves keeps its SQLite file read-only for the 7-day N-1 window| C-89 media | Media's SQLite file after the switch to PostgreSQL (2026-09-28 22:54 UTC, release `e3f2239`; import verified, counts and checksums as the rehearsal on a production copy: 3,173 media objects, 2,998 locations, 1,021 variants, 1,321 relationships, 792 vods, 324 clips, 891 pastes, 291 jobs, 789 outbox events, 29 tables with rows). The switch waited for recordings to be idle (switch-service.sh `019d853`). Also `media.pre-postgres-20260928T225438Z.db` | `/opt/openvibe.media/data/media.db` (0400) | Media | 7 days without a rollback: `ovhost archive push` both files, then delete | 2026-10-05 | `sudo ovhost rollback media` to `47b21bd` (reads the SQLite file; objects, jobs and views written since the switch would be lost, so export them first) || C-89 openre | OpenRe.Stream's SQLite file after the switch to PostgreSQL (2026-09-28 23:32 UTC, release `405682c`; import verified: 1 stream definition, 1 key, 1 session and its 4 transitions, 10 worker rows, 2 outbox events; every OpenRe unit stopped for the import through `SWITCH_UNITS`; worker generation 6 started after). Also `openre.pre-postgres-20260928T233208Z.db` | `/var/lib/openre/openre.db` (0400) | OpenRe | 7 days without a rollback: `ovhost archive push` both files, then delete | 2026-10-05 | `sudo ovhost rollback openre` to `d9f4240` and `deploy.sh workers` (reads the SQLite file; definitions and keys created since the switch would be lost) |++**Archive step unblocked (2026-09-28).** `ovhost archive push` refuses a path the inventory declares as a++live SQLite database (`lib/archive.js`: it refuses a file that is any non-postgresql `databases[].path`).++Until the switch, that rule covered these frozen `.db` files, so the "archive both files, then delete"++step of C-89 could not run. The 19 services above now declare `engine: postgresql` with no `path`, so their++sqlite entries no longer name a file and archive push accepts the frozen `.db` files. The archive-then-delete++step of each C-89 row can proceed once its 7-day window closes; the removal condition and target of every++row stand unchanged.++## Kept by design (not shims)These match the search terms but are permanent. They are listed so the next review does not re-open them.@@@ -1,6 +1,51 @@--# Database files on the host (WS-S task 6, hazard H14)++# Databases: the inventory model, and the files left on the host (WS-S task 6, hazard H14)--Inventory of 2026-09-26: every SQLite file under /opt, /var/lib, /root, /home and /srv that no process had open. 42 open files (the live databases) are not listed. Paths, sizes and dates only; nothing was read.++`host.example.json` declares each service's databases as a `databases[]` list. An item is a union of two++**engines** ([lib/inventory.js](../lib/inventory.js) normalises them, [lib/dbengine.js](../lib/dbengine.js)++holds the per-engine helpers):++++- **SQLite** — a file the service user's worker copies:++ ```json++ { "name": "live", "path": "/opt/openvibe.live/data/live.db" }++ ```++- **PostgreSQL** — a database in the host's cluster, named, with **no `path`**:++ ```json++ { "name": "trade", "engine": "postgresql", "database": "ov_trade" }++ ```++++`engine` defaults to `sqlite`. A PostgreSQL `database` must match `/^ov_[a-z0-9_]+$/`; a PostgreSQL entry++has no `path`, and a SQLite entry has no `database`. The data lives in the cluster, so ovhost never opens++it by copy: pgBackRest backs the whole cluster up, the nightly run verifies that, and a per-service++logical `.dump` feeds the drills ([backups.md](backups.md#postgresql-services)).++++**Which engine.** With `host.example.json`:++++| Engine | Services |++|---|---|++| PostgreSQL | tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media, openre (19) |++| SQLite | network, chat, tools, games, live (5) |++++**The two engines appear in four more places in the inventory:**++++- `drill.databases`. A SQLite database keeps a string (the env var that points the service at the restored++ file) or `{ "env": "DATA_DIR", "dir": true }` for a service that takes a data directory; a PostgreSQL++ database is `{ "trade": { "url": "DATABASE_URL", "directUrl": "DATABASE_DIRECT_URL" } }`, and the drill++ creates a scratch database and role and points both vars at it ([restore-drills.md](restore-drills.md)).++- `drill.countsTolerance` — a whole number of rows, default `0`. Production keeps writing between the read++ that makes the backup and the dump, so a busy table may differ by up to this many rows without failing++ the drill.++- `nginx.tenants.database` — the Stage B tenant database (the Host API is on PostgreSQL): a SQLite path++ keeps its string form, and PostgreSQL is the object `{ "engine": "postgresql", "database": "ov_host" }`.++- the `protected` probe kind `postgresql-count` — `{ "kind": "postgresql-count", "database": "ov_openre",++ "sql": "SELECT count(*) …", "label": "…" }`: a read-only SELECT against a database in the cluster, run as++ the postgres OS user. OpenRe's protected probe (its active ingest sessions) is one, on `ov_openre`. The++ other kinds are `http-json-count`, `sqlite-count` and `sum` ([lib/probes.js](../lib/probes.js)).++++---++++The rest of this page is the 2026-09-26 sweep of SQLite **files**: every file under /opt, /var/lib, /root,++/home and /srv that no process had open. 42 open files (the live databases) are not listed. Paths, sizes++and dates only; nothing was read. A PostgreSQL service has no row here: its rows are in the cluster.`ovhost archive push <file>...` encrypts files with the backup key and copies them to `<BACKUP_S3_PREFIX>-archive/<host>/<stamp>/`, outside the backup runs so retention never prunes them; `ovhost archive list` and `ovhost archive restore <stamp> <path> --out <dir>` read them back. It never deletes a local file.@@@ -90,7 +90,8 @@ block (see `host.example.json`):Backups are root-only (directories 0700, files 0600, see [backups.md](backups.md)), so root makesthe copy with `install -o <service user> -m 0600`, and the copy belongs to the service user.`PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;-- otherwise the drill fails before anything starts.++ otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into a++ scratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).2. **Start.** A second instance starts from the production checkout through `systemd-run`, as theservice user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the productionunit's non-secret `Environment=` values and the production env file. After that it loads@@@ -124,6 +125,58 @@ cleaned up.A drill compares the backup with production *now*. If writes landed after the backup, row counts andlist endpoints differ, and the drill reports that honestly. Take the backup right before the drill.++### A PostgreSQL drill++++Step 1 above is the SQLite path. A service whose `databases[]` entry is `engine: postgresql`++([db-inventory.md](db-inventory.md)) restores a **logical dump** instead of a file copy:++++1. **Restore.** The drill takes the service's newest `<name>.dump` (`pg_dump -Fc`, from++ `<stateDir>/backups/<service>.jsonl`, or the `--backup <dir>` directory) and creates a scratch++ database `ov_<id>_drill_<stamp>` owned by a **freshly created login role** `ov_<id>_drill_<stamp>`,++ whose password is generated with `crypto.randomBytes` and never printed. It then runs++ `pg_restore --no-owner --no-privileges --role=<drill role>` into that database: `--no-owner` because the++ archive names the production role, `--no-privileges` so none of the archive's GRANTs hands a production++ role anything, `--role` so every object ends up owned by the scratch role the drilled instance connects++ as. Before restoring, scratch databases and roles an earlier, killed drill of the same service left++ behind (`ov_<id>_drill_*`) are dropped, and every drop uses `dropdb --force`.++2. **Start.** `drill.env` overrides the three values the drilled release would otherwise take from++ production:++ - `DATABASE_URL` and `DATABASE_DIRECT_URL` — the scratch database on `127.0.0.1:5432`;++ - `VALKEY_URL=redis://127.0.0.1:9/0` — a closed port.++++ The drill **refuses to start** if any of the three would keep its production value. They are not++ optional, because the sandbox runs the release's real boot path:++ - **`DATABASE_DIRECT_URL` un-overridden would migrate production.** The release runs its migrations++ from the direct URL at boot, and a drill is deliberately pointed at a copy precisely so that boot++ can run them there.++ - **`VALKEY_URL` un-neutralised would write into the production keyspace.** The drill sandbox allows++ loopback (services need Network's public keys there), so a Valkey URL left at production would let++ the second instance write locks, caches and queues into production's keyspace.++ - **`DATABASE_URL` un-overridden would serve production data.** The drilled instance would read and++ write the live database instead of the restored copy, and every comparison would be vacuous.++3. **Compare.** `counts` are read from production and from the scratch database over `psql` (the SQLite++ drills compare files). `countsTolerance` in the drill block (default `0`) is the number of rows that++ may differ: production keeps writing between the read that makes the backup and the read that counts++ it, so a busy table is allowed to drift by that many rows instead of failing the drill.++++The connection is `127.0.0.1:5432`, the cluster itself, and deliberately **not PgBouncer's 6432**:++PgBouncer pools the production app roles and databases, and the scratch database and its generated role++exist only in the cluster, not in PgBouncer's configuration. Connecting directly to the port the++`pg_restore` used is also what guarantees the instance reaches the scratch database and not a pool that++happens to route back to production.++++Free space is checked before the restore: the drill refuses below **2x the dump size + 1 GB**, because++`pg_restore` needs room for the restored tables on top of the archive and the database's own WAL and++temporary files.++++The scratch database and its role are dropped in a `finally`, together with stopping the instance, on++success and on failure. A PostgreSQL drill always cleans up after itself; `--keep` keeps the drill++directory (the copy, `drill.env`, `drill.log`) but never the scratch database or role.++++For such a service the **Checks** column in the table above reports `pg_restore --list` (that the dump is++a readable archive) rather than `pragma integrity_check`, and each count is read over `psql` from++production and from the scratch database.++### ServicesSupported with declared overrides (the table above records which have passed on the host):@@@ -238,7 +291,9 @@ games as of 22 Sep) have entries built from their repositories' `deploy/` directport.- `databases.<name>: { "env": "DATA_DIR", "dir": true }`, or `"dir": "{tmp}/<name>"`: the copy keepsits production file name inside that directory, and the env var, if one is given, points at the-- directory.++ directory. For a `postgresql` database the entry is instead `{ "url": "<NAME>", "directUrl":++ "<NAME>" }`: the two env vars that point the drilled service at a database URL, which the drill sets++ to the scratch database (see [A PostgreSQL drill](#a-postgresql-drill)).- `bind: [{ "from": "{tmp}/…", "to": "<path inside the checkout>" }]`: `BindPaths=` in the drillunit's own mount namespace. Use it for files a service opens relative to its checkout with no envoverride: the restored copy appears at the production path for the drill instance only. Tools yt,@@@ -15,8 +15,31 @@ ADR-007 (`OpenVibe.Contracts/docs/adr/ADR-007-data-ownership.md`) decides:host, or sizes SQLite handles poorly.- Money paths use SERIALIZABLE transactions and row locks when they move.--Every service below runs on SQLite (better-sqlite3), with one database per service. No repository has a--PostgreSQL driver or config. **This page is the Wave 22 review of that state.**++As reviewed on 2026-09-23 every service ran on SQLite (better-sqlite3), with one database per service.++Since 2026-09-28 nineteen of them run on the host's **PostgreSQL cluster** (ADR-035), and five stay on++SQLite: network, chat, tools, games and live. The per-service table below is the Wave 22 review of the++SQLite state; where a service switched, its SQLite file is now frozen read-only for the 7-day rollback++window (compatibility-register.md, C-89), and the live rows are in the cluster. **This page remains the++Wave 22 review of that state.**++++## Where the data lives now++++- **PostgreSQL services** (tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes,++ vip, ai, host, events, community, billing, media, openre) keep their rows in the cluster on the data++ role (`roles/data/`, ADR-035): PostgreSQL 18 on loopback, one database per service named `ov_<name>`++ (`ov_trade`, `ov_media`, …). The service reaches it through **PgBouncer** on `127.0.0.1:6432`, which++ fronts the cluster for the app roles (and carries the app role's SCRAM password). Their durability is++ **pgBackRest**, off-host: a full cluster backup weekly, a differential nightly and the WAL stream, which++ the nightly `ovhost backup` verifies rather than copies ([backups.md](backups.md#postgresql-services)).++- **What stays on the local filesystem** for those services is the rest of the data: object and media++ files under `*_PATH` and the object stores (Media's B2/R2), blobs (`HOST_STORAGE_DIR`), uploads and++ outputs, plus the frozen pre-switch SQLite file kept read-only for rollback. Those files are not in the++ cluster and are backed up (or not) exactly as before.++- **SQLite services** are unchanged: one file per service, copied by the online backup API as the service++ user ([backups.md](backups.md)).++++`openvibe-shared/ready` is still what each service's `/ready` reports ([below](#per-service)); a switched++service's `db` check reads the cluster as its `DATABASE_URL`/`DATABASE_DIRECT_URL` do.Sources:- Each repository's `server/config.js`, `.env.example` and `deploy/systemd/*.service`.@@@ -81,9 +104,9 @@ Sources:production path. Whether `OPENVIBELIVE_DB_PATH` is set is unknown: check `/etc/openvibe/network.env`.- **Chat reads Live's files** at `/opt/openvibe.live/data/sounds`. These are files, not a database(`deploy/systemd/openvibe-chat.service:43`).--7. **Canonical money is still in Live.** Billing's `billing.db` is a shadow (Billing-- `STATUS.json: "authoritative": false`) until the cutover in [cutover-runbook.md](cutover-runbook.md)-- step B.++7. **Canonical money is still in Live.** Billing's ledger (was `billing.db`, now the cluster database++ `ov_billing` after the 2026-09-28 switch) is a shadow (Billing `STATUS.json: "authoritative": false`)++ until the cutover in [cutover-runbook.md](cutover-runbook.md) step B.## Per service@@@ -91,6 +114,14 @@ Sources:the operator notes unless cited. Where a repo's `STATUS.json` disagrees, the operator notes and the drilllog win.++**The rows record the 2026-09-23 SQLite state.** For a service that has since switched to PostgreSQL —++tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events,++community, billing, media, openre (C-89) — the "Backing store" path below is now the **frozen pre-switch++SQLite file** kept read-only for the 7-day rollback, and the live store is the cluster database++`ov_<name>` reached through PgBouncer, as in [Where the data lives now](#where-the-data-lives-now). The++five SQLite rows (network, chat, tools, games, live) are current. The handler citations are the ones++reviewed on 2026-09-23.++| Service | Backing store (production path; env, default) | Canonical? | ADR-007 status | Readiness checks (handler) | Store checked? | Backup / drill ||---|---|---|---|---|---|---|| **Network** | `/opt/openvibe.network/data/network.db`; `DB_PATH`, default `./data/network.db` (`server/config.js:55`). Analytics lives in the same file (`README.md:306-307`). | Yes: identity, OAuth clients, subjects, OpenCoins wallets, themes, notifications, analytics | SQLite, the reviewed single-host state. `docs/platform-plan.md:74,111` defers PostgreSQL "until a room needs it". | `/api/ready` (`server/observability.js:58-96`). Required: `db` (`SELECT COUNT(*) FROM oauth_clients`), `signing_key` in production (signs and verifies an RS256 probe; fails on the HS256 dev key). Optional: `registry_poll`, `discord_bot`. | **Yes** | Drill passed 05:22. ovhost's deploy readiness polls the static `/api/health` (finding 3). |@@@ -1,11 +1,12 @@{"_about": "Example ovhost inventory for the single OpenVibe host (openvibe-oregon). Copy to /etc/openvibe/host.json (root:root, 0640) and correct anything that differs on the machine; the real file is never committed. Paths, units, ports and env file locations come from OpenVibe.Network docs/roadmap-baseline/data/services.json and prod-snapshot.json (22 Sep 2026) and each repository's deploy/ directory. env.required lists NAMES only.",++ "_databases": "Databases (lib/inventory.js): each services.<id>.databases[] item is one of two engines. SQLite (the default) is a file the service user's worker copies: { \"name\": \"live\", \"path\": \"/opt/openvibe.live/data/live.db\" }. PostgreSQL is a database in the cluster: { \"name\": \"trade\", \"engine\": \"postgresql\", \"database\": \"ov_trade\" } — the name must match ov_[a-z0-9_]+, and there is no path (ovhost reaches it as the postgres OS user over peer auth, and pgBackRest, not ovhost, copies it). tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media and openre are PostgreSQL; network, chat, tools, games and live are still SQLite. A drill block follows the engine: a PostgreSQL database maps to { \"url\": \"DATABASE_URL\", \"directUrl\": \"DATABASE_DIRECT_URL\" } and the drill creates a scratch database and role to restore into.","host": "openvibe-oregon","stateDir": "/var/lib/openvibe-host","backupDir": "/var/backups/openvibe","backupStagingDir": "/var/backups/openvibe.staging","drillDir": "/var/lib/openvibe-drills",-- "_drill": "Restore drills (ovhost drill <service>): each service with a database has a drill block. The drill port is the service port + 10000 on 127.0.0.1. http://127.0.0.1:9 is a closed port, used where an empty value would fall back to a production service on loopback. An empty OV_OAUTH_CLIENT_SECRET stops a service from minting service tokens, so it cannot call other services with its identity. Loopback stays reachable from the drill sandbox (services need Network's public keys), so these overrides are what keep a drill away from production. _notes explain what is left on and why.",++ "_drill": "Restore drills (ovhost drill <service>): each service with a database has a drill block. The drill port is the service port + 10000 on 127.0.0.1. http://127.0.0.1:9 is a closed port, used where an empty value would fall back to a production service on loopback. An empty OV_OAUTH_CLIENT_SECRET stops a service from minting service tokens, so it cannot call other services with its identity. Loopback stays reachable from the drill sandbox (services need Network's public keys), so these overrides are what keep a drill away from production. A PostgreSQL drill restores the dump into a scratch database under a freshly created role, then sets DATABASE_URL and DATABASE_DIRECT_URL to it and VALKEY_URL to the closed port; it refuses to start if any of the three would keep its production value, because the drilled release would otherwise run its migrations against production (DATABASE_DIRECT_URL) and write into the production Valkey keyspace (VALKEY_URL). _notes explain what is left on and why.","_lifecycle": "Lifecycle (ovhost validate, WS-P task 1): each service's lifecycle comes from its manifest in openvibe-contracts (the service's `manifest` id, default its key), from openvibe-contracts 0.55.0 on. A `lifecycle` block in a service entry replaces the manifest's on this host; none is needed today. validate compares shutdown.deadlineSeconds with each unit's effective TimeoutStopSec and KillSignal, and the checkout's installed openvibe-contracts with contracts.range.","nginx": {"sitesAvailable": "/etc/nginx/sites-available",@@@ -305,7 +306,6 @@"envFile": "/etc/openvibe/media.env","env": {"required": [-- "DB_PATH","VOD_PATH","MEDIA_B2_ENDPOINT","MEDIA_B2_BUCKET",@@@ -342,7 +342,8 @@"databases": [{"name": "media",-- "path": "/opt/openvibe.media/data/media.db"++ "engine": "postgresql",++ "database": "ov_media"}],"nginx": {@@@ -351,10 +352,13 @@"variant": "http"},"drill": {-- "_notes": "Needs OpenVibe.Media 6e74eb0 or later (MEDIA_DRILL, server/drill.js; requires checks the deployed checkout). MEDIA_DRILL=1 refuses to start unless DB_PATH is outside the checkout and /opt/openvibe.media, HOST is loopback and PORT is not 4100. It then serves HTTP only: no app seeding or JWKS refresh, no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant scans, disk guardian, thumbnail cleanup, object purge, backfill, orphan-recording finalize or Events relay, and no webhook (the apps table's URLs are production's). It writes no file and creates no directory, and never opens a file path from the database: every byte route (/v and /c bytes, /t, /a, /f, /o, paste screenshots, live frames) answers 503; watch pages and /browse render from the copy. Every method but GET/HEAD/OPTIONS answers 403, /auth/* 503; net, fetch, child_process (git only: no ffmpeg or ffprobe), dgram and listen are guarded. /api/ready drops the storage and remote-tier checks (mode: drill). The storage paths from media.env move under {tmp}/storage (never created) as a second switch; THUMBNAILS_PATH stays production's, read-only in the sandbox, because /browse counts the thumbnail directory in its tab bar (the drill lists it and serves none of it). /browse caches its tab counts for 60 s in production: take the backup right before the drill. EVENTS_URL and the client secret are blanked too. Not run on the host yet.",++ "_notes": "Needs OpenVibe.Media 6e74eb0 or later (MEDIA_DRILL, server/drill.js; requires checks the deployed checkout). MEDIA_DRILL=1 refuses to start unless its database is outside the checkout and /opt/openvibe.media, HOST is loopback and PORT is not 4100. It then serves HTTP only: no app seeding or JWKS refresh, no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant scans, disk guardian, thumbnail cleanup, object purge, backfill, orphan-recording finalize or Events relay, and no webhook (the apps table's URLs are production's). It writes no file and creates no directory, and never opens a file path from the database: every byte route (/v and /c bytes, /t, /a, /f, /o, paste screenshots, live frames) answers 503; watch pages and /browse render from the copy. Every method but GET/HEAD/OPTIONS answers 403, /auth/* 503; net, fetch, child_process (git only: no ffmpeg or ffprobe), dgram and listen are guarded. /api/ready drops the storage and remote-tier checks (mode: drill). The storage paths from media.env move under {tmp}/storage (never created) as a second switch; THUMBNAILS_PATH stays production's, read-only in the sandbox, because /browse counts the thumbnail directory in its tab bar (the drill lists it and serves none of it). /browse caches its tab counts for 60 s in production: take the backup right before the drill. EVENTS_URL and the client secret are blanked too. Not run on the host yet.","port": 14100,"databases": {-- "media": "DB_PATH"++ "media": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -367,13 +371,18 @@"FILES_PATH": "{tmp}/storage/files","OBJECTS_PATH": "{tmp}/storage/objects","EVENTS_URL": "",-- "OV_OAUTH_CLIENT_SECRET": ""++ "OV_OAUTH_CLIENT_SECRET": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"requires": [{"file": "server/drill.js","contains": "MEDIA_DRILL"},++ {++ "file": "server/drill.js",++ "contains": "DATABASE_URL"++ },{"file": "server/public/routes.js","contains": "refuseBytes"@@@ -663,14 +672,11 @@"databases": [{"name": "community",-- "path": "/var/lib/openvibe-community/community.db"-- },-- {-- "name": "community-legacy-path",-- "path": "/opt/openvibe.community/data/community.db"++ "engine": "postgresql",++ "database": "ov_community"}],-- "_databasesNote": "The repo unit puts the database in /var/lib/openvibe-community; the 22 Sep production snapshot found it under /opt/openvibe.community/data. Keep the one that exists (backup skips a missing path).",++ "_databasesNote": "Community is one PostgreSQL database (ov_community) now. Before the switch it declared two SQLite paths (the repo unit's /var/lib/openvibe-community/community.db and the /opt/openvibe.community/data/community.db the 22 Sep production snapshot found); neither is declared any more, and the frozen files are for `ovhost archive push`.","nginx": {"vhost": "openvibe.community.conf","repoVhost": "deploy/nginx/openvibe.community.conf",@@@ -679,7 +685,10 @@"drill": {"port": 14200,"databases": {-- "community": "COMMUNITY_DB_PATH"++ "community": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -687,7 +696,8 @@"DISCORD_RELAY_ENABLED": "0","OV_OAUTH_CLIENT_SECRET": "","OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9",-- "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9"++ "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/pastes?limit=5",@@@ -712,7 +722,7 @@"table": "posts"}],-- "_notes": "If the host keeps the database at the legacy path, map \"community-legacy-path\" instead of \"community\" (and in counts). An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."++ "_notes": "Community is one PostgreSQL database (ov_community) now: the second, legacy SQLite path it used to declare is gone. An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."}},"events": {@@@ -753,7 +763,8 @@"databases": [{"name": "events",-- "path": "/var/lib/openvibe-events/events.db"++ "engine": "postgresql",++ "database": "ov_events"}],"nginx": {@@@ -768,12 +779,16 @@"drill": {"port": 14300,"databases": {-- "events": "EVENTS_DB_PATH"++ "events": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}","HOST": "127.0.0.1",-- "EVENTS_WORKER": "off"++ "EVENTS_WORKER": "off",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"counts": [{@@@ -921,7 +936,7 @@}},"host": {-- "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its database and content-addressed objects live in /var/lib/openvibe-host-api, which is NOT ovhost's own stateDir (/var/lib/openvibe-host). Tenant vhosts: `ovhost nginx tenants host`.",++ "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api, which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.","repo": "/opt/openvibe.host","units": ["openvibe-host.service"@@@ -953,7 +968,8 @@"databases": [{"name": "host",-- "path": "/var/lib/openvibe-host-api/host.db"++ "engine": "postgresql",++ "database": "ov_host"}],"nginx": {@@@ -961,7 +977,10 @@"sitesDomain": "openvibe.host","dashboardDomain": "openvibe.host","wildcardCert": "openvibe.host",-- "database": "/var/lib/openvibe-host-api/host.db",++ "database": {++ "engine": "postgresql",++ "database": "ov_host"++ },"maxUpload": "110m","replaces": ["openvibe.host-tenants-pending.conf"@@@ -971,7 +990,10 @@"drill": {"port": 14910,"databases": {-- "host": "HOST_DB_PATH"++ "host": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -979,7 +1001,8 @@"HOST_WORKER": "off","EVENTS_URL": "","OV_OAUTH_CLIENT_SECRET": "",-- "HOST_STORAGE_DIR": "{tmp}/objects"++ "HOST_STORAGE_DIR": "{tmp}/objects",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"dirs": ["{tmp}/objects"@@@ -1002,11 +1025,11 @@"table": "host_domains"}],-- "_notes": "Restores host.db only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."++ "_notes": "Restores the ov_host database only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."}},"billing": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.billing","units": ["openvibe-billing.service"@@@ -1031,20 +1054,25 @@"databases": [{"name": "billing",-- "path": "/var/lib/openvibe-billing/billing.db"++ "engine": "postgresql",++ "database": "ov_billing"}],"drill": {"port": 14600,"databases": {-- "billing": "BILLING_DB_PATH"++ "billing": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}","HOST": "127.0.0.1","BILLING_JOBS": "off","EVENTS_URL": "",-- "OV_OAUTH_CLIENT_SECRET": ""++ "OV_OAUTH_CLIENT_SECRET": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/rates"@@@ -1156,7 +1184,7 @@}},"search": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.search","units": ["openvibe-search.service"@@@ -1181,20 +1209,25 @@"databases": [{"name": "search",-- "path": "/var/lib/openvibe-search/search.db"++ "engine": "postgresql",++ "database": "ov_search"}],"drill": {"port": 14710,"databases": {-- "search": "SEARCH_DB_PATH"++ "search": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}","HOST": "127.0.0.1","EVENTS_URL": "","SEARCH_EVENTS_SECRET": "",-- "OV_OAUTH_CLIENT_SECRET": ""++ "OV_OAUTH_CLIENT_SECRET": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/search?q=openvibe&limit=10",@@@ -1218,7 +1251,7 @@}},"sources": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.sources","units": ["openvibe-sources.service"@@@ -1243,20 +1276,25 @@"databases": [{"name": "sources",-- "path": "/var/lib/openvibe-sources/sources.db"++ "engine": "postgresql",++ "database": "ov_sources"}],"drill": {"port": 14720,"databases": {-- "sources": "SOURCES_DB_PATH"++ "sources": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}","HOST": "127.0.0.1","SOURCES_WORKER": "off","EVENTS_URL": "",-- "OV_OAUTH_CLIENT_SECRET": ""++ "OV_OAUTH_CLIENT_SECRET": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"counts": [{@@@ -1280,7 +1318,7 @@}},"wiki": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.wiki","units": ["openvibe-wiki.service"@@@ -1305,13 +1343,17 @@"databases": [{"name": "wiki",-- "path": "/var/lib/openvibe-wiki/wiki.db"++ "engine": "postgresql",++ "database": "ov_wiki"}],"drill": {"port": 14800,"databases": {-- "wiki": "WIKI_DB_PATH"++ "wiki": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1320,7 +1362,8 @@"OV_OAUTH_CLIENT_SECRET": "","OV_MEDIA_INTERNAL_URL": "","OV_SOURCES_INTERNAL_URL": "",-- "WIKI_SCHEDULE_INTERVAL_MS": "2147483647"++ "WIKI_SCHEDULE_INTERVAL_MS": "2147483647",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/spaces",@@@ -1344,7 +1387,7 @@}},"blog": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.blog","units": ["openvibe-blog.service"@@@ -1369,13 +1412,17 @@"databases": [{"name": "blog",-- "path": "/var/lib/openvibe-blog/blog.db"++ "engine": "postgresql",++ "database": "ov_blog"}],"drill": {"port": 14810,"databases": {-- "blog": "BLOG_DB_PATH"++ "blog": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1384,7 +1431,8 @@"EVENTS_URL": "","OV_OAUTH_CLIENT_SECRET": "","OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9",-- "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9"++ "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/blogs/openvibe/posts?limit=10",@@@ -1408,7 +1456,7 @@}},"tips": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.tips","units": ["openvibe-tips.service"@@@ -1433,13 +1481,17 @@"databases": [{"name": "tips",-- "path": "/var/lib/openvibe-tips/tips.db"++ "engine": "postgresql",++ "database": "ov_tips"}],"drill": {"port": 14610,"databases": {-- "tips": "TIPS_DB_PATH"++ "tips": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1450,7 +1502,8 @@"TIPS_CHAT_ADAPTER": "none","OV_OAUTH_CLIENT_SECRET": "","BILLING_URL": "http://127.0.0.1:9",-- "LIVE_INTERNAL_URL": "http://127.0.0.1:9"++ "LIVE_INTERNAL_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/sitemap.xml"@@@ -1473,7 +1526,7 @@}},"vip": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.vip","units": ["openvibe-vip.service"@@@ -1498,13 +1551,17 @@"databases": [{"name": "vip",-- "path": "/var/lib/openvibe-vip/vip.db"++ "engine": "postgresql",++ "database": "ov_vip"}],"drill": {"port": 14620,"databases": {-- "vip": "VIP_DB_PATH"++ "vip": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1514,7 +1571,8 @@"VIP_EVENTS_SECRET": "","OV_OAUTH_CLIENT_SECRET": "","BILLING_URL": "http://127.0.0.1:9",-- "VIP_CHECKOUT_PROVIDERS": ""++ "VIP_CHECKOUT_PROVIDERS": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/creators/network",@@@ -1539,7 +1597,7 @@}},"news": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.news","units": ["openvibe-news.service"@@@ -1564,13 +1622,17 @@"databases": [{"name": "news",-- "path": "/var/lib/openvibe-news/news.db"++ "engine": "postgresql",++ "database": "ov_news"}],"drill": {"port": 14820,"databases": {-- "news": "NEWS_DB_PATH"++ "news": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1580,7 +1642,8 @@"NEWS_EVENTS_SECRET": "","OV_AI_INTERNAL_URL": "","OV_OAUTH_CLIENT_SECRET": "",-- "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9"++ "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/stories?limit=10",@@@ -1603,7 +1666,7 @@}},"reviews": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.reviews","units": ["openvibe-reviews.service"@@@ -1628,13 +1691,17 @@"databases": [{"name": "reviews",-- "path": "/var/lib/openvibe-reviews/reviews.db"++ "engine": "postgresql",++ "database": "ov_reviews"}],"drill": {"port": 14830,"databases": {-- "reviews": "REVIEWS_DB_PATH"++ "reviews": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1643,7 +1710,8 @@"EVENTS_URL": "","REVIEWS_EVENTS_SECRET": "","OV_OAUTH_CLIENT_SECRET": "",-- "OV_SOURCES_INTERNAL_URL": ""++ "OV_SOURCES_INTERNAL_URL": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/entities?limit=10",@@@ -1667,7 +1735,7 @@}},"deals": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.deals","units": ["openvibe-deals.service"@@@ -1692,13 +1760,17 @@"databases": [{"name": "deals",-- "path": "/var/lib/openvibe-deals/deals.db"++ "engine": "postgresql",++ "database": "ov_deals"}],"drill": {"port": 14840,"databases": {-- "deals": "DEALS_DB_PATH"++ "deals": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1709,7 +1781,8 @@"DEALS_EVENTS_SECRET": "","OV_OAUTH_CLIENT_SECRET": "","OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9",-- "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9"++ "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": [{@@@ -1737,7 +1810,7 @@}},"coupons": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.coupons","units": ["openvibe-coupons.service"@@@ -1762,13 +1835,17 @@"databases": [{"name": "coupons",-- "path": "/var/lib/openvibe-coupons/coupons.db"++ "engine": "postgresql",++ "database": "ov_coupons"}],"drill": {"port": 14850,"databases": {-- "coupons": "COUPONS_DB_PATH"++ "coupons": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1776,7 +1853,8 @@"COUPONS_WORKER": "off","EVENTS_URL": "","OV_OAUTH_CLIENT_SECRET": "",-- "OV_SOURCES_INTERNAL_URL": ""++ "OV_SOURCES_INTERNAL_URL": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/sitemaps/merchants.xml",@@@ -1799,7 +1877,7 @@}},"trade": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.trade","units": ["openvibe-trade.service"@@@ -1824,13 +1902,17 @@"databases": [{"name": "trade",-- "path": "/var/lib/openvibe-trade/trade.db"++ "engine": "postgresql",++ "database": "ov_trade"}],"drill": {"port": 14860,"databases": {-- "trade": "TRADE_DB_PATH"++ "trade": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1839,7 +1921,8 @@"EVENTS_URL": "","TRADE_EVENTS_WEBHOOK_SECRET": "","OV_OAUTH_CLIENT_SECRET": "",-- "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9"++ "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/v1/instruments?limit=10",@@@ -1862,7 +1945,7 @@}},"codes": {-- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.","repo": "/opt/openvibe.codes","units": ["openvibe-codes.service"@@@ -1887,19 +1970,24 @@"databases": [{"name": "codes",-- "path": "/var/lib/openvibe-codes/codes.db"++ "engine": "postgresql",++ "database": "ov_codes"}],"drill": {"port": 14900,"databases": {-- "codes": "CODES_DB_PATH"++ "codes": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}","HOST": "127.0.0.1","EVENTS_URL": "",-- "OV_OAUTH_CLIENT_SECRET": ""++ "OV_OAUTH_CLIENT_SECRET": "",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/sitemap.xml"@@@ -1946,14 +2034,18 @@"databases": [{"name": "ai",-- "path": "/var/lib/openvibe-ai/ai.db"++ "engine": "postgresql",++ "database": "ov_ai"}],"drill": {"_notes": "Every provider key, base URL and the HTTP seam are blanked and AI_ENABLED=0, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and EVENTS_URL are blanked (no service calls, no events); WHISPER_BIN is blanked (no transcription). Boot marks interrupted runs failed in the restored copy only. Counts cover configuration and citations only: runs, requests and usage grow by the minute, so they are checked by integrity_check, not by an equal count. Passed on the host 2026-09-24 05:40 UTC.","port": 14700,"databases": {-- "ai": "AI_DB_PATH"++ "ai": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -1971,7 +2063,8 @@"WHISPER_BIN": "","OV_OAUTH_CLIENT_SECRET": "","EVENTS_URL": "",-- "AI_STUB_FALLBACK": "1"++ "AI_STUB_FALLBACK": "1",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/health"@@@ -2046,7 +2139,8 @@"databases": [{"name": "openre",-- "path": "/var/lib/openre/openre.db"++ "engine": "postgresql",++ "database": "ov_openre"}],"drill": {@@@ -2056,7 +2150,10 @@"server/index.js"],"databases": {-- "openre": "OPENRE_DB_PATH"++ "openre": {++ "url": "DATABASE_URL",++ "directUrl": "DATABASE_DIRECT_URL"++ }},"env": {"PORT": "{port}",@@@ -2066,7 +2163,8 @@"OV_OAUTH_CLIENT_SECRET": "","MEDIA_URL": "http://127.0.0.1:9","MEDIA_API_KEY": "",-- "OPENRE_RECORDING": "off"++ "OPENRE_RECORDING": "off",++ "VALKEY_URL": "redis://127.0.0.1:9/0"},"compare": ["/api/health",@@@ -2093,8 +2191,8 @@"_notes": "Only openre-api starts (command; the working directory and the non-secret Environment= come from openre-api.service). It never runs a transport, a coordinator or the event relay in any mode. OPENRE_DRILL=1 (OpenRe 5e86ea7 and later) also makes it read-only: writes, /play/ (which would pull from production's worker on loopback) and sign-in (which would redeem codes at Network) answer 503. The coordinator and the workers refuse to start. The event relay and Media calls are off. EVENTS_URL and OV_OAUTH_CLIENT_SECRET empty and MEDIA_URL on the closed port are the same switches for releases before 5e86ea7. Left on (read-only): the Network JWKS GET on loopback. Not compared: ingest_sessions and workers (they change whenever someone streams or a generation rolls over) and /api/ready (it reports live worker heartbeats)."},"protected": {-- "kind": "sqlite-count",-- "db": "/var/lib/openre/openre.db",++ "kind": "postgresql-count",++ "database": "ov_openre","sql": "SELECT count(*) AS n FROM ingest_sessions WHERE state IN ('starting', 'live', 'ending')","label": "ingest sessions"},@@@ -42,7 +42,10 @@ async function refusal(exec, inv, file) {if (typeof file !== 'string' || !path.isAbsolute(file) || path.normalize(file) !== file) return 'not a normalised absolute path';if (!ALLOWED_ROOTS.some((r) => file.startsWith(r))) return `outside ${ALLOWED_ROOTS.join(', ')}`;if (!NAME_RE.test(objectName(file))) return 'the path has characters an object name cannot carry';-- const declared = Object.values(inv.services || {}).flatMap((s) => (s.databases || []).map((d) => d.path || d));++ // A live SQLite database is a file the inventory declares, and archiving it is refused. A++ // PostgreSQL database has no file (its own entry carries no path), so the C-89 switch — archive++ // the frozen SQLite file, then delete it — is no longer blocked by the inventory that moved on.++ const declared = Object.values(inv.services || {}).flatMap((s) => (s.databases || []).filter((d) => d.engine !== 'postgresql').map((d) => d.path));if (declared.includes(file)) return 'a database the inventory declares (a live database)';const st = await exec.stat(file);if (!st) return 'no such file';@@@ -59,11 +59,13 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]for the VERIFIED custom domains (read from the service'sdatabase); lists domains still waiting for a certificatesnapshot <service> [--out <file>] config tarball: units, vhost, env NAMES, sha — no secret values-- backup <service> sqlite .backup of the declared databases to a dated directory-- backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]++ backup <service> [--logical] .backup of the declared databases to a dated directory;++ --logical also takes a PostgreSQL pg_dump (else weekly)++ backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>] [--logical]every service with databases; one failure never stops theothers; keeps the last 7 daily + 4 weekly good backups per-- service; JSON summary in <stateDir>/backup-runs/<run>.json;++ service; --logical also dumps every PostgreSQL database;++ JSON summary in <stateDir>/backup-runs/<run>.json;--offsite then encrypts and uploads the run (root)backup-metrics [--run <run>] rewrite the Prometheus textfile (node_exporter collector) from arecorded backup --all run (default: the latest)@@@ -398,13 +400,13 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procconst cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });return lib.offsite.push(ctx, summary, { cfg, client: s3(cfg) });} : null;-- const r = await lib.backupAll(ctx, { daily: count(opts, 'keep-daily', 7), weekly: count(opts, 'keep-weekly', 4), prune: !opts['no-prune'], offsite: offsiteRun });++ const r = await lib.backupAll(ctx, { daily: count(opts, 'keep-daily', 7), weekly: count(opts, 'keep-weekly', 4), prune: !opts['no-prune'], offsite: offsiteRun, logical: !!opts.logical });if (json) emit(r);else {for (const s of r.services) {const copied = s.files.filter((f) => f.dest && !f.error);const size = copied.reduce((a, f) => a + (f.bytes || 0), 0);-- const detail = s.status === 'failed' ? s.error : s.status === 'skipped' ? 'no database found' : `${copied.length} file(s), ${bytes(size)}`;++ const detail = s.status === 'failed' ? s.error : s.status === 'skipped' ? 'no database found' : s.status === 'verified' ? 'pgBackRest verified (no dump due)' : `${copied.length} file(s), ${bytes(size)}`;out(`${s.status.toUpperCase().padEnd(8)} ${s.service.padEnd(10)} ${detail}${s.pruned.length ? `; pruned ${s.pruned.length}` : ''}`);}if (r.offsite) {@@@ -416,8 +418,14 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc}return r.ok ? 0 : 2;}-- const r = await lib.backup(ctx, needService());-- if (json) emit(r); else for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);++ const r = await lib.backup(ctx, needService(), { logical: !!opts.logical });++ if (json) emit(r);++ else {++ for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);++ // A PostgreSQL run with no dump due writes no file: say what it did instead, so a++ // silent command is never mistaken for one that did nothing.++ if (!r.files.length && r.verified) out(`pgBackRest verified: newest ${r.verified.type} backup stopped ${r.verified.stop}, WAL archiver ok: ${r.verified.walOk}; no logical dump due (--logical forces one)`);++ }return 0;}case 'backup-metrics': {@@@ -1,19 +1,34 @@'use strict';/**-- * ovhost backup <service>: an online SQLite `.backup` of every database the inventory declares for-- * the service, into <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.db. The copy is made by a worker-- * running as the service user (root never opens a service database — see sqlite-worker.js), checked-- * with PRAGMA quick_check, and recorded in <stateDir>/backups/<service>.jsonl.++ * ovhost backup <service>: an online copy of every database the inventory declares for the service,++ * into <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in++ * <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root++ * never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.++ * PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups++ * and its WAL (roles/data/provision.sh), so a run that finds a PostgreSQL database first VERIFIES++ * that layer (dbengine.verifyPgBackRest: a stanza that is ok and whose newest backup is under 26 h++ * old; dbengine.walFailedCount: pg_stat_archiver.failed_count has not risen since the last run) and++ * only then, on a slower cadence (LOGICAL_DUMP_INTERVAL_DAYS, or --logical), takes a per-service++ * pg_dump -Fc archive — the artifact the off-site code uploads and a drill can restore.*-- * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600,-- * with no -wal/-shm beside it. The worker writes into <backupStagingDir>/<service>-<stamp> (its own,-- * 0700, under a root 0711 parent); root then takes that directory over, checks each copy is a plain-- * file with one link, chowns and chmods it, and renames the directory into place.++ * A PostgreSQL service's record always carries `verified` (type + stop + walOk when it passed, an++ * error when it did not), and a run with nothing to dump is verification-only: ok: true, files: [],++ * and no directory at all. A failed verification (or a logical dump that failed) is recorded++ * (ok: false) and then throws a BackupError carrying the record, so backup --all reports the service++ * failed — never skipped.++ *++ * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600, with++ * no -wal/-shm beside it. SQLite's worker writes into <backupStagingDir>/<service>-<stamp> (its own,++ * 0700, under a root 0711 parent) as the service user. A service with any PostgreSQL database gets a++ * staging directory owned by postgres instead, because pg_dump runs as that OS user. In both cases++ * root then takes that directory over (lockDown, root:root 0700) BEFORE anything inside it is++ * checked — the service user (or postgres) can no longer reach it while root validates the copies,++ * chowns and chmods them and renames the directory into place, which is what keeps the property.** Every declared database is attempted even when an earlier one fails. A backup with any failed* database is still recorded (ok: false, so a drill and retention never treat it as good) and then-- * throws a BackupError carrying the record. A backup that found none of its databases is recorded-- * as ok: false, empty: true and leaves no directory behind.++ * throws a BackupError carrying the record. A SQLite backup that found none of its databases is++ * recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.** ovhost backup --all (backupAll): every service that declares databases, one after another. A* failing service never stops the others. Afterwards each service's backups are pruned to the last@@@ -22,9 +37,16 @@*/const path = require('path');const { service } = require('../inventory');++const dbengine = require('../dbengine');const lock = require('../lock');const retention = require('../retention');++const DAY_MS = 86400000;++// A PostgreSQL database's own logical dump is on a slower cadence than pgBackRest's nightly++// differentials: it exists for the off-site copy and per-service drills, not for durability.++const LOGICAL_DUMP_INTERVAL_DAYS = 7;++const ARCHIVER_STATE = 'pgbackrest-archiver.json';++class BackupError extends Error {constructor(message, record) { super(message); this.record = record; }}@@@ -47,15 +69,105 @@ async function privateDir(exec, p) {await lockDown(exec, p, 0o700);}--async function backup(ctx, id, { reason = 'manual' } = {}) {++/** The last pg_stat_archiver.failed_count this host recorded, or null on the first run. */++async function readArchiver(exec, inv) {++ const text = await exec.readFile(path.join(inv.stateDir, ARCHIVER_STATE));++ if (text == null) return null;++ try {++ const v = JSON.parse(text);++ return v && typeof v.failedCount === 'number' && Number.isFinite(v.failedCount) ? v.failedCount : null;++ } catch { return null; }++}++++/** Remember this run's failed_count (mode 0640: nobody but root needs it) for the next run's compare. */++async function writeArchiver(exec, inv, failedCount) {++ await exec.mkdir(inv.stateDir, { mode: 0o750 });++ await exec.writeFile(path.join(inv.stateDir, ARCHIVER_STATE), `${JSON.stringify({ failedCount, at: new Date(exec.now()).toISOString() })}\n`, { mode: 0o640 });++}++++/**++ * The every-run pgBackRest verification for a service with PostgreSQL databases. Nothing is copied++ * until this passes: a stale stanza, a bad status or a risen WAL archiver failure count all mean the++ * cluster's durability layer is not healthy, and that is what the record must say.++ * -> { type, stop, walOk, ok: true } or { ok: false, error }++ */++async function verifyPostgres(ctx) {++ const { exec, inv, log } = ctx;++ try {++ const v = await dbengine.verifyPgBackRest(exec, { now: exec.now() });++ const failed = await dbengine.walFailedCount(exec);++ const previous = await readArchiver(exec, inv);++ const walOk = previous === null || failed <= previous;++ // Record this run's value even when it rose: the next run compares against the latest, so a++ // single archiving blip fails one night rather than every night thereafter.++ await writeArchiver(exec, inv, failed);++ if (!walOk) {++ const error = `the WAL archiver failed_count rose from ${previous} to ${failed}`;++ log(`PostgreSQL verification FAILED: ${error}`);++ return { type: v.type, stop: v.stop, walOk, ok: false, error };++ }++ log(`pgBackRest ok: newest ${v.type} backup stopped ${v.stop}; WAL archiver failed_count ${failed}`);++ return { type: v.type, stop: v.stop, walOk, ok: true };++ } catch (err) {++ log(`PostgreSQL verification FAILED: ${err.message}`);++ return { ok: false, error: err.message };++ }++}++++/** When this service last recorded a GOOD logical dump (`at` ISO), or null: the weekly cadence input. */++async function lastGoodDumpAt(exec, inv, id) {++ const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`));++ const lines = String(text || '').split('\n').filter(Boolean);++ for (let i = lines.length - 1; i >= 0; i--) {++ let r;++ try { r = JSON.parse(lines[i]); } catch { continue; }++ if (!r || r.ok === false) continue;++ if ((r.files || []).some((f) => typeof f.file === 'string' && f.file.endsWith('.dump'))) return r.at || null;++ }++ return null;++}++++async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {const { exec, inv, log } = ctx;const svc = service(inv, id);if (!svc.databases.length) throw new Error(`${id} declares no databases`);++ const postgres = svc.databases.filter((d) => d.engine === 'postgresql');++ const hasPostgres = postgres.length > 0;++ const engine = hasPostgres ? 'postgresql' : 'sqlite';++++ // (a) Verify the cluster's own backup layer BEFORE copying anything. A failed verification is++ // itself a record (ok: false): backupAll then marks the service failed, never skipped.++ let verified = null;++ if (hasPostgres) {++ verified = await verifyPostgres(ctx);++ if (!verified.ok) {++ const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: false, files: [], verified };++ await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);++ throw new BackupError(`PostgreSQL verification failed: ${verified.error}`, record);++ }++ }++++ // (b) A logical dump only when the last good one is older than LOGICAL_DUMP_INTERVAL_DAYS, or the++ // caller passed --logical: pgBackRest already holds a nightly differential, so this exists for the++ // off-site copy and per-service drills.++ const dump = [];++ if (hasPostgres) {++ const last = await lastGoodDumpAt(exec, inv, id);++ const at = last === null ? NaN : Date.parse(last);++ if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);++ }++ // A verification-only run (PostgreSQL, no dump due, no SQLite database) leaves nothing on disk.++ if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite')) {++ const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };++ await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);++ return record;++ }++const st = stamp(exec.now());const dir = path.join(inv.backupDir, id, st);if (await exec.stat(dir)) throw new Error(`${dir} already exists; a backup never overwrites another`);// Backups hold user data: every directory under backupDir is root:root 0700 and every copy is-- // root:root 0600. The worker must run as the service user (root never opens a service database),++ // root:root 0600. SQLite's worker runs as the service user (root never opens a service database),// so it writes into a staging directory of its own, which root then takes over before the move.await privateDir(exec, inv.backupDir);await privateDir(exec, path.join(inv.backupDir, id));@@@ -63,27 +175,44 @@ async function backup(ctx, id, { reason = 'manual' } = {}) {await lockDown(exec, inv.backupStagingDir, 0o711);const stage = path.join(inv.backupStagingDir, `${id}-${st}`);if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });-- await exec.mkdir(stage, { owner: svc.runAs, mode: 0o700 });++ // pg_dump runs as the postgres OS user, so a service with any PostgreSQL database needs a staging++ // directory postgres can write into (SQLite keeps the service user). The lockDown(stage, 0o700)++ // below takes the whole directory back to root:root BEFORE anything inside it is checked, so++ // neither the service user nor postgres can reach it while root validates and moves it.++ await exec.mkdir(stage, { owner: hasPostgres ? dbengine.PG_USER : svc.runAs, mode: 0o700 });const files = [];try {for (const db of svc.databases) {-- if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }-- // Named by the inventory name: two databases may share a file name (community.db twice).-- const file = `${db.name}${path.extname(db.path) || '.db'}`;-- log(`backing up ${db.path} → ${path.join(dir, file)} (as ${svc.runAs})`);++ const file = dbengine.fileName(db);++ if (db.engine === 'sqlite') {++ if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }++ log(`backing up ${db.path} → ${path.join(dir, file)} (as ${svc.runAs})`);++ const t0 = exec.now();++ try {++ // The worker runs PRAGMA quick_check on the copy and fails unless it answers ok.++ await exec.sqliteBackup(db.path, path.join(stage, file), { as: svc.runAs });++ } catch (err) {++ files.push({ name: db.name, source: db.path, dest: path.join(dir, file), error: err.message });++ log(`FAILED ${db.path}: ${err.message}`);++ continue;++ }++ files.push({ name: db.name, source: db.path, dest: path.join(dir, file), file, check: 'quick_check ok', seconds: Math.round((exec.now() - t0) / 1000) });++ continue;++ }++ if (!dump.includes(db)) { log(`${db.database} is covered by pgBackRest today; no logical dump due`); continue; }const t0 = exec.now();try {-- // The worker runs PRAGMA quick_check on the copy and fails unless it answers ok.-- await exec.sqliteBackup(db.path, path.join(stage, file), { as: svc.runAs });++ // dbengine.dump runs pg_dump -w -Fc -f <stage>/<name>.dump as postgres and logs it.++ await dbengine.dump(exec, db, path.join(stage, file), { log });} catch (err) {-- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), error: err.message });-- log(`FAILED ${db.path}: ${err.message}`);++ files.push({ name: db.name, source: db.database, dest: path.join(dir, file), error: err.message });++ log(`FAILED ${db.database}: ${err.message}`);continue;}-- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), file, check: 'quick_check ok', seconds: Math.round((exec.now() - t0) / 1000) });++ files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });}-- // Take the staging directory back first: after this the service user can no longer reach-- // anything inside it, so what is checked below cannot change underneath.++ // Take the staging directory back first: after this the service user (or postgres) can no++ // longer reach anything inside it, so what is checked below cannot change underneath.await lockDown(exec, stage, 0o700);for (const f of files.filter((x) => x.file && !x.error)) {const p = path.join(stage, f.file);@@@ -106,13 +235,14 @@ async function backup(ctx, id, { reason = 'manual' } = {}) {} finally {if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });}-- for (const f of files) delete f.file;const failed = files.filter((f) => f.error);const copied = files.filter((f) => f.dest && !f.error);-- const record = { service: id, at: new Date(exec.now()).toISOString(), reason, dir, ok: !failed.length && copied.length > 0, files };-- // Nothing existed to back up (a service not deployed yet): no directory is left, so an empty-- // backup can never stand in for a real one in retention or a drill.-- if (!copied.length && !failed.length) record.empty = true;++ // `file` is kept on each entry: the weekly cadence reads it back to find the last good dump.++ const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: !failed.length && copied.length > 0, dir, files };++ if (verified) record.verified = verified;++ // Nothing existed to back up (a SQLite service not deployed yet): no directory is left, so an++ // empty backup can never stand in for a real one in retention or a drill. PostgreSQL never empty.++ if (!copied.length && !failed.length && !hasPostgres) record.empty = true;await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);if (failed.length) throw new BackupError(failed.map((f) => `${f.name}: ${f.error}`).join('; '), record);return record;@@@ -160,7 +290,7 @@ async function prune(ctx, id, { daily = 7, weekly = 4 } = {}) {return { kept: dirs.filter((d) => !remove.includes(d)), removed: remove, unrecorded };}--async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, offsite = null } = {}) {++async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, offsite = null, logical = false } = {}) {const { exec, inv, log } = ctx;const held = await lock.acquire(exec, inv, '_backup-all');const started = exec.now();@@@ -173,14 +303,22 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, ofconst entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };summary.services.push(entry);try {-- const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}` });-- entry.dir = rec.empty ? null : rec.dir;++ const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}`, logical });++ entry.dir = rec.empty ? null : (rec.dir || null);entry.files = rec.files;++ if (rec.verified) entry.verified = rec.verified;++ // A PostgreSQL service with nothing to dump this run was still verified, not backed++ // up: it has its own status so the summary distinguishes the two.if (rec.empty) entry.status = 'skipped';++ else if (rec.engine === 'postgresql' && !rec.dir) entry.status = 'verified';} catch (err) {entry.status = 'failed';entry.error = err.message;-- if (err.record) { entry.dir = err.record.dir; entry.files = err.record.files; }++ if (err.record) {++ entry.dir = err.record.dir || null;++ entry.files = err.record.files;++ if (err.record.verified) entry.verified = err.record.verified;++ }log(`✗ ${svc.id}: ${err.message}`);}if (doPrune) {@@@ -220,7 +358,9 @@ const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';* openvibe_backup_last_run_timestamp_seconds when the last run finished* openvibe_backup_last_run_ok 1 when every service (and the off-site copy) succeeded* openvibe_backup_last_success_timestamp_seconds the last run that succeeded (read back from backup-runs)-- * openvibe_backup_services{status} services ok, failed and skipped in the last run++ * openvibe_backup_services{status} services ok, failed, skipped and verified in the last run++ * openvibe_backup_pg_verified{service} 1 when that service's PostgreSQL verification passed++ * openvibe_pgbackrest_last_backup_age_seconds age of the newest pgBackRest backup, at write time* openvibe_backup_offsite_ok 1/0 for the last run's off-site copy, when it made one** Written only where the collector's directory exists (inventory `textfileDir`, default the Debian@@@ -241,6 +381,7 @@ async function writeMetrics(ctx, summary) {}}const count = (st) => summary.services.filter((x) => x.status === st).length;++ const verified = summary.services.filter((x) => x.verified);const lines = ['# HELP openvibe_backup_last_run_timestamp_seconds When the last ovhost backup --all run finished.','# TYPE openvibe_backup_last_run_timestamp_seconds gauge',@@@ -250,8 +391,22 @@ async function writeMetrics(ctx, summary) {`openvibe_backup_last_run_ok ${summary.ok ? 1 : 0}`,'# HELP openvibe_backup_services Services by outcome in the last run.','# TYPE openvibe_backup_services gauge',-- ...['ok', 'failed', 'skipped'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),++ ...['ok', 'failed', 'skipped', 'verified'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),];++ if (verified.length) {++ lines.push(++ '# HELP openvibe_backup_pg_verified 1 when a service\'s PostgreSQL backup verification passed in the last run.',++ '# TYPE openvibe_backup_pg_verified gauge',++ ...verified.map((s) => `openvibe_backup_pg_verified{service="${s.service}"} ${s.verified.ok ? 1 : 0}`),++ );++ // The newest verified stop's age at this run (a snapshot: it does not advance between runs; a run++ // that stops happening is caught by OpenVibeBackupStale).++ const stops = verified.filter((s) => s.verified.stop).map((s) => Date.parse(s.verified.stop)).filter(Number.isFinite);++ if (stops.length) {++ const age = Math.max(0, Math.floor((exec.now() - Math.max(...stops)) / 1000));++ lines.push('# HELP openvibe_pgbackrest_last_backup_age_seconds Age of the newest pgBackRest backup at the last run.', '# TYPE openvibe_pgbackrest_last_backup_age_seconds gauge', `openvibe_pgbackrest_last_backup_age_seconds ${age}`);++ }++ }if (lastOk) lines.push('# HELP openvibe_backup_last_success_timestamp_seconds When the last fully successful run finished.', '# TYPE openvibe_backup_last_success_timestamp_seconds gauge', `openvibe_backup_last_success_timestamp_seconds ${at(lastOk)}`);if (summary.offsite) lines.push('# HELP openvibe_backup_offsite_ok 1 when the last run\'s off-site copy succeeded.', '# TYPE openvibe_backup_offsite_ok gauge', `openvibe_backup_offsite_ok ${summary.offsite.ok ? 1 : 0}`);await exec.writeFile(path.join(dir, 'openvibe_backup.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });@@@ -4,36 +4,50 @@* the service from a backup next to production and checks that the rebuilt copy answers like* production does:*-- * 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Copy each database the-- * drill block names into a fresh directory under <drillDir> owned by the service user, and-- * run PRAGMA integrity_check on each copy. The result must be `ok`.++ * 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Every database the drill++ * block names is restored, per engine (lib/dbengine.js):++ * SQLite copy the file into a fresh directory under <drillDir> owned by the service user++ * and run PRAGMA integrity_check on the copy. The result must be `ok`.++ * PostgreSQL check free space first, copy the pg_dump -Fc archive into <tmp>/pg owned by++ * postgres, verify it with `pg_restore --list`, then create a scratch database and++ * a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the++ * archive into the scratch database. Both are dropped again in a finally.* 2. Start a second instance from the production checkout, as the service user, through* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production* env file, and a second env file with the drill's overrides: its port, the restored database-- * paths, and the switches that turn side effects off. The instance runs sandboxed:-- * ProtectSystem=strict with only the drill directory writable, a private /tmp, binding only its-- * drill port, and no addresses beyond loopback unless the inventory says otherwise.-- * RuntimeMaxSec stops it even if ovhost dies. Loopback stays open (the instance needs Network's-- * public keys), so the inventory's overrides are what keep it away from production services.++ * paths (or the scratch database's URLs), and the switches that turn side effects off. The++ * instance runs sandboxed: ProtectSystem=strict with only the drill directory writable, a++ * private /tmp, binding only its drill port, and no addresses beyond loopback unless the++ * inventory says otherwise. RuntimeMaxSec stops it even if ovhost dies. Loopback stays open++ * (the instance needs Network's public keys), so the inventory's overrides are what keep it away++ * from production services.* 3. Wait for its readiness path. Then compare the declared read-only GETs with production, byte* for byte or as JSON without the listed volatile keys, and compare declared row counts-- * (production against the restored copy).-- * 4. Stop the instance by its own MainPID (never pkill -f), remove the drill directory unless-- * --keep, append the result to <stateDir>/drills/<service>.jsonl, and return a Markdown row-- * for docs/restore-drills.md.++ * (production against the restored copy, allowing countsTolerance rows).++ * 4. Stop the instance by its own MainPID (never pkill -f), drop the scratch database and role,++ * remove the drill directory unless --keep, append the result to <stateDir>/drills/<service>.jsonl,++ * and return a Markdown row for docs/restore-drills.md.** ovhost itself writes only inside the drill directory, the lock file and the drill log. It never* writes to the checkout, the env file, the database or the backup. It never reads an env value:* systemd loads the production env file for the instance, and the override file holds only* inventory values. Environment= values from the unit are passed on only if their names do not look-- * secret and the drill does not set them itself (OpenRe's unit sets OPENRE_DB_PATH to production's++ * secret and the drill does not set them itself (OpenRe's unit sets DATABASE_URL to production's* database: that line never reaches the drill instance).++ *++ * The production env file is loaded FIRST, so any variable the drill does not set keeps its++ * production value. For a PostgreSQL service the drill therefore refuses to start unless the drill++ * block overrides all three of the database URL, the direct database URL and VALKEY_URL: an++ * un-overridden direct URL would migrate production, an un-overridden URL would serve production++ * data, and an un-neutralised Valkey URL would write into production's keyspace (the sandbox still++ * allows loopback).*/const path = require('path');const { service } = require('../inventory');const envfile = require('../envfile');const systemd = require('../systemd');const lock = require('../lock');++const dbengine = require('../dbengine');const { stamp } = require('./backup');class DrillError extends Error {@@@ -135,19 +149,26 @@ async function countRows(exec, db, table, as) {// ── inputs ───────────────────────────────────────────────────────────────────--/** The backup to restore: --backup <dir>, else the last record in <stateDir>/backups/<id>.jsonl. */++/** The backup to restore: --backup <dir>, else the newest usable record in <stateDir>/backups/<id>.jsonl. */async function resolveBackup(exec, inv, svc, dirArg) {let dir = dirArg;let takenAt = null;++ const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));++ const pgTargets = restored.filter((d) => d.engine === 'postgresql');if (dir) {if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');dir = path.normalize(dir);} else {const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${svc.id}.jsonl`));-- // A failed or empty backup (ok: false) is never restored.++ // A failed or empty backup (ok: false) is never restored. PostgreSQL records a verification-++ // only run too (ok: true, no dir, files: []), which holds no artifact to restore: pick the++ // newest record that carries a .dump entry for every PostgreSQL database, not merely the last.const records = String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir && r.ok !== false);-- if (!records.length) throw new DrillError(`no good ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);-- const last = records[records.length - 1];++ const usable = pgTargets.length++ ? records.filter((r) => pgTargets.every((d) => (r.files || []).some((f) => f.file === dbengine.fileName(d))))++ : records;++ if (!usable.length) throw new DrillError(`no good ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);++ const last = usable[usable.length - 1];dir = last.dir;takenAt = last.at || null;}@@@ -156,11 +177,12 @@ async function resolveBackup(exec, inv, svc, dirArg) {const files = [];for (const name of Object.keys(svc.drill.databases)) {const db = svc.databases.find((d) => d.name === name);-- const file = path.join(dir, `${name}${path.extname(db.path) || '.db'}`);++ const target = svc.drill.databases[name];++ const file = path.join(dir, db.engine === 'postgresql' ? dbengine.fileName(db) : `${name}${path.extname(db.path) || '.db'}`);const fst = await exec.stat(file);if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);-- const target = svc.drill.databases[name];-- files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db' });++ if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });++ else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });}return { dir, takenAt, files };}@@@ -220,6 +242,24 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);const d = svc.drill;if (d.compare.length && !d.productionPort) throw new DrillError(`${id} declares compare paths but no production port`);++ // The production env file is loaded first, so an override the drill does not set keeps its++ // production value. Refuse before creating or starting anything if a PostgreSQL service is not++ // fully pointed at its scratch database and away from production Valkey.++ const pgRestored = svc.databases.filter((db) => db.engine === 'postgresql' && Object.prototype.hasOwnProperty.call(d.databases, db.name));++ // A PostgreSQL database the drill does not restore would leave the instance on production's URLs++ // (and its migrations on production's direct URL): every one must be in drill.databases.++ const pgUnmapped = svc.databases.filter((db) => db.engine === 'postgresql' && !Object.prototype.hasOwnProperty.call(d.databases, db.name));++ if (pgUnmapped.length) throw new DrillError(`${id}: drill.databases must map every PostgreSQL database (${pgUnmapped.map((db) => db.name).join(', ')} is missing): otherwise the drill instance would run on production's database`);++ if (pgRestored.length) {++ const missingEnv = [];++ for (const db of pgRestored) {++ const t = d.databases[db.name];++ if (!t.url) missingEnv.push(`${db.engine} ${db.database} has no url env var`);++ if (!t.directUrl) missingEnv.push(`${db.engine} ${db.database} has no directUrl env var`);++ }++ if (typeof d.env.VALKEY_URL !== 'string' || d.env.VALKEY_URL === '') missingEnv.push('VALKEY_URL: without it the drill instance would write into the production Valkey keyspace');++ if (missingEnv.length) throw new DrillError(`${id}: drill.env must set ${missingEnv.join('; ')}`);++ }const missing = await missingRequirements(exec, svc);if (missing.length) throw new DrillError(`${id}: the deployed checkout (${svc.codeDir}) does not have what this drill's overrides rely on: ${missing.join('; ')}. Deploy a release with the drill switch first.`);if (svc.envFile && !(await exec.stat(svc.envFile))) throw new DrillError(`${svc.envFile} not found`);@@@ -252,6 +292,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {counts: [],acceptance: [],stop: null,++ scratch: null,kept: false,result: 'failed',failure: null,@@@ -259,23 +300,76 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const fail = (stage, message) => { if (!record.failure) record.failure = { stage, message }; log(`✗ ${stage}: ${message}`); return new Abort(message); };let created = false;let productionPids = [];++ const scratches = [];try {++ // Free space before anything is restored: pg_restore needs room for the restored tables on++ // top of the archive plus the database's own WAL and temporary files.++ const pgFiles = bk.files.filter((f) => f.engine === 'postgresql');++ if (pgFiles.length) {++ let dumpSize = 0;++ for (const f of pgFiles) { const s = await exec.stat(f.file); dumpSize += s ? s.size : 0; }++ const needed = 2 * dumpSize + 1024 ** 3;++ // The drill directory may not exist yet on a host that has never run a drill, and statfs++ // works on an existing path; create it first (the mkdir below is then a harmless no-op).++ await exec.mkdir(inv.drillDir, { mode: 0o711 });++ const free = (await exec.statfs(inv.drillDir)).free;++ if (free < needed) throw fail('space', `not enough free space under ${inv.drillDir}: ${free} bytes free, need ${needed} (2 × ${dumpSize} bytes of dump + 1 GiB)`);++ // A drill killed before its finally ran leaves its scratch database and role behind. This++ // service's drill lock is held, so anything under its prefix belongs to no running drill.++ const prefix = `ov_${id.replace(/[^a-z0-9_]/g, '_')}_drill_`;++ const left = await dbengine.scratchLeftovers(exec, prefix);++ for (const name of left.databases) { log(`dropping a scratch database an earlier drill left: ${name}`); await dbengine.dropDatabase(exec, name); }++ for (const name of left.roles) { log(`dropping a scratch role an earlier drill left: ${name}`); await dbengine.dropRole(exec, name); }++ }if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);await exec.mkdir(inv.drillDir, { mode: 0o711 });-- await exec.mkdir(tmp, { owner: svc.runAs, mode: 0o700 });++ // postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;++ // every file inside keeps its own restrictive mode). A SQLite-only drill keeps <tmp> at 0700.++ await exec.mkdir(tmp, { owner: svc.runAs, mode: pgFiles.length ? 0o711 : 0o700 });created = true;await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });++ // postgres (not the service user) reads a PostgreSQL dump, so it gets its own directory; the++ // copy inside is 0600 postgres (the backup itself is root-only, which postgres cannot read).++ if (pgFiles.length) await exec.mkdir(path.join(tmp, 'pg'), { owner: dbengine.PG_USER, mode: 0o700 });const vars = { tmp, port: d.port, db: {} };// A data-directory database keeps its production file name: the service finds it by name.const dirOf = (f) => (f.dir ? expand(f.dir, vars) : null);-- for (const f of bk.files) vars.db[f.name] = f.dir ? path.join(dirOf(f), path.basename(f.production)) : path.join(tmp, 'db', `${f.name}${f.ext}`);++ for (const f of bk.files) { if (f.engine === 'postgresql') continue; vars.db[f.name] = f.dir ? path.join(dirOf(f), path.basename(f.production)) : path.join(tmp, 'db', `${f.name}${f.ext}`); }for (const dir of new Set(bk.files.filter((f) => f.dir).map(dirOf))) await exec.mkdir(dir, { owner: svc.runAs, mode: 0o700 });for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });for (const b of d.bind) if (!(await exec.stat(expand(b.from, vars)))) await exec.mkdir(expand(b.from, vars), { owner: svc.runAs, mode: 0o700 });// 1. restore and check every database copy++ const sid = id.replace(/[^a-z0-9_]/g, '_');++ const scratchFor = (name) => scratches.find((s) => s.name === name);for (const f of bk.files) {++ if (f.engine === 'postgresql') {++ // A PostgreSQL restore replaces the file copy: root copies the root-only archive into++ // <tmp>/pg owned by postgres, verifies it, then creates a scratch database and role and++ // restores the archive into them (--no-owner, --role: the archive names production's++ // role, and every object must end up owned by the drill role the instance connects as).++ const copy = path.join(tmp, 'pg', `${f.name}.dump`);++ log(`restoring ${f.file} → ${copy} (owned by ${dbengine.PG_USER})`);++ const cp = await exec.run('install', ['-o', dbengine.PG_USER, '-m', '0600', '-T', '--', f.file, copy], { privileged: true });++ if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);++ const st = await exec.stat(copy);++ const entry = { name: f.name, source: f.file, copy, bytes: st ? st.size : null, integrity: null, engine: 'postgresql' };++ record.databases.push(entry);++ try { await dbengine.verifyDump(exec, copy, { as: dbengine.PG_USER }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }++ entry.integrity = 'pg_restore --list ok';++ log(`pg_restore --list ${f.name}: ok`);++ // The role and the database share the same name (separate namespaces); dbengine always++ // quotes the identifier, so the stamp's hyphen is fine.++ const name = `ov_${sid}_drill_${at}${pgFiles.length > 1 ? `_${pgFiles.indexOf(f)}` : ''}`;++ const { password } = await dbengine.createRole(exec, name, {});++ const scratch = { name: f.name, database: name, role: name, password, dropped: null, error: null };++ scratches.push(scratch);++ if (!record.scratch) record.scratch = { database: name, role: name };++ await dbengine.createDatabase(exec, name, { owner: name });++ await dbengine.restore(exec, f.production, { dump: copy, database: name, role: name, log });++ continue;++ }const dest = vars.db[f.name];log(`restoring ${f.file} → ${dest} (owned by ${svc.runAs})`);// Backups are root-only (0600), so root copies; install unlinks whatever is at dest@@@ -283,7 +377,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const cp = await exec.run('install', ['-o', svc.runAs, '-m', '0600', '-T', '--', f.file, dest], { privileged: true });if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);const st = await exec.stat(dest);-- const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null };++ const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null, engine: 'sqlite' };record.databases.push(entry);let rows;try { rows = await exec.sqlite(dest, 'PRAGMA integrity_check', { as: svc.runAs }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }@@@ -292,13 +386,28 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {log(`integrity_check ${f.name}: ok`);}const restoredCounts = [];-- for (const c of d.counts) restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));++ for (const c of d.counts) {++ const rdb = svc.databases.find((x) => x.name === c.db);++ // PostgreSQL counts come from the scratch database over psql, SQLite from the restored file.++ if (rdb.engine === 'postgresql') restoredCounts.push(await dbengine.countRows(exec, { engine: 'postgresql', database: scratchFor(c.db).database }, countSql(c.table)));++ else restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));++ }// 2. start the drill instanceconst lines = ['# written by ovhost drill; no secret values: the production env file is loaded separately'];for (const [k, v] of Object.entries(d.env)) lines.push(envLine(k, expand(v, vars)));const dbEnv = new Map();-- for (const f of bk.files) if (f.envName) dbEnv.set(f.envName, f.dir ? dirOf(f) : vars.db[f.name]);++ for (const f of bk.files) {++ if (f.engine === 'postgresql') {++ // Both names point at the scratch database over the direct cluster port (5432, never++ // PgBouncer's 6432: the scratch database and role exist only in the cluster). The++ // password is generated per drill and lives only in this file the instance reads.++ const s = scratchFor(f.name);++ const url = dbengine.databaseUrl({ database: s.database, role: s.role, password: s.password });++ dbEnv.set(f.url, url);++ dbEnv.set(f.directUrl, url);++ } else if (f.envName) dbEnv.set(f.envName, f.dir ? dirOf(f) : vars.db[f.name]);++ }for (const [k, v] of dbEnv) lines.push(envLine(k, v));const envPath = path.join(tmp, 'drill.env');await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });@@@ -374,11 +483,17 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {const db = svc.databases.find((x) => x.name === c.db);let production = null;let error = null;-- try { production = await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }-- const row = { db: c.db, table: c.table, production, restored: restoredCounts[i], match: production === restoredCounts[i] };++ // PostgreSQL counts come from the cluster over psql (the inventory entry itself, never++ // .path, which a PostgreSQL entry does not have); SQLite from the production file.++ try { production = db.engine === 'postgresql' ? await dbengine.countRows(exec, db, countSql(c.table)) : await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }++ const tol = d.countsTolerance || 0;++ // A busy table drifts between the backup and now: countsTolerance (default 0) allows it.++ const match = production != null && restoredCounts[i] != null && Math.abs(production - restoredCounts[i]) <= tol;++ const row = { db: c.db, table: c.table, production, restored: restoredCounts[i], match };++ if (tol) row.tolerance = tol;if (error) row.error = error;record.counts.push(row);-- log(`${row.match ? 'same' : 'DIFFERENT'} ${c.table}: production ${production == null ? `unknown (${error})` : production}, restored ${restoredCounts[i]}`);++ log(`${row.match ? 'same' : 'DIFFERENT'} ${c.table}: production ${production == null ? `unknown (${error})` : production}, restored ${restoredCounts[i]}${tol ? ` (±${tol})` : ''}`);}const bad = [...record.compare.filter((r) => !r.match).map((r) => `${r.path} (${r.detail})`), ...record.counts.filter((r) => !r.match).map((r) => `${r.table} ${r.production} ≠ ${r.restored}`)];if (bad.length) throw fail('compare', `differs from production: ${bad.join('; ')}`);@@@ -401,10 +516,27 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {if (unaccepted.length) throw fail('acceptance', unaccepted.join('; '));} catch (err) {if (!(err instanceof Abort)) fail('error', err.message);++ } finally {++ // Cleanup ALWAYS, on success and on failure: stop the drill instance first (so nothing is++ // connected to the scratch database), then drop the scratch database and role. dropdb is++ // attempted before dropRole so a dropdb refusal (a session still connected) cannot stop the++ // role attempt. A failure to drop is logged and recorded, never thrown: it must not mask the++ // drill's own outcome.++ // A throw while stopping must not skip the drops below (the leftover sweep only runs on the next drill).++ try { record.stop = await stopInstance(exec, record.pid, d.port, productionPids, log); } catch (err) { record.stop = { ok: false, detail: `stop failed: ${err.message}` }; log(`could not stop the drill instance: ${err.message}`); }++ for (const s of scratches) {++ try { await dbengine.dropDatabase(exec, s.database); } catch (err) { s.dropped = false; s.error = `dropdb: ${err.message}`; log(`could not drop the scratch database ${s.database}: ${err.message}`); }++ try { await dbengine.dropRole(exec, s.role); } catch (err) { s.dropped = false; s.error = [s.error, `droprole: ${err.message}`].filter(Boolean).join('; '); log(`could not drop the scratch role ${s.role}: ${err.message}`); }++ if (s.dropped === null) s.dropped = true;++ }++ if (record.scratch && scratches.length) {++ record.scratch.dropped = scratches.every((s) => s.dropped);++ const dropErrors = scratches.filter((s) => s.error).map((s) => s.error);++ if (dropErrors.length) record.scratch.error = dropErrors.join('; ');++ }}-- // 4. stop, clean up, record-- record.stop = await stopInstance(exec, record.pid, d.port, productionPids, log);++ // 4. stop outcome, remove the drill directory, recordif (record.stop && !record.stop.ok) fail('stop', record.stop.detail);if (created) {const safe = tmp.startsWith(`${inv.drillDir}/`) && /^[a-z][a-z0-9-]*-\d{8}-\d{6}$/.test(path.basename(tmp));@@@ -454,13 +586,15 @@ function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }/** A row for the table in docs/restore-drills.md. */function markdownRow(r) {const checks = [];-- for (const db of r.databases) checks.push(`\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);++ for (const db of r.databases) checks.push(db.engine === 'postgresql'++ ? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`++ : `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);if (same.length) checks.push(`${same.join(', ')} identical to production${r.compare.some((c) => c.match && c.mode && c.mode !== 'bytes') ? ' (JSON without volatile keys where declared)' : ''}`);checks.push(...diff);-- for (const c of r.counts) checks.push(`${c.table} ${c.production == null ? '?' : c.production} ${c.match ? '=' : '≠'} ${c.restored}`);++ for (const c of r.counts) checks.push(`${c.table} ${c.production == null ? '?' : c.production} ${c.match ? '=' : '≠'} ${c.restored}${c.tolerance ? ` (±${c.tolerance})` : ''}`);const accepted = (r.acceptance || []).filter((a) => a.ok).map((a) => `\`${a.path}\``);if (accepted.length) checks.push(`acceptance ${accepted.join(', ')}`);if (r.failure && r.failure.stage !== 'compare' && r.failure.stage !== 'acceptance') checks.push(`${r.failure.stage}: ${cell(r.failure.message)}`);nnew file mode 100644@@@ -0,0 +1,222 @@++'use strict';++/**++ * Per-engine database helpers. lib/inventory.js normalises every entry in `databases[]` to an++ * `engine` of "sqlite" (the default: a file) or "postgresql" (a database in the cluster), so no++ * caller has to know which commands an engine needs.++ *++ * SQLite keeps the existing worker (lib/sqlite-worker.js): every file operation runs as the service++ * user, and root never opens a service database. PostgreSQL runs as the postgres OS user over peer++ * auth (`local postgres peer` in roles/data/files/pg_hba.conf) — the same property: ovhost reads no++ * service env value, and never holds a database password except the one it generates itself for a++ * restore drill.++ *++ * pgBackRest, not ovhost, is the durability layer for the cluster (roles/data/provision.sh: one++ * stanza `openvibe`, full weekly + differential nightly + WAL to object storage). What ovhost does++ * per night is verify that layer (verifyPgBackRest, walFailedCount) and, on a slower cadence, take++ * a per-service logical dump (dump) that the off-site code encrypts and uploads unchanged.++ */++const crypto = require('crypto');++const path = require('path');++++// The OS user PostgreSQL's own tools run as (peer auth), the stanza provision.sh creates, and the++// direct connection a drill must use: 5432 is the cluster, 6432 is PgBouncer.++const PG_USER = 'postgres';++const PG_STANZA = 'openvibe';++const PG_DIRECT_HOST = '127.0.0.1';++const PG_DIRECT_PORT = 5432;++// A port nothing listens on: a drill instance keeps its production Valkey prefix, so it must not be++// able to reach a real Valkey (it would write into the production keyspace).++const CLOSED_VALKEY_URL = 'redis://127.0.0.1:9/0';++++const WORK_TIMEOUT_MS = 6 * 3600 * 1000;++++const ident = (name) => `"${String(name).replace(/"/g, '""')}"`;++const literal = (value) => `'${String(value).replace(/'/g, "''")}'`;++const scrubbed = (r) => ((r.stderr || r.stdout || '').trim().slice(0, 300) || `exit ${r.code}`);++++/** What a log line and a record call this database. */++function label(db) {++ return db.engine === 'postgresql' ? db.database : db.path;++}++++/** The file name this database's artifact gets inside a backup directory. */++function fileName(db) {++ return db.engine === 'postgresql' ? `${db.name}.dump` : `${db.name}${path.extname(db.path) || '.db'}`;++}++++/** Whether the database is still there (SQLite: the file; PostgreSQL: the database in the cluster). */++async function exists(exec, db) {++ if (db.engine === 'sqlite') return !!(await exec.stat(db.path));++ const rows = await exec.psql('postgres', `SELECT 1 AS present FROM pg_database WHERE datname = ${literal(db.database)}`);++ return rows.length > 0;++}++++/**++ * A logical dump of one database into `dest` (a `pg_dump -Fc` archive: compressed, and restorable++ * table by table with pg_restore). It runs as postgres, so `dest` must be writable by postgres —++ * the caller's staging directory is owned by postgres when any database is PostgreSQL.++ */++async function dump(exec, db, dest, { log } = {}) {++ if (db.engine !== 'postgresql') throw new Error('sqlite dumps go through exec.sqliteBackup (the worker runs as the service user)');++ // -w: peer auth needs no password, and a prompt in the nightly unit would hang until the timeout.++ const r = await exec.run('pg_dump', ['-w', '-Fc', '-f', dest, '-d', db.database], { as: PG_USER, timeoutMs: WORK_TIMEOUT_MS });++ if (r.code !== 0) throw new Error(`pg_dump ${db.database} failed: ${scrubbed(r)}`);++ if (log) log(`pg_dump ${db.database} → ${dest}`);++ return dest;++}++++/**++ * Whether `file` is a readable pg_dump archive (pg_restore --list parses its table of contents).++ * --list only reads the file and contacts no server, so a caller holding a root-only copy can pass++ * `as: null` and check it as itself rather than making it readable to postgres.++ */++async function verifyDump(exec, file, { as = PG_USER } = {}) {++ const r = await exec.run('pg_restore', ['--list', file], { as, timeoutMs: 120000 });++ if (r.code !== 0) throw new Error(`pg_restore --list ${file} failed: ${scrubbed(r)}`);++ return true;++}++++/** A database role with a freshly generated password. -> { role, password } */++async function createRole(exec, role, { password = null } = {}) {++ const pw = password || crypto.randomBytes(24).toString('hex');++ // The password reaches psql on stdin, never in argv: /proc/<pid>/cmdline is world-readable.++ const sql = `CREATE ROLE ${ident(role)} LOGIN PASSWORD ${literal(pw)};\n`;++ const r = await exec.run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '-d', 'postgres', '-f', '-'], { as: PG_USER, input: sql, timeoutMs: 120000 });++ if (r.code !== 0) throw new Error(`creating role ${role} failed: ${scrubbed(r)}`);++ return { role, password: pw };++}++++async function createDatabase(exec, database, { owner = null } = {}) {++ const args = ['-w'];++ if (owner) args.push(`--owner=${owner}`);++ args.push(database);++ const r = await exec.run('createdb', args, { as: PG_USER, timeoutMs: 120000 });++ if (r.code !== 0) throw new Error(`createdb ${database} failed: ${scrubbed(r)}`);++ return database;++}++++async function dropDatabase(exec, database) {++ // --force ends any session still connected (a drill instance that did not stop): only ever called++ // with a scratch database's name, never a production one.++ const r = await exec.run('dropdb', ['-w', '--if-exists', '--force', database], { as: PG_USER, timeoutMs: 120000 });++ if (r.code !== 0) throw new Error(`dropdb ${database} failed: ${scrubbed(r)}`);++}++++async function dropRole(exec, role) {++ const sql = `DROP ROLE IF EXISTS ${ident(role)};\n`;++ const r = await exec.run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '-d', 'postgres', '-f', '-'], { as: PG_USER, input: sql, timeoutMs: 120000 });++ if (r.code !== 0) throw new Error(`dropping role ${role} failed: ${scrubbed(r)}`);++}++++/**++ * Restore a pg_dump archive into `database`, under `role`. --no-owner because the archive names the++ * production role (which the scratch database must not depend on); --role makes every object land++ * owned by the drill role, so the drilled service can read and write it.++ */++async function restore(exec, db, { dump: dumpFile, database, role, log } = {}) {++ if (db.engine !== 'postgresql') throw new Error('sqlite restore is a file copy, not pg_restore');++ // --no-privileges: the archive's GRANTs name production roles; the scratch copy must not hand them anything.++ const args = ['-w', '--no-owner', '--no-privileges'];++ if (role) args.push(`--role=${role}`);++ args.push('-d', database, dumpFile);++ const r = await exec.run('pg_restore', args, { as: PG_USER, timeoutMs: WORK_TIMEOUT_MS });++ if (r.code !== 0) throw new Error(`pg_restore into ${database} failed: ${scrubbed(r)}`);++ if (log) log(`pg_restore ${dumpFile} → ${database} (as ${role || PG_USER})`);++ return database;++}++++/**++ * Scratch databases and roles an earlier drill left behind (killed before its cleanup ran): every name++ * that starts with `prefix` (ov_<service>_drill_). -> { databases: [], roles: [] }++ */++async function scratchLeftovers(exec, prefix) {++ const p = literal(prefix);++ const databases = await exec.psql('postgres', `SELECT datname AS name FROM pg_database WHERE left(datname, length(${p})) = ${p}`);++ const roles = await exec.psql('postgres', `SELECT rolname AS name FROM pg_roles WHERE left(rolname, length(${p})) = ${p}`);++ const names = (rows) => (rows || []).map((r) => r.name).filter((n) => typeof n === 'string' && n.startsWith(prefix));++ return { databases: names(databases), roles: names(roles) };++}++++/** A row count from either engine. `sql` must return the number in its first column. */++async function countRows(exec, db, sql, { as } = {}) {++ const rows = db.engine === 'postgresql' ? await exec.psql(db.database, sql) : await exec.sqlite(db.path, sql, { as });++ const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);++ if (!Number.isFinite(n)) throw new Error(`count of ${label(db)} returned no number`);++ return n;++}++++/** The connection URL a drill instance must use: the cluster directly, never PgBouncer. */++function databaseUrl({ database, role, password }) {++ return `postgresql://${encodeURIComponent(role)}:${encodeURIComponent(password)}@${PG_DIRECT_HOST}:${PG_DIRECT_PORT}/${encodeURIComponent(database)}`;++}++++/** `pgbackrest info --output=json` as postgres. */++async function pgbackrestInfo(exec) {++ const r = await exec.run('pgbackrest', [`--stanza=${PG_STANZA}`, 'info', '--output=json'], { as: PG_USER, timeoutMs: 300000 });++ if (r.code !== 0) throw new Error(`pgbackrest info failed: ${scrubbed(r)}`);++ let stanzas;++ try { stanzas = JSON.parse(r.stdout); } catch { throw new Error(`pgbackrest info did not answer JSON: ${String(r.stdout).trim().slice(0, 200)}`); }++ return Array.isArray(stanzas) ? stanzas : [];++}++++/** The newest backup in a stanza's `info`: { type, stop (epoch seconds) } or null. */++function newestBackup(stanza) {++ let best = null;++ for (const b of stanza.backup || []) {++ const stop = b && b.timestamp ? Number(b.timestamp.stop) : NaN;++ if (!Number.isFinite(stop)) continue;++ if (!best || stop > best.stop) best = { type: b.type || 'unknown', stop };++ }++ return best;++}++++/**++ * The every-run pgBackRest verification: the stanza exists and is ok, and its newest backup stopped++ * within `maxAgeSeconds` (26 h: nightly differentials at 02:15, with room for a slow one).++ * -> { type, stop (ISO), ageSeconds }++ */++async function verifyPgBackRest(exec, { now, maxAgeSeconds = 26 * 3600 } = {}) {++ const stanzas = await pgbackrestInfo(exec);++ const stanza = stanzas.find((s) => s && s.name === PG_STANZA);++ if (!stanza) throw new Error(`pgBackRest has no stanza ${PG_STANZA}: the cluster is not being backed up`);++ const code = Number((stanza.status && stanza.status.code) || 0);++ if (code !== 0) throw new Error(`pgBackRest stanza ${PG_STANZA} is not ok: ${(stanza.status && stanza.status.message) || `status code ${code}`}`);++ const newest = newestBackup(stanza);++ if (!newest) throw new Error(`pgBackRest stanza ${PG_STANZA} holds no backup at all`);++ const age = Math.floor(now / 1000) - newest.stop;++ if (age > maxAgeSeconds) throw new Error(`the newest pgBackRest backup (${newest.type}) stopped ${(age / 3600).toFixed(1)}h ago (limit ${maxAgeSeconds / 3600}h)`);++ return { type: newest.type, stop: new Date(newest.stop * 1000).toISOString(), ageSeconds: age };++}++++/** `pg_stat_archiver.failed_count`: the cluster-wide count of WAL segments archiving ever failed on. */++async function walFailedCount(exec) {++ const rows = await exec.psql('postgres', 'SELECT failed_count::text AS n FROM pg_stat_archiver');++ const n = Number(rows && rows[0] ? rows[0].n : NaN);++ if (!Number.isFinite(n)) throw new Error('pg_stat_archiver returned no failed_count');++ return n;++}++++module.exports = {++ PG_USER,++ PG_STANZA,++ PG_DIRECT_HOST,++ PG_DIRECT_PORT,++ CLOSED_VALKEY_URL,++ label,++ fileName,++ exists,++ dump,++ verifyDump,++ restore,++ createRole,++ createDatabase,++ dropDatabase,++ dropRole,++ scratchLeftovers,++ countRows,++ databaseUrl,++ pgbackrestInfo,++ verifyPgBackRest,++ newestBackup,++ walFailedCount,++};@@@ -23,6 +23,11 @@* any method, http:// or https:// (release notifications: Network's token endpoint and Events'* publish API). Headers and body may carry a secret: nothing here logs them.* sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })++ * psql(database, sql, { as }) -> rows. PostgreSQL is reached as the postgres OS user over peer++ * auth (`local postgres peer`, roles/data/files/pg_hba.conf) — the same no-secret property the++ * SQLite worker gives the service user. Everything else PostgreSQL (pg_dump, pg_restore,++ * createdb, dropdb, pgbackrest) goes through run() with { as: 'postgres' }.++ * statfs(path) -> { free, size } bytes, for the free-space check before a restore* listeners(port) -> [{ pid, process }]* kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)* sleep(ms) ; now() -> ms ; isRoot() ; userName() ; hostname()@@@ -232,6 +237,28 @@ function createSystemExecutor({ env = process.env } = {}) {return out;}++ /** `psql --csv`: a header row and one line per row, so a caller reads columns by name. */++ function parseCsv(text) {++ const rows = [];++ let row = [];++ let field = '';++ let quoted = false;++ for (let i = 0; i < text.length; i++) {++ const c = text[i];++ if (quoted) {++ if (c === '"') { if (text[i + 1] === '"') { field += '"'; i++; } else quoted = false; }++ else field += c;++ } else if (c === '"') quoted = true;++ else if (c === ',') { row.push(field); field = ''; }++ else if (c === '\n') { row.push(field); rows.push(row); row = []; field = ''; }++ else if (c !== '\r') field += c;++ }++ if (field !== '' || row.length) { row.push(field); rows.push(row); }++ if (!rows.length) return [];++ const header = rows[0];++ return rows.slice(1).filter((r) => r.length > 1 || r[0] !== '').map((r) => Object.fromEntries(header.map((h, i) => [h, r[i] === undefined ? null : r[i]])));++ }++async function listeners(port) {const r = await run('ss', ['-H', '-ltnp', `sport = :${port}`], { privileged: true });if (r.code !== 0) throw new Error(`ss failed: ${r.stderr.trim()}`);@@@ -266,6 +293,15 @@ function createSystemExecutor({ env = process.env } = {}) {request,sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },++ psql: async (database, sql, { as = 'postgres' } = {}) => {++ const r = await run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '--csv', '-d', database, '-c', sql], { as, timeoutMs: 6 * 3600 * 1000 });++ if (r.code !== 0) throw new Error(`psql ${database} failed: ${(r.stderr || r.stdout).trim().slice(0, 300)}`);++ return parseCsv(r.stdout);++ },++ statfs: async (p) => {++ const s = fs.statfsSync(p);++ return { free: s.bavail * s.bsize, size: s.blocks * s.bsize };++ },listeners,kill: async (pid, signal = 'SIGTERM') => {if (!Number.isInteger(pid) || pid <= 1) throw new Error(`refusing to signal pid ${pid}`);@@@ -20,7 +20,10 @@ const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;const DRAIN_POLICIES = ['refuse', 'wait', 'report'];--const PROBE_KINDS = ['http-json-count', 'sqlite-count', 'sum'];++const PROBE_KINDS = ['http-json-count', 'sqlite-count', 'postgresql-count', 'sum'];++// A PostgreSQL database an ovhost service owns: one per service in the cluster pgBackRest backs up++// (roles/data/provision.sh), reached as the postgres OS user over peer auth (no secret is ever read).++const PG_DATABASE_RE = /^ov_[a-z0-9_]+$/;function abs(value, where) {if (typeof value !== 'string' || !path.isAbsolute(value) || value.includes('\0')) throw new InventoryError(`${where} must be an absolute path`);@@@ -209,10 +212,28 @@ function normaliseService(id, raw, defaults) {noRestartPaths: s.noRestartPaths || [],protected: s.protected || null,drain: { policy: 'refuse', waitMaxSeconds: 8 * 3600, pollSeconds: 60, quietChecks: 2, ...(s.drain || {}) },++ // A database is one of two engines. SQLite (the default) is a file the service user's worker++ // copies; PostgreSQL is a database in the cluster, named, and backed up by pgBackRest as a++ // whole — ovhost never opens it directly. Nothing outside the engine's own code may assume++ // `.path` exists.databases: (s.databases || []).map((d, i) => {-- const name = d.name || path.basename(String(d.path), '.db');-- if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${where}.databases[${i}].name must be a plain name`);-- return { name, path: abs(d.path, `${where}.databases[${i}].path`) };++ const at = `${where}.databases[${i}]`;++ const engine = d.engine == null ? 'sqlite' : d.engine;++ if (engine === 'sqlite') {++ if (d.database != null) throw new InventoryError(`${at}: a sqlite database is a path, not a database name`);++ const name = d.name || path.basename(String(d.path), '.db');++ if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);++ return { name, engine, path: abs(d.path, `${at}.path`) };++ }++ if (engine === 'postgresql') {++ if (d.path != null) throw new InventoryError(`${at}: a postgresql database has no path (it lives in the cluster, named by "database")`);++ if (!PG_DATABASE_RE.test(d.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_trade)`);++ const name = d.name || String(d.database).slice(3);++ if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);++ if (d.role != null && !/^[a-z_][a-z0-9_-]*$/.test(String(d.role))) throw new InventoryError(`${at}.role must be a database role name`);++ return { name, engine, database: d.database, role: d.role == null ? null : String(d.role) };++ }++ throw new InventoryError(`${at}.engine must be sqlite or postgresql`);}),backupOnChange: s.backupOnChange || [],nginx: s.nginx || null,@@@ -273,6 +294,10 @@ function normaliseService(id, raw, defaults) {abs(p.db, `${at}.db`);if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);}++ if (p.kind === 'postgresql-count') {++ if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);++ if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);++ }if (p.kind === 'sum') {if (!Array.isArray(p.probes) || !p.probes.length) throw new InventoryError(`${at}.probes must be a non-empty array`);p.probes.forEach((q, i) => checkProbe(q || {}, `${at}.probes[${i}]`, true));@@@ -287,6 +312,18 @@ function normaliseService(id, raw, defaults) {if (n.repoVhosts) rel(n.repoVhosts.replace(/\*\.conf$/, 'x.conf'), `${where}.nginx.repoVhosts`);if (n.variant && !['http', 'sse', 'websocket'].includes(n.variant)) throw new InventoryError(`${where}.nginx.variant must be http, sse or websocket`);if (n.certName && !FILE_NAME_RE.test(n.certName)) throw new InventoryError(`${where}.nginx.certName must be a name`);++ // The tenants database may be a PostgreSQL database (the Host API is on PostgreSQL): the++ // verified custom domains are then read over psql as the postgres OS user, not from a file.++ if (n.tenants && n.tenants.database != null) {++ const db = n.tenants.database;++ let tenantDb;++ if (typeof db === 'string') tenantDb = { engine: 'sqlite', path: abs(db, `${where}.nginx.tenants.database`) };++ else if (db && typeof db === 'object' && !Array.isArray(db) && db.engine === 'postgresql' && PG_DATABASE_RE.test(db.database || '')) tenantDb = { engine: 'postgresql', database: db.database };++ else throw new InventoryError(`${where}.nginx.tenants.database must be a path or { "engine": "postgresql", "database": "ov_host" }`);++ // A fresh object every time: normalise() must never rewrite the document it was handed++ // (the example inventory is a shared require() cache, and a drill test normalises it twice).++ out.nginx = { ...n, tenants: { ...n.tenants, database: tenantDb } };++ }}if (out.lifecycle !== null && (typeof out.lifecycle !== 'object' || Array.isArray(out.lifecycle))) throw new InventoryError(`${where}.lifecycle must be an object (the manifest's lifecycle block)`);for (const p of out.backupOnChange) rel(p, `${where}.backupOnChange`);@@@ -309,6 +346,12 @@ const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;* DIRECTORY: the copy lands in {tmp}/data/ (or "dir": "{tmp}/<name>") under its* production file name and the env var (optional) points at that directory. Without* "env", pair the directory with `bind`.++ * A database with engine postgresql instead names the two env vars that point the++ * service at a database URL: { "url": "DATABASE_URL", "directUrl":++ * "DATABASE_DIRECT_URL" }. The drill creates a scratch database and a scratch role,++ * restores the dump into it, and writes both vars (and VALKEY_URL, which the drill++ * must neutralise itself) into drill.env — the service's own env file is loaded first,++ * so any name the drill does not override keeps its production value.* env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}* and {db:<name>}* dirs directories to create (owned by the service user) inside {tmp} before the start@@@ -349,19 +392,28 @@ function normaliseDrill(raw, svc, where) {const inDataDir = new Map();for (const [name, target] of Object.entries(rawDatabases)) {if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);++ const defined = svc.databases.find((d) => d.name === name);++ if (defined.engine === 'postgresql') {++ if (!target || typeof target !== 'object' || Array.isArray(target)) throw new InventoryError(`${w}.databases.${name} must be { "url": "<NAME>", "directUrl": "<NAME>" } for a postgresql database`);++ if (!ENV_NAME_RE.test(target.url || '')) throw new InventoryError(`${w}.databases.${name}.url must be an env var name`);++ if (!ENV_NAME_RE.test(target.directUrl || '')) throw new InventoryError(`${w}.databases.${name}.directUrl must be an env var name`);++ if (target.url === target.directUrl) throw new InventoryError(`${w}.databases.${name}: url and directUrl must be different env vars`);++ databases[name] = { engine: 'postgresql', url: target.url, directUrl: target.directUrl };++ continue;++ }if (typeof target === 'string') {if (!ENV_NAME_RE.test(target)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);-- databases[name] = { env: target, dir: false };++ databases[name] = { engine: 'sqlite', env: target, dir: false };continue;}const dir = target && target.dir === true ? '{tmp}/data' : target && target.dir;if (!target || typeof target !== 'object' || typeof dir !== 'string') throw new InventoryError(`${w}.databases.${name} must be an env var name or { "env": "<NAME>", "dir": true }`);if (!/^\{tmp\}\/[A-Za-z0-9._-]+$/.test(dir)) throw new InventoryError(`${w}.databases.${name}.dir must be true or "{tmp}/<name>"`);if (target.env != null && !ENV_NAME_RE.test(target.env)) throw new InventoryError(`${w}.databases.${name}.env must be an env var name`);-- const dest = `${dir}/${path.basename(svc.databases.find((d) => d.name === name).path)}`;++ const dest = `${dir}/${path.basename(defined.path)}`;if (inDataDir.has(dest)) throw new InventoryError(`${w}.databases: ${name} and ${inDataDir.get(dest)} would both be ${dest}`);inDataDir.set(dest, name);-- databases[name] = { env: target.env || null, dir };++ databases[name] = { engine: 'sqlite', env: target.env || null, dir };}const dirEnvs = new Map();for (const t of Object.values(databases)) {@@@ -379,7 +431,12 @@ function normaliseDrill(raw, svc, where) {}}if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);-- for (const t of Object.values(databases)) if (t.env && t.env in env) throw new InventoryError(`${w}.env.${t.env} is already set by drill.databases`);++ for (const t of Object.values(databases)) {++ // The drill writes these itself; declaring one in drill.env as well would put two lines with++ // the same name in the file, and which wins is not something an operator should have to know.++ const owned = t.engine === 'postgresql' ? [t.url, t.directUrl] : t.env ? [t.env] : [];++ for (const name of owned) if (name in env) throw new InventoryError(`${w}.env.${name} is already set by drill.databases`);++ }const dirs = raw.dirs || [];for (const d of dirs) if (typeof d !== 'string' || !/^\{tmp\}\/[A-Za-z0-9._/-]+$/.test(d) || d.includes('..')) throw new InventoryError(`${w}.dirs entries must be paths under {tmp}`);if (raw.command != null && (!Array.isArray(raw.command) || !raw.command.length || raw.command.some((a) => typeof a !== 'string'))) throw new InventoryError(`${w}.command must be an argv array`);@@@ -434,6 +491,10 @@ function normaliseDrill(raw, svc, where) {});if (!compare.length && !counts.length && !acceptance.length) throw new InventoryError(`${w} needs at least one compare path, row count or acceptance check`);const readyTimeoutSeconds = Number(raw.readyTimeoutSeconds || (svc.ready && svc.ready.timeoutSeconds) || 90);++ // A write-heavy service moves between the production read and the backup's dump (Events took++ // 18.7k rows in a day): countsTolerance lets the operator allow for that instead of flapping.++ const countsTolerance = raw.countsTolerance == null ? 0 : Number(raw.countsTolerance);++ if (!(Number.isInteger(countsTolerance) && countsTolerance >= 0)) throw new InventoryError(`${w}.countsTolerance must be a whole number of rows`);return {supported: true,port,@@@ -449,6 +510,7 @@ function normaliseDrill(raw, svc, where) {readyTimeoutSeconds,compare,counts,++ countsTolerance,acceptance,outbound: raw.outbound === true,outboundReason: raw.outboundReason || null,@@@ -9,6 +9,7 @@const fs = require('fs');const path = require('path');const systemd = require('./systemd');++const dbengine = require('./dbengine');const TEMPLATE_DIR = path.join(__dirname, '..', 'templates', 'nginx');@@@ -235,7 +236,7 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {if (withoutCert.length) servers += fill(readTemplate('custom-http.tmpl'), { names: withoutCert.map((d) => d.hostname).join(' '), fallback: 'return 404;' });for (const d of withCert) servers += fill(readTemplate('custom-https.tmpl'), { ...base, hostname: d.hostname, certDir: path.join(t.certBase, d.hostname), tenantLocation });const custom = fill(readTemplate('custom-domains.conf.tmpl'), {-- service: svc.id, database: t.database || '(the Host database)', withCert: withCert.length, withoutCert: withoutCert.length,++ service: svc.id, database: t.database ? dbengine.label(t.database) : '(the Host database)', withCert: withCert.length, withoutCert: withoutCert.length,servers: servers || '\n# No verified custom domains yet.\n',});return {@@@ -247,11 +248,17 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {};}--/** Verified custom domains from the Host database (read-only, as the service user) + certificate presence. */++/**++ * Verified custom domains from the Host database + certificate presence. A PostgreSQL database is++ * read as the postgres OS user over peer auth (Host moved to PostgreSQL; its service role's password++ * is not ovhost's to read), a SQLite file as the service user.++ */async function tenantDomains(exec, svc) {const t = tenantsConfig(svc);if (!t.database) throw new Error(`${svc.id}: nginx.tenants.database is not set in the inventory`);-- const rows = await exec.sqlite(t.database, VERIFIED_SQL, { as: svc.runAs });++ const rows = t.database.engine === 'postgresql'++ ? await exec.psql(t.database.database, VERIFIED_SQL)++ : await exec.sqlite(t.database.path, VERIFIED_SQL, { as: svc.runAs });const out = [];for (const r of rows || []) {const h = String(r.hostname || '').toLowerCase();@@@ -27,6 +27,7 @@const path = require('path');const { pipeline } = require('stream/promises');const bcrypto = require('./backup-crypto');++const dbengine = require('./dbengine');const { STAMP_RE, parseStamp } = require('./retention');const DEFAULT_ENV_FILE = '/etc/openvibe/backup.env';@@@ -325,6 +326,8 @@ async function readManifest(client, cfg, master, host, run) {function unsafeOut(inv, out) {for (const svc of Object.values(inv.services)) {for (const db of svc.databases) {++ // A PostgreSQL database is in the cluster, not at a path: it has no directory to avoid.++ if (!db.path) continue;const dir = path.dirname(db.path);if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;}@@@ -385,13 +388,20 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:await exec.removeFile(dest);throw new OffsiteError(`${f.file}: the download does not match the manifest's hashes`);}++ // A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which++ // needs no database and is run as the caller — the copy is root-only 0600, and postgres could++ // not read it); a SQLite copy is quick_checked through the worker.let check;++ const isDump = path.extname(f.file) === '.dump';try {-- const rows = await exec.sqlite(dest, 'PRAGMA quick_check');-- check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';++ if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }++ else {++ const rows = await exec.sqlite(dest, 'PRAGMA quick_check');++ check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';++ }} catch (err) { check = `error: ${err.message}`; }record.files.push({ name: f.name, file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: check });-- if (check !== 'ok') throw new OffsiteError(`${dest}: quick_check = ${check}`);++ if (isDump ? check.startsWith('error') : check !== 'ok') throw new OffsiteError(`${dest}: ${check}`);}return record;}@@@ -416,6 +426,7 @@ module.exports = {pruneRemote,listRuns,restoreDownload,++ unsafeOut,check,uploadIterable,parseValues,@@@ -7,7 +7,11 @@* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.* sqlite-count a read-only SELECT against the service's database, run as the service user.* Media: SELECT count(*) FROM vods WHERE is_recording = 1.-- * sum the total of `probes` (each an http-json-count or sqlite-count): Tools' running++ * postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS++ * user over peer auth (no secret is read, and a PostgreSQL service's database has++ * no file for a sqlite-count to open). OpenRe: ingest sessions in flight.++ * sum the total of `probes` (each an http-json-count, sqlite-count or++ * postgresql-count): Tools' running* jobs in each satellite's jobs.db. Unknown when any part is.** A probe that cannot answer while the service is running is UNKNOWN, never zero: "could not ask"@@@ -75,6 +79,15 @@ async function countOne(exec, svc, p, label) {if (!Number.isFinite(n)) return { count: null, label, unknown: 'query returned no number' };return { count: n, label };}++ if (p.kind === 'postgresql-count') {++ // As the postgres OS user over peer auth: the service's own role needs a password, which++ // ovhost must never read.++ const rows = await exec.psql(p.database, p.sql);++ const first = rows && rows[0] ? Object.values(rows[0])[0] : undefined;++ const n = Number(first);++ if (!Number.isFinite(n)) return { count: null, label, unknown: 'query returned no number' };++ return { count: n, label };++ }} catch (err) {return { count: null, label, unknown: err.message };}@@@ -10,6 +10,9 @@ const crypto = require('crypto');const { scenario, test, runTests, SECRET } = require('./helpers');const { main } = require('../lib/cli');const { mockS3 } = require('./s3-mock');++const { refusal } = require('../lib/archive');++const offsite = require('../lib/offsite');++const { normalise } = require('../lib/inventory');const KEY_HEX = crypto.createHash('sha256').update('archive test key').digest('hex');const S3_SECRET = `${SECRET}-s3`;@@@ -78,7 +81,7 @@ runTests([test('refusals: live databases, open files, symlinks, outside roots, relative paths; nothing uploaded', async () => {const host = archiveHost();-- const liveDb = Object.values(host.inv.services).flatMap((s) => s.databases.map((d) => d.path))[0];++ const liveDb = Object.values(host.inv.services).flatMap((s) => s.databases.filter((d) => d.engine !== 'postgresql').map((d) => d.path))[0];assert.ok(liveDb, 'the scenario declares a database');host.put(liveDb, 'SQLite format 3\0 live', { owner: 'ubuntu' });await host.exec.symlink('/opt/backups/old/hobostreamer.db', '/opt/backups/old/link.db');@@@ -89,4 +92,26 @@ runTests([assert.strictEqual(host.s3.objects.size, 0, `${file}: nothing is uploaded when any file is refused`);}}),++++ test('a PostgreSQL entry has no path, so it neither blocks archiving the file it replaced nor makes unsafeOut throw', async () => {++ const host = archiveHost();++ const raw = JSON.parse(host.read('/etc/openvibe/host.json'));++ // Media moved to PostgreSQL (C-89): its databases[] entry names a database in the cluster.++ raw.services.media.databases = [{ name: 'media', engine: 'postgresql', database: 'ov_media' }];++ const inv = normalise(raw);++ const frozen = '/opt/openvibe.media/data/media.db'; // Media's old SQLite file, frozen at the switch++ host.put(frozen, 'SQLite format 3\0 frozen', { owner: 'ubuntu' });++ assert.strictEqual(await refusal(host.exec, inv, frozen), null, 'the frozen SQLite file is archivable again');++++ // A database the inventory still declares as SQLite is refused, message unchanged.++ const live = inv.services.live.databases[0];++ assert.strictEqual(live.engine, 'sqlite');++ host.put(live.path, 'SQLite format 3\0 live', { owner: 'ubuntu' });++ assert.match(await refusal(host.exec, inv, live.path), /a database the inventory declares \(a live database\)/);++++ // unsafeOut only avoids database directories; a database in the cluster has none.++ let out;++ assert.doesNotThrow(() => { out = offsite.unsafeOut(inv, '/var/lib/openvibe-restore/media-2026'); });++ assert.strictEqual(out, null);++ }),]);@@@ -58,6 +58,19 @@ function dirsOf(host, dir) {return [...host.files.keys()].filter((k) => path.dirname(k) === dir).map((k) => path.basename(k)).sort();}++/** backupHost plus a PostgreSQL-only service "trade" and a fresh pgBackRest differential. */++function pgOffsiteHost({ dumpContent = null } = {}) {++ const host = backupHost();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.trade = { repo: '/opt/openvibe.trade', units: [], databases: [{ name: 'trade', engine: 'postgresql', database: 'ov_trade' }] };++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++ host.inv = normalise(doc);++ host.pgDatabases.add('ov_trade');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];++ if (dumpContent) host.dumpContent = dumpContent;++ return host;++}++function noSecrets(text, what) {for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);}@@@ -451,6 +464,29 @@ runTests([assert.match(none.out, /no off-host run of openre/);}),++ test('restore-download: a PostgreSQL .dump is verified with pg_restore --list; a corrupt one is refused', async () => {++ const host = pgOffsiteHost();++ const r = await host.cli('backup', '--all', '--offsite');++ assert.strictEqual(r.code, 0, r.out);++ assert.ok([...host.s3.objects.keys()].some((k) => k.endsWith('/trade/trade.dump.ovbk')), 'the pg_dump archive was uploaded');++++ const d = await host.cli('restore-download', 'trade', 'latest');++ assert.strictEqual(d.code, 0, d.out);++ const dest = '/var/lib/openvibe-restore/trade-20260922-120000/trade.dump';++ assert.ok(host.files.has(dest));++ assert.match(d.out, /pg_restore --list ok/);++ // --list only reads the file, so root checks the root-only copy itself, not as postgres.++ assert.ok(host.calls.some((c) => c.cmd === 'pg_restore' && c.args.includes('--list') && c.args.includes(dest) && c.as === null));++ assert.deepStrictEqual([host.files.get(dest).owner, host.files.get(dest).mode], ['root', 0o600]);++++ // A dump that is not a pg_dump archive: the manifest hashes match, so only pg_restore --list refuses it.++ const bad = pgOffsiteHost({ dumpContent: () => 'this is not a pg_dump archive' });++ await bad.cli('backup', '--all', '--offsite');++ const refused = await bad.cli('restore-download', 'trade', 'latest');++ assert.strictEqual(refused.code, 2, refused.out);++ assert.match(refused.out, /pg_restore --list .* failed|text format dump/);++ }),++// ── configuration ──────────────────────────────────────────────────────────test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {let host = backupHost({ envMode: 0o644 });nnew file mode 100644@@@ -0,0 +1,192 @@++'use strict';++/**++ * The PostgreSQL side of `ovhost backup`: every run verifies pgBackRest (a fresh stanza, and a WAL++ * archiver failed_count that has not risen), and a per-service pg_dump -Fc only on a slower cadence++ * (the weekly one, or --logical). SQLite services must be untouched. Against the fake host; no++ * pg_dump, pgbackrest or psql ever runs for real.++ */++const assert = require('assert');++const path = require('path');++const { scenario, test, runTests } = require('./helpers');++const { normalise } = require('../lib/inventory');++++const DAY = 86400000;++const STATE = '/var/lib/openvibe-host';++const PROM = '/var/lib/prometheus/node-exporter/openvibe_backup.prom';++++/** The standard fake host plus a PostgreSQL-only service "trade", and a fresh nightly differential. */++function pgHost() {++ const host = scenario();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.trade = { repo: '/opt/openvibe.trade', units: [], databases: [{ name: 'trade', engine: 'postgresql', database: 'ov_trade' }] };++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++ host.inv = normalise(doc);++ host.pgDatabases.add('ov_trade');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];++ return host;++}++++/** The newest record in a service's backup log. */++function lastRecord(host, id) {++ const lines = host.read(`${STATE}/backups/${id}.jsonl`).trim().split('\n');++ return JSON.parse(lines[lines.length - 1]);++}++++/** A prior GOOD logical-dump record `ageMs` old, as an earlier weekly run would have written. */++function seedDump(host, ageMs) {++ const dir = '/var/backups/openvibe/trade/20200101-000000';++ const rec = {++ service: 'trade', at: new Date(host.exec.now() - ageMs).toISOString(), reason: 'scheduled', engine: 'postgresql', ok: true, dir,++ files: [{ name: 'trade', source: 'ov_trade', dest: `${dir}/trade.dump`, file: 'trade.dump', check: 'pg_dump -Fc', bytes: 1 }],++ };++ host.put(`${STATE}/backups/trade.jsonl`, `${JSON.stringify(rec)}\n`);++}++++runTests([++ test('a fresh pgBackRest backup: verified, recorded on the run, and left for Prometheus', async () => {++ const host = pgHost();++ host.put('/var/lib/prometheus/node-exporter/.keep', '');++ const r = await host.cli('backup', '--all');++ assert.strictEqual(r.code, 0, r.out);++++ const rec = lastRecord(host, 'trade');++ assert.strictEqual(rec.engine, 'postgresql');++ assert.strictEqual(rec.verified.ok, true);++ assert.strictEqual(rec.verified.type, 'diff');++ assert.strictEqual(rec.verified.walOk, true);++ assert.strictEqual(rec.verified.stop, new Date((Math.floor(host.exec.now() / 1000) - 3600) * 1000).toISOString());++ // No prior record: a logical dump was due on the first run.++ assert.ok(rec.dir);++ assert.strictEqual(host.dumps.length, 1);++ assert.ok(host.files.has(`${rec.dir}/trade.dump`));++++ const prom = host.read(PROM);++ assert.match(prom, /^openvibe_backup_pg_verified\{service="trade"\} 1$/m);++ assert.match(prom, /^openvibe_backup_services\{status="verified"\} 0$/m);++ assert.match(prom, /^openvibe_pgbackrest_last_backup_age_seconds 3600$/m);++ }),++++ test('a stale pgBackRest backup: the run is failed, never skipped, and the summary says so', async () => {++ const host = pgHost();++ host.pgBackups = [{ type: 'full', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 27 * 3600 } }];++ const r = await host.cli('backup', '--all', '--json');++ assert.strictEqual(r.code, 2, r.out);++ const s = JSON.parse(r.out);++ assert.strictEqual(s.ok, false);++ const trade = s.services.find((x) => x.service === 'trade');++ assert.strictEqual(trade.status, 'failed', 'a verification failure is failed, not skipped');++ assert.strictEqual(trade.verified.ok, false);++ assert.match(trade.error, /stopped 27\.0h ago/);++++ const rec = lastRecord(host, 'trade');++ assert.strictEqual(rec.ok, false);++ assert.strictEqual(rec.verified.ok, false);++ assert.match(rec.verified.error, /stopped 27\.0h ago/);++ assert.ok(!('dir' in rec), 'a failed verification leaves no directory');++ assert.strictEqual(host.dumps.length, 0);++ }),++++ test('the WAL archiver failed_count: a rise fails the run, an unchanged value does not', async () => {++ const host = pgHost();++ const first = await host.cli('backup', 'trade');++ assert.strictEqual(first.code, 0, first.out);++ assert.strictEqual(lastRecord(host, 'trade').verified.walOk, true);++ const state = host.files.get(`${STATE}/pgbackrest-archiver.json`);++ assert.strictEqual(JSON.parse(state.content).failedCount, 0);++ assert.strictEqual(state.mode, 0o640, 'the recorded count is root-only 0640');++++ // The same value: still ok.++ host.advance(1000);++ const same = await host.cli('backup', 'trade');++ assert.strictEqual(same.code, 0, same.out);++ assert.strictEqual(lastRecord(host, 'trade').verified.walOk, true);++++ // It rose: failed, and the new value is remembered for the next run.++ host.advance(1000);++ host.pgArchiver.failed_count = 3;++ const risen = await host.cli('backup', 'trade');++ assert.strictEqual(risen.code, 2, risen.out);++ const rec = lastRecord(host, 'trade');++ assert.strictEqual(rec.ok, false);++ assert.strictEqual(rec.verified.walOk, false);++ assert.match(rec.verified.error, /failed_count rose from 0 to 3/);++ assert.strictEqual(JSON.parse(host.files.get(`${STATE}/pgbackrest-archiver.json`).content).failedCount, 3);++ }),++++ test('weekly cadence: no dump while the last good one is recent, a dump when it is stale, --logical forces one', async () => {++ const host = pgHost();++++ seedDump(host, 3 * DAY);++ const recent = await host.cli('backup', 'trade');++ assert.strictEqual(recent.code, 0, recent.out);++ assert.strictEqual(host.dumps.length, 0, 'no pg_dump three days after the last good one');++ const rec = lastRecord(host, 'trade');++ assert.strictEqual(rec.ok, true);++ assert.deepStrictEqual(rec.files, []);++ assert.strictEqual(rec.verified.ok, true);++ assert.ok(!('dir' in rec), 'a verification-only run writes no directory');++ assert.ok(!('empty' in rec), 'a verification-only run is never empty');++++ seedDump(host, 8 * DAY);++ host.advance(1000);++ const stale = await host.cli('backup', 'trade');++ assert.strictEqual(stale.code, 0, stale.out);++ assert.strictEqual(host.dumps.length, 1);++ assert.strictEqual(host.dumps[0].database, 'ov_trade');++ assert.strictEqual(host.dumps[0].as, 'postgres');++ const dumped = lastRecord(host, 'trade');++ assert.strictEqual(dumped.files.length, 1);++ assert.strictEqual(dumped.files[0].file, 'trade.dump');++ assert.strictEqual(dumped.files[0].check, 'pg_dump -Fc');++ assert.ok(host.files.has(`${dumped.dir}/trade.dump`));++++ // --logical ignores the cadence.++ seedDump(host, 3 * DAY);++ host.advance(1000);++ const forced = await host.cli('backup', 'trade', '--logical');++ assert.strictEqual(forced.code, 0, forced.out);++ assert.strictEqual(host.dumps.length, 2);++ }),++++ test('staging ownership: postgres owns the directory while pg_dump runs; SQLite stays the service user', async () => {++ const host = pgHost();++ const stages = [];++ const mkdir = host.exec.mkdir;++ host.exec.mkdir = async (p, o) => { if (String(p).startsWith('/var/backups/openvibe.staging/')) stages.push([path.basename(p), o && o.owner]); return mkdir(p, o); };++++ await host.cli('backup', 'trade');++ assert.deepStrictEqual(stages, [['trade-20260922-120000', 'postgres']], 'pg_dump must have a postgres-owned staging directory');++ assert.strictEqual(host.dumps[0].as, 'postgres');++++ await host.cli('backup', 'live');++ assert.strictEqual(stages[1][1], 'ubuntu', 'a SQLite service keeps the service user');++ assert.strictEqual(stages[1][0], 'live-20260922-120000');++++ // Root takes every copy back once the staging directory is back to root:root.++ const copy = host.files.get(`${lastRecord(host, 'trade').dir}/trade.dump`);++ assert.deepStrictEqual([copy.owner, copy.mode], ['root', 0o600]);++ }),++++ test('backup --all: SQLite and PostgreSQL services in one run; verified differs from ok and failed', async () => {++ const host = pgHost();++ seedDump(host, 3 * DAY); // the weekly dump is not due: trade is verification-only this run++ const r = await host.cli('backup', '--all', '--json');++ assert.strictEqual(r.code, 0, r.out);++ const s = JSON.parse(r.out);++ assert.strictEqual(s.ok, true);++ assert.deepStrictEqual(s.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'ok'], ['trade', 'verified']]);++ const trade = s.services.find((x) => x.service === 'trade');++ assert.strictEqual(trade.dir, null);++ assert.deepStrictEqual(trade.files, []);++ assert.strictEqual(trade.verified.ok, true);++++ // A stale stanza then fails that service; the others are unaffected and the run exits 2.++ host.advance(1000);++ host.pgBackups = [{ type: 'full', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 30 * 3600 } }];++ const bad = await host.cli('backup', '--all', '--json');++ assert.strictEqual(bad.code, 2);++ const s2 = JSON.parse(bad.out);++ assert.strictEqual(s2.ok, false);++ assert.deepStrictEqual(s2.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'ok'], ['trade', 'failed']]);++ }),++]);@@@ -18,7 +18,6 @@ const { normalise } = require('../lib/inventory');const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));const LIVE_DB = '/opt/openvibe.live/data/live.db';--const MEDIA_DB = '/opt/openvibe.media/data/media.db';const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));@@@ -95,7 +94,7 @@ async function mediaHost({ switchInCheckout = true } = {}) {doc.services.media = JSON.parse(JSON.stringify(EXAMPLE.services.media));host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });if (switchInCheckout) {-- host.put('/opt/openvibe.media/server/drill.js', "const enabled = parse(process.env.MEDIA_DRILL);\n", { owner: 'ubuntu' });++ host.put('/opt/openvibe.media/server/drill.js', "const enabled = parse(process.env.MEDIA_DRILL);\nif (!env.DATABASE_URL) out.push('DATABASE_URL');\n", { owner: 'ubuntu' });host.put('/opt/openvibe.media/server/public/routes.js', "if (drill.refuseBytes(res)) return;\n", { owner: 'ubuntu' });}host.put('/etc/systemd/system/openvibe-media.service', [@@@ -105,15 +104,17 @@ async function mediaHost({ switchInCheckout = true } = {}) {const unit = host.units.get('openvibe-media.service');host.alivePids.add(unit.mainPid);host.listeners.set(4100, [{ pid: unit.mainPid, process: 'node' }]);-- host.put(MEDIA_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });++ // Media is PostgreSQL now (host.example.json): the production database is in the cluster, and the++ // drill restores a logical dump into a scratch database instead of copying a file.++ host.pgDatabases.add('ov_media');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(Date.parse('2026-09-22T12:00:00Z') / 1000) } }];host.http.set('http://127.0.0.1:4100/api/ready', () => ({ status: 200, body: { ready: true } }));for (const tab of ['videos', 'clips']) host.http.set(`http://127.0.0.1:4100/browse?tab=${tab}`, () => ({ status: 200, body: BROWSE(tab) }));const counts = { media_objects: 6021, vods: 1300, clips: 4700, apps: 4 };-- host.sqliteHandler = (db, sql) => {-- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];-- if (/is_recording/.test(sql)) return [{ n: 0 }];++ host.psqlHandler = (database, sql) => {++ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];const m = /FROM "([a-z_]+)"/.exec(sql);-- return [{ n: m ? counts[m[1]] ?? 0 : 0 }];++ return [{ n: String(m ? counts[m[1]] ?? 0 : 0) }];};host.onSystemdRun = (spec) => {const envFile = spec.envFiles[spec.envFiles.length - 1];@@@ -137,7 +138,7 @@ runTests([const live = inv.services.live.drill;assert.strictEqual(live.supported, true);assert.strictEqual(live.port, 13000);-- assert.deepStrictEqual(live.databases, { live: { env: 'DB_PATH', dir: false } });++ assert.deepStrictEqual(live.databases, { live: { engine: 'sqlite', env: 'DB_PATH', dir: false } });assert.strictEqual(live.env.LIVE_DRILL, '1');assert.strictEqual(live.env.DATA_DIR, '{tmp}/data');assert.strictEqual(live.env.HOST, '127.0.0.1');@@@ -148,11 +149,11 @@ runTests([const media = inv.services.media.drill;assert.strictEqual(media.supported, true);assert.strictEqual(media.port, 14100);-- assert.deepStrictEqual(media.databases, { media: { env: 'DB_PATH', dir: false } });++ assert.deepStrictEqual(media.databases, { media: { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } });assert.strictEqual(media.env.MEDIA_DRILL, '1');for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'ASSETS_PATH', 'FILES_PATH', 'OBJECTS_PATH']) assert.match(media.env[k], /^\{tmp\}\/storage\//, k);assert.ok(!('THUMBNAILS_PATH' in media.env), '/browse lists production\'s thumbnail directory');-- assert.deepStrictEqual(media.requires.map((r) => r.contains), ['MEDIA_DRILL', 'refuseBytes']);++ assert.deepStrictEqual(media.requires.map((r) => r.contains), ['MEDIA_DRILL', 'DATABASE_URL', 'refuseBytes']);assert.deepStrictEqual(media.compare.map((c) => c.path), ['/browse?tab=videos', '/browse?tab=clips']);assert.deepStrictEqual(media.counts.map((c) => c.table), ['media_objects', 'vods', 'clips', 'apps']);}),@@@ -217,15 +218,24 @@ runTests([assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.media/server/index.js']);assert.ok(run.props.includes('SocketBindAllow=tcp:14100'));const env = envOf(host.drillEnv);-- assert.deepStrictEqual(env, {++ const { DATABASE_URL, DATABASE_DIRECT_URL, ...envRest } = env;++ assert.deepStrictEqual(envRest, {PORT: '14100', HOST: '127.0.0.1', MEDIA_DRILL: '1',VOD_PATH: `${rec.dir}/storage/vods`, CLIPS_PATH: `${rec.dir}/storage/clips`, PASTES_PATH: `${rec.dir}/storage/pastes`,ASSETS_PATH: `${rec.dir}/storage/assets`, FILES_PATH: `${rec.dir}/storage/files`, OBJECTS_PATH: `${rec.dir}/storage/objects`,-- EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', DB_PATH: `${rec.dir}/db/media.db`,++ EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', VALKEY_URL: 'redis://127.0.0.1:9/0',});-- assert.deepStrictEqual(host.drillDirAtStart, ['db', 'db/media.db', 'drill.env'], 'nothing but the copy and the override file; storage is never created');++ // The generated URL names the scratch database and role; both direct and pooled vars point at++ // the cluster port (5432), never PgBouncer.++ const scratch = rec.scratch.database;++ assert.match(DATABASE_URL, new RegExp(`^postgresql://${scratch}:[0-9a-f]+@127\\.0\\.0\\.1:5432/${scratch}$`));++ assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);++ assert.strictEqual(rec.databases[0].integrity, 'pg_restore --list ok');++ assert.deepStrictEqual(host.drillDirAtStart, ['db', 'drill.env', 'pg', 'pg/media.dump'], 'the dump copy and the override file; storage is never created');assert.deepStrictEqual(stateChanges(host), []);-- assert.strictEqual(host.read(MEDIA_DB), 'sqlite-production');++ assert.ok(host.pgDatabases.has('ov_media'), 'the production database is untouched');++ assert.ok(host.droppedDatabases.some((d) => d.name === scratch), 'the scratch database is dropped');++ assert.ok(!host.pgRoles.has(scratch), 'the scratch role is dropped');assert.ok(!(r.out + JSON.stringify(host.systemdRuns)).includes(SECRET));}),nnew file mode 100644@@@ -0,0 +1,327 @@++'use strict';++/**++ * ovhost drill for a PostgreSQL service (lib/dbengine.js): the artifact is a `.dump` in the backup,++ * not a SQLite file. The drill copies it into <tmp>/pg owned by postgres, verifies it with++ * `pg_restore --list`, creates a scratch database and login role, restores into them, points++ * drill.env at the scratch database (and at a closed Valkey), and always drops both again in a++ * finally. Covered: the generated URLs and their password, the refusal when an override is missing,++ * cleanup on failure, counts with tolerance, artifact selection and the free-space refusal.++ */++const assert = require('assert');++const path = require('path');++const { scenario, test, runTests } = require('./helpers');++++const ITEMS = '[{"id":"it_1","title":"hello"},{"id":"it_2","title":"world"}]';++// The production password in trade.env: the drill's generated password must never be this.++const PROD_PASSWORD = 'prod-password-must-not-be-reused';++++function tradeEntry(drillOverrides = {}) {++ return {++ repo: '/opt/openvibe.trade',++ units: ['openvibe-trade.service'],++ envFile: '/etc/openvibe/trade.env',++ port: 4500,++ ready: { url: 'http://127.0.0.1:4500/api/ready', timeoutSeconds: 30 },++ databases: [{ name: 'trade', engine: 'postgresql', database: 'ov_trade' }],++ drill: {++ port: 14500,++ databases: { trade: { url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } },++ env: { PORT: '{port}', HOST: '127.0.0.1', TRADE_DRILL: '1', VALKEY_URL: 'redis://127.0.0.1:9/0' },++ compare: [{ path: '/api/items?limit=5' }],++ counts: [{ db: 'trade', table: 'items' }],++ ...drillOverrides,++ },++ };++}++++/**++ * The standard fake host plus Trade in production, its cluster database and a good pgBackRest layer,++ * with one `ovhost backup trade` taken (a logical pg_dump). The drill instance's answers and the two++ * row counts are controlled through `host.drill`.++ */++async function pgScenario({ drill: drillOverrides, backup = true } = {}) {++ const host = scenario();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.trade = tradeEntry(drillOverrides);++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++++ const repo = host.createRepo('/opt/openvibe.trade', { owner: 'ubuntu' });++ const sha = repo.commit({ 'package.json': '{"name":"openvibe-trade"}', 'server/index.js': 'trade();' }, { message: 'initial' });++ repo.publish(sha);++ repo.checkout(sha);++ host.addUnit('openvibe-trade.service', { mainPid: 2222 });++ host.alivePids.add(2222);++ host.listeners.set(4500, [{ pid: 2222, process: 'node' }]);++ host.put('/etc/systemd/system/openvibe-trade.service', [++ '[Service]',++ 'User=ubuntu',++ 'WorkingDirectory=/opt/openvibe.trade',++ 'EnvironmentFile=/etc/openvibe/trade.env',++ 'ExecStart=/usr/bin/node server/index.js',++ '',++ ].join('\n'));++ // The production app's own connection strings: the drill must not let either reach the instance.++ host.put('/etc/openvibe/trade.env', `DATABASE_URL=postgresql://trade_app:${PROD_PASSWORD}@127.0.0.1:6432/ov_trade\nDATABASE_DIRECT_URL=postgresql://trade_app:${PROD_PASSWORD}@127.0.0.1:5432/ov_trade\nVALKEY_URL=redis://valkey-production:6379/7\n`, { mode: 0o600 });++++ // The cluster: the production database, and a healthy pgBackRest stanza so `ovhost backup` passes.++ host.pgDatabases.add('ov_trade');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(Date.parse('2026-09-22T12:00:00Z') / 1000) } }];++ host.drill = {++ productionCount: 500,++ restoredCount: 500,++ ready: () => ({ status: 200, body: { status: 'ready' } }),++ items: () => ({ status: 200, body: ITEMS }),++ };++ host.psqlHandler = (database, sql) => {++ if (/left\((datname|rolname)/.test(sql)) return undefined; // leftover-scratch lookups: the fake's catalog++ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];++ if (/FROM "([a-z_]+)"/.test(sql)) return [{ n: String(database === 'ov_trade' ? host.drill.productionCount : host.drill.restoredCount) }];++ return [{ n: '0' }];++ };++ host.http.set('http://127.0.0.1:4500/api/items?limit=5', () => ({ status: 200, body: ITEMS }));++ host.onSystemdRun = (spec) => {++ const envFile = spec.envFiles[spec.envFiles.length - 1];++ host.drillEnv = host.read(envFile);++ host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);++ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });++ host.http.set('http://127.0.0.1:14500/api/ready', alive(() => host.drill.ready()));++ host.http.set('http://127.0.0.1:14500/api/items?limit=5', alive(() => host.drill.items()));++ return undefined;++ };++++ if (backup) {++ const b = await host.cli('backup', 'trade', '--json');++ assert.strictEqual(b.code, 0, b.out);++ host.backupDir = JSON.parse(b.out).dir;++ }++ host.advance(60 * 1000);++ return host;++}++++const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));++const lastLog = (host) => JSON.parse(host.read('/var/lib/openvibe-host/drills/trade.jsonl').trim().split('\n').pop());++++function fsSnapshot(host) {++ const m = new Map();++ for (const [k, v] of host.files) m.set(k, `${v.type}:${v.owner}:${v.mode}:${v.type === 'file' ? String(v.content) : v.target || ''}`);++ return m;++}++++function changedPaths(before, after) {++ const out = [];++ for (const [k, v] of after) if (before.get(k) !== v) out.push(k);++ for (const k of before.keys()) if (!after.has(k)) out.push(k);++ return out.sort();++}++++runTests([++ test('leftovers of a killed drill are dropped first; restores skip GRANTs; scratch drops are forced', async () => {++ const host = await pgScenario();++ const old = 'ov_trade_drill_20260101-000000';++ host.pgDatabases.add(old);++ host.pgRoles.set(old, { password: 'x' });++ host.pgDatabases.add('ov_tradex'); // another name that only looks alike stays++ const callsBefore = host.calls.length;++ const r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 0, r.out);++ assert.ok(!host.pgDatabases.has(old), 'the leftover scratch database is dropped');++ assert.ok(!host.pgRoles.has(old), 'the leftover scratch role is dropped');++ assert.ok(host.pgDatabases.has('ov_trade') && host.pgDatabases.has('ov_tradex'), 'production and look-alike databases are untouched');++ const calls = host.calls.slice(callsBefore);++ const restore = calls.find((c) => c.cmd === 'pg_restore' && !c.args.includes('--list'));++ assert.ok(restore.args.includes('--no-privileges') && restore.args.includes('--no-owner'), restore.args.join(' '));++ for (const c of calls.filter((x) => x.cmd === 'dropdb')) assert.ok(c.args.includes('--force'), c.args.join(' '));++ }),++ test('full flow: pg_restore --list, a scratch database and role, drill.env pointed at them, cleanup', async () => {++ const host = await pgScenario();++ const callsBefore = host.calls.length;++ const r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 0, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));++ assert.strictEqual(rec.failure, null);++++ const tmp = rec.dir;++ assert.match(tmp, /^\/var\/lib\/openvibe-drills\/trade-\d{8}-\d{6}$/);++ assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.engine, d.integrity, d.copy]), [['trade', 'postgresql', 'pg_restore --list ok', `${tmp}/pg/trade.dump`]]);++ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 500, restored: 500, match: true }]);++ assert.strictEqual(rec.ready.ok, true);++ assert.strictEqual(rec.stop.ok, true);++++ // The scratch database and role share the derived name and are recorded on the drill record.++ const at = path.basename(tmp).slice('trade-'.length);++ const scratch = `ov_trade_drill_${at}`;++ assert.deepStrictEqual(rec.scratch, { database: scratch, role: scratch, dropped: true });++ assert.ok(host.restores.some((x) => x.database === scratch && x.role === scratch && x.file === `${tmp}/pg/trade.dump`));++ assert.ok(host.createdDatabases.some((c) => c.name === scratch && c.owner === scratch && c.as === 'postgres'));++ // The scratch database and role are dropped again; production's database is untouched.++ assert.ok(host.droppedDatabases.some((c) => c.name === scratch && c.as === 'postgres'));++ assert.ok(!host.pgRoles.has(scratch), 'the scratch role is gone');++ assert.ok(host.pgDatabases.has('ov_trade'), 'production database still there');++++ // drill.env points the instance at the scratch database over the direct port, and at a closed++ // Valkey from the inventory. The password is generated, not production's, and never in argv.++ const env = envOf(host.drillEnv);++ assert.strictEqual(env.VALKEY_URL, 'redis://127.0.0.1:9/0');++ const url = `postgresql://${scratch}:`;++ assert.ok(env.DATABASE_URL.startsWith(url), env.DATABASE_URL);++ assert.ok(env.DATABASE_URL.endsWith(`@127.0.0.1:5432/${scratch}`), env.DATABASE_URL);++ assert.strictEqual(env.DATABASE_DIRECT_URL, env.DATABASE_URL);++ const password = /^postgresql:\/\/[^:]+:([^@]+)@/.exec(env.DATABASE_URL)[1];++ assert.notStrictEqual(password, PROD_PASSWORD);++ assert.ok(!JSON.stringify(host.calls.slice(callsBefore)).includes(password), 'the generated password never reaches an argv');++ assert.ok(!host.drillEnv.includes(PROD_PASSWORD), 'the production password never reaches drill.env');++++ // The dump is copied as postgres (the backup is root-only, so postgres cannot read it directly).++ const cp = host.calls.slice(callsBefore).find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/pg/trade.dump`));++ assert.deepStrictEqual(cp.args, ['-o', 'postgres', '-m', '0600', '-T', '--', `${host.backupDir}/trade.dump`, `${tmp}/pg/trade.dump`]);++ assert.strictEqual(cp.privileged, true);++++ // The Markdown row names pg_restore --list, not pragma integrity_check.++ assert.match(r.out, /`pg_restore --list` \(trade\) = ok/);++ assert.strictEqual(rec.markdown, r.out.split('\n').pop());++ }),++++ test('refuses before anything is created when drill.env would leave a production value', async () => {++ const host = await pgScenario({ backup: false, drill: { env: { PORT: '{port}', HOST: '127.0.0.1', TRADE_DRILL: '1' } } });++ const before = fsSnapshot(host);++ const r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 1, r.out);++ assert.match(r.out, /VALKEY_URL/);++ assert.match(r.out, /production Valkey keyspace/);++ assert.strictEqual(host.createdDatabases.length, 0, 'nothing created');++ assert.strictEqual(host.dumps.length, 0, 'no pg_dump');++ assert.strictEqual(host.restores.length, 0, 'no pg_restore');++ assert.strictEqual(host.systemdRuns.length, 0, 'nothing started');++ assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');++ }),++++ test('refuses when a PostgreSQL database is not mapped in drill.databases (it would run on production)', async () => {++ const host = await pgScenario({ backup: false });++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.trade.databases.push({ name: 'ledger', engine: 'postgresql', database: 'ov_trade_ledger' });++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++ const before = fsSnapshot(host);++ const r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 1, r.out);++ assert.match(r.out, /must map every PostgreSQL database \(ledger is missing\)/);++ assert.strictEqual(host.createdDatabases.length + host.restores.length + host.systemdRuns.length, 0, 'nothing created or started');++ assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');++ }),++++ test('the drill directory is traversable by postgres (0711) and the dump copy stays 0600 postgres', async () => {++ const host = await pgScenario();++ const r = await host.cli('drill', 'trade', '--keep');++ assert.strictEqual(r.code, 0, r.out);++ const tmp = lastLog(host).dir;++ assert.strictEqual(host.files.get(tmp).mode, 0o711);++ const copy = host.files.get(`${tmp}/pg/trade.dump`);++ assert.strictEqual(copy.owner, 'postgres');++ assert.strictEqual(copy.mode, 0o600);++ }),++++ test('cleanup on failure: a failed comparison still drops the scratch database and role', async () => {++ const host = await pgScenario();++ host.drill.productionCount = 500;++ host.drill.restoredCount = 400;++ const r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.result, 'failed');++ assert.strictEqual(rec.failure.stage, 'compare');++ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 500, restored: 400, match: false }]);++ // The scratch database and role are dropped by the finally, even though the drill failed.++ assert.strictEqual(rec.scratch.dropped, true);++ assert.ok(host.droppedDatabases.some((c) => c.name === rec.scratch.database));++ assert.ok(!host.pgRoles.has(rec.scratch.role), 'the drill role was dropped');++ assert.ok(host.pgDatabases.has('ov_trade'), 'production database still there');++ }),++++ test('cleanup on failure: an instance that never becomes ready still drops the scratch', async () => {++ const host = await pgScenario();++ host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });++ const r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'ready');++ assert.ok(host.droppedDatabases.some((c) => c.name === rec.scratch.database));++ assert.ok(!host.pgRoles.has(rec.scratch.role));++ assert.ok(!host.files.has(rec.dir), 'the drill directory is removed');++ }),++++ test('counts: a match passes; 2 rows off fails at tolerance 0 but passes at 3', async () => {++ let host = await pgScenario({ drill: { countsTolerance: 0 } });++ host.drill.productionCount = 502;++ host.drill.restoredCount = 500;++ let r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 2, r.out);++ let rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'compare');++ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 502, restored: 500, match: false }], 'no tolerance field at 0');++++ host = await pgScenario({ drill: { countsTolerance: 3 } });++ host.drill.productionCount = 502;++ host.drill.restoredCount = 500;++ r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 0, r.out);++ rec = lastLog(host);++ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 502, restored: 500, match: true, tolerance: 3 }]);++ assert.match(rec.markdown, /items 502 = 500 \(±3\)/);++ assert.match(r.out, /items: production 502, restored 500 \(±3\)/);++ }),++++ test('artifact selection: --backup uses <dir>/<name>.dump; a verification-only record is skipped', async () => {++ // (a) --backup selects the .dump file in the named directory.++ let host = await pgScenario({ backup: false });++ host.ensureDir('/srv/old-backup');++ host.put('/srv/old-backup/trade.dump', 'pg_dump -Fc of ov_trade', { owner: 'root', mode: 0o600 });++ let r = await host.cli('drill', 'trade', '--backup', '/srv/old-backup', '--json');++ assert.strictEqual(r.code, 0, r.out);++ assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');++ assert.strictEqual(lastLog(host).databases[0].source, '/srv/old-backup/trade.dump');++++ // (b) a newer verification-only record (no dir, no dump) is not picked: the newest record that++ // actually holds a .dump entry wins.++ host = await pgScenario({ backup: false });++ const goodDir = '/var/backups/openvibe/trade/20260922-115500';++ host.ensureDir(goodDir);++ host.put(`${goodDir}/trade.dump`, 'pg_dump -Fc of ov_trade', { owner: 'root', mode: 0o600 });++ const good = { service: 'trade', at: '2026-09-22T11:55:00.000Z', engine: 'postgresql', ok: true, dir: goodDir, files: [{ name: 'trade', source: 'ov_trade', dest: `${goodDir}/trade.dump`, file: 'trade.dump', bytes: 23 }] };++ const verificationOnly = { service: 'trade', at: '2026-09-22T12:00:30.000Z', engine: 'postgresql', ok: true, files: [], verified: { type: 'diff', stop: '2026-09-22T02:20:11.000Z', walOk: true, ok: true } };++ host.put('/var/lib/openvibe-host/backups/trade.jsonl', `${JSON.stringify(good)}\n${JSON.stringify(verificationOnly)}\n`, { mode: 0o640 });++ r = await host.cli('drill', 'trade', '--json');++ assert.strictEqual(r.code, 0, r.out);++ assert.strictEqual(JSON.parse(r.out).backup, goodDir);++++ // (c) only a verification-only record: refused, as if there were no backup at all.++ host = await pgScenario({ backup: false });++ host.put('/var/lib/openvibe-host/backups/trade.jsonl', `${JSON.stringify(verificationOnly)}\n`, { mode: 0o640 });++ r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 1, r.out);++ assert.match(r.out, /no good ovhost backup recorded for trade/);++ }),++++ test('free-space: a drill root that does not exist yet is created before measuring, so the first drill runs', async () => {++ const host = await pgScenario();++ // A host that has never run a drill has no <drillDir>; statfs() works on existing paths only.++ assert.ok(!host.files.has('/var/lib/openvibe-drills'), 'the drill root does not exist yet');++ const r = await host.cli('drill', 'trade');++ assert.strictEqual(r.code, 0, r.out);++ assert.ok(host.files.has('/var/lib/openvibe-drills'), 'the drill root was created before measuring');++ assert.strictEqual(lastLog(host).result, 'passed');++ }),++++ test('free-space: refuses before restoring when the drill directory has almost no room', async () => {++ const host = await pgScenario({ backup: false });++ host.ensureDir('/srv/old-backup');++ host.put('/srv/old-backup/trade.dump', 'pg_dump -Fc of ov_trade', { owner: 'root', mode: 0o600 });++ host.statfsFree = 1024;++ const r = await host.cli('drill', 'trade', '--backup', '/srv/old-backup');++ assert.strictEqual(r.code, 2, r.out);++ assert.match(r.out, /not enough free space under \/var\/lib\/openvibe-drills/);++ assert.match(r.out, /1024 bytes free/);++ assert.strictEqual(host.restores.length, 0, 'no pg_restore ran');++ assert.strictEqual(host.dumps.length, 0, 'no pg_dump ran');++ assert.strictEqual(host.systemdRuns.length, 0);++ }),++]);@@@ -13,55 +13,55 @@ const { normalise } = require('../lib/inventory');const { compareBodies } = require('../lib/commands/drill');const DRILL_DIR = '/var/lib/openvibe-drills';--const PROD_DB = '/var/lib/openvibe-community/community.db';++const PROD_DB = '/var/lib/openvibe-tools/tools.db';const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';--function communityEntry(drillOverrides = {}) {++function toolsEntry(drillOverrides = {}) {return {-- repo: '/opt/openvibe.community',-- units: ['openvibe-community.service'],-- envFile: '/etc/openvibe/community.env',++ repo: '/opt/openvibe.tools',++ units: ['openvibe-tools.service'],++ envFile: '/etc/openvibe/tools.env',port: 4200,ready: { url: 'http://127.0.0.1:4200/api/ready', timeoutSeconds: 30 },-- databases: [{ name: 'community', path: PROD_DB }],++ databases: [{ name: 'tools', path: PROD_DB }],drill: {port: 14200,-- databases: { community: 'COMMUNITY_DB_PATH' },++ databases: { tools: 'TOOLS_DB_PATH' },env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words' },dirs: ['{tmp}/data'],compare: [{ path: '/api/pastes?limit=5' }, { path: '/api/v1/pulse?limit=5', ignore: ['generated_at'] }],-- counts: [{ db: 'community', table: 'pastes' }],++ counts: [{ db: 'tools', table: 'pastes' }],...drillOverrides,},};}/**-- * The standard fake host plus Community in production, with one `ovhost backup community` taken.++ * The standard fake host plus Tools in production, with one `ovhost backup tools` taken.* The drill instance's behaviour is controlled through `host.drill`.*/async function drillScenario({ drill: drillOverrides, backup = true } = {}) {const host = scenario();const doc = JSON.parse(host.read('/etc/openvibe/host.json'));-- doc.services.community = communityEntry(drillOverrides);++ doc.services.tools = toolsEntry(drillOverrides);host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });-- const repo = host.createRepo('/opt/openvibe.community', { owner: 'ubuntu' });-- const sha = repo.commit({ 'package.json': '{"name":"openvibe-community"}', 'server/index.js': 'community();' }, { message: 'initial' });++ const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });++ const sha = repo.commit({ 'package.json': '{"name":"openvibe-tools"}', 'server/index.js': 'tools();' }, { message: 'initial' });repo.publish(sha);repo.checkout(sha);-- host.addUnit('openvibe-community.service', { mainPid: 2222 });++ host.addUnit('openvibe-tools.service', { mainPid: 2222 });host.alivePids.add(2222);host.listeners.set(4200, [{ pid: 2222, process: 'node' }]);-- host.put('/etc/systemd/system/openvibe-community.service', [++ host.put('/etc/systemd/system/openvibe-tools.service', ['[Service]','User=ubuntu',-- 'WorkingDirectory=/opt/openvibe.community',++ 'WorkingDirectory=/opt/openvibe.tools','ExecStart=/usr/bin/node server/index.js',`Environment=NODE_ENV=production "SESSION_SECRET=${SECRET}-unit"`,'',].join('\n'));-- host.put('/etc/openvibe/community.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.community\n`, { mode: 0o600 });++ host.put('/etc/openvibe/tools.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.tools\n`, { mode: 0o600 });host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });host.http.set('http://127.0.0.1:4200/api/ready', () => ({ status: 200, body: { status: 'ready' } }));@@@ -101,7 +101,7 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {};if (backup) {-- const b = await host.cli('backup', 'community', '--json');++ const b = await host.cli('backup', 'tools', '--json');assert.strictEqual(b.code, 0, b.out);host.backupDir = JSON.parse(b.out).dir;}@@@ -201,31 +201,31 @@ function changedPaths(before, after) {}function lastLog(host) {-- const text = host.read('/var/lib/openvibe-host/drills/community.jsonl');++ const text = host.read('/var/lib/openvibe-host/drills/tools.jsonl');const lines = text.trim().split('\n');return JSON.parse(lines[lines.length - 1]);}const allowedWrite = (p, tmp) => p === DRILL_DIR || p === tmp || p.startsWith(`${tmp}/`)-- || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/community.jsonl'-- || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/community-drill.lock';++ || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/tools.jsonl'++ || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/tools-drill.lock';runTests([test('full flow: restore, integrity, sandboxed second instance, identical answers, stop by pid, cleanup, log row', async () => {const host = await drillScenario();const before = fsSnapshot(host);const callsBefore = host.calls.length;-- const r = await host.cli('drill', 'community');++ const r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 0, r.out);const rec = lastLog(host);assert.strictEqual(rec.result, 'passed');assert.strictEqual(rec.failure, null);assert.strictEqual(rec.backup, host.backupDir);const tmp = rec.dir;-- assert.match(tmp, /^\/var\/lib\/openvibe-drills\/community-\d{8}-\d{6}$/);-- assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['community', 'ok', `${tmp}/db/community.db`]]);++ assert.match(tmp, /^\/var\/lib\/openvibe-drills\/tools-\d{8}-\d{6}$/);++ assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['tools', 'ok', `${tmp}/db/tools.db`]]);assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/pastes?limit=5', true], ['/api/v1/pulse?limit=5', true]]);-- assert.deepStrictEqual(rec.counts, [{ db: 'community', table: 'pastes', production: 892, restored: 892, match: true }]);++ assert.deepStrictEqual(rec.counts, [{ db: 'tools', table: 'pastes', production: 892, restored: 892, match: true }]);assert.strictEqual(rec.ready.ok, true);assert.strictEqual(rec.stop.ok, true);assert.strictEqual(rec.stop.signal, 'SIGTERM');@@@ -233,20 +233,20 @@ runTests([const calls = host.calls.slice(callsBefore);// The copy is made as the service user; integrity and counts run as the service user.// Backups are root-only: root copies, and the copy belongs to the service user, 0600.-- const cp = calls.find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/db/community.db`));++ const cp = calls.find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/db/tools.db`));assert.strictEqual(cp.privileged, true);-- assert.deepStrictEqual(cp.args, ['-o', 'ubuntu', '-m', '0600', '-T', '--', `${host.backupDir}/community.db`, `${tmp}/db/community.db`]);++ assert.deepStrictEqual(cp.args, ['-o', 'ubuntu', '-m', '0600', '-T', '--', `${host.backupDir}/tools.db`, `${tmp}/db/tools.db`]);const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));-- assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/community.db`, 'ubuntu']);++ assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/tools.db`, 'ubuntu']);// systemd-run: production ExecStart/WorkingDirectory, production env file then the override file,// sandbox properties, loopback-only networking, the service user.assert.strictEqual(host.systemdRuns.length, 1);const run = host.systemdRuns[0];assert.strictEqual(run.uid, 'ubuntu');-- assert.strictEqual(run.cwd, '/opt/openvibe.community');++ assert.strictEqual(run.cwd, '/opt/openvibe.tools');assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js']);-- assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/community.env', `${tmp}/drill.env`]);++ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/tools.env', `${tmp}/drill.env`]);for (const p of ['ProtectSystem=strict', `ReadWritePaths=${tmp}`, 'PrivateTmp=yes', 'NoNewPrivileges=yes', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'SocketBindAllow=tcp:14200', 'SocketBindDeny=any', 'Environment=NODE_ENV=production']) {assert.ok(run.props.includes(p), `systemd-run -p ${p}`);}@@@ -259,7 +259,7 @@ runTests([assert.deepStrictEqual(kills.map((c) => c.args), [['SIGTERM', run.pid]]);assert.ok(!calls.some((c) => /pkill|killall/.test(c.cmd)));assert.ok(host.alivePids.has(2222), 'production still running');-- assert.strictEqual(host.units.get('openvibe-community.service').restarts, 0);++ assert.strictEqual(host.units.get('openvibe-tools.service').restarts, 0);assert.ok(!host.restarts().length, 'nothing restarted');// Only the drill directory (now removed), the lock (released) and the drill log changed.@@@ -267,33 +267,33 @@ runTests([const changed = changedPaths(before, after);for (const p of changed) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);assert.ok(!host.files.has(tmp), 'temp dir removed');-- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'), 'lock released');++ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'), 'lock released');// Secrets: nothing ovhost printed, logged or passed on carries a value.-- const everything = [r.out, host.read('/var/lib/openvibe-host/drills/community.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');++ const everything = [r.out, host.read('/var/lib/openvibe-host/drills/tools.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');assert.ok(!everything.includes(SECRET), 'a secret value leaked');-- assert.ok(!host.reads.includes('/etc/openvibe/community.env'), 'ovhost never reads the env file for a drill');++ assert.ok(!host.reads.includes('/etc/openvibe/tools.env'), 'ovhost never reads the env file for a drill');// Markdown row ready for docs/restore-drills.md.-- assert.match(r.out, /\| 2026-09-22 12:01 \| community \| `\/var\/backups\/openvibe\/community\/20260922-120000\/` \(`ovhost drill community`\) \| `pragma integrity_check` \(community\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);++ assert.match(r.out, /\| 2026-09-22 12:01 \| tools \| `\/var\/backups\/openvibe\/tools\/20260922-120000\/` \(`ovhost drill tools`\) \| `pragma integrity_check` \(tools\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);assert.strictEqual(rec.markdown, r.out.split('\n').pop());}),test('--keep leaves the drill directory; the override file holds inventory values only', async () => {const host = await drillScenario();const before = fsSnapshot(host);-- const r = await host.cli('drill', 'community', '--keep');++ const r = await host.cli('drill', 'tools', '--keep');assert.strictEqual(r.code, 0, r.out);const rec = lastLog(host);assert.strictEqual(rec.kept, true);const tmp = rec.dir;assert.strictEqual(host.files.get(tmp).owner, 'ubuntu');assert.strictEqual(host.files.get(tmp).mode, 0o700);-- assert.strictEqual(host.files.get(`${tmp}/db/community.db`).owner, 'ubuntu');-- assert.strictEqual(host.read(`${tmp}/db/community.db`), host.read(`${host.backupDir}/community.db`));++ assert.strictEqual(host.files.get(`${tmp}/db/tools.db`).owner, 'ubuntu');++ assert.strictEqual(host.read(`${tmp}/db/tools.db`), host.read(`${host.backupDir}/tools.db`));assert.strictEqual(host.files.get(`${tmp}/data`).owner, 'ubuntu');const env = host.read(`${tmp}/drill.env`);-- for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `COMMUNITY_DB_PATH=${tmp}/db/community.db`]) {++ for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `TOOLS_DB_PATH=${tmp}/db/tools.db`]) {assert.ok(env.split('\n').includes(line), `drill.env has ${line}`);}assert.ok(!env.includes(SECRET));@@@ -305,7 +305,7 @@ runTests([test('integrity failure: nothing is started, the copy is removed, the log says failed', async () => {const host = await drillScenario();host.drill.integrity = [{ integrity_check: '*** in database main ***' }, { integrity_check: 'Page 5: btree page has a bad cell' }];-- const r = await host.cli('drill', 'community');++ const r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 2, r.out);const rec = lastLog(host);assert.strictEqual(rec.result, 'failed');@@@ -319,7 +319,7 @@ runTests([test('readiness timeout: the instance is stopped by its pid and the directory removed', async () => {const host = await drillScenario();host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });-- const r = await host.cli('drill', 'community');++ const r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 2, r.out);const rec = lastLog(host);assert.strictEqual(rec.failure.stage, 'ready');@@@ -336,7 +336,7 @@ runTests([const host = await drillScenario();let probes = 0;host.drill.ready = () => { probes += 1; if (probes === 2) host.alivePids.delete(host.systemdRuns[0].pid); return { status: 503 }; };-- const r = await host.cli('drill', 'community');++ const r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 2, r.out);const rec = lastLog(host);assert.strictEqual(rec.failure.stage, 'ready');@@@ -350,7 +350,7 @@ runTests([host.drill.pastes = () => ({ status: 200, body: PASTES.replace('world', 'World') });host.drill.pulse = () => ({ status: 200, body: { items: [{ id: 2 }], generated_at: 'different, but ignored' } });host.drill.productionCount = 900;-- const r = await host.cli('drill', 'community');++ const r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 2, r.out);const rec = lastLog(host);assert.strictEqual(rec.failure.stage, 'compare');@@@ -360,7 +360,7 @@ runTests([assert.strictEqual(pulse.match, false);assert.match(pulse.detail, /\$\.items\[0\]\.id/);assert.ok(!/generated_at/.test(pulse.detail));-- assert.deepStrictEqual(rec.counts[0], { db: 'community', table: 'pastes', production: 900, restored: 892, match: false });++ assert.deepStrictEqual(rec.counts[0], { db: 'tools', table: 'pastes', production: 900, restored: 892, match: false });assert.match(rec.markdown, /`\/api\/pastes\?limit=5` differs \(bodies differ/);assert.match(rec.markdown, /pastes 900 ≠ 892 \| failed \|$/);// Still stopped and cleaned up.@@@ -371,14 +371,14 @@ runTests([test('acceptance: the restored instance alone answers each declared GET with its status and JSON keys (WS-S task 2)', async () => {const accept = [{ path: '/api/ready', keys: ['status'] }, { path: '/api/v1/pulse?limit=5', keys: ['items', 'generated_at'] }];let host = await drillScenario({ drill: { acceptance: accept } });-- let r = await host.cli('drill', 'community');++ let r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 0, r.out);let rec = lastLog(host);assert.deepStrictEqual(rec.acceptance, [{ path: '/api/ready', ok: true }, { path: '/api/v1/pulse?limit=5', ok: true }]);assert.match(rec.markdown, /acceptance `\/api\/ready`, `\/api\/v1\/pulse\?limit=5`/);host = await drillScenario({ drill: { acceptance: [{ path: '/api/v1/pulse?limit=5', keys: ['items', 'cursor'] }, { path: '/api/ready', status: 204 }] } });-- r = await host.cli('drill', 'community');++ r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 2, r.out);rec = lastLog(host);assert.strictEqual(rec.failure.stage, 'acceptance');@@@ -389,7 +389,7 @@ runTests([test('acceptance entries are validated in the inventory', async () => {const host = await drillScenario({ drill: { acceptance: [{ path: 'no-slash' }] }, backup: false });-- const r = await host.cli('drill', 'community');++ const r = await host.cli('drill', 'tools');assert.notStrictEqual(r.code, 0);assert.match(r.out, /acceptance\[0\]\.path must be a path/);}),@@@ -406,24 +406,24 @@ runTests([const host = await drillScenario();const before = fsSnapshot(host);host.drill.onStart = () => ({ code: 1, stderr: 'Failed to start transient service unit: Unit already exists.' });-- const r = await host.cli('drill', 'community');++ const r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 2, r.out);const rec = lastLog(host);assert.strictEqual(rec.failure.stage, 'start');assert.match(rec.failure.message, /systemd-run failed/);assert.ok(!host.files.has(rec.dir));-- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'));++ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'));for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, rec.dir), `unexpected write: ${p}`);// A second drill can run straight after.host.drill.onStart = null;host.advance(1000);-- assert.strictEqual((await host.cli('drill', 'community')).code, 0);++ assert.strictEqual((await host.cli('drill', 'tools')).code, 0);}),test('an instance that ignores SIGTERM gets SIGKILL; one that survives SIGKILL keeps its directory and fails', async () => {let host = await drillScenario();host.drill.ignoreTerm = true;-- let r = await host.cli('drill', 'community');++ let r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 0, r.out);let rec = lastLog(host);assert.strictEqual(rec.stop.signal, 'SIGKILL');@@@ -432,7 +432,7 @@ runTests([host = await drillScenario();host.drill.ignoreTerm = true;host.drill.ignoreKill = true;-- r = await host.cli('drill', 'community');++ r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 2, r.out);rec = lastLog(host);assert.strictEqual(rec.failure.stage, 'stop');@@@ -445,7 +445,7 @@ runTests([test('refusals: not root, port in use, unsupported service, no drill block, no backup, missing database copy', async () => {let host = await drillScenario();host.exec.isRoot = async () => false;-- let r = await host.cli('drill', 'community');++ let r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 1);assert.match(r.out, /must run as root/);assert.strictEqual(host.systemdRuns.length, 0);@@@ -453,14 +453,14 @@ runTests([host = await drillScenario();host.listeners.set(14200, [{ pid: 777, process: 'python3' }]);const before = fsSnapshot(host);-- r = await host.cli('drill', 'community');++ r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 1);assert.match(r.out, /port 14200 is already in use \(python3 pid 777\)/);assert.strictEqual(host.systemdRuns.length, 0);assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');host = await drillScenario({ drill: { supported: false, reason: 'workers have no off switch' } });-- r = await host.cli('drill', 'community');++ r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 1);assert.match(r.out, /does not support restore drills: workers have no off switch/);@@@ -470,20 +470,20 @@ runTests([assert.match(r.out, /media has no drill block/);host = await drillScenario({ backup: false });-- r = await host.cli('drill', 'community');++ r = await host.cli('drill', 'tools');assert.strictEqual(r.code, 1);-- assert.match(r.out, /no good ovhost backup recorded for community/);++ assert.match(r.out, /no good ovhost backup recorded for tools/);host = await drillScenario({ backup: false });host.ensureDir('/srv/old-backup');-- r = await host.cli('drill', 'community', '--backup', '/srv/old-backup');++ r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup');assert.strictEqual(r.code, 1);-- assert.match(r.out, /has no copy of the community database/);-- host.put('/srv/old-backup/community.db', 'older sqlite', { owner: 'ubuntu' });-- r = await host.cli('drill', 'community', '--backup', '/srv/old-backup', '--json');++ assert.match(r.out, /has no copy of the tools database/);++ host.put('/srv/old-backup/tools.db', 'older sqlite', { owner: 'ubuntu' });++ r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup', '--json');assert.strictEqual(r.code, 0, r.out);assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');-- r = await host.cli('drill', 'community', '--backup', 'relative/dir');++ r = await host.cli('drill', 'tools', '--backup', 'relative/dir');assert.strictEqual(r.code, 1);}),@@@ -554,18 +554,18 @@ runTests([test('inventory: drill blocks are validated', () => {const base = { owner: 'ubuntu' };-- const svc = (drill, extra = {}) => normalise({ services: { community: { ...base, ...communityEntry(), ...extra, drill: { ...communityEntry().drill, ...drill } } } });++ const svc = (drill, extra = {}) => normalise({ services: { tools: { ...base, ...toolsEntry(), ...extra, drill: { ...toolsEntry().drill, ...drill } } } });assert.throws(() => svc({ port: 4200 }), /must differ from the service's own port/);-- assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of community's databases/);++ assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of tools's databases/);assert.throws(() => svc({ env: { HOST: '127.0.0.1' } }), /\{port\}/);-- assert.throws(() => svc({ env: { PORT: '{port}', COMMUNITY_DB_PATH: '/x' } }), /already set by drill.databases/);++ assert.throws(() => svc({ env: { PORT: '{port}', TOOLS_DB_PATH: '/x' } }), /already set by drill.databases/);assert.throws(() => svc({ env: { PORT: '{port}\nEVIL=1' } }), /one-line/);-- assert.throws(() => svc({ dirs: ['/opt/openvibe.community/data'] }), /under \{tmp\}/);-- assert.throws(() => svc({ counts: [{ db: 'community', table: 'x; drop table y' }] }), /table name/);++ assert.throws(() => svc({ dirs: ['/opt/openvibe.tools/data'] }), /under \{tmp\}/);++ assert.throws(() => svc({ counts: [{ db: 'tools', table: 'x; drop table y' }] }), /table name/);assert.throws(() => svc({ compare: [{ path: 'http://evil/' }] }), /path/);assert.throws(() => svc({ supported: false }), /reason is required/);-- assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, community: { ...base, ...communityEntry() } } }), /14200 is a's port/);-- assert.strictEqual(svc({}).services.community.drill.ready, '/api/ready');++ assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, tools: { ...base, ...toolsEntry() } } }), /14200 is a's port/);++ assert.strictEqual(svc({}).services.tools.drill.ready, '/api/ready');}),test('host.example.json: every service with a database has a drill block; unsupported ones give a reason', () => {@@@ -1,9 +1,14 @@'use strict';/*** A fake host implementing the executor interface (lib/executor.js) in memory: a filesystem, git-- * repositories, systemd units, npm, nginx, ss, HTTP endpoints and SQLite. Nothing here touches the-- * real machine. Every run() call is recorded in `calls` so tests can assert what would have been-- * executed (and as whom).++ * repositories, systemd units, npm, nginx, ss, HTTP endpoints, SQLite and PostgreSQL. Nothing here++ * touches the real machine. Every run() call is recorded in `calls` so tests can assert what would++ * have been executed (and as whom).++ *++ * PostgreSQL is modelled the way lib/dbengine.js uses it: `exec.psql(database, sql)` answers rows++ * (queries only), and the cluster's own tools — pg_dump, pg_restore, createdb, dropdb, pgbackrest —++ * go through run() as the postgres OS user, which is also how the role password reaches psql++ * (on stdin, so it never appears in an argv this fake records).*/const crypto = require('crypto');const path = require('path');@@@ -26,6 +31,26 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stnpmRewritesLockfile: true,sqliteHandler: () => [{ n: 0 }],sqliteCalls: [],++ // ── PostgreSQL ──++ // The cluster's databases and roles, the archiver's failed WAL count, the stanza's backups++ // and its status. A test drives them directly (host.pgDatabases.add('ov_trade')) or through++ // the tools below, exactly as production would.++ pgDatabases: new Set(),++ pgRoles: new Map(), // role -> { password }++ pgArchiver: { failed_count: 0 },++ pgBackups: [], // [{ type, timestamp: { stop } }] (epoch seconds)++ pgStanzaStatus: { code: 0, message: 'ok' },++ pgbackrestCode: 0,++ pgbackrestStderr: '',++ psqlHandler: null, // (database, sql, as) -> rows | undefined (undefined: the engine defaults)++ psqlCalls: [],++ adminSql: [], // SQL sent to psql on stdin (role creation, where the password must not be argv)++ dumps: [], // { database, dest, as }++ restores: [], // { database, role, file, as }++ createdDatabases: [], // { name, owner, as }++ droppedDatabases: [], // { name, as }++ statfsFree: 100 * 1024 ** 3,++ statfsSize: 500 * 1024 ** 3,reads: [],onRestart: null, // (unit) -> voidonSystemdRun: null, // (spec) -> undefined | { code, stderr } ; spec = { unit, uid, cwd, props, argv, pid }@@@ -406,6 +431,66 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', streturn r ? { stdout: '', stderr: '', ...r } : { code: 0, stdout: '', stderr: '' };}case 'fuser': return { code: (host.openFiles || new Set()).has(path.resolve(args[args.length - 1])) ? 0 : 1, stdout: '', stderr: '' };++ // ── PostgreSQL tools (lib/dbengine.js runs each of these as the postgres OS user) ──++ case 'psql': {++ // Admin SQL on stdin (CREATE ROLE … LOGIN PASSWORD …, DROP ROLE …). Queries never come++ // this way: they go through exec.psql(). The password stays out of argv, as in life.++ const sql = String(opts.input || '');++ host.adminSql.push(sql);++ const create = /CREATE ROLE "([^"]+)" LOGIN PASSWORD '([^']*)'/.exec(sql);++ if (create) host.pgRoles.set(create[1], { password: create[2] });++ const drop = /DROP ROLE IF EXISTS "([^"]+)"/.exec(sql);++ if (drop) host.pgRoles.delete(drop[1]);++ if (host.onAdminSql) { const r = host.onAdminSql(sql, opts); if (r) return { stdout: '', stderr: '', ...r }; }++ return { code: 0, stdout: '', stderr: '' };++ }++ case 'createdb': {++ const name = args[args.length - 1];++ const owner = (args.find((a) => a.startsWith('--owner=')) || '').slice(8) || null;++ if (host.pgDatabases.has(name)) return { code: 1, stdout: '', stderr: `createdb: error: database creation failed: ERROR: database "${name}" already exists` };++ host.pgDatabases.add(name);++ host.createdDatabases.push({ name, owner, as: opts.as || null });++ return { code: 0, stdout: '', stderr: '' };++ }++ case 'dropdb': {++ const name = args[args.length - 1];++ host.pgDatabases.delete(name);++ host.droppedDatabases.push({ name, as: opts.as || null });++ return { code: 0, stdout: '', stderr: '' };++ }++ case 'pg_dump': {++ const dest = args[args.indexOf('-f') + 1];++ const database = args[args.indexOf('-d') + 1];++ if (!host.pgDatabases.has(database)) return { code: 1, stdout: '', stderr: `pg_dump: error: connection to server failed: FATAL: database "${database}" does not exist` };++ const parent = files.get(path.dirname(path.resolve(dest)));++ if ((opts.as || 'root') !== 'root' && parent && parent.owner !== (opts.as || 'root')) return { code: 1, stdout: '', stderr: `pg_dump: error: could not open output file "${dest}": Permission denied` };++ host.dumps.push({ database, dest, as: opts.as || null });++ put(dest, host.dumpContent ? host.dumpContent(database) : `pg_dump -Fc of ${database}`, { owner: opts.as || 'root', mode: 0o600 });++ return { code: 0, stdout: '', stderr: '' };++ }++ case 'pg_restore': {++ if (args.includes('--list')) {++ const e = get(args[args.length - 1]);++ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `pg_restore: error: could not open input file "${args[args.length - 1]}"` };++ // A text dump is what pg_restore refuses; the fake's archives all start with its marker.++ if (!String(e.content).startsWith('pg_dump -Fc')) return { code: 1, stdout: '', stderr: 'pg_restore: error: input file appears to be a text format dump. Please use psql.' };++ return { code: 0, stdout: ';\n; Archive created at 2026-09-28 03:30:00 UTC\n', stderr: '' };++ }++ const database = args[args.indexOf('-d') + 1];++ const file = args[args.length - 1];++ const role = (args.find((a) => a.startsWith('--role=')) || '').slice(7) || null;++ if (!host.pgDatabases.has(database)) return { code: 1, stdout: '', stderr: `pg_restore: error: connection to server failed: database "${database}" does not exist` };++ if (!get(file)) return { code: 1, stdout: '', stderr: `pg_restore: error: could not open input file "${file}"` };++ if (role && !host.pgRoles.has(role)) return { code: 1, stdout: '', stderr: `pg_restore: error: role "${role}" does not exist` };++ host.restores.push({ database, role, file, as: opts.as || null });++ return { code: 0, stdout: '', stderr: '' };++ }++ case 'pgbackrest': {++ if (!args.includes('info')) return { code: 1, stdout: '', stderr: `fake pgbackrest: ${args.join(' ')}` };++ if (host.pgbackrestCode !== 0) return { code: host.pgbackrestCode, stdout: '', stderr: host.pgbackrestStderr || 'pgbackrest: error' };++ const stanzas = host.pgbackrestStanzas ? host.pgbackrestStanzas() : [{ name: 'openvibe', status: host.pgStanzaStatus, backup: host.pgBackups }];++ return { code: 0, stdout: `${JSON.stringify(stanzas)}\n`, stderr: '' };++ }case 'systemd-run': return systemdRunCmd(args);case 'cp': {const src = args[args.length - 2];@@@ -491,6 +576,24 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stif (r && r.error) return { status: 0, error: r.error };return { status: r.status, body: typeof r.body === 'string' ? r.body : JSON.stringify(r.body || {}) };},++ /** Queries only, as the postgres OS user: the engine defaults answer `datname`/archiver asks. */++ async psql(database, sql, { as = 'postgres' } = {}) {++ host.psqlCalls.push({ database, sql, as });++ calls.push({ cmd: 'psql', args: [database, sql], as, privileged: false, cwd: null });++ if (host.psqlHandler) { const r = host.psqlHandler(database, sql, as); if (r !== undefined) return r; }++ const left = /FROM (pg_database|pg_roles) WHERE left\((?:datname|rolname), length\('([^']*)'\)\)/.exec(sql);++ if (left) return [...(left[1] === 'pg_database' ? host.pgDatabases : host.pgRoles.keys())].filter((n) => n.startsWith(left[2])).map((name) => ({ name }));++ const named = /datname = '([^']*)'/.exec(sql);++ if (/pg_database/i.test(sql) && named) return host.pgDatabases.has(named[1]) ? [{ present: '1' }] : [];++ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];++ return [{ n: 0 }];++ },++ async statfs(p) {++ // The real statfs() runs on an existing path only: a missing path is an ENOENT, not a++ // free-space figure. Mirror that so a space check before the directory exists is caught.++ if (!get(p)) { const err = new Error(`ENOENT: no such file or directory, statfs '${p}'`); err.code = 'ENOENT'; throw err; }++ return { free: host.statfsFree, size: host.statfsSize };++ },async sqlite(db, sql, { as } = {}) {host.sqliteCalls.push({ op: 'query', db, sql, as });calls.push({ cmd: 'sqlite', args: [db, sql], as: as || null, privileged: false, cwd: null });@@@ -7,6 +7,8 @@*/const assert = require('assert');const { scenario, test, runTests } = require('./helpers');++const { normalise } = require('../lib/inventory');++const nginx = require('../lib/nginx');function withHost(host, rows) {const doc = JSON.parse(host.read('/etc/openvibe/host.json'));@@@ -52,6 +54,35 @@ runTests([assert.strictEqual(host.read('/etc/nginx/sites-available/openvibe.host.conf'), null, 'render alone writes nothing');}),++ test('a PostgreSQL tenant database reads the verified domains over psql, never a SQLite file', async () => {++ const host = scenario();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ // Host moved to PostgreSQL (ADR-035): its tenants database is a database in the cluster, not++ // a file, and its service role's password is not ovhost's to read — so psql as postgres.++ doc.services.host = {++ repo: '/opt/openvibe.host',++ units: ['openvibe-host.service'],++ port: 4910,++ ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },++ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],++ nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: { engine: 'postgresql', database: 'ov_host' }, maxUpload: '110m' } },++ };++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++ host.psqlHandler = (database) => {++ assert.strictEqual(database, 'ov_host', 'the Host database in the cluster');++ return [{ hostname: 'www.tenant-a.org' }, { hostname: 'docs.tenant-b.net' }];++ };++ host.put('/etc/letsencrypt/live/www.tenant-a.org/fullchain.pem', 'CERT', { mode: 0o644 });++ const domains = await nginx.tenantDomains(host.exec, normalise(doc).services.host);++ assert.deepStrictEqual(domains, [{ hostname: 'www.tenant-a.org', hasCert: true }, { hostname: 'docs.tenant-b.net', hasCert: false }], 'the same [{ hostname, hasCert }] shape');++ assert.ok(host.psqlCalls.some((c) => c.database === 'ov_host' && c.sql === nginx.VERIFIED_SQL), 'the verified domains come from psql');++ assert.strictEqual(host.sqliteCalls.length, 0, 'a database in the cluster is never opened as a file');++ const r = await host.cli('nginx', 'tenants', 'host');++ assert.strictEqual(r.code, 0, r.out);++ assert.match(r.out, /server_name www\.tenant-a\.org;/);++ assert.match(r.out, /server_name docs\.tenant-b\.net;/);++ }),++test('hostile or misplaced values from the database never reach nginx config', async () => {const host = withHost(scenario(), [{ hostname: 'ok.example.org' }, { hostname: 'evil.org; } server { listen 80; root /; }' }, { hostname: 'x.openvibe.host' },@@@ -15,7 +15,6 @@ const { normalise } = require('../lib/inventory');const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));const RELEASE = '655b98a10aaa';const OLD = '0123456789ab';--const PROD_DB = '/var/lib/openre/openre.db';const WORKERS = [`openre-rtmp-ingest@${RELEASE}.service`, `openre-restream-worker@${RELEASE}.service`];/** The standard fake host plus OpenRe as on openvibe-ovh: release layout, two units, worker instances. */@@@ -37,7 +36,7 @@ async function openreHost({ workersRunning = true } = {}) {'EnvironmentFile=/etc/openvibe/openre.env','Environment=NODE_ENV=production','Environment=PORT=4500',-- `Environment=OPENRE_DB_PATH=${PROD_DB}`,++ `Environment=DATABASE_URL=postgresql://openre_app:${SECRET}@127.0.0.1:6432/ov_openre`,'Environment=PATH=/usr/local/bin:/usr/bin:/bin','ExecStart=/usr/bin/env node server/index.js','',@@@ -50,18 +49,23 @@ async function openreHost({ workersRunning = true } = {}) {host.addUnit(`openre-rtmp-ingest@${RELEASE}.service`, { mainPid: 3102, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });host.addUnit(`openre-restream-worker@${RELEASE}.service`, { mainPid: 3103, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });host.addUnit(`openre-rtmp-ingest@${OLD}.service`, { active: 'inactive', sub: 'dead', mainPid: 0 });-- host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });++ // openre is PostgreSQL: the production database and a healthy pgBackRest stanza so that++ // `ovhost backup openre` (and the drill's restore of its .dump) can succeed.++ host.pgDatabases.add('ov_openre');++ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(Date.parse('2026-09-22T11:00:00Z') / 1000) } }];const health = '{"status":"ok","service":"openre-api","version":"0.1.0","release":"dev"}';const robots = 'User-agent: *\nAllow: /$\n';host.http.set('http://127.0.0.1:4500/api/ready', () => ({ status: 200, body: { status: 'ready' } }));host.http.set('http://127.0.0.1:4500/api/health', () => ({ status: 200, body: health }));host.http.set('http://127.0.0.1:4500/robots.txt', () => ({ status: 200, body: robots }));++ // The API's own status probe counts protected sessions over psql; the drill counts the same++ // tables in the restored scratch database.const counts = { stream_definitions: 1, ingest_keys: 1, destinations: 0, migration_map: 0 };-- host.sqliteHandler = (db, sql) => {-- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];++ host.psqlHandler = (database, sql) => {++ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];const m = /FROM "([a-z_]+)"/.exec(sql);-- return [{ n: m ? counts[m[1]] ?? 0 : 0 }];++ return [{ n: String(m ? (counts[m[1]] || 0) : 0) }];};host.onSystemdRun = (spec) => {host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);@@@ -93,7 +97,7 @@ runTests([assert.strictEqual(o.drill.env.OPENRE_DRILL, '1');for (const k of ['EVENTS_URL', 'OV_OAUTH_CLIENT_SECRET', 'MEDIA_API_KEY']) assert.strictEqual(o.drill.env[k], '', k);assert.strictEqual(o.drill.env.MEDIA_URL, 'http://127.0.0.1:9');-- assert.deepStrictEqual(o.drill.databases, { openre: { env: 'OPENRE_DB_PATH', dir: false } });++ assert.deepStrictEqual(o.drill.databases, { openre: { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } });// every other service keeps an empty listassert.deepStrictEqual(inv.services.live.workerUnits, []);}),@@@ -175,12 +179,18 @@ runTests([assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/openre.env', `${rec.dir}/drill.env`]);for (const p of ['SocketBindAllow=tcp:14500', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);assert.ok(run.props.includes('Environment=PATH=/usr/local/bin:/usr/bin:/bin'), 'other unit Environment= is passed on');-- assert.ok(!run.props.some((p) => /OPENRE_DB_PATH|Environment=PORT=/.test(p)), 'the production database path and port from the unit never reach the drill instance');++ assert.ok(!run.props.some((p) => /Environment=DATABASE_URL=|Environment=PORT=/.test(p)), 'the production connection string and port from the unit never reach the drill instance');const env = Object.fromEntries(host.drillEnv.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));-- assert.deepStrictEqual(env, {++ const { DATABASE_URL, DATABASE_DIRECT_URL, ...envRest } = env;++ assert.deepStrictEqual(envRest, {PORT: '14500', HOST: '127.0.0.1', OPENRE_DRILL: '1', EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', MEDIA_API_KEY: '', OPENRE_RECORDING: 'off',-- OPENRE_DB_PATH: `${rec.dir}/db/openre.db`,++ VALKEY_URL: 'redis://127.0.0.1:9/0',});++ // The instance is pointed at the restored scratch database over loopback, never at production.++ const scratch = `ov_openre_drill_${path.basename(rec.dir).slice('openre-'.length)}`;++ assert.match(DATABASE_URL, new RegExp(`^postgresql://${scratch}:[0-9a-f]+@127\\.0\\.0\\.1:5432/${scratch}$`));++ assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);++ assert.strictEqual(rec.databases[0].integrity, 'pg_restore --list ok');const calls = host.calls.slice(callsBefore);assert.deepStrictEqual(calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', run.pid]], 'only the drill instance is signalled');@@@ -191,7 +201,7 @@ runTests([assert.ok(!calls.some((c) => c.args && c.args.includes(u) && c.args[0] !== 'show'), `${u} is only ever read`);}assert.ok(host.alivePids.has(3100) && host.alivePids.has(3101));-- assert.strictEqual(host.read(PROD_DB), 'sqlite-production');++ assert.ok(host.pgDatabases.has('ov_openre'), 'the production database is untouched');assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/openre.jsonl')).includes(SECRET));}),]);nnew file mode 100644@@@ -0,0 +1,117 @@++'use strict';++/**++ * The two-engine inventory (ADR-035): a `databases[]` entry is SQLite (the default, a file) or++ * PostgreSQL (a database in the cluster, named — its object carries NO `path`). The same engine++ * choice reaches the drill block ({ url, directUrl }), nginx tenants (read over psql, not a file)++ * and the protected `postgresql-count` probe. These are the normalised shapes the rest of ovhost++ * depends on, and every code path outside the engine's own code is written to assume no `.path`.++ */++const assert = require('assert');++const { test, runTests } = require('./helpers');++const { normalise } = require('../lib/inventory');++++const base = { owner: 'ubuntu', repo: '/opt/openvibe.trade' };++const withDatabases = (databases, extra = {}) => ({ services: { trade: { ...base, databases, ...extra } } });++const first = (doc) => normalise(doc).services.trade.databases[0];++++// A service whose single database is PostgreSQL and that drills it: the drill names the two env vars++// the drill instance must be pointed at, instead of a file/env pair.++function pgDrill(overrides = {}) {++ return {++ services: {++ trade: {++ ...base,++ databases: [{ engine: 'postgresql', database: 'ov_trade' }],++ drill: {++ port: 5555,++ databases: { trade: { url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } },++ env: { PORT: '{port}' },++ ready: '/api/ready',++ compare: ['/api/items'],++ ...overrides,++ },++ },++ },++ };++}++++runTests([++ test('a PostgreSQL entry normalises to { name, engine, database, role } and carries no path', () => {++ const db = first(withDatabases([{ engine: 'postgresql', database: 'ov_trade' }]));++ assert.deepStrictEqual(db, { name: 'trade', engine: 'postgresql', database: 'ov_trade', role: null });++ assert.ok(!('path' in db), 'a database in the cluster has no file, so no path key at all');++ }),++++ test('name defaults to the database minus its ov_ prefix; an explicit name is kept', () => {++ assert.strictEqual(first(withDatabases([{ engine: 'postgresql', database: 'ov_openre' }])).name, 'openre');++ assert.deepStrictEqual(first(withDatabases([{ name: 'ingest', engine: 'postgresql', database: 'ov_openre' }])), { name: 'ingest', engine: 'postgresql', database: 'ov_openre', role: null });++ }),++++ test('an optional role is accepted and validated', () => {++ assert.strictEqual(first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', role: 'ov_trade_app' }])).role, 'ov_trade_app');++ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', role: 'Trade' }])), /role must be a database role name/);++ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', role: '1bad' }])), /role must be a database role name/);++ }),++++ test('rejections name what is wrong with the entry', () => {++ // A path on a PostgreSQL entry: the database lives in the cluster, not at a file.++ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', path: '/x/trade.db' }])), /has no path/);++ // A database name that is not an owned ov_ database.++ for (const database of ['trade', 'ov-Trade', 'public']) {++ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database }])), /database must match/, database);++ }++ // An engine that is neither.++ assert.throws(() => first(withDatabases([{ engine: 'mysql', database: 'ov_trade' }])), /engine must be sqlite or postgresql/);++ // A SQLite entry cannot name a database, and must have a path.++ assert.throws(() => first(withDatabases([{ path: '/x/live.db', database: 'ov_live' }])), /a sqlite database is a path, not a database name/);++ assert.throws(() => first(withDatabases([{ name: 'live' }])), /path must be an absolute path/);++ }),++++ test('the SQLite default is unchanged', () => {++ assert.deepStrictEqual(first(withDatabases([{ name: 'live', path: '/x/live.db' }])), { name: 'live', engine: 'sqlite', path: '/x/live.db' });++ }),++++ test('drill.databases for a PostgreSQL database is { engine, url, directUrl }', () => {++ const d = normalise(pgDrill()).services.trade.drill;++ assert.deepStrictEqual(d.databases.trade, { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' });++ }),++++ test('drill.databases rejects the SQLite shape and bad env var names for a PostgreSQL database', () => {++ const drill = (databases) => pgDrill({ databases });++ assert.throws(() => normalise(drill({ trade: 'DATABASE_URL' })), /for a postgresql database/);++ assert.throws(() => normalise(drill({ trade: { url: 'DATABASE_URL' } })), /directUrl must be an env var name/);++ assert.throws(() => normalise(drill({ trade: { directUrl: 'DATABASE_DIRECT_URL' } })), /url must be an env var name/);++ assert.throws(() => normalise(drill({ trade: { url: '1BAD', directUrl: 'OTHER' } })), /url must be an env var name/);++ assert.throws(() => normalise(drill({ trade: { url: 'DATABASE_URL', directUrl: 'DATABASE_URL' } })), /url and directUrl must be different env vars/);++ }),++++ test('countsTolerance is a whole number >= 0, default 0', () => {++ assert.strictEqual(normalise(pgDrill()).services.trade.drill.countsTolerance, 0);++ assert.strictEqual(normalise(pgDrill({ countsTolerance: 25 })).services.trade.drill.countsTolerance, 25);++ assert.throws(() => normalise(pgDrill({ countsTolerance: -1 })), /countsTolerance must be a whole number of rows/);++ assert.throws(() => normalise(pgDrill({ countsTolerance: 1.5 })), /countsTolerance must be a whole number of rows/);++ }),++++ test('nginx.tenants.database accepts a path (SQLite) or a PostgreSQL database object', () => {++ const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } }).services.host.nginx.tenants.database;++ assert.deepStrictEqual(tenants('/var/lib/openvibe-host-api/host.db'), { engine: 'sqlite', path: '/var/lib/openvibe-host-api/host.db' });++ assert.deepStrictEqual(tenants({ engine: 'postgresql', database: 'ov_host' }), { engine: 'postgresql', database: 'ov_host' });++ }),++++ test('nginx.tenants.database rejects anything else', () => {++ const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } });++ for (const database of [{ engine: 'mysql', database: 'ov_host' }, { engine: 'postgresql', database: 'host' }, { database: 'ov_host' }, 42, []]) {++ assert.throws(() => tenants(database), /nginx\.tenants\.database must be a path or/, JSON.stringify(database));++ }++ }),++++ test('the postgresql-count probe validates its database and SQL, alone and inside a sum', () => {++ const probe = (protectedBlock) => normalise({ services: { openre: { ...base, protected: protectedBlock } } }).services.openre.protected;++ const valid = { kind: 'postgresql-count', database: 'ov_openre', sql: 'SELECT count(*) AS n FROM ingest_sessions', label: 'ingest sessions' };++ assert.deepStrictEqual(probe(valid), valid);++ assert.throws(() => probe({ ...valid, database: 'openre' }), /database must match/);++ assert.throws(() => probe({ ...valid, sql: 'DELETE FROM ingest_sessions' }), /sql must be a SELECT/);++ // A sum may hold a postgresql-count part (nested kinds only rule out a sum inside a sum).++ assert.doesNotThrow(() => probe({ kind: 'sum', probes: [valid] }));++ }),++]);@@@ -5,6 +5,7 @@const assert = require('assert');const { countProtected } = require('../lib/probes');const { normalise } = require('../lib/inventory');++const { createFakeHost } = require('./fake-host');const counts = { '/opt/t/apps/img/data/jobs.db': 2, '/opt/t/apps/audio/data/jobs.db': 0, '/opt/t/apps/docs/data/jobs.db': 1 };const exec = {@@@ -24,6 +25,26 @@ const svc = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { kassert.strictEqual(r.count, null, 'one part that cannot answer makes the total unknown, never a smaller number');assert.match(r.unknown, /^part 2: no such database/);++ // A postgresql-count part asks the cluster over psql, as the postgres OS user (peer auth); the++ // number it answers is the count, and a psql failure is unknown — never zero.++ const host = createFakeHost();++ host.addUnit('openvibe-tools.service');++ host.sqliteHandler = () => [{ n: 2 }];++ host.psqlHandler = (database, sql) => { assert.strictEqual(database, 'ov_tools'); assert.match(sql, /^SELECT/); return [{ n: 7 }]; };++ const pgPart = { kind: 'postgresql-count', database: 'ov_tools', sql: 'SELECT count(*) AS n FROM tool_jobs', label: 'running tool jobs' };++ assert.deepStrictEqual(await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart }), { count: 7, label: 'running tool jobs' });++ assert.deepStrictEqual(host.psqlCalls.at(-1), { database: 'ov_tools', sql: pgPart.sql, as: 'postgres' });++ const pgSum = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), pgPart] } };++ assert.strictEqual((await countProtected(host.exec, pgSum)).count, 9, 'a postgresql-count part adds to the sum');++ host.psqlHandler = () => { throw new Error('psql: could not connect to server'); };++ const down = await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart });++ assert.strictEqual(down.count, null, 'a probe that cannot ask is unknown');++ assert.notStrictEqual(down.count, 0, 'never read as zero');++ assert.match(down.unknown, /could not connect/);++ const downSum = await countProtected(host.exec, pgSum);++ assert.strictEqual(downSum.count, null);++ assert.match(downSum.unknown, /^part 2: psql: could not connect/);++// The inventory checks each part.const base = { owner: 'ubuntu', repo: '/opt/t', units: ['openvibe-tools.service'] };assert.doesNotThrow(() => normalise({ services: { tools: { ...base, protected: svc.protected } } }));
@@ -15,14 +15,14 @@ Every repository ships its own deploy script, and each one encodes rules learned
@@ -15,14 +15,14 @@ Every repository ships its own deploy script, and each one encodes rules learned
|---|---|---|
|---|---|---|
| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
| Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
| Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
-| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`sqlite-count`: `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only **as the service user**. |
+| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`sqlite-count` of a file, or `postgresql-count` of a database in the cluster: Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only **as the owning user** (the service user, or the postgres OS user for the cluster). |
| Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
| Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
| Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
| Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
| Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
| Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
| Build, install vhosts, `nginx -t`, reload | Sites `deploy.sh` | `build` hooks and `nginx.installOnDeploy`. Vhost installs are transactional: if `nginx -t` fails, the previous files come back and nginx is not reloaded. |
| Build, install vhosts, `nginx -t`, reload | Sites `deploy.sh` | `build` hooks and `nginx.installOnDeploy`. Vhost installs are transactional: if `nginx -t` fails, the previous files come back and nginx is not reloaded. |
| Build a release while the old one serves; switch a symlink; roll back to a release with its own `node_modules`; pid 1 must hold the socket | Live `deploy.sh` (release layout), OpenRe `deploy.sh release`/`api` | Strategy `release-layout` ([docs/deploy-strategies.md](docs/deploy-strategies.md)). The socket unit is restarted only when its unit file changed or systemd does not hold the port. |
| Build a release while the old one serves; switch a symlink; roll back to a release with its own `node_modules`; pid 1 must hold the socket | Live `deploy.sh` (release layout), OpenRe `deploy.sh release`/`api` | Strategy `release-layout` ([docs/deploy-strategies.md](docs/deploy-strategies.md)). The socket unit is restarted only when its unit file changed or systemd does not hold the port. |
| Per-app installs (not `apps/_shared`), jobs runtime and guard load before the restart, every `openvibe-tools*` unit | Tools `deploy.sh` | Strategy `multi-app`: `skipPackages`, `preflight`, `unitsMatch`, `ready.release`. |
| Per-app installs (not `apps/_shared`), jobs runtime and guard load before the restart, every `openvibe-tools*` unit | Tools `deploy.sh` | Strategy `multi-app`: `skipPackages`, `preflight`, `unitsMatch`, `ready.release`. |
-| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm, builds, SQLite queries and backups all run as the declared owner/service user (`runuser` from root, `sudo -u` otherwise). |
+| git as the checkout owner; never leave root-owned files under `/opt` | Community/Events/Tools checkouts | git, npm, builds, SQLite queries and backups all run as the declared owner/service user (`runuser` from root, `sudo -u` otherwise); PostgreSQL dumps and probes run as the postgres OS user. |
Stage A adds a few safety rules of its own:
Stage A adds a few safety rules of its own:
@@ -33,7 +33,7 @@ Stage A adds a few safety rules of its own:
@@ -33,7 +33,7 @@ Stage A adds a few safety rules of its own:
## Owns
## Owns
-- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled SQLite backups with retention and encrypted off-host copies, restore drills.
+- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, scheduled backups (online SQLite copies, pgBackRest verification and logical PostgreSQL dumps) with retention and encrypted off-host copies, restore drills.
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user.
- Host roles, provisioned from code so any machine can take them: the **data role** (`roles/data/`, ADR-035): PostgreSQL 18, PgBouncer, pgBackRest, Valkey and their exporters, plus `add-service.sh` giving each service its database, roles and Valkey user.
@@ -92,10 +92,11 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak
@@ -92,10 +92,11 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak
- The socket unit is **never stopped or restarted**, in any flow, except the release-layout rebind: only when its unit file changed or systemd does not hold the port (`test/strategy-release-layout.test.js`).
- The socket unit is **never stopped or restarted**, in any flow, except the release-layout rebind: only when its unit file changed or systemd does not hold the port (`test/strategy-release-layout.test.js`).
- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.
- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.
- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.
- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.
-- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok`. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.
+- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok` (a SQLite copy) or `pg_restore --list` on the `.dump` of a PostgreSQL service. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.
- **Backups:**
- **Backups:**
- `backup --all` backs up every service with databases. One failing service leaves the others backed up and exits 2.
- `backup --all` backs up every service with databases. One failing service leaves the others backed up and exits 2.
+ - A PostgreSQL service is verified, not copied: every run checks pgBackRest (newest backup within 26 h, stanza ok, `pg_stat_archiver.failed_count` not rising) and reports `verified`; a logical `pg_dump -Fc` per database is taken weekly or with `--logical`, and a **failed verification** is `failed`, never `skipped`.
- Retention keeps the newest good backup of each of the last 7 days and 4 ISO weeks. A service whose backups keep failing never loses its last good ones. Directories ovhost did not record are never touched.
- Retention keeps the newest good backup of each of the last 7 days and 4 ISO weeks. A service whose backups keep failing never loses its last good ones. Directories ovhost did not record are never touched.
- Every backup directory is root 0700 and every copy is root 0600, with no `-wal`/`-shm`. A copy that is a link is refused.
- Every backup directory is root 0700 and every copy is root 0600, with no `-wal`/`-shm`. A copy that is a link is refused.
- Off-host copies round-trip through AES-256-GCM. Tampering, truncation, swapped objects, a forged manifest and another key are all refused.
- Off-host copies round-trip through AES-256-GCM. Tampering, truncation, swapped objects, a forged manifest and another key are all refused.
@@ -124,7 +125,7 @@ ovhost nginx render <service> [--variant http|sse|websocket] [--manifest <file>]
@@ -124,7 +125,7 @@ ovhost nginx render <service> [--variant http|sse|websocket] [--manifest <file>]
ovhost nginx tenants <service> [--wildcard-cert <name|dir>] [--install] Stage B tenant vhosts
ovhost nginx tenants <service> [--wildcard-cert <name|dir>] [--install] Stage B tenant vhosts
ovhost snapshot <service> [--out <file>]
ovhost snapshot <service> [--out <file>]
ovhost backup <service>
ovhost backup <service>
-ovhost backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]
+ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]
ovhost offsite push [--run <run>] | offsite list [<service>] | offsite check
ovhost offsite push [--run <run>] | offsite list [<service>] | offsite check
ovhost archive push <file>... [--note <text>] | archive list | archive restore <stamp> <path> --out <dir>
ovhost archive push <file>... [--note <text>] | archive list | archive restore <stamp> <path> --out <dir>
ovhost restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
ovhost restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
@@ -163,10 +164,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
@@ -163,10 +164,11 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
`sudo ovhost backup --all --offsite` runs daily from `deploy/systemd/openvibe-backup.timer` (03:30 UTC). It works like this:
`sudo ovhost backup --all --offsite` runs daily from `deploy/systemd/openvibe-backup.timer` (03:30 UTC). It works like this:
- It backs up every service that declares databases. A failing service never stops the others; exit `2` if anything failed.
- It backs up every service that declares databases. A failing service never stops the others; exit `2` if anything failed.
+- A **SQLite** database is copied online (as the service user). A **PostgreSQL** database is verified, not copied: every run checks pgBackRest for the whole cluster and records `verified`, and a logical `pg_dump -Fc` per database is taken when the last good one is over 7 days old or on any run with `--logical`. A failed verification is `failed`, never `skipped`.
- It keeps the last 7 daily and 4 weekly good backups per service.
- It keeps the last 7 daily and 4 weekly good backups per service.
- It writes a JSON summary to `<stateDir>/backup-runs/<run>.json`.
- It writes a JSON summary to `<stateDir>/backup-runs/<run>.json`.
- It encrypts every copy on the host (AES-256-GCM, Node crypto, with a key that stays on the host) and uploads it to S3-compatible storage (Backblaze B2). Off-host copies are kept for 30 days.
- It encrypts every copy on the host (AES-256-GCM, Node crypto, with a key that stays on the host) and uploads it to S3-compatible storage (Backblaze B2). Off-host copies are kept for 30 days.
-- Backups are `root:root`: directories 0700, files 0600, no `-wal`/`-shm`.
+- Backups are `root:root`: directories 0700, files 0600, no `-wal`/`-shm`. PostgreSQL dumps are made as the postgres OS user over peer auth; ovhost reads no service env value.
`ovhost restore-download` fetches, verifies and decrypts a run into a new root-only directory, and never touches a live database. See [docs/backups.md](docs/backups.md) for the install steps, the env names (`/etc/openvibe/backup.env`), the encryption format and the restore procedure.
`ovhost restore-download` fetches, verifies and decrypts a run into a new root-only directory, and never touches a live database. See [docs/backups.md](docs/backups.md) for the install steps, the env names (`/etc/openvibe/backup.env`), the encryption format and the restore procedure.
@@ -177,21 +179,22 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
@@ -177,21 +179,22 @@ Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passe
Each inventory entry's `drill` block declares:
Each inventory entry's `drill` block declares:
- `port`: the service port + 10000 in the example;
- `port`: the service port + 10000 in the example;
-- `databases`: the env var that points the service at each restored copy;
+- `databases`: the env var that points the service at each restored copy, or `{ "url": "<NAME>", "directUrl": "<NAME>" }` for a PostgreSQL database (the drill creates a scratch database and role, restores the `.dump` into it, and sets both vars to it);
- `env`: overrides that turn side effects off. Values may use `{tmp}`, `{port}` and `{db:<name>}`;
- `env`: overrides that turn side effects off. Values may use `{tmp}`, `{port}` and `{db:<name>}`;
- `dirs` to create inside the drill directory;
- `dirs` to create inside the drill directory;
- `ready`: the readiness path;
- `ready`: the readiness path;
- `compare`: paths, plus volatile `ignore` keys where needed;
- `compare`: paths, plus volatile `ignore` keys where needed;
- `counts`: `{ db, table }` pairs;
- `counts`: `{ db, table }` pairs;
+- `countsTolerance` (default `0`): how many rows a busy table may differ by, because production keeps writing between the backup read and the drill;
- `supported: false` with a `reason` when a second instance cannot run without side effects.
- `supported: false` with a `reason` when a second instance cannot run without side effects.
-In `host.example.json`, 24 services have drill blocks, and 23 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`), `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.
+In `host.example.json`, 24 services have drill blocks, and 23 can be drilled. ai is marked unsupported, and the reason is in the file. live and media rely on their own drill switches (`LIVE_DRILL`, `MEDIA_DRILL`), which `requires` checks for in the deployed checkout. For a service with several units, `unit` (as tools does: only the docs app runs) or `command` (an absolute argv, as openre does: only `openre-api` runs) says which one process the drill starts. `productionPort` says which production port to compare against, `databases` entries can be data directories (`{ "env": "DATA_DIR", "dir": true }`) or a PostgreSQL database's two URL env vars, `bind` redirects paths a service opens relative to its checkout into the drill directory (`BindPaths=`, in the drill unit's own mount namespace), and `requires` refuses the drill unless the deployed checkout has each switch the overrides rely on. A PostgreSQL drill always overrides `DATABASE_URL`, `DATABASE_DIRECT_URL` and `VALKEY_URL` and drops its scratch database and role in a `finally` ([docs/restore-drills.md](docs/restore-drills.md#a-postgresql-drill)). The Tools and Games drills rely only on switches those repositories already have. [docs/restore-drills.md](docs/restore-drills.md) lists what each repository could add to make its drill stricter.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
### Inventory
### Inventory
-`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `sqlite-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRe ingest sessions, Tools running jobs), the `drain` policy, `databases`, `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRe's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
+`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `sqlite-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRe ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a SQLite file `{ "name", "path" }` or a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRe's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
@@ -46,6 +46,17 @@ groups:
@@ -46,6 +46,17 @@ groups:
labels: { severity: page }
labels: { severity: page }
annotations:
annotations:
summary: "No successful nightly backup for more than 30 hours (or no backup metrics at all)"
summary: "No successful nightly backup for more than 30 hours (or no backup metrics at all)"
+ # A service whose databases are gone (the 2026-09 SQLite→PostgreSQL switch archived the .db files
+ # and dropped the path from the inventory) is reported `skipped` by backup --all. openvibe_backup_last_run_ok
+ # stays 1 (a skip is not a failure) and no other backup alert fires, so production can sit with no backups
+ # at all. 30 h is longer than the 24 h between runs, so ONE skipped run never reaches `for` and TWO
+ # consecutive ones always do.
+ - alert: OpenVibeBackupSkipped
+ expr: openvibe_backup_services{status="skipped"} > 0
+ for: 30h
+ labels: { severity: ticket }
+ annotations:
+ summary: "A service has been skipped by two nightly backups in a row ({{ with query \"openvibe_backup_services{status='skipped'}\" }}{{ . | first | value }}{{ end }} service(s)): its databases are gone, so it has no backups"
# Production drift (OpenVibe.Network registry/deploy-drift.js, hourly against GitHub; WS-S task 7).
# Production drift (OpenVibe.Network registry/deploy-drift.js, hourly against GitHub; WS-S task 7).
# max_over_time: Network restarts drop the series for a minute, which reset `for` and made the alert
# max_over_time: Network restarts drop the series for a minute, which reset `for` and made the alert
# resolve and fire again every few hours (2026-09-28). Documentation-only drift is not counted.
# resolve and fire again every few hours (2026-09-28). Documentation-only drift is not counted.
@@ -149,12 +160,17 @@ groups:
@@ -149,12 +160,17 @@ groups:
labels: { severity: page }
labels: { severity: page }
annotations:
annotations:
summary: "WAL archiving to the pgBackRest repository is failing: point-in-time recovery is at risk"
summary: "WAL archiving to the pgBackRest repository is failing: point-in-time recovery is at risk"
+ # Cluster-wide pgBackRest staleness. openvibe_pgbackrest_last_backup_age_seconds is written by
+ # `ovhost backup --all` (the nightly verify pass reads the stanza and records the age of its newest
+ # backup), NOT by the pgbackrest-backup@ unit's own backup-metric.sh: that keeps writing
+ # openvibe_pgbackrest_last_success_timestamp_seconds, which stays. absent() covers "no backup --all
+ # run has written the age at all" — a host where the nightly verify never ran.
- alert: DataBackupStale
- alert: DataBackupStale
- expr: time() - max(openvibe_pgbackrest_last_success_timestamp_seconds) > 26 * 3600
+ expr: max(openvibe_pgbackrest_last_backup_age_seconds) > 30 * 3600 or absent(openvibe_pgbackrest_last_backup_age_seconds)
for: 30m
for: 30m
labels: { severity: ticket }
labels: { severity: ticket }
annotations:
annotations:
- summary: "No successful pgBackRest backup for more than 26 hours"
+ summary: "No pgBackRest backup of the cluster has stopped within 30 hours (point-in-time recovery is at risk)"
- alert: DataValkeyMemoryHigh
- alert: DataValkeyMemoryHigh
expr: redis_memory_used_bytes / clamp_min(redis_memory_max_bytes, 1) > 0.9
expr: redis_memory_used_bytes / clamp_min(redis_memory_max_bytes, 1) > 0.9
for: 15m
for: 15m
@@ -1,12 +1,13 @@
@@ -1,12 +1,13 @@
[Unit]
[Unit]
-Description=OpenVibe backups: every service's SQLite databases, then an encrypted off-host copy
+Description=OpenVibe backups: verify pgBackRest for the PostgreSQL services, dump SQLite databases and the scheduled logical PostgreSQL dumps, then an encrypted off-host copy
Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.md
Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/backups.md
After=network-online.target
After=network-online.target
Wants=network-online.target
Wants=network-online.target
[Service]
[Service]
Type=oneshot
Type=oneshot
-# Root: ovhost drops to each service user for the SQLite backup (runuser) and is the only reader of
+# Root: ovhost drops to each service user for the SQLite backup and to the postgres OS user for the
+# PostgreSQL dumps and the pgBackRest verification (runuser) and is the only reader of
# /etc/openvibe/backup.env, the encryption key and the root-only backups. The env file is read by
# /etc/openvibe/backup.env, the encryption key and the root-only backups. The env file is read by
# ovhost itself, not loaded into this unit's environment.
# ovhost itself, not loaded into this unit's environment.
User=root
User=root
@@ -18,7 +19,8 @@ UMask=0077
@@ -18,7 +19,8 @@ UMask=0077
Nice=10
Nice=10
IOSchedulingClass=best-effort
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOSchedulingPriority=7
-# media.db and live.db are the large ones; the SQLite worker's own limit is 6 h per database.
+# The PostgreSQL logical dumps are the large ones now (they run as the postgres OS user), and the
+# SQLite worker's own limit is 6 h per database; a pg_dump is given the same. 6 h covers a whole run.
TimeoutStartSec=6h
TimeoutStartSec=6h
PrivateTmp=true
PrivateTmp=true
NoNewPrivileges=true
NoNewPrivileges=true
@@ -2,8 +2,10 @@
@@ -2,8 +2,10 @@
`ovhost backup --all --offsite` runs once a day from a systemd timer. It backs up every database
`ovhost backup --all --offsite` runs once a day from a systemd timer. It backs up every database
the inventory declares, keeps a week of daily and a month of weekly copies on the host, and uploads
the inventory declares, keeps a week of daily and a month of weekly copies on the host, and uploads
-an encrypted copy of each run to S3-compatible object storage (Backblaze B2). This page covers what it
-does, how to install it, and how to restore from the off-host copy.
+an encrypted copy of each run to S3-compatible object storage (Backblaze B2). A SQLite database is
+copied; a PostgreSQL service is covered by the cluster's pgBackRest backups, which every run verifies,
+with a per-service logical dump on a slower cadence. This page covers what it does, how to install it,
+and how to restore from the off-host copy.
Restore drills (a second instance started from a backup and compared with production) are in
Restore drills (a second instance started from a backup and compared with production) are in
[restore-drills.md](restore-drills.md). A drill works on an off-host copy too: see
[restore-drills.md](restore-drills.md). A drill works on an off-host copy too: see
@@ -12,18 +14,22 @@ Restore drills (a second instance started from a backup and compared with produc
@@ -12,18 +14,22 @@ Restore drills (a second instance started from a backup and compared with produc
## What is backed up
## What is backed up
`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has
`ovhost backup --all` backs up every service in the inventory (`/etc/openvibe/host.json`) that has
-a `databases` list. With `host.example.json` that means:
-
-| Service | Databases |
-|---|---|
-| network | `network.db` |
-| live | `live.db` (includes the money ledger while `BILLING_AUTHORITY=live`), `rs-companion.db`. `data/analytics.db` is left out on purpose ([storage-state.md](storage-state.md): analytics, hard-coded path, not canonical). |
-| media | `media.db` (the media files themselves are in B2/R2, not covered here) |
-| tools | seven `apps/*/data/analytics.db`, and `jobs.db` for img, audio and docs |
-| games | `world.db` |
-| ai | `ai.db` |
-| openre | `openre.db` |
-| community, events, chat, billing, host, search, sources, wiki, blog, tips, vip, news, reviews, deals, coupons, trade, codes | one database each |
+a `databases` list. `databases[]` items are one of two **engines** ([db-inventory.md](db-inventory.md)):
+SQLite, a file the service user's worker copies; and PostgreSQL, a database in the host's cluster
+(ADR-035) that ovhost verifies through pgBackRest and periodically dumps. With `host.example.json`:
+
+| Service | Engine | Databases |
+|---|---|---|
+| network | SQLite | `network.db` |
+| chat | SQLite | `chat.db` |
+| tools | SQLite | seven `apps/*/data/analytics.db`, and `jobs.db` for img, audio and docs |
+| games | SQLite | `world.db` |
+| live | SQLite | `live.db` (includes the money ledger while `BILLING_AUTHORITY=live`), `rs-companion.db`. `data/analytics.db` is left out on purpose ([storage-state.md](storage-state.md): analytics, hard-coded path, not canonical). |
+| tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media, openre | PostgreSQL | one database each in the cluster, named `ov_<name>` (for example `ov_trade`); there is no local file. The media files themselves are in B2/R2, not covered here. |
+
+The 19 PostgreSQL services each run the every-night pgBackRest verification and a scheduled logical
+dump; the five SQLite services keep the online copy described below. [PostgreSQL
+services](#postgresql-services) covers what that means.
A database that does not exist yet (a service that is not deployed) is reported as `skipped` and
A database that does not exist yet (a service that is not deployed) is reported as `skipped` and
does not fail the run. Adding a database to the inventory is all it takes to include it.
does not fail the run. Adding a database to the inventory is all it takes to include it.
@@ -46,6 +52,10 @@ For each service, one after another:
@@ -46,6 +52,10 @@ For each service, one after another:
`ok: false` when any database failed.
`ok: false` when any database failed.
5. **Prune** (see [Retention](#retention)).
5. **Prune** (see [Retention](#retention)).
+Steps 1-3 are the **SQLite** path. A PostgreSQL service does not copy a file at all: every run first
+verifies pgBackRest, and a logical dump is taken only when the last good one is old enough (see
+[PostgreSQL services](#postgresql-services)). `drill` uses the same `.dump` artifacts.
+
**Permissions.** Backups hold user data. Every directory under `/var/backups/openvibe` is
**Permissions.** Backups hold user data. Every directory under `/var/backups/openvibe` is
`root:root 0700` and every copy is `root:root 0600`, with no `-wal`/`-shm` next to it. Directories left
`root:root 0700` and every copy is `root:root 0600`, with no `-wal`/`-shm` next to it. Directories left
by older versions (`ubuntu`, 0750) are locked down on the next run. The staging parent
by older versions (`ubuntu`, 0750) are locked down on the next run. The staging parent
@@ -55,21 +65,85 @@ by older versions (`ubuntu`, 0750) are locked down on the next run. The staging
@@ -55,21 +65,85 @@ by older versions (`ubuntu`, 0750) are locked down on the next run. The staging
service never stops the other services. The run's exit code is `2` when anything failed and `0`
service never stops the other services. The run's exit code is `2` when anything failed and `0`
otherwise. If a service fails halfway (one of two databases), the good copy is kept but the backup is
otherwise. If a service fails halfway (one of two databases), the good copy is kept but the backup is
marked `ok: false`. Such a backup is never picked by a drill, and it is pruned once a newer good backup
marked `ok: false`. Such a backup is never picked by a drill, and it is pruned once a newer good backup
-exists.
+exists. A PostgreSQL service whose pgBackRest verification fails is reported `failed`, **never
+`skipped`**: a broken backup layer is a failure, not an absent database.
**Summary.** Each run writes `/var/lib/openvibe-host/backup-runs/<run>.json` (`root`, 0640). It holds
**Summary.** Each run writes `/var/lib/openvibe-host/backup-runs/<run>.json` (`root`, 0640). It holds
per-service status, files, sizes, errors, what was pruned, and the off-host result: objects, bytes,
per-service status, files, sizes, errors, what was pruned, and the off-host result: objects, bytes,
-failures, and the key id. The summary never holds a secret. The run id is the start time in UTC, for
-example `20260924-033412`.
+failures, and the key id. A PostgreSQL service that only verified pgBackRest is reported `verified`.
+The summary never holds a secret. The run id is the start time in UTC, for example `20260924-033412`.
```
```
sudo ovhost backup --all # local only
sudo ovhost backup --all # local only
sudo ovhost backup --all --offsite # what the timer runs
sudo ovhost backup --all --offsite # what the timer runs
sudo ovhost backup --all --json # the summary on stdout
sudo ovhost backup --all --json # the summary on stdout
+sudo ovhost backup --all --logical # force the logical PostgreSQL dumps too
sudo ovhost backup --all --keep-daily 14 --keep-weekly 8
sudo ovhost backup --all --keep-daily 14 --keep-weekly 8
sudo ovhost backup --all --no-prune
sudo ovhost backup --all --no-prune
```
```
+## PostgreSQL services
+
+A PostgreSQL service's data does not live in a file ovhost can copy. It lives in the host's cluster, so
+ovhost does not make the durability layer — [pgBackRest](#pgbackrest-is-the-durability-layer) already
+does — it **verifies** that layer on every run, and takes a per-service logical dump on a slower cadence
+for restores and drills. Everything runs as the `postgres` OS user over peer auth
+(`local postgres peer` in `roles/data/files/pg_hba.conf`), so ovhost still reads no service env value.
+
+**Every night: verify pgBackRest.** Before anything else, the run asks pgBackRest about the whole-cluster
+stanza (`runuser -u postgres -- pgbackrest --stanza=openvibe info --output=json`). Two conditions must
+hold:
+
+- the stanza's newest backup **stopped within 26 hours** and the stanza **status is ok** (newest first,
+ whatever its type);
+- `pg_stat_archiver.failed_count` has **not increased** since the last run. The last value ovhost saw is
+ kept in `<stateDir>/pgbackrest-archiver.json`, and each run compares against it, so a WAL segment that
+ failed to archive is caught even though it is not visible in a single night's snapshot.
+
+The result is recorded per service in `<stateDir>/backups/<svc>.jsonl` as `verified: { type, stop, walOk }`.
+A **failed verification** makes that service's status `failed` — never `skipped` — so `ovhost backup --all`
+exits `2`, `openvibe_backup_last_run_ok` goes to `0` and `OpenVibeBackupFailed` pages. A run where every
+PostgreSQL service only verified is `verified` in the summary.
+
+**Weekly, or on demand: the logical dump.** A `pg_dump -Fc` of each of the service's databases is taken
+when the last good dump is older than **7 days**, or on any run with `--logical`. It lands in the existing
+layout, one file per database:
+
+```
+/var/backups/openvibe/<service>/<stamp>/<name>.dump # pg_dump -Fc, owned root:root 0600
+```
+
+The off-site code then encrypts and uploads each `.dump` exactly as it does a `.db` copy, so a `.dump`
+reaches the bucket as `<service>/<name>.dump.ovbk` and is restored by `restore-download` unchanged.
+`pg_restore --list` proves a `.dump` is a readable archive.
+
+**Metrics and alerts.** The run rewrites `/var/lib/prometheus/node-exporter/openvibe_backup.prom` for
+node_exporter's textfile collector:
+
+- `openvibe_backup_pg_verified{service}` — `1` when that PostgreSQL service's last verification passed,
+ `0` when it failed;
+- `openvibe_pgbackrest_last_backup_age_seconds` — the age of the stanza's newest backup (gauge, seconds);
+- `openvibe_backup_services{status}` gains the value `verified` (a service with a verification-only run).
+
+`deploy/prometheus/openvibe-rules.yml` reads those:
+
+- `DataBackupStale` fires when `openvibe_pgbackrest_last_backup_age_seconds` is over **30 hours**, or the
+ metric is absent (no run has written it). It is the cluster staleness alert; the per-service
+ `openvibe_backup_pg_verified` says which service's verification failed.
+- `OpenVibeBackupSkipped` fires when a service has been `skipped` for **30 hours** — two nightly runs in a
+ row. A service whose databases are gone (for example one whose SQLite file was archived after the
+ 2026-09 switch to PostgreSQL) is `skipped`, `openvibe_backup_last_run_ok` stays `1` and nothing else
+ fires, so this is what catches "production with no backups". The rules live in
+ [`deploy/prometheus/openvibe-rules.yml`](../deploy/prometheus/openvibe-rules.yml).
+
+### pgBackRest is the durability layer
+
+It backs up the whole cluster — encrypted, off-host, with point-in-time recovery: a full backup every
+Sunday, a differential nightly, and the WAL stream. The per-service `.dump` files are **not** the
+durability layer; they are smaller, service-scoped, portable restore units (a logical archive survives a
+PostgreSQL major-version change, and it is what the drills restore and compare). Losing the cluster is
+recovered from pgBackRest; losing one service's rows, or drilling a service, uses its `.dump`.
+
## Retention
## Retention
**On the host** (per service). Kept: the newest good backup of each of the last **7 days** that have
**On the host** (per service). Kept: the newest good backup of each of the last **7 days** that have
@@ -93,7 +167,8 @@ To leave deletion to a bucket lifecycle rule instead, set `BACKUP_OFFSITE_PRUNE=
@@ -93,7 +167,8 @@ To leave deletion to a bucket lifecycle rule instead, set `BACKUP_OFFSITE_PRUNE=
### Layout
### Layout
```
```
-s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.db.ovbk
+s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.db.ovbk # a SQLite copy
+s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/<service>/<name>.dump.ovbk # a PostgreSQL dump
s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/manifest.json
s3://<BACKUP_S3_BUCKET>/<BACKUP_S3_PREFIX>/<host>/<run>/manifest.json
```
```
@@ -228,8 +303,11 @@ sudo ls -t /var/lib/openvibe-host/backup-runs | head -1
@@ -228,8 +303,11 @@ sudo ls -t /var/lib/openvibe-host/backup-runs | head -1
sudo jq '{ok, services: [.services[] | {service, status, error}], offsite: {ok: .offsite.ok, failures: .offsite.failures}}' /var/lib/openvibe-host/backup-runs/<run>.json
sudo jq '{ok, services: [.services[] | {service, status, error}], offsite: {ok: .offsite.ok, failures: .offsite.failures}}' /var/lib/openvibe-host/backup-runs/<run>.json
```
```
-Nothing alerts on a failed run yet. The unit fails (exit 2) and the summary says `"ok": false`. Hook
-an `OnFailure=` unit or the monitoring stack to that when one exists.
+The monitoring stack alerts on it (all in `deploy/prometheus/openvibe-rules.yml`):
+`OpenVibeBackupFailed` pages when the last run failed, `OpenVibeBackupMissed` when no run has succeeded
+for 30 hours, `OpenVibeBackupSkipped` when a service has no databases left to back up, and
+`DataBackupStale` when pgBackRest itself is stale. The unit also fails (exit 2) and the summary says
+`"ok": false`.
### Bucket (Backblaze B2)
### Bucket (Backblaze B2)
@@ -280,10 +358,10 @@ For each file, the command:
@@ -280,10 +358,10 @@ For each file, the command:
- checks the manifest's HMAC;
- checks the manifest's HMAC;
- streams the object through GCM decryption;
- streams the object through GCM decryption;
- checks both SHA-256 hashes against the manifest;
- checks both SHA-256 hashes against the manifest;
-- runs `PRAGMA quick_check` on the result.
+- runs `PRAGMA quick_check` on the result of a `.db` copy, or `pg_restore --list` on a `.dump`.
Any failure removes that file and exits non-zero. The output lists each file with its size, its hash,
Any failure removes that file and exits non-zero. The output lists each file with its size, its hash,
-`quick_check ok` and the production path it came from.
+its check and the production path it came from.
### 3. Put it in place (manual, one service at a time)
### 3. Put it in place (manual, one service at a time)
@@ -343,10 +421,11 @@ sudo rm -rf /var/lib/openvibe-restore/drill-community
@@ -343,10 +421,11 @@ sudo rm -rf /var/lib/openvibe-restore/drill-community
```
```
The drill:
The drill:
-- copies the databases into its sandbox as the service user (`install -o <user> -m 0600`);
-- runs `integrity_check`;
+- restores each database. A SQLite copy goes into the sandbox as the service user (`install -o <user>
+ -m 0600`) and is checked with `integrity_check`; a PostgreSQL `.dump` is restored with `pg_restore`
+ into a scratch database created for the drill ([restore-drills.md](restore-drills.md));
- starts a sandboxed second instance and compares it with production;
- starts a sandboxed second instance and compares it with production;
-- stops it and cleans up.
+- stops it and cleans up (a PostgreSQL drill always drops its scratch database and role).
Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
@@ -369,7 +448,7 @@ Do this at least once after installing, and again after any change to the key or
@@ -369,7 +448,7 @@ Do this at least once after installing, and again after any change to the key or
| Command | What it does | Exit |
| Command | What it does | Exit |
|---|---|---|
|---|---|---|
-| `ovhost backup --all [--offsite] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases; prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
+| `ovhost backup --all [--offsite] [--logical] [--no-prune] [--keep-daily n] [--keep-weekly n]` | back up every service with databases (verify pgBackRest, dump SQLite, dump PostgreSQL when due or `--logical`); prune; summary; optionally upload | 0 all ok · 1 lock/usage · 2 any failure |
| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
| `ovhost offsite push [--run <run>]` | upload a run (default: the latest) again, for example after a failed upload | 0 · 1 config · 2 failure |
| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
| `ovhost offsite list [<service>] [--from-host <h>]` | runs off-host, newest first | 0 · 1 config · 2 bucket error |
| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
| `ovhost offsite check` | config, key and bucket access; uploads nothing | 0 · 1 config · 2 bucket error |
@@ -163,6 +163,14 @@ Each service that moves keeps its SQLite file read-only for the 7-day N-1 window
@@ -163,6 +163,14 @@ Each service that moves keeps its SQLite file read-only for the 7-day N-1 window
| C-89 media | Media's SQLite file after the switch to PostgreSQL (2026-09-28 22:54 UTC, release `e3f2239`; import verified, counts and checksums as the rehearsal on a production copy: 3,173 media objects, 2,998 locations, 1,021 variants, 1,321 relationships, 792 vods, 324 clips, 891 pastes, 291 jobs, 789 outbox events, 29 tables with rows). The switch waited for recordings to be idle (switch-service.sh `019d853`). Also `media.pre-postgres-20260928T225438Z.db` | `/opt/openvibe.media/data/media.db` (0400) | Media | 7 days without a rollback: `ovhost archive push` both files, then delete | 2026-10-05 | `sudo ovhost rollback media` to `47b21bd` (reads the SQLite file; objects, jobs and views written since the switch would be lost, so export them first) |
| C-89 media | Media's SQLite file after the switch to PostgreSQL (2026-09-28 22:54 UTC, release `e3f2239`; import verified, counts and checksums as the rehearsal on a production copy: 3,173 media objects, 2,998 locations, 1,021 variants, 1,321 relationships, 792 vods, 324 clips, 891 pastes, 291 jobs, 789 outbox events, 29 tables with rows). The switch waited for recordings to be idle (switch-service.sh `019d853`). Also `media.pre-postgres-20260928T225438Z.db` | `/opt/openvibe.media/data/media.db` (0400) | Media | 7 days without a rollback: `ovhost archive push` both files, then delete | 2026-10-05 | `sudo ovhost rollback media` to `47b21bd` (reads the SQLite file; objects, jobs and views written since the switch would be lost, so export them first) |
| C-89 openre | OpenRe.Stream's SQLite file after the switch to PostgreSQL (2026-09-28 23:32 UTC, release `405682c`; import verified: 1 stream definition, 1 key, 1 session and its 4 transitions, 10 worker rows, 2 outbox events; every OpenRe unit stopped for the import through `SWITCH_UNITS`; worker generation 6 started after). Also `openre.pre-postgres-20260928T233208Z.db` | `/var/lib/openre/openre.db` (0400) | OpenRe | 7 days without a rollback: `ovhost archive push` both files, then delete | 2026-10-05 | `sudo ovhost rollback openre` to `d9f4240` and `deploy.sh workers` (reads the SQLite file; definitions and keys created since the switch would be lost) |
| C-89 openre | OpenRe.Stream's SQLite file after the switch to PostgreSQL (2026-09-28 23:32 UTC, release `405682c`; import verified: 1 stream definition, 1 key, 1 session and its 4 transitions, 10 worker rows, 2 outbox events; every OpenRe unit stopped for the import through `SWITCH_UNITS`; worker generation 6 started after). Also `openre.pre-postgres-20260928T233208Z.db` | `/var/lib/openre/openre.db` (0400) | OpenRe | 7 days without a rollback: `ovhost archive push` both files, then delete | 2026-10-05 | `sudo ovhost rollback openre` to `d9f4240` and `deploy.sh workers` (reads the SQLite file; definitions and keys created since the switch would be lost) |
+**Archive step unblocked (2026-09-28).** `ovhost archive push` refuses a path the inventory declares as a
+live SQLite database (`lib/archive.js`: it refuses a file that is any non-postgresql `databases[].path`).
+Until the switch, that rule covered these frozen `.db` files, so the "archive both files, then delete"
+step of C-89 could not run. The 19 services above now declare `engine: postgresql` with no `path`, so their
+sqlite entries no longer name a file and archive push accepts the frozen `.db` files. The archive-then-delete
+step of each C-89 row can proceed once its 7-day window closes; the removal condition and target of every
+row stand unchanged.
+
## Kept by design (not shims)
## Kept by design (not shims)
These match the search terms but are permanent. They are listed so the next review does not re-open them.
These match the search terms but are permanent. They are listed so the next review does not re-open them.
@@ -1,6 +1,51 @@
@@ -1,6 +1,51 @@
-# Database files on the host (WS-S task 6, hazard H14)
+# Databases: the inventory model, and the files left on the host (WS-S task 6, hazard H14)
-Inventory of 2026-09-26: every SQLite file under /opt, /var/lib, /root, /home and /srv that no process had open. 42 open files (the live databases) are not listed. Paths, sizes and dates only; nothing was read.
+`host.example.json` declares each service's databases as a `databases[]` list. An item is a union of two
+**engines** ([lib/inventory.js](../lib/inventory.js) normalises them, [lib/dbengine.js](../lib/dbengine.js)
+holds the per-engine helpers):
+
+- **SQLite** — a file the service user's worker copies:
+ ```json
+ { "name": "live", "path": "/opt/openvibe.live/data/live.db" }
+ ```
+- **PostgreSQL** — a database in the host's cluster, named, with **no `path`**:
+ ```json
+ { "name": "trade", "engine": "postgresql", "database": "ov_trade" }
+ ```
+
+`engine` defaults to `sqlite`. A PostgreSQL `database` must match `/^ov_[a-z0-9_]+$/`; a PostgreSQL entry
+has no `path`, and a SQLite entry has no `database`. The data lives in the cluster, so ovhost never opens
+it by copy: pgBackRest backs the whole cluster up, the nightly run verifies that, and a per-service
+logical `.dump` feeds the drills ([backups.md](backups.md#postgresql-services)).
+
+**Which engine.** With `host.example.json`:
+
+| Engine | Services |
+|---|---|
+| PostgreSQL | tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media, openre (19) |
+| SQLite | network, chat, tools, games, live (5) |
+
+**The two engines appear in four more places in the inventory:**
+
+- `drill.databases`. A SQLite database keeps a string (the env var that points the service at the restored
+ file) or `{ "env": "DATA_DIR", "dir": true }` for a service that takes a data directory; a PostgreSQL
+ database is `{ "trade": { "url": "DATABASE_URL", "directUrl": "DATABASE_DIRECT_URL" } }`, and the drill
+ creates a scratch database and role and points both vars at it ([restore-drills.md](restore-drills.md)).
+- `drill.countsTolerance` — a whole number of rows, default `0`. Production keeps writing between the read
+ that makes the backup and the dump, so a busy table may differ by up to this many rows without failing
+ the drill.
+- `nginx.tenants.database` — the Stage B tenant database (the Host API is on PostgreSQL): a SQLite path
+ keeps its string form, and PostgreSQL is the object `{ "engine": "postgresql", "database": "ov_host" }`.
+- the `protected` probe kind `postgresql-count` — `{ "kind": "postgresql-count", "database": "ov_openre",
+ "sql": "SELECT count(*) …", "label": "…" }`: a read-only SELECT against a database in the cluster, run as
+ the postgres OS user. OpenRe's protected probe (its active ingest sessions) is one, on `ov_openre`. The
+ other kinds are `http-json-count`, `sqlite-count` and `sum` ([lib/probes.js](../lib/probes.js)).
+
+---
+
+The rest of this page is the 2026-09-26 sweep of SQLite **files**: every file under /opt, /var/lib, /root,
+/home and /srv that no process had open. 42 open files (the live databases) are not listed. Paths, sizes
+and dates only; nothing was read. A PostgreSQL service has no row here: its rows are in the cluster.
`ovhost archive push <file>...` encrypts files with the backup key and copies them to `<BACKUP_S3_PREFIX>-archive/<host>/<stamp>/`, outside the backup runs so retention never prunes them; `ovhost archive list` and `ovhost archive restore <stamp> <path> --out <dir>` read them back. It never deletes a local file.
`ovhost archive push <file>...` encrypts files with the backup key and copies them to `<BACKUP_S3_PREFIX>-archive/<host>/<stamp>/`, outside the backup runs so retention never prunes them; `ovhost archive list` and `ovhost archive restore <stamp> <path> --out <dir>` read them back. It never deletes a local file.
@@ -90,7 +90,8 @@ block (see `host.example.json`):
@@ -90,7 +90,8 @@ block (see `host.example.json`):
Backups are root-only (directories 0700, files 0600, see [backups.md](backups.md)), so root makes
Backups are root-only (directories 0700, files 0600, see [backups.md](backups.md)), so root makes
the copy with `install -o <service user> -m 0600`, and the copy belongs to the service user.
the copy with `install -o <service user> -m 0600`, and the copy belongs to the service user.
`PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;
`PRAGMA integrity_check`, run as the service user, must return `ok` for each copy;
- otherwise the drill fails before anything starts.
+ otherwise the drill fails before anything starts. A PostgreSQL service restores its `.dump` into a
+ scratch database instead — see [A PostgreSQL drill](#a-postgresql-drill).
2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
unit's non-secret `Environment=` values and the production env file. After that it loads
unit's non-secret `Environment=` values and the production env file. After that it loads
@@ -124,6 +125,58 @@ cleaned up.
@@ -124,6 +125,58 @@ cleaned up.
A drill compares the backup with production *now*. If writes landed after the backup, row counts and
A drill compares the backup with production *now*. If writes landed after the backup, row counts and
list endpoints differ, and the drill reports that honestly. Take the backup right before the drill.
list endpoints differ, and the drill reports that honestly. Take the backup right before the drill.
+### A PostgreSQL drill
+
+Step 1 above is the SQLite path. A service whose `databases[]` entry is `engine: postgresql`
+([db-inventory.md](db-inventory.md)) restores a **logical dump** instead of a file copy:
+
+1. **Restore.** The drill takes the service's newest `<name>.dump` (`pg_dump -Fc`, from
+ `<stateDir>/backups/<service>.jsonl`, or the `--backup <dir>` directory) and creates a scratch
+ database `ov_<id>_drill_<stamp>` owned by a **freshly created login role** `ov_<id>_drill_<stamp>`,
+ whose password is generated with `crypto.randomBytes` and never printed. It then runs
+ `pg_restore --no-owner --no-privileges --role=<drill role>` into that database: `--no-owner` because the
+ archive names the production role, `--no-privileges` so none of the archive's GRANTs hands a production
+ role anything, `--role` so every object ends up owned by the scratch role the drilled instance connects
+ as. Before restoring, scratch databases and roles an earlier, killed drill of the same service left
+ behind (`ov_<id>_drill_*`) are dropped, and every drop uses `dropdb --force`.
+2. **Start.** `drill.env` overrides the three values the drilled release would otherwise take from
+ production:
+ - `DATABASE_URL` and `DATABASE_DIRECT_URL` — the scratch database on `127.0.0.1:5432`;
+ - `VALKEY_URL=redis://127.0.0.1:9/0` — a closed port.
+
+ The drill **refuses to start** if any of the three would keep its production value. They are not
+ optional, because the sandbox runs the release's real boot path:
+ - **`DATABASE_DIRECT_URL` un-overridden would migrate production.** The release runs its migrations
+ from the direct URL at boot, and a drill is deliberately pointed at a copy precisely so that boot
+ can run them there.
+ - **`VALKEY_URL` un-neutralised would write into the production keyspace.** The drill sandbox allows
+ loopback (services need Network's public keys there), so a Valkey URL left at production would let
+ the second instance write locks, caches and queues into production's keyspace.
+ - **`DATABASE_URL` un-overridden would serve production data.** The drilled instance would read and
+ write the live database instead of the restored copy, and every comparison would be vacuous.
+3. **Compare.** `counts` are read from production and from the scratch database over `psql` (the SQLite
+ drills compare files). `countsTolerance` in the drill block (default `0`) is the number of rows that
+ may differ: production keeps writing between the read that makes the backup and the read that counts
+ it, so a busy table is allowed to drift by that many rows instead of failing the drill.
+
+The connection is `127.0.0.1:5432`, the cluster itself, and deliberately **not PgBouncer's 6432**:
+PgBouncer pools the production app roles and databases, and the scratch database and its generated role
+exist only in the cluster, not in PgBouncer's configuration. Connecting directly to the port the
+`pg_restore` used is also what guarantees the instance reaches the scratch database and not a pool that
+happens to route back to production.
+
+Free space is checked before the restore: the drill refuses below **2x the dump size + 1 GB**, because
+`pg_restore` needs room for the restored tables on top of the archive and the database's own WAL and
+temporary files.
+
+The scratch database and its role are dropped in a `finally`, together with stopping the instance, on
+success and on failure. A PostgreSQL drill always cleans up after itself; `--keep` keeps the drill
+directory (the copy, `drill.env`, `drill.log`) but never the scratch database or role.
+
+For such a service the **Checks** column in the table above reports `pg_restore --list` (that the dump is
+a readable archive) rather than `pragma integrity_check`, and each count is read over `psql` from
+production and from the scratch database.
+
### Services
### Services
Supported with declared overrides (the table above records which have passed on the host):
Supported with declared overrides (the table above records which have passed on the host):
@@ -238,7 +291,9 @@ games as of 22 Sep) have entries built from their repositories' `deploy/` direct
@@ -238,7 +291,9 @@ games as of 22 Sep) have entries built from their repositories' `deploy/` direct
port.
port.
- `databases.<name>: { "env": "DATA_DIR", "dir": true }`, or `"dir": "{tmp}/<name>"`: the copy keeps
- `databases.<name>: { "env": "DATA_DIR", "dir": true }`, or `"dir": "{tmp}/<name>"`: the copy keeps
its production file name inside that directory, and the env var, if one is given, points at the
its production file name inside that directory, and the env var, if one is given, points at the
- directory.
+ directory. For a `postgresql` database the entry is instead `{ "url": "<NAME>", "directUrl":
+ "<NAME>" }`: the two env vars that point the drilled service at a database URL, which the drill sets
+ to the scratch database (see [A PostgreSQL drill](#a-postgresql-drill)).
- `bind: [{ "from": "{tmp}/…", "to": "<path inside the checkout>" }]`: `BindPaths=` in the drill
- `bind: [{ "from": "{tmp}/…", "to": "<path inside the checkout>" }]`: `BindPaths=` in the drill
unit's own mount namespace. Use it for files a service opens relative to its checkout with no env
unit's own mount namespace. Use it for files a service opens relative to its checkout with no env
override: the restored copy appears at the production path for the drill instance only. Tools yt,
override: the restored copy appears at the production path for the drill instance only. Tools yt,
@@ -15,8 +15,31 @@ ADR-007 (`OpenVibe.Contracts/docs/adr/ADR-007-data-ownership.md`) decides:
@@ -15,8 +15,31 @@ ADR-007 (`OpenVibe.Contracts/docs/adr/ADR-007-data-ownership.md`) decides:
host, or sizes SQLite handles poorly.
host, or sizes SQLite handles poorly.
- Money paths use SERIALIZABLE transactions and row locks when they move.
- Money paths use SERIALIZABLE transactions and row locks when they move.
-Every service below runs on SQLite (better-sqlite3), with one database per service. No repository has a
-PostgreSQL driver or config. **This page is the Wave 22 review of that state.**
+As reviewed on 2026-09-23 every service ran on SQLite (better-sqlite3), with one database per service.
+Since 2026-09-28 nineteen of them run on the host's **PostgreSQL cluster** (ADR-035), and five stay on
+SQLite: network, chat, tools, games and live. The per-service table below is the Wave 22 review of the
+SQLite state; where a service switched, its SQLite file is now frozen read-only for the 7-day rollback
+window (compatibility-register.md, C-89), and the live rows are in the cluster. **This page remains the
+Wave 22 review of that state.**
+
+## Where the data lives now
+
+- **PostgreSQL services** (tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes,
+ vip, ai, host, events, community, billing, media, openre) keep their rows in the cluster on the data
+ role (`roles/data/`, ADR-035): PostgreSQL 18 on loopback, one database per service named `ov_<name>`
+ (`ov_trade`, `ov_media`, …). The service reaches it through **PgBouncer** on `127.0.0.1:6432`, which
+ fronts the cluster for the app roles (and carries the app role's SCRAM password). Their durability is
+ **pgBackRest**, off-host: a full cluster backup weekly, a differential nightly and the WAL stream, which
+ the nightly `ovhost backup` verifies rather than copies ([backups.md](backups.md#postgresql-services)).
+- **What stays on the local filesystem** for those services is the rest of the data: object and media
+ files under `*_PATH` and the object stores (Media's B2/R2), blobs (`HOST_STORAGE_DIR`), uploads and
+ outputs, plus the frozen pre-switch SQLite file kept read-only for rollback. Those files are not in the
+ cluster and are backed up (or not) exactly as before.
+- **SQLite services** are unchanged: one file per service, copied by the online backup API as the service
+ user ([backups.md](backups.md)).
+
+`openvibe-shared/ready` is still what each service's `/ready` reports ([below](#per-service)); a switched
+service's `db` check reads the cluster as its `DATABASE_URL`/`DATABASE_DIRECT_URL` do.
Sources:
Sources:
- Each repository's `server/config.js`, `.env.example` and `deploy/systemd/*.service`.
- Each repository's `server/config.js`, `.env.example` and `deploy/systemd/*.service`.
@@ -81,9 +104,9 @@ Sources:
@@ -81,9 +104,9 @@ Sources:
production path. Whether `OPENVIBELIVE_DB_PATH` is set is unknown: check `/etc/openvibe/network.env`.
production path. Whether `OPENVIBELIVE_DB_PATH` is set is unknown: check `/etc/openvibe/network.env`.
- **Chat reads Live's files** at `/opt/openvibe.live/data/sounds`. These are files, not a database
- **Chat reads Live's files** at `/opt/openvibe.live/data/sounds`. These are files, not a database
(`deploy/systemd/openvibe-chat.service:43`).
(`deploy/systemd/openvibe-chat.service:43`).
-7. **Canonical money is still in Live.** Billing's `billing.db` is a shadow (Billing
- `STATUS.json: "authoritative": false`) until the cutover in [cutover-runbook.md](cutover-runbook.md)
- step B.
+7. **Canonical money is still in Live.** Billing's ledger (was `billing.db`, now the cluster database
+ `ov_billing` after the 2026-09-28 switch) is a shadow (Billing `STATUS.json: "authoritative": false`)
+ until the cutover in [cutover-runbook.md](cutover-runbook.md) step B.
## Per service
## Per service
@@ -91,6 +114,14 @@ Sources:
@@ -91,6 +114,14 @@ Sources:
the operator notes unless cited. Where a repo's `STATUS.json` disagrees, the operator notes and the drill
the operator notes unless cited. Where a repo's `STATUS.json` disagrees, the operator notes and the drill
log win.
log win.
+**The rows record the 2026-09-23 SQLite state.** For a service that has since switched to PostgreSQL —
+tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events,
+community, billing, media, openre (C-89) — the "Backing store" path below is now the **frozen pre-switch
+SQLite file** kept read-only for the 7-day rollback, and the live store is the cluster database
+`ov_<name>` reached through PgBouncer, as in [Where the data lives now](#where-the-data-lives-now). The
+five SQLite rows (network, chat, tools, games, live) are current. The handler citations are the ones
+reviewed on 2026-09-23.
+
| Service | Backing store (production path; env, default) | Canonical? | ADR-007 status | Readiness checks (handler) | Store checked? | Backup / drill |
| Service | Backing store (production path; env, default) | Canonical? | ADR-007 status | Readiness checks (handler) | Store checked? | Backup / drill |
|---|---|---|---|---|---|---|
|---|---|---|---|---|---|---|
| **Network** | `/opt/openvibe.network/data/network.db`; `DB_PATH`, default `./data/network.db` (`server/config.js:55`). Analytics lives in the same file (`README.md:306-307`). | Yes: identity, OAuth clients, subjects, OpenCoins wallets, themes, notifications, analytics | SQLite, the reviewed single-host state. `docs/platform-plan.md:74,111` defers PostgreSQL "until a room needs it". | `/api/ready` (`server/observability.js:58-96`). Required: `db` (`SELECT COUNT(*) FROM oauth_clients`), `signing_key` in production (signs and verifies an RS256 probe; fails on the HS256 dev key). Optional: `registry_poll`, `discord_bot`. | **Yes** | Drill passed 05:22. ovhost's deploy readiness polls the static `/api/health` (finding 3). |
| **Network** | `/opt/openvibe.network/data/network.db`; `DB_PATH`, default `./data/network.db` (`server/config.js:55`). Analytics lives in the same file (`README.md:306-307`). | Yes: identity, OAuth clients, subjects, OpenCoins wallets, themes, notifications, analytics | SQLite, the reviewed single-host state. `docs/platform-plan.md:74,111` defers PostgreSQL "until a room needs it". | `/api/ready` (`server/observability.js:58-96`). Required: `db` (`SELECT COUNT(*) FROM oauth_clients`), `signing_key` in production (signs and verifies an RS256 probe; fails on the HS256 dev key). Optional: `registry_poll`, `discord_bot`. | **Yes** | Drill passed 05:22. ovhost's deploy readiness polls the static `/api/health` (finding 3). |
@@ -1,11 +1,12 @@
@@ -1,11 +1,12 @@
{
{
"_about": "Example ovhost inventory for the single OpenVibe host (openvibe-oregon). Copy to /etc/openvibe/host.json (root:root, 0640) and correct anything that differs on the machine; the real file is never committed. Paths, units, ports and env file locations come from OpenVibe.Network docs/roadmap-baseline/data/services.json and prod-snapshot.json (22 Sep 2026) and each repository's deploy/ directory. env.required lists NAMES only.",
"_about": "Example ovhost inventory for the single OpenVibe host (openvibe-oregon). Copy to /etc/openvibe/host.json (root:root, 0640) and correct anything that differs on the machine; the real file is never committed. Paths, units, ports and env file locations come from OpenVibe.Network docs/roadmap-baseline/data/services.json and prod-snapshot.json (22 Sep 2026) and each repository's deploy/ directory. env.required lists NAMES only.",
+ "_databases": "Databases (lib/inventory.js): each services.<id>.databases[] item is one of two engines. SQLite (the default) is a file the service user's worker copies: { \"name\": \"live\", \"path\": \"/opt/openvibe.live/data/live.db\" }. PostgreSQL is a database in the cluster: { \"name\": \"trade\", \"engine\": \"postgresql\", \"database\": \"ov_trade\" } — the name must match ov_[a-z0-9_]+, and there is no path (ovhost reaches it as the postgres OS user over peer auth, and pgBackRest, not ovhost, copies it). tips, deals, coupons, reviews, trade, news, wiki, blog, search, sources, codes, vip, ai, host, events, community, billing, media and openre are PostgreSQL; network, chat, tools, games and live are still SQLite. A drill block follows the engine: a PostgreSQL database maps to { \"url\": \"DATABASE_URL\", \"directUrl\": \"DATABASE_DIRECT_URL\" } and the drill creates a scratch database and role to restore into.",
"host": "openvibe-oregon",
"host": "openvibe-oregon",
"stateDir": "/var/lib/openvibe-host",
"stateDir": "/var/lib/openvibe-host",
"backupDir": "/var/backups/openvibe",
"backupDir": "/var/backups/openvibe",
"backupStagingDir": "/var/backups/openvibe.staging",
"backupStagingDir": "/var/backups/openvibe.staging",
"drillDir": "/var/lib/openvibe-drills",
"drillDir": "/var/lib/openvibe-drills",
- "_drill": "Restore drills (ovhost drill <service>): each service with a database has a drill block. The drill port is the service port + 10000 on 127.0.0.1. http://127.0.0.1:9 is a closed port, used where an empty value would fall back to a production service on loopback. An empty OV_OAUTH_CLIENT_SECRET stops a service from minting service tokens, so it cannot call other services with its identity. Loopback stays reachable from the drill sandbox (services need Network's public keys), so these overrides are what keep a drill away from production. _notes explain what is left on and why.",
+ "_drill": "Restore drills (ovhost drill <service>): each service with a database has a drill block. The drill port is the service port + 10000 on 127.0.0.1. http://127.0.0.1:9 is a closed port, used where an empty value would fall back to a production service on loopback. An empty OV_OAUTH_CLIENT_SECRET stops a service from minting service tokens, so it cannot call other services with its identity. Loopback stays reachable from the drill sandbox (services need Network's public keys), so these overrides are what keep a drill away from production. A PostgreSQL drill restores the dump into a scratch database under a freshly created role, then sets DATABASE_URL and DATABASE_DIRECT_URL to it and VALKEY_URL to the closed port; it refuses to start if any of the three would keep its production value, because the drilled release would otherwise run its migrations against production (DATABASE_DIRECT_URL) and write into the production Valkey keyspace (VALKEY_URL). _notes explain what is left on and why.",
"_lifecycle": "Lifecycle (ovhost validate, WS-P task 1): each service's lifecycle comes from its manifest in openvibe-contracts (the service's `manifest` id, default its key), from openvibe-contracts 0.55.0 on. A `lifecycle` block in a service entry replaces the manifest's on this host; none is needed today. validate compares shutdown.deadlineSeconds with each unit's effective TimeoutStopSec and KillSignal, and the checkout's installed openvibe-contracts with contracts.range.",
"_lifecycle": "Lifecycle (ovhost validate, WS-P task 1): each service's lifecycle comes from its manifest in openvibe-contracts (the service's `manifest` id, default its key), from openvibe-contracts 0.55.0 on. A `lifecycle` block in a service entry replaces the manifest's on this host; none is needed today. validate compares shutdown.deadlineSeconds with each unit's effective TimeoutStopSec and KillSignal, and the checkout's installed openvibe-contracts with contracts.range.",
"nginx": {
"nginx": {
"sitesAvailable": "/etc/nginx/sites-available",
"sitesAvailable": "/etc/nginx/sites-available",
@@ -305,7 +306,6 @@
@@ -305,7 +306,6 @@
"envFile": "/etc/openvibe/media.env",
"envFile": "/etc/openvibe/media.env",
"env": {
"env": {
"required": [
"required": [
- "DB_PATH",
"VOD_PATH",
"VOD_PATH",
"MEDIA_B2_ENDPOINT",
"MEDIA_B2_ENDPOINT",
"MEDIA_B2_BUCKET",
"MEDIA_B2_BUCKET",
@@ -342,7 +342,8 @@
@@ -342,7 +342,8 @@
"databases": [
"databases": [
{
{
"name": "media",
"name": "media",
- "path": "/opt/openvibe.media/data/media.db"
+ "engine": "postgresql",
+ "database": "ov_media"
}
}
],
],
"nginx": {
"nginx": {
@@ -351,10 +352,13 @@
@@ -351,10 +352,13 @@
"variant": "http"
"variant": "http"
},
},
"drill": {
"drill": {
- "_notes": "Needs OpenVibe.Media 6e74eb0 or later (MEDIA_DRILL, server/drill.js; requires checks the deployed checkout). MEDIA_DRILL=1 refuses to start unless DB_PATH is outside the checkout and /opt/openvibe.media, HOST is loopback and PORT is not 4100. It then serves HTTP only: no app seeding or JWKS refresh, no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant scans, disk guardian, thumbnail cleanup, object purge, backfill, orphan-recording finalize or Events relay, and no webhook (the apps table's URLs are production's). It writes no file and creates no directory, and never opens a file path from the database: every byte route (/v and /c bytes, /t, /a, /f, /o, paste screenshots, live frames) answers 503; watch pages and /browse render from the copy. Every method but GET/HEAD/OPTIONS answers 403, /auth/* 503; net, fetch, child_process (git only: no ffmpeg or ffprobe), dgram and listen are guarded. /api/ready drops the storage and remote-tier checks (mode: drill). The storage paths from media.env move under {tmp}/storage (never created) as a second switch; THUMBNAILS_PATH stays production's, read-only in the sandbox, because /browse counts the thumbnail directory in its tab bar (the drill lists it and serves none of it). /browse caches its tab counts for 60 s in production: take the backup right before the drill. EVENTS_URL and the client secret are blanked too. Not run on the host yet.",
+ "_notes": "Needs OpenVibe.Media 6e74eb0 or later (MEDIA_DRILL, server/drill.js; requires checks the deployed checkout). MEDIA_DRILL=1 refuses to start unless its database is outside the checkout and /opt/openvibe.media, HOST is loopback and PORT is not 4100. It then serves HTTP only: no app seeding or JWKS refresh, no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, invariant scans, disk guardian, thumbnail cleanup, object purge, backfill, orphan-recording finalize or Events relay, and no webhook (the apps table's URLs are production's). It writes no file and creates no directory, and never opens a file path from the database: every byte route (/v and /c bytes, /t, /a, /f, /o, paste screenshots, live frames) answers 503; watch pages and /browse render from the copy. Every method but GET/HEAD/OPTIONS answers 403, /auth/* 503; net, fetch, child_process (git only: no ffmpeg or ffprobe), dgram and listen are guarded. /api/ready drops the storage and remote-tier checks (mode: drill). The storage paths from media.env move under {tmp}/storage (never created) as a second switch; THUMBNAILS_PATH stays production's, read-only in the sandbox, because /browse counts the thumbnail directory in its tab bar (the drill lists it and serves none of it). /browse caches its tab counts for 60 s in production: take the backup right before the drill. EVENTS_URL and the client secret are blanked too. Not run on the host yet.",
"port": 14100,
"port": 14100,
"databases": {
"databases": {
- "media": "DB_PATH"
+ "media": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -367,13 +371,18 @@
@@ -367,13 +371,18 @@
"FILES_PATH": "{tmp}/storage/files",
"FILES_PATH": "{tmp}/storage/files",
"OBJECTS_PATH": "{tmp}/storage/objects",
"OBJECTS_PATH": "{tmp}/storage/objects",
"EVENTS_URL": "",
"EVENTS_URL": "",
- "OV_OAUTH_CLIENT_SECRET": ""
+ "OV_OAUTH_CLIENT_SECRET": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"requires": [
"requires": [
{
{
"file": "server/drill.js",
"file": "server/drill.js",
"contains": "MEDIA_DRILL"
"contains": "MEDIA_DRILL"
},
},
+ {
+ "file": "server/drill.js",
+ "contains": "DATABASE_URL"
+ },
{
{
"file": "server/public/routes.js",
"file": "server/public/routes.js",
"contains": "refuseBytes"
"contains": "refuseBytes"
@@ -663,14 +672,11 @@
@@ -663,14 +672,11 @@
"databases": [
"databases": [
{
{
"name": "community",
"name": "community",
- "path": "/var/lib/openvibe-community/community.db"
- },
- {
- "name": "community-legacy-path",
- "path": "/opt/openvibe.community/data/community.db"
+ "engine": "postgresql",
+ "database": "ov_community"
}
}
],
],
- "_databasesNote": "The repo unit puts the database in /var/lib/openvibe-community; the 22 Sep production snapshot found it under /opt/openvibe.community/data. Keep the one that exists (backup skips a missing path).",
+ "_databasesNote": "Community is one PostgreSQL database (ov_community) now. Before the switch it declared two SQLite paths (the repo unit's /var/lib/openvibe-community/community.db and the /opt/openvibe.community/data/community.db the 22 Sep production snapshot found); neither is declared any more, and the frozen files are for `ovhost archive push`.",
"nginx": {
"nginx": {
"vhost": "openvibe.community.conf",
"vhost": "openvibe.community.conf",
"repoVhost": "deploy/nginx/openvibe.community.conf",
"repoVhost": "deploy/nginx/openvibe.community.conf",
@@ -679,7 +685,10 @@
@@ -679,7 +685,10 @@
"drill": {
"drill": {
"port": 14200,
"port": 14200,
"databases": {
"databases": {
- "community": "COMMUNITY_DB_PATH"
+ "community": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -687,7 +696,8 @@
@@ -687,7 +696,8 @@
"DISCORD_RELAY_ENABLED": "0",
"DISCORD_RELAY_ENABLED": "0",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9",
"OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9",
- "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9"
+ "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/pastes?limit=5",
"/api/pastes?limit=5",
@@ -712,7 +722,7 @@
@@ -712,7 +722,7 @@
"table": "posts"
"table": "posts"
}
}
],
],
- "_notes": "If the host keeps the database at the legacy path, map \"community-legacy-path\" instead of \"community\" (and in counts). An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."
+ "_notes": "Community is one PostgreSQL database (ov_community) now: the second, legacy SQLite path it used to declare is gone. An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."
}
}
},
},
"events": {
"events": {
@@ -753,7 +763,8 @@
@@ -753,7 +763,8 @@
"databases": [
"databases": [
{
{
"name": "events",
"name": "events",
- "path": "/var/lib/openvibe-events/events.db"
+ "engine": "postgresql",
+ "database": "ov_events"
}
}
],
],
"nginx": {
"nginx": {
@@ -768,12 +779,16 @@
@@ -768,12 +779,16 @@
"drill": {
"drill": {
"port": 14300,
"port": 14300,
"databases": {
"databases": {
- "events": "EVENTS_DB_PATH"
+ "events": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
"HOST": "127.0.0.1",
"HOST": "127.0.0.1",
- "EVENTS_WORKER": "off"
+ "EVENTS_WORKER": "off",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"counts": [
"counts": [
{
{
@@ -921,7 +936,7 @@
@@ -921,7 +936,7 @@
}
}
},
},
"host": {
"host": {
- "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its database and content-addressed objects live in /var/lib/openvibe-host-api, which is NOT ovhost's own stateDir (/var/lib/openvibe-host). Tenant vhosts: `ovhost nginx tenants host`.",
+ "_note": "OpenVibe.Host Stage B (tenant static hosting): the Host API service on port 4910. Not deployed as of 22 Sep 2026. Its content-addressed objects live in /var/lib/openvibe-host-api, which is NOT ovhost's own stateDir (/var/lib/openvibe-host); its database is the PostgreSQL database ov_host. Tenant vhosts: `ovhost nginx tenants host`.",
"repo": "/opt/openvibe.host",
"repo": "/opt/openvibe.host",
"units": [
"units": [
"openvibe-host.service"
"openvibe-host.service"
@@ -953,7 +968,8 @@
@@ -953,7 +968,8 @@
"databases": [
"databases": [
{
{
"name": "host",
"name": "host",
- "path": "/var/lib/openvibe-host-api/host.db"
+ "engine": "postgresql",
+ "database": "ov_host"
}
}
],
],
"nginx": {
"nginx": {
@@ -961,7 +977,10 @@
@@ -961,7 +977,10 @@
"sitesDomain": "openvibe.host",
"sitesDomain": "openvibe.host",
"dashboardDomain": "openvibe.host",
"dashboardDomain": "openvibe.host",
"wildcardCert": "openvibe.host",
"wildcardCert": "openvibe.host",
- "database": "/var/lib/openvibe-host-api/host.db",
+ "database": {
+ "engine": "postgresql",
+ "database": "ov_host"
+ },
"maxUpload": "110m",
"maxUpload": "110m",
"replaces": [
"replaces": [
"openvibe.host-tenants-pending.conf"
"openvibe.host-tenants-pending.conf"
@@ -971,7 +990,10 @@
@@ -971,7 +990,10 @@
"drill": {
"drill": {
"port": 14910,
"port": 14910,
"databases": {
"databases": {
- "host": "HOST_DB_PATH"
+ "host": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -979,7 +1001,8 @@
@@ -979,7 +1001,8 @@
"HOST_WORKER": "off",
"HOST_WORKER": "off",
"EVENTS_URL": "",
"EVENTS_URL": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
- "HOST_STORAGE_DIR": "{tmp}/objects"
+ "HOST_STORAGE_DIR": "{tmp}/objects",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"dirs": [
"dirs": [
"{tmp}/objects"
"{tmp}/objects"
@@ -1002,11 +1025,11 @@
@@ -1002,11 +1025,11 @@
"table": "host_domains"
"table": "host_domains"
}
}
],
],
- "_notes": "Restores host.db only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."
+ "_notes": "Restores the ov_host database only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."
}
}
},
},
"billing": {
"billing": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.billing",
"repo": "/opt/openvibe.billing",
"units": [
"units": [
"openvibe-billing.service"
"openvibe-billing.service"
@@ -1031,20 +1054,25 @@
@@ -1031,20 +1054,25 @@
"databases": [
"databases": [
{
{
"name": "billing",
"name": "billing",
- "path": "/var/lib/openvibe-billing/billing.db"
+ "engine": "postgresql",
+ "database": "ov_billing"
}
}
],
],
"drill": {
"drill": {
"port": 14600,
"port": 14600,
"databases": {
"databases": {
- "billing": "BILLING_DB_PATH"
+ "billing": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
"HOST": "127.0.0.1",
"HOST": "127.0.0.1",
"BILLING_JOBS": "off",
"BILLING_JOBS": "off",
"EVENTS_URL": "",
"EVENTS_URL": "",
- "OV_OAUTH_CLIENT_SECRET": ""
+ "OV_OAUTH_CLIENT_SECRET": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/rates"
"/api/v1/rates"
@@ -1156,7 +1184,7 @@
@@ -1156,7 +1184,7 @@
}
}
},
},
"search": {
"search": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.search",
"repo": "/opt/openvibe.search",
"units": [
"units": [
"openvibe-search.service"
"openvibe-search.service"
@@ -1181,20 +1209,25 @@
@@ -1181,20 +1209,25 @@
"databases": [
"databases": [
{
{
"name": "search",
"name": "search",
- "path": "/var/lib/openvibe-search/search.db"
+ "engine": "postgresql",
+ "database": "ov_search"
}
}
],
],
"drill": {
"drill": {
"port": 14710,
"port": 14710,
"databases": {
"databases": {
- "search": "SEARCH_DB_PATH"
+ "search": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
"HOST": "127.0.0.1",
"HOST": "127.0.0.1",
"EVENTS_URL": "",
"EVENTS_URL": "",
"SEARCH_EVENTS_SECRET": "",
"SEARCH_EVENTS_SECRET": "",
- "OV_OAUTH_CLIENT_SECRET": ""
+ "OV_OAUTH_CLIENT_SECRET": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/search?q=openvibe&limit=10",
"/api/v1/search?q=openvibe&limit=10",
@@ -1218,7 +1251,7 @@
@@ -1218,7 +1251,7 @@
}
}
},
},
"sources": {
"sources": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.sources",
"repo": "/opt/openvibe.sources",
"units": [
"units": [
"openvibe-sources.service"
"openvibe-sources.service"
@@ -1243,20 +1276,25 @@
@@ -1243,20 +1276,25 @@
"databases": [
"databases": [
{
{
"name": "sources",
"name": "sources",
- "path": "/var/lib/openvibe-sources/sources.db"
+ "engine": "postgresql",
+ "database": "ov_sources"
}
}
],
],
"drill": {
"drill": {
"port": 14720,
"port": 14720,
"databases": {
"databases": {
- "sources": "SOURCES_DB_PATH"
+ "sources": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
"HOST": "127.0.0.1",
"HOST": "127.0.0.1",
"SOURCES_WORKER": "off",
"SOURCES_WORKER": "off",
"EVENTS_URL": "",
"EVENTS_URL": "",
- "OV_OAUTH_CLIENT_SECRET": ""
+ "OV_OAUTH_CLIENT_SECRET": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"counts": [
"counts": [
{
{
@@ -1280,7 +1318,7 @@
@@ -1280,7 +1318,7 @@
}
}
},
},
"wiki": {
"wiki": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.wiki",
"repo": "/opt/openvibe.wiki",
"units": [
"units": [
"openvibe-wiki.service"
"openvibe-wiki.service"
@@ -1305,13 +1343,17 @@
@@ -1305,13 +1343,17 @@
"databases": [
"databases": [
{
{
"name": "wiki",
"name": "wiki",
- "path": "/var/lib/openvibe-wiki/wiki.db"
+ "engine": "postgresql",
+ "database": "ov_wiki"
}
}
],
],
"drill": {
"drill": {
"port": 14800,
"port": 14800,
"databases": {
"databases": {
- "wiki": "WIKI_DB_PATH"
+ "wiki": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1320,7 +1362,8 @@
@@ -1320,7 +1362,8 @@
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_MEDIA_INTERNAL_URL": "",
"OV_MEDIA_INTERNAL_URL": "",
"OV_SOURCES_INTERNAL_URL": "",
"OV_SOURCES_INTERNAL_URL": "",
- "WIKI_SCHEDULE_INTERVAL_MS": "2147483647"
+ "WIKI_SCHEDULE_INTERVAL_MS": "2147483647",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/spaces",
"/api/v1/spaces",
@@ -1344,7 +1387,7 @@
@@ -1344,7 +1387,7 @@
}
}
},
},
"blog": {
"blog": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.blog",
"repo": "/opt/openvibe.blog",
"units": [
"units": [
"openvibe-blog.service"
"openvibe-blog.service"
@@ -1369,13 +1412,17 @@
@@ -1369,13 +1412,17 @@
"databases": [
"databases": [
{
{
"name": "blog",
"name": "blog",
- "path": "/var/lib/openvibe-blog/blog.db"
+ "engine": "postgresql",
+ "database": "ov_blog"
}
}
],
],
"drill": {
"drill": {
"port": 14810,
"port": 14810,
"databases": {
"databases": {
- "blog": "BLOG_DB_PATH"
+ "blog": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1384,7 +1431,8 @@
@@ -1384,7 +1431,8 @@
"EVENTS_URL": "",
"EVENTS_URL": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9",
"OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9",
- "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9"
+ "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/blogs/openvibe/posts?limit=10",
"/api/v1/blogs/openvibe/posts?limit=10",
@@ -1408,7 +1456,7 @@
@@ -1408,7 +1456,7 @@
}
}
},
},
"tips": {
"tips": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.tips",
"repo": "/opt/openvibe.tips",
"units": [
"units": [
"openvibe-tips.service"
"openvibe-tips.service"
@@ -1433,13 +1481,17 @@
@@ -1433,13 +1481,17 @@
"databases": [
"databases": [
{
{
"name": "tips",
"name": "tips",
- "path": "/var/lib/openvibe-tips/tips.db"
+ "engine": "postgresql",
+ "database": "ov_tips"
}
}
],
],
"drill": {
"drill": {
"port": 14610,
"port": 14610,
"databases": {
"databases": {
- "tips": "TIPS_DB_PATH"
+ "tips": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1450,7 +1502,8 @@
@@ -1450,7 +1502,8 @@
"TIPS_CHAT_ADAPTER": "none",
"TIPS_CHAT_ADAPTER": "none",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"BILLING_URL": "http://127.0.0.1:9",
"BILLING_URL": "http://127.0.0.1:9",
- "LIVE_INTERNAL_URL": "http://127.0.0.1:9"
+ "LIVE_INTERNAL_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/sitemap.xml"
"/sitemap.xml"
@@ -1473,7 +1526,7 @@
@@ -1473,7 +1526,7 @@
}
}
},
},
"vip": {
"vip": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.vip",
"repo": "/opt/openvibe.vip",
"units": [
"units": [
"openvibe-vip.service"
"openvibe-vip.service"
@@ -1498,13 +1551,17 @@
@@ -1498,13 +1551,17 @@
"databases": [
"databases": [
{
{
"name": "vip",
"name": "vip",
- "path": "/var/lib/openvibe-vip/vip.db"
+ "engine": "postgresql",
+ "database": "ov_vip"
}
}
],
],
"drill": {
"drill": {
"port": 14620,
"port": 14620,
"databases": {
"databases": {
- "vip": "VIP_DB_PATH"
+ "vip": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1514,7 +1571,8 @@
@@ -1514,7 +1571,8 @@
"VIP_EVENTS_SECRET": "",
"VIP_EVENTS_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"BILLING_URL": "http://127.0.0.1:9",
"BILLING_URL": "http://127.0.0.1:9",
- "VIP_CHECKOUT_PROVIDERS": ""
+ "VIP_CHECKOUT_PROVIDERS": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/creators/network",
"/api/v1/creators/network",
@@ -1539,7 +1597,7 @@
@@ -1539,7 +1597,7 @@
}
}
},
},
"news": {
"news": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.news",
"repo": "/opt/openvibe.news",
"units": [
"units": [
"openvibe-news.service"
"openvibe-news.service"
@@ -1564,13 +1622,17 @@
@@ -1564,13 +1622,17 @@
"databases": [
"databases": [
{
{
"name": "news",
"name": "news",
- "path": "/var/lib/openvibe-news/news.db"
+ "engine": "postgresql",
+ "database": "ov_news"
}
}
],
],
"drill": {
"drill": {
"port": 14820,
"port": 14820,
"databases": {
"databases": {
- "news": "NEWS_DB_PATH"
+ "news": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1580,7 +1642,8 @@
@@ -1580,7 +1642,8 @@
"NEWS_EVENTS_SECRET": "",
"NEWS_EVENTS_SECRET": "",
"OV_AI_INTERNAL_URL": "",
"OV_AI_INTERNAL_URL": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
- "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9"
+ "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/stories?limit=10",
"/api/v1/stories?limit=10",
@@ -1603,7 +1666,7 @@
@@ -1603,7 +1666,7 @@
}
}
},
},
"reviews": {
"reviews": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.reviews",
"repo": "/opt/openvibe.reviews",
"units": [
"units": [
"openvibe-reviews.service"
"openvibe-reviews.service"
@@ -1628,13 +1691,17 @@
@@ -1628,13 +1691,17 @@
"databases": [
"databases": [
{
{
"name": "reviews",
"name": "reviews",
- "path": "/var/lib/openvibe-reviews/reviews.db"
+ "engine": "postgresql",
+ "database": "ov_reviews"
}
}
],
],
"drill": {
"drill": {
"port": 14830,
"port": 14830,
"databases": {
"databases": {
- "reviews": "REVIEWS_DB_PATH"
+ "reviews": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1643,7 +1710,8 @@
@@ -1643,7 +1710,8 @@
"EVENTS_URL": "",
"EVENTS_URL": "",
"REVIEWS_EVENTS_SECRET": "",
"REVIEWS_EVENTS_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
- "OV_SOURCES_INTERNAL_URL": ""
+ "OV_SOURCES_INTERNAL_URL": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/entities?limit=10",
"/api/v1/entities?limit=10",
@@ -1667,7 +1735,7 @@
@@ -1667,7 +1735,7 @@
}
}
},
},
"deals": {
"deals": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.deals",
"repo": "/opt/openvibe.deals",
"units": [
"units": [
"openvibe-deals.service"
"openvibe-deals.service"
@@ -1692,13 +1760,17 @@
@@ -1692,13 +1760,17 @@
"databases": [
"databases": [
{
{
"name": "deals",
"name": "deals",
- "path": "/var/lib/openvibe-deals/deals.db"
+ "engine": "postgresql",
+ "database": "ov_deals"
}
}
],
],
"drill": {
"drill": {
"port": 14840,
"port": 14840,
"databases": {
"databases": {
- "deals": "DEALS_DB_PATH"
+ "deals": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1709,7 +1781,8 @@
@@ -1709,7 +1781,8 @@
"DEALS_EVENTS_SECRET": "",
"DEALS_EVENTS_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9",
"OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9",
- "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9"
+ "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
{
{
@@ -1737,7 +1810,7 @@
@@ -1737,7 +1810,7 @@
}
}
},
},
"coupons": {
"coupons": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.coupons",
"repo": "/opt/openvibe.coupons",
"units": [
"units": [
"openvibe-coupons.service"
"openvibe-coupons.service"
@@ -1762,13 +1835,17 @@
@@ -1762,13 +1835,17 @@
"databases": [
"databases": [
{
{
"name": "coupons",
"name": "coupons",
- "path": "/var/lib/openvibe-coupons/coupons.db"
+ "engine": "postgresql",
+ "database": "ov_coupons"
}
}
],
],
"drill": {
"drill": {
"port": 14850,
"port": 14850,
"databases": {
"databases": {
- "coupons": "COUPONS_DB_PATH"
+ "coupons": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1776,7 +1853,8 @@
@@ -1776,7 +1853,8 @@
"COUPONS_WORKER": "off",
"COUPONS_WORKER": "off",
"EVENTS_URL": "",
"EVENTS_URL": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
- "OV_SOURCES_INTERNAL_URL": ""
+ "OV_SOURCES_INTERNAL_URL": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/sitemaps/merchants.xml",
"/sitemaps/merchants.xml",
@@ -1799,7 +1877,7 @@
@@ -1799,7 +1877,7 @@
}
}
},
},
"trade": {
"trade": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.trade",
"repo": "/opt/openvibe.trade",
"units": [
"units": [
"openvibe-trade.service"
"openvibe-trade.service"
@@ -1824,13 +1902,17 @@
@@ -1824,13 +1902,17 @@
"databases": [
"databases": [
{
{
"name": "trade",
"name": "trade",
- "path": "/var/lib/openvibe-trade/trade.db"
+ "engine": "postgresql",
+ "database": "ov_trade"
}
}
],
],
"drill": {
"drill": {
"port": 14860,
"port": 14860,
"databases": {
"databases": {
- "trade": "TRADE_DB_PATH"
+ "trade": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1839,7 +1921,8 @@
@@ -1839,7 +1921,8 @@
"EVENTS_URL": "",
"EVENTS_URL": "",
"TRADE_EVENTS_WEBHOOK_SECRET": "",
"TRADE_EVENTS_WEBHOOK_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
- "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9"
+ "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/v1/instruments?limit=10",
"/api/v1/instruments?limit=10",
@@ -1862,7 +1945,7 @@
@@ -1862,7 +1945,7 @@
}
}
},
},
"codes": {
"codes": {
- "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database from the repository's deploy/ directory.",
"repo": "/opt/openvibe.codes",
"repo": "/opt/openvibe.codes",
"units": [
"units": [
"openvibe-codes.service"
"openvibe-codes.service"
@@ -1887,19 +1970,24 @@
@@ -1887,19 +1970,24 @@
"databases": [
"databases": [
{
{
"name": "codes",
"name": "codes",
- "path": "/var/lib/openvibe-codes/codes.db"
+ "engine": "postgresql",
+ "database": "ov_codes"
}
}
],
],
"drill": {
"drill": {
"port": 14900,
"port": 14900,
"databases": {
"databases": {
- "codes": "CODES_DB_PATH"
+ "codes": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
"HOST": "127.0.0.1",
"HOST": "127.0.0.1",
"EVENTS_URL": "",
"EVENTS_URL": "",
- "OV_OAUTH_CLIENT_SECRET": ""
+ "OV_OAUTH_CLIENT_SECRET": "",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/sitemap.xml"
"/sitemap.xml"
@@ -1946,14 +2034,18 @@
@@ -1946,14 +2034,18 @@
"databases": [
"databases": [
{
{
"name": "ai",
"name": "ai",
- "path": "/var/lib/openvibe-ai/ai.db"
+ "engine": "postgresql",
+ "database": "ov_ai"
}
}
],
],
"drill": {
"drill": {
"_notes": "Every provider key, base URL and the HTTP seam are blanked and AI_ENABLED=0, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and EVENTS_URL are blanked (no service calls, no events); WHISPER_BIN is blanked (no transcription). Boot marks interrupted runs failed in the restored copy only. Counts cover configuration and citations only: runs, requests and usage grow by the minute, so they are checked by integrity_check, not by an equal count. Passed on the host 2026-09-24 05:40 UTC.",
"_notes": "Every provider key, base URL and the HTTP seam are blanked and AI_ENABLED=0, so no run can reach a paid provider; OV_OAUTH_CLIENT_SECRET and EVENTS_URL are blanked (no service calls, no events); WHISPER_BIN is blanked (no transcription). Boot marks interrupted runs failed in the restored copy only. Counts cover configuration and citations only: runs, requests and usage grow by the minute, so they are checked by integrity_check, not by an equal count. Passed on the host 2026-09-24 05:40 UTC.",
"port": 14700,
"port": 14700,
"databases": {
"databases": {
- "ai": "AI_DB_PATH"
+ "ai": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -1971,7 +2063,8 @@
@@ -1971,7 +2063,8 @@
"WHISPER_BIN": "",
"WHISPER_BIN": "",
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"EVENTS_URL": "",
"EVENTS_URL": "",
- "AI_STUB_FALLBACK": "1"
+ "AI_STUB_FALLBACK": "1",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/health"
"/api/health"
@@ -2046,7 +2139,8 @@
@@ -2046,7 +2139,8 @@
"databases": [
"databases": [
{
{
"name": "openre",
"name": "openre",
- "path": "/var/lib/openre/openre.db"
+ "engine": "postgresql",
+ "database": "ov_openre"
}
}
],
],
"drill": {
"drill": {
@@ -2056,7 +2150,10 @@
@@ -2056,7 +2150,10 @@
"server/index.js"
"server/index.js"
],
],
"databases": {
"databases": {
- "openre": "OPENRE_DB_PATH"
+ "openre": {
+ "url": "DATABASE_URL",
+ "directUrl": "DATABASE_DIRECT_URL"
+ }
},
},
"env": {
"env": {
"PORT": "{port}",
"PORT": "{port}",
@@ -2066,7 +2163,8 @@
@@ -2066,7 +2163,8 @@
"OV_OAUTH_CLIENT_SECRET": "",
"OV_OAUTH_CLIENT_SECRET": "",
"MEDIA_URL": "http://127.0.0.1:9",
"MEDIA_URL": "http://127.0.0.1:9",
"MEDIA_API_KEY": "",
"MEDIA_API_KEY": "",
- "OPENRE_RECORDING": "off"
+ "OPENRE_RECORDING": "off",
+ "VALKEY_URL": "redis://127.0.0.1:9/0"
},
},
"compare": [
"compare": [
"/api/health",
"/api/health",
@@ -2093,8 +2191,8 @@
@@ -2093,8 +2191,8 @@
"_notes": "Only openre-api starts (command; the working directory and the non-secret Environment= come from openre-api.service). It never runs a transport, a coordinator or the event relay in any mode. OPENRE_DRILL=1 (OpenRe 5e86ea7 and later) also makes it read-only: writes, /play/ (which would pull from production's worker on loopback) and sign-in (which would redeem codes at Network) answer 503. The coordinator and the workers refuse to start. The event relay and Media calls are off. EVENTS_URL and OV_OAUTH_CLIENT_SECRET empty and MEDIA_URL on the closed port are the same switches for releases before 5e86ea7. Left on (read-only): the Network JWKS GET on loopback. Not compared: ingest_sessions and workers (they change whenever someone streams or a generation rolls over) and /api/ready (it reports live worker heartbeats)."
"_notes": "Only openre-api starts (command; the working directory and the non-secret Environment= come from openre-api.service). It never runs a transport, a coordinator or the event relay in any mode. OPENRE_DRILL=1 (OpenRe 5e86ea7 and later) also makes it read-only: writes, /play/ (which would pull from production's worker on loopback) and sign-in (which would redeem codes at Network) answer 503. The coordinator and the workers refuse to start. The event relay and Media calls are off. EVENTS_URL and OV_OAUTH_CLIENT_SECRET empty and MEDIA_URL on the closed port are the same switches for releases before 5e86ea7. Left on (read-only): the Network JWKS GET on loopback. Not compared: ingest_sessions and workers (they change whenever someone streams or a generation rolls over) and /api/ready (it reports live worker heartbeats)."
},
},
"protected": {
"protected": {
- "kind": "sqlite-count",
- "db": "/var/lib/openre/openre.db",
+ "kind": "postgresql-count",
+ "database": "ov_openre",
"sql": "SELECT count(*) AS n FROM ingest_sessions WHERE state IN ('starting', 'live', 'ending')",
"sql": "SELECT count(*) AS n FROM ingest_sessions WHERE state IN ('starting', 'live', 'ending')",
"label": "ingest sessions"
"label": "ingest sessions"
},
},
@@ -42,7 +42,10 @@ async function refusal(exec, inv, file) {
@@ -42,7 +42,10 @@ async function refusal(exec, inv, file) {
if (typeof file !== 'string' || !path.isAbsolute(file) || path.normalize(file) !== file) return 'not a normalised absolute path';
if (typeof file !== 'string' || !path.isAbsolute(file) || path.normalize(file) !== file) return 'not a normalised absolute path';
if (!ALLOWED_ROOTS.some((r) => file.startsWith(r))) return `outside ${ALLOWED_ROOTS.join(', ')}`;
if (!ALLOWED_ROOTS.some((r) => file.startsWith(r))) return `outside ${ALLOWED_ROOTS.join(', ')}`;
if (!NAME_RE.test(objectName(file))) return 'the path has characters an object name cannot carry';
if (!NAME_RE.test(objectName(file))) return 'the path has characters an object name cannot carry';
- const declared = Object.values(inv.services || {}).flatMap((s) => (s.databases || []).map((d) => d.path || d));
+ // A live SQLite database is a file the inventory declares, and archiving it is refused. A
+ // PostgreSQL database has no file (its own entry carries no path), so the C-89 switch — archive
+ // the frozen SQLite file, then delete it — is no longer blocked by the inventory that moved on.
+ const declared = Object.values(inv.services || {}).flatMap((s) => (s.databases || []).filter((d) => d.engine !== 'postgresql').map((d) => d.path));
if (declared.includes(file)) return 'a database the inventory declares (a live database)';
if (declared.includes(file)) return 'a database the inventory declares (a live database)';
const st = await exec.stat(file);
const st = await exec.stat(file);
if (!st) return 'no such file';
if (!st) return 'no such file';
@@ -59,11 +59,13 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
@@ -59,11 +59,13 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
for the VERIFIED custom domains (read from the service's
for the VERIFIED custom domains (read from the service's
database); lists domains still waiting for a certificate
database); lists domains still waiting for a certificate
snapshot <service> [--out <file>] config tarball: units, vhost, env NAMES, sha — no secret values
snapshot <service> [--out <file>] config tarball: units, vhost, env NAMES, sha — no secret values
- backup <service> sqlite .backup of the declared databases to a dated directory
- backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>]
+ backup <service> [--logical] .backup of the declared databases to a dated directory;
+ --logical also takes a PostgreSQL pg_dump (else weekly)
+ backup --all [--offsite] [--no-prune] [--keep-daily <n>] [--keep-weekly <n>] [--logical]
every service with databases; one failure never stops the
every service with databases; one failure never stops the
others; keeps the last 7 daily + 4 weekly good backups per
others; keeps the last 7 daily + 4 weekly good backups per
- service; JSON summary in <stateDir>/backup-runs/<run>.json;
+ service; --logical also dumps every PostgreSQL database;
+ JSON summary in <stateDir>/backup-runs/<run>.json;
--offsite then encrypts and uploads the run (root)
--offsite then encrypts and uploads the run (root)
backup-metrics [--run <run>] rewrite the Prometheus textfile (node_exporter collector) from a
backup-metrics [--run <run>] rewrite the Prometheus textfile (node_exporter collector) from a
recorded backup --all run (default: the latest)
recorded backup --all run (default: the latest)
@@ -398,13 +400,13 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -398,13 +400,13 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
const cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });
const cfg = await lib.offsite.loadConfig(exec, { file: opts['backup-env'], env });
return lib.offsite.push(ctx, summary, { cfg, client: s3(cfg) });
return lib.offsite.push(ctx, summary, { cfg, client: s3(cfg) });
} : null;
} : null;
- const r = await lib.backupAll(ctx, { daily: count(opts, 'keep-daily', 7), weekly: count(opts, 'keep-weekly', 4), prune: !opts['no-prune'], offsite: offsiteRun });
+ const r = await lib.backupAll(ctx, { daily: count(opts, 'keep-daily', 7), weekly: count(opts, 'keep-weekly', 4), prune: !opts['no-prune'], offsite: offsiteRun, logical: !!opts.logical });
if (json) emit(r);
if (json) emit(r);
else {
else {
for (const s of r.services) {
for (const s of r.services) {
const copied = s.files.filter((f) => f.dest && !f.error);
const copied = s.files.filter((f) => f.dest && !f.error);
const size = copied.reduce((a, f) => a + (f.bytes || 0), 0);
const size = copied.reduce((a, f) => a + (f.bytes || 0), 0);
- const detail = s.status === 'failed' ? s.error : s.status === 'skipped' ? 'no database found' : `${copied.length} file(s), ${bytes(size)}`;
+ const detail = s.status === 'failed' ? s.error : s.status === 'skipped' ? 'no database found' : s.status === 'verified' ? 'pgBackRest verified (no dump due)' : `${copied.length} file(s), ${bytes(size)}`;
out(`${s.status.toUpperCase().padEnd(8)} ${s.service.padEnd(10)} ${detail}${s.pruned.length ? `; pruned ${s.pruned.length}` : ''}`);
out(`${s.status.toUpperCase().padEnd(8)} ${s.service.padEnd(10)} ${detail}${s.pruned.length ? `; pruned ${s.pruned.length}` : ''}`);
}
}
if (r.offsite) {
if (r.offsite) {
@@ -416,8 +418,14 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -416,8 +418,14 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
}
}
return r.ok ? 0 : 2;
return r.ok ? 0 : 2;
}
}
- const r = await lib.backup(ctx, needService());
- if (json) emit(r); else for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);
+ const r = await lib.backup(ctx, needService(), { logical: !!opts.logical });
+ if (json) emit(r);
+ else {
+ for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);
+ // A PostgreSQL run with no dump due writes no file: say what it did instead, so a
+ // silent command is never mistaken for one that did nothing.
+ if (!r.files.length && r.verified) out(`pgBackRest verified: newest ${r.verified.type} backup stopped ${r.verified.stop}, WAL archiver ok: ${r.verified.walOk}; no logical dump due (--logical forces one)`);
+ }
return 0;
return 0;
}
}
case 'backup-metrics': {
case 'backup-metrics': {
@@ -1,19 +1,34 @@
@@ -1,19 +1,34 @@
'use strict';
'use strict';
/**
/**
- * ovhost backup <service>: an online SQLite `.backup` of every database the inventory declares for
- * the service, into <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.db. The copy is made by a worker
- * running as the service user (root never opens a service database — see sqlite-worker.js), checked
- * with PRAGMA quick_check, and recorded in <stateDir>/backups/<service>.jsonl.
+ * ovhost backup <service>: an online copy of every database the inventory declares for the service,
+ * into <backupDir>/<service>/<YYYYMMDD-HHMMSS>/<name>.<ext>, recorded in
+ * <stateDir>/backups/<service>.jsonl. SQLite is copied by a worker running as the service user (root
+ * never opens a service database — see sqlite-worker.js) and checked with PRAGMA quick_check.
+ * PostgreSQL is not dumped every night: pgBackRest holds the cluster's full + differential backups
+ * and its WAL (roles/data/provision.sh), so a run that finds a PostgreSQL database first VERIFIES
+ * that layer (dbengine.verifyPgBackRest: a stanza that is ok and whose newest backup is under 26 h
+ * old; dbengine.walFailedCount: pg_stat_archiver.failed_count has not risen since the last run) and
+ * only then, on a slower cadence (LOGICAL_DUMP_INTERVAL_DAYS, or --logical), takes a per-service
+ * pg_dump -Fc archive — the artifact the off-site code uploads and a drill can restore.
*
*
- * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600,
- * with no -wal/-shm beside it. The worker writes into <backupStagingDir>/<service>-<stamp> (its own,
- * 0700, under a root 0711 parent); root then takes that directory over, checks each copy is a plain
- * file with one link, chowns and chmods it, and renames the directory into place.
+ * A PostgreSQL service's record always carries `verified` (type + stop + walOk when it passed, an
+ * error when it did not), and a run with nothing to dump is verification-only: ok: true, files: [],
+ * and no directory at all. A failed verification (or a logical dump that failed) is recorded
+ * (ok: false) and then throws a BackupError carrying the record, so backup --all reports the service
+ * failed — never skipped.
+ *
+ * Permissions: every directory under backupDir is root:root 0700 and every copy root:root 0600, with
+ * no -wal/-shm beside it. SQLite's worker writes into <backupStagingDir>/<service>-<stamp> (its own,
+ * 0700, under a root 0711 parent) as the service user. A service with any PostgreSQL database gets a
+ * staging directory owned by postgres instead, because pg_dump runs as that OS user. In both cases
+ * root then takes that directory over (lockDown, root:root 0700) BEFORE anything inside it is
+ * checked — the service user (or postgres) can no longer reach it while root validates the copies,
+ * chowns and chmods them and renames the directory into place, which is what keeps the property.
*
*
* Every declared database is attempted even when an earlier one fails. A backup with any failed
* Every declared database is attempted even when an earlier one fails. A backup with any failed
* database is still recorded (ok: false, so a drill and retention never treat it as good) and then
* database is still recorded (ok: false, so a drill and retention never treat it as good) and then
- * throws a BackupError carrying the record. A backup that found none of its databases is recorded
- * as ok: false, empty: true and leaves no directory behind.
+ * throws a BackupError carrying the record. A SQLite backup that found none of its databases is
+ * recorded as ok: false, empty: true and leaves no directory behind; PostgreSQL never sets empty.
*
*
* ovhost backup --all (backupAll): every service that declares databases, one after another. A
* ovhost backup --all (backupAll): every service that declares databases, one after another. A
* failing service never stops the others. Afterwards each service's backups are pruned to the last
* failing service never stops the others. Afterwards each service's backups are pruned to the last
@@ -22,9 +37,16 @@
@@ -22,9 +37,16 @@
*/
*/
const path = require('path');
const path = require('path');
const { service } = require('../inventory');
const { service } = require('../inventory');
+const dbengine = require('../dbengine');
const lock = require('../lock');
const lock = require('../lock');
const retention = require('../retention');
const retention = require('../retention');
+const DAY_MS = 86400000;
+// A PostgreSQL database's own logical dump is on a slower cadence than pgBackRest's nightly
+// differentials: it exists for the off-site copy and per-service drills, not for durability.
+const LOGICAL_DUMP_INTERVAL_DAYS = 7;
+const ARCHIVER_STATE = 'pgbackrest-archiver.json';
+
class BackupError extends Error {
class BackupError extends Error {
constructor(message, record) { super(message); this.record = record; }
constructor(message, record) { super(message); this.record = record; }
}
}
@@ -47,15 +69,105 @@ async function privateDir(exec, p) {
@@ -47,15 +69,105 @@ async function privateDir(exec, p) {
await lockDown(exec, p, 0o700);
await lockDown(exec, p, 0o700);
}
}
-async function backup(ctx, id, { reason = 'manual' } = {}) {
+/** The last pg_stat_archiver.failed_count this host recorded, or null on the first run. */
+async function readArchiver(exec, inv) {
+ const text = await exec.readFile(path.join(inv.stateDir, ARCHIVER_STATE));
+ if (text == null) return null;
+ try {
+ const v = JSON.parse(text);
+ return v && typeof v.failedCount === 'number' && Number.isFinite(v.failedCount) ? v.failedCount : null;
+ } catch { return null; }
+}
+
+/** Remember this run's failed_count (mode 0640: nobody but root needs it) for the next run's compare. */
+async function writeArchiver(exec, inv, failedCount) {
+ await exec.mkdir(inv.stateDir, { mode: 0o750 });
+ await exec.writeFile(path.join(inv.stateDir, ARCHIVER_STATE), `${JSON.stringify({ failedCount, at: new Date(exec.now()).toISOString() })}\n`, { mode: 0o640 });
+}
+
+/**
+ * The every-run pgBackRest verification for a service with PostgreSQL databases. Nothing is copied
+ * until this passes: a stale stanza, a bad status or a risen WAL archiver failure count all mean the
+ * cluster's durability layer is not healthy, and that is what the record must say.
+ * -> { type, stop, walOk, ok: true } or { ok: false, error }
+ */
+async function verifyPostgres(ctx) {
+ const { exec, inv, log } = ctx;
+ try {
+ const v = await dbengine.verifyPgBackRest(exec, { now: exec.now() });
+ const failed = await dbengine.walFailedCount(exec);
+ const previous = await readArchiver(exec, inv);
+ const walOk = previous === null || failed <= previous;
+ // Record this run's value even when it rose: the next run compares against the latest, so a
+ // single archiving blip fails one night rather than every night thereafter.
+ await writeArchiver(exec, inv, failed);
+ if (!walOk) {
+ const error = `the WAL archiver failed_count rose from ${previous} to ${failed}`;
+ log(`PostgreSQL verification FAILED: ${error}`);
+ return { type: v.type, stop: v.stop, walOk, ok: false, error };
+ }
+ log(`pgBackRest ok: newest ${v.type} backup stopped ${v.stop}; WAL archiver failed_count ${failed}`);
+ return { type: v.type, stop: v.stop, walOk, ok: true };
+ } catch (err) {
+ log(`PostgreSQL verification FAILED: ${err.message}`);
+ return { ok: false, error: err.message };
+ }
+}
+
+/** When this service last recorded a GOOD logical dump (`at` ISO), or null: the weekly cadence input. */
+async function lastGoodDumpAt(exec, inv, id) {
+ const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`));
+ const lines = String(text || '').split('\n').filter(Boolean);
+ for (let i = lines.length - 1; i >= 0; i--) {
+ let r;
+ try { r = JSON.parse(lines[i]); } catch { continue; }
+ if (!r || r.ok === false) continue;
+ if ((r.files || []).some((f) => typeof f.file === 'string' && f.file.endsWith('.dump'))) return r.at || null;
+ }
+ return null;
+}
+
+async function backup(ctx, id, { reason = 'manual', logical = false } = {}) {
const { exec, inv, log } = ctx;
const { exec, inv, log } = ctx;
const svc = service(inv, id);
const svc = service(inv, id);
if (!svc.databases.length) throw new Error(`${id} declares no databases`);
if (!svc.databases.length) throw new Error(`${id} declares no databases`);
+ const postgres = svc.databases.filter((d) => d.engine === 'postgresql');
+ const hasPostgres = postgres.length > 0;
+ const engine = hasPostgres ? 'postgresql' : 'sqlite';
+
+ // (a) Verify the cluster's own backup layer BEFORE copying anything. A failed verification is
+ // itself a record (ok: false): backupAll then marks the service failed, never skipped.
+ let verified = null;
+ if (hasPostgres) {
+ verified = await verifyPostgres(ctx);
+ if (!verified.ok) {
+ const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: false, files: [], verified };
+ await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
+ throw new BackupError(`PostgreSQL verification failed: ${verified.error}`, record);
+ }
+ }
+
+ // (b) A logical dump only when the last good one is older than LOGICAL_DUMP_INTERVAL_DAYS, or the
+ // caller passed --logical: pgBackRest already holds a nightly differential, so this exists for the
+ // off-site copy and per-service drills.
+ const dump = [];
+ if (hasPostgres) {
+ const last = await lastGoodDumpAt(exec, inv, id);
+ const at = last === null ? NaN : Date.parse(last);
+ if (logical || !Number.isFinite(at) || exec.now() - at > LOGICAL_DUMP_INTERVAL_DAYS * DAY_MS) dump.push(...postgres);
+ }
+ // A verification-only run (PostgreSQL, no dump due, no SQLite database) leaves nothing on disk.
+ if (!dump.length && !svc.databases.some((d) => d.engine === 'sqlite')) {
+ const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: true, files: [], verified };
+ await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
+ return record;
+ }
+
const st = stamp(exec.now());
const st = stamp(exec.now());
const dir = path.join(inv.backupDir, id, st);
const dir = path.join(inv.backupDir, id, st);
if (await exec.stat(dir)) throw new Error(`${dir} already exists; a backup never overwrites another`);
if (await exec.stat(dir)) throw new Error(`${dir} already exists; a backup never overwrites another`);
// Backups hold user data: every directory under backupDir is root:root 0700 and every copy is
// Backups hold user data: every directory under backupDir is root:root 0700 and every copy is
- // root:root 0600. The worker must run as the service user (root never opens a service database),
+ // root:root 0600. SQLite's worker runs as the service user (root never opens a service database),
// so it writes into a staging directory of its own, which root then takes over before the move.
// so it writes into a staging directory of its own, which root then takes over before the move.
await privateDir(exec, inv.backupDir);
await privateDir(exec, inv.backupDir);
await privateDir(exec, path.join(inv.backupDir, id));
await privateDir(exec, path.join(inv.backupDir, id));
@@ -63,27 +175,44 @@ async function backup(ctx, id, { reason = 'manual' } = {}) {
@@ -63,27 +175,44 @@ async function backup(ctx, id, { reason = 'manual' } = {}) {
await lockDown(exec, inv.backupStagingDir, 0o711);
await lockDown(exec, inv.backupStagingDir, 0o711);
const stage = path.join(inv.backupStagingDir, `${id}-${st}`);
const stage = path.join(inv.backupStagingDir, `${id}-${st}`);
if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
- await exec.mkdir(stage, { owner: svc.runAs, mode: 0o700 });
+ // pg_dump runs as the postgres OS user, so a service with any PostgreSQL database needs a staging
+ // directory postgres can write into (SQLite keeps the service user). The lockDown(stage, 0o700)
+ // below takes the whole directory back to root:root BEFORE anything inside it is checked, so
+ // neither the service user nor postgres can reach it while root validates and moves it.
+ await exec.mkdir(stage, { owner: hasPostgres ? dbengine.PG_USER : svc.runAs, mode: 0o700 });
const files = [];
const files = [];
try {
try {
for (const db of svc.databases) {
for (const db of svc.databases) {
- if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }
- // Named by the inventory name: two databases may share a file name (community.db twice).
- const file = `${db.name}${path.extname(db.path) || '.db'}`;
- log(`backing up ${db.path} → ${path.join(dir, file)} (as ${svc.runAs})`);
+ const file = dbengine.fileName(db);
+ if (db.engine === 'sqlite') {
+ if (!(await exec.stat(db.path))) { files.push({ name: db.name, source: db.path, skipped: 'not found' }); log(`skip ${db.path}: not found`); continue; }
+ log(`backing up ${db.path} → ${path.join(dir, file)} (as ${svc.runAs})`);
+ const t0 = exec.now();
+ try {
+ // The worker runs PRAGMA quick_check on the copy and fails unless it answers ok.
+ await exec.sqliteBackup(db.path, path.join(stage, file), { as: svc.runAs });
+ } catch (err) {
+ files.push({ name: db.name, source: db.path, dest: path.join(dir, file), error: err.message });
+ log(`FAILED ${db.path}: ${err.message}`);
+ continue;
+ }
+ files.push({ name: db.name, source: db.path, dest: path.join(dir, file), file, check: 'quick_check ok', seconds: Math.round((exec.now() - t0) / 1000) });
+ continue;
+ }
+ if (!dump.includes(db)) { log(`${db.database} is covered by pgBackRest today; no logical dump due`); continue; }
const t0 = exec.now();
const t0 = exec.now();
try {
try {
- // The worker runs PRAGMA quick_check on the copy and fails unless it answers ok.
- await exec.sqliteBackup(db.path, path.join(stage, file), { as: svc.runAs });
+ // dbengine.dump runs pg_dump -w -Fc -f <stage>/<name>.dump as postgres and logs it.
+ await dbengine.dump(exec, db, path.join(stage, file), { log });
} catch (err) {
} catch (err) {
- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), error: err.message });
- log(`FAILED ${db.path}: ${err.message}`);
+ files.push({ name: db.name, source: db.database, dest: path.join(dir, file), error: err.message });
+ log(`FAILED ${db.database}: ${err.message}`);
continue;
continue;
}
}
- files.push({ name: db.name, source: db.path, dest: path.join(dir, file), file, check: 'quick_check ok', seconds: Math.round((exec.now() - t0) / 1000) });
+ files.push({ name: db.name, source: db.database, dest: path.join(dir, file), file, check: 'pg_dump -Fc', seconds: Math.round((exec.now() - t0) / 1000) });
}
}
- // Take the staging directory back first: after this the service user can no longer reach
- // anything inside it, so what is checked below cannot change underneath.
+ // Take the staging directory back first: after this the service user (or postgres) can no
+ // longer reach anything inside it, so what is checked below cannot change underneath.
await lockDown(exec, stage, 0o700);
await lockDown(exec, stage, 0o700);
for (const f of files.filter((x) => x.file && !x.error)) {
for (const f of files.filter((x) => x.file && !x.error)) {
const p = path.join(stage, f.file);
const p = path.join(stage, f.file);
@@ -106,13 +235,14 @@ async function backup(ctx, id, { reason = 'manual' } = {}) {
@@ -106,13 +235,14 @@ async function backup(ctx, id, { reason = 'manual' } = {}) {
} finally {
} finally {
if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
if (await exec.stat(stage)) await exec.run('rm', ['-rf', '--', stage], { privileged: true });
}
}
- for (const f of files) delete f.file;
const failed = files.filter((f) => f.error);
const failed = files.filter((f) => f.error);
const copied = files.filter((f) => f.dest && !f.error);
const copied = files.filter((f) => f.dest && !f.error);
- const record = { service: id, at: new Date(exec.now()).toISOString(), reason, dir, ok: !failed.length && copied.length > 0, files };
- // Nothing existed to back up (a service not deployed yet): no directory is left, so an empty
- // backup can never stand in for a real one in retention or a drill.
- if (!copied.length && !failed.length) record.empty = true;
+ // `file` is kept on each entry: the weekly cadence reads it back to find the last good dump.
+ const record = { service: id, at: new Date(exec.now()).toISOString(), reason, engine, ok: !failed.length && copied.length > 0, dir, files };
+ if (verified) record.verified = verified;
+ // Nothing existed to back up (a SQLite service not deployed yet): no directory is left, so an
+ // empty backup can never stand in for a real one in retention or a drill. PostgreSQL never empty.
+ if (!copied.length && !failed.length && !hasPostgres) record.empty = true;
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
await exec.appendFile(path.join(inv.stateDir, 'backups', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
if (failed.length) throw new BackupError(failed.map((f) => `${f.name}: ${f.error}`).join('; '), record);
if (failed.length) throw new BackupError(failed.map((f) => `${f.name}: ${f.error}`).join('; '), record);
return record;
return record;
@@ -160,7 +290,7 @@ async function prune(ctx, id, { daily = 7, weekly = 4 } = {}) {
@@ -160,7 +290,7 @@ async function prune(ctx, id, { daily = 7, weekly = 4 } = {}) {
return { kept: dirs.filter((d) => !remove.includes(d)), removed: remove, unrecorded };
return { kept: dirs.filter((d) => !remove.includes(d)), removed: remove, unrecorded };
}
}
-async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, offsite = null } = {}) {
+async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, offsite = null, logical = false } = {}) {
const { exec, inv, log } = ctx;
const { exec, inv, log } = ctx;
const held = await lock.acquire(exec, inv, '_backup-all');
const held = await lock.acquire(exec, inv, '_backup-all');
const started = exec.now();
const started = exec.now();
@@ -173,14 +303,22 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, of
@@ -173,14 +303,22 @@ async function backupAll(ctx, { daily = 7, weekly = 4, prune: doPrune = true, of
const entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };
const entry = { service: svc.id, status: 'ok', dir: null, files: [], error: null, pruned: [] };
summary.services.push(entry);
summary.services.push(entry);
try {
try {
- const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}` });
- entry.dir = rec.empty ? null : rec.dir;
+ const rec = await backup(ctx, svc.id, { reason: `scheduled ${run}`, logical });
+ entry.dir = rec.empty ? null : (rec.dir || null);
entry.files = rec.files;
entry.files = rec.files;
+ if (rec.verified) entry.verified = rec.verified;
+ // A PostgreSQL service with nothing to dump this run was still verified, not backed
+ // up: it has its own status so the summary distinguishes the two.
if (rec.empty) entry.status = 'skipped';
if (rec.empty) entry.status = 'skipped';
+ else if (rec.engine === 'postgresql' && !rec.dir) entry.status = 'verified';
} catch (err) {
} catch (err) {
entry.status = 'failed';
entry.status = 'failed';
entry.error = err.message;
entry.error = err.message;
- if (err.record) { entry.dir = err.record.dir; entry.files = err.record.files; }
+ if (err.record) {
+ entry.dir = err.record.dir || null;
+ entry.files = err.record.files;
+ if (err.record.verified) entry.verified = err.record.verified;
+ }
log(`✗ ${svc.id}: ${err.message}`);
log(`✗ ${svc.id}: ${err.message}`);
}
}
if (doPrune) {
if (doPrune) {
@@ -220,7 +358,9 @@ const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
@@ -220,7 +358,9 @@ const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
* openvibe_backup_last_run_timestamp_seconds when the last run finished
* openvibe_backup_last_run_timestamp_seconds when the last run finished
* openvibe_backup_last_run_ok 1 when every service (and the off-site copy) succeeded
* openvibe_backup_last_run_ok 1 when every service (and the off-site copy) succeeded
* openvibe_backup_last_success_timestamp_seconds the last run that succeeded (read back from backup-runs)
* openvibe_backup_last_success_timestamp_seconds the last run that succeeded (read back from backup-runs)
- * openvibe_backup_services{status} services ok, failed and skipped in the last run
+ * openvibe_backup_services{status} services ok, failed, skipped and verified in the last run
+ * openvibe_backup_pg_verified{service} 1 when that service's PostgreSQL verification passed
+ * openvibe_pgbackrest_last_backup_age_seconds age of the newest pgBackRest backup, at write time
* openvibe_backup_offsite_ok 1/0 for the last run's off-site copy, when it made one
* openvibe_backup_offsite_ok 1/0 for the last run's off-site copy, when it made one
*
*
* Written only where the collector's directory exists (inventory `textfileDir`, default the Debian
* Written only where the collector's directory exists (inventory `textfileDir`, default the Debian
@@ -241,6 +381,7 @@ async function writeMetrics(ctx, summary) {
@@ -241,6 +381,7 @@ async function writeMetrics(ctx, summary) {
}
}
}
}
const count = (st) => summary.services.filter((x) => x.status === st).length;
const count = (st) => summary.services.filter((x) => x.status === st).length;
+ const verified = summary.services.filter((x) => x.verified);
const lines = [
const lines = [
'# HELP openvibe_backup_last_run_timestamp_seconds When the last ovhost backup --all run finished.',
'# HELP openvibe_backup_last_run_timestamp_seconds When the last ovhost backup --all run finished.',
'# TYPE openvibe_backup_last_run_timestamp_seconds gauge',
'# TYPE openvibe_backup_last_run_timestamp_seconds gauge',
@@ -250,8 +391,22 @@ async function writeMetrics(ctx, summary) {
@@ -250,8 +391,22 @@ async function writeMetrics(ctx, summary) {
`openvibe_backup_last_run_ok ${summary.ok ? 1 : 0}`,
`openvibe_backup_last_run_ok ${summary.ok ? 1 : 0}`,
'# HELP openvibe_backup_services Services by outcome in the last run.',
'# HELP openvibe_backup_services Services by outcome in the last run.',
'# TYPE openvibe_backup_services gauge',
'# TYPE openvibe_backup_services gauge',
- ...['ok', 'failed', 'skipped'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),
+ ...['ok', 'failed', 'skipped', 'verified'].map((st) => `openvibe_backup_services{status="${st}"} ${count(st)}`),
];
];
+ if (verified.length) {
+ lines.push(
+ '# HELP openvibe_backup_pg_verified 1 when a service\'s PostgreSQL backup verification passed in the last run.',
+ '# TYPE openvibe_backup_pg_verified gauge',
+ ...verified.map((s) => `openvibe_backup_pg_verified{service="${s.service}"} ${s.verified.ok ? 1 : 0}`),
+ );
+ // The newest verified stop's age at this run (a snapshot: it does not advance between runs; a run
+ // that stops happening is caught by OpenVibeBackupStale).
+ const stops = verified.filter((s) => s.verified.stop).map((s) => Date.parse(s.verified.stop)).filter(Number.isFinite);
+ if (stops.length) {
+ const age = Math.max(0, Math.floor((exec.now() - Math.max(...stops)) / 1000));
+ lines.push('# HELP openvibe_pgbackrest_last_backup_age_seconds Age of the newest pgBackRest backup at the last run.', '# TYPE openvibe_pgbackrest_last_backup_age_seconds gauge', `openvibe_pgbackrest_last_backup_age_seconds ${age}`);
+ }
+ }
if (lastOk) lines.push('# HELP openvibe_backup_last_success_timestamp_seconds When the last fully successful run finished.', '# TYPE openvibe_backup_last_success_timestamp_seconds gauge', `openvibe_backup_last_success_timestamp_seconds ${at(lastOk)}`);
if (lastOk) lines.push('# HELP openvibe_backup_last_success_timestamp_seconds When the last fully successful run finished.', '# TYPE openvibe_backup_last_success_timestamp_seconds gauge', `openvibe_backup_last_success_timestamp_seconds ${at(lastOk)}`);
if (summary.offsite) lines.push('# HELP openvibe_backup_offsite_ok 1 when the last run\'s off-site copy succeeded.', '# TYPE openvibe_backup_offsite_ok gauge', `openvibe_backup_offsite_ok ${summary.offsite.ok ? 1 : 0}`);
if (summary.offsite) lines.push('# HELP openvibe_backup_offsite_ok 1 when the last run\'s off-site copy succeeded.', '# TYPE openvibe_backup_offsite_ok gauge', `openvibe_backup_offsite_ok ${summary.offsite.ok ? 1 : 0}`);
await exec.writeFile(path.join(dir, 'openvibe_backup.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });
await exec.writeFile(path.join(dir, 'openvibe_backup.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });
@@ -4,36 +4,50 @@
@@ -4,36 +4,50 @@
* the service from a backup next to production and checks that the rebuilt copy answers like
* the service from a backup next to production and checks that the rebuilt copy answers like
* production does:
* production does:
*
*
- * 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Copy each database the
- * drill block names into a fresh directory under <drillDir> owned by the service user, and
- * run PRAGMA integrity_check on each copy. The result must be `ok`.
+ * 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Every database the drill
+ * block names is restored, per engine (lib/dbengine.js):
+ * SQLite copy the file into a fresh directory under <drillDir> owned by the service user
+ * and run PRAGMA integrity_check on the copy. The result must be `ok`.
+ * PostgreSQL check free space first, copy the pg_dump -Fc archive into <tmp>/pg owned by
+ * postgres, verify it with `pg_restore --list`, then create a scratch database and
+ * a login role `ov_<id>_drill_<stamp>` and `pg_restore --no-owner --role=…` the
+ * archive into the scratch database. Both are dropped again in a finally.
* 2. Start a second instance from the production checkout, as the service user, through
* 2. Start a second instance from the production checkout, as the service user, through
* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
* systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
* env file, and a second env file with the drill's overrides: its port, the restored database
* env file, and a second env file with the drill's overrides: its port, the restored database
- * paths, and the switches that turn side effects off. The instance runs sandboxed:
- * ProtectSystem=strict with only the drill directory writable, a private /tmp, binding only its
- * drill port, and no addresses beyond loopback unless the inventory says otherwise.
- * RuntimeMaxSec stops it even if ovhost dies. Loopback stays open (the instance needs Network's
- * public keys), so the inventory's overrides are what keep it away from production services.
+ * paths (or the scratch database's URLs), and the switches that turn side effects off. The
+ * instance runs sandboxed: ProtectSystem=strict with only the drill directory writable, a
+ * private /tmp, binding only its drill port, and no addresses beyond loopback unless the
+ * inventory says otherwise. RuntimeMaxSec stops it even if ovhost dies. Loopback stays open
+ * (the instance needs Network's public keys), so the inventory's overrides are what keep it away
+ * from production services.
* 3. Wait for its readiness path. Then compare the declared read-only GETs with production, byte
* 3. Wait for its readiness path. Then compare the declared read-only GETs with production, byte
* for byte or as JSON without the listed volatile keys, and compare declared row counts
* for byte or as JSON without the listed volatile keys, and compare declared row counts
- * (production against the restored copy).
- * 4. Stop the instance by its own MainPID (never pkill -f), remove the drill directory unless
- * --keep, append the result to <stateDir>/drills/<service>.jsonl, and return a Markdown row
- * for docs/restore-drills.md.
+ * (production against the restored copy, allowing countsTolerance rows).
+ * 4. Stop the instance by its own MainPID (never pkill -f), drop the scratch database and role,
+ * remove the drill directory unless --keep, append the result to <stateDir>/drills/<service>.jsonl,
+ * and return a Markdown row for docs/restore-drills.md.
*
*
* ovhost itself writes only inside the drill directory, the lock file and the drill log. It never
* ovhost itself writes only inside the drill directory, the lock file and the drill log. It never
* writes to the checkout, the env file, the database or the backup. It never reads an env value:
* writes to the checkout, the env file, the database or the backup. It never reads an env value:
* systemd loads the production env file for the instance, and the override file holds only
* systemd loads the production env file for the instance, and the override file holds only
* inventory values. Environment= values from the unit are passed on only if their names do not look
* inventory values. Environment= values from the unit are passed on only if their names do not look
- * secret and the drill does not set them itself (OpenRe's unit sets OPENRE_DB_PATH to production's
+ * secret and the drill does not set them itself (OpenRe's unit sets DATABASE_URL to production's
* database: that line never reaches the drill instance).
* database: that line never reaches the drill instance).
+ *
+ * The production env file is loaded FIRST, so any variable the drill does not set keeps its
+ * production value. For a PostgreSQL service the drill therefore refuses to start unless the drill
+ * block overrides all three of the database URL, the direct database URL and VALKEY_URL: an
+ * un-overridden direct URL would migrate production, an un-overridden URL would serve production
+ * data, and an un-neutralised Valkey URL would write into production's keyspace (the sandbox still
+ * allows loopback).
*/
*/
const path = require('path');
const path = require('path');
const { service } = require('../inventory');
const { service } = require('../inventory');
const envfile = require('../envfile');
const envfile = require('../envfile');
const systemd = require('../systemd');
const systemd = require('../systemd');
const lock = require('../lock');
const lock = require('../lock');
+const dbengine = require('../dbengine');
const { stamp } = require('./backup');
const { stamp } = require('./backup');
class DrillError extends Error {
class DrillError extends Error {
@@ -135,19 +149,26 @@ async function countRows(exec, db, table, as) {
@@ -135,19 +149,26 @@ async function countRows(exec, db, table, as) {
// ── inputs ───────────────────────────────────────────────────────────────────
// ── inputs ───────────────────────────────────────────────────────────────────
-/** The backup to restore: --backup <dir>, else the last record in <stateDir>/backups/<id>.jsonl. */
+/** The backup to restore: --backup <dir>, else the newest usable record in <stateDir>/backups/<id>.jsonl. */
async function resolveBackup(exec, inv, svc, dirArg) {
async function resolveBackup(exec, inv, svc, dirArg) {
let dir = dirArg;
let dir = dirArg;
let takenAt = null;
let takenAt = null;
+ const restored = Object.keys(svc.drill.databases).map((name) => svc.databases.find((d) => d.name === name));
+ const pgTargets = restored.filter((d) => d.engine === 'postgresql');
if (dir) {
if (dir) {
if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');
if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');
dir = path.normalize(dir);
dir = path.normalize(dir);
} else {
} else {
const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${svc.id}.jsonl`));
const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${svc.id}.jsonl`));
- // A failed or empty backup (ok: false) is never restored.
+ // A failed or empty backup (ok: false) is never restored. PostgreSQL records a verification-
+ // only run too (ok: true, no dir, files: []), which holds no artifact to restore: pick the
+ // newest record that carries a .dump entry for every PostgreSQL database, not merely the last.
const records = String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir && r.ok !== false);
const records = String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir && r.ok !== false);
- if (!records.length) throw new DrillError(`no good ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);
- const last = records[records.length - 1];
+ const usable = pgTargets.length
+ ? records.filter((r) => pgTargets.every((d) => (r.files || []).some((f) => f.file === dbengine.fileName(d))))
+ : records;
+ if (!usable.length) throw new DrillError(`no good ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);
+ const last = usable[usable.length - 1];
dir = last.dir;
dir = last.dir;
takenAt = last.at || null;
takenAt = last.at || null;
}
}
@@ -156,11 +177,12 @@ async function resolveBackup(exec, inv, svc, dirArg) {
@@ -156,11 +177,12 @@ async function resolveBackup(exec, inv, svc, dirArg) {
const files = [];
const files = [];
for (const name of Object.keys(svc.drill.databases)) {
for (const name of Object.keys(svc.drill.databases)) {
const db = svc.databases.find((d) => d.name === name);
const db = svc.databases.find((d) => d.name === name);
- const file = path.join(dir, `${name}${path.extname(db.path) || '.db'}`);
+ const target = svc.drill.databases[name];
+ const file = path.join(dir, db.engine === 'postgresql' ? dbengine.fileName(db) : `${name}${path.extname(db.path) || '.db'}`);
const fst = await exec.stat(file);
const fst = await exec.stat(file);
if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);
if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);
- const target = svc.drill.databases[name];
- files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db' });
+ if (db.engine === 'postgresql') files.push({ name, file, production: db, engine: 'postgresql', url: target.url, directUrl: target.directUrl });
+ else files.push({ name, file, production: db.path, envName: target.env, dir: target.dir, ext: path.extname(db.path) || '.db', engine: 'sqlite' });
}
}
return { dir, takenAt, files };
return { dir, takenAt, files };
}
}
@@ -220,6 +242,24 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -220,6 +242,24 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);
if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);
const d = svc.drill;
const d = svc.drill;
if (d.compare.length && !d.productionPort) throw new DrillError(`${id} declares compare paths but no production port`);
if (d.compare.length && !d.productionPort) throw new DrillError(`${id} declares compare paths but no production port`);
+ // The production env file is loaded first, so an override the drill does not set keeps its
+ // production value. Refuse before creating or starting anything if a PostgreSQL service is not
+ // fully pointed at its scratch database and away from production Valkey.
+ const pgRestored = svc.databases.filter((db) => db.engine === 'postgresql' && Object.prototype.hasOwnProperty.call(d.databases, db.name));
+ // A PostgreSQL database the drill does not restore would leave the instance on production's URLs
+ // (and its migrations on production's direct URL): every one must be in drill.databases.
+ const pgUnmapped = svc.databases.filter((db) => db.engine === 'postgresql' && !Object.prototype.hasOwnProperty.call(d.databases, db.name));
+ if (pgUnmapped.length) throw new DrillError(`${id}: drill.databases must map every PostgreSQL database (${pgUnmapped.map((db) => db.name).join(', ')} is missing): otherwise the drill instance would run on production's database`);
+ if (pgRestored.length) {
+ const missingEnv = [];
+ for (const db of pgRestored) {
+ const t = d.databases[db.name];
+ if (!t.url) missingEnv.push(`${db.engine} ${db.database} has no url env var`);
+ if (!t.directUrl) missingEnv.push(`${db.engine} ${db.database} has no directUrl env var`);
+ }
+ if (typeof d.env.VALKEY_URL !== 'string' || d.env.VALKEY_URL === '') missingEnv.push('VALKEY_URL: without it the drill instance would write into the production Valkey keyspace');
+ if (missingEnv.length) throw new DrillError(`${id}: drill.env must set ${missingEnv.join('; ')}`);
+ }
const missing = await missingRequirements(exec, svc);
const missing = await missingRequirements(exec, svc);
if (missing.length) throw new DrillError(`${id}: the deployed checkout (${svc.codeDir}) does not have what this drill's overrides rely on: ${missing.join('; ')}. Deploy a release with the drill switch first.`);
if (missing.length) throw new DrillError(`${id}: the deployed checkout (${svc.codeDir}) does not have what this drill's overrides rely on: ${missing.join('; ')}. Deploy a release with the drill switch first.`);
if (svc.envFile && !(await exec.stat(svc.envFile))) throw new DrillError(`${svc.envFile} not found`);
if (svc.envFile && !(await exec.stat(svc.envFile))) throw new DrillError(`${svc.envFile} not found`);
@@ -252,6 +292,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -252,6 +292,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
counts: [],
counts: [],
acceptance: [],
acceptance: [],
stop: null,
stop: null,
+ scratch: null,
kept: false,
kept: false,
result: 'failed',
result: 'failed',
failure: null,
failure: null,
@@ -259,23 +300,76 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -259,23 +300,76 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const fail = (stage, message) => { if (!record.failure) record.failure = { stage, message }; log(`✗ ${stage}: ${message}`); return new Abort(message); };
const fail = (stage, message) => { if (!record.failure) record.failure = { stage, message }; log(`✗ ${stage}: ${message}`); return new Abort(message); };
let created = false;
let created = false;
let productionPids = [];
let productionPids = [];
+ const scratches = [];
try {
try {
+ // Free space before anything is restored: pg_restore needs room for the restored tables on
+ // top of the archive plus the database's own WAL and temporary files.
+ const pgFiles = bk.files.filter((f) => f.engine === 'postgresql');
+ if (pgFiles.length) {
+ let dumpSize = 0;
+ for (const f of pgFiles) { const s = await exec.stat(f.file); dumpSize += s ? s.size : 0; }
+ const needed = 2 * dumpSize + 1024 ** 3;
+ // The drill directory may not exist yet on a host that has never run a drill, and statfs
+ // works on an existing path; create it first (the mkdir below is then a harmless no-op).
+ await exec.mkdir(inv.drillDir, { mode: 0o711 });
+ const free = (await exec.statfs(inv.drillDir)).free;
+ if (free < needed) throw fail('space', `not enough free space under ${inv.drillDir}: ${free} bytes free, need ${needed} (2 × ${dumpSize} bytes of dump + 1 GiB)`);
+ // A drill killed before its finally ran leaves its scratch database and role behind. This
+ // service's drill lock is held, so anything under its prefix belongs to no running drill.
+ const prefix = `ov_${id.replace(/[^a-z0-9_]/g, '_')}_drill_`;
+ const left = await dbengine.scratchLeftovers(exec, prefix);
+ for (const name of left.databases) { log(`dropping a scratch database an earlier drill left: ${name}`); await dbengine.dropDatabase(exec, name); }
+ for (const name of left.roles) { log(`dropping a scratch role an earlier drill left: ${name}`); await dbengine.dropRole(exec, name); }
+ }
if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
await exec.mkdir(inv.drillDir, { mode: 0o711 });
await exec.mkdir(inv.drillDir, { mode: 0o711 });
- await exec.mkdir(tmp, { owner: svc.runAs, mode: 0o700 });
+ // postgres reads the dump copy in <tmp>/pg, so it must be able to traverse <tmp> (0711: no listing;
+ // every file inside keeps its own restrictive mode). A SQLite-only drill keeps <tmp> at 0700.
+ await exec.mkdir(tmp, { owner: svc.runAs, mode: pgFiles.length ? 0o711 : 0o700 });
created = true;
created = true;
await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });
await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });
+ // postgres (not the service user) reads a PostgreSQL dump, so it gets its own directory; the
+ // copy inside is 0600 postgres (the backup itself is root-only, which postgres cannot read).
+ if (pgFiles.length) await exec.mkdir(path.join(tmp, 'pg'), { owner: dbengine.PG_USER, mode: 0o700 });
const vars = { tmp, port: d.port, db: {} };
const vars = { tmp, port: d.port, db: {} };
// A data-directory database keeps its production file name: the service finds it by name.
// A data-directory database keeps its production file name: the service finds it by name.
const dirOf = (f) => (f.dir ? expand(f.dir, vars) : null);
const dirOf = (f) => (f.dir ? expand(f.dir, vars) : null);
- for (const f of bk.files) vars.db[f.name] = f.dir ? path.join(dirOf(f), path.basename(f.production)) : path.join(tmp, 'db', `${f.name}${f.ext}`);
+ for (const f of bk.files) { if (f.engine === 'postgresql') continue; vars.db[f.name] = f.dir ? path.join(dirOf(f), path.basename(f.production)) : path.join(tmp, 'db', `${f.name}${f.ext}`); }
for (const dir of new Set(bk.files.filter((f) => f.dir).map(dirOf))) await exec.mkdir(dir, { owner: svc.runAs, mode: 0o700 });
for (const dir of new Set(bk.files.filter((f) => f.dir).map(dirOf))) await exec.mkdir(dir, { owner: svc.runAs, mode: 0o700 });
for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });
for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });
for (const b of d.bind) if (!(await exec.stat(expand(b.from, vars)))) await exec.mkdir(expand(b.from, vars), { owner: svc.runAs, mode: 0o700 });
for (const b of d.bind) if (!(await exec.stat(expand(b.from, vars)))) await exec.mkdir(expand(b.from, vars), { owner: svc.runAs, mode: 0o700 });
// 1. restore and check every database copy
// 1. restore and check every database copy
+ const sid = id.replace(/[^a-z0-9_]/g, '_');
+ const scratchFor = (name) => scratches.find((s) => s.name === name);
for (const f of bk.files) {
for (const f of bk.files) {
+ if (f.engine === 'postgresql') {
+ // A PostgreSQL restore replaces the file copy: root copies the root-only archive into
+ // <tmp>/pg owned by postgres, verifies it, then creates a scratch database and role and
+ // restores the archive into them (--no-owner, --role: the archive names production's
+ // role, and every object must end up owned by the drill role the instance connects as).
+ const copy = path.join(tmp, 'pg', `${f.name}.dump`);
+ log(`restoring ${f.file} → ${copy} (owned by ${dbengine.PG_USER})`);
+ const cp = await exec.run('install', ['-o', dbengine.PG_USER, '-m', '0600', '-T', '--', f.file, copy], { privileged: true });
+ if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
+ const st = await exec.stat(copy);
+ const entry = { name: f.name, source: f.file, copy, bytes: st ? st.size : null, integrity: null, engine: 'postgresql' };
+ record.databases.push(entry);
+ try { await dbengine.verifyDump(exec, copy, { as: dbengine.PG_USER }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }
+ entry.integrity = 'pg_restore --list ok';
+ log(`pg_restore --list ${f.name}: ok`);
+ // The role and the database share the same name (separate namespaces); dbengine always
+ // quotes the identifier, so the stamp's hyphen is fine.
+ const name = `ov_${sid}_drill_${at}${pgFiles.length > 1 ? `_${pgFiles.indexOf(f)}` : ''}`;
+ const { password } = await dbengine.createRole(exec, name, {});
+ const scratch = { name: f.name, database: name, role: name, password, dropped: null, error: null };
+ scratches.push(scratch);
+ if (!record.scratch) record.scratch = { database: name, role: name };
+ await dbengine.createDatabase(exec, name, { owner: name });
+ await dbengine.restore(exec, f.production, { dump: copy, database: name, role: name, log });
+ continue;
+ }
const dest = vars.db[f.name];
const dest = vars.db[f.name];
log(`restoring ${f.file} → ${dest} (owned by ${svc.runAs})`);
log(`restoring ${f.file} → ${dest} (owned by ${svc.runAs})`);
// Backups are root-only (0600), so root copies; install unlinks whatever is at dest
// Backups are root-only (0600), so root copies; install unlinks whatever is at dest
@@ -283,7 +377,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -283,7 +377,7 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const cp = await exec.run('install', ['-o', svc.runAs, '-m', '0600', '-T', '--', f.file, dest], { privileged: true });
const cp = await exec.run('install', ['-o', svc.runAs, '-m', '0600', '-T', '--', f.file, dest], { privileged: true });
if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
const st = await exec.stat(dest);
const st = await exec.stat(dest);
- const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null };
+ const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null, engine: 'sqlite' };
record.databases.push(entry);
record.databases.push(entry);
let rows;
let rows;
try { rows = await exec.sqlite(dest, 'PRAGMA integrity_check', { as: svc.runAs }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }
try { rows = await exec.sqlite(dest, 'PRAGMA integrity_check', { as: svc.runAs }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }
@@ -292,13 +386,28 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -292,13 +386,28 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
log(`integrity_check ${f.name}: ok`);
log(`integrity_check ${f.name}: ok`);
}
}
const restoredCounts = [];
const restoredCounts = [];
- for (const c of d.counts) restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));
+ for (const c of d.counts) {
+ const rdb = svc.databases.find((x) => x.name === c.db);
+ // PostgreSQL counts come from the scratch database over psql, SQLite from the restored file.
+ if (rdb.engine === 'postgresql') restoredCounts.push(await dbengine.countRows(exec, { engine: 'postgresql', database: scratchFor(c.db).database }, countSql(c.table)));
+ else restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));
+ }
// 2. start the drill instance
// 2. start the drill instance
const lines = ['# written by ovhost drill; no secret values: the production env file is loaded separately'];
const lines = ['# written by ovhost drill; no secret values: the production env file is loaded separately'];
for (const [k, v] of Object.entries(d.env)) lines.push(envLine(k, expand(v, vars)));
for (const [k, v] of Object.entries(d.env)) lines.push(envLine(k, expand(v, vars)));
const dbEnv = new Map();
const dbEnv = new Map();
- for (const f of bk.files) if (f.envName) dbEnv.set(f.envName, f.dir ? dirOf(f) : vars.db[f.name]);
+ for (const f of bk.files) {
+ if (f.engine === 'postgresql') {
+ // Both names point at the scratch database over the direct cluster port (5432, never
+ // PgBouncer's 6432: the scratch database and role exist only in the cluster). The
+ // password is generated per drill and lives only in this file the instance reads.
+ const s = scratchFor(f.name);
+ const url = dbengine.databaseUrl({ database: s.database, role: s.role, password: s.password });
+ dbEnv.set(f.url, url);
+ dbEnv.set(f.directUrl, url);
+ } else if (f.envName) dbEnv.set(f.envName, f.dir ? dirOf(f) : vars.db[f.name]);
+ }
for (const [k, v] of dbEnv) lines.push(envLine(k, v));
for (const [k, v] of dbEnv) lines.push(envLine(k, v));
const envPath = path.join(tmp, 'drill.env');
const envPath = path.join(tmp, 'drill.env');
await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });
await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });
@@ -374,11 +483,17 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -374,11 +483,17 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
const db = svc.databases.find((x) => x.name === c.db);
const db = svc.databases.find((x) => x.name === c.db);
let production = null;
let production = null;
let error = null;
let error = null;
- try { production = await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }
- const row = { db: c.db, table: c.table, production, restored: restoredCounts[i], match: production === restoredCounts[i] };
+ // PostgreSQL counts come from the cluster over psql (the inventory entry itself, never
+ // .path, which a PostgreSQL entry does not have); SQLite from the production file.
+ try { production = db.engine === 'postgresql' ? await dbengine.countRows(exec, db, countSql(c.table)) : await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }
+ const tol = d.countsTolerance || 0;
+ // A busy table drifts between the backup and now: countsTolerance (default 0) allows it.
+ const match = production != null && restoredCounts[i] != null && Math.abs(production - restoredCounts[i]) <= tol;
+ const row = { db: c.db, table: c.table, production, restored: restoredCounts[i], match };
+ if (tol) row.tolerance = tol;
if (error) row.error = error;
if (error) row.error = error;
record.counts.push(row);
record.counts.push(row);
- log(`${row.match ? 'same' : 'DIFFERENT'} ${c.table}: production ${production == null ? `unknown (${error})` : production}, restored ${restoredCounts[i]}`);
+ log(`${row.match ? 'same' : 'DIFFERENT'} ${c.table}: production ${production == null ? `unknown (${error})` : production}, restored ${restoredCounts[i]}${tol ? ` (±${tol})` : ''}`);
}
}
const bad = [...record.compare.filter((r) => !r.match).map((r) => `${r.path} (${r.detail})`), ...record.counts.filter((r) => !r.match).map((r) => `${r.table} ${r.production} ≠ ${r.restored}`)];
const bad = [...record.compare.filter((r) => !r.match).map((r) => `${r.path} (${r.detail})`), ...record.counts.filter((r) => !r.match).map((r) => `${r.table} ${r.production} ≠ ${r.restored}`)];
if (bad.length) throw fail('compare', `differs from production: ${bad.join('; ')}`);
if (bad.length) throw fail('compare', `differs from production: ${bad.join('; ')}`);
@@ -401,10 +516,27 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -401,10 +516,27 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
if (unaccepted.length) throw fail('acceptance', unaccepted.join('; '));
if (unaccepted.length) throw fail('acceptance', unaccepted.join('; '));
} catch (err) {
} catch (err) {
if (!(err instanceof Abort)) fail('error', err.message);
if (!(err instanceof Abort)) fail('error', err.message);
+ } finally {
+ // Cleanup ALWAYS, on success and on failure: stop the drill instance first (so nothing is
+ // connected to the scratch database), then drop the scratch database and role. dropdb is
+ // attempted before dropRole so a dropdb refusal (a session still connected) cannot stop the
+ // role attempt. A failure to drop is logged and recorded, never thrown: it must not mask the
+ // drill's own outcome.
+ // A throw while stopping must not skip the drops below (the leftover sweep only runs on the next drill).
+ try { record.stop = await stopInstance(exec, record.pid, d.port, productionPids, log); } catch (err) { record.stop = { ok: false, detail: `stop failed: ${err.message}` }; log(`could not stop the drill instance: ${err.message}`); }
+ for (const s of scratches) {
+ try { await dbengine.dropDatabase(exec, s.database); } catch (err) { s.dropped = false; s.error = `dropdb: ${err.message}`; log(`could not drop the scratch database ${s.database}: ${err.message}`); }
+ try { await dbengine.dropRole(exec, s.role); } catch (err) { s.dropped = false; s.error = [s.error, `droprole: ${err.message}`].filter(Boolean).join('; '); log(`could not drop the scratch role ${s.role}: ${err.message}`); }
+ if (s.dropped === null) s.dropped = true;
+ }
+ if (record.scratch && scratches.length) {
+ record.scratch.dropped = scratches.every((s) => s.dropped);
+ const dropErrors = scratches.filter((s) => s.error).map((s) => s.error);
+ if (dropErrors.length) record.scratch.error = dropErrors.join('; ');
+ }
}
}
- // 4. stop, clean up, record
- record.stop = await stopInstance(exec, record.pid, d.port, productionPids, log);
+ // 4. stop outcome, remove the drill directory, record
if (record.stop && !record.stop.ok) fail('stop', record.stop.detail);
if (record.stop && !record.stop.ok) fail('stop', record.stop.detail);
if (created) {
if (created) {
const safe = tmp.startsWith(`${inv.drillDir}/`) && /^[a-z][a-z0-9-]*-\d{8}-\d{6}$/.test(path.basename(tmp));
const safe = tmp.startsWith(`${inv.drillDir}/`) && /^[a-z][a-z0-9-]*-\d{8}-\d{6}$/.test(path.basename(tmp));
@@ -454,13 +586,15 @@ function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }
@@ -454,13 +586,15 @@ function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }
/** A row for the table in docs/restore-drills.md. */
/** A row for the table in docs/restore-drills.md. */
function markdownRow(r) {
function markdownRow(r) {
const checks = [];
const checks = [];
- for (const db of r.databases) checks.push(`\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
+ for (const db of r.databases) checks.push(db.engine === 'postgresql'
+ ? `\`pg_restore --list\` (${db.name}) = ${db.integrity === 'pg_restore --list ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`
+ : `\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
if (same.length) checks.push(`${same.join(', ')} identical to production${r.compare.some((c) => c.match && c.mode && c.mode !== 'bytes') ? ' (JSON without volatile keys where declared)' : ''}`);
if (same.length) checks.push(`${same.join(', ')} identical to production${r.compare.some((c) => c.match && c.mode && c.mode !== 'bytes') ? ' (JSON without volatile keys where declared)' : ''}`);
checks.push(...diff);
checks.push(...diff);
- for (const c of r.counts) checks.push(`${c.table} ${c.production == null ? '?' : c.production} ${c.match ? '=' : '≠'} ${c.restored}`);
+ for (const c of r.counts) checks.push(`${c.table} ${c.production == null ? '?' : c.production} ${c.match ? '=' : '≠'} ${c.restored}${c.tolerance ? ` (±${c.tolerance})` : ''}`);
const accepted = (r.acceptance || []).filter((a) => a.ok).map((a) => `\`${a.path}\``);
const accepted = (r.acceptance || []).filter((a) => a.ok).map((a) => `\`${a.path}\``);
if (accepted.length) checks.push(`acceptance ${accepted.join(', ')}`);
if (accepted.length) checks.push(`acceptance ${accepted.join(', ')}`);
if (r.failure && r.failure.stage !== 'compare' && r.failure.stage !== 'acceptance') checks.push(`${r.failure.stage}: ${cell(r.failure.message)}`);
if (r.failure && r.failure.stage !== 'compare' && r.failure.stage !== 'acceptance') checks.push(`${r.failure.stage}: ${cell(r.failure.message)}`);
new file mode 100644
new file mode 100644
@@ -0,0 +1,222 @@
@@ -0,0 +1,222 @@
+'use strict';
+/**
+ * Per-engine database helpers. lib/inventory.js normalises every entry in `databases[]` to an
+ * `engine` of "sqlite" (the default: a file) or "postgresql" (a database in the cluster), so no
+ * caller has to know which commands an engine needs.
+ *
+ * SQLite keeps the existing worker (lib/sqlite-worker.js): every file operation runs as the service
+ * user, and root never opens a service database. PostgreSQL runs as the postgres OS user over peer
+ * auth (`local postgres peer` in roles/data/files/pg_hba.conf) — the same property: ovhost reads no
+ * service env value, and never holds a database password except the one it generates itself for a
+ * restore drill.
+ *
+ * pgBackRest, not ovhost, is the durability layer for the cluster (roles/data/provision.sh: one
+ * stanza `openvibe`, full weekly + differential nightly + WAL to object storage). What ovhost does
+ * per night is verify that layer (verifyPgBackRest, walFailedCount) and, on a slower cadence, take
+ * a per-service logical dump (dump) that the off-site code encrypts and uploads unchanged.
+ */
+const crypto = require('crypto');
+const path = require('path');
+
+// The OS user PostgreSQL's own tools run as (peer auth), the stanza provision.sh creates, and the
+// direct connection a drill must use: 5432 is the cluster, 6432 is PgBouncer.
+const PG_USER = 'postgres';
+const PG_STANZA = 'openvibe';
+const PG_DIRECT_HOST = '127.0.0.1';
+const PG_DIRECT_PORT = 5432;
+// A port nothing listens on: a drill instance keeps its production Valkey prefix, so it must not be
+// able to reach a real Valkey (it would write into the production keyspace).
+const CLOSED_VALKEY_URL = 'redis://127.0.0.1:9/0';
+
+const WORK_TIMEOUT_MS = 6 * 3600 * 1000;
+
+const ident = (name) => `"${String(name).replace(/"/g, '""')}"`;
+const literal = (value) => `'${String(value).replace(/'/g, "''")}'`;
+const scrubbed = (r) => ((r.stderr || r.stdout || '').trim().slice(0, 300) || `exit ${r.code}`);
+
+/** What a log line and a record call this database. */
+function label(db) {
+ return db.engine === 'postgresql' ? db.database : db.path;
+}
+
+/** The file name this database's artifact gets inside a backup directory. */
+function fileName(db) {
+ return db.engine === 'postgresql' ? `${db.name}.dump` : `${db.name}${path.extname(db.path) || '.db'}`;
+}
+
+/** Whether the database is still there (SQLite: the file; PostgreSQL: the database in the cluster). */
+async function exists(exec, db) {
+ if (db.engine === 'sqlite') return !!(await exec.stat(db.path));
+ const rows = await exec.psql('postgres', `SELECT 1 AS present FROM pg_database WHERE datname = ${literal(db.database)}`);
+ return rows.length > 0;
+}
+
+/**
+ * A logical dump of one database into `dest` (a `pg_dump -Fc` archive: compressed, and restorable
+ * table by table with pg_restore). It runs as postgres, so `dest` must be writable by postgres —
+ * the caller's staging directory is owned by postgres when any database is PostgreSQL.
+ */
+async function dump(exec, db, dest, { log } = {}) {
+ if (db.engine !== 'postgresql') throw new Error('sqlite dumps go through exec.sqliteBackup (the worker runs as the service user)');
+ // -w: peer auth needs no password, and a prompt in the nightly unit would hang until the timeout.
+ const r = await exec.run('pg_dump', ['-w', '-Fc', '-f', dest, '-d', db.database], { as: PG_USER, timeoutMs: WORK_TIMEOUT_MS });
+ if (r.code !== 0) throw new Error(`pg_dump ${db.database} failed: ${scrubbed(r)}`);
+ if (log) log(`pg_dump ${db.database} → ${dest}`);
+ return dest;
+}
+
+/**
+ * Whether `file` is a readable pg_dump archive (pg_restore --list parses its table of contents).
+ * --list only reads the file and contacts no server, so a caller holding a root-only copy can pass
+ * `as: null` and check it as itself rather than making it readable to postgres.
+ */
+async function verifyDump(exec, file, { as = PG_USER } = {}) {
+ const r = await exec.run('pg_restore', ['--list', file], { as, timeoutMs: 120000 });
+ if (r.code !== 0) throw new Error(`pg_restore --list ${file} failed: ${scrubbed(r)}`);
+ return true;
+}
+
+/** A database role with a freshly generated password. -> { role, password } */
+async function createRole(exec, role, { password = null } = {}) {
+ const pw = password || crypto.randomBytes(24).toString('hex');
+ // The password reaches psql on stdin, never in argv: /proc/<pid>/cmdline is world-readable.
+ const sql = `CREATE ROLE ${ident(role)} LOGIN PASSWORD ${literal(pw)};\n`;
+ const r = await exec.run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '-d', 'postgres', '-f', '-'], { as: PG_USER, input: sql, timeoutMs: 120000 });
+ if (r.code !== 0) throw new Error(`creating role ${role} failed: ${scrubbed(r)}`);
+ return { role, password: pw };
+}
+
+async function createDatabase(exec, database, { owner = null } = {}) {
+ const args = ['-w'];
+ if (owner) args.push(`--owner=${owner}`);
+ args.push(database);
+ const r = await exec.run('createdb', args, { as: PG_USER, timeoutMs: 120000 });
+ if (r.code !== 0) throw new Error(`createdb ${database} failed: ${scrubbed(r)}`);
+ return database;
+}
+
+async function dropDatabase(exec, database) {
+ // --force ends any session still connected (a drill instance that did not stop): only ever called
+ // with a scratch database's name, never a production one.
+ const r = await exec.run('dropdb', ['-w', '--if-exists', '--force', database], { as: PG_USER, timeoutMs: 120000 });
+ if (r.code !== 0) throw new Error(`dropdb ${database} failed: ${scrubbed(r)}`);
+}
+
+async function dropRole(exec, role) {
+ const sql = `DROP ROLE IF EXISTS ${ident(role)};\n`;
+ const r = await exec.run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '-d', 'postgres', '-f', '-'], { as: PG_USER, input: sql, timeoutMs: 120000 });
+ if (r.code !== 0) throw new Error(`dropping role ${role} failed: ${scrubbed(r)}`);
+}
+
+/**
+ * Restore a pg_dump archive into `database`, under `role`. --no-owner because the archive names the
+ * production role (which the scratch database must not depend on); --role makes every object land
+ * owned by the drill role, so the drilled service can read and write it.
+ */
+async function restore(exec, db, { dump: dumpFile, database, role, log } = {}) {
+ if (db.engine !== 'postgresql') throw new Error('sqlite restore is a file copy, not pg_restore');
+ // --no-privileges: the archive's GRANTs name production roles; the scratch copy must not hand them anything.
+ const args = ['-w', '--no-owner', '--no-privileges'];
+ if (role) args.push(`--role=${role}`);
+ args.push('-d', database, dumpFile);
+ const r = await exec.run('pg_restore', args, { as: PG_USER, timeoutMs: WORK_TIMEOUT_MS });
+ if (r.code !== 0) throw new Error(`pg_restore into ${database} failed: ${scrubbed(r)}`);
+ if (log) log(`pg_restore ${dumpFile} → ${database} (as ${role || PG_USER})`);
+ return database;
+}
+
+/**
+ * Scratch databases and roles an earlier drill left behind (killed before its cleanup ran): every name
+ * that starts with `prefix` (ov_<service>_drill_). -> { databases: [], roles: [] }
+ */
+async function scratchLeftovers(exec, prefix) {
+ const p = literal(prefix);
+ const databases = await exec.psql('postgres', `SELECT datname AS name FROM pg_database WHERE left(datname, length(${p})) = ${p}`);
+ const roles = await exec.psql('postgres', `SELECT rolname AS name FROM pg_roles WHERE left(rolname, length(${p})) = ${p}`);
+ const names = (rows) => (rows || []).map((r) => r.name).filter((n) => typeof n === 'string' && n.startsWith(prefix));
+ return { databases: names(databases), roles: names(roles) };
+}
+
+/** A row count from either engine. `sql` must return the number in its first column. */
+async function countRows(exec, db, sql, { as } = {}) {
+ const rows = db.engine === 'postgresql' ? await exec.psql(db.database, sql) : await exec.sqlite(db.path, sql, { as });
+ const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);
+ if (!Number.isFinite(n)) throw new Error(`count of ${label(db)} returned no number`);
+ return n;
+}
+
+/** The connection URL a drill instance must use: the cluster directly, never PgBouncer. */
+function databaseUrl({ database, role, password }) {
+ return `postgresql://${encodeURIComponent(role)}:${encodeURIComponent(password)}@${PG_DIRECT_HOST}:${PG_DIRECT_PORT}/${encodeURIComponent(database)}`;
+}
+
+/** `pgbackrest info --output=json` as postgres. */
+async function pgbackrestInfo(exec) {
+ const r = await exec.run('pgbackrest', [`--stanza=${PG_STANZA}`, 'info', '--output=json'], { as: PG_USER, timeoutMs: 300000 });
+ if (r.code !== 0) throw new Error(`pgbackrest info failed: ${scrubbed(r)}`);
+ let stanzas;
+ try { stanzas = JSON.parse(r.stdout); } catch { throw new Error(`pgbackrest info did not answer JSON: ${String(r.stdout).trim().slice(0, 200)}`); }
+ return Array.isArray(stanzas) ? stanzas : [];
+}
+
+/** The newest backup in a stanza's `info`: { type, stop (epoch seconds) } or null. */
+function newestBackup(stanza) {
+ let best = null;
+ for (const b of stanza.backup || []) {
+ const stop = b && b.timestamp ? Number(b.timestamp.stop) : NaN;
+ if (!Number.isFinite(stop)) continue;
+ if (!best || stop > best.stop) best = { type: b.type || 'unknown', stop };
+ }
+ return best;
+}
+
+/**
+ * The every-run pgBackRest verification: the stanza exists and is ok, and its newest backup stopped
+ * within `maxAgeSeconds` (26 h: nightly differentials at 02:15, with room for a slow one).
+ * -> { type, stop (ISO), ageSeconds }
+ */
+async function verifyPgBackRest(exec, { now, maxAgeSeconds = 26 * 3600 } = {}) {
+ const stanzas = await pgbackrestInfo(exec);
+ const stanza = stanzas.find((s) => s && s.name === PG_STANZA);
+ if (!stanza) throw new Error(`pgBackRest has no stanza ${PG_STANZA}: the cluster is not being backed up`);
+ const code = Number((stanza.status && stanza.status.code) || 0);
+ if (code !== 0) throw new Error(`pgBackRest stanza ${PG_STANZA} is not ok: ${(stanza.status && stanza.status.message) || `status code ${code}`}`);
+ const newest = newestBackup(stanza);
+ if (!newest) throw new Error(`pgBackRest stanza ${PG_STANZA} holds no backup at all`);
+ const age = Math.floor(now / 1000) - newest.stop;
+ if (age > maxAgeSeconds) throw new Error(`the newest pgBackRest backup (${newest.type}) stopped ${(age / 3600).toFixed(1)}h ago (limit ${maxAgeSeconds / 3600}h)`);
+ return { type: newest.type, stop: new Date(newest.stop * 1000).toISOString(), ageSeconds: age };
+}
+
+/** `pg_stat_archiver.failed_count`: the cluster-wide count of WAL segments archiving ever failed on. */
+async function walFailedCount(exec) {
+ const rows = await exec.psql('postgres', 'SELECT failed_count::text AS n FROM pg_stat_archiver');
+ const n = Number(rows && rows[0] ? rows[0].n : NaN);
+ if (!Number.isFinite(n)) throw new Error('pg_stat_archiver returned no failed_count');
+ return n;
+}
+
+module.exports = {
+ PG_USER,
+ PG_STANZA,
+ PG_DIRECT_HOST,
+ PG_DIRECT_PORT,
+ CLOSED_VALKEY_URL,
+ label,
+ fileName,
+ exists,
+ dump,
+ verifyDump,
+ restore,
+ createRole,
+ createDatabase,
+ dropDatabase,
+ dropRole,
+ scratchLeftovers,
+ countRows,
+ databaseUrl,
+ pgbackrestInfo,
+ verifyPgBackRest,
+ newestBackup,
+ walFailedCount,
+};
@@ -23,6 +23,11 @@
@@ -23,6 +23,11 @@
* any method, http:// or https:// (release notifications: Network's token endpoint and Events'
* any method, http:// or https:// (release notifications: Network's token endpoint and Events'
* publish API). Headers and body may carry a secret: nothing here logs them.
* publish API). Headers and body may carry a secret: nothing here logs them.
* sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })
* sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })
+ * psql(database, sql, { as }) -> rows. PostgreSQL is reached as the postgres OS user over peer
+ * auth (`local postgres peer`, roles/data/files/pg_hba.conf) — the same no-secret property the
+ * SQLite worker gives the service user. Everything else PostgreSQL (pg_dump, pg_restore,
+ * createdb, dropdb, pgbackrest) goes through run() with { as: 'postgres' }.
+ * statfs(path) -> { free, size } bytes, for the free-space check before a restore
* listeners(port) -> [{ pid, process }]
* listeners(port) -> [{ pid, process }]
* kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)
* kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)
* sleep(ms) ; now() -> ms ; isRoot() ; userName() ; hostname()
* sleep(ms) ; now() -> ms ; isRoot() ; userName() ; hostname()
@@ -232,6 +237,28 @@ function createSystemExecutor({ env = process.env } = {}) {
@@ -232,6 +237,28 @@ function createSystemExecutor({ env = process.env } = {}) {
return out;
return out;
}
}
+ /** `psql --csv`: a header row and one line per row, so a caller reads columns by name. */
+ function parseCsv(text) {
+ const rows = [];
+ let row = [];
+ let field = '';
+ let quoted = false;
+ for (let i = 0; i < text.length; i++) {
+ const c = text[i];
+ if (quoted) {
+ if (c === '"') { if (text[i + 1] === '"') { field += '"'; i++; } else quoted = false; }
+ else field += c;
+ } else if (c === '"') quoted = true;
+ else if (c === ',') { row.push(field); field = ''; }
+ else if (c === '\n') { row.push(field); rows.push(row); row = []; field = ''; }
+ else if (c !== '\r') field += c;
+ }
+ if (field !== '' || row.length) { row.push(field); rows.push(row); }
+ if (!rows.length) return [];
+ const header = rows[0];
+ return rows.slice(1).filter((r) => r.length > 1 || r[0] !== '').map((r) => Object.fromEntries(header.map((h, i) => [h, r[i] === undefined ? null : r[i]])));
+ }
+
async function listeners(port) {
async function listeners(port) {
const r = await run('ss', ['-H', '-ltnp', `sport = :${port}`], { privileged: true });
const r = await run('ss', ['-H', '-ltnp', `sport = :${port}`], { privileged: true });
if (r.code !== 0) throw new Error(`ss failed: ${r.stderr.trim()}`);
if (r.code !== 0) throw new Error(`ss failed: ${r.stderr.trim()}`);
@@ -266,6 +293,15 @@ function createSystemExecutor({ env = process.env } = {}) {
@@ -266,6 +293,15 @@ function createSystemExecutor({ env = process.env } = {}) {
request,
request,
sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,
sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,
sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },
sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },
+ psql: async (database, sql, { as = 'postgres' } = {}) => {
+ const r = await run('psql', ['-X', '-q', '-v', 'ON_ERROR_STOP=1', '--csv', '-d', database, '-c', sql], { as, timeoutMs: 6 * 3600 * 1000 });
+ if (r.code !== 0) throw new Error(`psql ${database} failed: ${(r.stderr || r.stdout).trim().slice(0, 300)}`);
+ return parseCsv(r.stdout);
+ },
+ statfs: async (p) => {
+ const s = fs.statfsSync(p);
+ return { free: s.bavail * s.bsize, size: s.blocks * s.bsize };
+ },
listeners,
listeners,
kill: async (pid, signal = 'SIGTERM') => {
kill: async (pid, signal = 'SIGTERM') => {
if (!Number.isInteger(pid) || pid <= 1) throw new Error(`refusing to signal pid ${pid}`);
if (!Number.isInteger(pid) || pid <= 1) throw new Error(`refusing to signal pid ${pid}`);
@@ -20,7 +20,10 @@ const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
@@ -20,7 +20,10 @@ const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
-const PROBE_KINDS = ['http-json-count', 'sqlite-count', 'sum'];
+const PROBE_KINDS = ['http-json-count', 'sqlite-count', 'postgresql-count', 'sum'];
+// A PostgreSQL database an ovhost service owns: one per service in the cluster pgBackRest backs up
+// (roles/data/provision.sh), reached as the postgres OS user over peer auth (no secret is ever read).
+const PG_DATABASE_RE = /^ov_[a-z0-9_]+$/;
function abs(value, where) {
function abs(value, where) {
if (typeof value !== 'string' || !path.isAbsolute(value) || value.includes('\0')) throw new InventoryError(`${where} must be an absolute path`);
if (typeof value !== 'string' || !path.isAbsolute(value) || value.includes('\0')) throw new InventoryError(`${where} must be an absolute path`);
@@ -209,10 +212,28 @@ function normaliseService(id, raw, defaults) {
@@ -209,10 +212,28 @@ function normaliseService(id, raw, defaults) {
noRestartPaths: s.noRestartPaths || [],
noRestartPaths: s.noRestartPaths || [],
protected: s.protected || null,
protected: s.protected || null,
drain: { policy: 'refuse', waitMaxSeconds: 8 * 3600, pollSeconds: 60, quietChecks: 2, ...(s.drain || {}) },
drain: { policy: 'refuse', waitMaxSeconds: 8 * 3600, pollSeconds: 60, quietChecks: 2, ...(s.drain || {}) },
+ // A database is one of two engines. SQLite (the default) is a file the service user's worker
+ // copies; PostgreSQL is a database in the cluster, named, and backed up by pgBackRest as a
+ // whole — ovhost never opens it directly. Nothing outside the engine's own code may assume
+ // `.path` exists.
databases: (s.databases || []).map((d, i) => {
databases: (s.databases || []).map((d, i) => {
- const name = d.name || path.basename(String(d.path), '.db');
- if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${where}.databases[${i}].name must be a plain name`);
- return { name, path: abs(d.path, `${where}.databases[${i}].path`) };
+ const at = `${where}.databases[${i}]`;
+ const engine = d.engine == null ? 'sqlite' : d.engine;
+ if (engine === 'sqlite') {
+ if (d.database != null) throw new InventoryError(`${at}: a sqlite database is a path, not a database name`);
+ const name = d.name || path.basename(String(d.path), '.db');
+ if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);
+ return { name, engine, path: abs(d.path, `${at}.path`) };
+ }
+ if (engine === 'postgresql') {
+ if (d.path != null) throw new InventoryError(`${at}: a postgresql database has no path (it lives in the cluster, named by "database")`);
+ if (!PG_DATABASE_RE.test(d.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_trade)`);
+ const name = d.name || String(d.database).slice(3);
+ if (!FILE_NAME_RE.test(name)) throw new InventoryError(`${at}.name must be a plain name`);
+ if (d.role != null && !/^[a-z_][a-z0-9_-]*$/.test(String(d.role))) throw new InventoryError(`${at}.role must be a database role name`);
+ return { name, engine, database: d.database, role: d.role == null ? null : String(d.role) };
+ }
+ throw new InventoryError(`${at}.engine must be sqlite or postgresql`);
}),
}),
backupOnChange: s.backupOnChange || [],
backupOnChange: s.backupOnChange || [],
nginx: s.nginx || null,
nginx: s.nginx || null,
@@ -273,6 +294,10 @@ function normaliseService(id, raw, defaults) {
@@ -273,6 +294,10 @@ function normaliseService(id, raw, defaults) {
abs(p.db, `${at}.db`);
abs(p.db, `${at}.db`);
if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
}
}
+ if (p.kind === 'postgresql-count') {
+ if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);
+ if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
+ }
if (p.kind === 'sum') {
if (p.kind === 'sum') {
if (!Array.isArray(p.probes) || !p.probes.length) throw new InventoryError(`${at}.probes must be a non-empty array`);
if (!Array.isArray(p.probes) || !p.probes.length) throw new InventoryError(`${at}.probes must be a non-empty array`);
p.probes.forEach((q, i) => checkProbe(q || {}, `${at}.probes[${i}]`, true));
p.probes.forEach((q, i) => checkProbe(q || {}, `${at}.probes[${i}]`, true));
@@ -287,6 +312,18 @@ function normaliseService(id, raw, defaults) {
@@ -287,6 +312,18 @@ function normaliseService(id, raw, defaults) {
if (n.repoVhosts) rel(n.repoVhosts.replace(/\*\.conf$/, 'x.conf'), `${where}.nginx.repoVhosts`);
if (n.repoVhosts) rel(n.repoVhosts.replace(/\*\.conf$/, 'x.conf'), `${where}.nginx.repoVhosts`);
if (n.variant && !['http', 'sse', 'websocket'].includes(n.variant)) throw new InventoryError(`${where}.nginx.variant must be http, sse or websocket`);
if (n.variant && !['http', 'sse', 'websocket'].includes(n.variant)) throw new InventoryError(`${where}.nginx.variant must be http, sse or websocket`);
if (n.certName && !FILE_NAME_RE.test(n.certName)) throw new InventoryError(`${where}.nginx.certName must be a name`);
if (n.certName && !FILE_NAME_RE.test(n.certName)) throw new InventoryError(`${where}.nginx.certName must be a name`);
+ // The tenants database may be a PostgreSQL database (the Host API is on PostgreSQL): the
+ // verified custom domains are then read over psql as the postgres OS user, not from a file.
+ if (n.tenants && n.tenants.database != null) {
+ const db = n.tenants.database;
+ let tenantDb;
+ if (typeof db === 'string') tenantDb = { engine: 'sqlite', path: abs(db, `${where}.nginx.tenants.database`) };
+ else if (db && typeof db === 'object' && !Array.isArray(db) && db.engine === 'postgresql' && PG_DATABASE_RE.test(db.database || '')) tenantDb = { engine: 'postgresql', database: db.database };
+ else throw new InventoryError(`${where}.nginx.tenants.database must be a path or { "engine": "postgresql", "database": "ov_host" }`);
+ // A fresh object every time: normalise() must never rewrite the document it was handed
+ // (the example inventory is a shared require() cache, and a drill test normalises it twice).
+ out.nginx = { ...n, tenants: { ...n.tenants, database: tenantDb } };
+ }
}
}
if (out.lifecycle !== null && (typeof out.lifecycle !== 'object' || Array.isArray(out.lifecycle))) throw new InventoryError(`${where}.lifecycle must be an object (the manifest's lifecycle block)`);
if (out.lifecycle !== null && (typeof out.lifecycle !== 'object' || Array.isArray(out.lifecycle))) throw new InventoryError(`${where}.lifecycle must be an object (the manifest's lifecycle block)`);
for (const p of out.backupOnChange) rel(p, `${where}.backupOnChange`);
for (const p of out.backupOnChange) rel(p, `${where}.backupOnChange`);
@@ -309,6 +346,12 @@ const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;
@@ -309,6 +346,12 @@ const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;
* DIRECTORY: the copy lands in {tmp}/data/ (or "dir": "{tmp}/<name>") under its
* DIRECTORY: the copy lands in {tmp}/data/ (or "dir": "{tmp}/<name>") under its
* production file name and the env var (optional) points at that directory. Without
* production file name and the env var (optional) points at that directory. Without
* "env", pair the directory with `bind`.
* "env", pair the directory with `bind`.
+ * A database with engine postgresql instead names the two env vars that point the
+ * service at a database URL: { "url": "DATABASE_URL", "directUrl":
+ * "DATABASE_DIRECT_URL" }. The drill creates a scratch database and a scratch role,
+ * restores the dump into it, and writes both vars (and VALKEY_URL, which the drill
+ * must neutralise itself) into drill.env — the service's own env file is loaded first,
+ * so any name the drill does not override keeps its production value.
* env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}
* env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}
* and {db:<name>}
* and {db:<name>}
* dirs directories to create (owned by the service user) inside {tmp} before the start
* dirs directories to create (owned by the service user) inside {tmp} before the start
@@ -349,19 +392,28 @@ function normaliseDrill(raw, svc, where) {
@@ -349,19 +392,28 @@ function normaliseDrill(raw, svc, where) {
const inDataDir = new Map();
const inDataDir = new Map();
for (const [name, target] of Object.entries(rawDatabases)) {
for (const [name, target] of Object.entries(rawDatabases)) {
if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);
if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);
+ const defined = svc.databases.find((d) => d.name === name);
+ if (defined.engine === 'postgresql') {
+ if (!target || typeof target !== 'object' || Array.isArray(target)) throw new InventoryError(`${w}.databases.${name} must be { "url": "<NAME>", "directUrl": "<NAME>" } for a postgresql database`);
+ if (!ENV_NAME_RE.test(target.url || '')) throw new InventoryError(`${w}.databases.${name}.url must be an env var name`);
+ if (!ENV_NAME_RE.test(target.directUrl || '')) throw new InventoryError(`${w}.databases.${name}.directUrl must be an env var name`);
+ if (target.url === target.directUrl) throw new InventoryError(`${w}.databases.${name}: url and directUrl must be different env vars`);
+ databases[name] = { engine: 'postgresql', url: target.url, directUrl: target.directUrl };
+ continue;
+ }
if (typeof target === 'string') {
if (typeof target === 'string') {
if (!ENV_NAME_RE.test(target)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);
if (!ENV_NAME_RE.test(target)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);
- databases[name] = { env: target, dir: false };
+ databases[name] = { engine: 'sqlite', env: target, dir: false };
continue;
continue;
}
}
const dir = target && target.dir === true ? '{tmp}/data' : target && target.dir;
const dir = target && target.dir === true ? '{tmp}/data' : target && target.dir;
if (!target || typeof target !== 'object' || typeof dir !== 'string') throw new InventoryError(`${w}.databases.${name} must be an env var name or { "env": "<NAME>", "dir": true }`);
if (!target || typeof target !== 'object' || typeof dir !== 'string') throw new InventoryError(`${w}.databases.${name} must be an env var name or { "env": "<NAME>", "dir": true }`);
if (!/^\{tmp\}\/[A-Za-z0-9._-]+$/.test(dir)) throw new InventoryError(`${w}.databases.${name}.dir must be true or "{tmp}/<name>"`);
if (!/^\{tmp\}\/[A-Za-z0-9._-]+$/.test(dir)) throw new InventoryError(`${w}.databases.${name}.dir must be true or "{tmp}/<name>"`);
if (target.env != null && !ENV_NAME_RE.test(target.env)) throw new InventoryError(`${w}.databases.${name}.env must be an env var name`);
if (target.env != null && !ENV_NAME_RE.test(target.env)) throw new InventoryError(`${w}.databases.${name}.env must be an env var name`);
- const dest = `${dir}/${path.basename(svc.databases.find((d) => d.name === name).path)}`;
+ const dest = `${dir}/${path.basename(defined.path)}`;
if (inDataDir.has(dest)) throw new InventoryError(`${w}.databases: ${name} and ${inDataDir.get(dest)} would both be ${dest}`);
if (inDataDir.has(dest)) throw new InventoryError(`${w}.databases: ${name} and ${inDataDir.get(dest)} would both be ${dest}`);
inDataDir.set(dest, name);
inDataDir.set(dest, name);
- databases[name] = { env: target.env || null, dir };
+ databases[name] = { engine: 'sqlite', env: target.env || null, dir };
}
}
const dirEnvs = new Map();
const dirEnvs = new Map();
for (const t of Object.values(databases)) {
for (const t of Object.values(databases)) {
@@ -379,7 +431,12 @@ function normaliseDrill(raw, svc, where) {
@@ -379,7 +431,12 @@ function normaliseDrill(raw, svc, where) {
}
}
}
}
if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);
if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);
- for (const t of Object.values(databases)) if (t.env && t.env in env) throw new InventoryError(`${w}.env.${t.env} is already set by drill.databases`);
+ for (const t of Object.values(databases)) {
+ // The drill writes these itself; declaring one in drill.env as well would put two lines with
+ // the same name in the file, and which wins is not something an operator should have to know.
+ const owned = t.engine === 'postgresql' ? [t.url, t.directUrl] : t.env ? [t.env] : [];
+ for (const name of owned) if (name in env) throw new InventoryError(`${w}.env.${name} is already set by drill.databases`);
+ }
const dirs = raw.dirs || [];
const dirs = raw.dirs || [];
for (const d of dirs) if (typeof d !== 'string' || !/^\{tmp\}\/[A-Za-z0-9._/-]+$/.test(d) || d.includes('..')) throw new InventoryError(`${w}.dirs entries must be paths under {tmp}`);
for (const d of dirs) if (typeof d !== 'string' || !/^\{tmp\}\/[A-Za-z0-9._/-]+$/.test(d) || d.includes('..')) throw new InventoryError(`${w}.dirs entries must be paths under {tmp}`);
if (raw.command != null && (!Array.isArray(raw.command) || !raw.command.length || raw.command.some((a) => typeof a !== 'string'))) throw new InventoryError(`${w}.command must be an argv array`);
if (raw.command != null && (!Array.isArray(raw.command) || !raw.command.length || raw.command.some((a) => typeof a !== 'string'))) throw new InventoryError(`${w}.command must be an argv array`);
@@ -434,6 +491,10 @@ function normaliseDrill(raw, svc, where) {
@@ -434,6 +491,10 @@ function normaliseDrill(raw, svc, where) {
});
});
if (!compare.length && !counts.length && !acceptance.length) throw new InventoryError(`${w} needs at least one compare path, row count or acceptance check`);
if (!compare.length && !counts.length && !acceptance.length) throw new InventoryError(`${w} needs at least one compare path, row count or acceptance check`);
const readyTimeoutSeconds = Number(raw.readyTimeoutSeconds || (svc.ready && svc.ready.timeoutSeconds) || 90);
const readyTimeoutSeconds = Number(raw.readyTimeoutSeconds || (svc.ready && svc.ready.timeoutSeconds) || 90);
+ // A write-heavy service moves between the production read and the backup's dump (Events took
+ // 18.7k rows in a day): countsTolerance lets the operator allow for that instead of flapping.
+ const countsTolerance = raw.countsTolerance == null ? 0 : Number(raw.countsTolerance);
+ if (!(Number.isInteger(countsTolerance) && countsTolerance >= 0)) throw new InventoryError(`${w}.countsTolerance must be a whole number of rows`);
return {
return {
supported: true,
supported: true,
port,
port,
@@ -449,6 +510,7 @@ function normaliseDrill(raw, svc, where) {
@@ -449,6 +510,7 @@ function normaliseDrill(raw, svc, where) {
readyTimeoutSeconds,
readyTimeoutSeconds,
compare,
compare,
counts,
counts,
+ countsTolerance,
acceptance,
acceptance,
outbound: raw.outbound === true,
outbound: raw.outbound === true,
outboundReason: raw.outboundReason || null,
outboundReason: raw.outboundReason || null,
@@ -9,6 +9,7 @@
@@ -9,6 +9,7 @@
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
const systemd = require('./systemd');
const systemd = require('./systemd');
+const dbengine = require('./dbengine');
const TEMPLATE_DIR = path.join(__dirname, '..', 'templates', 'nginx');
const TEMPLATE_DIR = path.join(__dirname, '..', 'templates', 'nginx');
@@ -235,7 +236,7 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {
@@ -235,7 +236,7 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {
if (withoutCert.length) servers += fill(readTemplate('custom-http.tmpl'), { names: withoutCert.map((d) => d.hostname).join(' '), fallback: 'return 404;' });
if (withoutCert.length) servers += fill(readTemplate('custom-http.tmpl'), { names: withoutCert.map((d) => d.hostname).join(' '), fallback: 'return 404;' });
for (const d of withCert) servers += fill(readTemplate('custom-https.tmpl'), { ...base, hostname: d.hostname, certDir: path.join(t.certBase, d.hostname), tenantLocation });
for (const d of withCert) servers += fill(readTemplate('custom-https.tmpl'), { ...base, hostname: d.hostname, certDir: path.join(t.certBase, d.hostname), tenantLocation });
const custom = fill(readTemplate('custom-domains.conf.tmpl'), {
const custom = fill(readTemplate('custom-domains.conf.tmpl'), {
- service: svc.id, database: t.database || '(the Host database)', withCert: withCert.length, withoutCert: withoutCert.length,
+ service: svc.id, database: t.database ? dbengine.label(t.database) : '(the Host database)', withCert: withCert.length, withoutCert: withoutCert.length,
servers: servers || '\n# No verified custom domains yet.\n',
servers: servers || '\n# No verified custom domains yet.\n',
});
});
return {
return {
@@ -247,11 +248,17 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {
@@ -247,11 +248,17 @@ function renderTenants(inv, svc, { wildcardCert, domains = [] } = {}) {
};
};
}
}
-/** Verified custom domains from the Host database (read-only, as the service user) + certificate presence. */
+/**
+ * Verified custom domains from the Host database + certificate presence. A PostgreSQL database is
+ * read as the postgres OS user over peer auth (Host moved to PostgreSQL; its service role's password
+ * is not ovhost's to read), a SQLite file as the service user.
+ */
async function tenantDomains(exec, svc) {
async function tenantDomains(exec, svc) {
const t = tenantsConfig(svc);
const t = tenantsConfig(svc);
if (!t.database) throw new Error(`${svc.id}: nginx.tenants.database is not set in the inventory`);
if (!t.database) throw new Error(`${svc.id}: nginx.tenants.database is not set in the inventory`);
- const rows = await exec.sqlite(t.database, VERIFIED_SQL, { as: svc.runAs });
+ const rows = t.database.engine === 'postgresql'
+ ? await exec.psql(t.database.database, VERIFIED_SQL)
+ : await exec.sqlite(t.database.path, VERIFIED_SQL, { as: svc.runAs });
const out = [];
const out = [];
for (const r of rows || []) {
for (const r of rows || []) {
const h = String(r.hostname || '').toLowerCase();
const h = String(r.hostname || '').toLowerCase();
@@ -27,6 +27,7 @@
@@ -27,6 +27,7 @@
const path = require('path');
const path = require('path');
const { pipeline } = require('stream/promises');
const { pipeline } = require('stream/promises');
const bcrypto = require('./backup-crypto');
const bcrypto = require('./backup-crypto');
+const dbengine = require('./dbengine');
const { STAMP_RE, parseStamp } = require('./retention');
const { STAMP_RE, parseStamp } = require('./retention');
const DEFAULT_ENV_FILE = '/etc/openvibe/backup.env';
const DEFAULT_ENV_FILE = '/etc/openvibe/backup.env';
@@ -325,6 +326,8 @@ async function readManifest(client, cfg, master, host, run) {
@@ -325,6 +326,8 @@ async function readManifest(client, cfg, master, host, run) {
function unsafeOut(inv, out) {
function unsafeOut(inv, out) {
for (const svc of Object.values(inv.services)) {
for (const svc of Object.values(inv.services)) {
for (const db of svc.databases) {
for (const db of svc.databases) {
+ // A PostgreSQL database is in the cluster, not at a path: it has no directory to avoid.
+ if (!db.path) continue;
const dir = path.dirname(db.path);
const dir = path.dirname(db.path);
if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;
if (out === dir || db.path.startsWith(`${out}/`) || out.startsWith(`${dir}/`)) return `${out} holds or sits inside ${svc.id}'s database directory ${dir}`;
}
}
@@ -385,13 +388,20 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:
@@ -385,13 +388,20 @@ async function restoreDownload(ctx, id, runArg, { cfg, client, out = null, host:
await exec.removeFile(dest);
await exec.removeFile(dest);
throw new OffsiteError(`${f.file}: the download does not match the manifest's hashes`);
throw new OffsiteError(`${f.file}: the download does not match the manifest's hashes`);
}
}
+ // A PostgreSQL artifact is a pg_dump archive, checked by parsing it (pg_restore --list, which
+ // needs no database and is run as the caller — the copy is root-only 0600, and postgres could
+ // not read it); a SQLite copy is quick_checked through the worker.
let check;
let check;
+ const isDump = path.extname(f.file) === '.dump';
try {
try {
- const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
- check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
+ if (isDump) { await dbengine.verifyDump(exec, dest, { as: null }); check = 'pg_restore --list ok'; }
+ else {
+ const rows = await exec.sqlite(dest, 'PRAGMA quick_check');
+ check = rows && rows[0] ? String(Object.values(rows[0])[0]) : 'no answer';
+ }
} catch (err) { check = `error: ${err.message}`; }
} catch (err) { check = `error: ${err.message}`; }
record.files.push({ name: f.name, file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: check });
record.files.push({ name: f.name, file: dest, source: f.source, bytes: p.bytes, sha256: p.sha256, quickCheck: check });
- if (check !== 'ok') throw new OffsiteError(`${dest}: quick_check = ${check}`);
+ if (isDump ? check.startsWith('error') : check !== 'ok') throw new OffsiteError(`${dest}: ${check}`);
}
}
return record;
return record;
}
}
@@ -416,6 +426,7 @@ module.exports = {
@@ -416,6 +426,7 @@ module.exports = {
pruneRemote,
pruneRemote,
listRuns,
listRuns,
restoreDownload,
restoreDownload,
+ unsafeOut,
check,
check,
uploadIterable,
uploadIterable,
parseValues,
parseValues,
@@ -7,7 +7,11 @@
@@ -7,7 +7,11 @@
* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
* sqlite-count a read-only SELECT against the service's database, run as the service user.
* sqlite-count a read-only SELECT against the service's database, run as the service user.
* Media: SELECT count(*) FROM vods WHERE is_recording = 1.
* Media: SELECT count(*) FROM vods WHERE is_recording = 1.
- * sum the total of `probes` (each an http-json-count or sqlite-count): Tools' running
+ * postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS
+ * user over peer auth (no secret is read, and a PostgreSQL service's database has
+ * no file for a sqlite-count to open). OpenRe: ingest sessions in flight.
+ * sum the total of `probes` (each an http-json-count, sqlite-count or
+ * postgresql-count): Tools' running
* jobs in each satellite's jobs.db. Unknown when any part is.
* jobs in each satellite's jobs.db. Unknown when any part is.
*
*
* A probe that cannot answer while the service is running is UNKNOWN, never zero: "could not ask"
* A probe that cannot answer while the service is running is UNKNOWN, never zero: "could not ask"
@@ -75,6 +79,15 @@ async function countOne(exec, svc, p, label) {
@@ -75,6 +79,15 @@ async function countOne(exec, svc, p, label) {
if (!Number.isFinite(n)) return { count: null, label, unknown: 'query returned no number' };
if (!Number.isFinite(n)) return { count: null, label, unknown: 'query returned no number' };
return { count: n, label };
return { count: n, label };
}
}
+ if (p.kind === 'postgresql-count') {
+ // As the postgres OS user over peer auth: the service's own role needs a password, which
+ // ovhost must never read.
+ const rows = await exec.psql(p.database, p.sql);
+ const first = rows && rows[0] ? Object.values(rows[0])[0] : undefined;
+ const n = Number(first);
+ if (!Number.isFinite(n)) return { count: null, label, unknown: 'query returned no number' };
+ return { count: n, label };
+ }
} catch (err) {
} catch (err) {
return { count: null, label, unknown: err.message };
return { count: null, label, unknown: err.message };
}
}
@@ -10,6 +10,9 @@ const crypto = require('crypto');
@@ -10,6 +10,9 @@ const crypto = require('crypto');
const { scenario, test, runTests, SECRET } = require('./helpers');
const { scenario, test, runTests, SECRET } = require('./helpers');
const { main } = require('../lib/cli');
const { main } = require('../lib/cli');
const { mockS3 } = require('./s3-mock');
const { mockS3 } = require('./s3-mock');
+const { refusal } = require('../lib/archive');
+const offsite = require('../lib/offsite');
+const { normalise } = require('../lib/inventory');
const KEY_HEX = crypto.createHash('sha256').update('archive test key').digest('hex');
const KEY_HEX = crypto.createHash('sha256').update('archive test key').digest('hex');
const S3_SECRET = `${SECRET}-s3`;
const S3_SECRET = `${SECRET}-s3`;
@@ -78,7 +81,7 @@ runTests([
@@ -78,7 +81,7 @@ runTests([
test('refusals: live databases, open files, symlinks, outside roots, relative paths; nothing uploaded', async () => {
test('refusals: live databases, open files, symlinks, outside roots, relative paths; nothing uploaded', async () => {
const host = archiveHost();
const host = archiveHost();
- const liveDb = Object.values(host.inv.services).flatMap((s) => s.databases.map((d) => d.path))[0];
+ const liveDb = Object.values(host.inv.services).flatMap((s) => s.databases.filter((d) => d.engine !== 'postgresql').map((d) => d.path))[0];
assert.ok(liveDb, 'the scenario declares a database');
assert.ok(liveDb, 'the scenario declares a database');
host.put(liveDb, 'SQLite format 3\0 live', { owner: 'ubuntu' });
host.put(liveDb, 'SQLite format 3\0 live', { owner: 'ubuntu' });
await host.exec.symlink('/opt/backups/old/hobostreamer.db', '/opt/backups/old/link.db');
await host.exec.symlink('/opt/backups/old/hobostreamer.db', '/opt/backups/old/link.db');
@@ -89,4 +92,26 @@ runTests([
@@ -89,4 +92,26 @@ runTests([
assert.strictEqual(host.s3.objects.size, 0, `${file}: nothing is uploaded when any file is refused`);
assert.strictEqual(host.s3.objects.size, 0, `${file}: nothing is uploaded when any file is refused`);
}
}
}),
}),
+
+ test('a PostgreSQL entry has no path, so it neither blocks archiving the file it replaced nor makes unsafeOut throw', async () => {
+ const host = archiveHost();
+ const raw = JSON.parse(host.read('/etc/openvibe/host.json'));
+ // Media moved to PostgreSQL (C-89): its databases[] entry names a database in the cluster.
+ raw.services.media.databases = [{ name: 'media', engine: 'postgresql', database: 'ov_media' }];
+ const inv = normalise(raw);
+ const frozen = '/opt/openvibe.media/data/media.db'; // Media's old SQLite file, frozen at the switch
+ host.put(frozen, 'SQLite format 3\0 frozen', { owner: 'ubuntu' });
+ assert.strictEqual(await refusal(host.exec, inv, frozen), null, 'the frozen SQLite file is archivable again');
+
+ // A database the inventory still declares as SQLite is refused, message unchanged.
+ const live = inv.services.live.databases[0];
+ assert.strictEqual(live.engine, 'sqlite');
+ host.put(live.path, 'SQLite format 3\0 live', { owner: 'ubuntu' });
+ assert.match(await refusal(host.exec, inv, live.path), /a database the inventory declares \(a live database\)/);
+
+ // unsafeOut only avoids database directories; a database in the cluster has none.
+ let out;
+ assert.doesNotThrow(() => { out = offsite.unsafeOut(inv, '/var/lib/openvibe-restore/media-2026'); });
+ assert.strictEqual(out, null);
+ }),
]);
]);
@@ -58,6 +58,19 @@ function dirsOf(host, dir) {
@@ -58,6 +58,19 @@ function dirsOf(host, dir) {
return [...host.files.keys()].filter((k) => path.dirname(k) === dir).map((k) => path.basename(k)).sort();
return [...host.files.keys()].filter((k) => path.dirname(k) === dir).map((k) => path.basename(k)).sort();
}
}
+/** backupHost plus a PostgreSQL-only service "trade" and a fresh pgBackRest differential. */
+function pgOffsiteHost({ dumpContent = null } = {}) {
+ const host = backupHost();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.trade = { repo: '/opt/openvibe.trade', units: [], databases: [{ name: 'trade', engine: 'postgresql', database: 'ov_trade' }] };
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+ host.inv = normalise(doc);
+ host.pgDatabases.add('ov_trade');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
+ if (dumpContent) host.dumpContent = dumpContent;
+ return host;
+}
+
function noSecrets(text, what) {
function noSecrets(text, what) {
for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);
for (const s of [S3_SECRET, SECRET, KEY_HEX]) assert.ok(!String(text).includes(s), `${what} leaked a secret value`);
}
}
@@ -451,6 +464,29 @@ runTests([
@@ -451,6 +464,29 @@ runTests([
assert.match(none.out, /no off-host run of openre/);
assert.match(none.out, /no off-host run of openre/);
}),
}),
+ test('restore-download: a PostgreSQL .dump is verified with pg_restore --list; a corrupt one is refused', async () => {
+ const host = pgOffsiteHost();
+ const r = await host.cli('backup', '--all', '--offsite');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.ok([...host.s3.objects.keys()].some((k) => k.endsWith('/trade/trade.dump.ovbk')), 'the pg_dump archive was uploaded');
+
+ const d = await host.cli('restore-download', 'trade', 'latest');
+ assert.strictEqual(d.code, 0, d.out);
+ const dest = '/var/lib/openvibe-restore/trade-20260922-120000/trade.dump';
+ assert.ok(host.files.has(dest));
+ assert.match(d.out, /pg_restore --list ok/);
+ // --list only reads the file, so root checks the root-only copy itself, not as postgres.
+ assert.ok(host.calls.some((c) => c.cmd === 'pg_restore' && c.args.includes('--list') && c.args.includes(dest) && c.as === null));
+ assert.deepStrictEqual([host.files.get(dest).owner, host.files.get(dest).mode], ['root', 0o600]);
+
+ // A dump that is not a pg_dump archive: the manifest hashes match, so only pg_restore --list refuses it.
+ const bad = pgOffsiteHost({ dumpContent: () => 'this is not a pg_dump archive' });
+ await bad.cli('backup', '--all', '--offsite');
+ const refused = await bad.cli('restore-download', 'trade', 'latest');
+ assert.strictEqual(refused.code, 2, refused.out);
+ assert.match(refused.out, /pg_restore --list .* failed|text format dump/);
+ }),
+
// ── configuration ──────────────────────────────────────────────────────────
// ── configuration ──────────────────────────────────────────────────────────
test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {
test('config: root-only env file and key; missing names listed by name; not root refused; no value ever printed', async () => {
let host = backupHost({ envMode: 0o644 });
let host = backupHost({ envMode: 0o644 });
new file mode 100644
new file mode 100644
@@ -0,0 +1,192 @@
@@ -0,0 +1,192 @@
+'use strict';
+/**
+ * The PostgreSQL side of `ovhost backup`: every run verifies pgBackRest (a fresh stanza, and a WAL
+ * archiver failed_count that has not risen), and a per-service pg_dump -Fc only on a slower cadence
+ * (the weekly one, or --logical). SQLite services must be untouched. Against the fake host; no
+ * pg_dump, pgbackrest or psql ever runs for real.
+ */
+const assert = require('assert');
+const path = require('path');
+const { scenario, test, runTests } = require('./helpers');
+const { normalise } = require('../lib/inventory');
+
+const DAY = 86400000;
+const STATE = '/var/lib/openvibe-host';
+const PROM = '/var/lib/prometheus/node-exporter/openvibe_backup.prom';
+
+/** The standard fake host plus a PostgreSQL-only service "trade", and a fresh nightly differential. */
+function pgHost() {
+ const host = scenario();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.trade = { repo: '/opt/openvibe.trade', units: [], databases: [{ name: 'trade', engine: 'postgresql', database: 'ov_trade' }] };
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+ host.inv = normalise(doc);
+ host.pgDatabases.add('ov_trade');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 3600 } }];
+ return host;
+}
+
+/** The newest record in a service's backup log. */
+function lastRecord(host, id) {
+ const lines = host.read(`${STATE}/backups/${id}.jsonl`).trim().split('\n');
+ return JSON.parse(lines[lines.length - 1]);
+}
+
+/** A prior GOOD logical-dump record `ageMs` old, as an earlier weekly run would have written. */
+function seedDump(host, ageMs) {
+ const dir = '/var/backups/openvibe/trade/20200101-000000';
+ const rec = {
+ service: 'trade', at: new Date(host.exec.now() - ageMs).toISOString(), reason: 'scheduled', engine: 'postgresql', ok: true, dir,
+ files: [{ name: 'trade', source: 'ov_trade', dest: `${dir}/trade.dump`, file: 'trade.dump', check: 'pg_dump -Fc', bytes: 1 }],
+ };
+ host.put(`${STATE}/backups/trade.jsonl`, `${JSON.stringify(rec)}\n`);
+}
+
+runTests([
+ test('a fresh pgBackRest backup: verified, recorded on the run, and left for Prometheus', async () => {
+ const host = pgHost();
+ host.put('/var/lib/prometheus/node-exporter/.keep', '');
+ const r = await host.cli('backup', '--all');
+ assert.strictEqual(r.code, 0, r.out);
+
+ const rec = lastRecord(host, 'trade');
+ assert.strictEqual(rec.engine, 'postgresql');
+ assert.strictEqual(rec.verified.ok, true);
+ assert.strictEqual(rec.verified.type, 'diff');
+ assert.strictEqual(rec.verified.walOk, true);
+ assert.strictEqual(rec.verified.stop, new Date((Math.floor(host.exec.now() / 1000) - 3600) * 1000).toISOString());
+ // No prior record: a logical dump was due on the first run.
+ assert.ok(rec.dir);
+ assert.strictEqual(host.dumps.length, 1);
+ assert.ok(host.files.has(`${rec.dir}/trade.dump`));
+
+ const prom = host.read(PROM);
+ assert.match(prom, /^openvibe_backup_pg_verified\{service="trade"\} 1$/m);
+ assert.match(prom, /^openvibe_backup_services\{status="verified"\} 0$/m);
+ assert.match(prom, /^openvibe_pgbackrest_last_backup_age_seconds 3600$/m);
+ }),
+
+ test('a stale pgBackRest backup: the run is failed, never skipped, and the summary says so', async () => {
+ const host = pgHost();
+ host.pgBackups = [{ type: 'full', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 27 * 3600 } }];
+ const r = await host.cli('backup', '--all', '--json');
+ assert.strictEqual(r.code, 2, r.out);
+ const s = JSON.parse(r.out);
+ assert.strictEqual(s.ok, false);
+ const trade = s.services.find((x) => x.service === 'trade');
+ assert.strictEqual(trade.status, 'failed', 'a verification failure is failed, not skipped');
+ assert.strictEqual(trade.verified.ok, false);
+ assert.match(trade.error, /stopped 27\.0h ago/);
+
+ const rec = lastRecord(host, 'trade');
+ assert.strictEqual(rec.ok, false);
+ assert.strictEqual(rec.verified.ok, false);
+ assert.match(rec.verified.error, /stopped 27\.0h ago/);
+ assert.ok(!('dir' in rec), 'a failed verification leaves no directory');
+ assert.strictEqual(host.dumps.length, 0);
+ }),
+
+ test('the WAL archiver failed_count: a rise fails the run, an unchanged value does not', async () => {
+ const host = pgHost();
+ const first = await host.cli('backup', 'trade');
+ assert.strictEqual(first.code, 0, first.out);
+ assert.strictEqual(lastRecord(host, 'trade').verified.walOk, true);
+ const state = host.files.get(`${STATE}/pgbackrest-archiver.json`);
+ assert.strictEqual(JSON.parse(state.content).failedCount, 0);
+ assert.strictEqual(state.mode, 0o640, 'the recorded count is root-only 0640');
+
+ // The same value: still ok.
+ host.advance(1000);
+ const same = await host.cli('backup', 'trade');
+ assert.strictEqual(same.code, 0, same.out);
+ assert.strictEqual(lastRecord(host, 'trade').verified.walOk, true);
+
+ // It rose: failed, and the new value is remembered for the next run.
+ host.advance(1000);
+ host.pgArchiver.failed_count = 3;
+ const risen = await host.cli('backup', 'trade');
+ assert.strictEqual(risen.code, 2, risen.out);
+ const rec = lastRecord(host, 'trade');
+ assert.strictEqual(rec.ok, false);
+ assert.strictEqual(rec.verified.walOk, false);
+ assert.match(rec.verified.error, /failed_count rose from 0 to 3/);
+ assert.strictEqual(JSON.parse(host.files.get(`${STATE}/pgbackrest-archiver.json`).content).failedCount, 3);
+ }),
+
+ test('weekly cadence: no dump while the last good one is recent, a dump when it is stale, --logical forces one', async () => {
+ const host = pgHost();
+
+ seedDump(host, 3 * DAY);
+ const recent = await host.cli('backup', 'trade');
+ assert.strictEqual(recent.code, 0, recent.out);
+ assert.strictEqual(host.dumps.length, 0, 'no pg_dump three days after the last good one');
+ const rec = lastRecord(host, 'trade');
+ assert.strictEqual(rec.ok, true);
+ assert.deepStrictEqual(rec.files, []);
+ assert.strictEqual(rec.verified.ok, true);
+ assert.ok(!('dir' in rec), 'a verification-only run writes no directory');
+ assert.ok(!('empty' in rec), 'a verification-only run is never empty');
+
+ seedDump(host, 8 * DAY);
+ host.advance(1000);
+ const stale = await host.cli('backup', 'trade');
+ assert.strictEqual(stale.code, 0, stale.out);
+ assert.strictEqual(host.dumps.length, 1);
+ assert.strictEqual(host.dumps[0].database, 'ov_trade');
+ assert.strictEqual(host.dumps[0].as, 'postgres');
+ const dumped = lastRecord(host, 'trade');
+ assert.strictEqual(dumped.files.length, 1);
+ assert.strictEqual(dumped.files[0].file, 'trade.dump');
+ assert.strictEqual(dumped.files[0].check, 'pg_dump -Fc');
+ assert.ok(host.files.has(`${dumped.dir}/trade.dump`));
+
+ // --logical ignores the cadence.
+ seedDump(host, 3 * DAY);
+ host.advance(1000);
+ const forced = await host.cli('backup', 'trade', '--logical');
+ assert.strictEqual(forced.code, 0, forced.out);
+ assert.strictEqual(host.dumps.length, 2);
+ }),
+
+ test('staging ownership: postgres owns the directory while pg_dump runs; SQLite stays the service user', async () => {
+ const host = pgHost();
+ const stages = [];
+ const mkdir = host.exec.mkdir;
+ host.exec.mkdir = async (p, o) => { if (String(p).startsWith('/var/backups/openvibe.staging/')) stages.push([path.basename(p), o && o.owner]); return mkdir(p, o); };
+
+ await host.cli('backup', 'trade');
+ assert.deepStrictEqual(stages, [['trade-20260922-120000', 'postgres']], 'pg_dump must have a postgres-owned staging directory');
+ assert.strictEqual(host.dumps[0].as, 'postgres');
+
+ await host.cli('backup', 'live');
+ assert.strictEqual(stages[1][1], 'ubuntu', 'a SQLite service keeps the service user');
+ assert.strictEqual(stages[1][0], 'live-20260922-120000');
+
+ // Root takes every copy back once the staging directory is back to root:root.
+ const copy = host.files.get(`${lastRecord(host, 'trade').dir}/trade.dump`);
+ assert.deepStrictEqual([copy.owner, copy.mode], ['root', 0o600]);
+ }),
+
+ test('backup --all: SQLite and PostgreSQL services in one run; verified differs from ok and failed', async () => {
+ const host = pgHost();
+ seedDump(host, 3 * DAY); // the weekly dump is not due: trade is verification-only this run
+ const r = await host.cli('backup', '--all', '--json');
+ assert.strictEqual(r.code, 0, r.out);
+ const s = JSON.parse(r.out);
+ assert.strictEqual(s.ok, true);
+ assert.deepStrictEqual(s.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'ok'], ['trade', 'verified']]);
+ const trade = s.services.find((x) => x.service === 'trade');
+ assert.strictEqual(trade.dir, null);
+ assert.deepStrictEqual(trade.files, []);
+ assert.strictEqual(trade.verified.ok, true);
+
+ // A stale stanza then fails that service; the others are unaffected and the run exits 2.
+ host.advance(1000);
+ host.pgBackups = [{ type: 'full', timestamp: { stop: Math.floor(host.exec.now() / 1000) - 30 * 3600 } }];
+ const bad = await host.cli('backup', '--all', '--json');
+ assert.strictEqual(bad.code, 2);
+ const s2 = JSON.parse(bad.out);
+ assert.strictEqual(s2.ok, false);
+ assert.deepStrictEqual(s2.services.map((x) => [x.service, x.status]), [['live', 'ok'], ['media', 'ok'], ['trade', 'failed']]);
+ }),
+]);
@@ -18,7 +18,6 @@ const { normalise } = require('../lib/inventory');
@@ -18,7 +18,6 @@ const { normalise } = require('../lib/inventory');
const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
const LIVE_DB = '/opt/openvibe.live/data/live.db';
const LIVE_DB = '/opt/openvibe.live/data/live.db';
-const MEDIA_DB = '/opt/openvibe.media/data/media.db';
const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));
const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));
@@ -95,7 +94,7 @@ async function mediaHost({ switchInCheckout = true } = {}) {
@@ -95,7 +94,7 @@ async function mediaHost({ switchInCheckout = true } = {}) {
doc.services.media = JSON.parse(JSON.stringify(EXAMPLE.services.media));
doc.services.media = JSON.parse(JSON.stringify(EXAMPLE.services.media));
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
if (switchInCheckout) {
if (switchInCheckout) {
- host.put('/opt/openvibe.media/server/drill.js', "const enabled = parse(process.env.MEDIA_DRILL);\n", { owner: 'ubuntu' });
+ host.put('/opt/openvibe.media/server/drill.js', "const enabled = parse(process.env.MEDIA_DRILL);\nif (!env.DATABASE_URL) out.push('DATABASE_URL');\n", { owner: 'ubuntu' });
host.put('/opt/openvibe.media/server/public/routes.js', "if (drill.refuseBytes(res)) return;\n", { owner: 'ubuntu' });
host.put('/opt/openvibe.media/server/public/routes.js', "if (drill.refuseBytes(res)) return;\n", { owner: 'ubuntu' });
}
}
host.put('/etc/systemd/system/openvibe-media.service', [
host.put('/etc/systemd/system/openvibe-media.service', [
@@ -105,15 +104,17 @@ async function mediaHost({ switchInCheckout = true } = {}) {
@@ -105,15 +104,17 @@ async function mediaHost({ switchInCheckout = true } = {}) {
const unit = host.units.get('openvibe-media.service');
const unit = host.units.get('openvibe-media.service');
host.alivePids.add(unit.mainPid);
host.alivePids.add(unit.mainPid);
host.listeners.set(4100, [{ pid: unit.mainPid, process: 'node' }]);
host.listeners.set(4100, [{ pid: unit.mainPid, process: 'node' }]);
- host.put(MEDIA_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
+ // Media is PostgreSQL now (host.example.json): the production database is in the cluster, and the
+ // drill restores a logical dump into a scratch database instead of copying a file.
+ host.pgDatabases.add('ov_media');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(Date.parse('2026-09-22T12:00:00Z') / 1000) } }];
host.http.set('http://127.0.0.1:4100/api/ready', () => ({ status: 200, body: { ready: true } }));
host.http.set('http://127.0.0.1:4100/api/ready', () => ({ status: 200, body: { ready: true } }));
for (const tab of ['videos', 'clips']) host.http.set(`http://127.0.0.1:4100/browse?tab=${tab}`, () => ({ status: 200, body: BROWSE(tab) }));
for (const tab of ['videos', 'clips']) host.http.set(`http://127.0.0.1:4100/browse?tab=${tab}`, () => ({ status: 200, body: BROWSE(tab) }));
const counts = { media_objects: 6021, vods: 1300, clips: 4700, apps: 4 };
const counts = { media_objects: 6021, vods: 1300, clips: 4700, apps: 4 };
- host.sqliteHandler = (db, sql) => {
- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
- if (/is_recording/.test(sql)) return [{ n: 0 }];
+ host.psqlHandler = (database, sql) => {
+ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];
const m = /FROM "([a-z_]+)"/.exec(sql);
const m = /FROM "([a-z_]+)"/.exec(sql);
- return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
+ return [{ n: String(m ? counts[m[1]] ?? 0 : 0) }];
};
};
host.onSystemdRun = (spec) => {
host.onSystemdRun = (spec) => {
const envFile = spec.envFiles[spec.envFiles.length - 1];
const envFile = spec.envFiles[spec.envFiles.length - 1];
@@ -137,7 +138,7 @@ runTests([
@@ -137,7 +138,7 @@ runTests([
const live = inv.services.live.drill;
const live = inv.services.live.drill;
assert.strictEqual(live.supported, true);
assert.strictEqual(live.supported, true);
assert.strictEqual(live.port, 13000);
assert.strictEqual(live.port, 13000);
- assert.deepStrictEqual(live.databases, { live: { env: 'DB_PATH', dir: false } });
+ assert.deepStrictEqual(live.databases, { live: { engine: 'sqlite', env: 'DB_PATH', dir: false } });
assert.strictEqual(live.env.LIVE_DRILL, '1');
assert.strictEqual(live.env.LIVE_DRILL, '1');
assert.strictEqual(live.env.DATA_DIR, '{tmp}/data');
assert.strictEqual(live.env.DATA_DIR, '{tmp}/data');
assert.strictEqual(live.env.HOST, '127.0.0.1');
assert.strictEqual(live.env.HOST, '127.0.0.1');
@@ -148,11 +149,11 @@ runTests([
@@ -148,11 +149,11 @@ runTests([
const media = inv.services.media.drill;
const media = inv.services.media.drill;
assert.strictEqual(media.supported, true);
assert.strictEqual(media.supported, true);
assert.strictEqual(media.port, 14100);
assert.strictEqual(media.port, 14100);
- assert.deepStrictEqual(media.databases, { media: { env: 'DB_PATH', dir: false } });
+ assert.deepStrictEqual(media.databases, { media: { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } });
assert.strictEqual(media.env.MEDIA_DRILL, '1');
assert.strictEqual(media.env.MEDIA_DRILL, '1');
for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'ASSETS_PATH', 'FILES_PATH', 'OBJECTS_PATH']) assert.match(media.env[k], /^\{tmp\}\/storage\//, k);
for (const k of ['VOD_PATH', 'CLIPS_PATH', 'PASTES_PATH', 'ASSETS_PATH', 'FILES_PATH', 'OBJECTS_PATH']) assert.match(media.env[k], /^\{tmp\}\/storage\//, k);
assert.ok(!('THUMBNAILS_PATH' in media.env), '/browse lists production\'s thumbnail directory');
assert.ok(!('THUMBNAILS_PATH' in media.env), '/browse lists production\'s thumbnail directory');
- assert.deepStrictEqual(media.requires.map((r) => r.contains), ['MEDIA_DRILL', 'refuseBytes']);
+ assert.deepStrictEqual(media.requires.map((r) => r.contains), ['MEDIA_DRILL', 'DATABASE_URL', 'refuseBytes']);
assert.deepStrictEqual(media.compare.map((c) => c.path), ['/browse?tab=videos', '/browse?tab=clips']);
assert.deepStrictEqual(media.compare.map((c) => c.path), ['/browse?tab=videos', '/browse?tab=clips']);
assert.deepStrictEqual(media.counts.map((c) => c.table), ['media_objects', 'vods', 'clips', 'apps']);
assert.deepStrictEqual(media.counts.map((c) => c.table), ['media_objects', 'vods', 'clips', 'apps']);
}),
}),
@@ -217,15 +218,24 @@ runTests([
@@ -217,15 +218,24 @@ runTests([
assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.media/server/index.js']);
assert.deepStrictEqual(run.argv, ['/usr/bin/env', 'node', '/opt/openvibe.media/server/index.js']);
assert.ok(run.props.includes('SocketBindAllow=tcp:14100'));
assert.ok(run.props.includes('SocketBindAllow=tcp:14100'));
const env = envOf(host.drillEnv);
const env = envOf(host.drillEnv);
- assert.deepStrictEqual(env, {
+ const { DATABASE_URL, DATABASE_DIRECT_URL, ...envRest } = env;
+ assert.deepStrictEqual(envRest, {
PORT: '14100', HOST: '127.0.0.1', MEDIA_DRILL: '1',
PORT: '14100', HOST: '127.0.0.1', MEDIA_DRILL: '1',
VOD_PATH: `${rec.dir}/storage/vods`, CLIPS_PATH: `${rec.dir}/storage/clips`, PASTES_PATH: `${rec.dir}/storage/pastes`,
VOD_PATH: `${rec.dir}/storage/vods`, CLIPS_PATH: `${rec.dir}/storage/clips`, PASTES_PATH: `${rec.dir}/storage/pastes`,
ASSETS_PATH: `${rec.dir}/storage/assets`, FILES_PATH: `${rec.dir}/storage/files`, OBJECTS_PATH: `${rec.dir}/storage/objects`,
ASSETS_PATH: `${rec.dir}/storage/assets`, FILES_PATH: `${rec.dir}/storage/files`, OBJECTS_PATH: `${rec.dir}/storage/objects`,
- EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', DB_PATH: `${rec.dir}/db/media.db`,
+ EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', VALKEY_URL: 'redis://127.0.0.1:9/0',
});
});
- assert.deepStrictEqual(host.drillDirAtStart, ['db', 'db/media.db', 'drill.env'], 'nothing but the copy and the override file; storage is never created');
+ // The generated URL names the scratch database and role; both direct and pooled vars point at
+ // the cluster port (5432), never PgBouncer.
+ const scratch = rec.scratch.database;
+ assert.match(DATABASE_URL, new RegExp(`^postgresql://${scratch}:[0-9a-f]+@127\\.0\\.0\\.1:5432/${scratch}$`));
+ assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);
+ assert.strictEqual(rec.databases[0].integrity, 'pg_restore --list ok');
+ assert.deepStrictEqual(host.drillDirAtStart, ['db', 'drill.env', 'pg', 'pg/media.dump'], 'the dump copy and the override file; storage is never created');
assert.deepStrictEqual(stateChanges(host), []);
assert.deepStrictEqual(stateChanges(host), []);
- assert.strictEqual(host.read(MEDIA_DB), 'sqlite-production');
+ assert.ok(host.pgDatabases.has('ov_media'), 'the production database is untouched');
+ assert.ok(host.droppedDatabases.some((d) => d.name === scratch), 'the scratch database is dropped');
+ assert.ok(!host.pgRoles.has(scratch), 'the scratch role is dropped');
assert.ok(!(r.out + JSON.stringify(host.systemdRuns)).includes(SECRET));
assert.ok(!(r.out + JSON.stringify(host.systemdRuns)).includes(SECRET));
}),
}),
new file mode 100644
new file mode 100644
@@ -0,0 +1,327 @@
@@ -0,0 +1,327 @@
+'use strict';
+/**
+ * ovhost drill for a PostgreSQL service (lib/dbengine.js): the artifact is a `.dump` in the backup,
+ * not a SQLite file. The drill copies it into <tmp>/pg owned by postgres, verifies it with
+ * `pg_restore --list`, creates a scratch database and login role, restores into them, points
+ * drill.env at the scratch database (and at a closed Valkey), and always drops both again in a
+ * finally. Covered: the generated URLs and their password, the refusal when an override is missing,
+ * cleanup on failure, counts with tolerance, artifact selection and the free-space refusal.
+ */
+const assert = require('assert');
+const path = require('path');
+const { scenario, test, runTests } = require('./helpers');
+
+const ITEMS = '[{"id":"it_1","title":"hello"},{"id":"it_2","title":"world"}]';
+// The production password in trade.env: the drill's generated password must never be this.
+const PROD_PASSWORD = 'prod-password-must-not-be-reused';
+
+function tradeEntry(drillOverrides = {}) {
+ return {
+ repo: '/opt/openvibe.trade',
+ units: ['openvibe-trade.service'],
+ envFile: '/etc/openvibe/trade.env',
+ port: 4500,
+ ready: { url: 'http://127.0.0.1:4500/api/ready', timeoutSeconds: 30 },
+ databases: [{ name: 'trade', engine: 'postgresql', database: 'ov_trade' }],
+ drill: {
+ port: 14500,
+ databases: { trade: { url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } },
+ env: { PORT: '{port}', HOST: '127.0.0.1', TRADE_DRILL: '1', VALKEY_URL: 'redis://127.0.0.1:9/0' },
+ compare: [{ path: '/api/items?limit=5' }],
+ counts: [{ db: 'trade', table: 'items' }],
+ ...drillOverrides,
+ },
+ };
+}
+
+/**
+ * The standard fake host plus Trade in production, its cluster database and a good pgBackRest layer,
+ * with one `ovhost backup trade` taken (a logical pg_dump). The drill instance's answers and the two
+ * row counts are controlled through `host.drill`.
+ */
+async function pgScenario({ drill: drillOverrides, backup = true } = {}) {
+ const host = scenario();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.trade = tradeEntry(drillOverrides);
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+
+ const repo = host.createRepo('/opt/openvibe.trade', { owner: 'ubuntu' });
+ const sha = repo.commit({ 'package.json': '{"name":"openvibe-trade"}', 'server/index.js': 'trade();' }, { message: 'initial' });
+ repo.publish(sha);
+ repo.checkout(sha);
+ host.addUnit('openvibe-trade.service', { mainPid: 2222 });
+ host.alivePids.add(2222);
+ host.listeners.set(4500, [{ pid: 2222, process: 'node' }]);
+ host.put('/etc/systemd/system/openvibe-trade.service', [
+ '[Service]',
+ 'User=ubuntu',
+ 'WorkingDirectory=/opt/openvibe.trade',
+ 'EnvironmentFile=/etc/openvibe/trade.env',
+ 'ExecStart=/usr/bin/node server/index.js',
+ '',
+ ].join('\n'));
+ // The production app's own connection strings: the drill must not let either reach the instance.
+ host.put('/etc/openvibe/trade.env', `DATABASE_URL=postgresql://trade_app:${PROD_PASSWORD}@127.0.0.1:6432/ov_trade\nDATABASE_DIRECT_URL=postgresql://trade_app:${PROD_PASSWORD}@127.0.0.1:5432/ov_trade\nVALKEY_URL=redis://valkey-production:6379/7\n`, { mode: 0o600 });
+
+ // The cluster: the production database, and a healthy pgBackRest stanza so `ovhost backup` passes.
+ host.pgDatabases.add('ov_trade');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(Date.parse('2026-09-22T12:00:00Z') / 1000) } }];
+ host.drill = {
+ productionCount: 500,
+ restoredCount: 500,
+ ready: () => ({ status: 200, body: { status: 'ready' } }),
+ items: () => ({ status: 200, body: ITEMS }),
+ };
+ host.psqlHandler = (database, sql) => {
+ if (/left\((datname|rolname)/.test(sql)) return undefined; // leftover-scratch lookups: the fake's catalog
+ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];
+ if (/FROM "([a-z_]+)"/.test(sql)) return [{ n: String(database === 'ov_trade' ? host.drill.productionCount : host.drill.restoredCount) }];
+ return [{ n: '0' }];
+ };
+ host.http.set('http://127.0.0.1:4500/api/items?limit=5', () => ({ status: 200, body: ITEMS }));
+ host.onSystemdRun = (spec) => {
+ const envFile = spec.envFiles[spec.envFiles.length - 1];
+ host.drillEnv = host.read(envFile);
+ host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);
+ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
+ host.http.set('http://127.0.0.1:14500/api/ready', alive(() => host.drill.ready()));
+ host.http.set('http://127.0.0.1:14500/api/items?limit=5', alive(() => host.drill.items()));
+ return undefined;
+ };
+
+ if (backup) {
+ const b = await host.cli('backup', 'trade', '--json');
+ assert.strictEqual(b.code, 0, b.out);
+ host.backupDir = JSON.parse(b.out).dir;
+ }
+ host.advance(60 * 1000);
+ return host;
+}
+
+const envOf = (text) => Object.fromEntries(text.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
+const lastLog = (host) => JSON.parse(host.read('/var/lib/openvibe-host/drills/trade.jsonl').trim().split('\n').pop());
+
+function fsSnapshot(host) {
+ const m = new Map();
+ for (const [k, v] of host.files) m.set(k, `${v.type}:${v.owner}:${v.mode}:${v.type === 'file' ? String(v.content) : v.target || ''}`);
+ return m;
+}
+
+function changedPaths(before, after) {
+ const out = [];
+ for (const [k, v] of after) if (before.get(k) !== v) out.push(k);
+ for (const k of before.keys()) if (!after.has(k)) out.push(k);
+ return out.sort();
+}
+
+runTests([
+ test('leftovers of a killed drill are dropped first; restores skip GRANTs; scratch drops are forced', async () => {
+ const host = await pgScenario();
+ const old = 'ov_trade_drill_20260101-000000';
+ host.pgDatabases.add(old);
+ host.pgRoles.set(old, { password: 'x' });
+ host.pgDatabases.add('ov_tradex'); // another name that only looks alike stays
+ const callsBefore = host.calls.length;
+ const r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.ok(!host.pgDatabases.has(old), 'the leftover scratch database is dropped');
+ assert.ok(!host.pgRoles.has(old), 'the leftover scratch role is dropped');
+ assert.ok(host.pgDatabases.has('ov_trade') && host.pgDatabases.has('ov_tradex'), 'production and look-alike databases are untouched');
+ const calls = host.calls.slice(callsBefore);
+ const restore = calls.find((c) => c.cmd === 'pg_restore' && !c.args.includes('--list'));
+ assert.ok(restore.args.includes('--no-privileges') && restore.args.includes('--no-owner'), restore.args.join(' '));
+ for (const c of calls.filter((x) => x.cmd === 'dropdb')) assert.ok(c.args.includes('--force'), c.args.join(' '));
+ }),
+ test('full flow: pg_restore --list, a scratch database and role, drill.env pointed at them, cleanup', async () => {
+ const host = await pgScenario();
+ const callsBefore = host.calls.length;
+ const r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));
+ assert.strictEqual(rec.failure, null);
+
+ const tmp = rec.dir;
+ assert.match(tmp, /^\/var\/lib\/openvibe-drills\/trade-\d{8}-\d{6}$/);
+ assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.engine, d.integrity, d.copy]), [['trade', 'postgresql', 'pg_restore --list ok', `${tmp}/pg/trade.dump`]]);
+ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 500, restored: 500, match: true }]);
+ assert.strictEqual(rec.ready.ok, true);
+ assert.strictEqual(rec.stop.ok, true);
+
+ // The scratch database and role share the derived name and are recorded on the drill record.
+ const at = path.basename(tmp).slice('trade-'.length);
+ const scratch = `ov_trade_drill_${at}`;
+ assert.deepStrictEqual(rec.scratch, { database: scratch, role: scratch, dropped: true });
+ assert.ok(host.restores.some((x) => x.database === scratch && x.role === scratch && x.file === `${tmp}/pg/trade.dump`));
+ assert.ok(host.createdDatabases.some((c) => c.name === scratch && c.owner === scratch && c.as === 'postgres'));
+ // The scratch database and role are dropped again; production's database is untouched.
+ assert.ok(host.droppedDatabases.some((c) => c.name === scratch && c.as === 'postgres'));
+ assert.ok(!host.pgRoles.has(scratch), 'the scratch role is gone');
+ assert.ok(host.pgDatabases.has('ov_trade'), 'production database still there');
+
+ // drill.env points the instance at the scratch database over the direct port, and at a closed
+ // Valkey from the inventory. The password is generated, not production's, and never in argv.
+ const env = envOf(host.drillEnv);
+ assert.strictEqual(env.VALKEY_URL, 'redis://127.0.0.1:9/0');
+ const url = `postgresql://${scratch}:`;
+ assert.ok(env.DATABASE_URL.startsWith(url), env.DATABASE_URL);
+ assert.ok(env.DATABASE_URL.endsWith(`@127.0.0.1:5432/${scratch}`), env.DATABASE_URL);
+ assert.strictEqual(env.DATABASE_DIRECT_URL, env.DATABASE_URL);
+ const password = /^postgresql:\/\/[^:]+:([^@]+)@/.exec(env.DATABASE_URL)[1];
+ assert.notStrictEqual(password, PROD_PASSWORD);
+ assert.ok(!JSON.stringify(host.calls.slice(callsBefore)).includes(password), 'the generated password never reaches an argv');
+ assert.ok(!host.drillEnv.includes(PROD_PASSWORD), 'the production password never reaches drill.env');
+
+ // The dump is copied as postgres (the backup is root-only, so postgres cannot read it directly).
+ const cp = host.calls.slice(callsBefore).find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/pg/trade.dump`));
+ assert.deepStrictEqual(cp.args, ['-o', 'postgres', '-m', '0600', '-T', '--', `${host.backupDir}/trade.dump`, `${tmp}/pg/trade.dump`]);
+ assert.strictEqual(cp.privileged, true);
+
+ // The Markdown row names pg_restore --list, not pragma integrity_check.
+ assert.match(r.out, /`pg_restore --list` \(trade\) = ok/);
+ assert.strictEqual(rec.markdown, r.out.split('\n').pop());
+ }),
+
+ test('refuses before anything is created when drill.env would leave a production value', async () => {
+ const host = await pgScenario({ backup: false, drill: { env: { PORT: '{port}', HOST: '127.0.0.1', TRADE_DRILL: '1' } } });
+ const before = fsSnapshot(host);
+ const r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 1, r.out);
+ assert.match(r.out, /VALKEY_URL/);
+ assert.match(r.out, /production Valkey keyspace/);
+ assert.strictEqual(host.createdDatabases.length, 0, 'nothing created');
+ assert.strictEqual(host.dumps.length, 0, 'no pg_dump');
+ assert.strictEqual(host.restores.length, 0, 'no pg_restore');
+ assert.strictEqual(host.systemdRuns.length, 0, 'nothing started');
+ assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');
+ }),
+
+ test('refuses when a PostgreSQL database is not mapped in drill.databases (it would run on production)', async () => {
+ const host = await pgScenario({ backup: false });
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.trade.databases.push({ name: 'ledger', engine: 'postgresql', database: 'ov_trade_ledger' });
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+ const before = fsSnapshot(host);
+ const r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 1, r.out);
+ assert.match(r.out, /must map every PostgreSQL database \(ledger is missing\)/);
+ assert.strictEqual(host.createdDatabases.length + host.restores.length + host.systemdRuns.length, 0, 'nothing created or started');
+ assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');
+ }),
+
+ test('the drill directory is traversable by postgres (0711) and the dump copy stays 0600 postgres', async () => {
+ const host = await pgScenario();
+ const r = await host.cli('drill', 'trade', '--keep');
+ assert.strictEqual(r.code, 0, r.out);
+ const tmp = lastLog(host).dir;
+ assert.strictEqual(host.files.get(tmp).mode, 0o711);
+ const copy = host.files.get(`${tmp}/pg/trade.dump`);
+ assert.strictEqual(copy.owner, 'postgres');
+ assert.strictEqual(copy.mode, 0o600);
+ }),
+
+ test('cleanup on failure: a failed comparison still drops the scratch database and role', async () => {
+ const host = await pgScenario();
+ host.drill.productionCount = 500;
+ host.drill.restoredCount = 400;
+ const r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.result, 'failed');
+ assert.strictEqual(rec.failure.stage, 'compare');
+ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 500, restored: 400, match: false }]);
+ // The scratch database and role are dropped by the finally, even though the drill failed.
+ assert.strictEqual(rec.scratch.dropped, true);
+ assert.ok(host.droppedDatabases.some((c) => c.name === rec.scratch.database));
+ assert.ok(!host.pgRoles.has(rec.scratch.role), 'the drill role was dropped');
+ assert.ok(host.pgDatabases.has('ov_trade'), 'production database still there');
+ }),
+
+ test('cleanup on failure: an instance that never becomes ready still drops the scratch', async () => {
+ const host = await pgScenario();
+ host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });
+ const r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'ready');
+ assert.ok(host.droppedDatabases.some((c) => c.name === rec.scratch.database));
+ assert.ok(!host.pgRoles.has(rec.scratch.role));
+ assert.ok(!host.files.has(rec.dir), 'the drill directory is removed');
+ }),
+
+ test('counts: a match passes; 2 rows off fails at tolerance 0 but passes at 3', async () => {
+ let host = await pgScenario({ drill: { countsTolerance: 0 } });
+ host.drill.productionCount = 502;
+ host.drill.restoredCount = 500;
+ let r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 2, r.out);
+ let rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'compare');
+ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 502, restored: 500, match: false }], 'no tolerance field at 0');
+
+ host = await pgScenario({ drill: { countsTolerance: 3 } });
+ host.drill.productionCount = 502;
+ host.drill.restoredCount = 500;
+ r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 0, r.out);
+ rec = lastLog(host);
+ assert.deepStrictEqual(rec.counts, [{ db: 'trade', table: 'items', production: 502, restored: 500, match: true, tolerance: 3 }]);
+ assert.match(rec.markdown, /items 502 = 500 \(±3\)/);
+ assert.match(r.out, /items: production 502, restored 500 \(±3\)/);
+ }),
+
+ test('artifact selection: --backup uses <dir>/<name>.dump; a verification-only record is skipped', async () => {
+ // (a) --backup selects the .dump file in the named directory.
+ let host = await pgScenario({ backup: false });
+ host.ensureDir('/srv/old-backup');
+ host.put('/srv/old-backup/trade.dump', 'pg_dump -Fc of ov_trade', { owner: 'root', mode: 0o600 });
+ let r = await host.cli('drill', 'trade', '--backup', '/srv/old-backup', '--json');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');
+ assert.strictEqual(lastLog(host).databases[0].source, '/srv/old-backup/trade.dump');
+
+ // (b) a newer verification-only record (no dir, no dump) is not picked: the newest record that
+ // actually holds a .dump entry wins.
+ host = await pgScenario({ backup: false });
+ const goodDir = '/var/backups/openvibe/trade/20260922-115500';
+ host.ensureDir(goodDir);
+ host.put(`${goodDir}/trade.dump`, 'pg_dump -Fc of ov_trade', { owner: 'root', mode: 0o600 });
+ const good = { service: 'trade', at: '2026-09-22T11:55:00.000Z', engine: 'postgresql', ok: true, dir: goodDir, files: [{ name: 'trade', source: 'ov_trade', dest: `${goodDir}/trade.dump`, file: 'trade.dump', bytes: 23 }] };
+ const verificationOnly = { service: 'trade', at: '2026-09-22T12:00:30.000Z', engine: 'postgresql', ok: true, files: [], verified: { type: 'diff', stop: '2026-09-22T02:20:11.000Z', walOk: true, ok: true } };
+ host.put('/var/lib/openvibe-host/backups/trade.jsonl', `${JSON.stringify(good)}\n${JSON.stringify(verificationOnly)}\n`, { mode: 0o640 });
+ r = await host.cli('drill', 'trade', '--json');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.strictEqual(JSON.parse(r.out).backup, goodDir);
+
+ // (c) only a verification-only record: refused, as if there were no backup at all.
+ host = await pgScenario({ backup: false });
+ host.put('/var/lib/openvibe-host/backups/trade.jsonl', `${JSON.stringify(verificationOnly)}\n`, { mode: 0o640 });
+ r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 1, r.out);
+ assert.match(r.out, /no good ovhost backup recorded for trade/);
+ }),
+
+ test('free-space: a drill root that does not exist yet is created before measuring, so the first drill runs', async () => {
+ const host = await pgScenario();
+ // A host that has never run a drill has no <drillDir>; statfs() works on existing paths only.
+ assert.ok(!host.files.has('/var/lib/openvibe-drills'), 'the drill root does not exist yet');
+ const r = await host.cli('drill', 'trade');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.ok(host.files.has('/var/lib/openvibe-drills'), 'the drill root was created before measuring');
+ assert.strictEqual(lastLog(host).result, 'passed');
+ }),
+
+ test('free-space: refuses before restoring when the drill directory has almost no room', async () => {
+ const host = await pgScenario({ backup: false });
+ host.ensureDir('/srv/old-backup');
+ host.put('/srv/old-backup/trade.dump', 'pg_dump -Fc of ov_trade', { owner: 'root', mode: 0o600 });
+ host.statfsFree = 1024;
+ const r = await host.cli('drill', 'trade', '--backup', '/srv/old-backup');
+ assert.strictEqual(r.code, 2, r.out);
+ assert.match(r.out, /not enough free space under \/var\/lib\/openvibe-drills/);
+ assert.match(r.out, /1024 bytes free/);
+ assert.strictEqual(host.restores.length, 0, 'no pg_restore ran');
+ assert.strictEqual(host.dumps.length, 0, 'no pg_dump ran');
+ assert.strictEqual(host.systemdRuns.length, 0);
+ }),
+]);
@@ -13,55 +13,55 @@ const { normalise } = require('../lib/inventory');
@@ -13,55 +13,55 @@ const { normalise } = require('../lib/inventory');
const { compareBodies } = require('../lib/commands/drill');
const { compareBodies } = require('../lib/commands/drill');
const DRILL_DIR = '/var/lib/openvibe-drills';
const DRILL_DIR = '/var/lib/openvibe-drills';
-const PROD_DB = '/var/lib/openvibe-community/community.db';
+const PROD_DB = '/var/lib/openvibe-tools/tools.db';
const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';
const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';
-function communityEntry(drillOverrides = {}) {
+function toolsEntry(drillOverrides = {}) {
return {
return {
- repo: '/opt/openvibe.community',
- units: ['openvibe-community.service'],
- envFile: '/etc/openvibe/community.env',
+ repo: '/opt/openvibe.tools',
+ units: ['openvibe-tools.service'],
+ envFile: '/etc/openvibe/tools.env',
port: 4200,
port: 4200,
ready: { url: 'http://127.0.0.1:4200/api/ready', timeoutSeconds: 30 },
ready: { url: 'http://127.0.0.1:4200/api/ready', timeoutSeconds: 30 },
- databases: [{ name: 'community', path: PROD_DB }],
+ databases: [{ name: 'tools', path: PROD_DB }],
drill: {
drill: {
port: 14200,
port: 14200,
- databases: { community: 'COMMUNITY_DB_PATH' },
+ databases: { tools: 'TOOLS_DB_PATH' },
env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words' },
env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words' },
dirs: ['{tmp}/data'],
dirs: ['{tmp}/data'],
compare: [{ path: '/api/pastes?limit=5' }, { path: '/api/v1/pulse?limit=5', ignore: ['generated_at'] }],
compare: [{ path: '/api/pastes?limit=5' }, { path: '/api/v1/pulse?limit=5', ignore: ['generated_at'] }],
- counts: [{ db: 'community', table: 'pastes' }],
+ counts: [{ db: 'tools', table: 'pastes' }],
...drillOverrides,
...drillOverrides,
},
},
};
};
}
}
/**
/**
- * The standard fake host plus Community in production, with one `ovhost backup community` taken.
+ * The standard fake host plus Tools in production, with one `ovhost backup tools` taken.
* The drill instance's behaviour is controlled through `host.drill`.
* The drill instance's behaviour is controlled through `host.drill`.
*/
*/
async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
const host = scenario();
const host = scenario();
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
- doc.services.community = communityEntry(drillOverrides);
+ doc.services.tools = toolsEntry(drillOverrides);
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
- const repo = host.createRepo('/opt/openvibe.community', { owner: 'ubuntu' });
- const sha = repo.commit({ 'package.json': '{"name":"openvibe-community"}', 'server/index.js': 'community();' }, { message: 'initial' });
+ const repo = host.createRepo('/opt/openvibe.tools', { owner: 'ubuntu' });
+ const sha = repo.commit({ 'package.json': '{"name":"openvibe-tools"}', 'server/index.js': 'tools();' }, { message: 'initial' });
repo.publish(sha);
repo.publish(sha);
repo.checkout(sha);
repo.checkout(sha);
- host.addUnit('openvibe-community.service', { mainPid: 2222 });
+ host.addUnit('openvibe-tools.service', { mainPid: 2222 });
host.alivePids.add(2222);
host.alivePids.add(2222);
host.listeners.set(4200, [{ pid: 2222, process: 'node' }]);
host.listeners.set(4200, [{ pid: 2222, process: 'node' }]);
- host.put('/etc/systemd/system/openvibe-community.service', [
+ host.put('/etc/systemd/system/openvibe-tools.service', [
'[Service]',
'[Service]',
'User=ubuntu',
'User=ubuntu',
- 'WorkingDirectory=/opt/openvibe.community',
+ 'WorkingDirectory=/opt/openvibe.tools',
'ExecStart=/usr/bin/node server/index.js',
'ExecStart=/usr/bin/node server/index.js',
`Environment=NODE_ENV=production "SESSION_SECRET=${SECRET}-unit"`,
`Environment=NODE_ENV=production "SESSION_SECRET=${SECRET}-unit"`,
'',
'',
].join('\n'));
].join('\n'));
- host.put('/etc/openvibe/community.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.community\n`, { mode: 0o600 });
+ host.put('/etc/openvibe/tools.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.tools\n`, { mode: 0o600 });
host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
host.http.set('http://127.0.0.1:4200/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
host.http.set('http://127.0.0.1:4200/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
@@ -101,7 +101,7 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
@@ -101,7 +101,7 @@ async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
};
};
if (backup) {
if (backup) {
- const b = await host.cli('backup', 'community', '--json');
+ const b = await host.cli('backup', 'tools', '--json');
assert.strictEqual(b.code, 0, b.out);
assert.strictEqual(b.code, 0, b.out);
host.backupDir = JSON.parse(b.out).dir;
host.backupDir = JSON.parse(b.out).dir;
}
}
@@ -201,31 +201,31 @@ function changedPaths(before, after) {
@@ -201,31 +201,31 @@ function changedPaths(before, after) {
}
}
function lastLog(host) {
function lastLog(host) {
- const text = host.read('/var/lib/openvibe-host/drills/community.jsonl');
+ const text = host.read('/var/lib/openvibe-host/drills/tools.jsonl');
const lines = text.trim().split('\n');
const lines = text.trim().split('\n');
return JSON.parse(lines[lines.length - 1]);
return JSON.parse(lines[lines.length - 1]);
}
}
const allowedWrite = (p, tmp) => p === DRILL_DIR || p === tmp || p.startsWith(`${tmp}/`)
const allowedWrite = (p, tmp) => p === DRILL_DIR || p === tmp || p.startsWith(`${tmp}/`)
- || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/community.jsonl'
- || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/community-drill.lock';
+ || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/tools.jsonl'
+ || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/tools-drill.lock';
runTests([
runTests([
test('full flow: restore, integrity, sandboxed second instance, identical answers, stop by pid, cleanup, log row', async () => {
test('full flow: restore, integrity, sandboxed second instance, identical answers, stop by pid, cleanup, log row', async () => {
const host = await drillScenario();
const host = await drillScenario();
const before = fsSnapshot(host);
const before = fsSnapshot(host);
const callsBefore = host.calls.length;
const callsBefore = host.calls.length;
- const r = await host.cli('drill', 'community');
+ const r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
const rec = lastLog(host);
const rec = lastLog(host);
assert.strictEqual(rec.result, 'passed');
assert.strictEqual(rec.result, 'passed');
assert.strictEqual(rec.failure, null);
assert.strictEqual(rec.failure, null);
assert.strictEqual(rec.backup, host.backupDir);
assert.strictEqual(rec.backup, host.backupDir);
const tmp = rec.dir;
const tmp = rec.dir;
- assert.match(tmp, /^\/var\/lib\/openvibe-drills\/community-\d{8}-\d{6}$/);
- assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['community', 'ok', `${tmp}/db/community.db`]]);
+ assert.match(tmp, /^\/var\/lib\/openvibe-drills\/tools-\d{8}-\d{6}$/);
+ assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['tools', 'ok', `${tmp}/db/tools.db`]]);
assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/pastes?limit=5', true], ['/api/v1/pulse?limit=5', true]]);
assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/pastes?limit=5', true], ['/api/v1/pulse?limit=5', true]]);
- assert.deepStrictEqual(rec.counts, [{ db: 'community', table: 'pastes', production: 892, restored: 892, match: true }]);
+ assert.deepStrictEqual(rec.counts, [{ db: 'tools', table: 'pastes', production: 892, restored: 892, match: true }]);
assert.strictEqual(rec.ready.ok, true);
assert.strictEqual(rec.ready.ok, true);
assert.strictEqual(rec.stop.ok, true);
assert.strictEqual(rec.stop.ok, true);
assert.strictEqual(rec.stop.signal, 'SIGTERM');
assert.strictEqual(rec.stop.signal, 'SIGTERM');
@@ -233,20 +233,20 @@ runTests([
@@ -233,20 +233,20 @@ runTests([
const calls = host.calls.slice(callsBefore);
const calls = host.calls.slice(callsBefore);
// The copy is made as the service user; integrity and counts run as the service user.
// The copy is made as the service user; integrity and counts run as the service user.
// Backups are root-only: root copies, and the copy belongs to the service user, 0600.
// Backups are root-only: root copies, and the copy belongs to the service user, 0600.
- const cp = calls.find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/db/community.db`));
+ const cp = calls.find((c) => c.cmd === 'install' && c.args.includes(`${tmp}/db/tools.db`));
assert.strictEqual(cp.privileged, true);
assert.strictEqual(cp.privileged, true);
- assert.deepStrictEqual(cp.args, ['-o', 'ubuntu', '-m', '0600', '-T', '--', `${host.backupDir}/community.db`, `${tmp}/db/community.db`]);
+ assert.deepStrictEqual(cp.args, ['-o', 'ubuntu', '-m', '0600', '-T', '--', `${host.backupDir}/tools.db`, `${tmp}/db/tools.db`]);
const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));
const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));
- assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/community.db`, 'ubuntu']);
+ assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/tools.db`, 'ubuntu']);
// systemd-run: production ExecStart/WorkingDirectory, production env file then the override file,
// systemd-run: production ExecStart/WorkingDirectory, production env file then the override file,
// sandbox properties, loopback-only networking, the service user.
// sandbox properties, loopback-only networking, the service user.
assert.strictEqual(host.systemdRuns.length, 1);
assert.strictEqual(host.systemdRuns.length, 1);
const run = host.systemdRuns[0];
const run = host.systemdRuns[0];
assert.strictEqual(run.uid, 'ubuntu');
assert.strictEqual(run.uid, 'ubuntu');
- assert.strictEqual(run.cwd, '/opt/openvibe.community');
+ assert.strictEqual(run.cwd, '/opt/openvibe.tools');
assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js']);
assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js']);
- assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/community.env', `${tmp}/drill.env`]);
+ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/tools.env', `${tmp}/drill.env`]);
for (const p of ['ProtectSystem=strict', `ReadWritePaths=${tmp}`, 'PrivateTmp=yes', 'NoNewPrivileges=yes', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'SocketBindAllow=tcp:14200', 'SocketBindDeny=any', 'Environment=NODE_ENV=production']) {
for (const p of ['ProtectSystem=strict', `ReadWritePaths=${tmp}`, 'PrivateTmp=yes', 'NoNewPrivileges=yes', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'SocketBindAllow=tcp:14200', 'SocketBindDeny=any', 'Environment=NODE_ENV=production']) {
assert.ok(run.props.includes(p), `systemd-run -p ${p}`);
assert.ok(run.props.includes(p), `systemd-run -p ${p}`);
}
}
@@ -259,7 +259,7 @@ runTests([
@@ -259,7 +259,7 @@ runTests([
assert.deepStrictEqual(kills.map((c) => c.args), [['SIGTERM', run.pid]]);
assert.deepStrictEqual(kills.map((c) => c.args), [['SIGTERM', run.pid]]);
assert.ok(!calls.some((c) => /pkill|killall/.test(c.cmd)));
assert.ok(!calls.some((c) => /pkill|killall/.test(c.cmd)));
assert.ok(host.alivePids.has(2222), 'production still running');
assert.ok(host.alivePids.has(2222), 'production still running');
- assert.strictEqual(host.units.get('openvibe-community.service').restarts, 0);
+ assert.strictEqual(host.units.get('openvibe-tools.service').restarts, 0);
assert.ok(!host.restarts().length, 'nothing restarted');
assert.ok(!host.restarts().length, 'nothing restarted');
// Only the drill directory (now removed), the lock (released) and the drill log changed.
// Only the drill directory (now removed), the lock (released) and the drill log changed.
@@ -267,33 +267,33 @@ runTests([
@@ -267,33 +267,33 @@ runTests([
const changed = changedPaths(before, after);
const changed = changedPaths(before, after);
for (const p of changed) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);
for (const p of changed) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);
assert.ok(!host.files.has(tmp), 'temp dir removed');
assert.ok(!host.files.has(tmp), 'temp dir removed');
- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'), 'lock released');
+ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'), 'lock released');
// Secrets: nothing ovhost printed, logged or passed on carries a value.
// Secrets: nothing ovhost printed, logged or passed on carries a value.
- const everything = [r.out, host.read('/var/lib/openvibe-host/drills/community.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');
+ const everything = [r.out, host.read('/var/lib/openvibe-host/drills/tools.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');
assert.ok(!everything.includes(SECRET), 'a secret value leaked');
assert.ok(!everything.includes(SECRET), 'a secret value leaked');
- assert.ok(!host.reads.includes('/etc/openvibe/community.env'), 'ovhost never reads the env file for a drill');
+ assert.ok(!host.reads.includes('/etc/openvibe/tools.env'), 'ovhost never reads the env file for a drill');
// Markdown row ready for docs/restore-drills.md.
// Markdown row ready for docs/restore-drills.md.
- assert.match(r.out, /\| 2026-09-22 12:01 \| community \| `\/var\/backups\/openvibe\/community\/20260922-120000\/` \(`ovhost drill community`\) \| `pragma integrity_check` \(community\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);
+ assert.match(r.out, /\| 2026-09-22 12:01 \| tools \| `\/var\/backups\/openvibe\/tools\/20260922-120000\/` \(`ovhost drill tools`\) \| `pragma integrity_check` \(tools\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);
assert.strictEqual(rec.markdown, r.out.split('\n').pop());
assert.strictEqual(rec.markdown, r.out.split('\n').pop());
}),
}),
test('--keep leaves the drill directory; the override file holds inventory values only', async () => {
test('--keep leaves the drill directory; the override file holds inventory values only', async () => {
const host = await drillScenario();
const host = await drillScenario();
const before = fsSnapshot(host);
const before = fsSnapshot(host);
- const r = await host.cli('drill', 'community', '--keep');
+ const r = await host.cli('drill', 'tools', '--keep');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
const rec = lastLog(host);
const rec = lastLog(host);
assert.strictEqual(rec.kept, true);
assert.strictEqual(rec.kept, true);
const tmp = rec.dir;
const tmp = rec.dir;
assert.strictEqual(host.files.get(tmp).owner, 'ubuntu');
assert.strictEqual(host.files.get(tmp).owner, 'ubuntu');
assert.strictEqual(host.files.get(tmp).mode, 0o700);
assert.strictEqual(host.files.get(tmp).mode, 0o700);
- assert.strictEqual(host.files.get(`${tmp}/db/community.db`).owner, 'ubuntu');
- assert.strictEqual(host.read(`${tmp}/db/community.db`), host.read(`${host.backupDir}/community.db`));
+ assert.strictEqual(host.files.get(`${tmp}/db/tools.db`).owner, 'ubuntu');
+ assert.strictEqual(host.read(`${tmp}/db/tools.db`), host.read(`${host.backupDir}/tools.db`));
assert.strictEqual(host.files.get(`${tmp}/data`).owner, 'ubuntu');
assert.strictEqual(host.files.get(`${tmp}/data`).owner, 'ubuntu');
const env = host.read(`${tmp}/drill.env`);
const env = host.read(`${tmp}/drill.env`);
- for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `COMMUNITY_DB_PATH=${tmp}/db/community.db`]) {
+ for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `TOOLS_DB_PATH=${tmp}/db/tools.db`]) {
assert.ok(env.split('\n').includes(line), `drill.env has ${line}`);
assert.ok(env.split('\n').includes(line), `drill.env has ${line}`);
}
}
assert.ok(!env.includes(SECRET));
assert.ok(!env.includes(SECRET));
@@ -305,7 +305,7 @@ runTests([
@@ -305,7 +305,7 @@ runTests([
test('integrity failure: nothing is started, the copy is removed, the log says failed', async () => {
test('integrity failure: nothing is started, the copy is removed, the log says failed', async () => {
const host = await drillScenario();
const host = await drillScenario();
host.drill.integrity = [{ integrity_check: '*** in database main ***' }, { integrity_check: 'Page 5: btree page has a bad cell' }];
host.drill.integrity = [{ integrity_check: '*** in database main ***' }, { integrity_check: 'Page 5: btree page has a bad cell' }];
- const r = await host.cli('drill', 'community');
+ const r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
const rec = lastLog(host);
const rec = lastLog(host);
assert.strictEqual(rec.result, 'failed');
assert.strictEqual(rec.result, 'failed');
@@ -319,7 +319,7 @@ runTests([
@@ -319,7 +319,7 @@ runTests([
test('readiness timeout: the instance is stopped by its pid and the directory removed', async () => {
test('readiness timeout: the instance is stopped by its pid and the directory removed', async () => {
const host = await drillScenario();
const host = await drillScenario();
host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });
host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });
- const r = await host.cli('drill', 'community');
+ const r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
const rec = lastLog(host);
const rec = lastLog(host);
assert.strictEqual(rec.failure.stage, 'ready');
assert.strictEqual(rec.failure.stage, 'ready');
@@ -336,7 +336,7 @@ runTests([
@@ -336,7 +336,7 @@ runTests([
const host = await drillScenario();
const host = await drillScenario();
let probes = 0;
let probes = 0;
host.drill.ready = () => { probes += 1; if (probes === 2) host.alivePids.delete(host.systemdRuns[0].pid); return { status: 503 }; };
host.drill.ready = () => { probes += 1; if (probes === 2) host.alivePids.delete(host.systemdRuns[0].pid); return { status: 503 }; };
- const r = await host.cli('drill', 'community');
+ const r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
const rec = lastLog(host);
const rec = lastLog(host);
assert.strictEqual(rec.failure.stage, 'ready');
assert.strictEqual(rec.failure.stage, 'ready');
@@ -350,7 +350,7 @@ runTests([
@@ -350,7 +350,7 @@ runTests([
host.drill.pastes = () => ({ status: 200, body: PASTES.replace('world', 'World') });
host.drill.pastes = () => ({ status: 200, body: PASTES.replace('world', 'World') });
host.drill.pulse = () => ({ status: 200, body: { items: [{ id: 2 }], generated_at: 'different, but ignored' } });
host.drill.pulse = () => ({ status: 200, body: { items: [{ id: 2 }], generated_at: 'different, but ignored' } });
host.drill.productionCount = 900;
host.drill.productionCount = 900;
- const r = await host.cli('drill', 'community');
+ const r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
const rec = lastLog(host);
const rec = lastLog(host);
assert.strictEqual(rec.failure.stage, 'compare');
assert.strictEqual(rec.failure.stage, 'compare');
@@ -360,7 +360,7 @@ runTests([
@@ -360,7 +360,7 @@ runTests([
assert.strictEqual(pulse.match, false);
assert.strictEqual(pulse.match, false);
assert.match(pulse.detail, /\$\.items\[0\]\.id/);
assert.match(pulse.detail, /\$\.items\[0\]\.id/);
assert.ok(!/generated_at/.test(pulse.detail));
assert.ok(!/generated_at/.test(pulse.detail));
- assert.deepStrictEqual(rec.counts[0], { db: 'community', table: 'pastes', production: 900, restored: 892, match: false });
+ assert.deepStrictEqual(rec.counts[0], { db: 'tools', table: 'pastes', production: 900, restored: 892, match: false });
assert.match(rec.markdown, /`\/api\/pastes\?limit=5` differs \(bodies differ/);
assert.match(rec.markdown, /`\/api\/pastes\?limit=5` differs \(bodies differ/);
assert.match(rec.markdown, /pastes 900 ≠ 892 \| failed \|$/);
assert.match(rec.markdown, /pastes 900 ≠ 892 \| failed \|$/);
// Still stopped and cleaned up.
// Still stopped and cleaned up.
@@ -371,14 +371,14 @@ runTests([
@@ -371,14 +371,14 @@ runTests([
test('acceptance: the restored instance alone answers each declared GET with its status and JSON keys (WS-S task 2)', async () => {
test('acceptance: the restored instance alone answers each declared GET with its status and JSON keys (WS-S task 2)', async () => {
const accept = [{ path: '/api/ready', keys: ['status'] }, { path: '/api/v1/pulse?limit=5', keys: ['items', 'generated_at'] }];
const accept = [{ path: '/api/ready', keys: ['status'] }, { path: '/api/v1/pulse?limit=5', keys: ['items', 'generated_at'] }];
let host = await drillScenario({ drill: { acceptance: accept } });
let host = await drillScenario({ drill: { acceptance: accept } });
- let r = await host.cli('drill', 'community');
+ let r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
let rec = lastLog(host);
let rec = lastLog(host);
assert.deepStrictEqual(rec.acceptance, [{ path: '/api/ready', ok: true }, { path: '/api/v1/pulse?limit=5', ok: true }]);
assert.deepStrictEqual(rec.acceptance, [{ path: '/api/ready', ok: true }, { path: '/api/v1/pulse?limit=5', ok: true }]);
assert.match(rec.markdown, /acceptance `\/api\/ready`, `\/api\/v1\/pulse\?limit=5`/);
assert.match(rec.markdown, /acceptance `\/api\/ready`, `\/api\/v1\/pulse\?limit=5`/);
host = await drillScenario({ drill: { acceptance: [{ path: '/api/v1/pulse?limit=5', keys: ['items', 'cursor'] }, { path: '/api/ready', status: 204 }] } });
host = await drillScenario({ drill: { acceptance: [{ path: '/api/v1/pulse?limit=5', keys: ['items', 'cursor'] }, { path: '/api/ready', status: 204 }] } });
- r = await host.cli('drill', 'community');
+ r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
rec = lastLog(host);
rec = lastLog(host);
assert.strictEqual(rec.failure.stage, 'acceptance');
assert.strictEqual(rec.failure.stage, 'acceptance');
@@ -389,7 +389,7 @@ runTests([
@@ -389,7 +389,7 @@ runTests([
test('acceptance entries are validated in the inventory', async () => {
test('acceptance entries are validated in the inventory', async () => {
const host = await drillScenario({ drill: { acceptance: [{ path: 'no-slash' }] }, backup: false });
const host = await drillScenario({ drill: { acceptance: [{ path: 'no-slash' }] }, backup: false });
- const r = await host.cli('drill', 'community');
+ const r = await host.cli('drill', 'tools');
assert.notStrictEqual(r.code, 0);
assert.notStrictEqual(r.code, 0);
assert.match(r.out, /acceptance\[0\]\.path must be a path/);
assert.match(r.out, /acceptance\[0\]\.path must be a path/);
}),
}),
@@ -406,24 +406,24 @@ runTests([
@@ -406,24 +406,24 @@ runTests([
const host = await drillScenario();
const host = await drillScenario();
const before = fsSnapshot(host);
const before = fsSnapshot(host);
host.drill.onStart = () => ({ code: 1, stderr: 'Failed to start transient service unit: Unit already exists.' });
host.drill.onStart = () => ({ code: 1, stderr: 'Failed to start transient service unit: Unit already exists.' });
- const r = await host.cli('drill', 'community');
+ const r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
const rec = lastLog(host);
const rec = lastLog(host);
assert.strictEqual(rec.failure.stage, 'start');
assert.strictEqual(rec.failure.stage, 'start');
assert.match(rec.failure.message, /systemd-run failed/);
assert.match(rec.failure.message, /systemd-run failed/);
assert.ok(!host.files.has(rec.dir));
assert.ok(!host.files.has(rec.dir));
- assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'));
+ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/tools-drill.lock'));
for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, rec.dir), `unexpected write: ${p}`);
for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, rec.dir), `unexpected write: ${p}`);
// A second drill can run straight after.
// A second drill can run straight after.
host.drill.onStart = null;
host.drill.onStart = null;
host.advance(1000);
host.advance(1000);
- assert.strictEqual((await host.cli('drill', 'community')).code, 0);
+ assert.strictEqual((await host.cli('drill', 'tools')).code, 0);
}),
}),
test('an instance that ignores SIGTERM gets SIGKILL; one that survives SIGKILL keeps its directory and fails', async () => {
test('an instance that ignores SIGTERM gets SIGKILL; one that survives SIGKILL keeps its directory and fails', async () => {
let host = await drillScenario();
let host = await drillScenario();
host.drill.ignoreTerm = true;
host.drill.ignoreTerm = true;
- let r = await host.cli('drill', 'community');
+ let r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
let rec = lastLog(host);
let rec = lastLog(host);
assert.strictEqual(rec.stop.signal, 'SIGKILL');
assert.strictEqual(rec.stop.signal, 'SIGKILL');
@@ -432,7 +432,7 @@ runTests([
@@ -432,7 +432,7 @@ runTests([
host = await drillScenario();
host = await drillScenario();
host.drill.ignoreTerm = true;
host.drill.ignoreTerm = true;
host.drill.ignoreKill = true;
host.drill.ignoreKill = true;
- r = await host.cli('drill', 'community');
+ r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 2, r.out);
assert.strictEqual(r.code, 2, r.out);
rec = lastLog(host);
rec = lastLog(host);
assert.strictEqual(rec.failure.stage, 'stop');
assert.strictEqual(rec.failure.stage, 'stop');
@@ -445,7 +445,7 @@ runTests([
@@ -445,7 +445,7 @@ runTests([
test('refusals: not root, port in use, unsupported service, no drill block, no backup, missing database copy', async () => {
test('refusals: not root, port in use, unsupported service, no drill block, no backup, missing database copy', async () => {
let host = await drillScenario();
let host = await drillScenario();
host.exec.isRoot = async () => false;
host.exec.isRoot = async () => false;
- let r = await host.cli('drill', 'community');
+ let r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 1);
assert.strictEqual(r.code, 1);
assert.match(r.out, /must run as root/);
assert.match(r.out, /must run as root/);
assert.strictEqual(host.systemdRuns.length, 0);
assert.strictEqual(host.systemdRuns.length, 0);
@@ -453,14 +453,14 @@ runTests([
@@ -453,14 +453,14 @@ runTests([
host = await drillScenario();
host = await drillScenario();
host.listeners.set(14200, [{ pid: 777, process: 'python3' }]);
host.listeners.set(14200, [{ pid: 777, process: 'python3' }]);
const before = fsSnapshot(host);
const before = fsSnapshot(host);
- r = await host.cli('drill', 'community');
+ r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 1);
assert.strictEqual(r.code, 1);
assert.match(r.out, /port 14200 is already in use \(python3 pid 777\)/);
assert.match(r.out, /port 14200 is already in use \(python3 pid 777\)/);
assert.strictEqual(host.systemdRuns.length, 0);
assert.strictEqual(host.systemdRuns.length, 0);
assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');
assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');
host = await drillScenario({ drill: { supported: false, reason: 'workers have no off switch' } });
host = await drillScenario({ drill: { supported: false, reason: 'workers have no off switch' } });
- r = await host.cli('drill', 'community');
+ r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 1);
assert.strictEqual(r.code, 1);
assert.match(r.out, /does not support restore drills: workers have no off switch/);
assert.match(r.out, /does not support restore drills: workers have no off switch/);
@@ -470,20 +470,20 @@ runTests([
@@ -470,20 +470,20 @@ runTests([
assert.match(r.out, /media has no drill block/);
assert.match(r.out, /media has no drill block/);
host = await drillScenario({ backup: false });
host = await drillScenario({ backup: false });
- r = await host.cli('drill', 'community');
+ r = await host.cli('drill', 'tools');
assert.strictEqual(r.code, 1);
assert.strictEqual(r.code, 1);
- assert.match(r.out, /no good ovhost backup recorded for community/);
+ assert.match(r.out, /no good ovhost backup recorded for tools/);
host = await drillScenario({ backup: false });
host = await drillScenario({ backup: false });
host.ensureDir('/srv/old-backup');
host.ensureDir('/srv/old-backup');
- r = await host.cli('drill', 'community', '--backup', '/srv/old-backup');
+ r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup');
assert.strictEqual(r.code, 1);
assert.strictEqual(r.code, 1);
- assert.match(r.out, /has no copy of the community database/);
- host.put('/srv/old-backup/community.db', 'older sqlite', { owner: 'ubuntu' });
- r = await host.cli('drill', 'community', '--backup', '/srv/old-backup', '--json');
+ assert.match(r.out, /has no copy of the tools database/);
+ host.put('/srv/old-backup/tools.db', 'older sqlite', { owner: 'ubuntu' });
+ r = await host.cli('drill', 'tools', '--backup', '/srv/old-backup', '--json');
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(r.code, 0, r.out);
assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');
assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');
- r = await host.cli('drill', 'community', '--backup', 'relative/dir');
+ r = await host.cli('drill', 'tools', '--backup', 'relative/dir');
assert.strictEqual(r.code, 1);
assert.strictEqual(r.code, 1);
}),
}),
@@ -554,18 +554,18 @@ runTests([
@@ -554,18 +554,18 @@ runTests([
test('inventory: drill blocks are validated', () => {
test('inventory: drill blocks are validated', () => {
const base = { owner: 'ubuntu' };
const base = { owner: 'ubuntu' };
- const svc = (drill, extra = {}) => normalise({ services: { community: { ...base, ...communityEntry(), ...extra, drill: { ...communityEntry().drill, ...drill } } } });
+ const svc = (drill, extra = {}) => normalise({ services: { tools: { ...base, ...toolsEntry(), ...extra, drill: { ...toolsEntry().drill, ...drill } } } });
assert.throws(() => svc({ port: 4200 }), /must differ from the service's own port/);
assert.throws(() => svc({ port: 4200 }), /must differ from the service's own port/);
- assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of community's databases/);
+ assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of tools's databases/);
assert.throws(() => svc({ env: { HOST: '127.0.0.1' } }), /\{port\}/);
assert.throws(() => svc({ env: { HOST: '127.0.0.1' } }), /\{port\}/);
- assert.throws(() => svc({ env: { PORT: '{port}', COMMUNITY_DB_PATH: '/x' } }), /already set by drill.databases/);
+ assert.throws(() => svc({ env: { PORT: '{port}', TOOLS_DB_PATH: '/x' } }), /already set by drill.databases/);
assert.throws(() => svc({ env: { PORT: '{port}\nEVIL=1' } }), /one-line/);
assert.throws(() => svc({ env: { PORT: '{port}\nEVIL=1' } }), /one-line/);
- assert.throws(() => svc({ dirs: ['/opt/openvibe.community/data'] }), /under \{tmp\}/);
- assert.throws(() => svc({ counts: [{ db: 'community', table: 'x; drop table y' }] }), /table name/);
+ assert.throws(() => svc({ dirs: ['/opt/openvibe.tools/data'] }), /under \{tmp\}/);
+ assert.throws(() => svc({ counts: [{ db: 'tools', table: 'x; drop table y' }] }), /table name/);
assert.throws(() => svc({ compare: [{ path: 'http://evil/' }] }), /path/);
assert.throws(() => svc({ compare: [{ path: 'http://evil/' }] }), /path/);
assert.throws(() => svc({ supported: false }), /reason is required/);
assert.throws(() => svc({ supported: false }), /reason is required/);
- assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, community: { ...base, ...communityEntry() } } }), /14200 is a's port/);
- assert.strictEqual(svc({}).services.community.drill.ready, '/api/ready');
+ assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, tools: { ...base, ...toolsEntry() } } }), /14200 is a's port/);
+ assert.strictEqual(svc({}).services.tools.drill.ready, '/api/ready');
}),
}),
test('host.example.json: every service with a database has a drill block; unsupported ones give a reason', () => {
test('host.example.json: every service with a database has a drill block; unsupported ones give a reason', () => {
@@ -1,9 +1,14 @@
@@ -1,9 +1,14 @@
'use strict';
'use strict';
/**
/**
* A fake host implementing the executor interface (lib/executor.js) in memory: a filesystem, git
* A fake host implementing the executor interface (lib/executor.js) in memory: a filesystem, git
- * repositories, systemd units, npm, nginx, ss, HTTP endpoints and SQLite. Nothing here touches the
- * real machine. Every run() call is recorded in `calls` so tests can assert what would have been
- * executed (and as whom).
+ * repositories, systemd units, npm, nginx, ss, HTTP endpoints, SQLite and PostgreSQL. Nothing here
+ * touches the real machine. Every run() call is recorded in `calls` so tests can assert what would
+ * have been executed (and as whom).
+ *
+ * PostgreSQL is modelled the way lib/dbengine.js uses it: `exec.psql(database, sql)` answers rows
+ * (queries only), and the cluster's own tools — pg_dump, pg_restore, createdb, dropdb, pgbackrest —
+ * go through run() as the postgres OS user, which is also how the role password reaches psql
+ * (on stdin, so it never appears in an argv this fake records).
*/
*/
const crypto = require('crypto');
const crypto = require('crypto');
const path = require('path');
const path = require('path');
@@ -26,6 +31,26 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -26,6 +31,26 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
npmRewritesLockfile: true,
npmRewritesLockfile: true,
sqliteHandler: () => [{ n: 0 }],
sqliteHandler: () => [{ n: 0 }],
sqliteCalls: [],
sqliteCalls: [],
+ // ── PostgreSQL ──
+ // The cluster's databases and roles, the archiver's failed WAL count, the stanza's backups
+ // and its status. A test drives them directly (host.pgDatabases.add('ov_trade')) or through
+ // the tools below, exactly as production would.
+ pgDatabases: new Set(),
+ pgRoles: new Map(), // role -> { password }
+ pgArchiver: { failed_count: 0 },
+ pgBackups: [], // [{ type, timestamp: { stop } }] (epoch seconds)
+ pgStanzaStatus: { code: 0, message: 'ok' },
+ pgbackrestCode: 0,
+ pgbackrestStderr: '',
+ psqlHandler: null, // (database, sql, as) -> rows | undefined (undefined: the engine defaults)
+ psqlCalls: [],
+ adminSql: [], // SQL sent to psql on stdin (role creation, where the password must not be argv)
+ dumps: [], // { database, dest, as }
+ restores: [], // { database, role, file, as }
+ createdDatabases: [], // { name, owner, as }
+ droppedDatabases: [], // { name, as }
+ statfsFree: 100 * 1024 ** 3,
+ statfsSize: 500 * 1024 ** 3,
reads: [],
reads: [],
onRestart: null, // (unit) -> void
onRestart: null, // (unit) -> void
onSystemdRun: null, // (spec) -> undefined | { code, stderr } ; spec = { unit, uid, cwd, props, argv, pid }
onSystemdRun: null, // (spec) -> undefined | { code, stderr } ; spec = { unit, uid, cwd, props, argv, pid }
@@ -406,6 +431,66 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -406,6 +431,66 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
return r ? { stdout: '', stderr: '', ...r } : { code: 0, stdout: '', stderr: '' };
return r ? { stdout: '', stderr: '', ...r } : { code: 0, stdout: '', stderr: '' };
}
}
case 'fuser': return { code: (host.openFiles || new Set()).has(path.resolve(args[args.length - 1])) ? 0 : 1, stdout: '', stderr: '' };
case 'fuser': return { code: (host.openFiles || new Set()).has(path.resolve(args[args.length - 1])) ? 0 : 1, stdout: '', stderr: '' };
+ // ── PostgreSQL tools (lib/dbengine.js runs each of these as the postgres OS user) ──
+ case 'psql': {
+ // Admin SQL on stdin (CREATE ROLE … LOGIN PASSWORD …, DROP ROLE …). Queries never come
+ // this way: they go through exec.psql(). The password stays out of argv, as in life.
+ const sql = String(opts.input || '');
+ host.adminSql.push(sql);
+ const create = /CREATE ROLE "([^"]+)" LOGIN PASSWORD '([^']*)'/.exec(sql);
+ if (create) host.pgRoles.set(create[1], { password: create[2] });
+ const drop = /DROP ROLE IF EXISTS "([^"]+)"/.exec(sql);
+ if (drop) host.pgRoles.delete(drop[1]);
+ if (host.onAdminSql) { const r = host.onAdminSql(sql, opts); if (r) return { stdout: '', stderr: '', ...r }; }
+ return { code: 0, stdout: '', stderr: '' };
+ }
+ case 'createdb': {
+ const name = args[args.length - 1];
+ const owner = (args.find((a) => a.startsWith('--owner=')) || '').slice(8) || null;
+ if (host.pgDatabases.has(name)) return { code: 1, stdout: '', stderr: `createdb: error: database creation failed: ERROR: database "${name}" already exists` };
+ host.pgDatabases.add(name);
+ host.createdDatabases.push({ name, owner, as: opts.as || null });
+ return { code: 0, stdout: '', stderr: '' };
+ }
+ case 'dropdb': {
+ const name = args[args.length - 1];
+ host.pgDatabases.delete(name);
+ host.droppedDatabases.push({ name, as: opts.as || null });
+ return { code: 0, stdout: '', stderr: '' };
+ }
+ case 'pg_dump': {
+ const dest = args[args.indexOf('-f') + 1];
+ const database = args[args.indexOf('-d') + 1];
+ if (!host.pgDatabases.has(database)) return { code: 1, stdout: '', stderr: `pg_dump: error: connection to server failed: FATAL: database "${database}" does not exist` };
+ const parent = files.get(path.dirname(path.resolve(dest)));
+ if ((opts.as || 'root') !== 'root' && parent && parent.owner !== (opts.as || 'root')) return { code: 1, stdout: '', stderr: `pg_dump: error: could not open output file "${dest}": Permission denied` };
+ host.dumps.push({ database, dest, as: opts.as || null });
+ put(dest, host.dumpContent ? host.dumpContent(database) : `pg_dump -Fc of ${database}`, { owner: opts.as || 'root', mode: 0o600 });
+ return { code: 0, stdout: '', stderr: '' };
+ }
+ case 'pg_restore': {
+ if (args.includes('--list')) {
+ const e = get(args[args.length - 1]);
+ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `pg_restore: error: could not open input file "${args[args.length - 1]}"` };
+ // A text dump is what pg_restore refuses; the fake's archives all start with its marker.
+ if (!String(e.content).startsWith('pg_dump -Fc')) return { code: 1, stdout: '', stderr: 'pg_restore: error: input file appears to be a text format dump. Please use psql.' };
+ return { code: 0, stdout: ';\n; Archive created at 2026-09-28 03:30:00 UTC\n', stderr: '' };
+ }
+ const database = args[args.indexOf('-d') + 1];
+ const file = args[args.length - 1];
+ const role = (args.find((a) => a.startsWith('--role=')) || '').slice(7) || null;
+ if (!host.pgDatabases.has(database)) return { code: 1, stdout: '', stderr: `pg_restore: error: connection to server failed: database "${database}" does not exist` };
+ if (!get(file)) return { code: 1, stdout: '', stderr: `pg_restore: error: could not open input file "${file}"` };
+ if (role && !host.pgRoles.has(role)) return { code: 1, stdout: '', stderr: `pg_restore: error: role "${role}" does not exist` };
+ host.restores.push({ database, role, file, as: opts.as || null });
+ return { code: 0, stdout: '', stderr: '' };
+ }
+ case 'pgbackrest': {
+ if (!args.includes('info')) return { code: 1, stdout: '', stderr: `fake pgbackrest: ${args.join(' ')}` };
+ if (host.pgbackrestCode !== 0) return { code: host.pgbackrestCode, stdout: '', stderr: host.pgbackrestStderr || 'pgbackrest: error' };
+ const stanzas = host.pgbackrestStanzas ? host.pgbackrestStanzas() : [{ name: 'openvibe', status: host.pgStanzaStatus, backup: host.pgBackups }];
+ return { code: 0, stdout: `${JSON.stringify(stanzas)}\n`, stderr: '' };
+ }
case 'systemd-run': return systemdRunCmd(args);
case 'systemd-run': return systemdRunCmd(args);
case 'cp': {
case 'cp': {
const src = args[args.length - 2];
const src = args[args.length - 2];
@@ -491,6 +576,24 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -491,6 +576,24 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
if (r && r.error) return { status: 0, error: r.error };
if (r && r.error) return { status: 0, error: r.error };
return { status: r.status, body: typeof r.body === 'string' ? r.body : JSON.stringify(r.body || {}) };
return { status: r.status, body: typeof r.body === 'string' ? r.body : JSON.stringify(r.body || {}) };
},
},
+ /** Queries only, as the postgres OS user: the engine defaults answer `datname`/archiver asks. */
+ async psql(database, sql, { as = 'postgres' } = {}) {
+ host.psqlCalls.push({ database, sql, as });
+ calls.push({ cmd: 'psql', args: [database, sql], as, privileged: false, cwd: null });
+ if (host.psqlHandler) { const r = host.psqlHandler(database, sql, as); if (r !== undefined) return r; }
+ const left = /FROM (pg_database|pg_roles) WHERE left\((?:datname|rolname), length\('([^']*)'\)\)/.exec(sql);
+ if (left) return [...(left[1] === 'pg_database' ? host.pgDatabases : host.pgRoles.keys())].filter((n) => n.startsWith(left[2])).map((name) => ({ name }));
+ const named = /datname = '([^']*)'/.exec(sql);
+ if (/pg_database/i.test(sql) && named) return host.pgDatabases.has(named[1]) ? [{ present: '1' }] : [];
+ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];
+ return [{ n: 0 }];
+ },
+ async statfs(p) {
+ // The real statfs() runs on an existing path only: a missing path is an ENOENT, not a
+ // free-space figure. Mirror that so a space check before the directory exists is caught.
+ if (!get(p)) { const err = new Error(`ENOENT: no such file or directory, statfs '${p}'`); err.code = 'ENOENT'; throw err; }
+ return { free: host.statfsFree, size: host.statfsSize };
+ },
async sqlite(db, sql, { as } = {}) {
async sqlite(db, sql, { as } = {}) {
host.sqliteCalls.push({ op: 'query', db, sql, as });
host.sqliteCalls.push({ op: 'query', db, sql, as });
calls.push({ cmd: 'sqlite', args: [db, sql], as: as || null, privileged: false, cwd: null });
calls.push({ cmd: 'sqlite', args: [db, sql], as: as || null, privileged: false, cwd: null });
@@ -7,6 +7,8 @@
@@ -7,6 +7,8 @@
*/
*/
const assert = require('assert');
const assert = require('assert');
const { scenario, test, runTests } = require('./helpers');
const { scenario, test, runTests } = require('./helpers');
+const { normalise } = require('../lib/inventory');
+const nginx = require('../lib/nginx');
function withHost(host, rows) {
function withHost(host, rows) {
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
@@ -52,6 +54,35 @@ runTests([
@@ -52,6 +54,35 @@ runTests([
assert.strictEqual(host.read('/etc/nginx/sites-available/openvibe.host.conf'), null, 'render alone writes nothing');
assert.strictEqual(host.read('/etc/nginx/sites-available/openvibe.host.conf'), null, 'render alone writes nothing');
}),
}),
+ test('a PostgreSQL tenant database reads the verified domains over psql, never a SQLite file', async () => {
+ const host = scenario();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ // Host moved to PostgreSQL (ADR-035): its tenants database is a database in the cluster, not
+ // a file, and its service role's password is not ovhost's to read — so psql as postgres.
+ doc.services.host = {
+ repo: '/opt/openvibe.host',
+ units: ['openvibe-host.service'],
+ port: 4910,
+ ready: { url: 'http://127.0.0.1:4910/api/ready', timeoutSeconds: 30 },
+ databases: [{ name: 'host', engine: 'postgresql', database: 'ov_host' }],
+ nginx: { tenants: { sitesDomain: 'openvibe.host', wildcardCert: 'openvibe.host', database: { engine: 'postgresql', database: 'ov_host' }, maxUpload: '110m' } },
+ };
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+ host.psqlHandler = (database) => {
+ assert.strictEqual(database, 'ov_host', 'the Host database in the cluster');
+ return [{ hostname: 'www.tenant-a.org' }, { hostname: 'docs.tenant-b.net' }];
+ };
+ host.put('/etc/letsencrypt/live/www.tenant-a.org/fullchain.pem', 'CERT', { mode: 0o644 });
+ const domains = await nginx.tenantDomains(host.exec, normalise(doc).services.host);
+ assert.deepStrictEqual(domains, [{ hostname: 'www.tenant-a.org', hasCert: true }, { hostname: 'docs.tenant-b.net', hasCert: false }], 'the same [{ hostname, hasCert }] shape');
+ assert.ok(host.psqlCalls.some((c) => c.database === 'ov_host' && c.sql === nginx.VERIFIED_SQL), 'the verified domains come from psql');
+ assert.strictEqual(host.sqliteCalls.length, 0, 'a database in the cluster is never opened as a file');
+ const r = await host.cli('nginx', 'tenants', 'host');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.match(r.out, /server_name www\.tenant-a\.org;/);
+ assert.match(r.out, /server_name docs\.tenant-b\.net;/);
+ }),
+
test('hostile or misplaced values from the database never reach nginx config', async () => {
test('hostile or misplaced values from the database never reach nginx config', async () => {
const host = withHost(scenario(), [
const host = withHost(scenario(), [
{ hostname: 'ok.example.org' }, { hostname: 'evil.org; } server { listen 80; root /; }' }, { hostname: 'x.openvibe.host' },
{ hostname: 'ok.example.org' }, { hostname: 'evil.org; } server { listen 80; root /; }' }, { hostname: 'x.openvibe.host' },
@@ -15,7 +15,6 @@ const { normalise } = require('../lib/inventory');
@@ -15,7 +15,6 @@ const { normalise } = require('../lib/inventory');
const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
const RELEASE = '655b98a10aaa';
const RELEASE = '655b98a10aaa';
const OLD = '0123456789ab';
const OLD = '0123456789ab';
-const PROD_DB = '/var/lib/openre/openre.db';
const WORKERS = [`openre-rtmp-ingest@${RELEASE}.service`, `openre-restream-worker@${RELEASE}.service`];
const WORKERS = [`openre-rtmp-ingest@${RELEASE}.service`, `openre-restream-worker@${RELEASE}.service`];
/** The standard fake host plus OpenRe as on openvibe-ovh: release layout, two units, worker instances. */
/** The standard fake host plus OpenRe as on openvibe-ovh: release layout, two units, worker instances. */
@@ -37,7 +36,7 @@ async function openreHost({ workersRunning = true } = {}) {
@@ -37,7 +36,7 @@ async function openreHost({ workersRunning = true } = {}) {
'EnvironmentFile=/etc/openvibe/openre.env',
'EnvironmentFile=/etc/openvibe/openre.env',
'Environment=NODE_ENV=production',
'Environment=NODE_ENV=production',
'Environment=PORT=4500',
'Environment=PORT=4500',
- `Environment=OPENRE_DB_PATH=${PROD_DB}`,
+ `Environment=DATABASE_URL=postgresql://openre_app:${SECRET}@127.0.0.1:6432/ov_openre`,
'Environment=PATH=/usr/local/bin:/usr/bin:/bin',
'Environment=PATH=/usr/local/bin:/usr/bin:/bin',
'ExecStart=/usr/bin/env node server/index.js',
'ExecStart=/usr/bin/env node server/index.js',
'',
'',
@@ -50,18 +49,23 @@ async function openreHost({ workersRunning = true } = {}) {
@@ -50,18 +49,23 @@ async function openreHost({ workersRunning = true } = {}) {
host.addUnit(`openre-rtmp-ingest@${RELEASE}.service`, { mainPid: 3102, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
host.addUnit(`openre-rtmp-ingest@${RELEASE}.service`, { mainPid: 3102, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
host.addUnit(`openre-restream-worker@${RELEASE}.service`, { mainPid: 3103, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
host.addUnit(`openre-restream-worker@${RELEASE}.service`, { mainPid: 3103, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
host.addUnit(`openre-rtmp-ingest@${OLD}.service`, { active: 'inactive', sub: 'dead', mainPid: 0 });
host.addUnit(`openre-rtmp-ingest@${OLD}.service`, { active: 'inactive', sub: 'dead', mainPid: 0 });
- host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
+ // openre is PostgreSQL: the production database and a healthy pgBackRest stanza so that
+ // `ovhost backup openre` (and the drill's restore of its .dump) can succeed.
+ host.pgDatabases.add('ov_openre');
+ host.pgBackups = [{ type: 'diff', timestamp: { stop: Math.floor(Date.parse('2026-09-22T11:00:00Z') / 1000) } }];
const health = '{"status":"ok","service":"openre-api","version":"0.1.0","release":"dev"}';
const health = '{"status":"ok","service":"openre-api","version":"0.1.0","release":"dev"}';
const robots = 'User-agent: *\nAllow: /$\n';
const robots = 'User-agent: *\nAllow: /$\n';
host.http.set('http://127.0.0.1:4500/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
host.http.set('http://127.0.0.1:4500/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
host.http.set('http://127.0.0.1:4500/api/health', () => ({ status: 200, body: health }));
host.http.set('http://127.0.0.1:4500/api/health', () => ({ status: 200, body: health }));
host.http.set('http://127.0.0.1:4500/robots.txt', () => ({ status: 200, body: robots }));
host.http.set('http://127.0.0.1:4500/robots.txt', () => ({ status: 200, body: robots }));
+ // The API's own status probe counts protected sessions over psql; the drill counts the same
+ // tables in the restored scratch database.
const counts = { stream_definitions: 1, ingest_keys: 1, destinations: 0, migration_map: 0 };
const counts = { stream_definitions: 1, ingest_keys: 1, destinations: 0, migration_map: 0 };
- host.sqliteHandler = (db, sql) => {
- if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
+ host.psqlHandler = (database, sql) => {
+ if (/pg_stat_archiver/i.test(sql)) return [{ n: String(host.pgArchiver.failed_count) }];
const m = /FROM "([a-z_]+)"/.exec(sql);
const m = /FROM "([a-z_]+)"/.exec(sql);
- return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
+ return [{ n: String(m ? (counts[m[1]] || 0) : 0) }];
};
};
host.onSystemdRun = (spec) => {
host.onSystemdRun = (spec) => {
host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);
host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);
@@ -93,7 +97,7 @@ runTests([
@@ -93,7 +97,7 @@ runTests([
assert.strictEqual(o.drill.env.OPENRE_DRILL, '1');
assert.strictEqual(o.drill.env.OPENRE_DRILL, '1');
for (const k of ['EVENTS_URL', 'OV_OAUTH_CLIENT_SECRET', 'MEDIA_API_KEY']) assert.strictEqual(o.drill.env[k], '', k);
for (const k of ['EVENTS_URL', 'OV_OAUTH_CLIENT_SECRET', 'MEDIA_API_KEY']) assert.strictEqual(o.drill.env[k], '', k);
assert.strictEqual(o.drill.env.MEDIA_URL, 'http://127.0.0.1:9');
assert.strictEqual(o.drill.env.MEDIA_URL, 'http://127.0.0.1:9');
- assert.deepStrictEqual(o.drill.databases, { openre: { env: 'OPENRE_DB_PATH', dir: false } });
+ assert.deepStrictEqual(o.drill.databases, { openre: { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } });
// every other service keeps an empty list
// every other service keeps an empty list
assert.deepStrictEqual(inv.services.live.workerUnits, []);
assert.deepStrictEqual(inv.services.live.workerUnits, []);
}),
}),
@@ -175,12 +179,18 @@ runTests([
@@ -175,12 +179,18 @@ runTests([
assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/openre.env', `${rec.dir}/drill.env`]);
assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/openre.env', `${rec.dir}/drill.env`]);
for (const p of ['SocketBindAllow=tcp:14500', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);
for (const p of ['SocketBindAllow=tcp:14500', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);
assert.ok(run.props.includes('Environment=PATH=/usr/local/bin:/usr/bin:/bin'), 'other unit Environment= is passed on');
assert.ok(run.props.includes('Environment=PATH=/usr/local/bin:/usr/bin:/bin'), 'other unit Environment= is passed on');
- assert.ok(!run.props.some((p) => /OPENRE_DB_PATH|Environment=PORT=/.test(p)), 'the production database path and port from the unit never reach the drill instance');
+ assert.ok(!run.props.some((p) => /Environment=DATABASE_URL=|Environment=PORT=/.test(p)), 'the production connection string and port from the unit never reach the drill instance');
const env = Object.fromEntries(host.drillEnv.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
const env = Object.fromEntries(host.drillEnv.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
- assert.deepStrictEqual(env, {
+ const { DATABASE_URL, DATABASE_DIRECT_URL, ...envRest } = env;
+ assert.deepStrictEqual(envRest, {
PORT: '14500', HOST: '127.0.0.1', OPENRE_DRILL: '1', EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', MEDIA_API_KEY: '', OPENRE_RECORDING: 'off',
PORT: '14500', HOST: '127.0.0.1', OPENRE_DRILL: '1', EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', MEDIA_API_KEY: '', OPENRE_RECORDING: 'off',
- OPENRE_DB_PATH: `${rec.dir}/db/openre.db`,
+ VALKEY_URL: 'redis://127.0.0.1:9/0',
});
});
+ // The instance is pointed at the restored scratch database over loopback, never at production.
+ const scratch = `ov_openre_drill_${path.basename(rec.dir).slice('openre-'.length)}`;
+ assert.match(DATABASE_URL, new RegExp(`^postgresql://${scratch}:[0-9a-f]+@127\\.0\\.0\\.1:5432/${scratch}$`));
+ assert.strictEqual(DATABASE_DIRECT_URL, DATABASE_URL);
+ assert.strictEqual(rec.databases[0].integrity, 'pg_restore --list ok');
const calls = host.calls.slice(callsBefore);
const calls = host.calls.slice(callsBefore);
assert.deepStrictEqual(calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', run.pid]], 'only the drill instance is signalled');
assert.deepStrictEqual(calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', run.pid]], 'only the drill instance is signalled');
@@ -191,7 +201,7 @@ runTests([
@@ -191,7 +201,7 @@ runTests([
assert.ok(!calls.some((c) => c.args && c.args.includes(u) && c.args[0] !== 'show'), `${u} is only ever read`);
assert.ok(!calls.some((c) => c.args && c.args.includes(u) && c.args[0] !== 'show'), `${u} is only ever read`);
}
}
assert.ok(host.alivePids.has(3100) && host.alivePids.has(3101));
assert.ok(host.alivePids.has(3100) && host.alivePids.has(3101));
- assert.strictEqual(host.read(PROD_DB), 'sqlite-production');
+ assert.ok(host.pgDatabases.has('ov_openre'), 'the production database is untouched');
assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/openre.jsonl')).includes(SECRET));
assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/openre.jsonl')).includes(SECRET));
}),
}),
]);
]);
new file mode 100644
new file mode 100644
@@ -0,0 +1,117 @@
@@ -0,0 +1,117 @@
+'use strict';
+/**
+ * The two-engine inventory (ADR-035): a `databases[]` entry is SQLite (the default, a file) or
+ * PostgreSQL (a database in the cluster, named — its object carries NO `path`). The same engine
+ * choice reaches the drill block ({ url, directUrl }), nginx tenants (read over psql, not a file)
+ * and the protected `postgresql-count` probe. These are the normalised shapes the rest of ovhost
+ * depends on, and every code path outside the engine's own code is written to assume no `.path`.
+ */
+const assert = require('assert');
+const { test, runTests } = require('./helpers');
+const { normalise } = require('../lib/inventory');
+
+const base = { owner: 'ubuntu', repo: '/opt/openvibe.trade' };
+const withDatabases = (databases, extra = {}) => ({ services: { trade: { ...base, databases, ...extra } } });
+const first = (doc) => normalise(doc).services.trade.databases[0];
+
+// A service whose single database is PostgreSQL and that drills it: the drill names the two env vars
+// the drill instance must be pointed at, instead of a file/env pair.
+function pgDrill(overrides = {}) {
+ return {
+ services: {
+ trade: {
+ ...base,
+ databases: [{ engine: 'postgresql', database: 'ov_trade' }],
+ drill: {
+ port: 5555,
+ databases: { trade: { url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' } },
+ env: { PORT: '{port}' },
+ ready: '/api/ready',
+ compare: ['/api/items'],
+ ...overrides,
+ },
+ },
+ },
+ };
+}
+
+runTests([
+ test('a PostgreSQL entry normalises to { name, engine, database, role } and carries no path', () => {
+ const db = first(withDatabases([{ engine: 'postgresql', database: 'ov_trade' }]));
+ assert.deepStrictEqual(db, { name: 'trade', engine: 'postgresql', database: 'ov_trade', role: null });
+ assert.ok(!('path' in db), 'a database in the cluster has no file, so no path key at all');
+ }),
+
+ test('name defaults to the database minus its ov_ prefix; an explicit name is kept', () => {
+ assert.strictEqual(first(withDatabases([{ engine: 'postgresql', database: 'ov_openre' }])).name, 'openre');
+ assert.deepStrictEqual(first(withDatabases([{ name: 'ingest', engine: 'postgresql', database: 'ov_openre' }])), { name: 'ingest', engine: 'postgresql', database: 'ov_openre', role: null });
+ }),
+
+ test('an optional role is accepted and validated', () => {
+ assert.strictEqual(first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', role: 'ov_trade_app' }])).role, 'ov_trade_app');
+ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', role: 'Trade' }])), /role must be a database role name/);
+ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', role: '1bad' }])), /role must be a database role name/);
+ }),
+
+ test('rejections name what is wrong with the entry', () => {
+ // A path on a PostgreSQL entry: the database lives in the cluster, not at a file.
+ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database: 'ov_trade', path: '/x/trade.db' }])), /has no path/);
+ // A database name that is not an owned ov_ database.
+ for (const database of ['trade', 'ov-Trade', 'public']) {
+ assert.throws(() => first(withDatabases([{ engine: 'postgresql', database }])), /database must match/, database);
+ }
+ // An engine that is neither.
+ assert.throws(() => first(withDatabases([{ engine: 'mysql', database: 'ov_trade' }])), /engine must be sqlite or postgresql/);
+ // A SQLite entry cannot name a database, and must have a path.
+ assert.throws(() => first(withDatabases([{ path: '/x/live.db', database: 'ov_live' }])), /a sqlite database is a path, not a database name/);
+ assert.throws(() => first(withDatabases([{ name: 'live' }])), /path must be an absolute path/);
+ }),
+
+ test('the SQLite default is unchanged', () => {
+ assert.deepStrictEqual(first(withDatabases([{ name: 'live', path: '/x/live.db' }])), { name: 'live', engine: 'sqlite', path: '/x/live.db' });
+ }),
+
+ test('drill.databases for a PostgreSQL database is { engine, url, directUrl }', () => {
+ const d = normalise(pgDrill()).services.trade.drill;
+ assert.deepStrictEqual(d.databases.trade, { engine: 'postgresql', url: 'DATABASE_URL', directUrl: 'DATABASE_DIRECT_URL' });
+ }),
+
+ test('drill.databases rejects the SQLite shape and bad env var names for a PostgreSQL database', () => {
+ const drill = (databases) => pgDrill({ databases });
+ assert.throws(() => normalise(drill({ trade: 'DATABASE_URL' })), /for a postgresql database/);
+ assert.throws(() => normalise(drill({ trade: { url: 'DATABASE_URL' } })), /directUrl must be an env var name/);
+ assert.throws(() => normalise(drill({ trade: { directUrl: 'DATABASE_DIRECT_URL' } })), /url must be an env var name/);
+ assert.throws(() => normalise(drill({ trade: { url: '1BAD', directUrl: 'OTHER' } })), /url must be an env var name/);
+ assert.throws(() => normalise(drill({ trade: { url: 'DATABASE_URL', directUrl: 'DATABASE_URL' } })), /url and directUrl must be different env vars/);
+ }),
+
+ test('countsTolerance is a whole number >= 0, default 0', () => {
+ assert.strictEqual(normalise(pgDrill()).services.trade.drill.countsTolerance, 0);
+ assert.strictEqual(normalise(pgDrill({ countsTolerance: 25 })).services.trade.drill.countsTolerance, 25);
+ assert.throws(() => normalise(pgDrill({ countsTolerance: -1 })), /countsTolerance must be a whole number of rows/);
+ assert.throws(() => normalise(pgDrill({ countsTolerance: 1.5 })), /countsTolerance must be a whole number of rows/);
+ }),
+
+ test('nginx.tenants.database accepts a path (SQLite) or a PostgreSQL database object', () => {
+ const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } }).services.host.nginx.tenants.database;
+ assert.deepStrictEqual(tenants('/var/lib/openvibe-host-api/host.db'), { engine: 'sqlite', path: '/var/lib/openvibe-host-api/host.db' });
+ assert.deepStrictEqual(tenants({ engine: 'postgresql', database: 'ov_host' }), { engine: 'postgresql', database: 'ov_host' });
+ }),
+
+ test('nginx.tenants.database rejects anything else', () => {
+ const tenants = (database) => normalise({ services: { host: { ...base, nginx: { tenants: { database } } } } });
+ for (const database of [{ engine: 'mysql', database: 'ov_host' }, { engine: 'postgresql', database: 'host' }, { database: 'ov_host' }, 42, []]) {
+ assert.throws(() => tenants(database), /nginx\.tenants\.database must be a path or/, JSON.stringify(database));
+ }
+ }),
+
+ test('the postgresql-count probe validates its database and SQL, alone and inside a sum', () => {
+ const probe = (protectedBlock) => normalise({ services: { openre: { ...base, protected: protectedBlock } } }).services.openre.protected;
+ const valid = { kind: 'postgresql-count', database: 'ov_openre', sql: 'SELECT count(*) AS n FROM ingest_sessions', label: 'ingest sessions' };
+ assert.deepStrictEqual(probe(valid), valid);
+ assert.throws(() => probe({ ...valid, database: 'openre' }), /database must match/);
+ assert.throws(() => probe({ ...valid, sql: 'DELETE FROM ingest_sessions' }), /sql must be a SELECT/);
+ // A sum may hold a postgresql-count part (nested kinds only rule out a sum inside a sum).
+ assert.doesNotThrow(() => probe({ kind: 'sum', probes: [valid] }));
+ }),
+]);
@@ -5,6 +5,7 @@
@@ -5,6 +5,7 @@
const assert = require('assert');
const assert = require('assert');
const { countProtected } = require('../lib/probes');
const { countProtected } = require('../lib/probes');
const { normalise } = require('../lib/inventory');
const { normalise } = require('../lib/inventory');
+const { createFakeHost } = require('./fake-host');
const counts = { '/opt/t/apps/img/data/jobs.db': 2, '/opt/t/apps/audio/data/jobs.db': 0, '/opt/t/apps/docs/data/jobs.db': 1 };
const counts = { '/opt/t/apps/img/data/jobs.db': 2, '/opt/t/apps/audio/data/jobs.db': 0, '/opt/t/apps/docs/data/jobs.db': 1 };
const exec = {
const exec = {
@@ -24,6 +25,26 @@ const svc = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { k
@@ -24,6 +25,26 @@ const svc = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { k
assert.strictEqual(r.count, null, 'one part that cannot answer makes the total unknown, never a smaller number');
assert.strictEqual(r.count, null, 'one part that cannot answer makes the total unknown, never a smaller number');
assert.match(r.unknown, /^part 2: no such database/);
assert.match(r.unknown, /^part 2: no such database/);
+ // A postgresql-count part asks the cluster over psql, as the postgres OS user (peer auth); the
+ // number it answers is the count, and a psql failure is unknown — never zero.
+ const host = createFakeHost();
+ host.addUnit('openvibe-tools.service');
+ host.sqliteHandler = () => [{ n: 2 }];
+ host.psqlHandler = (database, sql) => { assert.strictEqual(database, 'ov_tools'); assert.match(sql, /^SELECT/); return [{ n: 7 }]; };
+ const pgPart = { kind: 'postgresql-count', database: 'ov_tools', sql: 'SELECT count(*) AS n FROM tool_jobs', label: 'running tool jobs' };
+ assert.deepStrictEqual(await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart }), { count: 7, label: 'running tool jobs' });
+ assert.deepStrictEqual(host.psqlCalls.at(-1), { database: 'ov_tools', sql: pgPart.sql, as: 'postgres' });
+ const pgSum = { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: { kind: 'sum', label: 'running tool jobs', probes: [part('img'), pgPart] } };
+ assert.strictEqual((await countProtected(host.exec, pgSum)).count, 9, 'a postgresql-count part adds to the sum');
+ host.psqlHandler = () => { throw new Error('psql: could not connect to server'); };
+ const down = await countProtected(host.exec, { units: ['openvibe-tools.service'], runAs: 'ubuntu', protected: pgPart });
+ assert.strictEqual(down.count, null, 'a probe that cannot ask is unknown');
+ assert.notStrictEqual(down.count, 0, 'never read as zero');
+ assert.match(down.unknown, /could not connect/);
+ const downSum = await countProtected(host.exec, pgSum);
+ assert.strictEqual(downSum.count, null);
+ assert.match(downSum.unknown, /^part 2: psql: could not connect/);
+
// The inventory checks each part.
// The inventory checks each part.
const base = { owner: 'ubuntu', repo: '/opt/t', units: ['openvibe-tools.service'] };
const base = { owner: 'ubuntu', repo: '/opt/t', units: ['openvibe-tools.service'] };
assert.doesNotThrow(() => normalise({ services: { tools: { ...base, protected: svc.protected } } }));
assert.doesNotThrow(() => normalise({ services: { tools: { ...base, protected: svc.protected } } }));