openvibers/openvibe.host Commits
ovhost drill: restore drills (Wave 22)
Changes
@@@ -31,7 +31,7 @@ Stage A adds a few safety rules of its own:## Owns--- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, SQLite backup hooks.++- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, SQLite backup hooks, restore drills.- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.@@@ -56,11 +56,11 @@ Stage A adds a few safety rules of its own:| Stage | Scope | State ||---|---|---|--| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp SQLite/HTTP/git). Installed on the host; used read-only so far. |++| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup, restore drill. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp SQLite/HTTP/git). Installed on the host; used read-only so far. || **B: tenant static hosting** | The Host API service (port 4910): projects, sites, immutable content-addressed deploys, activation and rollback, `<site>.openvibe.host` and TXT-verified custom domains, quotas, upload logs, events, a server-rendered dashboard; tenant vhosts via `ovhost nginx tenants`. | **alpha**: written and tested (`npm test`, against a temp database, a mock Network and a DNS table). Not deployed. || **C: sandboxed user code** | Isolation profiles, CPU/memory/time/network/storage budgets, secret references, outbound policy, metering, kill/revoke without touching platform services. | **not started**, deliberately. Blocked until isolation and metering are proven. Nothing in Stage B runs tenant code. |--Not in Stage A yet (from the charter/roadmap list): container adapters, DNS adapters, certificate **renewal**, logs/metrics links, incident/maintenance controls, the release-manifest/active-client work in §15.18, `host.release.deployed|rolled_back` events (they need an Events outbox and a service principal), and the restore drills and cutover runbook that Wave 22 will add.++Not in Stage A yet (from the charter/roadmap list): container adapters, DNS adapters, certificate **renewal**, logs/metrics links, incident/maintenance controls, the release-manifest/active-client work in §15.18, `host.release.deployed|rolled_back` events (they need an Events outbox and a service principal), and the Wave 22 cutover runbook. Restore drills (`ovhost drill`) are written and tested against the fake host but have not been run on the host yet.## Acceptance (Stage A)@@@ -74,6 +74,7 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak- The socket unit is **never stopped or restarted**, in any flow.- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.++- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok`. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.Not demonstrated yet: any of this on the production host. The Wave 21 exit criterion ("deploy, restart and roll back one service without interrupting unrelated runtimes or protected sessions") still needs a run on the host, with evidence.@@@ -92,9 +93,10 @@ ovhost nginx render <service> [--variant http|sse|websocket] [--manifest <file>]ovhost nginx tenants <service> [--wildcard-cert <name|dir>] [--install] Stage B tenant vhostsovhost snapshot <service> [--out <file>]ovhost backup <service>++ovhost drill <service> [--backup <dir>] [--keep] restore drill (root only)```--Every command accepts `--json` and `--inventory <file>`. Exit codes: `0` ok · `1` usage/precondition (including a held lock) · `2` validation failed, nothing restarted · `3` not ready, rolled back and serving · `4` rollback failed, **manual intervention** · `5` protected sessions active (refused, or `--wait-idle` gave up).++Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passed, `1` refused (not root, port in use, no backup, unsupported), `2` failed. Other exit codes: `0` ok · `1` usage/precondition (including a held lock) · `2` validation failed, nothing restarted · `3` not ready, rolled back and serving · `4` rollback failed, **manual intervention** · `5` protected sessions active (refused, or `--wait-idle` gave up).### Deploy sequence@@@ -109,6 +111,25 @@ Every command accepts `--json` and `--inventory <file>`. Exit codes: `0` ok · `9. If readiness fails, run the automatic rollback (restore, reinstall, restart, poll).10. Append the release record to `<stateDir>/releases/<service>.jsonl`.++### Restore drills (Wave 22)++++`sudo ovhost drill <service>` restores the latest `ovhost backup` (or `--backup <dir>`) and starts a second instance on a spare loopback port. It compares that instance with production, stops it and logs the result to `<stateDir>/drills/<service>.jsonl`. It also prints a Markdown row for [docs/restore-drills.md](docs/restore-drills.md), which covers the steps, the sandbox and the per-service status.++++Each inventory entry's `drill` block declares:++++- `port`: the service port + 10000 in the example;++- `databases`: the env var that points the service at each restored copy;++- `env`: overrides that turn side effects off. Values may use `{tmp}`, `{port}` and `{db:<name>}`;++- `dirs` to create inside the drill directory;++- `ready`: the readiness path;++- `compare`: paths, plus volatile `ignore` keys where needed;++- `counts`: `{ db, table }` pairs;++- `supported: false` with a `reason` when a second instance cannot run without side effects.++++In `host.example.json`, 18 services have drill blocks. live, media, tools and games are marked unsupported, and the reasons are in the file.++++The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.++### Inventory`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange` and `nginx`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.@@@ -127,7 +148,7 @@ Nothing here has been done yet. These are the steps:```The tree must stay world-readable (default 0755/0644). The SQLite worker runs as `ubuntu` from this directory.2. **Inventory:** `sudo install -o root -g root -m 0640 host.example.json /etc/openvibe/host.json`, then correct it against the machine. Check the Community database path, the Tools unit list, and Events, which was not deployed as of 22 Sep.--3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks and snapshots. `/var/backups/openvibe/<service>` is created per service, owned by the service user, so the backup worker can write there. Both directories are created on first use.++3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots and drill logs. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe/<service>` is created per service, owned by the service user, so the backup worker can write there. Both directories are created on first use.4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm/SQLite (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.5. **First run:** `sudo ovhost status`, then `sudo ovhost validate <each service>`, then `sudo ovhost plan live`, all read-only apart from `git fetch`. Adopt `deploy` one service at a time, starting with one that has no protected sessions.@@@ -27,7 +27,7 @@"host.domain.verified"],"stages": {-- "A": "alpha: ovhost CLI installed on the production host 2026-09-23 (/usr/local/bin/ovhost, inventory /etc/openvibe/host.json); used read-only (status, validate) so far \u2014 no service deploy has moved to it",++ "A": "alpha: ovhost CLI installed on the production host 2026-09-23 (/usr/local/bin/ovhost, inventory /etc/openvibe/host.json); used read-only (status, validate) so far — no service deploy has moved to it; restore drills (`ovhost drill`, Wave 22) written and tested against the fake host, not yet run on the host","B": "alpha: tenant static hosting service written and tested (tenant isolation, traversal/symlink refusal, quotas, atomic rollback, TXT-verified custom domains, no secrets in responses); not deployed","C": "not started (deliberately: blocked until isolation and metering are proven)"},@@@ -19,5 +19,95 @@ The production instance is never touched.5. Stop the drill instance by its port's pid (not `pkill -f`, which can match the operator's own shell),delete the temp directory.--Open work: an `ovhost drill <service>` command that automates the steps above for every service with--declared databases and read-only comparison paths in the inventory.++Looking back at that manual run: `EVENTS_URL=` had no effect, because Community has no events relay. The++instance could also still reach production Network over loopback (JWKS, service tokens, identity++resolve-batch). The inventory's `community` drill block now also empties `OV_OAUTH_CLIENT_SECRET` and++points `OV_LIVE_INTERNAL_URL` and `OV_MEDIA_INTERNAL_URL` at a closed port.++++## Running a drill with `ovhost drill`++++```++sudo ovhost backup community # a fresh backup keeps the comparison meaningful++sudo ovhost drill community # restore, start, compare, stop, clean up, log++sudo ovhost drill community --keep # keep the drill directory (copy, drill.env, drill.log)++sudo ovhost drill community --backup /var/backups/openvibe/community/20260923-034341++```++++The command does the steps above automatically for any service whose inventory entry has a `drill`++block (see `host.example.json`):++++1. **Restore.** The command picks the latest `ovhost backup` from `<stateDir>/backups/<service>.jsonl`,++ or the `--backup` directory. It copies every database named in `drill.databases` into++ `/var/lib/openvibe-drills/<service>-<stamp>/db/`, a directory owned by the service user (mode 0700).++ The copy is made as the service user. `PRAGMA integrity_check` must return `ok` for each copy;++ otherwise the drill fails before anything starts.++2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the++ service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production++ unit's non-secret `Environment=` values and the production env file. After that it loads++ `drill.env`, which holds the drill port, the restored database paths and the switches that turn++ side effects off. `drill.env` holds inventory values only and is loaded last, so its values win.++ ovhost never reads the env file's values.++3. **Sandbox.** The instance runs with:++ - `ProtectSystem=strict`, with only the drill directory writable;++ - `PrivateTmp`, `ProtectHome=read-only` and `NoNewPrivileges`;++ - binding allowed only on its drill port (`SocketBindAllow`);++ - no addresses beyond loopback (`IPAddressDeny=any`, `IPAddressAllow=localhost`);++ - `RuntimeMaxSec`, which stops it even if ovhost dies.++++ Loopback stays open because services fetch Network's public keys there. Production services++ are therefore still reachable on loopback, and the inventory's overrides are what keep the drill++ away from them.++4. **Compare.** Once the instance answers its readiness path, every `drill.compare` path is fetched++ from production and from the drill instance. Bodies must match byte for byte, or as JSON with the++ listed volatile keys removed. Statuses must match, and production must answer 2xx. Each++ `drill.counts` table is counted in production and in the restored copy.++5. **Stop and record.** The instance's own MainPID gets SIGTERM, then SIGKILL after 20 s. ovhost++ never uses `pkill -f` and never signals a production pid. The drill directory is removed unless++ `--keep` is set, or unless the instance could not be confirmed stopped. The result is appended to++ `/var/lib/openvibe-host/drills/<service>.jsonl`, and a row for the table above is printed.++++Exit codes are `0` passed, `1` refused, and `2` failed. A refusal happens when ovhost is not run as++root, the port is in use, there is no backup, or the service's drill is unsupported. A failure is++integrity, readiness, a mismatch, or a stop that did not complete. Every failure is still logged and++cleaned up.++++A drill compares the backup with production *now*. If writes landed after the backup, row counts and++list endpoints differ, and the drill reports that honestly. Take the backup right before the drill.++++### Services++++Supported with declared overrides (not yet run on the host with `ovhost drill`):++++- network++- community++- events++- host (the Host API: `host.db` only, counts only)++- billing++- chat++- search++- sources++- wiki++- blog++- tips++- vip++- news++- reviews++- deals++- coupons++- trade++- codes++++Services that are not deployed yet (everything except network, live, media, tools, community and++games as of 22 Sep) have entries built from their repositories' `deploy/` directories.++++Marked `drill: { supported: false }`, with the reason in the inventory:++++- **live:** `data/analytics.db` is opened relative to the checkout at boot. It is read-only in the++ sandbox, and without the sandbox it would be production's file. RTMP always binds. Restreams, relays,++ AI jobs and Media reconcilers resume from rows in the copy.++- **media:** webhooks to Live take their URL from the `apps` table and have no env switch. The health++ job, junk sweep and clip jobs run ffmpeg against production files through absolute paths stored in++ the database.++- **tools:** eight units. yt, food and maps open `analytics.db` relative to the checkout. The job++ runtimes re-queue and prune with no switch.++- **games:** its runtime side effects have not been reviewed.@@@ -3,6 +3,8 @@"host": "openvibe-oregon","stateDir": "/var/lib/openvibe-host","backupDir": "/var/backups/openvibe",++ "drillDir": "/var/lib/openvibe-drills",++ "_drill": "Restore drills (ovhost drill <service>): each service with a database has a drill block. The drill port is the service port + 10000 on 127.0.0.1. http://127.0.0.1:9 is a closed port, used where an empty value would fall back to a production service on loopback. An empty OV_OAUTH_CLIENT_SECRET stops a service from minting service tokens, so it cannot call other services with its identity. Loopback stays reachable from the drill sandbox (services need Network's public keys), so these overrides are what keep a drill away from production. _notes explain what is left on and why.","nginx": {"sitesAvailable": "/etc/nginx/sites-available","sitesEnabled": "/etc/nginx/sites-enabled",@@@ -25,7 +27,24 @@"ready": { "url": "http://127.0.0.1:4000/api/health", "timeoutSeconds": 30 },"noRestartPaths": ["docs/", "test/", "**/*.md"],"databases": [{ "name": "network", "path": "/opt/openvibe.network/data/network.db" }],-- "nginx": { "vhost": "openvibe.network.conf", "repoVhost": "deploy/nginx/openvibe.network.conf", "variant": "http" }++ "nginx": { "vhost": "openvibe.network.conf", "repoVhost": "deploy/nginx/openvibe.network.conf", "variant": "http" },++ "drill": {++ "port": 14000,++ "ready": "/api/ready",++ "databases": { "network": "DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "OV_EVENTS_INTERNAL_URL": "", "INTERNAL_API_KEY": "", "OV_TOOLS_INTERNAL_URL": "http://127.0.0.1:9", "AVATAR_PATH": "{tmp}/avatars" },++ "dirs": ["{tmp}/avatars"],++ "compare": [++ "/api/health",++ "/api/.well-known/jwks",++ "/api/brand",++ "/api/themes?limit=500",++ "/api/domains",++ "/api/v1/registry/capabilities"++ ],++ "counts": [{ "db": "network", "table": "users" }, { "db": "network", "table": "oauth_clients" }, { "db": "network", "table": "wallets" }, { "db": "network", "table": "coin_transactions" }],++ "_notes": "OV_EVENTS_INTERNAL_URL empty stops the dev_audit relay; INTERNAL_API_KEY empty falls back to the placeholder key, which makes the chrome site-copy job (paid LLM call through Live), avatar import and Live analytics calls return early. No switch: the ecosystem poller GETs every service readiness and release.json on loopback every 60 s (read-only). Discord bot login and the Resend email queue are gated by settings in the copy and reach only remote hosts, which the sandbox blocks. JWT_PRIVATE_KEY/JWT_PUBLIC_KEY come from the production env file and are only read."++ }},"live": {"repo": "/opt/openvibe.live",@@@ -55,7 +74,11 @@],"_databasesNote": "analytics.db (1.7 GB, append-only pageviews) is left out of routine backups on purpose; add it here to include it.","backupOnChange": ["server/db/migrations.js", "server/db/schema.sql"],-- "nginx": { "vhost": "openvibe.live.conf", "repoVhost": "deploy/nginx/openvibe.live.conf", "variant": "websocket", "wsPaths": ["/ws/"], "maxBody": "3m" }++ "nginx": { "vhost": "openvibe.live.conf", "repoVhost": "deploy/nginx/openvibe.live.conf", "variant": "websocket", "wsPaths": ["/ws/"], "maxBody": "3m" },++ "drill": {++ "supported": false,++ "reason": "server/index.js opens data/analytics.db relative to the checkout (not movable by env) at module load: in the sandbox the checkout is read-only and boot fails, and without the sandbox the drill would write production's analytics database. It also always binds RTMP (RTMP_PORT and RTMP_PORT+8000), resumes restreams/RobotStreamer relays and AI jobs for rows in the copy, and runs Media reconcilers; several have no env switch. Needs a BindPaths over data/ plus DB-level switches before it can be drilled."++ }},"media": {"repo": "/opt/openvibe.media",@@@ -74,7 +97,11 @@},"drain": { "policy": "refuse", "waitMaxSeconds": 43200, "pollSeconds": 60, "quietChecks": 2 },"databases": [{ "name": "media", "path": "/opt/openvibe.media/data/media.db" }],-- "nginx": { "vhost": "openvibe.media.conf", "repoVhost": "deploy/nginx/openvibe.media.conf", "variant": "http" }++ "nginx": { "vhost": "openvibe.media.conf", "repoVhost": "deploy/nginx/openvibe.media.conf", "variant": "http" },++ "drill": {++ "supported": false,++ "reason": "Webhooks to Live (orphan-recording finalize at boot, clip re-cuts, storage alerts) read their URL from the apps table and have no env switch; the health job, junk sweep and clip jobs run ffmpeg against production files through absolute paths stored in the database; the tiering sweep and health job start on a timer with no switch. A side-effect-free instance needs the copy edited (webhook_url NULL, storage_tier off) first, which ovhost does not do."++ }},"tools": {"repo": "/opt/openvibe.tools",@@@ -102,7 +129,11 @@{ "name": "tools-maps-analytics", "path": "/opt/openvibe.tools/apps/maps/data/analytics.db" },{ "name": "tools-text-analytics", "path": "/opt/openvibe.tools/apps/text/data/analytics.db" },{ "name": "tools-yt-analytics", "path": "/opt/openvibe.tools/apps/yt/data/analytics.db" }-- ]++ ],++ "drill": {++ "supported": false,++ "reason": "Eight units (gateway plus seven apps) run as separate processes and a drill starts one; yt, food and maps open apps/<app>/data/analytics.db relative to the checkout with no env override (read-only in the sandbox, production's file without it); the job runtimes prune and re-queue jobs at start with no switch. Its only databases are per-app analytics."++ }},"community": {"repo": "/opt/openvibe.community",@@@ -118,7 +149,19 @@{ "name": "community-legacy-path", "path": "/opt/openvibe.community/data/community.db" }],"_databasesNote": "The repo unit puts the database in /var/lib/openvibe-community; the 22 Sep production snapshot found it under /opt/openvibe.community/data. Keep the one that exists (backup skips a missing path).",-- "nginx": { "vhost": "openvibe.community.conf", "repoVhost": "deploy/nginx/openvibe.community.conf", "variant": "http" }++ "nginx": { "vhost": "openvibe.community.conf", "repoVhost": "deploy/nginx/openvibe.community.conf", "variant": "http" },++ "drill": {++ "port": 14200,++ "databases": { "community": "COMMUNITY_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "DISCORD_RELAY_ENABLED": "0", "OV_OAUTH_CLIENT_SECRET": "", "OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9", "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9" },++ "compare": [++ "/api/pastes?limit=5",++ "/api/v1/spaces",++ "/api/v1/pulse?limit=5"++ ],++ "counts": [{ "db": "community", "table": "pastes" }, { "db": "community", "table": "spaces" }, { "db": "community", "table": "threads" }, { "db": "community", "table": "posts" }],++ "_notes": "If the host keeps the database at the legacy path, map \"community-legacy-path\" instead of \"community\" (and in counts). An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."++ }},"events": {"_note": "Not deployed as of the 22 Sep 2026 production snapshot (no events.env on the host).",@@@ -138,7 +181,14 @@},"drain": { "policy": "report" },"databases": [{ "name": "events", "path": "/var/lib/openvibe-events/events.db" }],-- "nginx": { "vhost": "events.openvibe.network.conf", "repoVhost": "deploy/nginx/events.openvibe.network.conf", "variant": "sse", "ssePaths": ["/realtime/stream"], "certName": "openvibe.network" }++ "nginx": { "vhost": "events.openvibe.network.conf", "repoVhost": "deploy/nginx/events.openvibe.network.conf", "variant": "sse", "ssePaths": ["/realtime/stream"], "certName": "openvibe.network" },++ "drill": {++ "port": 14300,++ "databases": { "events": "EVENTS_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_WORKER": "off" },++ "counts": [{ "db": "events", "table": "events" }, { "db": "events", "table": "subscriptions" }, { "db": "events", "table": "deliveries" }, { "db": "events", "table": "consumer_checkpoints" }],++ "_notes": "EVENTS_WORKER=off stops signed deliveries to subscribers. /api/ready requires Network JWKS (read-only GET on loopback). The retention prune interval always runs but only touches the copy. No public read endpoint is stable enough to compare; row counts only."++ }},"games": {"repo": "/opt/openvibe.games",@@@ -148,7 +198,11 @@"envFile": "/etc/openvibe/games.env","port": 8000,"ready": { "url": "http://127.0.0.1:8000/healthz", "timeoutSeconds": 60 },-- "databases": [{ "name": "world", "path": "/opt/openvibe.games/data/world.db" }]++ "databases": [{ "name": "world", "path": "/opt/openvibe.games/data/world.db" }],++ "drill": {++ "supported": false,++ "reason": "Not reviewed: a pnpm workspace with a TypeScript build whose runtime side effects (world simulation loop, realtime ports) have not been read, so no side-effect-free overrides can be declared without guessing."++ }},"sites": {"repo": "/opt/openvibe.sites",@@@ -168,7 +222,317 @@"ready": { "url": "http://127.0.0.1:4910/api/ready", "timeoutSeconds": 30 },"noRestartPaths": ["docs/", "test/", "**/*.md"],"databases": [{ "name": "host", "path": "/var/lib/openvibe-host-api/host.db" }],-- "nginx": { "tenants": { "sitesDomain": "openvibe.host", "dashboardDomain": "openvibe.host", "wildcardCert": "openvibe.host", "database": "/var/lib/openvibe-host-api/host.db", "maxUpload": "110m" } }++ "nginx": { "tenants": { "sitesDomain": "openvibe.host", "dashboardDomain": "openvibe.host", "wildcardCert": "openvibe.host", "database": "/var/lib/openvibe-host-api/host.db", "maxUpload": "110m" } },++ "drill": {++ "port": 14910,++ "databases": { "host": "HOST_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "HOST_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "HOST_STORAGE_DIR": "{tmp}/objects" },++ "dirs": ["{tmp}/objects"],++ "counts": [{ "db": "host", "table": "host_projects" }, { "db": "host", "table": "host_sites" }, { "db": "host", "table": "host_deploys" }, { "db": "host", "table": "host_domains" }],++ "_notes": "Restores host.db only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."++ }++ },++ "billing": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.billing",++ "units": ["openvibe-billing.service"],++ "unitSources": { "openvibe-billing.service": "deploy/systemd/openvibe-billing.service" },++ "envFile": "/etc/openvibe/billing.env",++ "env": { "required": "from-example" },++ "port": 4600,++ "ready": { "url": "http://127.0.0.1:4600/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "billing", "path": "/var/lib/openvibe-billing/billing.db" }],++ "drill": {++ "port": 14600,++ "databases": { "billing": "BILLING_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "BILLING_JOBS": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "" },++ "compare": ["/api/v1/rates"],++ "counts": [{ "db": "billing", "table": "accounts" }, { "db": "billing", "table": "transactions" }, { "db": "billing", "table": "ledger_entries" }, { "db": "billing", "table": "subscriptions" }, { "db": "billing", "table": "payment_intents" }],++ "_notes": "BILLING_JOBS=off stops provider-event settlement, the renewal sweep and the outbox relay. Payment providers are called only on incoming requests (and are remote, which the sandbox blocks). /api/health is not compared: it reports the relay state, which the drill turns off."++ }++ },++ "chat": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.chat",++ "units": ["openvibe-chat.service"],++ "unitSources": { "openvibe-chat.service": "deploy/systemd/openvibe-chat.service" },++ "envFile": "/etc/openvibe/chat.env",++ "env": { "required": "from-example" },++ "port": 4400,++ "ready": { "url": "http://127.0.0.1:4400/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "chat", "path": "/var/lib/openvibe-chat/chat.db" }],++ "drill": {++ "port": 14400,++ "databases": { "chat": "CHAT_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "LIVE_MIRROR": "0", "OV_OAUTH_CLIENT_SECRET": "", "OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9", "CHAT_CACHE_DIR": "{tmp}/cache", "SOUNDS_PATH": "{tmp}/sounds" },++ "dirs": ["{tmp}/cache", "{tmp}/sounds"],++ "compare": ["/api/chat/filters/friendly"],++ "counts": [{ "db": "chat", "table": "dm_conversations" }, { "db": "chat", "table": "dm_messages" }, { "db": "chat", "table": "emotes" }, { "db": "chat", "table": "moderation_actions" }],++ "_notes": "An empty client secret and a dead Live URL stop the 2 s Live projection sync and the effects POSTed to Live. History endpoints are not compared: cosmetics come from Live (cut off here) and expired messages are swept at boot, so chat_messages is not counted either."++ }++ },++ "search": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.search",++ "units": ["openvibe-search.service"],++ "unitSources": { "openvibe-search.service": "deploy/systemd/openvibe-search.service" },++ "envFile": "/etc/openvibe/search.env",++ "env": { "required": "from-example" },++ "port": 4710,++ "ready": { "url": "http://127.0.0.1:4710/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "search", "path": "/var/lib/openvibe-search/search.db" }],++ "drill": {++ "port": 14710,++ "databases": { "search": "SEARCH_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "SEARCH_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "" },++ "compare": [++ "/api/v1/search?q=openvibe&limit=10",++ "/api/v1/suggest?q=open&limit=10"++ ],++ "counts": [{ "db": "search", "table": "documents" }, { "db": "search", "table": "doc_acl" }, { "db": "search", "table": "doc_facets" }],++ "_notes": "/api/ready needs the Network public key (read-only JWKS GET on loopback). No switch: the 6-hourly outbox prune (copy only)."++ }++ },++ "sources": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.sources",++ "units": ["openvibe-sources.service"],++ "unitSources": { "openvibe-sources.service": "deploy/systemd/openvibe-sources.service" },++ "envFile": "/etc/openvibe/sources.env",++ "env": { "required": "from-example" },++ "port": 4720,++ "ready": { "url": "http://127.0.0.1:4720/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "sources", "path": "/var/lib/openvibe-sources/sources.db" }],++ "drill": {++ "port": 14720,++ "databases": { "sources": "SOURCES_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "SOURCES_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "" },++ "counts": [{ "db": "sources", "table": "sources" }, { "db": "sources", "table": "items" }, { "db": "sources", "table": "item_revisions" }, { "db": "sources", "table": "fetch_runs" }],++ "_notes": "SOURCES_WORKER=off stops the crawler (feeds, sitemaps, robots.txt; production may allow loopback sources). Every /api/v1 read needs a service token, so row counts only."++ }++ },++ "wiki": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.wiki",++ "units": ["openvibe-wiki.service"],++ "unitSources": { "openvibe-wiki.service": "deploy/systemd/openvibe-wiki.service" },++ "envFile": "/etc/openvibe/wiki.env",++ "env": { "required": "from-example" },++ "port": 4800,++ "ready": { "url": "http://127.0.0.1:4800/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "wiki", "path": "/var/lib/openvibe-wiki/wiki.db" }],++ "drill": {++ "port": 14800,++ "databases": { "wiki": "WIKI_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_MEDIA_INTERNAL_URL": "", "OV_SOURCES_INTERNAL_URL": "", "WIKI_SCHEDULE_INTERVAL_MS": "2147483647" },++ "compare": [++ "/api/v1/spaces",++ "/feed.json"++ ],++ "counts": [{ "db": "wiki", "table": "wiki_spaces" }, { "db": "wiki", "table": "wiki_pages" }, { "db": "wiki", "table": "wiki_page_revisions" }],++ "_notes": "The schedule worker has no off switch; the interval is set to the largest value Node accepts (anything larger becomes 1 ms). An empty client secret also stops Community thread creation on page views."++ }++ },++ "blog": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.blog",++ "units": ["openvibe-blog.service"],++ "unitSources": { "openvibe-blog.service": "deploy/systemd/openvibe-blog.service" },++ "envFile": "/etc/openvibe/blog.env",++ "env": { "required": "from-example" },++ "port": 4810,++ "ready": { "url": "http://127.0.0.1:4810/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "blog", "path": "/var/lib/openvibe-blog/blog.db" }],++ "drill": {++ "port": 14810,++ "databases": { "blog": "BLOG_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "BLOG_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9", "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9" },++ "compare": [++ "/api/v1/blogs/openvibe/posts?limit=10",++ "/feed.json"++ ],++ "counts": [{ "db": "blog", "table": "blogs" }, { "db": "blog", "table": "blog_posts" }, { "db": "blog", "table": "blog_post_revisions" }],++ "_notes": "An empty client secret stops Community thread creation and Network identity lookups on page reads; author names then come from the subject_projections cache in the copy."++ }++ },++ "tips": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.tips",++ "units": ["openvibe-tips.service"],++ "unitSources": { "openvibe-tips.service": "deploy/systemd/openvibe-tips.service" },++ "envFile": "/etc/openvibe/tips.env",++ "env": { "required": "from-example" },++ "port": 4610,++ "ready": { "url": "http://127.0.0.1:4610/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "tips", "path": "/var/lib/openvibe-tips/tips.db" }],++ "drill": {++ "port": 14610,++ "databases": { "tips": "TIPS_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "TIPS_JOBS": "off", "EVENTS_URL": "", "TIPS_EVENTS_SECRET": "", "TIPS_CHAT_ADAPTER": "none", "OV_OAUTH_CLIENT_SECRET": "", "BILLING_URL": "http://127.0.0.1:9", "LIVE_INTERNAL_URL": "http://127.0.0.1:9" },++ "compare": ["/sitemap.xml"],++ "counts": [{ "db": "tips", "table": "creator_tip_profiles" }, { "db": "tips", "table": "tip_interactions" }, { "db": "tips", "table": "tip_goals" }],++ "_notes": "Money: TIPS_JOBS=off stops due Billing transfers and chat deliveries, BILLING_URL points at a closed port (an empty value falls back to production Billing), TIPS_EVENTS_SECRET empty refuses funding webhooks. /api/ready needs the Network public key (read-only JWKS GET on loopback)."++ }++ },++ "vip": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.vip",++ "units": ["openvibe-vip.service"],++ "unitSources": { "openvibe-vip.service": "deploy/systemd/openvibe-vip.service" },++ "envFile": "/etc/openvibe/vip.env",++ "env": { "required": "from-example" },++ "port": 4620,++ "ready": { "url": "http://127.0.0.1:4620/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "vip", "path": "/var/lib/openvibe-vip/vip.db" }],++ "drill": {++ "port": 14620,++ "databases": { "vip": "VIP_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "VIP_JOBS": "off", "EVENTS_URL": "", "VIP_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "BILLING_URL": "http://127.0.0.1:9", "VIP_CHECKOUT_PROVIDERS": "" },++ "compare": [++ "/api/v1/creators/network",++ "/api/v1/plans?creator=network",++ "/api/v1/perks?creator=network"++ ],++ "counts": [{ "db": "vip", "table": "vip_creators" }, { "db": "vip", "table": "vip_plans" }, { "db": "vip", "table": "vip_memberships" }],++ "_notes": "VIP_FORM_SECRET must stay as production sets it (the service exits without it). BILLING_URL points at a closed port: the Billing client has no off switch and an empty value falls back to production Billing."++ }++ },++ "news": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.news",++ "units": ["openvibe-news.service"],++ "unitSources": { "openvibe-news.service": "deploy/systemd/openvibe-news.service" },++ "envFile": "/etc/openvibe/news.env",++ "env": { "required": "from-example" },++ "port": 4820,++ "ready": { "url": "http://127.0.0.1:4820/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "news", "path": "/var/lib/openvibe-news/news.db" }],++ "drill": {++ "port": 14820,++ "databases": { "news": "NEWS_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "NEWS_WORKER": "off", "EVENTS_URL": "", "NEWS_EVENTS_SECRET": "", "OV_AI_INTERNAL_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9" },++ "compare": [++ "/api/v1/stories?limit=10",++ "/api/v1/topics"++ ],++ "counts": [{ "db": "news", "table": "news_topics" }, { "db": "news", "table": "news_stories" }, { "db": "news", "table": "news_source_items" }]++ }++ },++ "reviews": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.reviews",++ "units": ["openvibe-reviews.service"],++ "unitSources": { "openvibe-reviews.service": "deploy/systemd/openvibe-reviews.service" },++ "envFile": "/etc/openvibe/reviews.env",++ "env": { "required": "from-example" },++ "port": 4830,++ "ready": { "url": "http://127.0.0.1:4830/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "reviews", "path": "/var/lib/openvibe-reviews/reviews.db" }],++ "drill": {++ "port": 14830,++ "databases": { "reviews": "REVIEWS_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "REVIEWS_SOURCES_SYNC": "false", "EVENTS_URL": "", "REVIEWS_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "" },++ "compare": [++ "/api/v1/entities?limit=10",++ "/feed.json"++ ],++ "counts": [{ "db": "reviews", "table": "review_entities" }, { "db": "reviews", "table": "review_signals" }, { "db": "reviews", "table": "review_summaries" }],++ "_notes": "reconcileIndex() at boot may write index revisions and outbox rows into the copy (no switch; copy only)."++ }++ },++ "deals": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.deals",++ "units": ["openvibe-deals.service"],++ "unitSources": { "openvibe-deals.service": "deploy/systemd/openvibe-deals.service" },++ "envFile": "/etc/openvibe/deals.env",++ "env": { "required": "from-example" },++ "port": 4840,++ "ready": { "url": "http://127.0.0.1:4840/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "deals", "path": "/var/lib/openvibe-deals/deals.db" }],++ "drill": {++ "port": 14840,++ "databases": { "deals": "DEALS_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "DEALS_WORKER": "off", "DEALS_IMPORT": "off", "EVENTS_URL": "", "DEALS_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9", "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9" },++ "compare": [{ "path": "/api/v1/offers?sort=new&page=1", "ignore": ["age_ms"] }],++ "counts": [{ "db": "deals", "table": "deal_offers" }, { "db": "deals", "table": "deal_products" }, { "db": "deals", "table": "deal_votes" }],++ "_notes": "The client secret MUST be empty: with it, an anonymous GET /d/:slug can create a thread in production Community. Offers carry freshness.age_ms (now - observed_at), ignored in the comparison."++ }++ },++ "coupons": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.coupons",++ "units": ["openvibe-coupons.service"],++ "unitSources": { "openvibe-coupons.service": "deploy/systemd/openvibe-coupons.service" },++ "envFile": "/etc/openvibe/coupons.env",++ "env": { "required": "from-example" },++ "port": 4850,++ "ready": { "url": "http://127.0.0.1:4850/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "coupons", "path": "/var/lib/openvibe-coupons/coupons.db" }],++ "drill": {++ "port": 14850,++ "databases": { "coupons": "COUPONS_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "COUPONS_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "" },++ "compare": [++ "/sitemaps/merchants.xml",++ "/feed.json"++ ],++ "counts": [{ "db": "coupons", "table": "coupons" }, { "db": "coupons", "table": "coupon_merchants" }, { "db": "coupons", "table": "coupon_merchant_domains" }]++ }++ },++ "trade": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.trade",++ "units": ["openvibe-trade.service"],++ "unitSources": { "openvibe-trade.service": "deploy/systemd/openvibe-trade.service" },++ "envFile": "/etc/openvibe/trade.env",++ "env": { "required": "from-example" },++ "port": 4860,++ "ready": { "url": "http://127.0.0.1:4860/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "trade", "path": "/var/lib/openvibe-trade/trade.db" }],++ "drill": {++ "port": 14860,++ "databases": { "trade": "TRADE_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "TRADE_WORKER": "off", "EVENTS_URL": "", "TRADE_EVENTS_WEBHOOK_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9" },++ "compare": [++ "/api/v1/instruments?limit=10",++ "/feed.json"++ ],++ "counts": [{ "db": "trade", "table": "trade_instruments" }, { "db": "trade", "table": "trade_market_observations" }, { "db": "trade", "table": "trade_source_documents" }]++ }++ },++ "codes": {++ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",++ "repo": "/opt/openvibe.codes",++ "units": ["openvibe-codes.service"],++ "unitSources": { "openvibe-codes.service": "deploy/systemd/openvibe-codes.service" },++ "envFile": "/etc/openvibe/codes.env",++ "env": { "required": "from-example" },++ "port": 4900,++ "ready": { "url": "http://127.0.0.1:4900/api/ready", "timeoutSeconds": 30 },++ "noRestartPaths": ["docs/", "test/", "**/*.md"],++ "databases": [{ "name": "codes", "path": "/var/lib/openvibe-codes/codes.db" }],++ "drill": {++ "port": 14900,++ "databases": { "codes": "CODES_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "" },++ "compare": ["/sitemap.xml"],++ "counts": [{ "db": "codes", "table": "manifests" }, { "db": "codes", "table": "releases" }, { "db": "codes", "table": "trust" }],++ "_notes": "Left on (read-only): JWKS and registry GETs to Network on loopback. The docs pages carry their boot time, so they are not compared."++ }}}}@@@ -30,13 +30,18 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]database); lists domains still waiting for a certificatesnapshot <service> [--out <file>] config tarball: units, vhost, env NAMES, sha — no secret valuesbackup <service> sqlite .backup of the declared databases to a dated directory++ drill <service> [--backup <dir>] [--keep]++ restore drill (root): restore the latest backup into a temp++ dir, start a sandboxed second instance on the drill port, compare++ it with production, stop it, log the resultInventory: --inventory, else $OVHOST_INVENTORY, else /etc/openvibe/host.json.Exit codes: 0 ok · 1 usage/precondition · 2 validation failed (nothing restarted) · 3 not ready,--rolled back · 4 rollback failed (manual intervention) · 5 protected sessions active++rolled back · 4 rollback failed (manual intervention) · 5 protected sessions active ·++drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported) · 2 drill failed`;--const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert']);++const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup']);function parseArgs(argv) {const pos = [];@@@ -199,6 +204,16 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procif (json) emit(r); else for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);return 0;}++ case 'drill': {++ const r = await lib.drill(ctx, needService(), { backup: opts.backup, keep: !!opts.keep });++ if (json) emit(r.record);++ else {++ out(`[ovhost] drill ${r.record.result}${r.record.failure ? ` at ${r.record.failure.stage}: ${r.record.failure.message}` : ''}`);++ out('Row for docs/restore-drills.md:');++ out(r.record.markdown);++ }++ return r.exitCode;++ }default:throw new UsageError(`unknown command "${cmd}"`);}@@@ -206,6 +221,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procif (err instanceof UsageError) { out(err.message); return 1; }if (err instanceof InventoryError || err instanceof LockedError) { out(`[ovhost] ${err.message}`); return 1; }if (err instanceof SocketGuardError) { out(`[ovhost] ${err.message}`); return 1; }++ if (err instanceof lib.DrillError) { out(`[ovhost] ${err.message}`); return err.exitCode; }if (err instanceof lib.OpError) { out(`[ovhost] ✗ ${err.message}`); return err.exitCode; }out(`[ovhost] ✗ ${err.message}`);return 2;nnew file mode 100644@@@ -0,0 +1,416 @@++'use strict';++/**++ * ovhost drill <service> [--backup <dir>] [--keep]: a restore drill (roadmap Wave 22). It rebuilds++ * the service from a backup next to production and checks that the rebuilt copy answers like++ * production does:++ *++ * 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Copy each database the++ * drill block names into a fresh directory under <drillDir> owned by the service user, and++ * run PRAGMA integrity_check on each copy. The result must be `ok`.++ * 2. Start a second instance from the production checkout, as the service user, through++ * systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production++ * env file, and a second env file with the drill's overrides: its port, the restored database++ * paths, and the switches that turn side effects off. The instance runs sandboxed:++ * ProtectSystem=strict with only the drill directory writable, a private /tmp, binding only its++ * drill port, and no addresses beyond loopback unless the inventory says otherwise.++ * RuntimeMaxSec stops it even if ovhost dies. Loopback stays open (the instance needs Network's++ * public keys), so the inventory's overrides are what keep it away from production services.++ * 3. Wait for its readiness path. Then compare the declared read-only GETs with production, byte++ * for byte or as JSON without the listed volatile keys, and compare declared row counts++ * (production against the restored copy).++ * 4. Stop the instance by its own MainPID (never pkill -f), remove the drill directory unless++ * --keep, append the result to <stateDir>/drills/<service>.jsonl, and return a Markdown row++ * for docs/restore-drills.md.++ *++ * ovhost itself writes only inside the drill directory, the lock file and the drill log. It never++ * writes to the checkout, the env file, the database or the backup. It never reads an env value:++ * systemd loads the production env file for the instance, and the override file holds only++ * inventory values. Environment= values from the unit are passed on only if their names do not look++ * secret.++ */++const path = require('path');++const { service } = require('../inventory');++const envfile = require('../envfile');++const systemd = require('../systemd');++const lock = require('../lock');++const { stamp } = require('./backup');++++class DrillError extends Error {++ constructor(message, exitCode = 1) {++ super(message);++ this.exitCode = exitCode;++ }++}++++/** Raised inside the run to jump to cleanup once record.failure is set. */++class Abort extends Error {}++++const STOP_GRACE_MS = 20000;++++// ── helpers ──────────────────────────────────────────────────────────────────++++function expand(value, vars) {++ return value.replace(/\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g, (_, k) => {++ if (k === 'tmp') return vars.tmp;++ if (k === 'port') return String(vars.port);++ return vars.db[k.slice(3)];++ });++}++++/** A line for a systemd EnvironmentFile. Values are quoted only when they need it. */++function envLine(name, value) {++ if (value === '') return `${name}=`;++ if (/^[A-Za-z0-9_./:,@%+=-]+$/.test(value)) return `${name}=${value}`;++ return `${name}="${value.replace(/["\\$`]/g, '\\$&')}"`;++}++++function environmentProperty(name, value) {++ const a = `${name}=${value}`;++ return /[\s"'\\]/.test(a) ? `Environment="${a.replace(/["\\]/g, '\\$&')}"` : `Environment=${a}`;++}++++function stripKeys(v, keys) {++ if (Array.isArray(v)) return v.map((x) => stripKeys(x, keys));++ if (v && typeof v === 'object') {++ const out = {};++ for (const [k, x] of Object.entries(v)) if (!keys.includes(k)) out[k] = stripKeys(x, keys);++ return out;++ }++ return v;++}++++function diffPaths(a, b, p = '$', out = []) {++ if (out.length >= 5) return out;++ if (Array.isArray(a) && Array.isArray(b)) {++ if (a.length !== b.length) { out.push(`${p}.length ${b.length} ≠ ${a.length}`); return out; }++ a.forEach((x, i) => diffPaths(x, b[i], `${p}[${i}]`, out));++ return out;++ }++ if (a && b && typeof a === 'object' && typeof b === 'object' && !Array.isArray(a) && !Array.isArray(b)) {++ for (const k of new Set([...Object.keys(a), ...Object.keys(b)])) {++ if (!(k in a)) out.push(`${p}.${k} only in the drill instance`);++ else if (!(k in b)) out.push(`${p}.${k} missing in the drill instance`);++ else diffPaths(a[k], b[k], `${p}.${k}`, out);++ if (out.length >= 5) break;++ }++ return out;++ }++ if (JSON.stringify(a) !== JSON.stringify(b)) out.push(p);++ return out;++}++++/** Compare one path. `prod` and `drill` are exec.http results. */++function compareBodies(entry, prod, drill) {++ const res = { path: entry.path, productionStatus: prod.status || 0, drillStatus: drill.status || 0, match: false };++ if (!(prod.status >= 200 && prod.status < 300)) { res.detail = `production answered ${prod.status || prod.error || 'nothing'}`; return res; }++ if (prod.status !== drill.status) { res.detail = `status ${drill.status || drill.error || 'no answer'} ≠ production ${prod.status}`; return res; }++ if (!entry.ignore.length) {++ if (prod.body === drill.body) { res.match = true; res.mode = 'bytes'; return res; }++ const a = Buffer.from(prod.body || '');++ const b = Buffer.from(drill.body || '');++ let i = 0;++ while (i < a.length && i < b.length && a[i] === b[i]) i++;++ res.detail = `bodies differ: ${b.length} bytes vs production ${a.length}, first difference at byte ${i}`;++ return res;++ }++ let pa;++ let pb;++ try { pa = JSON.parse(prod.body); pb = JSON.parse(drill.body); } catch { res.detail = 'ignore keys are declared but a body is not JSON'; return res; }++ const diffs = diffPaths(stripKeys(pa, entry.ignore), stripKeys(pb, entry.ignore));++ res.mode = `json without ${entry.ignore.join(', ')}`;++ if (!diffs.length) { res.match = true; return res; }++ res.detail = `JSON differs at ${diffs.join('; ')}`;++ return res;++}++++function countSql(table) { return `SELECT count(*) AS n FROM "${table}"`; }++++async function countRows(exec, db, table, as) {++ const rows = await exec.sqlite(db, countSql(table), { as });++ const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);++ if (!Number.isFinite(n)) throw new Error(`count of ${table} returned no number`);++ return n;++}++++// ── inputs ───────────────────────────────────────────────────────────────────++++/** The backup to restore: --backup <dir>, else the last record in <stateDir>/backups/<id>.jsonl. */++async function resolveBackup(exec, inv, svc, dirArg) {++ let dir = dirArg;++ let takenAt = null;++ if (dir) {++ if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');++ dir = path.normalize(dir);++ } else {++ const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${svc.id}.jsonl`));++ const records = String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir);++ if (!records.length) throw new DrillError(`no ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);++ const last = records[records.length - 1];++ dir = last.dir;++ takenAt = last.at || null;++ }++ const st = await exec.stat(dir);++ if (!st || !st.isDir) throw new DrillError(`backup directory ${dir} not found`);++ const files = [];++ for (const name of Object.keys(svc.drill.databases)) {++ const db = svc.databases.find((d) => d.name === name);++ const file = path.join(dir, `${name}${path.extname(db.path) || '.db'}`);++ const fst = await exec.stat(file);++ if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);++ files.push({ name, file, production: db.path, envName: svc.drill.databases[name], ext: path.extname(db.path) || '.db' });++ }++ return { dir, takenAt, files };++}++++/** How production starts the service: ExecStart, WorkingDirectory and non-secret Environment=. */++async function startSpec(exec, svc) {++ const d = svc.drill;++ if (!svc.units.length && !d.command) throw new DrillError(`${svc.id} has no unit to copy ExecStart from; set drill.command`);++ if (svc.units.length > 1 && !d.command) throw new DrillError(`${svc.id} runs ${svc.units.length} units; a drill starts one process, so set drill.command (or mark the drill unsupported)`);++ let directives = { execStart: null, rawExecStart: null, workingDirectory: null, user: null, environment: [] };++ if (svc.units.length) {++ const st = await systemd.show(exec, svc.units[0]);++ const texts = [];++ for (const f of [st.fragmentPath, ...st.dropIns].filter(Boolean)) {++ const t = await exec.readFile(f, { privileged: true });++ if (t != null) texts.push(t);++ }++ directives = envfile.unitDirectives(texts);++ }++ let argv = d.command;++ if (!argv) {++ if (!directives.execStart || !directives.execStart.length) throw new DrillError(`no ExecStart found for ${svc.units[0]}; set drill.command`);++ if (/[$%]/.test(directives.rawExecStart) || /^[^-:+!]*@/.test(directives.rawExecStart.split(/\s/)[0])) throw new DrillError(`${svc.units[0]} ExecStart uses substitutions or specifiers; set drill.command`);++ argv = directives.execStart;++ }++ if (!path.isAbsolute(argv[0])) throw new DrillError(`the drill command must start with an absolute path (got ${argv[0]}); set drill.command`);++ const environment = [];++ const skippedEnvironment = [];++ for (const e of directives.environment) {++ if (envfile.SECRET_NAME_RE.test(e.name)) skippedEnvironment.push(e.name);++ else environment.push(e);++ }++ return { argv, cwd: directives.workingDirectory || svc.repo, unitUser: directives.user, environment, skippedEnvironment };++}++++// ── the drill ────────────────────────────────────────────────────────────────++++async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {++ const { exec, inv, log } = ctx;++ const svc = service(inv, id);++ if (!(await exec.isRoot())) throw new DrillError('ovhost drill must run as root (sudo ovhost drill …): it copies databases as the service user and starts the drill instance through systemd-run');++ if (!svc.drill) throw new DrillError(`${id} has no drill block in the inventory`);++ if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);++ const d = svc.drill;++ if (d.compare.length && !svc.port) throw new DrillError(`${id} declares compare paths but no production port`);++ if (svc.envFile && !(await exec.stat(svc.envFile))) throw new DrillError(`${svc.envFile} not found`);++ const busy = await exec.listeners(d.port);++ if (busy.length) throw new DrillError(`port ${d.port} is already in use (${busy.map((l) => `${l.process || '?'} pid ${l.pid || '?'}`).join(', ')}); free it or change drill.port`);++ const bk = await resolveBackup(exec, inv, svc, backupArg);++ const spec = await startSpec(exec, svc);++ if (spec.unitUser && spec.unitUser !== svc.runAs) log(`warning: ${svc.units[0]} runs as ${spec.unitUser}, the inventory says ${svc.runAs}; the drill runs as ${svc.runAs}`);++ if (spec.skippedEnvironment.length) log(`not passing on unit Environment= with secret-looking names: ${spec.skippedEnvironment.join(', ')}`);++++ const held = await lock.acquire(exec, inv, `${id}-drill`);++ const at = stamp(exec.now());++ const tmp = path.join(inv.drillDir, `${id}-${at}`);++ const unitName = `ovhost-drill-${id}-${at}.service`;++ const logFile = path.join(tmp, 'drill.log');++ const record = {++ service: id,++ startedAt: new Date(exec.now()).toISOString(),++ operator: process.env.SUDO_USER || (await exec.userName()),++ host: inv.host,++ backup: bk.dir,++ backupTakenAt: bk.takenAt,++ port: d.port,++ dir: tmp,++ unit: unitName,++ pid: null,++ databases: [],++ ready: null,++ compare: [],++ counts: [],++ stop: null,++ kept: false,++ result: 'failed',++ failure: null,++ };++ const fail = (stage, message) => { if (!record.failure) record.failure = { stage, message }; log(`✗ ${stage}: ${message}`); return new Abort(message); };++ let created = false;++ let productionPids = [];++++ try {++ if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);++ await exec.mkdir(inv.drillDir, { mode: 0o711 });++ await exec.mkdir(tmp, { owner: svc.runAs, mode: 0o700 });++ created = true;++ await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });++ const vars = { tmp, port: d.port, db: {} };++ for (const f of bk.files) vars.db[f.name] = path.join(tmp, 'db', `${f.name}${f.ext}`);++ for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });++++ // 1. restore and check every database copy++ for (const f of bk.files) {++ const dest = vars.db[f.name];++ log(`restoring ${f.file} → ${dest} (as ${svc.runAs})`);++ const cp = await exec.run('cp', ['--no-preserve=mode,ownership', '--', f.file, dest], { as: svc.runAs });++ if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);++ const st = await exec.stat(dest);++ const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null };++ record.databases.push(entry);++ let rows;++ try { rows = await exec.sqlite(dest, 'PRAGMA integrity_check', { as: svc.runAs }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }++ entry.integrity = (rows || []).map((r) => String(Object.values(r)[0])).join('; ') || 'no result';++ if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);++ log(`integrity_check ${f.name}: ok`);++ }++ const restoredCounts = [];++ for (const c of d.counts) restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));++++ // 2. start the drill instance++ const lines = ['# written by ovhost drill; no secret values: the production env file is loaded separately'];++ for (const [k, v] of Object.entries(d.env)) lines.push(envLine(k, expand(v, vars)));++ for (const f of bk.files) lines.push(envLine(f.envName, vars.db[f.name]));++ const envPath = path.join(tmp, 'drill.env');++ await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });++ const props = [++ 'Type=exec',++ ...spec.environment.map((e) => environmentProperty(e.name, e.value)),++ ...(svc.envFile ? [`EnvironmentFile=${svc.envFile}`] : []),++ `EnvironmentFile=${envPath}`,++ `StandardOutput=append:${logFile}`,++ `StandardError=append:${logFile}`,++ 'ProtectSystem=strict',++ `ReadWritePaths=${tmp}`,++ 'PrivateTmp=yes',++ 'ProtectHome=read-only',++ 'NoNewPrivileges=yes',++ `SocketBindAllow=tcp:${d.port}`,++ 'SocketBindDeny=any',++ `RuntimeMaxSec=${d.runtimeMaxSeconds}`,++ 'TimeoutStopSec=15',++ ...(d.outbound ? [] : ['IPAddressDeny=any', 'IPAddressAllow=localhost']),++ ];++ if (d.outbound) log(`warning: the drill instance may reach non-loopback addresses (drill.outbound${d.outboundReason ? `: ${d.outboundReason}` : ''})`);++ const args = ['--unit', unitName, '--description', `ovhost restore drill of ${id}`, '--collect', '--quiet', `--uid=${svc.runAs}`, `--working-directory=${spec.cwd}`];++ for (const p of props) args.push('-p', p);++ args.push('--', ...spec.argv.map((a) => expand(a, vars)));++ for (const u of svc.units) { const st = await systemd.show(exec, u); if (st.mainPid) productionPids.push(st.mainPid); }++ log(`starting ${spec.argv.join(' ')} in ${spec.cwd} as ${svc.runAs} on port ${d.port} (${unitName})`);++ const run = await exec.run('systemd-run', args, { privileged: true });++ if (run.code !== 0) throw fail('start', `systemd-run failed: ${(run.stderr || '').trim().slice(0, 300)}`);++ const st = await systemd.show(exec, unitName);++ if (!st.mainPid) throw fail('start', `the drill instance exited right away (${st.active}/${st.sub}); rerun with --keep and read ${logFile}`);++ if (productionPids.includes(st.mainPid)) throw fail('start', `pid ${st.mainPid} is production's; not touching it`);++ record.pid = st.mainPid;++++ // 3. readiness, then comparisons++ const headers = (svc.ready && svc.ready.headers) || {};++ const readyUrl = `http://127.0.0.1:${d.port}${d.ready}`;++ const t0 = exec.now();++ let last = null;++ record.ready = { url: readyUrl, ok: false };++ while (exec.now() - t0 < d.readyTimeoutSeconds * 1000) {++ last = await exec.http(readyUrl, { timeoutMs: 5000, headers });++ if (last.status >= 200 && last.status < 300) break;++ if (!(await exec.pidAlive(record.pid))) {++ record.ready = { url: readyUrl, ok: false, status: last.status || 0, seconds: Math.round((exec.now() - t0) / 1000) };++ record.pid = null;++ throw fail('ready', `the drill instance exited before it was ready; rerun with --keep and read ${logFile}`);++ }++ await exec.sleep(1000);++ }++ record.ready = { url: readyUrl, ok: !!(last && last.status >= 200 && last.status < 300), status: last ? last.status || 0 : 0, seconds: Math.round((exec.now() - t0) / 1000) };++ if (!record.ready.ok) throw fail('ready', `${readyUrl} not ready after ${d.readyTimeoutSeconds}s (${last ? last.status || last.error : 'never probed'})`);++ log(`drill instance ready: ${d.ready} ${record.ready.status} after ${record.ready.seconds}s`);++++ for (const c of d.compare) {++ const h = { ...headers, ...c.headers };++ const prod = await exec.http(`http://127.0.0.1:${svc.port}${c.path}`, { timeoutMs: 10000, headers: h });++ const drl = await exec.http(`http://127.0.0.1:${d.port}${c.path}`, { timeoutMs: 10000, headers: h });++ const res = compareBodies(c, prod, drl);++ record.compare.push(res);++ log(`${res.match ? 'same' : 'DIFFERENT'} ${c.path}${res.detail ? `: ${res.detail}` : ''}`);++ }++ for (let i = 0; i < d.counts.length; i++) {++ const c = d.counts[i];++ const db = svc.databases.find((x) => x.name === c.db);++ let production = null;++ let error = null;++ try { production = await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }++ const row = { db: c.db, table: c.table, production, restored: restoredCounts[i], match: production === restoredCounts[i] };++ if (error) row.error = error;++ record.counts.push(row);++ log(`${row.match ? 'same' : 'DIFFERENT'} ${c.table}: production ${production == null ? `unknown (${error})` : production}, restored ${restoredCounts[i]}`);++ }++ const bad = [...record.compare.filter((r) => !r.match).map((r) => `${r.path} (${r.detail})`), ...record.counts.filter((r) => !r.match).map((r) => `${r.table} ${r.production} ≠ ${r.restored}`)];++ if (bad.length) throw fail('compare', `differs from production: ${bad.join('; ')}`);++ } catch (err) {++ if (!(err instanceof Abort)) fail('error', err.message);++ }++++ // 4. stop, clean up, record++ record.stop = await stopInstance(exec, record.pid, d.port, productionPids, log);++ if (record.stop && !record.stop.ok) fail('stop', record.stop.detail);++ if (created) {++ const safe = tmp.startsWith(`${inv.drillDir}/`) && /^[a-z][a-z0-9-]*-\d{8}-\d{6}$/.test(path.basename(tmp));++ if (keep) { record.kept = true; log(`kept ${tmp} (--keep)`); } else if (record.stop && !record.stop.ok) {++ record.kept = true;++ log(`kept ${tmp}: the drill instance could not be confirmed stopped`);++ } else if (safe) {++ const rm = await exec.run('rm', ['-rf', '--one-file-system', '--', tmp], { privileged: true });++ if (rm.code !== 0) { record.kept = true; log(`could not remove ${tmp}: ${(rm.stderr || '').trim()}`); }++ }++ }++ record.finishedAt = new Date(exec.now()).toISOString();++ record.result = record.failure ? 'failed' : 'passed';++ record.markdown = markdownRow(record);++ try {++ await exec.appendFile(path.join(inv.stateDir, 'drills', `${id}.jsonl`), `${JSON.stringify(record)}\n`);++ } finally {++ await held.release();++ }++ return { exitCode: record.failure ? 2 : 0, record };++}++++/** SIGTERM the drill's own MainPID, then SIGKILL after the grace period. Never anything else. */++async function stopInstance(exec, pid, port, productionPids, log) {++ if (!pid) return null;++ if (productionPids.includes(pid)) return { ok: false, detail: `refusing to signal production pid ${pid}` };++ log(`stopping the drill instance (pid ${pid})`);++ await exec.kill(pid, 'SIGTERM');++ let signal = 'SIGTERM';++ const t0 = exec.now();++ while (await exec.pidAlive(pid)) {++ if (exec.now() - t0 >= STOP_GRACE_MS) {++ if (signal === 'SIGKILL') return { ok: false, pid, signal, detail: `pid ${pid} survived SIGKILL` };++ log(`pid ${pid} still running after ${STOP_GRACE_MS / 1000}s; SIGKILL`);++ await exec.kill(pid, 'SIGKILL');++ signal = 'SIGKILL';++ }++ await exec.sleep(500);++ }++ const left = await exec.listeners(port);++ if (left.length) return { ok: false, pid, signal, detail: `port ${port} is still in use after pid ${pid} stopped (${left.map((l) => l.pid).join(', ')})` };++ return { ok: true, pid, signal };++}++++function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }++++/** A row for the table in docs/restore-drills.md. */++function markdownRow(r) {++ const checks = [];++ for (const db of r.databases) checks.push(`\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);++ if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);++ const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);++ const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);++ if (same.length) checks.push(`${same.join(', ')} identical to production${r.compare.some((c) => c.match && c.mode && c.mode !== 'bytes') ? ' (JSON without volatile keys where declared)' : ''}`);++ checks.push(...diff);++ for (const c of r.counts) checks.push(`${c.table} ${c.production == null ? '?' : c.production} ${c.match ? '=' : '≠'} ${c.restored}`);++ if (r.failure && r.failure.stage !== 'compare') checks.push(`${r.failure.stage}: ${cell(r.failure.message)}`);++ const when = r.startedAt.replace('T', ' ').slice(0, 16);++ return `| ${when} | ${r.service} | \`${r.backup}/\` (\`ovhost drill ${r.service}\`) | ${checks.join('; ')} | ${r.result} |`;++}++++module.exports = { drill, DrillError, compareBodies };@@@ -65,6 +65,39 @@ function splitAssignments(s) {return out;}++/**++ * The directives a restore drill needs from a unit file and its drop-ins, in systemd's order:++ * the last ExecStart/WorkingDirectory/User wins (an empty ExecStart= resets it), and Environment=++ * assignments accumulate. Returns { execStart: argv|null, rawExecStart, workingDirectory, user,++ * environment: [{ name, value }] }. The caller decides which Environment= values it may pass on —++ * ovhost never prints them.++ */++function unitDirectives(texts) {++ const out = { execStart: null, rawExecStart: null, workingDirectory: null, user: null, environment: [] };++ for (const text of texts) {++ for (const line of String(text || '').split('\n')) {++ const m = line.match(/^\s*(ExecStart|WorkingDirectory|User|Environment)=(.*)$/);++ if (!m) continue;++ const value = m[2].trim();++ if (m[1] === 'ExecStart') {++ out.rawExecStart = value || null;++ out.execStart = value ? splitAssignments(value.replace(/^[-@:+!]+/, '')) : null;++ } else if (m[1] === 'WorkingDirectory') out.workingDirectory = value.replace(/^-/, '') || null;++ else if (m[1] === 'User') out.user = value || null;++ else {++ for (const part of splitAssignments(value)) {++ const i = part.indexOf('=');++ if (i <= 0) continue;++ const name = part.slice(0, i);++ out.environment = out.environment.filter((e) => e.name !== name);++ out.environment.push({ name, value: part.slice(i + 1) });++ }++ }++ }++ }++ return out;++}++/*** Unit text with the value of every secret-looking Environment= assignment replaced by <redacted>.* Used by snapshots: unit files are config worth keeping, but an operator may have put a secret@@@ -84,4 +117,4 @@ function redactUnit(unitText) {}).join('\n');}--module.exports = { parseNames, parseExample, unitEnvironmentNames, redactUnit, SECRET_NAME_RE };++module.exports = { parseNames, parseExample, unitEnvironmentNames, unitDirectives, redactUnit, SECRET_NAME_RE };@@@ -19,6 +19,7 @@* http(url, { timeoutMs, headers }) -> { status, body } | { status: 0, error }* sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })* listeners(port) -> [{ pid, process }]++ * kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)* sleep(ms) ; now() -> ms ; isRoot() ; userName() ; hostname()*/const { spawn } = require('child_process');@@@ -224,6 +225,10 @@ function createSystemExecutor({ env = process.env } = {}) {sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },listeners,++ kill: async (pid, signal = 'SIGTERM') => {++ if (!Number.isInteger(pid) || pid <= 1) throw new Error(`refusing to signal pid ${pid}`);++ try { process.kill(pid, signal); return true; } catch (err) { if (err.code === 'ESRCH') return false; throw err; }++ },sleep: (ms) => new Promise((r) => setTimeout(r, ms)),now: () => Date.now(),isRoot: async () => isRoot(),@@@ -9,6 +9,7 @@ const { validate, envNames } = require('./commands/validate');const { status } = require('./commands/status');const { backup } = require('./commands/backup');const { snapshot } = require('./commands/snapshot');++const { drill, DrillError } = require('./commands/drill');const releases = require('./releases');const certs = require('./certs');const nginx = require('./nginx');@@@ -25,6 +26,8 @@ module.exports = {status,backup,snapshot,++ drill,++ DrillError,releases,certs,nginx,@@@ -70,6 +70,7 @@ function normaliseService(id, raw, defaults) {}),backupOnChange: s.backupOnChange || [],nginx: s.nginx || null,++ drill: null,};if (typeof out.owner !== 'string' || !/^[a-z_][a-z0-9_-]*$/.test(out.owner)) throw new InventoryError(`${where}.owner must be a user name`);if (!/^[a-z_][a-z0-9_-]*$/.test(out.runAs)) throw new InventoryError(`${where}.runAs must be a user name`);@@@ -110,9 +111,96 @@ function normaliseService(id, raw, defaults) {if (n.certName && !FILE_NAME_RE.test(n.certName)) throw new InventoryError(`${where}.nginx.certName must be a name`);}for (const p of out.backupOnChange) rel(p, `${where}.backupOnChange`);++ if (s.drill) out.drill = normaliseDrill(s.drill, out, where);return out;}++const ENV_NAME_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;++const TABLE_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;++const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;++++/**++ * The `drill` block (Wave 22 restore drills): how to start a second, side-effect-free instance of++ * the service against restored copies of its databases, and what to compare with production.++ *++ * supported false (with a reason) when a second instance cannot run safely++ * port spare loopback port for the drill instance (never the service's own port)++ * databases { <database name>: <ENV VAR that points the service at that file> }++ * env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}++ * and {db:<name>}++ * dirs directories to create (owned by the service user) inside {tmp} before the start++ * command argv to start the service; default: the production unit's ExecStart++ * ready readiness path; default: the path of the service's ready URL++ * compare [{ path, ignore: [volatile keys], headers }] read-only GETs compared with production++ * counts [{ db, table }] row counts compared between production and the restored copy++ * outbound true lets the instance reach non-loopback addresses (default: loopback only)++ */++function normaliseDrill(raw, svc, where) {++ const w = `${where}.drill`;++ if (!raw || typeof raw !== 'object') throw new InventoryError(`${w} must be an object`);++ if (raw.supported === false) {++ if (typeof raw.reason !== 'string' || !raw.reason.trim()) throw new InventoryError(`${w}.reason is required when supported is false`);++ return { supported: false, reason: raw.reason };++ }++ const port = Number(raw.port);++ if (!(Number.isInteger(port) && port >= 1024 && port < 65536)) throw new InventoryError(`${w}.port must be a TCP port from 1024`);++ if (port === svc.port) throw new InventoryError(`${w}.port must differ from the service's own port`);++ if (!svc.databases.length) throw new InventoryError(`${w}: a drill restores databases, and ${svc.id} declares none`);++ const dbNames = svc.databases.map((d) => d.name);++ const databases = raw.databases || {};++ if (typeof databases !== 'object' || Array.isArray(databases) || !Object.keys(databases).length) throw new InventoryError(`${w}.databases must map at least one database name to an env var name`);++ for (const [name, envName] of Object.entries(databases)) {++ if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);++ if (!ENV_NAME_RE.test(envName)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);++ }++ const env = raw.env || {};++ for (const [k, v] of Object.entries(env)) {++ if (!ENV_NAME_RE.test(k)) throw new InventoryError(`${w}.env: "${k}" is not an env var name`);++ if (typeof v !== 'string' || /[\n\r\0]/.test(v)) throw new InventoryError(`${w}.env.${k} must be a one-line string`);++ for (const m of v.matchAll(PLACEHOLDER_RE)) {++ if (m[1].startsWith('db:') && !Object.keys(databases).includes(m[1].slice(3))) throw new InventoryError(`${w}.env.${k}: {${m[1]}} is not a restored database`);++ }++ }++ if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);++ for (const envName of Object.values(databases)) if (envName in env) throw new InventoryError(`${w}.env.${envName} is already set by drill.databases`);++ const dirs = raw.dirs || [];++ for (const d of dirs) if (typeof d !== 'string' || !/^\{tmp\}\/[A-Za-z0-9._/-]+$/.test(d) || d.includes('..')) throw new InventoryError(`${w}.dirs entries must be paths under {tmp}`);++ if (raw.command != null && (!Array.isArray(raw.command) || !raw.command.length || raw.command.some((a) => typeof a !== 'string'))) throw new InventoryError(`${w}.command must be an argv array`);++ if (raw.command && !path.isAbsolute(raw.command[0])) throw new InventoryError(`${w}.command[0] must be an absolute path`);++ let ready = raw.ready || null;++ if (!ready && svc.ready) ready = new URL(svc.ready.url).pathname;++ if (!ready || !/^\/[^/]/.test(ready)) throw new InventoryError(`${w}.ready must be a path such as /api/ready (or declare the service's ready URL)`);++ const compare = (raw.compare || []).map((c, i) => {++ const cc = typeof c === 'string' ? { path: c } : c;++ if (!cc || typeof cc.path !== 'string' || !/^\/[^/]/.test(cc.path) || /\s/.test(cc.path)) throw new InventoryError(`${w}.compare[${i}].path must be a path such as /api/items?limit=5`);++ const ignore = cc.ignore || [];++ if (!Array.isArray(ignore) || ignore.some((k) => typeof k !== 'string' || !k)) throw new InventoryError(`${w}.compare[${i}].ignore must be a list of key names`);++ return { path: cc.path, ignore, headers: cc.headers || {} };++ });++ const counts = (raw.counts || []).map((c, i) => {++ if (!c || !Object.keys(databases).includes(c.db)) throw new InventoryError(`${w}.counts[${i}].db must be a restored database`);++ if (!TABLE_RE.test(c.table || '')) throw new InventoryError(`${w}.counts[${i}].table must be a table name`);++ return { db: c.db, table: c.table };++ });++ if (!compare.length && !counts.length) throw new InventoryError(`${w} needs at least one compare path or row count`);++ const readyTimeoutSeconds = Number(raw.readyTimeoutSeconds || (svc.ready && svc.ready.timeoutSeconds) || 90);++ return {++ supported: true,++ port,++ databases,++ env,++ dirs,++ command: raw.command || null,++ ready,++ readyTimeoutSeconds,++ compare,++ counts,++ outbound: raw.outbound === true,++ outboundReason: raw.outboundReason || null,++ runtimeMaxSeconds: Number(raw.runtimeMaxSeconds || 1800),++ };++}++function normalise(raw) {if (!raw || typeof raw !== 'object' || !raw.services || typeof raw.services !== 'object') throw new InventoryError('inventory needs a "services" object');const defaults = raw.defaults || {};@@@ -120,6 +208,7 @@ function normalise(raw) {host: raw.host || null,stateDir: abs(raw.stateDir || '/var/lib/openvibe-host', 'stateDir'),backupDir: abs(raw.backupDir || '/var/backups/openvibe', 'backupDir'),++ drillDir: abs(raw.drillDir || '/var/lib/openvibe-drills', 'drillDir'),nginx: {sitesAvailable: abs((raw.nginx && raw.nginx.sitesAvailable) || '/etc/nginx/sites-available', 'nginx.sitesAvailable'),sitesEnabled: abs((raw.nginx && raw.nginx.sitesEnabled) || '/etc/nginx/sites-enabled', 'nginx.sitesEnabled'),@@@ -131,6 +220,14 @@ function normalise(raw) {if (id.startsWith('_')) continue;inv.services[id] = normaliseService(id, svc, defaults);}++ const ports = new Map(Object.values(inv.services).filter((s) => s.port).map((s) => [s.port, s.id]));++ const drillPorts = new Map();++ for (const s of Object.values(inv.services)) {++ if (!s.drill || !s.drill.supported) continue;++ if (ports.has(s.drill.port)) throw new InventoryError(`services.${s.id}.drill.port ${s.drill.port} is ${ports.get(s.drill.port)}'s port`);++ if (drillPorts.has(s.drill.port)) throw new InventoryError(`services.${s.id}.drill.port ${s.drill.port} is also ${drillPorts.get(s.drill.port)}'s drill port`);++ drillPorts.set(s.drill.port, s.id);++ }return inv;}@@@ -17,7 +17,7 @@ runTests([test('host.example.json is a valid inventory for every service on the host', () => {const raw = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));const inv = normalise(raw);-- assert.deepStrictEqual(Object.keys(inv.services), ['network', 'live', 'media', 'tools', 'community', 'events', 'games', 'sites', 'host']);++ assert.deepStrictEqual(Object.keys(inv.services), ['network', 'live', 'media', 'tools', 'community', 'events', 'games', 'sites', 'host', 'billing', 'chat', 'search', 'sources', 'wiki', 'blog', 'tips', 'vip', 'news', 'reviews', 'deals', 'coupons', 'trade', 'codes']);assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');nnew file mode 100644@@@ -0,0 +1,413 @@++'use strict';++/**++ * ovhost drill against the fake host: the full flow, integrity failure, readiness timeout, an++ * instance that dies, mismatch reporting, cleanup on failure, refusals (not root, port in use,++ * unsupported, no backup), and the guarantee that nothing outside the drill directory, the lock and++ * the drill log is written.++ */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const { scenario, test, runTests, SECRET } = require('./helpers');++const { normalise } = require('../lib/inventory');++const { compareBodies } = require('../lib/commands/drill');++++const DRILL_DIR = '/var/lib/openvibe-drills';++const PROD_DB = '/var/lib/openvibe-community/community.db';++const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';++++function communityEntry(drillOverrides = {}) {++ return {++ repo: '/opt/openvibe.community',++ units: ['openvibe-community.service'],++ envFile: '/etc/openvibe/community.env',++ port: 4200,++ ready: { url: 'http://127.0.0.1:4200/api/ready', timeoutSeconds: 30 },++ databases: [{ name: 'community', path: PROD_DB }],++ drill: {++ port: 14200,++ databases: { community: 'COMMUNITY_DB_PATH' },++ env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words' },++ dirs: ['{tmp}/data'],++ compare: [{ path: '/api/pastes?limit=5' }, { path: '/api/v1/pulse?limit=5', ignore: ['generated_at'] }],++ counts: [{ db: 'community', table: 'pastes' }],++ ...drillOverrides,++ },++ };++}++++/**++ * The standard fake host plus Community in production, with one `ovhost backup community` taken.++ * The drill instance's behaviour is controlled through `host.drill`.++ */++async function drillScenario({ drill: drillOverrides, backup = true } = {}) {++ const host = scenario();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.community = communityEntry(drillOverrides);++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++++ const repo = host.createRepo('/opt/openvibe.community', { owner: 'ubuntu' });++ const sha = repo.commit({ 'package.json': '{"name":"openvibe-community"}', 'server/index.js': 'community();' }, { message: 'initial' });++ repo.publish(sha);++ repo.checkout(sha);++ host.addUnit('openvibe-community.service', { mainPid: 2222 });++ host.alivePids.add(2222);++ host.listeners.set(4200, [{ pid: 2222, process: 'node' }]);++ host.put('/etc/systemd/system/openvibe-community.service', [++ '[Service]',++ 'User=ubuntu',++ 'WorkingDirectory=/opt/openvibe.community',++ 'ExecStart=/usr/bin/node server/index.js',++ `Environment=NODE_ENV=production "SESSION_SECRET=${SECRET}-unit"`,++ '',++ ].join('\n'));++ host.put('/etc/openvibe/community.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.community\n`, { mode: 0o600 });++ host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });++++ host.http.set('http://127.0.0.1:4200/api/ready', () => ({ status: 200, body: { status: 'ready' } }));++ host.http.set('http://127.0.0.1:4200/api/pastes?limit=5', () => ({ status: 200, body: PASTES }));++ host.http.set('http://127.0.0.1:4200/api/v1/pulse?limit=5', () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:00.000Z' } }));++++ host.drill = {++ integrity: [{ integrity_check: 'ok' }],++ productionCount: 892,++ restoredCount: 892,++ ready: () => ({ status: 200, body: { status: 'ready' } }),++ pastes: () => ({ status: 200, body: PASTES }),++ pulse: () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:07.000Z' } }),++ onStart: null,++ ignoreTerm: false,++ ignoreKill: false,++ };++ host.sqliteHandler = (db, sql) => {++ if (/integrity_check/.test(sql)) return host.drill.integrity;++ if (/count\(\*\) AS n FROM "pastes"/.test(sql)) return [{ n: db === PROD_DB ? host.drill.productionCount : host.drill.restoredCount }];++ if (/is_recording/.test(sql)) return [{ n: host.recording }];++ return [{ n: 0 }];++ };++ host.onSystemdRun = (spec) => {++ if (host.drill.onStart) { const r = host.drill.onStart(spec); if (r) return r; }++ host.listeners.set(14200, [{ pid: spec.pid, process: 'node' }]);++ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });++ host.http.set('http://127.0.0.1:14200/api/ready', alive(() => host.drill.ready()));++ host.http.set('http://127.0.0.1:14200/api/pastes?limit=5', alive(() => host.drill.pastes()));++ host.http.set('http://127.0.0.1:14200/api/v1/pulse?limit=5', alive(() => host.drill.pulse()));++ return undefined;++ };++ host.onKill = (pid, signal) => {++ if (signal === 'SIGTERM' && host.drill.ignoreTerm) return false;++ if (signal === 'SIGKILL' && host.drill.ignoreKill) return false;++ return undefined;++ };++++ if (backup) {++ const b = await host.cli('backup', 'community', '--json');++ assert.strictEqual(b.code, 0, b.out);++ host.backupDir = JSON.parse(b.out).dir;++ }++ host.advance(60 * 1000);++ return host;++}++++/** Every file entry as "type:owner:mode:content", to diff the fake filesystem. */++function fsSnapshot(host) {++ const m = new Map();++ for (const [k, v] of host.files) m.set(k, `${v.type}:${v.owner}:${v.mode}:${v.type === 'file' ? String(v.content) : v.target || ''}`);++ return m;++}++++function changedPaths(before, after) {++ const out = [];++ for (const [k, v] of after) if (before.get(k) !== v) out.push(k);++ for (const k of before.keys()) if (!after.has(k)) out.push(k);++ return out.sort();++}++++function lastLog(host) {++ const text = host.read('/var/lib/openvibe-host/drills/community.jsonl');++ const lines = text.trim().split('\n');++ return JSON.parse(lines[lines.length - 1]);++}++++const allowedWrite = (p, tmp) => p === DRILL_DIR || p === tmp || p.startsWith(`${tmp}/`)++ || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/community.jsonl'++ || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/community-drill.lock';++++runTests([++ test('full flow: restore, integrity, sandboxed second instance, identical answers, stop by pid, cleanup, log row', async () => {++ const host = await drillScenario();++ const before = fsSnapshot(host);++ const callsBefore = host.calls.length;++ const r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 0, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.result, 'passed');++ assert.strictEqual(rec.failure, null);++ assert.strictEqual(rec.backup, host.backupDir);++ const tmp = rec.dir;++ assert.match(tmp, /^\/var\/lib\/openvibe-drills\/community-\d{8}-\d{6}$/);++ assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['community', 'ok', `${tmp}/db/community.db`]]);++ assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/pastes?limit=5', true], ['/api/v1/pulse?limit=5', true]]);++ assert.deepStrictEqual(rec.counts, [{ db: 'community', table: 'pastes', production: 892, restored: 892, match: true }]);++ assert.strictEqual(rec.ready.ok, true);++ assert.strictEqual(rec.stop.ok, true);++ assert.strictEqual(rec.stop.signal, 'SIGTERM');++++ const calls = host.calls.slice(callsBefore);++ // The copy is made as the service user; integrity and counts run as the service user.++ const cp = calls.find((c) => c.cmd === 'cp');++ assert.strictEqual(cp.as, 'ubuntu');++ assert.deepStrictEqual(cp.args.slice(-2), [`${host.backupDir}/community.db`, `${tmp}/db/community.db`]);++ const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));++ assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/community.db`, 'ubuntu']);++++ // systemd-run: production ExecStart/WorkingDirectory, production env file then the override file,++ // sandbox properties, loopback-only networking, the service user.++ assert.strictEqual(host.systemdRuns.length, 1);++ const run = host.systemdRuns[0];++ assert.strictEqual(run.uid, 'ubuntu');++ assert.strictEqual(run.cwd, '/opt/openvibe.community');++ assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js']);++ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/community.env', `${tmp}/drill.env`]);++ for (const p of ['ProtectSystem=strict', `ReadWritePaths=${tmp}`, 'PrivateTmp=yes', 'NoNewPrivileges=yes', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'SocketBindAllow=tcp:14200', 'SocketBindDeny=any', 'Environment=NODE_ENV=production']) {++ assert.ok(run.props.includes(p), `systemd-run -p ${p}`);++ }++ assert.ok(!run.props.some((p) => p.includes('SESSION_SECRET')), 'a secret-looking unit Environment= is not passed on');++ const runCall = calls.find((c) => c.cmd === 'systemd-run');++ assert.strictEqual(runCall.privileged, true);++++ // Stopped by its own pid with SIGTERM; production's pid is never signalled; no pkill.++ const kills = calls.filter((c) => c.cmd === 'kill');++ assert.deepStrictEqual(kills.map((c) => c.args), [['SIGTERM', run.pid]]);++ assert.ok(!calls.some((c) => /pkill|killall/.test(c.cmd)));++ assert.ok(host.alivePids.has(2222), 'production still running');++ assert.strictEqual(host.units.get('openvibe-community.service').restarts, 0);++ assert.ok(!host.restarts().length, 'nothing restarted');++++ // Only the drill directory (now removed), the lock (released) and the drill log changed.++ const after = fsSnapshot(host);++ const changed = changedPaths(before, after);++ for (const p of changed) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);++ assert.ok(!host.files.has(tmp), 'temp dir removed');++ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'), 'lock released');++++ // Secrets: nothing ovhost printed, logged or passed on carries a value.++ const everything = [r.out, host.read('/var/lib/openvibe-host/drills/community.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');++ assert.ok(!everything.includes(SECRET), 'a secret value leaked');++ assert.ok(!host.reads.includes('/etc/openvibe/community.env'), 'ovhost never reads the env file for a drill');++++ // Markdown row ready for docs/restore-drills.md.++ assert.match(r.out, /\| 2026-09-22 12:01 \| community \| `\/var\/backups\/openvibe\/community\/20260922-120000\/` \(`ovhost drill community`\) \| `pragma integrity_check` \(community\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);++ assert.strictEqual(rec.markdown, r.out.split('\n').pop());++ }),++++ test('--keep leaves the drill directory; the override file holds inventory values only', async () => {++ const host = await drillScenario();++ const before = fsSnapshot(host);++ const r = await host.cli('drill', 'community', '--keep');++ assert.strictEqual(r.code, 0, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.kept, true);++ const tmp = rec.dir;++ assert.strictEqual(host.files.get(tmp).owner, 'ubuntu');++ assert.strictEqual(host.files.get(tmp).mode, 0o700);++ assert.strictEqual(host.files.get(`${tmp}/db/community.db`).owner, 'ubuntu');++ assert.strictEqual(host.read(`${tmp}/db/community.db`), host.read(`${host.backupDir}/community.db`));++ assert.strictEqual(host.files.get(`${tmp}/data`).owner, 'ubuntu');++ const env = host.read(`${tmp}/drill.env`);++ for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `COMMUNITY_DB_PATH=${tmp}/db/community.db`]) {++ assert.ok(env.split('\n').includes(line), `drill.env has ${line}`);++ }++ assert.ok(!env.includes(SECRET));++ for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);++ // The production database and the backup are untouched.++ assert.strictEqual(host.read(PROD_DB), 'sqlite-production');++ }),++++ test('integrity failure: nothing is started, the copy is removed, the log says failed', async () => {++ const host = await drillScenario();++ host.drill.integrity = [{ integrity_check: '*** in database main ***' }, { integrity_check: 'Page 5: btree page has a bad cell' }];++ const r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.result, 'failed');++ assert.strictEqual(rec.failure.stage, 'integrity');++ assert.match(rec.failure.message, /bad cell/);++ assert.strictEqual(host.systemdRuns.length, 0);++ assert.ok(!host.files.has(rec.dir));++ assert.match(r.out, /\| failed \|$/);++ }),++++ test('readiness timeout: the instance is stopped by its pid and the directory removed', async () => {++ const host = await drillScenario();++ host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });++ const r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'ready');++ assert.match(rec.failure.message, /not ready after 30s \(503\)/);++ assert.strictEqual(rec.compare.length, 0, 'nothing compared');++ const pid = host.systemdRuns[0].pid;++ assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', pid]]);++ assert.ok(!host.alivePids.has(pid));++ assert.ok(!host.files.has(rec.dir));++ assert.ok(!host.listeners.has(14200));++ }),++++ test('an instance that exits before it is ready fails the drill without signalling anything', async () => {++ const host = await drillScenario();++ let probes = 0;++ host.drill.ready = () => { probes += 1; if (probes === 2) host.alivePids.delete(host.systemdRuns[0].pid); return { status: 503 }; };++ const r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'ready');++ assert.match(rec.failure.message, /exited before it was ready/);++ assert.ok(!host.calls.some((c) => c.cmd === 'kill'));++ assert.ok(!host.files.has(rec.dir));++ }),++++ test('mismatches are reported per path and per table; volatile keys are ignored only where declared', async () => {++ const host = await drillScenario();++ host.drill.pastes = () => ({ status: 200, body: PASTES.replace('world', 'World') });++ host.drill.pulse = () => ({ status: 200, body: { items: [{ id: 2 }], generated_at: 'different, but ignored' } });++ host.drill.productionCount = 900;++ const r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'compare');++ const [pastes, pulse] = rec.compare;++ assert.strictEqual(pastes.match, false);++ assert.match(pastes.detail, /first difference at byte 55/);++ assert.strictEqual(pulse.match, false);++ assert.match(pulse.detail, /\$\.items\[0\]\.id/);++ assert.ok(!/generated_at/.test(pulse.detail));++ assert.deepStrictEqual(rec.counts[0], { db: 'community', table: 'pastes', production: 900, restored: 892, match: false });++ assert.match(rec.markdown, /`\/api\/pastes\?limit=5` differs \(bodies differ/);++ assert.match(rec.markdown, /pastes 900 ≠ 892 \| failed \|$/);++ // Still stopped and cleaned up.++ assert.ok(!host.alivePids.has(host.systemdRuns[0].pid));++ assert.ok(!host.files.has(rec.dir));++ }),++++ test('compare: a status mismatch or a failing production answer is never a match', () => {++ const e = { path: '/x', ignore: [] };++ assert.strictEqual(compareBodies(e, { status: 500, body: 'e' }, { status: 500, body: 'e' }).match, false);++ assert.match(compareBodies(e, { status: 200, body: 'a' }, { status: 404, body: 'a' }).detail, /status 404/);++ assert.strictEqual(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '{"a":1,"t":1}' }, { status: 200, body: '{"a":1,"t":2}' }).match, true);++ assert.match(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '<html>' }, { status: 200, body: '<html>' }).detail, /not JSON/);++ }),++++ test('cleanup on failure: systemd-run fails → directory removed, lock released, row logged', async () => {++ const host = await drillScenario();++ const before = fsSnapshot(host);++ host.drill.onStart = () => ({ code: 1, stderr: 'Failed to start transient service unit: Unit already exists.' });++ const r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 2, r.out);++ const rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'start');++ assert.match(rec.failure.message, /systemd-run failed/);++ assert.ok(!host.files.has(rec.dir));++ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'));++ for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, rec.dir), `unexpected write: ${p}`);++ // A second drill can run straight after.++ host.drill.onStart = null;++ host.advance(1000);++ assert.strictEqual((await host.cli('drill', 'community')).code, 0);++ }),++++ test('an instance that ignores SIGTERM gets SIGKILL; one that survives SIGKILL keeps its directory and fails', async () => {++ let host = await drillScenario();++ host.drill.ignoreTerm = true;++ let r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 0, r.out);++ let rec = lastLog(host);++ assert.strictEqual(rec.stop.signal, 'SIGKILL');++ assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args[0]), ['SIGTERM', 'SIGKILL']);++++ host = await drillScenario();++ host.drill.ignoreTerm = true;++ host.drill.ignoreKill = true;++ r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 2, r.out);++ rec = lastLog(host);++ assert.strictEqual(rec.failure.stage, 'stop');++ assert.strictEqual(rec.kept, true);++ assert.ok(host.files.has(rec.dir), 'directory kept while the instance may still hold it');++ assert.ok(host.alivePids.has(2222), 'production untouched');++ assert.ok(!host.calls.some((c) => c.cmd === 'kill' && c.args[1] === 2222));++ }),++++ test('refusals: not root, port in use, unsupported service, no drill block, no backup, missing database copy', async () => {++ let host = await drillScenario();++ host.exec.isRoot = async () => false;++ let r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 1);++ assert.match(r.out, /must run as root/);++ assert.strictEqual(host.systemdRuns.length, 0);++++ host = await drillScenario();++ host.listeners.set(14200, [{ pid: 777, process: 'python3' }]);++ const before = fsSnapshot(host);++ r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 1);++ assert.match(r.out, /port 14200 is already in use \(python3 pid 777\)/);++ assert.strictEqual(host.systemdRuns.length, 0);++ assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');++++ host = await drillScenario({ drill: { supported: false, reason: 'workers have no off switch' } });++ r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 1);++ assert.match(r.out, /does not support restore drills: workers have no off switch/);++++ host = await drillScenario();++ r = await host.cli('drill', 'media');++ assert.strictEqual(r.code, 1);++ assert.match(r.out, /media has no drill block/);++++ host = await drillScenario({ backup: false });++ r = await host.cli('drill', 'community');++ assert.strictEqual(r.code, 1);++ assert.match(r.out, /no ovhost backup recorded for community/);++++ host = await drillScenario({ backup: false });++ host.ensureDir('/srv/old-backup');++ r = await host.cli('drill', 'community', '--backup', '/srv/old-backup');++ assert.strictEqual(r.code, 1);++ assert.match(r.out, /has no copy of the community database/);++ host.put('/srv/old-backup/community.db', 'older sqlite', { owner: 'ubuntu' });++ r = await host.cli('drill', 'community', '--backup', '/srv/old-backup', '--json');++ assert.strictEqual(r.code, 0, r.out);++ assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');++ r = await host.cli('drill', 'community', '--backup', 'relative/dir');++ assert.strictEqual(r.code, 1);++ }),++++ test('inventory: drill blocks are validated', () => {++ const base = { owner: 'ubuntu' };++ const svc = (drill, extra = {}) => normalise({ services: { community: { ...base, ...communityEntry(), ...extra, drill: { ...communityEntry().drill, ...drill } } } });++ assert.throws(() => svc({ port: 4200 }), /must differ from the service's own port/);++ assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of community's databases/);++ assert.throws(() => svc({ env: { HOST: '127.0.0.1' } }), /\{port\}/);++ assert.throws(() => svc({ env: { PORT: '{port}', COMMUNITY_DB_PATH: '/x' } }), /already set by drill.databases/);++ assert.throws(() => svc({ env: { PORT: '{port}\nEVIL=1' } }), /one-line/);++ assert.throws(() => svc({ dirs: ['/opt/openvibe.community/data'] }), /under \{tmp\}/);++ assert.throws(() => svc({ counts: [{ db: 'community', table: 'x; drop table y' }] }), /table name/);++ assert.throws(() => svc({ compare: [{ path: 'http://evil/' }] }), /path/);++ assert.throws(() => svc({ supported: false }), /reason is required/);++ assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, community: { ...base, ...communityEntry() } } }), /14200 is a's port/);++ assert.strictEqual(svc({}).services.community.drill.ready, '/api/ready');++ }),++++ test('host.example.json: every service with a database has a drill block; unsupported ones give a reason', () => {++ const raw = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));++ const inv = normalise(raw);++ for (const s of Object.values(inv.services)) {++ if (!s.databases.length) continue;++ assert.ok(s.drill, `${s.id} has a drill block`);++ if (!s.drill.supported) { assert.ok(s.drill.reason.length > 20, `${s.id} explains why`); continue; }++ assert.ok(s.drill.compare.length || s.drill.counts.length);++ assert.ok(Object.keys(s.drill.env).length >= 2, `${s.id} overrides at least its port and one side effect`);++ }++ }),++]);@@@ -45,6 +45,27 @@ runTests([db.close();}),++ test('drill primitives: integrity_check through the worker, kill() signals exactly one pid', async () => {++ const dbPath = path.join(dir, 'restored.db');++ const db = new Database(dbPath);++ db.pragma('journal_mode = WAL');++ db.exec('CREATE TABLE pastes (id INTEGER PRIMARY KEY); INSERT INTO pastes DEFAULT VALUES;');++ db.close();++ const me = await exec.userName();++ assert.deepStrictEqual(await exec.sqlite(dbPath, 'PRAGMA integrity_check', { as: me }), [{ integrity_check: 'ok' }]);++ assert.deepStrictEqual(await exec.sqlite(dbPath, 'SELECT count(*) AS n FROM "pastes"', { as: me }), [{ n: 1 }]);++++ const { spawn } = require('child_process');++ const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' });++ const exited = new Promise((r) => child.on('exit', (code, signal) => r(signal)));++ assert.strictEqual(await exec.pidAlive(child.pid), true);++ assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), true);++ assert.strictEqual(await exited, 'SIGTERM');++ assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), false, 'a pid that is gone');++ await assert.rejects(exec.kill(1, 'SIGTERM'), /refusing/);++ await assert.rejects(exec.kill(0, 'SIGTERM'), /refusing/);++ }),++test('http, files, locks and run() behave as the fake host assumes', async () => {const server = http.createServer((req, res) => { res.writeHead(req.url === '/ok' ? 200 : 503); res.end(JSON.stringify({ host: req.headers.host })); });await new Promise((r) => server.listen(0, '127.0.0.1', r));@@@ -27,6 +27,10 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stsqliteCalls: [],reads: [],onRestart: null, // (unit) -> void++ onSystemdRun: null, // (spec) -> undefined | { code, stderr } ; spec = { unit, uid, cwd, props, argv, pid }++ onKill: null, // (pid, signal) -> false to ignore the signal++ systemdRuns: [],++ nextPid: 40000,};// ── filesystem ──@@@ -186,6 +190,28 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st}}++ // ── systemd-run (transient units: restore drills) ──++ function systemdRunCmd(args) {++ const spec = { unit: null, uid: null, cwd: null, props: [], argv: [], pid: null };++ for (let i = 0; i < args.length; i++) {++ const a = args[i];++ if (a === '--') { spec.argv = args.slice(i + 1); break; }++ if (a === '--unit') spec.unit = args[++i];++ else if (a === '--description') i += 1;++ else if (a === '-p') spec.props.push(args[++i]);++ else if (a.startsWith('--uid=')) spec.uid = a.slice(6);++ else if (a.startsWith('--working-directory=')) spec.cwd = a.slice(20);++ }++ spec.envFiles = spec.props.filter((p) => p.startsWith('EnvironmentFile=')).map((p) => p.slice(16));++ spec.pid = host.nextPid++;++ host.systemdRuns.push(spec);++ const r = host.onSystemdRun ? host.onSystemdRun(spec) : undefined;++ if (r && r.code) return { stdout: '', stderr: '', ...r };++ if (!(r && r.exitedAtStart)) alivePids.add(spec.pid);++ host.addUnit(spec.unit, { mainPid: spec.pid, active: r && r.exitedAtStart ? 'failed' : 'active', transient: true });++ return { code: 0, stdout: '', stderr: '' };++ }++host.socketViolations = () => calls.filter((c) => c.cmd === 'systemctl' && ['stop', 'restart', 'kill', 'try-restart', 'reload-or-restart', 'disable', 'mask'].includes(c.args[0]) && String(c.args[1]).endsWith('.socket'));host.restarts = () => calls.filter((c) => c.cmd === 'systemctl' && c.args[0] === 'restart').map((c) => c.args[1]);@@@ -201,6 +227,15 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stcase 'rm': rmTree(args[args.length - 1]); return { code: 0, stdout: '', stderr: '' };case 'install': { const src = args[args.length - 2]; const dest = args[args.length - 1]; put(dest, host.read(src)); return { code: 0, stdout: '', stderr: '' }; }case 'node': return { code: 0, stdout: '', stderr: '' };++ case 'systemd-run': return systemdRunCmd(args);++ case 'cp': {++ const src = args[args.length - 2];++ const dest = args[args.length - 1];++ const e = get(src);++ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `cp: cannot stat '${src}': No such file or directory` };++ put(dest, e.content, { owner: opts.as || user, mode: 0o644 });++ return { code: 0, stdout: '', stderr: '' };++ }default: return { code: 127, stdout: '', stderr: `fake host: no ${cmd}` };}},@@@ -251,6 +286,18 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stput(dest, `backup of ${db}`, { owner: as || user, mode: 0o640 });},async listeners(port) { return listeners.get(port) || []; },++ async kill(pid, signal = 'SIGTERM') {++ calls.push({ cmd: 'kill', args: [signal, pid], as: null, privileged: true, cwd: null });++ if (!alivePids.has(pid)) return false;++ if (host.onKill && host.onKill(pid, signal) === false) return true;++ alivePids.delete(pid);++ for (const u of units.values()) if (u.mainPid === pid) { u.active = 'inactive'; u.sub = 'dead'; }++ for (const [port, list] of listeners) {++ const rest = list.filter((l) => l.pid !== pid);++ if (rest.length) listeners.set(port, rest); else listeners.delete(port);++ }++ return true;++ },async sleep(ms) { clock += ms; },now: () => clock,async isRoot() { return root; },
@@ -31,7 +31,7 @@ Stage A adds a few safety rules of its own:
@@ -31,7 +31,7 @@ Stage A adds a few safety rules of its own:
## Owns
## Owns
-- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, SQLite backup hooks.
+- Stage A: the host inventory (services, units, env names, ports, probes, drain policy), environment validation, release install and rollback for git-checkout services, readiness and drain orchestration, the release log, certificate inventory, nginx vhost rendering and transactional install, config snapshots, SQLite backup hooks, restore drills.
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Stage B: tenant projects (keyed by project id), static sites, immutable content-addressed deploy artifacts, activation and rollback, default and custom domains (DNS TXT verification), per-project quotas, upload/validation logs, the tenant vhosts (`ovhost nginx tenants`).
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
- Later (Stage C): sandbox profiles, budgets, secret references, outbound policy.
@@ -56,11 +56,11 @@ Stage A adds a few safety rules of its own:
@@ -56,11 +56,11 @@ Stage A adds a few safety rules of its own:
| Stage | Scope | State |
| Stage | Scope | State |
|---|---|---|
|---|---|---|
-| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp SQLite/HTTP/git). Installed on the host; used read-only so far. |
+| **A: operator plane** | `ovhost` CLI and library: inventory, validate, plan, deploy (`--wait-idle`/`--force`), automatic rollback, rollback, status, releases, certs, nginx render/install, snapshot, backup, restore drill. No daemon and no server. Port **4910** is reserved for a later operator API. | **alpha**: written and tested (fake host, plus the real executor against temp SQLite/HTTP/git). Installed on the host; used read-only so far. |
| **B: tenant static hosting** | The Host API service (port 4910): projects, sites, immutable content-addressed deploys, activation and rollback, `<site>.openvibe.host` and TXT-verified custom domains, quotas, upload logs, events, a server-rendered dashboard; tenant vhosts via `ovhost nginx tenants`. | **alpha**: written and tested (`npm test`, against a temp database, a mock Network and a DNS table). Not deployed. |
| **B: tenant static hosting** | The Host API service (port 4910): projects, sites, immutable content-addressed deploys, activation and rollback, `<site>.openvibe.host` and TXT-verified custom domains, quotas, upload logs, events, a server-rendered dashboard; tenant vhosts via `ovhost nginx tenants`. | **alpha**: written and tested (`npm test`, against a temp database, a mock Network and a DNS table). Not deployed. |
| **C: sandboxed user code** | Isolation profiles, CPU/memory/time/network/storage budgets, secret references, outbound policy, metering, kill/revoke without touching platform services. | **not started**, deliberately. Blocked until isolation and metering are proven. Nothing in Stage B runs tenant code. |
| **C: sandboxed user code** | Isolation profiles, CPU/memory/time/network/storage budgets, secret references, outbound policy, metering, kill/revoke without touching platform services. | **not started**, deliberately. Blocked until isolation and metering are proven. Nothing in Stage B runs tenant code. |
-Not in Stage A yet (from the charter/roadmap list): container adapters, DNS adapters, certificate **renewal**, logs/metrics links, incident/maintenance controls, the release-manifest/active-client work in §15.18, `host.release.deployed|rolled_back` events (they need an Events outbox and a service principal), and the restore drills and cutover runbook that Wave 22 will add.
+Not in Stage A yet (from the charter/roadmap list): container adapters, DNS adapters, certificate **renewal**, logs/metrics links, incident/maintenance controls, the release-manifest/active-client work in §15.18, `host.release.deployed|rolled_back` events (they need an Events outbox and a service principal), and the Wave 22 cutover runbook. Restore drills (`ovhost drill`) are written and tested against the fake host but have not been run on the host yet.
## Acceptance (Stage A)
## Acceptance (Stage A)
@@ -74,6 +74,7 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak
@@ -74,6 +74,7 @@ What the tests demonstrate (`npm test`, every system call made against `test/fak
- The socket unit is **never stopped or restarted**, in any flow.
- The socket unit is **never stopped or restarted**, in any flow.
- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.
- Only the service being deployed is restarted. Static-only changes (for example Live `public/`) are deployed without a restart, even while streams are live.
- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.
- `nginx -t` failures restore the previous vhost state. `certs` never reads a key file. Snapshots contain no secret values or remote-URL credentials.
+- A **restore drill** restores the latest backup into a service-user-owned temp directory and requires `integrity_check = ok`. It starts a sandboxed second instance through `systemd-run` with the production env file plus an override file. It compares the declared paths and row counts with production, then stops the instance by its own pid and removes the directory. Failed integrity, a readiness timeout, an instance that dies, mismatches and a failed `systemd-run` are all reported, logged and cleaned up. It refuses to run without root or on a port already in use. A filesystem diff of the fake host shows it writes nothing outside the drill directory, its lock and its log, and it never reads the env file or passes on a secret-looking unit `Environment=`.
Not demonstrated yet: any of this on the production host. The Wave 21 exit criterion ("deploy, restart and roll back one service without interrupting unrelated runtimes or protected sessions") still needs a run on the host, with evidence.
Not demonstrated yet: any of this on the production host. The Wave 21 exit criterion ("deploy, restart and roll back one service without interrupting unrelated runtimes or protected sessions") still needs a run on the host, with evidence.
@@ -92,9 +93,10 @@ ovhost nginx render <service> [--variant http|sse|websocket] [--manifest <file>]
@@ -92,9 +93,10 @@ ovhost nginx render <service> [--variant http|sse|websocket] [--manifest <file>]
ovhost nginx tenants <service> [--wildcard-cert <name|dir>] [--install] Stage B tenant vhosts
ovhost nginx tenants <service> [--wildcard-cert <name|dir>] [--install] Stage B tenant vhosts
ovhost snapshot <service> [--out <file>]
ovhost snapshot <service> [--out <file>]
ovhost backup <service>
ovhost backup <service>
+ovhost drill <service> [--backup <dir>] [--keep] restore drill (root only)
```
```
-Every command accepts `--json` and `--inventory <file>`. Exit codes: `0` ok · `1` usage/precondition (including a held lock) · `2` validation failed, nothing restarted · `3` not ready, rolled back and serving · `4` rollback failed, **manual intervention** · `5` protected sessions active (refused, or `--wait-idle` gave up).
+Every command accepts `--json` and `--inventory <file>`. `drill` exits `0` passed, `1` refused (not root, port in use, no backup, unsupported), `2` failed. Other exit codes: `0` ok · `1` usage/precondition (including a held lock) · `2` validation failed, nothing restarted · `3` not ready, rolled back and serving · `4` rollback failed, **manual intervention** · `5` protected sessions active (refused, or `--wait-idle` gave up).
### Deploy sequence
### Deploy sequence
@@ -109,6 +111,25 @@ Every command accepts `--json` and `--inventory <file>`. Exit codes: `0` ok · `
@@ -109,6 +111,25 @@ Every command accepts `--json` and `--inventory <file>`. Exit codes: `0` ok · `
9. If readiness fails, run the automatic rollback (restore, reinstall, restart, poll).
9. If readiness fails, run the automatic rollback (restore, reinstall, restart, poll).
10. Append the release record to `<stateDir>/releases/<service>.jsonl`.
10. Append the release record to `<stateDir>/releases/<service>.jsonl`.
+### Restore drills (Wave 22)
+
+`sudo ovhost drill <service>` restores the latest `ovhost backup` (or `--backup <dir>`) and starts a second instance on a spare loopback port. It compares that instance with production, stops it and logs the result to `<stateDir>/drills/<service>.jsonl`. It also prints a Markdown row for [docs/restore-drills.md](docs/restore-drills.md), which covers the steps, the sandbox and the per-service status.
+
+Each inventory entry's `drill` block declares:
+
+- `port`: the service port + 10000 in the example;
+- `databases`: the env var that points the service at each restored copy;
+- `env`: overrides that turn side effects off. Values may use `{tmp}`, `{port}` and `{db:<name>}`;
+- `dirs` to create inside the drill directory;
+- `ready`: the readiness path;
+- `compare`: paths, plus volatile `ignore` keys where needed;
+- `counts`: `{ db, table }` pairs;
+- `supported: false` with a `reason` when a second instance cannot run without side effects.
+
+In `host.example.json`, 18 services have drill blocks. live, media, tools and games are marked unsupported, and the reasons are in the file.
+
+The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
+
### Inventory
### Inventory
`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange` and `nginx`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange` and `nginx`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
@@ -127,7 +148,7 @@ Nothing here has been done yet. These are the steps:
@@ -127,7 +148,7 @@ Nothing here has been done yet. These are the steps:
```
```
The tree must stay world-readable (default 0755/0644). The SQLite worker runs as `ubuntu` from this directory.
The tree must stay world-readable (default 0755/0644). The SQLite worker runs as `ubuntu` from this directory.
2. **Inventory:** `sudo install -o root -g root -m 0640 host.example.json /etc/openvibe/host.json`, then correct it against the machine. Check the Community database path, the Tools unit list, and Events, which was not deployed as of 22 Sep.
2. **Inventory:** `sudo install -o root -g root -m 0640 host.example.json /etc/openvibe/host.json`, then correct it against the machine. Check the Community database path, the Tools unit list, and Events, which was not deployed as of 22 Sep.
-3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks and snapshots. `/var/backups/openvibe/<service>` is created per service, owned by the service user, so the backup worker can write there. Both directories are created on first use.
+3. **State:** `/var/lib/openvibe-host` (root, 0750) holds the release logs, locks, snapshots and drill logs. `/var/lib/openvibe-drills` (root, 0711) holds one directory per running drill, owned by the service user and removed afterwards. `/var/backups/openvibe/<service>` is created per service, owned by the service user, so the backup worker can write there. Both directories are created on first use.
4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm/SQLite (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.
4. **Run as root:** `sudo ovhost …`. `ovhost` drops to `ubuntu` for git/npm/SQLite (`runuser`) and needs root for `systemctl`, `nginx -t`, reading `/etc/openvibe/*.env` (names only) and writing vhosts. For least privilege, give operator accounts a sudoers rule for `/usr/local/bin/ovhost` alone instead of a general `ALL`. Do **not** grant it to the `ubuntu` service user if that account has no sudo today.
5. **First run:** `sudo ovhost status`, then `sudo ovhost validate <each service>`, then `sudo ovhost plan live`, all read-only apart from `git fetch`. Adopt `deploy` one service at a time, starting with one that has no protected sessions.
5. **First run:** `sudo ovhost status`, then `sudo ovhost validate <each service>`, then `sudo ovhost plan live`, all read-only apart from `git fetch`. Adopt `deploy` one service at a time, starting with one that has no protected sessions.
@@ -27,7 +27,7 @@
@@ -27,7 +27,7 @@
"host.domain.verified"
"host.domain.verified"
],
],
"stages": {
"stages": {
- "A": "alpha: ovhost CLI installed on the production host 2026-09-23 (/usr/local/bin/ovhost, inventory /etc/openvibe/host.json); used read-only (status, validate) so far \u2014 no service deploy has moved to it",
+ "A": "alpha: ovhost CLI installed on the production host 2026-09-23 (/usr/local/bin/ovhost, inventory /etc/openvibe/host.json); used read-only (status, validate) so far — no service deploy has moved to it; restore drills (`ovhost drill`, Wave 22) written and tested against the fake host, not yet run on the host",
"B": "alpha: tenant static hosting service written and tested (tenant isolation, traversal/symlink refusal, quotas, atomic rollback, TXT-verified custom domains, no secrets in responses); not deployed",
"B": "alpha: tenant static hosting service written and tested (tenant isolation, traversal/symlink refusal, quotas, atomic rollback, TXT-verified custom domains, no secrets in responses); not deployed",
"C": "not started (deliberately: blocked until isolation and metering are proven)"
"C": "not started (deliberately: blocked until isolation and metering are proven)"
},
},
@@ -19,5 +19,95 @@ The production instance is never touched.
@@ -19,5 +19,95 @@ The production instance is never touched.
5. Stop the drill instance by its port's pid (not `pkill -f`, which can match the operator's own shell),
5. Stop the drill instance by its port's pid (not `pkill -f`, which can match the operator's own shell),
delete the temp directory.
delete the temp directory.
-Open work: an `ovhost drill <service>` command that automates the steps above for every service with
-declared databases and read-only comparison paths in the inventory.
+Looking back at that manual run: `EVENTS_URL=` had no effect, because Community has no events relay. The
+instance could also still reach production Network over loopback (JWKS, service tokens, identity
+resolve-batch). The inventory's `community` drill block now also empties `OV_OAUTH_CLIENT_SECRET` and
+points `OV_LIVE_INTERNAL_URL` and `OV_MEDIA_INTERNAL_URL` at a closed port.
+
+## Running a drill with `ovhost drill`
+
+```
+sudo ovhost backup community # a fresh backup keeps the comparison meaningful
+sudo ovhost drill community # restore, start, compare, stop, clean up, log
+sudo ovhost drill community --keep # keep the drill directory (copy, drill.env, drill.log)
+sudo ovhost drill community --backup /var/backups/openvibe/community/20260923-034341
+```
+
+The command does the steps above automatically for any service whose inventory entry has a `drill`
+block (see `host.example.json`):
+
+1. **Restore.** The command picks the latest `ovhost backup` from `<stateDir>/backups/<service>.jsonl`,
+ or the `--backup` directory. It copies every database named in `drill.databases` into
+ `/var/lib/openvibe-drills/<service>-<stamp>/db/`, a directory owned by the service user (mode 0700).
+ The copy is made as the service user. `PRAGMA integrity_check` must return `ok` for each copy;
+ otherwise the drill fails before anything starts.
+2. **Start.** A second instance starts from the production checkout through `systemd-run`, as the
+ service user. It uses the production unit's `ExecStart` and `WorkingDirectory`, the production
+ unit's non-secret `Environment=` values and the production env file. After that it loads
+ `drill.env`, which holds the drill port, the restored database paths and the switches that turn
+ side effects off. `drill.env` holds inventory values only and is loaded last, so its values win.
+ ovhost never reads the env file's values.
+3. **Sandbox.** The instance runs with:
+ - `ProtectSystem=strict`, with only the drill directory writable;
+ - `PrivateTmp`, `ProtectHome=read-only` and `NoNewPrivileges`;
+ - binding allowed only on its drill port (`SocketBindAllow`);
+ - no addresses beyond loopback (`IPAddressDeny=any`, `IPAddressAllow=localhost`);
+ - `RuntimeMaxSec`, which stops it even if ovhost dies.
+
+ Loopback stays open because services fetch Network's public keys there. Production services
+ are therefore still reachable on loopback, and the inventory's overrides are what keep the drill
+ away from them.
+4. **Compare.** Once the instance answers its readiness path, every `drill.compare` path is fetched
+ from production and from the drill instance. Bodies must match byte for byte, or as JSON with the
+ listed volatile keys removed. Statuses must match, and production must answer 2xx. Each
+ `drill.counts` table is counted in production and in the restored copy.
+5. **Stop and record.** The instance's own MainPID gets SIGTERM, then SIGKILL after 20 s. ovhost
+ never uses `pkill -f` and never signals a production pid. The drill directory is removed unless
+ `--keep` is set, or unless the instance could not be confirmed stopped. The result is appended to
+ `/var/lib/openvibe-host/drills/<service>.jsonl`, and a row for the table above is printed.
+
+Exit codes are `0` passed, `1` refused, and `2` failed. A refusal happens when ovhost is not run as
+root, the port is in use, there is no backup, or the service's drill is unsupported. A failure is
+integrity, readiness, a mismatch, or a stop that did not complete. Every failure is still logged and
+cleaned up.
+
+A drill compares the backup with production *now*. If writes landed after the backup, row counts and
+list endpoints differ, and the drill reports that honestly. Take the backup right before the drill.
+
+### Services
+
+Supported with declared overrides (not yet run on the host with `ovhost drill`):
+
+- network
+- community
+- events
+- host (the Host API: `host.db` only, counts only)
+- billing
+- chat
+- search
+- sources
+- wiki
+- blog
+- tips
+- vip
+- news
+- reviews
+- deals
+- coupons
+- trade
+- codes
+
+Services that are not deployed yet (everything except network, live, media, tools, community and
+games as of 22 Sep) have entries built from their repositories' `deploy/` directories.
+
+Marked `drill: { supported: false }`, with the reason in the inventory:
+
+- **live:** `data/analytics.db` is opened relative to the checkout at boot. It is read-only in the
+ sandbox, and without the sandbox it would be production's file. RTMP always binds. Restreams, relays,
+ AI jobs and Media reconcilers resume from rows in the copy.
+- **media:** webhooks to Live take their URL from the `apps` table and have no env switch. The health
+ job, junk sweep and clip jobs run ffmpeg against production files through absolute paths stored in
+ the database.
+- **tools:** eight units. yt, food and maps open `analytics.db` relative to the checkout. The job
+ runtimes re-queue and prune with no switch.
+- **games:** its runtime side effects have not been reviewed.
@@ -3,6 +3,8 @@
@@ -3,6 +3,8 @@
"host": "openvibe-oregon",
"host": "openvibe-oregon",
"stateDir": "/var/lib/openvibe-host",
"stateDir": "/var/lib/openvibe-host",
"backupDir": "/var/backups/openvibe",
"backupDir": "/var/backups/openvibe",
+ "drillDir": "/var/lib/openvibe-drills",
+ "_drill": "Restore drills (ovhost drill <service>): each service with a database has a drill block. The drill port is the service port + 10000 on 127.0.0.1. http://127.0.0.1:9 is a closed port, used where an empty value would fall back to a production service on loopback. An empty OV_OAUTH_CLIENT_SECRET stops a service from minting service tokens, so it cannot call other services with its identity. Loopback stays reachable from the drill sandbox (services need Network's public keys), so these overrides are what keep a drill away from production. _notes explain what is left on and why.",
"nginx": {
"nginx": {
"sitesAvailable": "/etc/nginx/sites-available",
"sitesAvailable": "/etc/nginx/sites-available",
"sitesEnabled": "/etc/nginx/sites-enabled",
"sitesEnabled": "/etc/nginx/sites-enabled",
@@ -25,7 +27,24 @@
@@ -25,7 +27,24 @@
"ready": { "url": "http://127.0.0.1:4000/api/health", "timeoutSeconds": 30 },
"ready": { "url": "http://127.0.0.1:4000/api/health", "timeoutSeconds": 30 },
"noRestartPaths": ["docs/", "test/", "**/*.md"],
"noRestartPaths": ["docs/", "test/", "**/*.md"],
"databases": [{ "name": "network", "path": "/opt/openvibe.network/data/network.db" }],
"databases": [{ "name": "network", "path": "/opt/openvibe.network/data/network.db" }],
- "nginx": { "vhost": "openvibe.network.conf", "repoVhost": "deploy/nginx/openvibe.network.conf", "variant": "http" }
+ "nginx": { "vhost": "openvibe.network.conf", "repoVhost": "deploy/nginx/openvibe.network.conf", "variant": "http" },
+ "drill": {
+ "port": 14000,
+ "ready": "/api/ready",
+ "databases": { "network": "DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "OV_EVENTS_INTERNAL_URL": "", "INTERNAL_API_KEY": "", "OV_TOOLS_INTERNAL_URL": "http://127.0.0.1:9", "AVATAR_PATH": "{tmp}/avatars" },
+ "dirs": ["{tmp}/avatars"],
+ "compare": [
+ "/api/health",
+ "/api/.well-known/jwks",
+ "/api/brand",
+ "/api/themes?limit=500",
+ "/api/domains",
+ "/api/v1/registry/capabilities"
+ ],
+ "counts": [{ "db": "network", "table": "users" }, { "db": "network", "table": "oauth_clients" }, { "db": "network", "table": "wallets" }, { "db": "network", "table": "coin_transactions" }],
+ "_notes": "OV_EVENTS_INTERNAL_URL empty stops the dev_audit relay; INTERNAL_API_KEY empty falls back to the placeholder key, which makes the chrome site-copy job (paid LLM call through Live), avatar import and Live analytics calls return early. No switch: the ecosystem poller GETs every service readiness and release.json on loopback every 60 s (read-only). Discord bot login and the Resend email queue are gated by settings in the copy and reach only remote hosts, which the sandbox blocks. JWT_PRIVATE_KEY/JWT_PUBLIC_KEY come from the production env file and are only read."
+ }
},
},
"live": {
"live": {
"repo": "/opt/openvibe.live",
"repo": "/opt/openvibe.live",
@@ -55,7 +74,11 @@
@@ -55,7 +74,11 @@
],
],
"_databasesNote": "analytics.db (1.7 GB, append-only pageviews) is left out of routine backups on purpose; add it here to include it.",
"_databasesNote": "analytics.db (1.7 GB, append-only pageviews) is left out of routine backups on purpose; add it here to include it.",
"backupOnChange": ["server/db/migrations.js", "server/db/schema.sql"],
"backupOnChange": ["server/db/migrations.js", "server/db/schema.sql"],
- "nginx": { "vhost": "openvibe.live.conf", "repoVhost": "deploy/nginx/openvibe.live.conf", "variant": "websocket", "wsPaths": ["/ws/"], "maxBody": "3m" }
+ "nginx": { "vhost": "openvibe.live.conf", "repoVhost": "deploy/nginx/openvibe.live.conf", "variant": "websocket", "wsPaths": ["/ws/"], "maxBody": "3m" },
+ "drill": {
+ "supported": false,
+ "reason": "server/index.js opens data/analytics.db relative to the checkout (not movable by env) at module load: in the sandbox the checkout is read-only and boot fails, and without the sandbox the drill would write production's analytics database. It also always binds RTMP (RTMP_PORT and RTMP_PORT+8000), resumes restreams/RobotStreamer relays and AI jobs for rows in the copy, and runs Media reconcilers; several have no env switch. Needs a BindPaths over data/ plus DB-level switches before it can be drilled."
+ }
},
},
"media": {
"media": {
"repo": "/opt/openvibe.media",
"repo": "/opt/openvibe.media",
@@ -74,7 +97,11 @@
@@ -74,7 +97,11 @@
},
},
"drain": { "policy": "refuse", "waitMaxSeconds": 43200, "pollSeconds": 60, "quietChecks": 2 },
"drain": { "policy": "refuse", "waitMaxSeconds": 43200, "pollSeconds": 60, "quietChecks": 2 },
"databases": [{ "name": "media", "path": "/opt/openvibe.media/data/media.db" }],
"databases": [{ "name": "media", "path": "/opt/openvibe.media/data/media.db" }],
- "nginx": { "vhost": "openvibe.media.conf", "repoVhost": "deploy/nginx/openvibe.media.conf", "variant": "http" }
+ "nginx": { "vhost": "openvibe.media.conf", "repoVhost": "deploy/nginx/openvibe.media.conf", "variant": "http" },
+ "drill": {
+ "supported": false,
+ "reason": "Webhooks to Live (orphan-recording finalize at boot, clip re-cuts, storage alerts) read their URL from the apps table and have no env switch; the health job, junk sweep and clip jobs run ffmpeg against production files through absolute paths stored in the database; the tiering sweep and health job start on a timer with no switch. A side-effect-free instance needs the copy edited (webhook_url NULL, storage_tier off) first, which ovhost does not do."
+ }
},
},
"tools": {
"tools": {
"repo": "/opt/openvibe.tools",
"repo": "/opt/openvibe.tools",
@@ -102,7 +129,11 @@
@@ -102,7 +129,11 @@
{ "name": "tools-maps-analytics", "path": "/opt/openvibe.tools/apps/maps/data/analytics.db" },
{ "name": "tools-maps-analytics", "path": "/opt/openvibe.tools/apps/maps/data/analytics.db" },
{ "name": "tools-text-analytics", "path": "/opt/openvibe.tools/apps/text/data/analytics.db" },
{ "name": "tools-text-analytics", "path": "/opt/openvibe.tools/apps/text/data/analytics.db" },
{ "name": "tools-yt-analytics", "path": "/opt/openvibe.tools/apps/yt/data/analytics.db" }
{ "name": "tools-yt-analytics", "path": "/opt/openvibe.tools/apps/yt/data/analytics.db" }
- ]
+ ],
+ "drill": {
+ "supported": false,
+ "reason": "Eight units (gateway plus seven apps) run as separate processes and a drill starts one; yt, food and maps open apps/<app>/data/analytics.db relative to the checkout with no env override (read-only in the sandbox, production's file without it); the job runtimes prune and re-queue jobs at start with no switch. Its only databases are per-app analytics."
+ }
},
},
"community": {
"community": {
"repo": "/opt/openvibe.community",
"repo": "/opt/openvibe.community",
@@ -118,7 +149,19 @@
@@ -118,7 +149,19 @@
{ "name": "community-legacy-path", "path": "/opt/openvibe.community/data/community.db" }
{ "name": "community-legacy-path", "path": "/opt/openvibe.community/data/community.db" }
],
],
"_databasesNote": "The repo unit puts the database in /var/lib/openvibe-community; the 22 Sep production snapshot found it under /opt/openvibe.community/data. Keep the one that exists (backup skips a missing path).",
"_databasesNote": "The repo unit puts the database in /var/lib/openvibe-community; the 22 Sep production snapshot found it under /opt/openvibe.community/data. Keep the one that exists (backup skips a missing path).",
- "nginx": { "vhost": "openvibe.community.conf", "repoVhost": "deploy/nginx/openvibe.community.conf", "variant": "http" }
+ "nginx": { "vhost": "openvibe.community.conf", "repoVhost": "deploy/nginx/openvibe.community.conf", "variant": "http" },
+ "drill": {
+ "port": 14200,
+ "databases": { "community": "COMMUNITY_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "DISCORD_RELAY_ENABLED": "0", "OV_OAUTH_CLIENT_SECRET": "", "OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9", "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9" },
+ "compare": [
+ "/api/pastes?limit=5",
+ "/api/v1/spaces",
+ "/api/v1/pulse?limit=5"
+ ],
+ "counts": [{ "db": "community", "table": "pastes" }, { "db": "community", "table": "spaces" }, { "db": "community", "table": "threads" }, { "db": "community", "table": "posts" }],
+ "_notes": "If the host keeps the database at the legacy path, map \"community-legacy-path\" instead of \"community\" (and in counts). An empty client secret stops identity resolve-batch calls to Network; the dead Live URL makes a PASTES_AUTHORITY=live proxy fail visibly instead of matching production through production. Community has no events relay (EVENTS_URL is unused)."
+ }
},
},
"events": {
"events": {
"_note": "Not deployed as of the 22 Sep 2026 production snapshot (no events.env on the host).",
"_note": "Not deployed as of the 22 Sep 2026 production snapshot (no events.env on the host).",
@@ -138,7 +181,14 @@
@@ -138,7 +181,14 @@
},
},
"drain": { "policy": "report" },
"drain": { "policy": "report" },
"databases": [{ "name": "events", "path": "/var/lib/openvibe-events/events.db" }],
"databases": [{ "name": "events", "path": "/var/lib/openvibe-events/events.db" }],
- "nginx": { "vhost": "events.openvibe.network.conf", "repoVhost": "deploy/nginx/events.openvibe.network.conf", "variant": "sse", "ssePaths": ["/realtime/stream"], "certName": "openvibe.network" }
+ "nginx": { "vhost": "events.openvibe.network.conf", "repoVhost": "deploy/nginx/events.openvibe.network.conf", "variant": "sse", "ssePaths": ["/realtime/stream"], "certName": "openvibe.network" },
+ "drill": {
+ "port": 14300,
+ "databases": { "events": "EVENTS_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_WORKER": "off" },
+ "counts": [{ "db": "events", "table": "events" }, { "db": "events", "table": "subscriptions" }, { "db": "events", "table": "deliveries" }, { "db": "events", "table": "consumer_checkpoints" }],
+ "_notes": "EVENTS_WORKER=off stops signed deliveries to subscribers. /api/ready requires Network JWKS (read-only GET on loopback). The retention prune interval always runs but only touches the copy. No public read endpoint is stable enough to compare; row counts only."
+ }
},
},
"games": {
"games": {
"repo": "/opt/openvibe.games",
"repo": "/opt/openvibe.games",
@@ -148,7 +198,11 @@
@@ -148,7 +198,11 @@
"envFile": "/etc/openvibe/games.env",
"envFile": "/etc/openvibe/games.env",
"port": 8000,
"port": 8000,
"ready": { "url": "http://127.0.0.1:8000/healthz", "timeoutSeconds": 60 },
"ready": { "url": "http://127.0.0.1:8000/healthz", "timeoutSeconds": 60 },
- "databases": [{ "name": "world", "path": "/opt/openvibe.games/data/world.db" }]
+ "databases": [{ "name": "world", "path": "/opt/openvibe.games/data/world.db" }],
+ "drill": {
+ "supported": false,
+ "reason": "Not reviewed: a pnpm workspace with a TypeScript build whose runtime side effects (world simulation loop, realtime ports) have not been read, so no side-effect-free overrides can be declared without guessing."
+ }
},
},
"sites": {
"sites": {
"repo": "/opt/openvibe.sites",
"repo": "/opt/openvibe.sites",
@@ -168,7 +222,317 @@
@@ -168,7 +222,317 @@
"ready": { "url": "http://127.0.0.1:4910/api/ready", "timeoutSeconds": 30 },
"ready": { "url": "http://127.0.0.1:4910/api/ready", "timeoutSeconds": 30 },
"noRestartPaths": ["docs/", "test/", "**/*.md"],
"noRestartPaths": ["docs/", "test/", "**/*.md"],
"databases": [{ "name": "host", "path": "/var/lib/openvibe-host-api/host.db" }],
"databases": [{ "name": "host", "path": "/var/lib/openvibe-host-api/host.db" }],
- "nginx": { "tenants": { "sitesDomain": "openvibe.host", "dashboardDomain": "openvibe.host", "wildcardCert": "openvibe.host", "database": "/var/lib/openvibe-host-api/host.db", "maxUpload": "110m" } }
+ "nginx": { "tenants": { "sitesDomain": "openvibe.host", "dashboardDomain": "openvibe.host", "wildcardCert": "openvibe.host", "database": "/var/lib/openvibe-host-api/host.db", "maxUpload": "110m" } },
+ "drill": {
+ "port": 14910,
+ "databases": { "host": "HOST_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "HOST_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "HOST_STORAGE_DIR": "{tmp}/objects" },
+ "dirs": ["{tmp}/objects"],
+ "counts": [{ "db": "host", "table": "host_projects" }, { "db": "host", "table": "host_sites" }, { "db": "host", "table": "host_deploys" }, { "db": "host", "table": "host_domains" }],
+ "_notes": "Restores host.db only: `ovhost backup host` does not copy the content-addressed objects, so tenant content cannot be compared (every /api/v1 read needs auth). HOST_WORKER=off stops DNS rechecks and the object sweep."
+ }
+ },
+ "billing": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.billing",
+ "units": ["openvibe-billing.service"],
+ "unitSources": { "openvibe-billing.service": "deploy/systemd/openvibe-billing.service" },
+ "envFile": "/etc/openvibe/billing.env",
+ "env": { "required": "from-example" },
+ "port": 4600,
+ "ready": { "url": "http://127.0.0.1:4600/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "billing", "path": "/var/lib/openvibe-billing/billing.db" }],
+ "drill": {
+ "port": 14600,
+ "databases": { "billing": "BILLING_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "BILLING_JOBS": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "" },
+ "compare": ["/api/v1/rates"],
+ "counts": [{ "db": "billing", "table": "accounts" }, { "db": "billing", "table": "transactions" }, { "db": "billing", "table": "ledger_entries" }, { "db": "billing", "table": "subscriptions" }, { "db": "billing", "table": "payment_intents" }],
+ "_notes": "BILLING_JOBS=off stops provider-event settlement, the renewal sweep and the outbox relay. Payment providers are called only on incoming requests (and are remote, which the sandbox blocks). /api/health is not compared: it reports the relay state, which the drill turns off."
+ }
+ },
+ "chat": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.chat",
+ "units": ["openvibe-chat.service"],
+ "unitSources": { "openvibe-chat.service": "deploy/systemd/openvibe-chat.service" },
+ "envFile": "/etc/openvibe/chat.env",
+ "env": { "required": "from-example" },
+ "port": 4400,
+ "ready": { "url": "http://127.0.0.1:4400/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "chat", "path": "/var/lib/openvibe-chat/chat.db" }],
+ "drill": {
+ "port": 14400,
+ "databases": { "chat": "CHAT_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "LIVE_MIRROR": "0", "OV_OAUTH_CLIENT_SECRET": "", "OV_LIVE_INTERNAL_URL": "http://127.0.0.1:9", "CHAT_CACHE_DIR": "{tmp}/cache", "SOUNDS_PATH": "{tmp}/sounds" },
+ "dirs": ["{tmp}/cache", "{tmp}/sounds"],
+ "compare": ["/api/chat/filters/friendly"],
+ "counts": [{ "db": "chat", "table": "dm_conversations" }, { "db": "chat", "table": "dm_messages" }, { "db": "chat", "table": "emotes" }, { "db": "chat", "table": "moderation_actions" }],
+ "_notes": "An empty client secret and a dead Live URL stop the 2 s Live projection sync and the effects POSTed to Live. History endpoints are not compared: cosmetics come from Live (cut off here) and expired messages are swept at boot, so chat_messages is not counted either."
+ }
+ },
+ "search": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.search",
+ "units": ["openvibe-search.service"],
+ "unitSources": { "openvibe-search.service": "deploy/systemd/openvibe-search.service" },
+ "envFile": "/etc/openvibe/search.env",
+ "env": { "required": "from-example" },
+ "port": 4710,
+ "ready": { "url": "http://127.0.0.1:4710/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "search", "path": "/var/lib/openvibe-search/search.db" }],
+ "drill": {
+ "port": 14710,
+ "databases": { "search": "SEARCH_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "SEARCH_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "" },
+ "compare": [
+ "/api/v1/search?q=openvibe&limit=10",
+ "/api/v1/suggest?q=open&limit=10"
+ ],
+ "counts": [{ "db": "search", "table": "documents" }, { "db": "search", "table": "doc_acl" }, { "db": "search", "table": "doc_facets" }],
+ "_notes": "/api/ready needs the Network public key (read-only JWKS GET on loopback). No switch: the 6-hourly outbox prune (copy only)."
+ }
+ },
+ "sources": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.sources",
+ "units": ["openvibe-sources.service"],
+ "unitSources": { "openvibe-sources.service": "deploy/systemd/openvibe-sources.service" },
+ "envFile": "/etc/openvibe/sources.env",
+ "env": { "required": "from-example" },
+ "port": 4720,
+ "ready": { "url": "http://127.0.0.1:4720/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "sources", "path": "/var/lib/openvibe-sources/sources.db" }],
+ "drill": {
+ "port": 14720,
+ "databases": { "sources": "SOURCES_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "SOURCES_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "" },
+ "counts": [{ "db": "sources", "table": "sources" }, { "db": "sources", "table": "items" }, { "db": "sources", "table": "item_revisions" }, { "db": "sources", "table": "fetch_runs" }],
+ "_notes": "SOURCES_WORKER=off stops the crawler (feeds, sitemaps, robots.txt; production may allow loopback sources). Every /api/v1 read needs a service token, so row counts only."
+ }
+ },
+ "wiki": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.wiki",
+ "units": ["openvibe-wiki.service"],
+ "unitSources": { "openvibe-wiki.service": "deploy/systemd/openvibe-wiki.service" },
+ "envFile": "/etc/openvibe/wiki.env",
+ "env": { "required": "from-example" },
+ "port": 4800,
+ "ready": { "url": "http://127.0.0.1:4800/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "wiki", "path": "/var/lib/openvibe-wiki/wiki.db" }],
+ "drill": {
+ "port": 14800,
+ "databases": { "wiki": "WIKI_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_MEDIA_INTERNAL_URL": "", "OV_SOURCES_INTERNAL_URL": "", "WIKI_SCHEDULE_INTERVAL_MS": "2147483647" },
+ "compare": [
+ "/api/v1/spaces",
+ "/feed.json"
+ ],
+ "counts": [{ "db": "wiki", "table": "wiki_spaces" }, { "db": "wiki", "table": "wiki_pages" }, { "db": "wiki", "table": "wiki_page_revisions" }],
+ "_notes": "The schedule worker has no off switch; the interval is set to the largest value Node accepts (anything larger becomes 1 ms). An empty client secret also stops Community thread creation on page views."
+ }
+ },
+ "blog": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.blog",
+ "units": ["openvibe-blog.service"],
+ "unitSources": { "openvibe-blog.service": "deploy/systemd/openvibe-blog.service" },
+ "envFile": "/etc/openvibe/blog.env",
+ "env": { "required": "from-example" },
+ "port": 4810,
+ "ready": { "url": "http://127.0.0.1:4810/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "blog", "path": "/var/lib/openvibe-blog/blog.db" }],
+ "drill": {
+ "port": 14810,
+ "databases": { "blog": "BLOG_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "BLOG_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_MEDIA_INTERNAL_URL": "http://127.0.0.1:9", "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9" },
+ "compare": [
+ "/api/v1/blogs/openvibe/posts?limit=10",
+ "/feed.json"
+ ],
+ "counts": [{ "db": "blog", "table": "blogs" }, { "db": "blog", "table": "blog_posts" }, { "db": "blog", "table": "blog_post_revisions" }],
+ "_notes": "An empty client secret stops Community thread creation and Network identity lookups on page reads; author names then come from the subject_projections cache in the copy."
+ }
+ },
+ "tips": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.tips",
+ "units": ["openvibe-tips.service"],
+ "unitSources": { "openvibe-tips.service": "deploy/systemd/openvibe-tips.service" },
+ "envFile": "/etc/openvibe/tips.env",
+ "env": { "required": "from-example" },
+ "port": 4610,
+ "ready": { "url": "http://127.0.0.1:4610/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "tips", "path": "/var/lib/openvibe-tips/tips.db" }],
+ "drill": {
+ "port": 14610,
+ "databases": { "tips": "TIPS_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "TIPS_JOBS": "off", "EVENTS_URL": "", "TIPS_EVENTS_SECRET": "", "TIPS_CHAT_ADAPTER": "none", "OV_OAUTH_CLIENT_SECRET": "", "BILLING_URL": "http://127.0.0.1:9", "LIVE_INTERNAL_URL": "http://127.0.0.1:9" },
+ "compare": ["/sitemap.xml"],
+ "counts": [{ "db": "tips", "table": "creator_tip_profiles" }, { "db": "tips", "table": "tip_interactions" }, { "db": "tips", "table": "tip_goals" }],
+ "_notes": "Money: TIPS_JOBS=off stops due Billing transfers and chat deliveries, BILLING_URL points at a closed port (an empty value falls back to production Billing), TIPS_EVENTS_SECRET empty refuses funding webhooks. /api/ready needs the Network public key (read-only JWKS GET on loopback)."
+ }
+ },
+ "vip": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.vip",
+ "units": ["openvibe-vip.service"],
+ "unitSources": { "openvibe-vip.service": "deploy/systemd/openvibe-vip.service" },
+ "envFile": "/etc/openvibe/vip.env",
+ "env": { "required": "from-example" },
+ "port": 4620,
+ "ready": { "url": "http://127.0.0.1:4620/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "vip", "path": "/var/lib/openvibe-vip/vip.db" }],
+ "drill": {
+ "port": 14620,
+ "databases": { "vip": "VIP_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "VIP_JOBS": "off", "EVENTS_URL": "", "VIP_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "BILLING_URL": "http://127.0.0.1:9", "VIP_CHECKOUT_PROVIDERS": "" },
+ "compare": [
+ "/api/v1/creators/network",
+ "/api/v1/plans?creator=network",
+ "/api/v1/perks?creator=network"
+ ],
+ "counts": [{ "db": "vip", "table": "vip_creators" }, { "db": "vip", "table": "vip_plans" }, { "db": "vip", "table": "vip_memberships" }],
+ "_notes": "VIP_FORM_SECRET must stay as production sets it (the service exits without it). BILLING_URL points at a closed port: the Billing client has no off switch and an empty value falls back to production Billing."
+ }
+ },
+ "news": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.news",
+ "units": ["openvibe-news.service"],
+ "unitSources": { "openvibe-news.service": "deploy/systemd/openvibe-news.service" },
+ "envFile": "/etc/openvibe/news.env",
+ "env": { "required": "from-example" },
+ "port": 4820,
+ "ready": { "url": "http://127.0.0.1:4820/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "news", "path": "/var/lib/openvibe-news/news.db" }],
+ "drill": {
+ "port": 14820,
+ "databases": { "news": "NEWS_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "NEWS_WORKER": "off", "EVENTS_URL": "", "NEWS_EVENTS_SECRET": "", "OV_AI_INTERNAL_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9" },
+ "compare": [
+ "/api/v1/stories?limit=10",
+ "/api/v1/topics"
+ ],
+ "counts": [{ "db": "news", "table": "news_topics" }, { "db": "news", "table": "news_stories" }, { "db": "news", "table": "news_source_items" }]
+ }
+ },
+ "reviews": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.reviews",
+ "units": ["openvibe-reviews.service"],
+ "unitSources": { "openvibe-reviews.service": "deploy/systemd/openvibe-reviews.service" },
+ "envFile": "/etc/openvibe/reviews.env",
+ "env": { "required": "from-example" },
+ "port": 4830,
+ "ready": { "url": "http://127.0.0.1:4830/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "reviews", "path": "/var/lib/openvibe-reviews/reviews.db" }],
+ "drill": {
+ "port": 14830,
+ "databases": { "reviews": "REVIEWS_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "REVIEWS_SOURCES_SYNC": "false", "EVENTS_URL": "", "REVIEWS_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "" },
+ "compare": [
+ "/api/v1/entities?limit=10",
+ "/feed.json"
+ ],
+ "counts": [{ "db": "reviews", "table": "review_entities" }, { "db": "reviews", "table": "review_signals" }, { "db": "reviews", "table": "review_summaries" }],
+ "_notes": "reconcileIndex() at boot may write index revisions and outbox rows into the copy (no switch; copy only)."
+ }
+ },
+ "deals": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.deals",
+ "units": ["openvibe-deals.service"],
+ "unitSources": { "openvibe-deals.service": "deploy/systemd/openvibe-deals.service" },
+ "envFile": "/etc/openvibe/deals.env",
+ "env": { "required": "from-example" },
+ "port": 4840,
+ "ready": { "url": "http://127.0.0.1:4840/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "deals", "path": "/var/lib/openvibe-deals/deals.db" }],
+ "drill": {
+ "port": 14840,
+ "databases": { "deals": "DEALS_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "DEALS_WORKER": "off", "DEALS_IMPORT": "off", "EVENTS_URL": "", "DEALS_EVENTS_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_COMMUNITY_INTERNAL_URL": "http://127.0.0.1:9", "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9" },
+ "compare": [{ "path": "/api/v1/offers?sort=new&page=1", "ignore": ["age_ms"] }],
+ "counts": [{ "db": "deals", "table": "deal_offers" }, { "db": "deals", "table": "deal_products" }, { "db": "deals", "table": "deal_votes" }],
+ "_notes": "The client secret MUST be empty: with it, an anonymous GET /d/:slug can create a thread in production Community. Offers carry freshness.age_ms (now - observed_at), ignored in the comparison."
+ }
+ },
+ "coupons": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.coupons",
+ "units": ["openvibe-coupons.service"],
+ "unitSources": { "openvibe-coupons.service": "deploy/systemd/openvibe-coupons.service" },
+ "envFile": "/etc/openvibe/coupons.env",
+ "env": { "required": "from-example" },
+ "port": 4850,
+ "ready": { "url": "http://127.0.0.1:4850/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "coupons", "path": "/var/lib/openvibe-coupons/coupons.db" }],
+ "drill": {
+ "port": 14850,
+ "databases": { "coupons": "COUPONS_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "COUPONS_WORKER": "off", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "" },
+ "compare": [
+ "/sitemaps/merchants.xml",
+ "/feed.json"
+ ],
+ "counts": [{ "db": "coupons", "table": "coupons" }, { "db": "coupons", "table": "coupon_merchants" }, { "db": "coupons", "table": "coupon_merchant_domains" }]
+ }
+ },
+ "trade": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.trade",
+ "units": ["openvibe-trade.service"],
+ "unitSources": { "openvibe-trade.service": "deploy/systemd/openvibe-trade.service" },
+ "envFile": "/etc/openvibe/trade.env",
+ "env": { "required": "from-example" },
+ "port": 4860,
+ "ready": { "url": "http://127.0.0.1:4860/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "trade", "path": "/var/lib/openvibe-trade/trade.db" }],
+ "drill": {
+ "port": 14860,
+ "databases": { "trade": "TRADE_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "TRADE_WORKER": "off", "EVENTS_URL": "", "TRADE_EVENTS_WEBHOOK_SECRET": "", "OV_OAUTH_CLIENT_SECRET": "", "OV_SOURCES_INTERNAL_URL": "http://127.0.0.1:9" },
+ "compare": [
+ "/api/v1/instruments?limit=10",
+ "/feed.json"
+ ],
+ "counts": [{ "db": "trade", "table": "trade_instruments" }, { "db": "trade", "table": "trade_market_observations" }, { "db": "trade", "table": "trade_source_documents" }]
+ }
+ },
+ "codes": {
+ "_note": "Not deployed as of the 22 Sep 2026 production snapshot; unit, env file, port and database path from the repository's deploy/ directory.",
+ "repo": "/opt/openvibe.codes",
+ "units": ["openvibe-codes.service"],
+ "unitSources": { "openvibe-codes.service": "deploy/systemd/openvibe-codes.service" },
+ "envFile": "/etc/openvibe/codes.env",
+ "env": { "required": "from-example" },
+ "port": 4900,
+ "ready": { "url": "http://127.0.0.1:4900/api/ready", "timeoutSeconds": 30 },
+ "noRestartPaths": ["docs/", "test/", "**/*.md"],
+ "databases": [{ "name": "codes", "path": "/var/lib/openvibe-codes/codes.db" }],
+ "drill": {
+ "port": 14900,
+ "databases": { "codes": "CODES_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "" },
+ "compare": ["/sitemap.xml"],
+ "counts": [{ "db": "codes", "table": "manifests" }, { "db": "codes", "table": "releases" }, { "db": "codes", "table": "trust" }],
+ "_notes": "Left on (read-only): JWKS and registry GETs to Network on loopback. The docs pages carry their boot time, so they are not compared."
+ }
}
}
}
}
}
}
@@ -30,13 +30,18 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
@@ -30,13 +30,18 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
database); lists domains still waiting for a certificate
database); lists domains still waiting for a certificate
snapshot <service> [--out <file>] config tarball: units, vhost, env NAMES, sha — no secret values
snapshot <service> [--out <file>] config tarball: units, vhost, env NAMES, sha — no secret values
backup <service> sqlite .backup of the declared databases to a dated directory
backup <service> sqlite .backup of the declared databases to a dated directory
+ drill <service> [--backup <dir>] [--keep]
+ restore drill (root): restore the latest backup into a temp
+ dir, start a sandboxed second instance on the drill port, compare
+ it with production, stop it, log the result
Inventory: --inventory, else $OVHOST_INVENTORY, else /etc/openvibe/host.json.
Inventory: --inventory, else $OVHOST_INVENTORY, else /etc/openvibe/host.json.
Exit codes: 0 ok · 1 usage/precondition · 2 validation failed (nothing restarted) · 3 not ready,
Exit codes: 0 ok · 1 usage/precondition · 2 validation failed (nothing restarted) · 3 not ready,
-rolled back · 4 rollback failed (manual intervention) · 5 protected sessions active
+rolled back · 4 rollback failed (manual intervention) · 5 protected sessions active ·
+drill: 0 passed · 1 precondition (not root, port in use, no backup, unsupported) · 2 drill failed
`;
`;
-const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert']);
+const FLAGS_WITH_VALUE = new Set(['inventory', 'to', 'limit', 'warn-days', 'variant', 'manifest', 'out', 'ready-timeout', 'wildcard-cert', 'backup']);
function parseArgs(argv) {
function parseArgs(argv) {
const pos = [];
const pos = [];
@@ -199,6 +204,16 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -199,6 +204,16 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
if (json) emit(r); else for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);
if (json) emit(r); else for (const f of r.files) out(f.skipped ? `skipped ${f.source}: ${f.skipped}` : `${f.source} → ${f.dest} (${f.bytes} bytes)`);
return 0;
return 0;
}
}
+ case 'drill': {
+ const r = await lib.drill(ctx, needService(), { backup: opts.backup, keep: !!opts.keep });
+ if (json) emit(r.record);
+ else {
+ out(`[ovhost] drill ${r.record.result}${r.record.failure ? ` at ${r.record.failure.stage}: ${r.record.failure.message}` : ''}`);
+ out('Row for docs/restore-drills.md:');
+ out(r.record.markdown);
+ }
+ return r.exitCode;
+ }
default:
default:
throw new UsageError(`unknown command "${cmd}"`);
throw new UsageError(`unknown command "${cmd}"`);
}
}
@@ -206,6 +221,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -206,6 +221,7 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
if (err instanceof UsageError) { out(err.message); return 1; }
if (err instanceof UsageError) { out(err.message); return 1; }
if (err instanceof InventoryError || err instanceof LockedError) { out(`[ovhost] ${err.message}`); return 1; }
if (err instanceof InventoryError || err instanceof LockedError) { out(`[ovhost] ${err.message}`); return 1; }
if (err instanceof SocketGuardError) { out(`[ovhost] ${err.message}`); return 1; }
if (err instanceof SocketGuardError) { out(`[ovhost] ${err.message}`); return 1; }
+ if (err instanceof lib.DrillError) { out(`[ovhost] ${err.message}`); return err.exitCode; }
if (err instanceof lib.OpError) { out(`[ovhost] ✗ ${err.message}`); return err.exitCode; }
if (err instanceof lib.OpError) { out(`[ovhost] ✗ ${err.message}`); return err.exitCode; }
out(`[ovhost] ✗ ${err.message}`);
out(`[ovhost] ✗ ${err.message}`);
return 2;
return 2;
new file mode 100644
new file mode 100644
@@ -0,0 +1,416 @@
@@ -0,0 +1,416 @@
+'use strict';
+/**
+ * ovhost drill <service> [--backup <dir>] [--keep]: a restore drill (roadmap Wave 22). It rebuilds
+ * the service from a backup next to production and checks that the rebuilt copy answers like
+ * production does:
+ *
+ * 1. Take the latest `ovhost backup` of the service (or --backup <dir>). Copy each database the
+ * drill block names into a fresh directory under <drillDir> owned by the service user, and
+ * run PRAGMA integrity_check on each copy. The result must be `ok`.
+ * 2. Start a second instance from the production checkout, as the service user, through
+ * systemd-run. It uses the production unit's ExecStart and WorkingDirectory, the production
+ * env file, and a second env file with the drill's overrides: its port, the restored database
+ * paths, and the switches that turn side effects off. The instance runs sandboxed:
+ * ProtectSystem=strict with only the drill directory writable, a private /tmp, binding only its
+ * drill port, and no addresses beyond loopback unless the inventory says otherwise.
+ * RuntimeMaxSec stops it even if ovhost dies. Loopback stays open (the instance needs Network's
+ * public keys), so the inventory's overrides are what keep it away from production services.
+ * 3. Wait for its readiness path. Then compare the declared read-only GETs with production, byte
+ * for byte or as JSON without the listed volatile keys, and compare declared row counts
+ * (production against the restored copy).
+ * 4. Stop the instance by its own MainPID (never pkill -f), remove the drill directory unless
+ * --keep, append the result to <stateDir>/drills/<service>.jsonl, and return a Markdown row
+ * for docs/restore-drills.md.
+ *
+ * ovhost itself writes only inside the drill directory, the lock file and the drill log. It never
+ * writes to the checkout, the env file, the database or the backup. It never reads an env value:
+ * systemd loads the production env file for the instance, and the override file holds only
+ * inventory values. Environment= values from the unit are passed on only if their names do not look
+ * secret.
+ */
+const path = require('path');
+const { service } = require('../inventory');
+const envfile = require('../envfile');
+const systemd = require('../systemd');
+const lock = require('../lock');
+const { stamp } = require('./backup');
+
+class DrillError extends Error {
+ constructor(message, exitCode = 1) {
+ super(message);
+ this.exitCode = exitCode;
+ }
+}
+
+/** Raised inside the run to jump to cleanup once record.failure is set. */
+class Abort extends Error {}
+
+const STOP_GRACE_MS = 20000;
+
+// ── helpers ──────────────────────────────────────────────────────────────────
+
+function expand(value, vars) {
+ return value.replace(/\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g, (_, k) => {
+ if (k === 'tmp') return vars.tmp;
+ if (k === 'port') return String(vars.port);
+ return vars.db[k.slice(3)];
+ });
+}
+
+/** A line for a systemd EnvironmentFile. Values are quoted only when they need it. */
+function envLine(name, value) {
+ if (value === '') return `${name}=`;
+ if (/^[A-Za-z0-9_./:,@%+=-]+$/.test(value)) return `${name}=${value}`;
+ return `${name}="${value.replace(/["\\$`]/g, '\\$&')}"`;
+}
+
+function environmentProperty(name, value) {
+ const a = `${name}=${value}`;
+ return /[\s"'\\]/.test(a) ? `Environment="${a.replace(/["\\]/g, '\\$&')}"` : `Environment=${a}`;
+}
+
+function stripKeys(v, keys) {
+ if (Array.isArray(v)) return v.map((x) => stripKeys(x, keys));
+ if (v && typeof v === 'object') {
+ const out = {};
+ for (const [k, x] of Object.entries(v)) if (!keys.includes(k)) out[k] = stripKeys(x, keys);
+ return out;
+ }
+ return v;
+}
+
+function diffPaths(a, b, p = '$', out = []) {
+ if (out.length >= 5) return out;
+ if (Array.isArray(a) && Array.isArray(b)) {
+ if (a.length !== b.length) { out.push(`${p}.length ${b.length} ≠ ${a.length}`); return out; }
+ a.forEach((x, i) => diffPaths(x, b[i], `${p}[${i}]`, out));
+ return out;
+ }
+ if (a && b && typeof a === 'object' && typeof b === 'object' && !Array.isArray(a) && !Array.isArray(b)) {
+ for (const k of new Set([...Object.keys(a), ...Object.keys(b)])) {
+ if (!(k in a)) out.push(`${p}.${k} only in the drill instance`);
+ else if (!(k in b)) out.push(`${p}.${k} missing in the drill instance`);
+ else diffPaths(a[k], b[k], `${p}.${k}`, out);
+ if (out.length >= 5) break;
+ }
+ return out;
+ }
+ if (JSON.stringify(a) !== JSON.stringify(b)) out.push(p);
+ return out;
+}
+
+/** Compare one path. `prod` and `drill` are exec.http results. */
+function compareBodies(entry, prod, drill) {
+ const res = { path: entry.path, productionStatus: prod.status || 0, drillStatus: drill.status || 0, match: false };
+ if (!(prod.status >= 200 && prod.status < 300)) { res.detail = `production answered ${prod.status || prod.error || 'nothing'}`; return res; }
+ if (prod.status !== drill.status) { res.detail = `status ${drill.status || drill.error || 'no answer'} ≠ production ${prod.status}`; return res; }
+ if (!entry.ignore.length) {
+ if (prod.body === drill.body) { res.match = true; res.mode = 'bytes'; return res; }
+ const a = Buffer.from(prod.body || '');
+ const b = Buffer.from(drill.body || '');
+ let i = 0;
+ while (i < a.length && i < b.length && a[i] === b[i]) i++;
+ res.detail = `bodies differ: ${b.length} bytes vs production ${a.length}, first difference at byte ${i}`;
+ return res;
+ }
+ let pa;
+ let pb;
+ try { pa = JSON.parse(prod.body); pb = JSON.parse(drill.body); } catch { res.detail = 'ignore keys are declared but a body is not JSON'; return res; }
+ const diffs = diffPaths(stripKeys(pa, entry.ignore), stripKeys(pb, entry.ignore));
+ res.mode = `json without ${entry.ignore.join(', ')}`;
+ if (!diffs.length) { res.match = true; return res; }
+ res.detail = `JSON differs at ${diffs.join('; ')}`;
+ return res;
+}
+
+function countSql(table) { return `SELECT count(*) AS n FROM "${table}"`; }
+
+async function countRows(exec, db, table, as) {
+ const rows = await exec.sqlite(db, countSql(table), { as });
+ const n = Number(rows && rows[0] ? Object.values(rows[0])[0] : NaN);
+ if (!Number.isFinite(n)) throw new Error(`count of ${table} returned no number`);
+ return n;
+}
+
+// ── inputs ───────────────────────────────────────────────────────────────────
+
+/** The backup to restore: --backup <dir>, else the last record in <stateDir>/backups/<id>.jsonl. */
+async function resolveBackup(exec, inv, svc, dirArg) {
+ let dir = dirArg;
+ let takenAt = null;
+ if (dir) {
+ if (!path.isAbsolute(dir)) throw new DrillError('--backup must be an absolute path');
+ dir = path.normalize(dir);
+ } else {
+ const text = await exec.readFile(path.join(inv.stateDir, 'backups', `${svc.id}.jsonl`));
+ const records = String(text || '').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter((r) => r && r.dir);
+ if (!records.length) throw new DrillError(`no ovhost backup recorded for ${svc.id}; run \`ovhost backup ${svc.id}\` first or pass --backup <dir>`);
+ const last = records[records.length - 1];
+ dir = last.dir;
+ takenAt = last.at || null;
+ }
+ const st = await exec.stat(dir);
+ if (!st || !st.isDir) throw new DrillError(`backup directory ${dir} not found`);
+ const files = [];
+ for (const name of Object.keys(svc.drill.databases)) {
+ const db = svc.databases.find((d) => d.name === name);
+ const file = path.join(dir, `${name}${path.extname(db.path) || '.db'}`);
+ const fst = await exec.stat(file);
+ if (!fst || !fst.isFile) throw new DrillError(`backup ${dir} has no copy of the ${name} database (${path.basename(file)})`);
+ files.push({ name, file, production: db.path, envName: svc.drill.databases[name], ext: path.extname(db.path) || '.db' });
+ }
+ return { dir, takenAt, files };
+}
+
+/** How production starts the service: ExecStart, WorkingDirectory and non-secret Environment=. */
+async function startSpec(exec, svc) {
+ const d = svc.drill;
+ if (!svc.units.length && !d.command) throw new DrillError(`${svc.id} has no unit to copy ExecStart from; set drill.command`);
+ if (svc.units.length > 1 && !d.command) throw new DrillError(`${svc.id} runs ${svc.units.length} units; a drill starts one process, so set drill.command (or mark the drill unsupported)`);
+ let directives = { execStart: null, rawExecStart: null, workingDirectory: null, user: null, environment: [] };
+ if (svc.units.length) {
+ const st = await systemd.show(exec, svc.units[0]);
+ const texts = [];
+ for (const f of [st.fragmentPath, ...st.dropIns].filter(Boolean)) {
+ const t = await exec.readFile(f, { privileged: true });
+ if (t != null) texts.push(t);
+ }
+ directives = envfile.unitDirectives(texts);
+ }
+ let argv = d.command;
+ if (!argv) {
+ if (!directives.execStart || !directives.execStart.length) throw new DrillError(`no ExecStart found for ${svc.units[0]}; set drill.command`);
+ if (/[$%]/.test(directives.rawExecStart) || /^[^-:+!]*@/.test(directives.rawExecStart.split(/\s/)[0])) throw new DrillError(`${svc.units[0]} ExecStart uses substitutions or specifiers; set drill.command`);
+ argv = directives.execStart;
+ }
+ if (!path.isAbsolute(argv[0])) throw new DrillError(`the drill command must start with an absolute path (got ${argv[0]}); set drill.command`);
+ const environment = [];
+ const skippedEnvironment = [];
+ for (const e of directives.environment) {
+ if (envfile.SECRET_NAME_RE.test(e.name)) skippedEnvironment.push(e.name);
+ else environment.push(e);
+ }
+ return { argv, cwd: directives.workingDirectory || svc.repo, unitUser: directives.user, environment, skippedEnvironment };
+}
+
+// ── the drill ────────────────────────────────────────────────────────────────
+
+async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
+ const { exec, inv, log } = ctx;
+ const svc = service(inv, id);
+ if (!(await exec.isRoot())) throw new DrillError('ovhost drill must run as root (sudo ovhost drill …): it copies databases as the service user and starts the drill instance through systemd-run');
+ if (!svc.drill) throw new DrillError(`${id} has no drill block in the inventory`);
+ if (!svc.drill.supported) throw new DrillError(`${id} does not support restore drills: ${svc.drill.reason}`);
+ const d = svc.drill;
+ if (d.compare.length && !svc.port) throw new DrillError(`${id} declares compare paths but no production port`);
+ if (svc.envFile && !(await exec.stat(svc.envFile))) throw new DrillError(`${svc.envFile} not found`);
+ const busy = await exec.listeners(d.port);
+ if (busy.length) throw new DrillError(`port ${d.port} is already in use (${busy.map((l) => `${l.process || '?'} pid ${l.pid || '?'}`).join(', ')}); free it or change drill.port`);
+ const bk = await resolveBackup(exec, inv, svc, backupArg);
+ const spec = await startSpec(exec, svc);
+ if (spec.unitUser && spec.unitUser !== svc.runAs) log(`warning: ${svc.units[0]} runs as ${spec.unitUser}, the inventory says ${svc.runAs}; the drill runs as ${svc.runAs}`);
+ if (spec.skippedEnvironment.length) log(`not passing on unit Environment= with secret-looking names: ${spec.skippedEnvironment.join(', ')}`);
+
+ const held = await lock.acquire(exec, inv, `${id}-drill`);
+ const at = stamp(exec.now());
+ const tmp = path.join(inv.drillDir, `${id}-${at}`);
+ const unitName = `ovhost-drill-${id}-${at}.service`;
+ const logFile = path.join(tmp, 'drill.log');
+ const record = {
+ service: id,
+ startedAt: new Date(exec.now()).toISOString(),
+ operator: process.env.SUDO_USER || (await exec.userName()),
+ host: inv.host,
+ backup: bk.dir,
+ backupTakenAt: bk.takenAt,
+ port: d.port,
+ dir: tmp,
+ unit: unitName,
+ pid: null,
+ databases: [],
+ ready: null,
+ compare: [],
+ counts: [],
+ stop: null,
+ kept: false,
+ result: 'failed',
+ failure: null,
+ };
+ const fail = (stage, message) => { if (!record.failure) record.failure = { stage, message }; log(`✗ ${stage}: ${message}`); return new Abort(message); };
+ let created = false;
+ let productionPids = [];
+
+ try {
+ if (await exec.stat(tmp)) throw fail('prepare', `${tmp} already exists`);
+ await exec.mkdir(inv.drillDir, { mode: 0o711 });
+ await exec.mkdir(tmp, { owner: svc.runAs, mode: 0o700 });
+ created = true;
+ await exec.mkdir(path.join(tmp, 'db'), { owner: svc.runAs, mode: 0o700 });
+ const vars = { tmp, port: d.port, db: {} };
+ for (const f of bk.files) vars.db[f.name] = path.join(tmp, 'db', `${f.name}${f.ext}`);
+ for (const dir of d.dirs) await exec.mkdir(expand(dir, vars), { owner: svc.runAs, mode: 0o700 });
+
+ // 1. restore and check every database copy
+ for (const f of bk.files) {
+ const dest = vars.db[f.name];
+ log(`restoring ${f.file} → ${dest} (as ${svc.runAs})`);
+ const cp = await exec.run('cp', ['--no-preserve=mode,ownership', '--', f.file, dest], { as: svc.runAs });
+ if (cp.code !== 0) throw fail('restore', `copying ${f.file} failed: ${(cp.stderr || '').trim().slice(0, 200)}`);
+ const st = await exec.stat(dest);
+ const entry = { name: f.name, source: f.file, copy: dest, bytes: st ? st.size : null, integrity: null };
+ record.databases.push(entry);
+ let rows;
+ try { rows = await exec.sqlite(dest, 'PRAGMA integrity_check', { as: svc.runAs }); } catch (err) { entry.integrity = `error: ${err.message}`; throw fail('integrity', `${f.name}: ${err.message}`); }
+ entry.integrity = (rows || []).map((r) => String(Object.values(r)[0])).join('; ') || 'no result';
+ if (entry.integrity !== 'ok') throw fail('integrity', `${f.name}: integrity_check = ${entry.integrity.slice(0, 300)}`);
+ log(`integrity_check ${f.name}: ok`);
+ }
+ const restoredCounts = [];
+ for (const c of d.counts) restoredCounts.push(await countRows(exec, vars.db[c.db], c.table, svc.runAs));
+
+ // 2. start the drill instance
+ const lines = ['# written by ovhost drill; no secret values: the production env file is loaded separately'];
+ for (const [k, v] of Object.entries(d.env)) lines.push(envLine(k, expand(v, vars)));
+ for (const f of bk.files) lines.push(envLine(f.envName, vars.db[f.name]));
+ const envPath = path.join(tmp, 'drill.env');
+ await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });
+ const props = [
+ 'Type=exec',
+ ...spec.environment.map((e) => environmentProperty(e.name, e.value)),
+ ...(svc.envFile ? [`EnvironmentFile=${svc.envFile}`] : []),
+ `EnvironmentFile=${envPath}`,
+ `StandardOutput=append:${logFile}`,
+ `StandardError=append:${logFile}`,
+ 'ProtectSystem=strict',
+ `ReadWritePaths=${tmp}`,
+ 'PrivateTmp=yes',
+ 'ProtectHome=read-only',
+ 'NoNewPrivileges=yes',
+ `SocketBindAllow=tcp:${d.port}`,
+ 'SocketBindDeny=any',
+ `RuntimeMaxSec=${d.runtimeMaxSeconds}`,
+ 'TimeoutStopSec=15',
+ ...(d.outbound ? [] : ['IPAddressDeny=any', 'IPAddressAllow=localhost']),
+ ];
+ if (d.outbound) log(`warning: the drill instance may reach non-loopback addresses (drill.outbound${d.outboundReason ? `: ${d.outboundReason}` : ''})`);
+ const args = ['--unit', unitName, '--description', `ovhost restore drill of ${id}`, '--collect', '--quiet', `--uid=${svc.runAs}`, `--working-directory=${spec.cwd}`];
+ for (const p of props) args.push('-p', p);
+ args.push('--', ...spec.argv.map((a) => expand(a, vars)));
+ for (const u of svc.units) { const st = await systemd.show(exec, u); if (st.mainPid) productionPids.push(st.mainPid); }
+ log(`starting ${spec.argv.join(' ')} in ${spec.cwd} as ${svc.runAs} on port ${d.port} (${unitName})`);
+ const run = await exec.run('systemd-run', args, { privileged: true });
+ if (run.code !== 0) throw fail('start', `systemd-run failed: ${(run.stderr || '').trim().slice(0, 300)}`);
+ const st = await systemd.show(exec, unitName);
+ if (!st.mainPid) throw fail('start', `the drill instance exited right away (${st.active}/${st.sub}); rerun with --keep and read ${logFile}`);
+ if (productionPids.includes(st.mainPid)) throw fail('start', `pid ${st.mainPid} is production's; not touching it`);
+ record.pid = st.mainPid;
+
+ // 3. readiness, then comparisons
+ const headers = (svc.ready && svc.ready.headers) || {};
+ const readyUrl = `http://127.0.0.1:${d.port}${d.ready}`;
+ const t0 = exec.now();
+ let last = null;
+ record.ready = { url: readyUrl, ok: false };
+ while (exec.now() - t0 < d.readyTimeoutSeconds * 1000) {
+ last = await exec.http(readyUrl, { timeoutMs: 5000, headers });
+ if (last.status >= 200 && last.status < 300) break;
+ if (!(await exec.pidAlive(record.pid))) {
+ record.ready = { url: readyUrl, ok: false, status: last.status || 0, seconds: Math.round((exec.now() - t0) / 1000) };
+ record.pid = null;
+ throw fail('ready', `the drill instance exited before it was ready; rerun with --keep and read ${logFile}`);
+ }
+ await exec.sleep(1000);
+ }
+ record.ready = { url: readyUrl, ok: !!(last && last.status >= 200 && last.status < 300), status: last ? last.status || 0 : 0, seconds: Math.round((exec.now() - t0) / 1000) };
+ if (!record.ready.ok) throw fail('ready', `${readyUrl} not ready after ${d.readyTimeoutSeconds}s (${last ? last.status || last.error : 'never probed'})`);
+ log(`drill instance ready: ${d.ready} ${record.ready.status} after ${record.ready.seconds}s`);
+
+ for (const c of d.compare) {
+ const h = { ...headers, ...c.headers };
+ const prod = await exec.http(`http://127.0.0.1:${svc.port}${c.path}`, { timeoutMs: 10000, headers: h });
+ const drl = await exec.http(`http://127.0.0.1:${d.port}${c.path}`, { timeoutMs: 10000, headers: h });
+ const res = compareBodies(c, prod, drl);
+ record.compare.push(res);
+ log(`${res.match ? 'same' : 'DIFFERENT'} ${c.path}${res.detail ? `: ${res.detail}` : ''}`);
+ }
+ for (let i = 0; i < d.counts.length; i++) {
+ const c = d.counts[i];
+ const db = svc.databases.find((x) => x.name === c.db);
+ let production = null;
+ let error = null;
+ try { production = await countRows(exec, db.path, c.table, svc.runAs); } catch (err) { error = err.message; }
+ const row = { db: c.db, table: c.table, production, restored: restoredCounts[i], match: production === restoredCounts[i] };
+ if (error) row.error = error;
+ record.counts.push(row);
+ log(`${row.match ? 'same' : 'DIFFERENT'} ${c.table}: production ${production == null ? `unknown (${error})` : production}, restored ${restoredCounts[i]}`);
+ }
+ const bad = [...record.compare.filter((r) => !r.match).map((r) => `${r.path} (${r.detail})`), ...record.counts.filter((r) => !r.match).map((r) => `${r.table} ${r.production} ≠ ${r.restored}`)];
+ if (bad.length) throw fail('compare', `differs from production: ${bad.join('; ')}`);
+ } catch (err) {
+ if (!(err instanceof Abort)) fail('error', err.message);
+ }
+
+ // 4. stop, clean up, record
+ record.stop = await stopInstance(exec, record.pid, d.port, productionPids, log);
+ if (record.stop && !record.stop.ok) fail('stop', record.stop.detail);
+ if (created) {
+ const safe = tmp.startsWith(`${inv.drillDir}/`) && /^[a-z][a-z0-9-]*-\d{8}-\d{6}$/.test(path.basename(tmp));
+ if (keep) { record.kept = true; log(`kept ${tmp} (--keep)`); } else if (record.stop && !record.stop.ok) {
+ record.kept = true;
+ log(`kept ${tmp}: the drill instance could not be confirmed stopped`);
+ } else if (safe) {
+ const rm = await exec.run('rm', ['-rf', '--one-file-system', '--', tmp], { privileged: true });
+ if (rm.code !== 0) { record.kept = true; log(`could not remove ${tmp}: ${(rm.stderr || '').trim()}`); }
+ }
+ }
+ record.finishedAt = new Date(exec.now()).toISOString();
+ record.result = record.failure ? 'failed' : 'passed';
+ record.markdown = markdownRow(record);
+ try {
+ await exec.appendFile(path.join(inv.stateDir, 'drills', `${id}.jsonl`), `${JSON.stringify(record)}\n`);
+ } finally {
+ await held.release();
+ }
+ return { exitCode: record.failure ? 2 : 0, record };
+}
+
+/** SIGTERM the drill's own MainPID, then SIGKILL after the grace period. Never anything else. */
+async function stopInstance(exec, pid, port, productionPids, log) {
+ if (!pid) return null;
+ if (productionPids.includes(pid)) return { ok: false, detail: `refusing to signal production pid ${pid}` };
+ log(`stopping the drill instance (pid ${pid})`);
+ await exec.kill(pid, 'SIGTERM');
+ let signal = 'SIGTERM';
+ const t0 = exec.now();
+ while (await exec.pidAlive(pid)) {
+ if (exec.now() - t0 >= STOP_GRACE_MS) {
+ if (signal === 'SIGKILL') return { ok: false, pid, signal, detail: `pid ${pid} survived SIGKILL` };
+ log(`pid ${pid} still running after ${STOP_GRACE_MS / 1000}s; SIGKILL`);
+ await exec.kill(pid, 'SIGKILL');
+ signal = 'SIGKILL';
+ }
+ await exec.sleep(500);
+ }
+ const left = await exec.listeners(port);
+ if (left.length) return { ok: false, pid, signal, detail: `port ${port} is still in use after pid ${pid} stopped (${left.map((l) => l.pid).join(', ')})` };
+ return { ok: true, pid, signal };
+}
+
+function cell(s) { return String(s).replace(/\|/g, '\\|').replace(/\n/g, ' '); }
+
+/** A row for the table in docs/restore-drills.md. */
+function markdownRow(r) {
+ const checks = [];
+ for (const db of r.databases) checks.push(`\`pragma integrity_check\` (${db.name}) = ${db.integrity === 'ok' ? 'ok' : cell(db.integrity).slice(0, 80)}`);
+ if (r.ready) checks.push(r.ready.ok ? `drill instance \`${new URL(r.ready.url).pathname}\` ${r.ready.status} after ${r.ready.seconds}s` : `drill instance not ready (${r.ready.status || 'no answer'})`);
+ const same = r.compare.filter((c) => c.match).map((c) => `\`${c.path}\``);
+ const diff = r.compare.filter((c) => !c.match).map((c) => `\`${c.path}\` differs (${cell(c.detail)})`);
+ if (same.length) checks.push(`${same.join(', ')} identical to production${r.compare.some((c) => c.match && c.mode && c.mode !== 'bytes') ? ' (JSON without volatile keys where declared)' : ''}`);
+ checks.push(...diff);
+ for (const c of r.counts) checks.push(`${c.table} ${c.production == null ? '?' : c.production} ${c.match ? '=' : '≠'} ${c.restored}`);
+ if (r.failure && r.failure.stage !== 'compare') checks.push(`${r.failure.stage}: ${cell(r.failure.message)}`);
+ const when = r.startedAt.replace('T', ' ').slice(0, 16);
+ return `| ${when} | ${r.service} | \`${r.backup}/\` (\`ovhost drill ${r.service}\`) | ${checks.join('; ')} | ${r.result} |`;
+}
+
+module.exports = { drill, DrillError, compareBodies };
@@ -65,6 +65,39 @@ function splitAssignments(s) {
@@ -65,6 +65,39 @@ function splitAssignments(s) {
return out;
return out;
}
}
+/**
+ * The directives a restore drill needs from a unit file and its drop-ins, in systemd's order:
+ * the last ExecStart/WorkingDirectory/User wins (an empty ExecStart= resets it), and Environment=
+ * assignments accumulate. Returns { execStart: argv|null, rawExecStart, workingDirectory, user,
+ * environment: [{ name, value }] }. The caller decides which Environment= values it may pass on —
+ * ovhost never prints them.
+ */
+function unitDirectives(texts) {
+ const out = { execStart: null, rawExecStart: null, workingDirectory: null, user: null, environment: [] };
+ for (const text of texts) {
+ for (const line of String(text || '').split('\n')) {
+ const m = line.match(/^\s*(ExecStart|WorkingDirectory|User|Environment)=(.*)$/);
+ if (!m) continue;
+ const value = m[2].trim();
+ if (m[1] === 'ExecStart') {
+ out.rawExecStart = value || null;
+ out.execStart = value ? splitAssignments(value.replace(/^[-@:+!]+/, '')) : null;
+ } else if (m[1] === 'WorkingDirectory') out.workingDirectory = value.replace(/^-/, '') || null;
+ else if (m[1] === 'User') out.user = value || null;
+ else {
+ for (const part of splitAssignments(value)) {
+ const i = part.indexOf('=');
+ if (i <= 0) continue;
+ const name = part.slice(0, i);
+ out.environment = out.environment.filter((e) => e.name !== name);
+ out.environment.push({ name, value: part.slice(i + 1) });
+ }
+ }
+ }
+ }
+ return out;
+}
+
/**
/**
* Unit text with the value of every secret-looking Environment= assignment replaced by <redacted>.
* Unit text with the value of every secret-looking Environment= assignment replaced by <redacted>.
* Used by snapshots: unit files are config worth keeping, but an operator may have put a secret
* Used by snapshots: unit files are config worth keeping, but an operator may have put a secret
@@ -84,4 +117,4 @@ function redactUnit(unitText) {
@@ -84,4 +117,4 @@ function redactUnit(unitText) {
}).join('\n');
}).join('\n');
}
}
-module.exports = { parseNames, parseExample, unitEnvironmentNames, redactUnit, SECRET_NAME_RE };
+module.exports = { parseNames, parseExample, unitEnvironmentNames, unitDirectives, redactUnit, SECRET_NAME_RE };
@@ -19,6 +19,7 @@
@@ -19,6 +19,7 @@
* http(url, { timeoutMs, headers }) -> { status, body } | { status: 0, error }
* http(url, { timeoutMs, headers }) -> { status, body } | { status: 0, error }
* sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })
* sqlite(db, sql, { as }) -> rows ; sqliteBackup(db, dest, { as })
* listeners(port) -> [{ pid, process }]
* listeners(port) -> [{ pid, process }]
+ * kill(pid, signal) -> bool (signal one pid; used only for a restore drill's own instance)
* sleep(ms) ; now() -> ms ; isRoot() ; userName() ; hostname()
* sleep(ms) ; now() -> ms ; isRoot() ; userName() ; hostname()
*/
*/
const { spawn } = require('child_process');
const { spawn } = require('child_process');
@@ -224,6 +225,10 @@ function createSystemExecutor({ env = process.env } = {}) {
@@ -224,6 +225,10 @@ function createSystemExecutor({ env = process.env } = {}) {
sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,
sqlite: async (db, sql, { as } = {}) => (await sqliteCall({ op: 'query', db, sql }, as)).rows,
sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },
sqliteBackup: async (db, dest, { as } = {}) => { await sqliteCall({ op: 'backup', db, dest }, as); },
listeners,
listeners,
+ kill: async (pid, signal = 'SIGTERM') => {
+ if (!Number.isInteger(pid) || pid <= 1) throw new Error(`refusing to signal pid ${pid}`);
+ try { process.kill(pid, signal); return true; } catch (err) { if (err.code === 'ESRCH') return false; throw err; }
+ },
sleep: (ms) => new Promise((r) => setTimeout(r, ms)),
sleep: (ms) => new Promise((r) => setTimeout(r, ms)),
now: () => Date.now(),
now: () => Date.now(),
isRoot: async () => isRoot(),
isRoot: async () => isRoot(),
@@ -9,6 +9,7 @@ const { validate, envNames } = require('./commands/validate');
@@ -9,6 +9,7 @@ const { validate, envNames } = require('./commands/validate');
const { status } = require('./commands/status');
const { status } = require('./commands/status');
const { backup } = require('./commands/backup');
const { backup } = require('./commands/backup');
const { snapshot } = require('./commands/snapshot');
const { snapshot } = require('./commands/snapshot');
+const { drill, DrillError } = require('./commands/drill');
const releases = require('./releases');
const releases = require('./releases');
const certs = require('./certs');
const certs = require('./certs');
const nginx = require('./nginx');
const nginx = require('./nginx');
@@ -25,6 +26,8 @@ module.exports = {
@@ -25,6 +26,8 @@ module.exports = {
status,
status,
backup,
backup,
snapshot,
snapshot,
+ drill,
+ DrillError,
releases,
releases,
certs,
certs,
nginx,
nginx,
@@ -70,6 +70,7 @@ function normaliseService(id, raw, defaults) {
@@ -70,6 +70,7 @@ function normaliseService(id, raw, defaults) {
}),
}),
backupOnChange: s.backupOnChange || [],
backupOnChange: s.backupOnChange || [],
nginx: s.nginx || null,
nginx: s.nginx || null,
+ drill: null,
};
};
if (typeof out.owner !== 'string' || !/^[a-z_][a-z0-9_-]*$/.test(out.owner)) throw new InventoryError(`${where}.owner must be a user name`);
if (typeof out.owner !== 'string' || !/^[a-z_][a-z0-9_-]*$/.test(out.owner)) throw new InventoryError(`${where}.owner must be a user name`);
if (!/^[a-z_][a-z0-9_-]*$/.test(out.runAs)) throw new InventoryError(`${where}.runAs must be a user name`);
if (!/^[a-z_][a-z0-9_-]*$/.test(out.runAs)) throw new InventoryError(`${where}.runAs must be a user name`);
@@ -110,9 +111,96 @@ function normaliseService(id, raw, defaults) {
@@ -110,9 +111,96 @@ function normaliseService(id, raw, defaults) {
if (n.certName && !FILE_NAME_RE.test(n.certName)) throw new InventoryError(`${where}.nginx.certName must be a name`);
if (n.certName && !FILE_NAME_RE.test(n.certName)) throw new InventoryError(`${where}.nginx.certName must be a name`);
}
}
for (const p of out.backupOnChange) rel(p, `${where}.backupOnChange`);
for (const p of out.backupOnChange) rel(p, `${where}.backupOnChange`);
+ if (s.drill) out.drill = normaliseDrill(s.drill, out, where);
return out;
return out;
}
}
+const ENV_NAME_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
+const TABLE_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
+const PLACEHOLDER_RE = /\{(tmp|port|db:[A-Za-z0-9._-]+)\}/g;
+
+/**
+ * The `drill` block (Wave 22 restore drills): how to start a second, side-effect-free instance of
+ * the service against restored copies of its databases, and what to compare with production.
+ *
+ * supported false (with a reason) when a second instance cannot run safely
+ * port spare loopback port for the drill instance (never the service's own port)
+ * databases { <database name>: <ENV VAR that points the service at that file> }
+ * env { NAME: value } overrides that turn side effects off; values may use {tmp}, {port}
+ * and {db:<name>}
+ * dirs directories to create (owned by the service user) inside {tmp} before the start
+ * command argv to start the service; default: the production unit's ExecStart
+ * ready readiness path; default: the path of the service's ready URL
+ * compare [{ path, ignore: [volatile keys], headers }] read-only GETs compared with production
+ * counts [{ db, table }] row counts compared between production and the restored copy
+ * outbound true lets the instance reach non-loopback addresses (default: loopback only)
+ */
+function normaliseDrill(raw, svc, where) {
+ const w = `${where}.drill`;
+ if (!raw || typeof raw !== 'object') throw new InventoryError(`${w} must be an object`);
+ if (raw.supported === false) {
+ if (typeof raw.reason !== 'string' || !raw.reason.trim()) throw new InventoryError(`${w}.reason is required when supported is false`);
+ return { supported: false, reason: raw.reason };
+ }
+ const port = Number(raw.port);
+ if (!(Number.isInteger(port) && port >= 1024 && port < 65536)) throw new InventoryError(`${w}.port must be a TCP port from 1024`);
+ if (port === svc.port) throw new InventoryError(`${w}.port must differ from the service's own port`);
+ if (!svc.databases.length) throw new InventoryError(`${w}: a drill restores databases, and ${svc.id} declares none`);
+ const dbNames = svc.databases.map((d) => d.name);
+ const databases = raw.databases || {};
+ if (typeof databases !== 'object' || Array.isArray(databases) || !Object.keys(databases).length) throw new InventoryError(`${w}.databases must map at least one database name to an env var name`);
+ for (const [name, envName] of Object.entries(databases)) {
+ if (!dbNames.includes(name)) throw new InventoryError(`${w}.databases: "${name}" is not one of ${svc.id}'s databases (${dbNames.join(', ')})`);
+ if (!ENV_NAME_RE.test(envName)) throw new InventoryError(`${w}.databases.${name} must be an env var name`);
+ }
+ const env = raw.env || {};
+ for (const [k, v] of Object.entries(env)) {
+ if (!ENV_NAME_RE.test(k)) throw new InventoryError(`${w}.env: "${k}" is not an env var name`);
+ if (typeof v !== 'string' || /[\n\r\0]/.test(v)) throw new InventoryError(`${w}.env.${k} must be a one-line string`);
+ for (const m of v.matchAll(PLACEHOLDER_RE)) {
+ if (m[1].startsWith('db:') && !Object.keys(databases).includes(m[1].slice(3))) throw new InventoryError(`${w}.env.${k}: {${m[1]}} is not a restored database`);
+ }
+ }
+ if (!Object.values(env).some((v) => v.includes('{port}')) && !(raw.command || []).some((a) => String(a).includes('{port}'))) throw new InventoryError(`${w}.env must point the instance at its port with {port} (e.g. "PORT": "{port}")`);
+ for (const envName of Object.values(databases)) if (envName in env) throw new InventoryError(`${w}.env.${envName} is already set by drill.databases`);
+ const dirs = raw.dirs || [];
+ for (const d of dirs) if (typeof d !== 'string' || !/^\{tmp\}\/[A-Za-z0-9._/-]+$/.test(d) || d.includes('..')) throw new InventoryError(`${w}.dirs entries must be paths under {tmp}`);
+ if (raw.command != null && (!Array.isArray(raw.command) || !raw.command.length || raw.command.some((a) => typeof a !== 'string'))) throw new InventoryError(`${w}.command must be an argv array`);
+ if (raw.command && !path.isAbsolute(raw.command[0])) throw new InventoryError(`${w}.command[0] must be an absolute path`);
+ let ready = raw.ready || null;
+ if (!ready && svc.ready) ready = new URL(svc.ready.url).pathname;
+ if (!ready || !/^\/[^/]/.test(ready)) throw new InventoryError(`${w}.ready must be a path such as /api/ready (or declare the service's ready URL)`);
+ const compare = (raw.compare || []).map((c, i) => {
+ const cc = typeof c === 'string' ? { path: c } : c;
+ if (!cc || typeof cc.path !== 'string' || !/^\/[^/]/.test(cc.path) || /\s/.test(cc.path)) throw new InventoryError(`${w}.compare[${i}].path must be a path such as /api/items?limit=5`);
+ const ignore = cc.ignore || [];
+ if (!Array.isArray(ignore) || ignore.some((k) => typeof k !== 'string' || !k)) throw new InventoryError(`${w}.compare[${i}].ignore must be a list of key names`);
+ return { path: cc.path, ignore, headers: cc.headers || {} };
+ });
+ const counts = (raw.counts || []).map((c, i) => {
+ if (!c || !Object.keys(databases).includes(c.db)) throw new InventoryError(`${w}.counts[${i}].db must be a restored database`);
+ if (!TABLE_RE.test(c.table || '')) throw new InventoryError(`${w}.counts[${i}].table must be a table name`);
+ return { db: c.db, table: c.table };
+ });
+ if (!compare.length && !counts.length) throw new InventoryError(`${w} needs at least one compare path or row count`);
+ const readyTimeoutSeconds = Number(raw.readyTimeoutSeconds || (svc.ready && svc.ready.timeoutSeconds) || 90);
+ return {
+ supported: true,
+ port,
+ databases,
+ env,
+ dirs,
+ command: raw.command || null,
+ ready,
+ readyTimeoutSeconds,
+ compare,
+ counts,
+ outbound: raw.outbound === true,
+ outboundReason: raw.outboundReason || null,
+ runtimeMaxSeconds: Number(raw.runtimeMaxSeconds || 1800),
+ };
+}
+
function normalise(raw) {
function normalise(raw) {
if (!raw || typeof raw !== 'object' || !raw.services || typeof raw.services !== 'object') throw new InventoryError('inventory needs a "services" object');
if (!raw || typeof raw !== 'object' || !raw.services || typeof raw.services !== 'object') throw new InventoryError('inventory needs a "services" object');
const defaults = raw.defaults || {};
const defaults = raw.defaults || {};
@@ -120,6 +208,7 @@ function normalise(raw) {
@@ -120,6 +208,7 @@ function normalise(raw) {
host: raw.host || null,
host: raw.host || null,
stateDir: abs(raw.stateDir || '/var/lib/openvibe-host', 'stateDir'),
stateDir: abs(raw.stateDir || '/var/lib/openvibe-host', 'stateDir'),
backupDir: abs(raw.backupDir || '/var/backups/openvibe', 'backupDir'),
backupDir: abs(raw.backupDir || '/var/backups/openvibe', 'backupDir'),
+ drillDir: abs(raw.drillDir || '/var/lib/openvibe-drills', 'drillDir'),
nginx: {
nginx: {
sitesAvailable: abs((raw.nginx && raw.nginx.sitesAvailable) || '/etc/nginx/sites-available', 'nginx.sitesAvailable'),
sitesAvailable: abs((raw.nginx && raw.nginx.sitesAvailable) || '/etc/nginx/sites-available', 'nginx.sitesAvailable'),
sitesEnabled: abs((raw.nginx && raw.nginx.sitesEnabled) || '/etc/nginx/sites-enabled', 'nginx.sitesEnabled'),
sitesEnabled: abs((raw.nginx && raw.nginx.sitesEnabled) || '/etc/nginx/sites-enabled', 'nginx.sitesEnabled'),
@@ -131,6 +220,14 @@ function normalise(raw) {
@@ -131,6 +220,14 @@ function normalise(raw) {
if (id.startsWith('_')) continue;
if (id.startsWith('_')) continue;
inv.services[id] = normaliseService(id, svc, defaults);
inv.services[id] = normaliseService(id, svc, defaults);
}
}
+ const ports = new Map(Object.values(inv.services).filter((s) => s.port).map((s) => [s.port, s.id]));
+ const drillPorts = new Map();
+ for (const s of Object.values(inv.services)) {
+ if (!s.drill || !s.drill.supported) continue;
+ if (ports.has(s.drill.port)) throw new InventoryError(`services.${s.id}.drill.port ${s.drill.port} is ${ports.get(s.drill.port)}'s port`);
+ if (drillPorts.has(s.drill.port)) throw new InventoryError(`services.${s.id}.drill.port ${s.drill.port} is also ${drillPorts.get(s.drill.port)}'s drill port`);
+ drillPorts.set(s.drill.port, s.id);
+ }
return inv;
return inv;
}
}
@@ -17,7 +17,7 @@ runTests([
@@ -17,7 +17,7 @@ runTests([
test('host.example.json is a valid inventory for every service on the host', () => {
test('host.example.json is a valid inventory for every service on the host', () => {
const raw = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
const raw = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
const inv = normalise(raw);
const inv = normalise(raw);
- assert.deepStrictEqual(Object.keys(inv.services), ['network', 'live', 'media', 'tools', 'community', 'events', 'games', 'sites', 'host']);
+ assert.deepStrictEqual(Object.keys(inv.services), ['network', 'live', 'media', 'tools', 'community', 'events', 'games', 'sites', 'host', 'billing', 'chat', 'search', 'sources', 'wiki', 'blog', 'tips', 'vip', 'news', 'reviews', 'deals', 'coupons', 'trade', 'codes']);
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
new file mode 100644
new file mode 100644
@@ -0,0 +1,413 @@
@@ -0,0 +1,413 @@
+'use strict';
+/**
+ * ovhost drill against the fake host: the full flow, integrity failure, readiness timeout, an
+ * instance that dies, mismatch reporting, cleanup on failure, refusals (not root, port in use,
+ * unsupported, no backup), and the guarantee that nothing outside the drill directory, the lock and
+ * the drill log is written.
+ */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const { scenario, test, runTests, SECRET } = require('./helpers');
+const { normalise } = require('../lib/inventory');
+const { compareBodies } = require('../lib/commands/drill');
+
+const DRILL_DIR = '/var/lib/openvibe-drills';
+const PROD_DB = '/var/lib/openvibe-community/community.db';
+const PASTES = '[{"id":"pst_1","title":"hello"},{"id":"pst_2","title":"world"}]';
+
+function communityEntry(drillOverrides = {}) {
+ return {
+ repo: '/opt/openvibe.community',
+ units: ['openvibe-community.service'],
+ envFile: '/etc/openvibe/community.env',
+ port: 4200,
+ ready: { url: 'http://127.0.0.1:4200/api/ready', timeoutSeconds: 30 },
+ databases: [{ name: 'community', path: PROD_DB }],
+ drill: {
+ port: 14200,
+ databases: { community: 'COMMUNITY_DB_PATH' },
+ env: { PORT: '{port}', HOST: '127.0.0.1', DISCORD_RELAY_ENABLED: '0', EVENTS_URL: '', UPLOAD_DIR: '{tmp}/data/uploads', GREETING: 'two words' },
+ dirs: ['{tmp}/data'],
+ compare: [{ path: '/api/pastes?limit=5' }, { path: '/api/v1/pulse?limit=5', ignore: ['generated_at'] }],
+ counts: [{ db: 'community', table: 'pastes' }],
+ ...drillOverrides,
+ },
+ };
+}
+
+/**
+ * The standard fake host plus Community in production, with one `ovhost backup community` taken.
+ * The drill instance's behaviour is controlled through `host.drill`.
+ */
+async function drillScenario({ drill: drillOverrides, backup = true } = {}) {
+ const host = scenario();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.community = communityEntry(drillOverrides);
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+
+ const repo = host.createRepo('/opt/openvibe.community', { owner: 'ubuntu' });
+ const sha = repo.commit({ 'package.json': '{"name":"openvibe-community"}', 'server/index.js': 'community();' }, { message: 'initial' });
+ repo.publish(sha);
+ repo.checkout(sha);
+ host.addUnit('openvibe-community.service', { mainPid: 2222 });
+ host.alivePids.add(2222);
+ host.listeners.set(4200, [{ pid: 2222, process: 'node' }]);
+ host.put('/etc/systemd/system/openvibe-community.service', [
+ '[Service]',
+ 'User=ubuntu',
+ 'WorkingDirectory=/opt/openvibe.community',
+ 'ExecStart=/usr/bin/node server/index.js',
+ `Environment=NODE_ENV=production "SESSION_SECRET=${SECRET}-unit"`,
+ '',
+ ].join('\n'));
+ host.put('/etc/openvibe/community.env', `OV_OAUTH_CLIENT_SECRET=${SECRET}\nBASE_URL=https://openvibe.community\n`, { mode: 0o600 });
+ host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
+
+ host.http.set('http://127.0.0.1:4200/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
+ host.http.set('http://127.0.0.1:4200/api/pastes?limit=5', () => ({ status: 200, body: PASTES }));
+ host.http.set('http://127.0.0.1:4200/api/v1/pulse?limit=5', () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:00.000Z' } }));
+
+ host.drill = {
+ integrity: [{ integrity_check: 'ok' }],
+ productionCount: 892,
+ restoredCount: 892,
+ ready: () => ({ status: 200, body: { status: 'ready' } }),
+ pastes: () => ({ status: 200, body: PASTES }),
+ pulse: () => ({ status: 200, body: { items: [{ id: 1 }], generated_at: '2026-09-22T12:00:07.000Z' } }),
+ onStart: null,
+ ignoreTerm: false,
+ ignoreKill: false,
+ };
+ host.sqliteHandler = (db, sql) => {
+ if (/integrity_check/.test(sql)) return host.drill.integrity;
+ if (/count\(\*\) AS n FROM "pastes"/.test(sql)) return [{ n: db === PROD_DB ? host.drill.productionCount : host.drill.restoredCount }];
+ if (/is_recording/.test(sql)) return [{ n: host.recording }];
+ return [{ n: 0 }];
+ };
+ host.onSystemdRun = (spec) => {
+ if (host.drill.onStart) { const r = host.drill.onStart(spec); if (r) return r; }
+ host.listeners.set(14200, [{ pid: spec.pid, process: 'node' }]);
+ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
+ host.http.set('http://127.0.0.1:14200/api/ready', alive(() => host.drill.ready()));
+ host.http.set('http://127.0.0.1:14200/api/pastes?limit=5', alive(() => host.drill.pastes()));
+ host.http.set('http://127.0.0.1:14200/api/v1/pulse?limit=5', alive(() => host.drill.pulse()));
+ return undefined;
+ };
+ host.onKill = (pid, signal) => {
+ if (signal === 'SIGTERM' && host.drill.ignoreTerm) return false;
+ if (signal === 'SIGKILL' && host.drill.ignoreKill) return false;
+ return undefined;
+ };
+
+ if (backup) {
+ const b = await host.cli('backup', 'community', '--json');
+ assert.strictEqual(b.code, 0, b.out);
+ host.backupDir = JSON.parse(b.out).dir;
+ }
+ host.advance(60 * 1000);
+ return host;
+}
+
+/** Every file entry as "type:owner:mode:content", to diff the fake filesystem. */
+function fsSnapshot(host) {
+ const m = new Map();
+ for (const [k, v] of host.files) m.set(k, `${v.type}:${v.owner}:${v.mode}:${v.type === 'file' ? String(v.content) : v.target || ''}`);
+ return m;
+}
+
+function changedPaths(before, after) {
+ const out = [];
+ for (const [k, v] of after) if (before.get(k) !== v) out.push(k);
+ for (const k of before.keys()) if (!after.has(k)) out.push(k);
+ return out.sort();
+}
+
+function lastLog(host) {
+ const text = host.read('/var/lib/openvibe-host/drills/community.jsonl');
+ const lines = text.trim().split('\n');
+ return JSON.parse(lines[lines.length - 1]);
+}
+
+const allowedWrite = (p, tmp) => p === DRILL_DIR || p === tmp || p.startsWith(`${tmp}/`)
+ || p === '/var/lib/openvibe-host/drills' || p === '/var/lib/openvibe-host/drills/community.jsonl'
+ || p === '/var/lib/openvibe-host/locks' || p === '/var/lib/openvibe-host/locks/community-drill.lock';
+
+runTests([
+ test('full flow: restore, integrity, sandboxed second instance, identical answers, stop by pid, cleanup, log row', async () => {
+ const host = await drillScenario();
+ const before = fsSnapshot(host);
+ const callsBefore = host.calls.length;
+ const r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.result, 'passed');
+ assert.strictEqual(rec.failure, null);
+ assert.strictEqual(rec.backup, host.backupDir);
+ const tmp = rec.dir;
+ assert.match(tmp, /^\/var\/lib\/openvibe-drills\/community-\d{8}-\d{6}$/);
+ assert.deepStrictEqual(rec.databases.map((d) => [d.name, d.integrity, d.copy]), [['community', 'ok', `${tmp}/db/community.db`]]);
+ assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/pastes?limit=5', true], ['/api/v1/pulse?limit=5', true]]);
+ assert.deepStrictEqual(rec.counts, [{ db: 'community', table: 'pastes', production: 892, restored: 892, match: true }]);
+ assert.strictEqual(rec.ready.ok, true);
+ assert.strictEqual(rec.stop.ok, true);
+ assert.strictEqual(rec.stop.signal, 'SIGTERM');
+
+ const calls = host.calls.slice(callsBefore);
+ // The copy is made as the service user; integrity and counts run as the service user.
+ const cp = calls.find((c) => c.cmd === 'cp');
+ assert.strictEqual(cp.as, 'ubuntu');
+ assert.deepStrictEqual(cp.args.slice(-2), [`${host.backupDir}/community.db`, `${tmp}/db/community.db`]);
+ const integrity = host.sqliteCalls.find((c) => /integrity_check/.test(c.sql));
+ assert.deepStrictEqual([integrity.db, integrity.as], [`${tmp}/db/community.db`, 'ubuntu']);
+
+ // systemd-run: production ExecStart/WorkingDirectory, production env file then the override file,
+ // sandbox properties, loopback-only networking, the service user.
+ assert.strictEqual(host.systemdRuns.length, 1);
+ const run = host.systemdRuns[0];
+ assert.strictEqual(run.uid, 'ubuntu');
+ assert.strictEqual(run.cwd, '/opt/openvibe.community');
+ assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js']);
+ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/community.env', `${tmp}/drill.env`]);
+ for (const p of ['ProtectSystem=strict', `ReadWritePaths=${tmp}`, 'PrivateTmp=yes', 'NoNewPrivileges=yes', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'SocketBindAllow=tcp:14200', 'SocketBindDeny=any', 'Environment=NODE_ENV=production']) {
+ assert.ok(run.props.includes(p), `systemd-run -p ${p}`);
+ }
+ assert.ok(!run.props.some((p) => p.includes('SESSION_SECRET')), 'a secret-looking unit Environment= is not passed on');
+ const runCall = calls.find((c) => c.cmd === 'systemd-run');
+ assert.strictEqual(runCall.privileged, true);
+
+ // Stopped by its own pid with SIGTERM; production's pid is never signalled; no pkill.
+ const kills = calls.filter((c) => c.cmd === 'kill');
+ assert.deepStrictEqual(kills.map((c) => c.args), [['SIGTERM', run.pid]]);
+ assert.ok(!calls.some((c) => /pkill|killall/.test(c.cmd)));
+ assert.ok(host.alivePids.has(2222), 'production still running');
+ assert.strictEqual(host.units.get('openvibe-community.service').restarts, 0);
+ assert.ok(!host.restarts().length, 'nothing restarted');
+
+ // Only the drill directory (now removed), the lock (released) and the drill log changed.
+ const after = fsSnapshot(host);
+ const changed = changedPaths(before, after);
+ for (const p of changed) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);
+ assert.ok(!host.files.has(tmp), 'temp dir removed');
+ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'), 'lock released');
+
+ // Secrets: nothing ovhost printed, logged or passed on carries a value.
+ const everything = [r.out, host.read('/var/lib/openvibe-host/drills/community.jsonl'), JSON.stringify(host.calls), JSON.stringify(host.systemdRuns)].join('\n');
+ assert.ok(!everything.includes(SECRET), 'a secret value leaked');
+ assert.ok(!host.reads.includes('/etc/openvibe/community.env'), 'ovhost never reads the env file for a drill');
+
+ // Markdown row ready for docs/restore-drills.md.
+ assert.match(r.out, /\| 2026-09-22 12:01 \| community \| `\/var\/backups\/openvibe\/community\/20260922-120000\/` \(`ovhost drill community`\) \| `pragma integrity_check` \(community\) = ok; drill instance `\/api\/ready` 200 after 0s; `\/api\/pastes\?limit=5`, `\/api\/v1\/pulse\?limit=5` identical to production \(JSON without volatile keys where declared\); pastes 892 = 892 \| passed \|/);
+ assert.strictEqual(rec.markdown, r.out.split('\n').pop());
+ }),
+
+ test('--keep leaves the drill directory; the override file holds inventory values only', async () => {
+ const host = await drillScenario();
+ const before = fsSnapshot(host);
+ const r = await host.cli('drill', 'community', '--keep');
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.kept, true);
+ const tmp = rec.dir;
+ assert.strictEqual(host.files.get(tmp).owner, 'ubuntu');
+ assert.strictEqual(host.files.get(tmp).mode, 0o700);
+ assert.strictEqual(host.files.get(`${tmp}/db/community.db`).owner, 'ubuntu');
+ assert.strictEqual(host.read(`${tmp}/db/community.db`), host.read(`${host.backupDir}/community.db`));
+ assert.strictEqual(host.files.get(`${tmp}/data`).owner, 'ubuntu');
+ const env = host.read(`${tmp}/drill.env`);
+ for (const line of ['PORT=14200', 'HOST=127.0.0.1', 'DISCORD_RELAY_ENABLED=0', 'EVENTS_URL=', `UPLOAD_DIR=${tmp}/data/uploads`, 'GREETING="two words"', `COMMUNITY_DB_PATH=${tmp}/db/community.db`]) {
+ assert.ok(env.split('\n').includes(line), `drill.env has ${line}`);
+ }
+ assert.ok(!env.includes(SECRET));
+ for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, tmp), `unexpected write: ${p}`);
+ // The production database and the backup are untouched.
+ assert.strictEqual(host.read(PROD_DB), 'sqlite-production');
+ }),
+
+ test('integrity failure: nothing is started, the copy is removed, the log says failed', async () => {
+ const host = await drillScenario();
+ host.drill.integrity = [{ integrity_check: '*** in database main ***' }, { integrity_check: 'Page 5: btree page has a bad cell' }];
+ const r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.result, 'failed');
+ assert.strictEqual(rec.failure.stage, 'integrity');
+ assert.match(rec.failure.message, /bad cell/);
+ assert.strictEqual(host.systemdRuns.length, 0);
+ assert.ok(!host.files.has(rec.dir));
+ assert.match(r.out, /\| failed \|$/);
+ }),
+
+ test('readiness timeout: the instance is stopped by its pid and the directory removed', async () => {
+ const host = await drillScenario();
+ host.drill.ready = () => ({ status: 503, body: { status: 'not_ready' } });
+ const r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'ready');
+ assert.match(rec.failure.message, /not ready after 30s \(503\)/);
+ assert.strictEqual(rec.compare.length, 0, 'nothing compared');
+ const pid = host.systemdRuns[0].pid;
+ assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', pid]]);
+ assert.ok(!host.alivePids.has(pid));
+ assert.ok(!host.files.has(rec.dir));
+ assert.ok(!host.listeners.has(14200));
+ }),
+
+ test('an instance that exits before it is ready fails the drill without signalling anything', async () => {
+ const host = await drillScenario();
+ let probes = 0;
+ host.drill.ready = () => { probes += 1; if (probes === 2) host.alivePids.delete(host.systemdRuns[0].pid); return { status: 503 }; };
+ const r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'ready');
+ assert.match(rec.failure.message, /exited before it was ready/);
+ assert.ok(!host.calls.some((c) => c.cmd === 'kill'));
+ assert.ok(!host.files.has(rec.dir));
+ }),
+
+ test('mismatches are reported per path and per table; volatile keys are ignored only where declared', async () => {
+ const host = await drillScenario();
+ host.drill.pastes = () => ({ status: 200, body: PASTES.replace('world', 'World') });
+ host.drill.pulse = () => ({ status: 200, body: { items: [{ id: 2 }], generated_at: 'different, but ignored' } });
+ host.drill.productionCount = 900;
+ const r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'compare');
+ const [pastes, pulse] = rec.compare;
+ assert.strictEqual(pastes.match, false);
+ assert.match(pastes.detail, /first difference at byte 55/);
+ assert.strictEqual(pulse.match, false);
+ assert.match(pulse.detail, /\$\.items\[0\]\.id/);
+ assert.ok(!/generated_at/.test(pulse.detail));
+ assert.deepStrictEqual(rec.counts[0], { db: 'community', table: 'pastes', production: 900, restored: 892, match: false });
+ assert.match(rec.markdown, /`\/api\/pastes\?limit=5` differs \(bodies differ/);
+ assert.match(rec.markdown, /pastes 900 ≠ 892 \| failed \|$/);
+ // Still stopped and cleaned up.
+ assert.ok(!host.alivePids.has(host.systemdRuns[0].pid));
+ assert.ok(!host.files.has(rec.dir));
+ }),
+
+ test('compare: a status mismatch or a failing production answer is never a match', () => {
+ const e = { path: '/x', ignore: [] };
+ assert.strictEqual(compareBodies(e, { status: 500, body: 'e' }, { status: 500, body: 'e' }).match, false);
+ assert.match(compareBodies(e, { status: 200, body: 'a' }, { status: 404, body: 'a' }).detail, /status 404/);
+ assert.strictEqual(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '{"a":1,"t":1}' }, { status: 200, body: '{"a":1,"t":2}' }).match, true);
+ assert.match(compareBodies({ path: '/x', ignore: ['t'] }, { status: 200, body: '<html>' }, { status: 200, body: '<html>' }).detail, /not JSON/);
+ }),
+
+ test('cleanup on failure: systemd-run fails → directory removed, lock released, row logged', async () => {
+ const host = await drillScenario();
+ const before = fsSnapshot(host);
+ host.drill.onStart = () => ({ code: 1, stderr: 'Failed to start transient service unit: Unit already exists.' });
+ const r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 2, r.out);
+ const rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'start');
+ assert.match(rec.failure.message, /systemd-run failed/);
+ assert.ok(!host.files.has(rec.dir));
+ assert.ok(!host.files.has('/var/lib/openvibe-host/locks/community-drill.lock'));
+ for (const p of changedPaths(before, fsSnapshot(host))) assert.ok(allowedWrite(p, rec.dir), `unexpected write: ${p}`);
+ // A second drill can run straight after.
+ host.drill.onStart = null;
+ host.advance(1000);
+ assert.strictEqual((await host.cli('drill', 'community')).code, 0);
+ }),
+
+ test('an instance that ignores SIGTERM gets SIGKILL; one that survives SIGKILL keeps its directory and fails', async () => {
+ let host = await drillScenario();
+ host.drill.ignoreTerm = true;
+ let r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 0, r.out);
+ let rec = lastLog(host);
+ assert.strictEqual(rec.stop.signal, 'SIGKILL');
+ assert.deepStrictEqual(host.calls.filter((c) => c.cmd === 'kill').map((c) => c.args[0]), ['SIGTERM', 'SIGKILL']);
+
+ host = await drillScenario();
+ host.drill.ignoreTerm = true;
+ host.drill.ignoreKill = true;
+ r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 2, r.out);
+ rec = lastLog(host);
+ assert.strictEqual(rec.failure.stage, 'stop');
+ assert.strictEqual(rec.kept, true);
+ assert.ok(host.files.has(rec.dir), 'directory kept while the instance may still hold it');
+ assert.ok(host.alivePids.has(2222), 'production untouched');
+ assert.ok(!host.calls.some((c) => c.cmd === 'kill' && c.args[1] === 2222));
+ }),
+
+ test('refusals: not root, port in use, unsupported service, no drill block, no backup, missing database copy', async () => {
+ let host = await drillScenario();
+ host.exec.isRoot = async () => false;
+ let r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 1);
+ assert.match(r.out, /must run as root/);
+ assert.strictEqual(host.systemdRuns.length, 0);
+
+ host = await drillScenario();
+ host.listeners.set(14200, [{ pid: 777, process: 'python3' }]);
+ const before = fsSnapshot(host);
+ r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 1);
+ assert.match(r.out, /port 14200 is already in use \(python3 pid 777\)/);
+ assert.strictEqual(host.systemdRuns.length, 0);
+ assert.deepStrictEqual(changedPaths(before, fsSnapshot(host)), [], 'a refusal writes nothing');
+
+ host = await drillScenario({ drill: { supported: false, reason: 'workers have no off switch' } });
+ r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 1);
+ assert.match(r.out, /does not support restore drills: workers have no off switch/);
+
+ host = await drillScenario();
+ r = await host.cli('drill', 'media');
+ assert.strictEqual(r.code, 1);
+ assert.match(r.out, /media has no drill block/);
+
+ host = await drillScenario({ backup: false });
+ r = await host.cli('drill', 'community');
+ assert.strictEqual(r.code, 1);
+ assert.match(r.out, /no ovhost backup recorded for community/);
+
+ host = await drillScenario({ backup: false });
+ host.ensureDir('/srv/old-backup');
+ r = await host.cli('drill', 'community', '--backup', '/srv/old-backup');
+ assert.strictEqual(r.code, 1);
+ assert.match(r.out, /has no copy of the community database/);
+ host.put('/srv/old-backup/community.db', 'older sqlite', { owner: 'ubuntu' });
+ r = await host.cli('drill', 'community', '--backup', '/srv/old-backup', '--json');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.strictEqual(JSON.parse(r.out).backup, '/srv/old-backup');
+ r = await host.cli('drill', 'community', '--backup', 'relative/dir');
+ assert.strictEqual(r.code, 1);
+ }),
+
+ test('inventory: drill blocks are validated', () => {
+ const base = { owner: 'ubuntu' };
+ const svc = (drill, extra = {}) => normalise({ services: { community: { ...base, ...communityEntry(), ...extra, drill: { ...communityEntry().drill, ...drill } } } });
+ assert.throws(() => svc({ port: 4200 }), /must differ from the service's own port/);
+ assert.throws(() => svc({ databases: { nope: 'X_DB' } }), /not one of community's databases/);
+ assert.throws(() => svc({ env: { HOST: '127.0.0.1' } }), /\{port\}/);
+ assert.throws(() => svc({ env: { PORT: '{port}', COMMUNITY_DB_PATH: '/x' } }), /already set by drill.databases/);
+ assert.throws(() => svc({ env: { PORT: '{port}\nEVIL=1' } }), /one-line/);
+ assert.throws(() => svc({ dirs: ['/opt/openvibe.community/data'] }), /under \{tmp\}/);
+ assert.throws(() => svc({ counts: [{ db: 'community', table: 'x; drop table y' }] }), /table name/);
+ assert.throws(() => svc({ compare: [{ path: 'http://evil/' }] }), /path/);
+ assert.throws(() => svc({ supported: false }), /reason is required/);
+ assert.throws(() => normalise({ services: { a: { ...base, repo: '/opt/a', port: 14200 }, community: { ...base, ...communityEntry() } } }), /14200 is a's port/);
+ assert.strictEqual(svc({}).services.community.drill.ready, '/api/ready');
+ }),
+
+ test('host.example.json: every service with a database has a drill block; unsupported ones give a reason', () => {
+ const raw = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
+ const inv = normalise(raw);
+ for (const s of Object.values(inv.services)) {
+ if (!s.databases.length) continue;
+ assert.ok(s.drill, `${s.id} has a drill block`);
+ if (!s.drill.supported) { assert.ok(s.drill.reason.length > 20, `${s.id} explains why`); continue; }
+ assert.ok(s.drill.compare.length || s.drill.counts.length);
+ assert.ok(Object.keys(s.drill.env).length >= 2, `${s.id} overrides at least its port and one side effect`);
+ }
+ }),
+]);
@@ -45,6 +45,27 @@ runTests([
@@ -45,6 +45,27 @@ runTests([
db.close();
db.close();
}),
}),
+ test('drill primitives: integrity_check through the worker, kill() signals exactly one pid', async () => {
+ const dbPath = path.join(dir, 'restored.db');
+ const db = new Database(dbPath);
+ db.pragma('journal_mode = WAL');
+ db.exec('CREATE TABLE pastes (id INTEGER PRIMARY KEY); INSERT INTO pastes DEFAULT VALUES;');
+ db.close();
+ const me = await exec.userName();
+ assert.deepStrictEqual(await exec.sqlite(dbPath, 'PRAGMA integrity_check', { as: me }), [{ integrity_check: 'ok' }]);
+ assert.deepStrictEqual(await exec.sqlite(dbPath, 'SELECT count(*) AS n FROM "pastes"', { as: me }), [{ n: 1 }]);
+
+ const { spawn } = require('child_process');
+ const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' });
+ const exited = new Promise((r) => child.on('exit', (code, signal) => r(signal)));
+ assert.strictEqual(await exec.pidAlive(child.pid), true);
+ assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), true);
+ assert.strictEqual(await exited, 'SIGTERM');
+ assert.strictEqual(await exec.kill(child.pid, 'SIGTERM'), false, 'a pid that is gone');
+ await assert.rejects(exec.kill(1, 'SIGTERM'), /refusing/);
+ await assert.rejects(exec.kill(0, 'SIGTERM'), /refusing/);
+ }),
+
test('http, files, locks and run() behave as the fake host assumes', async () => {
test('http, files, locks and run() behave as the fake host assumes', async () => {
const server = http.createServer((req, res) => { res.writeHead(req.url === '/ok' ? 200 : 503); res.end(JSON.stringify({ host: req.headers.host })); });
const server = http.createServer((req, res) => { res.writeHead(req.url === '/ok' ? 200 : 503); res.end(JSON.stringify({ host: req.headers.host })); });
await new Promise((r) => server.listen(0, '127.0.0.1', r));
await new Promise((r) => server.listen(0, '127.0.0.1', r));
@@ -27,6 +27,10 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -27,6 +27,10 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
sqliteCalls: [],
sqliteCalls: [],
reads: [],
reads: [],
onRestart: null, // (unit) -> void
onRestart: null, // (unit) -> void
+ onSystemdRun: null, // (spec) -> undefined | { code, stderr } ; spec = { unit, uid, cwd, props, argv, pid }
+ onKill: null, // (pid, signal) -> false to ignore the signal
+ systemdRuns: [],
+ nextPid: 40000,
};
};
// ── filesystem ──
// ── filesystem ──
@@ -186,6 +190,28 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -186,6 +190,28 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
}
}
}
}
+ // ── systemd-run (transient units: restore drills) ──
+ function systemdRunCmd(args) {
+ const spec = { unit: null, uid: null, cwd: null, props: [], argv: [], pid: null };
+ for (let i = 0; i < args.length; i++) {
+ const a = args[i];
+ if (a === '--') { spec.argv = args.slice(i + 1); break; }
+ if (a === '--unit') spec.unit = args[++i];
+ else if (a === '--description') i += 1;
+ else if (a === '-p') spec.props.push(args[++i]);
+ else if (a.startsWith('--uid=')) spec.uid = a.slice(6);
+ else if (a.startsWith('--working-directory=')) spec.cwd = a.slice(20);
+ }
+ spec.envFiles = spec.props.filter((p) => p.startsWith('EnvironmentFile=')).map((p) => p.slice(16));
+ spec.pid = host.nextPid++;
+ host.systemdRuns.push(spec);
+ const r = host.onSystemdRun ? host.onSystemdRun(spec) : undefined;
+ if (r && r.code) return { stdout: '', stderr: '', ...r };
+ if (!(r && r.exitedAtStart)) alivePids.add(spec.pid);
+ host.addUnit(spec.unit, { mainPid: spec.pid, active: r && r.exitedAtStart ? 'failed' : 'active', transient: true });
+ return { code: 0, stdout: '', stderr: '' };
+ }
+
host.socketViolations = () => calls.filter((c) => c.cmd === 'systemctl' && ['stop', 'restart', 'kill', 'try-restart', 'reload-or-restart', 'disable', 'mask'].includes(c.args[0]) && String(c.args[1]).endsWith('.socket'));
host.socketViolations = () => calls.filter((c) => c.cmd === 'systemctl' && ['stop', 'restart', 'kill', 'try-restart', 'reload-or-restart', 'disable', 'mask'].includes(c.args[0]) && String(c.args[1]).endsWith('.socket'));
host.restarts = () => calls.filter((c) => c.cmd === 'systemctl' && c.args[0] === 'restart').map((c) => c.args[1]);
host.restarts = () => calls.filter((c) => c.cmd === 'systemctl' && c.args[0] === 'restart').map((c) => c.args[1]);
@@ -201,6 +227,15 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -201,6 +227,15 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
case 'rm': rmTree(args[args.length - 1]); return { code: 0, stdout: '', stderr: '' };
case 'rm': rmTree(args[args.length - 1]); return { code: 0, stdout: '', stderr: '' };
case 'install': { const src = args[args.length - 2]; const dest = args[args.length - 1]; put(dest, host.read(src)); return { code: 0, stdout: '', stderr: '' }; }
case 'install': { const src = args[args.length - 2]; const dest = args[args.length - 1]; put(dest, host.read(src)); return { code: 0, stdout: '', stderr: '' }; }
case 'node': return { code: 0, stdout: '', stderr: '' };
case 'node': return { code: 0, stdout: '', stderr: '' };
+ case 'systemd-run': return systemdRunCmd(args);
+ case 'cp': {
+ const src = args[args.length - 2];
+ const dest = args[args.length - 1];
+ const e = get(src);
+ if (!e || e.type !== 'file') return { code: 1, stdout: '', stderr: `cp: cannot stat '${src}': No such file or directory` };
+ put(dest, e.content, { owner: opts.as || user, mode: 0o644 });
+ return { code: 0, stdout: '', stderr: '' };
+ }
default: return { code: 127, stdout: '', stderr: `fake host: no ${cmd}` };
default: return { code: 127, stdout: '', stderr: `fake host: no ${cmd}` };
}
}
},
},
@@ -251,6 +286,18 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -251,6 +286,18 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
put(dest, `backup of ${db}`, { owner: as || user, mode: 0o640 });
put(dest, `backup of ${db}`, { owner: as || user, mode: 0o640 });
},
},
async listeners(port) { return listeners.get(port) || []; },
async listeners(port) { return listeners.get(port) || []; },
+ async kill(pid, signal = 'SIGTERM') {
+ calls.push({ cmd: 'kill', args: [signal, pid], as: null, privileged: true, cwd: null });
+ if (!alivePids.has(pid)) return false;
+ if (host.onKill && host.onKill(pid, signal) === false) return true;
+ alivePids.delete(pid);
+ for (const u of units.values()) if (u.mainPid === pid) { u.active = 'inactive'; u.sub = 'dead'; }
+ for (const [port, list] of listeners) {
+ const rest = list.filter((l) => l.pid !== pid);
+ if (rest.length) listeners.set(port, rest); else listeners.delete(port);
+ }
+ return true;
+ },
async sleep(ms) { clock += ms; },
async sleep(ms) { clock += ms; },
now: () => clock,
now: () => clock,
async isRoot() { return root; },
async isRoot() { return root; },