openvibers/openvibe.host Commits

OpenRe on ovhost: workerUnits and a restore drill.

OP OpenVibers committed 23 Sept 2026 · ed7de69879 · 13 files, +294 −17

Inventory: workerUnits declares units or templates ([email protected], [email protected]) that belong to a service but are never started, stopped or restarted. status and validate list their instances read-only (systemctl list-units, unprivileged); validate warns when none runs. The inventory refuses a worker unit, or an instance of a worker template, in units, which are the only units ovhost ever restarts.

Drill: the openre drill is supported. Only openre-api starts (drill.command, /usr/bin/node server/index.js), on 127.0.0.1:14500, with OPENRE_DRILL=1 (OpenRe 5e86ea7: read-only API; the coordinator and the workers refuse to start; no event relay, no Media calls). EVENTS_URL and OV_OAUTH_CLIENT_SECRET are empty and MEDIA_URL points at the closed port. It compares /api/health and /robots.txt and four table counts.

Hardening: a drill no longer passes on a unit Environment= that it sets itself. OpenRe's unit sets OPENRE_DB_PATH to the production database, and only the override file's path reaches the instance.

Docs: the Host cutover runbook row F now points to OpenRe docs/cutover.md.

Changes

ddiff --git a/README.md b/README.md
iindex 51d72f8..7d3218a 100644
---- a/README.md
++++ b/README.md
@@@ -150,13 +150,13 @@ Each inventory entry's `drill` block declares:
- `counts`: `{ db, table }` pairs;
- `supported: false` with a `reason` when a second instance cannot run without side effects.
--In `host.example.json`, 18 services have drill blocks. live, media, tools and games are marked unsupported, and the reasons are in the file.
++In `host.example.json`, 24 services have drill blocks, and 19 can be drilled. live, media, tools, games and ai are marked unsupported, and the reasons are in the file. `command` (an absolute argv) is required when a service has several units, as openre does: only `openre-api` runs in its drill.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
### Inventory
--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange` and `nginx`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRe's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
`games` is listed as `managed: false`. It is a pnpm workspace with a TypeScript build, and those steps are not encoded yet. `status`, `validate`, `snapshot` and `backup` work for it; `deploy` and `rollback` refuse it.
ddiff --git a/docs/backups.md b/docs/backups.md
iindex 2740e71..d66d4db 100644
---- a/docs/backups.md
++++ b/docs/backups.md
@@@ -350,7 +350,7 @@ The drill:
Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
--For services without a drill (live, media, tools, games, openre), download the copy and check it by
++For services without a drill (live, media, tools, games, ai), download the copy and check it by
hand:
```
ddiff --git a/docs/cutover-runbook.md b/docs/cutover-runbook.md
iindex 1c4ec40..e529f41 100644
---- a/docs/cutover-runbook.md
++++ b/docs/cutover-runbook.md
@@@ -29,7 +29,7 @@ State markers:
| C | AI → OpenVibe.AI (Wave 13) | **DONE** 2026-09-23 ~15:40 UTC (`AI_SERVICE=remote`) | AI [`docs/migration.md`](https://github.com/OpenVibers/OpenVibe.AI/blob/main/docs/migration.md) | Unset `AI_SERVICE`, restart Live |
| D | Events delivery signature v2 | **IN PROGRESS**: Events sends v2; consumer deploys unknown; v1 still sent | Events [`docs/replay-window-rollout.md`](https://github.com/OpenVibers/OpenVibe.Events/blob/main/docs/replay-window-rollout.md) | The consumer redeploys its previous release; replay the DLQ |
| E | Money → OpenVibe.Billing | **BLOCKED (owner)**: PowerChat dashboard. Billing runs in shadow. | Billing [`docs/live-cutover.md`](https://github.com/OpenVibers/OpenVibe.Billing/blob/main/docs/live-cutover.md) | Depends on the step reached. After step 9 there is **no tool** |
--| F | RTMP ingest → OpenRe.Stream | **BLOCKED (owner)**: DNS `ingest.openre.stream` and port 1936. OpenRe is deployed. | OpenRe [`README.md` "Cutover runbook"](https://github.com/OpenVibers/OpenRe.Stream/blob/main/README.md#cutover-runbook) | Per slot `{"authority":"live"}`. All slots: unset `OPENRE_URL` |
++| F | RTMP ingest → OpenRe.Stream | **BLOCKED (owner)**: port 1936 at the provider edge. DNS `ingest.openre.stream` is done and OpenRe is deployed. | OpenRe [`docs/cutover.md`](https://github.com/OpenVibers/OpenRe.Stream/blob/main/docs/cutover.md) (steps, checks and rollback; `scripts/cutover-preflight.js`) | Per slot `{"authority":"live"}` + `set-definition-state.js --state disabled`. All slots: unset `OPENRE_URL` |
| G | Retire the shims | **NOT STARTED**. Dated in [compatibility-register.md](compatibility-register.md) | Network [`docs/retirement.md`](https://github.com/OpenVibers/OpenVibe.Network/blob/main/docs/retirement.md) | per entry |
The remaining order is D → E → F → G.
@@@ -320,7 +320,7 @@ Live carries the integration (`f0ca18b`). It is inert until `OPENRE_URL` is set.
defaults to `live` on every slot (`server/openre/schema.js:5-7`), and only RTMP moves
(`server/openre/authority.js:18`). WHIP, JSMPEG and SFU are **not ported** (OpenRe `README.md:207-209`).
--**Once, before the first slot** (OpenRe `README.md:188-194`):
++**Once, before the first slot.** OpenRe's [`docs/cutover.md`](https://github.com/OpenVibers/OpenRe.Stream/blob/main/docs/cutover.md) supersedes the table below: it has the exact commands, the owner and agent steps (A1–A7), and a read-only preflight whose output on 2026-09-23 settles the "Unknown" rows (no subscription, no `OPENRE_*` in `live.env`).
| Step | State |
|---|---|
ddiff --git a/docs/restore-drills.md b/docs/restore-drills.md
iindex 3fb2fc4..fd2abcf 100644
---- a/docs/restore-drills.md
++++ b/docs/restore-drills.md
@@@ -130,6 +130,14 @@ Supported with declared overrides (not yet run on the host with `ovhost drill`):
- coupons
- trade
- codes
++- openre (not run yet). Only openre-api starts (`drill.command`), with `OPENRE_DRILL=1`. From OpenRe
++ `5e86ea7` on, that makes it read-only: writes, `/play/` and sign-in answer 503. The coordinator
++ and every transport worker refuse to start in a drill, and the event relay and Media calls are
++ off. The API never runs a transport or the relay in any case. Empty `EVENTS_URL` and
++ `OV_OAUTH_CLIENT_SECRET`, plus `MEDIA_URL` on the closed port, cover older releases. The workers
++ stay out of every ovhost command: they are `workerUnits`, which are listed and never restarted.
++ Run it after the OpenRe release with `OPENRE_DRILL` is deployed (OpenRe `docs/cutover.md`
++ phase C).
Services that are not deployed yet (everything except network, live, media, tools, community and
games as of 22 Sep) have entries built from their repositories' `deploy/` directories.
ddiff --git a/host.example.json b/host.example.json
iindex 95dae7b..d0f6c71 100644
---- a/host.example.json
++++ b/host.example.json
@@@ -551,17 +551,26 @@
"drill": { "supported": false, "reason": "Not written yet: a drill must start AI with every provider key blanked, or queued and new runs would call paid providers. Backups work." }
},
"openre": {
-- "_note": "Release layout: the service runs from /opt/openre.stream/current and deploys with its own deploy/deploy.sh (release | api | workers). The templated ingest and restream workers belong to the session coordinator; ovhost never restarts them.",
++ "_note": "Release layout: the service runs from /opt/openre.stream/current and deploys with its own deploy/scripts/deploy.sh (release | api | workers). units are the API and the session coordinator. workerUnits are the transport workers: one instance per release (openre-rtmp-ingest@<release>), started by deploy.sh workers, drained by the coordinator when a newer generation is ready, and exiting on their own. ovhost lists them in status/validate and never starts, stops or restarts them (the inventory refuses a worker unit in units).",
"repo": "/opt/openre.stream",
"layout": "release",
"units": ["openre-api.service", "openre-session-coordinator.service"],
++ "workerUnits": ["[email protected]", "[email protected]"],
"envFile": "/etc/openvibe/openre.env",
"env": { "required": "from-example" },
"port": 4500,
"ready": { "url": "http://127.0.0.1:4500/api/ready", "timeoutSeconds": 30 },
"noRestartPaths": ["docs/", "test/", "**/*.md"],
"databases": [{ "name": "openre", "path": "/var/lib/openre/openre.db" }],
-- "drill": { "supported": false, "reason": "The session coordinator reconciles slots and starts workers at boot; a drill needs a switch that keeps it away from live ingest slots." }
++ "drill": {
++ "port": 14500,
++ "command": ["/usr/bin/node", "server/index.js"],
++ "databases": { "openre": "OPENRE_DB_PATH" },
++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "OPENRE_DRILL": "1", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "MEDIA_URL": "http://127.0.0.1:9", "MEDIA_API_KEY": "", "OPENRE_RECORDING": "off" },
++ "compare": ["/api/health", "/robots.txt"],
++ "counts": [{ "db": "openre", "table": "stream_definitions" }, { "db": "openre", "table": "ingest_keys" }, { "db": "openre", "table": "destinations" }, { "db": "openre", "table": "migration_map" }],
++ "_notes": "Only openre-api starts (command; the working directory and the non-secret Environment= come from openre-api.service). It never runs a transport, a coordinator or the event relay in any mode. OPENRE_DRILL=1 (OpenRe 5e86ea7 and later) also makes it read-only: writes, /play/ (which would pull from production's worker on loopback) and sign-in (which would redeem codes at Network) answer 503. The coordinator and the workers refuse to start. The event relay and Media calls are off. EVENTS_URL and OV_OAUTH_CLIENT_SECRET empty and MEDIA_URL on the closed port are the same switches for releases before 5e86ea7. Left on (read-only): the Network JWKS GET on loopback. Not compared: ingest_sessions and workers (they change whenever someone streams or a generation rolls over) and /api/ready (it reports live worker heartbeats)."
++ }
}
}
}
ddiff --git a/lib/cli.js b/lib/cli.js
iindex 8007679..621e7da 100644
---- a/lib/cli.js
++++ b/lib/cli.js
@@@ -168,9 +168,10 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
for (const r of rows) {
const units = r.units.map((u) => `${u.unit.replace(/\.service$/, '')}=${u.active}${u.restarts ? `(${u.restarts} restarts)` : ''}`).join(' ');
const sock = r.socket ? ` socket=${r.socket.active}` : '';
++ const workers = r.workers && r.workers.length ? ` workers[${r.workers.map((w) => `${w.unit.replace(/\.service$/, '')}=${w.active}`).join(' ')}]` : '';
const ready = r.ready ? (r.ready.ok ? 'ready' : `NOT READY (${r.ready.status || r.ready.error})`) : '-';
const prot = r.protected ? (r.protected.count == null ? ` ${r.protected.label}=unknown` : ` ${r.protected.label}=${r.protected.count}`) : '';
-- out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${prot}${r.managed ? '' : ' [unmanaged]'}`);
++ out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${workers}${prot}${r.managed ? '' : ' [unmanaged]'}`);
}
return 0;
}
ddiff --git a/lib/commands/drill.js b/lib/commands/drill.js
iindex 493f9c6..0018dd3 100644
---- a/lib/commands/drill.js
++++ b/lib/commands/drill.js
@@@ -26,7 +26,8 @@
* writes to the checkout, the env file, the database or the backup. It never reads an env value:
* systemd loads the production env file for the instance, and the override file holds only
* inventory values. Environment= values from the unit are passed on only if their names do not look
-- * secret.
++ * secret and the drill does not set them itself (OpenRe's unit sets OPENRE_DB_PATH to production's
++ * database: that line never reaches the drill instance).
*/
const path = require('path');
const { service } = require('../inventory');
@@@ -277,9 +278,13 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
for (const f of bk.files) lines.push(envLine(f.envName, vars.db[f.name]));
const envPath = path.join(tmp, 'drill.env');
await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });
++ // A unit Environment= the drill sets itself (its port, a restored database path) is not passed
++ // on at all: the override file would win anyway (EnvironmentFile= beats Environment=), but a
++ // production path must never reach the drill instance's environment.
++ const drillNames = new Set([...Object.keys(d.env), ...bk.files.map((f) => f.envName)]);
const props = [
'Type=exec',
-- ...spec.environment.map((e) => environmentProperty(e.name, e.value)),
++ ...spec.environment.filter((e) => !drillNames.has(e.name)).map((e) => environmentProperty(e.name, e.value)),
...(svc.envFile ? [`EnvironmentFile=${svc.envFile}`] : []),
`EnvironmentFile=${envPath}`,
`StandardOutput=append:${logFile}`,
ddiff --git a/lib/commands/status.js b/lib/commands/status.js
iindex 9cbc387..c46ee59 100644
---- a/lib/commands/status.js
++++ b/lib/commands/status.js
@@@ -7,11 +7,19 @@ const { git } = require('../git');
async function statusOne(ctx, svc) {
const { exec } = ctx;
-- const row = { service: svc.id, managed: svc.managed, units: [], socket: null, sha: null, branch: null, ready: null, protected: null };
++ const row = { service: svc.id, managed: svc.managed, units: [], workers: [], socket: null, sha: null, branch: null, ready: null, protected: null };
for (const u of svc.units) {
const st = await systemd.show(exec, u);
row.units.push({ unit: u, active: st.active, sub: st.sub, restarts: st.restarts });
}
++ // Worker units are only listed: ovhost never starts, stops or restarts them.
++ for (const w of svc.workerUnits) {
++ try {
++ for (const i of await systemd.instances(exec, w)) row.workers.push({ unit: i.unit, active: i.active, sub: i.sub });
++ } catch (err) {
++ row.workers.push({ unit: w, active: 'unknown', error: err.message.split('\n')[0] });
++ }
++ }
if (svc.socketUnit) {
const st = await systemd.show(exec, svc.socketUnit);
row.socket = { unit: svc.socketUnit, active: st.active, sub: st.sub };
ddiff --git a/lib/commands/validate.js b/lib/commands/validate.js
iindex 0462a9e..e0a383e 100644
---- a/lib/commands/validate.js
++++ b/lib/commands/validate.js
@@@ -6,7 +6,8 @@
* non-empty (names from the unit's Environment= lines count as present). Values are
* never read into the result — only { name, empty }.
* units every unit is loaded from a unit file; the socket unit is active; repo unit files that
-- * differ from the installed ones are reported.
++ * differ from the installed ones are reported. Instances of workerUnits are listed (a warning
++ * when none runs); they are never started, stopped or restarted.
* port free (service down) or held by the service / its socket — nothing else.
* nginx the vhost is in sites-available and linked in sites-enabled; `nginx -t` is clean.
* checkout owned by the declared owner, on the declared branch, no tracked changes.
@@@ -98,6 +99,13 @@ async function validate(ctx, id) {
else add(st.active === 'active' ? 'ok' : 'warn', 'units', `${u} ${st.active}/${st.sub} from ${st.fragmentPath}`);
if (st.mainPid) pids.add(st.mainPid);
}
++ for (const w of svc.workerUnits) {
++ let list = [];
++ try { list = await systemd.instances(exec, w); } catch (err) { add('warn', 'units', `could not list ${w}: ${err.message}`); continue; }
++ const running = list.filter((i) => i.active === 'active');
++ if (!running.length) add('warn', 'units', `no running instance of worker unit ${w}`);
++ for (const i of list) add('info', 'units', `worker ${i.unit} ${i.active}/${i.sub} (ovhost never starts, stops or restarts it)`);
++ }
if (svc.socketUnit) {
const st = await systemd.show(exec, svc.socketUnit);
if (st.load !== 'loaded') add('error', 'units', `${svc.socketUnit} is not loaded (${st.load})`);
ddiff --git a/lib/inventory.js b/lib/inventory.js
iindex 4b61ea4..8bec135 100644
---- a/lib/inventory.js
++++ b/lib/inventory.js
@@@ -16,6 +16,8 @@ class InventoryError extends Error {}
const ID_RE = /^[a-z][a-z0-9-]{0,39}$/;
const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
++// A worker unit is a template (`[email protected]`) or a plain service unit.
++const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
const PROBE_KINDS = ['http-json-count', 'sqlite-count'];
@@@ -49,6 +51,10 @@ function normaliseService(id, raw, defaults) {
managed: s.layout === 'release' ? false : s.managed !== false,
unmanagedReason: s.unmanagedReason || (s.layout === 'release' ? 'release layout: deployed by its own deploy script' : null),
units: s.units || [],
++ // Units that belong to the service but that ovhost never starts, stops or restarts: e.g.
++ // OpenRe's transport workers ([email protected], one instance per release), which
++ // drain on their own. status/validate list their instances read-only.
++ workerUnits: s.workerUnits || [],
socketUnit: s.socketUnit || null,
unitSources: s.unitSources || {},
envFile: s.envFile ? abs(s.envFile, `${where}.envFile`) : null,
@@@ -83,6 +89,15 @@ function normaliseService(id, raw, defaults) {
if (u.endsWith('.socket')) throw new InventoryError(`${where}.units: "${u}" is a socket unit — declare it as socketUnit; ovhost never restarts or stops socket units`);
}
if (out.socketUnit && !(UNIT_RE.test(out.socketUnit) && out.socketUnit.endsWith('.socket'))) throw new InventoryError(`${where}.socketUnit must be a .socket unit`);
++ if (!Array.isArray(out.workerUnits)) throw new InventoryError(`${where}.workerUnits must be an array`);
++ for (const w of out.workerUnits) {
++ if (typeof w !== 'string' || !WORKER_UNIT_RE.test(w)) throw new InventoryError(`${where}.workerUnits: "${w}" is not a .service unit or template`);
++ for (const u of out.units) {
++ if (u === w || (w.endsWith('@.service') && u.startsWith(w.slice(0, -'.service'.length)))) {
++ throw new InventoryError(`${where}.units: "${u}" is a worker unit (${w}); ovhost restarts every unit in units, and never a worker unit`);
++ }
++ }
++ }
for (const [unit, src] of Object.entries(out.unitSources)) {
if (!UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit or drop-in name`);
rel(src, `${where}.unitSources.${unit}`);
ddiff --git a/lib/systemd.js b/lib/systemd.js
iindex 2b64efa..5150ca5 100644
---- a/lib/systemd.js
++++ b/lib/systemd.js
@@@ -5,8 +5,9 @@
* the .socket unit so HTTP connections queue in the kernel through a restart; stopping that unit is
* what turns a restart into Cloudflare 502s. Only the .service unit is ever restarted.
*/
--const SOCKET_SAFE_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'cat', 'status']);
--const ALLOWED_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'restart', 'reload', 'cat', 'status', 'daemon-reload']);
++const SOCKET_SAFE_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'cat', 'status', 'list-units']);
++const ALLOWED_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'restart', 'reload', 'cat', 'status', 'daemon-reload', 'list-units']);
++const READ_ONLY_ACTIONS = ['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'];
const PROPS = ['LoadState', 'ActiveState', 'SubState', 'MainPID', 'FragmentPath', 'UnitFileState', 'DropInPaths', 'PartOf', 'NRestarts'];
class SocketGuardError extends Error {}
@@@ -16,8 +17,11 @@ async function systemctl(exec, action, unit) {
if (unit && unit.endsWith('.socket') && !SOCKET_SAFE_ACTIONS.has(action)) {
throw new SocketGuardError(`refusing "systemctl ${action} ${unit}": socket units are never stopped or restarted (they hold the listener that keeps HTTP up through a restart)`);
}
-- const args = action === 'show' ? ['show', unit, `--property=${PROPS.join(',')}`] : unit ? [action, unit] : [action];
-- const privileged = !['show', 'is-active', 'is-enabled', 'cat', 'status'].includes(action);
++ let args;
++ if (action === 'show') args = ['show', unit, `--property=${PROPS.join(',')}`];
++ else if (action === 'list-units') args = ['list-units', '--all', '--plain', '--no-legend', '--no-pager', unit];
++ else args = unit ? [action, unit] : [action];
++ const privileged = !READ_ONLY_ACTIONS.includes(action);
return exec.run('systemctl', args, { privileged });
}
@@@ -42,6 +46,22 @@ async function show(exec, unit) {
};
}
++/**
++ * Loaded instances of a worker unit (read-only): `[email protected]` lists every `name@*.service`, a
++ * plain unit lists itself. -> [{ unit, load, active, sub }]
++ */
++async function instances(exec, workerUnit) {
++ const pattern = workerUnit.endsWith('@.service') ? `${workerUnit.slice(0, -'.service'.length)}*.service` : workerUnit;
++ const r = await systemctl(exec, 'list-units', pattern);
++ if (r.code !== 0) throw new Error(`systemctl list-units ${pattern} failed: ${(r.stderr || '').trim()}`);
++ const out = [];
++ for (const line of String(r.stdout || '').split('\n')) {
++ const [unit, load, active, sub] = line.trim().split(/\s+/);
++ if (unit && unit.endsWith('.service')) out.push({ unit, load, active, sub });
++ }
++ return out;
++}
++
async function restart(exec, unit) {
const r = await systemctl(exec, 'restart', unit);
if (r.code !== 0) throw new Error(`systemctl restart ${unit} failed: ${r.stderr.trim()}`);
@@@ -66,4 +86,4 @@ async function reloadNginx(exec) {
if (r.code !== 0) throw new Error(`systemctl reload nginx failed: ${r.stderr.trim()}`);
}
--module.exports = { systemctl, show, restart, ensureSocketActive, daemonReload, reloadNginx, SocketGuardError };
++module.exports = { systemctl, show, instances, restart, ensureSocketActive, daemonReload, reloadNginx, SocketGuardError };
ddiff --git a/test/fake-host.js b/test/fake-host.js
iindex 97ab420..12fd927 100644
---- a/test/fake-host.js
++++ b/test/fake-host.js
@@@ -188,6 +188,13 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
case 'stop': if (u) { u.active = 'inactive'; u.sub = 'dead'; } return { code: 0, stdout: '', stderr: '' };
case 'reload': case 'daemon-reload': return { code: 0, stdout: '', stderr: '' };
case 'is-active': return { code: u && u.active === 'active' ? 0 : 3, stdout: '', stderr: '' };
++ case 'list-units': {
++ // systemctl list-units --all --plain --no-legend --no-pager <glob>
++ const glob = args[args.length - 1];
++ const re = new RegExp(`^${glob.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*')}$`);
++ const rows = [...units.entries()].filter(([name]) => re.test(name)).map(([name, x]) => `${name} ${x.load} ${x.active} ${x.sub} ${name}`);
++ return { code: 0, stdout: rows.length ? `${rows.join('\n')}\n` : '', stderr: '' };
++ }
default: return { code: 1, stdout: '', stderr: `fake systemctl: ${action}` };
}
}
ddiff --git a/test/openre-workers.test.js b/test/openre-workers.test.js
nnew file mode 100644
iindex 0000000..c2647f7
---- /dev/null
++++ b/test/openre-workers.test.js
@@@ -0,0 +1,196 @@
++'use strict';
++/**
++ * OpenRe on ovhost: `workerUnits` (the transport worker templates) are listed by status and
++ * validate and never started, stopped or restarted; the inventory refuses a worker unit in
++ * `units`; a deploy of a git-layout service with worker units restarts only its units; and the
++ * example inventory's openre drill starts openre-api alone with OPENRE_DRILL=1 and the side-effect
++ * switches, leaving the API, the coordinator and every worker untouched.
++ */
++const assert = require('assert');
++const fs = require('fs');
++const path = require('path');
++const { scenario, push, test, runTests, SECRET } = require('./helpers');
++const { normalise } = require('../lib/inventory');
++
++const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
++const RELEASE = '655b98a10aaa';
++const OLD = '0123456789ab';
++const PROD_DB = '/var/lib/openre/openre.db';
++const WORKERS = [`openre-rtmp-ingest@${RELEASE}.service`, `openre-restream-worker@${RELEASE}.service`];
++
++/** The standard fake host plus OpenRe as on openvibe-ovh: release layout, two units, worker instances. */
++async function openreHost({ workersRunning = true } = {}) {
++ const host = scenario();
++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++ doc.services.openre = JSON.parse(JSON.stringify(EXAMPLE.services.openre));
++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
++
++ host.put(`/opt/openre.stream/releases/${RELEASE}/package.json`, '{"name":"openre-stream"}', { owner: 'ubuntu' });
++ host.put(`/opt/openre.stream/releases/${RELEASE}/.env.example`, 'OV_OAUTH_CLIENT_ID=openre\nOV_OAUTH_CLIENT_SECRET=\nOPENRE_SECRETS_KEY=\n#OPENRE_DRILL=\n', { owner: 'ubuntu' });
++ host.files.get(`/opt/openre.stream/releases/${RELEASE}`).owner = 'ubuntu';
++ await host.exec.symlink(`/opt/openre.stream/releases/${RELEASE}`, '/opt/openre.stream/current');
++ host.put('/etc/openvibe/openre.env', `OV_OAUTH_CLIENT_ID=openre\nOV_OAUTH_CLIENT_SECRET=${SECRET}\nOPENRE_SECRETS_KEY=${SECRET}-k\nOPENRE_RTMP_BIND=0.0.0.0\n`, { mode: 0o600 });
++ host.put('/etc/systemd/system/openre-api.service', [
++ '[Service]',
++ 'User=ubuntu',
++ 'WorkingDirectory=/opt/openre.stream/current',
++ 'EnvironmentFile=/etc/openvibe/openre.env',
++ 'Environment=NODE_ENV=production',
++ 'Environment=PORT=4500',
++ `Environment=OPENRE_DB_PATH=${PROD_DB}`,
++ 'Environment=PATH=/usr/local/bin:/usr/bin:/bin',
++ 'ExecStart=/usr/bin/env node server/index.js',
++ '',
++ ].join('\n'));
++ host.addUnit('openre-api.service', { mainPid: 3100 });
++ host.addUnit('openre-session-coordinator.service', { mainPid: 3101 });
++ host.alivePids.add(3100);
++ host.alivePids.add(3101);
++ host.listeners.set(4500, [{ pid: 3100, process: 'node' }]);
++ host.addUnit(`openre-rtmp-ingest@${RELEASE}.service`, { mainPid: 3102, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
++ host.addUnit(`openre-restream-worker@${RELEASE}.service`, { mainPid: 3103, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
++ host.addUnit(`openre-rtmp-ingest@${OLD}.service`, { active: 'inactive', sub: 'dead', mainPid: 0 });
++ host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
++
++ const health = '{"status":"ok","service":"openre-api","version":"0.1.0","release":"dev"}';
++ const robots = 'User-agent: *\nAllow: /$\n';
++ host.http.set('http://127.0.0.1:4500/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
++ host.http.set('http://127.0.0.1:4500/api/health', () => ({ status: 200, body: health }));
++ host.http.set('http://127.0.0.1:4500/robots.txt', () => ({ status: 200, body: robots }));
++ const counts = { stream_definitions: 1, ingest_keys: 1, destinations: 0, migration_map: 0 };
++ host.sqliteHandler = (db, sql) => {
++ if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
++ const m = /FROM "([a-z_]+)"/.exec(sql);
++ return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
++ };
++ host.onSystemdRun = (spec) => {
++ host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);
++ host.drillEnv = host.read(spec.envFiles[spec.envFiles.length - 1]);
++ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
++ host.http.set('http://127.0.0.1:14500/api/ready', alive(() => ({ status: 200, body: { status: 'ready', mode: 'drill' } })));
++ host.http.set('http://127.0.0.1:14500/api/health', alive(() => ({ status: 200, body: health })));
++ host.http.set('http://127.0.0.1:14500/robots.txt', alive(() => ({ status: 200, body: robots })));
++ return undefined;
++ };
++ return host;
++}
++
++/** systemctl calls that could change a unit's state. */
++const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));
++
++runTests([
++ test('the example inventory declares the OpenRe worker templates and a supported drill', () => {
++ const inv = normalise(EXAMPLE);
++ const o = inv.services.openre;
++ assert.deepStrictEqual(o.units, ['openre-api.service', 'openre-session-coordinator.service']);
++ assert.deepStrictEqual(o.workerUnits, ['[email protected]', '[email protected]']);
++ assert.strictEqual(o.layout, 'release');
++ assert.strictEqual(o.managed, false);
++ assert.strictEqual(o.drill.supported, true);
++ assert.strictEqual(o.drill.port, 14500);
++ assert.deepStrictEqual(o.drill.command, ['/usr/bin/node', 'server/index.js']);
++ assert.strictEqual(o.drill.env.OPENRE_DRILL, '1');
++ for (const k of ['EVENTS_URL', 'OV_OAUTH_CLIENT_SECRET', 'MEDIA_API_KEY']) assert.strictEqual(o.drill.env[k], '', k);
++ assert.strictEqual(o.drill.env.MEDIA_URL, 'http://127.0.0.1:9');
++ assert.deepStrictEqual(o.drill.databases, { openre: 'OPENRE_DB_PATH' });
++ // every other service keeps an empty list
++ assert.deepStrictEqual(inv.services.live.workerUnits, []);
++ }),
++
++ test('a worker unit, or an instance of a worker template, is refused in units', () => {
++ const base = { owner: 'ubuntu', repo: '/opt/x' };
++ assert.throws(() => normalise({ services: { x: { ...base, units: ['x-api.service', '[email protected]'], workerUnits: ['[email protected]'] } } }), /"x-worker@abc\.service" is a worker unit \(x-worker@\.service\)/);
++ assert.throws(() => normalise({ services: { x: { ...base, units: ['x-w.service'], workerUnits: ['x-w.service'] } } }), /is a worker unit/);
++ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: ['x.socket'] } } }), /not a \.service unit or template/);
++ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: ['../x.service'] } } }), /not a \.service unit or template/);
++ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: '[email protected]' } } }), /must be an array/);
++ const ok = normalise({ services: { x: { ...base, units: ['x-worker-api.service'], workerUnits: ['[email protected]'] } } });
++ assert.deepStrictEqual(ok.services.x.workerUnits, ['[email protected]'], 'a unit that only shares a prefix without the @ is fine');
++ }),
++
++ test('status lists worker instances read-only and never changes a unit', async () => {
++ const host = await openreHost();
++ const r = await host.cli('status', 'openre');
++ assert.strictEqual(r.code, 0, r.out);
++ assert.match(r.out, new RegExp(`openre\\s+${RELEASE}\\s+ready\\s+openre-api=active openre-session-coordinator=active workers\\[openre-rtmp-ingest@${RELEASE}=active openre-rtmp-ingest@${OLD}=inactive openre-restream-worker@${RELEASE}=active\\] \\[unmanaged\\]`));
++ const json = JSON.parse((await host.cli('status', 'openre', '--json')).out);
++ assert.deepStrictEqual(json[0].workers.map((w) => [w.unit, w.active]), [[`openre-rtmp-ingest@${RELEASE}.service`, 'active'], [`openre-rtmp-ingest@${OLD}.service`, 'inactive'], [`openre-restream-worker@${RELEASE}.service`, 'active']]);
++ const lists = host.calls.filter((c) => c.cmd === 'systemctl' && c.args[0] === 'list-units');
++ assert.deepStrictEqual(lists[0].args, ['list-units', '--all', '--plain', '--no-legend', '--no-pager', 'openre-rtmp-ingest@*.service']);
++ assert.ok(lists.every((c) => !c.privileged), 'listing needs no privileges');
++ assert.deepStrictEqual(stateChanges(host), []);
++ }),
++
++ test('validate lists the workers as info, warns when none runs, and changes nothing', async () => {
++ let host = await openreHost();
++ let res = JSON.parse((await host.cli('validate', 'openre', '--json')).out);
++ const workers = res.findings.filter((f) => f.area === 'units' && /^worker /.test(f.message));
++ assert.deepStrictEqual(workers.map((f) => f.level), ['info', 'info', 'info']);
++ assert.match(workers[0].message, /never starts, stops or restarts it/);
++ assert.ok(!res.findings.some((f) => /no running instance/.test(f.message)));
++ assert.deepStrictEqual(stateChanges(host), []);
++ host = await openreHost({ workersRunning: false });
++ res = JSON.parse((await host.cli('validate', 'openre', '--json')).out);
++ assert.deepStrictEqual(res.findings.filter((f) => /no running instance/.test(f.message)).map((f) => [f.level, f.message]), [
++ ['warn', 'no running instance of worker unit [email protected]'],
++ ['warn', 'no running instance of worker unit [email protected]'],
++ ]);
++ assert.deepStrictEqual(stateChanges(host), []);
++ assert.ok(!JSON.stringify(res).includes(SECRET));
++ }),
++
++ test('a deploy of a git-layout service with worker units restarts its units only', async () => {
++ const host = scenario();
++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
++ doc.services.events.workerUnits = ['[email protected]'];
++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
++ host.addUnit('[email protected]', { mainPid: 5001 });
++ push(host, 'events', { 'server/index.js': 'events(2);' });
++ const r = await host.cli('deploy', 'events');
++ assert.strictEqual(r.code, 0, r.out);
++ assert.deepStrictEqual(host.restarts(), ['openvibe-events.service']);
++ assert.ok(!host.calls.some((c) => c.cmd === 'systemctl' && c.args.includes('[email protected]')), 'the worker instance is never named in a systemctl call');
++ assert.strictEqual(host.units.get('[email protected]').restarts, 0);
++ }),
++
++ test('ovhost drill openre: openre-api alone, OPENRE_DRILL=1, side effects off, production untouched', async () => {
++ const host = await openreHost();
++ const b = await host.cli('backup', 'openre', '--json');
++ assert.strictEqual(b.code, 0, b.out);
++ host.advance(60 * 1000);
++ const callsBefore = host.calls.length;
++ const r = await host.cli('drill', 'openre');
++ assert.strictEqual(r.code, 0, r.out);
++ const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/openre.jsonl').trim().split('\n').pop());
++ assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));
++ assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/health', true], ['/robots.txt', true]]);
++ assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.match]), [['stream_definitions', true], ['ingest_keys', true], ['destinations', true], ['migration_map', true]]);
++
++ assert.strictEqual(host.systemdRuns.length, 1, 'one process');
++ const run = host.systemdRuns[0];
++ assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js'], 'the API, never a worker or the coordinator');
++ assert.strictEqual(run.cwd, '/opt/openre.stream/current');
++ assert.strictEqual(run.uid, 'ubuntu');
++ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/openre.env', `${rec.dir}/drill.env`]);
++ for (const p of ['SocketBindAllow=tcp:14500', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);
++ assert.ok(run.props.includes('Environment=PATH=/usr/local/bin:/usr/bin:/bin'), 'other unit Environment= is passed on');
++ assert.ok(!run.props.some((p) => /OPENRE_DB_PATH|Environment=PORT=/.test(p)), 'the production database path and port from the unit never reach the drill instance');
++ const env = Object.fromEntries(host.drillEnv.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
++ assert.deepStrictEqual(env, {
++ PORT: '14500', HOST: '127.0.0.1', OPENRE_DRILL: '1', EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', MEDIA_API_KEY: '', OPENRE_RECORDING: 'off',
++ OPENRE_DB_PATH: `${rec.dir}/db/openre.db`,
++ });
++
++ const calls = host.calls.slice(callsBefore);
++ assert.deepStrictEqual(calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', run.pid]], 'only the drill instance is signalled');
++ assert.deepStrictEqual(stateChanges(host), [], 'no unit was started, stopped or restarted');
++ for (const u of ['openre-api.service', 'openre-session-coordinator.service', ...WORKERS]) {
++ assert.strictEqual(host.units.get(u).active, 'active', u);
++ assert.strictEqual(host.units.get(u).restarts, 0, u);
++ assert.ok(!calls.some((c) => c.args && c.args.includes(u) && c.args[0] !== 'show'), `${u} is only ever read`);
++ }
++ assert.ok(host.alivePids.has(3100) && host.alivePids.has(3101));
++ assert.strictEqual(host.read(PROD_DB), 'sqlite-production');
++ assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/openre.jsonl')).includes(SECRET));
++ }),
++]);