openvibers/openvibe.host Commits
OpenRe on ovhost: workerUnits and a restore drill.
Changes
@@@ -150,13 +150,13 @@ Each inventory entry's `drill` block declares:- `counts`: `{ db, table }` pairs;- `supported: false` with a `reason` when a second instance cannot run without side effects.--In `host.example.json`, 18 services have drill blocks. live, media, tools and games are marked unsupported, and the reasons are in the file.++In `host.example.json`, 24 services have drill blocks, and 19 can be drilled. live, media, tools, games and ai are marked unsupported, and the reasons are in the file. `command` (an absolute argv) is required when a service has several units, as openre does: only `openre-api` runs in its drill.The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.### Inventory--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange` and `nginx`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRe's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.`games` is listed as `managed: false`. It is a pnpm workspace with a TypeScript build, and those steps are not encoded yet. `status`, `validate`, `snapshot` and `backup` work for it; `deploy` and `rollback` refuse it.@@@ -350,7 +350,7 @@ The drill:Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.--For services without a drill (live, media, tools, games, openre), download the copy and check it by++For services without a drill (live, media, tools, games, ai), download the copy and check it byhand:```@@@ -29,7 +29,7 @@ State markers:| C | AI → OpenVibe.AI (Wave 13) | **DONE** 2026-09-23 ~15:40 UTC (`AI_SERVICE=remote`) | AI [`docs/migration.md`](https://github.com/OpenVibers/OpenVibe.AI/blob/main/docs/migration.md) | Unset `AI_SERVICE`, restart Live || D | Events delivery signature v2 | **IN PROGRESS**: Events sends v2; consumer deploys unknown; v1 still sent | Events [`docs/replay-window-rollout.md`](https://github.com/OpenVibers/OpenVibe.Events/blob/main/docs/replay-window-rollout.md) | The consumer redeploys its previous release; replay the DLQ || E | Money → OpenVibe.Billing | **BLOCKED (owner)**: PowerChat dashboard. Billing runs in shadow. | Billing [`docs/live-cutover.md`](https://github.com/OpenVibers/OpenVibe.Billing/blob/main/docs/live-cutover.md) | Depends on the step reached. After step 9 there is **no tool** |--| F | RTMP ingest → OpenRe.Stream | **BLOCKED (owner)**: DNS `ingest.openre.stream` and port 1936. OpenRe is deployed. | OpenRe [`README.md` "Cutover runbook"](https://github.com/OpenVibers/OpenRe.Stream/blob/main/README.md#cutover-runbook) | Per slot `{"authority":"live"}`. All slots: unset `OPENRE_URL` |++| F | RTMP ingest → OpenRe.Stream | **BLOCKED (owner)**: port 1936 at the provider edge. DNS `ingest.openre.stream` is done and OpenRe is deployed. | OpenRe [`docs/cutover.md`](https://github.com/OpenVibers/OpenRe.Stream/blob/main/docs/cutover.md) (steps, checks and rollback; `scripts/cutover-preflight.js`) | Per slot `{"authority":"live"}` + `set-definition-state.js --state disabled`. All slots: unset `OPENRE_URL` || G | Retire the shims | **NOT STARTED**. Dated in [compatibility-register.md](compatibility-register.md) | Network [`docs/retirement.md`](https://github.com/OpenVibers/OpenVibe.Network/blob/main/docs/retirement.md) | per entry |The remaining order is D → E → F → G.@@@ -320,7 +320,7 @@ Live carries the integration (`f0ca18b`). It is inert until `OPENRE_URL` is set.defaults to `live` on every slot (`server/openre/schema.js:5-7`), and only RTMP moves(`server/openre/authority.js:18`). WHIP, JSMPEG and SFU are **not ported** (OpenRe `README.md:207-209`).--**Once, before the first slot** (OpenRe `README.md:188-194`):++**Once, before the first slot.** OpenRe's [`docs/cutover.md`](https://github.com/OpenVibers/OpenRe.Stream/blob/main/docs/cutover.md) supersedes the table below: it has the exact commands, the owner and agent steps (A1–A7), and a read-only preflight whose output on 2026-09-23 settles the "Unknown" rows (no subscription, no `OPENRE_*` in `live.env`).| Step | State ||---|---|@@@ -130,6 +130,14 @@ Supported with declared overrides (not yet run on the host with `ovhost drill`):- coupons- trade- codes++- openre (not run yet). Only openre-api starts (`drill.command`), with `OPENRE_DRILL=1`. From OpenRe++ `5e86ea7` on, that makes it read-only: writes, `/play/` and sign-in answer 503. The coordinator++ and every transport worker refuse to start in a drill, and the event relay and Media calls are++ off. The API never runs a transport or the relay in any case. Empty `EVENTS_URL` and++ `OV_OAUTH_CLIENT_SECRET`, plus `MEDIA_URL` on the closed port, cover older releases. The workers++ stay out of every ovhost command: they are `workerUnits`, which are listed and never restarted.++ Run it after the OpenRe release with `OPENRE_DRILL` is deployed (OpenRe `docs/cutover.md`++ phase C).Services that are not deployed yet (everything except network, live, media, tools, community andgames as of 22 Sep) have entries built from their repositories' `deploy/` directories.@@@ -551,17 +551,26 @@"drill": { "supported": false, "reason": "Not written yet: a drill must start AI with every provider key blanked, or queued and new runs would call paid providers. Backups work." }},"openre": {-- "_note": "Release layout: the service runs from /opt/openre.stream/current and deploys with its own deploy/deploy.sh (release | api | workers). The templated ingest and restream workers belong to the session coordinator; ovhost never restarts them.",++ "_note": "Release layout: the service runs from /opt/openre.stream/current and deploys with its own deploy/scripts/deploy.sh (release | api | workers). units are the API and the session coordinator. workerUnits are the transport workers: one instance per release (openre-rtmp-ingest@<release>), started by deploy.sh workers, drained by the coordinator when a newer generation is ready, and exiting on their own. ovhost lists them in status/validate and never starts, stops or restarts them (the inventory refuses a worker unit in units).","repo": "/opt/openre.stream","layout": "release","units": ["openre-api.service", "openre-session-coordinator.service"],++ "workerUnits": ["[email protected]", "[email protected]"],"envFile": "/etc/openvibe/openre.env","env": { "required": "from-example" },"port": 4500,"ready": { "url": "http://127.0.0.1:4500/api/ready", "timeoutSeconds": 30 },"noRestartPaths": ["docs/", "test/", "**/*.md"],"databases": [{ "name": "openre", "path": "/var/lib/openre/openre.db" }],-- "drill": { "supported": false, "reason": "The session coordinator reconciles slots and starts workers at boot; a drill needs a switch that keeps it away from live ingest slots." }++ "drill": {++ "port": 14500,++ "command": ["/usr/bin/node", "server/index.js"],++ "databases": { "openre": "OPENRE_DB_PATH" },++ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "OPENRE_DRILL": "1", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "MEDIA_URL": "http://127.0.0.1:9", "MEDIA_API_KEY": "", "OPENRE_RECORDING": "off" },++ "compare": ["/api/health", "/robots.txt"],++ "counts": [{ "db": "openre", "table": "stream_definitions" }, { "db": "openre", "table": "ingest_keys" }, { "db": "openre", "table": "destinations" }, { "db": "openre", "table": "migration_map" }],++ "_notes": "Only openre-api starts (command; the working directory and the non-secret Environment= come from openre-api.service). It never runs a transport, a coordinator or the event relay in any mode. OPENRE_DRILL=1 (OpenRe 5e86ea7 and later) also makes it read-only: writes, /play/ (which would pull from production's worker on loopback) and sign-in (which would redeem codes at Network) answer 503. The coordinator and the workers refuse to start. The event relay and Media calls are off. EVENTS_URL and OV_OAUTH_CLIENT_SECRET empty and MEDIA_URL on the closed port are the same switches for releases before 5e86ea7. Left on (read-only): the Network JWKS GET on loopback. Not compared: ingest_sessions and workers (they change whenever someone streams or a generation rolls over) and /api/ready (it reports live worker heartbeats)."++ }}}}@@@ -168,9 +168,10 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procfor (const r of rows) {const units = r.units.map((u) => `${u.unit.replace(/\.service$/, '')}=${u.active}${u.restarts ? `(${u.restarts} restarts)` : ''}`).join(' ');const sock = r.socket ? ` socket=${r.socket.active}` : '';++ const workers = r.workers && r.workers.length ? ` workers[${r.workers.map((w) => `${w.unit.replace(/\.service$/, '')}=${w.active}`).join(' ')}]` : '';const ready = r.ready ? (r.ready.ok ? 'ready' : `NOT READY (${r.ready.status || r.ready.error})`) : '-';const prot = r.protected ? (r.protected.count == null ? ` ${r.protected.label}=unknown` : ` ${r.protected.label}=${r.protected.count}`) : '';-- out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${prot}${r.managed ? '' : ' [unmanaged]'}`);++ out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${workers}${prot}${r.managed ? '' : ' [unmanaged]'}`);}return 0;}@@@ -26,7 +26,8 @@* writes to the checkout, the env file, the database or the backup. It never reads an env value:* systemd loads the production env file for the instance, and the override file holds only* inventory values. Environment= values from the unit are passed on only if their names do not look-- * secret.++ * secret and the drill does not set them itself (OpenRe's unit sets OPENRE_DB_PATH to production's++ * database: that line never reaches the drill instance).*/const path = require('path');const { service } = require('../inventory');@@@ -277,9 +278,13 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {for (const f of bk.files) lines.push(envLine(f.envName, vars.db[f.name]));const envPath = path.join(tmp, 'drill.env');await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });++ // A unit Environment= the drill sets itself (its port, a restored database path) is not passed++ // on at all: the override file would win anyway (EnvironmentFile= beats Environment=), but a++ // production path must never reach the drill instance's environment.++ const drillNames = new Set([...Object.keys(d.env), ...bk.files.map((f) => f.envName)]);const props = ['Type=exec',-- ...spec.environment.map((e) => environmentProperty(e.name, e.value)),++ ...spec.environment.filter((e) => !drillNames.has(e.name)).map((e) => environmentProperty(e.name, e.value)),...(svc.envFile ? [`EnvironmentFile=${svc.envFile}`] : []),`EnvironmentFile=${envPath}`,`StandardOutput=append:${logFile}`,@@@ -7,11 +7,19 @@ const { git } = require('../git');async function statusOne(ctx, svc) {const { exec } = ctx;-- const row = { service: svc.id, managed: svc.managed, units: [], socket: null, sha: null, branch: null, ready: null, protected: null };++ const row = { service: svc.id, managed: svc.managed, units: [], workers: [], socket: null, sha: null, branch: null, ready: null, protected: null };for (const u of svc.units) {const st = await systemd.show(exec, u);row.units.push({ unit: u, active: st.active, sub: st.sub, restarts: st.restarts });}++ // Worker units are only listed: ovhost never starts, stops or restarts them.++ for (const w of svc.workerUnits) {++ try {++ for (const i of await systemd.instances(exec, w)) row.workers.push({ unit: i.unit, active: i.active, sub: i.sub });++ } catch (err) {++ row.workers.push({ unit: w, active: 'unknown', error: err.message.split('\n')[0] });++ }++ }if (svc.socketUnit) {const st = await systemd.show(exec, svc.socketUnit);row.socket = { unit: svc.socketUnit, active: st.active, sub: st.sub };@@@ -6,7 +6,8 @@* non-empty (names from the unit's Environment= lines count as present). Values are* never read into the result — only { name, empty }.* units every unit is loaded from a unit file; the socket unit is active; repo unit files that-- * differ from the installed ones are reported.++ * differ from the installed ones are reported. Instances of workerUnits are listed (a warning++ * when none runs); they are never started, stopped or restarted.* port free (service down) or held by the service / its socket — nothing else.* nginx the vhost is in sites-available and linked in sites-enabled; `nginx -t` is clean.* checkout owned by the declared owner, on the declared branch, no tracked changes.@@@ -98,6 +99,13 @@ async function validate(ctx, id) {else add(st.active === 'active' ? 'ok' : 'warn', 'units', `${u} ${st.active}/${st.sub} from ${st.fragmentPath}`);if (st.mainPid) pids.add(st.mainPid);}++ for (const w of svc.workerUnits) {++ let list = [];++ try { list = await systemd.instances(exec, w); } catch (err) { add('warn', 'units', `could not list ${w}: ${err.message}`); continue; }++ const running = list.filter((i) => i.active === 'active');++ if (!running.length) add('warn', 'units', `no running instance of worker unit ${w}`);++ for (const i of list) add('info', 'units', `worker ${i.unit} ${i.active}/${i.sub} (ovhost never starts, stops or restarts it)`);++ }if (svc.socketUnit) {const st = await systemd.show(exec, svc.socketUnit);if (st.load !== 'loaded') add('error', 'units', `${svc.socketUnit} is not loaded (${st.load})`);@@@ -16,6 +16,8 @@ class InventoryError extends Error {}const ID_RE = /^[a-z][a-z0-9-]{0,39}$/;const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;++// A worker unit is a template (`[email protected]`) or a plain service unit.++const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;const DRAIN_POLICIES = ['refuse', 'wait', 'report'];const PROBE_KINDS = ['http-json-count', 'sqlite-count'];@@@ -49,6 +51,10 @@ function normaliseService(id, raw, defaults) {managed: s.layout === 'release' ? false : s.managed !== false,unmanagedReason: s.unmanagedReason || (s.layout === 'release' ? 'release layout: deployed by its own deploy script' : null),units: s.units || [],++ // Units that belong to the service but that ovhost never starts, stops or restarts: e.g.++ // OpenRe's transport workers ([email protected], one instance per release), which++ // drain on their own. status/validate list their instances read-only.++ workerUnits: s.workerUnits || [],socketUnit: s.socketUnit || null,unitSources: s.unitSources || {},envFile: s.envFile ? abs(s.envFile, `${where}.envFile`) : null,@@@ -83,6 +89,15 @@ function normaliseService(id, raw, defaults) {if (u.endsWith('.socket')) throw new InventoryError(`${where}.units: "${u}" is a socket unit — declare it as socketUnit; ovhost never restarts or stops socket units`);}if (out.socketUnit && !(UNIT_RE.test(out.socketUnit) && out.socketUnit.endsWith('.socket'))) throw new InventoryError(`${where}.socketUnit must be a .socket unit`);++ if (!Array.isArray(out.workerUnits)) throw new InventoryError(`${where}.workerUnits must be an array`);++ for (const w of out.workerUnits) {++ if (typeof w !== 'string' || !WORKER_UNIT_RE.test(w)) throw new InventoryError(`${where}.workerUnits: "${w}" is not a .service unit or template`);++ for (const u of out.units) {++ if (u === w || (w.endsWith('@.service') && u.startsWith(w.slice(0, -'.service'.length)))) {++ throw new InventoryError(`${where}.units: "${u}" is a worker unit (${w}); ovhost restarts every unit in units, and never a worker unit`);++ }++ }++ }for (const [unit, src] of Object.entries(out.unitSources)) {if (!UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit or drop-in name`);rel(src, `${where}.unitSources.${unit}`);@@@ -5,8 +5,9 @@* the .socket unit so HTTP connections queue in the kernel through a restart; stopping that unit is* what turns a restart into Cloudflare 502s. Only the .service unit is ever restarted.*/--const SOCKET_SAFE_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'cat', 'status']);--const ALLOWED_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'restart', 'reload', 'cat', 'status', 'daemon-reload']);++const SOCKET_SAFE_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'cat', 'status', 'list-units']);++const ALLOWED_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'restart', 'reload', 'cat', 'status', 'daemon-reload', 'list-units']);++const READ_ONLY_ACTIONS = ['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'];const PROPS = ['LoadState', 'ActiveState', 'SubState', 'MainPID', 'FragmentPath', 'UnitFileState', 'DropInPaths', 'PartOf', 'NRestarts'];class SocketGuardError extends Error {}@@@ -16,8 +17,11 @@ async function systemctl(exec, action, unit) {if (unit && unit.endsWith('.socket') && !SOCKET_SAFE_ACTIONS.has(action)) {throw new SocketGuardError(`refusing "systemctl ${action} ${unit}": socket units are never stopped or restarted (they hold the listener that keeps HTTP up through a restart)`);}-- const args = action === 'show' ? ['show', unit, `--property=${PROPS.join(',')}`] : unit ? [action, unit] : [action];-- const privileged = !['show', 'is-active', 'is-enabled', 'cat', 'status'].includes(action);++ let args;++ if (action === 'show') args = ['show', unit, `--property=${PROPS.join(',')}`];++ else if (action === 'list-units') args = ['list-units', '--all', '--plain', '--no-legend', '--no-pager', unit];++ else args = unit ? [action, unit] : [action];++ const privileged = !READ_ONLY_ACTIONS.includes(action);return exec.run('systemctl', args, { privileged });}@@@ -42,6 +46,22 @@ async function show(exec, unit) {};}++/**++ * Loaded instances of a worker unit (read-only): `[email protected]` lists every `name@*.service`, a++ * plain unit lists itself. -> [{ unit, load, active, sub }]++ */++async function instances(exec, workerUnit) {++ const pattern = workerUnit.endsWith('@.service') ? `${workerUnit.slice(0, -'.service'.length)}*.service` : workerUnit;++ const r = await systemctl(exec, 'list-units', pattern);++ if (r.code !== 0) throw new Error(`systemctl list-units ${pattern} failed: ${(r.stderr || '').trim()}`);++ const out = [];++ for (const line of String(r.stdout || '').split('\n')) {++ const [unit, load, active, sub] = line.trim().split(/\s+/);++ if (unit && unit.endsWith('.service')) out.push({ unit, load, active, sub });++ }++ return out;++}++async function restart(exec, unit) {const r = await systemctl(exec, 'restart', unit);if (r.code !== 0) throw new Error(`systemctl restart ${unit} failed: ${r.stderr.trim()}`);@@@ -66,4 +86,4 @@ async function reloadNginx(exec) {if (r.code !== 0) throw new Error(`systemctl reload nginx failed: ${r.stderr.trim()}`);}--module.exports = { systemctl, show, restart, ensureSocketActive, daemonReload, reloadNginx, SocketGuardError };++module.exports = { systemctl, show, instances, restart, ensureSocketActive, daemonReload, reloadNginx, SocketGuardError };@@@ -188,6 +188,13 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', stcase 'stop': if (u) { u.active = 'inactive'; u.sub = 'dead'; } return { code: 0, stdout: '', stderr: '' };case 'reload': case 'daemon-reload': return { code: 0, stdout: '', stderr: '' };case 'is-active': return { code: u && u.active === 'active' ? 0 : 3, stdout: '', stderr: '' };++ case 'list-units': {++ // systemctl list-units --all --plain --no-legend --no-pager <glob>++ const glob = args[args.length - 1];++ const re = new RegExp(`^${glob.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*')}$`);++ const rows = [...units.entries()].filter(([name]) => re.test(name)).map(([name, x]) => `${name} ${x.load} ${x.active} ${x.sub} ${name}`);++ return { code: 0, stdout: rows.length ? `${rows.join('\n')}\n` : '', stderr: '' };++ }default: return { code: 1, stdout: '', stderr: `fake systemctl: ${action}` };}}nnew file mode 100644@@@ -0,0 +1,196 @@++'use strict';++/**++ * OpenRe on ovhost: `workerUnits` (the transport worker templates) are listed by status and++ * validate and never started, stopped or restarted; the inventory refuses a worker unit in++ * `units`; a deploy of a git-layout service with worker units restarts only its units; and the++ * example inventory's openre drill starts openre-api alone with OPENRE_DRILL=1 and the side-effect++ * switches, leaving the API, the coordinator and every worker untouched.++ */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const { scenario, push, test, runTests, SECRET } = require('./helpers');++const { normalise } = require('../lib/inventory');++++const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));++const RELEASE = '655b98a10aaa';++const OLD = '0123456789ab';++const PROD_DB = '/var/lib/openre/openre.db';++const WORKERS = [`openre-rtmp-ingest@${RELEASE}.service`, `openre-restream-worker@${RELEASE}.service`];++++/** The standard fake host plus OpenRe as on openvibe-ovh: release layout, two units, worker instances. */++async function openreHost({ workersRunning = true } = {}) {++ const host = scenario();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.openre = JSON.parse(JSON.stringify(EXAMPLE.services.openre));++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++++ host.put(`/opt/openre.stream/releases/${RELEASE}/package.json`, '{"name":"openre-stream"}', { owner: 'ubuntu' });++ host.put(`/opt/openre.stream/releases/${RELEASE}/.env.example`, 'OV_OAUTH_CLIENT_ID=openre\nOV_OAUTH_CLIENT_SECRET=\nOPENRE_SECRETS_KEY=\n#OPENRE_DRILL=\n', { owner: 'ubuntu' });++ host.files.get(`/opt/openre.stream/releases/${RELEASE}`).owner = 'ubuntu';++ await host.exec.symlink(`/opt/openre.stream/releases/${RELEASE}`, '/opt/openre.stream/current');++ host.put('/etc/openvibe/openre.env', `OV_OAUTH_CLIENT_ID=openre\nOV_OAUTH_CLIENT_SECRET=${SECRET}\nOPENRE_SECRETS_KEY=${SECRET}-k\nOPENRE_RTMP_BIND=0.0.0.0\n`, { mode: 0o600 });++ host.put('/etc/systemd/system/openre-api.service', [++ '[Service]',++ 'User=ubuntu',++ 'WorkingDirectory=/opt/openre.stream/current',++ 'EnvironmentFile=/etc/openvibe/openre.env',++ 'Environment=NODE_ENV=production',++ 'Environment=PORT=4500',++ `Environment=OPENRE_DB_PATH=${PROD_DB}`,++ 'Environment=PATH=/usr/local/bin:/usr/bin:/bin',++ 'ExecStart=/usr/bin/env node server/index.js',++ '',++ ].join('\n'));++ host.addUnit('openre-api.service', { mainPid: 3100 });++ host.addUnit('openre-session-coordinator.service', { mainPid: 3101 });++ host.alivePids.add(3100);++ host.alivePids.add(3101);++ host.listeners.set(4500, [{ pid: 3100, process: 'node' }]);++ host.addUnit(`openre-rtmp-ingest@${RELEASE}.service`, { mainPid: 3102, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });++ host.addUnit(`openre-restream-worker@${RELEASE}.service`, { mainPid: 3103, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });++ host.addUnit(`openre-rtmp-ingest@${OLD}.service`, { active: 'inactive', sub: 'dead', mainPid: 0 });++ host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });++++ const health = '{"status":"ok","service":"openre-api","version":"0.1.0","release":"dev"}';++ const robots = 'User-agent: *\nAllow: /$\n';++ host.http.set('http://127.0.0.1:4500/api/ready', () => ({ status: 200, body: { status: 'ready' } }));++ host.http.set('http://127.0.0.1:4500/api/health', () => ({ status: 200, body: health }));++ host.http.set('http://127.0.0.1:4500/robots.txt', () => ({ status: 200, body: robots }));++ const counts = { stream_definitions: 1, ingest_keys: 1, destinations: 0, migration_map: 0 };++ host.sqliteHandler = (db, sql) => {++ if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];++ const m = /FROM "([a-z_]+)"/.exec(sql);++ return [{ n: m ? counts[m[1]] ?? 0 : 0 }];++ };++ host.onSystemdRun = (spec) => {++ host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);++ host.drillEnv = host.read(spec.envFiles[spec.envFiles.length - 1]);++ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });++ host.http.set('http://127.0.0.1:14500/api/ready', alive(() => ({ status: 200, body: { status: 'ready', mode: 'drill' } })));++ host.http.set('http://127.0.0.1:14500/api/health', alive(() => ({ status: 200, body: health })));++ host.http.set('http://127.0.0.1:14500/robots.txt', alive(() => ({ status: 200, body: robots })));++ return undefined;++ };++ return host;++}++++/** systemctl calls that could change a unit's state. */++const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));++++runTests([++ test('the example inventory declares the OpenRe worker templates and a supported drill', () => {++ const inv = normalise(EXAMPLE);++ const o = inv.services.openre;++ assert.deepStrictEqual(o.units, ['openre-api.service', 'openre-session-coordinator.service']);++ assert.deepStrictEqual(o.workerUnits, ['[email protected]', '[email protected]']);++ assert.strictEqual(o.layout, 'release');++ assert.strictEqual(o.managed, false);++ assert.strictEqual(o.drill.supported, true);++ assert.strictEqual(o.drill.port, 14500);++ assert.deepStrictEqual(o.drill.command, ['/usr/bin/node', 'server/index.js']);++ assert.strictEqual(o.drill.env.OPENRE_DRILL, '1');++ for (const k of ['EVENTS_URL', 'OV_OAUTH_CLIENT_SECRET', 'MEDIA_API_KEY']) assert.strictEqual(o.drill.env[k], '', k);++ assert.strictEqual(o.drill.env.MEDIA_URL, 'http://127.0.0.1:9');++ assert.deepStrictEqual(o.drill.databases, { openre: 'OPENRE_DB_PATH' });++ // every other service keeps an empty list++ assert.deepStrictEqual(inv.services.live.workerUnits, []);++ }),++++ test('a worker unit, or an instance of a worker template, is refused in units', () => {++ const base = { owner: 'ubuntu', repo: '/opt/x' };++ assert.throws(() => normalise({ services: { x: { ...base, units: ['x-api.service', '[email protected]'], workerUnits: ['[email protected]'] } } }), /"x-worker@abc\.service" is a worker unit \(x-worker@\.service\)/);++ assert.throws(() => normalise({ services: { x: { ...base, units: ['x-w.service'], workerUnits: ['x-w.service'] } } }), /is a worker unit/);++ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: ['x.socket'] } } }), /not a \.service unit or template/);++ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: ['../x.service'] } } }), /not a \.service unit or template/);++ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: '[email protected]' } } }), /must be an array/);++ const ok = normalise({ services: { x: { ...base, units: ['x-worker-api.service'], workerUnits: ['[email protected]'] } } });++ assert.deepStrictEqual(ok.services.x.workerUnits, ['[email protected]'], 'a unit that only shares a prefix without the @ is fine');++ }),++++ test('status lists worker instances read-only and never changes a unit', async () => {++ const host = await openreHost();++ const r = await host.cli('status', 'openre');++ assert.strictEqual(r.code, 0, r.out);++ assert.match(r.out, new RegExp(`openre\\s+${RELEASE}\\s+ready\\s+openre-api=active openre-session-coordinator=active workers\\[openre-rtmp-ingest@${RELEASE}=active openre-rtmp-ingest@${OLD}=inactive openre-restream-worker@${RELEASE}=active\\] \\[unmanaged\\]`));++ const json = JSON.parse((await host.cli('status', 'openre', '--json')).out);++ assert.deepStrictEqual(json[0].workers.map((w) => [w.unit, w.active]), [[`openre-rtmp-ingest@${RELEASE}.service`, 'active'], [`openre-rtmp-ingest@${OLD}.service`, 'inactive'], [`openre-restream-worker@${RELEASE}.service`, 'active']]);++ const lists = host.calls.filter((c) => c.cmd === 'systemctl' && c.args[0] === 'list-units');++ assert.deepStrictEqual(lists[0].args, ['list-units', '--all', '--plain', '--no-legend', '--no-pager', 'openre-rtmp-ingest@*.service']);++ assert.ok(lists.every((c) => !c.privileged), 'listing needs no privileges');++ assert.deepStrictEqual(stateChanges(host), []);++ }),++++ test('validate lists the workers as info, warns when none runs, and changes nothing', async () => {++ let host = await openreHost();++ let res = JSON.parse((await host.cli('validate', 'openre', '--json')).out);++ const workers = res.findings.filter((f) => f.area === 'units' && /^worker /.test(f.message));++ assert.deepStrictEqual(workers.map((f) => f.level), ['info', 'info', 'info']);++ assert.match(workers[0].message, /never starts, stops or restarts it/);++ assert.ok(!res.findings.some((f) => /no running instance/.test(f.message)));++ assert.deepStrictEqual(stateChanges(host), []);++ host = await openreHost({ workersRunning: false });++ res = JSON.parse((await host.cli('validate', 'openre', '--json')).out);++ assert.deepStrictEqual(res.findings.filter((f) => /no running instance/.test(f.message)).map((f) => [f.level, f.message]), [++ ['warn', 'no running instance of worker unit [email protected]'],++ ['warn', 'no running instance of worker unit [email protected]'],++ ]);++ assert.deepStrictEqual(stateChanges(host), []);++ assert.ok(!JSON.stringify(res).includes(SECRET));++ }),++++ test('a deploy of a git-layout service with worker units restarts its units only', async () => {++ const host = scenario();++ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));++ doc.services.events.workerUnits = ['[email protected]'];++ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });++ host.addUnit('[email protected]', { mainPid: 5001 });++ push(host, 'events', { 'server/index.js': 'events(2);' });++ const r = await host.cli('deploy', 'events');++ assert.strictEqual(r.code, 0, r.out);++ assert.deepStrictEqual(host.restarts(), ['openvibe-events.service']);++ assert.ok(!host.calls.some((c) => c.cmd === 'systemctl' && c.args.includes('[email protected]')), 'the worker instance is never named in a systemctl call');++ assert.strictEqual(host.units.get('[email protected]').restarts, 0);++ }),++++ test('ovhost drill openre: openre-api alone, OPENRE_DRILL=1, side effects off, production untouched', async () => {++ const host = await openreHost();++ const b = await host.cli('backup', 'openre', '--json');++ assert.strictEqual(b.code, 0, b.out);++ host.advance(60 * 1000);++ const callsBefore = host.calls.length;++ const r = await host.cli('drill', 'openre');++ assert.strictEqual(r.code, 0, r.out);++ const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/openre.jsonl').trim().split('\n').pop());++ assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));++ assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/health', true], ['/robots.txt', true]]);++ assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.match]), [['stream_definitions', true], ['ingest_keys', true], ['destinations', true], ['migration_map', true]]);++++ assert.strictEqual(host.systemdRuns.length, 1, 'one process');++ const run = host.systemdRuns[0];++ assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js'], 'the API, never a worker or the coordinator');++ assert.strictEqual(run.cwd, '/opt/openre.stream/current');++ assert.strictEqual(run.uid, 'ubuntu');++ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/openre.env', `${rec.dir}/drill.env`]);++ for (const p of ['SocketBindAllow=tcp:14500', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);++ assert.ok(run.props.includes('Environment=PATH=/usr/local/bin:/usr/bin:/bin'), 'other unit Environment= is passed on');++ assert.ok(!run.props.some((p) => /OPENRE_DB_PATH|Environment=PORT=/.test(p)), 'the production database path and port from the unit never reach the drill instance');++ const env = Object.fromEntries(host.drillEnv.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));++ assert.deepStrictEqual(env, {++ PORT: '14500', HOST: '127.0.0.1', OPENRE_DRILL: '1', EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', MEDIA_API_KEY: '', OPENRE_RECORDING: 'off',++ OPENRE_DB_PATH: `${rec.dir}/db/openre.db`,++ });++++ const calls = host.calls.slice(callsBefore);++ assert.deepStrictEqual(calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', run.pid]], 'only the drill instance is signalled');++ assert.deepStrictEqual(stateChanges(host), [], 'no unit was started, stopped or restarted');++ for (const u of ['openre-api.service', 'openre-session-coordinator.service', ...WORKERS]) {++ assert.strictEqual(host.units.get(u).active, 'active', u);++ assert.strictEqual(host.units.get(u).restarts, 0, u);++ assert.ok(!calls.some((c) => c.args && c.args.includes(u) && c.args[0] !== 'show'), `${u} is only ever read`);++ }++ assert.ok(host.alivePids.has(3100) && host.alivePids.has(3101));++ assert.strictEqual(host.read(PROD_DB), 'sqlite-production');++ assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/openre.jsonl')).includes(SECRET));++ }),++]);
@@ -150,13 +150,13 @@ Each inventory entry's `drill` block declares:
@@ -150,13 +150,13 @@ Each inventory entry's `drill` block declares:
- `counts`: `{ db, table }` pairs;
- `counts`: `{ db, table }` pairs;
- `supported: false` with a `reason` when a second instance cannot run without side effects.
- `supported: false` with a `reason` when a second instance cannot run without side effects.
-In `host.example.json`, 18 services have drill blocks. live, media, tools and games are marked unsupported, and the reasons are in the file.
+In `host.example.json`, 24 services have drill blocks, and 19 can be drilled. live, media, tools, games and ai are marked unsupported, and the reasons are in the file. `command` (an absolute argv) is required when a service has several units, as openre does: only `openre-api` runs in its drill.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
The sandbox blocks writes outside the drill directory and addresses beyond loopback. Loopback stays open, so the overrides are what keep a drill away from production services. Common overrides are `OV_OAUTH_CLIENT_SECRET=` (no service tokens) and `http://127.0.0.1:9` for URLs whose empty value would fall back to a production service.
### Inventory
### Inventory
-`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange` and `nginx`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
+`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe, the `drain` policy, `databases`, `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service deployed by its own script, which ovhost does not manage) and `workerUnits`. `workerUnits` are units, or templates such as OpenRe's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
`games` is listed as `managed: false`. It is a pnpm workspace with a TypeScript build, and those steps are not encoded yet. `status`, `validate`, `snapshot` and `backup` work for it; `deploy` and `rollback` refuse it.
`games` is listed as `managed: false`. It is a pnpm workspace with a TypeScript build, and those steps are not encoded yet. `status`, `validate`, `snapshot` and `backup` work for it; `deploy` and `rollback` refuse it.
@@ -350,7 +350,7 @@ The drill:
@@ -350,7 +350,7 @@ The drill:
Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
Record the result in [restore-drills.md](restore-drills.md) and note "off-host" in the Backup column.
-For services without a drill (live, media, tools, games, openre), download the copy and check it by
+For services without a drill (live, media, tools, games, ai), download the copy and check it by
hand:
hand:
```
```
@@ -29,7 +29,7 @@ State markers:
@@ -29,7 +29,7 @@ State markers:
| C | AI → OpenVibe.AI (Wave 13) | **DONE** 2026-09-23 ~15:40 UTC (`AI_SERVICE=remote`) | AI [`docs/migration.md`](https://github.com/OpenVibers/OpenVibe.AI/blob/main/docs/migration.md) | Unset `AI_SERVICE`, restart Live |
| C | AI → OpenVibe.AI (Wave 13) | **DONE** 2026-09-23 ~15:40 UTC (`AI_SERVICE=remote`) | AI [`docs/migration.md`](https://github.com/OpenVibers/OpenVibe.AI/blob/main/docs/migration.md) | Unset `AI_SERVICE`, restart Live |
| D | Events delivery signature v2 | **IN PROGRESS**: Events sends v2; consumer deploys unknown; v1 still sent | Events [`docs/replay-window-rollout.md`](https://github.com/OpenVibers/OpenVibe.Events/blob/main/docs/replay-window-rollout.md) | The consumer redeploys its previous release; replay the DLQ |
| D | Events delivery signature v2 | **IN PROGRESS**: Events sends v2; consumer deploys unknown; v1 still sent | Events [`docs/replay-window-rollout.md`](https://github.com/OpenVibers/OpenVibe.Events/blob/main/docs/replay-window-rollout.md) | The consumer redeploys its previous release; replay the DLQ |
| E | Money → OpenVibe.Billing | **BLOCKED (owner)**: PowerChat dashboard. Billing runs in shadow. | Billing [`docs/live-cutover.md`](https://github.com/OpenVibers/OpenVibe.Billing/blob/main/docs/live-cutover.md) | Depends on the step reached. After step 9 there is **no tool** |
| E | Money → OpenVibe.Billing | **BLOCKED (owner)**: PowerChat dashboard. Billing runs in shadow. | Billing [`docs/live-cutover.md`](https://github.com/OpenVibers/OpenVibe.Billing/blob/main/docs/live-cutover.md) | Depends on the step reached. After step 9 there is **no tool** |
-| F | RTMP ingest → OpenRe.Stream | **BLOCKED (owner)**: DNS `ingest.openre.stream` and port 1936. OpenRe is deployed. | OpenRe [`README.md` "Cutover runbook"](https://github.com/OpenVibers/OpenRe.Stream/blob/main/README.md#cutover-runbook) | Per slot `{"authority":"live"}`. All slots: unset `OPENRE_URL` |
+| F | RTMP ingest → OpenRe.Stream | **BLOCKED (owner)**: port 1936 at the provider edge. DNS `ingest.openre.stream` is done and OpenRe is deployed. | OpenRe [`docs/cutover.md`](https://github.com/OpenVibers/OpenRe.Stream/blob/main/docs/cutover.md) (steps, checks and rollback; `scripts/cutover-preflight.js`) | Per slot `{"authority":"live"}` + `set-definition-state.js --state disabled`. All slots: unset `OPENRE_URL` |
| G | Retire the shims | **NOT STARTED**. Dated in [compatibility-register.md](compatibility-register.md) | Network [`docs/retirement.md`](https://github.com/OpenVibers/OpenVibe.Network/blob/main/docs/retirement.md) | per entry |
| G | Retire the shims | **NOT STARTED**. Dated in [compatibility-register.md](compatibility-register.md) | Network [`docs/retirement.md`](https://github.com/OpenVibers/OpenVibe.Network/blob/main/docs/retirement.md) | per entry |
The remaining order is D → E → F → G.
The remaining order is D → E → F → G.
@@ -320,7 +320,7 @@ Live carries the integration (`f0ca18b`). It is inert until `OPENRE_URL` is set.
@@ -320,7 +320,7 @@ Live carries the integration (`f0ca18b`). It is inert until `OPENRE_URL` is set.
defaults to `live` on every slot (`server/openre/schema.js:5-7`), and only RTMP moves
defaults to `live` on every slot (`server/openre/schema.js:5-7`), and only RTMP moves
(`server/openre/authority.js:18`). WHIP, JSMPEG and SFU are **not ported** (OpenRe `README.md:207-209`).
(`server/openre/authority.js:18`). WHIP, JSMPEG and SFU are **not ported** (OpenRe `README.md:207-209`).
-**Once, before the first slot** (OpenRe `README.md:188-194`):
+**Once, before the first slot.** OpenRe's [`docs/cutover.md`](https://github.com/OpenVibers/OpenRe.Stream/blob/main/docs/cutover.md) supersedes the table below: it has the exact commands, the owner and agent steps (A1–A7), and a read-only preflight whose output on 2026-09-23 settles the "Unknown" rows (no subscription, no `OPENRE_*` in `live.env`).
| Step | State |
| Step | State |
|---|---|
|---|---|
@@ -130,6 +130,14 @@ Supported with declared overrides (not yet run on the host with `ovhost drill`):
@@ -130,6 +130,14 @@ Supported with declared overrides (not yet run on the host with `ovhost drill`):
- coupons
- coupons
- trade
- trade
- codes
- codes
+- openre (not run yet). Only openre-api starts (`drill.command`), with `OPENRE_DRILL=1`. From OpenRe
+ `5e86ea7` on, that makes it read-only: writes, `/play/` and sign-in answer 503. The coordinator
+ and every transport worker refuse to start in a drill, and the event relay and Media calls are
+ off. The API never runs a transport or the relay in any case. Empty `EVENTS_URL` and
+ `OV_OAUTH_CLIENT_SECRET`, plus `MEDIA_URL` on the closed port, cover older releases. The workers
+ stay out of every ovhost command: they are `workerUnits`, which are listed and never restarted.
+ Run it after the OpenRe release with `OPENRE_DRILL` is deployed (OpenRe `docs/cutover.md`
+ phase C).
Services that are not deployed yet (everything except network, live, media, tools, community and
Services that are not deployed yet (everything except network, live, media, tools, community and
games as of 22 Sep) have entries built from their repositories' `deploy/` directories.
games as of 22 Sep) have entries built from their repositories' `deploy/` directories.
@@ -551,17 +551,26 @@
@@ -551,17 +551,26 @@
"drill": { "supported": false, "reason": "Not written yet: a drill must start AI with every provider key blanked, or queued and new runs would call paid providers. Backups work." }
"drill": { "supported": false, "reason": "Not written yet: a drill must start AI with every provider key blanked, or queued and new runs would call paid providers. Backups work." }
},
},
"openre": {
"openre": {
- "_note": "Release layout: the service runs from /opt/openre.stream/current and deploys with its own deploy/deploy.sh (release | api | workers). The templated ingest and restream workers belong to the session coordinator; ovhost never restarts them.",
+ "_note": "Release layout: the service runs from /opt/openre.stream/current and deploys with its own deploy/scripts/deploy.sh (release | api | workers). units are the API and the session coordinator. workerUnits are the transport workers: one instance per release (openre-rtmp-ingest@<release>), started by deploy.sh workers, drained by the coordinator when a newer generation is ready, and exiting on their own. ovhost lists them in status/validate and never starts, stops or restarts them (the inventory refuses a worker unit in units).",
"repo": "/opt/openre.stream",
"repo": "/opt/openre.stream",
"layout": "release",
"layout": "release",
"units": ["openre-api.service", "openre-session-coordinator.service"],
"units": ["openre-api.service", "openre-session-coordinator.service"],
+ "workerUnits": ["[email protected]", "[email protected]"],
"envFile": "/etc/openvibe/openre.env",
"envFile": "/etc/openvibe/openre.env",
"env": { "required": "from-example" },
"env": { "required": "from-example" },
"port": 4500,
"port": 4500,
"ready": { "url": "http://127.0.0.1:4500/api/ready", "timeoutSeconds": 30 },
"ready": { "url": "http://127.0.0.1:4500/api/ready", "timeoutSeconds": 30 },
"noRestartPaths": ["docs/", "test/", "**/*.md"],
"noRestartPaths": ["docs/", "test/", "**/*.md"],
"databases": [{ "name": "openre", "path": "/var/lib/openre/openre.db" }],
"databases": [{ "name": "openre", "path": "/var/lib/openre/openre.db" }],
- "drill": { "supported": false, "reason": "The session coordinator reconciles slots and starts workers at boot; a drill needs a switch that keeps it away from live ingest slots." }
+ "drill": {
+ "port": 14500,
+ "command": ["/usr/bin/node", "server/index.js"],
+ "databases": { "openre": "OPENRE_DB_PATH" },
+ "env": { "PORT": "{port}", "HOST": "127.0.0.1", "OPENRE_DRILL": "1", "EVENTS_URL": "", "OV_OAUTH_CLIENT_SECRET": "", "MEDIA_URL": "http://127.0.0.1:9", "MEDIA_API_KEY": "", "OPENRE_RECORDING": "off" },
+ "compare": ["/api/health", "/robots.txt"],
+ "counts": [{ "db": "openre", "table": "stream_definitions" }, { "db": "openre", "table": "ingest_keys" }, { "db": "openre", "table": "destinations" }, { "db": "openre", "table": "migration_map" }],
+ "_notes": "Only openre-api starts (command; the working directory and the non-secret Environment= come from openre-api.service). It never runs a transport, a coordinator or the event relay in any mode. OPENRE_DRILL=1 (OpenRe 5e86ea7 and later) also makes it read-only: writes, /play/ (which would pull from production's worker on loopback) and sign-in (which would redeem codes at Network) answer 503. The coordinator and the workers refuse to start. The event relay and Media calls are off. EVENTS_URL and OV_OAUTH_CLIENT_SECRET empty and MEDIA_URL on the closed port are the same switches for releases before 5e86ea7. Left on (read-only): the Network JWKS GET on loopback. Not compared: ingest_sessions and workers (they change whenever someone streams or a generation rolls over) and /api/ready (it reports live worker heartbeats)."
+ }
}
}
}
}
}
}
@@ -168,9 +168,10 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -168,9 +168,10 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
for (const r of rows) {
for (const r of rows) {
const units = r.units.map((u) => `${u.unit.replace(/\.service$/, '')}=${u.active}${u.restarts ? `(${u.restarts} restarts)` : ''}`).join(' ');
const units = r.units.map((u) => `${u.unit.replace(/\.service$/, '')}=${u.active}${u.restarts ? `(${u.restarts} restarts)` : ''}`).join(' ');
const sock = r.socket ? ` socket=${r.socket.active}` : '';
const sock = r.socket ? ` socket=${r.socket.active}` : '';
+ const workers = r.workers && r.workers.length ? ` workers[${r.workers.map((w) => `${w.unit.replace(/\.service$/, '')}=${w.active}`).join(' ')}]` : '';
const ready = r.ready ? (r.ready.ok ? 'ready' : `NOT READY (${r.ready.status || r.ready.error})`) : '-';
const ready = r.ready ? (r.ready.ok ? 'ready' : `NOT READY (${r.ready.status || r.ready.error})`) : '-';
const prot = r.protected ? (r.protected.count == null ? ` ${r.protected.label}=unknown` : ` ${r.protected.label}=${r.protected.count}`) : '';
const prot = r.protected ? (r.protected.count == null ? ` ${r.protected.label}=unknown` : ` ${r.protected.label}=${r.protected.count}`) : '';
- out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${prot}${r.managed ? '' : ' [unmanaged]'}`);
+ out(`${r.service.padEnd(10)} ${short(r.sha)} ${ready.padEnd(12)} ${units || '(no units)'}${sock}${workers}${prot}${r.managed ? '' : ' [unmanaged]'}`);
}
}
return 0;
return 0;
}
}
@@ -26,7 +26,8 @@
@@ -26,7 +26,8 @@
* writes to the checkout, the env file, the database or the backup. It never reads an env value:
* writes to the checkout, the env file, the database or the backup. It never reads an env value:
* systemd loads the production env file for the instance, and the override file holds only
* systemd loads the production env file for the instance, and the override file holds only
* inventory values. Environment= values from the unit are passed on only if their names do not look
* inventory values. Environment= values from the unit are passed on only if their names do not look
- * secret.
+ * secret and the drill does not set them itself (OpenRe's unit sets OPENRE_DB_PATH to production's
+ * database: that line never reaches the drill instance).
*/
*/
const path = require('path');
const path = require('path');
const { service } = require('../inventory');
const { service } = require('../inventory');
@@ -277,9 +278,13 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
@@ -277,9 +278,13 @@ async function drill(ctx, id, { backup: backupArg = null, keep = false } = {}) {
for (const f of bk.files) lines.push(envLine(f.envName, vars.db[f.name]));
for (const f of bk.files) lines.push(envLine(f.envName, vars.db[f.name]));
const envPath = path.join(tmp, 'drill.env');
const envPath = path.join(tmp, 'drill.env');
await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });
await exec.writeFile(envPath, `${lines.join('\n')}\n`, { mode: 0o640 });
+ // A unit Environment= the drill sets itself (its port, a restored database path) is not passed
+ // on at all: the override file would win anyway (EnvironmentFile= beats Environment=), but a
+ // production path must never reach the drill instance's environment.
+ const drillNames = new Set([...Object.keys(d.env), ...bk.files.map((f) => f.envName)]);
const props = [
const props = [
'Type=exec',
'Type=exec',
- ...spec.environment.map((e) => environmentProperty(e.name, e.value)),
+ ...spec.environment.filter((e) => !drillNames.has(e.name)).map((e) => environmentProperty(e.name, e.value)),
...(svc.envFile ? [`EnvironmentFile=${svc.envFile}`] : []),
...(svc.envFile ? [`EnvironmentFile=${svc.envFile}`] : []),
`EnvironmentFile=${envPath}`,
`EnvironmentFile=${envPath}`,
`StandardOutput=append:${logFile}`,
`StandardOutput=append:${logFile}`,
@@ -7,11 +7,19 @@ const { git } = require('../git');
@@ -7,11 +7,19 @@ const { git } = require('../git');
async function statusOne(ctx, svc) {
async function statusOne(ctx, svc) {
const { exec } = ctx;
const { exec } = ctx;
- const row = { service: svc.id, managed: svc.managed, units: [], socket: null, sha: null, branch: null, ready: null, protected: null };
+ const row = { service: svc.id, managed: svc.managed, units: [], workers: [], socket: null, sha: null, branch: null, ready: null, protected: null };
for (const u of svc.units) {
for (const u of svc.units) {
const st = await systemd.show(exec, u);
const st = await systemd.show(exec, u);
row.units.push({ unit: u, active: st.active, sub: st.sub, restarts: st.restarts });
row.units.push({ unit: u, active: st.active, sub: st.sub, restarts: st.restarts });
}
}
+ // Worker units are only listed: ovhost never starts, stops or restarts them.
+ for (const w of svc.workerUnits) {
+ try {
+ for (const i of await systemd.instances(exec, w)) row.workers.push({ unit: i.unit, active: i.active, sub: i.sub });
+ } catch (err) {
+ row.workers.push({ unit: w, active: 'unknown', error: err.message.split('\n')[0] });
+ }
+ }
if (svc.socketUnit) {
if (svc.socketUnit) {
const st = await systemd.show(exec, svc.socketUnit);
const st = await systemd.show(exec, svc.socketUnit);
row.socket = { unit: svc.socketUnit, active: st.active, sub: st.sub };
row.socket = { unit: svc.socketUnit, active: st.active, sub: st.sub };
@@ -6,7 +6,8 @@
@@ -6,7 +6,8 @@
* non-empty (names from the unit's Environment= lines count as present). Values are
* non-empty (names from the unit's Environment= lines count as present). Values are
* never read into the result — only { name, empty }.
* never read into the result — only { name, empty }.
* units every unit is loaded from a unit file; the socket unit is active; repo unit files that
* units every unit is loaded from a unit file; the socket unit is active; repo unit files that
- * differ from the installed ones are reported.
+ * differ from the installed ones are reported. Instances of workerUnits are listed (a warning
+ * when none runs); they are never started, stopped or restarted.
* port free (service down) or held by the service / its socket — nothing else.
* port free (service down) or held by the service / its socket — nothing else.
* nginx the vhost is in sites-available and linked in sites-enabled; `nginx -t` is clean.
* nginx the vhost is in sites-available and linked in sites-enabled; `nginx -t` is clean.
* checkout owned by the declared owner, on the declared branch, no tracked changes.
* checkout owned by the declared owner, on the declared branch, no tracked changes.
@@ -98,6 +99,13 @@ async function validate(ctx, id) {
@@ -98,6 +99,13 @@ async function validate(ctx, id) {
else add(st.active === 'active' ? 'ok' : 'warn', 'units', `${u} ${st.active}/${st.sub} from ${st.fragmentPath}`);
else add(st.active === 'active' ? 'ok' : 'warn', 'units', `${u} ${st.active}/${st.sub} from ${st.fragmentPath}`);
if (st.mainPid) pids.add(st.mainPid);
if (st.mainPid) pids.add(st.mainPid);
}
}
+ for (const w of svc.workerUnits) {
+ let list = [];
+ try { list = await systemd.instances(exec, w); } catch (err) { add('warn', 'units', `could not list ${w}: ${err.message}`); continue; }
+ const running = list.filter((i) => i.active === 'active');
+ if (!running.length) add('warn', 'units', `no running instance of worker unit ${w}`);
+ for (const i of list) add('info', 'units', `worker ${i.unit} ${i.active}/${i.sub} (ovhost never starts, stops or restarts it)`);
+ }
if (svc.socketUnit) {
if (svc.socketUnit) {
const st = await systemd.show(exec, svc.socketUnit);
const st = await systemd.show(exec, svc.socketUnit);
if (st.load !== 'loaded') add('error', 'units', `${svc.socketUnit} is not loaded (${st.load})`);
if (st.load !== 'loaded') add('error', 'units', `${svc.socketUnit} is not loaded (${st.load})`);
@@ -16,6 +16,8 @@ class InventoryError extends Error {}
@@ -16,6 +16,8 @@ class InventoryError extends Error {}
const ID_RE = /^[a-z][a-z0-9-]{0,39}$/;
const ID_RE = /^[a-z][a-z0-9-]{0,39}$/;
const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
const UNIT_RE = /^[A-Za-z0-9@._-]+\.(service|socket)$/;
+// A worker unit is a template (`[email protected]`) or a plain service unit.
+const WORKER_UNIT_RE = /^[A-Za-z0-9@._-]+\.service$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const FILE_NAME_RE = /^[A-Za-z0-9._-]+$/;
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
const DRAIN_POLICIES = ['refuse', 'wait', 'report'];
const PROBE_KINDS = ['http-json-count', 'sqlite-count'];
const PROBE_KINDS = ['http-json-count', 'sqlite-count'];
@@ -49,6 +51,10 @@ function normaliseService(id, raw, defaults) {
@@ -49,6 +51,10 @@ function normaliseService(id, raw, defaults) {
managed: s.layout === 'release' ? false : s.managed !== false,
managed: s.layout === 'release' ? false : s.managed !== false,
unmanagedReason: s.unmanagedReason || (s.layout === 'release' ? 'release layout: deployed by its own deploy script' : null),
unmanagedReason: s.unmanagedReason || (s.layout === 'release' ? 'release layout: deployed by its own deploy script' : null),
units: s.units || [],
units: s.units || [],
+ // Units that belong to the service but that ovhost never starts, stops or restarts: e.g.
+ // OpenRe's transport workers ([email protected], one instance per release), which
+ // drain on their own. status/validate list their instances read-only.
+ workerUnits: s.workerUnits || [],
socketUnit: s.socketUnit || null,
socketUnit: s.socketUnit || null,
unitSources: s.unitSources || {},
unitSources: s.unitSources || {},
envFile: s.envFile ? abs(s.envFile, `${where}.envFile`) : null,
envFile: s.envFile ? abs(s.envFile, `${where}.envFile`) : null,
@@ -83,6 +89,15 @@ function normaliseService(id, raw, defaults) {
@@ -83,6 +89,15 @@ function normaliseService(id, raw, defaults) {
if (u.endsWith('.socket')) throw new InventoryError(`${where}.units: "${u}" is a socket unit — declare it as socketUnit; ovhost never restarts or stops socket units`);
if (u.endsWith('.socket')) throw new InventoryError(`${where}.units: "${u}" is a socket unit — declare it as socketUnit; ovhost never restarts or stops socket units`);
}
}
if (out.socketUnit && !(UNIT_RE.test(out.socketUnit) && out.socketUnit.endsWith('.socket'))) throw new InventoryError(`${where}.socketUnit must be a .socket unit`);
if (out.socketUnit && !(UNIT_RE.test(out.socketUnit) && out.socketUnit.endsWith('.socket'))) throw new InventoryError(`${where}.socketUnit must be a .socket unit`);
+ if (!Array.isArray(out.workerUnits)) throw new InventoryError(`${where}.workerUnits must be an array`);
+ for (const w of out.workerUnits) {
+ if (typeof w !== 'string' || !WORKER_UNIT_RE.test(w)) throw new InventoryError(`${where}.workerUnits: "${w}" is not a .service unit or template`);
+ for (const u of out.units) {
+ if (u === w || (w.endsWith('@.service') && u.startsWith(w.slice(0, -'.service'.length)))) {
+ throw new InventoryError(`${where}.units: "${u}" is a worker unit (${w}); ovhost restarts every unit in units, and never a worker unit`);
+ }
+ }
+ }
for (const [unit, src] of Object.entries(out.unitSources)) {
for (const [unit, src] of Object.entries(out.unitSources)) {
if (!UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit or drop-in name`);
if (!UNIT_RE.test(unit) && !/^[A-Za-z0-9@._-]+\.(service|socket)\.d\/[A-Za-z0-9._-]+\.conf$/.test(unit)) throw new InventoryError(`${where}.unitSources: "${unit}" is not a unit or drop-in name`);
rel(src, `${where}.unitSources.${unit}`);
rel(src, `${where}.unitSources.${unit}`);
@@ -5,8 +5,9 @@
@@ -5,8 +5,9 @@
* the .socket unit so HTTP connections queue in the kernel through a restart; stopping that unit is
* the .socket unit so HTTP connections queue in the kernel through a restart; stopping that unit is
* what turns a restart into Cloudflare 502s. Only the .service unit is ever restarted.
* what turns a restart into Cloudflare 502s. Only the .service unit is ever restarted.
*/
*/
-const SOCKET_SAFE_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'cat', 'status']);
-const ALLOWED_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'restart', 'reload', 'cat', 'status', 'daemon-reload']);
+const SOCKET_SAFE_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'cat', 'status', 'list-units']);
+const ALLOWED_ACTIONS = new Set(['show', 'is-active', 'is-enabled', 'start', 'restart', 'reload', 'cat', 'status', 'daemon-reload', 'list-units']);
+const READ_ONLY_ACTIONS = ['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'];
const PROPS = ['LoadState', 'ActiveState', 'SubState', 'MainPID', 'FragmentPath', 'UnitFileState', 'DropInPaths', 'PartOf', 'NRestarts'];
const PROPS = ['LoadState', 'ActiveState', 'SubState', 'MainPID', 'FragmentPath', 'UnitFileState', 'DropInPaths', 'PartOf', 'NRestarts'];
class SocketGuardError extends Error {}
class SocketGuardError extends Error {}
@@ -16,8 +17,11 @@ async function systemctl(exec, action, unit) {
@@ -16,8 +17,11 @@ async function systemctl(exec, action, unit) {
if (unit && unit.endsWith('.socket') && !SOCKET_SAFE_ACTIONS.has(action)) {
if (unit && unit.endsWith('.socket') && !SOCKET_SAFE_ACTIONS.has(action)) {
throw new SocketGuardError(`refusing "systemctl ${action} ${unit}": socket units are never stopped or restarted (they hold the listener that keeps HTTP up through a restart)`);
throw new SocketGuardError(`refusing "systemctl ${action} ${unit}": socket units are never stopped or restarted (they hold the listener that keeps HTTP up through a restart)`);
}
}
- const args = action === 'show' ? ['show', unit, `--property=${PROPS.join(',')}`] : unit ? [action, unit] : [action];
- const privileged = !['show', 'is-active', 'is-enabled', 'cat', 'status'].includes(action);
+ let args;
+ if (action === 'show') args = ['show', unit, `--property=${PROPS.join(',')}`];
+ else if (action === 'list-units') args = ['list-units', '--all', '--plain', '--no-legend', '--no-pager', unit];
+ else args = unit ? [action, unit] : [action];
+ const privileged = !READ_ONLY_ACTIONS.includes(action);
return exec.run('systemctl', args, { privileged });
return exec.run('systemctl', args, { privileged });
}
}
@@ -42,6 +46,22 @@ async function show(exec, unit) {
@@ -42,6 +46,22 @@ async function show(exec, unit) {
};
};
}
}
+/**
+ * Loaded instances of a worker unit (read-only): `[email protected]` lists every `name@*.service`, a
+ * plain unit lists itself. -> [{ unit, load, active, sub }]
+ */
+async function instances(exec, workerUnit) {
+ const pattern = workerUnit.endsWith('@.service') ? `${workerUnit.slice(0, -'.service'.length)}*.service` : workerUnit;
+ const r = await systemctl(exec, 'list-units', pattern);
+ if (r.code !== 0) throw new Error(`systemctl list-units ${pattern} failed: ${(r.stderr || '').trim()}`);
+ const out = [];
+ for (const line of String(r.stdout || '').split('\n')) {
+ const [unit, load, active, sub] = line.trim().split(/\s+/);
+ if (unit && unit.endsWith('.service')) out.push({ unit, load, active, sub });
+ }
+ return out;
+}
+
async function restart(exec, unit) {
async function restart(exec, unit) {
const r = await systemctl(exec, 'restart', unit);
const r = await systemctl(exec, 'restart', unit);
if (r.code !== 0) throw new Error(`systemctl restart ${unit} failed: ${r.stderr.trim()}`);
if (r.code !== 0) throw new Error(`systemctl restart ${unit} failed: ${r.stderr.trim()}`);
@@ -66,4 +86,4 @@ async function reloadNginx(exec) {
@@ -66,4 +86,4 @@ async function reloadNginx(exec) {
if (r.code !== 0) throw new Error(`systemctl reload nginx failed: ${r.stderr.trim()}`);
if (r.code !== 0) throw new Error(`systemctl reload nginx failed: ${r.stderr.trim()}`);
}
}
-module.exports = { systemctl, show, restart, ensureSocketActive, daemonReload, reloadNginx, SocketGuardError };
+module.exports = { systemctl, show, instances, restart, ensureSocketActive, daemonReload, reloadNginx, SocketGuardError };
@@ -188,6 +188,13 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
@@ -188,6 +188,13 @@ function createFakeHost({ root = true, user = 'root', hostname = 'fake-host', st
case 'stop': if (u) { u.active = 'inactive'; u.sub = 'dead'; } return { code: 0, stdout: '', stderr: '' };
case 'stop': if (u) { u.active = 'inactive'; u.sub = 'dead'; } return { code: 0, stdout: '', stderr: '' };
case 'reload': case 'daemon-reload': return { code: 0, stdout: '', stderr: '' };
case 'reload': case 'daemon-reload': return { code: 0, stdout: '', stderr: '' };
case 'is-active': return { code: u && u.active === 'active' ? 0 : 3, stdout: '', stderr: '' };
case 'is-active': return { code: u && u.active === 'active' ? 0 : 3, stdout: '', stderr: '' };
+ case 'list-units': {
+ // systemctl list-units --all --plain --no-legend --no-pager <glob>
+ const glob = args[args.length - 1];
+ const re = new RegExp(`^${glob.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*')}$`);
+ const rows = [...units.entries()].filter(([name]) => re.test(name)).map(([name, x]) => `${name} ${x.load} ${x.active} ${x.sub} ${name}`);
+ return { code: 0, stdout: rows.length ? `${rows.join('\n')}\n` : '', stderr: '' };
+ }
default: return { code: 1, stdout: '', stderr: `fake systemctl: ${action}` };
default: return { code: 1, stdout: '', stderr: `fake systemctl: ${action}` };
}
}
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,196 @@
@@ -0,0 +1,196 @@
+'use strict';
+/**
+ * OpenRe on ovhost: `workerUnits` (the transport worker templates) are listed by status and
+ * validate and never started, stopped or restarted; the inventory refuses a worker unit in
+ * `units`; a deploy of a git-layout service with worker units restarts only its units; and the
+ * example inventory's openre drill starts openre-api alone with OPENRE_DRILL=1 and the side-effect
+ * switches, leaving the API, the coordinator and every worker untouched.
+ */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const { scenario, push, test, runTests, SECRET } = require('./helpers');
+const { normalise } = require('../lib/inventory');
+
+const EXAMPLE = JSON.parse(fs.readFileSync(path.join(__dirname, '..', 'host.example.json'), 'utf8'));
+const RELEASE = '655b98a10aaa';
+const OLD = '0123456789ab';
+const PROD_DB = '/var/lib/openre/openre.db';
+const WORKERS = [`openre-rtmp-ingest@${RELEASE}.service`, `openre-restream-worker@${RELEASE}.service`];
+
+/** The standard fake host plus OpenRe as on openvibe-ovh: release layout, two units, worker instances. */
+async function openreHost({ workersRunning = true } = {}) {
+ const host = scenario();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.openre = JSON.parse(JSON.stringify(EXAMPLE.services.openre));
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+
+ host.put(`/opt/openre.stream/releases/${RELEASE}/package.json`, '{"name":"openre-stream"}', { owner: 'ubuntu' });
+ host.put(`/opt/openre.stream/releases/${RELEASE}/.env.example`, 'OV_OAUTH_CLIENT_ID=openre\nOV_OAUTH_CLIENT_SECRET=\nOPENRE_SECRETS_KEY=\n#OPENRE_DRILL=\n', { owner: 'ubuntu' });
+ host.files.get(`/opt/openre.stream/releases/${RELEASE}`).owner = 'ubuntu';
+ await host.exec.symlink(`/opt/openre.stream/releases/${RELEASE}`, '/opt/openre.stream/current');
+ host.put('/etc/openvibe/openre.env', `OV_OAUTH_CLIENT_ID=openre\nOV_OAUTH_CLIENT_SECRET=${SECRET}\nOPENRE_SECRETS_KEY=${SECRET}-k\nOPENRE_RTMP_BIND=0.0.0.0\n`, { mode: 0o600 });
+ host.put('/etc/systemd/system/openre-api.service', [
+ '[Service]',
+ 'User=ubuntu',
+ 'WorkingDirectory=/opt/openre.stream/current',
+ 'EnvironmentFile=/etc/openvibe/openre.env',
+ 'Environment=NODE_ENV=production',
+ 'Environment=PORT=4500',
+ `Environment=OPENRE_DB_PATH=${PROD_DB}`,
+ 'Environment=PATH=/usr/local/bin:/usr/bin:/bin',
+ 'ExecStart=/usr/bin/env node server/index.js',
+ '',
+ ].join('\n'));
+ host.addUnit('openre-api.service', { mainPid: 3100 });
+ host.addUnit('openre-session-coordinator.service', { mainPid: 3101 });
+ host.alivePids.add(3100);
+ host.alivePids.add(3101);
+ host.listeners.set(4500, [{ pid: 3100, process: 'node' }]);
+ host.addUnit(`openre-rtmp-ingest@${RELEASE}.service`, { mainPid: 3102, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
+ host.addUnit(`openre-restream-worker@${RELEASE}.service`, { mainPid: 3103, ...(workersRunning ? {} : { active: 'inactive', sub: 'dead' }) });
+ host.addUnit(`openre-rtmp-ingest@${OLD}.service`, { active: 'inactive', sub: 'dead', mainPid: 0 });
+ host.put(PROD_DB, 'sqlite-production', { owner: 'ubuntu', mode: 0o640 });
+
+ const health = '{"status":"ok","service":"openre-api","version":"0.1.0","release":"dev"}';
+ const robots = 'User-agent: *\nAllow: /$\n';
+ host.http.set('http://127.0.0.1:4500/api/ready', () => ({ status: 200, body: { status: 'ready' } }));
+ host.http.set('http://127.0.0.1:4500/api/health', () => ({ status: 200, body: health }));
+ host.http.set('http://127.0.0.1:4500/robots.txt', () => ({ status: 200, body: robots }));
+ const counts = { stream_definitions: 1, ingest_keys: 1, destinations: 0, migration_map: 0 };
+ host.sqliteHandler = (db, sql) => {
+ if (/integrity_check/.test(sql)) return [{ integrity_check: 'ok' }];
+ const m = /FROM "([a-z_]+)"/.exec(sql);
+ return [{ n: m ? counts[m[1]] ?? 0 : 0 }];
+ };
+ host.onSystemdRun = (spec) => {
+ host.listeners.set(14500, [{ pid: spec.pid, process: 'node' }]);
+ host.drillEnv = host.read(spec.envFiles[spec.envFiles.length - 1]);
+ const alive = (fn) => () => (host.alivePids.has(spec.pid) ? fn() : { status: 0, error: 'ECONNREFUSED' });
+ host.http.set('http://127.0.0.1:14500/api/ready', alive(() => ({ status: 200, body: { status: 'ready', mode: 'drill' } })));
+ host.http.set('http://127.0.0.1:14500/api/health', alive(() => ({ status: 200, body: health })));
+ host.http.set('http://127.0.0.1:14500/robots.txt', alive(() => ({ status: 200, body: robots })));
+ return undefined;
+ };
+ return host;
+}
+
+/** systemctl calls that could change a unit's state. */
+const stateChanges = (host) => host.calls.filter((c) => c.cmd === 'systemctl' && !['show', 'is-active', 'is-enabled', 'cat', 'status', 'list-units'].includes(c.args[0]));
+
+runTests([
+ test('the example inventory declares the OpenRe worker templates and a supported drill', () => {
+ const inv = normalise(EXAMPLE);
+ const o = inv.services.openre;
+ assert.deepStrictEqual(o.units, ['openre-api.service', 'openre-session-coordinator.service']);
+ assert.deepStrictEqual(o.workerUnits, ['[email protected]', '[email protected]']);
+ assert.strictEqual(o.layout, 'release');
+ assert.strictEqual(o.managed, false);
+ assert.strictEqual(o.drill.supported, true);
+ assert.strictEqual(o.drill.port, 14500);
+ assert.deepStrictEqual(o.drill.command, ['/usr/bin/node', 'server/index.js']);
+ assert.strictEqual(o.drill.env.OPENRE_DRILL, '1');
+ for (const k of ['EVENTS_URL', 'OV_OAUTH_CLIENT_SECRET', 'MEDIA_API_KEY']) assert.strictEqual(o.drill.env[k], '', k);
+ assert.strictEqual(o.drill.env.MEDIA_URL, 'http://127.0.0.1:9');
+ assert.deepStrictEqual(o.drill.databases, { openre: 'OPENRE_DB_PATH' });
+ // every other service keeps an empty list
+ assert.deepStrictEqual(inv.services.live.workerUnits, []);
+ }),
+
+ test('a worker unit, or an instance of a worker template, is refused in units', () => {
+ const base = { owner: 'ubuntu', repo: '/opt/x' };
+ assert.throws(() => normalise({ services: { x: { ...base, units: ['x-api.service', '[email protected]'], workerUnits: ['[email protected]'] } } }), /"x-worker@abc\.service" is a worker unit \(x-worker@\.service\)/);
+ assert.throws(() => normalise({ services: { x: { ...base, units: ['x-w.service'], workerUnits: ['x-w.service'] } } }), /is a worker unit/);
+ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: ['x.socket'] } } }), /not a \.service unit or template/);
+ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: ['../x.service'] } } }), /not a \.service unit or template/);
+ assert.throws(() => normalise({ services: { x: { ...base, workerUnits: '[email protected]' } } }), /must be an array/);
+ const ok = normalise({ services: { x: { ...base, units: ['x-worker-api.service'], workerUnits: ['[email protected]'] } } });
+ assert.deepStrictEqual(ok.services.x.workerUnits, ['[email protected]'], 'a unit that only shares a prefix without the @ is fine');
+ }),
+
+ test('status lists worker instances read-only and never changes a unit', async () => {
+ const host = await openreHost();
+ const r = await host.cli('status', 'openre');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.match(r.out, new RegExp(`openre\\s+${RELEASE}\\s+ready\\s+openre-api=active openre-session-coordinator=active workers\\[openre-rtmp-ingest@${RELEASE}=active openre-rtmp-ingest@${OLD}=inactive openre-restream-worker@${RELEASE}=active\\] \\[unmanaged\\]`));
+ const json = JSON.parse((await host.cli('status', 'openre', '--json')).out);
+ assert.deepStrictEqual(json[0].workers.map((w) => [w.unit, w.active]), [[`openre-rtmp-ingest@${RELEASE}.service`, 'active'], [`openre-rtmp-ingest@${OLD}.service`, 'inactive'], [`openre-restream-worker@${RELEASE}.service`, 'active']]);
+ const lists = host.calls.filter((c) => c.cmd === 'systemctl' && c.args[0] === 'list-units');
+ assert.deepStrictEqual(lists[0].args, ['list-units', '--all', '--plain', '--no-legend', '--no-pager', 'openre-rtmp-ingest@*.service']);
+ assert.ok(lists.every((c) => !c.privileged), 'listing needs no privileges');
+ assert.deepStrictEqual(stateChanges(host), []);
+ }),
+
+ test('validate lists the workers as info, warns when none runs, and changes nothing', async () => {
+ let host = await openreHost();
+ let res = JSON.parse((await host.cli('validate', 'openre', '--json')).out);
+ const workers = res.findings.filter((f) => f.area === 'units' && /^worker /.test(f.message));
+ assert.deepStrictEqual(workers.map((f) => f.level), ['info', 'info', 'info']);
+ assert.match(workers[0].message, /never starts, stops or restarts it/);
+ assert.ok(!res.findings.some((f) => /no running instance/.test(f.message)));
+ assert.deepStrictEqual(stateChanges(host), []);
+ host = await openreHost({ workersRunning: false });
+ res = JSON.parse((await host.cli('validate', 'openre', '--json')).out);
+ assert.deepStrictEqual(res.findings.filter((f) => /no running instance/.test(f.message)).map((f) => [f.level, f.message]), [
+ ['warn', 'no running instance of worker unit [email protected]'],
+ ['warn', 'no running instance of worker unit [email protected]'],
+ ]);
+ assert.deepStrictEqual(stateChanges(host), []);
+ assert.ok(!JSON.stringify(res).includes(SECRET));
+ }),
+
+ test('a deploy of a git-layout service with worker units restarts its units only', async () => {
+ const host = scenario();
+ const doc = JSON.parse(host.read('/etc/openvibe/host.json'));
+ doc.services.events.workerUnits = ['[email protected]'];
+ host.put('/etc/openvibe/host.json', JSON.stringify(doc, null, 2), { mode: 0o640, owner: 'root' });
+ host.addUnit('[email protected]', { mainPid: 5001 });
+ push(host, 'events', { 'server/index.js': 'events(2);' });
+ const r = await host.cli('deploy', 'events');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.deepStrictEqual(host.restarts(), ['openvibe-events.service']);
+ assert.ok(!host.calls.some((c) => c.cmd === 'systemctl' && c.args.includes('[email protected]')), 'the worker instance is never named in a systemctl call');
+ assert.strictEqual(host.units.get('[email protected]').restarts, 0);
+ }),
+
+ test('ovhost drill openre: openre-api alone, OPENRE_DRILL=1, side effects off, production untouched', async () => {
+ const host = await openreHost();
+ const b = await host.cli('backup', 'openre', '--json');
+ assert.strictEqual(b.code, 0, b.out);
+ host.advance(60 * 1000);
+ const callsBefore = host.calls.length;
+ const r = await host.cli('drill', 'openre');
+ assert.strictEqual(r.code, 0, r.out);
+ const rec = JSON.parse(host.read('/var/lib/openvibe-host/drills/openre.jsonl').trim().split('\n').pop());
+ assert.strictEqual(rec.result, 'passed', JSON.stringify(rec.failure));
+ assert.deepStrictEqual(rec.compare.map((c) => [c.path, c.match]), [['/api/health', true], ['/robots.txt', true]]);
+ assert.deepStrictEqual(rec.counts.map((c) => [c.table, c.match]), [['stream_definitions', true], ['ingest_keys', true], ['destinations', true], ['migration_map', true]]);
+
+ assert.strictEqual(host.systemdRuns.length, 1, 'one process');
+ const run = host.systemdRuns[0];
+ assert.deepStrictEqual(run.argv, ['/usr/bin/node', 'server/index.js'], 'the API, never a worker or the coordinator');
+ assert.strictEqual(run.cwd, '/opt/openre.stream/current');
+ assert.strictEqual(run.uid, 'ubuntu');
+ assert.deepStrictEqual(run.envFiles, ['/etc/openvibe/openre.env', `${rec.dir}/drill.env`]);
+ for (const p of ['SocketBindAllow=tcp:14500', 'SocketBindDeny=any', 'IPAddressDeny=any', 'IPAddressAllow=localhost', 'Environment=NODE_ENV=production']) assert.ok(run.props.includes(p), p);
+ assert.ok(run.props.includes('Environment=PATH=/usr/local/bin:/usr/bin:/bin'), 'other unit Environment= is passed on');
+ assert.ok(!run.props.some((p) => /OPENRE_DB_PATH|Environment=PORT=/.test(p)), 'the production database path and port from the unit never reach the drill instance');
+ const env = Object.fromEntries(host.drillEnv.trim().split('\n').filter((l) => !l.startsWith('#')).map((l) => [l.slice(0, l.indexOf('=')), l.slice(l.indexOf('=') + 1)]));
+ assert.deepStrictEqual(env, {
+ PORT: '14500', HOST: '127.0.0.1', OPENRE_DRILL: '1', EVENTS_URL: '', OV_OAUTH_CLIENT_SECRET: '', MEDIA_URL: 'http://127.0.0.1:9', MEDIA_API_KEY: '', OPENRE_RECORDING: 'off',
+ OPENRE_DB_PATH: `${rec.dir}/db/openre.db`,
+ });
+
+ const calls = host.calls.slice(callsBefore);
+ assert.deepStrictEqual(calls.filter((c) => c.cmd === 'kill').map((c) => c.args), [['SIGTERM', run.pid]], 'only the drill instance is signalled');
+ assert.deepStrictEqual(stateChanges(host), [], 'no unit was started, stopped or restarted');
+ for (const u of ['openre-api.service', 'openre-session-coordinator.service', ...WORKERS]) {
+ assert.strictEqual(host.units.get(u).active, 'active', u);
+ assert.strictEqual(host.units.get(u).restarts, 0, u);
+ assert.ok(!calls.some((c) => c.args && c.args.includes(u) && c.args[0] !== 'show'), `${u} is only ever read`);
+ }
+ assert.ok(host.alivePids.has(3100) && host.alivePids.has(3101));
+ assert.strictEqual(host.read(PROD_DB), 'sqlite-production');
+ assert.ok(!(r.out + JSON.stringify(host.systemdRuns) + host.read('/var/lib/openvibe-host/drills/openre.jsonl')).includes(SECRET));
+ }),
+]);