openvibers/openvibe.host Commits

Media probe counts chunked recordings (#286)

OP OpenVibers committed 8 days ago · b857cc07a7 · 7 files, +68 −12

Media's /api/ready reports ffmpeg recordings and open browser chunk sessions in two fields. The probe adds an addFields entry when it is present and falls back to recordings_in_progress when an older Media omits it.

Changes

ddiff --git a/README.md b/README.md
iindex d40d5bf..5e28ef2 100644
---- a/README.md
++++ b/README.md
@@@ -78,7 +78,7 @@ Repository deploy scripts call ovhost. Stage A applies the deploy rules for ever
|---|---|---|
| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
| Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
--| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`postgresql-count` of Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only as the postgres OS user. |
++| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`http-json-count` of Media's `/api/ready`: `recordings_in_progress` plus `chunked_recordings_in_progress` for a browser MediaRecorder session that received a chunk in the last five minutes). An older Media without the second field is counted by the first. |
| Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
| Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
| Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
@@@ -279,7 +279,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb
### Inventory
--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings and chunk sessions, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
ddiff --git a/host.example.json b/host.example.json
iindex b23eda9..82f8443 100644
---- a/host.example.json
++++ b/host.example.json
@@@ -338,6 +338,9 @@
"kind": "http-json-count",
"url": "http://127.0.0.1:4100/api/ready",
"field": "recordings_in_progress",
++ "addFields": [
++ "chunked_recordings_in_progress"
++ ],
"label": "recordings in progress"
},
"drain": {
ddiff --git a/lib/inventory.js b/lib/inventory.js
iindex 8a8164c..c9dad99 100644
---- a/lib/inventory.js
++++ b/lib/inventory.js
@@@ -301,6 +301,10 @@ function normaliseService(id, raw, defaults) {
const kinds = nested ? PROBE_KINDS.filter((k) => k !== 'sum') : PROBE_KINDS;
if (!kinds.includes(p.kind)) throw new InventoryError(`${at}.kind must be one of ${kinds.join(', ')}`);
if (p.kind === 'http-json-count' && !/^http:\/\/(127\.0\.0\.1|localhost|\[::1\])[:/]/.test(p.url || '')) throw new InventoryError(`${at}.url must be a loopback http:// URL`);
++ // Extra numeric fields summed into the count (Media: an older /api/ready has no
++ // chunked_recordings_in_progress, which must read as zero, not as an unanswerable probe).
++ if (p.addFields != null && (p.kind !== 'http-json-count' || p.array)) throw new InventoryError(`${at}.addFields is only for an http-json-count with a numeric field`);
++ if (p.addFields != null && (!Array.isArray(p.addFields) || p.addFields.some((f) => typeof f !== 'string' || !f))) throw new InventoryError(`${at}.addFields must be a list of dotted field names`);
if (p.kind === 'postgresql-count') {
if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);
if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
ddiff --git a/lib/probes.js b/lib/probes.js
iindex 3482eae..904627b 100644
---- a/lib/probes.js
++++ b/lib/probes.js
@@@ -3,8 +3,11 @@
* Protected-session probes: how many sessions a restart of this service would cut.
*
* http-json-count GET a loopback URL and count: `array` (dotted path) filtered by `where`
-- * (key: value equality), or a numeric `field` (dotted path).
++ * (key: value equality), or a numeric `field` (dotted path) plus every numeric
++ * `addFields` entry that is present. An absent addField is zero (an older service
++ * that does not report it yet); a present one that is not a number is unknown.
* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
++ * Media: /api/ready recordings_in_progress + chunked_recordings_in_progress.
* postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS
* user over peer auth (no secret is read, and a PostgreSQL service's database has
* no file to open). OpenRestream: ingest sessions in flight.
@@@ -67,7 +70,16 @@ async function countOne(exec, svc, p, label) {
}
const n = Number(at(body, p.field));
if (!Number.isFinite(n)) return { count: null, label, unknown: `no number at "${p.field}"` };
-- return { count: n, label };
++ let total = n;
++ for (const field of p.addFields || []) {
++ const raw = at(body, field);
++ // An older service omits the field: count what it does report, never fail the probe.
++ if (raw === undefined || raw === null) continue;
++ const extra = Number(raw);
++ if (!Number.isFinite(extra)) return { count: null, label, unknown: `no number at "${field}"` };
++ total += extra;
++ }
++ return { count: total, label };
}
if (p.kind === 'postgresql-count') {
// As the postgres OS user over peer auth: the service's own role needs a password, which
ddiff --git a/test/cli-inventory.test.js b/test/cli-inventory.test.js
iindex d26d02a..826c443 100644
---- a/test/cli-inventory.test.js
++++ b/test/cli-inventory.test.js
@@@ -25,8 +25,8 @@ runTests([
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
-- // Media on PostgreSQL (ADR-035): recordings in progress come from its /api/ready, through PostgreSQL.
-- assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field], ['http-json-count', 'recordings_in_progress']);
++ // Media: its /api/ready reports ffmpeg and chunked browser recordings in two fields; the probe sums them.
++ assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field, inv.services.media.protected.addFields], ['http-json-count', 'recordings_in_progress', ['chunked_recordings_in_progress']]);
// Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.
assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');
assert.deepStrictEqual(Object.fromEntries(['live', 'tools', 'games', 'openre', 'network'].map((id) => [id, inv.services[id].strategy])), { live: 'release-layout', tools: 'multi-app', games: 'pnpm-build', openre: 'release-layout', network: 'git-checkout' });
@@@ -74,6 +74,8 @@ runTests([
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { skipVhosts: ['../../etc/passwd'] } } } }), /vhost file names/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', ready: { url: 'http://evil.example/ready' } } } }), /loopback/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'DELETE FROM vods' } } } }), /SELECT/);
++ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'SELECT 1 AS n', addFields: ['n'] } } } }), /addFields/);
++ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'n', addFields: 'n' } } } }), /addFields/);
assert.throws(() => normalise({ services: { 'X Y': { ...base, repo: '/opt/x' } } }), /service id/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', unitSources: { 'a.service': '../../etc/shadow' } } } }), /inside the checkout/);
}),
ddiff --git a/test/deploy.test.js b/test/deploy.test.js
iindex acd6d95..8909e1f 100644
---- a/test/deploy.test.js
++++ b/test/deploy.test.js
@@@ -120,19 +120,45 @@ runTests([
assert.deepStrictEqual(host.restarts(), []);
}),
-- test('refuses to restart Media while a recording is in progress (read-only PostgreSQL query)', async () => {
++ test('refuses to restart Media while an ffmpeg or chunked browser recording is in progress', async () => {
const host = scenario();
const from = host.repo('media').head;
push(host, 'media', { 'server/index.js': 'media2();' });
host.recording = 1;
++ host.chunkedRecording = 2;
++ const r = await host.cli('deploy', 'media');
++ assert.strictEqual(r.code, 5, r.out);
++ assert.match(r.out, /3 recordings in progress — refusing/);
++ assert.strictEqual(host.repo('media').head, from);
++ assert.deepStrictEqual(host.restarts(), []);
++ assert.ok(host.calls.some((c) => c.cmd === 'curl' && c.args[0] === 'http://127.0.0.1:4100/api/ready'), 'the probe asks Media /api/ready, not PostgreSQL');
++ assert.deepStrictEqual(host.psqlCalls.filter((c) => c.database === 'ov_media'), [], 'nothing queries the media database');
++ }),
++
++ test('falls back to recordings_in_progress when Media does not report chunk sessions yet', async () => {
++ const host = scenario();
++ const from = host.repo('media').head;
++ push(host, 'media', { 'server/index.js': 'media2();' });
++ host.recording = 1;
++ host.chunkedRecording = 2;
++ host.mediaReportChunked = false; // an older Media: the additive field is absent
const r = await host.cli('deploy', 'media');
assert.strictEqual(r.code, 5, r.out);
assert.match(r.out, /1 recordings in progress — refusing/);
assert.strictEqual(host.repo('media').head, from);
assert.deepStrictEqual(host.restarts(), []);
-- const q = host.psqlCalls.find((c) => c.database === 'ov_media');
-- assert.strictEqual(q.as, 'postgres');
-- assert.match(q.sql, /^SELECT count\(\*\) AS n FROM vods WHERE is_recording = 1$/);
++ }),
++
++ test('a malformed chunked count is unknown (treated as active), never zero', async () => {
++ const host = scenario();
++ const from = host.repo('media').head;
++ push(host, 'media', { 'server/index.js': 'media2();' });
++ host.chunkedRecording = 'many';
++ const r = await host.cli('deploy', 'media');
++ assert.strictEqual(r.code, 5, r.out);
++ assert.match(r.out, /recordings in progress: unknown \(no number at "chunked_recordings_in_progress"\) — treated as active/);
++ assert.strictEqual(host.repo('media').head, from);
++ assert.deepStrictEqual(host.restarts(), []);
}),
test('Media deploys once no recording is running', async () => {
ddiff --git a/test/helpers.js b/test/helpers.js
iindex 022e127..51852d4 100644
---- a/test/helpers.js
++++ b/test/helpers.js
@@@ -1,7 +1,7 @@
'use strict';
/**
* A standard fake host with the shapes of the real estate: Live (socket-activated, /api/streams),
-- * Media (recordings counted in PostgreSQL), Tools (several packages and units), Sites (static build +
++ * Media (recordings counted by its /api/ready, ffmpeg and browser chunk sessions), Tools (several packages and units), Sites (static build +
* nginx vhosts, no units) and Events (SSE, drain policy "report").
*/
const path = require('path');
@@@ -60,7 +60,7 @@ function inventoryDoc() {
env: { required: 'from-example' },
port: 4100,
ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },
-- protected: { kind: 'postgresql-count', database: 'ov_media', sql: 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', label: 'recordings in progress' },
++ protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'recordings_in_progress', addFields: ['chunked_recordings_in_progress'], label: 'recordings in progress' },
databases: [{ name: 'media', engine: 'postgresql', database: 'ov_media' }],
lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),
},
@@@ -103,6 +103,8 @@ function scenario() {
host.badShas = new Set();
host.liveStreams = [];
host.recording = 0;
++ host.chunkedRecording = 0;
++ host.mediaReportChunked = true;
host.sseConnections = 0;
host.unitRepo = {};
@@@ -181,6 +183,13 @@ function scenario() {
host.http.set('http://127.0.0.1:3000/api/ready', readyFor('openvibe-live.service'));
host.http.set('http://127.0.0.1:3000/api/streams', () => ({ status: 200, body: { streams: host.liveStreams } }));
host.http.set('http://127.0.0.1:4100/healthz', readyFor('openvibe-media.service'));
++ host.http.set('http://127.0.0.1:4100/api/ready', () => {
++ const u = host.units.get('openvibe-media.service');
++ if (!u || u.active !== 'active') return { status: 502 };
++ const body = { status: 'ready', recordings_in_progress: host.recording };
++ if (host.mediaReportChunked) body.chunked_recordings_in_progress = host.chunkedRecording;
++ return { status: 200, body };
++ });
host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));
host.http.set('http://127.0.0.1:4300/api/ready', readyFor('openvibe-events.service'));
host.psqlHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : undefined);