openvibers/openvibe.host Commits
Media probe counts chunked recordings (#286)
Changes
@@@ -78,7 +78,7 @@ Repository deploy scripts call ovhost. Stage A applies the deploy rules for ever|---|---|---|| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. || Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |--| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`postgresql-count` of Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only as the postgres OS user. |++| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`http-json-count` of Media's `/api/ready`: `recordings_in_progress` plus `chunked_recordings_in_progress` for a browser MediaRecorder session that received a chunk in the last five minutes). An older Media without the second field is counted by the first. || Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. || Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. || Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |@@@ -279,7 +279,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb### Inventory--`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.++`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings and chunk sessions, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.@@@ -338,6 +338,9 @@"kind": "http-json-count","url": "http://127.0.0.1:4100/api/ready","field": "recordings_in_progress",++ "addFields": [++ "chunked_recordings_in_progress"++ ],"label": "recordings in progress"},"drain": {@@@ -301,6 +301,10 @@ function normaliseService(id, raw, defaults) {const kinds = nested ? PROBE_KINDS.filter((k) => k !== 'sum') : PROBE_KINDS;if (!kinds.includes(p.kind)) throw new InventoryError(`${at}.kind must be one of ${kinds.join(', ')}`);if (p.kind === 'http-json-count' && !/^http:\/\/(127\.0\.0\.1|localhost|\[::1\])[:/]/.test(p.url || '')) throw new InventoryError(`${at}.url must be a loopback http:// URL`);++ // Extra numeric fields summed into the count (Media: an older /api/ready has no++ // chunked_recordings_in_progress, which must read as zero, not as an unanswerable probe).++ if (p.addFields != null && (p.kind !== 'http-json-count' || p.array)) throw new InventoryError(`${at}.addFields is only for an http-json-count with a numeric field`);++ if (p.addFields != null && (!Array.isArray(p.addFields) || p.addFields.some((f) => typeof f !== 'string' || !f))) throw new InventoryError(`${at}.addFields must be a list of dotted field names`);if (p.kind === 'postgresql-count') {if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);@@@ -3,8 +3,11 @@* Protected-session probes: how many sessions a restart of this service would cut.** http-json-count GET a loopback URL and count: `array` (dotted path) filtered by `where`-- * (key: value equality), or a numeric `field` (dotted path).++ * (key: value equality), or a numeric `field` (dotted path) plus every numeric++ * `addFields` entry that is present. An absent addField is zero (an older service++ * that does not report it yet); a present one that is not a number is unknown.* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.++ * Media: /api/ready recordings_in_progress + chunked_recordings_in_progress.* postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS* user over peer auth (no secret is read, and a PostgreSQL service's database has* no file to open). OpenRestream: ingest sessions in flight.@@@ -67,7 +70,16 @@ async function countOne(exec, svc, p, label) {}const n = Number(at(body, p.field));if (!Number.isFinite(n)) return { count: null, label, unknown: `no number at "${p.field}"` };-- return { count: n, label };++ let total = n;++ for (const field of p.addFields || []) {++ const raw = at(body, field);++ // An older service omits the field: count what it does report, never fail the probe.++ if (raw === undefined || raw === null) continue;++ const extra = Number(raw);++ if (!Number.isFinite(extra)) return { count: null, label, unknown: `no number at "${field}"` };++ total += extra;++ }++ return { count: total, label };}if (p.kind === 'postgresql-count') {// As the postgres OS user over peer auth: the service's own role needs a password, which@@@ -25,8 +25,8 @@ runTests([assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');-- // Media on PostgreSQL (ADR-035): recordings in progress come from its /api/ready, through PostgreSQL.-- assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field], ['http-json-count', 'recordings_in_progress']);++ // Media: its /api/ready reports ffmpeg and chunked browser recordings in two fields; the probe sums them.++ assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field, inv.services.media.protected.addFields], ['http-json-count', 'recordings_in_progress', ['chunked_recordings_in_progress']]);// Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');assert.deepStrictEqual(Object.fromEntries(['live', 'tools', 'games', 'openre', 'network'].map((id) => [id, inv.services[id].strategy])), { live: 'release-layout', tools: 'multi-app', games: 'pnpm-build', openre: 'release-layout', network: 'git-checkout' });@@@ -74,6 +74,8 @@ runTests([assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { skipVhosts: ['../../etc/passwd'] } } } }), /vhost file names/);assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', ready: { url: 'http://evil.example/ready' } } } }), /loopback/);assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'DELETE FROM vods' } } } }), /SELECT/);++ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'SELECT 1 AS n', addFields: ['n'] } } } }), /addFields/);++ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'n', addFields: 'n' } } } }), /addFields/);assert.throws(() => normalise({ services: { 'X Y': { ...base, repo: '/opt/x' } } }), /service id/);assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', unitSources: { 'a.service': '../../etc/shadow' } } } }), /inside the checkout/);}),@@@ -120,19 +120,45 @@ runTests([assert.deepStrictEqual(host.restarts(), []);}),-- test('refuses to restart Media while a recording is in progress (read-only PostgreSQL query)', async () => {++ test('refuses to restart Media while an ffmpeg or chunked browser recording is in progress', async () => {const host = scenario();const from = host.repo('media').head;push(host, 'media', { 'server/index.js': 'media2();' });host.recording = 1;++ host.chunkedRecording = 2;++ const r = await host.cli('deploy', 'media');++ assert.strictEqual(r.code, 5, r.out);++ assert.match(r.out, /3 recordings in progress — refusing/);++ assert.strictEqual(host.repo('media').head, from);++ assert.deepStrictEqual(host.restarts(), []);++ assert.ok(host.calls.some((c) => c.cmd === 'curl' && c.args[0] === 'http://127.0.0.1:4100/api/ready'), 'the probe asks Media /api/ready, not PostgreSQL');++ assert.deepStrictEqual(host.psqlCalls.filter((c) => c.database === 'ov_media'), [], 'nothing queries the media database');++ }),++++ test('falls back to recordings_in_progress when Media does not report chunk sessions yet', async () => {++ const host = scenario();++ const from = host.repo('media').head;++ push(host, 'media', { 'server/index.js': 'media2();' });++ host.recording = 1;++ host.chunkedRecording = 2;++ host.mediaReportChunked = false; // an older Media: the additive field is absentconst r = await host.cli('deploy', 'media');assert.strictEqual(r.code, 5, r.out);assert.match(r.out, /1 recordings in progress — refusing/);assert.strictEqual(host.repo('media').head, from);assert.deepStrictEqual(host.restarts(), []);-- const q = host.psqlCalls.find((c) => c.database === 'ov_media');-- assert.strictEqual(q.as, 'postgres');-- assert.match(q.sql, /^SELECT count\(\*\) AS n FROM vods WHERE is_recording = 1$/);++ }),++++ test('a malformed chunked count is unknown (treated as active), never zero', async () => {++ const host = scenario();++ const from = host.repo('media').head;++ push(host, 'media', { 'server/index.js': 'media2();' });++ host.chunkedRecording = 'many';++ const r = await host.cli('deploy', 'media');++ assert.strictEqual(r.code, 5, r.out);++ assert.match(r.out, /recordings in progress: unknown \(no number at "chunked_recordings_in_progress"\) — treated as active/);++ assert.strictEqual(host.repo('media').head, from);++ assert.deepStrictEqual(host.restarts(), []);}),test('Media deploys once no recording is running', async () => {@@@ -1,7 +1,7 @@'use strict';/*** A standard fake host with the shapes of the real estate: Live (socket-activated, /api/streams),-- * Media (recordings counted in PostgreSQL), Tools (several packages and units), Sites (static build +++ * Media (recordings counted by its /api/ready, ffmpeg and browser chunk sessions), Tools (several packages and units), Sites (static build +* nginx vhosts, no units) and Events (SSE, drain policy "report").*/const path = require('path');@@@ -60,7 +60,7 @@ function inventoryDoc() {env: { required: 'from-example' },port: 4100,ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },-- protected: { kind: 'postgresql-count', database: 'ov_media', sql: 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', label: 'recordings in progress' },++ protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'recordings_in_progress', addFields: ['chunked_recordings_in_progress'], label: 'recordings in progress' },databases: [{ name: 'media', engine: 'postgresql', database: 'ov_media' }],lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),},@@@ -103,6 +103,8 @@ function scenario() {host.badShas = new Set();host.liveStreams = [];host.recording = 0;++ host.chunkedRecording = 0;++ host.mediaReportChunked = true;host.sseConnections = 0;host.unitRepo = {};@@@ -181,6 +183,13 @@ function scenario() {host.http.set('http://127.0.0.1:3000/api/ready', readyFor('openvibe-live.service'));host.http.set('http://127.0.0.1:3000/api/streams', () => ({ status: 200, body: { streams: host.liveStreams } }));host.http.set('http://127.0.0.1:4100/healthz', readyFor('openvibe-media.service'));++ host.http.set('http://127.0.0.1:4100/api/ready', () => {++ const u = host.units.get('openvibe-media.service');++ if (!u || u.active !== 'active') return { status: 502 };++ const body = { status: 'ready', recordings_in_progress: host.recording };++ if (host.mediaReportChunked) body.chunked_recordings_in_progress = host.chunkedRecording;++ return { status: 200, body };++ });host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));host.http.set('http://127.0.0.1:4300/api/ready', readyFor('openvibe-events.service'));host.psqlHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : undefined);
@@ -78,7 +78,7 @@ Repository deploy scripts call ovhost. Stage A applies the deploy rules for ever
@@ -78,7 +78,7 @@ Repository deploy scripts call ovhost. Stage A applies the deploy rules for ever
|---|---|---|
|---|---|---|
| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
| Never stop `openvibe-live.socket`; restart only the service unit | Live (socket activation, `deploy/systemd`) | Every `systemctl` call goes through a guard that refuses `stop`/`restart`/`reload` on a `.socket` unit. A socket that is down gets `start`ed. The one exception is the rebind Live's script also made (`systemd.rebindSocket`, release-layout only): when the socket unit file changed or pid 1 does not hold the port, the service is stopped, the socket restarted and the service started on it. |
| Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
| Don't restart Live while anyone is live; `--wait-idle` holds the restart | Live `deploy.sh --wait-idle` (`/api/streams`) | Protected-session probe (`http-json-count`). Checked before the checkout moves and again right before the restart. |
-| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`postgresql-count` of Media's `SELECT count(*) FROM vods WHERE is_recording = 1`), run read-only as the postgres OS user. |
+| Don't restart Media while it is recording | Media hazard H3 (`vods-orphans/` in B2) | Protected-session probe (`http-json-count` of Media's `/api/ready`: `recordings_in_progress` plus `chunked_recordings_in_progress` for a browser MediaRecorder session that received a chunk in the last five minutes). An older Media without the second field is counted by the first. |
| Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
| Poll `/api/ready` after a restart; roll back if it doesn't come up | Live `deploy.sh` | Readiness poll with a timeout; on failure the checkout goes back to the previous sha, deps are reinstalled if they differed, the unit is restarted again. |
| Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
| Install only when the lockfile or dependencies changed; restore `package-lock.json` afterwards | Network `deploy.sh` and host practice | `npm install --omit=dev` as the checkout owner, then `git checkout -- package-lock.json` if npm rewrote it. |
| Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
| Every dependency must resolve before anything restarts | Tools `deploy.sh` (an emptied `file:` link crash-looped every unit) | `node_modules/<dep>/package.json` must exist, parse and name the package. A broken dependency is reinstalled once, and if it still fails the deploy aborts and the checkout is restored. |
@@ -279,7 +279,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb
@@ -279,7 +279,7 @@ The sandbox blocks writes outside the drill directory and addresses beyond loopb
### Inventory
### Inventory
-`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count`, `postgresql-count`, or `sum` of several: Live streams, Media recordings, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
+`host.example.json` describes every service on the host. It records the checkout, owner/`runAs`, units and socket unit, `unitSources` (repo unit files), env file, `env.required` (names, or `"from-example"`), port, loopback `ready` URL (plus headers, since Tools needs `Host: openvibe.tools`), `packages` (`"apps/*"` for Tools), install command, `build` hooks, `noRestartPaths`, the `protected` probe (`http-json-count` — a numeric `field`, plus any present `addFields` —, `postgresql-count`, or `sum` of several: Live streams, Media recordings and chunk sessions, Events SSE connections, Chat sockets, Games players online, OpenRestream ingest sessions, Tools running jobs), the `drain` policy, `databases` (each a PostgreSQL database `{ "name", "engine": "postgresql", "database": "ov_<name>" }`; [docs/db-inventory.md](docs/db-inventory.md)), `backupOnChange`, `nginx`, `layout` (`release` for a `releases/<id>` + `current` service using the `release-layout` strategy) and `workerUnits`. `workerUnits` are units, or templates such as OpenRestream's `[email protected]`, that belong to the service but that ovhost **never starts, stops or restarts**: they drain and exit on their own. `status` and `validate` list their instances read-only, `validate` warns when none runs, and the inventory refuses a worker unit, or an instance of one, in `units`. The real file is `/etc/openvibe/host.json` and is **never committed**. When `ovhost` runs as root it refuses an inventory that is not root-owned or that anyone else can write, and it rejects relative paths, non-loopback probe URLs, non-SELECT probe queries and vhost names that contain a path.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
**Lifecycle** (roadmap WS-P task 1, `lib/lifecycle.js`). Each service's lifecycle is the `lifecycle` block of its manifest in openvibe-contracts (≥ 0.55.0): liveness, shutdown (signal, `deadlineSeconds`, drains, workers), startupRecovery, rollback, contracts and leases. A service entry may carry its own `lifecycle` block, which replaces the manifest's on this host, and `validate --manifest <file>` reads another manifest file. `ovhost validate` fails when no block is found or a field is missing, naming the service and the field (`live: lifecycle.shutdown.deadlineSeconds is missing`). It also fails when `deadlineSeconds` exceeds a unit's stop timeout (systemd's effective `TimeoutStopUSec`, else `TimeoutStopSec` in the unit source) or the unit's `KillSignal` is another signal, and when the checkout's installed openvibe-contracts is outside `contracts.range`. Worker units that drain longer than their stop timeout are a warning, since ovhost never stops them. This repository still pins openvibe-contracts v0.49.0, which has no lifecycle blocks, so until that pin moves to v0.55.0 or later `validate` reports every service as undeclared.
@@ -338,6 +338,9 @@
@@ -338,6 +338,9 @@
"kind": "http-json-count",
"kind": "http-json-count",
"url": "http://127.0.0.1:4100/api/ready",
"url": "http://127.0.0.1:4100/api/ready",
"field": "recordings_in_progress",
"field": "recordings_in_progress",
+ "addFields": [
+ "chunked_recordings_in_progress"
+ ],
"label": "recordings in progress"
"label": "recordings in progress"
},
},
"drain": {
"drain": {
@@ -301,6 +301,10 @@ function normaliseService(id, raw, defaults) {
@@ -301,6 +301,10 @@ function normaliseService(id, raw, defaults) {
const kinds = nested ? PROBE_KINDS.filter((k) => k !== 'sum') : PROBE_KINDS;
const kinds = nested ? PROBE_KINDS.filter((k) => k !== 'sum') : PROBE_KINDS;
if (!kinds.includes(p.kind)) throw new InventoryError(`${at}.kind must be one of ${kinds.join(', ')}`);
if (!kinds.includes(p.kind)) throw new InventoryError(`${at}.kind must be one of ${kinds.join(', ')}`);
if (p.kind === 'http-json-count' && !/^http:\/\/(127\.0\.0\.1|localhost|\[::1\])[:/]/.test(p.url || '')) throw new InventoryError(`${at}.url must be a loopback http:// URL`);
if (p.kind === 'http-json-count' && !/^http:\/\/(127\.0\.0\.1|localhost|\[::1\])[:/]/.test(p.url || '')) throw new InventoryError(`${at}.url must be a loopback http:// URL`);
+ // Extra numeric fields summed into the count (Media: an older /api/ready has no
+ // chunked_recordings_in_progress, which must read as zero, not as an unanswerable probe).
+ if (p.addFields != null && (p.kind !== 'http-json-count' || p.array)) throw new InventoryError(`${at}.addFields is only for an http-json-count with a numeric field`);
+ if (p.addFields != null && (!Array.isArray(p.addFields) || p.addFields.some((f) => typeof f !== 'string' || !f))) throw new InventoryError(`${at}.addFields must be a list of dotted field names`);
if (p.kind === 'postgresql-count') {
if (p.kind === 'postgresql-count') {
if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);
if (!PG_DATABASE_RE.test(p.database || '')) throw new InventoryError(`${at}.database must match ${PG_DATABASE_RE} (e.g. ov_openre)`);
if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
if (!/^\s*select\b/i.test(p.sql || '')) throw new InventoryError(`${at}.sql must be a SELECT`);
@@ -3,8 +3,11 @@
@@ -3,8 +3,11 @@
* Protected-session probes: how many sessions a restart of this service would cut.
* Protected-session probes: how many sessions a restart of this service would cut.
*
*
* http-json-count GET a loopback URL and count: `array` (dotted path) filtered by `where`
* http-json-count GET a loopback URL and count: `array` (dotted path) filtered by `where`
- * (key: value equality), or a numeric `field` (dotted path).
+ * (key: value equality), or a numeric `field` (dotted path) plus every numeric
+ * `addFields` entry that is present. An absent addField is zero (an older service
+ * that does not report it yet); a present one that is not a number is unknown.
* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
* Live: /api/streams, streams where is_live. Events: /api/ready realtime_connections.
+ * Media: /api/ready recordings_in_progress + chunked_recordings_in_progress.
* postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS
* postgresql-count a read-only SELECT against a database in the cluster, run as the postgres OS
* user over peer auth (no secret is read, and a PostgreSQL service's database has
* user over peer auth (no secret is read, and a PostgreSQL service's database has
* no file to open). OpenRestream: ingest sessions in flight.
* no file to open). OpenRestream: ingest sessions in flight.
@@ -67,7 +70,16 @@ async function countOne(exec, svc, p, label) {
@@ -67,7 +70,16 @@ async function countOne(exec, svc, p, label) {
}
}
const n = Number(at(body, p.field));
const n = Number(at(body, p.field));
if (!Number.isFinite(n)) return { count: null, label, unknown: `no number at "${p.field}"` };
if (!Number.isFinite(n)) return { count: null, label, unknown: `no number at "${p.field}"` };
- return { count: n, label };
+ let total = n;
+ for (const field of p.addFields || []) {
+ const raw = at(body, field);
+ // An older service omits the field: count what it does report, never fail the probe.
+ if (raw === undefined || raw === null) continue;
+ const extra = Number(raw);
+ if (!Number.isFinite(extra)) return { count: null, label, unknown: `no number at "${field}"` };
+ total += extra;
+ }
+ return { count: total, label };
}
}
if (p.kind === 'postgresql-count') {
if (p.kind === 'postgresql-count') {
// As the postgres OS user over peer auth: the service's own role needs a password, which
// As the postgres OS user over peer auth: the service's own role needs a password, which
@@ -25,8 +25,8 @@ runTests([
@@ -25,8 +25,8 @@ runTests([
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.strictEqual(inv.services.live.socketUnit, 'openvibe-live.socket');
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.ok(!inv.services.live.units.includes('openvibe-live.socket'));
assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
assert.strictEqual(inv.services.live.protected.url, 'http://127.0.0.1:3000/api/streams');
- // Media on PostgreSQL (ADR-035): recordings in progress come from its /api/ready, through PostgreSQL.
- assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field], ['http-json-count', 'recordings_in_progress']);
+ // Media: its /api/ready reports ffmpeg and chunked browser recordings in two fields; the probe sums them.
+ assert.deepStrictEqual([inv.services.media.protected.kind, inv.services.media.protected.field, inv.services.media.protected.addFields], ['http-json-count', 'recordings_in_progress', ['chunked_recordings_in_progress']]);
// Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.
// Every service on the host is deployed by a strategy (WS-N task 11): Games by pnpm-build.
assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');
assert.ok(Object.values(inv.services).every((s) => s.managed), 'every service is managed');
assert.deepStrictEqual(Object.fromEntries(['live', 'tools', 'games', 'openre', 'network'].map((id) => [id, inv.services[id].strategy])), { live: 'release-layout', tools: 'multi-app', games: 'pnpm-build', openre: 'release-layout', network: 'git-checkout' });
assert.deepStrictEqual(Object.fromEntries(['live', 'tools', 'games', 'openre', 'network'].map((id) => [id, inv.services[id].strategy])), { live: 'release-layout', tools: 'multi-app', games: 'pnpm-build', openre: 'release-layout', network: 'git-checkout' });
@@ -74,6 +74,8 @@ runTests([
@@ -74,6 +74,8 @@ runTests([
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { skipVhosts: ['../../etc/passwd'] } } } }), /vhost file names/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', nginx: { skipVhosts: ['../../etc/passwd'] } } } }), /vhost file names/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', ready: { url: 'http://evil.example/ready' } } } }), /loopback/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', ready: { url: 'http://evil.example/ready' } } } }), /loopback/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'DELETE FROM vods' } } } }), /SELECT/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'DELETE FROM vods' } } } }), /SELECT/);
+ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'postgresql-count', database: 'ov_x', sql: 'SELECT 1 AS n', addFields: ['n'] } } } }), /addFields/);
+ assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'n', addFields: 'n' } } } }), /addFields/);
assert.throws(() => normalise({ services: { 'X Y': { ...base, repo: '/opt/x' } } }), /service id/);
assert.throws(() => normalise({ services: { 'X Y': { ...base, repo: '/opt/x' } } }), /service id/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', unitSources: { 'a.service': '../../etc/shadow' } } } }), /inside the checkout/);
assert.throws(() => normalise({ services: { x: { ...base, repo: '/opt/x', unitSources: { 'a.service': '../../etc/shadow' } } } }), /inside the checkout/);
}),
}),
@@ -120,19 +120,45 @@ runTests([
@@ -120,19 +120,45 @@ runTests([
assert.deepStrictEqual(host.restarts(), []);
assert.deepStrictEqual(host.restarts(), []);
}),
}),
- test('refuses to restart Media while a recording is in progress (read-only PostgreSQL query)', async () => {
+ test('refuses to restart Media while an ffmpeg or chunked browser recording is in progress', async () => {
const host = scenario();
const host = scenario();
const from = host.repo('media').head;
const from = host.repo('media').head;
push(host, 'media', { 'server/index.js': 'media2();' });
push(host, 'media', { 'server/index.js': 'media2();' });
host.recording = 1;
host.recording = 1;
+ host.chunkedRecording = 2;
+ const r = await host.cli('deploy', 'media');
+ assert.strictEqual(r.code, 5, r.out);
+ assert.match(r.out, /3 recordings in progress — refusing/);
+ assert.strictEqual(host.repo('media').head, from);
+ assert.deepStrictEqual(host.restarts(), []);
+ assert.ok(host.calls.some((c) => c.cmd === 'curl' && c.args[0] === 'http://127.0.0.1:4100/api/ready'), 'the probe asks Media /api/ready, not PostgreSQL');
+ assert.deepStrictEqual(host.psqlCalls.filter((c) => c.database === 'ov_media'), [], 'nothing queries the media database');
+ }),
+
+ test('falls back to recordings_in_progress when Media does not report chunk sessions yet', async () => {
+ const host = scenario();
+ const from = host.repo('media').head;
+ push(host, 'media', { 'server/index.js': 'media2();' });
+ host.recording = 1;
+ host.chunkedRecording = 2;
+ host.mediaReportChunked = false; // an older Media: the additive field is absent
const r = await host.cli('deploy', 'media');
const r = await host.cli('deploy', 'media');
assert.strictEqual(r.code, 5, r.out);
assert.strictEqual(r.code, 5, r.out);
assert.match(r.out, /1 recordings in progress — refusing/);
assert.match(r.out, /1 recordings in progress — refusing/);
assert.strictEqual(host.repo('media').head, from);
assert.strictEqual(host.repo('media').head, from);
assert.deepStrictEqual(host.restarts(), []);
assert.deepStrictEqual(host.restarts(), []);
- const q = host.psqlCalls.find((c) => c.database === 'ov_media');
- assert.strictEqual(q.as, 'postgres');
- assert.match(q.sql, /^SELECT count\(\*\) AS n FROM vods WHERE is_recording = 1$/);
+ }),
+
+ test('a malformed chunked count is unknown (treated as active), never zero', async () => {
+ const host = scenario();
+ const from = host.repo('media').head;
+ push(host, 'media', { 'server/index.js': 'media2();' });
+ host.chunkedRecording = 'many';
+ const r = await host.cli('deploy', 'media');
+ assert.strictEqual(r.code, 5, r.out);
+ assert.match(r.out, /recordings in progress: unknown \(no number at "chunked_recordings_in_progress"\) — treated as active/);
+ assert.strictEqual(host.repo('media').head, from);
+ assert.deepStrictEqual(host.restarts(), []);
}),
}),
test('Media deploys once no recording is running', async () => {
test('Media deploys once no recording is running', async () => {
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
'use strict';
'use strict';
/**
/**
* A standard fake host with the shapes of the real estate: Live (socket-activated, /api/streams),
* A standard fake host with the shapes of the real estate: Live (socket-activated, /api/streams),
- * Media (recordings counted in PostgreSQL), Tools (several packages and units), Sites (static build +
+ * Media (recordings counted by its /api/ready, ffmpeg and browser chunk sessions), Tools (several packages and units), Sites (static build +
* nginx vhosts, no units) and Events (SSE, drain policy "report").
* nginx vhosts, no units) and Events (SSE, drain policy "report").
*/
*/
const path = require('path');
const path = require('path');
@@ -60,7 +60,7 @@ function inventoryDoc() {
@@ -60,7 +60,7 @@ function inventoryDoc() {
env: { required: 'from-example' },
env: { required: 'from-example' },
port: 4100,
port: 4100,
ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },
ready: { url: 'http://127.0.0.1:4100/healthz', timeoutSeconds: 30 },
- protected: { kind: 'postgresql-count', database: 'ov_media', sql: 'SELECT count(*) AS n FROM vods WHERE is_recording = 1', label: 'recordings in progress' },
+ protected: { kind: 'http-json-count', url: 'http://127.0.0.1:4100/api/ready', field: 'recordings_in_progress', addFields: ['chunked_recordings_in_progress'], label: 'recordings in progress' },
databases: [{ name: 'media', engine: 'postgresql', database: 'ov_media' }],
databases: [{ name: 'media', engine: 'postgresql', database: 'ov_media' }],
lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),
lifecycle: lifecycleDoc({ liveness: { endpoint: '/healthz', means: 'the process answers HTTP and reads its database' }, shutdown: { signal: 'SIGTERM', deadlineSeconds: 70, drains: ['recordings stopped', 'HTTP server closed'] } }),
},
},
@@ -103,6 +103,8 @@ function scenario() {
@@ -103,6 +103,8 @@ function scenario() {
host.badShas = new Set();
host.badShas = new Set();
host.liveStreams = [];
host.liveStreams = [];
host.recording = 0;
host.recording = 0;
+ host.chunkedRecording = 0;
+ host.mediaReportChunked = true;
host.sseConnections = 0;
host.sseConnections = 0;
host.unitRepo = {};
host.unitRepo = {};
@@ -181,6 +183,13 @@ function scenario() {
@@ -181,6 +183,13 @@ function scenario() {
host.http.set('http://127.0.0.1:3000/api/ready', readyFor('openvibe-live.service'));
host.http.set('http://127.0.0.1:3000/api/ready', readyFor('openvibe-live.service'));
host.http.set('http://127.0.0.1:3000/api/streams', () => ({ status: 200, body: { streams: host.liveStreams } }));
host.http.set('http://127.0.0.1:3000/api/streams', () => ({ status: 200, body: { streams: host.liveStreams } }));
host.http.set('http://127.0.0.1:4100/healthz', readyFor('openvibe-media.service'));
host.http.set('http://127.0.0.1:4100/healthz', readyFor('openvibe-media.service'));
+ host.http.set('http://127.0.0.1:4100/api/ready', () => {
+ const u = host.units.get('openvibe-media.service');
+ if (!u || u.active !== 'active') return { status: 502 };
+ const body = { status: 'ready', recordings_in_progress: host.recording };
+ if (host.mediaReportChunked) body.chunked_recordings_in_progress = host.chunkedRecording;
+ return { status: 200, body };
+ });
host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));
host.http.set('http://127.0.0.1:4001/api/health', readyFor('openvibe-tools.service'));
host.http.set('http://127.0.0.1:4300/api/ready', readyFor('openvibe-events.service'));
host.http.set('http://127.0.0.1:4300/api/ready', readyFor('openvibe-events.service'));
host.psqlHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : undefined);
host.psqlHandler = (db, sql) => (/is_recording/.test(sql) ? [{ n: host.recording }] : undefined);