openvibers/openvibe.host Commits
ovhost nodes report: this inventory's machines, reported to Network's node registry (roadmap WS-X1 step 3, ADR-034 §12). host.json gains nodeSource and nodes (network.node@1 without health: id, name, roles, region-level location, provider, beacon); the inventory refuses an address or ip field, duplicate ids and non-https beacons, because the registry is public. Health is measured on every run: each beacon's 2xx within 2 s is up, a slower 2xx degraded, anything else down. The report is validated against network.node-report-request@1 before a token is asked for, and is sent with Host's principal (scope network.node.report); no secret or token is printed. openvibe-nodes.timer runs it every 5 minutes; openvibe_nodes_report.prom feeds the new OpenVibeNodeReportStale and OpenVibeNodeDown alerts. openvibe-contracts pinned at v0.76.0. docs/nodes.md; test/nodes.test.js.
Changes
@@@ -101,6 +101,19 @@ groups:labels: { severity: page }annotations:summary: "Operator alerts were not delivered for more than 15 minutes (ovhost alerts relay: Prometheus, the token or Network)"++ # The node registry (openvibe-nodes.timer; WS-X1): products find the nearest node from it.++ - alert: OpenVibeNodeReportStale++ expr: time() - openvibe_nodes_report_last_success_timestamp_seconds > 60 * 60++ for: 10m++ labels: { severity: ticket }++ annotations:++ summary: "The node registry has not heard from this host for more than an hour (ovhost nodes report)"++ - alert: OpenVibeNodeDown++ expr: openvibe_nodes{status="down"} > 0++ for: 15m++ labels: { severity: ticket }++ annotations:++ summary: "A node's beacon has not answered for 15 minutes (ovhost nodes report --dry-run names it)"# The data role (roles/data, ADR-035): PostgreSQL, PgBouncer, Valkey and pgBackRest.- name: openvibe-datarules:nnew file mode 100644@@@ -0,0 +1,19 @@++[Unit]++Description=OpenVibe node registry: this inventory's machines and their measured health, reported to OpenVibe.Network++Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/nodes.md++After=network-online.target++Wants=network-online.target++++[Service]++Type=oneshot++# Root: /etc/openvibe/host.env (Host's service principal) is root-only and read by ovhost itself, and the++# textfile collector's directory is root's.++User=root++ExecStart=/usr/local/bin/ovhost nodes report++Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin++Environment=OVHOST_INVENTORY=/etc/openvibe/host.json++UMask=0077++TimeoutStartSec=90s++PrivateTmp=true++NoNewPrivileges=true++ProtectSystem=fullnnew file mode 100644@@@ -0,0 +1,10 @@++[Unit]++Description=Report this inventory's machines to the OpenVibe node registry every five minutes (WS-X1)++++[Timer]++OnBootSec=3min++OnUnitActiveSec=5min++AccuracySec=30s++++[Install]++WantedBy=timers.targetnnew file mode 100644@@@ -0,0 +1,49 @@++# The node registry reporter++++`ovhost nodes report` tells OpenVibe.Network which machines this inventory runs, and how healthy each one is right now.++++- It is roadmap WS-X1 step 3 and ADR-034 section 12.++- `openvibe-nodes.timer` runs it every five minutes.++- Products read the result at `GET https://openvibe.network/api/v1/nodes`, which returns `network.node-list-result@1`. It can be filtered with `?role=` and `?region=`. `openvibe-sdk/geo` uses it to pick the nearest node for a role, for example a game-server list or a ping tool with several locations.++++## The inventory++++`/etc/openvibe/host.json` holds two keys for this:++++- **`nodeSource`** names this inventory. The default is `primary`.++- **`nodes`** lists each machine's public description, which is `network.node@1` without `health` and `updated_at`.++++```json++"nodeSource": "oregon",++"nodes": [++ { "id": "oregon-1", "name": "Oregon 1", "roles": ["web", "app", "data", "ingest"],++ "location": { "region": "us-west", "country": "US" }, "provider": "ovh",++ "beacon": "https://openvibe.network/api/v1/nodes/oregon-1/beacon" }++]++```++++The registry is public, so the inventory refuses any node that carries an `address` or `ip`. Location stays at region level.++++## Health++++Health is measured, never assumed. Each node's beacon is fetched on every run:++++| Beacon result | Health |++|---|---|++| a 2xx within 2 s | `up` |++| a slower 2xx | `degraded` |++| anything else | `down` |++++A node removed from the inventory is marked `down` by Network. It is not deleted.++++## Running it++++```++sudo ovhost nodes report --dry-run # measure and print; nothing is sent++sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-nodes.{service,timer} /etc/systemd/system/++sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-nodes.timer++```++++- **Credentials:** it uses Host's principal from `/etc/openvibe/host.env`, the same one `ovhost alerts relay` uses, with the scope `network.node.report`. Neither the secret nor the token is ever printed.++- **Metrics:** each run writes `openvibe_nodes_report.prom` to the textfile collector.++- **Alerts:** `OpenVibeNodeReportStale` fires when no report has reached Network for an hour. `OpenVibeNodeDown` fires when a beacon has failed for 15 minutes.@@@ -93,6 +93,9 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]alerts relay [--prometheus <url>] [--dry-run]the alerts firing in Prometheus, sent to Network, which pagesthe operator (openvibe-alerts.timer, every 2 minutes)++ nodes report [--dry-run] this inventory's machines (host.json "nodes"), each beacon++ measured, sent to Network's node registry++ (openvibe-nodes.timer, every 5 minutes)restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]download, verify and decrypt one service's databases into aNEW root-only directory; never touches a live database@@@ -552,6 +555,18 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => procelse out(`ALERTS delivered: ${r.firing} firing; opened ${r.result.opened}, reminded ${r.result.reminded}, resolved ${r.result.resolved}; ${r.result.notified} notification(s)`);return r.ok ? 0 : 2;}++ case 'nodes': {++ // WS-X1 step 3: this inventory's machines, with measured health, to Network's node registry.++ if (pos[1] !== 'report') throw new UsageError('usage: ovhost nodes report [--dry-run] [--events-env <file>]');++ const r = await lib.nodes.report(ctx, { envFile: opts['events-env'], env, dryRun: !!opts['dry-run'] });++ if (json) emit(r);++ else if (!r.ok) out(`NODES not reported (${r.stage}): ${r.error}`);++ else {++ out(r.dryRun ? `${r.probes.length} node(s) of ${r.source} (dry run, nothing sent)` : `NODES reported: ${r.probes.length} node(s) of ${r.source}; the registry lists ${r.registered}${r.down.length ? `, down: ${r.down.join(', ')}` : ''}`);++ for (const p of r.probes) out(` ${p.status.padEnd(9)} ${p.id} ${p.detail}`);++ }++ return r.ok ? 0 : 2;++ }case 'archive': {// WS-S task 6: dead and backup database files, encrypted off-host before the owner deletes them.const sub = pos[1];@@@ -36,6 +36,7 @@ module.exports = {offsite,archive: require('./archive'),alerts: require('./alerts'),++ nodes: require('./nodes'),browserWatch: require('./browser-watch'),incidents: require('./incidents'),dns: require('./dns'),@@@ -456,6 +456,23 @@ function normaliseDrill(raw, svc, where) {};}++/** The public description of each machine (network.node@1 without health; lib/nodes.js validates the rest). */++function normaliseNodes(raw) {++ if (raw == null) return [];++ if (!Array.isArray(raw)) throw new InventoryError('nodes must be an array');++ const seen = new Set();++ return raw.map((n, i) => {++ if (!n || typeof n !== 'object' || !ID_RE.test(n.id || '')) throw new InventoryError(`nodes[${i}].id must match ${ID_RE}`);++ if (seen.has(n.id)) throw new InventoryError(`nodes[${i}].id "${n.id}" is listed twice`);++ seen.add(n.id);++ if ('address' in n || 'ip' in n) throw new InventoryError(`nodes[${i}] must not carry an address: the registry is public`);++ let u;++ try { u = new URL(String(n.beacon)); } catch { u = null; }++ if (!u || u.protocol !== 'https:') throw new InventoryError(`nodes[${i}].beacon must be an https:// URL`);++ return { ...n };++ });++}++function normalise(raw) {if (!raw || typeof raw !== 'object' || !raw.services || typeof raw.services !== 'object') throw new InventoryError('inventory needs a "services" object');const defaults = raw.defaults || {};@@@ -476,6 +493,9 @@ function normalise(raw) {events: { envFile: raw.events && raw.events.envFile ? abs(raw.events.envFile, 'events.envFile') : null },// The DNS adapter (lib/dns.js): names never proxied by Cloudflare (non-HTTP traffic, the CNAME target).dns: { dnsOnly: Array.isArray(raw.dns && raw.dns.dnsOnly) ? raw.dns.dnsOnly.map(String) : require('./dns').DEFAULT_DNS_ONLY },++ // The node registry (lib/nodes.js, ADR-034 §12): this inventory's machines, reported to Network.++ nodeSource: raw.nodeSource || 'primary',++ nodes: normaliseNodes(raw.nodes),services: {},};for (const [id, svc] of Object.entries(raw.services)) {nnew file mode 100644@@@ -0,0 +1,97 @@++'use strict';++/**++ * `ovhost nodes report` (roadmap WS-X1 step 3, ADR-034 §12): the machines this inventory knows, with their health++ * now, sent to Network's node registry (POST /internal/nodes/report, network.node.report, Host's principal).++ * openvibe-nodes.timer runs it every five minutes; products read the result at GET /api/v1/nodes.++ *++ * The inventory's "nodes" array holds each machine's public description (network.node@1 without health and++ * updated_at: id, name, roles, region-level location, provider, beacon); "nodeSource" names this inventory (default++ * "primary"). Health is measured, never assumed: each node's beacon is fetched; a 2xx within 2 s is up, a slower 2xx++ * degraded, anything else down. A node removed from the inventory is marked down by Network, not deleted.++ * Every run leaves openvibe_nodes_report.prom for the textfile collector (OpenVibeNodeReportStale).++ */++const path = require('path');++const { loadCredentials } = require('./announce');++++const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';++const SOURCE_RE = /^[a-z][a-z0-9-]{1,39}$/;++const SLOW_MS = 2000;++++async function measure(exec, beacon) {++ const t0 = exec.now();++ const r = await exec.request(beacon, { method: 'GET', headers: { 'user-agent': 'openvibe-host-nodes' }, timeoutMs: 5000 });++ const ms = exec.now() - t0;++ if (!(r.status >= 200 && r.status < 300)) return { status: 'down', detail: r.status ? `beacon answered ${r.status}` : `beacon unreachable (${r.error || 'no answer'})` };++ return { status: ms > SLOW_MS ? 'degraded' : 'up', detail: `${ms} ms` };++}++++/** The report body (network.node-report-request@1) from the inventory, with measured health. */++async function build(exec, inv) {++ const source = inv.nodeSource || 'primary';++ if (!SOURCE_RE.test(source)) throw new Error(`nodeSource "${source}" must match ${SOURCE_RE}`);++ const at = new Date(exec.now()).toISOString();++ const nodes = [];++ const probes = [];++ for (const n of inv.nodes || []) {++ const m = await measure(exec, n.beacon);++ probes.push({ id: n.id, ...m });++ nodes.push({ ...n, health: { status: m.status, checked_at: at }, updated_at: at });++ }++ const body = { source, nodes };++ const v = require('openvibe-contracts').validate('network.node-report-request@1', body);++ if (!v.valid) throw new Error(`the inventory's nodes do not match network.node-report-request@1: ${(v.errors || []).slice(0, 3).map((e) => `${e.path} ${e.message}`).join('; ')}`);++ return { body, probes };++}++++async function writeMetrics(exec, dir, r) {++ const now = Math.floor(exec.now() / 1000);++ let lastOk = r.ok ? now : null;++ if (!r.ok) {++ const prev = (await exec.readFile(path.join(dir, 'openvibe_nodes_report.prom'))) || '';++ const m = prev.match(/^openvibe_nodes_report_last_success_timestamp_seconds (\d+)$/m);++ if (m) lastOk = Number(m[1]);++ }++ const count = (s) => (r.probes || []).filter((p) => p.status === s).length;++ const lines = [++ '# HELP openvibe_nodes_report_last_run_ok 1 when the last ovhost nodes report reached Network.',++ '# TYPE openvibe_nodes_report_last_run_ok gauge',++ `openvibe_nodes_report_last_run_ok ${r.ok ? 1 : 0}`,++ '# HELP openvibe_nodes Nodes of this inventory by the health the last run measured.',++ '# TYPE openvibe_nodes gauge',++ ...['up', 'degraded', 'down'].map((s) => `openvibe_nodes{status="${s}"} ${count(s)}`),++ ];++ if (lastOk) lines.push('# HELP openvibe_nodes_report_last_success_timestamp_seconds When a report last reached Network.', '# TYPE openvibe_nodes_report_last_success_timestamp_seconds gauge', `openvibe_nodes_report_last_success_timestamp_seconds ${lastOk}`);++ await exec.writeFile(path.join(dir, 'openvibe_nodes_report.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });++}++++/** One run → { ok, source, probes, stage?, error?, dryRun? }. opts: envFile, env, dryRun, metrics (default true). */++async function report(ctx, { envFile = null, env = {}, dryRun = false, metrics = true } = {}) {++ const { exec, inv } = ctx;++ const done = async (r) => {++ if (metrics && !dryRun) { try { await writeMetrics(exec, inv.textfileDir || TEXTFILE_DIR, r); } catch { /* metrics never fail a run */ } }++ return r;++ };++ let built;++ try { built = await build(exec, inv); } catch (err) { return done({ ok: false, stage: 'inventory', error: err.message }); }++ const { body, probes } = built;++ if (dryRun) return { ok: true, dryRun: true, source: body.source, probes, body };++ const creds = await loadCredentials(exec, inv, { file: envFile, env });++ if (creds.error) return done({ ok: false, stage: 'credentials', source: body.source, probes, error: creds.error });++ const form = new URLSearchParams({ grant_type: 'client_credentials', client_id: creds.clientId, client_secret: creds.secret, audience: 'openvibe.network', scope: 'network.node.report' });++ const t = await exec.request(`${creds.networkUrl}/oauth/token`, { method: 'POST', headers: { 'Content-Type': 'application/x-www-form-urlencoded', Accept: 'application/json' }, body: form.toString(), timeoutMs: 5000 });++ let token = null;++ try { token = t.status === 200 ? JSON.parse(t.body).access_token : null; } catch { token = null; }++ if (typeof token !== 'string' || !token) return done({ ok: false, stage: 'token', source: body.source, probes, error: t.status ? `the token endpoint answered ${t.status}` : `Network unreachable (${t.error || 'no answer'})` });++ const r = await exec.request(`${creds.networkUrl}/internal/nodes/report`, {++ method: 'POST', headers: { 'Content-Type': 'application/json', Accept: 'application/json', Authorization: `Bearer ${token}` }, body: JSON.stringify(body), timeoutMs: 10000,++ });++ let result = null;++ try { result = JSON.parse(r.body); } catch { result = null; }++ if (r.status !== 200 || !result || !Array.isArray(result.nodes)) {++ return done({ ok: false, stage: 'deliver', source: body.source, probes, error: r.status ? `Network answered ${r.status}${result && result.error ? `: ${result.error}` : ''}` : `Network unreachable (${r.error || 'no answer'})` });++ }++ const down = result.nodes.filter((n) => n.health && n.health.status === 'down').map((n) => n.id);++ return done({ ok: true, source: body.source, probes, registered: result.nodes.length, down });++}++++module.exports = { report, build, measure, writeMetrics };@@@ -16,7 +16,7 @@"dotenv": "^16.4.7","express": "^4.21.2","express-rate-limit": "^7.5.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.66.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"},@@@ -1412,9 +1412,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.66.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.66.0",-- "integrity": "sha512-iArup+k2iUI4ddMDVgVrf8xTCbEB5LaV0Da/1Bt5k3eH2TJpJ7yIEtBgBee0UZ6taNd2bZsZ6zJOkamqCAa4nA==",++ "version": "0.76.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",++ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -31,7 +31,7 @@"dotenv": "^16.4.7","express": "^4.21.2","express-rate-limit": "^7.5.0",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.66.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"},nnew file mode 100644@@@ -0,0 +1,108 @@++'use strict';++/**++ * The node registry reporter (lib/nodes.js, WS-X1 step 3): `ovhost nodes report` measures each inventory node's++ * beacon (2xx fast = up, 2xx slow = degraded, anything else = down), sends network.node-report-request@1 to++ * Network with Host's token (network.node.report), never shows a secret, and leaves openvibe_nodes_report.prom.++ * The inventory refuses addresses and duplicate ids: the registry is public.++ */++const assert = require('assert');++const contracts = require('openvibe-contracts');++const { scenario, test, runTests, SECRET } = require('./helpers');++const { normalise } = require('../lib/inventory');++++const TOKEN = 'tok_NODES_TOKEN_MUST_NEVER_APPEAR_9c2f';++const HOST_SECRET = `${SECRET}-host`;++const PROM_FILE = '/var/lib/prometheus/node-exporter/openvibe_nodes_report.prom';++const NODES = [++ { id: 'oregon-1', name: 'Oregon 1', roles: ['web', 'app', 'data', 'ingest'], location: { region: 'us-west', country: 'US', city: 'Hillsboro' }, provider: 'ovh', beacon: 'https://openvibe.network/api/v1/nodes/oregon-1/beacon' },++ { id: 'probe-fra', name: 'Frankfurt probe', roles: ['edge-probe'], location: { region: 'eu-central', country: 'DE' }, beacon: 'https://fra.example.test/beacon' },++ { id: 'probe-sgp', name: 'Singapore probe', roles: ['edge-probe'], location: { region: 'ap-southeast', country: 'SG' }, beacon: 'https://sgp.example.test/beacon' },++];++++function withNetwork({ network = 'ok' } = {}) {++ const host = scenario();++ host.inv.nodes = normalise({ services: {}, nodes: NODES }).nodes;++ host.inv.nodeSource = 'oregon';++ host.put('/etc/openvibe/host.json', JSON.stringify({ ...JSON.parse(host.read('/etc/openvibe/host.json')), nodeSource: 'oregon', nodes: NODES }), { mode: 0o640, owner: 'root' });++ host.put('/etc/openvibe/host.env', `OV_OAUTH_CLIENT_ID=host\nOV_OAUTH_CLIENT_SECRET="${HOST_SECRET}"\nEVENTS_URL=http://127.0.0.1:4300\nOV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000\n`, { mode: 0o600, owner: 'root' });++ host.put('/var/lib/prometheus/node-exporter/.keep', '');++ host.tokenRequests = [];++ host.delivered = [];++ host.http.set(NODES[0].beacon, () => ({ status: 204, body: '' }));++ host.http.set(NODES[1].beacon, () => { host.exec.sleep(2500); return { status: 204, body: '' }; });++ host.http.set(NODES[2].beacon, () => ({ status: 502, body: 'bad gateway' }));++ host.http.set('http://127.0.0.1:4000/oauth/token', ({ body }) => {++ const form = Object.fromEntries(new URLSearchParams(body));++ host.tokenRequests.push(form);++ if (form.client_secret !== HOST_SECRET) return { status: 401, body: { error: 'invalid_client' } };++ return { status: 200, body: { access_token: TOKEN, token_type: 'Bearer', expires_in: 300 } };++ });++ host.http.set('http://127.0.0.1:4000/internal/nodes/report', ({ method, headers, body }) => {++ assert.strictEqual(method, 'POST');++ assert.strictEqual(headers.Authorization, `Bearer ${TOKEN}`);++ const doc = JSON.parse(body);++ host.delivered.push(doc);++ if (network !== 'ok') return network;++ return { status: 200, body: { nodes: doc.nodes, generated_at: '2026-09-28T12:00:00Z' } };++ });++ return host;++}++++function noSecrets(host, out) {++ for (const s of [HOST_SECRET, TOKEN]) {++ assert.ok(!out.includes(s), 'the output shows no secret');++ for (const [p, f] of host.files) if (p !== '/etc/openvibe/host.env' && f.type === 'file') assert.ok(!String(f.content).includes(s), `${p} holds no secret`);++ }++}++++runTests([++ test('inventory: nodes need ids and https beacons, never an address, no duplicates', async () => {++ assert.deepStrictEqual(normalise({ services: {} }).nodes, []);++ assert.strictEqual(normalise({ services: {} }).nodeSource, 'primary');++ assert.throws(() => normalise({ services: {}, nodes: [{ ...NODES[0], address: '15.0.0.1' }] }), /must not carry an address/);++ assert.throws(() => normalise({ services: {}, nodes: [NODES[0], NODES[0]] }), /listed twice/);++ assert.throws(() => normalise({ services: {}, nodes: [{ ...NODES[0], beacon: 'http://x.test/b' }] }), /https:\/\//);++ assert.throws(() => normalise({ services: {}, nodes: {} }), /must be an array/);++ }),++ test('nodes report: beacons measured, the complete set sent with network.node.report, metrics written', async () => {++ const host = withNetwork();++ const r = await host.cli('nodes', 'report');++ assert.strictEqual(r.code, 0, r.out);++ assert.match(r.out, /NODES reported: 3 node\(s\) of oregon; the registry lists 3, down: probe-sgp/);++ assert.match(r.out, /up +oregon-1 +\d+ ms/);++ assert.match(r.out, /degraded +probe-fra +2500 ms/);++ assert.match(r.out, /down +probe-sgp +beacon answered 502/);++ assert.deepStrictEqual(host.tokenRequests.map((t) => [t.grant_type, t.client_id, t.audience, t.scope]), [['client_credentials', 'host', 'openvibe.network', 'network.node.report']]);++ const doc = host.delivered[0];++ assert.ok(contracts.validate('network.node-report-request@1', doc).valid);++ assert.deepStrictEqual(doc.nodes.map((n) => [n.id, n.health.status]), [['oregon-1', 'up'], ['probe-fra', 'degraded'], ['probe-sgp', 'down']]);++ const prom = host.read(PROM_FILE);++ assert.match(prom, /openvibe_nodes_report_last_run_ok 1/);++ assert.match(prom, /openvibe_nodes\{status="down"\} 1/);++ assert.match(prom, /openvibe_nodes_report_last_success_timestamp_seconds \d+/);++ noSecrets(host, r.out);++ }),++ test('dry run sends nothing; a refused report fails loudly and keeps the last success time', async () => {++ const host = withNetwork({ network: { status: 403, body: { error: 'insufficient_scope' } } });++ const dry = await host.cli('nodes', 'report', '--dry-run');++ assert.strictEqual(dry.code, 0, dry.out);++ assert.match(dry.out, /3 node\(s\) of oregon \(dry run, nothing sent\)/);++ assert.strictEqual(host.delivered.length, 0);++ host.put(PROM_FILE, 'openvibe_nodes_report_last_success_timestamp_seconds 1790000000\n');++ const r = await host.cli('nodes', 'report');++ assert.strictEqual(r.code, 2);++ assert.match(r.out, /NODES not reported \(deliver\): Network answered 403: insufficient_scope/);++ const prom = host.read(PROM_FILE);++ assert.match(prom, /openvibe_nodes_report_last_run_ok 0/);++ assert.match(prom, /openvibe_nodes_report_last_success_timestamp_seconds 1790000000/);++ noSecrets(host, r.out);++ }),++ test('an inventory node the contract refuses stops the run before any token is asked for', async () => {++ const host = withNetwork();++ host.put('/etc/openvibe/host.json', JSON.stringify({ ...JSON.parse(host.read('/etc/openvibe/host.json')), nodes: [{ ...NODES[0], roles: ['gameserver'] }] }), { mode: 0o640, owner: 'root' });++ const r = await host.cli('nodes', 'report');++ assert.strictEqual(r.code, 2);++ assert.match(r.out, /NODES not reported \(inventory\): .*roles/);++ assert.strictEqual(host.tokenRequests.length, 0);++ }),++]);
@@ -101,6 +101,19 @@ groups:
@@ -101,6 +101,19 @@ groups:
labels: { severity: page }
labels: { severity: page }
annotations:
annotations:
summary: "Operator alerts were not delivered for more than 15 minutes (ovhost alerts relay: Prometheus, the token or Network)"
summary: "Operator alerts were not delivered for more than 15 minutes (ovhost alerts relay: Prometheus, the token or Network)"
+ # The node registry (openvibe-nodes.timer; WS-X1): products find the nearest node from it.
+ - alert: OpenVibeNodeReportStale
+ expr: time() - openvibe_nodes_report_last_success_timestamp_seconds > 60 * 60
+ for: 10m
+ labels: { severity: ticket }
+ annotations:
+ summary: "The node registry has not heard from this host for more than an hour (ovhost nodes report)"
+ - alert: OpenVibeNodeDown
+ expr: openvibe_nodes{status="down"} > 0
+ for: 15m
+ labels: { severity: ticket }
+ annotations:
+ summary: "A node's beacon has not answered for 15 minutes (ovhost nodes report --dry-run names it)"
# The data role (roles/data, ADR-035): PostgreSQL, PgBouncer, Valkey and pgBackRest.
# The data role (roles/data, ADR-035): PostgreSQL, PgBouncer, Valkey and pgBackRest.
- name: openvibe-data
- name: openvibe-data
rules:
rules:
new file mode 100644
new file mode 100644
@@ -0,0 +1,19 @@
@@ -0,0 +1,19 @@
+[Unit]
+Description=OpenVibe node registry: this inventory's machines and their measured health, reported to OpenVibe.Network
+Documentation=https://github.com/OpenVibers/OpenVibe.Host/blob/main/docs/nodes.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+# Root: /etc/openvibe/host.env (Host's service principal) is root-only and read by ovhost itself, and the
+# textfile collector's directory is root's.
+User=root
+ExecStart=/usr/local/bin/ovhost nodes report
+Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+Environment=OVHOST_INVENTORY=/etc/openvibe/host.json
+UMask=0077
+TimeoutStartSec=90s
+PrivateTmp=true
+NoNewPrivileges=true
+ProtectSystem=full
new file mode 100644
new file mode 100644
@@ -0,0 +1,10 @@
@@ -0,0 +1,10 @@
+[Unit]
+Description=Report this inventory's machines to the OpenVibe node registry every five minutes (WS-X1)
+
+[Timer]
+OnBootSec=3min
+OnUnitActiveSec=5min
+AccuracySec=30s
+
+[Install]
+WantedBy=timers.target
new file mode 100644
new file mode 100644
@@ -0,0 +1,49 @@
@@ -0,0 +1,49 @@
+# The node registry reporter
+
+`ovhost nodes report` tells OpenVibe.Network which machines this inventory runs, and how healthy each one is right now.
+
+- It is roadmap WS-X1 step 3 and ADR-034 section 12.
+- `openvibe-nodes.timer` runs it every five minutes.
+- Products read the result at `GET https://openvibe.network/api/v1/nodes`, which returns `network.node-list-result@1`. It can be filtered with `?role=` and `?region=`. `openvibe-sdk/geo` uses it to pick the nearest node for a role, for example a game-server list or a ping tool with several locations.
+
+## The inventory
+
+`/etc/openvibe/host.json` holds two keys for this:
+
+- **`nodeSource`** names this inventory. The default is `primary`.
+- **`nodes`** lists each machine's public description, which is `network.node@1` without `health` and `updated_at`.
+
+```json
+"nodeSource": "oregon",
+"nodes": [
+ { "id": "oregon-1", "name": "Oregon 1", "roles": ["web", "app", "data", "ingest"],
+ "location": { "region": "us-west", "country": "US" }, "provider": "ovh",
+ "beacon": "https://openvibe.network/api/v1/nodes/oregon-1/beacon" }
+]
+```
+
+The registry is public, so the inventory refuses any node that carries an `address` or `ip`. Location stays at region level.
+
+## Health
+
+Health is measured, never assumed. Each node's beacon is fetched on every run:
+
+| Beacon result | Health |
+|---|---|
+| a 2xx within 2 s | `up` |
+| a slower 2xx | `degraded` |
+| anything else | `down` |
+
+A node removed from the inventory is marked `down` by Network. It is not deleted.
+
+## Running it
+
+```
+sudo ovhost nodes report --dry-run # measure and print; nothing is sent
+sudo cp /usr/local/lib/openvibe-host/deploy/systemd/openvibe-nodes.{service,timer} /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now openvibe-nodes.timer
+```
+
+- **Credentials:** it uses Host's principal from `/etc/openvibe/host.env`, the same one `ovhost alerts relay` uses, with the scope `network.node.report`. Neither the secret nor the token is ever printed.
+- **Metrics:** each run writes `openvibe_nodes_report.prom` to the textfile collector.
+- **Alerts:** `OpenVibeNodeReportStale` fires when no report has reached Network for an hour. `OpenVibeNodeDown` fires when a beacon has failed for 15 minutes.
@@ -93,6 +93,9 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
@@ -93,6 +93,9 @@ Usage: ovhost <command> [args] [--inventory <file>] [--json]
alerts relay [--prometheus <url>] [--dry-run]
alerts relay [--prometheus <url>] [--dry-run]
the alerts firing in Prometheus, sent to Network, which pages
the alerts firing in Prometheus, sent to Network, which pages
the operator (openvibe-alerts.timer, every 2 minutes)
the operator (openvibe-alerts.timer, every 2 minutes)
+ nodes report [--dry-run] this inventory's machines (host.json "nodes"), each beacon
+ measured, sent to Network's node registry
+ (openvibe-nodes.timer, every 5 minutes)
restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
restore-download <service> <run|latest> [--out <dir>] [--from-host <name>]
download, verify and decrypt one service's databases into a
download, verify and decrypt one service's databases into a
NEW root-only directory; never touches a live database
NEW root-only directory; never touches a live database
@@ -552,6 +555,18 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
@@ -552,6 +555,18 @@ async function main(argv, { exec = lib.createSystemExecutor(), out = (s) => proc
else out(`ALERTS delivered: ${r.firing} firing; opened ${r.result.opened}, reminded ${r.result.reminded}, resolved ${r.result.resolved}; ${r.result.notified} notification(s)`);
else out(`ALERTS delivered: ${r.firing} firing; opened ${r.result.opened}, reminded ${r.result.reminded}, resolved ${r.result.resolved}; ${r.result.notified} notification(s)`);
return r.ok ? 0 : 2;
return r.ok ? 0 : 2;
}
}
+ case 'nodes': {
+ // WS-X1 step 3: this inventory's machines, with measured health, to Network's node registry.
+ if (pos[1] !== 'report') throw new UsageError('usage: ovhost nodes report [--dry-run] [--events-env <file>]');
+ const r = await lib.nodes.report(ctx, { envFile: opts['events-env'], env, dryRun: !!opts['dry-run'] });
+ if (json) emit(r);
+ else if (!r.ok) out(`NODES not reported (${r.stage}): ${r.error}`);
+ else {
+ out(r.dryRun ? `${r.probes.length} node(s) of ${r.source} (dry run, nothing sent)` : `NODES reported: ${r.probes.length} node(s) of ${r.source}; the registry lists ${r.registered}${r.down.length ? `, down: ${r.down.join(', ')}` : ''}`);
+ for (const p of r.probes) out(` ${p.status.padEnd(9)} ${p.id} ${p.detail}`);
+ }
+ return r.ok ? 0 : 2;
+ }
case 'archive': {
case 'archive': {
// WS-S task 6: dead and backup database files, encrypted off-host before the owner deletes them.
// WS-S task 6: dead and backup database files, encrypted off-host before the owner deletes them.
const sub = pos[1];
const sub = pos[1];
@@ -36,6 +36,7 @@ module.exports = {
@@ -36,6 +36,7 @@ module.exports = {
offsite,
offsite,
archive: require('./archive'),
archive: require('./archive'),
alerts: require('./alerts'),
alerts: require('./alerts'),
+ nodes: require('./nodes'),
browserWatch: require('./browser-watch'),
browserWatch: require('./browser-watch'),
incidents: require('./incidents'),
incidents: require('./incidents'),
dns: require('./dns'),
dns: require('./dns'),
@@ -456,6 +456,23 @@ function normaliseDrill(raw, svc, where) {
@@ -456,6 +456,23 @@ function normaliseDrill(raw, svc, where) {
};
};
}
}
+/** The public description of each machine (network.node@1 without health; lib/nodes.js validates the rest). */
+function normaliseNodes(raw) {
+ if (raw == null) return [];
+ if (!Array.isArray(raw)) throw new InventoryError('nodes must be an array');
+ const seen = new Set();
+ return raw.map((n, i) => {
+ if (!n || typeof n !== 'object' || !ID_RE.test(n.id || '')) throw new InventoryError(`nodes[${i}].id must match ${ID_RE}`);
+ if (seen.has(n.id)) throw new InventoryError(`nodes[${i}].id "${n.id}" is listed twice`);
+ seen.add(n.id);
+ if ('address' in n || 'ip' in n) throw new InventoryError(`nodes[${i}] must not carry an address: the registry is public`);
+ let u;
+ try { u = new URL(String(n.beacon)); } catch { u = null; }
+ if (!u || u.protocol !== 'https:') throw new InventoryError(`nodes[${i}].beacon must be an https:// URL`);
+ return { ...n };
+ });
+}
+
function normalise(raw) {
function normalise(raw) {
if (!raw || typeof raw !== 'object' || !raw.services || typeof raw.services !== 'object') throw new InventoryError('inventory needs a "services" object');
if (!raw || typeof raw !== 'object' || !raw.services || typeof raw.services !== 'object') throw new InventoryError('inventory needs a "services" object');
const defaults = raw.defaults || {};
const defaults = raw.defaults || {};
@@ -476,6 +493,9 @@ function normalise(raw) {
@@ -476,6 +493,9 @@ function normalise(raw) {
events: { envFile: raw.events && raw.events.envFile ? abs(raw.events.envFile, 'events.envFile') : null },
events: { envFile: raw.events && raw.events.envFile ? abs(raw.events.envFile, 'events.envFile') : null },
// The DNS adapter (lib/dns.js): names never proxied by Cloudflare (non-HTTP traffic, the CNAME target).
// The DNS adapter (lib/dns.js): names never proxied by Cloudflare (non-HTTP traffic, the CNAME target).
dns: { dnsOnly: Array.isArray(raw.dns && raw.dns.dnsOnly) ? raw.dns.dnsOnly.map(String) : require('./dns').DEFAULT_DNS_ONLY },
dns: { dnsOnly: Array.isArray(raw.dns && raw.dns.dnsOnly) ? raw.dns.dnsOnly.map(String) : require('./dns').DEFAULT_DNS_ONLY },
+ // The node registry (lib/nodes.js, ADR-034 §12): this inventory's machines, reported to Network.
+ nodeSource: raw.nodeSource || 'primary',
+ nodes: normaliseNodes(raw.nodes),
services: {},
services: {},
};
};
for (const [id, svc] of Object.entries(raw.services)) {
for (const [id, svc] of Object.entries(raw.services)) {
new file mode 100644
new file mode 100644
@@ -0,0 +1,97 @@
@@ -0,0 +1,97 @@
+'use strict';
+/**
+ * `ovhost nodes report` (roadmap WS-X1 step 3, ADR-034 §12): the machines this inventory knows, with their health
+ * now, sent to Network's node registry (POST /internal/nodes/report, network.node.report, Host's principal).
+ * openvibe-nodes.timer runs it every five minutes; products read the result at GET /api/v1/nodes.
+ *
+ * The inventory's "nodes" array holds each machine's public description (network.node@1 without health and
+ * updated_at: id, name, roles, region-level location, provider, beacon); "nodeSource" names this inventory (default
+ * "primary"). Health is measured, never assumed: each node's beacon is fetched; a 2xx within 2 s is up, a slower 2xx
+ * degraded, anything else down. A node removed from the inventory is marked down by Network, not deleted.
+ * Every run leaves openvibe_nodes_report.prom for the textfile collector (OpenVibeNodeReportStale).
+ */
+const path = require('path');
+const { loadCredentials } = require('./announce');
+
+const TEXTFILE_DIR = '/var/lib/prometheus/node-exporter';
+const SOURCE_RE = /^[a-z][a-z0-9-]{1,39}$/;
+const SLOW_MS = 2000;
+
+async function measure(exec, beacon) {
+ const t0 = exec.now();
+ const r = await exec.request(beacon, { method: 'GET', headers: { 'user-agent': 'openvibe-host-nodes' }, timeoutMs: 5000 });
+ const ms = exec.now() - t0;
+ if (!(r.status >= 200 && r.status < 300)) return { status: 'down', detail: r.status ? `beacon answered ${r.status}` : `beacon unreachable (${r.error || 'no answer'})` };
+ return { status: ms > SLOW_MS ? 'degraded' : 'up', detail: `${ms} ms` };
+}
+
+/** The report body (network.node-report-request@1) from the inventory, with measured health. */
+async function build(exec, inv) {
+ const source = inv.nodeSource || 'primary';
+ if (!SOURCE_RE.test(source)) throw new Error(`nodeSource "${source}" must match ${SOURCE_RE}`);
+ const at = new Date(exec.now()).toISOString();
+ const nodes = [];
+ const probes = [];
+ for (const n of inv.nodes || []) {
+ const m = await measure(exec, n.beacon);
+ probes.push({ id: n.id, ...m });
+ nodes.push({ ...n, health: { status: m.status, checked_at: at }, updated_at: at });
+ }
+ const body = { source, nodes };
+ const v = require('openvibe-contracts').validate('network.node-report-request@1', body);
+ if (!v.valid) throw new Error(`the inventory's nodes do not match network.node-report-request@1: ${(v.errors || []).slice(0, 3).map((e) => `${e.path} ${e.message}`).join('; ')}`);
+ return { body, probes };
+}
+
+async function writeMetrics(exec, dir, r) {
+ const now = Math.floor(exec.now() / 1000);
+ let lastOk = r.ok ? now : null;
+ if (!r.ok) {
+ const prev = (await exec.readFile(path.join(dir, 'openvibe_nodes_report.prom'))) || '';
+ const m = prev.match(/^openvibe_nodes_report_last_success_timestamp_seconds (\d+)$/m);
+ if (m) lastOk = Number(m[1]);
+ }
+ const count = (s) => (r.probes || []).filter((p) => p.status === s).length;
+ const lines = [
+ '# HELP openvibe_nodes_report_last_run_ok 1 when the last ovhost nodes report reached Network.',
+ '# TYPE openvibe_nodes_report_last_run_ok gauge',
+ `openvibe_nodes_report_last_run_ok ${r.ok ? 1 : 0}`,
+ '# HELP openvibe_nodes Nodes of this inventory by the health the last run measured.',
+ '# TYPE openvibe_nodes gauge',
+ ...['up', 'degraded', 'down'].map((s) => `openvibe_nodes{status="${s}"} ${count(s)}`),
+ ];
+ if (lastOk) lines.push('# HELP openvibe_nodes_report_last_success_timestamp_seconds When a report last reached Network.', '# TYPE openvibe_nodes_report_last_success_timestamp_seconds gauge', `openvibe_nodes_report_last_success_timestamp_seconds ${lastOk}`);
+ await exec.writeFile(path.join(dir, 'openvibe_nodes_report.prom'), `${lines.join('\n')}\n`, { mode: 0o644 });
+}
+
+/** One run → { ok, source, probes, stage?, error?, dryRun? }. opts: envFile, env, dryRun, metrics (default true). */
+async function report(ctx, { envFile = null, env = {}, dryRun = false, metrics = true } = {}) {
+ const { exec, inv } = ctx;
+ const done = async (r) => {
+ if (metrics && !dryRun) { try { await writeMetrics(exec, inv.textfileDir || TEXTFILE_DIR, r); } catch { /* metrics never fail a run */ } }
+ return r;
+ };
+ let built;
+ try { built = await build(exec, inv); } catch (err) { return done({ ok: false, stage: 'inventory', error: err.message }); }
+ const { body, probes } = built;
+ if (dryRun) return { ok: true, dryRun: true, source: body.source, probes, body };
+ const creds = await loadCredentials(exec, inv, { file: envFile, env });
+ if (creds.error) return done({ ok: false, stage: 'credentials', source: body.source, probes, error: creds.error });
+ const form = new URLSearchParams({ grant_type: 'client_credentials', client_id: creds.clientId, client_secret: creds.secret, audience: 'openvibe.network', scope: 'network.node.report' });
+ const t = await exec.request(`${creds.networkUrl}/oauth/token`, { method: 'POST', headers: { 'Content-Type': 'application/x-www-form-urlencoded', Accept: 'application/json' }, body: form.toString(), timeoutMs: 5000 });
+ let token = null;
+ try { token = t.status === 200 ? JSON.parse(t.body).access_token : null; } catch { token = null; }
+ if (typeof token !== 'string' || !token) return done({ ok: false, stage: 'token', source: body.source, probes, error: t.status ? `the token endpoint answered ${t.status}` : `Network unreachable (${t.error || 'no answer'})` });
+ const r = await exec.request(`${creds.networkUrl}/internal/nodes/report`, {
+ method: 'POST', headers: { 'Content-Type': 'application/json', Accept: 'application/json', Authorization: `Bearer ${token}` }, body: JSON.stringify(body), timeoutMs: 10000,
+ });
+ let result = null;
+ try { result = JSON.parse(r.body); } catch { result = null; }
+ if (r.status !== 200 || !result || !Array.isArray(result.nodes)) {
+ return done({ ok: false, stage: 'deliver', source: body.source, probes, error: r.status ? `Network answered ${r.status}${result && result.error ? `: ${result.error}` : ''}` : `Network unreachable (${r.error || 'no answer'})` });
+ }
+ const down = result.nodes.filter((n) => n.health && n.health.status === 'down').map((n) => n.id);
+ return done({ ok: true, source: body.source, probes, registered: result.nodes.length, down });
+}
+
+module.exports = { report, build, measure, writeMetrics };
@@ -16,7 +16,7 @@
@@ -16,7 +16,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"express-rate-limit": "^7.5.0",
"express-rate-limit": "^7.5.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.66.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
},
},
@@ -1412,9 +1412,9 @@
@@ -1412,9 +1412,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.66.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.66.0",
- "integrity": "sha512-iArup+k2iUI4ddMDVgVrf8xTCbEB5LaV0Da/1Bt5k3eH2TJpJ7yIEtBgBee0UZ6taNd2bZsZ6zJOkamqCAa4nA==",
+ "version": "0.76.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
+ "integrity": "sha512-zTdx776KWrcJKm1JTnf+0+BnsvswdvXy1ldPyse1hSqhnW/SSr98sX696eRetFUhTl8qVM1xSRCykqIUbqItPw==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -31,7 +31,7 @@
@@ -31,7 +31,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"express-rate-limit": "^7.5.0",
"express-rate-limit": "^7.5.0",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.66.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.76.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.12.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.25.0"
},
},
new file mode 100644
new file mode 100644
@@ -0,0 +1,108 @@
@@ -0,0 +1,108 @@
+'use strict';
+/**
+ * The node registry reporter (lib/nodes.js, WS-X1 step 3): `ovhost nodes report` measures each inventory node's
+ * beacon (2xx fast = up, 2xx slow = degraded, anything else = down), sends network.node-report-request@1 to
+ * Network with Host's token (network.node.report), never shows a secret, and leaves openvibe_nodes_report.prom.
+ * The inventory refuses addresses and duplicate ids: the registry is public.
+ */
+const assert = require('assert');
+const contracts = require('openvibe-contracts');
+const { scenario, test, runTests, SECRET } = require('./helpers');
+const { normalise } = require('../lib/inventory');
+
+const TOKEN = 'tok_NODES_TOKEN_MUST_NEVER_APPEAR_9c2f';
+const HOST_SECRET = `${SECRET}-host`;
+const PROM_FILE = '/var/lib/prometheus/node-exporter/openvibe_nodes_report.prom';
+const NODES = [
+ { id: 'oregon-1', name: 'Oregon 1', roles: ['web', 'app', 'data', 'ingest'], location: { region: 'us-west', country: 'US', city: 'Hillsboro' }, provider: 'ovh', beacon: 'https://openvibe.network/api/v1/nodes/oregon-1/beacon' },
+ { id: 'probe-fra', name: 'Frankfurt probe', roles: ['edge-probe'], location: { region: 'eu-central', country: 'DE' }, beacon: 'https://fra.example.test/beacon' },
+ { id: 'probe-sgp', name: 'Singapore probe', roles: ['edge-probe'], location: { region: 'ap-southeast', country: 'SG' }, beacon: 'https://sgp.example.test/beacon' },
+];
+
+function withNetwork({ network = 'ok' } = {}) {
+ const host = scenario();
+ host.inv.nodes = normalise({ services: {}, nodes: NODES }).nodes;
+ host.inv.nodeSource = 'oregon';
+ host.put('/etc/openvibe/host.json', JSON.stringify({ ...JSON.parse(host.read('/etc/openvibe/host.json')), nodeSource: 'oregon', nodes: NODES }), { mode: 0o640, owner: 'root' });
+ host.put('/etc/openvibe/host.env', `OV_OAUTH_CLIENT_ID=host\nOV_OAUTH_CLIENT_SECRET="${HOST_SECRET}"\nEVENTS_URL=http://127.0.0.1:4300\nOV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000\n`, { mode: 0o600, owner: 'root' });
+ host.put('/var/lib/prometheus/node-exporter/.keep', '');
+ host.tokenRequests = [];
+ host.delivered = [];
+ host.http.set(NODES[0].beacon, () => ({ status: 204, body: '' }));
+ host.http.set(NODES[1].beacon, () => { host.exec.sleep(2500); return { status: 204, body: '' }; });
+ host.http.set(NODES[2].beacon, () => ({ status: 502, body: 'bad gateway' }));
+ host.http.set('http://127.0.0.1:4000/oauth/token', ({ body }) => {
+ const form = Object.fromEntries(new URLSearchParams(body));
+ host.tokenRequests.push(form);
+ if (form.client_secret !== HOST_SECRET) return { status: 401, body: { error: 'invalid_client' } };
+ return { status: 200, body: { access_token: TOKEN, token_type: 'Bearer', expires_in: 300 } };
+ });
+ host.http.set('http://127.0.0.1:4000/internal/nodes/report', ({ method, headers, body }) => {
+ assert.strictEqual(method, 'POST');
+ assert.strictEqual(headers.Authorization, `Bearer ${TOKEN}`);
+ const doc = JSON.parse(body);
+ host.delivered.push(doc);
+ if (network !== 'ok') return network;
+ return { status: 200, body: { nodes: doc.nodes, generated_at: '2026-09-28T12:00:00Z' } };
+ });
+ return host;
+}
+
+function noSecrets(host, out) {
+ for (const s of [HOST_SECRET, TOKEN]) {
+ assert.ok(!out.includes(s), 'the output shows no secret');
+ for (const [p, f] of host.files) if (p !== '/etc/openvibe/host.env' && f.type === 'file') assert.ok(!String(f.content).includes(s), `${p} holds no secret`);
+ }
+}
+
+runTests([
+ test('inventory: nodes need ids and https beacons, never an address, no duplicates', async () => {
+ assert.deepStrictEqual(normalise({ services: {} }).nodes, []);
+ assert.strictEqual(normalise({ services: {} }).nodeSource, 'primary');
+ assert.throws(() => normalise({ services: {}, nodes: [{ ...NODES[0], address: '15.0.0.1' }] }), /must not carry an address/);
+ assert.throws(() => normalise({ services: {}, nodes: [NODES[0], NODES[0]] }), /listed twice/);
+ assert.throws(() => normalise({ services: {}, nodes: [{ ...NODES[0], beacon: 'http://x.test/b' }] }), /https:\/\//);
+ assert.throws(() => normalise({ services: {}, nodes: {} }), /must be an array/);
+ }),
+ test('nodes report: beacons measured, the complete set sent with network.node.report, metrics written', async () => {
+ const host = withNetwork();
+ const r = await host.cli('nodes', 'report');
+ assert.strictEqual(r.code, 0, r.out);
+ assert.match(r.out, /NODES reported: 3 node\(s\) of oregon; the registry lists 3, down: probe-sgp/);
+ assert.match(r.out, /up +oregon-1 +\d+ ms/);
+ assert.match(r.out, /degraded +probe-fra +2500 ms/);
+ assert.match(r.out, /down +probe-sgp +beacon answered 502/);
+ assert.deepStrictEqual(host.tokenRequests.map((t) => [t.grant_type, t.client_id, t.audience, t.scope]), [['client_credentials', 'host', 'openvibe.network', 'network.node.report']]);
+ const doc = host.delivered[0];
+ assert.ok(contracts.validate('network.node-report-request@1', doc).valid);
+ assert.deepStrictEqual(doc.nodes.map((n) => [n.id, n.health.status]), [['oregon-1', 'up'], ['probe-fra', 'degraded'], ['probe-sgp', 'down']]);
+ const prom = host.read(PROM_FILE);
+ assert.match(prom, /openvibe_nodes_report_last_run_ok 1/);
+ assert.match(prom, /openvibe_nodes\{status="down"\} 1/);
+ assert.match(prom, /openvibe_nodes_report_last_success_timestamp_seconds \d+/);
+ noSecrets(host, r.out);
+ }),
+ test('dry run sends nothing; a refused report fails loudly and keeps the last success time', async () => {
+ const host = withNetwork({ network: { status: 403, body: { error: 'insufficient_scope' } } });
+ const dry = await host.cli('nodes', 'report', '--dry-run');
+ assert.strictEqual(dry.code, 0, dry.out);
+ assert.match(dry.out, /3 node\(s\) of oregon \(dry run, nothing sent\)/);
+ assert.strictEqual(host.delivered.length, 0);
+ host.put(PROM_FILE, 'openvibe_nodes_report_last_success_timestamp_seconds 1790000000\n');
+ const r = await host.cli('nodes', 'report');
+ assert.strictEqual(r.code, 2);
+ assert.match(r.out, /NODES not reported \(deliver\): Network answered 403: insufficient_scope/);
+ const prom = host.read(PROM_FILE);
+ assert.match(prom, /openvibe_nodes_report_last_run_ok 0/);
+ assert.match(prom, /openvibe_nodes_report_last_success_timestamp_seconds 1790000000/);
+ noSecrets(host, r.out);
+ }),
+ test('an inventory node the contract refuses stops the run before any token is asked for', async () => {
+ const host = withNetwork();
+ host.put('/etc/openvibe/host.json', JSON.stringify({ ...JSON.parse(host.read('/etc/openvibe/host.json')), nodes: [{ ...NODES[0], roles: ['gameserver'] }] }), { mode: 0o640, owner: 'root' });
+ const r = await host.cli('nodes', 'report');
+ assert.strictEqual(r.code, 2);
+ assert.match(r.out, /NODES not reported \(inventory\): .*roles/);
+ assert.strictEqual(host.tokenRequests.length, 0);
+ }),
+]);